CN113935914A - Depth image restoration method, electronic device and storage medium - Google Patents
Depth image restoration method, electronic device and storage medium Download PDFInfo
- Publication number
- CN113935914A CN113935914A CN202111171194.8A CN202111171194A CN113935914A CN 113935914 A CN113935914 A CN 113935914A CN 202111171194 A CN202111171194 A CN 202111171194A CN 113935914 A CN113935914 A CN 113935914A
- Authority
- CN
- China
- Prior art keywords
- pixel points
- depth image
- filling
- hole
- priority
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 47
- 239000011800 void material Substances 0.000 claims description 54
- 230000000694 effects Effects 0.000 claims description 18
- 238000001914 filtration Methods 0.000 claims description 10
- 238000004590 computer program Methods 0.000 claims description 5
- 230000008439 repair process Effects 0.000 claims description 5
- 230000002146 bilateral effect Effects 0.000 claims description 4
- 239000000126 substance Substances 0.000 claims description 4
- 238000009499 grossing Methods 0.000 claims description 2
- 230000006872 improvement Effects 0.000 description 8
- 230000008569 process Effects 0.000 description 7
- 230000000877 morphologic effect Effects 0.000 description 6
- 230000006870 function Effects 0.000 description 4
- 230000007797 corrosion Effects 0.000 description 2
- 238000005260 corrosion Methods 0.000 description 2
- 238000001514 detection method Methods 0.000 description 2
- 230000003628 erosive effect Effects 0.000 description 2
- 230000004927 fusion Effects 0.000 description 2
- 230000002093 peripheral effect Effects 0.000 description 2
- 230000009467 reduction Effects 0.000 description 2
- 230000005540 biological transmission Effects 0.000 description 1
- 238000004422 calculation algorithm Methods 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 230000010339 dilation Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 230000009191 jumping Effects 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000003672 processing method Methods 0.000 description 1
Images
Classifications
-
- G06T5/77—
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10028—Range image; Depth image; 3D point clouds
Abstract
The embodiment of the invention relates to the field of image processing, and discloses a depth image restoration method, electronic equipment and a storage medium, wherein the method comprises the following steps: acquiring a depth image containing a hole pixel point; performing the following preferential filling operation on unfilled hole pixel points in the depth image: aiming at unfilled hole pixel points, determining a neighborhood by taking each hole pixel point as a center, and determining the filling priority of each hole pixel point according to the number and the pixel value of non-hole pixel points contained in the neighborhood; filling the specified number of hole pixel points with the highest filling priority; judging whether the depth image contains unfilled hole pixel points; if yes, continuing to perform the prior filling operation on the unfilled hole pixel points; and if not, taking the depth image obtained after the priority filling operation as the repaired depth image. According to the scheme, the filling priority is set for the cavity points in the depth image, so that the repairing precision and robustness are improved.
Description
Technical Field
The present invention relates to the field of image processing, and in particular, to a depth image restoration method, an electronic device, and a storage medium.
Background
The depth image has wide application in the fields of three-dimensional scene reconstruction, man-machine interaction, navigation, positioning and the like. Currently, there are 3 main depth image acquisition modes: 1. the laser radar technology, namely, transmitting signals to a target within a certain time interval and calculating depth information according to the distance covered; 2. the stereoscopic vision technology is mainly used for acquiring depth information by fixing a plurality of cameras and calculating parallax; 3. the main principle of the structured light technology is to generate recognizable stripe images by projecting a projection source into a scene space, and then an image sensor reads the stripes in the space and compares the stripes with reference stripes so as to calculate the depth information in the scene.
With the continuous development of science and technology, depth camera equipment tends to be more civilized. However, in the acquisition process, due to the reason of the device itself and the influence of the surrounding environment, a large number of holes often exist in the acquired depth information, and if the holes are directly applied without being processed, the accuracy of subsequent work is influenced, and even failure is caused.
Disclosure of Invention
An object of embodiments of the present invention is to provide a depth image restoration method, an electronic device, and a storage medium, which improve the accuracy and robustness of restoration by setting a filling priority for a hole point in a depth image.
In order to solve the above technical problem, an embodiment of the present invention provides a depth image restoration method, including:
acquiring a depth image containing a hole pixel point;
performing the following preferential filling operation on unfilled hole pixel points in the depth image:
aiming at the unfilled hole pixel points, respectively determining a neighborhood by taking each hole pixel point as a center, and determining the filling priority of each hole pixel point according to the number and the pixel value of non-hole pixel points contained in the neighborhood; the higher the filling priority is, the better the image restoration effect after the corresponding hole pixel point is filled is;
filling the specified number of hole pixel points with the highest filling priority;
judging whether the depth image contains unfilled hole pixel points;
if yes, continuing to execute the preferential filling operation on the unfilled hole pixel points; and if not, taking the depth image obtained after the priority filling operation as the repaired depth image.
An embodiment of the present invention also provides an electronic device, including:
at least one processor; and the number of the first and second groups,
a memory communicatively coupled to the at least one processor; wherein the content of the first and second substances,
the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the depth image restoration method as described above.
Embodiments of the present invention also provide a computer-readable storage medium storing a computer program which, when executed by a processor, implements the depth image restoration method as described above.
Compared with the prior art, the embodiment of the invention performs the following preferential filling operation on unfilled hole pixel points in the depth image after obtaining the depth image containing the hole pixel points: aiming at unfilled hole pixel points, determining a neighborhood by taking each hole pixel point as a center, and determining the filling priority of each hole pixel point according to the number and the pixel value of non-hole pixel points contained in the neighborhood; the higher the filling priority is, the better the image restoration effect after the corresponding hole pixel point is filled is; filling the specified number of hole pixel points with the highest filling priority; after the first preferential filling operation is executed, judging whether the depth image contains unfilled hole pixel points; if yes, continuing to execute the prior filling operation on unfilled hole pixel points; and if not, taking the depth image obtained after the priority filling operation as the repaired depth image. According to the scheme, the filling priority is set for the cavity points in the depth image, so that the higher part of the cavity pixel points with the filling priority can be filled preferentially each time, the filling priority of the unfilled cavity pixel points is updated after filling at each time, and the filling is continued preferentially until all the cavity pixel points are filled.
Drawings
FIG. 1 is a first flowchart illustrating a depth image restoration method according to an embodiment of the present invention;
FIG. 2 is a detailed flowchart II of a depth image restoration method according to an embodiment of the present invention;
FIG. 3 is a flowchart III illustrating a depth image restoration method according to an embodiment of the present invention;
FIGS. 4 (a-e) are neighborhood graphs of void pixel points according to embodiments of the present invention;
FIG. 5 is a detailed flowchart IV of a depth image restoration method according to an embodiment of the present invention;
FIG. 6 is a detailed flowchart of a depth image restoration method according to an embodiment of the present invention;
fig. 7 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention more apparent, embodiments of the present invention will be described in detail below with reference to the accompanying drawings. However, it will be appreciated by those of ordinary skill in the art that numerous technical details are set forth in order to provide a better understanding of the present application in various embodiments of the present invention. However, the technical solution claimed in the present application can be implemented without these technical details and various changes and modifications based on the following embodiments.
An embodiment of the present invention relates to a depth image restoration method, and as shown in fig. 1, the depth image restoration method provided in this embodiment includes the following steps.
Step 101: and obtaining a depth image containing the hole pixel points.
Specifically, the hole pixel points in this embodiment may be pixel points corresponding to holes in the original depth image determined based on a disparity map or the like, or may be hole pixel points in the depth image obtained by performing certain processing (such as noise reduction) on the original depth image. In this embodiment, the source of the depth image including the void pixel is not limited, and the pixel value of the void pixel may be set to 0 in order to describe the void pixel in the depth image.
Aiming at the acquired depth image containing the hole pixel points, the following preferential filling operation is carried out on the hole pixel points which are not filled in the depth image (steps 102-103).
Step 102: aiming at unfilled hole pixel points, determining a neighborhood by taking each hole pixel point as a center, and determining the filling priority of each hole pixel point according to the number and the pixel value of non-hole pixel points contained in the neighborhood; the higher the filling priority is, the better the image restoration effect after the corresponding hole pixel point is filled.
Specifically, the depth values in the depth image have area continuity, and the depth values in the same area do not change much, so the filling priority of the hole pixel point is necessarily related to the pixel value of the pixel point in the local area. Therefore, for unfilled hole pixels, a neighborhood can be determined in the region corresponding to 8 adjacent pixels by using each hole pixel as a center. And determining the filling priority of the hollow pixel point at the central position according to the number and the pixel value of the non-hollow pixel points (pixel points with the pixel value not being 0) contained in the neighborhood. The higher the filling priority is, the better the restoration effect of the depth image after the hole pixel point representing the central position is filled.
Wherein, to the understanding that the repair effect is good, after the hole pixel is filled, the degree of fusion of the filled hole pixel and other pixels in the neighborhood is higher, the overall pixel smoothness of the region is better, and the holes are less. And if the restoration effect is good, the image effect shown by the existing pixel points in the neighborhood before the hole pixel points are required to be restored, namely, the hole pixel points are filled is relatively good. The good effect is also expressed in that the fusion degree between the pixel points contained in the neighborhood is higher, the pixel smoothness of the whole area is better, and the holes are fewer. And the quality of the effects can be quantitatively evaluated through the number and the pixel value of the non-hole pixel points contained in the neighborhood. For example, the smaller the number of non-hole pixels, the smaller the overall variation range of the pixel values, the better the image effect is, and vice versa.
Based on this, in this embodiment, according to the number and the pixel value of the non-void pixel points included in the neighborhood, the quality of the image restoration effect after the void pixel points are filled is quantified, so that the filling priority of each void pixel point is determined. The higher the filling priority is, the better the image restoration effect after the corresponding hole pixel point is filled.
Step 103: and filling the specified number of hole pixel points with the highest filling priority.
Specifically, after the filling priority of each void pixel point is determined, filling can be performed on the specified number of void pixel points with the highest filling priority according to the principle that filling is preferentially performed when the filling priority is higher. When filling is performed for each hole pixel point, the filled pixel value may be a pixel average value of all non-hole pixel points in a neighborhood corresponding to the hole pixel point.
In an example, when filling is performed on a specified number of hole pixel points with the highest filling priority, filling may be performed on one hole pixel point with the highest filling priority among the hole pixel points.
Step 104: judging whether the depth image contains unfilled hole pixel points; if yes, jumping back to step 102, and continuing to execute the priority filling operation; if not, step 105 is performed.
After the one-time preferential filling operation is completed, unfilled void pixel points in the depth image are reduced, at this time, whether the depth image still contains unfilled void pixel points is judged, if yes, the step 102 needs to be skipped back, the preferential filling operation is continuously executed, and the process is circulated until the depth image does not contain unfilled void pixel points.
Step 105: and taking the depth image obtained after the preferential filling operation as the repaired depth image.
Specifically, when the depth image does not include unfilled hole pixel points, that is, no hole exists in the depth image, the depth image at this time can be used as a repaired depth image.
In addition, in order to further improve the repairing effect of the depth image, after step 104, if it is determined that the depth image does not include unfilled hole pixel points, the depth image obtained after the present preferential filling operation may be smoothed by using bilateral filtering, so as to obtain a final repaired depth image.
In particular, some frizz may appear in the depth image obtained after the filling of the cavity. Bilateral filtering is nonlinear filtering, and consists of two kernel functions, so that not only is the correlation in a spatial domain referred, but also the similarity of pixel values is referred. The bilateral filtering is adopted to carry out smoothing processing on the depth image, and the image edge information in the image information can be considered while filtering, so that the image edge is smoother.
Compared with the related art, in the embodiment, after the depth image containing the hole pixel points is obtained, the following preferential filling operation is performed on the hole pixel points which are not filled in the depth image: aiming at unfilled hole pixel points, determining a neighborhood by taking each hole pixel point as a center, and determining the filling priority of each hole pixel point according to the number and the pixel value of non-hole pixel points contained in the neighborhood; the higher the filling priority is, the better the image restoration effect after the corresponding hole pixel point is filled is; filling the specified number of hole pixel points with the highest filling priority; after the first preferential filling operation is executed, judging whether the depth image contains unfilled hole pixel points; if yes, continuing to execute the prior filling operation on unfilled hole pixel points; and if not, taking the depth image obtained after the priority filling operation as the repaired depth image. According to the scheme, the filling priority is set for the cavity points in the depth image, so that the higher part of the cavity pixel points with the filling priority can be filled preferentially each time, the filling priority of the unfilled cavity pixel points is updated after filling at each time, and the filling is continued preferentially until all the cavity pixel points are filled.
Another embodiment of the present invention relates to a depth image restoration method, as shown in fig. 2, which is an improvement of the method steps shown in fig. 1, and the improvement is to refine the process of obtaining a depth image containing hole pixel points. As shown in fig. 2, the above step 101 includes the following sub-steps.
Substep 1011: and obtaining a depth image containing the original hole pixel points.
Specifically, the original hole pixel points in this embodiment may be pixel points corresponding to holes in the original depth image determined based on a disparity map or the like, or may be hole pixel points in the depth image obtained by performing certain processing (such as noise reduction) on the original depth image. In this embodiment, the source of the depth image including the original hole pixel point is not limited, and the pixel value of the original hole pixel point may be 0 in order to describe the original hole pixel point in the depth image.
In one example, this step may be implemented as follows.
Step 1: an original depth image is acquired.
Specifically, the original depth image may be an original depth image determined based on a disparity map or the like.
Step 2: and dividing the original depth image into a plurality of regions according to the pixel value difference value of each pixel point and the adjacent pixel point in the original depth image.
Specifically, when the depth camera shoots a depth image, due to the reason of the depth camera and the influence of the surrounding environment, the collected depth information often contains a large amount of holes and noises, and if the holes and the noises are directly applied without being processed, the precision of subsequent work is directly influenced, and even failure is caused. Therefore, the depth image needs to be denoised before the hole repairing.
In this embodiment, a method of detecting a connected region is used to implement denoising processing. Firstly, setting a connected region detection threshold value; traversing each pixel point in the original depth image, and determining pixel difference values between all pixel points in a neighborhood formed by the current pixel point and 8 pixel points; and counting all pixel points with pixel difference values smaller than the detection threshold of the connected region in the same region until the original depth image is traversed and cut off. Finally, the original depth image is divided into several regions.
And step 3: and determining each pixel point in the region, in which the number of the included pixel points is less than the preset region pixel point threshold value, as an original hole pixel point.
Specifically, a region area size threshold is set, the number of pixels included in each region in the original depth image is compared with the region area size threshold, and the pixel values of the pixels in the regions smaller than the region area size threshold are assigned to 0, that is, the pixels are set as original hole pixels.
Substep 1012: and performing opening and closing operation on the depth image containing the original hole pixel points by adopting the circular structural elements to eliminate partial original hole pixel points to obtain the depth image containing the hole pixel points.
The morphological filtering is an image processing method which takes several basic operations in morphology as a core algorithm, and has better effect to a certain extent compared with the conventional linear filtering. The morphological filtering removes different types of noise signals contained in the target object using different structural elements. Most of the morphological processing is a binarized image and a grayscale image. The object processed in the binary operation is a set element, and the object processed in the grayscale image is an image function. The principle of processing depth images using morphology is similar to processing grayscale images.
The morphological filtering mainly includes dilation and erosion. The main effect of the expansion is to expand the edges of the target object, allowing an increased range of the object, so that isolated parts in contact with the object can be incorporated into the object itself. The erosion acts opposite to the expansion, which is mainly to shrink the edge of the target object inwards. The opening operation is to perform corrosion operation and then expansion operation, mainly remove single small points, burrs and small bridges in the image and smoothen the edge of the object. The closed operation is to perform expansion operation and then corrosion operation, mainly to fill the tiny holes existing in the target object, and to repair tiny cracks and smooth the edge of the object.
Specifically, in this embodiment, the round structural element is adopted to perform morphological processing of opening and closing operations on the denoised original depth image, so as to eliminate isolated and burred points in the depth image and repair a fine cavity region.
Compared with the related art, the depth image containing the original hole pixel points is obtained; and performing opening and closing operation on the depth image containing the original hole pixel points by adopting the circular structural elements to eliminate partial original hole pixel points to obtain the depth image containing the hole pixel points. By carrying out opening and closing operation morphological processing on the denoised depth image, isolated and burred points in the original depth image can be eliminated, and a fine cavity area can be repaired, so that the depth image after optimized processing is obtained, and the effect of the depth image after subsequent cavity filling is improved.
Another embodiment of the present invention relates to a depth image restoration method, as shown in fig. 3, which is an improvement of the method steps shown in fig. 1, and the improvement is to refine the determination process of the filling priority of each hole pixel point. As shown in fig. 3, the step 102 includes the following sub-steps.
Substep 1021: and aiming at unfilled hole pixel points, determining a neighborhood by taking each hole pixel point as a center respectively, and preferentially determining that the filling priority of the hole pixel points with less number of non-hole pixel points contained in the neighborhood is higher.
Specifically, as shown in fig. 4, there are 5 neighborhood images centered around a hole pixel (pixel value is 0). Aiming at the void pixel points positioned in the center position in the neighborhood images, the filling priority of the void pixel points with less number of non-void pixel points contained in the neighborhood can be preferentially determined to be higher. For example, in fig. 4, the number of non-hole pixels included in the neighborhoods (a) to (e) is 0, 2, 4, and 8 in order, and the filling priorities of the hole pixels corresponding to the center position in the drawing are (a), (b) > (c) > (d) > (e).
Substep 1022: and when the filling priorities of the void pixel points obtained based on the number of the non-void pixel points contained in the neighborhood are the same, determining that the filling priority of the void pixel points with smaller standard difference of the pixel values of the non-void pixel points contained in the neighborhood is higher.
Specifically, as shown in fig. 4, there are 5 neighborhood images centered around a hole pixel (pixel value is 0). For the void pixel points located in the center of the neighborhood images, when the filling priorities of the void pixel points obtained based on the number of the non-void pixel points included in the neighborhood are the same, for example, the filling priorities of (a) and (b) are the same, it can be continuously determined that the filling priority of the void pixel points included in the neighborhood and having a smaller standard deviation of the pixel values of the non-void pixel points is higher. For example, it can be known through calculation that the standard deviation of the pixel value of each pixel point in the field in (a) is smaller than the standard deviation of the pixel value of each pixel point in the field in (b), so that the filling priority ranking of the void pixel point at the center position in the corresponding graph can be determined to be (a) > (b), and the finally obtained filling priority ranking of the void pixel point at the center position in each corresponding graph is (a) > (b) > (c) > (d) > (e).
Compared with the related technology, the filling priority of the void pixel points with less number of non-void pixel points contained in the neighborhood is determined to be higher by priority; when the filling priorities of the void pixel points obtained based on the number of the non-void pixel points contained in the neighborhood are the same, the filling priority of the void pixel point with the smaller standard difference of the pixel values of the non-void pixel points contained in the neighborhood is determined to be higher, and therefore an implementation mode for determining the filling priority of each void pixel point is provided.
Another embodiment of the present invention relates to a depth image restoration method, as shown in fig. 5, which is an improvement of the method steps shown in fig. 1, and the improvement is that the determination process of the filling priority of each hole pixel point is refined. As shown in fig. 5, the step 102 includes the following sub-steps.
Substep 1023: aiming at unfilled hole pixel points, determining a neighborhood by taking each hole pixel point as a center, and determining a first filling priority of the hole pixel points based on the number of non-hole pixel points contained in the neighborhood; wherein, the smaller the number, the higher the corresponding padding priority.
The process executed in this step can refer to sub-step 1021, which is not described herein.
Substep 1024: aiming at unfilled hole pixel points, determining a neighborhood by taking each hole pixel point as a center, and determining a second filling priority of the hole pixel points based on the pixel value standard difference of non-hole pixel points contained in the neighborhood; wherein, the smaller the standard deviation of the pixel value, the higher the corresponding filling priority.
Specifically, as shown in fig. 4, there are 5 neighborhood images centered around a hole pixel (pixel value is 0). And respectively calculating the standard deviation of the pixel values of the non-hole pixel points contained in the neighborhood aiming at the hole pixel points positioned at the central position in the neighborhood images. For example, in fig. 4, the standard deviations of the pixel values of the non-hole pixels included in the neighborhoods (a) to (e) are 1.05, 41.14, 41.19, 0.5, and 0 in this order, and the second filling priorities of the hole pixels corresponding to the center positions in the drawing are (d) > (a), (b) > (c) > (e). The second shim priority for (e) is at a minimum by default in any case.
Sub-step 1025: and taking the average priority of the first filling priority and the second filling priority as the filling priority of the final hole pixel point.
For example, in fig. 4, the filling priority ranking of the void pixel point corresponding to the center position in the final map may be (a), (b) > (d) > (c) > (e). When the average priority is actually calculated, weights can be set for the first padding priority and the second padding priority respectively, and the weighted average priority is determined as the final padding priority.
Compared with the related art, the embodiment determines the first filling priority of the void pixel points based on the number of the non-void pixel points contained in the neighborhood; wherein, the smaller the number, the higher the corresponding filling priority; determining a second filling priority of the void pixel points based on the pixel value standard deviation of the non-void pixel points contained in the neighborhood; the smaller the standard deviation of the pixel value is, the higher the corresponding filling priority is; and taking the average priority of the first filling priority and the second filling priority as the filling priority of the final hole pixel point, thereby providing an implementation mode for determining the filling priority of each hole pixel point.
Another embodiment of the present invention relates to a depth image restoration method, as shown in fig. 6, which is an improvement of the method steps shown in fig. 1, and the improvement is to refine the determination process of the filling priority of each hole pixel point. As shown in fig. 6, the step 102 includes the following sub-steps.
Substep 1026: the filling priority of each hollow pixel point is calculated by adopting the following formula:
wherein the content of the first and second substances,R(s)the fill-in priority for the void pixel points,sthe light-emitting diode is a hollow pixel point,n v (s)the number of non-void pixels in the neighborhood is determined by centering on the void pixel,
is the standard deviation of the pixel values of the non-hole pixel points in the neighborhood,
is a constant factor.
Can make it possible to
Normalized to [0,1 ]]In the meantime.
Specifically, the filling priority ranking of the void pixel point at the center position in each corresponding graph in fig. 4 determined by the formula (1) is (a) > (b) > (c) > (d) > (e).
Compared with the related art, the embodiment determines the final filling priority of the void pixel points through the formula (1), thereby providing an implementation mode for determining the filling priority of each void pixel point.
Another embodiment of the invention relates to an electronic device, as shown in FIG. 7, comprising at least one processor 202; and a memory 201 communicatively coupled to the at least one processor 202; wherein the memory 201 stores instructions executable by the at least one processor 202, the instructions being executable by the at least one processor 202 to enable the at least one processor 202 to perform any of the method embodiments described above.
Where the memory 201 and the processor 202 are coupled in a bus, the bus may comprise any number of interconnected buses and bridges that couple one or more of the various circuits of the processor 202 and the memory 201 together. The bus may also connect various other circuits such as peripherals, voltage regulators, power management circuits, and the like, which are well known in the art, and therefore, will not be described any further herein. A bus interface provides an interface between the bus and the transceiver. The transceiver may be one element or a plurality of elements, such as a plurality of receivers and transmitters, providing a means for communicating with various other apparatus over a transmission medium. The data processed by the processor 202 is transmitted over a wireless medium through an antenna, which further receives the data and transmits the data to the processor 202.
The processor 202 is responsible for managing the bus and general processing and may also provide various functions including timing, peripheral interfaces, voltage regulation, power management, and other control functions. And memory 201 may be used to store data used by processor 202 in performing operations.
Another embodiment of the present invention relates to a computer-readable storage medium storing a computer program. The computer program realizes any of the above-described method embodiments when executed by a processor.
That is, as can be understood by those skilled in the art, all or part of the steps in the method for implementing the embodiments described above may be implemented by a program instructing related hardware, where the program is stored in a storage medium and includes several instructions to enable a device (which may be a single chip, a chip, or the like) or a processor (processor) to execute all or part of the steps of the method described in the embodiments of the present application. And the aforementioned storage medium includes: various media capable of storing program codes, such as a usb disk, a removable hard disk, a Read-only memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk.
It will be understood by those of ordinary skill in the art that the foregoing embodiments are specific examples for carrying out the invention, and that various changes in form and details may be made therein without departing from the spirit and scope of the invention in practice.
Claims (10)
1. A depth image restoration method, comprising:
acquiring a depth image containing a hole pixel point;
performing the following preferential filling operation on unfilled hole pixel points in the depth image: aiming at the unfilled hole pixel points, respectively determining a neighborhood by taking each hole pixel point as a center, and determining the filling priority of each hole pixel point according to the number and the pixel value of non-hole pixel points contained in the neighborhood; the higher the filling priority is, the better the image restoration effect after the corresponding hole pixel point is filled is; filling the specified number of hole pixel points with the highest filling priority;
judging whether the depth image contains unfilled hole pixel points;
if yes, continuing to execute the preferential filling operation on the unfilled hole pixel points; and if not, taking the depth image obtained after the priority filling operation as the repaired depth image.
2. The method of claim 1, wherein the obtaining the depth image containing the hole pixel points comprises:
acquiring a depth image containing original hole pixel points;
and carrying out opening and closing operation on the depth image containing the original hole pixel points by adopting a circular structural element so as to eliminate part of the original hole pixel points and obtain the depth image containing the hole pixel points.
3. The method of claim 2, wherein the obtaining the depth image containing the original hole pixel points comprises:
collecting an original depth image;
dividing the original depth image into a plurality of regions according to the pixel value difference value of each pixel point and adjacent pixel points in the original depth image;
and determining each pixel point in the region, in which the number of the included pixel points is less than the preset region pixel point threshold value, as the original hole pixel point.
4. The method according to any one of claims 1 to 3, wherein the determining the filling priority of each hole pixel point according to the number and the pixel value of the non-hole pixel points included in the neighborhood comprises:
preferentially determining that the filling priority of the void pixel points with less number of non-void pixel points contained in the neighborhood is higher;
and when the filling priorities of the void pixels obtained based on the number of the non-void pixels contained in the neighborhood are the same, determining that the filling priority of the void pixels with smaller standard deviation of the pixel values of the non-void pixels contained in the neighborhood is higher.
5. The method according to any one of claims 1 to 3, wherein the determining the filling priority of each hole pixel point according to the number and the pixel value of the non-hole pixel points included in the neighborhood comprises:
determining a first filling priority of the void pixel points based on the number of the non-void pixel points contained in the neighborhood; wherein, the smaller the number is, the higher the corresponding padding priority is;
determining a second filling priority of the void pixel points based on the pixel value standard deviation of the non-void pixel points contained in the neighborhood; wherein, the smaller the standard deviation of the pixel values is, the higher the corresponding padding priority is;
and taking the average priority of the first filling priority and the second filling priority as the final filling priority of the hollow pixel point.
6. The method according to any one of claims 1 to 3, wherein the determining the filling priority of each hole pixel point according to the number and the pixel value of the non-hole pixel points included in the neighborhood comprises:
and calculating the filling priority of each hollow pixel point by adopting the following formula:
wherein the content of the first and second substances,R(s)filling priorities for the hole pixel points,sthe number of the hollow pixel points is the number of the hollow pixel points,n v (s)the number of non-hole pixels in the neighborhood determined by taking the hole pixel as the center,
the standard deviation of the pixel values of the non-hole pixel points in the neighborhood,
is a constant factor.
7. The method according to any of claims 1-3, wherein said filling a specified number of hole pixels with highest fill priority comprises:
and filling one hole pixel point with the highest filling priority in all the hole pixel points.
8. The method according to any one of claims 1 to 3, wherein after determining whether the depth image contains unfilled hole pixels, the method further comprises:
and if not, performing smoothing treatment on the depth image obtained after the preferential filling operation by adopting bilateral filtering to obtain a final repaired depth image.
9. An electronic device, comprising:
at least one processor; and the number of the first and second groups,
a memory communicatively coupled to the at least one processor; wherein the content of the first and second substances,
the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the depth map repair method of any of claims 1 to 8.
10. A computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the depth map repair method of any one of claims 1 to 8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111171194.8A CN113935914A (en) | 2021-10-08 | 2021-10-08 | Depth image restoration method, electronic device and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111171194.8A CN113935914A (en) | 2021-10-08 | 2021-10-08 | Depth image restoration method, electronic device and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113935914A true CN113935914A (en) | 2022-01-14 |
Family
ID=79278168
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111171194.8A Pending CN113935914A (en) | 2021-10-08 | 2021-10-08 | Depth image restoration method, electronic device and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113935914A (en) |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103136775A (en) * | 2013-03-19 | 2013-06-05 | 武汉大学 | KINECT depth map cavity filling method based on local restriction reconstruction |
| CN103379350A (en) * | 2012-04-28 | 2013-10-30 | 中国科学院深圳先进技术研究院 | Virtual viewpoint image post-processing method |
| CN105825482A (en) * | 2016-03-15 | 2016-08-03 | 四川用联信息技术有限公司 | Depth image restoration algorithm |
| CN106651871A (en) * | 2016-11-18 | 2017-05-10 | 华东师范大学 | Automatic filling method for cavities in depth image |
| CN108629756A (en) * | 2018-04-28 | 2018-10-09 | 东北大学 | A kind of Kinect v2 depth images Null Spot restorative procedure |
| CN108648221A (en) * | 2018-05-10 | 2018-10-12 | 重庆大学 | A kind of depth map cavity restorative procedure based on mixed filtering |
| CN112116602A (en) * | 2020-08-31 | 2020-12-22 | 北京的卢深视科技有限公司 | Depth map repairing method and device and readable storage medium |
-
2021
- 2021-10-08 CN CN202111171194.8A patent/CN113935914A/en active Pending
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103379350A (en) * | 2012-04-28 | 2013-10-30 | 中国科学院深圳先进技术研究院 | Virtual viewpoint image post-processing method |
| CN103136775A (en) * | 2013-03-19 | 2013-06-05 | 武汉大学 | KINECT depth map cavity filling method based on local restriction reconstruction |
| CN105825482A (en) * | 2016-03-15 | 2016-08-03 | 四川用联信息技术有限公司 | Depth image restoration algorithm |
| CN106651871A (en) * | 2016-11-18 | 2017-05-10 | 华东师范大学 | Automatic filling method for cavities in depth image |
| CN108629756A (en) * | 2018-04-28 | 2018-10-09 | 东北大学 | A kind of Kinect v2 depth images Null Spot restorative procedure |
| CN108648221A (en) * | 2018-05-10 | 2018-10-12 | 重庆大学 | A kind of depth map cavity restorative procedure based on mixed filtering |
| CN112116602A (en) * | 2020-08-31 | 2020-12-22 | 北京的卢深视科技有限公司 | Depth map repairing method and device and readable storage medium |
Non-Patent Citations (1)
| Title |
|---|
| 李声锋: "《有理逼近方法及其在数值计算与图像修复中的应用》", 30 June 2019 * |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109461181B (en) | Depth image acquisition method and system based on speckle structured light | |
| CN111210429B (en) | Point cloud data partitioning method and device and obstacle detection method and device | |
| EP3712841A1 (en) | Image processing method, image processing apparatus, and computer-readable recording medium | |
| CN111582054B (en) | Point cloud data processing method and device and obstacle detection method and device | |
| US20130136338A1 (en) | Methods and Apparatus for Correcting Disparity Maps using Statistical Analysis on Local Neighborhoods | |
| US11042966B2 (en) | Method, electronic device, and storage medium for obtaining depth image | |
| US10708505B2 (en) | Image processing apparatus, method, and storage medium | |
| CN111553946B (en) | Method and device for removing ground point cloud and method and device for detecting obstacle | |
| CN114331912B (en) | Image restoration method and device | |
| CN113762253B (en) | Speckle extraction method and device, electronic device and storage medium | |
| CN114283089B (en) | Jump acceleration based depth recovery method, electronic device, and storage medium | |
| KR101086274B1 (en) | Apparatus and method for extracting depth information | |
| CN112508803A (en) | Denoising method and device for three-dimensional point cloud data and storage medium | |
| CN110188640B (en) | Face recognition method, face recognition device, server and computer readable medium | |
| CN113935914A (en) | Depth image restoration method, electronic device and storage medium | |
| CN114331919B (en) | Depth recovery method, electronic device, and storage medium | |
| CN116311135A (en) | Data dimension reduction method, data dimension reduction system and controller for semantic information | |
| CN114283081B (en) | Depth recovery method based on pyramid acceleration, electronic device and storage medium | |
| CN113936316B (en) | DOE (DOE-out-of-state) detection method, electronic device and computer-readable storage medium | |
| CN114693546A (en) | Image denoising method and device, electronic equipment and computer readable storage medium | |
| CN111784756A (en) | Parallax confidence estimation method and device, electronic equipment and medium | |
| CN113640823B (en) | Method and device for map drawing based on laser reflectivity base map | |
| CN114783041B (en) | Target object recognition method, electronic device, and computer-readable storage medium | |
| CN113965697B (en) | Parallax imaging method based on continuous frame information, electronic device and storage medium | |
| CN110599542A (en) | Method and device for local mapping of adaptive VSLAM (virtual local area model) facing to geometric area |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20220516 Address after: 230091 room 611-217, R & D center building, China (Hefei) international intelligent voice Industrial Park, 3333 Xiyou Road, high tech Zone, Hefei, Anhui Province Applicant after: Hefei lushenshi Technology Co.,Ltd. Address before: 100083 room 3032, North B, bungalow, building 2, A5 Xueyuan Road, Haidian District, Beijing Applicant before: BEIJING DILUSENSE TECHNOLOGY CO.,LTD. Applicant before: Hefei lushenshi Technology Co.,Ltd. |
|
| TA01 | Transfer of patent application right | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20220114 |
|
| RJ01 | Rejection of invention patent application after publication |