CN108648221B - Depth map hole repairing method based on hybrid filtering - Google Patents
Depth map hole repairing method based on hybrid filtering Download PDFInfo
- Publication number
- CN108648221B CN108648221B CN201810443580.XA CN201810443580A CN108648221B CN 108648221 B CN108648221 B CN 108648221B CN 201810443580 A CN201810443580 A CN 201810443580A CN 108648221 B CN108648221 B CN 108648221B
- Authority
- CN
- China
- Prior art keywords
- pixel
- pixels
- neighborhood
- hole
- point
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 59
- 238000001914 filtration Methods 0.000 title claims abstract description 42
- 230000002146 bilateral effect Effects 0.000 claims abstract description 18
- 238000004364 calculation method Methods 0.000 claims description 19
- 230000008439 repair process Effects 0.000 claims description 16
- 230000004927 fusion Effects 0.000 claims description 6
- 230000000694 effects Effects 0.000 abstract description 17
- 239000011800 void material Substances 0.000 abstract description 8
- 238000012545 processing Methods 0.000 abstract description 4
- 230000003044 adaptive effect Effects 0.000 abstract description 3
- 238000010586 diagram Methods 0.000 description 14
- 230000008569 process Effects 0.000 description 14
- 230000006870 function Effects 0.000 description 9
- 239000000126 substance Substances 0.000 description 5
- 230000004438 eyesight Effects 0.000 description 3
- 239000000463 material Substances 0.000 description 3
- 241000282414 Homo sapiens Species 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 238000002474 experimental method Methods 0.000 description 2
- 230000006872 improvement Effects 0.000 description 2
- 230000004075 alteration Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 230000000052 comparative effect Effects 0.000 description 1
- 230000008878 coupling Effects 0.000 description 1
- 238000010168 coupling process Methods 0.000 description 1
- 238000005859 coupling reaction Methods 0.000 description 1
- 230000003247 decreasing effect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000009472 formulation Methods 0.000 description 1
- 238000009499 grossing Methods 0.000 description 1
- 238000005286 illumination Methods 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- 239000000203 mixture Substances 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000016776 visual perception Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/50—Depth or shape recovery
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/11—Region-based segmentation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10028—Range image; Depth image; 3D point clouds
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Image Processing (AREA)
- Image Analysis (AREA)
Abstract
The invention discloses a depth map void repairing method based on hybrid filtering, which comprises the following steps: acquiring a depth image to be repaired; identifying a void region thereof; calculating the priority of pixels in the holes; putting the pixels with the priority higher than the threshold value into a priority queue for filling; after the hole area is repaired, repairing the non-hole area; for the hollow area and the non-hollow area, the non-edge area is repaired firstly, and then the edge area is repaired. The invention provides that priorities are set for all pixels in the hole, and the priorities depend on the support and credibility of neighborhood pixels to the pixels of the central hole. With the continuous filling of unknown pixels, the hole edge is continuously reduced, the priority of the pixels at the hole edge is updated, and the adaptive updating threshold ensures that the pixels with the highest priority are filled preferentially. The invention adds texture and structure information on the principle of joint bilateral filtering, and has better processing effect on images with complex texture and structure.
Description
Technical Field
The invention belongs to the technical field of depth map restoration in three-dimensional reconstruction, and particularly relates to a depth map hole restoration method based on hybrid filtering.
Background
The daily life scenes are three-dimensional stereo scenes, namely, the daily life scenes are formed by interweaving a plurality of three-dimensional information, and the information received by human beings is also based on the visual perception and processing of the human beings. At present, three-dimensional reconstruction mainly comprises converting two-dimensional information into three-dimensional information through a stereoscopic vision technology, and extracting data from an image to realize three-dimensional reconstruction of a two-dimensional object. Currently, there are two main techniques for reconstructing a three-dimensional scene: one method is to adopt a multi-view method for reconstruction, and infer and display the three-dimensional information of a scene or an object in the scene by using binocular or multi-view stereo vision through a video camera or a camera; the other method is to reconstruct a three-dimensional model on the basis of a color image and a depth image in a depth + texture mode. Unlike texture color images, which are texture information describing an object, the gray value of a depth image represents the distance between the object and the camera in the image, and larger values represent longer distances.
In the process of acquiring the depth image, due to the limitation of the device and the interference of external environment factors, there are a plurality of acquiring modes, such as: calibration errors of sensor hardware, errors of offset and measurement accuracy of the offset can be influenced under illumination conditions, the texture of the surface material of an object can be influenced, and the like, so that an area with a depth value of 0, namely a cavity, exists in an acquired depth image, and the effect of three-dimensional reconstruction is poor.
At present, aiming at image restoration in three-dimensional reconstruction, a filtering algorithm is mainly adopted, Tomasi and Maduci in 1998 put forward a theory of a bilateral filtering algorithm for the first time, then improvement of years is carried out, Anh Vu Le et al in 2014 propose a combined bilateral filtering algorithm based on direction and a combined bilateral filtering algorithm based on direction locally, edge direction factors are further added in calculation of spatial proximity in a formula, namely, a kernel is Gaussian filtering based on direction, in depth image hole restoration, whether pixels in an image are in holes or not and whether object boundaries are in object boundaries or not are divided into four types, and a combination mode of the algorithm is found through experiments, so that different algorithms are applied to different conditions, and hole restoration is carried out more effectively and accurately. However, the algorithm based on the joint bilateral filtering and the improved algorithm thereof focus on the computational improvement and perfection of the depth value of the central hole pixel, and the overall hole pixel filling strategy is not perfected. This may cause some unnecessary errors, for example, an unknown central pixel, and although the similarity between the surrounding neighboring pixels and the central pixel is high, the number of the neighboring pixels known as such is small, and if the calculation of the pixel is performed first, the error may be caused by the missing information of the surrounding neighboring pixels. In addition, the main structure information and the texture information of the image are not added in the repairing process of the current algorithm, which affects the accuracy of the edge pixel calculation and the overall repairing capability of the image.
Disclosure of Invention
The invention aims to at least solve the technical problems in the prior art, and particularly creatively provides a depth map hole repairing method based on hybrid filtering.
In order to achieve the above object, the present invention provides a depth map hole repairing method based on hybrid filtering, which includes the following steps:
s1, acquiring a depth image to be repaired;
s2, identifying a hole area, wherein the pixel with the depth value of 0 is a hole pixel;
s3, setting a priority initial threshold, and calculating the priority of all the pixels of the hole filling area, wherein the priority is determined by the support degree r1 and the reliability r2 of the hole pixels;
s4, placing the pixels with the priority greater than the threshold into a priority queue, and calculating the depth values of the pixels in the queue according to the sequence of the priorities from large to small;
s5, extracting main structure information and texture information from the color image through a total variation model;
s6, for the cavity area, firstly judging whether the cavity pixel is in the edge area, if it is in the non-edge area, adopting local direction-based local united bilateral filtering algorithm, the main calculation method is that the depth value of the depth value neighborhood pixel is weighted and averaged, the weight of each neighborhood pixel is obtained by the space proximity, the gray value similarity, the structure similarity and the texture similarity of the neighborhood pixel and the central pixel, if there is still unfilled pixel in the cavity non-edge area, returning to execute step S4; if not, starting to fill the edge region, adopting a combined bilateral filtering algorithm based on the direction and fusing the structure and the texture information, wherein the repair neighborhood window of the non-edge region repair algorithm is self-adaptive;
and S7, after finishing the repair of all the hole regions, repairing all the non-hole regions, wherein the repair of the non-hole region and the non-edge region is realized by a combined trilateral filtering algorithm of a fusion structure and texture information, and the filling method of the non-hole edge region is the same as the filling method of the edge region of the hole region.
The method solves the problem of the cavity in the process of acquiring the depth image, firstly sets priorities for all pixels in the cavity aiming at an integral hole filling strategy, and the priorities depend on the support and credibility of neighborhood pixels to the pixels of the central cavity. In the process of filling the cavity, the priority of the pixels at the edge of the cavity and the adaptive updating threshold value are updated, and the pixels with the highest priority can be always filled preferentially. The void filling priority strategy not only considers the position information, the gray value and the depth value information of the pixel points in the field, but also considers the number of the neighborhood pixels and circularly calculates the priority value, thereby avoiding the calculation error caused by neglecting the filling priority sequence of the void pixels.
The method adds main structure information and texture information of the image into the existing filtering algorithm, and carries out hole restoration on non-hole non-edge pixels, edge non-hole pixels, hole non-edge pixels and hole edge pixels respectively according to whether the pixels in the image are at the boundary or not and whether the pixels are in the holes or not, so as to obtain a restoration result image with better quality.
The depth map hole repairing method based on the hybrid filtering combines two kinds of information of texture and structure, and has a good processing result for images with complex texture.
Because the amount of information of the object edge in the image is large and the requirement on the accuracy of the pixel value is high, the invention fills the pixels of the non-edge area firstly, and when the completion is finished, the neighborhood pixels of the edge area are basically repaired. And repairing the edge area, wherein the edge area is repaired to obtain more information, and the repairing effect is better.
In a preferred fact mode of the invention, calculating neighborhood pixel support r1 and credibility r2 of a pixel point to be filled, wherein the neighborhood pixel support r1 is obtained by dividing the number of known neighborhood pixels around the pixel to be filled by the number of neighborhood pixels in a filter window, the size of the filter window is set to k x k, and k is a positive integer greater than 1; the confidence r2 is the similarity between the neighboring pixels and the pixels to be padded, including the spatial proximity and the gray value similarity, and the formula is as follows:
wherein the content of the first and second substances,
in order to be in spatial proximity to each other,
is the similarity of the gray-scale values,
wherein the superscript d indicates that the information comes from the depth map, the superscript c indicates that the information comes from the color map, the subscript s indicates the spatial domain, the subscript r indicates the pixel range domain,
representing spatial proximity between pixels, qxIs the abscissa of pixel point q, qyIs the ordinate of pixel point q, q being the neighborhood pixel point of the central pixel, pxIs the abscissa of pixel p, p being the central pixel, pyIs the ordinate of the pixel point p,
representing the gray value similarity between the pixel q point in the neighborhood and the pixel p point in the center, IpIs the gray value of p points, IqGray value of q point, σsIs a space domain parameter in the Euclidean distance formula, sigmarIs the standard deviation of the gaussian equation.
By the method, the support degree r1 and the credibility r2 of the neighborhood pixel points of the pixel points to be filled are accurately and quickly calculated, and the priority is guaranteed to be successfully calculated.
In another preferable embodiment of the present invention, in step S3, the calculation method of the priority m is: m-r 1+ λ × r2, λ >0,
wherein, λ is a parameter for determining the weight occupied by the support degree r1 and the reliability r2 when calculating the priority.
In the invention, before the depth of each hole pixel to be detected is calculated, priorities, namely the priority of the filling sequence, are set for all unknown hole pixels. All the hollow pixels in the hollow area are sorted according to the priorities, and when the hollow is filled, the depth value is calculated according to the priority sequence of each pixel to be measured from large to small, so that the filling efficiency is improved, and the calculation error caused by neglecting the filling priority sequence of the hollow pixels is avoided.
In another preferable embodiment of the present invention, the method for extracting the main structure information and the texture information includes: the image I is decomposed into a linear combination of two parts, namely main structure information S and texture information T, which are expressed as follows:
I＝S+T,
defining a total variation model:
wherein omega is the area of the image,
establishing an energy formula：
Obtaining
Let the main structure information S to be output be as close to the original image I as possible, λ 1 × TV (S) is TV regularization term, λ 1 is regularization parameter, the weights of the first term and the second term in the balance formula, the texture information in the image is obtained by the formula T ═ I-S,
based on the solved main structure information S and texture information T, for each pixel point of the cavity region, the structure similarity and the texture similarity between the pixel point and the neighborhood pixel point are obtained, and the specific method comprises the following steps:
wherein q is a neighborhood pixel of the central pixel, p is the central pixel,
representing the structural similarity between the pixel q point in the neighborhood and the pixel p point in the center, SpMain structural information, S, representing the center pixel point pqMain structure information representing the neighborhood pixel point q,
representing the texture similarity, T, between the pixel q point in the neighborhood and the pixel p point in the centerpTexture information, T, representing the center pixel pqAnd expressing the texture information of the neighborhood pixel point q.
The invention calculates the structural similarity and the texture similarity and adds the two information into the calculation of the depth value weight, so that the depth value of the central hole pixel is more accurately and effectively calculated.
In another preferable embodiment of the present invention, the method for filling the hole pixel includes:
1) and for the pixels in the edge area of the hollow part, filling by adopting a combined bilateral filtering algorithm based on the direction and fusing structure and texture information:
wherein D ispIs the depth value of the central pixel to be measured, the neighborhood phipComprises the following steps:
φp＝{q＝(qx,qy)px-w≤qx≤px+w,py-w≤qy≤py+ w, a filter window of (2w +1) × (2w +1) size is adopted, w is a positive integer;
in the method, because a Directional Gaussian Filter algorithm is used in the spatial proximity, in the process of calculating the weight of the neighborhood pixels, higher weight is given to the neighborhood pixels which are in the same direction or close to the gradient of the void pixels, so that the calculation of the weight is more accurate.
2) And for pixels of non-edge areas of the hollow part, filling by adopting a direction-based local joint bilateral filtering algorithm:
neighborhood ΩpThe window size is self-adaptive and is determined by the distance from the hole pixel to the nearest edge in the same plane with the hole pixel;
3) and for pixels in non-edge areas of the non-cavity part, repairing and filling by adopting a combined trilateral filtering algorithm of a fusion structure and texture information:
wherein the content of the first and second substances,
similarity of depth values of the p point and the q point;
4) and for pixels in the non-hollow part edge area, filling and repairing by adopting the same filling algorithm as the filling algorithm of the hollow part edge area.
In the image restoration process, due to the difference between the cavity and the noise and the particularity of the edge pixels, all the pixels to be restored are divided into four types according to whether the pixels are in the cavity or not and whether the pixels are at the edge or not, and the pixels are respectively restored by using the mixed filtering algorithm combined by different algorithms, so that the accuracy of the edge pixels is improved, the error of cavity filling is reduced, and the image restoration effect is improved.
In another preferred embodiment of the present invention, the method for adjusting the size of the repair neighborhood window of the non-edge region repair algorithm comprises: the window size is determined by the distance of the hole pixel to its nearest edge in the same plane.
Additional aspects and advantages of the invention will be set forth in part in the description which follows and, in part, will be obvious from the description, or may be learned by practice of the invention.
Drawings
The above and/or additional aspects and advantages of the present invention will become apparent and readily appreciated from the following description of the embodiments, taken in conjunction with the accompanying drawings of which:
FIG. 1 is a flow chart of a depth map hole repairing method based on hybrid filtering in a preferred embodiment of the present invention;
FIG. 2 is a diagram of the comparative effect of an experimental group in a preferred embodiment of the present invention, in which FIG. 2(a) is an original image to be restored, and FIG. 2(b) is a depth map with holes; FIG. 2(c) is a diagram of extracted texture information; FIG. 2(d) is a diagram of extracted structural information; FIG. 2 is a diagram illustrating the repairing effect of pixels in the edge area of the hollow portion; FIG. 2(f) is a diagram showing the repairing effect of the pixel in the non-edge area of the hollow portion; FIG. 2(g) is a diagram of the repairing effect of the pixel in the non-edge area of the non-hollow portion; FIG. 2(h) is a diagram showing the repairing effect of pixels in the edge area of the non-hollow portion; FIG. 2(i) is an enlarged view of the repairing effect; FIG. 2(j) is an original image;
fig. 3 is a schematic diagram of adaptively adjusting the size of a calculated pixel window in a preferred embodiment of the present invention.
Detailed Description
Reference will now be made in detail to embodiments of the present invention, examples of which are illustrated in the accompanying drawings, wherein like or similar reference numerals refer to the same or similar elements or elements having the same or similar function throughout. The embodiments described below with reference to the accompanying drawings are illustrative only for the purpose of explaining the present invention, and are not to be construed as limiting the present invention.
In the description of the present invention, unless otherwise specified and limited, it is to be noted that the terms "mounted," "connected," and "connected" are to be interpreted broadly, and may be, for example, a mechanical connection or an electrical connection, a communication between two elements, a direct connection, or an indirect connection via an intermediate medium, and specific meanings of the terms may be understood by those skilled in the art according to specific situations.
The invention provides a depth map hole repairing method based on hybrid filtering, and provides a corresponding repairing technology according to the problems of a depth image in the process of obtaining the depth image, so as to obtain a more accurate depth image and promote the development of the research fields of utilizing depth information, such as stereoscopic vision, three-dimensional reconstruction and the like, as shown in figure 1, the method comprises the following steps:
and S1, acquiring an image to be repaired, wherein the image to be repaired is provided with a hole. In a preferred embodiment of the present invention, the groudtruth may be used as a perfect depth map, a hole is dug on the perfect depth map, and the experimental effect may be determined by comparing the repair result map with the groudtruth. Therefore, the invention adopts the artificial simulated depth image acquisition equipment to dig the holes on the groudtruth (the holes appearing on the general depth acquisition equipment are easy to be in the edge area, so in a more preferable embodiment, the holes are also dug in the edge area), and the holes are taken as the depth map with the holes in the experiment, namely the image to be restored.
S2, identifying a hole region, wherein the region with a depth value of 0 is a hole.
And S3, calculating the priority of all the pixels of the hole filling area. The calculation method of the priority m comprises the following steps: m-r 1+ λ × r2, λ >0,
wherein, λ is a parameter for determining the weight occupied by the support degree r1 and the reliability r2 when calculating the priority.
The neighborhood pixel point support degree r1 is determined by the proportion of the known neighborhood pixels of the filled hole pixels to the total neighborhood pixels, the size of the filter window is set to k × k, and k is a positive integer greater than 1;
the confidence r2 is determined by the spatial proximity and gray value similarity between the neighboring pixels and the pixels to be filled, and the formula is as follows:
wherein the content of the first and second substances,
in order to be in spatial proximity to each other,
is the similarity of the gray-scale values,
wherein the superscript d indicates that the information comes from the depth map, the superscript c indicates that the information comes from the color map, the subscript s indicates the spatial domain, the subscript r indicates the pixel range domain,
representing spatial proximity between pixels, qxIs the abscissa of pixel point q, qyIs likeOrdinate of a pixel point q, q being a neighborhood pixel point of the central pixel, pxIs the abscissa of pixel p, p being the central pixel, pyIs the ordinate of the pixel point p,
representing the gray value similarity between the pixel q point in the neighborhood and the pixel p point in the center, IpIs the gray value of p points, IqGray value of q point, σsIs a space domain parameter in the Euclidean distance formula, sigmarThe degree of smoothing, σ, of the Gaussian filter function is determined for the standard deviation of the Gaussian formularThe larger the gaussian function, the wider the band, and the smoother it appears on the image. Through sigmarThe performance of the algorithm can be determined, and the variation range of each similarity factor among pixel points is limited.
In the invention, before the depth of each hole pixel to be detected is calculated, priorities, namely the priority of the filling sequence, are set for all unknown hole pixels. All the hollow pixels in the hollow area are placed in the priority queue according to the priorities of the hollow pixels and the pixels larger than the threshold value, and when the hollow is filled, the pixels in the priority queue are filled preferentially, so that the filling efficiency is improved, and the calculation error caused by neglecting the filling priority sequence of the hollow pixels is avoided.
In the calculation, an initial threshold value is set for the priority m, the initial threshold value is the value with the highest m according to the data obtained by calculating the priority for the first time, and the initial threshold value is 1.5. Then, the priority m of all pixels in the hole region is calculated by m-r 1+ λ × r 2. All hole pixels above the threshold are inserted into the priority queue TSet and the depth value of each pixel in the queue is calculated. Since each calculated pixel value affects the support and confidence of its neighboring hole pixels, the priority of all unknown pixels is not constant, and needs to be dynamically updated as more and more depths of hole pixels are calculated. Therefore, after all the hole pixels in the current queue are filled, the priority of the hole edge pixels and the adaptive update threshold are updated, and the hole pixels are arranged in the queue according to the new threshold. Here, the threshold is adaptively changed, if there are no pixels to be filled in the queue, that is, the number of pixels to be filled is 0, the threshold is decreased by 0.1, otherwise, the threshold is increased by 0.1, and the threshold is adjusted so that the pixel with the highest priority can always be preferentially calculated.
And S4, calculating the depth value of the hole pixel according to the sequence from large to small of the priority order.
And S5, extracting main structure information and texture information of the color image, and acquiring the structure similarity and the texture similarity of each pixel point of the cavity region and the adjacent pixel points.
In another preferred embodiment of the present invention, the method for extracting the main structure information and the texture information comprises:
the image I is decomposed into a linear combination of two parts, namely main structure information S and texture information T, which are expressed as follows:
I＝S+T,
defining a total variation model:
wherein omega is the area of the image,
establishing an energy formula:
the texture information in the image is obtained by the formula T-I-S,
based on the solved main structure information S and texture information T, for each pixel point of the cavity region, the structure similarity and the texture similarity between the pixel point and the neighborhood pixel point are obtained, and the specific method comprises the following steps:
wherein q is a neighborhood pixel of the central pixel, p is the central pixel,
representing the structural similarity between the pixel q point in the neighborhood and the pixel p point in the center, SpMain structural information, S, representing the center pixel point pqMain structure information representing the neighborhood pixel point q,
representing the texture similarity, T, between the pixel q point in the neighborhood and the pixel p point in the centerpTexture information, T, representing the center pixel pqAnd expressing the texture information of the neighborhood pixel point q.
The invention adds the structural similarity and the texture similarity into the existing algorithm, and adds the two information into the calculation of the depth value weight of the neighborhood pixels, so that the depth value calculation of the central hole pixel is more accurate and effective.
The specific calculation process is as follows: the energy function based on the image total variation realizes minimization under two conditions of the total variation item and the statistical characteristics of noise, so that u is an original image and is a noise image0For an image contaminated by noise, then
u0＝u+n
Where n is random gaussian noise with a mean value of 0, i.e., E (n) is 0, and a variance E (n)2)＝σ2Then the total variation model of the image u can be expressed as:
where Ω denotes an image region. The total variation considers the minimization of the energy function with a constraint that is the statistical information of the noise.
The statistical information of the noise comprises the mean value limiting condition ^ of the noiseΩ(u-u0) dxdy ═ μ and variance limiting condition fΩ(u-u0)2dxdy＝σ2. The minimization problem of the energy function under the defined conditions can be studied by coupling coefficients into the non-defined conditions, which is equivalent to solving the non-defined minimization problem:
where γ, λ are two regularization parameters, which are also lagrange multipliers. Since the mean value of the system noise is 0, in the above equation
∫Ω(u-u0)2dxdy＝∫Ωndxdy0
Due to the fact that
Is a constant term, so the above formula can be expressed as:
thus, the energy function model of the total variation of the image is:
and eliminating noise by solving the minimum value of the energy function so as to obtain a clear image u.
To find the minimum of the energy function E (u)The value of E (u) is obtained by the principle of variational theorem
Therefore, it is not only easy to use
Wherein the content of the first and second substances,
in order to be a gradient operator, the method comprises the following steps,
representing divergence of the content in parentheses, the formulation of the model of the total variation is therefore evolved as:
in order for the solution of the total variation formula to satisfy the constraint condition, the solution u (x, y, t) of the above formula at any time needs to satisfy the variance constraint condition, i.e., the solution u (x, y, t) of the above formula at any time needs to satisfy the variance constraint condition
∫Ω(u(x，y，t)-u0)2dxdy＝σ2
In the equation evolution process, it is necessary to ensure that the variance of u (x, y, t) is moderately constant, so
Can be unfolded to obtain
Will be provided with
Replacing u in the above formulatIs obtained by
Therefore, the above formula can be established only by the value of the regularization parameter lambda, so that the solution of the total variation original equation can be ensured to meet the variance limiting condition, and the problem of limiting minimization can be solved. The formula for calculating the parameter λ derived from the above equation is:
in the actual operation process, a small positive number epsilon is added into the denominator, so that the condition that 0 appears in the denominator can be avoided, and therefore, the condition that 0 appears in the denominator can be obtained
When the above formula is written in a discrete form, the (m +1) th discrete iteration formula is shown as follows, wherein deltat is the iteration step size.
The above general iteration formula illustrates that when the image u is unknown, the initial value of the iteration is u(0)＝u0(ii) a After successive iterations, so that
Description of the invention u(m+1)＝u(m)The iteration reaches a steady state, then u is equal to u(m)I.e. the stationary solution, i.e. the approximation value that ultimately restores the noise contaminated image.
The angle at which the fully-variant model is TV regularization can also be used to constrain structure and texture in the image. When the texture scale is small, the texture scale is generally represented as noise, and the total variation model is used for eliminating the noise; when the texture scale is larger, the full variation model can also effectively extract the main structure and the texture in the image.
S6, for the cavity area, firstly judging whether the cavity pixel is in the edge area, if it is in the non-edge area, adopting the local united bilateral filtering algorithm based on direction, the main calculation method is that the depth value of the depth value neighborhood pixel is weighted and averaged, the weight of each neighborhood pixel is obtained by the space proximity, the gray value similarity, the structure similarity and the texture similarity of the neighborhood pixel and the central pixel, if there is still the pixel which is not filled in the cavity non-edge area, returning to execute the step S4; if not, starting to fill the edge region, adopting a combined bilateral filtering algorithm based on directions and fusing structure and texture information, wherein a repair neighborhood window of a non-edge region repair algorithm is self-adaptive, and the range of the algorithm is fixed;
s7, after finishing the repair of all the hole areas, repairing all the non-hole areas, wherein the repair of the non-hole areas and the non-edge areas of the non-hole areas is realized by a combined trilateral filtering algorithm of a fusion structure and texture information, and the filling method of the non-hole edge areas is the same as the filling method of the edge areas of the hole areas
In another preferable embodiment of the present invention, the method for filling the hole pixel includes:
1) and for the pixels in the edge area of the hollow part, filling by adopting a combined bilateral filtering algorithm based on the direction and fusing structure and texture information:
wherein D ispIs the depth value of the central pixel to be measured, the neighborhood phipComprises the following steps:
φp＝{q＝(qx,qy)px-w≤qx≤px+w,py-w≤qy≤py+ w, a filter window of (2w +1) × (2w +1) size is adopted, w is a positive integer;
in the method, because a Directional Gaussian Filter algorithm is used in the spatial proximity, in the process of calculating the weight of the neighborhood pixels, higher weight is given to the neighborhood pixels which are in the same direction or close to the gradient of the void pixels, so that the calculation of the weight is more accurate.
2) And for pixels of non-edge areas of the hollow part, filling by adopting a direction-based local joint bilateral filtering algorithm:
neighborhood ΩpThe window size is self-adaptive and is determined by the distance from the hole pixel to the nearest edge in the same plane with the hole pixel;
3) and for pixels in non-edge areas of the non-cavity part, repairing and filling by adopting a combined trilateral filtering algorithm of a fusion structure and texture information:
wherein the content of the first and second substances,
similarity of depth values of the p point and the q point;
4) and for pixels in the non-hollow part edge area, filling and repairing by adopting the same filling algorithm as the filling algorithm of the hollow part edge area.
In the image restoration process, due to the difference between the cavity and the noise and the particularity of the edge pixels, all the pixels to be restored are divided into four types according to whether the pixels are in the cavity or not and whether the pixels are at the edge or not, and the pixels are respectively restored by using the mixed filtering algorithm combined by different algorithms, so that the accuracy of the edge pixels is improved, the error of cavity filling is reduced, and the image restoration effect is improved.
The method solves the problem of the cavity in the process of acquiring the depth image, firstly sets priorities for all pixels in the cavity aiming at an integral hole filling strategy, and the priorities depend on the support and credibility of neighborhood pixels to the pixels of the central cavity. In the process of filling the cavity, along with the continuous filling of unknown pixels, the edge of the cavity is continuously reduced, the priority of the pixels at the edge of the cavity can be continuously updated by the algorithm, and meanwhile, the threshold value can be adaptively updated, so that the pixels with the highest priority can be always filled preferentially. The void filling priority strategy not only considers the position information, the gray value and the depth value information of the pixel points in the field, but also considers the number of the neighborhood pixels and circularly calculates the priority value, thereby avoiding the calculation error caused by neglecting the filling priority sequence of the void pixels.
The depth map hole repairing method based on the hybrid filtering combines two kinds of information of texture and structure on the basis of the joint bilateral filtering principle, and has a good processing result on images with complex texture.
Because the amount of information of the object edge in the image is large and the requirement on the accuracy of the pixel value is high, the invention fills the pixels of the non-edge area firstly, and when the completion is finished, the neighborhood pixels of the edge area are basically repaired. And repairing the edge area, wherein the edge area is repaired to obtain more information, and the repairing effect is better.
Fig. 2 is a repairing example of an image to be repaired, in which fig. 2(a) is an original image to be repaired, and fig. 2(b) is a depth map with holes; FIG. 2(c) is a diagram of extracted texture information; FIG. 2(d) is a diagram of extracted structural information; FIG. 2 is a diagram illustrating the repairing effect of pixels in the edge area of the hollow portion; FIG. 2(f) is a diagram showing the repairing effect of the pixel in the non-edge area of the hollow portion; FIG. 2(g) is a diagram of the repairing effect of the pixel in the non-edge area of the non-hollow portion; FIG. 2(h) is a diagram showing the repairing effect of pixels in the edge area of the non-hollow portion; FIG. 2(i) is an enlarged view of the repairing effect; fig. 2(j) is an original image.
In the present embodiment, as shown in fig. 3, the method for adjusting the size of the repair neighborhood window in the hole non-edge area includes: the window size is determined by the distance of the hole pixel to its nearest edge in the same plane.
According to the method, main structure information and texture information of the image are added in the algorithm, the method is divided into four categories according to whether pixels in the image are located at the boundary or not and whether the pixels are located in the hole or not, and hole repairing is respectively carried out on non-hole non-edge pixels, non-hole-edge pixels at the edge, non-hole-edge pixels in the hole and edge pixels at the hole, so that a repairing result image with better quality is obtained.
In the description herein, references to the description of the term "one embodiment," "some embodiments," "an example," "a specific example," or "some examples," etc., mean that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the invention. In this specification, the schematic representations of the terms used above do not necessarily refer to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.
While embodiments of the invention have been shown and described, it will be understood by those of ordinary skill in the art that: various changes, modifications, substitutions and alterations can be made to the embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.
Claims (3)
1. A depth map hole repairing method based on hybrid filtering is characterized by comprising the following steps:
s1, acquiring a depth image to be repaired;
s2, identifying a hole area, wherein the pixel with the depth value of 0 is a hole pixel;
s3, setting a priority initial threshold, calculating the priority of all the pixels in the hole filling area, wherein the priority is determined by the support degree r1 and the credibility r2 of the neighborhood pixels of the hole pixels,
the support degree r1 and the reliability degree r2 of the neighborhood pixel points of the hole pixels,
the neighborhood pixel point support degree r1 is determined by the proportion of the known neighborhood pixels of the filled hole pixels to the total neighborhood pixels, the size of the filter window is set to k x k, and k is a positive integer greater than 1;
the confidence r2 is determined by the spatial proximity and gray value similarity between the neighboring pixels and the pixels to be filled, and the formula is as follows:
r2＝fs d(qx-px,py-qy)×fr c(Ip-Iq)，
wherein f iss d(qx-px,py-qy) Is spatial proximity, fr c(Ip-Iq) Is the similarity of the gray-scale values,
wherein, the superscript d indicates that the information comes from the depth map, the superscript c indicates that the information comes from the color map, the subscript s indicates the space domain, the subscript r indicates the pixel range domain, fs dRepresenting spatial proximity between pixels, qxIs the abscissa of pixel point q, qyIs the ordinate of pixel point q, q being the neighborhood pixel point of the central pixel, pxIs the abscissa of pixel p, p being the central pixel, pyIs the ordinate, f, of the pixel point pr c(Ip-Iq) Representing the gray value similarity between the pixel q point in the neighborhood and the pixel p point in the center, IpIs the gray value of p points, IqGray value of q point, σsIs a space domain parameter in the Euclidean distance formula, sigmarIs the standard deviation of the gaussian formula;
s4, placing the pixels with the priority greater than the threshold into a priority queue, and calculating the depth values of the pixels in the queue according to the sequence of the priorities from large to small;
s5, extracting main structure information and texture information from the color image through a total variation model;
s6, for the cavity area, firstly judging whether the cavity pixel is in the edge area, if it is in the non-edge area, adopting the local united bilateral filtering algorithm based on direction, the main calculation method is that the depth value of the depth value neighborhood pixel is weighted and averaged, the weight of each neighborhood pixel is obtained by the space proximity, the gray value similarity, the structure similarity and the texture similarity of the neighborhood pixel and the central pixel, if there is still the pixel which is not filled in the cavity non-edge area, returning to execute the step S4; if not, starting to fill the edge region, adopting a combined bilateral filtering algorithm based on the direction and fusing the structure and the texture information, wherein the repair neighborhood window of the non-edge region repair algorithm is self-adaptive;
s7, after finishing the repair of all the hole areas, repairing all the non-hole areas, wherein the repair of the non-hole areas and the non-edge areas of the non-hole areas is realized by a combined trilateral filtering algorithm of a fusion structure and texture information, and the filling method of the non-hole edge areas is the same as the filling method of the edge areas of the hole areas;
the method for filling the hollow pixel comprises the following steps:
1) and for the pixels in the edge area of the hollow part, filling by adopting a combined bilateral filtering algorithm based on the direction and fusing structure and texture information:
wherein D ispIs the depth value of the central pixel to be measured, the neighborhood phipComprises the following steps:
φp＝{q＝(qx,qy)|px-w≤qx≤px+w,py-w≤qy≤py+ w, with a filter window of (2w +1) × (2w +1), w being a positive integer, fs dRepresenting a pixelSpatial proximity of cells, fr c(Ip-Iq) Representing the gray value similarity between the pixel q point in the neighborhood and the pixel p point in the center, fr s(Sp-Sq) Representing the structural similarity between the pixel q point in the neighborhood and the pixel p point in the center, fr t(Tp-Tq) Representing the texture similarity between a pixel q point in the neighborhood and a central pixel p point;
2) and for pixels of non-edge areas of the hollow part, filling by adopting a direction-based local joint bilateral filtering algorithm:
neighborhood ΩpThe window size is self-adaptive and is determined by the distance from the hole pixel to the nearest edge in the same plane with the hole pixel;
3) and for pixels in non-edge areas of the non-cavity part, repairing and filling by adopting a combined trilateral filtering algorithm of a fusion structure and texture information:
wherein f isr d(D (p) -D (q)) is the depth value similarity of the p point and the q point;
4) and for pixels in the non-hollow part edge area, filling and repairing by adopting the same filling algorithm as the filling algorithm of the hollow part edge area.
2. The depth map hole repairing method based on hybrid filtering of claim 1, wherein in step S3, the method for calculating the priority m is:
m＝r1+λ×r2，λ>0，
wherein, λ is a parameter for determining the weight occupied by the support degree r1 and the reliability r2 when calculating the priority.
3. The depth map hole restoration method based on hybrid filtering according to claim 1, wherein the method for extracting main structure information and texture information is as follows:
the image I is decomposed into a linear combination of two parts, namely main structure information S and texture information T, which are expressed as follows:
I＝S+T,
definition of image structure information a fully variant model:
an energy formula is established for the image:
wherein omega is the area of the image,
let main structure information S to be output be as close to original image I as possible, λ 1 × TV (S) is TV regularization term, λ 1 is regularization parameter, weights of the first term and the second term in the balance formula,
the texture information in the image is obtained by the formula T-I-S,
based on the solved main structure information S and texture information T, for each pixel point of the cavity region, the structure similarity and the texture similarity between the pixel point and the neighborhood pixel point are obtained, and the specific method comprises the following steps:
wherein q is a neighborhood pixel of the central pixel, p is the central pixel, fr s(Sp-Sq) Representing the structural similarity between the pixel q point in the neighborhood and the pixel p point in the center, SpMain structural information, S, representing the center pixel point pqMain structural information representing neighborhood pixel q, fr t(Tp-Tq) Representing the texture similarity, T, between the pixel q point in the neighborhood and the pixel p point in the centerpTexture information, T, representing the center pixel pqTexture information, σ, representing neighborhood pixels qrIs the standard deviation of the gaussian equation.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810443580.XA CN108648221B (en) | 2018-05-10 | 2018-05-10 | Depth map hole repairing method based on hybrid filtering |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810443580.XA CN108648221B (en) | 2018-05-10 | 2018-05-10 | Depth map hole repairing method based on hybrid filtering |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN108648221A CN108648221A (en) | 2018-10-12 |
| CN108648221B true CN108648221B (en) | 2021-09-17 |
Family
ID=63754436
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810443580.XA Active CN108648221B (en) | 2018-05-10 | 2018-05-10 | Depth map hole repairing method based on hybrid filtering |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108648221B (en) |
Families Citing this family (19)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109636732B (en) * | 2018-10-24 | 2023-06-23 | 深圳先进技术研究院 | Hole repairing method of depth image and image processing device |
| CN109712067B (en) * | 2018-12-03 | 2021-05-28 | 北京航空航天大学 | Virtual viewpoint drawing method based on depth image |
| CN109685732B (en) * | 2018-12-18 | 2023-02-17 | 重庆邮电大学 | High-precision depth image restoration method based on boundary capture |
| CN111369611B (en) * | 2018-12-25 | 2023-06-02 | 曜科智能科技（上海）有限公司 | Image pixel depth value optimization method, device, equipment and storage medium thereof |
| CN111383185B (en) * | 2018-12-29 | 2023-09-22 | 海信集团有限公司 | Hole filling method based on dense parallax map and vehicle-mounted equipment |
| CN110599513B (en) * | 2019-09-04 | 2022-02-11 | 南京邮电大学 | Binocular vision image edge detection and target tracking method |
| CN110866882B (en) * | 2019-11-21 | 2021-09-07 | 湖南工程学院 | Layered joint bilateral filtering depth map repairing method based on depth confidence |
| CN111179195B (en) * | 2019-12-27 | 2023-05-30 | 西北大学 | Depth image cavity filling method and device, electronic equipment and storage medium thereof |
| CN111044988A (en) * | 2020-01-06 | 2020-04-21 | 青梧桐有限责任公司 | Fine processing method for laser radar scanning house type graph |
| US11074708B1 (en) | 2020-01-06 | 2021-07-27 | Hand Held Products, Inc. | Dark parcel dimensioning |
| CN111311521A (en) * | 2020-03-12 | 2020-06-19 | 京东方科技集团股份有限公司 | Image restoration method and device and electronic equipment |
| CN111986124B (en) * | 2020-09-07 | 2024-05-28 | 凌云光技术股份有限公司 | Filling method and device for missing pixels of depth image |
| CN112651897B (en) * | 2020-12-30 | 2024-05-03 | 成都星时代宇航科技有限公司 | Pixel repairing method, device, electronic equipment and computer readable storage medium |
| CN113242419A (en) * | 2021-04-30 | 2021-08-10 | 电子科技大学成都学院 | 2D-to-3D method and system based on static building |
| CN113763273B (en) * | 2021-09-07 | 2022-08-12 | 合肥的卢深视科技有限公司 | Face complementing method, electronic device and computer readable storage medium |
| CN113935914A (en) * | 2021-10-08 | 2022-01-14 | 北京的卢深视科技有限公司 | Depth image restoration method, electronic device and storage medium |
| CN114418902B (en) * | 2022-03-31 | 2022-07-08 | 深圳市华汉伟业科技有限公司 | Image restoration method and computer-readable storage medium |
| CN116721143B (en) * | 2023-08-04 | 2023-10-20 | 南京诺源医疗器械有限公司 | Depth information processing device and method for 3D medical image |
| CN117314792B (en) * | 2023-11-27 | 2024-04-19 | 广州市大湾区虚拟现实研究院 | Depth image restoration method based on RGB image edge capture |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103310420A (en) * | 2013-06-19 | 2013-09-18 | 武汉大学 | Method and system for repairing color image holes on basis of texture and geometrical similarities |
| CN103955891A (en) * | 2014-03-31 | 2014-07-30 | 中科创达软件股份有限公司 | Image restoration method based on block matching |
| CN104601972A (en) * | 2014-12-17 | 2015-05-06 | 清华大学深圳研究生院 | Method for synthesizing free viewpoint by image inpainting |
| CN106485672A (en) * | 2016-09-12 | 2017-03-08 | 西安电子科技大学 | Improved Block- matching reparation and three side Steerable filter image enchancing methods of joint |
| CN106851248A (en) * | 2017-02-13 | 2017-06-13 | 浙江工商大学 | Based on openness image repair priority computational methods |
| CN106920263A (en) * | 2017-03-10 | 2017-07-04 | 大连理工大学 | Undistorted integration imaging 3 D displaying method based on Kinect |
-
2018
- 2018-05-10 CN CN201810443580.XA patent/CN108648221B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103310420A (en) * | 2013-06-19 | 2013-09-18 | 武汉大学 | Method and system for repairing color image holes on basis of texture and geometrical similarities |
| CN103955891A (en) * | 2014-03-31 | 2014-07-30 | 中科创达软件股份有限公司 | Image restoration method based on block matching |
| CN104601972A (en) * | 2014-12-17 | 2015-05-06 | 清华大学深圳研究生院 | Method for synthesizing free viewpoint by image inpainting |
| CN106485672A (en) * | 2016-09-12 | 2017-03-08 | 西安电子科技大学 | Improved Block- matching reparation and three side Steerable filter image enchancing methods of joint |
| CN106851248A (en) * | 2017-02-13 | 2017-06-13 | 浙江工商大学 | Based on openness image repair priority computational methods |
| CN106920263A (en) * | 2017-03-10 | 2017-07-04 | 大连理工大学 | Undistorted integration imaging 3 D displaying method based on Kinect |
Non-Patent Citations (6)
| Title |
|---|
| "Depth map enhancement method based on joint bilateral filter";Yinghua Shen 等;《2014 7th International Congress on Image and Signal Processing》;20150108;第153-158页 * |
| "Directional Joint Bilateral Filter for Depth Images";Anh Vu Le 等;《Sensors (Basel)》;20140626;第14卷(第7期);第11363-11364页、第11369-11370页，表1，公式(1)-(10) * |
| "Structure Extraction from Texture via Relative Total Variation";Li Xu 等;《ACM Transaction on Graphics》;20121130;第31卷(第6期);第171:1-171:11页 * |
| "Structure guided fusion for depth map inpainting";Fei Qi 等;《Pattern Recognition Letters》;20130131;第34卷(第1期);第71-72页 * |
| "基于超像素分割的深度图像修复算法";胡天佑 等;《光电子·激光》;20161031;第27卷(第10期);第1120-1127页 * |
| "纹理图像的结构提取方法研究";陈丽;《中国优秀硕士学位论文全文数据库信息科技辑》;20160215;第2016年卷(第2期);第5-7页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN108648221A (en) | 2018-10-12 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108648221B (en) | Depth map hole repairing method based on hybrid filtering | |
| CN107833270B (en) | Real-time object three-dimensional reconstruction method based on depth camera | |
| CN105069804B (en) | Threedimensional model scan rebuilding method based on smart mobile phone | |
| CN110060283B (en) | Multi-measure semi-global dense matching method | |
| CN103473743B (en) | A kind of method obtaining image depth information | |
| CN106803275A (en) | Estimated based on camera pose and the 2D panoramic videos of spatial sampling are generated | |
| CN110211169B (en) | Reconstruction method of narrow baseline parallax based on multi-scale super-pixel and phase correlation | |
| CN110536142B (en) | Interframe interpolation method for non-rigid image sequence | |
| CN108596975A (en) | A kind of Stereo Matching Algorithm for weak texture region | |
| CN110866882B (en) | Layered joint bilateral filtering depth map repairing method based on depth confidence | |
| CN111179195B (en) | Depth image cavity filling method and device, electronic equipment and storage medium thereof | |
| CN111462030A (en) | Multi-image fused stereoscopic set vision new angle construction drawing method | |
| CN111369660A (en) | Seamless texture mapping method for three-dimensional model | |
| CN104079800A (en) | Shaking preventing method for video image in video surveillance | |
| CN110245199A (en) | A kind of fusion method of high inclination-angle video and 2D map | |
| CN111179333B (en) | Defocus blur kernel estimation method based on binocular stereo vision | |
| CN107155100B (en) | A kind of solid matching method and device based on image | |
| CN108805841B (en) | Depth map recovery and viewpoint synthesis optimization method based on color map guide | |
| CN109493426B (en) | Monocular imaging-based three-dimensional reconstruction method and system for blast furnace charge level | |
| KR101921608B1 (en) | Apparatus and method for generating depth information | |
| CN111369435B (en) | Color image depth up-sampling method and system based on self-adaptive stable model | |
| CN112802186A (en) | Dynamic scene real-time three-dimensional reconstruction method based on binarization characteristic coding matching | |
| CN112991504A (en) | Improved method for filling holes based on TOF camera three-dimensional reconstruction | |
| CN113034681A (en) | Three-dimensional reconstruction method and device for spatial plane relation constraint | |
| CN109218706B (en) | Method for generating stereoscopic vision image from single image |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |