CN105139355A - Method for enhancing depth images - Google Patents

Abstract

The invention provides a method for enhancing depth images, and relates to the image processing field. The method comprises: the step 1: shooting a depth image for an object through a depth camera, and shooting a color image for the object through a color camera which is synchronous with the depth camera; the step 2: constructing an energy function according to the depth image and the color image; the step 3: minimizing the energy function by using a least square method, and obtaining the optimal value of the enhanced depth value of the pixel in the depth image through calculation; and the step 4: outputting the optimal value of the enhanced depth value of the pixel in the depth image. The method for enhancing depth images can obtain a more accurate depth value of image pixel through calculation.

Description

A kind of Enhancement Method of depth image

Technical field

The present invention relates to image processing field, refer to a kind of Enhancement Method of depth image especially.

Background technology

Depth map represents the distance of each point distance video camera in scene.Obtaining scene depth figure is one of vital task of computer vision system.Existing depth measuring devices (such as Microsoft Kinect) limit by its precision, and the depth map of catching comprises noise and damaged area usually.The time complexity of traditional depth map acquisition methods based on solid geometry theory is high, and have dependence to the texture of scene, range of application is restricted.

Based on the depth map acquisition methods of hardware sensor, do not rely on scene texture, there is high speed, flexibly feature; Especially selling of nearly 2 years low price depth transducers, make these class methods more be tending towards practical, derivative application emerges in an endless stream.But be subject to the restriction of hardware precision and built-in algorithms, depth transducer can only obtain the low resolution depth map comprising noise and damaged area at a low price.

In order to strengthen low-quality depth map, existing method comprises: use synchronization from the coloured image of similar viewing angles as a reference, use visual angle mapping algorithm to be alignd with coloured image by depth map, and utilize filtering or global optimization strategy, improve the resolution of depth map.

The people such as Yang proposed in 2007 first on low resolution depth map, to construct a cost model, were then estimated by least consume and sub-pixel derivation algorithm increases depth map resolution iteratively.The people such as Park proposed in 2011 to utilize the depth map of color, marginal information and the linear interpolation extracted from coloured image to make reference, and be optimized, solve high-resolution depth map by markov random file theory.

There is open defect in above two kinds of methods:

First, they suppose that used coloured image can align with depth map in perfection, and this and actual observation are inconsistent.Map and the impact of the noise of depth transducer own and measuring error by camera lens distortion, visual angle, the coloured image after alignment and depth map there are differences in object edge observed reading.Use such coloured image can produce misguidance to the enhancing algorithm of depth map as clue.

Secondly, above algorithm hypothesis original depth-map is measured complete, and there is not large area damaged area, this is not inconsistent with actual observation yet.According to the difference of sensor measurement principle and object material, depth map presents corresponding breakage.Such as, using the depth transducer of Infrared survey principle cannot the low region of stably measured infrared reflecting, can there is breakage in this region in depth map.In addition, visual angle mapping algorithm can leave slight crack by object edge in depth map.

Summary of the invention

The technical problem to be solved in the present invention is, provides a kind of Enhancement Method of depth image, can calculate the depth value of the better image pixel of accuracy.

Described method comprises:

Step one, takes the depth image of an object by depth camera; And the coloured image of described object is taken by the colour imagery shot synchronous with described depth camera;

Step 2, according to described depth image and described coloured image, structure energy function;

Step 3, minimizes described energy function by least square method, calculates the optimal value of the depth value after the enhancing of pixel in described depth image;

Step 4, exports the optimal value of the depth value after the enhancing of pixel in described depth image.

Described energy function is:

E(D')＝E _d(D')+λ _hE _h(D')+λ _sE _s(D')

Wherein, E (D') is energy function; D' is the depth value after the enhancing of pixel in depth image; E _d(D') be data constraint item, E _h(D') be the damaged area bound term of depth image; E _s(D') be Smoothing Constraint item; λ _hfor the weight of the damaged area bound term of depth image; λ _sit is the weight of Smoothing Constraint item.

Described data constraint item is defined by following formula:

$E_d\left(D^{\′}\right)=\underset{x}{\Σ}{\left(C\right(x\left)\right)}^2{\left(D^{\′}\right(x)-D(x\left)\right)}^2;$

Wherein, x is the sequence number of pixel in depth image; The existing depth value that D (x) is pixel x in depth image, D'(x) be depth value after the enhancing of pixel x in depth image, C (x) is the degree of confidence of the existing depth value of pixel x in depth image:

Described damaged area bound term is defined by following formula:

$E_h\left(D^{\′}\right)=\underset{x\∈{\mathrm{\Ω}}_H}{\Σ}{\left(D^{\′}\right(x)-\tilde{D}(x\left)\right)}^2;$

Wherein, Ω _hthe pixel set of damaged area in depth image, it is the degree of depth initial value of pixel x in the damaged area of depth image;

Described Smoothing Constraint item is defined by following formula:

$E_s\left(D^{\′}\right)=\underset{x}{\Σ}\underset{x_i\∈N_1\left(x\right)}{\Σ}w(x_i,x){\left(D^{\′}\right(x_i)-D^{\′}(x\left)\right)}^2$

Wherein, N ₁x () is the 1 rank neighbour of pixel x;

X _iit is a pixel in the single order neighbour of pixel x; I is the index of pixel;

D'(x _i) be pixel x in depth image _ienhancing after depth value;

W (x _i, x) be weight coefficient;

Weight coefficient w (x _i, x) be color weight w _i(x _i, x), edge weights w _e(x _i, x) with segmentation weight w _s(x _i, the x) product of three;

$w_i(x_i,x)=\exp (-\frac{{\left(I\right(x_i)-I(x\left)\right)}^2}{2{\mathrm{\σ}}_i^2})$

Ω _eit is the pixel set of the fringe region of coloured image;

S (x) is the mark of the block that the pixel x of coloured image is corresponding; S (x _i) be the mark of the block that in coloured image, pixel is corresponding;

I (x) is the color value of pixel x in coloured image; I (x _i) be pixel x in coloured image _icolor value

σ _iit is the first function parameter.

Before described step 3, described method also comprises:

Card Buddhist nun Canny edge detection process is carried out to described coloured image, obtains the pixel set omega of the fringe region of described coloured image _e.

Before described step 3, described method also comprises:

Use average drifting MeanShift image segmentation, be super-pixel by described color images, obtain mark S (x) of the block that pixel x is corresponding in described coloured image.

Before described step 3, described method also comprises:

To the damaged area Ω of described depth image _h, use depth map restore design to fill, solve the degree of depth initial value of pixel x in the damaged area of described depth image.

Before described step 3, described method also comprises:

According to described depth image and described coloured image, calculate the degree of confidence C (x) of the existing depth value of pixel x in described depth image.

Described according to described depth image and described coloured image, the step calculating the degree of confidence C (x) of the existing depth value of pixel x in described depth image comprises:

Wherein, Ω _hit is the pixel set of damaged area in depth image; Ω _dbe in depth image do not belong to damaged area and distance damaged area is no more than the pixel set of d; D is the first distance threshold;

C _fx () is defined by following formula:

$C_f\left(x\right)=\frac{1}{\mathrm{\η}}\underset{x_i\∈N_r,x_i\≠x}{\Σ}{w}_f(x_i,x)c_f(x_i,x)$

$w_f(x_i,x)=\exp (-\frac{{\left(I\right(x_i)-I(x\left)\right)}^2}{2{\mathrm{\σ}}_I^2})$

$c_f(x_i,x)=\exp (-\frac{{\left(D\right(x_i)-D(x\left)\right)}^2}{2{\mathrm{\σ}}_D^2})$

N _rit is the set that distance pixel x is less than or equal to the pixel of r; R is second distance threshold value;

I (x) is the color value of pixel x in coloured image; I (x _i) be pixel x in coloured image _icolor value; D (x) is the existing depth value of pixel x in depth image, D (x _i) be pixel x in depth image _iexisting depth value; η is w _f(x _i, summation x), _{σ I}and σ _dthe second function parameter and the 3rd function parameter respectively;

C _hx () is defined by following formula:

$C_h\left(x\right)=\frac{1}{{\mathrm{\η}}^{\′}}\underset{x_i\∈N_r,x_i\≠x}{\Σ}{w}_h(x_i,x)c_h(x_i,x)$

$w_h(x_i,x)=\exp (-\frac{{\left(D\right(x_i)-D(x\left)\right)}^2}{2{{\mathrm{\σ}}^{\′}}_D^2})$

$c_h(x_i,x)=\exp (-\frac{{\left(I\right(x_i)-I(x\left)\right)}^2}{2{{\mathrm{\σ}}^{\′}}_I^2})$

Wherein, η ' is w _h(x _i, summation x), σ ' _iwith σ ' _dthe 4th function parameter and the 5th function parameter respectively.

Before described step 3, described method also comprises:

Spatial registration is carried out to the pixel of described depth image and the pixel of described coloured image;

Corresponding relation between the color value obtaining same pixel x in the depth value of pixel x in described depth image and described coloured image.

The beneficial effect of technique scheme of the present invention is as follows:

In the present invention, use markov random file theory to be optimized depth map, obtain the depth map that a pair is more complete, more credible, more consistent with coloured image observed result.Depth map after enhancing can be used as input data, is applied in more senior computer vision, augmented reality task.

Accompanying drawing explanation

Fig. 1 is the schematic flow sheet of the Enhancement Method of the depth image shown in one embodiment of the invention;

Fig. 2 is the schematic flow sheet of the Enhancement Method of the depth image shown in one embodiment of the invention;

Fig. 3 is the principle schematic of the Enhancement Method of depth image of the present invention.

Embodiment

For making the technical problem to be solved in the present invention, technical scheme and advantage clearly, be described in detail below in conjunction with the accompanying drawings and the specific embodiments.

As shown in Figure 1, the Enhancement Method of described depth image, comprising:

Step 11, takes the depth image of an object by depth camera; And the coloured image of described object is taken by the colour imagery shot synchronous with described depth camera;

Step 12, according to described depth image and described coloured image, structure energy function;

Step 13, minimizes described energy function by least square method, calculates the optimal value of the depth value after the enhancing of pixel in described depth image;

Step 14, exports the optimal value of the depth value after the enhancing of pixel in described depth image.

Wherein, described energy function is:

E(D')＝E _d(D')+λ _hE _h(D')+λ _sE _s(D')

Wherein, E (D') is energy function; D' is the depth value after the enhancing of pixel in depth image; E _d(D') be data constraint item, for retraining the difference between D ' and D.E _h(D') be the damaged area bound term of depth image; E _s(D') be Smoothing Constraint item; λ _hfor the weight of the damaged area bound term of depth image; λ _sit is the weight of Smoothing Constraint item.

Wherein, described data constraint item is defined by following formula:

$E_d\left(D^{\′}\right)=\underset{x}{\Σ}{\left(C\right(x\left)\right)}^2{\left(D^{\′}\right(x)-D(x\left)\right)}^2;$

Wherein, x is the sequence number of pixel in depth image; The existing depth value that D (x) is pixel x in depth image, D'(x) be depth value after the enhancing of pixel x in depth image, C (x) is the degree of confidence of the existing depth value of pixel x in depth image:

Described damaged area bound term is defined by following formula:

$E_h\left(D^{\′}\right)=\underset{x\∈{\mathrm{\Ω}}_H}{\Σ}{\left(D^{\′}\right(x)-\tilde{D}(x\left)\right)}^2;$

Wherein, Ω _hthe pixel set of damaged area in depth image, it is the degree of depth initial value of pixel x in the damaged area of depth image;

Described Smoothing Constraint item is defined by following formula:

$E_s\left(D^{\′}\right)=\underset{x}{\Σ}\underset{x_i\∈N_1\left(x\right)}{\Σ}w(x_i,x){\left(D^{\′}\right(x_i)-D^{\′}(x\left)\right)}^2$

Wherein, N ₁x () is the 1 rank neighbour of pixel x;

X _iit is a pixel in the single order neighbour of pixel x; I is the index of pixel;

D'(x _i) be pixel x in depth image _ienhancing after depth value;

W (x _i, x) be weight coefficient;

Weight coefficient w (x _i, x) be color weight w _i(x _i, x), edge weights w _e(x _i, x) with segmentation weight w _s(x _i, the x) product of three;

$w_i(x_i,x)=\exp (-\frac{{\left(I\right(x_i)-I(x\left)\right)}^2}{2{\mathrm{\σ}}_i^2})$

Ω _eit is the pixel set of the fringe region of coloured image;

S (x) is the mark of the block that the pixel x of coloured image is corresponding; S (x _i) be the mark of the block that in coloured image, pixel is corresponding;

I (x) is the color value of pixel x in coloured image; I (x _i) be pixel x in coloured image _icolor value

σ _ibe the first function parameter, carry out value by empirical value, can 30 be taken as.

Wherein, before described step 13, described method also comprises:

Step 131, carries out card Buddhist nun Canny edge detection process to described coloured image, obtains the pixel set omega of the fringe region of described coloured image _e.

Wherein, before described step 13, described method also comprises:

Step 132, uses average drifting MeanShift image segmentation, is super-pixel, obtains mark S (x) of the block that pixel x is corresponding in described coloured image by described color images.

Wherein, before described step 13, described method also comprises:

Step 133, to the damaged area Ω of described depth image _h, use depth map restore design to fill, solve the degree of depth initial value of pixel x in the damaged area of described depth image.

Wherein, before described step 13, described method also comprises:

Step 134, according to described depth image and described coloured image, calculates the degree of confidence C (x) of the existing depth value of pixel x in described depth image.

Step 134 is specially according to following formulae discovery:

Wherein, Ω _hit is the pixel set of damaged area in depth image; Ω _dbe in depth image do not belong to damaged area and distance damaged area is no more than the pixel set of d; D is the first distance threshold;

C _fx () is defined by following formula:

$C_f\left(x\right)=\frac{1}{\mathrm{\η}}\underset{x_i\∈N_r,x_i\≠x}{\Σ}{w}_f(x_i,x)c_f(x_i,x)$

$w_f(x_i,x)=\exp (-\frac{{\left(I\right(x_i)-I(x\left)\right)}^2}{2{\mathrm{\σ}}_I^2})$

$c_f(x_i,x)=\exp (-\frac{{\left(D\right(x_i)-D(x\left)\right)}^2}{2{\mathrm{\σ}}_D^2})$

N _rit is the set that distance pixel x is less than or equal to the pixel of r; R is second distance threshold value;

I (x) is the color value of pixel x in coloured image; I (x _i) be pixel x in coloured image _icolor value; D (x) is the existing depth value of pixel x in depth image, D (x _i) be pixel x in depth image _iexisting depth value; η is w _f(x _i, summation x), _{σ I}and σ _dthe second function parameter and the 3rd function parameter respectively; Carry out value by empirical value, 30 and 10 can be taken as respectively;

C _hx () is defined by following formula:

$C_h\left(x\right)=\frac{1}{{\mathrm{\η}}^{\′}}\underset{x_i\∈N_r,x_i\≠x}{\Σ}{w}_h(x_i,x)c_h(x_i,x)$

$w_h(x_i,x)=\exp (-\frac{{\left(D\right(x_i)-D(x\left)\right)}^2}{2{{\mathrm{\σ}}^{\′}}_D^2})$

$c_h(x_i,x)=\exp (-\frac{{\left(I\right(x_i)-I(x\left)\right)}^2}{2{{\mathrm{\σ}}^{\′}}_I^2})$

Wherein, η ' is w _h(x _i, summation x), σ ' _iwith σ ' _dbe the 4th function parameter and the 5th function parameter respectively, carry out value by empirical value, 30 and 10 can be taken as respectively.

Wherein, before described step 13, described method also comprises:

Step 135, carries out spatial registration to the pixel of described depth image and the pixel of described coloured image; Corresponding relation between the color value obtaining same pixel x in the depth value of pixel x in described depth image and described coloured image.

In above-described embodiment, disclose and a kind ofly utilize the hint information in coloured image to carry out the method for depth map of wild phase like visual angle.The coloured image that the present invention uses a pair visual angle similar and depth map, utilize the correlativity of regional area color and the degree of depth, the degree of confidence of depth map is estimated, and utilize the color of this degree of confidence and coloured image, edge and segmentation result as guiding clue, use markov random file theory to be optimized depth map, finally obtain the depth map that a pair is more complete, more credible, more consistent with coloured image observed result.Depth map after enhancing can be used as input data, is applied in more senior computer vision, augmented reality task.

As shown in Figure 2, the Enhancement Method of described depth image, comprising:

Step 21, takes the depth image of an object by depth camera; And the coloured image of described object is taken by the colour imagery shot synchronous with described depth camera;

Step 22, carries out spatial registration to the pixel of described depth image and the pixel of described coloured image; Corresponding relation between the color value obtaining same pixel x in the depth value of pixel x in described depth image and described coloured image.

Step 23, carries out card Buddhist nun Canny edge detection process to described coloured image, obtains the pixel set omega of the fringe region of described coloured image _e.

Step 24, uses average drifting MeanShift image segmentation, is super-pixel, obtains mark S (x) of the block that pixel x is corresponding in described coloured image by described color images.

Step 25, according to described depth image and described coloured image, calculates the degree of confidence C (x) of the existing depth value of pixel x in described depth image.

Step 26, to the damaged area Ω of described depth image _h, use depth map restore design to fill, solve the degree of depth initial value of pixel x in the damaged area of described depth image.

Step 27, according to described depth image and described coloured image, structure energy function;

Step 28, minimizes described energy function by least square method, calculates the optimal value of the depth value after the enhancing of pixel in described depth image;

Step 29, exports the optimal value of the depth value after the enhancing of pixel in described depth image.

Wherein, described energy function is:

E(D')＝E _d(D')+λ _hE _h(D')+λ _sE _s(D')

Wherein, E (D') is energy function; D' is the depth value after the enhancing of pixel in depth image; E _d(D') be data constraint item, for retraining the difference between D ' and D.E _h(D') be the damaged area bound term of depth image; E _s(D') be Smoothing Constraint item; λ _hfor the weight of the damaged area bound term of depth image; λ _sit is the weight of Smoothing Constraint item.

Wherein, described data constraint item is defined by following formula:

$E_d\left(D^{\′}\right)=\underset{x}{\Σ}{\left(C\right(x\left)\right)}^2{\left(D^{\′}\right(x)-D(x\left)\right)}^2;$

Wherein, x is the sequence number of pixel in depth image; The existing depth value that D (x) is pixel x in depth image, D'(x) be depth value after the enhancing of pixel x in depth image, C (x) is the degree of confidence of the existing depth value of pixel x in depth image:

Described damaged area bound term is defined by following formula:

$E_h\left(D^{\′}\right)=\underset{x\∈{\mathrm{\Ω}}_H}{\Σ}{\left(D^{\′}\right(x)-\tilde{D}(x\left)\right)}^2;$

Wherein, Ω _hthe pixel set of damaged area in depth image, it is the degree of depth initial value of pixel x in the damaged area of depth image;

Described Smoothing Constraint item is defined by following formula:

$E_s\left(D^{\′}\right)=\underset{x}{\Σ}\underset{x_i\∈N_1\left(x\right)}{\Σ}w(x_i,x){\left(D^{\′}\right(x_i)-D^{\′}(x\left)\right)}^2$

Wherein, N ₁x () is the 1 rank neighbour of pixel x;

X _iit is a pixel in the single order neighbour of pixel x; I is the index of pixel;

D'(x _i) be pixel x in depth image _ienhancing after depth value;

W (x _i, x) be weight coefficient;

Weight coefficient w (x _i, x) be color weight w _i(x _i, x), edge weights w _e(x _i, x) with segmentation weight w _s(x _i, the x) product of three;

$w_i(x_i,x)=\exp (-\frac{{\left(I\right(x_i)-I(x\left)\right)}^2}{2{\mathrm{\σ}}_i^2})$

Ω _eit is the pixel set of the fringe region of coloured image;

S (x) is the mark of the block that the pixel x of coloured image is corresponding; S (x _i) be the mark of the block that in coloured image, pixel is corresponding;

I (x) is the color value of pixel x in coloured image; I (x _i) be pixel x in coloured image _icolor value

σ _ibe the first function parameter, carry out value by empirical value, can 30 be taken as.

Thought of the present invention is below described.As shown in Figure 3, a kind ofly utilize the hint information in coloured image to carry out the method for depth map of wild phase like visual angle, based on color and depth consistency, the Enhancement Method of described depth map comprises the following steps:

Step 1, use depth camera shooting depth map to obtain depth data, and use takes synchronous coloured image with the colour imagery shot of depth camera registration, this results in impaired depth data and coloured image corresponding with it.

Step 2, according to the position relationship between colour imagery shot and depth camera, carries out pre-service to depth image, by position corresponding on depth image data-mapping to coloured image.

Step 3, carries out Boundary Detection and cutting operation to coloured image, extracts marginal information and the provincial characteristics of image.

Step 4, after the feature extracting coloured image, carries out " color--the degree of depth " consistency detection according to feature for the coloured image matched and depth data, calculates the degree of confidence of each depth value in original depth image data thus.

Step 5, for the leak part in original depth-map, uses easy depth map restore design to carry out filling up as initial value.

Step 6, according to the information such as image feature information, degree of confidence obtained, utilizes Markov random field theory building least energy function.

Step 7, solves energy equation, obtains the depth data after repairing.

In above-described embodiment, with coloured image as a reference, the depth map obtained from low side depth transducer is strengthened.The target strengthened not only comprises repairs the damaged area of depth map, and corrects " color-degree of depth " inconsistency produced because of measuring error and view transformation, obtains that align with coloured image, that marginal information is consistent depth map.Quoting markov random file theory for realizing this target, defining following energy function E (D'), and by minimizing the enhancing of this energy function realization to depth map:

E(D')＝E _d(D')+λ _hE _h(D')+λ _sE _s(D')(1)

Wherein, E _d(D') be data constraint item, each pixel value of the inferior quality depth map that it uses sensor to obtain is as constraint, and by inspection " color-degree of depth " consistance, confidence level estimation is carried out to each pixel in depth map, according to the power of the height definition data constraint of degree of confidence.E _h(D') be damaged area bound term, for repairing the large scale damaged area in depth map, it is tentatively repaired depth map by simple and easy digital picture restore design, and uses this reparation result as constraint.E _s(D') be Smoothing Constraint item, its uses the color of coloured image, edge and segmentation result as constraint, makes the depth value of color smooth region also level and smooth, and makes the edge of depth image and coloured image ensure to align.

Depth map after being repaired by the present invention is consistent with Color Image Edge, align accurately.Found by the comparison in different common data sets, the depth map average error after reparation of the present invention is only 0.32%, far below the reparation error using simple digital image repair algorithm average 7.52%.

Below embodiments of the invention are described.Said method comprising the steps of:

Step 1, uses common RGBD (red, green, blue, the degree of depth) camera (as Microsoft Kinect, Asus Xtion etc.) the shooting degree of depth/coloured image pair.Usual coloured image has higher resolution, and depth image resolution is low, and noise is serious and comprise more mistake.

Step 2, uses camera calibration technology to obtain camera interior and exterior parameter, and calculates the corresponding relation from depth map to coloured image based on this, the registration degree of depth and coloured image.The SDK (SDK (Software Development Kit)) of Kinect itself, with this function, therefore can directly use.Noting due to reasons such as deep errors, camera calibration errors, often there is certain error in the corresponding relation calculated by the method.

Step 3, the degree of confidence C (x) according to " color-degree of depth " consistance compute depth pixel:

Wherein, Ω _hthe pixel set of damaged area in depth map, Ω _dthe pixel set (d can be set to 7) itself not belonging to damaged area but be no more than d apart from damaged area, C _fx () is defined by following formula:

$C_f\left(x\right)=\frac{1}{\mathrm{\η}}\underset{x_i\∈N_r,x_i\≠x}{\Σ}{w}_f(x_i,x)c_f(x_i,x)---\left(4\right)$

$w_f(x_i,x)=\exp (-\frac{{\left(I\right(x_i)-I(x\left)\right)}^2}{2{\mathrm{\σ}}_I^2})---\left(5\right)$

$c_f(x_i,x)=\exp (-\frac{{\left(D\right(x_i)-D(x\left)\right)}^2}{2{\mathrm{\σ}}_D^2})---\left(6\right)$

Wherein, N _rbe the set (r can be set to 7) that distance pixel x is no more than the pixel of r, I (x) and D (x) is color value and the depth value of pixel x, and η is w in formula (4) _f(x _i, summation x), σ _iand σ _d15 gray scale differences and 30 millimeters can be set to respectively.

C in formula (3) _hx () is defined by following formula:

$C_h\left(x\right)=\frac{1}{{\mathrm{\η}}^{\′}}\underset{x_i\∈N_r,x_i\≠x}{\Σ}{w}_h(x_i,x)c_h(x_i,x)---\left(7\right)$

$w_h(x_i,x)=\exp (-\frac{{\left(D\right(x_i)-D(x\left)\right)}^2}{2{{\mathrm{\σ}}^{\′}}_D^2})---\left(8\right)$

$c_h(x_i,x)=\exp (-\frac{{\left(I\right(x_i)-I(x\left)\right)}^2}{2{{\mathrm{\σ}}^{\′}}_I^2})---\left(9\right)$

Wherein, the meaning of every symbol is identical with formula (5) (6), and η ' is w in formula (7) _h(x _i, summation x), σ ' _iwith σ ' _d15 gray scale differences and 15 millimeters can be set to respectively, can set based on experience value.

Step 4, to the damaged area Ω of depth map _h(namely depth camera fails correctly to obtain the region of the degree of depth), uses the technology of Simple depth figure reparation to carry out fillings acquisition degree of depth initial value.

Simple depth figure restore design using damaged area borderline pixel as Seed Points, according to Seed Points priority from high to low, to center, damaged area iteration diffusion mode carry out region reparation.The priority of Seed Points defines according to following formula:

Wherein, T (x _s) for Seed Points is to the distance on border, damaged area, D (x _s) be the depth value of Seed Points, D _maxfor the maximum depth value of scene, for the Laplace transform value of coloured image, λ ₁and λ ₂-10 and 7 can be set to respectively.

Step 5, to coloured image, adopts the set omega of all image borders of Canny edge detection calculation _e; And use average drifting (MeanShift) image partition method to be super-pixel by Iamge Segmentation, and remember that S (x) is for cut zone mark corresponding to pixel x.

Step 6, structure markov energy function.This energy function is by data constraint item E _d(D'), damaged area bound term E _hand Smoothing Constraint item E (D') _s(D') three compositions.Data constraint item in energy function is defined as follows:

$E_d\left(D^{\′}\right)=\underset{x}{\Σ}{\left(C\right(x\left)\right)}^2{\left(D^{\′}\right(x)-D(x\left)\right)}^2---\left(2\right)$

Wherein, D (x) is the existing depth value of pixel x in depth map, D'(x) be strengthen after depth value, C (x) is the degree of confidence of the existing depth value of pixel x:

Damaged area bound term in energy function is defined by following formula:

$E_h\left(D^{\′}\right)=\underset{x\∈{\mathrm{\Ω}}_H}{\Σ}{\left(D^{\′}\right(x)-\tilde{D}(x\left)\right)}^2---\left(10\right)$

Wherein, Ω _hthe pixel set of damaged area in depth map, it is the depth value that Simple depth figure restore design solves.

Energy function Smoothing Constraint item is defined by following formula:

$E_s\left(D^{\′}\right)=\underset{x}{\Σ}\underset{x_i\∈N_1\left(x\right)}{\Σ}w(x_i,x){\left(D^{\′}\right(x_i)-D^{\′}(x\left)\right)}^2---\left(12\right)$

Wherein, N ₁x () is the 1 rank neighbour of pixel x, weight coefficient w (x _i, x) be color weight w _i(x _i, x), edge weights w _e(x _i, x) with segmentation weight w _s(x _i, the x) product of three, they define according to following formula respectively:

$w_i(x_i,x)=\exp (-\frac{{\left(I\right(x_i)-I(x\left)\right)}^2}{2{\mathrm{\σ}}_i^2})---\left(13\right)$

Wherein, Ω _ebe the edge aggregation of coloured image, S (x) is the mark of color images block, σ _i10 gray scale differences can be set to.

Step 7, minimizes following energy function by least square method, obtain D'(x) optimum solution as Output rusults:

E(D')＝E _d(D')+λ _hE _h(D')+λ _sE _s(D')

Wherein, λ _hand λ _sbe the weight of damaged area bound term and Smoothing Constraint item respectively, 0.1 and 0.2 can be set to respectively.

The above is the preferred embodiment of the present invention; it should be pointed out that for those skilled in the art, under the prerequisite not departing from principle of the present invention; can also make some improvements and modifications, these improvements and modifications also should be considered as protection scope of the present invention.

Claims (9)

1. an Enhancement Method for depth image, is characterized in that, comprising:

Step one, takes the depth image of an object by depth camera; And the coloured image of described object is taken by the colour imagery shot synchronous with described depth camera;

Step 2, according to described depth image and described coloured image, structure energy function;

Step 3, minimizes described energy function by least square method, calculates the optimal value of the depth value after the enhancing of pixel in described depth image;

Step 4, exports the optimal value of the depth value after the enhancing of pixel in described depth image.

2. method according to claim 1, is characterized in that, described energy function is:

E(D')＝E _d(D')+λ _hE _h(D')+λ _sE _s(D')

Wherein, E (D') is energy function; D' is the depth value after the enhancing of pixel in depth image; E _d(D') be data constraint item, E _h(D') be the damaged area bound term of depth image; E _s(D') be Smoothing Constraint item; λ _hfor the weight of the damaged area bound term of depth image; λ _sit is the weight of Smoothing Constraint item.

3. method according to claim 2, is characterized in that,

Described data constraint item is defined by following formula:

$E_d\left(D^{\′}\right)=\underset{x}{\Σ}{\left(C\left(x\right)\right)}^2{(D^{\′}\left(x\right)-D\left(x\right))}^2;$

Wherein, x is the sequence number of pixel in depth image; The existing depth value that D (x) is pixel x in depth image, D'(x) be depth value after the enhancing of pixel x in depth image, C (x) is the degree of confidence of the existing depth value of pixel x in depth image:

Described damaged area bound term is defined by following formula:

$E_h\left(D^{\′}\right)=\underset{x\∈{\mathrm{\Ω}}_H}{\Σ}{(D^{\′}\left(x\right)-\tilde{D}\left(x\right))}^2;$

Wherein, Ω _hthe pixel set of damaged area in depth image, it is the degree of depth initial value of pixel x in the damaged area of depth image;

Described Smoothing Constraint item is defined by following formula:

$E_s\left(D^{\′}\right)=\underset{x}{\Σ}\underset{x_i\∈N_1\left(x\right)}{\Σ}w\left(x_i,x\right){\left(D^{\′}\left(x_i\right)-D^{\′}\left(x\right)\right)}^2$

Wherein, N ₁x () is the 1 rank neighbour of pixel x;

X _iit is a pixel in the single order neighbour of pixel x; I is the index of pixel;

D'(x _i) be pixel x in depth image _ienhancing after depth value;

W (x _i, x) be weight coefficient;

Weight coefficient w (x _i, x) be color weight w _i(x _i, x), edge weights w _e(x _i, x) with segmentation weight w _s(x _i, the x) product of three;

$w_i(x_i,x)=\exp \left(-\frac{{\left(I\right(x_i)-I(x\left)\right)}^2}{2{\mathrm{\σ}}_i^2}\right)$

Ω _eit is the pixel set of the fringe region of coloured image;

S (x) is the mark of the block that the pixel x of coloured image is corresponding; S (x _i) be in coloured image pixel to x _ithe mark of the block of answering;

I (x) is the color value of pixel x in coloured image; I (x _i) be pixel x in coloured image _icolor value

σ _iit is the first function parameter.

4. method according to claim 3, is characterized in that, before described step 3, described method also comprises:

Card Buddhist nun Canny edge detection process is carried out to described coloured image, obtains the pixel set omega of the fringe region of described coloured image _e.

5. method according to claim 3, is characterized in that, before described step 3, described method also comprises:

Use average drifting MeanShift image segmentation, be super-pixel by described color images, obtain mark S (x) of the block that pixel x is corresponding in described coloured image.

6. method according to claim 3, is characterized in that, before described step 3, described method also comprises:

To the damaged area Ω of described depth image _h, use depth map restore design to fill, solve the degree of depth initial value of pixel x in the damaged area of described depth image.

7. method according to claim 3, is characterized in that, before described step 3, described method also comprises:

According to described depth image and described coloured image, calculate the degree of confidence C (x) of the existing depth value of pixel x in described depth image.

8. method according to claim 7, is characterized in that, described according to described depth image and described coloured image, the step calculating the degree of confidence C (x) of the existing depth value of pixel x in described depth image comprises:

Wherein, Ω _hit is the pixel set of damaged area in depth image; Ω _dbe in depth image do not belong to damaged area and distance damaged area is no more than the pixel set of d; D is the first distance threshold;

C _fx () is defined by following formula:

$C_f\left(x\right)=\frac{1}{\mathrm{\η}}\underset{x_i\∈N_r,x_i\≠x}{\Σ}{w}_f(x_i,x)c_f(x_i,x)$

$w_f(x_i,x)=\exp \left(-\frac{{\left(I\right(x_i)-I(x\left)\right)}^2}{2{\mathrm{\σ}}_I^2}\right)$

$c_f(x_i,x)=\exp \left(-\frac{{\left(D\right(x_i)-D(x\left)\right)}^2}{2{\mathrm{\σ}}_D^2}\right)$

N _rit is the set that distance pixel x is less than or equal to the pixel of r; R is second distance threshold value;

I (x) is the color value of pixel x in coloured image; I (x _i) be pixel x in coloured image _icolor value;

D (x) is the existing depth value of pixel x in depth image, D (x _i) be pixel x in depth image _iexisting depth value; η is w _f(x _i, summation x), σ _iand σ _dthe second function parameter and the 3rd function parameter respectively;

C _hx () is defined by following formula:

$C_h\left(x\right)=\frac{1}{{\mathrm{\η}}^{\′}}\underset{x_i\∈N_r,x_i\≠x}{\Σ}{w}_h(x_i,x)c_h(x_i,x)$

$w_h(x_i,x)=\exp \left(-\frac{{\left(D\right(x_i)-D(x\left)\right)}^2}{2{\mathrm{\σ}}_D^{\′2}}\right)$

$c_h(x_i,x)=\exp \left(-\frac{{\left(I\right(x_i)-I(x\left)\right)}^2}{2{\mathrm{\σ}}_I^{\′2}}\right)$

Wherein, η ' is w _h(x _i, summation x), σ ' _iwith σ ' _dthe 4th function parameter and the 5th function parameter respectively.

9. method according to claim 8, is characterized in that, before described step 3, described method also comprises:

Spatial registration is carried out to the pixel of described depth image and the pixel of described coloured image;

Corresponding relation between the color value obtaining same pixel x in the depth value of pixel x in described depth image and described coloured image.