CN102831582A - Method for enhancing depth image of Microsoft somatosensory device - Google Patents

Abstract

The invention discloses a depth image enhancement method of a Microsoft somatosensory device. It includes the following steps: perform edge detection on the color image and the depth image respectively, and use the two edge images as input, use the region growing method to obtain the region where the error pixel is located; remove the depth value of the error pixel; use the region growing method to remove the invalid pixel A smooth area is constructed around it; the depth value of invalid pixels in the smooth area is estimated by bilateral filtering method; the depth value of remaining invalid pixels is estimated by bilateral filtering method, and an enhanced depth image is obtained. The invention points out for the first time that the problem of edge mismatch between the depth image and the corresponding color image is caused by wrong pixels, and further proposes a detection method for wrong pixels. The invention can effectively fill the holes of the Kinect depth image, well solve the problem of edge mismatch of the depth image, and greatly improve the quality of the Kinect depth image.

Description

A depth image enhancement method of Microsoft somatosensory device

technical field

The invention relates to a depth image enhancement method, in particular to a depth image enhancement method of a Microsoft somatosensory device.

Background technique

Kinect is an inexpensive depth image acquisition device released by Microsoft. It is capable of simultaneously producing color and depth images of size 640×480 at 30fps. Due to this cheap and real-time feature, Kinect was widely used in interactive places such as hospitals, libraries, and lecture halls shortly after its release.

Due to the limitation of the measurement principle, the depth image of Kinect will produce holes near the edge of the object and the surface with poor reflectivity, and the edge of the depth image and the edge of the corresponding color image often do not match.

To solve the hole-filling problem, researchers have tried several filling methods. Traditional methods are mainly divided into pixel-based methods and point cloud-based methods. The idea of the pixel-based method is to treat the depth image as a normal grayscale image, and regard the hole as the area to be repaired. In this way, the problem of hole filling is transformed into a traditional image restoration problem. This type of method mainly uses color information as a guide, and estimates the depth value of invalid points through image restoration methods such as interpolation, fast repair, and belief propagation. However, since the edge of the depth image does not match the edge of the color image, the depth information of the object edge is unreliable, and the estimated depth value is often inaccurate.

The idea of the point cloud-based method is to treat the depth image as the data describing the surface of the object, so that the problem of filling the hole is transformed into the problem of completing the surface of the object. This type of method first converts the depth image data into point cloud data, reconstructs the 3D surface through the point cloud, and then finds the hole and hole according to the characteristics of the surface structure (such as the similarity of shape, the relationship between surface normal vectors, etc.) The best matching image patch. Such methods alleviate the problem of inaccurate estimated depth values in the first type of methods, but do not completely solve the problem. Moreover, this type of method needs to reconstruct a 3D surface, which increases unnecessary calculation for applications that do not require 3D reconstruction.

For the mismatch between depth image edges and color image edges, the existing methods mainly explore the information of depth image sequences, and use longer time window filtering to obtain stable depth image edges. This method requires motion estimation of adjacent images. Due to the influence of image noise and other factors, the motion estimation of image sequences is not very accurate, and the amount of calculation is also large.

Contents of the invention

In order to solve the above-mentioned problems existing in the Kinect depth image, the present invention provides a depth image enhancement method of a Microsoft somatosensory device. The present invention can be widely used in various kinect actual systems as preprocessing to Kinect depth data.

The technical scheme that the present invention solves the problems of the technologies described above comprises the following steps:

1) Edge detection is performed on the Kinect color image and depth image respectively, and the edge of the color image and the edge of the depth image are obtained;

2) Take two edge images as input, and use the region growing method to obtain the middle area of the two edge images, that is, the area where the error pixel is located;

3) Remove the wrong pixel depth value;

4) Construct a smooth region around invalid pixels with the region growing method;

5) Estimate the depth value of invalid pixels in the smooth area by bilateral filtering method.

6) Estimate the depth value of the remaining invalid pixels by bilateral filtering method, and obtain a depth image without holes whose edges are consistent with the edges of the color image;

In the above-mentioned Microsoft somatosensory device depth image enhancement method, the step 1) is:

The color image and depth image collected from Kinect are converted into 8-bit grayscale images, and then edge detection is performed on the two 8-bit grayscale images using Canny edge detection algorithm. Among them, the upper threshold and lower threshold of Canny edge detection are 200 and 100, respectively.

In the above-mentioned Microsoft somatosensory device depth image enhancement method, the step 2) comprises the following steps:

a) Use the region growing method to build regions on the edge of the color image and the edge of the depth image respectively to form a mask image mask1 and a mask image mask2.

The method of constructing the region on the edge of the depth image is: using all the pixels on the edge of the depth image as seeds to grow the region until it touches the edge of the color image or reaches a specified distance.

The method of constructing the region on the edge of the color image is: using all the pixels on the edge of the color image as seeds to grow the region until the edge of the depth image is encountered or a specified distance is reached.

b) Perform a morphological dilation operation on the depth edge image.

c) The mask image mask1 and the mask image mask2 are summed by pixels to obtain the mask image mask4, and then the mask image mask4 and the mask image mask3 are ORed by pixels to obtain the mask image mask5, which is the wrong pixel The result of the detection, where non-zero pixels represent error pixels.

In the above-mentioned depth image enhancement method of the Microsoft somatosensory device, the step 4) is: perform region growing in a 5×5 window centered on each invalid pixel _Pi , and construct a smooth region around it.

In the above-mentioned Microsoft somatosensory device depth image enhancement method, the bilateral filtering method in the step 5) is:

${\mathrm{<span\; class="notranslate">\; D.</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}^{\mathrm{<span\; class="notranslate">\; E.</span>}}<span\; class="notranslate">\; =</span>\underset{{\mathrm{<span\; class="notranslate">\; C</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; C</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; <</span>\mathrm{<span\; class="notranslate">\; T</span>}}{\underset{\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; \&Element;</span>\mathrm{<span\; class="notranslate">\; \&Omega;</span>}{\mathrm{<span\; class="notranslate">\; D.</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; \&NotElement;</span>\mathrm{<span\; class="notranslate">\; 0</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}\frac{{\mathrm{<span\; class="notranslate">\; G</span>}}_{\mathrm{<span\; class="notranslate">\; the\; s</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; )</span>{\mathrm{<span\; class="notranslate">\; G</span>}}_{\mathrm{<span\; class="notranslate">\; c</span>}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; C</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; C</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; )</span>{\mathrm{<span\; class="notranslate">\; D.</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}}{{\mathrm{<span\; class="notranslate">\; G</span>}}_{\mathrm{<span\; class="notranslate">\; the\; s</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; )</span>{\mathrm{<span\; class="notranslate">\; G</span>}}_{\mathrm{<span\; class="notranslate">\; c</span>}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; C</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; C</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; )</span>}<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; )</span>$

为像素P_i的深度估计值，D_j为像素P_j的深度值，G_s和G_c为均值为0，方差为1.5和3的高斯函数。i-j为像素P_i与P_j的欧氏距离，C_i-C_j为像素P_i与P_j在彩色空间的欧氏距离。T为一个给定的阈值，其值为40。且参与计算的像素个数达到3时估计值才被采纳。where Ω is the smooth area around _Pi ,

is the estimated depth value of pixel P _i , D _j is the depth value of pixel P _j , G _s and G _c are Gaussian functions with mean value 0 and variance 1.5 and 3. ij is the Euclidean distance between pixels P _i and P _j , and C _i -C _j is the Euclidean distance between pixels P _i and P _j in the color space. T is a given threshold, whose value is 40. And the estimated value is adopted when the number of pixels involved in the calculation reaches 3.

Repeat the bilateral filtering until there are no invalid pixels in the smooth area or the estimated values of invalid pixels are not adopted even though there are invalid pixels.

In the above-mentioned Microsoft somatosensory device depth image enhancement method, the bilateral filtering method adopted for the remaining invalid pixels in the step 6) is:

${\mathrm{<span\; class="notranslate">\; D.</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}^{\mathrm{<span\; class="notranslate">\; E.</span>}}<span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; \&Element;</span>\mathrm{<span\; class="notranslate">\; \&Omega;</span>}{\mathrm{<span\; class="notranslate">\; D.</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; \&NotElement;</span>\mathrm{<span\; class="notranslate">\; 0</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}\frac{{\mathrm{<span\; class="notranslate">\; G</span>}}_{\mathrm{<span\; class="notranslate">\; the\; s</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; )</span>{\mathrm{<span\; class="notranslate">\; G</span>}}_{\mathrm{<span\; class="notranslate">\; c</span>}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; C</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; C</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; )</span>{\mathrm{<span\; class="notranslate">\; D.</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}}{{\mathrm{<span\; class="notranslate">\; G</span>}}_{\mathrm{<span\; class="notranslate">\; the\; s</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; )</span>{\mathrm{<span\; class="notranslate">\; G</span>}}_{\mathrm{<span\; class="notranslate">\; c</span>}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; C</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; C</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; )</span>}<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 2</span>}<span\; class="notranslate">\; )</span>$

Among them, P _i is the invalid pixel outside the smooth area, that is, the remaining invalid pixels; Ω is a neighborhood of pixel P _i , the size is 5×5, is the estimated depth value of pixel P _i , D _j is the depth value of pixel P _j , G _s and G _c are Gaussian functions with mean value 0, variance 1.5 and 3; ij is the Euclidean distance between pixel P _i and P _j ; C _i -C _j is the Euclidean distance between pixels P _i and P _j in the color space; T is a given threshold, its value is 40; and when the number of pixels involved in the calculation reaches 3, the estimated value is adopted ; Repeat the bilateral filtering until there are no remaining invalid pixels or the estimated values of invalid pixels are not adopted even though there are invalid pixels.

Due to the adoption of the above technical solution, the technical effect of the present invention is that the present invention uses the method of removing wrong pixels to avoid using wrong depth values to estimate the depth values of invalid points, so that the depth value estimation is more accurate. Furthermore, due to the removal of erroneous pixels, the depth image edges are matched to the corresponding color image edges. In order to estimate the depth value of the invalid point more accurately, the region growing method is used to construct a smooth area near the invalid point, and the effective pixels in the smooth area are used to estimate the depth value of the invalid point, so that the error of the estimated depth value is minimized, thus Get a complete high-accuracy depth image. The invention effectively fills the holes of the Kinect depth image, well solves the problem of edge mismatch of the depth image, greatly improves the quality of the Kinect depth image, and has great significance and practical value for the subsequent processing of the depth image.

The present invention will be further described below in conjunction with the accompanying drawings.

Description of drawings

Fig. 1 is a flowchart of the present invention.

FIG. 2 is a schematic diagram of an area where an error pixel is detected in an embodiment of the present invention.

Fig. 3 is a schematic diagram of hole filling in a depth image in an embodiment of the present invention.

Figure 4 is an example of image enhancement, where (a) is the image obtained by the bilateral filtering method, and (b) is the image obtained by the method of the present invention.

Detailed ways

Referring to Fig. 1, Fig. 1 is a flowchart of the present invention, and its specific implementation steps are as follows:

1. Perform edge detection on Kinect color image and depth image respectively, and obtain color image edge and depth image edge.

The color image and depth image collected from Kinect are respectively converted into 8-bit grayscale images, and then edge detection is performed on the two 8-bit grayscale images to obtain color edge images and depth edge images. The edge detection method specifically adopts the Canny edge detection algorithm (the specific implementation details of the Canny edge detection algorithm refer to the paper John Canny, "A computational approach to edge detection" published in IEEE Transactions on Pattern Analysis and Machine Intelligence in 1986. IEEE Trans.Pattern Analysis and Machine Intelligence, vol.8, no.6, pp.679-714.). Among them, the upper threshold and lower threshold of Canny edge detection are 200 and 100, respectively.

2 Take two edge images as input, and use the region growing method to obtain the area in the middle of the two edge images, that is, the area where the error pixel is located, as shown in Figure 2, specifically including:

1) Use the region growing method to construct regions on the edge of the color image and the edge of the depth image respectively to form mask image mask1 and mask image mask2.

Wherein, the method of constructing a region on the edge of the depth image is: for each pixel on the edge of the depth image, use this pixel as a seed to grow the region until it touches the edge of the color image or reaches the window boundary. The specific steps are:

Step 1: For each pixel on the edge of the depth image, if it is not on the edge of the color image, put it into the pixel set A to be investigated;

Step 2: For each pixel P in A, inspect its four neighborhoods separately. If the inspected point is not on the edge of the color image and is within the 9×9 inspection window centered on P, put the point into the pending Consider the pixel set A, and then remove P from A until A is an empty set.

The method of constructing the region on the edge of the color image is: using all the pixels on the edge of the color image as seeds to grow the region until the edge of the depth image is encountered or the specified distance is reached.

2) Perform morphological expansion on the depth edge image with a 3×3 template to obtain the mask image mask3.

3) The mask image mask1 and the mask image mask2 are summed by pixels to obtain the mask image mask4, and then the mask image mask4 and the mask image mask3 are ORed by pixels to obtain the mask image mask5, which is the wrong pixel The result of the detection, where non-zero pixels represent error pixels.

3 Remove wrong pixel depth values.

4 Construct smooth regions around invalid pixels by region growing method.

As shown in FIG. 3 , for each invalid pixel P _i , region growing is performed in a 5×5 window centered on this pixel, and a smooth region is constructed around it.

5 Estimate the depth value of invalid pixels in the smooth area by bilateral filtering method.

As shown in Figure 3, the following bilateral filtering method is used to estimate the depth value of this invalid pixel:

${\mathrm{<span\; class="notranslate">\; D.</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}^{\mathrm{<span\; class="notranslate">\; E.</span>}}<span\; class="notranslate">\; =</span>\underset{{\mathrm{<span\; class="notranslate">\; C</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; C</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; <</span>\mathrm{<span\; class="notranslate">\; T</span>}}{\underset{\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; \&Element;</span>\mathrm{<span\; class="notranslate">\; \&Omega;</span>}{\mathrm{<span\; class="notranslate">\; D.</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; \&NotElement;</span>\mathrm{<span\; class="notranslate">\; 0</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}\frac{{\mathrm{<span\; class="notranslate">\; G</span>}}_{\mathrm{<span\; class="notranslate">\; the\; s</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; )</span>{\mathrm{<span\; class="notranslate">\; G</span>}}_{\mathrm{<span\; class="notranslate">\; c</span>}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; C</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; C</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; )</span>{\mathrm{<span\; class="notranslate">\; D.</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}}{{\mathrm{<span\; class="notranslate">\; G</span>}}_{\mathrm{<span\; class="notranslate">\; the\; s</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; )</span>{\mathrm{<span\; class="notranslate">\; G</span>}}_{\mathrm{<span\; class="notranslate">\; c</span>}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; C</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; C</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; )</span>}<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; )</span>$

为像素P_i的深度估计值，D_j为像素P_j的深度值，G_s和G_c为均值为0，方差为1.5和3的高斯函数。i-j为像素P_i与P_j在图像空间的欧氏距离，C_i-C_j为像素P_i与P_j在彩色空间的欧氏距离。T为一个给定的阈值，其值为40。where Ω is the smooth area around _Pi ,

is the estimated depth value of pixel P _i , D _j is the depth value of pixel P _j , G _s and G _c are Gaussian functions with mean value 0 and variance 1.5 and 3. ij is the Euclidean distance between pixels P _i and P _j in the image space, and C _i -C _j is the Euclidean distance between pixels P _i and P _j in the color space. T is a given threshold, whose value is 40.

In order to accurately estimate the depth value of invalid points, the estimated value is adopted only when the number of pixels involved in the calculation reaches 3. In order to fill the larger holes, a loop method is adopted to implement bilateral filtering until there are no invalid pixels in the smooth area or the estimated values of invalid pixels are not adopted even though there are invalid pixels.

6 Estimate the depth value of the remaining invalid pixels by bilateral filtering method, and obtain a depth image without holes whose edges are consistent with the edges of the color image.

As shown in Figure 3, the following bilateral filtering is used to estimate its depth value:

${\mathrm{<span\; class="notranslate">\; D.</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}^{\mathrm{<span\; class="notranslate">\; E.</span>}}<span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; \&Element;</span>\mathrm{<span\; class="notranslate">\; \&Omega;</span>}{\mathrm{<span\; class="notranslate">\; D.</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; \&NotElement;</span>\mathrm{<span\; class="notranslate">\; 0</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}\frac{{\mathrm{<span\; class="notranslate">\; G</span>}}_{\mathrm{<span\; class="notranslate">\; the\; s</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; )</span>{\mathrm{<span\; class="notranslate">\; G</span>}}_{\mathrm{<span\; class="notranslate">\; c</span>}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; C</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; C</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; )</span>{\mathrm{<span\; class="notranslate">\; D.</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}}{{\mathrm{<span\; class="notranslate">\; G</span>}}_{\mathrm{<span\; class="notranslate">\; the\; s</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; )</span>{\mathrm{<span\; class="notranslate">\; G</span>}}_{\mathrm{<span\; class="notranslate">\; c</span>}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; C</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; C</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; )</span>}<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 2</span>}<span\; class="notranslate">\; )</span>$

为像素P_i的深度估计值，D_j为像素P_j的深度值，G_s和G_c为均值为0，方差为1.5和3的高斯函数；i-j为像素P_i与P_j的欧氏距离；C_i-C_j为像素P_i与P_j在彩色空间的欧氏距离；T为一个给定的阈值，其值为40；且参与计算的像素个数达到3时，估计值才被采纳；重复双边滤波，直至没有剩余无效像素或者虽有剩余无效像素但其估计值均不被采纳为止。Among them, P _i is the invalid pixel outside the smooth area, that is, the remaining invalid pixels; Ω is a neighborhood of pixel P _i , and the size is 5×5;

is the estimated depth value of pixel P _i , D _j is the depth value of pixel P _j , G _s and G _c are Gaussian functions with mean value 0, variance 1.5 and 3; ij is the Euclidean distance between pixel P _i and P _j ; C _i -C _j is the Euclidean distance between pixels P _i and P _j in the color space; T is a given threshold, its value is 40; and when the number of pixels involved in the calculation reaches 3, the estimated value is adopted ; Repeat the bilateral filtering until there are no remaining invalid pixels or the estimated values of the remaining invalid pixels are not adopted.

The method provided by the present invention is compared with the general bilateral filtering method, as shown in FIG. 4 . It can be seen from Figure 4 that this method not only effectively fills the hole, but also greatly enhances the stability of the edge, so that the edge of the depth image and the color image match well.

Claims (8)

1. Microsoft's body induction device depth image Enhancement Method may further comprise the steps:

1) Kinect coloured image and depth image are made rim detection respectively, obtain Color Image Edge and depth image edge;

2) be input with two edge images, adopt region-growing method to obtain the middle zone of two edge images, i.e. the zone at erroneous pixel place;

3) remove the erroneous pixel depth value;

4) around inactive pixels, make up smooth region with region-growing method;

5) estimate the depth value of inactive pixels in the smooth region with the bilateral filtering method;

6), obtain the depth image in the breadths edge nothing cavity consistent with Color Image Edge with bilateral filtering method estimated remaining inactive pixels depth value.

2. Microsoft according to claim 1 body induction device depth image Enhancement Method, said step 1) is:

To transfer 8 gray level images respectively to from coloured image and the depth image that Kinect gathers; Adopt the Canny edge detection algorithm to carry out rim detection respectively to two 8 gray level images then; Wherein, the upper limit threshold of Canny rim detection and lower threshold are respectively 200 and 100.

3. Microsoft according to claim 1 body induction device depth image Enhancement Method is characterized in that said step 2) be:

A) make up the zone in Color Image Edge and depth image edge respectively with region-growing method, form mask images mask1 and mask images mask2;

B) the depth edge image is carried out the morphology expansive working;

C) mask images mask1 and mask images mask2 are according to pixels asked with operation obtain mask images mask4; Then mask images mask4 and mask images mask3 are according to pixels asked or operate and obtain mask images mask5; This is the result that erroneous pixel detects, and wherein non-zero pixels is represented erroneous pixel.

4. Microsoft according to claim 3 body induction device depth image Enhancement Method; The method that said depth image edge makes up the zone is: with all pixels on the depth image edge is that seed carries out region growing, till running into Color Image Edge or reaching distance to a declared goal.

5. Microsoft according to claim 3 body induction device depth image Enhancement Method; The method that said Color Image Edge makes up the zone is: with all pixels on the Color Image Edge is that seed carries out region growing, till running into the depth image edge or reaching distance to a declared goal.

6. Microsoft according to claim 1 body induction device depth image Enhancement Method, said step 4) is: with each inactive pixels P _iCarry out region growing in 5 * 5 windows for the center, and around it, make up smooth region.

7. Microsoft according to claim 1 body induction device depth image Enhancement Method, the bilateral filtering method is in the said step 5):

$D_i^E=\underset{C_i-C_j<T}{\underset{j\&Element;\mathrm{\&Omega;}{D}_j\&NotElement;0}{\mathrm{\&Sigma;}}}\frac{G_s(i-j)G_c(C_i-C_j)D_j}{G_s(i-j)G_c(C_i-C_j)}---\left(1\right)$

Wherein Ω is P _iSmooth region on every side;

Be pixel P _iThe estimation of Depth value, D _jBe pixel P _jDepth value; G _sAnd G _cFor average is 0, variance is 1.5 and 3 Gaussian function; I-j is pixel P _iWith P _jEuclidean distance, C _i-C _jBe pixel P _iWith P _jEuclidean distance in the color space; T is a given threshold value, and its value is 40; And when the number of pixels of participating in calculating reached 3, estimated value was just adopted; Repeat bilateral filtering, though not having inactive pixels or having its estimated value of inactive pixels all not to be adopted as in smooth region ended.

8. Microsoft according to claim 1 body induction device depth image Enhancement Method, the bilateral filtering method that adopts for the residue inactive pixels in the said step 6) is:

$D_i^E=\underset{j\&Element;\mathrm{\&Omega;}{D}_j\&NotElement;0}{\mathrm{\&Sigma;}}\frac{G_s(i-j)G_c(C_i-C_j)D_j}{G_s(i-j)G_c(C_i-C_j)}---\left(2\right)$

Wherein, P _iFor the outer inactive pixels of smooth region, promptly remain inactive pixels; Ω is pixel P _iA neighborhood, size is 5 * 5; Be pixel P _iThe estimation of Depth value, D _jBe pixel P _jDepth value, G _sAnd G _cFor average is 0, variance is 1.5 and 3 Gaussian function; I-j is pixel P _iWith P _jEuclidean distance; C _i-C _jBe pixel P _iWith P _jEuclidean distance in the color space; T is a given threshold value, and its value is 40; And the number of pixels of participating in calculating reaches at 3 o'clock, and estimated value is just adopted; Repeat bilateral filtering, though end until not remaining inactive pixels or having its estimated value of residue inactive pixels all not to be adopted as.

Images