JP7489253B2 - Depth map generating device and program thereof, and depth map generating system - Google Patents

Description

The present invention relates to a depth map generating device and a program for generating a depth map, and a depth map generating system.

In recent years, there has been active research into technology for acquiring the three-dimensional shape (depth map) of a subject existing in space. This technology is expected to be applied to a variety of fields, including three-dimensional video production, augmented reality (AR), virtual reality (VR), and robotics. Approaches for acquiring the three-dimensional shape of a subject can be broadly divided into active and passive methods (Non-Patent Document 1).

In active methods, the measuring device has a light source and uses reflected light from the subject to measure depth. Specific methods include pattern light projection, time of flight (ToF), and photometric stereo. Of these, the method using a ToF camera has been attracting attention in recent years. A ToF camera determines the distance from the subject to the ToF camera by measuring the time it takes for light emitted from a light source to be reflected by the subject and return. The advantage of active methods is that they can obtain highly accurate distances in real time without the need for advanced calculation processing. On the other hand, the disadvantages of active methods are that they are vulnerable to external light, measurement errors can occur depending on the reflectance and distance of the subject, and scale calibration may be required.

Passive methods involve moving multiple color cameras (hereafter referred to as "RGB cameras") or a single RGB camera and measuring the depth distance from the parallax. Specific methods include the stereo method (multiple-eye stereo) and motion stereo. These methods are based on the same principle as stereo methods, which calculate depth from the parallax of two or more cameras. The advantages of passive methods are that they do not require special light to be shone on the subject, they are not affected by external light, and they can be achieved with just a general color camera and a computer. On the other hand, the disadvantages of passive methods are that the obtained depth can be ambiguous (texture-less, occlusion areas), and the calculation costs are high.

Another known camera is an RGB-D camera that can acquire RGB images and depth images from multiple viewpoints by arranging an RGB camera and a depth camera on the same optical axis and using a lens array (Patent Document 1). In this method, the light entering through the camera lens is split by a mirror (e.g., a half mirror or a dichroic mirror) and received by the RGB camera and the depth camera.

JP 2009-300268 A

Digital Image Processing (Revised New Edition), CG-ARTS Association, 2015

As mentioned above, since obtaining three-dimensional shapes can be applied in a wide range of fields, various methods have been proposed, but none have yet been established. For general-purpose purposes, it is more convenient to have RGB images and depth maps from various viewpoints, rather than just a color image (hereafter referred to as an RGB image) and depth map from one viewpoint. In other words, having a set of RGB images and depth maps from multiple viewpoints improves versatility.

The accuracy of the depth map is also important. Depth images obtained with RGB-D cameras are represented by pixel values (brightness values), so these pixel values must be converted into a real-scale depth map. However, the conversion function to real scale has a significant impact on the accuracy of the depth map. Furthermore, the accuracy of the depth map is also affected by the shooting environment and the type of subject. Note that real scale refers to the distance (depth) in real space.

The present invention aims to solve the above problems and provide a depth map generation device and program thereof, as well as a depth map generation system, that can easily obtain images captured from multiple viewpoints and highly accurate depth maps.

To solve the above problem, the depth map generating device according to the present invention is a depth map generating device that generates a depth map corresponding to the captured images at each viewpoint using captured images and depth images captured by an imaging device consisting of a imaging camera and a depth camera with the same optical axis and an optical element array, and is configured to include a cost volume generating means, a depth conversion means, a cost weight calculation means, a visibility weight calculation means, a weight application means, and a final depth map generating means.

According to this configuration, the cost volume generation means calculates a cost representing the similarity between the captured images projected onto the depth layer for each pixel position of the captured image and the depth layer at a predetermined interval in the depth direction, and generates a cost volume in which the costs are arranged three-dimensionally by depth layer and pixel position.
The depth conversion means converts the depth image into an intermediate depth map by a depth conversion function that converts the pixel value of each pixel of the depth image into a depth.
The cost weight calculation means calculates a cost weight that represents the weight of the intermediate depth map using a normal distribution function.

Also, the visibility weight calculation means calculates, from the intermediate depth map, a visibility weight that reduces the cost when occlusion occurs.
The weight application means applies a cost weight and a visibility weight to the cost volume.
The final depth map generating means generates a final depth map indicating the depth of the depth layer having the smallest cost in a cost sequence at the same pixel position in the cost volume after the weights are applied.

In other words, the depth map generating device performs a refinement process that constrains the cost volume generated from the captured image with two weights based on the depth map generated from the depth image. This refinement process enables the depth map generating device to generate a highly accurate depth map that corresponds to the captured image from each viewpoint.

The present invention can be realized by a program that causes a computer to function as the depth map generating device described above.

The present invention can also be realized in a depth map generation system that includes an imaging device that is composed of a imaging camera and a depth camera with the same optical axis and an optical element array, and the depth map generation device described above.

The present invention makes it easy to obtain images captured from multiple viewpoints and highly accurate depth maps.

1 is an overall configuration diagram of a three-dimensional shape acquisition system according to an embodiment. 1 is a block diagram showing a configuration of a three-dimensional shape acquisition apparatus according to an embodiment. 5A and 5B are explanatory diagrams for explaining the photographing of a calibration pattern by an RGB-D camera, in which (a) shows calibration data A and (b) shows calibration data B. 1A and 1B are diagrams illustrating division of an image obtained by capturing a calibration pattern, in which (a) shows an RGB image and (b) shows a depth image. 5A and 5B are diagrams illustrating calculation of a scale conversion function, in which FIG. 5A shows a distance from a virtual camera to a calibration pattern, and FIG. 5B shows an example of the scale conversion function. 1A and 1B are diagrams illustrating division of an image of a subject, where (a) shows an RGB image and (b) shows a depth image. FIG. 11 is an explanatory diagram illustrating an example of a depth layer. FIG. 13 is an explanatory diagram for explaining cost volume. FIG. 1 is an explanatory diagram for explaining a normal distribution function. FIG. 4A is an explanatory diagram illustrating an example of a cost weight function, and FIG. 4B is an explanatory diagram illustrating an example of a visibility function. 4 is a flowchart showing a camera calibration process in the embodiment. 4 is a flow chart illustrating a refinement process in an embodiment.

The following describes an embodiment of the present invention with reference to the drawings. However, the embodiment described below is intended to embody the technical concept of the present invention, and unless otherwise specified, the present invention is not limited to the following.

[Overview of 3D shape acquisition system]
An overview of a three-dimensional shape acquisition system (depth map generation system) 1 according to an embodiment will be described with reference to FIG.
The three-dimensional shape acquisition system 1 acquires RGB images (captured images) and a depth map from multiple viewpoints of a subject 9, and camera parameters of a virtual camera C. As shown in FIG 1, the three-dimensional shape acquisition system 1 includes an RGB-D camera (capturing device) 2 and a three-dimensional shape acquisition device (depth map generating device) 3.

When multiple RGB cameras and depth cameras are arranged to capture images from multiple viewpoints, the system becomes large-scale and expensive. Therefore, the 3D shape acquisition system 1 uses a single RGB-D camera (image capture device) 2 (described below) to achieve a configuration equivalent to that of multiple RGB cameras and depth cameras, simplifying the system configuration.

In fields such as three-dimensional video production, the camera parameters of the virtual camera C are required. Furthermore, because a depth image is represented by pixel values (brightness values), a scale conversion function is also required to convert these pixel values into a real-scale depth map. Therefore, in the three-dimensional shape acquisition system 1, the three-dimensional shape acquisition device 3 performs a camera calibration process using a calibration pattern to calculate the camera parameters and scale conversion function of the virtual camera C.

The accuracy of the depth map is also important. As mentioned above, the scale conversion function has a large effect on the accuracy of the depth map. Furthermore, the accuracy of the depth map is greatly reduced depending on the shooting environment and the type of subject. Therefore, in the three-dimensional shape acquisition system 1, the three-dimensional shape acquisition device 3 described below uses RGB images and depth images from multiple viewpoints to improve the accuracy of the depth map (refinement processing). At this time, the three-dimensional shape acquisition device 3 divides one RGB image captured by one RGB-D camera 2 for each viewpoint and matches them, so errors caused by color differences can be suppressed compared to matching images captured by multiple RGB cameras.

First, the configuration of the RGB-D camera 2 will be described. Next, the camera calibration process performed by the three-dimensional shape acquisition device 3 will be described. This camera calibration process is a process for calculating the camera parameters and scale conversion function of each virtual camera C. Finally, the refinement process performed by the three-dimensional shape acquisition device 3 to improve the accuracy of the depth map will be described.

[RGB-D Camera Configuration]
As shown in FIG. 1, the RGB-D camera 2 is an imaging device including a camera body 20 and a lens system 21. In this embodiment, the camera body 20 is an RGB camera and a depth camera (not shown) arranged on the same optical axis. The camera body 20 also disperses light from a subject 9 using a spectroscopic element (not shown), and receives the dispersed light with the RGB camera and the depth camera, respectively. For example, the RGB camera may be a general color camera. The spectroscopic element may be a half mirror or a dichroic mirror.

In this embodiment, a ToF camera is used as the depth camera. This ToF camera is equipped with an infrared LED array 25 for irradiating the subject 9 with infrared rays when measuring distance. A depth image can be obtained by calculating the inter-frame difference of the infrared images captured by the ToF camera.

The lens system 21 includes a Fresnel lens 22 and a lens array (optical element array) 23. The lens array 23 is a two-dimensional array of N _x N _Y element lenses 24. The RGB-D camera 2 can acquire RGB images and depth images for N _x N _Y viewpoints via the lens array 23. That is, the RGB-D camera 2 realizes a configuration equivalent to an arrangement of N _x N _Y virtual cameras C. In this embodiment, it is assumed that there are four viewpoints (four virtual cameras C) corresponding to the 2 x 2 element lenses 24.

The angle of view of the virtual camera C can be adjusted by adjusting the positional relationship between the camera body 20 and the lens system 21. Also, in FIG. 1, only two of the four virtual cameras C are illustrated.

[Configuration of the three-dimensional shape acquisition device]
The configuration of the three-dimensional shape acquisition device 3 will be described with reference to FIG.
The three-dimensional shape acquisition device 3 generates a depth map corresponding to the RGB images of each viewpoint by using RGB images and depth images of the subject 9 captured at each viewpoint by the RGB-D camera 2. As shown in Fig. 2, the three-dimensional shape acquisition device 3 includes a camera calibration means 4 that performs a camera calibration process and a refinement means 5 that performs a refinement process.

<Camera calibration method>
The camera calibration means 4 estimates two types of parameters. The first is the camera parameters of the virtual camera C. The camera parameters of the virtual camera C represent the focal length of the lens, the lens distortion, the position and the orientation of the virtual camera C, and the like. The second is a scale conversion function of each virtual camera C. Furthermore, the camera calibration means 4 corrects the angle of view of the RGB image and the depth image as necessary. Note that the camera calibration means 4 does not need to perform camera calibration processing every time shooting is performed, and it is sufficient to perform camera calibration processing when the focal length of the RGB-D camera 2 or the relationship between the position and orientation of the RGB-D camera 2, the Fresnel lens 22, and the lens array 23 changes.

As shown in FIG. 3(a), the camera calibration means 4 receives an RGB image and a depth image captured by the RGB-D camera 2 of a calibration pattern 90. The calibration pattern 90 is a planar pattern with a known arrangement of feature points (for example, a chessboard pattern). At this time, the RGB-D camera 2 captures the calibration pattern 90 by changing its posture two or more times (shown by the dashed line). Note that, when the skew of the internal parameters is other than 0, the RGB-D camera 2 captures the calibration pattern 90 by changing its posture three or more times. As shown in FIG. 3(a), the RGB image and the depth image captured by disposing the lens system 21 are called calibration data A. When performing the above-mentioned angle of view correction, the calibration pattern 90 is captured by removing the lens system 21 as shown in FIG. 3(b). The RGB image and the depth image captured by removing the lens system 21 in this way are called calibration data B.

As shown in FIG. 2, the camera calibration means 4 includes an angle of view correction means 40, an image division means 41, an initial camera parameter calculation means 42, a camera parameter optimization means 43, and a scale conversion function calculation means (depth conversion function calculation means) 44.

The angle of view correction means 40 performs projective transformation on the depth image so that the angle of view of the depth image input from the RGB-D camera 2 coincides with the angle of view of the RGB image. Due to the installation accuracy of the RGB-D camera 2, the angle of view of the RGB image captured by the RGB camera and the depth image captured by the depth camera may be slightly shifted. For this reason, the angle of view correction means 40 corrects this slight shift in the angle of view using the calibration data B. Specifically, the angle of view correction means 40 calculates a homography matrix based on four or more corresponding points (feature points of the calibration pattern 90) between the RGB image and the depth image (Reference 1). Then, the angle of view correction means 40 performs projective transformation on the depth image using this homography matrix to make the angle of view of the depth image coincide with the angle of view of the RGB image.
It should be noted that, when the angles of view of the RGB camera and the depth camera are the same, the angle-of-view correction means 40 does not need to perform the angle-of-view correction process described above.

Reference 1: "OpenCV", [online], [searched June 24, 2020], Internet <URL: https://opencv.org/>

The angle-of-view correction means 40 can also remove lens distortion using the calibration data B. For example, the angle-of-view correction means 40 calculates the lens distortion coefficient of the RGB-D camera 2 using Zhang's method, and removes the lens distortion from the RGB image and the depth image (Reference 2).

Reference 2: Z. Zhang, “A flexible new technique for camera calibration”, IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 22, No. 11, pp. 1330-1334 (2000)

The image division means 41 divides the RGB image and the depth image input from the angle of view correction means 40 for each viewpoint (element lens 24). That is, the image division means 41 divides the RGB image and the depth image for each virtual camera C to generate an RGB image and a depth image virtually captured by the virtual camera C. In this embodiment, the image division means 41 divides the RGB image P _C and the depth image P _D into four, as shown in Figures 4(a) and 4(b).

The region α into which the RGB image P _C and the depth image P _D are divided is set manually. In this case, since the outside of the lens array 23 and the gaps between the element lenses 24 are not necessary in the divided RGB image P _C and the depth image P _D , it is not necessary to divide these unnecessary regions. To simplify the following explanation, it is assumed that the divided RGB image P _C and the depth image P _D have the same image size.

The initial camera parameter calculation means 42 calculates initial camera parameters of the virtual camera C corresponding to each viewpoint by performing camera calibration processing on the RGB image P _C of each viewpoint input from the image division means 41. For example, the initial camera parameter calculation means 42 applies Zhang's method to the RGB image P _C of each viewpoint to calculate initial camera parameters including the camera parameters of each virtual camera C and the position and orientation of each calibration pattern 90.

The camera parameter optimization means 43 optimizes the camera parameters between each virtual camera C through a camera calibration process using the initial camera parameters input from the initial camera parameter calculation means 42 as initial values. The initial camera parameter calculation means 42 described above calculates the camera parameters of each virtual camera C individually, but optimizing the camera parameters between all virtual cameras C improves the accuracy of the camera parameters.

Here, the position and orientation of the calibration pattern 90 are the common parameters. The camera parameters to be optimized include the camera parameters of each virtual camera C and the position and orientation of the common calibration pattern 90. Specifically, the camera parameter optimization means 43 uses the average values of the camera parameters of each virtual camera C and the position and orientation of the calibration pattern 90 as initial values, and uses the position and orientation of the virtual camera C included in the initial camera parameters. The camera parameter optimization means 43 then optimizes the camera parameters by bundle adjusting these initial values.

The scale conversion function calculation means 44 calculates a scale conversion function by associating the distance from the position of the virtual camera C indicated by the camera parameters input from the camera parameter optimization means 43 to the calibration pattern 90 with the pixel value of each pixel of the depth image P _D. That is, the scale conversion function calculation means 44 calculates a scale conversion function for converting the depth image P _D into a real-scale depth map. As described above, since the position and orientation of the virtual camera C and the position and orientation of the calibration pattern 90 are known in the camera parameters, the distance r from the virtual camera C to the calibration pattern 90 can be calculated in real scale.

Specifically, as shown in FIG. 5A, the scale conversion function calculation means 44 associates the distance r from the virtual camera C to the calibration pattern 90 with the brightness value q (pixel value) of each pixel of the depth image P _D. At this time, the calibration pattern 90 included in the depth image P _D has a low reflectance in the black pattern portion, making it difficult to perform accurate association. For this reason, it is preferable that the scale conversion function calculation means 44 performs association only in the white portion of the calibration pattern 90 included in the depth image P _D. Here, the scale conversion function calculation means 44 performs association in all depth images P _D in which the calibration pattern 90 is captured, thereby obtaining a graph as shown in FIG. 5B. Then, the scale conversion function calculation means 44 can calculate the scale conversion function h(q) by approximating this graph with a function (for example, a quintic function). Note that the scale conversion function calculation means 44 may use a look-up table instead of approximating this graph with a scale conversion function.

Then, the camera calibration means 4 outputs the calculated scale conversion function to the scale conversion means 54, and outputs the camera parameters of the virtual camera C to the cost volume generation means 51 and the weight application means 59.

<Refinement Means>
The refinement means 5 receives an RGB image P _C and a depth image P _D captured by the RGB-D camera 2 of a subject 9. The refinement means 5 improves the accuracy of the depth map by restricting the cost volume generated from the RGB image P _C with two weights based on the depth map generated from the depth image P _D. The refinement means 5 performs refinement processing each time an image is captured.

As shown in FIG. 2, the refinement means 5 includes an image division means 50, a cost volume generation means 51, an initial depth map generation means 52, a smoothing means 53, a scale conversion means (depth conversion means) 54, a layering processing means 55, a scale correction means (intermediate depth map correction means) 56, a cost weight calculation means 57, a visibility weight calculation means 58, a weight application means 59, and a final depth map generation means 60.

The image division means 50 divides the RGB image P _C and the depth image P _D input from the RGB-D camera 2 for each viewpoint. As shown in Figures 6(a) and 6(b), the image division means 50 divides the RGB image P _C and the depth image P _D of the subject 9, similar to the image division means 41.

6, the subject 9 appears as an inverted image in the RGB image _PC and the depth image _PD due to the lens system 21. In this case, the RGB image _PC and the depth image _PD may be subjected to an inversion process so that the subject 9 appears as an upright image.

The cost volume generating means 51 calculates a cost for each pixel position of a depth layer and an RGB image _PC described below, and generates a cost volume in which the costs are arranged three-dimensionally by the depth layer and the pixel position. In this embodiment, the cost volume generating means 51 uses a plane sweep method, which is one of the methods for estimating a cost volume (Reference 3).

Reference 3: David Gallup, et al., "Real-time plane-sweeping stereo with multiple sweeping directions", IEEE Conference on Computer Vision and Pattern Recognition, pp. 1-8 (2007)

First, as shown in FIG. 7, the cost volume generating means 51 sets a plurality of depth layers N _D at a predetermined interval in the depth direction in the space in which the subject 9 is arranged. In the example of FIG. 7, five depth layers N _D are set (D=1, ..., 5). In FIG. 7, the x-axis indicates the horizontal direction, the y-axis indicates the vertical direction, and the z-axis indicates the depth direction. Next, the cost volume generating means 51 sets a camera pair with one of all virtual cameras C as a reference camera and another virtual camera C. Then, the cost volume generating means 51 projects the RGB images P _C of each virtual camera C constituting the camera pair onto the depth layer _{N D} by projective transformation. Furthermore, the cost volume generating means 51 calculates the cost by calculating the difference (for example, SAD: Sum of Absolute Difference) between the pixel values of each pixel of the two RGB images P _C projected onto the depth layer N D. This cost represents the similarity between the two RGB images P _C projected onto the depth layer N _D , and the smaller the value, the higher the possibility that the depth of the subject 9 exists in the depth layer N _D .

The cost volume generating means 51 can generate a cost volume by performing the above-mentioned process for all depth layers N _D. As shown in FIG. 8, if the size of the RGB image P _C is U×V pixels, the cost volume 91 is a three-dimensional array of U×V×N _D costs. In addition, in the cost volume 91, costs arranged in the depth direction at the same pixel position are set as a cost column 92. In other words, the cost column 92 is a three-dimensional array of 1×1×N _D costs. Then, the cost volume generating means 51 applies a guided filter to the cost volume 91 using the RGB image P _C of the reference camera as a guide (Reference 4). This allows the cost volume 91 to be smoothed while maintaining edges, thereby reducing noise in the cost volume 91.

Reference 4: Kaiming He, Sun Jian, and Tang Xiaoou, "Guided image filtering", European conference on computer vision. Springer, pp. 1-10, (2010)

If a set of virtual cameras C around a reference camera is S, then the number of camera pairs that can be set is the number of elements in the set |S|. In this case, the number of cost volumes 91 can be the same as the number of camera pairs. For example, when there are four virtual cameras C, there are three camera pairs for one reference camera, and there are also three cost volumes 91. For example, when virtual camera _C1 is the reference camera, the camera pairs are ( _C1 , _C2 ), ( _C1 , _C3 ), and ( _C1 , _C4 ).

The initial depth map generating means 52 generates an initial depth map indicating the depth of the depth layer _ND having the smallest cost in the cost column 92 at the same pixel position in the cost volume 91 input from the cost volume generating means 51 .

Here, since there are multiple cost volumes 91 for one reference camera, the initial depth map generating means 52 calculates the sum of each cost volume 91 as the final cost volume 91 of the reference camera. Then, the initial depth map generating means 52 calculates the depth layer N _D having the smallest cost in each cost column 92 as the correct depth, and generates an initial depth map D ^C of the reference camera.

The initial depth map generator 52 then outputs the initial depth map ^DC to the scale corrector 56 and outputs the final cost volume 91 to the weight applier 59 .

The smoothing means 53 smoothes the depth image _PD input from the image dividing means 50. Here, since the depth image _PD contains noise such as shot noise of the depth camera, the smoothing means 53 smoothes the depth image _PD by filtering. For example, a guided filter is used as the filtering process. This guided filter is a type of smoothing filter, and smoothes the target image using a guide image. Here, an RGB image _PC is used as the guide image.

Although noise can be removed by filtering, excessive smoothing may reduce the accuracy of the depth image _PD . For this reason, the smoothing means 53 may perform filtering only as necessary.

The scale conversion means 54 converts the depth image P _D into an intermediate depth map by a scale conversion function that converts the pixel value of each pixel of the depth image P _D into a real-scale depth. In this embodiment, the scale conversion means 54 converts the depth image P _D input from the smoothing means 53 into a real-scale depth map by the scale conversion function input from the scale conversion function calculation means 44. Note that, when a scale conversion function is provided by the manufacturer of the RGB-D camera 2, the scale conversion means 54 may use this function.

The layering processing means 55 performs layering processing to replace the depth of the intermediate depth map input from the scale conversion means 54 with the depth of the closest depth layer N _D. Specifically, since the camera parameters are known, the layering processing means 55 can convert the intermediate depth map of real scale into a three-dimensional point cloud. Here, when the intermediate depth map represents a distance from the optical center, rather than a distance in the optical axis direction (generally the z direction) in the camera coordinate system, the layering processing means 55 converts the intermediate depth map into a three-dimensional point cloud taking into account the distance. Then, the layering processing means 55 represents the intermediate depth map in the depth layer N _D by making the depth of each point belong to the closest depth layer N _D. Hereinafter, the intermediate depth map subjected to layering processing is referred to as D ^D.

The scale correction means 56 calculates an average of the depth differences in each depth layer N _D as a correction value for pixels where the depth difference between the initial depth map D ^C and the intermediate depth map D ^D is equal to or less than a threshold, and corrects the depth of the intermediate depth map D ^D with the correction value. In other words, when the accuracy of the scale conversion function is low, the scale correction means 56 corrects the intermediate depth map D ^D generated from the depth image P _D to match it with the initial depth map D ^C generated from the RGB image P _C.

Specifically, the scale correction means 56 calculates the depth difference D ^Sub = D ^C - D ^D for each pixel between the initial depth map D ^C and the intermediate depth map D ^D. Next, the scale correction means 56 calculates the average of the depth differences D Sub at each depth d (d = 1, 2, ..., N _D ) of the initial depth map D ^C for only pixels that satisfy | ^{D Sub |} ≤ threshold, and sets the average as a correction value. Note that the threshold value threshold is set manually. Then, the scale correction means 56 applies the correction depth value D ^Cor to the intermediate depth map D ^D _Old before correction, such as D ^D _New = D ^D _Old + D ^Cor , to obtain the intermediate depth map D ^D _New after correction (hereinafter, intermediate depth map D ^D ).
It should be noted that the scale correction means 56 does not need to perform the process if the scale conversion function has high accuracy.

The cost weight calculation means 57 calculates a cost weight W ^C which represents the weight of the intermediate depth map D ^D input from the scale correction means 56 by a normal distribution function. As described above, the cost volume 91 is generated only from the RGB image P _C , and does not take the depth map into consideration. Therefore, by applying the cost weight W ^C calculated from the intermediate depth map D ^D to the cost volume 91, the cost volume 91 takes into consideration both the RGB image P _C and the depth map.

The cost weight W ^C is expressed as a normal distribution with the weight of the intermediate depth map D ^D as the minimum value, assuming that the intermediate depth map D D has a high possibility of having a correct depth value. As shown in Fig. 9, the maximum value of the normal distribution is 1, and the normal distribution function g(d) of the depth layer d is defined by the following formula (1).

Here, μ represents the mean, ^σ2 represents the variance, and σ represents the standard deviation. Using this normal distribution function g(d), a cost weight function f _C (d) is defined by the following formula (2). Note that a _c is a parameter that determines the cost weight W ^C. Also, as shown in FIG. 10(a), in the normal distribution function g(d) of formula (2), the mean μ represents the mean of the depth values D ^D (u, v) of the pixels (u, v) of the intermediate depth map D ^D , and the variance σ ² is set in advance according to the design policy of the cost weight function f _C (d) (for example, σ ² = N _D /3).

The cost weight W ^C is a three-dimensional array of the same size as the cost volume 91. Each element of the cost weight W ^C contains the value of the cost weight function f _C (d), as shown in the following formula (3). From the above, the cost weight calculation means 57 calculates the cost weight W ^C using formula (3).

The visibility weight calculation means 58 calculates a visibility weight ^WV that reduces the cost when occlusion occurs, from the intermediate depth map D ^D input from the cost weight calculation means 57 .

Here, occlusion was not taken into consideration when the cost volume 91 was generated, and the cost of the occluded parts becomes noise, causing an error in the layering process described above. This error also occurs when the sum of the cost volumes 91 is calculated for multiple camera pairs. Note that occlusion refers to the occurrence of an area that is visible from one virtual camera C but not visible from the other virtual camera C.

On the other hand, the intermediate depth map D 1 ^D is not affected by occlusion because it is generated from a single depth camera. Therefore, the visibility weight calculation means 58 calculates a visibility weight W 1 ^V from the intermediate depth map D ^{1 D} in order to mitigate the effect of occlusion (to reduce the cost of the portion where occlusion occurs).

As shown in FIG. 10B, the visibility weight function f _V (d) is defined by the following formula (4). Note that a _V is a parameter that determines the visibility weight W ^V. In the normal distribution function g(d) of formula (4), the mean μ represents a value D ^D (u, v)+shift obtained by adding a constant shift to the average of the depth values D ^D (u, v) (where shift ≧0). In addition, the variance σ ² is set in advance according to the design policy of the visibility weight function f _V (d) (for example, σ ² = N _D /10). By increasing the value of the constant shift, the presence of an error in the intermediate depth map D ^D is tolerated, but the effect of the visibility weight W ^V is reduced.

The visibility weight ^WV is a three-dimensional array of the same size as the cost volume 91. Each element of the visibility weight ^WV contains the value of the visibility weight function _fV (d) as shown in the following formula (5). From the above, the visibility weight calculation means 58 calculates the visibility weight ^WV of formula (5).

The weight application means 59 applies the cost weight W ^C and the visibility weight W ^V to the cost volume 91 input from the initial depth map generation means 52. Here, the final cost volume E ^S is the cost volume 91 integrated for all camera pairs with the reference camera C. That is, the weight application means 59 calculates the final cost volume E ^S using the cost weight W ^C (x, y, z), cost volume E _j , and visibility weight W ^V of the reference camera, as shown in the following formula (6).

The cost volume _Ej is a cost volume 91 between the reference camera C and a virtual camera _Cj (jεS) included in the surrounding camera set S. Furthermore, warp represents a projective transformation from the virtual camera _Cj to the reference camera C with each depth layer N _D as a plane.

The final depth map generating means 60 generates a final depth map indicating the depth of the depth layer N _D having the smallest cost in the cost column 92 at the same pixel position in the cost volume 91 input from the weight application means 59. In other words, the final depth map generating means 60 determines the depth layer N _D having the smallest cost in each cost column 92 as the correct depth, and generates the final depth map.
It should be noted that the final depth map generating means 60 generates the final depth map in a similar manner to that of the initial depth map generating means 52, and therefore further explanation thereof will be omitted.

Thereafter, the refinement means 5 outputs a set of the RGB images P _C for each viewpoint, the final depth map, and the camera parameters of the virtual camera C input from the camera calibration means 4 .

[Camera calibration process]
The camera calibration process will be described with reference to FIG.
11, in step S1, the angle-of-view correction means 40 performs projective transformation on the depth image P _D input from the RGB-D camera 2 so that the angle of view of the depth image P _D matches the angle of view of the RGB image P _C. Note that the process of step S1 is not essential and is therefore illustrated by a dashed line.

In step S2, the image dividing means 41 divides the RGB image _PC and the depth image P _D for each viewpoint.
In step S3, the initial camera parameter calculation means 42 performs camera calibration processing on the RGB image _PC of each viewpoint to calculate initial camera parameters of the virtual camera C corresponding to each viewpoint.
In step S4, the camera parameter optimization means 43 optimizes the camera parameters between the virtual cameras C by a camera calibration process using the initial camera parameters as initial values.
In step S5, the scale conversion function calculation means 44 calculates a scale conversion function by associating the distance from the position of the virtual camera C indicated by the camera parameters to the calibration pattern with the pixel value of each pixel of the depth image _PD .

[Refinement Processing]
The refinement process will now be described with reference to FIG.
As shown in FIG. 12, in step S10, the image dividing means 50 divides the RGB image P _{1 C} and the depth image P _{1 D} for each virtual camera C.
In step S11, the cost volume generating means 51 calculates the cost for each pixel of the depth layer and the RGB image _PC , and generates a cost volume 91 which is a three-dimensional array of the costs.

In step S<b>12 , the initial depth map generating means 52 generates an initial depth map indicating the depth of the depth layer with the smallest cost in the cost column 92 at the same pixel position in the cost volume 91 .
The processes in steps S11 and S12 and the processes in steps S13 to S18 described below can be executed in parallel.

In step S13, the smoothing means 53 smoothes the depth image _PD .
In step S14, the scale conversion means 54 converts the depth image _PD into an intermediate depth map using a scale conversion function that converts the pixel value of each pixel of the depth image _PD into a real-scale depth.
In step S15, the layering processing means 55 performs layering processing to replace the depth of the intermediate depth map with the depth of the closest depth layer.

In step S16, the scale correction means 56 calculates an average of the depth differences in each depth layer N _D as a correction value for pixels where the depth difference between the initial depth map D ^C and the intermediate depth map D ^D is equal to or less than a threshold value, and corrects the depth of the intermediate depth map D ^D with the correction value. Note that the process of step S16 is not essential and is therefore illustrated by a dashed line.
In step S17, the cost weight calculation means 57 calculates a cost weight W ^C which represents the weight of the intermediate depth map D ^D by a normal distribution function.
In step S18, the visibility weight calculation means 58 calculates, from the intermediate depth map ^D1D , a visibility weight ^W1V that reduces the cost when occlusion occurs.

In step S 19 , the weight application means 59 applies the cost weight ^W_C and the visibility weight ^W_V to the cost volume 91 .
In step S20, the final depth map generating means 60 generates a final depth map indicating the depth of the depth layer _ND that has the smallest cost in the cost column 92 at the same pixel position in the cost volume 91.

[Action and Effects]
As described above, the three-dimensional shape acquisition system 1 can easily acquire the RGB images P _C from multiple viewpoints, the highly accurate depth map, and the camera parameters of the virtual camera C. That is, the three-dimensional shape acquisition system 1 realizes a simple system configuration and can provide the RGB images P _C from multiple viewpoints, the highly accurate depth map, and the camera parameters of the virtual camera C. These data can be used in various applications. For example, when generating a three-dimensional image, a dense multi-view RGB image is required. The data provided by the three-dimensional shape acquisition system 1 includes the camera parameters of the virtual camera C and the highly accurate depth map, so that a three-dimensional image can be generated by simple processing.

Although the embodiment of the present invention has been described in detail above, the present invention is not limited to this, and includes design modifications and the like that do not deviate from the gist of the present invention.

In the above embodiment, the depth camera is described as a ToF camera, but this is not limited to this. For example, the depth camera may be a stereo camera.

The present invention can also be realized by a program that causes hardware resources such as a CPU, memory, and hard disk of a computer to operate as the above-mentioned three-dimensional shape acquisition device. These programs may be distributed via a communication line, or written onto a recording medium such as a CD-ROM or flash memory and distributed.

1. 3D shape acquisition system (depth map generation system)
2 RGB-D camera (photography device)
20 Camera body 21 Lens system 22 Fresnel lens 23 Lens array 24 Element lens 25 Infrared LED array 3 Three-dimensional shape acquisition device (depth map generation device)
4 Camera calibration means 40 View angle correction means 41 Image division means 42 Initial camera parameter calculation means 43 Camera parameter optimization means 44 Scale conversion function calculation means (depth conversion function calculation means)
5 Refinement means 50 Image division means 51 Cost volume generation means 52 Initial depth map generation means 53 Smoothing means 54 Scale conversion means (depth conversion means)
55 Layering processing means 56 Scale correction means (intermediate depth map correction means)
57 Cost weight calculation means 58 Visibility weight calculation means 59 Weight application means 60 Final depth map generation means 9 Object 90 Calibration pattern 91 Cost volume 92 Cost sequence C Virtual camera D ^C Initial depth map D ^D Intermediate depth map N _D Depth layer

Claims (8)

A depth map generating device that generates a depth map corresponding to the captured images at each viewpoint using captured images and depth images captured by an imaging device including a imaging camera and a depth camera on the same optical axis and an optical element array, the device comprising:
a cost volume generating means for calculating a cost representing a similarity between the captured images projected onto the depth layer for each pixel position of the depth layer and the captured images at a predetermined interval in the depth direction, and generating a cost volume in which the costs are three-dimensionally arranged in the depth layer and the pixel positions;
a depth conversion means for converting the depth image into an intermediate depth map using a depth conversion function that converts a pixel value of each pixel of the depth image into a depth;
a cost weight calculation means for calculating a cost weight obtained by expressing the weight of the intermediate depth map as a normal distribution function;
a visibility weight calculation means for calculating a visibility weight that reduces the cost when an occlusion occurs from the intermediate depth map;
weight application means for applying the cost weight and the visibility weight to the cost volume;
a final depth map generating means for generating a final depth map indicating a depth of the depth layer in which the cost is minimum in a cost sequence at the same pixel position in the cost volume after applying a weight;
A depth map generating device comprising: A smoothing means for smoothing the depth image is further provided,
2. The depth map generating device according to claim 1, wherein the depth conversion means converts the depth image smoothed by the smoothing means into the intermediate depth map using the depth conversion function. an initial depth map generating means for generating an initial depth map indicating a depth of the depth layer in which the cost is minimum in a cost sequence at the same pixel position in the cost volume generated by the cost volume generating means;
an intermediate depth map correction means for calculating an average of depth differences between the depth layers as a correction value for pixels in which a depth difference between the initial depth map and the intermediate depth map is equal to or smaller than a threshold, and correcting the depth of the intermediate depth map with the correction value;
The depth map generating device according to claim 1 or 2, further comprising: A layering processing means for performing a layering process of replacing a depth of the intermediate depth map with a depth of the closest depth layer,
4. The depth map generating device according to claim 3, wherein the intermediate depth map correcting means corrects the depth of the intermediate depth map, which has been subjected to layering processing by the layering processing means, with the correction value. an initial camera parameter calculation means for calculating initial camera parameters of a virtual camera corresponding to each viewpoint by performing a camera calibration process on a captured image of a calibration pattern captured by the image capture device from each viewpoint;
a camera parameter optimization means for optimizing camera parameters between the virtual cameras through the camera calibration process using the initial camera parameters as initial values;
a depth conversion function calculation means for calculating the depth conversion function by making the distance from the position of the virtual camera indicated by the optimized camera parameters to the calibration pattern correspond to the pixel value of each pixel of the depth image;
The depth map generating device according to claim 1 , further comprising: and a field-of-view correction unit that performs projective transformation on the depth image so that the field of view of the depth image, which is obtained by photographing the calibration pattern from each viewpoint by the photographing device, coincides with the field of view of the photographed image,
The depth map generating device according to claim 5, characterized in that the depth conversion function calculation means calculates the depth conversion function by making the depth from the position of the virtual camera to the calibration pattern correspond to the pixel values of each pixel of the depth image projected by the angle of view correction means. A program for causing a computer to function as a depth map generating device according to any one of claims 1 to 6. A photographing device including a photographing camera and a depth camera on the same optical axis and an optical element array;
A depth map generating device according to any one of claims 1 to 6,
A depth map generating system comprising:

Images