JP2022024688A - Depth map generation device and program thereof, and depth map generation system - Google Patents

Abstract

To provide a depth map generation device capable of easily acquiring photographic images at a plurality of viewpoints and a highly accurate depth map.SOLUTION: A three-dimensional shape acquisition device 3 includes cost volume generation means 51 for generating a cost volume, scale conversion means 54 for converting a depth image into an intermediate depth map using a scale conversion function, cost weight calculation means 57 for calculating a cost weight, visibility weight calculation means 58 for calculating a visibility weight, weight application means 59 for applying the cost weight and the visibility weight to the cost volume, and final depth map generation means 60 for generating a final depth map indicating the depth of a depth layer that minimizes the cost in a cost column at the same pixel position in the cost volume.SELECTED DRAWING: Figure 2

Description

The present invention relates to a depth map generator and a program thereof for generating a depth map, and a depth map generation system.

In recent years, techniques for acquiring a three-dimensional shape (depth map) of a subject existing in space have been actively studied. This technology is expected to be applied to various fields such as 3D video production, AR (Augmented Reality), VR (Virtual Reality), and robotics. The approach for acquiring the three-dimensional shape of the subject is roughly classified into an active method and a passive method (Non-Patent Document 1).

In the active method, the measuring device has a light source and the depth is measured by using the reflected light from the subject. Specific methods include pattern light projection, light flight time method (ToF: Time of Flight), and illuminance difference stereo method. Among these, the method using a ToF camera has been attracting attention in recent years. The ToF camera obtains the distance from the ToF camera to the subject by measuring the time until the light emitted from the light source is reflected by the subject and returned. The merit of the active method is that a highly accurate distance can be obtained in real time without performing advanced calculation processing. On the other hand, the disadvantages of the active method are that it is vulnerable to ambient light, measurement errors occur depending on the reflectance and distance of the subject, and scale calibration may be required.

The passive method is to move a plurality of color cameras (hereinafter referred to as "RGB cameras") or one RGB camera and measure the depth distance from the parallax. Specific methods include a stereo method (multi-eye stereo) and a motion stereo. These principles are the stereo method, which calculates the depth from the parallax of two or more cameras. The merit of the passive method is that it does not need to irradiate the subject with special light, is not affected by ambient light, and can be realized only with a general color camera and computer. On the other hand, the disadvantage of the passive method is that the obtained depth remains ambiguous (textureless, occlusion area) and the calculation cost is high.

In addition, there is known an RGB-D camera capable of acquiring RGB images and depth images for a plurality of viewpoints by arranging an RGB camera and a depth camera on the same optical axis and using a lens array (Patent Document 1). In this method, light rays incident from a camera lens are separated by a mirror (for example, a half mirror or a dichroic mirror) and received by an RGB camera and a depth camera.

Japanese Unexamined Patent Publication No. 2009-300268

Digital Image Processing (Revised New Edition), CG-ARTS Association, 2015

As described above, since the acquisition of a three-dimensional shape has a wide range of applications, various methods have been proposed, but have not yet been established. Considering a general purpose, it is convenient to have RGB images and depth maps of various viewpoints as well as color images (hereinafter, RGB images) and depth maps of one viewpoint. That is, having a set of RGB images and depth maps from a plurality of viewpoints improves versatility.

The accuracy of the depth map is also important. Since the depth image obtained by the RGB-D camera is represented by a pixel value (luminance value), it is necessary to convert this pixel value into a depth map of an actual scale. However, the conversion function to real scale has a great influence on the accuracy of the depth map. Furthermore, the accuracy of the depth map is also affected by the shooting environment and the type of subject. The actual scale is a distance (depth) in real space.

An object of the present invention is to provide a depth map generation device and a program thereof, and a depth map generation system, which can solve the above-mentioned problems and easily acquire a photographed image of a plurality of viewpoints and a highly accurate depth map.

In order to solve the above-mentioned problems, the depth map generation device according to the present invention is a photographed image and a depth image in which a photographing device composed of a photographing camera having the same optical axis and a depth camera and an optical element array captures a subject from each viewpoint. It is a depth map generation device that generates a depth map corresponding to a captured image of each viewpoint by using, and is a cost volume generation means, a depth conversion means, a cost weight calculation means, a visibility weight calculation means, and a weight application. The configuration includes a means and a final depth map generation means.

According to such a configuration, the cost volume generating means calculates the cost representing the similarity between the captured images projected on the depth layer for each of the depth layers and the pixel positions of the captured images at predetermined intervals in the depth direction, and calculates the cost. Generate a cost volume that is three-dimensionally arranged by the depth layer and the pixel position.
The depth conversion means converts the depth image into an intermediate depth map by a depth conversion function that converts the pixel value of each pixel of the depth image into the depth.
The cost weight calculation means calculates the cost weight in which the weight of the intermediate depth map is expressed by a normal distribution function.

In addition, the visibility weight calculation means calculates the visibility weight that reduces the cost when occlusion occurs from the intermediate depth map.
The weight application means applies the cost weight and the visibility weight to the cost volume.
The final depth map generation means generates a final depth map showing the depth of the depth layer that minimizes the cost in the cost column at the same pixel position in the cost volume after weight application.

That is, the depth map generation device performs refinement processing in which the cost volume generated from the captured image is constrained by two weights based on the depth map generated from the depth image. By this refinement processing, the depth map generator can generate a highly accurate depth map corresponding to the captured image of each viewpoint.

The present invention can be realized by a program for making a computer function as the depth map generator described above.

Further, the present invention is realized by a depth map generation system including a photographing device including a photographing camera having the same optical axis, a depth camera, and an optical element array, and the depth map generating device described above. You can also do it.

According to the present invention, it is possible to easily acquire a photographed image from a plurality of viewpoints and a highly accurate depth map.

It is an overall block diagram of the 3D shape acquisition system which concerns on embodiment. It is a block diagram which shows the structure of the 3D shape acquisition apparatus which concerns on embodiment. It is explanatory drawing explaining the photographing of the calibration pattern by the RGB-D camera, (a) shows calibration data A, and (b) shows calibration data B. It is explanatory drawing explaining the division of the image which photographed the calibration pattern, (a) shows an RGB image, (b) shows a depth image. It is explanatory drawing explaining the calculation of a scale conversion function, (a) shows the distance from a virtual camera to a calibration pattern, and (b) shows an example of a scale conversion function. It is explanatory drawing explaining division of the image which photographed a subject, (a) shows an RGB image, (b) shows a depth image. It is explanatory drawing explaining an example of a depth layer. It is explanatory drawing explaining the cost volume. It is explanatory drawing explaining a normal distribution function. (A) is an explanatory diagram for explaining an example of a cost weight function, and (b) is an explanatory diagram for explaining an example of a visibility function. In the embodiment, it is a flowchart which shows the camera calibration process. In the embodiment, it is a flowchart which shows the refinement procedure.

Hereinafter, embodiments of the present invention will be described with reference to the drawings. However, the embodiments described below are for embodying the technical idea of the present invention, and the present invention is not limited to the following unless otherwise specified.

[Overview of 3D shape acquisition system]
An outline of the three-dimensional shape acquisition system (depth map generation system) 1 according to the embodiment will be described with reference to FIG.
The three-dimensional shape acquisition system 1 acquires RGB images (photographed images) and depth maps of a plurality of viewpoints and camera parameters of the virtual camera C for the subject 9. As shown in FIG. 1, the three-dimensional shape acquisition system 1 includes an RGB-D camera (shooting device) 2 and a three-dimensional shape acquisition device (depth map generation device) 3.

If a large number of RGB cameras and depth cameras are arranged for shooting from multiple viewpoints, the system becomes large and the cost becomes high. Therefore, in the three-dimensional shape acquisition system 1, one RGB-D camera (shooting device) 2 described later realizes a configuration equivalent to arranging a large number of RGB cameras and depth cameras, and simplifies the system configuration. can.

In fields such as 3D video production, the camera parameters of the virtual camera C are required. Further, since the depth image is represented by a pixel value (luminance value), a scale conversion function for converting this pixel value into a depth map of an actual scale is also required. Therefore, in the three-dimensional shape acquisition system 1, the three-dimensional shape acquisition device 3 performs camera calibration processing using the calibration pattern, and calculates the camera parameters and the scale conversion function of the virtual camera C.

The accuracy of the depth map is also important. As mentioned above, the scale conversion function has a great influence on the accuracy of the depth map. Furthermore, the accuracy of the depth map is greatly reduced depending on the shooting environment and the type of subject. Therefore, in the three-dimensional shape acquisition system 1, the accuracy of the depth map is improved by using the RGB image and the depth image of a plurality of viewpoints by the three-dimensional shape acquisition device 3 described later (refining process). At this time, in the three-dimensional shape acquisition device 3, one RGB image taken by one RGB-D camera 2 is divided and matched for each viewpoint, so that the images taken by a plurality of RGB cameras are matched. Compared to the case, the error caused by the color difference can be suppressed.

First, the configuration of the RGB-D camera 2 will be described. Next, the camera calibration process by the three-dimensional shape acquisition device 3 will be described. This camera calibration process is a process of calculating the camera parameters and the scale conversion function of each virtual camera C. Finally, the refinement process for improving the accuracy of the depth map by the three-dimensional shape acquisition device 3 will be described.

[Structure of RGB-D camera]
As shown in FIG. 1, the RGB-D camera 2 is an imaging device including a camera body 20 and a lens system 21. In the present embodiment, the camera body 20 has an RGB camera and a depth camera (not shown) arranged on the same optical axis. Further, the camera body 20 disperses the light rays from the subject 9 by a spectroscopic element (not shown), and the separated light rays are received by the RGB camera and the depth camera, respectively. For example, as an RGB camera, a general color camera can be mentioned. Further, examples of the spectroscopic element include a half mirror or a dichroic mirror.

In this embodiment, a ToF camera is used as the depth camera. This ToF camera includes an infrared LED array 25 for irradiating the subject 9 with infrared rays at the time of distance measurement. A depth image can be acquired by obtaining the difference between frames of the infrared image taken by the ToF camera.

The lens system 21 includes a Fresnel lens 22 and a lens array (optical element array) 23. The lens array 23 is a two-dimensional arrangement of _NX × _NY element lenses 24. The RGB-D camera 2 can acquire RGB images and depth images for _NX × _NY viewpoints via the lens array 23. That is, the RGB-D camera 2 realizes a configuration equivalent to the arrangement of _NX × _NY virtual cameras C. In the present embodiment, it is assumed that there are four viewpoints (four virtual cameras C) corresponding to the 2 × 2 element lenses 24.

By adjusting the positional relationship between the camera body 20 and the lens system 21, the angle of view of the virtual camera C can be adjusted. Further, in FIG. 1, only two virtual cameras C out of the four virtual cameras C are shown.

[Configuration of 3D shape acquisition device]
The configuration of the three-dimensional shape acquisition device 3 will be described with reference to FIG.
The three-dimensional shape acquisition device 3 generates a depth map corresponding to the RGB image of each viewpoint by using the RGB image and the depth image of the subject 9 taken by the RGB-D camera 2 at each viewpoint. As shown in FIG. 2, the three-dimensional shape acquisition device 3 includes a camera calibration means 4 for performing a camera calibration process and a refinement means 5 for performing a refinement process.

<Camera calibration means>
The camera calibration means 4 estimates two types of parameters. The first is the camera parameters of the virtual camera C. The camera parameters of the virtual camera C represent the focal length of the lens, the lens distortion, the position and orientation of the virtual camera C, and the like. The second is the scale conversion function of each virtual camera C. Further, the camera calibration means 4 corrects the angle of view of the RGB image and the depth image as necessary. The camera calibration means 4 does not need to perform camera calibration processing each time a photograph is taken, and is related to the focal length of the RGB-D camera 2 and the position / orientation of the RGB-D camera 2, the Frenel lens 22 and the lens array 23. The camera may be calibrated when the value changes.

As shown in FIG. 3A, an RGB image and a depth image obtained by capturing the calibration pattern 90 with the RGB-D camera 2 are input to the camera calibration means 4. The calibration pattern 90 is a planar pattern in which the arrangement of feature points is known (for example, a chess board pattern). At this time, the RGB-D camera 2 changes the posture of the calibration pattern 90 two or more times to take a picture (shown by a broken line). When the skew of the internal parameter is set to other than 0, the RGB-D camera 2 changes the posture of the calibration pattern 90 three times or more to take a picture. As shown in FIG. 3A, the RGB image and the depth image taken by arranging the lens system 21 are referred to as calibration data A. When performing the above-mentioned angle of view correction, as shown in FIG. 3B, the lens system 21 is removed and the calibration pattern 90 is photographed. The RGB image and the depth image taken with the lens system 21 removed in this way are referred to as calibration data B.

As shown in FIG. 2, the camera calibration means 4 includes an angle of view correction means 40, an image division means 41, an initial camera parameter calculation means 42, a camera parameter optimization means 43, and a scale conversion function calculation means (depth conversion). The function calculation means) 44 is provided.

The angle of view correction means 40 projects and converts the depth image so that the angle of view of the depth image input from the RGB-D camera 2 matches the angle of view of the RGB image. Due to the mounting accuracy of the RGB-D camera 2, the angle of view between the RGB image taken by the RGB camera and the depth image taken by the depth camera may be slightly different. Therefore, the angle of view correction means 40 uses the calibration data B to correct this delicate angle of view deviation. Specifically, the angle-of-view correction means 40 calculates a homography matrix based on four or more corresponding points (characteristic points of the calibration pattern 90) between the RGB image and the depth image (Reference 1). Then, the angle of view correction means 40 projects and transforms the depth image by this homography matrix to match the angle of view of the depth image with the angle of view of the RGB image.
The angle of view correction means 40 does not need to perform the above-mentioned angle of view correction process when the angle of view of the RGB camera and the depth camera match.

Reference 1: "OpenCV", [online], [Search on June 24, 2nd year of Reiwa], Internet <URL: https://opencv.org/>

Further, the angle of view correction means 40 can remove the lens distortion by using the calibration data B. For example, the angle of view correction means 40 calculates the lens distortion coefficient of the RGB-D camera 2 by the method of Zhang, and removes the lens distortion from the RGB image and the depth image (Reference 2).

Reference 2: Z. Zhang, “A flexible new technique for camera calibration”, IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 22, No. 11, pp. 1330-1334 (2000)

The image segmentation means 41 divides the RGB image and the depth image input from the angle of view correction means 40 for each viewpoint (element lens 24). That is, the image segmentation means 41 divides the RGB image and the depth image into each virtual camera C to generate the RGB image and the depth image virtually taken by the virtual camera C. In the present embodiment, the image segmentation means 41 divides the RGB image _{PC and the depth image P D into} four as shown in FIGS. 4A and 4B.

The area α for dividing the RGB image _{PC and the depth image P D is} manually set. At this time, since the RGB image _CC and the depth image P _D after division do not require a gap on the outside of the lens array 23 or between the element lenses 24, it is not necessary to divide these unnecessary regions. For the sake of simplification of the following description, it is assumed that the RGB image _PC and the depth image _PD after division have the same image size.

The initial camera parameter calculation means 42 calculates the initial camera parameters of the virtual camera _C corresponding to each viewpoint by performing a camera calibration process on the RGB image PC of each viewpoint input from the image segmentation means 41. .. For example, the initial camera parameter calculation means 42 applies Zhang's method to the RGB image PC of each viewpoint, and calculates the camera parameters of each virtual camera _C and the initial camera parameters including the position / orientation of each calibration pattern 90. ..

The camera parameter optimizing means 43 optimizes the camera parameters among the virtual cameras C by the camera calibration process using the initial camera parameters input from the initial camera parameter calculating means 42 as initial values. In the above-mentioned initial camera parameter calculation means 42, the camera parameters of each virtual camera C are calculated individually, but by optimizing the camera parameters among all the virtual cameras C, the accuracy of the camera parameters is improved. ..

Here, the position and orientation of the calibration pattern 90 are set as common parameters. The camera parameters to be optimized include the camera parameters of each virtual camera C and the position / orientation of the common calibration pattern 90. Specifically, the camera parameter optimizing means 43 sets the position / posture of the virtual camera C included in the initial camera parameters as the initial value with the average value of the position / posture of the camera parameter of each virtual camera C and the calibration pattern 90 as the initial value. use. Then, the camera parameter optimizing means 43 optimizes the camera parameters by bundle-adjusting these initial values.

The scale conversion function calculation means 44 makes the distance from the position of the virtual camera _C indicated by the camera parameters input from the camera parameter optimization means 43 to the calibration pattern 90 correspond to the pixel value of each pixel of the depth image PD. , Calculates the scale conversion function. That is, the scale conversion function calculation means 44 calculates a scale conversion function for converting the depth image P _D into a depth map of an actual scale. As described above, since the position / posture of the virtual camera C and the position / posture of the calibration pattern 90 are known in the camera parameters, the distance r from the virtual camera C to the calibration pattern 90 can be calculated on an actual scale.

Specifically, as shown in FIG. 5A, the scale conversion function calculation means 44 has a distance r from the virtual camera C to the calibration pattern 90 and a luminance value q (pixel value) of each pixel of the depth image P _D. ) And. At this time, in the calibration pattern 90 included in the depth image _PD , the reflectance is lowered in the black pattern portion, so that accurate mapping is difficult. Therefore, it is preferable that the scale conversion function calculation means 44 associates only with the white portion of the calibration pattern 90 included in the depth image _PD . Here, the scale conversion function calculation means 44 obtains a graph as shown in FIG. 5 (b) by associating all the depth images _PD in which the calibration pattern 90 is captured. Then, the scale conversion function calculation means 44 can calculate the scale conversion function h (q) by approximating this graph with a function (for example, a quintic function). The scale conversion function calculation means 44 may not approximate this graph with the scale conversion function and may use it as a lookup table.

After that, the camera calibration means 4 outputs the calculated scale conversion function to the scale conversion means 54, and outputs the camera parameters of the virtual camera C to the cost volume generation means 51 and the weight application means 59.

<Refinement means>
The refinement means 5 inputs an RGB image _{PC and a depth image P D obtained} by photographing the subject 9 with the RGB-D camera 2. Then, the refinement means 5 improves the accuracy of the depth map by constraining the cost volume generated from the _RGB image PC with two weights based on the depth map generated from the depth image _PD . The refinement means 5 performs a refinement process each time a photograph is taken.

As shown in FIG. 2, the refinement means 5 includes an image segmentation means 50, a cost volume generation means 51, an initial depth map generation means 52, a smoothing means 53, and a scale conversion means (depth conversion means) 54. , A layering processing means 55, a scale correction means (intermediate depth map correction means) 56, a cost weight calculation means 57, a visibility weight calculation means 58, a weight application means 59, and a final depth map generation means 60. ..

The image segmentation means 50 divides the RGB image PC and the depth image P _D input from the RGB- _D camera 2 for each viewpoint. As shown in FIGS. 6A and 6B, the image segmentation means 50 divides the RGB image _PC and the depth image PD in which the subject 9 is captured, similarly to the image _segmentation means 41.

In FIG. 6, since the subject 9 is interposed in the lens system 21, the subject 9 is an upright image in the RGB image _PC and the depth image _PD . In this case, the RGB image _{PC and the depth image P D may} be inverted so that the subject 9 becomes an upright image.

The cost volume generating means 51 calculates the cost for each pixel position of the depth layer and the RGB image PC described later, and generates a cost volume in which the cost is three _- dimensionally arranged in the depth layer and the pixel position. In the present embodiment, the cost volume generating means 51 uses the plain sweep method, which is one of the methods for estimating the cost volume (Reference 3).

Reference 3: David Gallup, et al., "Real-time plane-sweeping stereo with multiple sweeping directions", IEEE Conference on Computer Vision and Pattern Recognition, pp. 1-8 (2007)

First, as shown in FIG. 7, the cost volume generating means 51 sets a plurality of depth layers _ND at predetermined intervals in the depth direction in the space where the subject 9 is arranged. In the example of FIG. 7, five depth layers ND are set ( _D = 1, ..., 5). In FIG. 7, the x-axis indicates the horizontal direction, the y-axis indicates the vertical direction, and the z-axis indicates the depth direction. Next, the cost volume generation means 51 uses any one of all the virtual cameras C as a reference camera, and sets a camera pair with this reference camera and another virtual camera C. Then, the cost volume generation means 51 projects the RGB image CC of each virtual camera _C constituting the camera pair onto the depth layer _ND by projective transformation. Further, the cost volume generating means 51 calculates the cost by obtaining the difference (for example, SAD: Sum of Absolute Difference) of the pixel value of each pixel of the two RGB image _{PCs projected} on the depth layer ND. This cost represents the similarity between the two _RGB images PC projected on the depth layer _ND , and the smaller the value, the higher the possibility that the depth of the subject 9 exists in the depth layer _ND . Represents.

The cost volume generating means 51 can generate a cost volume by performing the above- _mentioned processing in all the depth layers ND. As shown in FIG. 8, assuming that the size of the RGB image PC is U × V pixels, the cost volume 91 is a three _- dimensional array of costs of U × V × N _D. Further, in the cost volume 91, the costs arranged in the depth direction at the same pixel position are referred to as the cost column 92. That is, the cost column 92 is a three-dimensional _array of costs of 1 × 1 × ND. Then, the cost volume generating means 51 applies the guided filter to the cost volume 91 using the _RGB image PC of the reference camera as a guide (Reference 4). As a result, the cost volume 91 can be smoothed while holding the edge, so that the noise of the cost volume 91 can be reduced.

Reference 4: Kaiming He, Sun Jian, and Tang Xiaoou, "Guided image filtering", European conference on computer vision. Springer, pp. 1-10, (2010)

Assuming that the set of virtual cameras C around the reference camera is S, the camera pair can be set by the number of elements | S | of the set. At this time, the cost volume 91 can be the same as that of the camera pair. For example, when there are four virtual cameras C, there are three camera pairs and three cost volumes 91 for one reference camera. For example, when the virtual camera C ₁ is a reference camera, the camera pairs are (C ₁ , C ₂ ), (C ₁ , C ₃ ), and (C ₁ , C ₄ ).

The initial depth map generation means 52 generates an initial depth map showing the depth of the depth layer _ND that minimizes the cost in the cost column 92 at the same pixel position with the cost volume 91 input from the cost volume generation means 51. Is.

Here, since the initial depth map generation means 52 has a plurality of cost volumes 91 for one reference camera, the sum of the cost volumes 91 is obtained as the final cost volume 91 of the reference camera. Then, the initial depth map generation means 52 obtains the depth layer N _D having the minimum cost in each cost column 92 as the correct depth, and generates the initial depth map ^DC of the reference camera.

After that, the initial depth map generation means 52 outputs the initial depth map ^DC to the scale correction means 56, and outputs the final cost volume 91 to the weight application means 59.

The smoothing means 53 _smoothes the depth image PD input from the image segmentation means 50. Here, the smoothing means 53 smoothes the depth image P _D by filtering because noise such as shot noise of the depth camera is included in the depth image P _D. For example, as a filtering process, a guided filter can be mentioned. This guided filter is a kind of smoothing filter and smoothes a target image by using a guide image. Here, an RGB image _PC is used as the guide image.

While noise can be removed by filtering, the accuracy of the depth image _PD may decrease due to excessive smoothing. Therefore, the smoothing means 53 may execute the filtering process as needed.

The scale conversion means 54 converts the depth image P _D into an intermediate depth map by a scale conversion function that converts the pixel value of each pixel of the depth image P _D into the depth of the actual scale. In the present embodiment, the scale conversion means 54 converts the depth image _PD input from the smoothing means 53 into an actual scale depth map by the scale conversion function input from the scale conversion function calculation means 44. The scale conversion means 54 may use the scale conversion function when the manufacturer of the RGB-D camera 2 provides the scale conversion function.

The layering processing means 55 performs layering processing in which the depth of the intermediate depth map input from the scale conversion means 54 is replaced with the depth of the nearest depth layer _ND . Specifically, since the layering processing means 55 has known camera parameters, it is possible to form a real-scale intermediate depth map into a three-dimensional point cloud. Here, the layering processing means 55 considers the distance from the optical center when the intermediate depth map represents the distance from the optical center instead of the distance in the optical axis direction (generally the z direction) in the camera coordinate system. 3D point cloud. Then, the _layering processing means 55 expresses the intermediate depth map by the depth layer ND by making the depth of each point belong to the nearest depth layer _ND . Hereinafter, the intermediate depth map that has been layered is referred to as ^DD .

The scale correction means 56 obtains the average of the depth differences in each depth layer N _D for the pixels whose depth difference between the initial depth map DC and the intermediate depth map ^{D D is} equal to or less than the threshold value, and obtains the average of the depth differences as the correction value of the intermediate depth map D ^D. The depth is corrected by the correction value. That is, when the accuracy of the scale conversion function is low, the scale correction means 56 ^corrects the intermediate depth map _DD generated from the depth image ^PD to match the initial depth map DC generated from the _RGB image PC.

Specifically, the scale correction means 56 calculates the depth difference ^{D Sub} = DC ^−DD of each pixel between the initial depth map DC and the intermediate depth map ^{D D.} Next, the scale correction means 56 targets only the pixels satisfying | D ^Sub | ≤thold, and the depth difference ^D at each depth d (d = 1, 2, ..., N _D ) of the initial depth map DC. The average of ^Sub is calculated and used as a correction value. The threshold threshold is set manually. Then, the scale correction means 56 applies the correction depth value D ^Cor to the intermediate depth map D ^D _Old before correction, such as D ^D _New = D ^D _Old + ^{DC Cor} , and obtains the corrected intermediate depth map D ^D _New . Find (hereafter, intermediate depth map ^DD ).
The scale correction means 56 does not have to perform processing when the accuracy of the scale conversion function is high.

The cost weight calculation means 57 calculates the cost weight ^WC in which the weights of the intermediate depth maps ^DD input from the scale correction means 56 are represented by a normal distribution function. As described above, the cost volume 91 is generated only from the _RGB image PC and does not consider the depth map. Therefore, by applying the cost weight ^WC calculated from the intermediate depth map ^DD to the cost volume 91, the cost volume 91 is obtained in consideration of both the RGB image _PC and the depth map.

The cost weight ^WC is expressed as a normal distribution with the weight of the depth as the minimum value, assuming that the intermediate depth map ^DD is likely to have the correct depth value. As shown in FIG. 9, the maximum value of the normal distribution is set to 1, and the normal distribution function g (d) of the depth layer d is defined by the following equation (1).

Here, μ represents the mean, σ ² represents the variance, and σ represents the standard deviation. Using this normal distribution function g (d), the cost weight function f _C (d) is defined by the following equation (2). Note that a _c is a parameter that determines the cost weight ^WC . Further, as shown in FIG. 10A, in the normal distribution function g (d) of the equation (2), the depth values ^DD (u, v) of the pixels (u, v) whose average μ is the intermediate depth map ^DD . ), And the variance σ ² is preset according to the design policy of the cost weight function f _C (d) (for example, σ ² = N _D / 3).

The cost weight ^WC is a three-dimensional array having the same size as the cost volume 91. Then, as shown in the following equation (3), the value of the cost weight function f _C (d) is input to each element of the cost weight ^WC . From the above, the cost weight calculation means 57 calculates the cost weight ^WC using the equation (3).

The visibility weight calculation means 58 calculates the visibility weight ^WV that reduces the cost when occlusion occurs from the intermediate depth map ^DD input from the cost weight calculation means 57.

Here, occlusion is not taken into consideration when the cost volume 91 is generated, the cost of the portion where occlusion occurs becomes noise, and an error occurs even in the layering process described above. This error also occurs when the sum of the cost volumes 91 is calculated for a plurality of camera pairs. Note that occlusion means that a region that can be seen by one virtual camera C and cannot be seen by the other virtual camera C is generated.

On the other hand, since the intermediate depth map ^DD is generated from one depth camera, it is not affected by occlusion. Therefore, the visibility weight calculation means 58 calculates the visibility weight ^WV from the intermediate depth map ^DD in order to mitigate the influence of occlusion (reduce the cost of the portion where occlusion occurs).

As shown in FIG. 10 (b), the visibility weight function f _V (d) is defined by the following equation (4). Note that a _V is a parameter that determines the visibility weight W ^V. In the normal distribution function g (d) of the equation (4), the mean μ represents the value ^DD (u, v) + shift obtained by adding the constant shift to the mean of the depth values ^DD (u, v) (however, shift). ≧ 0). Further, the variance σ ² is set in advance according to the design policy of the visibility weight function f _V (d) (for example, σ ² = N _D / 10). By increasing the value of the constant shift, it is permissible even if there is an error in the intermediate depth map ^DD , but the effect of the visibility weight ^WV becomes smaller.

The visibility weight ^WV is a three-dimensional array having the same size as the cost volume 91. Then, as shown in the following equation (5), the value of the visibility weight function f ^V (d) is input to each element of the visibility weight _WV . From the above, the visibility weight calculation means 58 calculates the visibility weight ^WV of the equation (5).

The weight applying means 59 applies the cost weight ^WC and the visibility weight ^WV to the cost volume 91 input from the initial depth map generating means 52. Here, the final cost volume ^ES is the cost volume 91 integrated in all camera pairs as the reference camera C. That is, as shown in the following equation (6), the weight applying means 59 has a final cost volume due to the cost weight ^WC (x, y, _z ), the cost volume EJ, and the visibility weight ^WV of the reference camera. Calculate ^ES .

The cost volume E _j is a cost volume 91 of the reference camera C and the virtual camera C _j (j ∈ S) included in the surrounding camera set S. Further, warp represents a projective transformation with each depth layer ND as a plane from the virtual camera _{C j} to the reference camera C.

The final depth map generation means 60 generates a final depth map showing the depth of the depth layer _ND that minimizes the cost in the cost column 92 at the same pixel position with the cost volume 91 input from the weight application means 59. be. That is, the final depth map generation means 60 obtains the depth layer _ND having the lowest cost in each cost column 92 as the correct depth, and generates the final depth map.
Since the final depth map generation means 60 generates the final depth map by the same method as the initial depth map generation means 52, further description thereof will be omitted.

After that, the refinement means 5 outputs the RGB image PC of each viewpoint, the final depth map, and the camera parameters of the virtual camera _C input from the camera calibration means 4 as a set.

[Camera calibration process]
The camera calibration process will be described with reference to FIG.
As shown in FIG. 11, in step S1, the angle-of-view correction means 40 uses the depth image so that the angle of view of the depth image PD input from the RGB _{- D} camera 2 matches the angle of view of the RGB image PC. _Projective conversion of PD. Since the process of step S1 is not essential, it is shown by a broken line.

In step S2, the image segmentation means 41 divides the RGB image _PC and the depth image _PD for each viewpoint.
In step S3, the initial camera parameter calculation means 42 calculates the initial camera parameters of the virtual camera _C corresponding to each viewpoint by performing camera calibration processing on the RGB image PC of each viewpoint.
In step S4, the camera parameter optimizing means 43 optimizes the camera parameters among the virtual cameras C by the camera calibration process with the initial camera parameters as the initial values.
In step S5, the scale conversion function calculation means 44 calculates the scale conversion function by making the distance from the position of the virtual camera _C indicated by the camera parameters to the calibration pattern correspond to the pixel value of each pixel of the depth image PD. ..

[Refinement processing]
The refinement process will be described with reference to FIG.
As shown in FIG. 12, in step S10, the image segmentation means 50 divides the RGB image PC and the depth image _PD into each virtual camera _C.
In step S11, the cost volume generating means 51 calculates the cost for each pixel of the depth layer and the RGB image PC, and generates the cost volume 91 which is a three _- dimensional array of costs.

In step S12, the initial depth map generation means 52 generates an initial depth map showing the depth of the depth layer that minimizes the cost in the cost column 92 at the same pixel position with the cost volume 91.
The processes of steps S11 and S12 and the processes of steps S13 to S18 described later can be executed in parallel.

In step S13, the smoothing means 53 _smoothes the depth image PD.
In step S14, the scale conversion means 54 converts the depth image P _D into an intermediate depth map by a scale conversion function that converts the pixel value of each pixel of the depth image P _D into the depth of the actual scale.
In step S15, the layering processing means 55 performs a layering process of replacing the depth of the intermediate depth map with the depth of the nearest depth layer.

In step S16, the scale correction means 56 obtains the average of the depth differences in each depth layer N ^D as a correction value for the pixels whose depth difference between the initial depth map _DC and the intermediate depth map ^DD is equal to or less than the threshold value, and obtains the intermediate depth. Correct the depth of the map ^DD with the correction value. Since the process of step S16 is not essential, it is shown by a broken line.
In step S17, the cost weight calculation means 57 calculates the cost weight ^WC in which the weights of the intermediate depth maps ^DD are represented by a normal distribution function.
In step S18, the visibility weight calculation means 58 calculates the visibility weight ^WV that reduces the cost when occlusion occurs from the intermediate depth map ^DD .

In step S19, the weight applying means 59 applies the cost weight ^WC and the visibility weight ^WV to the cost volume 91.
In step S20, the final depth map generation means 60 generates a final depth map showing the depth of the depth layer _ND that minimizes the cost in the cost column 92 at the same pixel position with the cost volume 91.

[Action / Effect]
As described above, the three-dimensional shape acquisition system 1 can easily acquire the RGB image _CC of a plurality of viewpoints, the highly accurate depth map, and the camera parameters of the virtual camera C. That is, the three-dimensional shape acquisition system 1 can realize a simple system configuration, and can provide RGB image PCs for a plurality of viewpoints, a highly accurate depth map, and camera parameters of a virtual camera _C. These data are available in various applications. For example, when generating a three-dimensional image, a dense multi-viewpoint RGB image is required. Since the data provided by the three-dimensional shape acquisition system 1 includes the camera parameters of the virtual camera C and the highly accurate depth map, a three-dimensional image can be generated by a simple process.

Although the embodiments of the present invention have been described in detail above, the present invention is not limited to this, and includes design changes and the like within a range that does not deviate from the gist of the present invention.

In the above-described embodiment, the depth camera is described as a ToF camera, but the present invention is not limited to this. For example, the depth camera may be a stereo camera.

The present invention can also be realized by a program that operates the hardware resources such as the CPU, memory, and hard disk of the computer as the above-mentioned three-dimensional shape acquisition device. These programs may be distributed via a communication line, or may be written and distributed on a recording medium such as a CD-ROM or a flash memory.

1 3D shape acquisition system (depth map generation system)
2 RGB-D camera (shooting device)
20 Camera body 21 Lens system 22 Fresnel lens 23 Lens array 24 Element lens 25 Infrared LED array 3 Three-dimensional shape acquisition device (depth map generator)
4 Camera calibration means 40 Angle of view correction means 41 Image division means 42 Initial camera parameter calculation means 43 Camera parameter optimization means 44 Scale conversion function calculation means (depth conversion function calculation means)
5 Refinement means 50 Image segmentation means 51 Cost volume generation means 52 Initial depth map generation means 53 Smoothing means 54 Scale conversion means (depth conversion means)
55 Layering processing means 56 Scale correction means (intermediate depth map correction means)
57 Cost weight calculation means 58 Visibility weight calculation means 59 Weight application means 60 Final depth map generation means 9 Subject 90 Calibration pattern 91 Cost volume 92 Cost column ^C Virtual camera DC Initial depth map D ^D Intermediate depth map N _D Depth layer

Claims (8)

A shooting device composed of a shooting camera with the same optical axis, a depth camera, and an optical element array generates a depth map corresponding to the shot image of each viewpoint by using the shot image and the depth image of the subject taken from each viewpoint. Depth map generator
A cost representing the similarity between the captured images projected on the depth layer is calculated for each of the depth layers at predetermined intervals in the depth direction and the pixel positions of the captured images, and the cost is calculated at the depth layer and the pixel positions. A cost volume generation means that generates a three-dimensional array of cost volumes,
Depth conversion means for converting the depth image into an intermediate depth map by a depth conversion function that converts the pixel value of each pixel of the depth image into depth.
A cost weight calculation means for calculating a cost weight in which the weight of the intermediate depth map is expressed by a normal distribution function, and a cost weight calculation means.
A visibility weight calculation means that calculates a visibility weight that reduces the cost when an occlusion occurs from the intermediate depth map.
A weight application means for applying the cost weight and the visibility weight to the cost volume,
A final depth map generation means that generates a final depth map showing the depth of the depth layer that minimizes the cost in the cost column at the same pixel position in the cost volume after weight application.
A depth map generator characterized by being equipped with. Further provided with a smoothing means for smoothing the depth image,
The depth map generation device according to claim 1, wherein the depth conversion means converts a depth image smoothed by the smoothing means into the intermediate depth map by the depth conversion function. An initial depth map generation means that generates an initial depth map showing the depth of the depth layer that minimizes the cost in a cost column at the same pixel position in the cost volume generated by the cost volume generation means.
For pixels whose depth difference between the initial depth map and the intermediate depth map is equal to or less than the threshold value, the average of the depth differences between the depth layers is obtained as a correction value, and the depth of the intermediate depth map is corrected by the correction value. Map correction means and
The depth map generator according to claim 1 or 2, further comprising. Further provided is a layering processing means for performing a layering process for replacing the depth of the intermediate depth map with the depth of the nearest depth layer.
The depth map generation device according to claim 3, wherein the intermediate depth map correction means corrects the depth of the intermediate depth map to which the layering processing means has been layered with the correction value. An initial camera parameter calculation means for calculating the initial camera parameters of the virtual camera corresponding to each viewpoint by performing a camera calibration process on the captured image obtained by the photographing device from each viewpoint.
A camera parameter optimization means that optimizes camera parameters among virtual cameras by the camera calibration process with the initial camera parameters as initial values.
Depth conversion function calculation means for calculating the depth conversion function by associating the distance from the position of the virtual camera indicated by the optimized camera parameters to the calibration pattern with the pixel value of each pixel of the depth image.
The depth map generator according to any one of claims 1 to 4, further comprising. Further, the photographing apparatus further includes an angle-of-view correction means for projecting and converting the depth image so that the angle of view of the depth image obtained by photographing the calibration pattern at each viewpoint matches the angle of view of the photographed image.
The depth conversion function calculating means makes the depth from the position of the virtual camera to the calibration pattern correspond to the pixel value of each pixel of the depth image projected and converted by the angle of view correction means, thereby causing the depth conversion function. The depth map generator according to claim 5, wherein the depth map generator is calculated. A program for causing a computer to function as the depth map generator according to any one of claims 1 to 6. An imaging device composed of a photographing camera with the same optical axis, a depth camera, and an optical element array, and
The depth map generator according to any one of claims 1 to 6.
Depth map generation system characterized by being equipped with.

Images