JP2015033047A - Depth estimation device employing plural cameras - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a depth estimation device capable of accurately and fast estimating the depth of an object while using camera images captured by a plurality of cameras.SOLUTION: The depth estimation device comprises: a camera image acquisition part 1 for acquiring camera images from each camera; an object region determination part 6 for determining an object region in a three-dimensional voxel space on the basis of the camera images; an object region projection part 7 for producing a projection image of the object region for each camera image; a corresponding point detection part 61 for detecting a corresponding point by performing corresponding point matching defining the same line as a search clearance between projection images; and a depth estimation part 8 for estimating the depth of the object region on the basis of a parallax of the corresponding points.

Description

  The present invention relates to an apparatus for estimating the depth of an object based on images taken by a plurality of cameras, and in particular, a depth that can contribute to high-quality motion parallax reproduction of a real space image using a multi-view 3D display or the like. The present invention relates to an estimation device.

  In real-world 3D modeling and virtual viewpoint rendering between cameras, which is indispensable for realizing the space-sharing communication expected as the next-generation communication service, depth estimation that estimates the three-dimensional distance from each camera to the subject is performed. It is essential.

  As a representative method for 3D modeling of real space, Non-Patent Document 1 describes matching of corresponding points between cameras on the premise of a dense camera arrangement in which the camera interval is about the human eye width. An interpolated image generation method between cameras by 3D warping using depth estimation based on depth information is disclosed.

  Patent Document 1 discloses a method of restoring a three-dimensional shape model by polygonizing a depth map obtained by a distance image sensor based on a depth value for each pixel. If the depth can be accurately estimated, it is possible to contribute to high-quality motion parallax reproduction of a real space image using a multi-view 3D display or the like.

Japanese Patent Application No. 2012-214194

W. R. Mark et al. `` Post-Rendering 3D Warping ',' in Proc of Symposium on Interactive 3D Graphics, pp. 7-16, 1997.

  In the method proposed in Non-Patent Document 1, depth estimation for each pixel is performed based on matching of corresponding points between cameras, and parallax compensation is performed on a camera array arranged at an interval approximately equal to the human eye width. A method for generating an interpolated image between cameras has been proposed. However, the problem that a hole is generated in the occlusion area is unavoidable. In particular, when the camera interval is increased, the synthesized image quality is greatly deteriorated.

  The technique proposed in Patent Document 1 shows that rough depth estimation of an object surface can be performed robustly by using a distance image sensor. However, there are problems that the accuracy in the vicinity of the edge is insufficient and it is difficult to determine the boundary between the object and the floor surface.

  SUMMARY OF THE INVENTION An object of the present invention is to provide a depth estimation apparatus using a plurality of cameras that can solve the above-described problems of the prior art and can accurately and quickly estimate the depth of an object using camera images taken by a plurality of cameras. It is in.

  In order to achieve the above object, the present invention provides an apparatus for estimating the depth of an object based on the camera image of the object captured by a plurality of cameras in which the lines of the camera image are synchronized with each other. Camera image acquisition means for acquiring a camera image, object area determination means for determining an object area in a three-dimensional voxel space based on each camera image, and an object area for generating a projection image of the object area for each camera image Projection means, corresponding point detection means for detecting corresponding points by matching corresponding points on the same line between projected images, and depth estimation for estimating the depth of an object region based on the parallax of each corresponding point And a means.

  According to the present invention, since the search space when detecting corresponding points from each camera image is limited to the same line of the camera image, corresponding points can be detected at high speed and with a high matching system, and the depth of the object can be reduced. Fast and accurate estimation can be performed. As a result, high-accuracy video rendering at any viewpoint between cameras is possible.

It is a functional block diagram of the depth estimation apparatus to which the present invention is applied. It is the figure which showed an example of the camera image. It is the figure which showed an example of the empty stage image. It is the figure which showed typically the calculation method of the background likelihood of each voxel. It is the figure which showed an example of the projection image. It is the figure which showed the method of corresponding point matching. It is the figure which showed the method of calculating parallax based on a corresponding point. It is the figure which showed an example of the depth image acquired by this invention.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. FIG. 1 is a functional block diagram showing a configuration of a main part of a depth estimation apparatus according to an embodiment of the present invention. Here, illustration of components unnecessary for the description of the present invention is omitted.

  The camera image acquisition unit 1 acquires camera images Ica1, Ica2, Ica3,... From a plurality of cameras CA1, CA2, CA3,... That capture a stage space (voxel space) in which an object such as a person to be rendered can exist. The cameras CA1, CA2, CA3... Are all adjusted in advance and parallelized so that the positions of the horizontal lines are synchronized with each other in the camera images Ica1, Ica2, Ica3.

  FIG. 2 is a diagram showing an example of each camera image Ica1, Ica2, Ica3 in which a person is photographed as an object, and the same line of the object is displayed on the same corresponding line of each camera image Ica1, Ica2, Ica3. ing. In the following, in order to make the explanation easy to understand, a case where there are three cameras (CA1, CA2, CA3) will be described as an example.

  The calibration unit 2 includes each voxel V in the stage space (individual minimum block in the three-dimensional space), and each camera image Ica1, Ica2, Ica3 projected from the voxel V based on the central projection matrix. Are associated with each other based on the following equation (1).

  The sky stage image storage unit 3 stores in advance sky stage images Ik1, Ik2, and Ik3 obtained by photographing the sky stage in which no object to be rendered exists with the cameras CA1, CA2, and CA3. FIG. 3 shows sky stage images Ik1, Ik2, and Ik3 corresponding to the camera images Ica1, Ica2, and Ica3 shown in FIG.

  In the background likelihood calculating section 4, the background likelihood function generating section 41 is provided for each camera based on a plurality of frames of empty stage images Ik1, Ik2, Ik3 taken over a predetermined time from each camera CA1, CA2, CA3. An average vector u and a covariance matrix Σ of each pixel value (RGB vector) are calculated. The function calculation unit 42 applies the pixel values x of the camera images Ica1, Ica2, and Ica3 in which the object is captured, the average vector u, and the covariance matrix Σ to the following equation (2), thereby obtaining the camera images Ica1, Ica2, The likelihood that each pixel is a background image is calculated as background likelihood f (x; u, Σ) for each Ica3.

  The voxel background likelihood calculation unit 5 calculates the background likelihood of each voxel V in the stage space based on the background likelihood of the pixel corresponding to the voxel V in each camera image Ica1, Ica2, Ica3.

  FIG. 4 is a diagram schematically showing a method of calculating the background likelihood of each voxel V. Each voxel V in the voxel space 10 set in the stage space and the voxel V are based on the central projection matrix. The pixels of the camera images Ica1, Ica2, and Ica3 projected in this way are associated in advance by the calibration unit 2 and given the correspondence relationship of the above equation (1). In the present embodiment, the background likelihood ρv of each voxel V is calculated by applying the background likelihood of the corresponding coordinates of the camera images Ica1, Ica2, and Ica3 to the following equation (3).

  The object region determination unit 6 introduces the concept of the energy function E (v, α) in consideration of the adjacency relationship of each voxel, and each voxel V so that the value of the energy function E (v, α) is minimized. The background likelihood is binarized (whether or not it is a background). The following expression (4) is a calculation expression for binarizing the voxel background likelihood and is composed of the sum of the data term (first term on the right side) and the smoothing term (second term on the right side).

Here, the symbol α is a binary variable to which “0” is assigned if each voxel V is a background, and “1” is assigned if an object (person). Both the data term and the smoothing term are determined in accordance with the selection of α, and the optimal allocation is determined in terms of how α can be selected to minimize the energy in the entire voxel space.

  The data term is obtained by the following equations (5) and (6), and the smoothing term is obtained by the following equations (7) and (8) depending on the likelihood value difference between adjacent voxels. The symbol N represents a combination of adjacent voxels in the voxel space 10 (6 patterns on the left, right, front, back, top and bottom, 26 patterns including diagonal), and i and j represent the combination. The sign K is a positive constant (parameter).

  The object area projection unit 7 projects the voxels V of the object area onto the camera images Ica1, Ica2, and Ica3, and acquires a projection image. In the present embodiment, the projection image Jca is obtained by searching for the light ray R by the central projection matrix for each pixel of the camera images Ica1, Ica2, and Ica3 and extracting the pixel of the light ray R that intersects the object candidate. FIG. 5 is a diagram illustrating an example of projection images Jca1 and Jca2 corresponding to the camera images Ica1 and Ica2.

  The depth estimation unit 8 includes a corresponding point detection unit 61, a depth data calculation unit 62, and a clustering unit 63, and estimates the depth of the voxel corresponding to the corresponding point based on the parallax of the corresponding point between the projected images.

  The corresponding point detection unit 61 extracts a local feature amount for each pixel of the projection images Jca1 and Jca2, and detects a corresponding point by performing corresponding point matching based on the local feature amount for each identical horizontal line.

  FIG. 6 is a diagram for explaining a method of detecting corresponding points by the corresponding point detection unit 61. All pixels located on each horizontal line L between projected images (here, Jca1 and Jca2) are shown. Corresponding point matching is performed on the target based on the local feature amount. In the illustrated example, the pixel f1 of the projection image Jca1 and the pixel f2 of the projection shadow image Jca2 are detected as corresponding points on the horizontal line Lx.

  The depth data calculation unit 62 calculates the depth of the voxel corresponding to the corresponding point based on the distance between the corresponding points f1 and f2, that is, the parallax d, to obtain depth data. FIG. 7 is a diagram illustrating a method for calculating the parallax d, in which the corresponding point f1 is projected onto the projection image Jca2, and the parallax d is calculated as the distance between the corresponding points f1 and f2. In this embodiment, the depth is estimated by applying the relative positional relationship between the parallax d and each camera CA to the triangulation method.

  The clustering unit 63 clusters (quantizes) the large number of depth estimation values obtained as described above into discrete values, and outputs them. FIG. 8 is a diagram showing an example of the depth images Dca1 and Dca2 expressed by the depth estimation values obtained as described above, and corresponding depth estimation values are obtained according to the part of the object. I understand.

  According to the present embodiment, the search space for detecting corresponding points from each camera image is limited to the same line of the camera image, and corresponding points can be detected at high speed and with a high matching system. Fast and accurate estimation can be performed. As a result, high-accuracy video rendering at an arbitrary viewpoint between cameras is possible.

  DESCRIPTION OF SYMBOLS 1 ... Camera image acquisition part, 2 ... Calibration part, 3 ... Empty stage image storage part, 4 ... Background likelihood calculation part, 5 ... Voxel background likelihood calculation part, 6 ... Object area | region determination part, 7 ... Object area projection , 8 ... Depth estimation part, 41 ... Background likelihood function generation part, 42 ... Function calculation part, 61 ... Corresponding point detection part, 62 ... Depth data calculation part, 63 ... Clustering part

Claims (4)

In an apparatus for estimating the depth of an object based on camera images of an object photographed by a plurality of cameras in which each line of the camera image is synchronized with each other,
Camera image acquisition means for acquiring a camera image from each camera;
An object region determining means for determining an object region in a three-dimensional voxel space based on each camera image;
Object region projection means for generating a projection image of the object region for each camera image;
Corresponding point detection means for detecting corresponding points by performing corresponding point matching between the projected images on the same line as a search space;
A depth estimation device comprising: depth estimation means for estimating the depth of an object region based on the parallax of each corresponding point. Means for calculating the background likelihood for each pixel of each camera image;
Means for projecting each voxel in the voxel space onto each camera image based on a central projection matrix;
2. The apparatus according to claim 1, wherein the object area determination unit determines whether each voxel is an object area based on a background likelihood of the projection destination image.   2. The object area projecting means searches for a ray based on a central projection matrix for each pixel of each camera image, obtains a pixel where each ray intersects the object area, and uses it as a projection image. 2. A heel departure estimation apparatus according to 2.   The said depth estimation means clusters each estimated value of depth into a discrete value, and outputs it, The droop estimation apparatus in any one of Claim 1 thru | or 3 characterized by the above-mentioned.