JP2009186369A - Depth information acquisition method, depth information acquiring device, program, and recording medium - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a method, or the like, of acquiring the depth information on an image of a desired direction from a virtual view point. <P>SOLUTION: Images acquired by photographing the same object and the same scenery by using n cameras (n≥2) are input, and a plurality of virtual surfaces are installed in the virtual view point and in the field of view from the virtual view point. When a point of each virtual surface, corresponding to any image in the images, as seen from the virtual view point, is assumed to be an investigatory point, each corresponding point on each image imaged by each of n cameras, corresponding to the investigating point for each virtual surface is calculated, two cameras are selected from n cameras, the difference degree of each corresponding point on the images imaged with two cameras of all groups is calculated; the calculated difference degree is converted into similarity that decreases with increase with difference degree and becomes 0, when the difference degree is infinity, the similarities of all the groups are integrated and the integrated similarity is output; and the distance to the virtual surface having the largest integrated similarity from the virtual view point is output as depth information on any pixel in the image, as seen from the virtual view point. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to a depth information acquisition method, a depth information acquisition device, a program, and a recording medium. It relates to a method for obtaining information.

As a method of acquiring depth information of an image viewed from a certain viewpoint using a plurality of image data taken by a plurality of cameras, a multi-baseline stereo method (see Non-Patent Document 1 below) or a multilayer reliability map is used. There is known a method (see Non-Patent Document 2 below).
A conventional method for acquiring depth information of an image viewed from a certain viewpoint using a plurality of image data taken by a plurality of cameras will be described below.
FIG. 1 is a diagram for explaining a conventional method for acquiring depth information of an image viewed from a certain viewpoint using a plurality of image data taken by a plurality of cameras.
In Figure 1, the reference camera _C R, 2002 is the base camera _{C R} lenses 2001, 2003 lens center of the base camera _{C R} (reference viewpoint), the imaging element of the base camera _{C R} 2004, 2005 base camera _{C R} origin of imaging device, the central point of any pixel of the image Motojo of the base camera _{C R} 2006, 2010 line of sight of the base camera _{C R,} 2011 survey point, the optical axis (reference viewpoint of the reference camera _{C R} 2012 the pass, the base camera _C a line perpendicular to the imaging element _R), 1101 other cameras (camera _C 1), 1102 is the lens camera _{C 1,} 1103 the lens center of the camera _{C 1,} 1104 is the camera _{C 1} imaging device, 1105 of the image pickup device of the camera _{C 1} origin, 1106 imaging point survey point 2011 in the image pickup Motojo cameras _{C 1,} 1110 the view of the camera _{C 1} It is.
One of the plurality of cameras is a reference camera C _R (2001), and the other cameras are C _i . However, i is a number which distinguishes a camera. The position and direction of the camera can be freely determined.
D in the figure, a reference from the camera _C R (2001), represents the distance to the plane in which there is survey point 2011, the intersection of the reference camera _C R perpendicular line drawn from the survey point 2011 to the optical axis 2012 of the (2001) When, in a where the distance between the base camera _C R lens center 2003 (reference viewpoint) of (2001), want to get the base camera _C R image incident on the image sensor (2004) of the (2001) Depth information for each pixel.

As prior art documents related to the invention of the present application, there are the following.
M. Okutomi, T. Kanade, “A Multiple-Baseline Stereo”, IEEE Trans. Pattren Analysis and Machine Intelligence, Vol. 15, No. 4, April 1993. Yutaka Kunida et al., “Real-time 3D video generation system using multi-layer reliability map”, Journal of the Institute of Image Information and Television Engineers, Vol. 60, No. 7, 2006, pp1102-1110.

However, the method described in Non-Patent Document 1 described above, since the reference camera C _R is required to be real camera, it is necessary to adjust so that the viewpoint and the optical axis to be measured actual camera. For this reason, there is a problem in that it is impossible to obtain depth information of an image captured by a virtual camera having a virtual viewpoint (virtual viewpoint) and an optical axis (virtual optical axis) once the camera is arranged.
In addition, since the integrated difference shown in the above equation (2) is the sum of the differences of the cameras, even one camera is generated due to the occurrence of occlusion (there is a portion that cannot be captured from the actual camera) or the like. When the dissimilarity is small, the integrated dissimilarity is affected and becomes small, and there is a problem that an error is likely to occur when the depth is determined.
Non-Patent Document 2 described above, although solving the problem that it is necessary base camera C _R of the Non-Patent Document 1 above is real camera, not be solved a problem that weak occlusion.
The present invention has been made to solve the problems of the prior art, and an object of the present invention is to provide a depth information acquisition method and a depth information acquisition device that are less affected by occlusion and viewed from a virtual viewpoint. Another object of the present invention is to provide a technique capable of acquiring depth information of an image in a desired direction.
Another object of the present invention is to provide a program for causing a computer to execute the above-described depth information acquisition method, and a recording medium on which the program is recorded.
The above and other objects and novel features of the present invention will become apparent from the description of this specification and the accompanying drawings.

Of the inventions disclosed in this application, the outline of typical ones will be briefly described as follows.
The present invention is a depth information acquisition method for acquiring depth information of an image viewed from a virtual viewpoint that is an arbitrary viewpoint using image information captured by n (n ≧ 2) cameras having different viewpoints, Step 1 for inputting images obtained by photographing the same object or landscape using n cameras, Step 2 for setting the virtual viewpoint, and arranging a plurality of virtual planes in a field of view viewed from the virtual viewpoint. Step 3 and selecting a point of each virtual plane corresponding to an arbitrary pixel in the image viewed from the virtual viewpoint as a survey point, select one of the virtual planes, and the survey in the selected virtual plane Step 4 for calculating each corresponding point on each image captured by each of the n cameras corresponding to a point, and selecting two cameras from the n cameras, and selecting the selected two That on the image taken by the camera Step 5 for calculating the dissimilarity between the corresponding points, and Step 6 for converting the dissimilarity calculated in Step 5 into a similarity that becomes a smaller value as the dissimilarity increases and becomes 0 when the dissimilarity is infinite. Executing step 5 and step 6 on all sets of n (n−1) / 2 sets for selecting two cameras from the n cameras; Step 8 for integrating the similarities of all sets of n (n-1) / 2 sets, and outputting the integrated similarity, and Step 9 for executing Steps 4 to 8 for all the virtual surfaces And an arbitrary pixel in the image viewed from the virtual viewpoint, the distance from the virtual viewpoint to the virtual surface having the largest integrated similarity among the integrated similarities in each virtual surface obtained in the step 9 Step 10 as depth information of And a step 11 to perform steps 4 10 said to all survey points that the user wants.

In the present invention, the step 8 adds a parameter to the similarity calculated in all the n (n-1) / 2 sets, and adds the parameter to the n (n-1) / 2. This is a step of multiplying the similarities calculated for all the pairs and outputting them as integrated similarities.
In the present invention, the step 8 selects the highest similarity among the similarities calculated in all the n (n−1) / 2 pairs, and the selected similarity is used as the integrated similarity. As the output step.
In addition, the present invention is a depth information acquisition device that implements the depth information acquisition method described above.
Moreover, this invention is a program for making a computer perform the above-mentioned depth information acquisition method.
The present invention is also a computer-readable recording medium that records the above-described program.

The effects obtained by the representative ones of the inventions disclosed in the present application will be briefly described as follows.
According to the depth information acquisition method and the depth information acquisition apparatus of the present invention, it is possible to acquire depth information of an image in a desired direction viewed from a virtual viewpoint by a method that is less affected by occlusion.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.
In all the drawings for explaining the embodiments, parts having the same functions are given the same reference numerals, and repeated explanation thereof is omitted.
FIG. 2 is a diagram for explaining an information acquisition method according to the embodiment of this invention.
2, 1001 virtual camera _C v, 1002 is a lens of a virtual camera _{C v,} 1003 the lens center of the virtual camera _{C v} (virtual viewpoint), 1004 image sensor of the virtual camera _{C v,} 1005 is the virtual camera _{C v} origin of imaging device, 1006 a center point of an arbitrary pixel of the imaging Motojo of the virtual camera _{C v,} 1010 the visual axis of the virtual camera _{C v,} 1011 survey point, 1012 in the spatial extent wants to see the user virtual surface group, 1101 other cameras (camera _C 1), 1102 is the lens camera _{C 1,} 1103 the lens center of the camera _{C 1,} 1104 the image pickup device of the camera _{C 1,} 1105 is the image pickup device of the camera _{C 1} origin, 1106 imaging point survey point 1011 in the image pickup Motojo the camera _{C 1,} 1110 is a line of sight camera _{C 1.}
In this embodiment, instead of the reference camera C _R is a real camera shown in FIG. 1, using the virtual camera C _v. In this embodiment, two or more real cameras are prepared, and a combination for selecting two from the group of real cameras is made. If there are n actual cameras, there are n (n-1) / 2 combinations, so that combination is made.

Hereinafter, the depth information acquisition method of the present embodiment will be described with reference to the flowchart of FIG.
(1) Step 101
The n (n ≧ 2) real cameras Ci (i = 1,..., n) photograph the same object or landscape, and input the photographed image to a depth information acquisition apparatus described later. At this time, in each camera Ci (i = 1,..., N), parameters relating to the inside of the camera (camera internal parameters) known in the field of computer vision and parameters relating to the position and orientation of the camera (external to the camera) Parameter) is assumed to be known. The camera internal parameter is expressed by the following equation (3) when expressed in a matrix.

(2) Step 102
In this step, a place where the user wants to take his / her eyes is determined, and a virtual viewpoint 1003 is set there.
(3) Step 103
In this step, a plurality of virtual surfaces 1012 are installed in layers in the field of view viewed from the virtual viewpoint 1003.
(4) Step 104
In this step, when a point on each virtual plane 1012 corresponding to an arbitrary pixel in the image viewed from the virtual viewpoint 1003 is set as the survey point 1011, one virtual plane is selected and the survey point 1011 in the selected virtual plane is selected. The coordinates of the corresponding points on the plurality of images captured by each of the n cameras corresponding to are calculated.
In this step, a virtual plane that is within the field of view viewed from the virtual viewpoint 1003 and that represents the range of the image that the user wants to generate is installed. This can be regarded as the image sensor 1004 of the virtual camera 1001 having the lens 1002 with the virtual viewpoint 1003 as the lens center.
Next, the definition of the image the user wants to see is determined, and the pixels on the surface of the image sensor 1004 of the virtual camera 1001 are arranged so as to satisfy the definition.
Thereby, the parameters of the virtual camera 1001 installed at the virtual viewpoint 1003 that is the viewpoint that the user wants to see are determined.
The internal / external parameters of the virtual camera 1001 are expressed by the following equation (5).

Here, since the content of the equation (5) is the same as the above-described equations (3) and (4), a detailed description thereof will be omitted.
Next, one arbitrary pixel 1006 of the image sensor 1004 of the virtual camera 1001 is selected. Then, an intersection point between a virtual line of sight 1010 that is a straight line passing through the virtual start point 1003 and the pixel 1006 and a virtual plane of each virtual plane group 1012 is set as an investigation point 1011.
Next, one virtual plane is selected from the virtual plane group 1012, and the coordinates of the corresponding points on the plurality of images captured by each of the n cameras corresponding to the survey point 1011 in the selected virtual plane are selected. Calculate

The viewpoint position input means 2 is an input device such as a mouse or a keyboard.
The subject image acquisition unit 12 is a unit that acquires an image of the same object or landscape, but it can also be acquired by reading out an image captured in advance from the recording device. Note that images taken from multiple viewpoint positions are preferably taken at the same time by synchronizing all cameras, but the changes in the position and orientation of the same object or landscape are sufficiently slow. However, this is not the case when it can be regarded as a stationary object.
The depth information acquisition unit 13 includes a corresponding point calculation unit 131, a dissimilarity calculation unit 132, a similarity conversion unit 133, an integrated similarity calculation unit 134, and a depth information output unit 135.
The corresponding point calculation unit 131 executes the process of step 104 in FIG. 3 for all n (≧ 2) cameras.
The dissimilarity calculation means 132 executes the process of step 105 in FIG. 3 for all sets of n (n−1) / 2 sets that select two cameras from the n cameras.
The similarity conversion unit 133 executes the process of step 106 in FIG. 3 for all sets of n (n−1) / 2 sets that select two cameras from the n cameras.
The integrated similarity calculation means 134 executes the processing of step 108 in FIG. 3 for all sets of n (n−1) / 2 sets that select two cameras from the n cameras.
The depth information output unit 135 executes the processing of step 110 and step 111 in FIG. 3 and outputs depth information.
Here, the depth information acquisition means 13 acquires depth information for all survey points desired by the user. That is, the corresponding point calculation unit 131, the dissimilarity calculation unit 132, the similarity conversion unit 133, the integrated similarity calculation unit 134, and the depth information output unit 135 are provided for all survey points desired by the user. The above-described process (the process of step 111 in FIG. 3) is executed.

As described above, according to the present embodiment, since the reference camera is not a real camera, it is not necessary for the user to place the real camera at the position of the virtual camera for which depth information is desired to be acquired, as compared with the conventional method. In addition, since the virtual camera is not a real camera, it is possible to acquire freely set virtual viewpoint and optical axis depth information.
In addition, the evaluation function for selecting the depth is expressed by (similarity of all the n (n−1) / 2 sets of selecting two cameras from n cameras (similarity of 0 or more). Since the product of +1) or the maximum value is used, the similarity of a pair whose similarity is low due to the influence of occlusion is approximately 1. Thereby, since the influence on the integrated similarity becomes small, it becomes difficult to be affected by the occlusion, and as a result, the accuracy of the depth information can be improved.
Further, in the calculation of the degree of difference, the calculation is performed for each group of n (n−1) / 2 sets for selecting two cameras from the n cameras. Since the method of selection is the smallest unit, it becomes easy to obtain a pair of cameras that can avoid occlusion, so that resistance to occlusion can be obtained, and as a result, accuracy of depth information can be increased. Become.
As mentioned above, the invention made by the present inventor has been specifically described based on the above embodiments. However, the present invention is not limited to the above embodiments, and various modifications can be made without departing from the scope of the invention. Of course.

It is a figure for demonstrating the conventional depth information acquisition method. It is a figure for demonstrating the depth information acquisition method of the Example of this invention. It is a flowchart which shows the process sequence of the depth information acquisition method of the Example of this invention. It is a figure of the pixel on the image sensor in the vicinity of the intersection of the virtual plane and the virtual line of sight and the image formation point (relationship diagram of a rectangle having the same size as the image sensor pixel around the point of interest and the pixel of the image sensor). It is a block diagram which shows schematic structure of the depth information acquisition apparatus of the Example of this invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Depth information acquisition apparatus 2 Viewpoint position input means 3 Subject imaging means 11 Virtual surface / virtual viewpoint determination means 12 Camera image acquisition means 13 Depth information acquisition means 131 Corresponding point calculation means 132 Dissimilarity calculation means 133 Similarity degree conversion means 134 Integrated similarity Degree calculation means 135 Depth information output means 1001 Virtual camera 1002 Virtual camera lens,
1003 Virtual camera lens center (virtual viewpoint)
DESCRIPTION OF SYMBOLS 1004 Image sensor of virtual camera 1005 Origin of image sensor of virtual camera 1006 Imaging point on image sensor of virtual camera of investigation point 1010 Line of sight of virtual camera 1011, 2011 Investigation point 1012 Virtual plane group 1101 Real camera 1102 Lens of real camera 1103 Lens center of lens of real camera 1104 Image sensor of real camera 1105 Origin point of image sensor of real camera 1106 Imaging point on image sensor of real camera of investigation point 1110 Line of sight of real camera 2001 Reference camera 2002 Lens of reference camera,
2003 Reference camera lens center (virtual viewpoint)
2004 Image pickup element of reference camera 2005 Origin of image pickup element of reference camera 2006 Imaging point on image pickup element of reference camera of survey point 2010 Line of sight of reference camera 5106 Image formation point of intersection of virtual plane and virtual line of sight 5111 Virtual plane and virtual A square 5112 having the same size as the pixel of the image sensor centering on the imaging point at the intersection of the line of sight 5112 A pixel group on the image sensor in the vicinity of the image formation point at the intersection of the virtual plane and the virtual line of sight

Claims (8)

A depth information acquisition method for acquiring depth information of an image viewed from a virtual viewpoint that is an arbitrary viewpoint using image information captured by n (n ≧ 2) cameras having different viewpoints,
Inputting an image obtained by photographing the same object or landscape using the n cameras;
Step 2 for setting the virtual viewpoint;
Installing a plurality of virtual planes in layers in the field of view viewed from the virtual viewpoint; and
When the point of each virtual surface corresponding to an arbitrary pixel in the image viewed from the virtual viewpoint is used as a survey point, the virtual surface is selected and the one corresponding to the survey point in the selected virtual surface is selected. calculating each corresponding point on each image captured by each of the n cameras;
Step 5 of selecting two cameras from the n cameras and calculating the difference between the corresponding points on the images captured by the two selected cameras.
Step 6 for converting the degree of difference calculated in Step 5 into a degree of similarity that becomes smaller as the degree of difference is larger and becomes 0 when the degree of difference is infinite;
Executing step 5 and step 6 for all sets of n (n-1) / 2 sets for selecting two cameras from the n cameras; and
Step 8 of integrating the similarities of all the n (n-1) / 2 sets, and outputting the integrated similarity;
Step 9 for executing steps 4 to 8 for all the virtual planes;
The distance from the virtual viewpoint to the virtual plane having the largest integrated similarity among the integrated similarities in each virtual plane obtained in step 9 is the depth of an arbitrary pixel in the image viewed from the virtual viewpoint. A depth information acquisition method comprising: step 10 for information.   The step 8 adds a parameter to the similarity calculated in all the n (n−1) / 2 sets, and adds the parameter to all the n (n−1) / 2 sets. The depth information acquisition method according to claim 1, which is a step of multiplying the calculated similarity and outputting as an integrated similarity.   The step 8 is a step of selecting the largest similarity among the similarities calculated in all the n (n-1) / 2 pairs and outputting the selected similarity as an integrated similarity. The depth information acquisition method according to claim 1. A depth information acquisition device that acquires depth information of an image viewed from a virtual viewpoint, which is an arbitrary viewpoint, using image information captured by n (n ≧ 2) cameras having different viewpoints,
Means 1 for inputting images obtained by photographing the same object or landscape using the n cameras;
Means 2 for setting the virtual viewpoint;
Means 3 for arranging a plurality of virtual surfaces in a layered manner in the visual field viewed from the virtual viewpoint;
When the point of each virtual surface corresponding to an arbitrary pixel in the image viewed from the virtual viewpoint is used as a survey point, the virtual surface is selected and the one corresponding to the survey point in the selected virtual surface is selected. means 4 for calculating corresponding points on each image captured by each of the n cameras;
Two cameras are selected from the n cameras, and the degree of difference between the corresponding points on the image captured by the selected two cameras is calculated from the two cameras. Means 5 for calculating for all pairs of n (n-1) / 2 pairs
Means for converting the dissimilarity of all the n (n-1) / 2 pairs calculated by the means 5 into a similarity that becomes smaller as the dissimilarity is larger and becomes 0 when the dissimilarity is infinite. 6 and
Means 7 for integrating the similarities of all the n (n-1) / 2 sets and outputting the integrated similarity;
Means 8 for executing said means 4 to means 7 for all said virtual surfaces;
Depth information of an arbitrary pixel in the image viewed from the virtual viewpoint, the distance from the virtual viewpoint to the virtual surface having the largest integrated similarity among the integrated similarities in each virtual surface obtained by the means 8 And a depth information acquiring apparatus.   The means 7 adds a parameter to the similarity calculated in all the n (n−1) / 2 sets, and adds the parameter to all the n (n−1) / 2 sets. The depth information acquisition apparatus according to claim 4, wherein the calculated similarity is multiplied and output as an integrated similarity.   The means 7 selects the largest similarity among the similarities calculated in all the n (n-1) / 2 pairs, and outputs the selected similarity as an integrated similarity. The depth information acquisition apparatus according to claim 4.   The program for making a computer perform the depth information acquisition method of any one of Claims 1 thru | or 3.   A computer-readable recording medium on which the program according to claim 7 is recorded.

Images