JP4732392B2 - 3D shape data generation method - Google Patents

Description

  The present invention relates to a technique for three-dimensionally acquiring the surface shape of an object using images captured by a plurality of cameras.

A conventional three-dimensional shape data generation method will be described below. (For example, Non-Patent Document 1)
A typical overall processing flow of the prior art will be described with reference to FIG. (A) is a flow when the object surface is obtained as a determined result, and (b) is a flow when the object surface is obtained as a probabilistic result.

First, the same object is imaged using a plurality of cameras.
Next, a three-dimensional space including a target object whose three-dimensional shape data is to be measured is divided by voxels as in the space carving method.
Then, a value indicating the possibility that an object surface exists in each voxel is calculated for all voxels from the similarity between the captured images.
Finally, a point where the object surface is highly likely to be present is set as the surface of the object, or the probability that the object surface exists is calculated from the possibility that the object surface exists.

  FIG. 1 is a diagram illustrating a relationship between a division by a voxel in a three-dimensional space and imaging positions of a plurality of cameras. A conventional three-dimensional data generation method will be described with reference to FIG. In this method, in calculating the possibility that an object surface exists, first, a three-dimensional space to be measured is divided by voxels. Reference numeral 101 denotes one divided voxel. It is assumed that there are a plurality of cameras that image an object in the space, and here there are n cameras. Reference numeral 111 denotes one of the cameras C1, which is the first camera for convenience. 112 is the lens of the camera C1, 113 is the lens center of the lens 112, 114 is the image sensor of the camera C1, 115 is the origin of the image sensor 114, and 116 is a point on the image sensor 114 on which the voxel 101 is imaged. . The same applies to the other cameras C2 to Cn.

In the camera C 1 111, the voxel 101 passes through the lens center 113 of the lens 112 and forms an image at a point 116 on the image sensor 114. However, the coordinates of the point 116 are coordinates having the point 115 as the origin. It is assumed that voxel 101 is v, and whether or not the object surface is present in this voxel is determined by the magnitude of the degree of difference in the vicinity of the point on the image sensor of each camera C1 to Cn on which voxel 101 is imaged. Here, the degree of difference is, for example, a standard deviation σ _v as follows.

Here, m _{v, Ci} camera C _i image forming position on the image pickup device when the imaging center of the voxel v of the three-dimensional space by the camera C _i, a is m _v on the image pickup device of the _{camera, Ci} Is a set of vectors having mv _{, Ci} as the origin, p is one vector in a, T (k) is a texture at the coordinate k of the pixel on the image sensor, Usually, a color vector having RGB color density as an element, n _C is the number of cameras, bar T _v (p) is an average of textures of all cameras with respect to coordinates p of points in the vicinity of m _{v, Ci} , | x | ² represents the sum of squares of each element of x. Note that the symbol T having a line at the top in the image data equation is described as “bar T” on the text.

Since the light emitted from a minute area on the object surface is considered to have almost the same texture in the corresponding area on the image sensor of each camera, the standard deviation σ _v becomes small. Conversely, the smaller the standard deviation σ _{v, the} higher the possibility that the corresponding voxel v has an object surface. Therefore, the standard deviation σ _v is set as a dissimilarity, and a threshold τ is provided for the dissimilarity σ _v . If σ _v ≧ τ, there is no object surface in the voxel v, and if σ _v <τ. It is determined that the object surface exists in the voxel v.

K. N. Kutulakos and S. M. Seitz, “A theory of shape by space carving”, Int. J. Comput. Vision, Vol.38, No.3, pp.199-218, 2000

  However, in the above conventional method, when the same texture exists on the object surface in the three-dimensional space, the degree of difference between voxels without the object surface becomes small, and it may cause an error in determining whether the object surface is the object surface There was a disadvantage that there was.

This will be described with reference to FIG. The upper curve in the figure is a real curve indicating the object surface, and the dotted line around it indicates the voxel boundary. Reference numeral 211 denotes a camera C1, 212, a lens, 213, a lens center, and 214, an image sensor. 203 is a small area a on the object surface, 204 is a small area b on the object surface, 2031 is an area a _C1 formed on the image sensor when the small area a of 203 is imaged by the camera C1, and 2032 is 203. Regions a _C2 and 2042 imaged on the image pickup device when the small region a is imaged by the camera C2 are regions b _C2 and 201 imaged on the image pickup device when the small region b of 204 is imaged by the camera C2. The voxels v1 and 202 including the intersection of the line connecting the object surface subregion a 203 and the lens center of the camera C1 and the line connecting the object surface subregion b 204 and the lens center of the camera C2 are shown in FIG. Voxel v2 including region a.

Since the small areas a _C1 and a _C2 on the image sensor on which the small area a on the object surface is imaged are the areas where the same small area a is imaged, the texture is almost equal and the dissimilarity σ _v is small. The voxel v2 including the small area a is correctly determined as the object surface. However, if the difference in texture between the small area a on the object surface and the small area b on the object surface is small, the textures of the small areas a _C1 and b _C2 are almost equal, and the dissimilarity σ _v becomes small. The voxel v1 not including the surface is erroneously determined as the object surface.

  As described above, the conventional method has a drawback that there is a possibility of making an error in determining whether or not the object surface is present when a similar texture exists on the object surface.

  The present invention has been made in view of the above, and an object of the present invention is to provide a three-dimensional shape data generation method for correctly determining an object surface even when a similar texture exists on the object surface. It is to provide.

  Of the inventions disclosed in the present application, the outline of typical ones will be briefly described as follows.

The first invention calculates the possibility that the surface of an object exists at a certain point in space from the similarity of images in which the same object is captured by a plurality of cameras, and determines that the object is likely to exist. This is a 3D shape data generation method in a 3D shape data generation apparatus that generates 3D shape data of the surface of the object by using the surface of the object, and shows the possibility that the surface of the object exists at a certain point in space The calculation of the value by the three-dimensional shape data generation apparatus shows a first step (corresponding to step 1 of the first embodiment) for dividing the space into voxels, and a first indicating the possibility that an object surface exists in each voxel. A second step (corresponding to step 2 in the first embodiment) for determining the value of the value from the similarity of the pixel values of the points on the image sensor of each camera corresponding to each voxel, and one of the voxels Focus on voxel, taking a plurality of points around the at any point is the point of interest in the voxel, from the lens center of the two cameras, a third step of pulling the camera line of sight, such as through the points ( (Corresponding to step 3 of the first embodiment) and for each intersection of the camera line of sight, a second value indicating the possibility of the existence of an object at the intersection is given as the first value in the voxel near the intersection. 4th step (corresponding to step 4 of the first embodiment) calculated from the value and the second value at the intersection point, a constraint condition that an object exists only at one intersection point on each camera line of sight Is applied to calculate the third value corrected for the possibility that the surface of the object exists at each intersection point, and the object surface is applied to the voxel to which the third value at the point of interest is noted. A fifth step (corresponding to steps 5-1 to 5-3 in the first embodiment) as a fourth value indicating the possibility of being present, and a third step with respect to the other voxels created in the first step. And a sixth step (corresponding to step 6 in the first embodiment) for executing the processing from the above steps to the fifth step.

  According to a second invention, in the three-dimensional shape data generation method of the first invention described above, in the first step, the space of the region where the surface of the object is supposed to exist is divided by voxels, and in the third step The point of interest is a center point of a voxel to be noticed, and a plurality of points are taken around the point of interest on an epipolar plane formed by the point of interest and the lens center of the two cameras.

  According to a third invention, in the three-dimensional shape data generation method according to the first or second invention, the plurality of cameras is three or more, and the processing from the third step to the fifth step is performed. Select two cameras from the multiple cameras in the third step by selecting multiple combinations with different combinations, calculate the fourth value for the voxel of interest in each combination, and integrate the fourth value in each combination The method further includes a seventh step (corresponding to steps 5-4 and 5-5 in the first embodiment) in which the obtained value is a value indicating the possibility that the object surface exists in the target voxel.

The fourth invention calculates the possibility that the surface of an object exists at a certain point in space from the similarity of images in which the same object is captured by a plurality of cameras, and determines that the object is likely to exist. This is a 3D shape data generation method in a 3D shape data generation apparatus that generates 3D shape data of the surface of the object by using the surface of the object, and shows the possibility that the surface of the object exists at a certain point in space The calculation of the value by the three-dimensional shape data generation apparatus pays attention to the first step (corresponding to step 1 of the second embodiment) of dividing the space into voxels, and one voxel of the voxels, and the inside of the voxel taking a plurality of points around the target point is any point, from the lens center of the two cameras, the second step (embodiment 2 steps pulling the camera line of sight, such as through the points 2 1) and the first value indicating the possibility of the existence of an object at the intersection for each intersection of the camera line of sight, the corresponding point on the image sensor of the corresponding two cameras corresponding to the intersection The third step (corresponding to step 2-2 in the second embodiment) calculated from the similarity of the pixel values of the first and the first value at the intersection point is only one intersection point on each camera line of sight. Apply an optimization algorithm that uses the constraint that an object exists, calculate a second value that corrects the possibility of the surface of the object at each intersection, and pay attention to the second value at the point of interest In the fourth step (corresponding to steps 2-3-1 to 2-3-3 in the second embodiment) as the third value indicating the possibility that the object surface exists in the voxel to be Second against other voxels you created Characterized by comprising from a fifth step of executing the processing from step to the fourth step (corresponding to step 3 of the second embodiment).

  According to a fifth aspect of the present invention, in the three-dimensional shape data generation method according to the fourth aspect of the present invention, in the first step, the space of the region where the surface of the object is supposed to exist is divided by voxels, and in the second step The point of interest is a center point of a voxel to be noticed, and a plurality of points are taken around the point of interest on an epipolar plane formed by the point of interest and the lens center of the two cameras.

  A sixth invention is the three-dimensional shape data generation method of the fourth or fifth invention, wherein the plurality of cameras are three or more, and the processing from the second step to the fourth step is performed, The two cameras are selected from the plurality of cameras by a plurality of selection methods with different combinations, and a third value for the voxel of interest in each combination is calculated and the third value in each combination is integrated, A sixth step (corresponding to steps 2-3-4 to 2-3-5 of the second embodiment) is further included, which is a value indicating the possibility that the object surface exists in the target voxel.

  The present invention optimizes the reliability of each voxel so that the reliability at the intersection of the line of sight of the camera pair has less geometric contradiction so as to have the three-dimensional consistency obtained from the camera pair As a result, the accuracy of the reliability of each voxel is improved. As a result, there is an advantage that the object surface can be correctly determined even when a similar texture exists on the object surface.

  Further, the present invention has an advantage that the reliability of the reliability is improved because the reliability of each voxel has a three-dimensional consistency obtained from the camera pair.

  Moreover, since the present invention integrates the results of a plurality of camera pairs, there is an advantage that the reliability of the reliability is improved.

[Embodiment 1]
The processing flow of the first embodiment will be described with reference to FIG. FIG. 8A shows a flow when the object surface is obtained as a determined result, and FIG. 8B shows a flow when the object surface is obtained as a probabilistic result.

First, the same object is imaged using a plurality of cameras.
Using the captured image data, a three-dimensional shape data generation device (not shown) performs the following calculation. The three-dimensional shape data generation apparatus can be configured by a program stored in a storage device and a computer, and hardware can be used in place of a part or all of the program.
A three-dimensional space in which the surface of a target object for which three-dimensional shape data is desired to be measured is divided by voxels as in the space carving method.
Next, a value indicating the possibility that an object surface exists in each voxel is calculated for all voxels from the similarity between the captured images.
Next, paying attention to one voxel, a plurality of points are taken around the voxel, and a camera line of sight passing through these points is drawn from the lens centers of the two selected cameras.
Next, a value indicating the possibility that the object surface exists is calculated for each intersection of the camera line of sight.
Next, an optimization algorithm is applied to calculate a value obtained by correcting a value indicating the possibility that the object surface exists at the point of interest and a value indicating the possibility that the object surface exists at the peripheral points. The value at the point of interest is a value indicating the existence probability of the object surface.
For all combinations in which two of the above are selected from all cameras, the object surface existence probability at the point of interest is obtained, and the result obtained by integrating these is set as the object surface existence probability at the point of interest.
A value indicating the possibility that an object surface exists in all voxels is calculated.

  Finally, a point where the object surface is highly likely to be present is set as the surface of the object, or the probability that the object surface exists is calculated from the possibility that the object surface exists.

Hereinafter, Embodiment 1 of the present invention will be described in detail with reference to the drawings.
A three-dimensional data generation method according to the first embodiment will be described with reference to FIGS. 1 and 3 to 6. This method captures the same object using multiple cameras, calculates the possibility of the surface of the object at a certain point in space from the similarity of the captured images, and is likely to exist Is the surface of the object, or the probability of existence of the surface of the object is calculated from the possibility of existence of the object, thereby generating three-dimensional shape data of the surface of the object. Hereinafter, a method for calculating the possibility that the surface of the object exists will be described in detail.

(Step 1: Voxel division of space)
As shown in FIG. 1, a three-dimensional space including a target object whose three-dimensional shape data is to be measured is divided by voxels as in the space carving method. Reference numeral 101 denotes one divided voxel. It is assumed that there are a plurality of cameras that image an object in the space, and here there are n cameras. Reference numeral 111 denotes one of the cameras C1, which is the first camera for convenience. 112 is the lens of the camera C1, 113 is the lens center of the lens 112, 114 is the image sensor of the camera C1, 115 is the origin of the image sensor 114, 116 is a corresponding point on the image sensor 114 that is calculated when the voxel 101 is imaged, It is. The same applies to the other cameras C2 to Cn.

  In the camera C 1 111, the voxel 101 passes through the lens center 113 of the lens 112 and forms an image at a point 116 on the image sensor 114. However, the coordinates of the point 116 are expressed in a coordinate system with the point 115 as the origin.

(Step 2: Temporary calculation of possibility of object surface in voxel)
A value (= reliability) indicating the possibility of whether or not an object surface is present in the voxel is set to v, and a point on the image sensor of each camera C1 to Cn on which the voxel 101 forms an image, or This is determined by the similarity of pixel values of points in the vicinity of the point. For example, the value indicating similarity is similar to the standard deviation of the pixel values of the corresponding points on the image pickup devices of all the cameras and the neighboring points calculated that the same voxel 101 is imaged. Value.

  Hereinafter, a method for calculating similarity using the dissimilarity disR will be described. The dissimilarity disR is calculated as follows, for example.

Here, m _{v, Ci} camera C _i image forming position on the image pickup device when the imaging center of the voxel v of the three-dimensional space by the camera C _i, a is m _v on the image pickup device of the _{camera, Ci} Is a set of vectors having mv _{, Ci} as the origin, p is one vector in a, T (k) is a texture at the coordinate k of the pixel on the image sensor, Usually, a color vector having RGB color density as an element, n _C is the number of cameras, bar T _v (k) is an average of textures of all cameras with respect to a coordinate p of a point in the vicinity of m _{v, Ci} , | x | Represents the magnitude of x, for example, the square root of the sum of squares of each element, m is a value that changes how disR and the reliability change with respect to the texture variance (expressed as | T−T average |), It is.

An example of how to obtain the texture T ( _{mv, Ci} + p) will be described with reference to FIG. Here, 501 is the intersection of the camera lines of sight of cameras C1 and C2, 502 is the camera line of sight of camera C1, 503 is the intersection of the camera line of sight of camera C1 and the image sensor, and 504 is on the image sensor near the camera line of sight of camera C1. A pixel group, 505 is centered on the intersection of the camera line of sight of the camera C1 and the image sensor, and is a square having the same size as the pixel of the image sensor, 506 is an image sensor centered on the intersection of the camera line of sight of the camera C1 and the image sensor A square of the same size as the pixel of the image sensor in the vicinity of the square of the same size as the pixel of, 512 is a camera line of sight of the camera C2, 513 is an intersection of the camera line of sight of the camera C2 and the image sensor, and 514 is a camera of the camera C2 A pixel group 515 on the image sensor in the vicinity of the line of sight is a quadrangle having the same size as the pixel of the image sensor centering on the intersection of the camera line of sight of the camera C2 and the image sensor.

Since the coordinates _{mv, Ci} of the pixels on the image sensor do not always coincide with the coordinates of the pixels of the image sensor, even if a is a set of vectors with the pixel width as a unit, m _{v, Ci +} p is Does not match the coordinates of the pixels. This situation is shown in FIG. In FIG. 5A, two lines of sight passing through the camera line of sight intersection 501 intersect at the points indicated by 503 and 513 on the image sensors of the cameras C1 and C2, respectively. A state where the center of the upper pixel is deviated is shown.

FIG. 5B is a view of the image sensor of the camera C1 as viewed from directly above. A quadrangle having the same size as that of the image sensor including an intersection with the camera sight line 505, and a quadrangle 506 having the same size around the image sensor. Is drawn. _{mv and Ci} correspond to the coordinates of 503, and a corresponds to the center coordinates of the respective rectangles 505 and 506. However, as shown in FIG. 5B, this is a case where a is nine coordinates centered on the intersection.

FIG. 5C is a diagram showing a relationship between a quadrangle having the same size as the pixel of the image sensor centered on the intersection of the camera line of sight and the image sensor, and m _{v and Ci} are The coordinates of the pixels of the image sensor, _{mn1 (v), Ci} , _{mn2 (v), Ci} , _{mn3 (v), Ci} , _{mn4 (v), Ci} are centered on the intersection of the camera line of sight and the image sensor. The coordinates of the center of the pixel of the image sensor that overlaps a quadrangle of the same size as the image sensor. In the figure, s and t indicate the overlapping ratio, and 0 ≦ s and t ≦ 1. At this time, the texture T ( _{mv, Ci} ) can be calculated using the bilinear method as follows.

Although the bilinear method is shown here, other existing interpolation methods such as the nearest neighbor method and the bicubic method may be used. The texture T ( _{mv, Ci} + p) when p is changed in a can be calculated in the same manner.

  Since the light emitted from a minute area on the object surface is considered to have almost the same texture in the corresponding area on the image sensor of each camera, the dissimilarity disR becomes small. Conversely, the smaller the dissimilarity disR, the higher the possibility that the corresponding voxel v has an object surface.

  For example, the similarity R having the opposite property is obtained from the dissimilarity disR as follows.

Here, σ is a parameter for adjusting the influence of the dissimilarity disR on the similarity R, and the smaller the σ is, the smaller the degree of similarity changes.

  The higher the degree of similarity, the higher the possibility that the object surface is in the voxel. This possibility is called reliability and expressed by p. Let p be as follows: This p is different from the vector p in equation (8.1). These can be distinguished in the expression which is the image data, but cannot be distinguished on the text. Therefore, hereinafter, “reliability p” is described on the text.

  Reliability is calculated for all voxels.

  In order to increase the accuracy of the voxel reliability, correction by recalculation of reliability is performed by the following processing.

(Step 3: Determination of points around the target voxel)
One camera pair is selected from a plurality of cameras, and a camera line of sight as shown below is drawn to determine a neighboring point of the target voxel that is the intersection. How to draw the camera line of sight will be described with reference to FIG.

  Reference numeral 301 denotes a target voxel v whose reliability is to be recalculated, and 302 is a range in a three-dimensional space in which the presence or absence of the object surface is to be examined, and a range in a three-dimensional space to be divided by voxels. There are a plurality of cameras that image an object in the space. Here, only one pair C1 and C2 of the camera pair is shown here. Reference numeral 311 denotes one of the cameras C 1 and 312, a lens of the camera C 1, 313 denotes a lens center of the lens 312, 314 denotes an image sensor of the camera C 1, and 315 denotes a pixel of the image sensor 314. The same applies to the other cameras C2. Reference numeral 303 denotes a line L passing through the target voxel v and parallel to a line passing through the lens centers of the cameras C1 and C2.

(Step 3-1: Determination of epipolar plane)
The target voxel v for which the accuracy of the reliability p is desired to be increased is determined. At this time, the plane including the center of each camera lens of the voxel v and the cameras C1 and C2 is determined. This plane is called an epipolar plane.

(Step 3-2: Determination of a line L parallel to two camera centers)
A line L that is on the epipolar plane and is parallel to the line passing through the center of each of the cameras C1 and C2 and passing through the center of the target voxel v is drawn.

(Step 3-3: Determination of points around the target voxel on the line L)
Points are drawn on the line L at regular intervals starting from the center point of the voxel v on the line L around the center point of the target voxel v (attention point, triple circle in FIG. 3) (see FIG. 3). double circle). This point includes a point at the center of the voxel. The attention point is not limited to the center point of the attention voxel v, but may be an arbitrary point in the attention voxel v. The interval may be, for example, the interval between the sides of the voxel, but is not limited thereto. Furthermore, although it was set as equal intervals here, it does not necessarily need to be equal intervals. Further, the geometric range for taking points is not particularly limited, and may be a range that covers the entire voxel.

(Step 3-4: Camera line drawing passing on the line L and determination of points around the target voxel)
A line (camera line of sight) passing through the above points (points around the target voxel on the line L) is drawn from the lens center of the cameras C1 and C2. FIG. 3 is a diagram in the case where the line L is parallel to one side of the voxel, and the alternate long and short dash line is a camera line of sight connecting the above-mentioned points at equal intervals and the lens centers of C1 and C2. .

  The points on the line L that are equally spaced and the camera line of sight connecting these points and the lens centers of C1 and C2 are on the same epipolar plane, so the camera lines of sight intersect each other. White circles, double circles, and triple circles in the figure indicate their intersections.

(Step 4: Calculation of reliability at each intersection of camera line of sight)
The reliability at the intersection on each camera line of sight is obtained from the reliability of voxels near each intersection. An example is shown in FIG. 401 is an intersection of camera line of sight, 402 is a solid including the intersection of camera line of sight, 403 is a voxel group near the intersection of camera line of sight, and is a voxel group having a portion overlapping with the solid of 402.

Reliability p _{C1C2d (i, j) at} the intersection of 401 (where (i, j) is a number for distinguishing the line of sight from the cameras C1 and C2, and (i, j) = (i _v , j _v ) Indicates a camera line of sight passing through the center of the _target voxel v), voxel groups in the vicinity of 403 are denoted by v ₁ to v _m, and their reliability levels are the reliability levels p _{C1C2,1, (i, j),} respectively. _{~p C1C2, m, (i,} j), calculates the ratio of solid of 402 for overlapping and volume of the solid of and their voxels 402 as follows and _r 1 ~r _m.

  If the 402 solid is the same size as the voxel centered at the intersection of 401, then m = 8.

(Step 5: Reliability correction)
On each camera line of sight, the surface of the object should exist at only one point. Therefore, using this constraint condition, the reliability is corrected to increase the accuracy of the reliability at each intersection.

  When the object surface is on the line of sight of the camera C1 in FIG. 3, it is desirable that the possibility that the object surface exists at each intersection is high only at the intersection of the line of sight close to the object surface, and the other intersections are small. . The following processing is performed so that the reliability at each intersection is the same.

  FIG. 6 is a diagram for explaining this processing. FIG. 6A shows a reproduction of the intersection of the camera line of sight and the line of sight in FIG. The alternate long and short dash line indicates the camera line of sight, and the white circle and the triple circle indicate the intersection of the camera line of sight. In particular, the triple circle indicates the center point of the target voxel v. FIG. 6B shows a result of geometric transformation so that the camera line of sight is parallel for convenience. Although the camera line of sight shown in FIG. 6A does not intersect, the part indicated by a black circle in FIG. 6B is that part. The C1 side and the C2 side indicate the side where the camera is placed.

(Step 5-1: Replacement of intersection of camera line of sight)
The original reliability shown in FIG. 6 (a) is assigned to the intersection of the camera line of sight indicated by the white circle and the triple circle in FIG. 6 (b), and is not actually intersected by the black circle in FIG. 6 (b). Is assigned 0.

(Step 5-2: Optimization of reliability of intersection of camera line of sight)
Since it is desirable that the reliability of the intersection point of the camera line of sight in FIG. 6B is increased by one in each row and each column, for example, the following energy function E is created and considered to be minimized. .

Here, the reliability p _{C1C2d (i, j)} is the reliability of the point in the i row and j column shown in FIG. 6, n _C1 and n _C2 are the numbers of lines on the C1 and C2 sides, and A and B are appropriate. Indicates a constant size.

Using this equation, 1 = (1−δ _ij ) + δ _ij is substituted for each of the first and second terms using the Kronecker delta function, and converted into a multiple primary form as follows.

  Using this energy function, the following dynamical system is created.

  In order to minimize the energy function using this dynamic system, for example, the Euler method may be used as follows.

First, the reliability p _{C1C2d (i, j)} is initialized with the reliability calculated by the equation (8.5), and then

And then

Calculate However, Δ is the maximum value of the time interval and is a constant determined by the user. w is the norm of the vector w = (w ₁₁ ,..., w _C1C2 ), and may be, for example, the sum of squares of each element of the vector.

Next, the reliability _{pd (i, j)} is updated as follows.

  For example, the following conditions may be used as the termination conditions.

Here, ε is a threshold determined by the user.

  Until the above termination condition is satisfied, the equations (8.9), (8.10), and (8.11) are repeatedly performed and expressed as the equation (8.6) or (8.7). The reliability is updated while reducing the energy function, and the iteration is terminated when the termination condition is satisfied.

  In this embodiment, the energy function is minimized using the Euler method and the correction of the reliability is obtained. However, the present invention is not limited to the method shown here, and the combination optimization using the neural network is performed. A method for solving the problem may be used.

(Step 5-3: Replacing the optimized reliability with the reliability of the target voxel)
The reliability of the center of the noticed voxel v of the triple circle shown in FIG. 6B calculated using step 5-2 is newly set as the reliability of the noticed voxel of the triple circle shown in FIG. 6A.

  It is also possible to obtain the same reliability for a plurality of pairs without fixing the camera pair selected in step 3, and to integrate the obtained reliability to further improve the accuracy of the reliability. In this case, the following steps 5-4 and 5-5 are added.

(Step 5-4: Optimization of reliability in other camera pairs)
Steps 3 to 5-3 are performed for a plurality of camera pairs with respect to one voxel.

(Step 5-5: Integration of reliability calculated from a plurality of camera pairs)
For one voxel, the reliability obtained from a plurality of camera pairs is integrated.
For example, the integration is as follows.

Here, the reliability _{p CrCsd (iv, jv)} camera pairs Cr, obtained by Cs, a reliability of the voxel v to be examined, the reliability _{p d} is the confidence that integrates them.

  The reliability may be calculated as follows.

Bias is a constant determined by the user. If Bias is set to 1, for example, the influence of a low reliability _{pCrCsd (iv, jv)} can be reduced.

(Step 6: Optimization and integration of reliability for all voxels)
Repeat steps 3-5 for all voxels.

  Through the above steps, the reliability of all voxels is calculated.

  Steps 3 to 6 can be repeated to further increase the reliability of the reliability of all voxels. In this case, the following step 7 is added.

(Step 7: Repeated execution of maximum calculation of reliability)
Repeat steps 3-6.

(Step 8: Output of 3D shape data)
Finally, the reliability of all voxels calculated by recalculation is set as three-dimensional shape data.

  Furthermore, binary data such as whether or not there is a surface can be obtained using the reliability of all the voxels, and the presence or absence of this surface can also be used as three-dimensional shape data. In this case, the following step 9 is added.

(Step 9: Discrimination of surface presence and output of data)
For this purpose, for example, using a predetermined threshold value, it is determined that an object surface exists in a voxel having a reliability equal to or higher than the threshold value, and no object surface exists in other voxels. The result is taken as three-dimensional shape data.

  The threshold value can be determined in advance as described above, but in addition, using a probability density function of reliability of all voxels (a function having the magnitude of reliability as an input and its occurrence probability as an output), Use an existing binarization algorithm that minimizes the least square error (for example, Otsu's threshold selection method) or an existing binarization algorithm that minimizes the average misclassification rate (for example, the kittler threshold selection method) The threshold may be determined.

  As an effect of the present embodiment, the reliability at the intersection of the line of sight of the camera pair has little geometric contradiction so that the reliability of each voxel has a three-dimensional consistency obtained from the camera pair. As a result, the accuracy of the reliability of each voxel is improved, and as a result, even when a similar texture exists on the object surface, the object surface can be correctly determined. There are advantages.

Further, as an effect of the present embodiment, since the reliability of each voxel has a three-dimensional consistency obtained from the camera pair, there is an advantage that the reliability of the reliability is improved.
Further, as an effect of the present embodiment, since the results of a plurality of camera pairs are integrated, there is an advantage that reliability of reliability is improved.

[Embodiment 2]
The processing flow of the second embodiment will be described with reference to FIG. (A) is a flow when the object surface is obtained as a determined result, and (b) is a flow when the object surface is obtained as a probabilistic result.

First, the same object is imaged using a plurality of cameras.
Using the captured image data, a three-dimensional shape data generation device (not shown) performs the following calculation. The three-dimensional shape data generation apparatus can be configured by a program stored in a storage device and a computer, and hardware can be used in place of a part or all of the program.
A three-dimensional space in which the surface of a target object for which three-dimensional shape data is desired to be measured is divided by voxels as in the space carving method.
Next, paying attention to one voxel, a plurality of points are taken around the voxel, and a camera line of sight passing through these points is drawn from the lens centers of the two selected cameras.
Next, a value indicating the possibility that the object surface exists is calculated for each intersection of the camera line of sight.
Next, an optimization algorithm is applied to calculate a value obtained by correcting a value indicating the possibility that the object surface exists at the point of interest and a value indicating the possibility that the object surface exists at the peripheral points. The value at the point of interest is a value indicating the existence probability of the object surface.
For all combinations in which two of the above are selected from all cameras, the object surface existence probability at the point of interest is obtained, and the result obtained by integrating these is set as the object surface existence probability at the point of interest.
A value indicating the possibility that an object surface exists in all voxels is calculated.
Finally, a point where the object surface is highly likely to be present is set as the surface of the object, or the probability that the object surface exists is calculated from the possibility that the object surface exists.

  Hereinafter, the second embodiment of the present invention will be described in detail with reference to the drawings.

  A three-dimensional data generation method according to the second embodiment will be described with reference to FIGS. 1, 3, 5, and 6. This method captures the same object using multiple cameras, calculates the possibility of the surface of the object at a certain point in space from the similarity of the captured images, and is likely to exist Is the surface of the object, or the probability of existence of the surface of the object is calculated from the possibility of existence of the object, thereby generating three-dimensional shape data of the surface of the object. Hereinafter, a method for calculating the possibility that the surface of the object exists will be described in detail.

(Step 1: Voxel division of space)
As shown in FIG. 1, a three-dimensional space including a target object whose three-dimensional shape data is to be measured is divided by voxels as in the space carving method. Reference numeral 101 denotes one divided voxel. It is assumed that there are a plurality of cameras that image an object in the space, and here there are n cameras. Reference numeral 111 denotes one of the cameras C1, which is the first camera for convenience. 112 is the lens of the camera C1, 113 is the lens center of the lens 112, 114 is the image sensor of the camera C1, 115 is the origin of the image sensor 114, 116 is a corresponding point on the image sensor 114 that is calculated when the voxel 101 is imaged, It is. The same applies to the other cameras C2 to Cn.

  In the camera C 1 111, the voxel 101 passes through the lens center 113 of the lens 112 and forms an image at a point 116 on the image sensor 114. However, the coordinates of the point 116 are expressed in a coordinate system with the point 115 as the origin.

(Step 2: Calculation of possibility of object surface in target voxel)
A value (= reliability) indicating the possibility of whether or not an object surface is present in the voxel is set to v, and a point on the image sensor of each camera C1 to Cn on which the voxel 101 forms an image, or Then, tentative determination is made based on the similarity of the pixel values of the points in the vicinity of the point, and then correction by recalculation of reliability as described later is performed in order to increase the reliability of the reliability.

(Step 2-1: Determination of points around the target voxel)
One camera pair is selected from a plurality of cameras, and a camera line of sight as shown below is drawn to determine a neighboring point of the target voxel that is the intersection. How to draw the camera line of sight will be described with reference to FIG.

  Reference numeral 301 denotes a target voxel v whose reliability is to be recalculated, and 302 is a range in a three-dimensional space in which the presence or absence of the object surface is to be examined, and a range in a three-dimensional space to be divided by voxels. There are a plurality of cameras that image an object in the space. Here, only one pair C1 and C2 of the camera pair is shown here. Reference numeral 311 denotes one of the cameras C 1 and 312, a lens of the camera C 1, 313 denotes a lens center of the lens 312, 314 denotes an image sensor of the camera C 1, and 315 denotes a pixel of the image sensor 314. The same applies to the other cameras C2. Reference numeral 303 denotes a line L passing through the target voxel v and parallel to a line passing through the lens centers of the cameras C1 and C2.

(Step 2-1-1: Determination of epipolar plane)
The target voxel v for which the accuracy of the reliability p is desired to be increased is determined. At this time, the plane including the center of each camera lens of the voxel v and the cameras C1 and C2 is determined. This plane is called an epipolar plane.

(Step 2-1-2: Determination of a line L parallel to two camera centers)
A line L that is on the epipolar plane and is parallel to the line passing through the center of each of the cameras C1 and C2 and passing through the center of the target voxel v is drawn.

(Step 2-1-3: Determination of points around the target voxel on the line L)
Points are drawn on the line L at regular intervals starting from the center point of the voxel v on the line L around the center point of the target voxel v (attention point, triple circle in FIG. 3) (see FIG. 3). double circle). This point includes a point at the center of the voxel. The attention point is not limited to the center point of the attention voxel v, but may be an arbitrary point in the attention voxel v. The interval may be, for example, the interval between the sides of the voxel, but is not limited thereto. Furthermore, although it was set as equal intervals here, it does not necessarily need to be equal intervals. Further, the geometric range for taking points is not particularly limited, and may be a range that covers the entire voxel.

(Step 2-1-4: Camera line-of-sight drawing on line L and determination of points around target voxel)
A line (camera line of sight) passing through the above points (points around the target voxel on the line L) is drawn from the lens center of the cameras C1 and C2. FIG. 3 is a diagram in the case where the line L is parallel to one side of the voxel, and the alternate long and short dash line is a camera line of sight connecting the above-mentioned points at equal intervals and the lens centers of C1 and C2. .

  The points on the line L that are equally spaced and the camera line of sight connecting these points and the lens centers of C1 and C2 are on the same epipolar plane, so the camera lines of sight intersect each other. White circles, double circles, and triple circles in the figure indicate their intersections.

(Step 2-2: Provisional calculation of the possibility that there is an object surface at each intersection of the center point of the target voxel and the camera line of sight)
A camera pair C1 in which each intersection point forms a value (= reliability) indicating the possibility of the presence of an object surface at each intersection point of the voxel v of interest and the camera line of sight around it. , And the pixel similarity of a point on the image sensor of C2 or a point in the vicinity of the point. For example, the value indicating similarity is a standard of pixel values in the vicinity of the corresponding point on the image sensor of the camera pair C1, C2 that is calculated that the same point is imaged. It is a value similar to a deviation.

Hereinafter, a method of calculating similarity using the dissimilarity disR _{C1C2 (i, j)} will be described. However, (i, j) is the number to distinguish the line of sight is out of the camera C1, C2, (i, j ) = (i v, j v) it is a camera line-of-sight passing through the center of the voxel of interest v Indicates. The dissimilarity disR _{C1C2 (i, j)} is calculated as follows, for example.

Here, m _{(i, j), Ck} is the imaging position on the image sensor of the camera Ck when the camera Ck images the center point of the _target voxel v and each intersection of the camera gazes around it. a is a set of vectors indicating the vicinity of m _{(i, j) and Ck} on the image sensor of the camera, a set of vectors having m _{(i, j) and Ck} as the origin, and p is one of a The vector, T (k), is a texture at the coordinate k of the pixel on the image sensor, and is usually a color vector whose elements are RGB color densities, and the bar T _{(i, j)} (k) is m _{(i, j),} The average of all camera textures for the coordinates p of points in the vicinity of _Ck , | x | represents the magnitude of x, for example, the square root of the sum of squares of each element, and m is the variance of the texture (| T− Change how disR and reliability change with respect to the average of T That value is.

An example of how to obtain the texture T (m _{(i, j), Ck} + p) will be described with reference to FIG. Here, 501 is the intersection of the camera lines of sight of cameras C1 and C2, 502 is the camera line of sight of camera C1, 503 is the intersection of the camera line of sight of camera C1 and the image sensor, and 504 is on the image sensor near the camera line of sight of camera C1. A pixel group, 505 is centered on the intersection of the camera line of sight of the camera C1 and the image sensor, and is a square having the same size as the pixel of the image sensor, 506 is an image sensor centered on the intersection of the camera line of sight of the camera C1 and the image sensor A square of the same size as the pixel of the image sensor in the vicinity of the square of the same size as the pixel of, 512 is a camera line of sight of the camera C2, 513 is an intersection of the camera line of sight of the camera C2 and the image sensor, and 514 is a camera of the camera C2 A pixel group 515 on the image sensor in the vicinity of the line of sight is a quadrangle having the same size as the pixel of the image sensor centering on the intersection of the camera line of sight of the camera C2 and the image sensor.

Since the coordinates m _{(i, j), Ck} of the pixels on the image sensor do not necessarily match the coordinates of the pixels of the image sensor, even if a is a set of vectors in units of pixel width, m _{(i, j), Ck} + p does not match the coordinates of the pixels of the image sensor. This situation is shown in FIG. In FIG. 5A, two lines of sight passing through the camera line of sight intersection 501 intersect at the points indicated by 503 and 513 on the image sensors of the cameras C1 and C2, respectively. A state where the center of the upper pixel is deviated is shown.

FIG. 5B is a view of the image sensor of the camera C1 as viewed from directly above. A quadrangle having the same size as that of the image sensor including an intersection with the camera sight line 505, and a quadrangle 506 having the same size around the image sensor. Is drawn. m _{(i, j) and Ck} correspond to the coordinates of 503, and a corresponds to the center coordinates of the respective squares 505 and 506. However, as shown in FIG. 5B, this is a case where a is nine coordinates centered on the intersection.

FIG. 5D is a diagram showing a relationship between a square of the same size as the pixel of the image sensor centered on the intersection of the camera line of sight and the image sensor, and m _{(i, j) , Ck} are the coordinates of the pixels of the image sensor _{, mn1 (i, j), Ck} , _{mn2 (i, j), Ck} , _{mn3 (i, j), Ck} , _{mn4 (i, j), Ck} Is the coordinates of the center of the pixel of the image sensor that overlaps a quadrangle of the same size as the image sensor centered on the intersection of the camera line of sight and the image sensor. In the figure, s and t indicate the overlapping ratio, and 0 ≦ s and t ≦ 1. At this time, the texture T (m _{(i, j), Ck} ) can be calculated using the bilinear method as follows.

Although the bilinear method is shown here, other existing interpolation methods such as the nearest neighbor method and the bicubic method may be used. The texture T (m _{(i, j), Ck} + p) when p is changed in a can be calculated in the same manner.

Since the light emitted from a minute area on the object surface is considered to have almost the same texture in the corresponding area on the image sensor of each camera, the dissimilarity disR _{C1C2 (i, j)} becomes small. Conversely, the smaller the dissimilarity disR _{C1C2 (i, j), the} higher the possibility that the corresponding voxel v has an object surface. From the dissimilarity disR _{C1C2 (i, j), the} similarity R _{C1C2 (i, j)} having the opposite property is obtained as follows, for example.

Here, σ is a parameter for adjusting the influence on the similarity R _{C1C2 (i, j)} with respect to the dissimilarity disR _{C1C2 (i, j)} , and the smaller the σ, the larger the similarity. Changes.

The higher the degree of similarity, the higher the possibility that the object surface is in the voxel. This possibility is referred to as reliability and expressed as p _{C1C2d (i, j)} . Let p _{C1C2d (i, j) be} as follows: This p is different from the vector p in (8.101). These can be distinguished in the expression which is the image data, but cannot be distinguished on the text. Therefore, hereinafter, “reliability p” is described on the text.

In order to increase the accuracy of the voxel reliability, correction by recalculation of reliability is performed by the following processing.

(Step 2-3: Correction of reliability)
On each camera line of sight, the surface of the object should exist at only one point. Therefore, using this constraint condition, the reliability is corrected to increase the accuracy of the reliability at each intersection.

  When the object surface is on the line of sight of the camera C1 in FIG. 3, it is desirable that the possibility that the object surface exists at each intersection is high only at the intersection of the line of sight close to the object surface, and the other intersections are small. . The following processing is performed so that the reliability at each intersection is in the desired state as described above.

  FIG. 6 is a diagram for explaining this processing. FIG. 6A shows a reproduction of the intersection of the camera line of sight and the line of sight in FIG. The alternate long and short dash line indicates the camera line of sight, and the white circle and the triple circle indicate the intersection of the camera line of sight. In particular, the triple circle indicates the center point of the target voxel v. FIG. 6B shows a result of geometric transformation so that the camera line of sight is parallel for convenience. Although the camera line of sight shown in FIG. 6A does not intersect, the part indicated by a black circle in FIG. 6B is that part. The C1 side and the C2 side indicate the side where the camera is placed.

(Step 2-3-1: Replacement of intersection of camera line of sight)
The original reliability shown in FIG. 6 (a) is assigned to the intersection of the camera line of sight indicated by the white circle and the triple circle in FIG. 6 (b), and is not actually intersected by the black circle in FIG. 6 (b). Is assigned 0.

(Step 2-3-2: Optimization of reliability of intersection of camera line of sight)
Since it is desirable that the reliability of the intersection point of the camera line of sight in FIG. 6B is increased by one in each row and each column, for example, the following energy function E is created and considered to be minimized. .

Here, the reliability p _{C1C2d (i, j)} is the reliability of the point in the i row and j column shown in FIG. 6, n _C1 and n _C2 are the numbers of lines on the C1 and C2 sides, and A and B are appropriate. Indicates a constant size.

Using this equation, 1 = (1−δ _ij ) + δ _ij is substituted for each of the first and second terms using the Kronecker delta function, and converted into a multiple primary form as follows.

  Using this energy function, the following dynamical system is created.

  In order to minimize the energy function using this dynamic system, for example, the Euler method may be used as follows.

First, the reliability p _{C1C2d (i, j)} is initialized with the reliability calculated by the equation (8.104), and then

And then

Calculate However, Δ is the maximum value of the time interval and is a constant determined by the user. w is the norm of the vector w = (w ₁₁ ,..., w _C1C2 ), and may be, for example, the sum of squares of each element of the vector.

Next, the reliability p _{C1C2d (i, j)} is updated as follows.

  For example, the following conditions may be used as the termination conditions.

Here, ε is a threshold determined by the user.

  Until the end condition as described above is satisfied, the expressions (8.109), (8.110), and (8.111) are repeatedly performed and expressed as the expression (8.106) or (8.107). The reliability is updated while reducing the energy function, and the iteration is terminated when the termination condition is satisfied.

  In this embodiment, the energy function is minimized using the Euler method and the correction of the reliability is obtained. However, the present invention is not limited to the method shown here, and the combination optimization using the neural network is performed. A method for solving the problem may be used.

(Step 2-3-3: Replacing the Optimized Reliability with the Reliability of the Voxel of Interest)
The reliability of the center of the noticed voxel v of the triple circle shown in FIG. 6B calculated using step 2-3-2 is newly set as the reliability of the noticed voxel of the triple circle shown in FIG. 6A.

  It is also possible to obtain the same reliability for a plurality of pairs without fixing the camera pair selected in step 2-1, and integrate the obtained reliability to further improve the accuracy of the reliability. In this case, the following steps 2-3-4 and 2-3-5 are added.

(Step 2-3-4: Optimization of reliability in other camera pairs)
Steps 2-1 to 2-3-3 are performed for a plurality of camera pairs with respect to one voxel.

(Step 2-3-5: Integration of reliability calculated from a plurality of camera pairs)
For one voxel, the reliability obtained from a plurality of camera pairs is integrated. For example, the integration is as follows.

Here, the reliability _{p CrCsd (iv, jv)} camera pairs Cr, obtained by Cs, a reliability of the voxel v to be examined, the reliability _{p d} is the confidence that integrates them.

  The reliability may be calculated as follows.

Bias is a constant determined by the user. If Bias is set to 1, for example, the influence of a low reliability _{pCrCsd (iv, jv)} can be reduced.

(Step 3: Optimization and integration of reliability for all voxels)
Step 2 is performed for all voxels.

  Through the above steps, the reliability of all voxels is calculated.

  Thereafter, Steps 2 to 3 can be repeatedly performed to further increase the reliability of the reliability of all voxels. In this case, the following step 4 is added.

(Step 4: Repeated execution of maximum calculation of reliability)
Repeat steps 2-3.

(Step 5: Output of 3D shape data)
Finally, the reliability of all voxels calculated by recalculation is set as three-dimensional shape data.

  Furthermore, binary data such as whether or not there is a surface can be obtained using the reliability of all the voxels, and the presence or absence of this surface can also be used as three-dimensional shape data. In this case, the following step 6 is added.

(Step 6: Discrimination of presence / absence of surface and output of data)
For this purpose, for example, using a predetermined threshold value, it is determined that an object surface exists in a voxel having a reliability equal to or higher than the threshold value, and no object surface exists in other voxels. The result is taken as three-dimensional shape data.

  The threshold value can be determined in advance as described above, but in addition, using a probability density function of reliability of all voxels (a function having the magnitude of reliability as an input and its occurrence probability as an output), Use an existing binarization algorithm that minimizes the least square error (for example, Otsu's threshold selection method) or an existing binarization algorithm that minimizes the average misidentification rate (for example, the kittler threshold selection method) The threshold may be determined.

  As an effect of the present embodiment, the reliability at the intersection of the line of sight of the camera pair has little geometric contradiction so that the reliability of each voxel has a three-dimensional consistency obtained from the camera pair. As a result, the accuracy of the reliability of each voxel is improved, and as a result, even when a similar texture exists on the object surface, the object surface can be correctly determined. There are advantages.

  Further, as an effect of the present embodiment, since the reliability of each voxel has a three-dimensional consistency obtained from the camera pair, there is an advantage that the reliability of the reliability is improved.

  Further, as an effect of the present embodiment, since the results of a plurality of camera pairs are integrated, there is an advantage that reliability of reliability is improved.

  As mentioned above, the invention made by the present inventor has been specifically described based on the embodiment. However, the invention is not limited to the embodiment, and various modifications can be made without departing from the scope of the invention. Of course.

It is a figure which shows the relationship between the division | segmentation by the voxel of three-dimensional space, and the imaging position of several cameras. It is a figure which shows the trouble of the conventional method. It is a figure which shows arrangement | positioning of the intersection of a camera line of sight used for the calculation for taking the three-dimensional positional alignment of the similarity using a camera pair. It is a figure which shows the intersection of eyes | visual_axis, and the voxel of the vicinity. It is a figure which shows the pixel on the image pick-up element of the visual line vicinity which passes through the intersection of a visual line (the relationship between the intersection of visual lines, and the image sensor in two cameras). It is a figure which shows the pixel on the image pick-up element of the visual line vicinity which passes through the intersection of a visual line (the intersection of a camera visual line and an image sensor, and the pixel of the vicinity). It is a figure which shows the pixel on the image pick-up element in the vicinity of a visual line passing through the intersection of the line of sight (the relationship between the square of the same size as the image pickup element pixel around the point of interest and the pixel of the image pickup element). It is a figure which shows the pixel on the image pick-up element in the vicinity of a visual line passing through the intersection of the line of sight (relationship between a rectangle having the same size as the image pickup element image centering on the intersection of the lines of sight i and j and the image of the image pickup element). . It is a figure which shows a mode that the arrangement | positioning of an intersection and the line of sight were made parallel, and the intersection was replaced. It is a figure which shows the flow of the typical whole process of a prior art. It is a figure which shows the flow (when obtaining as a result of determining the object surface) of the whole process 1 by Embodiment 1. FIG. It is a figure which shows the flow (when obtaining the object surface as a stochastic result) of the whole process 1 by Embodiment 1. FIG. It is a figure which shows the flow (when obtaining as a result of having determined the object surface) of the whole process 2 by Embodiment 2. FIG. It is a figure which shows the flow (when obtaining the object surface as a stochastic result) of the whole process 2 by Embodiment 2. FIG.

Explanation of symbols

101 is a voxel v, 111 is a camera, 112 is a lens, 113 is a lens center, 114 is an image sensor, 115 is an image sensor origin, 116 is an image point of the voxel, 201 is a voxel v1 and 202 that do not include the object surface, and 202 is an object Voxels v2 and 203 including the surface are a small area a on the object surface, 204 is a small area b on the object surface, 211 is a camera, 212 is a lens, 213 is a camera center, 214 is an image sensor, and 2031 is an object in the camera C1. An imaging region a _C1 , 2032 of the small surface area a is an imaging region a _C2 , 2033 of the small object surface region a in the camera C2, and 2033 is an imaging region b _C2 , 301 of the small object surface region b of the camera C2. The target voxel v, 302 is a range to be divided by the voxel, 303 is a line passing through the center of the camera C1, C2 through the center of the target voxel. Parallel lines, 311 is a camera, 312 is a lens, 313 is a lens center, 314 is an image sensor, 315 is an image sensor pixel, 401 is an intersection of camera lines of sight, and 402 is the same size as a voxel centered at the intersection of camera lines of sight 403 is a group of voxels near the intersection of the camera line of sight, 501 is the intersection of the camera lines of sight of the cameras C1 and C2, 502 is the camera line of sight of the camera C1, 503 is the intersection of the camera line of sight of the camera C1 and the image sensor, and 504 is A pixel group on the image sensor in the vicinity of the camera line of sight of the camera C1, 505 is a quadrangle having the same size as the pixel of the image sensor centering on the intersection of the camera line of sight of the camera C1 and the image sensor, and 506 is a camera line of sight of the camera C1. And a square of the same size as the pixels of the image sensor in the vicinity of the square of the same size as the pixels of the image sensor centering on the intersection of the image sensor and 512 La C2 camera line of sight 513 is the intersection of the camera line of sight of the camera C2 and the image sensor, 514 is a pixel group on the image sensor near the camera line of sight of the camera C2, 515 is centered on the intersection of the camera line of sight of the camera C2 and the image sensor A quadrangle having the same size as the pixels of the image sensor.

Claims (6)

By calculating the possibility that the surface of an object exists at a certain point in space from the similarity of images in which the same object is captured by multiple cameras, and setting the point that is highly likely to exist as the surface of the object In the three-dimensional shape data generation method in the three-dimensional shape data generation device that generates the three-dimensional shape data of the surface of the object,
The calculation by the three-dimensional shape data generation device of the value indicating the possibility that the surface of the object exists at a certain point in the space,
A first step of dividing the space by voxels;
A second step of determining a first value indicating the possibility that an object surface exists in each voxel from the similarity of pixel values of points on the image sensor of each camera corresponding to each voxel;
Pay attention to one of the voxels, take a plurality of points around the point of interest, which is an arbitrary point in the voxel, and take a camera line of sight that passes through those points from the lens center of the two cameras A third step to draw,
A fourth step of calculating, for each intersection between the camera lines of sight, a second value indicating the possibility of an object at the intersection from the first value in a voxel near the intersection;
Applying an optimization algorithm using a constraint that an object exists only at one intersection point on each camera line of sight to the second value at the intersection point, the possibility that the surface of the object exists at each intersection point Calculating a third value obtained by correcting the value of the fifth, and the fifth value as a fourth value indicating the possibility that the object surface exists in the voxel to which the third point at the point of interest is noted;
A method for generating three-dimensional shape data, comprising: a sixth step for executing the processing from the third step to the fifth step on the other voxels created in the first step. The three-dimensional shape data generation method according to claim 1,
In the first step, the space of the region where the surface of the object is supposed to exist is divided by voxels,
In the third step, the target point is set as a center point of the target voxel, and a plurality of points are taken around the target point on an epipolar plane formed by the target point and the lens center of the two cameras. A method for generating three-dimensional shape data. The three-dimensional shape data generation method according to claim 1 or 2,
The plurality of cameras are three or more,
In the processes from the third step to the fifth step, two cameras are selected from the plurality of cameras by a plurality of selection methods with different combinations, and a fourth value for the voxel to be noticed in each combination is selected. 3D shape data generation characterized by further comprising a seventh step of calculating and integrating the fourth value in each combination as a value indicating the possibility that an object surface exists in the target voxel Method. By calculating the possibility that the surface of an object exists at a certain point in space from the similarity of images in which the same object is captured by multiple cameras, and setting the point that is highly likely to exist as the surface of the object In the three-dimensional shape data generation method in the three-dimensional shape data generation device that generates the three-dimensional shape data of the surface of the object,
The calculation by the three-dimensional shape data generation device of the value indicating the possibility that the surface of the object exists at a certain point in the space,
A first step to voxel-divide the space;
Pay attention to one of the voxels, take a plurality of points around the point of interest, which is an arbitrary point in the voxel, and take a camera line of sight that passes through those points from the lens center of the two cameras The second step to draw,
For each intersection of the camera line of sight, the first value indicating the possibility that an object exists at the intersection is the similarity of the pixel values of the corresponding points on the image sensor of the corresponding two cameras corresponding to the intersection. A third step to calculate from
Applying an optimization algorithm using a constraint condition that an object exists only at one intersection point on each camera line of sight with respect to the first value at the intersection point, the possibility that the surface of the object exists at each intersection point A fourth step of calculating a second value corrected value, and setting the second value at the point of interest as a third value indicating the possibility that the object surface exists in the target voxel;
A fifth step for performing the processing from the second step to the fourth step on the other voxels created in the first step;
A three-dimensional shape data generating method comprising: The three-dimensional shape data generation method according to claim 4,
In the first step, the space of the region where the surface of the object is supposed to exist is divided by voxels,
In the second step, the target point is set as a center point of the target voxel, and a plurality of points are taken around the target point on an epipolar plane formed by the target point and the lens center of the two cameras. A method for generating three-dimensional shape data. The three-dimensional shape data generation method according to claim 4 or 5,
The plurality of cameras are three or more,
In the processes from the second step to the fourth step, two cameras are selected from the plurality of cameras by a plurality of selection methods with different combinations, and a third value for the voxel to be noticed in each combination is selected. 3D shape data generation characterized by further comprising a sixth step of calculating and integrating a third value in each combination as a value indicating the possibility that an object surface exists in the target voxel Method.

Images