JP2018181047A - Three-dimensional shape model generating device, three-dimensional shape model generating method and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a three-dimensional shape model generating device that improves both the processing time and restoration accuracy for generation of a three-dimensional shape model from a multi-viewpoint image.SOLUTION: The three-dimensional shape model generating device for generating a three-dimensional shape model of an object from two or more multi-viewpoint images obtained by capturing the object from different viewpoints and camera parameters of an image capturing device that captures multi-viewpoint images includes: a depth map generating unit for generating a depth map corresponding to a multi-viewpoint image for each viewpoint from each of the multi-viewpoint images and the camera parameters; a three-dimensional point group generating unit for generating a three-dimensional point group from the depth map and the camera parameters; a viewpoint selecting unit for selecting a viewpoint using matching processing of three-dimensional points of the three-dimensional point group from the three-dimensional point group and the camera parameters; and a three-dimensional coordinates calculating unit for re-calculating three-dimensional coordinates in a three-dimensional space of the three-dimensional points by performing three-dimensional point matching based on the selected viewpoint based on phase-only correlation.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a three-dimensional shape model generating device, a three-dimensional shape model generating method, and a program for generating a three-dimensional shape model of an object from a plurality of captured images of two or more sheets where the object is imaged from different viewpoints. .

There is a three-dimensional reconstruction method for generating a three-dimensional shape model of an object based on a plurality of two-dimensional captured images (hereinafter referred to as "multi-viewpoint images") in which the object is captured from different viewpoints.
As one of the above three-dimensional reconstruction methods, normalized mutual correlation of local regions in each of a plurality of multi-viewpoint images is performed to obtain three-dimensional coordinates of each point in a three-dimensional point group indicating the three-dimensional shape of the object There is a method (the first three-dimensional reconstruction method). For example, in this first three-dimensional restoration method, three-dimensional coordinates in three-dimensional space of pixels in a captured image corresponding to a certain viewpoint are calculated using plain sweeping for a local region including a plurality of feature points. Find the 3D coordinates of the included pixel. That is, in plain sweeping, on the premise of a full search, three-dimensional coordinates and a normal vector are obtained for which the numerical value obtained by adding the correlation value of the normalized cross-correlation between the local regions of each of a plurality of multiview images is highest. There is.

Further, as another method of three-dimensional reconstruction (second three-dimensional reconstruction method), each of the local regions of a plurality of multi-viewpoint images is subjected to Fourier transform, and the phase difference of the phase spectrum between each of the Fourier-transformed local regions is There is a method of obtaining three-dimensional coordinates in a three-dimensional space based on the inclination (see, for example, Non-Patent Document 1).
This second three-dimensional reconstruction method can determine the three-dimensional coordinates of the pixels included in the local region with high accuracy, as compared to the normalized cross-correlation between the local regions described above. Thereby, the second three-dimensional reconstruction method generates the three-dimensional shape model of the object with higher accuracy, as compared to the first three-dimensional reconstruction method of performing the normalized mutual correlation between the local regions described above. can do.

  However, since the second three-dimensional reconstruction method is performed based on the phase difference of the phase spectrum between the local regions in each of a plurality of multi-view images, it is necessary to perform Fourier transform to obtain the spatial frequency of the local region . For this reason, this second three-dimensional reconstruction method has the drawback that the computational cost is high as compared to the first three-dimensional reconstruction method using normalized cross correlation between local regions. Therefore, the second three-dimensional reconstruction method is not suitable for plain sweeping on the premise of a full search, in consideration of the time for performing the Fourier transform of each local region.

Therefore, in order to perform three-dimensional reconstruction using the second three-dimensional reconstruction method in a realistic processing time, processing time has conventionally been shortened by coarse-to-fine search using an image pyramid (for example, patent) Reference 1).
In the coarse-to-fine search using this image pyramid, the multi-viewpoint image is converted to a lower resolution image of lower resolution, the Fourier transformation is performed on the local region of the low resolution image, and the phase between the local regions is performed by plane sweeping. Based on the phase difference of the spectrum, the three-dimensional coordinates of the pixels in the local region are determined. As a result, the entire search in plain sweeping is limited to only the low resolution image, and the time taken for the three-dimensional reconstruction processing using the second three-dimensional reconstruction method can be shortened.

Patent No. 5787398 gazette

Shuji Sakai, Koichi Ito, Takafumi Aoki, Takafumi Watanabe and Hiroki Unten, "Phase-based window matching with geometric correction for multi-view stereo," IEICE Transactions on Information and Systems, Vol. E98-D, no. 10, pp. 1818-1828, 2015.

However, in Patent Document 1 above, in order to further shorten the processing time, the number of layers of low resolution images in the image pyramid is increased. In this case, the error generated in the lower resolution image of lower resolution propagates to the lower resolution image in the higher resolution layer, and the restoration accuracy of the three-dimensional shape model of the object captured in the multi-viewpoint image is degraded. .
On the other hand, if the number of layers of the image pyramid is reduced to suppress the error propagation area, the number of pixels for which the entire search is performed in plain sweeping is increased. As the number of pixels increases, the processing time for plain sweeping increases.

  As described above, in the generation of a three-dimensional shape model from multi-viewpoint images, the processing time becomes longer when attempting to improve the restoration accuracy of the three-dimensional shape model, and the processing time for restoration of the three-dimensional shape model decreases. When it is made to make it do, recovery accuracy falls, and there is a trade-off between processing time and recovery accuracy.

  The present invention has been made in view of such a situation, and is capable of generating a three-dimensional shape model from multi-viewpoint images while satisfying both processing time and restoration accuracy. , And a three-dimensional shape model generation method and program.

  In order to solve the problems described above, a three-dimensional shape model generating apparatus according to the present invention includes two or more multi-viewpoint images obtained by capturing an object from different viewpoints, and camera parameters of an imaging apparatus capturing the multi-viewpoint image. From the multi-viewpoint image and the camera parameters, a depth map corresponding to the multi-viewpoint image for each viewpoint from the multi-viewpoint image and the camera parameter. From a map generation unit, a three-dimensional point group generation unit for generating a three-dimensional point group indicating the three-dimensional shape of the object from the depth map and the camera parameter, and from the three-dimensional point group and the camera parameter A viewpoint selecting unit for selecting two or more viewpoints for performing matching processing of each three-dimensional point in the three-dimensional point group; and a map of the three-dimensional point according to the selected viewpoint. Ing is carried out for the phase-only correlation method, characterized in that it comprises a three-dimensional coordinate calculation unit for re-calculating the three-dimensional coordinates in the three-dimensional space of each three-dimensional point in the three-dimensional point cloud.

  In the three-dimensional shape model generation device of the present invention, in the three-dimensional coordinate system in which the three-dimensional shape model is formed by the viewpoint selecting unit, a viewpoint direction similar to a normal of the three-dimensional point of the object to be subjected to matching processing. It is characterized by selecting the viewpoint of

  In the three-dimensional shape model generation device of the present invention, when another different three-dimensional point is present between the three-dimensional point of interest and the viewpoint, the viewpoint selecting unit excludes the viewpoint from the viewpoint of selecting the viewpoint. It is characterized by

  In the three-dimensional shape model generation device according to the present invention, the three-dimensional point group generation unit generates three-dimensional points of outliers included in a three-dimensional point group generated from each of the depth maps corresponding to the multi-viewpoint image. , And removing each of three-dimensional points that are not consistent with other depth maps.

  The three-dimensional shape model generation device of the present invention is characterized in that the depth map generation unit generates a depth map corresponding to each of the multi-viewpoint images using a patch match stereo (PatchMatch Stereo) method. Do.

  In the three-dimensional shape model generation method of the present invention, the three-dimensional shape of the object is obtained from two or more multi-viewpoint images obtained by imaging the object from different viewpoints and camera parameters of an imaging device which images the multi-viewpoint image. A three-dimensional shape model generating method for generating a model, wherein a depth map generating unit generates a depth map corresponding to the multi-viewpoint image for each viewpoint from the multi-viewpoint image and the camera parameter. And a three-dimensional point group generation process in which a three-dimensional point group generation unit generates a three-dimensional point group indicating the three-dimensional shape of the object from the depth map and the camera parameters; A viewpoint selection process for selecting two or more viewpoints for performing matching processing of each three-dimensional point in the three-dimensional point group from the three-dimensional point group and the camera parameter; three-dimensional coordinates A three-dimensional coordinate calculation process in which the output unit performs matching of the three-dimensional points according to the selected viewpoint by the phase limited correlation method and recalculates three-dimensional coordinates in three-dimensional space of each three-dimensional point in the three-dimensional point group And including.

  A program according to the present invention generates a three-dimensional shape model of an object from two or more multi-viewpoint images obtained by capturing an object from different viewpoints and camera parameters of an imaging device capturing the multi-viewpoint image. A program for operating a computer as a source shape model generation device, wherein the computer generates a depth map generating a depth map corresponding to the multi-viewpoint image for each viewpoint from the multi-viewpoint image and the camera parameter; Three-dimensional point group generation means for generating a three-dimensional point group indicating the three-dimensional shape of the object from the depth map and the camera parameter, the three-dimensional point group from the three-dimensional point group and the camera parameter Viewpoint selecting means for selecting two or more viewpoints for performing matching processing of each three-dimensional point in Matching three-dimensional point performed by the phase-only correlation method, a program for operating as a three-dimensional coordinate calculation means for re-calculating the three-dimensional coordinates in the three-dimensional space of each three-dimensional point in the three-dimensional point cloud.

  As described above, according to the present invention, it is possible to improve the generation of a three-dimensional shape model from a multi-viewpoint image as well as the processing time and the restoration accuracy as compared with the conventional example. A generation device, a three-dimensional shape model generation method, and a program can be provided.

It is a block diagram showing an example of composition of a three-dimensional shape model generating device by one embodiment of the present invention. FIG. 6 is a diagram showing an example of configuration of a multi-viewpoint image table written and stored in the multi-viewpoint image storage unit 17; It is a figure which shows the structural example of the three-dimensional point-group table memorize | stored in the three-dimensional point-group memory | storage part 18. FIG. FIG. 6 is a diagram showing an example of the configuration of a three-dimensional shape model table stored in the three-dimensional shape model storage unit 19; It is a flowchart which shows the operation example of the process which produces | generates the three-dimensional shape model which the three-dimensional shape model production | generation apparatus 1 by this embodiment performs. It is a figure which shows the image of the three-dimensional-shaped model of this target object produced | generated from the multi-viewpoint image of a target object.

Hereinafter, an embodiment of the present invention will be described with reference to the drawings.
FIG. 1 is a block diagram showing an example of the configuration of a three-dimensional shape model generation apparatus according to an embodiment of the present invention. In FIG. 1, the three-dimensional shape model generation apparatus 1 includes a multi-viewpoint image input unit 11, a depth map generation unit 12, a three-dimensional point group generation unit 13, a viewpoint selection unit 14, a three-dimensional coordinate calculation unit 15, and a three-dimensional shape model. Each of the generation unit 16, the multi-viewpoint image storage unit 17, the three-dimensional point group storage unit 18, and the three-dimensional shape model storage unit 19 is provided.

  The multi-viewpoint image input unit 11 inputs data of multi-viewpoint images captured from a plurality of different viewpoints captured by an imaging device (not shown), adds viewpoint identification information to the multi-viewpoint image, and performs multi-viewpoint image The memory unit 17 is written and stored. In addition, the multi-viewpoint image input unit 11 writes and stores camera parameters of the imaging device that has input the multi-viewpoint image in the multi-viewpoint image storage unit 17. The camera parameters include external parameters and internal parameters, and the external parameters are the center coordinates of the lens in the world coordinate system, the direction of the optical axis of the lens, etc. The internal parameters are the focal length, the image center, the resolution of the image (pixels Number, distortion aberration coefficient, and the like.

FIG. 2 is a view showing a configuration example of a multi-viewpoint image table written and stored in the multi-viewpoint image storage unit 17. As shown in FIG. In FIG. 2, the multi-viewpoint image table has fields of viewpoint coordinates, viewpoint directions, camera parameters, and image indexes in a record corresponding to viewpoint identification information.
Here, the viewpoint identification information is information for identifying a viewpoint at which a multi-viewpoint image is captured. The viewpoint coordinates indicate three-dimensional coordinates of a viewpoint obtained by imaging a multi-viewpoint image in the world coordinate system. The viewpoint direction indicates the direction (viewpoint vector) in three-dimensional coordinates of a viewpoint obtained by imaging a multi-viewpoint image in the world coordinate system. The camera parameters are the above-described external parameters and internal parameters. The image index is an address in the multi-view image storage unit 17 of the area where the data of the multi-view image is written and stored.

Returning to FIG. 1, the depth map generation unit 12 generates a depth map in the world coordinate system corresponding to the multi-viewpoint image of each viewpoint using each of the multi-viewpoint image and the camera parameter. This depth map is a map in which information of depth in three-dimensional coordinates for each pixel of each multi-viewpoint image of each viewpoint is shown.
In the present embodiment, the depth map generation unit 12 uses, for example, a patch match stereo (PatchMatch Stereo) method when generating a depth map. In the patch match stereo method, the depth map and the normal vector map of each pixel of each multi-view image of each view are generated by initializing each with a random number.

  Then, with respect to the depth map and the normal vector map generated by the random number, the depth map generation unit 12 performs spatial propagation of numerical values for other pixels adjacent to each pixel, and numerical space in multi-viewpoint images of different viewpoints. The propagation and fine adjustment of the depth information (depth information) and the normal vector of each pixel are repeatedly performed using the adjustment value obtained by the random number. The depth map generation unit 12 generates a depth map and a normal vector map at which the matching score between the same pixels in the multi-view images of different viewpoints becomes maximum, and the depth map and the normal vector map of the multi-view images at each viewpoint. (See, for example, the description of Japanese Patent Application No. 2016-124820). The depth map generation unit 12 writes and stores each of the generated depth map and normal vector map in the three-dimensional point group storage unit 18.

  Here, as the matching score, a normalized cross correlation in a local area (patch) of an image, an SSD (Sum of Squared Differences) in a local area of the image, or the like is used. The patch match stereo method can generate a depth map equivalent to a full search (for example, plain sweeping which is an example of a multi-view stereo algorithm) with a short processing time compared to a full search, and the initial stage of the object in this embodiment Is a suitable method for estimating the three-dimensional shape of Since it is important to generate a three-dimensional shape in a short time compared to the accuracy of the estimated three-dimensional shape of the object in the process of generating the depth map and normal vector map in the present embodiment, the matching score It is desirable to use a normalized cross-correlation, SSD, which is low in accuracy of matching as compared with the phase-only correlation method described later, but is simple in calculation and can obtain a high-speed matching score.

  FIG. 3 is a view showing a configuration example of the three-dimensional point group table stored in the three-dimensional point group storage unit 18. In FIG. 3A, the three-dimensional point group generation table has fields of depth map index and normal vector map index in the record corresponding to the viewpoint identification information. The viewpoint identification information is information for identifying a viewpoint at which a multi-viewpoint image is captured. The depth map index is an address in the three-dimensional point group storage unit 18 of a region where data of the depth map indicating depth information of each three-dimensional pixel in the multi-viewpoint image is written and stored. The normal vector map index is an address in the three-dimensional point group storage unit 18 of the area in which the data of the normal vector map indicating the normal vector of the three-dimensional point of each pixel in the multi-viewpoint image is written and stored. .

  In FIG. 3B, the three-dimensional point group table has three columns of three-dimensional point group index, normal vector map index, and recalculation point group index in the record corresponding to the three-dimensional point group identification information. doing. The three-dimensional point group index is a three-dimensional point group of a region in which data of a three-dimensional point group indicating the three-dimensional shape of the object is written and stored, in which three-dimensional point groups corresponding to a plurality of viewpoints are combined. This is an address in the storage unit 18. The normal vector index is stored by writing data of normal vectors of three-dimensional points in the three-dimensional point group indicating the three-dimensional shape of the object, in which three-dimensional point groups corresponding to a plurality of viewpoints are combined. Address in the three-dimensional point group storage unit 18 of the area being stored. As the recalculation point group index, data of the three-dimensional point group obtained by re-calculating coordinates of each three-dimensional point of the three-dimensional point group by the three-dimensional coordinate calculation unit 15 described later is written and stored. This is an address in the three-dimensional point group storage unit 18 of the area.

  Returning to FIG. 1, the three-dimensional point group generation unit 13 integrates each of the depth map and the normal vector map in all the viewpoints, and generates a three-dimensional point group corresponding to the three-dimensional shape of the object. Here, the three-dimensional point group generation unit 13 generates the three-dimensional point group by a predetermined calculation using each of the depth map and the camera parameter of the imaging device. That is, the three-dimensional point group generation unit 13 obtains a three-dimensional point group from each of the depth maps of multi-viewpoint images of the respective viewpoints, and sets three-dimensional coordinates of three-dimensional points in this three-dimensional point group as respective camera parameters. Coordinate conversion is performed on the basis of the three-dimensional coordinates in the world coordinate system.

  Thereby, the three-dimensional point group generation unit 13 integrates the three-dimensional point group based on the depth map corresponding to each multi-viewpoint image in the world coordinate system. Then, the three-dimensional point group generation unit 13 adds three-dimensional point group identification information to the combined three-dimensional point group, and writes and stores the three-dimensional point group table of the three-dimensional point group storage unit 18 . Further, the three-dimensional point group generation unit 13 writes the address of the area in which the three-dimensional point group is written together with the three-dimensional point group identification information as a three-dimensional point group index in the three-dimensional point group table .

  Here, when the three-dimensional point group generation unit 13 integrates the depth maps of each of the multi-viewpoint images of a plurality of different viewpoints to generate a three-dimensional point group, different third orders are generated in the same region in different multi-viewpoint images. A plurality of three-dimensional points generated as source points are processed as one three-dimensional point. For example, the three-dimensional point group generation unit 13 integrates a plurality of three-dimensional points in the same area described above using a voxel grid filter. That is, the three-dimensional point group generation unit 13 divides the three-dimensional space in the world coordinate system by a voxel grid of a predetermined size. Then, the three-dimensional point group generation unit 13 extracts the three-dimensional coordinates of the three-dimensional point contained in the interior for each divided voxel, and only the three-dimensional point having the average value of the three-dimensional coordinates of the three-dimensional point in the voxel Leave and remove other 3D points.

  Thereby, a plurality of three-dimensional points generated as different three-dimensional points in the same region in different multi-viewpoint images can be made into one three-dimensional point, and the number of three-dimensional points in the three-dimensional point group Can be reduced. By performing this process, the three-dimensional point group generation unit 13 is not required to correspond to unnecessary three-dimensional points, so each of the three-dimensional points in the three-dimensional point group is performed by the three-dimensional coordinate calculation unit 15 described later. The process of recalculation of three-dimensional coordinates of can be shortened.

  Further, when generating a three-dimensional point group by integrating depth maps generated from multi-viewpoint images of respective viewpoints, the three-dimensional point group generation unit 13 removes outliers from the three-dimensional point in the generated three-dimensional point group Processing may be performed. That is, the three-dimensional point group generation unit 13 is a processing frame such as a circle of a predetermined size centered on the three-dimensional point to be processed, or a voxel (a size larger than the voxel grid unlike voxels of the voxel grid described above) It is determined whether the three-dimensional point which is the target point is to be deleted depending on whether the number of three-dimensional points included in the processing frame is equal to or less than a preset threshold value. At this time, the three-dimensional point group generation unit 13 deletes the three-dimensional point that is the target point when the number of three-dimensional points included in the processing frame is equal to or less than a preset threshold. On the other hand, the three-dimensional point group generation unit 13 does not delete the three-dimensional point which is the target point, when the number of three-dimensional points included in the processing frame exceeds a preset threshold.

  In addition, when the three-dimensional point group generation unit 13 combines the depth maps generated from the multi-viewpoint images of the respective viewpoints to generate the three-dimensional point group, the multiple viewpoints of a plurality of different viewpoints in the combined three-dimensional point group The configuration may be such that three-dimensional points that are not consistent between three-dimensional point groups of each image are deleted. That is, the three-dimensional point group generation unit 13 compares the depth maps of each multi-viewpoint image, and the three-dimensional points which should be at the same coordinate value in each depth map of the multi-view image are different in each depth map It is judged whether or not it has depth information.

  Here, when the three-dimensional point group generation unit 13 determines that three-dimensional points that should be at the same coordinate value have different depth information in each depth map, the three-dimensional point group is generated from the three-dimensional point group. Remove. By performing this process, when the same three-dimensional point has different depth information between multi-view images of different views, the three-dimensional point of the three-dimensional point group in the multi-view image of one view is the other view It is possible to prevent occlusion (invisible) of the appearance of the three-dimensional point of the three-dimensional point group in the multi-viewpoint image.

Also, the three-dimensional point group generation unit 13 estimates normal vectors of three-dimensional points in the three-dimensional point group after integration. Here, for example, for the estimation of the normal vector of each three-dimensional point, the three-dimensional point group generation unit 13 mainly applies to other three-dimensional points in the vicinity of the three-dimensional point for which the normal vector is estimated. Use a method that applies component analysis. That is, the three-dimensional point group generation unit 13 applies a plane to a set of local three-dimensional points (a set of three-dimensional point groups in the vicinity of a target three-dimensional point) by the least squares method. It is regarded as a surface of a three-dimensional shape of an object, and a direction perpendicular to this surface is a normal vector.
Then, the three-dimensional point group generation unit 13 writes and stores the normal vectors estimated for each of the three-dimensional points of the three-dimensional point group as a normal vector map in the three-dimensional point group storage unit 18. The three-dimensional point group generation unit 13 acquires the address of the area where the normal vector map is written, and writes and stores the address as a normal vector map index in the three-dimensional point group table.

The viewpoint selecting unit 14 selects a multi-viewpoint image of two or more plural viewpoints used for matching in order to perform matching for each three-dimensional point in the three-dimensional point group after integration. That is, the viewpoint selecting unit 14 selects the required number of viewpoints in the order of the highest selection cost as a viewpoint to be used for matching, from the viewpoint with the highest selection cost obtained by the following equation (1). “·” In the equation (1) indicates an inner product of vectors. "*" Simply indicates multiplication.
S _ij = (C _j · r _ij ) * (− n _i · r _ij ) (1)

In equation (1), a selection score S _ij is obtained by combining the i-th three-dimensional point in the three-dimensional point group and the j-th viewpoint in a plurality of viewpoints. C _j shows the vector of the optical axis of the j-th viewpoint. n _i indicates the normal vector of the ith three-dimensional point. r _ij indicates a viewpoint vector heading from the j-th viewpoint to the i-th three-dimensional point. The first term "C _j · r _ij " in the above equation (1) has a high cost of the viewpoint corresponding to the multi-view image in which the pixel corresponding to the i-th three-dimensional point to be targeted is in the central vicinity Become. In the second term “n _i · r _ij ” in the equation (1), the cost of the viewpoint in front of the i-th three-dimensional point to be processed becomes higher. Therefore, according to the above equation (1), as a viewpoint suitable for 3D point matching, a viewpoint that is as far forward as possible in front of the 3D point to be matched and a pixel corresponding to the 3D point is a multiview image A viewpoint located at a position closer to the center is selected.

Further, the viewpoint selecting unit 14 may be configured to obtain the selection score S _ij by the following equation (2) instead of the equation (1).
S _ij = V _ij * (C _j · r _ij ) * (− n _i · r _ij ) (2)
In equation (2), the coefficient V _ij is a visibility parameter indicating whether or not the i-th three-dimensional point is visible from the j-th viewpoint. Here, the coefficient V _ij is “1” when the i th three-dimensional point is seen from the j th viewpoint, and “0” when the i th three dimensional point is not visible from the j th viewpoint . That is, the coefficient V _ij determines whether the i-th three-dimensional point can be seen from the j-th viewpoint. At this time, the viewpoint selection unit 14 detects the presence or absence of another three-dimensional point between the j-th viewpoint and the i-th three-dimensional point, whereby the i-th three-dimensional point can be seen from the j-th viewpoint It is judged whether or not it is. Then, the viewpoint selecting unit 14 determines that no other three-dimensional point exists between the j-th viewpoint and the i-th three-dimensional point (for a viewpoint having pixels corresponding to the three-dimensional point in the multi-view image). The coefficient V _{ij is set} to “1”, and the equation (2) is calculated. On the other hand, the viewpoint selecting unit 14 determines that another three-dimensional point exists between the j-th viewpoint and the i-th three-dimensional point (for a viewpoint having no pixel corresponding to the three-dimensional point in the multi-view image). The coefficient V _{ij is set} to “0”, and the equation (2) is calculated.

  The three-dimensional coordinate calculation unit 15 performs three-dimensional coordinate matching for each three-dimensional point in the three-dimensional point group using the multi-viewpoint image of each of the selected viewpoints, and performs re-calculation processing of the three-dimensional coordinate It takes place. In the present embodiment, matching processing of three-dimensional points between multi-viewpoint images is performed using a phase only correlation method. This phase-only correlation method makes it possible to obtain three-dimensional coordinates with higher accuracy, although the processing time is longer compared to matching based on normalized cross-correlation and matching score obtained by SSD (for example, Japanese Patent Application No. 2015-141533 See the description in the In the matching by the phase limited correlation method, the three-dimensional coordinate calculation unit 15 sets the local region in each of the multi-viewpoint images of the selected viewpoint by each of the three-dimensional coordinates and the normal vector at each of the three-dimensional points. Do. Then, the three-dimensional coordinate calculation unit 15 estimates the amount of slight parallel movement of the images of the local region between the multi-viewpoint images by calculating the phase only correlation function from the images of the respective local regions of the multi-viewpoint image. The three-dimensional coordinate calculation unit 15 recalculates three-dimensional coordinates of the three-dimensional point based on the estimated amount of parallel movement.

Also, 3D points with low matching scores in the phase limited correlation function are likely to be inaccurate 3D points called artifact outliers. Therefore, the three-dimensional coordinate calculation unit 15 compares the matching score with a score threshold which is a predetermined threshold for the matching score set in advance, in order to remove a three-dimensional point having a low matching score. Then, the three-dimensional coordinate calculation unit 15 does not delete the three-dimensional point when the matching score is equal to or more than the score threshold. On the other hand, the three-dimensional coordinate calculation unit 15 deletes the three-dimensional point when the matching score is less than the score threshold. This can prevent the occurrence of incorrect three-dimensional points corresponding to the artifact outliers.
The three-dimensional coordinate calculation unit 15 writes a three-dimensional point group based on the re-calculated three-dimensional coordinates into the three-dimensional point group storage unit 18 and stores the three-dimensional point group address using the address of the written area as a re-calculation point group index. Write and store in the point cloud table.

The three-dimensional shape model generation unit 16 generates a three-dimensional shape model (three-dimensional mesh model) using a three-dimensional point group consisting of three-dimensional points obtained by recalculating three-dimensional coordinates. The three-dimensional shape model generation unit 16 generates a three-dimensional shape model from the three-dimensional point group using, for example, a method of mesh reconstruction (Poisson Surface Reconstruction).
Then, the three-dimensional shape model generation unit 16 adds shape model identification information to the generated three-dimensional shape model, and writes data of this three-dimensional shape model in the three-dimensional shape model storage unit 19 for storage. Also, the three-dimensional shape model generation unit 16 uses the address of the area in which the data of the three-dimensional shape model is written in the three-dimensional shape model storage unit 19 as a shape model index. Write to the table for storage.

  FIG. 4 is a diagram showing a configuration example of the three-dimensional shape model table stored in the three-dimensional shape model storage unit 19. In FIG. 4, the three-dimensional shape model table has a field of shape model index in a record corresponding to shape model identification information. The shape model identification information is information for identifying the three-dimensional shape model generated by the three-dimensional shape model generation unit 16. The shape model index indicates the address of the area where the data of the three-dimensional shape model is written in the three-dimensional shape model storage unit 19.

  FIG. 5 is a flowchart showing an operation example of processing for generating a three-dimensional shape model performed by the three-dimensional shape model generation device 1 according to the present embodiment. The flowchart shown below generates a depth map from a plurality of different multi-view images, then generates a three-dimensional point group, and generates a three-dimensional shape model based on the three-dimensional point group. The flow of processing of the model generation device 1 is shown.

Step S1:
The multi-viewpoint image input unit 11 inputs, from an external device (not shown), a plurality of multi-viewpoint images captured by the imaging device from different viewpoints, and assigns viewpoint identification information to each of the viewpoints capturing the multi-viewpoint image. Do.
Then, the multi-viewpoint image input unit 11 writes and stores each of the input multi-viewpoint images in the multi-viewpoint image storage unit 17, and acquires the address of the area where each of the multi-viewpoint images is written. The multi-viewpoint image input unit 11 associates the viewpoint coordinates of the viewpoint, the viewpoint direction of the viewpoint, the camera parameters of the imaging device, and the address of the area where the multi-view image is written in correspondence with the viewpoint identification information added to each of the multiview images. The multi-viewpoint image table of the multi-viewpoint image storage unit 17 is written and stored as an image index.

Step S2:
The depth map generation unit 12 reads out each of the viewpoint coordinates, the viewpoint direction, the camera parameter, and the multi-view image index from the multi-view image table of the multi-view image storage unit 17. Then, the depth map generation unit 12 reads each of the multi-viewpoint images from the multi-viewpoint image storage unit 17 based on the multi-viewpoint image index.
The depth map generation unit 12 generates a depth map and a normal vector map corresponding to each viewpoint using each of the viewpoint coordinates, the viewpoint direction, the camera parameter, and the multi-viewpoint image. The depth map generation unit 12 writes and stores each of the generated depth map and normal vector map in the three-dimensional point group storage unit 18. The depth map generation unit 12 acquires the address of the area in which each of the depth map and the normal vector map is written, and corresponds to the viewpoint identification information as a depth map index and a normal vector map index to obtain a three-dimensional point group storage unit Write and store in 18 three-dimensional point cloud generation tables.

  Step S3: The three-dimensional point group generation unit 13 refers to the three-dimensional point group generation table of the three-dimensional point group storage unit 18 and reads out the depth map index of all the viewpoints. Then, the three-dimensional point group generation unit 13 reads the depth map of each of the viewpoints by each of the depth map indexes. The three-dimensional point group generation unit 13 integrates the depth maps of all the viewpoints to generate a three-dimensional point group indicating the three-dimensional shape of the object, which is composed of three-dimensional points corresponding to each pixel of the multi-viewpoint image. .

Step S4:
The three-dimensional point group generation unit 13 combines the generated three-dimensional point group with noise by selecting a plurality of three-dimensional points regarded as the same three-dimensional point using the voxel grid filter described above. Adjustment processing such as removal of three-dimensional points that are outliers and removal of three-dimensional points that are not consistent with the depth map between multiple viewpoints are performed.
Then, the three-dimensional point group generation unit 13 writes the generated three-dimensional point group in the three-dimensional point group storage unit 18 to be stored, and acquires the address of the written area. The three-dimensional point group generation unit 13 writes and stores the acquired address in the three-dimensional point group table as a three-dimensional point group index.
Also, the three-dimensional point group generation unit 13 estimates normal vectors of each of the three-dimensional points in the created three-dimensional point group, writes the three-dimensional point group in the three-dimensional point group storage unit 18, and stores it. Get the address of the area you wrote. The three-dimensional point group generation unit 13 writes and stores the acquired address in the three-dimensional point group table as a normal vector map index.

Step S5:
The viewpoint selecting unit 14 selects a plurality of, for example, two viewpoints for matching each three-dimensional point in the three-dimensional point group.
At this time, the viewpoint selecting unit 14 is a viewpoint as far in front as possible of the three-dimensional point to be matched, and the pixel corresponding to the three-dimensional point is as central as possible, and the pixel corresponding to this three-dimensional point is present The viewpoint which imaged the multi-viewpoint image to be selected is selected.
That is, the viewpoint selecting unit 14 reads the normal vector of the three-dimensional point to be matched in the three-dimensional point group from the normal vector map of the three-dimensional point group storage unit 18.

Then, the viewpoint selecting unit 14 sequentially selects viewpoints from the multi-viewpoint image table of the multi-viewpoint image storage unit 17 in the order described in the multi-viewpoint image table, and selects each of viewpoint coordinates and viewpoint directions of the selected viewpoints. Read out.
The viewpoint selecting unit 14 calculates a selection cost S _ij for selecting a viewpoint to be used for three-dimensional point matching, for each of all the viewpoints in the multi-viewpoint image table, using Equation (1) or (2). At this time, when the viewpoint selecting unit 14 calculates the selection cost S _ij using the equation (2), a coefficient V indicating whether the three-dimensional point i is visible from the viewpoint j (visible or not visible from the viewpoint) _{Find ij} .
After _obtaining the selection costs S _ij of all the viewpoints, the viewpoint selecting unit 14 selects a predetermined number of viewpoints from among all the viewpoints in the order in which the selection cost S _ij is the largest.

Step S6:
The three-dimensional coordinate calculation unit 15 reads the normal vector of the three-dimensional point to be matched in the three-dimensional point group from the normal vector map of the three-dimensional point group storage unit 18.
Then, the three-dimensional coordinate calculation unit 15 sets, from each of the three-dimensional coordinates of the three-dimensional point and the normal vector, a local region of the same size in each of the multi-viewpoint images in the plurality of selected viewpoints. The three-dimensional coordinate calculation unit 15 calculates the phase-only correlation function from the phase spectrum of the image of each local region of the selected multi-viewpoint image, and determines the peak of the correlation value to obtain the minute between the multi-viewpoint images ( Estimate the amount of translation in sub-pixel units).
Thereby, the three-dimensional coordinate calculation unit 15 recalculates the three-dimensional coordinates of the target three-dimensional point with high accuracy in units of subpixels.

Step S7:
The three-dimensional coordinate calculation unit 15 calculates in advance the correlation value (matching score) obtained by calculating the phase-only correlation function in step S6 from the phase spectrum of the image in each local region of the selected multi-viewpoint image. It is determined whether or not it is equal to or more than the set threshold. At this time, the three-dimensional coordinate calculation unit 15 advances the process to step S8 when the matching score is equal to or more than the threshold. On the other hand, if the matching score is less than the threshold value, the three-dimensional coordinate calculation unit 15 advances the process to step S9.

Step S8:
When the matching score is equal to or more than the threshold value, the three-dimensional coordinates calculation unit 15 temporarily stores the three-dimensional coordinates of the three-dimensional point of the target recalculated in step S6 in the storage unit in itself. Write as a point.

Step S9:
When the matching score is less than the threshold, the three-dimensional coordinate calculation unit 15 removes the three-dimensional point of this object from the three-dimensional point group.

Step S10:
The three-dimensional coordinate calculation unit 15 determines whether recalculation of three-dimensional coordinates of all three-dimensional points in the three-dimensional point group is completed. At this time, when the recalculation of the three-dimensional coordinates of all the three-dimensional points in the three-dimensional point group is completed, the process proceeds to step S11. If the recalculation of the three-dimensional coordinates of all the three-dimensional points in the three-dimensional point group is not completed, the process proceeds to step S6, and the three-dimensional coordinates of the next three-dimensional point in the three-dimensional point group are recalculated. .

Step S11:
The three-dimensional coordinate calculation unit 15 determines that all three-dimensional points in the three-dimensional point group stored in the three-dimensional point group index of the three-dimensional point group storage unit 18 have been recalculated. The recalculation point group stored in the storage unit is written and stored in the three-dimensional point group storage unit 18. Then, the three-dimensional coordinate calculation unit 15 obtains the address of the area where the recalculation point group is written, and writes and stores the address of the recalculation point group index in the three-dimensional point group table of the three-dimensional point group storage unit 18 .

Step S12:
The three-dimensional shape model generation unit 16 refers to the three-dimensional point group table of the three-dimensional point group storage unit 18 and reads the recalculation point group index.
Then, the three-dimensional shape model generation unit 16 reads data of the recalculation point group from the three-dimensional point group storage unit 18 based on the recalculation point group index.
The three-dimensional shape model generation unit 16 generates a three-dimensional shape model as a three-dimensional mesh model using, for example, a method of mesh reconstruction from the read recalculation point group. The three-dimensional shape model generation unit 16 adds shape model identification information to the generated three-dimensional shape model, and writes and stores data of the three-dimensional shape model in the three-dimensional shape model storage unit 19. The three-dimensional shape model generation unit 16 acquires an address at which data of the three-dimensional shape model is written, makes the shape model identification information correspond, and the three-dimensional shape model of the three-dimensional shape model storage unit 19 as a three-dimensional shape model index. Write to the table for storage.

FIG. 6 is a view showing an image of a three-dimensional shape model of the object generated from multi-viewpoint images of the object. Each of FIG. 6A and FIG. 6B is a diagram showing an example of multi-viewpoint images captured from different viewpoints. In the example of FIG. 6, including FIG. 6 (a) and FIG. 6 (b), FIG. 6 (a) and FIG. 6 (a) are used using 60 multi-viewpoint images captured from different viewpoints and camera parameters of the imaging device. A three-dimensional shape model of the object shown in FIG. 6 (b) was generated.
Each of FIG. 6 (c) and FIG. 6 (d) observes a three-dimensional shape model generated by the method described in Non-Patent Document 1 from the respective viewpoints of FIG. 6 (a) and FIG. 6 (b). Shows an image. In the three-dimensional shape of the image of FIGS. 6C and 6D, noise due to an error can be confirmed at the occlusion boundary or at the end of the three-dimensional shape model. The time taken to generate the three-dimensional shape model of the image shown in FIGS. 6 (c) and 6 (d) was about 11,300 seconds.

On the other hand, each of FIG. 6 (e) and FIG. 6 (f) is a three-dimensional shape generated by the method of this embodiment (using equation (2) in the selection of the line of sight used for matching three-dimensional points). The model shows images observed from the respective viewpoints of FIG. 6 (a) and FIG. 6 (b). In the three-dimensional shape of the image in FIG. 6C and FIG. 6D according to this embodiment, noise due to an error at the occlusion boundary or at the end of the three-dimensional shape model is shown in FIG. It can be confirmed that it is reduced compared to). The time taken to generate the three-dimensional shape model of the image shown in FIGS. 6 (e) and 6 (f) was about 600 seconds.
As can be seen from the results shown in FIG. 6, according to the three-dimensional shape model generating device of the present embodiment, the time required to generate a three-dimensional shape model is reduced and the three-dimensional shape of the object is compared to the conventional example. It is possible to generate a highly accurate three-dimensional shape model close to.

  With the above-described configuration, according to the present embodiment, a depth map corresponding to each of a plurality of multi-viewpoint images captured at different viewpoints is used as a matching score, for example, normalized cross correlation or SSD in PatchMatch Stereo method. Since the matching accuracy is lower than that of the phase only correlation method, the time taken to generate the depth map can be shortened. Then, according to the present embodiment, the depth map generated as described above is integrated at all viewpoints to generate a three-dimensional point group, and the phase-only correlation method is applied to each of the three-dimensional points in this three-dimensional point group. In order to recalculate three-dimensional coordinates by using the above method, the number of three-dimensional points to be calculated is reduced as compared with the case of generating a depth map in the phase limited correlation method, and the phase only correlation method is used to generate the depth map. Recalculation of three-dimensional coordinates of each of the three-dimensional point groups in the three-dimensional point group can be performed at high speed with the same accuracy as that used.

  Further, according to the present embodiment, a plurality of three-dimensional points regarded as the same three-dimensional point using the voxel grid filter with respect to the three-dimensional point group in the three-dimensional point group generated by integrating the depth map in all the viewpoints Adjustments such as integration by selection of points, removal of three-dimensional points that are noise outliers, and removal of three-dimensional points that are not consistent with the depth map between multiple viewpoints, etc. Since the three-dimensional points that interfere with the processing are performed before performing the matching processing by the phase-only correlation method, a highly accurate re-computed point group closer to the three-dimensional shape of the object compared to the prior art (recalculated (3D point group) can be generated.

  Further, according to the present embodiment, since the viewpoint with high selection cost obtained from the equations (1) and (2) is used for the matching process of the three-dimensional point in the three-dimensional point group, the three-dimensional point group Since the direction of the normal vector of the point and the viewpoint direction are closer, ie, a viewpoint having a better view of the three-dimensional point is selected, the selection of the viewpoint to be observed as the contour of the three-dimensional shape is reduced. It is possible to reduce noise generated at occlusion boundaries and contours in the shape.

  Further, according to the present embodiment, by using the equation (2) for selecting the viewpoint, it is possible to determine whether the viewpoint is in the visible state without being blocked by another three-dimensional point Since a viewpoint is selected in which the direction of the normal vector of the three-dimensional point in the point group is closer to the viewpoint direction, noise generated at an occlusion boundary in a three-dimensional shape is compared to when using equation (1). It can be reduced.

  A program for realizing the function of the three-dimensional shape model generation apparatus 1 of FIG. 1 in the present invention is recorded in a computer readable recording medium, and the program recorded in the recording medium is read into a computer system. By executing the processing, processing may be performed to generate a three-dimensional shape model of the object from the determined three-dimensional point group. Note that the “computer system” referred to here includes hardware such as an operating system (OS) and peripheral devices. The "computer system" also includes a WWW (World Wide Web) system provided with a homepage providing environment (or display environment). The "computer readable recording medium" is a portable medium such as a flexible disk, a magneto-optical disk, a ROM (Read Only Memory), a CD-ROM (Compact Disc-Read Only Memory), etc., and is incorporated in a computer system. It refers to a storage device such as a hard disk. Furthermore, “computer-readable recording medium” refers to volatile memory (RAM (Random Access (RAM) within a computer system serving as a server or client when a program is transmitted via a network such as the Internet or a communication line such as a telephone line). Memory), etc., includes those which hold a program for a certain period of time.

  The program may be transmitted from a computer system in which the program is stored in a storage device or the like to another computer system via a transmission medium or by transmission waves in the transmission medium. Here, the “transmission medium” for transmitting the program is a medium having a function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line. Further, the program may be for realizing a part of the functions described above. Furthermore, it may be a so-called difference file (difference program) that can realize the above-described functions in combination with a program already recorded in the computer system.

  The embodiment of the present invention has been described in detail with reference to the drawings. However, the specific configuration is not limited to this embodiment, and design changes and the like within the scope of the present invention are also included.

DESCRIPTION OF SYMBOLS 1 ... 3D shape model generation apparatus 11 ... multi-viewpoint image input part 12 ... depth map generation part 13 ... 3D point group generation part 14 ... viewpoint selection part 15 ... 3D coordinate calculation part 16 ... 3D shape model generation part 17 ... Multi-viewpoint image storage unit 18 ... Three-dimensional point group storage unit 19 ... Three-dimensional shape model storage unit

Claims (7)

A three-dimensional shape model generating device for generating a three-dimensional shape model of an object from two or more multi-viewpoint images obtained by imaging an object from different viewpoints and camera parameters of an imaging device capturing the multi-viewpoint image Yes,
A depth map generation unit that generates a depth map corresponding to the multi-viewpoint image for each of the viewpoints from the multi-viewpoint image and the camera parameter;
A three-dimensional point group generation unit that generates a three-dimensional point group indicating the three-dimensional shape of the object from the depth map and the camera parameter;
A viewpoint selection unit that selects two or more viewpoints for performing matching processing of each three-dimensional point in the three-dimensional point group from the three-dimensional point group and the camera parameter;
A three-dimensional coordinate calculation unit which performs matching of the three-dimensional points by the phase-only correlation method based on the selected viewpoint, and recalculates three-dimensional coordinates in three-dimensional space of each three-dimensional point in the three-dimensional point group; Three-dimensional shape model generation device characterized by The viewpoint selection unit
In the three-dimensional coordinate system in which the three-dimensional shape model is formed, the viewpoint selecting unit selects a viewpoint in a viewpoint direction similar to a normal of the three-dimensional point to be subjected to matching processing. Item 3. The three-dimensional shape model generation device according to Item 1. The viewpoint selection unit
3. The three-dimensional shape model generation device according to claim 2, wherein when there is another different three-dimensional point between the three-dimensional point of interest and the view point, the view point is excluded from the view point to be selected. . The three-dimensional point group generation unit
Remove each of the three-dimensional points of outliers included in the three-dimensional point group generated from each of the depth maps corresponding to the multi-viewpoint image and the three-dimensional points which are not consistent with other depth maps The three-dimensional shape model generation device according to any one of claims 1 to 3, characterized in that: The depth map generation unit
The three-dimensional image according to any one of claims 1 to 4, wherein a depth map corresponding to each of the multi-view images is generated using a patch match stereo (PatchMatch Stereo) method. Shape model generator. A three-dimensional shape model generation method for generating a three-dimensional shape model of an object from two or more multi-viewpoint images obtained by capturing an object from different viewpoints and camera parameters of an imaging device capturing the multi-viewpoint image Yes,
A depth map generating step of generating a depth map corresponding to the multi-viewpoint image for each viewpoint from the multi-viewpoint image and the camera parameter;
A three-dimensional point group generation process of generating a three-dimensional point group indicating the three-dimensional shape of the object from the depth map and the camera parameter;
A viewpoint selection process of selecting two or more viewpoints for performing matching processing of each three-dimensional point in the three-dimensional point group from the three-dimensional point group and the camera parameter;
A three-dimensional coordinate calculation unit performs matching of the three-dimensional points by the phase-only correlation method according to the selected viewpoint, and re-calculates three-dimensional coordinates of each three-dimensional point in the three-dimensional point group in three-dimensional space A three-dimensional shape model generating method characterized by including a coordinate calculation process. As a three-dimensional shape model generating device for generating a three-dimensional shape model of an object from two or more multi-viewpoint images obtained by capturing an object from different viewpoints and camera parameters of an imaging device capturing the multi-viewpoint image A program that runs a computer,
The computer,
A depth map generation unit configured to generate a depth map corresponding to the multi-viewpoint image for each of the viewpoints from the multi-viewpoint image and the camera parameter;
Three-dimensional point group generation means for generating a three-dimensional point group indicating the three-dimensional shape of the object from the depth map and the camera parameter;
Viewpoint selecting means for selecting two or more viewpoints for performing matching processing of each three-dimensional point in the three-dimensional point group from the three-dimensional point group and the camera parameter;
In order to perform matching of the three-dimensional point by the phase limited correlation method based on the selected viewpoint and to recalculate three-dimensional coordinates in the three-dimensional space of each three-dimensional point in the three-dimensional point group Programs.