JP2023000111A - Three-dimensional model restoration device, method, and program - Google Patents

Abstract

To provide a three-dimensional model restoration device, a method, and a program that estimate a three-dimensional shape of an object in advance when generating a parallax image of the object by stereo matching, and restrict a pixel for the stereo matching to a vicinity of a two-dimensional projection area of a three-dimensional object shape, and restrict a range of parallax searched for each pixel to a vicinity of a surface of the three-dimensional object shape.SOLUTION: In a three-dimensional model restoration device 1 that restores, on the basis of a plurality of camera images obtained by photographing an object from different viewpoints, a three-dimensional model of the object, a three-dimensional object shape estimation unit 10 estimates, on the basis of the plurality of camera images, the three-dimensional shape of the object. A stereo matching unit 20 generates a depth image by stereo matching using an estimation result of the three-dimensional shape of the object for each camera. An object model generation unit 30 restores the three-dimensional model of the object using the depth image generated for each camera.SELECTED DRAWING: Figure 1

Description

The present invention relates to an apparatus, method, and program for restoring a three-dimensional model of an object captured in camera images, and in particular, using multi-viewpoint images obtained by photographing an object from various directions with a plurality of cameras whose camera parameters have been calibrated. The present invention relates to a three-dimensional model restoration device, method, and program for restoring a three-dimensional model of an object at high speed and with low noise.

Methods for 3D reconstruction from a 2D image of an object can be classified into an active method that reconstructs the object by projecting a laser or light onto the object, and a passive method that reconstructs the image only from an ordinary camera image.

As an active method, Patent Document 1 discloses a method of projecting a dot pattern onto an object, photographing it with a stereo camera, and obtaining parallax by stereo matching to perform three-dimensional restoration. Patent Document 2 discloses a method for three-dimensional reconstruction of an object using a depth sensor.

As a passive method, a method of extracting the silhouette of the object using multi-viewpoint images of the object and determining the three-dimensional shape by the SfS (Shape-from-Silhouette) method is generally known. Patent Document 3 discloses a method of improving SfS so that a highly accurate 3D shape can be obtained even when silhouette extraction is inaccurate, and performing 3D restoration by globally evaluating multiple silhouette images. ing.

As another passive method, a method of generating parallax images by stereo matching between multiple viewpoint images (stereo images) of an object and obtaining a three-dimensional shape by synthesizing the parallax images is also known. Generally speaking, compared to the SfS method, the stereo matching tends to have lower restoration accuracy in areas with few patterns, but it is possible to restore concave structures, and an improvement in restoration accuracy can be expected as the resolution increases. It is also possible to improve stability by taking into account continuity of shape.

Patent Document 4 discloses a three-dimensional reconstruction method using the PatchMatch Stereo method (Non-Patent Document 1), which performs high-speed stereo matching in consideration of the normal direction of the object surface by random search. Patent Literature 5 proposes a technique for shortening the matching time by limiting the search range for matching to within the silhouette area using silhouette information of a subject extracted by another means during stereo matching.

Patent document 6 describes a method of shortening the matching time by using both a stereo camera and a depth sensor and limiting the parallax range searched for each pixel for stereo matching to the vicinity of the depth value obtained by the depth sensor. is proposed.

JP 2020-71034 A Japanese Patent Application Laid-Open No. 2014-67372 JP 2013-25458 A Japanese Patent Application Laid-Open No. 2018-181047 JP-A-2000-331160 JP 2009-47495 A

M.Bleyer, C.Rhemann, C.Rother, "PatchMatch Stereo - Stereo Matching with Slanted Support Windows", The British Machine Vision Conference (BMVC), 2011

However, in any of the above methods, it is difficult to restore a three-dimensional model of an object at high speed and with high accuracy while allowing flexibility in the type and arrangement of cameras. For example, Patent Documents 1 and 2 assume the use of special sensors, and cannot be applied to devices using ordinary RGB cameras.

In the depth sensor used in Patent Document 2, the estimation accuracy may vary depending on the surface material of the object (error generally increases in black areas) and area (error generally increases near edges). The SfS-based method disclosed in Patent Document 3 is based on the premise that the cameras surround the entire circumference of the object at substantially equal intervals, so it lacks flexibility in camera placement and requires a large number of cameras.

In the stereo matching-based method disclosed in Patent Document 4, noise may occur due to processing speed and matching errors. In the stereo-matching-based method utilizing silhouette information disclosed in Patent Document 5, the accuracy of stereo matching depends on the accuracy of silhouette extraction, so when a loss occurs in silhouette extraction, a loss may occur in the 3D model as well.

In the stereo-matching-based method that uses a depth sensor together, which is disclosed in Patent Document 6, the accuracy of stereo matching depends on the accuracy of the depth sensor, so the estimation accuracy may vary depending on the surface material and area of the object.

An object of the present invention is to solve the above technical problems by estimating the three-dimensional shape of an object in advance when generating a parallax image of the object by stereo matching, and performing stereo matching on pixels of the three-dimensional object shape. A three-dimensional model reconstruction apparatus, method, and method that does not require an additional camera other than a stereo camera by limiting the range of parallax to be searched for in each pixel to the vicinity of the surface of the three-dimensional object shape while limiting it to the vicinity of the dimensional projection area. to provide the program.

In order to achieve the above object, the present invention provides a three-dimensional model restoration device for restoring a three-dimensional model of an object based on a plurality of camera images photographing the object from different viewpoints, comprising the following configuration. is characterized by

(1) means for estimating the three-dimensional shape of an object based on a plurality of camera images; and means for generating a depth image by stereo matching using the estimated three-dimensional shape of the object for each camera, A three-dimensional model of an object is reconstructed using depth images generated by each camera.

(2) The means for generating the depth image selects the reference image among the other camera images when the camera image is used as the reference image based on the camera image and the estimation result of the three-dimensional shape of the object for each camera. using the camera image as a reference image for each camera, performing stereo matching with the selected reference image to generate the depth image.

(3) The means for generating the depth image specifies the object region in the stereo matching reference image based on the three-dimensional shape of the object, and limits the stereo matching search range to the object region.

(4) The means for generating the depth image includes means for converting the estimation result of the three-dimensional shape of the object into an object parallax image using the camera parameters of each camera, and the reference image to be focused on in stereo matching. For each pixel, the search range of corresponding pixels in the reference image is limited based on the estimation result of the object parallax image.

(1) Since depth images are generated by stereo matching using the results of estimating the 3D shape of an object for each camera, noise can be detected from multi-viewpoint images captured from various directions using calibrated cameras with flexible camera placement. A 3D model with a small amount of data can be restored at high speed.

(2) Since the reference image is selected based on the 3D shape of the object, it is possible to select images with similar appearances as a stereo pair at a low computational cost, improving the 3D reconstruction accuracy. Become.

(3) Since the pixel search range in stereo matching is selected based on the three-dimensional shape of the object, it is possible to suppress accuracy deterioration due to erroneously excluding a necessary region from the matching range.

(4) Since the pixel search range in stereo matching can be limited to the vicinity of the surface of the object, quality improvement and processing load reduction can be achieved.

1 is a functional block diagram of a 3D model restoration device according to a first embodiment of the present invention; FIG. FIG. 2 schematically shows a first method for estimating the three-dimensional shape of an object; FIG. 4 is a diagram schematically showing a second method of estimating the three-dimensional shape of an object; FIG. 10 is a diagram schematically showing a third method of estimating the three-dimensional shape of an object; FIG. 4 is a diagram showing a method of estimating three-dimensional shapes of multiple objects with two cameras; FIG. 4 is a diagram showing a method of estimating three-dimensional shapes of multiple objects using three cameras; FIG. 4 is a diagram showing an example of multi-viewpoint stereo matching; FIG. 4 is a diagram schematically showing a method of selecting a reference image based on the degree of difference in joint point coordinates of three-dimensional object shapes for each camera; FIG. 10 is a diagram showing an example of performing window matching within a parallax range corresponding to the size of a voxel space to be restored for all pixels of a reference image; FIG. 10 is a diagram showing an example of reducing the processing load of matching by omitting matching for windows in which no object is shown;

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. FIG. 1 is a functional block diagram showing the configuration of a three-dimensional model restoration device 1 according to the first embodiment of the present invention, which mainly comprises a three-dimensional object shape estimation unit 10, a stereo matching unit 20 and an object model generation unit 30. configuration.

Such a three-dimensional model restoration device 1 can be configured by installing an application (program) that realizes each function in at least one general-purpose computer or server. Alternatively, a part of the application can be configured as a dedicated machine or a single-function machine that is made into hardware or software.

The three-dimensional object shape estimation unit 10 receives a plurality of camera images with different viewpoints (multi-viewpoint images) as input, roughly estimates the three-dimensional shape of the object included in each camera image, and sends the estimation result to the stereo matching unit 20. Output. In this embodiment, a shape such as a solid model, a mesh (surface) model, or a wireframe model is estimated as a three-dimensional model in a three-dimensional space containing an object.

The three-dimensional object shape estimation unit 10 can estimate any three-dimensional shape model. A certain case will be described as an example.

In addition, the present invention can be applied to model restoration of any object (human body, partial regions such as a human face and upper body, animals such as dogs and cats, artificial objects such as cars and furniture, etc.). The restoration of a person's shape will be described as an example.

The three-dimensional object shape estimation unit 10 includes a position/orientation estimation unit 101 and a three-dimensional shape estimation unit 102, and roughly estimates the three-dimensional shape of an object captured in a camera image. The position and orientation estimation unit 101 estimates the position and orientation of an object based on the camera image for each camera. A three-dimensional shape estimation unit 102 estimates only one three-dimensional shape of an object based on the position and orientation estimated for each camera. As a three-dimensional shape estimation method, for example, the following three types of methods can be adopted.

(1) First Estimation Method As shown in FIG. 2, first, a rectangular area containing an object is detected from each camera image, and two-dimensional coordinates of the central position of the rectangular area are obtained. The two-dimensional coordinates of each camera image are then backprojected to three-dimensional coordinates by triangulation to obtain the three-dimensional position of the object center. Triangulation can be performed from pairs of 2D coordinates, so if there are three or more camera images, the 3D position is calculated for each combination of pairs, and the average value of multiple 3D positions is used to calculate the 3D It may represent the position. Finally, an object approximation three-dimensional model designed in advance so as to include the object shape around the three-dimensional position of the object center is placed as the three-dimensional object shape.

Rectangular regions of objects can be detected by arbitrary algorithms such as R-CNN, YOLO, SSD, etc., which are commonly used in the field of computer vision. A prismatic model, a cylindrical model, or the like can be used as the object approximation three-dimensional model.

(2) Second method As shown in Fig. 3, first, the positions of joint points of a predefined object are estimated from each camera image (posture estimation), and each joint point is reversed to three-dimensional coordinates by triangulation. Project. Next, an approximate three-dimensional model of each partial area of the object (designed in advance to include the partial area of the object) is placed around the three-dimensional coordinates of each joint point, and this is taken as a three-dimensional object shape.

The pose of an object can be estimated by any algorithm such as OpenPose, which is commonly used in the field of computer vision. A prismatic model, a cylindrical model, or the like can be used as the approximate three-dimensional model of each partial region of the object.

(3) Third method As shown in FIG. 4, first using a parametric model of the object, using a classifier trained to estimate the parameters of the parametric model from the image for each camera, The 3D pose and 3D object shape of the object are simultaneously estimated for each camera image. Then, the three-dimensional posture and the three-dimensional object shape estimated for each camera image are averaged or otherwise integrated into one three-dimensional posture and three-dimensional object shape, which is output. Alternatively, a classifier trained to estimate parameters of a parametric model from multiple camera images may be used to simultaneously estimate the 3D pose and 3D object shape of the object and output them.

Here, the parametric model of an object is a mesh model that can express a three-dimensional shape model of an object using the three-dimensional posture of the object as a parameter. model) is commonly used. SMPL is capable of controlling a 3D human shape model represented by 6890 vertices with a 72-dimensional posture parameter θ and a 10-dimensional body shape parameter β. For example, by a method such as SPIN (SMPL oOptimization IN the loop), it is possible to estimate θ and β using an image as an input, and obtain a three-dimensional human shape from the θ and β.

In the above explanation, it is assumed that only one object is captured in each camera image, but as shown in FIG. 5, it can be similarly applied to the case where a plurality of objects are captured. However, when there are only two cameras, if the same object is not identified (associated) between images, triangulation points (object center in the first method, each joint in the second method) points) pairs cannot be identified.

Here, it is possible to specify pairs of points for which triangulation is performed by associating the same object between each camera image based on a measure such as the degree of similarity of image features. In the case of a system configuration with three or more cameras, triangulation may be performed by associating identical objects in the same manner as in the case of two cameras. By using triangulation using information from three cameras, it is possible to specify the 3D position of the object center and each joint point only by geometric processing without identifying the same object between images. You can do it.

The stereo matching unit 20 performs stereo matching between camera images using N camera images from different viewpoints and the estimation result of the three-dimensional object shape (if there are M objects in the shooting scene, a total of M objects). generates N parallax images corresponding to each camera image, and further generates a depth image using the parallax images. The generated N depth images are output to the object model generation unit 30 .

Stereo matching processing generally involves selecting an image for which a parallax image is to be obtained (reference image) and a reference image to be used for matching as a stereo pair, and performing image parallelization (Stereo Rectification) using camera parameters and window matching. to generate parallax images.

One or a plurality of reference images (multi-viewpoint stereo matching) may be used for the reference image. When there are two reference images, as shown in FIG. 7, the standard image can generate a more accurate parallax image by performing window matching on the two reference images.

It is desirable that the reference image is selected from other camera images (candidate images) for each camera image in descending order of overlap between the reference image and the shooting area. As a simple method, the angle difference Δθ between the shooting direction rb of the reference image and the shooting direction rc of each candidate image is calculated by the following equation (1), and the candidate image with the smaller angle difference Δθ is preferentially selected as the reference image. There is a way.

Also, feature point matching may be performed between the reference image and each candidate image, and the candidate image from which more corresponding points are obtained may be preferentially selected as the reference image.

In this embodiment, the stereo matching unit 20 includes a reference image selection unit 201, which selects a reference image from other camera images based on the camera image and the estimation result of the three-dimensional object shape for each camera.

The reference image selection unit 201 converts each vertex V (=[[x1, y1, z1], [x2, y2, z2], ...]) of the estimated three-dimensional object shape for each camera into the camera parameter of the camera. is used to two-dimensionally project onto the camera image to obtain a two-dimensional projection point P (=[[u1, v1], [u2, v2]...]). Then, for each camera, the difference between the two-dimensional projection point Pb of the three-dimensional object shape in the camera image (reference image) and the two-dimensional projection point Pc of the three-dimensional object shape in the camera image (candidate image) of another camera A degree (eg Procrustes distance) d ^p (Pb, Pc) is calculated, and at least one candidate image with a smaller dissimilarity is selected as a reference image.

As described above, in this embodiment, the stereo matching unit 20 includes the reference image selection unit 201, and in consideration of the estimation result of the three-dimensional object shape for each camera, a reference image is selected when each camera image is used as a reference image. Since the images are selected, it is possible to select as a stereo pair images that are close in actual appearance, that is, images having a larger overlapping area, and as a result, it is possible to improve the three-dimensional reconstruction accuracy.

Note that, in the present invention, the method of selecting a reference image in consideration of the three-dimensional object shape for each camera is not limited to the above method. For example, as shown in FIG. 8, each vertex V of the three-dimensional object shape estimated by the three-dimensional object shape estimation unit 10 is two-dimensionally projected onto the camera image of the camera using the camera parameters for each camera. It can be obtained as a two-dimensional coordinate group P' (=[[u'1, v'1], [u'2, v'2], ….]) of each joint point, which is an index for posture estimation. .

Then, for each camera, the degree of difference d ^p (P'b, P'c) between the orientation P'b of the object in the camera image and the orientation P'c of the object in the other candidate images is calculated. A small candidate image may be preferentially selected as a reference image for each camera. In the example of FIG. 8, since d ^p (P ₁ , P ₂ )<d ^p (P ₁ , P ₃ ), the candidate image cam2 is selected as the reference image. This makes it possible to select images that are similar in appearance as a stereo pair with less computational cost.

At this time, a predetermined threshold (first threshold) is set for each dissimilarity d ^p , all candidate images with dissimilarity d ^p less than the first threshold are selected as reference images, and multi-viewpoint stereo matching is performed. You can do it. In general, noise may increase when stereo pairs with small overlapping regions are included. Since the number of reference images for an image group can be automatically adjusted, it is possible to perform three-dimensional reconstruction with less noise.

On the other hand, if the baseline length (inter-camera distance) is extremely short with respect to the depth (shooting distance), the accuracy of parallax estimation may deteriorate. In general, there is a positive correlation between the baseline length and each of the dissimilarities d ^p , and the shorter the baseline length, the smaller the dissimilarity d ^p tends to be. Therefore, in the present embodiment, a second threshold lower than the first threshold is set as a threshold for excluding a camera whose baseline length is so short as to degrade the accuracy of parallax estimation, and the degree of dissimilarity d ^p is the second threshold. A candidate image with a smaller degree of dissimilarity may be preferentially selected as a reference image from among the candidate images exceeding the threshold.

Alternatively, all candidate images whose dissimilarity d ^p exceeds the second threshold and is lower than the first threshold may be selected as reference images to perform multi-viewpoint stereo matching. As a result, while avoiding the problem of degraded parallax estimation accuracy due to insufficient baseline length, it is possible to select images that are close to the actual appearance as a stereo pair, resulting in improved 3D reconstruction accuracy. becomes possible.

When a reference image is selected in each of the processes described above, the stereo matching unit 20 performs window matching between the reference image and each reference image for each camera. In window matching, as shown in Fig. 9, for all pixels of the reference image, window regions are matched with the reference image within the range of parallax according to the size of the voxel space to be restored. Parallax is calculated from the distance. Note that SSD and NCC are generally used as matching functions.

In this embodiment, in order to reduce the processing load of stereo matching, the stereo matching unit 20 is provided with a search range limiting unit 202, and as shown in FIG. The processing load of matching is reduced by omitting matching of windows that are not

The search range limiting unit 202 two-dimensionally projects the estimation result of the three-dimensional object shape common to each camera onto the camera image (reference image) using the camera parameters for each camera, thereby limiting the object region in the reference image. If the center coordinates of the window are outside the object area, the processing load is reduced by skipping the matching process.

Although the process itself for reducing the matching range is disclosed in Patent Document 5 and the like, this embodiment is characterized by using a two-dimensional projection image of the three-dimensional object shape as reference information for determining the search range for matching.

It is generally difficult to estimate the object region (silhouette) robustly against noise without prior knowledge of the general shape of the object to be reconstructed. However, in this embodiment, since the object area is identified using the two-dimensional projection image of the three-dimensional object shape, it is possible to suppress accuracy deterioration due to erroneously excluding a necessary area from the matching range.

However, since the object region does not necessarily include the object in the reference image, it cannot be denied that the restoration accuracy may be degraded by the above processing. Therefore, in this embodiment, the object region is converted into a binary image (with 1 inside the region and 0 outside the region), and the binary image is subjected to erosion filter processing to convert the region of the two-dimensional projection image into Expand. Defects of the object are suppressed by determining whether or not to skip the matching process based on the region after the dilation.

Note that the processing load of stereo matching increases in proportion to the depth width (parallax range) of the voxel space. Furthermore, the wider the range of parallax, the higher the risk of matching error, and the problem of deteriorating the reconstruction accuracy of the three-dimensional model. Therefore, in the present embodiment, the search range limiting unit 202 limits the parallax range searched in window matching based on the information of the estimated three-dimensional object shape, thereby reducing the processing load and improving accuracy at the same time. .

In this embodiment, the estimation result of the three-dimensional object shape for each camera is converted into a parallax image (object parallax image) of a reference image using the camera parameters of the camera, and each pixel of the reference image is converted to each pixel of the reference image. When window matching is performed, the parallax search range is limited within a certain threshold range based on the parallax value of the pixel in the object parallax image.

Here, the "object parallax image" means a parallax image that can be easily estimated based on prior knowledge of the object shape without performing stereo matching processing, which is generally considered to have a high processing load. It is different from the parallax image (output) that the unit 20 outputs as a result of window matching processing.

Therefore, the object parallax image is only positioned as an auxiliary input (input) for the stereo matching unit 20 to perform window matching processing at high speed and with low noise, and generally the output parallax image is the input object parallax image. It is expected to be stricter and more accurate (lower noise) than

Then, for example, if the parallax in the object parallax image of the pixel is d _o , a pixel range of ± _{αd 0} (α is a predetermined coefficient) to the left and right around the pixel shifted by d 0 along the epipolar line from the pixel. Window matching is performed by narrowing the search range to only As a result, the parallax search range can be narrowed, and quality improvement and processing load reduction can be achieved.

The process itself for reducing the parallax search range is disclosed in Patent Document 6 and the like, but in this embodiment, as reference information for limiting the search range, an object parallax image generated based on the estimation result of the three-dimensional object shape is used. They differ in terms of use. It is generally difficult to estimate a depth map robustly against noise without prior knowledge of the general shape of the object to be reconstructed. It is possible to suppress the deterioration of accuracy caused by doing so.

However, since the area occupied by the three-dimensional object shape in the object parallax image does not necessarily include the object in the reference image, the above processing may degrade the restoration accuracy. Therefore, in the present embodiment, when window matching is performed for each pixel of the reference image, the parallax search range is limited within a certain threshold range based on the parallax value of the region near the pixel in the object parallax image.

Specifically, the minimum and maximum values of effective parallax values (parallax of pixels outside the range of the three-dimensional object shape are set to invalid values) existing in the neighboring region are calculated, and (minimum value - threshold value) (maximum value+threshold value) to suppress object loss by limiting the parallax search range.

More specifically, for example, a 5×5 pixel range (different from the window size) centered on the pixel of interest in the reference image is defined as the neighboring region, and the threshold for the parallax search range is set to 5. If the object If the maximum value (parallax) corresponding to all 25 pixels of the parallax image is "10" and the minimum value is "5", the minimum search value is 0 (5-5) and the maximum value is 15 (10+ 5), window matching is performed for pixels with a parallax in the range of 0 to 15 on the epipolar line of the reference image.

The transformation unit 203 finally transforms the parallax image into the depth image by transforming the parallax d into the depth Z by the following equation (2). Here, B is the distance between focal points in the collimated image, and f is the focal length.

Stereo matching 20 performs each of the above processes on N camera images, and finally outputs N depth images to object model generation unit 30 .

The object model generation unit 30 generates a three-dimensional shape model of the object using the N depth images output by the stereo matching unit 20, synthesizes the camera image as a texture, and outputs the three-dimensional model restoration device 1. do. There are no restrictions on the method of generating a three-dimensional shape model from multiple depth images, and any method can be adopted.

For example, TSDF (Truncated Signed Distance Functions) can be used as a technique for generating single voxel data from N depth images. In TSDF, each depth image is back-projected to voxel space and synthesized into single voxel data by weighted averaging in voxel space.

Also, a marching cube method can be used as a technique for generating a three-dimensional shape model (polygon data) from voxel data. In the marching cube method, a cube with 8 adjacent voxels as vertices is treated as a unit, and voxel data is converted to cubic by repeating the process of converting into 15 patterns of polygons predefined according to the voxel values of the 8 vertices. It can be converted to the original shape model.

In addition, according to the above-described embodiments, it is possible to accurately restore a three-dimensional model of an object captured by multiple cameras, so that various services and entertainment can be provided to many people overcoming geographical or economic disparities. be able to provide. As a result, the UN-led Sustainable Development Goals (SDGs) include Goal 9 “Build resilient infrastructure and promote inclusive and sustainable industrialization” and Goal 11 “Make cities inclusive, safe and resilient.” and make it sustainable.

10... Three-dimensional object shape estimation unit 20... Stereo matching unit 201... Reference image selection unit 202... Search range limitation unit 203... Conversion unit 30... Object model generation unit

Claims (15)

A three-dimensional model restoration device for restoring a three-dimensional model of an object based on a plurality of camera images taken from different viewpoints of the object,
means for estimating a three-dimensional shape of an object based on multiple camera images;
means for generating a depth image by stereo matching using the estimation result of the three-dimensional shape of the object for each camera;
A three-dimensional model restoration device that restores a three-dimensional model of an object using a depth image generated for each camera. The means for generating the depth image selects a reference image from other camera images when the camera image is used as a reference image based on the camera image and the estimation result of the three-dimensional shape of the object for each camera. have the means to
2. The three-dimensional model restoration apparatus according to claim 1, wherein a depth image is generated by performing stereo matching with the selected reference image using the camera image as a reference image for each camera. means for generating the depth image,
means for two-dimensionally projecting each vertex of the estimated three-dimensional shape of the object onto the camera image using the camera parameters for each camera;
With the camera image of one camera as the reference image and the camera images of the other cameras as candidate images, the degree of difference between the two-dimensional projection point of each vertex in the reference image and the two-dimensional projection point of each vertex in each candidate image and means for calculating
3. The three-dimensional model reconstruction apparatus according to claim 2, wherein at least one candidate image having the smaller degree of difference is selected as a reference image. means for generating the depth image,
means for two-dimensionally projecting each joint point of the estimated three-dimensional shape of the object onto the camera image using the camera parameters for each camera;
Using the camera image of one camera as a reference image and the camera images of the other cameras as candidate images, the distance between the two-dimensional projection point of each joint point in the reference image and the two-dimensional projection point of each joint point in each candidate image is and means for calculating the degree of dissimilarity,
3. The three-dimensional model reconstruction apparatus according to claim 2, wherein at least one candidate image having the smaller degree of difference is selected as a reference image. 5. The 3D model reconstruction apparatus according to claim 3, wherein the means for generating the depth image selects all the candidate images whose dissimilarity is less than a first threshold as reference images. 3. The means for generating the depth image selects, as the reference images, all images whose dissimilarity exceeds a second threshold smaller than the first threshold and falls below the first threshold. 6. The three-dimensional model restoration device according to 5. 4. The means for generating the depth image selects at least one candidate image having a relatively smaller dissimilarity from among the reference images whose dissimilarity exceeds a second threshold as the reference image. 5. The three-dimensional model restoration device according to 4. 8. The three-dimensional model reconstruction apparatus according to claim 3, wherein the means for generating the depth image uses a Procrustes distance as the dissimilarity. 9. The method according to any one of claims 1 to 8, wherein the means for generating the depth image specifies an object region in the stereo matching reference image based on the three-dimensional shape, and limits a stereo matching search range to the object region. 3D model reconstruction device according to any one of the above. 10. The three-dimensional image according to claim 9, wherein the means for generating the depth image identifies the object area by two-dimensionally projecting the three-dimensional shape onto the reference image using the camera parameters of each camera. Model restoration device. The means for generating the depth image comprises means for converting the estimation result of the three-dimensional shape into an object parallax image using the camera parameters of each camera,
9. The method according to any one of claims 1 to 8, wherein for each pixel of the reference image to be focused on in stereo matching, a search range of corresponding pixels in the reference image is limited based on the estimation result of the object parallax image. 3D model reconstruction device. The means for generating the depth image searches the search range of the corresponding pixel in the reference image for each pixel of the reference image of interest in stereo matching according to the parallax of the pixel of the object parallax image corresponding to the pixel of interest. 12. The three-dimensional model reconstruction device according to claim 11, wherein the range is restricted. The means for generating the depth image converts a search range of corresponding pixels in the reference image for each pixel of the reference image of interest in stereo matching to the object parallax image corresponding to a predetermined pixel range including the pixel of interest. 12. The three-dimensional model restoration apparatus according to claim 11, wherein the search range is limited according to the parallax range of each pixel within the pixel range. In a three-dimensional model restoration method in which a computer restores a three-dimensional model of an object based on a plurality of camera images photographing the object from different viewpoints,
estimating the three-dimensional shape of an object based on multiple camera images;
generating a depth image by stereo matching using the estimation result of the three-dimensional shape of the object for each camera;
A three-dimensional model restoration method comprising restoring a three-dimensional model of an object using depth images generated for each camera. In a three-dimensional model restoration program for restoring a three-dimensional model of an object based on multiple camera images of the object taken from different viewpoints,
estimating a three-dimensional shape of an object based on multiple camera images;
a procedure for generating a depth image by stereo matching using the results of estimating the three-dimensional shape of the object for each camera;
A three-dimensional model restoration program for causing a computer to execute a procedure for restoring a three-dimensional model of an object using depth images generated for each camera.

Images