JP2021015559A - Three-dimensional shape model generation device, three-dimensional shape model generation method, and program - Google Patents

Abstract

To provide a three-dimensional shape model generation device, a three-dimensional shape model generation method, and a program, that can highly-accurately estimate an actual dimension of a three-dimensional shape created by a three-dimensional restoring method using a multi-viewpoint image.SOLUTION: A device comprises: a focus area detection section for detecting an area came into focus in each image for a plurality of multi-viewpoint images in which an object is photographed from different viewpoints one another; a distance calculation section in which, as to pixels of the area came into focus in each image of the multi-viewpoint images, calculating a virtual distance representing a virtual distance between a projected image in which a three-dimensional shape model of the object is projected onto a two-dimensional plane, and a viewpoint position in the projected image; a scale estimation section for estimating a scale value representing a ratio between the projected image against the virtual distance, and an actual real distance to the viewpoint position in the projected image; and a scale conversion section for converting the virtual distance to the real distance using the scale value.SELECTED DRAWING: Figure 1

Description

The present invention relates to a three-dimensional shape model generation device, a three-dimensional shape model generation method, and a program.

Conventionally, there is a three-dimensional restoration method for generating a three-dimensional shape model of an object by using a plurality of images (multi-view images) obtained by capturing the object from different viewpoints. In this method, the appearance of the object differs for each multi-viewpoint image, so the three-dimensional shape of the object is created (restored) by calculating the depth value of each pixel in the image using the principle of a stereo camera. be able to. In the three-dimensional restoration method, a three-dimensional shape can be created, but the actual size (scale) of the object cannot be obtained. This is because the actual size of the object cannot be obtained only by capturing the object in the image.

One of the methods for estimating the scale of an object from an image is to use a marker (for example, Patent Document 1). Patent Document 1 discloses a technique for estimating the actual size of an object by using an image obtained by capturing an image of a marker whose actual size (actual size) is known together with the object.

There is also a method of estimating the scale of an object using the depth of field of an image (for example, Patent Document 2). Depth of field is the range of the actual distance from the camera that appears to be in focus to the object. In Patent Document 2, a plurality of images having different in-focus positions are acquired by using the Depth from defocus method, and the correlation value of each other's focus in the acquired plurality of images is calculated to obtain an object from the camera. The technique of calculating the actual distance is disclosed.

JP-A-2018-57532 Japanese Patent No. 5923476

However, if the marker and the object are simultaneously captured and a plurality of images are taken from different viewpoints, the object may be hidden behind the marker. If the object is hidden behind the marker, it becomes difficult to accurately create the three-dimensional shape of that part.
In addition, the depth of field varies. Therefore, the distance estimated from the depth of field contains an error, and there is a problem that the distance cannot be estimated accurately.

The present invention has been made in view of such a situation, and a three-dimensional shape model generation capable of accurately estimating the actual dimensions of the three-dimensional shape created by the three-dimensional restoration method using a multi-viewpoint image. A device, a three-dimensional shape model generation method, and a program are provided.

The three-dimensional shape model generation device of the present invention includes a three-dimensional shape generator that generates a three-dimensional shape model of the object from a plurality of multi-viewpoint images obtained by capturing the object from different viewpoints, and the multi-viewpoint image. A projection in which the three-dimensional shape model is projected onto a two-dimensional plane with respect to a focus area detection unit that detects an in-focus area in each image and pixels in an in-focus area in each image of the multi-viewpoint image. A distance calculation unit that calculates a virtual distance that is a virtual distance between the image and the viewpoint position in the projected image, and the actual actual distance between the projected image and the viewpoint position in the projected image with respect to the virtual distance. It is characterized by including a scale estimation unit that estimates a scale value indicating the ratio of, and a scale conversion unit that converts the virtual distance into the real distance using the scale value.

In the three-dimensional shape model generation device of the present invention, the focus region detection unit responds to each image of the multi-viewpoint image according to the region focused by using the focus function of the camera that captured the multi-viewpoint image. A focused region in each image of the multi-viewpoint image is detected.

In the three-dimensional shape model generation device of the present invention, the focus area detection unit detects an in-focus area in each image of the multi-viewpoint image by image processing.

In the three-dimensional shape model generation device of the present invention, the focus area detection unit uses a trained model generated by performing machine learning using a training data set in which inputs and outputs are associated with each other. The in-focus area in each image of the multi-viewpoint image is detected, the input of the learning data set is an input image for learning, and the output of the learning data set is the in-focus area of the input image. This is information indicating the area.

In the three-dimensional shape model generator of the present invention, the input of the training data set is each image of the multi-viewpoint image, and the output of the training data set is the three-dimensional shape of the marker whose actual dimensions are known. The correction was determined based on the depth value of each pixel in the corrected projected image, in which the model was scale-corrected based on the actual dimensions of the marker, and the corrected three-dimensional shape model was projected onto a two-dimensional plane. This is information indicating an in-focus area in the completed projected image.

In the three-dimensional shape model generation device of the present invention, the distance calculation unit determines whether or not the pixel determined to be in focus by the focus region detection unit is an edge, and the focus region detection unit. Of the determination results of whether or not the corresponding pixel of the projected image obtained by projecting the three-dimensional shape model onto the two-dimensional plane is an edge, which is a corresponding pixel corresponding to the pixel determined to be in focus. Based on the determination result of at least one of them, it is determined whether or not the virtual distance in the corresponding pixel of the projected image is used for calculating the virtual distance in the projected image.

In the three-dimensional shape model generation device of the present invention, the scale estimation unit derives the actual distance based on the depth of field calculated from the camera parameters of the camera that captured the multi-viewpoint image.

In the three-dimensional shape model generator of the present invention, the actual distance is derived based on the corrected three-dimensional shape model in which the three-dimensional shape model of the marker whose actual dimensions are known is scale-corrected based on the actual dimensions. A marker scale estimation unit is further provided, and the scale estimation unit estimates the scale value using the actual distance derived by the marker scale estimation unit.

In the three-dimensional shape model generation device of the present invention, the marker scale estimation unit projects the corrected three-dimensional shape model onto a two-dimensional plane based on the corrected three-dimensional shape model, and the corrected projected image. The actual distance is derived based on the depth value of the pixel determined to be in focus among the pixels in.

In the three-dimensional shape model generation device of the present invention, the scale estimation unit derives the actual distance based on the distance obtained from the focus function of the camera that has captured the multi-viewpoint image.

In the three-dimensional shape model generation device of the present invention, the distance calculation unit makes the distance corresponding to the depth value of the pixels included in the detected in-focus region correspond to each corresponding pixel of the projected image. As a result, the virtual distance for each pixel of the projected image is calculated.

In the three-dimensional shape model generation device of the present invention, the focus area detection unit detects the degree of focus for each pixel, and the distance calculation unit detects pixels included in the detected in-focus area. By weighting the distance according to the depth value of the above according to the degree of focus and making the weighted distance correspond to each corresponding pixel of the projected image, the virtual distance for each pixel of the projected image can be obtained. calculate.

In the three-dimensional shape model generation device of the present invention, the distance calculation unit weights the distance according to the depth value of the pixels included in the detected region in focus according to the magnitude of the distance. Then, by associating the weighted distance with each corresponding pixel of the projected image, the virtual distance for each pixel of the projected image is calculated.

In the three-dimensional shape model generation device of the present invention, the distance calculation unit calculates a distance according to the depth value of each pixel in the focused region in each image of the multi-viewpoint image, and the calculated distance is projected. When there are a plurality of distances corresponding to the pixels of the image and corresponding to the pixels of the projected image, the plurality of distances are compared, and the virtual in the pixels of the projected image is compared according to the degree of variation of the plurality of distances. It is determined whether or not the distance is used for calculating the virtual distance in the projected image.

In the three-dimensional shape model generation method of the present invention, the three-dimensional shape generation unit generates a three-dimensional shape model of the object from a plurality of multi-viewpoint images obtained by capturing the object from different viewpoints. The process, the focus area detection step in which the focus area detection unit detects the in-focus area in each image of the multi-viewpoint image, and the distance calculation unit in the focus area in each image of the multi-viewpoint image. A distance calculation process for calculating a virtual distance, which is a virtual distance between a projected image obtained by projecting the three-dimensional shape model onto a two-dimensional plane and a viewpoint position in the projected image, and a scale estimation unit A scale estimation step for estimating the ratio of the projected image to the actual actual distance to the viewpoint position in the projected image with respect to the virtual distance, and a scale conversion unit using the scale value. It is characterized by including a scale conversion step of converting the virtual distance into the real distance.

The program of the present invention is a three-dimensional shape generating means for generating a three-dimensional shape model of the object from a plurality of multi-view images obtained by capturing an object from different viewpoints, and each image of the multi-view image. A focus area detecting means for detecting an in-focus area in the above, a projected image obtained by projecting the three-dimensional shape model onto a two-dimensional plane with respect to pixels in the in-focus area in each image of the multi-viewpoint image, and the above. The ratio of the distance calculation means for calculating the virtual distance, which is a virtual distance, to the viewpoint position in the projected image, and the ratio of the projected image to the actual actual distance to the viewpoint position in the projected image with respect to the virtual distance. It is a program for operating as a scale estimation means for estimating the indicated scale value, and a scale conversion means for converting the virtual distance into the real distance by using the scale value.

According to the present invention, the actual dimensions of the three-dimensional shape created by the three-dimensional restoration method using the multi-viewpoint image can be estimated with high accuracy.

It is a block diagram which shows the example of the structure of the 3D shape model generation apparatus 1 which concerns on 1st Embodiment. It is a figure which shows the example of the structure of the information stored in the scale information storage unit 109 which concerns on 1st Embodiment. It is a figure which shows the example of the plurality of multi-viewpoint image TG which concerns on 1st Embodiment. It is a figure which shows the example of the 3D shape model M which concerns on 1st Embodiment. It is a figure which shows the example of the multi-viewpoint image TG which concerns on 1st Embodiment. It is a figure which shows the example of the blur map BM of the multi-viewpoint image TG which concerns on 1st Embodiment. It is a figure which shows the example of the function of the depth of field which concerns on 1st Embodiment. This is an example of the distribution of the virtual distance in the pixels of the projected image of the three-dimensional shape model M according to the first embodiment. It is a flowchart which shows the flow of the process performed by the 3D shape model generation apparatus 1 which concerns on 1st Embodiment. It is a block diagram which shows the example of the structure of the 3D shape model generation apparatus 1A which concerns on 2nd Embodiment.

Hereinafter, the three-dimensional shape model generator of the embodiment will be described with reference to the drawings.

<First Embodiment>
First, the first embodiment will be described.
FIG. 1 is a block diagram showing an example of the configuration of the three-dimensional shape model generation device 1 according to the first embodiment. The three-dimensional shape model generation device 1 includes, for example, an image data acquisition unit 101, a three-dimensional shape generation unit 102, a focus area detection unit 103, a distance calculation unit 104, a scale estimation unit 105, and a scale conversion unit 106. The image data storage unit 107, the three-dimensional shape storage unit 108, and the scale information storage unit 109 are provided.

The image data acquisition unit 101 acquires the image information of the multi-viewpoint image TG (see FIG. 3A) from the image data storage unit 107. The multi-viewpoint image TG is an image in which the objects T are captured from different viewpoints. The object T is an object that can be imaged and has an arbitrary three-dimensional shape. The image information of the multi-viewpoint image TG includes information indicating a color such as an RGB value or a gray scale for each pixel of the multi-viewpoint image TG.

The image data acquisition unit 101 acquires the camera parameters of the multi-viewpoint image TG from the scale information storage unit 109. The camera parameter of the multi-viewpoint image TG is the attribute information of the multi-viewpoint image TG, and is the information indicated by the so-called Exif (Exchangeable image file format). For example, the camera parameter is information indicating the viewpoint position (position of the camera at the time of imaging), the imaging direction, the angle of view, and the like when the multi-viewpoint image TG is imaged. Further, the camera parameters may include information about an imaging device (camera) that has captured the multi-viewpoint image TG. The information about the image pickup device is information indicating the specifications of the components of the image pickup device and the state at the time of imaging. This is information indicating the distortion coefficient of the lens.

The image data acquisition unit 101 acquires image information and camera parameters in a plurality of multi-viewpoint image TGs, and outputs the acquired information to the three-dimensional shape generation unit 102.

The three-dimensional shape generation unit 102 creates a three-dimensional shape model M of the object T. First, the three-dimensional shape generation unit 102 generates the depth maps of each of the plurality of multi-viewpoint image TGs from the principle of stereo matching by using the image information of the plurality of multi-viewpoint image TGs and the camera parameters. The depth map is information (map) indicating the depth (depth) of each pixel of the image.

The three-dimensional shape generation unit 102 integrates each depth map of the multi-viewpoint image TG to generate a three-dimensional point cloud. The three-dimensional point cloud is a set of three-dimensional points corresponding to the three-dimensional shape of the object T. The three-dimensional shape generation unit 102 generates a mesh model using the three-dimensional point cloud. The mesh model is a three-dimensional shape model that shows the three-dimensional shape of an object as an aggregate of polygons. The three-dimensional shape generation unit 102 generates a mesh model from a three-dimensional point cloud, for example, by using a method of mesh reconstruction (Poisson Surface Reconstruction). The three-dimensional shape generation unit 102 uses the generated mesh model as a three-dimensional shape model.

The three-dimensional shape generation unit 102 stores the generated information about the three-dimensional point cloud and the mesh model in the three-dimensional shape storage unit 108. The information about the three-dimensional point cloud includes information indicating the coordinates (three-dimensional coordinates) of each point of the three-dimensional point cloud. Further, the information regarding the three-dimensional point cloud may include information indicating the color (for example, RGB value) of each point in the three-dimensional point cloud. The information about the mesh model includes information indicating the shape, coordinates (three-dimensional coordinates), colors, textures, and the like of polygons (polygons) constituting the mesh model.

The focus area detection unit 103 detects an in-focus area in each of the plurality of multi-viewpoint image TGs. The focus region detection unit 103 detects an in-focus region in each of the plurality of multi-viewpoint image TGs by using, for example, a machine learning method. In this case, the focus area detection unit 103 inputs the multi-viewpoint image TG into the trained model. The trained model is, for example, a model generated by training a training data set of a convolutional neural network (CNN) with a training model of a convolutional neural network (CNN). A data set for training is information in which inputs and outputs (answers to inputs) are combined (set).

Here, the input of the learning data set is an image in which an arbitrary object prepared for learning is captured, and is an image in which an in-focus part and an out-of-focus part are mixed. The output of the training data set is information indicating the in-focus part and the out-of-focus part of the training image. For example, information indicating whether or not the image is in focus is associated with each pixel. It is a data set. The output of the training data set is determined, for example, by the worker who creates the training data set. That is, the worker determines whether or not each pixel is in focus and sets it as the output of the learning data set.

By training such a training data set, the trained model estimates (outputs) the degree to which each pixel in the input (unlearned) image is in focus. It becomes.

The focus area detection unit 103 stores the information indicating the detection result (information indicating the in-focus area in each of the plurality of multi-viewpoint image TGs) in the image data storage unit 107. The information indicating the detection result is, for example, information in which the degree of focusing (hereinafter, also referred to as the blur amount) is associated with each pixel of each image of the multi-viewpoint image TG. The amount of blur is expressed by a real value from 0 to 1, for example, and when it is close to 0, it indicates that it is in focus, and when it is close to 1, it indicates that it is out of focus. That is, the amount of blur is smaller when the subject is in focus.

In the above description, the case where the focus area detection unit 103 detects the in-focus area by using the machine learning method has been described as an example, but the present invention is not limited to this. The focus area detection unit 103 may detect an in-focus area by using an arbitrary method. For example, the focus area detection unit 103 may determine the focused area as the focused area by the focus function of the camera. This focused area is, for example, an area inside a frame indicating a focused area, which is displayed so as to be superimposed on the imaging range on a rear display that displays the imaging range at the time of imaging.

For example, the focus area detection unit 103 may detect an in-focus area by using image processing. In this case, the focus region detection unit 103 detects the degree of color change in the image by frequency analysis. The focus region detection unit 103 performs a Fourier transform on the RGB value for each pixel for a local region in the vicinity of the pixel, and extracts a high frequency component in the local region. Then, the focus region detection unit 103 determines whether or not the high frequency component in the extracted local region is in focus depending on whether or not the high frequency component is equal to or higher than a predetermined threshold value. When the level of the high frequency component in the local region is equal to or higher than a predetermined threshold value, the focus region detection unit 103 determines that the local region is in focus, and the level of the high frequency component in the local region is less than the predetermined threshold value. If, it is determined that the local region is out of focus.

The distance calculation unit 104 calculates the virtual distance. The virtual distance here is a virtual distance from a predetermined position to a corresponding portion of the object T corresponding to an arbitrary point of the three-dimensional point cloud constituting the three-dimensional shape model. The predetermined position is a virtual viewpoint position in the projected image generated by reprojecting the three-dimensional shape model onto the two-dimensional plane.

The virtual distance is a value obtained by multiplying the actual distance (real distance) by a constant. This is because the distance calculation unit 104 calculates the virtual distance according to the depth value (depth value) of each point in the three-dimensional point cloud. The depth value of the three-dimensional point cloud is a value calculated based on the relative positional relationship of the object T imaged in each image from the multi-viewpoint image TG. Therefore, it is not a positional relationship according to the actual dimensions, but a relative value based on some value. Therefore, the distance calculated according to the depth value of the three-dimensional point cloud is a virtual distance that is proportional to the real distance.

The distance calculation unit 104 calculates the virtual distance using the pixels included in the in-focus region of each of the plurality of multi-viewpoint image TGs. The in-focus area is an area detected by the focus area detection unit 103.

The distance calculation unit 104 focuses on pixels in which the amount of blur associated with the pixels of the multi-viewpoint image TG detected by the focus area detection unit 103 is less than a predetermined threshold value (for example, 0.1). Pixel. The distance calculation unit 104 acquires the depth value (depth value) of the focused pixel. The pixel depth value (depth value) is, for example, the depth value (depth value) for each pixel calculated in the process of generating the three-dimensional shape model M by the three-dimensional shape generation unit 102.

The distance calculation unit 104 makes each of the depth values (depth values) of the in-focus pixels correspond to each pixel of the projected image. Each pixel of the projected image is associated with the depth value of each of the in-focus pixels in one or more multi-viewpoint image TGs. The distance calculation unit 104 calculates the virtual distance in each pixel of the projected image by integrating the pixels of the plurality of multi-viewpoint image TGs corresponding to each pixel of the projected image.

In general, a camera has a depth of field (see FIG. 4) determined according to its camera parameters, and a range of focus is predetermined. This depth of field depends on the actual distance from the viewpoint position to the portion in which the subject captured in the image is in focus. From this, it is possible to obtain the actual distance by using the depth of field. On the other hand, as described above, the virtual distance is a virtual distance from a predetermined position to the corresponding portion of the object T in the focused region. That is, the virtual distance can be said to be a value obtained by multiplying the depth of field by a constant, and the ratio of the real distance to the virtual distance (scale value SC) can be obtained via the depth of field.

However, the depth of field varies. Therefore, the value of the virtual distance varies from pixel to pixel (see FIG. 5). This variation becomes an error with respect to the true value of the virtual distance, and can be a factor of deteriorating the accuracy of the virtual distance. In addition, the focus region detection unit 103 estimates whether or not the focus is achieved by using the trained model. Therefore, the accuracy of estimation may be insufficient depending on the contents of the training data set to be trained by the trained model. If the estimation is inaccurate, it is possible that pixels outside the depth of field range (that is, out of focus) will be erroneously estimated to be in focus. If the pixel for which the virtual distance is calculated includes a pixel that is considered to be in focus due to incorrect estimation even though it is not actually in focus, the virtual distance in that pixel is virtual. It is an error with respect to the true value of the distance.

As a countermeasure, in the present embodiment, the true value of the virtual distance is searched for by performing statistical processing on the virtual distance of each pixel of the projected image.

For example, the distance calculation unit 104 searches for the true value of the virtual distance by applying RANSAC (RANdom SAmple Consensus) to the virtual distance of each pixel of the projected image. RANSAC is a method of estimating data that does not include outliers by repeating applying the least squares method to randomly extracted data samples for outliers, that is, data groups that include errors. .. The distance calculation unit 104 searches for the true value of the virtual distance by calculating the range of the depth of field as an aligner (range of error) in RANSAC.

However, the depth of field is a value calculated with the distance u to the subject (see FIG. 4) as a parameter. On the other hand, the virtual distance is a virtual distance before being converted into a real distance. Therefore, when the depth of field is used as the aligner in RANSAC, the distance calculation unit 104 sets a tentative value (for example, 200 mm or the like) as the distance u to the subject. The distance calculation unit 104 sets the distance obtained by applying RANSAC to the virtual distance of each pixel of the projected image as the true value of the virtual distance. The distance calculation unit 104 stores the virtual distance of each pixel of the projected image and the true value of the virtual distance in the three-dimensional shape storage unit 108.

In the above description, the case where the distance calculation unit 104 calculates the true value of the virtual distance by using RANSAC has been described as an example, but the present invention is not limited to this. The distance calculation unit 104 may calculate the most probable virtual distance from a set of virtual distances including variations by using at least a statistical method. For example, the distance calculation unit 104 may derive a representative value from the set of virtual distances and use the derived value as the true value of the virtual distance. The representative value may be an arbitrary value derived from a set of virtual distances by a statistical method, and is, for example, a simple addition mean value, a weighted mean value, a median value, a maximum value, a minimum value, or the like. Alternatively, the distance calculation unit 104 may calculate the true value of the virtual distance by using the virtual distance selected from the set of virtual distances. In this case, for example, when the variation of the virtual distance between a plurality of pixels corresponding to the same point in the three-dimensional point cloud is large, the true value of the virtual distance may not be calculated.

The scale estimation unit 105 estimates the scale value SC. The scale value SC is the real distance with respect to the virtual distance. The scale estimation unit 105 uses the true value of the virtual distance calculated by the distance calculation unit 104 as the virtual distance when estimating the scale value SC. The scale estimation unit 105 derives the actual distance when estimating the scale value SC by using the camera parameters. When the Exif of the multi-viewpoint image TG describes the distance u itself to the subject, the scale estimation unit 105 uses that information as the actual distance. Alternatively, when the Exif of the multi-viewpoint image TG indicates the depth of field, the focal length, the lens F value, and the permissible circle of confusion diameter, the scale estimation unit 105 is based on the relational expression of FIG. The distance u to the subject is calculated, and the calculated value is used as the actual distance. Alternatively, as a display function of the camera, the scale estimation unit 105 sets the value corresponding to the display as the actual distance when the information on the distance u to the subject is displayed on the rear display of the camera at the time of imaging. You may. The scale estimation unit 105 stores the actual distance used for estimating the scale value SC and the estimated scale value SC in the scale information storage unit 109.

In order to simplify the processing, it is desirable to unify (fix) the camera parameters in the plurality of multi-viewpoint image TGs used to generate the three-dimensional shape model M. When the contents of the camera parameters are set differently for each multi-viewpoint image TG, the scale value SC or the like is calculated for each contents of the camera parameters.

The scale conversion unit 106 performs scale conversion by converting the virtual distance into a real distance. The scale conversion unit 106 performs scale conversion by multiplying the virtual distance calculated by the distance calculation unit 104 by the scale value SC estimated by the scale estimation unit 105. Further, the scale conversion unit 106 makes the three-dimensional shape correspond to the coordinate system according to the actual dimensions by multiplying the three-dimensional coordinates of each pixel of the projected image by the scale value SC. This makes it possible to obtain the actual dimensions of the three-dimensional shape.

The image data storage unit 107 stores information related to the multi-viewpoint image TG. The information regarding the multi-viewpoint image TG includes information indicating a depth value (depth value) calculated for each pixel of the multi-viewpoint image TG and information indicating the degree of focusing (blurring amount).
The three-dimensional shape storage unit 108 stores information about the three-dimensional shape model. The information about the three-dimensional shape model includes information indicating the virtual distance in each pixel of the projected image and the true value of the virtual distance in the three-dimensional shape model.
The scale information storage unit 109 stores information related to scale conversion.

FIG. 2 is a diagram showing an example of the configuration of information (scale information) stored in the scale information storage unit 109 according to the first embodiment. For example, the scale information storage unit 109 is created for each multi-viewpoint image TG.
As shown in FIG. 2, the information related to scale conversion includes items such as camera parameters and scale conversion parameters. The camera parameters include attribute information at the time of camera and imaging, for example, information indicating Exif. The camera parameters include information indicating an image ID, a camera model, a focal length, a focus, a lens F value, an allowable circle of confusion, a depth of field, and the like. The scale conversion parameters include the scale value SC estimated by the scale estimation unit 105 and the information indicating the actual distance used for estimating the scale value SC.

FIG. 3A is a diagram showing an example of a plurality of multi-viewpoint image TGs according to the first embodiment. The object T here is bread placed on a wood board. As shown in FIG. 3A, the multi-viewpoint image TG is composed of a plurality of images obtained by capturing the object T from different viewpoints.

FIG. 3B is a diagram showing an example of the three-dimensional shape model M according to the first embodiment. As shown in FIG. 3B, the three-dimensional shape can be restored from the multi-viewpoint image TG. This three-dimensional shape model M restores the shape, but the actual dimensions are unknown. The actual size is the size obtained by enlarging or reducing the three-dimensional shape model M, but the specific coefficient thereof is unknown.

FIG. 3C is a diagram showing an example of the multi-viewpoint image TG according to the first embodiment. As shown in FIG. 3C, a part of the imaging region of the multi-viewpoint image TG, for example, the left end of the wood board imaged by the multi-viewpoint image TG shown in FIG. 3C, is out of focus. In addition, a part of the other imaging region of the multi-viewpoint image TG, for example, a portion on the right side from the center of the pan imaged by the multi-viewpoint image TG shown in FIG. 3C, is in focus.

FIG. 3D is a diagram showing an example of a blur map BM of the multi-viewpoint image TG according to the first embodiment. A blur map is an image (map) in which a blur amount is associated with each pixel in the image. In this example, the closer to white, the greater the amount of blur, that is, the greater the degree of out-of-focus. It also shows that the closer to black, the smaller the amount of blur, that is, the higher the degree of focus.

FIG. 4 is a diagram showing an example of a function of depth of field according to the first embodiment. In FIG. 4, DoF _f is the front depth of field, DoF _r is the rear depth of field, N is the lens F value, c is the permissible circle of confusion diameter, f is the focal length, and u is the distance to the subject.
As shown in FIG. 4, the depth of field is obtained by the sum of the front depth of field DoF _f and the rear depth of field DoF _r , centered on the distance u to the subject, and has a predetermined width. It becomes a value. The forward depth of field DoF _f is a range in which focus is achieved in the direction approaching the viewpoint position from the distance u to the subject. Rear depth of field DoF _r is a focus range in the direction away from the viewpoint position from the distance u to the object.

FIG. 5 is an example of the distribution of the virtual distance of each pixel of the projected image according to the first embodiment. As shown in FIG. 5, the virtual distance of each pixel of the projected image varies. By applying statistical processing such as RANSAC to a set of virtual distances including such an error, a probable virtual distance is calculated. As a result, the virtual distance with high accuracy can be obtained, and the actual dimensions of the three-dimensional shape model M can be obtained with high accuracy.

FIG. 6 is a flowchart showing a flow of processing performed by the three-dimensional shape model generation device 1 according to the first embodiment.
Step S101:
The image data acquisition unit 101 acquires a multi-viewpoint image TG of the object T.
Step S102:
The three-dimensional shape generation unit 102 generates a three-dimensional shape model M using the multi-viewpoint image TG.
Step S103:
The focus area detection unit 103 generates a blur map in the multi-viewpoint image TG.
Step S104:
The distance calculation unit 104 calculates the virtual distance for each pixel included in the focused region in the multi-viewpoint image TG.
Step S105:
The distance calculation unit 104 calculates the virtual distance of each pixel of the projected image by associating each pixel of the projected image of the three-dimensional shape model M with the virtual distance of each pixel of the multi-viewpoint image TG calculated in step S104. To do.
Step S106:
The distance calculation unit 104 calculates the most probable virtual distance (true value of the virtual distance) in the projected image by performing statistical processing on the variation in the virtual distance of each pixel of the projected image.
Step S107:
The scale estimation unit 105 estimates the scale value SC, which is the actual distance with respect to the virtual distance. The scale estimation unit 105 calculates the scale value SC by using, for example, the real distance derived by using the camera parameters and the virtual distance calculated in step S106.
Step S108:
The scale conversion unit 106 calculates the actual distance. The scale conversion unit 106 calculates the actual distance by multiplying the virtual distance calculated in step S106 by the scale value SC estimated in step S107.

In the above-described embodiment, as shown in the flowchart of FIG. 6, a case where a three-dimensional shape model is generated (step S102) and then a blur map in the multi-viewpoint image TG is generated (step S103) is illustrated. As explained, but not limited to this. For example, the three-dimensional shape generation unit 102 is set so that the weight of the image having a high pixel blur intensity is smaller than that of the image having a low blur intensity, so that only the region where the multi-viewpoint image TG is in focus is set. May be used to generate a three-dimensional shape model. In this case, each pixel of the projected image is composed of only the pixels that are in focus in the multi-viewpoint image TG. In this case, the process of "corresponding each pixel of the projected image of the three-dimensional shape model M with the virtual distance of each pixel of the multi-viewpoint image TG calculated in step S104" in step S105 can be omitted. That is, in step S105, the depth value of each pixel of the projected image can be used as the virtual distance as it is.

As described above, the three-dimensional shape model generation device 1 according to the first embodiment includes the three-dimensional shape generation unit 102, the focus area detection unit 103, the distance calculation unit 104, the scale estimation unit 105, and the scale. It includes a conversion unit 106. The three-dimensional shape generation unit 102 generates a three-dimensional shape model M of the object T from a plurality of multi-viewpoint images TG obtained by capturing the object T from different viewpoints. The focus area detection unit 103 detects an in-focus area in each image of the multi-viewpoint image TG. The distance calculation unit 104 is a virtual distance between the projected image obtained by projecting the three-dimensional shape model M onto a two-dimensional plane and the viewpoint position for the pixels in the in-focus region of each image of the multi-viewpoint image TG. Calculate the virtual distance. The scale estimation unit 105 estimates the scale value SC. The scale conversion unit 106 converts the virtual distance into an actual distance by using the scale value SC. As a result, the three-dimensional shape model generation device 1 of the first embodiment can improve the accuracy of the virtual distance by utilizing the fact that the in-focus region is within the range of the depth of field. Therefore, the actual dimensions of the three-dimensional shape model M restored by the three-dimensional restoration method using the multi-viewpoint image TG can be estimated with high accuracy.

Further, in the three-dimensional shape model generation device 1 of the first embodiment, the focus region detection unit 103 focuses on each image of the multi-viewpoint image TG by using the focus function of the camera that captured the multi-viewpoint image TG. The in-focus region in each image of the multi-viewpoint image TG may be detected according to the region. As a result, the in-focus area can be easily detected.

Further, in the three-dimensional shape model generation device 1 of the first embodiment, the focus area detection unit 103 may detect the in-focus area in each image of the multi-viewpoint image TG by image processing. As a result, even if the focused area is unknown, the in-focus area can be detected.

Further, in the three-dimensional shape model generation device 1 of the first embodiment, the focus area detection unit 103 is learning generated by performing machine learning using a learning data set in which inputs and outputs are associated with each other. The completed model may be used to detect an in-focus region in each image of the multi-viewpoint image TG. In this case, the input of the training data set is the input image for training. The output of the training data set is information indicating an in-focus area in the training input image. As a result, it is possible to detect an in-focus area without performing complicated image processing.

Further, in the three-dimensional shape model generation device 1 of the first embodiment, the distance calculation unit 104 is a corresponding pixel corresponding to a pixel determined to be in focus by the focus area detection unit 103, and is three-dimensional. The depth value in the corresponding pixel of the projected image obtained by projecting the shape model M on the two-dimensional plane may be set as the virtual distance in the corresponding pixel of the projected image. As a result, the virtual distance can be easily obtained from the depth value obtained in the process of generating the three-dimensional shape model M.

Further, in the three-dimensional shape model generation device 1 of the first embodiment, the scale estimation unit 105 derives the actual distance based on the depth of field calculated from the camera parameters of the camera that captured the multi-viewpoint image TG. You may do so. Thereby, the actual distance can be easily derived.

Further, in the three-dimensional shape model generation device 1 of the first embodiment, the scale estimation unit 105 may derive the actual distance based on the distance obtained from the focus function of the camera that has captured the multi-viewpoint image TG. Good. Thereby, the actual distance can be easily derived.

Further, in the three-dimensional shape model generation device 1 of the first embodiment, the distance calculation unit 104 corresponds to the projected image with a distance corresponding to the depth value of the pixels included in the detected in-focus region. The virtual distance for each pixel of the projected image may be calculated by corresponding to the points. Thereby, the actual distance can be easily derived.

Further, in the three-dimensional shape model generation device 1 of the first embodiment, the distance calculation unit 104 corresponds to the projected image with a distance corresponding to the depth value of the pixels included in the detected region in focus. The virtual distance for each pixel of the projected image is calculated by making it correspond to the points, and the virtual distance for each pixel is calculated by integrating the calculated virtual distance for each pixel using a statistical method. May be good. As a result, even if there is an error in the virtual distance for each pixel of the projected image, it is possible to calculate a more probable virtual distance with the error reduced by integrating using a statistical method. is there.

(Modification 1 of the first embodiment)
Next, a modification 1 of the first embodiment will be described. This modification is different from the above-described embodiment in that the virtual distance is calculated depending on whether or not the pixel determined to be in focus by the focus region detection unit 103 is an edge.

In general, the depth value of the pixel corresponding to the edge of the object T tends to include a large amount of error. Therefore, if the virtual distance is calculated according to whether or not the pixel corresponds to the edge of the object T, the accuracy of the calculated virtual distance can be improved.

For example, the distance calculation unit 104 does not use the depth value of the pixel corresponding to the edge in the calculation of the virtual distance, or the weighting for multiplying the depth value of the pixel corresponding to the edge is smaller than that of other pixels. Set to a value. The distance calculation unit 104 determines, for example, whether or not the pixel in the in-focus region in the multi-viewpoint image TG is an edge. The distance calculation unit 104 detects whether or not it is an edge by using, for example, a method of edge detection by the Canny method. The distance calculation unit 104 makes the depth values of the pixels determined not to be edges correspond to the pixels of the projected image among the pixels included in the in-focus region in each of the plurality of multi-view image TGs. Determine the virtual distance of each pixel in the projected image. The distance calculation unit 104 calculates the most probable virtual distance (true value of the virtual distance) in the projected image by performing statistical processing on each of the virtual distances in each pixel of the projected image.
Alternatively, the distance calculation unit 104 determines whether or not the corresponding pixel in the projected image is an edge, and uses only the pixel determined that the corresponding pixel is not an edge to use the most probable virtual distance (virtual distance) in the projected image. The true value of) may be calculated.

As described above, in the three-dimensional shape model generation device 1 according to the modified example of the first embodiment, in the distance calculation unit 104, the pixels determined by the focus area detection unit 103 to be in focus are edges. The virtual distance is calculated based on the determination result of determining whether or not the image is. Alternatively, the distance calculation unit 104 determines whether or not the corresponding pixel in the projected image is an edge, and uses only the pixel determined that the corresponding pixel is not an edge to use the most probable virtual distance (virtual distance) in the projected image. The true value of) may be calculated. That is, the distance calculation unit 104 at least determines whether or not the pixel in the focused region in the multi-viewpoint image TG is an edge, and whether or not the corresponding pixel in the projected image is an edge. Using one of the determination results, the most probable virtual distance (true value of the virtual distance) in the projected image is calculated. As a result, it is possible to reduce the influence of the depth value, which is likely to include an error, on the calculation of the virtual distance.

(Modification 2 of the first embodiment)
Next, a modification 2 of the first embodiment will be described. This modification differs from the above-described embodiment in that the depth value of the pixel determined to be in focus by the focus region detection unit 103 is weighted.

The distance calculation unit 104 weights the distance according to the depth value of the pixels included in the in-focus region according to the degree of focus. The distance calculation unit 104 sets the weighting so that the multiplication value of the weighting becomes large when the focus is higher (the degree of focusing is larger). The distance calculation unit 104 sets the weighting so that the multiplication value of the weighting becomes smaller when the focus is less (the degree of the focus is smaller). The distance calculation unit 104 calculates the virtual distance for each pixel of the projected image by associating the weighted distance with the point corresponding to the projected image.

Further, the distance calculation unit 104 may weight the distance corresponding to the depth value of the pixels included in the in-focus region according to the magnitude of the distance. The distance calculation unit 104 sets the weighting so that the multiplication value of the weighting becomes large when the distance is small (close to the viewpoint position of the camera). The distance calculation unit 104 sets the weighting so that the multiplication value of the weighting becomes small when the distance is large (far from the viewpoint position of the camera). The distance calculation unit 104 calculates the virtual distance for each pixel of the projected image by associating the weighted distance with the point corresponding to the projected image.

As described above, in the three-dimensional shape model generation device 1 according to the second modification of the first embodiment, the focus area detection unit 103 detects the degree of focus for each pixel, and the distance calculation unit 104 determines the degree of focus. , The distance according to the depth value of the pixel included in the in-focus area is weighted according to the degree of in-focus. As a result, the degree of focus can be reflected in the calculation of the virtual distance, and the virtual distance can be calculated more accurately.

Further, in the three-dimensional shape model generation device 1 according to the second modification of the first embodiment, the distance calculation unit 104 sets the distance according to the depth value of the pixels included in the in-focus area. Weight according to size. Generally, as the distance from the viewpoint position of the camera increases, the error included in the distance u to the subject tends to increase. Therefore, it is possible to calculate the virtual distance more accurately by weighting according to the distance from the viewpoint position.

(Modification 3 of the first embodiment)
Next, a modification 3 of the first embodiment will be described. The present embodiment differs from the above-described embodiment in that the virtual distance of the projected image is calculated in consideration of the variation in the depth values of the pixels of the plurality of multi-viewpoint image TGs corresponding to one pixel of the projected image. To do.

For example, the distance calculation unit 104 extracts the pixels of the multi-viewpoint image TG corresponding to the pixels of the projected image for each pixel. When there are a plurality of pixels of the multi-viewpoint image TG corresponding to one pixel of the projected image, the distance calculation unit 104 calculates the degree of variation in the depth value in the plurality of pixels. Any statistical method, such as variance, may be used to calculate the degree of variability. When the degree of variation in the depth value of the pixels of the multi-viewpoint image TG is larger than a predetermined threshold value, the distance calculation unit 104 sets the virtual distance of the pixel in the projected image as the virtual distance (true value of the virtual distance) of the projected image. Not used for calculation. On the other hand, when the degree of variation in the depth value is smaller than a predetermined threshold value, the distance calculation unit 104 uses the virtual distance of the pixel in the projected image to calculate the virtual distance (true value of the virtual distance) of the projected image. To do.

As described above, in the three-dimensional shape model generation device 1 according to the third modification of the first embodiment, the distance calculation unit 104 has a depth value in the pixels of a plurality of multi-viewpoint image TGs corresponding to the pixels of the projected image. It is determined whether or not the virtual distance of the pixel in the projected image is used for calculating the virtual distance of the projected image according to the degree of variation of. As a result, when the variation in the depth value for each pixel is large, it can be prevented from being used for calculating the virtual distance of the projected image, and the virtual distance of the projected image can be calculated more accurately.

(Second Embodiment)
Next, the second embodiment will be described. In the following description, only the parts different from the above-described embodiment will be described, and the same parts will be designated by the same reference numerals and the description thereof will be omitted.

The present embodiment differs from the above-described embodiment in that the actual distance used when estimating the scale value SC is obtained by calibration. By obtaining the actual distance by calibration, the actual distance can be obtained more accurately, and the actual dimensions of the three-dimensional shape model M can be estimated more accurately.

FIG. 7 is a block diagram showing an example of the configuration of the three-dimensional shape model generation device 1A according to the second embodiment. The three-dimensional shape model generation device 1A includes a marker scale estimation unit 110.
The marker scale estimation unit 110 estimates the scale value SC in the projected image of the marker three-dimensional shape model MM. The marker three-dimensional shape model MM is a three-dimensional shape model of the object T to which the marker is attached. The marker is a mark whose actual dimensions are known.

The marker scale estimation unit 110 estimates the scale value SC in the projected image of the marker three-dimensional shape model MM (see FIG. 8) by the same processing as that performed by the scale estimation unit 105 in the first embodiment. However, in the marker multi-viewpoint image MTG, the actual dimensions can be known by using the marker captured in the image as a clue. Based on this actual dimension, the real distance, not the virtual distance, is calculated.

Specifically, the image data acquisition unit 101 acquires a marker multi-viewpoint image MTG, which is a multi-viewpoint image of the object T with a marker.
The three-dimensional shape generation unit 102 generates a marker three-dimensional shape model MM using the marker multi-viewpoint image MTG.
The focus area detection unit 103 generates a blur map in the marker multi-viewpoint image MTG.
The distance calculation unit 104 calculates the actual distance for each pixel included in the focused region in the marker multi-viewpoint image MTG.
The distance calculation unit 104 calculates the actual distance of each pixel of the projected image by associating each pixel of the projected image of the marker three-dimensional shape model MM with the actual distance of each pixel calculated above.
The distance calculation unit 104 calculates the most probable real distance (true value of the real distance) in the projected image by performing statistical processing on the variation in the real distance of each pixel of the projected image.
The marker scale estimation unit 110 includes a real distance (true value of the real distance) calculated by the distance calculation unit 104 and a virtual distance (true value of the virtual distance) calculated by the distance calculation unit 104 in the first embodiment. Is used to estimate the scale value SC.

As described above, the three-dimensional shape model generation device 1A of the second embodiment includes the marker scale estimation unit 110. The marker scale estimation unit 110 derives the actual distance based on the corrected three-dimensional shape model in which the three-dimensional shape model M of the marker whose actual dimensions are known is scale-corrected based on the actual dimensions. The scale estimation unit 105 estimates the scale value SC using the actual distance estimated by the marker scale estimation unit 110. This makes it possible to estimate the scale value SC using a more accurate real distance.

(Modification 1 of the second embodiment)
Next, a modification 1 of the second embodiment will be described. In this modification, the content of the learning data set to be trained by the trained model is different from the above-described embodiment.

In this modification, the input of the training data set is a corrected projection image. The corrected projected image is an image obtained by projecting the corrected marker three-dimensional shape model MM onto a two-dimensional plane. The corrected marker three-dimensional shape model MM is an actual marker three-dimensional shape model MM generated by using the marker multi-view image MTG, which is a multi-view image of the object T to which the marker is attached. It is a model corrected to the actual size of the object T with a marker by enlarging or reducing it according to the value.

Further, the output of the training data set is used as information indicating a focused region in the corrected projection image, which is determined based on the depth value for each pixel in the corrected projection image.

As described above, in the three-dimensional shape model generation device 1A according to the first modification of the second embodiment, the input of the training data set is a three-dimensional shape model of a marker whose actual dimensions are known. This is a corrected projection image obtained by projecting the corrected 3D shape model onto a 2D plane based on the corrected 3D shape model scale-corrected based on the actual dimensions of, and the output of the training data set is , Information indicating an in-focus area in the projected image, which is determined based on the depth value for each pixel in the corrected projected image. As a result, in the three-dimensional shape model generation device 1A according to the first modification of the second embodiment, the corrected projection image (that is, the projection image having the actual dimensional information) and the corrected projection image are It is possible to learn whether or not the image is in focus according to the depth value for each pixel (that is, the distance to the actual subject). As a result, the trained model can be used as a model for estimating whether or not the subject is in focus according to the depth of field. By estimating whether or not the trained model is in focus according to the depth of field, it is possible to keep the virtual distance within the range of the depth of field and reduce the error included in the virtual distance. It is possible.

All or part of the three-dimensional shape model generation device 1 in the above-described embodiment may be realized by a computer. In that case, the program for realizing this function may be recorded on a computer-readable recording medium, and the program recorded on the recording medium may be read by the computer system and executed. The term "computer system" as used herein includes hardware such as an OS and peripheral devices. Further, the "computer-readable recording medium" refers to a portable medium such as a flexible disk, a magneto-optical disk, a ROM, or a CD-ROM, or a storage device such as a hard disk built in a computer system. Further, a "computer-readable recording medium" is a communication line for transmitting a program via a network such as the Internet or a communication line such as a telephone line, and dynamically holds the program for a short period of time. It may also include a program that holds a program for a certain period of time, such as a volatile memory inside a computer system that serves as a server or a client in that case. Further, the above program may be for realizing a part of the above-mentioned functions, and may be further realized for realizing the above-mentioned functions in combination with a program already recorded in the computer system. It may be realized by using a programmable logic device such as FPGA.

Although the embodiments of the present invention have been described in detail with reference to the drawings, the specific configuration is not limited to this embodiment, and includes designs and the like within a range that does not deviate from the gist of the present invention.

1 ... Three-dimensional shape model generation device 101 ... Image data acquisition unit 102 ... Three-dimensional shape generation unit 103 ... Focus area detection unit 104 ... Distance calculation unit 105 ... Scale estimation unit 106 ... Scale conversion unit 107 ... Image data storage unit 108 ... Three-dimensional shape storage unit 109 ... Scale information storage unit 110 ... Marker scale estimation unit

Claims (16)

A three-dimensional shape generator that generates a three-dimensional shape model of the object from a plurality of multi-viewpoint images obtained by capturing the object from different viewpoints.
A focus area detection unit that detects an in-focus area in each image of the multi-viewpoint image,
It is a virtual distance between the projected image obtained by projecting the three-dimensional shape model onto a two-dimensional plane and the viewpoint position in the projected image for the pixels in the in-focus region of each image of the multi-viewpoint image. The distance calculation unit that calculates the virtual distance and
A scale estimation unit that estimates a scale value indicating the ratio of the projected image to the actual actual distance to the viewpoint position in the projected image with respect to the virtual distance.
A scale conversion unit that converts the virtual distance into the real distance using the scale value, and
A three-dimensional shape model generator equipped with. The focus region detection unit focuses on each image of the multi-viewpoint image according to the region focused by using the focus function of the camera that captured the multi-viewpoint image. Detect the area
The three-dimensional shape model generator according to claim 1. The focus area detection unit detects an in-focus area in each image of the multi-viewpoint image by image processing.
The three-dimensional shape model generator according to claim 1. The focus area detection unit uses a trained model generated by performing machine learning using a learning data set in which inputs and outputs are associated with each other, and focuses on each image of the multi-viewpoint image. Detects the area
The input of the training data set is an input image for training, and is
The output of the training data set is information indicating a focused region in the input image.
The three-dimensional shape model generator according to claim 1. The input of the training data set is each image of the multi-viewpoint image.
The output of the training data set is to project a corrected 3D shape model on a 2D plane, which is a scale-corrected 3D shape model of a marker whose actual dimensions are known based on the actual dimensions of the marker. Further, it is information indicating an in-focus area in the corrected projection image determined based on the depth value for each pixel in the corrected projection image.
The three-dimensional shape model generator according to claim 4. The distance calculation unit determines whether or not the pixel determined to be in focus by the focus area detection unit is an edge, and the pixel determined to be in focus by the focus area detection unit. Based on the determination result of at least one of the determination results of whether or not the corresponding pixel of the projected image obtained by projecting the three-dimensional shape model onto the two-dimensional plane is an edge. It is determined whether or not the virtual distance in the corresponding pixel of the projected image is used for calculating the virtual distance in the projected image.
The three-dimensional shape model generator according to any one of claims 1 to 5. The scale estimation unit derives the actual distance based on the depth of field calculated from the camera parameters of the camera that captured the multi-viewpoint image.
The three-dimensional shape model generator according to any one of claims 1 to 6. A marker scale estimation unit for deriving the actual distance based on the corrected 3D shape model in which the 3D shape model of the marker whose actual dimensions are known is scale-corrected based on the actual dimensions is further provided.
The scale estimation unit estimates the scale value using the actual distance derived by the marker scale estimation unit.
The three-dimensional shape model generator according to any one of claims 1 to 6. The marker scale estimation unit determines that the pixels in the corrected projected image obtained by projecting the corrected three-dimensional shape model onto a two-dimensional plane based on the corrected three-dimensional shape model are in focus. The actual distance is derived based on the depth value of the determined pixel.
The three-dimensional shape model generator according to claim 8. The scale estimation unit derives the actual distance based on the distance obtained from the focus function of the camera that captured the multi-viewpoint image.
The three-dimensional shape model generator according to any one of claims 1 to 6. The distance calculation unit makes the distance corresponding to the depth value of the pixels included in the detected focused region correspond to each pixel of the projected image, whereby the virtual of each pixel of the projected image. Calculate the distance,
The three-dimensional shape model generator according to any one of claims 1 to 10. The focus area detection unit detects the degree of focus for each pixel and determines the degree of focus.
The distance calculation unit weights the distance according to the depth value of the pixel included in the detected in-focus area according to the degree of the in-focus, and the weighted distance is the projected image. The virtual distance for each pixel of the projected image is calculated by associating each pixel with.
The three-dimensional shape model generator according to claim 11. The distance calculation unit weights the distance according to the depth value of the pixels included in the detected in-focus region according to the magnitude of the distance, and the weighted distance is the projected image. The virtual distance for each pixel of the projected image is calculated by associating each pixel.
The three-dimensional shape model generator according to claim 11 or 12. The distance calculation unit calculates a distance corresponding to the depth value of each pixel in the focused region in each image of the multi-viewpoint image, and associates the calculated distance with the pixels of the projected image to cause the projected image. When there are a plurality of distances corresponding to the pixels, the plurality of distances are compared, and the virtual distance in the pixels of the projected image is changed to the virtual distance in the projected image according to the degree of variation in the plurality of distances. Judge whether to use for calculation,
The three-dimensional shape model generator according to any one of claims 1 to 13. A three-dimensional shape generation step in which the three-dimensional shape generation unit generates a three-dimensional shape model of the object from a plurality of multi-viewpoint images obtained by capturing the object from different viewpoints.
A focus area detection step in which the focus area detection unit detects an in-focus area in each image of the multi-viewpoint image,
The distance calculation unit virtualizes the projected image obtained by projecting the three-dimensional shape model onto a two-dimensional plane and the viewpoint position in the projected image for the pixels in the in-focus region of each image of the multi-viewpoint image. A distance calculation process that calculates a virtual distance, which is a typical distance,
A scale estimation step in which the scale estimation unit estimates a scale value indicating the ratio of the projected image to the actual actual distance to the viewpoint position in the projected image with respect to the virtual distance.
A scale conversion step in which the scale conversion unit converts the virtual distance into the real distance using the scale value.
3D shape model generation method including. Computer,
A three-dimensional shape generating means for generating a three-dimensional shape model of the object from a plurality of multi-viewpoint images obtained by capturing the object from different viewpoints.
Focus area detecting means for detecting a focused area in each image of the multi-viewpoint image,
It is a virtual distance between the projected image obtained by projecting the three-dimensional shape model onto a two-dimensional plane and the viewpoint position in the projected image for the pixels in the in-focus region of each image of the multi-viewpoint image. Distance calculation means for calculating virtual distance and
A scale estimation means for estimating a scale value indicating the ratio of the projected image to the actual actual distance to the viewpoint position in the projected image with respect to the virtual distance.
A scale conversion means for converting the virtual distance into the real distance using the scale value.
A program to operate as.