JP2020102044A - Three-dimensional shape model generation system, three-dimensional shape model generation method, and program - Google Patents

Abstract

To provide a three-dimensional shape model generation system capable of reducing the data capacity of a multi-viewpoint image used for generating a three-dimensional shape model while suppressing a decrease in accuracy of the generated three-dimensional shape model.SOLUTION: The system includes an image information acquisition unit configured to acquire, from a multi-viewpoint image that is two or more color images of an object captured from different viewpoints, information of a color image including information of a grayscale image excluding color information indicating colors in the multi-viewpoint image, and the color information in the multi-viewpoint image, a capacity reduction unit configured to reduce the data capacity of the color image, a three-dimensional model generation unit configured to use the information of the grayscale image to generate a three-dimensional model of the object, and a coloring unit configured to use the information of the color image whose data capacity has been reduced by the capacity reduction unit to color the three-dimensional model generated by the three-dimensional model generation unit.SELECTED DRAWING: Figure 1

Description

The present invention relates to a three-dimensional shape model generation system, a three-dimensional shape model generation method, and a program for generating a three-dimensional shape model from a plurality of two or more images of an object taken from different viewpoints.

There is a three-dimensional reconstruction method that generates a three-dimensional shape model of an object based on a plurality of images of the object taken from different viewpoints (hereinafter referred to as “multi-view images”). Depending on the size and shape of the object, a large number of multi-viewpoint images are required to generate the three-dimensional shape model, and the data capacity of the multi-viewpoint images used to generate the three-dimensional shape model tends to increase. Therefore, in the case of a configuration in which an imaging device that captures a multi-viewpoint image and a server device that generates a three-dimensional shape model are connected via a communication network, the data capacity increases as the number of multi-viewpoint images increases. However, the communication time required to transmit the information of the multi-viewpoint image via the communication network increases. As a result, the time for generating the three-dimensional shape model is increased as a whole due to the increase in communication time for transmitting the multi-viewpoint images. Therefore, when generating a three-dimensional shape model, it is important to reduce the communication time required for transmitting the multi-viewpoint images necessary for generating the three-dimensional shape model.

As a technique for reducing the communication time required to transmit a multi-viewpoint image used to generate a three-dimensional shape model, feature points are extracted from an image, and the data volume is reduced by transmitting only the three-dimensional information of the extracted feature points. There is one that reduces the communication time required for transmission (for example, Patent Document 1). In addition, there is a technique for reducing the data capacity and the communication time required for transmission by determining that an inappropriate image is not registered in the position and orientation estimation of the imaging device (for example, Patent Document 2).

JP, 2013-8137, A JP, 2017-187861, A

However, Patent Document 1 is information that does not assume that the three-dimensional information of the feature points to be transmitted generates a three-dimensional shape model of an arbitrary object. In Patent Document 1, since the three-dimensional shape model is generated using only the three-dimensional information of the feature points, only the feature points (for example, corner points) suitable for model generation can be used. However, this is insufficient for expressing the shape of the entire object, and the accuracy of restoration in the generated three-dimensional shape decreases.
If the object has a large number of feature points, the accuracy of restoration in the three-dimensional shape is unlikely to deteriorate even if the three-dimensional shape model is generated using only the three-dimensional information of the feature points. However, since the amount of features is large, the data capacity cannot be reduced to the extent that the communication time is reduced. Rather, since the information shown for each pixel changes, the data capacity of the three-dimensional information of the extracted feature points may increase rather than the data capacity of the multi-viewpoint image.
Further, in Patent Document 2, since all the images that are determined to be appropriate images in the position and orientation estimation of the imaging device are registered, it is not possible to reduce the data capacity of the multi-viewpoint image that satisfies this condition. In the case where various images are taken, the effect of reduction can hardly be obtained. On the other hand, if the data capacity is reduced by strictly setting the conditions that are determined to be inappropriate in the position and orientation estimation of the image pickup apparatus, the accuracy of the three-dimensional shape may decrease.

The present invention has been made in view of such a situation, and reduces the data volume of a multi-viewpoint image used for generating a three-dimensional geometric model while suppressing a decrease in accuracy of the three-dimensional geometric model to be generated. Provided are a three-dimensional shape model generation system, a three-dimensional shape model generation method, and a program capable of reducing the communication time required for transmitting information of a multi-viewpoint image.

The three-dimensional shape model generation system of the present invention is a multi-view image that is two or more color images of an object captured from different viewpoints, and is a grayscale image excluding color information indicating colors in the multi-view image. Information, and an image information acquisition unit that acquires information about a color image including the color information in the multi-viewpoint image, a capacity reduction unit that reduces the data capacity of the color image, and information about the grayscale image, A three-dimensional model generation unit that generates a three-dimensional model of the object, and a three-dimensional model generated by the three-dimensional model generation unit using the information of the color image whose data capacity has been reduced by the capacity reduction unit. And a coloring unit for coloring.

In the three-dimensional shape model generation system of the present invention, the capacity reduction unit may reduce a reduction ratio, which is a ratio of a reduction amount to a pre-reduction amount in the data volume of the grayscale image, to be smaller than the reduction ratio in the color image. As described above, the data capacity of the grayscale image is reduced, and the three-dimensional model generation unit uses the grayscale image whose data capacity is reduced by the capacity reduction unit to generate a three-dimensional model of the object. It is characterized by generating.

The three-dimensional shape model generation system of the present invention is characterized in that the capacity reduction unit reduces the image size in the color image or the grayscale image.

The three-dimensional shape model generation system of the present invention is characterized in that the capacity reducing unit selects the color image used for coloring the three-dimensional model from a plurality of the color images.

The three-dimensional shape model generation system of the present invention is characterized in that the capacity reduction unit deletes a partial area of the image in the color image or the grayscale image.

The three-dimensional shape model generation system of the present invention is characterized in that the capacity reduction unit deletes a region in the color image or the grayscale image that is different from a region in which the object is imaged.

In the three-dimensional shape model generation system of the present invention, the capacity reduction unit deletes a region where blur occurs in the color image or the grayscale image.

In the three-dimensional shape model generation system of the present invention, the multi-viewpoint image is an RGB image, the image information acquisition unit acquires an image showing the brightness in the multi-viewpoint image by gray scale as the grayscale image, An image indicating the color in the multi-view image by RGB values is acquired as the color image.

In the three-dimensional shape model generation system of the present invention, the multi-viewpoint image is a Lab image, and the image information acquisition unit uses the image showing the brightness in the multi-viewpoint image by the L channel of the Lab image as the grayscale image. It is characterized in that the color image in the multi-viewpoint image is acquired and the image showing the color of the Lab image by the ab channel is acquired as the color image.

The three-dimensional shape model generation system of the present invention includes the image information acquisition unit and the capacity reduction unit, and provides the server device with the information of the grayscale image and the information of the color image with a reduced data capacity. The client device for transmitting, the three-dimensional model generation unit, and the coloring unit, the three-dimensional model generation unit using the information of the grayscale image received from the client device three-dimensional And a server device for generating a model.

A three-dimensional shape model generation system of the present invention includes the image information acquisition unit, the capacity reduction unit, and the coloring unit, transmits information of the grayscale image to a server device, and the server device described above. A client device that receives a three-dimensional shape model of the object, the coloring unit performs coloring on the three-dimensional model of the object received from the server device using the information of the color image in which the data capacity is reduced, A server device that includes the three-dimensional model generation unit, and that transmits the three-dimensional model generated by the three-dimensional model generation unit using the grayscale image information received from the client device to the client device. It is characterized by

In the three-dimensional shape model generation method of the present invention, the image information acquisition unit indicates the color in the multi-viewpoint image from information indicating the multi-viewpoint image that is two or more color images of the object captured from different viewpoints. An image information acquisition process of acquiring grayscale information excluding color information and information of a color image including the color information in the multi-view image, and a capacity reduction unit that reduces the data capacity of the color image. And a 3D model generation process in which the 3D model generation unit generates a 3D model of the object using the information of the grayscale image, and a coloring unit reduces the data capacity by the capacity reduction unit. And a coloring step of coloring the three-dimensional model generated by the three-dimensional model generation unit using the information of the generated color image.

The program of the present invention, a computer, grayscale information that excludes color information indicating a color in the multi-viewpoint image from information indicating a multi-viewpoint image that is two or more color images obtained by capturing an object from different viewpoints. , And an image information acquisition unit that acquires information about a color image including the color information in the multi-viewpoint image, a capacity reduction unit that reduces the data capacity of the color image, and the object using the information about the grayscale image. 3D model generating means for generating the 3D model, and coloring means for coloring the 3D model generated by the 3D model generating means using the information of the color image whose data capacity has been reduced by the capacity reducing means. It is a program for operating as.

According to the present invention, it is possible to reduce the data capacity of a multi-viewpoint image used for generating a three-dimensional shape model while transmitting the information of the multi-viewpoint image while suppressing a decrease in accuracy of the three-dimensional shape model to be generated. The communication time required for can be reduced.

It is a block diagram showing an example of composition in three-dimensional geometric model generation system 1 by a 1st embodiment. It is a figure explaining the process which the three-dimensional geometric model generation system 1 by 1st Embodiment performs. It is a sequence diagram explaining the flow of the process which the three-dimensional geometric model generation system 1 by 1st Embodiment performs. It is a figure explaining the process which the three-dimensional geometric model generation system 1 by the modification 1 of 1st Embodiment performs. It is a figure explaining the process which the three-dimensional geometric model production|generation system 1 by the modification 2 of 1st Embodiment performs. It is a figure explaining the process which the three-dimensional geometric model generation system 1 by the modification 3 of 1st Embodiment performs. It is a figure explaining the process which the three-dimensional geometric model production|generation system 1 by the modification 4 of 1st Embodiment performs. It is a block diagram showing an example of composition in three-dimensional geometric model generation system 1A by a 2nd embodiment. It is a sequence diagram explaining the flow of the process which the 3D geometric model production|generation system 1A by 2nd Embodiment performs.

An embodiment of the present invention will be described below with reference to the drawings.

(First embodiment)
First, the first embodiment will be described.
FIG. 1 is a block diagram showing an example of the configuration of a three-dimensional shape model generation system 1 according to the first embodiment. The three-dimensional shape model generation system 1 includes, for example, an imaging device 10, a client device 20, a server device 30, and a communication network 40. The imaging device 10 and the client device 20 are communicatively connected by, for example, a USB (Universal Serial Bus) cable or the like. Further, the client device 20 and the server device 30 are communicably connected via a communication network 40.

The imaging device 10 is a device that captures multi-viewpoint images, and is, for example, a digital camera. The multi-viewpoint image is an image for generating a three-dimensional shape model of the object, and is two or more images obtained by capturing the object from different viewpoints. The imaging device 10 outputs the information of the captured multi-viewpoint image via a USB cable or the like.
In the present embodiment, the imaging device 10 captures a multi-viewpoint image with colors. That is, the multi-viewpoint image is a color image.

The information of the multi-viewpoint image is information indicating the color, the brightness, or the brightness of each pixel in the image. For example, when the multi-view image is an RGB (Red Green Blue) image, the information of the multi-view image is an RGB value for each pixel in the image. Alternatively, when the multi-view image is a Lab image, the information of the multi-view image is information of the L channel and the ab channel for each pixel in the image. Alternatively, when the multi-viewpoint image is a YCbCr image, the information of the multi-viewpoint image is information of the Y channel and the CbCr channel for each pixel in the image.

Further, the information of the multi-viewpoint image may include camera parameters. The camera parameter is attribute information of the image, and is, for example, information indicating the position of the viewpoint, the image capturing direction, the angle of view, and the like when the image is captured. In addition, the camera parameter may include information (so-called internal parameter) regarding the image capturing device that captured the image. The internal parameter is information indicating the specifications of the components of the image pickup apparatus and the state at the time of image pickup, and for example, the focal length of the lens at the time of image pickup, the shutter speed, the exposure state, the resolution of the image (the number of pixels), the distortion aberration of the lens. This is information indicating a coefficient and the like.

The client device 20 is a device that performs information processing using the information of the multi-viewpoint image acquired from the imaging device 10, and is, for example, a personal computer. The client device 20 includes, for example, a communication unit 201, an image information acquisition unit 202, a capacity reduction unit 203, and a data set generation unit 204.

The communication unit 201 communicates with the imaging device 10 via a USB cable or the like. The communication unit 201 acquires information about a multi-viewpoint image from the imaging device 10. The communication unit 201 outputs the acquired multi-viewpoint image information to the image information acquisition unit 202 and the data set generation unit 204.
The communication unit 201 also communicates with the server device 30 via the communication network 40. The communication unit 201 transmits the data set generated by the data set generation unit 204 described below to the server device 30. The communication unit 201 acquires information on the three-dimensional shape model from the server device 30.

The image information acquisition unit 202 acquires information about a grayscale image and information about a color image from a multi-viewpoint image. The grayscale image is an image in which color information is excluded from the multi-viewpoint image. The color information is information indicating colors in the multi-viewpoint image. The color image is an image including color information in the multi-viewpoint image. The image information acquisition unit 202 outputs the acquired grayscale image information and color image information to the capacity reduction unit 203.

For example, when the multi-viewpoint image is an RGB image, the grayscale image is an image showing the brightness of the image in grayscale. Further, the color image is an image (RGB image) in which colors in the image are represented by RGB values. In this case, the grayscale image information is information indicating a grayscale calculated for each pixel based on the RGB value. In addition, the information of the color image is an RGB value for each pixel.

When the multi-viewpoint image is a Lab image, the grayscale image is an image showing the brightness of the image by the L channel of the Lab image. The color image is an image showing the color in the image by the ab channel in the Lab image. In this case, the grayscale image information is the L component value for each pixel in the image. The information of the color image is the a component value and the b component value for each pixel in the image.

When the multi-view image is a YCbCr image, the grayscale image is an image showing the brightness in the image by the Y channel of the YCbCr image. The color image is an image showing the color in the image by the CbCr channel of the YCbCr image. In this case, the grayscale image information is the Y component value for each pixel in the image. The information of the color image is the Cb component value and Cr component value for each pixel in the image.

The capacity reduction unit 203 acquires the color image information from the image information acquisition unit 202 and reduces the data capacity of the acquired color image. The capacity reduction unit 203 reduces the data capacity by reducing the image size of the color image, for example. The capacity reduction unit 203 outputs the information of the color image with the reduced data capacity to the data set generation unit 204.

The data set generation unit 204 generates a data set using the information of the multi-viewpoint image and the information of the color image with the reduced data capacity. The data set is a set of information for generating a three-dimensional shape model and is information including a grayscale image group information, a color image group information, and a camera parameter of each image in the object. The data set generation unit 204 transmits the generated data set to the server device 30 via the communication unit 201.

The server device 30 is a device that creates a three-dimensional model using the data set received from the client device 20. The server device 30 includes, for example, a communication unit 301, a three-dimensional model generation unit 302, and a coloring unit 303.

The communication unit 301 communicates with the client device 20 via the communication network 40. The communication unit 301 receives a data set for generating a 3D geometric model from the client device 20. The communication unit 301 outputs the received data set to the three-dimensional model generation unit 302 and the coloring unit 303. The communication unit 301 transmits the information of the three-dimensional shape model to the client device 20.

The three-dimensional model generation unit 302 generates a three-dimensional model using the grayscale image included in the data set. For example, the three-dimensional model generation unit 302 first generates a depth map using the information on the grayscale image group and the camera parameter. The depth map is information (map) indicating the depth (depth) of each pixel in each grayscale image. Next, the 3D model generation unit 302 integrates the depth maps of the grayscale images to generate a 3D point cloud. The three-dimensional point cloud is a set of three-dimensional points corresponding to the three-dimensional shape of the object. The three-dimensional model generation unit 302 uses the three-dimensional point cloud to generate a mesh model. The mesh model is a three-dimensional shape model that shows the three-dimensional shape of an object as a collection of polygons. The three-dimensional model generation unit 302 generates a mesh model from the three-dimensional point cloud using, for example, a method of mesh reconstruction (Poisson Surface Reconstruction). The three-dimensional model generation unit 302 uses the generated mesh model as a three-dimensional model and outputs information about the three-dimensional model to the coloring unit 303.

The coloring unit 303 colors the three-dimensional model using the color image included in the data set. The coloring unit 303 projects the color image on a three-dimensional model viewed from the viewpoint of the color image, and colors the area of the three-dimensional model corresponding to the color image to the color of the color image. The viewpoint of the color image is acquired from the camera parameters included in the data set. The coloring unit 303 colors the entire three-dimensional model by performing a process of projecting each of the color images on the three-dimensional model. When a part or all areas of a plurality of color images overlap in the three-dimensional model, the coloring unit 303 colors the representative value of the color of the corresponding pixel in each of the plurality of color images in the three-dimensional model. The representative value may be a color that represents each color, and is, for example, a simple addition average value of each color, an average value by weighted addition, or a median value. The coloring unit 303 transmits the colored 3D model information to the client device 20 via the communication unit 301. Here, the colored three-dimensional model is an example of a “three-dimensional shape model”.

The communication network 40 is, for example, a communication network including a part or all of the Internet, a WAN (Wide Area Network), a LAN (Local Area Network), a provider device, a wireless base station, a dedicated line, and the like.

Here, the processing performed by the three-dimensional shape model generation system 1 will be described with reference to FIGS. 2 and 3.
FIG. 2 is a diagram illustrating processing performed by the three-dimensional shape model generation system 1 according to the first embodiment. FIG. 2 schematically shows the processing performed by the three-dimensional shape model generation system 1 according to the first embodiment.
As shown in FIG. 2, in the three-dimensional shape model generation system 1 according to the first embodiment, first, the image information acquisition unit 202 acquires (separates) the grayscale image K and the color image R from the multi-viewpoint image T. To do. Next, the capacity reduction unit 203 reduces the data size of the color image R by reducing the image size of the color image R. Next, the data set generation unit 204 generates a data set including the grayscale image K and the color image R with the reduced data capacity and transmits the data set to the server device 30. The three-dimensional model generation unit 302 generates a three-dimensional model M using the grayscale image K included in the data set. Then, the coloring unit 303 generates a colored three-dimensional model D, which is a color of the three-dimensional model M, using the color image R in which the data volume included in the data set is reduced.

FIG. 3 is a sequence diagram illustrating the flow of processing performed by the 3D geometric model generation system 1 according to the first embodiment.
Step S10:
The imaging device 10 captures the multi-viewpoint image T. The imaging device 10 outputs the information of the captured multi-viewpoint image T to the client device 20 via a USB cable or the like.

Step S11:
The client device 20 acquires the information of the multi-viewpoint image T by the communication unit 201. The communication unit 201 outputs the information of the multi-viewpoint image T to the image information acquisition unit 202 and the data set generation unit 204.
Step S12:
In the client device 20, the image information acquisition unit 202 acquires information on the grayscale image K and the color image R from the multi-viewpoint image T. The image information acquisition unit 202 outputs the acquired information of the grayscale image K and the color image R to the capacity reduction unit 203.
Step S13:
The client device 20 causes the capacity reduction unit 203 to reduce the data capacity of the color image R. 302 outputs the information of the color image R with the reduced data capacity to the data set generation unit 204.
Step S14:
The data set generation unit 204 generates a data set including the grayscale image K and the color image R with the reduced data capacity. The data set generation unit 204 transmits the generated data set to the server device 30.

Step S15:
The server device 30 acquires the data set by the communication unit 301. The communication unit 301 outputs the acquired data set to the three-dimensional model generation unit 302 and the coloring unit 303.
Step S16:
The server device 30 causes the three-dimensional model generation unit 302 to generate the three-dimensional model M using the grayscale image K included in the data set.
Step S17:
The server device 30 causes the coloring unit 303 to color the three-dimensional model M by using the color image R in which the data volume included in the data set is reduced, and generates the colored three-dimensional model D. The coloring unit 303 transmits the generated information of the colored three-dimensional model D to the client device 20.

As described above, the 3D geometric model generation system 1 according to the first embodiment includes the image information acquisition unit 202, the capacity reduction unit 203, the 3D model generation unit 302, and the coloring unit 303. The image information acquisition unit 202 acquires, from the multi-viewpoint image T, information on the grayscale image K excluding color information on the multi-viewpoint image T and information on a color image R indicating colors in the multi-viewpoint image. The capacity reduction unit 203 reduces the data capacity of the color image R. The three-dimensional model generation unit 302 uses the information of the grayscale image K to generate a three-dimensional model of the object. The coloring unit 303 colors the three-dimensional model generated by the three-dimensional model generation unit 302 using the information of the color image R whose data capacity has been reduced by the capacity reduction unit 203.
As a result, the three-dimensional shape model generation system 1 according to the first embodiment suppresses a decrease in the accuracy of the three-dimensional model M generated by not reducing the data capacity of the grayscale image K, By reducing the data volume of the color image R, the data volume of information (data set) used for generating the colored three-dimensional model D (three-dimensional shape model) can be reduced.

Further, in the three-dimensional shape model generation system 1 according to the first embodiment, the capacity reduction unit 203 reduces the data capacity by reducing the image size of the color image R. Here, reducing the image size means reducing the number of pixels in the image. For example, when the original image size is 300×600 pixels, reducing the image size to 1/4 means that the reduced image size is 150×300 pixels. When reducing the image size, the number of pixels in the image is reduced by, for example, integrating a plurality of pixels into one pixel. As a method of integrating a plurality of pixels into one pixel, it is possible to leave only the pixel having the representative value of the values of the plurality of pixels and delete the other pixels, or to position at a specific coordinate of the plurality of pixels. It is also possible to leave only the pixels to be processed and delete the other pixels.

Further, in the three-dimensional shape model generation system 1 according to the first embodiment, the multi-viewpoint image is an RGB image, and the image information acquisition unit 202 acquires an image indicating the brightness of the multi-viewpoint image in grayscale as a grayscale image. Then, an image showing the colors in the multi-viewpoint image by RGB values is acquired as a color image. As a result, the three-dimensional shape model generation system 1 according to the first embodiment can reduce the data capacity by an easy method of extracting the gray scale and the RGB value from the RGB image.

Further, in the three-dimensional shape model generation system 1 according to the first embodiment, the multi-viewpoint image is a Lab image, and the image information acquisition unit 202 acquires, as a grayscale image, an image showing the brightness in the multi-viewpoint image by the L channel. Then, an image showing the color in the multi-view image by the ab channel is acquired as a color image. As a result, the three-dimensional shape model generation system 1 in the first embodiment can reduce the data capacity by a simple method of extracting the L channel and the ab channel from the Lab image.

Further, the three-dimensional shape model generation system 1 according to the first embodiment may include the client device 20 and the server device 30. The client device 20 includes an image information acquisition unit 202 and a capacity reduction unit 203, and transmits the information of the grayscale image K and the information of the color image R with the reduced data capacity to the server device 30. The server device 30 includes a three-dimensional model generation unit 302 and a coloring unit 303. The three-dimensional model generation unit 302 uses the grayscale image K received from the client device 20 to generate a three-dimensional model M of the object. To do. As a result, the three-dimensional geometric model generation system 1 according to the first exemplary embodiment can generate data even if data is transmitted and received between the client device 20 and the server device 30 via the communication network 40 or the like. Since the information of the color image R with the reduced capacity can be used, it is possible to reduce the data capacity of data to be transmitted and received.

(Modification 1 of the first embodiment)
Next, a modified example 1 of the first embodiment will be described. This modification is different from the above-described embodiment in that the data capacity of the grayscale image K is reduced.
In this modification, the capacity reduction unit 203 reduces the data size of the grayscale image K by reducing the image size of the grayscale image K.

For example, in the case of an RGB image, a grayscale calculated using the RGB value of each pixel is used for the grayscale image. In this case, a monochrome image is often stored in a full-color format, and the data capacities of the grayscale image and the RGB image are almost the same. Even if the image size of such a grayscale image is reduced to about 1/2, the reduced grayscale image and the non-reduced grayscale image have almost the same accuracy as the generated three-dimensional model M. It is well known from experience that there is no difference.

Therefore, the capacity reduction unit 203 reduces the data capacity so that the reduction ratio of the data capacity of the grayscale image K is smaller than the reduction ratio of the color image, for example. The reduction ratio here is the ratio of the reduction amount before the reduction. For example, the capacity reduction unit 203 reduces the image size of the color image R to 1/4 and the image size of the grayscale image K to 1/2.
In this case, assuming that the data amount of the color image R before reduction is 1, the reduced data amount of the color image R is (1-1/4=)3/4, and the reduction ratio of the color image R is It becomes 3/4. Further, assuming that the data capacity before reduction in the grayscale image K is 1, the data capacity of the reduced grayscale image K is (1-1/2=)1/2, and the reduction ratio of the grayscale image K is Becomes 1/2. That is, the reduction ratio of the grayscale image K is smaller than the reduction ratio of the color image R.

FIG. 4 is a diagram illustrating a process performed by the three-dimensional shape model generation system 1 according to the first modification of the first embodiment.
As shown in FIG. 4, in the three-dimensional shape model generation system 1 according to the modified example 1 of the first embodiment, first, the image information acquisition unit 202 extracts the grayscale image K and the color image R from the multi-viewpoint image T. Get (separate). Next, the capacity reduction unit 203 reduces the data size of both images by reducing the image size of the grayscale image K and the color image R. Next, the data set generation unit 204 generates a data set including the grayscale image K and the color image R with the reduced data volume and transmits the data set to the server device 30. The three-dimensional model generation unit 302 generates a three-dimensional model M using the grayscale image K with the reduced data capacity included in the data set. Then, the coloring unit 303 generates a colored three-dimensional model D, which is a color of the three-dimensional model M, using the color image R in which the data volume included in the data set is reduced.

As described above, in the three-dimensional shape model generation system 1 in the modified example 1 of the first embodiment, the capacity reduction unit 203 reduces the data capacity of the grayscale image K, and the three-dimensional model generation unit 302 A three-dimensional model of the object is generated using the grayscale image K whose data capacity has been reduced by the capacity reduction unit 203. As a result, in the three-dimensional shape model generation system 1 in the modified example 1 of the first embodiment, the data capacity of the grayscale image K and the color image R can be reduced, so that the three-dimensional model M and the colored cubic The data capacity of the data set used to generate the original model D can be reduced.

Further, in the three-dimensional shape model generation system 1 in the modified example 1 of the first embodiment, the capacity reduction unit 203 sets the grayscale image K so that the reduction ratio is larger than the reduction ratio in the color image R. The data capacity of the scale image K may be reduced. As a result, in the three-dimensional shape model generation system 1 according to the modified example 1 of the first embodiment, for example, the grayscale image K has an image size that is ½ of the image size that does not affect the accuracy of the three-dimensional model M that is generated. It is possible to reduce the data capacity of the data set by compressing the color image R into the image size smaller than 1/2 and compressing the color image R into the image size smaller than 1/2.

(Modification 2 of the first embodiment)
Next, a modified example 2 of the first embodiment will be described. This modified example is different from the above-described embodiment in that the data amount is reduced by reducing the number of images in the color image R.
In the present modification, the capacity reduction unit 203 reduces the number of images of the color images R, which is the same number as the grayscale images K. For example, when there are 100 grayscale images K and 100 color images R, the capacity reduction unit 203 leaves 100 grayscale images K and selects 100 to 10 color images R. The capacity reducing unit 203, for example, a color having a viewpoint at a position moved from the front of the object in the horizontal direction by a predetermined angle (for example, 45°) from the front of the object from among the plurality of color images R. The image R and the color image R having the viewpoint at a position horizontally moved by a predetermined angle (for example, 60°) from the front surface of the object are selected. In other words, the capacity reduction unit 203 selects the color image R having the viewpoint on each side such as the front surface, the side surface, the top surface, and the back surface of the object in a well-balanced manner. This makes it possible to color the entire three-dimensional model M while reducing the data capacity of the color image R.

FIG. 5 is a diagram illustrating a process performed by the 3D geometric model generation system 1 according to the second modification of the first embodiment.
As shown in FIG. 5, in the three-dimensional shape model generation system 1 according to the second modification of the first embodiment, first, the image information acquisition unit 202 extracts the grayscale image K and the color image R from the multi-viewpoint image T. Get (separate). Next, the capacity reduction unit 203 reduces the data capacity of the color image R by reducing the number of images of the color image R. Next, the data set generation unit 204 generates a data set including the grayscale image K and the color image R with the reduced data capacity and transmits the data set to the server device 30. The three-dimensional model generation unit 302 generates a three-dimensional model M using the grayscale image K included in the data set. Then, the coloring unit 303 generates a colored three-dimensional model D, which is a color of the three-dimensional model M, using the color image R in which the data volume included in the data set is reduced.

As described above, in the three-dimensional shape model generation system 1 according to the modified example 2 of the first embodiment, the capacity reduction unit 203 selects some color images R from the plurality of color images R, The data capacity of the color image R is reduced. As a result, the three-dimensional shape model generation system 1 according to the modified example 2 of the first embodiment selects, for example, the color image R from the plurality of color images R so that the viewpoint exists in each direction in a well-balanced manner. As a result, it is possible to color the entire three-dimensional model M and reduce the data capacity as compared with the case where all the color images R are used.

(Modification 3 of the first embodiment)
Next, a modified example 3 of the first embodiment will be described. This modification is different from the above-described embodiment in that the data capacity is reduced by deleting (trimming) a part of the image in the color image R.
In the present modification, the capacity reduction unit 203 sets, for example, a predetermined area (for example, an N×N pixel area) at the center of the image in the color image R (where N is an arbitrary natural number that does not exceed the number of pixels of the image). Retain and delete the area different from the area.

Further, the capacity reduction unit 203 may leave an area in the color image R in which the object is imaged and delete an area different from the area. In this case, for example, the capacity reduction unit 203 detects the contour line (edge) of the object in the color image R by applying a Laplacian filter or a Sobel filter to the color image R. Then, the capacity reduction unit 203 deletes the area outside the contour line detected from the color image R.

Further, the capacity reduction unit 203 may leave an area in the color image R where blurring (blurring or blurring) has not occurred in the image and delete the area in which blurring has occurred.
In this case, for example, the capacity reduction unit 203 uses a frequency analysis method to determine an area in which blurring has occurred and an area in which blurring has not occurred. The capacity reduction unit 203 performs a Fourier transform on the local area in the vicinity of the pixel for the RGB value of each pixel, and extracts the high frequency component in the local area. Then, the capacity reduction unit 203 determines whether or not blurring has occurred in the extracted local region, depending on whether or not the high frequency component in the extracted local region is equal to or greater than a predetermined threshold value. When the level of the high frequency component in the local area is equal to or higher than the predetermined threshold, the capacity reducing unit 203 determines that blur is not generated in the local area, and the level of the high frequency component in the local area is less than the predetermined threshold. In some cases, it is determined that blur has occurred in the local area.

Alternatively, the capacity reduction unit 203 may use a machine learning method to determine an area where blur occurs in an image and an area where blur does not occur in the image. For example, the capacity reduction unit 203 executes machine learning using learning data of a blur map corresponding to an input image by using a convolutional neural network (Convolutional Numerical Network, CNN), and thereby a blur degree for each pixel from the color image R. It is possible to generate a blur map indicating

FIG. 6 is a diagram illustrating a process performed by the three-dimensional shape model generation system 1 according to the modified example 3 of the first embodiment.
As shown in FIG. 6, in the three-dimensional shape model generation system 1 according to the modified example 3 of the first embodiment, first, the image information acquisition unit 202 extracts the grayscale image K and the color image R from the multi-viewpoint image T. Get (separate). Next, the capacity reduction unit 203 reduces the data capacity of the color image R by trimming the color image R. Next, the data set generation unit 204 generates a data set including the grayscale image K and the color image R with the reduced data capacity and transmits the data set to the server device 30. The three-dimensional model generation unit 302 generates a three-dimensional model M using the grayscale image K included in the data set. Then, the coloring unit 303 generates a colored three-dimensional model D, which is a color of the three-dimensional model M, using the color image R in which the data volume included in the data set is reduced.

As described above, in the three-dimensional shape model generation system 1 according to the modified example 3 of the first embodiment, the capacity reduction unit 203 reduces the data capacity by deleting a partial area of the image in the color image R. To do. As a result, the three-dimensional shape model generation system 1 according to the modified example 3 of the first embodiment is easy to leave, for example, a central N×N pixel region of an image in a multi-viewpoint image and delete other regions. Any method can reduce the data capacity of the data set.

Further, in the three-dimensional shape model generation system 1 according to the modified example 3 of the first embodiment, the capacity reduction unit 203 reduces the data capacity by deleting a range in the color image R different from the range in which the object is captured. Reduce. As a result, the three-dimensional shape model generation system 1 according to the modified example 3 of the first embodiment improves the accuracy of the generated model by deleting the information other than the target object for which the colored three-dimensional model D is generated. The data capacity of the data set can be reduced without degradation.

Further, in the three-dimensional shape model generation system 1 according to the modified example 3 of the first embodiment, the capacity reduction unit 203 reduces the data capacity by deleting the area where blurring is occurring for each pixel in the multi-viewpoint image. .. As a result, the three-dimensional shape model generation system 1 according to the modified example 3 of the first embodiment reduces the accuracy of the color three-dimensional model D to be generated by deleting the region in the color image R where the blur or blur occurs. It is possible to reduce the data capacity without doing so.

(Modification 4 of the first embodiment)
Next, a modified example 4 of the first embodiment will be described. This modification is different from the above-described embodiment in that the data capacity is reduced by deleting (trimming) a partial area of the image in the grayscale image K.
In this modification, as in the modification 3 of the first embodiment described above, the capacity reducing unit 203 leaves a predetermined area (for example, an area of N×N pixels) at the center of the image in the grayscale image K, and Delete an area that is different from the area.
Further, the capacity reduction unit 203 leaves the area in which the object is imaged in the grayscale image K and deletes the area different from the area by the same method as in the third modification of the first embodiment described above. You may
In addition, the capacity reducing unit 203 leaves a region in the grayscale image K where blurring (blurring or blurring) does not occur in the grayscale image K by the same method as that in the modification 3 of the first embodiment, and blurring occurs. The existing area may be deleted.

FIG. 7 is a diagram illustrating a process performed by the three-dimensional geometric model generation system 1 according to the modified example 4 of the first embodiment.
As shown in FIG. 7, in the three-dimensional shape model generation system 1 according to the modified example 4 of the first embodiment, first, the image information acquisition unit 202 extracts the grayscale image K and the color image R from the multi-viewpoint image T. Get (separate). Next, the capacity reduction unit 203 trims the grayscale image K and the color image R to reduce the data capacity of both images. Next, the data set generation unit 204 generates a data set including the grayscale image K and the color image R with the reduced data volume, and transmits the data set to the server device 30. The three-dimensional model generation unit 302 generates a three-dimensional model M using the grayscale image K with the reduced data capacity included in the data set. Then, the coloring unit 303 generates a colored three-dimensional model D, which is a color of the three-dimensional model M, using the color image R in which the data volume included in the data set is reduced.

As described above, in the three-dimensional geometric model generation system 1 according to the modified example 4 of the first embodiment, the capacity reduction unit 203 trims the grayscale image K to reduce the data capacity. As a result, in the three-dimensional shape model generation system 1 in the modified example 4 of the first embodiment, the data capacity of the grayscale image K can be reduced by an easy method of trimming the grayscale image K. , The data capacity of the data set used to generate the three-dimensional model M can be reduced. Further, when trimming the grayscale image K, by leaving a region in the center of the image, a region in which the object is imaged, or a region in which no blur occurs in the image, and deleting a region different from the region, It is possible to reduce the data capacity of the data set without lowering the accuracy of the three-dimensional model M to be generated.

(Second embodiment)
Next, a second embodiment will be described. The present embodiment is different from the above-described embodiments in that the client device 20 uses the color image R to color the three-dimensional model M. In the following description, configurations different from the above-described embodiments will be described, and configurations similar to the above-described embodiments will be denoted by the same reference numerals as those in the above-described embodiments and description thereof will be omitted.

FIG. 8 is a block diagram showing an example of the configuration of the 3D geometric model generation system 1A according to the second embodiment.
As shown in FIG. 8, the 3D geometric model generation system 1A includes, for example, an imaging device 10, a client device 20A, and a server device 30A.
The client device 20A includes a communication unit 201A, a data set generation unit 204A, and a coloring unit 205. The server device 30A includes a communication unit 301A.

The data set generation unit 204A generates a data set that includes the grayscale image K information but does not include the color image R information, and transmits the generated data set to the server device 30A.

The server device 30A receives the data set from the client device 20A, and generates the three-dimensional model M using the information of the grayscale image K included in the received data set. The communication unit 301A transmits the information of the three-dimensional model M generated using the information of the grayscale image K, that is, the information of the (non-colored) three-dimensional model M to the client device 20.

The communication unit 201 receives the information of the (non-colored) three-dimensional model M from the server device 30, and outputs the received information of the three-dimensional model M to the coloring unit 205.
The coloring unit 205 uses the information of the color image R and the three-dimensional model M to color the three-dimensional shape model and generate a colored three-dimensional model D. Since the function of the coloring unit 205 is the same as that of the coloring unit 303 described above, the description thereof will be omitted.

FIG. 9 is a sequence diagram illustrating the flow of processing performed by the 3D geometric model generation system 1A according to the second embodiment. Since steps S20 to S23 and S25 in the sequence diagram of FIG. 9 are the same as steps S10 to S13 and S15 in the sequence diagram of FIG. 3 described above, the description thereof will be omitted. That is, only steps S24, S26, and S27 will be described.

Step S24:
The client device 20 causes the data set generation unit 204A to generate a data set including the grayscale image K. The data set generation unit 204 transmits the generated data set to the server device 30.
Step S26:
In the server device 30, the three-dimensional model generation unit 302 generates the three-dimensional model M using the grayscale image K included in the data set, and the generated (non-colored) three-dimensional model M information is transmitted to the client device 20. Send.
Step S27:
The client device 20 receives the information of the (non-colored) three-dimensional model M through the communication unit 201A, and the coloring unit 303 uses the color image R in which the data capacity included in the data set is reduced to form the three-dimensional model M. Coloring is performed to generate a colored three-dimensional model D.

As described above, the 3D geometric model generation system 1A according to the second embodiment includes the client device 20A and the server device 30A. The client device 20A includes an image information acquisition unit 202, a capacity reduction unit 203, and a coloring unit 205, transmits grayscale image information to the server device 30A, and the server device 30A acquires the three-dimensional model M of the target object. Upon receiving the information, the coloring unit 205 colors the three-dimensional model M of the object received from the server device 30A using the information of the color image R whose data capacity has been reduced.
As a result, the three-dimensional shape model generation system 1A according to the second exemplary embodiment uses the color even when data is transmitted and received between the client device 20A and the server device 30A via the communication network 40 or the like. Since it is not necessary to send and receive the information of the image R via the communication network 40, it is possible to reduce the amount of data flowing through the communication network 40, and use the information of the color image R with the reduced data amount. By performing the coloring with the use of the color image R, it is possible to reduce the processing load associated with the generation of the colored three-dimensional model D, as compared with the case where the information of the color image R whose data capacity is not reduced is used.

All or part of the three-dimensional shape model generation system 1, 1A in the above-described embodiment may be realized by a computer. In that case, the program for realizing this function may be recorded in a computer-readable recording medium, and the program recorded in this recording medium may be read by a computer system and executed. It should be noted that the “computer system” mentioned here includes an OS and hardware such as peripheral devices. Further, the "computer-readable recording medium" refers to a portable medium such as a flexible disk, a magneto-optical disk, a ROM, a CD-ROM, or a storage device such as a hard disk built in a computer system. Further, the "computer-readable recording medium" means to hold a program dynamically for a short time like a communication line when transmitting the program through a network such as the Internet or a communication line such as a telephone line. In this case, a volatile memory inside a computer system that serves as a server or a client in that case may hold a program for a certain period of time. Further, the program may be for realizing a part of the above-mentioned functions, or may be a program for realizing the above-mentioned functions in combination with a program already recorded in a computer system, It may be realized using a programmable logic device such as FPGA.

Although the embodiment of the present invention has been described in detail above with reference to the drawings, the specific configuration is not limited to this embodiment and includes a design and the like within a range not departing from the gist of the present invention.

1, 1A... Three-dimensional shape model generation system 10... Imaging device 20, 20A... Client device 201, 201A... Communication unit 202... Image information acquisition unit 203... Capacity reduction unit 204, 204A... Data set generation unit 205... Coloring unit 30 , 30A... Server device 301, 301A... Communication unit 302... Three-dimensional model generation unit 303... Coloring unit 40... Communication network

Claims (13)

From a multi-viewpoint image that is two or more color images of an object captured from different viewpoints, grayscale image information excluding color information indicating colors in the multi-viewpoint image and the color information in the multi-viewpoint image are displayed. An image information acquisition unit that acquires information about the color image including
A capacity reduction unit that reduces the data capacity of the color image acquired by the image information acquisition unit,
Using the information of the grayscale image, a three-dimensional model generation unit that generates a three-dimensional model of the object,
A coloring unit that colors the three-dimensional model generated by the three-dimensional model generation unit using the information of the color image whose data capacity has been reduced by the capacity reduction unit,
A three-dimensional shape model generation system comprising: The capacity reduction unit reduces the data capacity of the grayscale image so that the reduction ratio, which is the ratio of the reduction amount to the amount before reduction, in the data amount of the grayscale image is smaller than the reduction ratio of the color image. Reduce,
The three-dimensional model generation unit generates a three-dimensional model of the object by using the information of the grayscale image whose data capacity has been reduced by the capacity reduction unit,
The three-dimensional shape model generation system according to claim 1, characterized in that. The capacity reduction unit reduces the image size in the color image or the grayscale image,
The three-dimensional shape model generation system according to claim 1 or 2, characterized in that. The capacity reduction unit selects a part of the color images from the plurality of color images,
The three-dimensional shape model generation system according to any one of claims 1 to 3, wherein. The capacity reduction unit deletes a partial area of the image in the color image or the grayscale image,
The three-dimensional geometric model generation system according to any one of claims 1 to 4, characterized in that The capacity reduction unit deletes a region in the color image or the grayscale image that is different from a region in which the object is captured.
The three-dimensional geometric model generation system according to claim 5, wherein. The capacity reduction unit deletes a blurring region in the color image or the grayscale image,
The three-dimensional shape model generation system according to claim 5 or 6, characterized in that. The multi-view image is an RGB image,
The image information acquisition unit acquires an image showing the brightness in the multi-viewpoint image by gray scale as the gray scale image, and acquires an image showing the color in the multi-viewpoint image by RGB values as the color image,
The three-dimensional shape model generation system according to any one of claims 1 to 7, characterized in that. The multi-view image is a Lab image,
The image information acquisition unit acquires, as the grayscale image, an image showing the brightness in the multi-viewpoint image by the L channel of the Lab image, and the image showing the color in the multi-viewpoint image by the ab channel of the Lab image in the color image. To get as
The three-dimensional shape model generation system according to any one of claims 1 to 7, characterized in that. A client device having the image information acquisition unit and the capacity reduction unit, and transmitting information of the grayscale image and information of the color image with reduced data capacity to a server device;
A server device that includes the three-dimensional model generation unit and the coloring unit, and the three-dimensional model generation unit generates a three-dimensional model of the object using the information of the grayscale image received from the client device. When,
The three-dimensional geometric model generation system according to any one of claims 1 to 9, further comprising: The image information acquisition unit, the capacity reduction unit, and the coloring unit, which transmits information of the grayscale image to a server device, receives a three-dimensional shape model of the object from the server device, and The coloring unit uses a client device that colors the three-dimensional model of the object received from the server device using the information of the color image whose data capacity has been reduced,
A server device that has the three-dimensional model generation unit, and that transmits the three-dimensional model generated using the information of the grayscale image received by the three-dimensional model generation unit from the client device to the client device;
The three-dimensional geometric model generation system according to any one of claims 1 to 9, further comprising: The image information acquisition unit, from the information indicating the multi-view image which is two or more color images obtained by capturing the object from different viewpoints, the grayscale image information excluding the color information indicating the color in the multi-view image, and An image information acquisition step of acquiring information of a color image including the color information in the multi-view image,
A capacity reduction unit, a capacity reduction process for reducing the data capacity of the color image,
A three-dimensional model generation process in which a three-dimensional model generation unit generates a three-dimensional model of the object using the information of the grayscale image,
A coloring process in which the coloring unit colors the three-dimensional model generated by the three-dimensional model generation unit using the information of the color image whose data capacity has been reduced by the capacity reduction unit;
A method for generating a three-dimensional shape model, including: Computer,
From information indicating a multi-viewpoint image that is two or more color images obtained by capturing an object from different viewpoints, information about a grayscale image excluding color information indicating colors in the multi-viewpoint image, and the multi-viewpoint image in the multi-viewpoint image. Image information acquisition means for acquiring information of a color image including color information,
Capacity reducing means for reducing the data capacity of the information of the color image,
Using the information of the grayscale image, a three-dimensional model generation means for generating a three-dimensional model of the object,
Coloring means for coloring the three-dimensional model generated by the three-dimensional model generating means using the information of the color image whose data capacity has been reduced by the capacity reducing means,
A program to operate as.