JP2022124941A - Three dimensional model generation device and method and program - Google Patents

Abstract

To provide a device, a method, and a program that create a 3D model of a subject from images of multiple cameras.SOLUTION: In a 3D model generation device 100, a silhouette image acquisition unit 10 acquires a silhouette image with a frame unit from a multi viewpoint video. A history DB 30 stores history information for model generation including a location where the 3D model is created in the past frame. An extension unit 41 of a computation area determination unit 40 determines a region obtained by extending a model generation region r in the past frame by a prescribed distance or the number of pixels therearound, to be a model computation region R in an input frame. A voxel computation unit 21 creates a 3D voxel model by the visual volume intersection method using the silhouette image that the silhouette image acquisition unit 10 has acquired, in a three dimensional model computation region. A history registration unit 50 registers a 3D model generation history for each frame as history information being referred to when determining the model computation region in the next and subsequent frames, into the history DB 30.SELECTED DRAWING: Figure 1

Description

The present invention relates to an apparatus, method, and program for generating a 3D model of an object at high speed from images captured by multiple cameras.

As an approach for generating a 3D model of an object from multiple camera images, the visual volume intersection method disclosed in Non-Patent Document 1 is widely known. In the visual volume intersection method, as shown in Fig. 14, a 3D model is generated by projecting a binary silhouette image obtained by extracting the subject area from each camera image onto a 3D space and leaving the intersection of the images. method.

The visual volume intersection method is used as one of the elemental techniques for restoring the shape of a 3D model in free-viewpoint imaging techniques such as those disclosed in Non-Patent Document 2. Free-viewpoint video technology is a technology that reconstructs 3D space from images from multiple cameras and enables viewing of images from angles where there are no cameras.

The smallest unit that constitutes a 3D model generated by the visual volume intersection method is called a voxel. A voxel is a cube with a small volume, and when generating a 3D model by the visual volume intersection method, a voxel grid is defined by filling the entire 3D area where the 3D model is created with the aforementioned cubes, and a model is created for each voxel grid. A determination is made as to whether or not it is generated.

Assuming that the length of one side of this cube (unit voxel size) is M, the larger the unit voxel size M is set, the more discrete the 3D space will be treated. is discretized, a rough 3D model is generated.

On the other hand, the smaller the unit voxel size M, the finer the shape can be restored, but the increase in the calculation unit increases the processing time explosively. In particular, when considering application to free-viewpoint video, calculation time tends to increase because voxels are generated in a wide space such as a sports stadium.

In order to solve such technical problems, Non-Patent Document 3 and Patent Document 1 disclose techniques for speeding up the processing of the visual volume intersection method. In Non-Patent Document 3, when generating a 3D voxel model by the visual volume intersection method, as shown in FIG. Get the bounding box in 3D space.

After that, we succeeded in greatly reducing the processing time by modeling the inside of each 3D bounding box using the visual volume intersection method with a small unit voxel size Ma (< Mb).

In addition, as a technique similar to Patent Document 1, a rough 3D voxel model is estimated for the 3D space in which the 3D model is created, a finer voxel grid is placed at the generation position of the rough 3D voxel model, and this voxel grid A technique for speeding up 3D model generation is disclosed by repeating the process of generating a 3D model again using the visual volume intersection method.

In Patent Document 2, in order to prevent the generation of a virtual image object in the visual volume intersection method, after estimating the object position by the visual volume intersection method as the first step and acquiring the object candidate, as the second step, the object is estimated for the candidate A model generation method based on two steps is disclosed to improve the model quality by determining whether it is a virtual object or not, and deleting the virtual object. Also disclosed is a mechanism that refers to the object generation position of the past frame at this time, and makes it difficult to determine that the object is a virtual image when the object is close to the reference position.

Non-Patent Document 4 discloses an invention for speeding up generation by balancing calculation loads when calculating a 3D model in a PC cluster.

JP 2018-063635 A Patent No. 5454573 Patent application No. 2019-153696

A. Laurentini, "The visual hull concept for silhouette based image understanding.", IEEE Transactions on Pattern Analysis and Machine Intelligence, 16, 150-162, (1994). J. Kilner, J. Starck, A. Hilton and O. Grau, "Dual-Mode Deformable Models for Free-Viewpoint Video of Sports Events," Sixth International Conference on 3-D Digital Imaging and Modeling (3DIM 2007), pp. 177-184, (2007). J. Chen, R. Watanabe, K. Nonaka, T. Konno, H. Sankoh, S. Naito, "A Fast Free-viewpoint Video Synthesis Algorithm for Sports Scenes", 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems ( IROS 2019), WeAT17.2, (2019). Tomomi Iwashita, Ryo Kurazume, Kenji Hara, Seiichi Uchida, Kenichi Morooka, Tsutomu Hasegawa, "High-speed reconstruction of 3D shape of real objects by parallel fast level set method", Proceedings of the 2016 Annual Conference on Robotics and Mechatronics, 2P1-C13, (2006). C. Stauffer and W. E. L. Grimson, "Adaptive background mixture models for real-time tracking," 1999 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 246-252, Vol. 2, (1999). J. Ruttle, M. Manzke and R. Dahyot, "Estimating 3D Scene Flow from Multiple 2D Optical Flows," 2009 13th International Machine Vision and Image Processing Conference, Dublin, 2009, pp. 1-6, doi: 10.1109/IMVIP. 2009.8. Arun, Somani; Thomas S. Huang; Steven D. Blostein, "Least-square fitting of two 3-D point sets". IEEE Pattern Analysis and Machine Intelligence, (1987).

Although speed-up methods such as non-patent document 3 and patent document 1 are effective, the experiments shown in non-patent document 3 show only real-time 3D model generation with a volleyball court size at the maximum. For example, it has not been verified whether real-time calculation is possible while maintaining quality in a wide area such as a soccer stadium.

Considering the application of free-viewpoint video technology, it is possible to convert sports stadiums into free-viewpoints, generate and view replay videos of arbitrary camerawork, and interactively view free-viewpoints from the viewpoint that the user wants to view according to the operation of each user. A use case of enjoying watching video is conceivable. In other words, real-time generation of 3D models in wide-area spaces such as soccer and baseball stadiums is essential.

By increasing the unit voxel size Ma and Mb, there is a possibility that real-time performance can be ensured, but increasing Ma and Mb invites deterioration in quality.

In addition, when a server with high computational processing ability is used, the cost (expense) of the server increases, which hinders practical use. In addition, in a use case such as 3D modeling of the entire stadium, the area is not only the generation time of fine voxels generated with the unit voxel size Ma, but also the generation time of coarse voxels generated with the unit voxel size Mb. It tended to increase due to its large size.

In addition, although the speed-up method using a PC cluster as in Non-Patent Document 4 is effective in speeding up, there is a problem of increased cost due to the need to prepare a large number of PCs.

The object of the present invention is to solve the above technical problems, and to estimate the movement direction and amount of movement of the target object between consecutive frames based on the generated positions of the 3D models of the past frames, and generate the 3D model in the input frame. 3D model generation is speeded up by determining the generation position and narrowing down the area where 3D model calculation should be performed.

In order to achieve the above object, the present invention provides a 3D model generation device for generating a 3D model of a subject based on silhouette images extracted in units of frames from moving images captured by a plurality of cameras with different viewpoints, wherein: It is characterized by having a structure.

(1) Means for storing a model generation history including the generation position of each 3D model in the past frame, means for determining a model calculation area in an input frame based on the model generation history, and 3D model calculation for the model calculation area and generating a 3D model.

(2) comprising means for classifying input frames into keyframes or non-keyframes, wherein said means for generating a 3D model comprises means for generating a relatively high resolution 3D model and generating a relatively low resolution 3D model; In the key frame, a high resolution 3D model is generated in the area where the low resolution 3D model was generated based on the silhouette image, and in the non-key frame, the 3D model calculation is performed for the model calculation area. Generates high resolution 3D models.

(3) In non-keyframes, a high-resolution 3D model is generated by performing 3D model calculation on the generation area of the low-resolution 3D model generated by performing 3D model calculation on the model calculation area. .

(4) As for the means for determining the model calculation area, the area obtained by extending the 3D model generation position in the past frame is determined as the model calculation area.

(5) A means for classifying each 3D model into classes is provided, and a means for determining the model calculation area expands the 3D model generation position in the past frame by an expansion amount corresponding to the class.

(6) The classification means classifies each 3D model into each class based on the moving speed assumed for the subject, and the means for determining the model calculation area is such that the faster the moving speed of the subject class, the larger the expansion amount. made it bigger.

(7) means for estimating the velocity field of each 3D model based on the generation history of each 3D model in a plurality of past frames; The model calculation area is determined.

(8) The number of silhouette images used to generate the low-resolution 3D model in the non-keyframes is smaller than the number of silhouette images used to generate the low-resolution 3D model in the keyframes.

According to the present invention, the following effects are achieved.

(1) The model calculation area in the input frame is determined based on the generation position of each 3D model in the past frame, and the 3D model calculation is performed only for that model calculation area, so the 3D model can be generated at high speed. .

(2) In keyframes, high-resolution 3D models are generated only in areas where low-resolution 3D models were generated based on silhouette images, while in non-keyframes, 3D model calculations are performed for model calculation areas. It produces high-resolution 3D models, which allows for faster 3D model generation, especially in non-keyframes, while maintaining quality.

(3) In non-keyframes, a high-resolution 3D model is generated by performing 3D model calculation on the low-resolution 3D model generation area generated by performing 3D model calculation on the model calculation area. can more accurately determine the model computational domain in

(4) Since the model calculation area is determined by extending the 3D model generation position in the past frame, the model calculation area can be determined by calculation with a light processing load, and the 3D model can be generated at high speed.

(5) Each 3D model is classified into classes, and the 3D model generation position in the past frame is expanded by the expansion amount according to the class, so that the model calculation area can be determined in an appropriate range without excess or deficiency for each 3D model. become.

(6) Each 3D model is classified into each class based on the moving speed assumed for the subject, and the class of the subject with faster moving speed has a larger expansion amount, so there is a difference in the moving speed of the 3D model. Even in such a case, the model calculation area can be determined in an appropriate range without excess or deficiency.

(7) The velocity field of each 3D model is estimated based on the generation history of each 3D model in multiple past frames, and the model calculation area is determined based on the velocity field for each 3D model. Even if there is a difference in the movement direction, the model calculation area can be determined in an appropriate range without excess or deficiency.

(8) Since the number of silhouette images used to generate low-resolution 3D models in non-keyframes is less than the number of silhouette images used to generate low-resolution 3D models in keyframes, It becomes possible to more accurately determine the model calculation region while minimizing the increase in processing load.

1 is a functional block diagram showing the configuration of a 3D model generation device according to a first embodiment of the present invention; FIG. FIG. 10 is a diagram showing an example of determining a 3D model calculation area by expanding a 3D model generation area in a past frame; FIG. 4 is a functional block diagram showing the configuration of a 3D model generation device according to a second embodiment of the present invention; FIG. 4 is a diagram showing an example of dividing each 3D model into 3D bounding boxes; FIG. 10 is a diagram showing an example of changing the expansion amount according to the class of the 3D model; FIG. 10 is a functional block diagram showing the configuration of a 3D model generation device according to a third embodiment of the present invention; It is the figure which showed the example which determines a model calculation area|region based on a velocity field. FIG. 10 is a functional block diagram showing the configuration of a 3D model generation device according to a fourth embodiment of the present invention; FIG. 10 is a diagram (part 1) showing an example in which a 3D model generation method is varied according to frames; FIG. 10 is a diagram (part 2) showing an example in which a 3D model generation method is varied according to frames; FIG. 10 is a diagram showing an example of determining the model calculation area of each 3D model in non-keyframes based on the expansion amount according to the class; FIG. 10 is a diagram showing an example of determining a model calculation region of each 3D model in non-keyframes based on a velocity field; FIG. 10 is a functional block diagram showing the configuration of a 3D model generation device according to a fifth embodiment of the present invention; FIG. 4 is a diagram showing a method of generating a 3D model by the visual volume intersection method; FIG. 4 is a diagram showing an example of generating a 3D model in two stages;

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. FIG. 1 is a functional block diagram showing the configuration of the main parts of a 3D model generation device 100 according to the first embodiment of the present invention. , a computational domain determination unit 40 and a history registration unit 50 are main components.

Such a 3D model generation device 100 can be configured by installing an application (program) that implements each function in at least one general-purpose computer or server. Alternatively, a part of the application can be configured as a dedicated machine or a single-function machine that is made into hardware or software. In this embodiment, an example will be described in which a 3D model is generated for each subject by photographing a sports scene with N cameras Cam1 to CamN.

The silhouette image acquisition unit 10 acquires silhouette images used for the visual volume intersection method in units of frames from a plurality of camera images (multi-viewpoint images) obtained by photographing a plurality of subjects from different viewpoints. It is desirable to acquire silhouette images from two or more cameras in order to form a 3D model by the visual volume intersection method. The silhouette image is acquired in the form of a binary mask image in which the subject area for which the 3D model is to be generated is represented in white, and the other areas are represented in black. Such a silhouette image can be calculated using a conventional technique such as the background subtraction method disclosed in Non-Patent Document 5.

The history DB 30 stores a model generation history including information on positions at which 3D models were generated in past frames before the current input frame. The computational region determination unit 40 determines the model computational region in the input frame based on the model generation history of past frames.

In this embodiment, the computational region determination unit 40 includes an expansion unit 41, and as shown in FIG. 2, for example, a predetermined distance or number of voxels (hereinafter referred to as , collectively referred to as an expansion amount P) is determined as the model calculation region R in the input frame. Although FIG. 2 is illustrated in two dimensions for the sake of convenience, expansion based on the expansion amount P is actually performed in a three-dimensional space.

In the 3D model generation unit 20, the voxel model calculation unit 21 arranges a voxel grid with a unit voxel size Ma suitable for generating a 3D model that satisfies the required quality in the three-dimensional model calculation area determined by the calculation area determination unit 40. Then, a high-resolution 3D voxel model is generated by the visual volume intersection method using the silhouette image acquired by the silhouette image acquisition unit 10 .

Although a voxel model is volume data formed by a large number of voxels, it is generally more convenient to handle 3D model data as a polygon model in many cases. Therefore, in the present embodiment, a 3D model output unit 22 is provided in the 3D model generation unit 20, and a voxel model is converted into a polygon model using a technique for converting a voxel model into a polygon model, such as the marching cube method. I am trying to output the model.

The history registration unit 50 registers the 3D model generation history obtained for each frame in the history DB 50 as history information to be referred to when determining the model calculation area in subsequent frames.

According to this embodiment, the model calculation area in the input frame is determined based on the generation position of each 3D model in the past frame, and the 3D model calculation is performed only for the model calculation area, so the 3D model is generated at high speed. become able to.

In addition, according to this embodiment, the 3D model generation position in the past frame is expanded to determine the model calculation area, so the model calculation area can be determined by calculation with a light processing load, and the 3D model can be generated even faster. become able to.

FIG. 3 is a functional block diagram showing the configuration of the second embodiment of the present invention, and since the same reference numerals as above denote the same or equivalent parts, description thereof will be omitted.

In the present embodiment, the history registration unit 50 includes a classification unit 51 that classifies the 3D model generated based on the silhouette image into classes based on, for example, the general movement speed thereof, and the expansion unit 41 classifies the 3D model for each 3D model It is characterized in that the expansion amount P when determining the model calculation region R by expanding the model generation region r is adaptively determined based on the result of the class classification.

The classification unit 51 classifies each 3D model generated for each frame into classes such as “person” and “ball”, and registers the classification result together with the generation position of each 3D model in the history DB 30 as a model generation history. In order to perform class classification processing, 3D models must be input separately for each subject. will be Alternatively, as shown in FIG. 4, class classification may be performed based on the 3D bounding box that contains each 3D model.

For class classification, (1) class classification based on the size of the 3D model, (2) class classification based on the position of the 3D model, (3) class classification based on deep learning, etc. can be applied.

(1) Classification based on 3D model size Each object can be classified based on the size and shape (overall size, length, width, height) of its 3D voxel model. For example, when classifying subjects into "person" and "ball", which are often seen in sports scenes, if the voxel model is larger than a predetermined threshold, it can be classified as "person", and if it is smaller, it can be classified as "ball". Alternatively, if the shape of the 3D bounding box is a cube, it can be classified as a "ball", and if it is a rectangular parallelepiped, it can be classified as a "person".

(2) Classification based on the position of the 3D model For example, a 3D model formed at a position exceeding 10m in height can be classified as a ball, and the probability of being a person or equipment can be underestimated.

(3) Classification based on deep learning, etc.
Utilizing the fact that the shape of the 3D model is characteristic for each subject, the relationship between the model shape and the subject is learned in advance by deep learning, etc., and a prediction model is constructed, and each 3D model is applied to the prediction model. Classification can be performed with

Each of the above classification methods may be used alone, or a plurality of classification methods may be combined as appropriate.

FIG. 5 is a diagram schematically showing an example of adaptively determining the extension amount P when the extension unit 41 determines the model calculation region R according to the class of the 3D model. In this embodiment, when the 3D models are generated, the classification unit 51 classifies each 3D model into "person" or "ball". In general, the moving speed of the "ball" is higher than the moving speed of the "person". is larger than the expansion amount of .

According to this embodiment, each 3D model is classified into classes, and the 3D model generation position in the past frame is expanded by the expansion amount according to the class. to be determined.

Further, according to the present embodiment, each 3D model is classified into each class based on the moving speed assumed for the subject, and the class of the subject with faster moving speed has a larger amount of expansion, so that the 3D model Even if there is a difference in movement speed, the model calculation area can be determined in an appropriate range that is neither excessive nor deficient.

FIG. 6 is a functional block diagram showing the configuration of the third embodiment of the present invention, and since the same reference numerals as above denote the same or equivalent parts, description thereof will be omitted.

In this embodiment, a velocity field estimating unit 42 is provided in the computational region determining unit 40, and as shown in FIG. 7, the moving field is estimated for each 3D model based on the change in the 3D model generation position between past frames, It is characterized in that the model calculation region in the input frame is determined based on this estimation result.

The velocity field estimation unit 42 calculates the 3D velocity field (v _x , v _y , v _z ). For estimating the 3D velocity field, a technique for reconstructing a 3D optical flow from the 2D optical flow of each camera, as disclosed in Non-Patent Document 6, for example, can be used.

In addition, 3D point cloud data is formed by assuming the voxel formation positions between the past frames as points, and point cloud data represented by the ICP (Iterative Closest Point) method disclosed in Non-Patent Document 7 is generated for each 3D model of the subject. Assuming that the voxel moves to the aligned position using the alignment method of (1), the velocity field of the voxel may be calculated by (position after movement) - (position before movement).

Alternatively, the 3D bounding boxes of each 3D model are tracked between the frames before and after, for example, by associating the closest 3D bounding boxes, and from the center of gravity of each 3D bounding box in the previous frame, the corresponding The movement vector to the centroid position of each 3D bounding box may be estimated as the velocity field of all voxels within the 3D bounding box of the subsequent frame.

The expansion unit 41 increases the expansion amount P when determining the model calculation area for a 3D model with a larger velocity field.

At this time, as in the second embodiment, the history registration unit 50 is provided with the classification unit 51 to classify each 3D model or its 3D bounding box into classes such as "person" and "ball", and A different velocity field may be calculated for each class by associating the closest 3D model or its 3D bounding box in the class.

According to this embodiment, the velocity field of each 3D model is estimated based on the generation history of each 3D model in a plurality of past frames, and the model calculation area is determined for each 3D model based on the velocity field. Even if there is a difference in the movement speed and movement direction of the model, the model calculation area can be determined in an appropriate range that is neither excessive nor deficient.

FIG. 8 is a functional block diagram showing the configuration of the fourth embodiment of the present invention, and since the same reference numerals as above denote the same or equivalent parts, description thereof will be omitted.

In this embodiment, the 3D model generation unit 20 serves as the voxel model calculation unit 21, a high resolution model generation unit 21a that generates a relatively high resolution 3D voxel model, and a low resolution model that generates a low resolution 3D voxel model. It is characterized in that it is provided with a generation unit 21b, and the model generation units 21a and 21b are selectively used or combined according to the cycle and type of input frames to generate a 3D voxel model.

Similar to the voxel model calculation unit 21 of the first embodiment, the high-resolution model generation unit 21a uses the silhouette image acquired by the silhouette image acquisition unit 10 in a three-dimensional space in which voxel grids with a unit voxel size of Ma are arranged. A high-resolution 3D voxel model is generated by the visual volume intersection method.

The low-resolution model generation unit 21b creates a low-resolution 3D image by the visual volume intersection method using the silhouette image acquired by the silhouette image acquisition unit 10 in a three-dimensional space in which voxel grids having a unit voxel size of Mb (>Ma) are arranged. Generate a voxel model.

The input frame identification unit 23 identifies whether the current input frame is a key frame or a non-key frame.

In this embodiment, as shown in FIG. 9, in the key frame, the low-resolution model generation unit 21b arranges a voxel grid with a unit voxel size of Mb in the 3D space for free-viewpoint production, and performs visual volume intersection using a silhouette image. A low-resolution 3D voxel model is generated by the method.

Next, the high-resolution model generation unit 21a arranges a voxel grid with a unit voxel size Ma only in the generation region of the low-resolution 3D voxel model or its 3D bounding box. Generate resolution 3D voxel models.

On the other hand, in non-key frames, the high-resolution model generation unit 21a creates a voxel grid with a unit voxel size of Ma only in the model calculation region determined by the calculation region determination unit 40 based on the generation history of the 3D model in the past frame. and generate a high-resolution 3D voxel model by the visual volume intersection method using the silhouette image.

The input frame identification unit 23 defines each frame image, for example, in the input order, as a key frame at a rate of one frame in a plurality of frames, and defines the other frame images as non-key frames. Then, as shown in FIG. 10, after generating a low-resolution 3D voxel model for keyframes, a high-resolution model is generated only for the generation area of the model, while for non-keyframes, the same as in the first embodiment. , generate a high-resolution model only for the model calculation area determined based on the model generation history.

According to this embodiment, in keyframes, high-resolution 3D models are generated only in areas where low-resolution 3D models have been generated based on silhouette images, while in non-keyframes, 3D model calculations are performed for model calculation areas. Since it generates a high-resolution 3D model, it is possible not only to speed up 3D model generation in two stages in keyframes, but also in 3D model generation in non-keyframes, while maintaining the quality. become.

When the history registration unit 50 includes the classification unit 51 as in the second embodiment, as shown in FIG. 11, the high-resolution model generation unit 21a generates a Generate a 3D model in the model computational domain that is adaptively expanded according to the

In addition, when the computational region determining unit 40 includes the velocity field estimating unit 42 in the same manner as in the third embodiment, as shown in FIG. You can use the results.

Furthermore, as in the fifth embodiment shown in FIG. 13, a low-resolution 3D model is generated for the model calculation region predicted by the calculation region determination unit 40 in non-key frames, and a low-resolution 3D voxel model or its The high-resolution model generation unit 21a may generate the high-resolution 3D model only for the generation area of the 3D bounding box.

At this time, if the low-resolution model generating unit 21b generates a low-resolution 3D model with the unit voxel size set to Mb in the same way as low-resolution 3D model generation in key frames, the calculation load may increase. Therefore, a voxel grid having a unit voxel size of Mc (>Mb) may be arranged in the predicted model calculation area, and a 3D model with a lower resolution may be generated.

Alternatively, the number of silhouette images (the number of cameras) used for model generation may be made smaller than that for low-resolution 3D model generation in non-key frames, while the unit voxel size remains Mb.

According to this embodiment, in the non-keyframes, the 3D model calculation is performed for the model calculation area, and the 3D model calculation is performed for the low-resolution 3D model generation area to generate the high-resolution 3D model. , the model computational region in non-keyframes can be more accurately determined.

In the present embodiment, it is determined for each voxel how many cameras out of the cameras that used the silhouette image when generating the low-resolution 3D model in the model calculation area in the non-key frame, the silhouette image was the foreground. , the model generation likelihood can be calculated. For example, for voxels that are the foreground in the silhouette images of 7 cameras out of 8 cameras, the model generation likelihood L _model is recorded as 7/8 (=0.875).

Furthermore, for each voxel of the low-resolution 3D model generated in the model calculation area in the non-keyframes, the distance from the 3D model generation area in the past frame (for example, the previous frame) is calculated, for example, the center of gravity of the 3D model generation area or the recent Measured as the distance from the neighboring voxels, the history-based generation likelihood L _history is calculated based on the reciprocal of the distance. At this time, the maximum likelihood (=1) can be given to voxels that overlap with the 3D model generation area.

Then, for each voxel of the low-resolution 3D model, only voxel regions where the model generation likelihood L _model and/or the history-based generation likelihood L _history exceed a preset threshold value T1, or as in the following equation (1) Only voxel regions in which the sum of the model generation likelihood L _model and the history-based generation likelihood L _history exceeds a preset threshold value T2 may be determined as the high-resolution model calculation region.

L _model + L _history ＞T2 (1)

Furthermore, according to each of the above embodiments, it is possible to provide high-quality 3D models of subjects in real time even via communication infrastructure, thereby providing diverse entertainment to many people beyond geographical or economic disparities. be able to provide. As a result, the UN-led Sustainable Development Goals (SDGs) include Goal 9 “Build resilient infrastructure and promote inclusive and sustainable industrialization” and Goal 11 “Make cities inclusive, safe and resilient.” and make it sustainable.

10... Silhouette image acquisition unit, 20... 3D model generation unit, 21... Voxel model calculation unit, 21a... High resolution model generation unit, 21b... Low resolution model generation unit, 22... 3D model output unit, 23... Input frame identification unit , 30... History database, 40... Calculation region determination unit, 41... Extension unit, 42... Velocity field estimation unit, 50... History registration unit, 51... Classification unit, 100... 3D model generation device

Claims (14)

A 3D model generation device that generates a 3D model of a subject based on silhouette images extracted frame by frame from moving images captured by multiple cameras with different viewpoints,
means for storing a model generation history including the generation position of each 3D model in past frames;
means for determining a model calculation region in an input frame based on the model generation history;
and means for generating a 3D model by performing 3D model calculation on the model calculation area. comprising means for classifying input frames as keyframes or non-keyframes;
The means for generating the 3D model comprises:
comprising means for generating a relatively high resolution 3D model and means for generating a low resolution 3D model;
In the key frame, a high resolution 3D model is generated in the area where the low resolution 3D model was generated based on the silhouette image,
2. The 3D model generating apparatus according to claim 1, wherein, in non-key frames, 3D model calculation is performed on the model calculation area to generate a high-resolution 3D model. In the non-keyframes, a high-resolution 3D model is generated by performing 3D model calculation on a low-resolution 3D model generation area generated by performing 3D model calculation on the model calculation area. 3. The 3D model generation device according to claim 2. 4. The 3D model generating apparatus according to any one of claims 1 to 3, wherein the means for determining the model calculation area determines an area obtained by extending a 3D model generation position in a past frame as the model calculation area. comprising means for classifying each 3D model into classes;
5. The 3D model generation device according to claim 4, wherein the means for determining the model calculation area expands the 3D model generation position in the past frame by an expansion amount according to the class. The means for classifying classifies each 3D model into each class based on a moving speed assumed for the subject,
6. The 3D model generating apparatus according to claim 5, wherein the means for determining the model calculation area increases the expansion amount for a subject class with a faster moving speed. Equipped with means for estimating the velocity field of each 3D model based on the generation history of each 3D model in a plurality of past frames,
7. The 3D model generating apparatus according to any one of claims 1 to 6, wherein the means for determining the model calculation area determines the model calculation area for each 3D model based on its velocity field. In the non-keyframes, the number of silhouette images used to generate a low-resolution 3D model for the model calculation area is smaller than the number of silhouette images used to generate a low-resolution 3D model in the keyframes. 4. The 3D model generation device according to claim 3, characterized by: wherein, in the non-keyframes, a unit voxel size for generating a low-resolution 3D model targeting the model calculation area is larger than a unit voxel size for generating a low-resolution 3D model in the keyframes; 4. The 3D model generation device according to claim 3. In the non-keyframe,
For each voxel of the low-resolution 3D model generated for the model calculation area, calculation of model generation likelihood based on how many cameras it was in the foreground, and based on the reciprocal of the distance from the model generation position in the past frame Calculating at least one of a history-based generation likelihood, and determining a high-resolution 3D model calculation region based on at least one of the model generation likelihood and the history-based generation likelihood 3. The 3D model generation device according to 3. In a 3D model generation method in which a computer generates a 3D model of a subject based on silhouette images extracted frame by frame from moving images shot by a plurality of cameras with different viewpoints,
Store the model generation history including the generation position of each 3D model in the past frame,
determining a model calculation region in an input frame based on the model generation history;
A 3D model generation method, wherein a 3D model is generated by performing 3D model calculation on the model calculation area. classifies input frames as keyframes or non-keyframes,
In the key frame, a high resolution 3D model is generated in the area where the low resolution 3D model was generated based on the silhouette image,
12. The 3D model generation method according to claim 11, wherein, in non-key frames, 3D model calculation is performed on the model calculation area to generate a high-resolution 3D model. In a 3D model generation program that generates a 3D model of a subject based on silhouette images extracted frame by frame from videos taken with multiple cameras with different viewpoints,
a procedure for storing a model generation history including the generation position of each 3D model in past frames;
determining a model calculation region in an input frame based on the model generation history;
A 3D model generation program that causes a computer to execute a procedure for performing 3D model calculations on the model calculation area and generating a 3D model. including a procedure for classifying input frames as keyframes or non-keyframes;
The procedure for generating the 3D model includes:
In the key frame, a high resolution 3D model is generated in the area where the low resolution 3D model was generated based on the silhouette image,
14. The 3D model generation program according to claim 13, wherein, in non-key frames, 3D model calculation is performed on the model calculation area to generate a high-resolution 3D model.

Images