JP7319939B2 - Free-viewpoint video generation method, device, and program - Google Patents

Description

The present invention relates to a method, apparatus, and program for generating a free-viewpoint video based on a plurality of camera images with different viewpoints. The present invention relates to a free-viewpoint video generation method, apparatus, and program that realize mapping.

Free-viewpoint video technology is a technology that enables video viewing from any viewpoint, including viewpoints where no camera exists, based on videos from a plurality of cameras with different viewpoints. As a method for realizing a free viewpoint video, there is a 3D model-based free viewpoint video generation method based on the visual volume intersection method disclosed in Non-Patent Document 1.

In the visual volume intersection method, as shown in Fig. 10, using a binary silhouette image obtained by extracting only the part of the subject from the video of each camera cam, the silhouette image of each camera cam is projected into a 3D space to obtain a visual volume This is a method of generating a 3D model by obtaining , and leaving only the intersection of the parts as a 3DCG model.

Such visual volume intersection methods include the full model method free viewpoint (= a method that faithfully expresses the shape of a 3D model) disclosed in Non-Patent Document 2, and the billboard method free viewpoint disclosed in Non-Patent Document 3. It is used as a basic technology to realize (= a method of creating a 3D model in the shape of a board called a billboard and mapping textures from a nearby camera onto the billboard).

As a silhouette image extraction method for obtaining a product set used in the visual volume intersection method, a method based on the background subtraction method represented by Non-Patent Document 4 is known. The background subtraction method is a method of extracting a subject based on the difference between a model in which the subject does not exist, called a background model, and an input image.

By the way, in sports scenes, for example, structures that do not move on the field (for example, goalposts in soccer or nets in volleyball) may appear. When applying the visual volume intersection method using silhouette images obtained by background subtraction-based silhouette extraction, such structures may adversely affect the quality of the free viewpoint.

For example, when a structure such as a goalpost overhangs a subject such as an athlete, the background subtraction method cannot extract a silhouette because the structure is stationary because it is determined as a background.

In the visual volume intersection method, the voxel grid is modeled when pixels of the silhouette image corresponding to the voxel grid for which it is determined whether or not to be modeled are determined to be the foreground in many cameras. If the number of cameras used as the threshold for determining the foreground decreases, it becomes easier to create a 3D model of an incorrect part. many. Therefore, if the silhouette is lost due to the structure, as shown in FIG. 11, the subject existing on the back side of the structure as seen from a certain camera may be lost.

Such technical problems tend to appear in silhouette extraction using the background subtraction method. However, there is a possibility that the portion shielded by the structure may not be extracted as a silhouette, and the method is not limited to the background subtraction method.

In order to solve such a technical problem, Patent Document 1 discloses that a silhouette image of a structure that shields a subject, such as a soccer goalpost (hereinafter sometimes referred to as a "shielding object silhouette image"), is captured for each camera. , and by performing the visual volume intersection method using the integrated silhouette image obtained by adding the silhouette image of the shielding object to the silhouette image of the subject obtained by the background subtraction method, it is possible to generate a 3D model without defects due to the shielding object. making it possible.

However, the 3D model of the goal post is also modeled by the visual volume intersection method using the integrated silhouette image. When the goal post is modeled, for example, when realizing the billboard free viewpoint of Non-Patent Document 3, a person who is in contact with the goal post model is integrated with the goal post model to generate a huge billboard. , there is a problem that the error of the display position of the subject becomes large.

In other words, in the billboard free viewpoint, a 3D object is labeled for each block of the model generated by the visual volume intersection method for the convenience of expressing by standing a board called a billboard at the position of the subject, and according to each block A billboard is formed. When the subject touches a huge structure, etc., the subject and the model of the structure are treated as one large mass and put together on one billboard.

Since this billboard rotates around the center of the board so as to face the user's selected viewpoint, it gives a sense of incongruity as if the structure and the person are rotating while attached to each other. In addition, the display position of the person changes significantly at the moment when the clump is eliminated, which causes a sense of incongruity.

In addition, in the visual volume intersection method using the integrated silhouette image, the goal post model is formed for each frame, so the data size of the 3D model increases.

In order to address such a technical problem, Patent Document 1 discloses a function of deleting the goalposts modeled by the visual volume intersection method after the 3D space is modeled by the visual volume intersection method. According to Patent Document 1, it is possible to reconstruct the 3D shape of an object without defects even when the object is obscured by an obstacle.

If the 3D model of a structure is deleted, the structure that should exist in the 3D space will no longer exist. This kind of implementation makes it possible to display a 3D model with a more accurate shape than using a structure model derived from the visual volume intersection method.

Japanese Patent Application Laid-Open No. 2019-106170

Laurentini, A. "The visual hull concept for silhouette based image understanding.", IEEE Transactions on Pattern Analysis and Machine Intelligence, 16, 150-162, (1994). J. Kilner, J. Starck, A. Hilton and O. Grau, "Dual-Mode Deformable Models for Free-Viewpoint Video of Sports Events," Sixth International Conference on 3-D Digital Imaging and Modeling (3DIM 2007), Montreal, QC, 2007, pp. 177-184. H. Sankoh, S. Naito, K. Nonaka, H. Sabirin, J. Chen, "Robust Billboard-based, Free-viewpoint Video Synthesis Algorithm to Overcome Occlusions under Challenging Outdoor Sport Scenes", Proceedings of the 26th ACM international conference on Multimedia, pp. 1724-1732, (2018) C. Stauffer and W. E. L. Grimson, "Adaptive background mixture models for real-time tracking," 1999 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 246-252 Vol. 2 (1999). D. Bolya, C. Zhou, F. Xiao, Y. J. Lee, "YOLACT: Real-Time Instance Segmentation", The IEEE International Conference on Computer Vision (ICCV), pp. 9157-9166, (2019). L. A. Lim and H. Y. Keles, "Learning multi-scale features for foreground segmentation," Pattern Analysis and Applications, pp. 1-12, (2019). Qiang Yao, Hiroshi Sankoh, Nonaka Keisuke, Sei Naito. "Automatic camera self-calibration for immersive navigation of free viewpoint sports video," 2016 IEEE 18th International Workshop on Multimedia Signal Processing (MMSP), 1-6, 2016. J. Chen, R. Watanabe, K. Nonaka, T. Konno, H. Sankoh, S. Naito, "A Fast Free-viewpoint Video Synthesis Algorithm for Sports Scenes", 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems ( IROS 2019), WeAT17.2, (2019).

Patent Literature 1 only discloses a mechanism relating to 3D model generation (processing for obtaining the shape of a 3D model), and does not disclose a method of texture mapping that considers obstructions.

Taking a soccer goal post as an example of a shield, it is necessary to prevent the texture of the goal post from being reflected in the human model behind the goal post. However, when texture mapping is performed using the mechanism disclosed in Patent Document 1, the texture of the goal post is mapped onto the 3D model of the person.

In addition, Patent Literature 1 does not disclose a mechanism for automatically generating shielding object silhouette images. In general, when a large number of cameras are used, manually creating silhouette images of shielding objects poses a significant problem in terms of human resources, so an automatic generation solution is essential.

An object of the present invention is to provide a free-viewpoint video generation method, apparatus, and program capable of solving the above technical problems, generating a 3D model without defects in the occlusion part, and realizing appropriate texture mapping to the occlusion part. to do.

In order to achieve the above object, the present invention provides a free-viewpoint video generating apparatus for generating a free-viewpoint video based on camera images of a subject and a shield that are synchronously captured by a plurality of cameras with different viewpoints, and has the following configuration. It is characterized by the fact that it is equipped.

(1) The present invention comprises means for generating a subject silhouette image for each camera, means for generating a shielding object silhouette image for each camera, and an integrated silhouette image by integrating the silhouette images of the subject and the shielding object for each camera. , means to generate an integrated 3D model by visual volume intersection method using each integrated silhouette image, and registration of whether each part of the integrated 3D model is visible or invisible from the viewpoint of each camera. The method includes means for generating occlusion information, and means for mapping textures acquired by a visible camera for each part of the integrated 3D model based on the occlusion information to a part invisible with a camera. There is a first feature.

(2) The present invention further comprises means for subtracting the 3D model of the occluder from the integrated 3D model, and the means for mapping uses the occlusion information for each part of the integrated 3D model from which the 3D model of the occluder has been reduced. The second feature is that the texture is mapped by using

According to the present invention, the following effects are achieved.

(1) Since the present invention has the first feature, in addition to being able to generate a 3D model with no loss considering the obstructing object, it is possible to perform texture mapping considering the obstruction caused by the existence of the obstructing object. , it is possible to generate a free-viewpoint video with excellent quality.

(2) Since the present invention has the second feature, it is possible to reduce the amount of 3D model data. You can prevent the phenomenon of rotating a huge billboard that is still integrated.

1 is a functional block diagram showing the configuration of required parts of a free-viewpoint video generating device according to a first embodiment of the invention; FIG. It is the figure which showed the generation method of the shielding object silhouette image. FIG. 4 is a diagram showing an example of camera parameters; It is the figure which showed the production|generation method of an integrated silhouette image. FIG. 4 is a diagram schematically showing a rendering method; FIG. 4 is a diagram comparing a rendering model generated according to the present invention with a rendering model generated according to the prior art; FIG. 10 is a functional block diagram showing the configuration of the required parts of the free viewpoint video generating device according to the second embodiment of the invention; FIG. 10 is a diagram showing an application example (part 1) to a multi-terminal distribution system that distributes rendering images with different virtual viewpoints to a plurality of viewing terminals; FIG. 10 is a diagram showing an application example (part 2) to a multi-terminal distribution system that distributes rendering images with different virtual viewpoints to a plurality of viewing terminals; It is a figure for demonstrating the visual volume intersection method. FIG. 10 is a diagram showing an example in which a subject silhouette image is deficient due to an obstructing object;

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. FIG. 1 is a functional block diagram showing the configuration of essential parts of a free-viewpoint video generating apparatus 1 according to the first embodiment of the present invention. An example will be described in which a free viewpoint video is generated based on video captured synchronously by a plurality of different cameras.

Such a free-viewpoint video generation device 1 can be realized by installing an application (program) that realizes each function described later in a general-purpose computer or mobile terminal equipped with a CPU, a memory, an interface, and a bus connecting them. Configurable. Alternatively, a part of the application can be configured as a dedicated machine or a single-function machine that is hardware or programmed.

A camera image acquisition unit 101 acquires camera images from a plurality of cameras that capture images of a competition field. In this embodiment, a full model free viewpoint is produced, all cameras Cam are fixed, and it is not assumed that the angle of view of each camera changes during the game.

A subject silhouette image generation unit 102 generates a silhouette image of a dynamic object (hereinafter referred to as a subject) that moves between frames for each camera image by, for example, the background subtraction method.

The shielding object silhouette image generation unit 103 generates a silhouette image of a static object that does not move between frames (hereafter referred to as a shielding object) for each camera using a predefined general-purpose 3D shielding model and camera parameters. automatically generated to . The camera parameters can be estimated based on matching results between each feature point extracted from a known structure represented by a shielding object and each feature point of the shielding object extracted from the camera image. For example, information about where the goalposts in a soccer match are located in the three-dimensional space of a stadium is known. Considering that the size of the goalposts is also determined by the standard, the 3D positions of feature points such as the corners of the goalposts are already known. By identifying such feature points in the 2D images obtained from each camera and matching the identified feature points with known 3D positions, the position and orientation of the camera can be determined (=camera calibration). can.

In this embodiment, since the camera is fixed, it is sufficient to generate the shielding object silhouette image only once at the beginning. The generated shielding object silhouette image is accumulated in the shielding object silhouette image DB 104 .

A general-purpose shield 3D model can be prepared in a general-purpose 3D model format such as .obj or .fbx. Therefore, instead of preparing a general-purpose 3D model, a plurality of 3D models of rectangular parallelepipeds and cylinders may be combined to generate a 3D model of a shield that resembles a goal post.

The shielding object silhouette image generation unit 103 arranges the shielding object 3D model at a predetermined position in a 3D space simulating a stadium, and as shown in FIG. , to generate the silhouette image of the shielding object. The camera parameters referred to here refer to a camera matrix (intrinsic parameter matrix) and an extrinsic parameter matrix, and are given in the format shown in FIG. 3, for example.

The camera parameters may be obtained manually or by auto-calibration as disclosed in Non-Patent Document 7. If this is combined with the method of auto-calibrating from the shape of the coat as in Non-Patent Document 7, the entire process including calibration can be performed fully automatically.

By applying the invention of the previous patent application by the present inventors (Japanese Patent Application No. 2019-231270) to the shielding object silhouette image of the goal post output by the shielding object silhouette image generation unit 103 for each camera, Image processing such as expanding the outline may be performed.

For example, a silhouette image obtained by back-projecting a 3D model can be inaccurate due to errors because the silhouette image itself can only represent discrete positions. If a 3D model is generated again by the visual volume intersection method using such a silhouette, there is a possibility that a post model smaller than the actual goal post will be generated. From the viewpoint of reducing such errors, silhouette image processing such as expanding the outline of the obtained silhouette may be performed.

As an example is shown in FIG. 4, the silhouette integration unit 105 generates an integrated silhouette image by integrating the shielding object silhouette image and the subject silhouette image frame by frame for each camera. For example, when the foreground of the silhouette is represented by 255 and the background is represented by 0, if either of the two input masks is 255, the object is the foreground.

The 3D model generating unit 106 generates a 3D voxel model (integrated 3D model) of the subject and the shielding objects by the visual volume intersection method using the N integrated silhouette images output by the silhouette integrating unit 105 . In this embodiment, voxel grids are arranged with a unit voxel size M in the target range of 3D model generation (for example, in the case of a sports video, the field where the sport is played, etc.), and whether or not to form a 3D model for each voxel grid is determined. is determined based on the visual volume intersection method.

The visual volume intersection method is a technique to acquire the common portion of the visual frustum when N silhouette images are projected onto the 3D world coordinates as the visual volume (Visual Hull) VH(I) based on the following equation (1). is.

In the above equation (1), set I is a set of silhouette images of each camera, and V _i is a viewing cone calculated based on the silhouette image obtained from the i-th camera. Also, normally, the part that is common to all N cameras is modeled, but the number of cameras that are modeled can be changed, such as modeling when N-1 cameras are common. good. By lowering the threshold for the number of cameras that generate the visual volume, it is possible to restore the 3D model even if the subject is missing in a small number of silhouette images, but there is a possibility that side effects such as increased noise will appear. be. This threshold for the number of cameras is set manually.

In the integrated 3D model output by the 3D model generation unit 106, the silhouettes of the goalposts are integrated, so it is possible to generate a defect-free 3D model of an object hidden behind an obstacle.

This visual volume intersection method processing may be applied to the two-stage visual volume intersection method as shown in Non-Patent Document 8. In this case, in both stages of the two-stage visual volume intersection method, modeling is performed by the visual volume intersection method using the integrated silhouette image generated by the silhouette integration unit.

At this time, a function of converting a voxel model into a polygon model using a method such as the Martin Cube method for converting a voxel model into a polygon model may be added, and a function of outputting a 3D model as a polygon model may be provided. . In this embodiment, after the visual volume intersection method is performed by the 3D model generation unit 106, the voxel model is converted into a polygon model based on the martin cube method.

The shielding object 3D model generation unit 107 generates a shielding object 3D model by the visual volume intersection method using the shielding object silhouette images accumulated for each camera in the shielding object silhouette image DB 104 . In this embodiment, since the cameras are fixed, the shielding object 3D model needs to be generated only once for each camera.

The generated shielding object 3D model is input to the shielding object 3D model subtraction unit 109, which will be described later. The general-purpose shielding object 3D model used for generation may be directly input to the shielding object 3D model subtraction unit 109 .

However, since it is desirable to input the same model as the integrated 3D model of the shield as the integrated 3D model to the shield 3D model subtraction unit 109, in this embodiment, the integrated 3D model of the shield is used as the visual volume A 3D model of the shielding object is generated using the silhouette image of the shielding object generated by the intersection method.

The occlusion information generator 108 calculates occlusion information of the 3D model. Occlusion information is information that records whether each part of the generated integrated 3D model is visible from each camera or invisible due to shielding. By referencing, texture mapping of invisible parts can be performed based on visible camera images.

In this embodiment, since a 3D polygon model is generated by the 3D model generation unit 106, the shielding relationship regarding each vertex portion of the 3D polygon model is recorded as occlusion information. For example, in an environment with N cameras, N pieces of occlusion information are recorded for each vertex part of the 3D polygon model.

In this embodiment, the occlusion information is recorded in a format such as "1" if the vertex part is visible and "0" if it is invisible. This makes it possible to express the occlusion information of each vertex part with visible/invisible 1 bit. The occlusion information takes into consideration not only the occlusion caused by the occluding object, but also all occlusion relationships including occlusion caused by other subjects.

For example, occlusion occurs when two players A and B overlap from a certain camera viewpoint, and if player A covers player B at this time, the texture of player A will not be reflected on player B. need to be mapped. In such a case, the occlusion information of the invisible vertex portion of player B is also recorded as "0" (invisible).

The shielding object 3D model subtraction unit 109 performs processing for removing a portion corresponding to the shielding object 3D model from the integrated 3D model generated by the 3D model generation unit 106 . In this embodiment, the position of the shielding 3D model generated by the shielding 3D model generation unit 107 is referred to, and the polygon existing at that position is deleted from the integrated 3D model.

In addition to the fact that this subtraction process can be expected to reduce the amount of 3D model data, when implementing the billboard free viewpoint, the 3D model of the post and the 3D model of the subject are connected, causing the huge billboard to rotate. It is possible to suppress the occurrence of the phenomenon that

The free-viewpoint rendering unit 110 renders an arbitrary virtual viewpoint p Render the composite image as seen from _v .

FIG. 5 is a diagram schematically showing a rendering method by the free viewpoint rendering section 110. As shown in FIG. In this embodiment, the visibility of each part (polygon in this embodiment) of the 3D model of the subject obtained by subtracting the shielding 3D model from the integrated 3D model is determined for each camera based on the occlusion information. , texture mapping is performed on an invisible part in some camera images using other visible camera images.

In this embodiment, two cameras Cam ₁ and Cam ₂ closest to the first requested virtual viewpoint p _v are selected, and each camera image Ic ₁ and Ic ₂ is mapped onto the polygon g of the 3D model M _j . . As a pre-process, in this embodiment, the visibility of the polygon g is determined using the occlusion information of all the vertices forming the polygon g. If the polygon g is a triangular polygon, visibility determination is made based on the occlusion information of each of the three vertices.

For example, when the visibility determination flag of polygon g for camera Cam1 is expressed as g _c1 , the flag g _c1 is visible if all three vertices that make up triangular polygon g are visible, and even one of the three vertices is invisible. , the flag g _c1 is made invisible. After obtaining the results of the visibility determination for each polygon in this manner, texture mapping is performed for each case as follows.

Case 1. If both flags g _c1 and g _c2 are visible:
Mapping by alpha blending is performed by the following equation (2).

Here, texture _c1 (g) and texture _c2 (g) indicate camera image areas corresponding to polygon g in cameras Cam ₁ and Cam ₂ , and texture (g) indicates the texture mapped to the polygon. Also, the alpha blend ratio a is calculated according to the ratio of the distances (angles) between the virtual viewpoint _pv and the camera viewpoints _pc1 and _pc2 .

Case 2. If only one of the flags g _c1 , g _c2 is visible:
Polygon g is rendered using only the camera textures that are visible. That is, in the above equation (2), the value of the alpha blend ratio a corresponding to the visible camera texture _ci (g) is set to 1. Alternatively, the camera _Cam3 , which is next closest to the virtual viewpoint _pv , is referred to instead of the invisible camera among the cameras _Cam1 and _Cam2 . At this time, the texture alpha-blending method is the same as the above formula (2).

Case 3. If all flags g _c1 , g _c2 are invisible:
Render using the texture of the camera Cam ₃ , which is next closest to the virtual viewpoint p _v . If the camera Cam ₃ is also invisible, the camera textures are referenced in order from the closest camera, such as the next closest camera Cam ₄ , and so on. At this time, the number of cameras to be sequentially referred to may be two or more, and the blending process may be performed according to the above equation (2).

In the above example, the number of nearby cameras to be initially referred to is two, but it may be changed by user settings. At that time, the above equation (2) is extended to the linear sum of the b cameras (sum of weights is 1) according to the initial number of reference cameras b. Also, textures are not mapped for polygons that are invisible to all cameras.

The display of the shielding 3D model in the free-viewpoint rendering unit 110 is performed by inputting a general-purpose 3D model prepared in advance and arranging it. This is because 3D models such as goalposts generally do not change significantly over time, and models derived from the visual volume intersection method are 3D models generated by synthesizing from N cameras. , is likely to be inferior to those prepared in advance in terms of quality.

FIG. 6 is a diagram comparing the rendering model [FIG. 6(b)] generated by the present embodiment with the rendering image [FIG. 6(a)] generated by the conventional technique.

In the conventional technology, the left leg portion of the silhouette image blocked by the goal post is missing, whereas in the rendering model generated by this embodiment, the texture is accurately mapped to the left leg portion, It can be seen that an accurate free-viewpoint video image is reproduced without loss or discomfort.

In the above-described first embodiment, the shielding object 3D model subtraction unit 109 is provided, the shielding object 3D model is removed from the integrated 3D model, and rendering is performed substantially only for the subject 3D model.

However, the present invention is not limited to this. As in the second embodiment shown in FIG. Free-viewpoint rendering may be performed on an integrated 3D model in which the model is not subtracted.

Even in this case, if the 3D model of the shielding object is covered with 3DCG input as a general-purpose 3D model or the like in the free-viewpoint rendering unit 110, the appearance of the 3D model is less likely to appear strange.

8 and 9 are diagrams showing examples of application to a multi-terminal distribution system that distributes rendering images with different virtual viewpoints to a plurality of viewing terminals.

In general, 3D model generation and occlusion information need only be calculated once for each frame, so high-speed calculations are performed on a high-end PC and saved. Then, by distributing this 3D model and occlusion information to viewing terminals that want to view the free viewpoint, and arranging a rendering unit on each viewing terminal, one high-end PC and multiple low-spec PCs can be used. Multi-terminal delivery can be realized with multiple viewing terminals.

The shielding relationship itself of the 3D model can also be recalculated by the rendering unit using the 3D model input to the free viewpoint rendering unit 110 . However, by storing the occlusion information in advance, the rendering unit can decipher the occluded relationship simply by referring to the occlusion information, so the processing load of the free viewpoint rendering unit 110 can be reduced. There is expected.

In the example of FIG. 8, a plurality of dedicated PCs specialized for rendering are prepared, and free viewpoint videos with different viewpoints are rendered and distributed in response to viewing requests from each viewing terminal.

In the example of FIG. 9, a free-viewpoint rendering unit 110 is installed in each viewing terminal so that rendering is executed for each viewing terminal.

In the above embodiment, the case where each camera is fixed has been described as an example, but the present invention is not limited to this, and can be similarly applied to the case of using a moving camera. The configuration that is changed from the first embodiment when using a moving camera will be described below.

The subject silhouette image generation unit 102 generates the subject silhouette image using the background subtraction method in the first embodiment. However, when a moving camera is used, the background changes, and it is difficult to generate a subject silhouette image by the background subtraction method. Therefore, as disclosed in Non-Patent Document 5, a silhouette extraction method capable of performing independent processing for each frame can be used.

In the first embodiment, the shielding object silhouette generation unit 103 has been described as generating the shielding object silhouette image only once at the beginning based on the 3D model of the shielding object and the camera parameters. However, with a moving camera the camera parameters change from frame to frame. A 3D model can then be placed in 3D space based on the latest camera parameters for each frame and backprojected onto each camera image.

Since it is difficult to manually calculate the camera parameters for each frame, the camera matrix and the extrinsic parameter matrix are calculated for each frame while performing auto-calibration, as disclosed in Non-Patent Document 7. Alternatively, a different shielding object silhouette image may be calculated for each frame.

In the first embodiment, the shielding object silhouette image generation unit 103 has been described as first generating a shielding object silhouette image only once based on the 3D model of the shielding object and the camera parameters. However, if a moving camera is used, the shielding object silhouette image generated by the shielding object silhouette image generating unit 103 changes for each frame, so the shielding object 3D model generating unit 107 also has a function of generating a shielding 3D model for each frame. may be provided.

Reference Signs List 1 Free-viewpoint video generation device 101 Camera video acquisition unit 102 Subject silhouette image generation unit 103 Shielding object silhouette image generation unit 104 Shielding object silhouette image DB 105 Silhouette integration unit 106 3D model Generating unit 107 : occlusion 3D model generation unit 108 : occlusion information generation unit 109 : shielding 3D model subtraction unit 110 : free viewpoint rendering unit

Claims (11)

A free-viewpoint video generation device that generates a free-viewpoint video based on camera images obtained by synchronously capturing a subject and a shield with a plurality of cameras with different viewpoints,
means for generating a subject silhouette image for each camera;
means for generating a shielding object silhouette image for each camera;
means for generating an integrated silhouette image by integrating the silhouette images of the subject and the shielding object for each camera;
means for generating an integrated 3D model by a visual volume intersection method using each integrated silhouette image;
means for generating occlusion information that registers whether each part of the integrated 3D model is visible or invisible at the viewpoint of each camera;
means for mapping, for each part of the integrated 3D model, a texture obtained by a camera in which the part is visible to a part that is invisible to some cameras, based on the occlusion information. Video production device. means for subtracting an occluder 3D model from the integrated 3D model;
2. The free-viewpoint video generation apparatus according to claim 1, wherein the mapping means uses the occlusion information to map a texture to each part of the integrated 3D model from which the 3D model of the obstructing object has been reduced. 3. The free-viewpoint video generation device according to claim 2, wherein the shielding object 3D model is a general-purpose 3D model imitating the shielding object. means for generating a 3D model of the shield based on the silhouette image of the shield;
3. The free-viewpoint video generation apparatus according to claim 2, wherein the means for subtracting the shielding object 3D model subtracts the generated shielding object 3D model from the integrated 3D model. the integrated 3D model is a polygon model,
5. The free-viewpoint video generation according to any one of claims 1 to 4, wherein in the occlusion information, for each vertex portion of each polygon, whether it is visible or invisible at the viewpoint of each camera is registered. Device. The means for generating the shield silhouette image arranges a separately prepared 3D model of the shield at a fixed position of the shield in a three-dimensional space, and reverses the shield at the fixed position to each camera based on the camera parameters. 6. The free-viewpoint video generating apparatus according to any one of claims 1 to 5, wherein each shielding object silhouette image is generated by projection. 6. The camera parameters are estimated based on matching results between each feature point extracted from a known structure that simulates the shield and each feature point of the shield extracted from the camera image. 3. The free-viewpoint video generation device according to . In a free-viewpoint video generation method in which a computer generates a free-viewpoint video based on camera images synchronously photographed by a plurality of cameras with different viewpoints of a subject and shielding objects,
Generate a subject silhouette image for each camera,
Generating shielding object silhouette images for each camera,
Generate an integrated silhouette image by integrating each silhouette image of the subject and the shielding object for each camera,
Generate an integrated 3D model by the visual volume intersection method using each integrated silhouette image,
generating occlusion information that registers whether each part of the integrated 3D model is visible or invisible at the viewpoint of each camera;
A free-viewpoint video generation method, characterized in that, based on the occlusion information, for each part of the integrated 3D model, a texture obtained by a camera that allows the part to be visible is mapped to a part that is invisible to a part of the cameras. subtracting an occluder 3D model from the integrated 3D model;
9. The free-viewpoint video generation method according to claim 8, wherein a texture is mapped to each part of the integrated 3D model from which the 3D model of the obstructing object is reduced, using the occlusion information. In a free-viewpoint video generation program that generates a free-viewpoint video based on camera images synchronously photographed by multiple cameras with different viewpoints of a subject and shields,
a procedure for generating a subject silhouette image for each camera;
A procedure for generating a shielding object silhouette image for each camera;
a procedure for generating an integrated silhouette image by integrating silhouette images of a subject and shielding objects for each camera;
A procedure for generating an integrated 3D model by the visual volume intersection method using each integrated silhouette image;
A procedure for generating occlusion information that registers whether each part of the integrated 3D model is visible or invisible at the viewpoint of each camera;
a step of mapping a texture acquired by a camera that is visible to a part that is invisible to some cameras for each part of the integrated 3D model based on the occlusion information;
A free-viewpoint video generation program that causes a computer to execute subtracting an occluder 3D model from the integrated 3D model;
11. The free-viewpoint video generating program according to claim 10, wherein, in the mapping step, textures are mapped to each part of the integrated 3D model from which the 3D model of the obstructing object has been reduced, using the occlusion information.

Images