JP2022038354A - Virtual viewpoint image generation device, method and program - Google Patents

Abstract

To provide a virtual viewpoint image without a sense of discomfort when generating a 3D model of a subject on the basis of a silhouette in such an environment that a long-shot camera and a close-up camera coexist and composing the virtual viewpoint image.SOLUTION: A boundary depending mapping unit 202 maps a texture from a camera image of a close-up camera to a 3D model within a field angle of the close-up camera without across the field angle boundary of the close-up camera, maps the texture from the camera image of the close-up camera to a 3D model within a field angle of a long-shot camera without across the field angle boundary of the close-up camera, maps only a texture extracted from the camera image of the long-shot camera by including a region within the field angle boundary of the close-up camera to a 3D model that is determined to be on the field angle boundary of the close-up camera, and thereby prevents quality deterioration of an image that may be generated due to mapping of both textures of the close-up camera and the long-shot camera to the 3D model on the field angle boundary.SELECTED DRAWING: Figure 1

Description

The present invention is a virtual viewpoint image generation device capable of generating a 3D model of a subject and providing a high-quality virtual viewpoint image that does not give a sense of discomfort even in an environment where a pulling camera and a close-up camera coexist when synthesizing the virtual viewpoint image. , Methods and programs.

Free viewpoint (virtual viewpoint) video technology is a technology that acquires video from multiple cameras and enables video viewing from any viewpoint, including viewpoints where no camera exists. As a method for realizing a free-viewpoint image, there is a 3D model-based free-viewpoint image generation method based on the visual volume crossing method disclosed in Non-Patent Document 1.

As shown in Fig. 9, the visual volume crossing method projects a binary silhouette image obtained by extracting only the subject part from each camera image into 3D space, and leaves only the part that is the product set as a 3DCG model. It is a method to generate a 3D model.

The visual volume crossing method includes a full model free viewpoint (a method that faithfully expresses the shape of a 3D model) disclosed in Patent Document 1 and a billboard method free viewpoint (3D model is referred to as a billboard) disclosed in Non-Patent Document 2. It is used as a basic technology to realize a method of producing in the shape of a board called, and mapping the texture from a nearby camera to the billboard).

In the free-viewpoint production disclosed in Non-Patent Document 1, first, the 3D space in which the free-viewpoint video is to be produced is filled with a voxel grid divided by a cubic grid. Next, the three-dimensional position of each voxel grid is back-projected onto the silhouette image of each camera, and the silhouette image of the corresponding position is referred to. Then, a voxel grid whose silhouette is determined to be white (the subject is present) is modeled by many cameras.

Such free-viewpoint video is available based on the determined camera work, taking advantage of the purpose of watching and enjoying sports from any viewpoint interactively in real time and the feature of being able to create video from any viewpoint. It has been used for the purpose of making a replay video with a feeling.

Japanese Unexamined Patent Publication No. 2018-063635 Japanese Patent Application No. 2020-133176

Laurentini, A. "The visual hull concept for silhouette based image understanding.", IEEE Transactions on Pattern Analysis and Machine Intelligence, 16, 150-162, (1994). H. Sankoh, S. Naito, K. Nonaka, H. Sabirin, J. Chen, "Robust Billboard-based, Free-viewpoint Video Synthesis Algorithm to Overcome Occlusions under Challenging Outdoor Sport Scenes", Proceedings of the 26th ACM international conference on Multimedia, pp. 1724-1732, (2018) J. Chen, R. Watanabe, K. Nonaka, T. Konno, H. Sankoh, S. Naito, "A Fast Free-viewpoint Video Synthesis Algorithm for Sports Scenes", 2019 IEEE / RSJ International Conference on Intelligent Robots and Systems ( IROS 2019), WeAT17.2, (2019). Qiang Yao, Hiroshi Sankoh, Nonaka Keisuke, Sei Naito. "Automatic camera self-calibration for immersive navigation of free viewpoint sports video," 2016 IEEE 18th International Workshop on Multimedia Signal Processing (MMSP), 1-6, 2016. C. Stauffer and W. E. L. Grimson, "Adaptive background mixture models for real-time tracking," 1999 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 246-252 Vol. 2 (1999). R. S. Yang et al., "Multi-Kinect scene reconstruction: Calibration and depth inconsistencies," 2013 28th International Conference on Image and Vision Computing New Zealand (IVCNZ 2013), Wellington, 2013, pp. 47-52. Shuang Sun et al., "Parametric Human Shape Reconstruction via Bidirectional Silhouette Guidance," Proceedings of the IEEE / CVF International Conference on Computer Vision (ICCV), 2019.

The placement of the camera is important in the production of free-viewpoint video. For example, if there is a camera that is very close to the subject, by creating a camera work that starts with the angle of view of the pulling camera and ends with the camera that is close to the subject, the work that gradually approaches the subject becomes highly clear. It is possible to realize with a texture with a degree.

The closer the camera is to the subject, the clearer the texture, while the closer the camera is to the subject, the smaller the area on the stadium that is reflected in the camera. In particular, in the case where only a specific camera is very close to the subject, the subject outside the angle of view of the close-up camera disappears without forming a 3D model.

FIG. 10 is a diagram comparing the range in which the intersection is formed between the case where all the cameras are pull cameras [the figure (a)] and the case where the camera includes one close-up camera [the figure (b)]. In Fig. (B), the range of the intersection is smaller than in Fig. (A), and it can be seen that the 3D model of the subject outside the angle of view of the close-up camera can disappear without being formed.

Such a technical problem can be solved by changing the threshold value of the number of cameras related to 3D model generation, such as modeling the part visible from N-1 units in an environment with N units.

Alternatively, as invented by the inventor of the present invention and already applied for a patent (Patent Document 2), when a 3D boxel model of a subject is generated by the visual volume crossing method using each camera image of a close-up camera and a pulling camera. As shown in Fig. 11, the 3D model is prevented from disappearing by treating it as if the subject exists outside the angle range of the close-up camera, while treating it as if the subject does not exist in the pulling camera. good.

On the other hand, even if a 3D model can be generated in an environment where the pull camera and the close camera coexist by making full use of each of the above methods, the subject in the close camera is only reflected in the pull camera from the close camera. The subject needs a rendering process that maps each from the pulling camera.

In addition, when the texture is mapped from each camera to the subject existing at the angle of view boundary between the close-up camera and the pulling camera, as shown in Fig. 12, the difference in the resolution of each camera and the difference in the mapping camera are the causes. There was a problem that the breaks were clearly noticeable at the boundary of the angle of view, which caused a sense of discomfort in the viewing quality.

An object of the present invention is to map a texture to a 3D model created in an environment in which a pulling camera and a close-up camera coexist to generate a virtual viewpoint image, and for each 3D model, the positional relationship with the angle-of-view boundary of the close-up camera. According to the present invention, it is an object of the present invention to provide a virtual viewpoint image generation device, a method and a program that do not cause a break in the angle of view boundary by mapping from only one of the pulling camera and the near camera.

In order to achieve the above object, the present invention is characterized in that it has the following configuration in a virtual viewpoint image generation device that generates a virtual viewpoint image based on a camera image of a subject taken from a plurality of viewpoints. ..

(1) A means for generating a 3D model based on each camera image of the close-up camera and a pull-back camera, a means for determining whether or not the 3D model of the subject exists on the angle of view boundary of the close-up camera, and a 3D model. On the other hand, it is equipped with a means for mapping the texture from each camera image, and the means for mapping is a texture from the camera image of the pulling camera including the inside of the angle of view of the close camera for the 3D model existing on the angle of view boundary. Was made to map.

(2) The judgment means is to judge whether or not the 3D bounding box containing the 3D model exists on the angle of view boundary of the camera.

(3) As a means to generate a 3D model, a visual volume crossing method based on each camera image of a close-up camera and a pulling camera is adopted, a means for constructing a low-resolution voxel model based on the silhouette of the subject, and a silhouette of the subject. A means for constructing a high-resolution voxel model in the area of a low-resolution voxel model based on I tried to judge whether or not to do it.

(4) When the 3D model is a polygon model, it generates occlusion information that records whether each polygon of the 3D model is visible or invisible from each camera, while the polygon of the 3D model existing on the angle boundary. As a means of rewriting and mapping the occlusion information of the close-up camera invisible, the texture is mapped to each polygon from the camera in which the polygon is visible.

(1) The virtual viewpoint image generator of the present invention performs texture mapping from the camera image of the pulling camera for the 3D model existing on the angle of view boundary of the near camera, including the inside of the angle of view of the near camera. Therefore, it is possible to prevent the quality deterioration that may occur due to the mapping of textures from both the camera and the pulling camera by leaning toward one 3D model.

(2) Judgment as to whether or not the 3D model of the subject exists on the angle of view boundary of the close-up camera based on whether or not the 3D bounding box containing the 3D model exists on the angle-of-view boundary of the close-up camera. Therefore, if all eight vertices of the 3D bounding box are within the angle of view range of the camera, or if all eight vertices are outside the angle of view range of the camera, it can be determined that they do not exist on the angle of view boundary. Therefore, a very high-speed determination is possible.

(3) Whether or not the 3D model of the subject exists on the angle of view boundary of the camera is determined at the time of the low-resolution voxel model, so it is not necessary to wait for the result of high-resolution voxel model generation. It becomes possible to perform boundary determination in parallel with the above, and it becomes possible to operate the processing at high speed.

(4) Since the occlusion information of the close-up camera related to the polygon of the 3D model existing on the angle of view boundary is rewritten invisible, the appropriate camera image can be obtained only by referring to the occlusion information without referring to the result of the boundary judgment. You will be able to map textures from.

It is a functional block diagram of the virtual viewpoint image generator which concerns on 1st Embodiment of this invention. It is a figure which showed the example which divides a 3D model for each subject by a bounding box. It is a figure which showed the method of performing the texture mapping to each 3D model from the camera image according to the angle of view. It is a functional block diagram of the virtual viewpoint image generator which concerns on 2nd Embodiment of this invention. It is a figure which showed the example of a camera parameter. It is a functional block diagram of the virtual viewpoint image generator which concerns on 3rd Embodiment of this invention. It is a functional block diagram of the virtual viewpoint image generator which concerns on 4th Embodiment of this invention. It is a figure which showed the example which rewrites the occlusion information based on the result of the boundary determination. It is a figure which showed the generation method of the 3D model by the visual volume crossing method. It is a figure which showed the example which the subject which is out of the angle of view range of a close-up camera disappears without forming a 3D model. It is a figure explaining the production method of the 3D model according to Patent Document 2. It is a figure which showed the example which the image quality deteriorates by mapping a texture from a close-up camera and a pulling camera to one 3D model.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. FIG. 1 is a functional block diagram showing the configuration of the main part of the virtual viewpoint image generation device 1 according to the first embodiment of the present invention, and has the 3D model production server 10 and the rendering server 20 as the main configurations. Here, a case where a sports scene is shot with N cameras Cam1 to CamN, a part of which is a close-up camera and the rest is a pulling camera will be described as an example.

Such a virtual viewpoint video generator 1 can be configured by implementing an application (program) that realizes each function on a general-purpose computer or server. Alternatively, it can be configured as a dedicated machine or a single-purpose machine in which a part of the application is made into hardware or software.

The 3D model production server 10 includes a silhouette image acquisition unit 101, a 3D model generation unit 102, and a boundary determination unit 103, and generates a 3D model for each subject and provides it to the rendering server 20. Further, the positional relationship with the angle of view boundary of the close-up camera is determined for each 3D model, and the determination result is provided to the rendering server 20.

The silhouette image acquisition unit 101 acquires a silhouette image used for generating a 3D model by the visual volume crossing method from the silhouette image database 30 from each camera image of the close-up camera and the pull camera. In order to generate a 3D model by the visual volume crossing method, it is desirable to acquire silhouette images from three or more cameras.

The silhouette image is given in the form of a binary mask image in which the subject area for generating the 3D model is represented by white (= 1) and the other areas are represented by black (= 0). Any existing method represented by the background subtraction method disclosed in Non-Patent Document 5 can be used to generate such a silhouette image.

The 3D model generation unit 102 calculates a 3D voxel model of the subject by the visual volume crossing method using N silhouette images based on the silhouette image acquired by the silhouette image acquisition unit 101 and the camera classification information separately given. .. Here, the camera classification information is information for identifying whether each camera is a close-up camera or a pulling camera.

In the visual volume crossing method, the intersection of the visual cones when N silhouette images are projected onto the 3D world coordinates is acquired as the visual volume (Visual Hull) VH (I), and the following equation (1) Indicated by. Here, the set I is a set of silhouette images of each camera, and Vi is a visual cone calculated from the silhouette images obtained from the i-th camera.

The voxel model generated in this way may be treated as a voxel as it is, or may be converted into a polygon model based on the Marching cube method or the like. Here, the explanation will be continued assuming that it is converted into a polygon model.

The boundary determination unit 103 determines whether or not the 3D model generated by the 3D model generation unit 102 exists on the angle of view boundary of the close-up camera. In the present embodiment, as shown in FIG. 2, a 3D bounding box containing an independent mass of each 3D model is defined, and it is determined whether or not the 3D model exists on the angle of view boundary for each 3D bounding box. do.

In the judgment targeting the 3D bounding box, if all 8 vertices are within the angle of view range of the close camera or all 8 vertices are outside the angle of view range of the close camera, the 3D bounding box does not exist on the angle of view boundary. Is determined. According to the judgment in units of the bounding box, only 8 vertices can be checked, which enables a very high-speed judgment.

On the other hand, the 3D bounding box is not exactly the same as the shape of the 3D model. Therefore, even though the included 3D model is within the angle of view of the close-up camera, if only the apex of the 3D bounding box leaks out of the angle of view of the close-up camera, a mistake may occur in the boundary determination.

Considering accuracy, it is desirable to use the voxel model contained in the 3D bounding box instead of the 3D bounding box unit. For example, if the center points of all voxels in the voxel model are back-projected toward the camera, and there is a center point that fits within the angle of view of the camera and a center point that does not fit, this subject exists in the boundary region. Judgment is made as a thing. If there are a plurality of close-up cameras, the boundary determination result may be calculated by multiplying the number of subjects by the number of close-up cameras.

The rendering server 20 renders a composite image viewed from a virtual viewpoint using the shape information of the subject 3D model produced by the 3D model production server 10 and each camera image (texture). In this embodiment, free viewpoint rendering is performed with a full model.

The rendering server 20 may be configured on the same computer as the 3D model production server 10, or may be configured on different servers. Generally, a 3D model only needs to be calculated once for a specific frame, so it is calculated and saved at high speed on a high-end PC, etc., and this 3D model is stored in a virtual viewpoint viewing terminal equipped with a rendering function. By configuring it for distribution, it is possible to realize video distribution to multiple terminals including one high-end PC and low-spec terminals.

In the rendering server 20, the virtual viewpoint selection unit 201 detects the viewpoint selection operation by the operator and acquires the position and orientation of the virtual viewpoint p _v . The boundary-dependent mapping unit 202 maps the texture from each camera image to each polygon of the 3D model based on the result of the virtual viewpoint _pv and the boundary determination. The virtual viewpoint video output unit 203 outputs the rendered composite video as a virtual viewpoint video.

FIG. 3 is a diagram schematically showing a texture mapping method by the boundary-dependent mapping unit 202. Only the texture extracted from the camera image of the close camera is mapped to the 3D model that is determined to be within the angle of view of the close camera without straddling the angle of view boundary. Further, for the 3D model determined to be within the angle of view of the pulling camera without straddling the angle of view boundary, only the texture extracted from the camera image of the pulling camera is mapped.

Of the 3D models that are determined to be within the angle of view of the pulling camera without straddling the angle of view boundary, for the 3D model that is also within the angle of view of the near-angle camera, only the close-up camera is used. The texture may be mapped.

On the other hand, for the 3D model determined to be on the angle of view boundary of the near camera, only the texture extracted from the camera image of the pull camera including the area within the angle of view of the near camera is available. It is mapped. This makes it possible to prevent the deterioration of image quality (Fig. 12) that may occur due to the mapping of the textures of both the close-up camera and the pulling camera to the 3D model on the angle of view boundary.

Although the camera classification information has been described separately in the first embodiment described above, the present invention is not limited to this, and as in the second embodiment shown in FIG. 4, the camera is used. A camera classification unit 104 that outputs camera classification information based on parameters may be provided so that the classification result can be adaptively changed according to the focal length that changes due to a zoom operation or the like.

The camera classification unit 104 automatically classifies N cameras into closer cameras or pull cameras by using the camera parameters given by the following equation (2).

Camera parameters are used to convert a point (X, Y, Z) on world coordinates to a 2D point (u, v) on the camera image, where r ₁₁ to r ₃₃ are rotation matrices indicating the direction of the camera. , T ₁ to t ₃ are parallel traveling matrices that represent the position of the camera, and the two are collectively called the external parameters of the camera.

f _x and f _y are focal lengths in pixel units that indicate the degree of zoom, and c _x and c _y are the principal points of the image, which are usually the center of the image. These parameters such as focal length and principal point are called internal parameters of the camera (often include parameters related to distortion caused by the camera lens on the image, but they are omitted here for simplicity).

s is a variable used for scaling to make [u, v, 1]. This camera parameter can be calculated in advance using the technique disclosed in Non-Patent Document 4. Figure 5 shows an example of the camera parameters that are actually input.

Here, since f _x and f _y are focal lengths in pixel units indicating the degree of zooming, it is highly possible that a camera with a large value is zoomed greatly. Therefore, the camera classification unit 104 can automatically classify the closer cameras by checking f _x and f _y .

The number of cameras classified as close-up cameras is not limited to one, and all several cameras (≤N) whose f _x and f _y are larger than a certain value may be classified as close-up cameras, or f _x and f. N1 units (≤N units) may be classified as cameras from the one with the larger _y . Furthermore, the cameras with L% of the total number of cameras (L is an arbitrary constant of 0 to 100) may be preferentially classified as a closer camera from the one with the larger f _x and f _y . Furthermore, the classification may be performed based on the camera position calculated from external parameters instead of f _x and f _y .

Alternatively, each camera may be classified by measuring the size of the subject 3D model created in the previous frame or the general-purpose 3D model such as a goal post prepared in advance reflected in each camera. For example, a 3D model prepared in advance can be placed in the area captured by all cameras including the close-up camera, and the 3D model can be classified based on the size of the silhouette that appears when it is back-projected toward the camera. ..

FIG. 6 is a functional block diagram showing the configuration of the main part of the virtual viewpoint image generation device 1 according to the third embodiment of the present invention, and the same reference numerals as those described above represent the same or equivalent parts. In this embodiment, the 3D model generation unit 102 includes a low-resolution voxel model generation unit 102a and a high-resolution voxel model generation unit 102b so that boundary determination is performed based on the low-resolution voxel model. There is a feature.

The low-resolution voxel model generation unit 102a generates a voxel model for a coarse voxel grid having a unit voxel size of M. The unit voxel size M is a value larger than the unit voxel size L in the high-resolution voxel generation unit 102b, and is set to, for example, M = 5 cm. In the present embodiment, a voxel grid is arranged in a target range of 3D model generation (for example, a field where the sport is performed in the case of sports video) with a unit voxel size M, and a 3D model is formed for this voxel grid. Whether or not it is determined based on the visual volume crossing method.

Next, for the formed coarse voxel model, labeling processing is performed for each block of the coarse voxel model by repeating the work of assuming that the connected voxels are the same subject.

Next, for the mass thus obtained, a 3D bounding box is defined so as to include it, and a voxel grid with a unit voxel size L is generated only inside this 3D bounding box, and the same as above is performed. Perform fine voxel generation. Such a two-step voxel generation method is disclosed in Non-Patent Document 3. The boundary determination unit 103 performs boundary determination in units of the 3D bounding box generated by the low-resolution voxel model generation unit 102a.

In this way, if the judgment is made at the time of the low-resolution voxel model, the boundary judgment can be performed in parallel without waiting for the result of the high-resolution voxel model generation, so that the processing is operated at high speed. be able to. However, the present invention does not prevent the boundary determination from being performed based on the high-resolution voxel model.

In this way, if the boundary determination is performed using the high-resolution voxel model, a more precise model shape can be obtained than in the case of using the low-resolution voxel model, so that the boundary determination can be performed more accurately.

FIG. 7 is a functional block diagram showing the configuration of the main part of the virtual viewpoint image generation device 1 according to the fourth embodiment of the present invention, and the same reference numerals as those of the third embodiment represent the same or equivalent parts. Therefore, the description thereof will be omitted.

In this embodiment, the 3D model production server 10 includes an occlusion information generation unit 105, the occlusion information is rewritten based on the determination result by the boundary determination unit 103, and the boundary dependence mapping unit 202 of the rendering server 20 is rewritten. The feature is that the texture is mapped based on the occlusion information of.

The occlusion information generation unit 105 generates occlusion information that separates each vertex of the 3D model into a visible camera and an invisible camera. In an environment where there are N cameras as in this embodiment, N occlusion information is calculated for each vertex of the 3D model, such as "1" for visible cameras and "0" for invisible cameras. Information is recorded.

When two players overlap in a soccer competition scene and player A obscures player B in a certain camera image, it is necessary to map the texture so that the texture of player A is not reflected in the 3D model of player B. In such a case, the occlusion information about the camera is recorded as "invisible" for the apex of the shielded part of the 3D model of player B. This occlusion information is calculated by using, for example, a method using a depth map as in Patent Document 1.

The boundary-dependent mapping unit 202 selects two cameras (c ₁ , c ₂ ) near the virtual viewpoint according to the result of the boundary determination, and maps these camera images to the polygon g of the 3D model. That is, if the 3D model to be mapped is not on the angle of view boundary and all of them are within the angle of view of the close-up camera, two cameras near the virtual viewpoint are selected for the close-up camera. On the other hand, if the 3D model to be mapped is on the angle of view boundary or all of them are within the angle of view of the pulling camera, two cameras near the virtual viewpoint are selected for the pulling camera. Will be done.

In the present embodiment, as the preprocessing, the visibility of the polygon is determined using the occlusion information of the three vertices constituting the polygon g (the three vertices are the case where the 3D model is formed by the triangular polygon, and the actual situation is high. Depends on the number of vertices that make up each polygon).

For example, when the visibility judgment flag of polygon g for camera c_1 is expressed as g_ (c_1), g_ (c_1) is visible if all three vertices constituting polygon g are visible, and any one of the three vertices is invisible. If so, g_ (c_1) is set to be invisible. In the present embodiment, texture mapping is performed as follows according to the result of such visibility determination of polygons for each camera.

Case 1: When the visibility judgment flags g _c1 and g _c2 of the cameras c ₁ and c ₂ related to the polygon g are both "visible", mapping by alpha blending is performed based on the following equation (3).

Here, texture _c1 (g) and texture _c2 (g) indicate the camera image area in which the polygon g corresponds to the cameras c ₁ and c ₂ , and texture (g) indicates the texture mapped to the polygon. The alpha blend ratio a is calculated according to the ratio of the distance (angle) between the virtual viewpoint p _v and each camera position p_ (c_1), p_ (c_2).

Case 2: When only one of the visibility judgment flags g _c1 and g _c2 is visible The polygon g is rendered using only the visible camera texture. That is, in the above equation (3), the value of the ratio a corresponding to the texture_ (c_i) of the visible camera is set to 1. Alternatively, the third camera c_3, which is the next closest to the virtual viewpoint p_v, is referred to instead of one of the invisible cameras, and mapping is performed by alpha blending based on the above equation (3) as in the case of Case 1.

Case 3: When both the visibility judgment flags g _c1 and g _c2 are invisible, at least one of the visibility judgment flags is visible to select another camera near the virtual viewpoint p _v (generally, the one with a close angle). The texture of the reference pixel position of each camera image is mapped to the polygon g by the alpha blend based on the above equation (3) as in the case of Case 1.

In the above embodiment, the number of nearby cameras to be initially referred to is two, but it may be changed by user setting. In that case, the above equation (1) is expanded to the linear sum of b cameras (the sum of the weights is 1) according to the number of initial reference cameras b. Also, textures are not mapped to polygons that are invisible in all cameras.

Here, in the present embodiment, the occlusion information is determined by the boundary determination unit 103 so that the boundary-dependent mapping unit 202 can map the texture from an appropriate camera image based on the occlusion and the boundary condition only by referring to the occlusion information. It can be rewritten according to the result.

FIG. 8 is a diagram showing an example of rewriting the occlusion information. Here, the visible / invisible (shielding) of the close-up camera is assigned to the least significant bit, and for each polygon constituting the 3D model located on the boundary. Is always rewritten to "0" indicating the shielded state regardless of whether each vertex is visible or invisible (shielded).

In each of the above embodiments, the 3D model generation unit 102 has been described as generating a 3D model by the visual volume crossing method using silhouette images based on the images of the cameras of the close-up camera and the pulling camera. However, the present invention is not limited to this, and is also applicable to texture mapping to a depth sensor-based 3D model (Non-Patent Document 6) and a neural network-based 3D model (Non-Patent Document 7). Applicable.

In Non-Patent Document 6, the 3D shape of an object is acquired by using a depth sensor that can acquire the distance to the object, such as Kinect (registered trademark). By combining multiple Kinect (registered trademarks), it is possible to acquire shapes with high accuracy from 360 degrees. In addition to the depth sensor, Kinect (registered trademark) also comes with an RGB camera, so you can use the Kinect (registered trademark) RGB camera for texture mapping. That is, only the shape is acquired by the depth sensor, and a normal camera can be used for texture mapping.

Non-Patent Document 7 discloses a method of estimating a 3D model shape by a neural network from a monocular or a plurality of cameras.

1 ... Virtual viewpoint video generator, 10 ... 3D model production server, 20 ... Rendering server, 101 ... Silhouette image acquisition unit, 102 ... 3D model generation unit, 102a ... Low resolution voxel model generation unit, 102b ... High resolution voxel Model generation unit, 103 ... boundary judgment unit, 104 ... camera classification unit, 105 ... occlusion information generation unit, 201 ... virtual viewpoint selection unit, 202 ... boundary-dependent mapping unit, 203 ... virtual viewpoint video output unit

Claims (15)

In a virtual viewpoint image generator that generates a virtual viewpoint image based on a camera image of a subject taken from multiple viewpoints.
A means to generate a 3D model based on each camera image of the close-up camera and the pull camera,
As a means of determining whether or not the 3D model of the subject is on the angle of view boundary of the close-up camera,
A means for mapping a texture from each camera image to the 3D model is provided.
The mapping means is characterized by mapping a texture from the camera image of the pulling camera including the inside of the angle of view of the near camera for a 3D model existing on the angle of view boundary of the near camera. Device. The virtual viewpoint image generation device according to claim 1, wherein the determination means determines whether or not a 3D bounding box containing a 3D model exists on the angle-of-view boundary of a close-up camera. The means for generating the 3D model is to generate a 3D model by the visual volume crossing method based on the images of the close-up camera and the pulling camera.
A means of building a low-resolution voxel model based on the silhouette of the subject,
It is equipped with a means for constructing a high-resolution voxel model in the area of the low-resolution voxel model based on the silhouette of the subject.
The virtual viewpoint image according to claim 1 or 2, wherein the determination means determines whether or not the low-resolution voxel model exists on the angle-of-view boundary of the close-up camera for each 3D model. Generator. The virtual means according to claim 3, wherein the means for generating the 3D model determines whether or not a 3D bounding box containing each low-resolution voxel model exists on the angle-of-view boundary of the close-up camera. Viewpoint image generator. The 3D model is a polygon model,
A means for generating occlusion information recording whether each polygon of the 3D model is visible or invisible from each camera is provided.
The means for generating the occlusion information is to invisiblely rewrite the occlusion information of the close-up camera regarding the polygon of the 3D model existing on the angle of view boundary.
The virtual viewpoint image generation device according to any one of claims 1 to 4, wherein the mapping means maps a texture to each polygon from a camera in which the polygon is visible. Equipped with a means to classify each camera as a close-up camera or a pull camera,
The means for determining is according to any one of claims 1 to 5, wherein the determination means determines whether or not each 3D model of the subject exists on the angle-of-view boundary of the close-up camera based on the result of the classification. Virtual viewpoint video generator. The virtual viewpoint image generation device according to claim 6, wherein the classification means classifies each camera into a close-up camera or a pulling camera based on a camera parameter. In a virtual viewpoint image generation method in which a computer generates a virtual viewpoint image based on a camera image of a subject taken from multiple viewpoints.
Generate a 3D model of the subject based on each camera image of the close-up camera and the pull camera,
Determines if the 3D model of the subject is on the angle of view boundary of the camera.
Textures are mapped from each camera image to the 3D model,
When mapping the texture, the virtual viewpoint is characterized in that the texture is mapped from the camera image of the pulling camera including the inside of the angle of view of the close camera for the 3D model existing on the angle of view boundary of the close camera. Video generation method. The feature is that it is determined whether or not the 3D model including the 3D model exists on the angle-of-view boundary of the close-up camera based on whether or not the 3D bounding box containing the 3D model exists on the angle-of-view boundary of the close-up camera. The virtual viewpoint image generation method according to claim 8. The 3D model is generated by the visual volume crossing method based on the images of the close-up camera and the pull camera.
Build a low-resolution voxel model based on the silhouette of the subject,
Build a high-resolution voxel model in the area of the low-resolution voxel model based on the silhouette of the subject,
The virtual viewpoint image generation method according to claim 8 or 9, wherein for each 3D model, it is determined whether or not the low-resolution voxel model exists on the angle-of-view boundary of the close-up camera. The 3D model is a polygon model,
Occlusion information is generated to record whether each polygon of the 3D model is visible or invisible from each camera.
Rewrite the occlusion information of the close-up camera regarding the polygon of the 3D model existing on the angle of view boundary invisible.
The virtual viewpoint image generation method according to any one of claims 8 to 10, wherein when the texture is mapped to each polygon, the texture is mapped from a camera in which the polygon is visible. In a virtual viewpoint image generation program that generates a virtual viewpoint image based on camera images taken from multiple viewpoints of the subject.
The procedure to generate a 3D model of the subject based on each camera image of the close-up camera and the pull camera,
The procedure for determining whether the 3D model of the subject is on the angle of view boundary of the close-up camera, and
Let the computer perform the procedure of mapping the texture from each camera image to the 3D model.
In the mapping procedure, virtual viewpoint image generation is characterized in that the texture is mapped from the camera image of the pulling camera including the inside of the angle of view of the near camera for the 3D model existing on the angle of view boundary of the near camera. program. The virtual viewpoint image generation program according to claim 12, wherein the determination procedure determines whether or not a 3D bounding box containing a 3D model exists on the angle-of-view boundary of a close-up camera. The procedure for generating the 3D model is to generate a 3D model by the visual volume crossing method based on the images of the close-up camera and the pulling camera.
The procedure for building a low-resolution voxel model based on the silhouette of the subject,
Including the procedure to build a high resolution voxel model in the area of the low resolution voxel model based on the silhouette of the subject, including
The virtual viewpoint image according to claim 12 or 13, wherein the determination procedure determines whether or not the low-resolution voxel model exists on the angle-of-view boundary of the close-up camera for each 3D model. Generation program. The 3D model is a polygon model,
Includes a procedure to generate occlusion information that records whether each polygon in the 3D model is visible or invisible from each camera.
In the procedure for generating the occlusion information, the occlusion information of the close-up camera regarding the polygon of the 3D model existing on the angle of view boundary is rewritten invisible.
The virtual viewpoint image generation program according to any one of claims 12 to 14, wherein the mapping procedure maps a texture to each polygon from a camera in which the polygon is visible.

Images