JP2022029730A - Three-dimensional (3d) model generation apparatus, virtual viewpoint video generation apparatus, method, and program - Google Patents

Abstract

To provide a virtual viewpoint video that does not feel strange even in an environment where a zoom-out camera and a zoom-in camera coexist when generating a 3D model of a subject and further generating a virtual viewpoint video.SOLUTION: A silhouette image acquisition unit 101 acquires a silhouette image from each of camera videos of a zoom-in camera and a zoom-out camera. A camera classification unit 103 classifies each of the camera in to the zoom-in camera and the zoom-out camera. A 3D model generation unit 102 generates a 3D model of a subject assuming that a silhouette of the subject exists outside an angle of view of the zoom-in camera based on a visual volume intersection method based on the camera video of the zoom-in camera and the zoom-out camera. A boundary determination unit 104 determines whether each of the 3D models of the subject exist on a boundary of the angle of view of the zoom-in camera. A rendering server 20 perform texture mapping from the camera video of the zoom-in camera including inside of the angle of view of the zoom-in camera to the 3D model existing on the boundary of the angle of view.SELECTED DRAWING: Figure 5

Description

INDUSTRIAL APPLICABILITY The present invention can provide a comfortable virtual viewpoint image without losing the 3D model even in an environment where a pulling camera and a close-up camera coexist when generating a 3D model of a subject and further generating a virtual viewpoint image. It relates to a model generator and a virtual viewpoint image generator, a method and a program.

Free viewpoint (virtual viewpoint) video technology is a technology that acquires video from multiple cameras and enables video viewing from any viewpoint, including viewpoints where no camera exists. As a method for realizing a free-viewpoint image, there is a 3D model-based free-viewpoint image generation method based on the visual volume crossing method disclosed in Non-Patent Document 1.

As shown in Fig. 11, the visual volume crossing method projects a binary silhouette image obtained by extracting only the subject part from each camera image into 3D space, and leaves only the part that is the product set as a 3DCG model. It is a method to generate a 3D model.

The visual volume crossing method includes a full model free viewpoint (a method that faithfully expresses the shape of a 3D model) disclosed in Patent Document 1 and a billboard method free viewpoint (3D model is referred to as a billboard) disclosed in Non-Patent Document 2. It is used as a basic technology to realize a method of creating a board in the shape of a so-called board and mapping the texture from a nearby camera to the billboard).

In the free-viewpoint production disclosed in Non-Patent Document 1, first, the 3D space in which the free-viewpoint video is to be produced is filled with a voxel grid divided by a cubic grid. Next, the three-dimensional position of each voxel grid is back-projected onto the silhouette image of each camera, and the silhouette image of the corresponding position is referred to. Then, a voxel grid whose silhouette is determined to be white (the subject is present) is modeled by many cameras.

Such free-viewpoint video is available based on the camera work that has been decided, taking advantage of the purpose of watching and enjoying sports from any viewpoint interactively in real time and the feature of being able to create video from any viewpoint. It has been used for the purpose of making a replay video with a feeling.

Japanese Unexamined Patent Publication No. 2018-063635

Laurentini, A. "The visual hull concept for silhouette based image understanding.", IEEE Transactions on Pattern Analysis and Machine Intelligence, 16, 150-162, (1994). H. Sankoh, S. Naito, K. Nonaka, H. Sabirin, J. Chen, "Robust Billboard-based, Free-viewpoint Video Synthesis Algorithm to Overcome Occlusions under Challenging Outdoor Sport Scenes", Proceedings of the 26th ACM international conference on Multimedia, pp. 1724-1732, (2018) J. Chen, R. Watanabe, K. Nonaka, T. Konno, H. Sankoh, S. Naito, "A Fast Free-viewpoint Video Synthesis Algorithm for Sports Scenes", 2019 IEEE / RSJ International Conference on Intelligent Robots and Systems ( IROS 2019), WeAT17.2, (2019). Qiang Yao, Hiroshi Sankoh, Nonaka Keisuke, Sei Naito. "Automatic camera self-calibration for immersive navigation of free viewpoint sports video," 2016 IEEE 18th International Workshop on Multimedia Signal Processing (MMSP), 1-6, 2016. C. Stauffer and W. E. L. Grimson, "Adaptive background mixture models for real-time tracking," 1999 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 246-252 Vol. 2 (1999).

The placement of the camera is important in the production of free-viewpoint video. For example, if there is a camera that is very close to the subject, by creating a camera work that starts with the angle of view of the pulling camera and ends with the camera that is close to the subject, the work that gradually approaches the subject becomes highly clear. It is possible to realize with a texture with a degree.

The closer the camera is to the subject, the clearer the texture will be, while the closer the camera is to the subject, the smaller the area on the stadium that will be reflected in the camera. In particular, in the case where only a specific camera is very close to the subject, the subject outside the angle of view of the close-up camera disappears without forming a 3D model.

FIG. 12 is a diagram comparing the range of formation of the intersection between the case where all the cameras are pull cameras [the figure (a)] and the case where the camera includes one close-up camera [the figure (b)]. In Fig. (B), the range of the intersection is smaller than in Fig. (A), and it can be seen that the 3D model of the subject outside the angle of view of the close-up camera can disappear without being formed.

Such a technical problem can be solved by changing the threshold value of the number of cameras related to 3D model generation, such as modeling the part visible from N-1 units in an environment with N units. However, as the number of units decreases, the increase in noise that the part that is not originally the subject is modeled in 3D becomes a problem, so changing the threshold value easily may lead to deterioration of quality.

Even if a 3D model is generated by controlling the threshold value of the number of cameras described above, the subject reflected in the close-up camera is mapped from the close-up camera, and the subject reflected only in the pull-down camera is mapped from the pull-down camera. A rendering process like this is required. In addition, for subjects existing at the angle of view boundary between the close-up camera and the pulling camera, as shown in Fig. 13, there is a clear break at the angle of view boundary due to the difference in resolution of each camera and the difference in the mapping camera. There was a problem that it became conspicuous and the viewing quality became uncomfortable.

An object of the present invention is to create a 3D model in an environment in which a pulling camera and a close-up camera coexist, and to generate a virtual viewpoint image. It is an object of the present invention to provide a 3D model generator as well as a free viewpoint image generator, a method and a program in which a camera treats a subject as if it does not exist so that the 3D model outside the angle of view of the close-up camera is not erased.

In order to achieve the above object, the present invention has the following configuration in a 3D model generation device that generates a 3D model based on a camera image of a subject taken from a plurality of viewpoints.

(1) The 3D model is generated by providing a means for classifying each camera into a close-up camera or a pull-down camera and a means for generating a 3D model of a subject by a visual volume crossing method based on the camera images of the close-up camera and the pull-down camera. As a means of doing so, it is considered that the subject exists outside the angle of view range of the close-up camera.

Further, the present invention has the following configuration in a virtual viewpoint image generation device that generates a virtual viewpoint image based on a camera image of a subject taken from a plurality of viewpoints.

(2) A means for classifying each camera into a close-up camera or a pull-down camera, a means for generating a 3D model of a subject by a visual volume crossing method based on the camera images of the close-up camera and the pull-down camera, and a means for generating a 3D model of the subject in the above-mentioned 3D model. It is equipped with a means for mapping textures and synthesizing a virtual viewpoint image, and the means for generating the 3D model is such that the subject exists outside the angle of view range of the close-up camera.

(3) Further provided with a means for determining whether or not each 3D model of the subject exists on the angle of view boundary of the close-up camera, and for a 3D model existing on the angle-of-view boundary, the angle of view of the close-up camera is provided. Texture mapping is performed from the camera image of the pull camera including the inside.

(1) The 3D model generator of the present invention considers that the subject exists outside the angle of view range of the close-up camera when generating a 3D model of the subject by the visual volume crossing method based on the camera images of the close-up camera and the pull-back camera. This makes it possible to prevent the loss of the subject 3D model that exists within the angle of view of the pulling camera and outside the angle of view of the close-up camera.

(2) When the virtual viewpoint image generator of the present invention generates a 3D model of a subject by the visual volume crossing method based on the camera images of the close-up camera and the pull-back camera, the subject exists outside the angle of view range of the close-up camera. Therefore, it is possible to prevent the virtual viewpoint image of the subject existing within the angle of view of the pulling camera and outside the angle of view of the camera from being synthesized.

(3) The virtual viewpoint image generator of the present invention performs texture mapping from the camera image of the pull camera for the 3D model existing on the angle of view boundary of the near camera, including the inside of the angle of view of the near camera. Therefore, it is possible to prevent the quality deterioration that may occur due to the mapping of textures from both the camera and the pulling camera by leaning toward one 3D model.

It is a functional block diagram of the 3D model generation apparatus which concerns on 1st Embodiment of this invention. It is a figure which showed the example which handles differently when a reference occurs outside the angle of view range between a pulling camera and a close-up camera. It is a functional block diagram of the 3D model generation apparatus which concerns on 2nd Embodiment of this invention. It is a figure which showed the example of a camera parameter. It is a functional block diagram of the virtual viewpoint image generator which concerns on 3rd Embodiment of this invention. It is a figure which showed the example which divides a 3D model for each subject by a bounding box. It is a figure which showed the method of performing the texture mapping to each 3D model from the camera image according to the angle of view. It is a functional block diagram of the virtual viewpoint image generator which concerns on 4th Embodiment of this invention. It is a functional block diagram of the virtual viewpoint image generator which concerns on 5th Embodiment of this invention. It is a figure which showed the example which rewrites the occlusion information based on the result of the boundary determination. It is a figure which showed the generation method of the 3D model by the visual volume crossing method. It is a figure which showed the example which the subject which is out of the angle of view range of a close-up camera disappears without forming a 3D model. It is a figure which showed the example which the image quality deteriorates by mapping a texture from a close-up camera and a pulling camera to one 3D model.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. FIG. 1 is a functional block diagram showing the configuration of the main part of the 3D model production server 10 according to the first embodiment of the present invention. Here, a sports scene is photographed by N cameras Cam1 to CamN, and one of them is taken. The case where the part is a close-up camera and the rest is a pulling camera will be described as an example.

The silhouette image acquisition unit 101 acquires a silhouette image used for generating a 3D model by the visual volume crossing method from the silhouette image database 30 from each camera image of the close-up camera and the pull camera. In order to generate a 3D model by the visual volume crossing method, it is desirable to acquire silhouette images from three or more cameras. The silhouette image is given in the form of a binary mask image in which the subject area for generating the 3D model is represented by white (= 1) and the other areas are represented by black (= 0). Any existing method represented by the background subtraction method disclosed in Non-Patent Document 5 can be used to generate such a silhouette image.

The 3D model generation unit 102 calculates a 3D voxel model of the subject by the visual volume crossing method using N silhouette images based on the silhouette image acquired by the silhouette image acquisition unit 101 and the camera classification information separately given. .. Here, the camera classification information is information for identifying whether each camera is a close-up camera or a pulling camera.

In the visual volume crossing method, the intersection of the visual cones when N silhouette images are projected onto the 3D world coordinates is acquired as the visual volume (Visual Hull) VH (I), and the following equation (1) Indicated by. Here, the set I is a set of silhouette images of each camera, and Vi is a visual cone calculated from the silhouette images obtained from the i-th camera.

In the visual volume crossing method, the silhouette images obtained from each camera are usually treated equally, but in the present embodiment, the silhouette image of the close-up camera and the silhouette image of the pulling camera are treated based on the camera classification information. It's different.

Specifically, as shown in Fig. 2, when determining whether to model each boxel grid arranged in 3D space by referring to the corresponding pixels of each silhouette image, it is out of the angle of view range (the boxel grid is the camera). When a reference occurs in an area that is not reflected in the camera, it is considered that the subject does not exist (black on the silhouette image) in the case of the pull camera, while the subject exists (white on the silhouette image) in the case of the close-up camera. By doing so, there is room for modeling outside the angle of view range of the close-up camera according to the reference result of the pulling camera.

According to the present embodiment, it is considered that the subject exists regardless of whether or not the subject actually exists outside the angle of view range of the close-up camera. However, in the area where the subject does not actually exist, the pulling camera is used. Since the reference result of is negative, only the voxel grid in which the subject actually existed is modeled. Therefore, it is possible to generate an accurate 3D model of the subject even in an environment where a close-up camera and a pulling camera coexist.

The voxel model generated in this way may be treated as a voxel as it is, or may be converted into a polygon model based on the Marching cube method or the like. Here, the explanation will be continued assuming that it is converted into a polygon model.

Although the camera classification information has been described separately in the first embodiment described above, the present invention is not limited to this, and the camera is as shown in the second embodiment shown in FIG. A camera classification unit 103 that outputs camera classification information based on parameters may be provided so that the classification result can be adaptively changed according to the focal length that changes due to a zoom operation or the like.

The camera classification unit 103 automatically classifies N cameras into closer cameras or pull cameras by using the camera parameters given by the following equation (2).

Camera parameters are used to convert a point (X, Y, Z) on world coordinates to a 2D point (u, v) on the camera image, where r ₁₁ to r ₃₃ are rotation matrices indicating the direction of the camera. , T ₁ to t ₃ are parallel traveling matrices that represent the position of the camera, and the two are collectively called the external parameters of the camera.

f _x and f _y are focal lengths in pixel units that indicate the degree of zoom, and c _x and c _y are the principal points of the image, and are usually the center of the image. These parameters such as focal length and principal point are called internal parameters of the camera (although they often include parameters related to distortion correction, they are omitted here for simplicity).

s is a variable used for scaling to make [u, v, 1]. This camera parameter can be calculated in advance using the technique disclosed in Non-Patent Document 4. Figure 4 shows an example of the camera parameters that are actually input.

Here, since f _x and f _y are focal lengths in pixel units indicating the degree of zooming, it is highly possible that a camera with a large value is zoomed greatly. Therefore, the camera classification unit 103 can automatically classify the closer cameras by checking f _x and f _y .

The number of cameras classified as close-up cameras is not limited to one, and all several cameras (≤N) whose f _x and f _y are larger than a certain value may be classified as close-up cameras, or f _x and f. M units (≤N units) may be classified as cameras from the one with the larger _y . Furthermore, the cameras with L% of the total number of cameras (L is an arbitrary constant of 0 to 100) may be preferentially classified as a closer camera from the one with the larger f _x and f _y . Furthermore, the classification may be performed based on the camera position calculated from external parameters instead of f _x and f _y .

Alternatively, each camera may be classified by measuring the size of the subject 3D model created in the previous frame or the general-purpose 3D model such as a goal post prepared in advance reflected in each camera. For example, a 3D model prepared in advance can be placed in the area captured by all cameras including the close-up camera, and the 3D model can be classified based on the size of the silhouette that appears when it is back-projected toward the camera. ..

If the number of cameras changes drastically due to a camera failure during the production of virtual viewpoint video, for example, the number of pulling cameras will become extremely small, and most cameras may become closer cameras. be. In such a case, since all silhouettes outside the angle of view of the close-up camera are considered to have a subject in the present embodiment, a 3D model that should not be originally modeled is placed in the area outside the angle of view of the close-up camera. It may be generated.

In order to avoid such a situation, if the number of close-up cameras or the ratio of close-up cameras to the total number of cameras is high, the 3D model generator 102 "outside the angle of view of the close-up camera silhouette image" The function of "considering that the subject exists" may be immobilized so that the subject does not exist outside the angle of view range as in the conventional technique.

According to the present embodiment, when the 3D model of the subject is generated by the visual volume crossing method based on the camera images of the close-up camera and the pull-back camera, it is considered that the subject exists outside the angle of view range of the close-up camera. It will be possible to prevent the loss of the subject 3D model that exists within the angle of view of the pulling camera and outside the angle of view of the close-up camera.

FIG. 5 is a functional block diagram showing the configuration of the main part of the virtual viewpoint image generation device 1 according to the third embodiment of the present invention, and has the 3D model production server 10 and the rendering server 20 as the main configurations.

The 3D model production server 10 includes the silhouette image acquisition unit 101, the 3D model generation unit 102, and the camera classification unit 103 as in the first or second embodiment, and the subject actually exists outside the angle of view range of the closer camera. Regardless of whether or not the subject exists, the 3D model is generated by treating the outside of the angle of view range of the silhouette image as white in which the subject exists.

The boundary determination unit 104 determines whether or not the 3D model generated by the 3D model generation unit 102 exists on the angle of view boundary of the close-up camera. In the present embodiment, as shown in FIG. 6, a 3D bounding box containing an independent mass of each 3D model is defined, and it is determined whether or not the 3D model exists on the angle of view boundary for each 3D bounding box. do.

In the judgment targeting the 3D bounding box, if all 8 vertices are within the angle of view range of the close camera or all 8 vertices are outside the angle of view range of the close camera, the 3D bounding box does not exist on the angle of view boundary. Is determined. According to the judgment in units of the bounding box, only 8 vertices can be checked, which enables a very high-speed judgment.

On the other hand, the 3D bounding box is not exactly the same as the shape of the 3D model. Therefore, even though the included 3D model is within the angle of view of the close-up camera, if only the apex of the 3D bounding box leaks out of the angle of view of the close-up camera, a mistake may occur in the boundary determination.

Considering accuracy, it is desirable to use the voxel model contained in the 3D bounding box instead of the 3D bounding box unit. For example, if the center points of all voxels in the voxel model are back-projected toward the camera, and there is a center point that fits within the angle of view of the camera and a center point that does not fit, this subject exists in the boundary region. Judgment is made as a thing. If there are a plurality of close-up cameras, the boundary determination result may be calculated by multiplying the number of subjects by the number of close-up cameras.

The rendering server 20 renders a composite image viewed from a virtual viewpoint using the shape information of the subject 3D model produced by the 3D model production server 10 and each camera image (texture). In this embodiment, free viewpoint rendering is performed with a full model.

The rendering server 20 may be configured on the same computer as the 3D model production server 10, or may be configured on different servers. Generally, a 3D model only needs to be calculated once for a specific frame, so it is calculated and saved at high speed on a high-end PC, etc., and this 3D model is stored in a virtual viewpoint viewing terminal equipped with a rendering function. By configuring it for distribution, it is possible to realize video distribution to multiple terminals including one high-end PC and low-spec terminals.

In the rendering server 20, the virtual viewpoint selection unit 201 detects the viewpoint selection operation by the operator and acquires the position and orientation of the virtual viewpoint p _v . The boundary-dependent mapping unit 202 maps the texture from each camera image to each polygon of the 3D model based on the result of the virtual viewpoint _pv and the boundary determination. The virtual viewpoint video output unit 203 outputs the rendered composite video as a virtual viewpoint video.

FIG. 7 is a diagram schematically showing a texture mapping method by the boundary-dependent mapping unit 202. Only the texture extracted from the camera image of the close camera is mapped to the 3D model that is determined to be within the angle of view of the close camera without straddling the angle of view boundary. Further, only the texture extracted from the camera image of the pulling camera is mapped to the 3D model determined to be within the angle of view of the pulling camera without straddling the angle of view boundary.

Of the 3D models that are determined to be within the angle of view of the pulling camera without straddling the angle of view boundary, for the 3D model that is also within the angle of view of the near-angle camera, only the close-up camera is used. The texture may be mapped.

On the other hand, for the 3D model determined to be on the angle of view boundary of the near camera, only the texture extracted from the camera image of the pull camera including the area within the angle of view of the near camera is available. It is mapped. This makes it possible to prevent the deterioration of image quality that may occur due to the mapping of the textures of both the close-up camera and the pulling camera to the 3D model on the angle of view boundary.

FIG. 8 is a functional block diagram showing the configuration of the main part of the virtual viewpoint image generation device 1 according to the fourth embodiment of the present invention, and the same reference numerals as those described above represent the same or equivalent parts. In this embodiment, the 3D model generation unit 102 includes a low-resolution voxel model generation unit 102a and a high-resolution voxel model generation unit 102b so that boundary determination is performed based on the low-resolution voxel model. There is a feature.

The low-resolution voxel model generation unit 102a generates a voxel model for a coarse voxel grid having a unit voxel size of M. The unit voxel size M is a value larger than the unit voxel size L in the high-resolution voxel generation unit 102b, and is set to, for example, M = 5 cm. In the present embodiment, a voxel grid is arranged in a target range of 3D model generation (for example, a field where the sport is performed in the case of sports video) with a unit voxel size M, and a 3D model is formed for this voxel grid. Whether or not it is determined based on the visual volume crossing method.

Next, for the formed coarse voxel model, labeling processing is performed for each block of the coarse voxel model by repeating the work of assuming that the connected voxels are the same subject.

Next, for the mass thus obtained, a 3D bounding box is defined so as to include it, and a voxel grid with a unit voxel size L is generated only inside this 3D bounding box, and the same as above is performed. Perform fine voxel generation. Such a two-step voxel generation method is disclosed in Non-Patent Document 3. The boundary determination unit 104 determines the boundary in units of the 3D bounding box generated by the low-resolution voxel model generation unit 102a.
In this way, if the judgment is made at the time of the low-resolution voxel model, the boundary judgment can be performed in parallel without waiting for the result of the high-resolution voxel model generation, so that the processing can be operated at high speed. can. However, the present invention does not prevent the boundary determination from being performed based on the high-resolution voxel model. In this way, if the boundary determination is performed using the high-resolution voxel model, a more precise model shape can be obtained than in the case of using the low-resolution voxel model, so that the boundary determination can be performed more accurately.

FIG. 9 is a functional block diagram showing the configuration of the main part of the virtual viewpoint image generation device 1 according to the fifth embodiment of the present invention, and the same reference numerals as those of the fourth embodiment represent the same or equivalent parts. Therefore, the description thereof will be omitted.

In this embodiment, the 3D model production server 10 includes an occlusion information generation unit 105, the occlusion information is rewritten based on the determination result by the boundary determination unit 104, and the boundary dependence mapping unit 202 of the rendering server 20 is rewritten. The feature is that the texture is mapped based on the occlusion information of.

The occlusion information generation unit 105 generates occlusion information that separates each vertex of the 3D model into a visible camera and an invisible camera. In an environment where there are N cameras as in this embodiment, N occlusion information is calculated for each vertex of the 3D model, such as "1" for visible cameras and "0" for invisible cameras. Information is recorded.

When two players overlap in a soccer competition scene and player A obscures player B in a certain camera image, it is necessary to map the texture so that the texture of player A is not reflected in the 3D model of player B. In such a case, the occlusion information about the camera is recorded as "invisible" for the apex of the shielded part of the 3D model of player B. This occlusion information is calculated by using, for example, a method using a depth map as in Patent Document 1.

The boundary-dependent mapping unit 202 selects two cameras (c_1, c_2) near the virtual viewpoint according to the result of the boundary determination, and maps these camera images to the polygon g of the 3D model. That is, if the 3D model to be mapped is not on the angle of view boundary and all of them are within the angle of view of the close-up camera, two cameras near the virtual viewpoint are selected for the close-up camera. On the other hand, if the 3D model to be mapped is on the angle of view boundary or all of them are within the angle of view of the pulling camera, two cameras near the virtual viewpoint are selected for the pulling camera. Will be done.

In the present embodiment, as the preprocessing, the visibility of the polygon is determined using the occlusion information of the three vertices constituting the polygon g (the three vertices are the case where the 3D model is formed by the triangular polygon, and the actual situation is high. Depends on the number of vertices that make up each polygon).

For example, when the visibility judgment flag of polygon g for camera c_1 is expressed as g_ (c_1), g_ (c_1) is visible if all three vertices constituting polygon g are visible, and any one of the three vertices is invisible. If so, g_ (c_1) is set to be invisible. In the present embodiment, texture mapping is performed as follows according to the result of such visibility determination of polygons for each camera.

Case 1: When the visibility judgment flags g _c1 and g _c2 of the cameras c ₁ and c ₂ related to the polygon g are both "visible", mapping by alpha blending is performed based on the following equation (3).

Here, texture _c1 (g) and texture _c2 (g) indicate the camera image area in which the polygon g corresponds to the cameras c ₁ and c ₂ , and texture (g) indicates the texture mapped to the polygon. The alpha blend ratio a is calculated according to the ratio of the distance (angle) between the virtual viewpoint p _v and each camera position p_ (c_1), p_ (c_2).

Case 2: When only one of the visibility judgment flags g _c1 and g _c2 is visible The polygon g is rendered using only the visible camera texture. That is, in the above equation (3), the value of the ratio a corresponding to the texture_ (c_i) of the visible camera is set to 1. Alternatively, the third camera c_3, which is the next closest to the virtual viewpoint p_v, is referred to instead of one of the invisible cameras, and mapping is performed by alpha blending based on the above equation (3) as in the case of Case 1.

Case 3: When both the visibility judgment flags g _c1 and g _c2 are invisible, at least one of the visibility judgment flags is visible to select another camera near the virtual viewpoint p _v (generally, the one with a close angle). The texture of the reference pixel position of each camera image is mapped to the polygon g by the alpha blend based on the above equation (3) as in the case of Case 1.

In the above embodiment, the number of nearby cameras to be initially referred to is two, but it may be changed by user setting. In that case, the above equation (1) is expanded to the linear sum of b cameras (the sum of the weights is 1) according to the number of initial reference cameras b. Also, textures are not mapped to polygons that are invisible in all cameras.

In the present embodiment, the occlusion information is the determination result of the boundary determination unit 104 so that the boundary-dependent mapping unit 202 can map the texture from an appropriate camera image based on the occlusion and the boundary condition only by referring to the occlusion information. It is designed to be rewritten according to.

FIG. 10 is a diagram showing an example of rewriting the occlusion information. Here, the visible / invisible (shielding) of the close-up camera is assigned to the least significant bit, and for each polygon constituting the 3D model located on the boundary. Is always rewritten to "0" indicating the shielded state, regardless of whether each vertex is visible or invisible (shielded).

1 ... Virtual viewpoint video generator, 10 ... 3D model production server, 20 ... Rendering server, 101 ... Silhouette image acquisition unit, 102 ... 3D model generation unit, 102a ... Low resolution voxel model generation unit, 102b ... High resolution voxel Model generation unit, 103 ... camera classification unit, 104 ... boundary determination unit, 105 ... occlusion information generation unit, 201 ... virtual viewpoint selection unit, 202 ... boundary-dependent mapping unit, 203 ... virtual viewpoint video output unit

Claims (15)

In a 3D model generator that generates a 3D model based on camera images taken from multiple viewpoints of the subject.
A means of classifying each camera into a close-up camera or a pulling camera,
It is equipped with a means for generating a 3D model of the subject by the visual volume crossing method based on the camera images of the close-up camera and the pull camera.
The means for generating the 3D model is a 3D model generation device characterized in that it considers that the subject exists outside the angle of view range of the close-up camera. A means for generating a silhouette image from the camera images of the close-up camera and the pulling camera is provided.
The 3D model generation device according to claim 1, wherein the means for generating the 3D model is to construct a voxel model by assuming that the silhouette of the subject exists outside the angle of view range of the close-up camera. The 3D model generator according to claim 1 or 2, wherein the classification means classifies each camera into a close-up camera or a pulling camera based on a camera parameter. In a virtual viewpoint image generator that generates a virtual viewpoint image based on a camera image of a subject taken from multiple viewpoints.
A means of classifying each camera into a close-up camera or a pulling camera,
A means to generate a 3D model of the subject by the visual volume crossing method based on the camera images of the close-up camera and the pull camera,
A means for synthesizing a virtual viewpoint image by mapping the texture of each camera image to the 3D model is provided.
The means for generating the 3D model is a virtual viewpoint image generation device characterized in that the subject is considered to exist outside the angle of view range of the close-up camera. A means for generating a silhouette image from each camera image of the close-up camera and the pulling camera is provided.
The virtual viewpoint image generation device according to claim 4, wherein the means for generating the 3D model is to construct a voxel model by assuming that the silhouette of the subject exists outside the angle of view range of the close-up camera. The virtual viewpoint image generation device according to claim 4, wherein the classification means classifies each camera into a close-up camera or a pulling camera based on a camera parameter. Further provided with a means for determining whether or not each 3D model of the subject is on the angle-of-view boundary of the close-up camera.
The virtual according to any one of claims 4 to 6, wherein the 3D model existing on the angle of view boundary is texture-mapped from the camera image of the pulling camera including the inside of the angle of view of the close-up camera. Viewpoint image generator. The virtual viewpoint image generation device according to claim 7, wherein the determination means determines whether or not a 3D bounding box containing each 3D model exists on the angle-of-view boundary of a close-up camera. The means for generating the 3D model is
A means of building a low-resolution voxel model based on the silhouette of the subject,
It is equipped with a means for constructing a high-resolution voxel model in the area of the low-resolution voxel model based on the silhouette of the subject.
The means according to any one of claims 4 to 8, wherein the determination means determines whether or not the low-resolution voxel model exists on the angle-of-view boundary of the close-up camera for each 3D model. Virtual viewpoint video generator. The virtual means according to claim 9, wherein the means for generating the 3D model determines whether or not the 3D bounding box containing each low-resolution voxel model exists on the angle-of-view boundary of the close-up camera. Viewpoint image generator. The 3D model is a polygon model,
It is provided with a means for generating occlusion information that records whether each polygon of each 3D model is visible or invisible from each camera.
The means for synthesizing the virtual viewpoint image is to map the texture to each polygon from the camera in which the polygon is visible.
The virtual viewpoint image generation device according to any one of claims 4 to 10, wherein the occlusion information of the close-up camera relating to the polygon of the 3D model existing on the angle of view boundary is invisiblely rewritten. In a virtual viewpoint image generation method in which a computer generates a virtual viewpoint image based on a camera image of a subject taken from multiple viewpoints.
Classify each camera as a close-up camera or a pull camera,
A 3D model of the subject is generated by the visual volume crossing method based on the camera images of the close-up camera and the pull camera.
By mapping the texture of each camera image to the above 3D model and synthesizing the virtual viewpoint image,
A virtual viewpoint image generation method characterized in that when the 3D model is generated, it is considered that the subject exists outside the angle of view range of the close-up camera. Determines whether each 3D model of the subject is on the angle of view boundary of the close-up camera.
The virtual viewpoint image generation method according to claim 12, wherein the 3D model existing on the angle of view boundary is texture-mapped from the camera image of the pulling camera including the inside of the angle of view of the close-up camera. In a virtual viewpoint image generation program that generates a virtual viewpoint image based on camera images taken from multiple viewpoints of the subject.
The procedure for classifying each camera as a close-up camera or a pull camera,
The procedure for generating a 3D model of the subject by the visual volume crossing method based on the camera images of the close-up camera and the pull camera,
Let the computer execute the procedure of mapping the texture of each camera image to the 3D model and synthesizing the virtual viewpoint image.
In the procedure for generating the 3D model, a virtual viewpoint image generation program is characterized in that the subject is considered to exist outside the angle of view range of the close-up camera. It also includes a procedure to determine if each 3D model of the subject is closer to the angle of view boundary of the camera.
The virtual viewpoint image generation program according to claim 14, wherein the 3D model existing on the angle of view boundary is texture-mapped from the camera image of the pulling camera including the inside of the angle of view of the close-up camera.

Images