JP2010020487A - Optional viewpoint video generation device and optional viewpoint video generation program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an optional viewpoint video generation device and an optional viewpoint video generation program, allowing generation of a high-quality interpolation video. <P>SOLUTION: This optional viewpoint video generation device 70 generating a video of a three-dimensional object 80 viewed from an optional virtual viewpoint by use of texture images obtained from a plurality of photographic points includes: a blend texture image selection means 40 selecting a plurality of texture images according to priority order based on a view line direction and a three-dimensional position of each photographic point, and a view line direction and a three-dimensional position of the virtual viewpoint; a blend texture image generation means 50 determining a blend rate based on the view line direction and the three-dimensional position in each photographic point, and the view line direction and the three-dimensional position of the virtual viewpoint, and blending the plurality of texture images to generate a blend texture image; and an interpolation video generation means 60 texture-mapping the blend texture image to a three-dimensional model 90, and generating the two-dimensional interpolation video of the object. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to an arbitrary viewpoint video generation device and an arbitrary viewpoint video generation program, and in particular, a specified arbitrary virtual viewpoint using a texture image obtained by shooting a three-dimensional object from a plurality of shooting points. The present invention relates to an arbitrary viewpoint video generation device and an arbitrary viewpoint video generation program for generating an image that can be seen from a user.

  Conventionally, when displaying an image of an object having a solid shape viewed from an arbitrary viewpoint designated by the user (hereinafter referred to as a “virtual viewpoint”), the object is imaged from a plurality of imaging points, There has been proposed a technique for generating and displaying a video by texture mapping on the surface of a three-dimensional model using a photographed image.

  In this method, the selection of texture data used in texture mapping does not depend on visual work, and appropriate texture data can be automatically selected, and the surface of a three-dimensional model is constructed in order to obtain a clear and realistic three-dimensional model. Compare the normal vector of the target surface with the vector of the line-of-sight direction of the shooting point, and paste the texture of the image shot from the shooting point where the optical axis vector of the shooting point is almost parallel to the surface vector of the surface to the surface of the model surface Has been proposed (see, for example, Patent Document 1).

In addition, in order to display a high-definition texture-mapped 3D model, priority is given to a camera image in which the vector (display vector) in the line-of-sight direction from the display viewpoint and the shooting vector are nearly parallel, Among them, viewpoint-dependent texture mapping is also proposed in which a video of a camera that is visible from each vertex is mapped (see, for example, Patent Document 2).
JP-A-9-134449 JP 2006-72805 A

  However, the general texture mapping method described above cannot accurately acquire parameters such as the three-dimensional position of the shooting point and the direction of the line of sight, or cannot generate an accurate shape of the three-dimensional model, There is a problem that the position of the texture to be mapped is shifted and an unnatural arbitrary viewpoint video is generated.

  In addition, the method of selecting texture image data to be mapped to each vertex and each surface of the 3D model surface described in Patent Document 1 described above depends on the 3D position and viewpoint direction of the viewpoint displaying the 3D model. However, there is a problem in that appropriate texture image data is not necessarily selected and a high-quality three-dimensional model cannot be displayed depending on the situation.

  In addition, the viewpoint-dependent texture mapping method described in Patent Document 2 has different visible states of each camera depending on the location of the surface of the three-dimensional model. Therefore, the texture image to be mapped is different, and the mapped texture image is different. A discontinuous texture is generated near the boundary of the part. Further, when the viewpoint is moved, the visible state is changed according to the position of the viewpoint, so that the texture image to be mapped is also changed, which causes a phenomenon such as flickering. Furthermore, when displaying a moving image using this method, the visible state of each camera changes over time on a part of the surface of the three-dimensional model at the same position. There is also a problem that the images to be switched also change, the continuity in the time axis direction becomes thin, and a phenomenon such as flickering occurs.

  The present invention has been made in view of the above-described problems, and an object thereof is to provide an arbitrary viewpoint video generation apparatus and an arbitrary viewpoint video generation program capable of generating a high-quality interpolated video.

In order to achieve the above object, an arbitrary viewpoint video generation device according to a first invention uses a texture image obtained by shooting a three-dimensional target object from a plurality of shooting points, and specifies an arbitrary virtual image. An arbitrary viewpoint video generation device that generates video viewed from a viewpoint,
A plurality of textures that give priority to each texture image based on the three-dimensional position and line-of-sight direction of each photographing point of the texture image and the three-dimensional position and line-of-sight direction of the virtual viewpoint and mix according to the priority order Blend texture image selection means for selecting an image;
Blending of texture images to be mixed based on the three-dimensional position and line-of-sight direction at each photographing point of the plurality of texture images selected by the blend texture image selection means and the three-dimensional position and line-of-sight direction of the virtual viewpoint Blend texture image generation means for determining a rate and mixing the plurality of texture images according to the blend rate to generate a blend texture image;
Interpolation video generation means for texture-mapping the blend texture image generated by the blend texture image generation means into a three-dimensional model and generating a two-dimensional interpolation video of the object seen from the virtual viewpoint; It is characterized by that.

  As a result, even if an error occurs in the camera parameters such as the 3D position and line-of-sight direction of the photographed camera, or an error occurs in the 3D model, the texture image mapped to the 3D model has a plurality of textures. Therefore, the discontinuity of the texture caused by the error does not appear remarkably. In addition, since the blend rate is in accordance with the relationship between the virtual viewpoint and the camera in the three-dimensional position and the line-of-sight direction, a high-quality interpolated video can be generated.

A second invention is an arbitrary viewpoint video generation device according to the first invention, wherein:
The blend texture image selection means assigns the priorities in order of increasing distance between the three-dimensional position of each shooting point and the three-dimensional position of the virtual viewpoint, or the line-of-sight direction of each shooting point is the virtual viewpoint. The priorities are assigned in order from the closest to the viewing direction.

  As a result, a texture photographed from a photographing point close to the virtual viewpoint or close to the virtual viewpoint direction is preferentially selected, and an appropriate texture image can be selected. In addition, a high-quality interpolated video can be generated and displayed.

3rd invention is the arbitrary viewpoint image | video production | generation apparatus which concerns on 1st or 2nd invention,
The blended texture image generation means determines that the blending rate increases as the distance between the three-dimensional position of the shooting point and the three-dimensional position of the virtual viewpoint is shorter, or as the line-of-sight direction of the shooting point is closer to the line-of-sight direction of the virtual viewpoint. Is determined to be high.

  As a result, the texture image shot from the shooting point closer to the virtual viewpoint or closer to the virtual viewpoint direction is mixed so as to be reflected in the blend texture image with higher weighting, so the reliability is higher and higher. A high quality interpolated video can be generated.

A fourth invention is an arbitrary viewpoint video generation device according to any one of the first to third inventions,
The blend texture image selection means re-selects the blend texture image when a predetermined time elapses or the position of the virtual viewpoint is moved by a predetermined distance or more.

  This allows the virtual viewpoint to move over time by re-selecting the blend texture image following the passage of time and movement of the virtual viewpoint, and re-executing the texture mapping using the re-selected blend texture image. In such a case, it is possible to generate a two-dimensional video interpolated with high quality at the virtual viewpoint at that time.

A fifth invention is an arbitrary viewpoint video generation device according to any one of the first to fourth inventions,
The blend texture image selection means reselects the blend texture image when the object has moved a predetermined distance or more.

  Thus, even when an arbitrary viewpoint video of a moving object such as a human being or an animal is generated and displayed as a moving image, a high-quality interpolated moving image can be generated and displayed.

A sixth invention is an arbitrary viewpoint video generation device according to any one of the first to fifth inventions,
The blend texture image selection means reselects the blend texture image when the virtual viewpoint and the three-dimensional model move relative to each other by a predetermined distance or more.

  As a result, when the virtual viewpoint and the target object move relatively, the blend texture image is reselected, and texture mapping is performed again using the reselected blend texture image. Even if the single movement itself is small, reselection can be appropriately performed when the movement is relatively large. In addition, if the blend texture image is reselected only when the virtual viewpoint and the target object move relatively, and texture mapping is executed using the reselected blend texture image, the internal efficiency can be improved. An inset image can be generated.

A seventh invention is an arbitrary viewpoint video generation device according to any one of the first to sixth inventions,
Visible area extraction means for acquiring a visible area from the photographing point for each texture drawing unit of the surface constituting the three-dimensional model,
The blend texture image selection means is set to select a predetermined number of texture images in order from the highest priority, and is selected from among the texture images included in the visible area acquired by the visible area extraction means. The predetermined number is selected.

  As a result, only texture images shot from shooting points that are visible from the mapping polygon (with visibility) can be selected and mixed for each polygon, and only appropriate and highly effective texture images can be selected. By selecting and mixing, it is possible to reliably generate a high-quality interpolated image.

An eighth invention is the arbitrary viewpoint video generation device according to any one of the first to sixth inventions,
Visible area extraction means for acquiring a visible area from the photographing point for each texture drawing unit of the surface constituting the three-dimensional model,
The blended texture image generating means excludes the texture image when the selected texture images include a texture image not within the visible area acquired by the visible area extracting means. The blend ratio is determined.

  As a result, if the selected texture image contains a texture image taken from a shooting point that is not visible from the polygon to be mapped, this is excluded, and the blend rate is determined only from the texture image that is visible. It is possible to determine a high-quality arbitrary viewpoint video by mixing texture images using only appropriate texture images.

An arbitrary viewpoint video generation program according to a ninth aspect of the invention generates an image that can be seen from a specified arbitrary virtual viewpoint, using texture images obtained by shooting a three-dimensional object from a plurality of shooting points. An arbitrary viewpoint video generation program for causing a computer to execute processing,
A plurality of textures that give priority to each texture image based on the three-dimensional position and line-of-sight direction of each photographing point of the texture image and the three-dimensional position and line-of-sight direction of the virtual viewpoint and mix according to the priority order Blend texture selection process to select an image,
The blend ratio of each texture image to be mixed based on the three-dimensional position and the line-of-sight direction at each photographing point of the plurality of texture images selected by the blend texture selection process and the three-dimensional position and the line-of-sight direction of the virtual viewpoint Blend texture image generation processing for generating a blend texture image by mixing the plurality of texture images according to the blend rate;
An interpolated image generating process for texture-mapping the blended texture image generated by the blended texture image generating process into a three-dimensional model and generating a two-dimensional interpolated image of the object seen from the virtual viewpoint; It is made to perform.

  As a result, the three-dimensional position and line-of-sight direction of a plurality of shooting points at which the object is photographed are compared with the three-dimensional position and line-of-sight direction of the virtual viewpoint, and priorities are given to the texture images, and the texture to be blended By outputting an image and texture-mapping the blended texture image obtained by blending these textures according to the blending ratio onto the three-dimensional model, a high-quality interpolated video can be displayed. In addition, it is possible to realize the interpolated image generation process at a low cost without requiring a special device configuration. Furthermore, by installing the program, it is possible to easily realize the interpolated video generation process.

  According to the present invention, it is possible to generate a high-quality image from an arbitrary designated viewpoint by interpolation using images photographed from a plurality of photographing points.

  The best mode for carrying out the present invention will be described below with reference to the drawings.

  FIG. 1 is a diagram illustrating an example of a camera arrangement in an arbitrary viewpoint video generation apparatus and an arbitrary viewpoint video program according to an embodiment to which the present invention is applied. In FIG. 1, a plurality of cameras Cam are arranged so as to surround the periphery of an object 80 having a three-dimensional shape. In FIG. 1, 40 cameras Cam are arranged around the object 80.

  In the arbitrary viewpoint video generation apparatus and the arbitrary viewpoint video generation program according to the present embodiment, such a plurality of cameras Cam are arranged around the object 80 having a three-dimensional shape, and each of the cameras Cam1 to Cam40 sets the object 80. Using each captured texture image, an image that three-dimensionally reproduces the object 80 is generated. In that case, the user who visually recognizes the image can freely select the viewpoint for visually recognizing the object 80 as a virtual viewpoint at an arbitrary three-dimensional position. The arbitrary viewpoint video generation apparatus and the arbitrary viewpoint video display program according to the present embodiment take images of the object 80 from various angles using, for example, a large number of cameras Cam as shown in FIG. Obtain as an image. Then, the texture image is pasted on the polygon that reproduces the object 80 to obtain a three-dimensional image. The object 80 has a substance and can be a three-dimensional object having various three-dimensional shapes as the object 80, and the shape and form thereof are not limited. In addition, in FIG. 1, among the 40 cameras Cam, Cam1 to Cam24 surround the object 80 in an annular shape and are arranged so as to surround the object 80 also above and below the object 80. The number of the cameras Cam may be larger or smaller, and the arrangement of the cameras Cam can be in various forms as long as the image of the object 80 can be acquired from various angles.

  FIG. 2 is a diagram illustrating a schematic configuration example of an embodiment of an arbitrary viewpoint video generation system including an arbitrary viewpoint video generation device 70 according to an example to which the present invention is applied. In FIG. 2, the arbitrary viewpoint video generation system includes a plurality of cameras Cam as imaging devices, a visible viewpoint information acquisition unit 10, a visible region extraction device 30, a three-dimensional shape generation device 20, and an arbitrary viewpoint video generation device 70. And have. The arbitrary viewpoint video generation device 70 is configured to include a blend texture image selection unit 40, a blend texture image generation unit 50, and an interpolated video generation unit 60. The visible region extraction unit 30 may be configured to be included in a part of the arbitrary viewpoint video generation device 70.

  The plurality of cameras Cam are image capturing means for capturing the object (entity) 80 from different image capturing points. If the object 80 can be photographed, various cameras such as a CCD (Charge Coupled Device) camera, a CMOS (Complementary Metal Oxide Semiconductor) camera, and an analog camera can be applied.

  FIG. 3 is a diagram illustrating an example of how the object 80 is photographed by a plurality of cameras Cam. In FIG. 3 and subsequent figures, the number of cameras Cam shown in the figure is four for ease of explanation, but as described above, the number of cameras applied to the arbitrary viewpoint video generation apparatus 70 according to the present embodiment. About the number of installation, you may make it provide more cameras Cam.

  As shown in FIG. 3, the plurality of cameras Cam <b> 1 to Cam <b> 4 perform shooting in synchronization with the object (entity) 80 from the respective shooting points CS <b> 1 to CS <b> 4 and acquire the shot multi-viewpoint images. In FIG. 3, a front view image of the cat object 80 is acquired from the shooting point CS1 where the camera Cam1 is provided, and a left side view image of the object 80 is acquired from the shooting point CS2 where the camera Cam2 is provided. Has been acquired. Similarly, a rear view image of the object 80 is taken from the photographing point CS3 where the camera Cam3 is arranged, and a right side view image of the cat which is the object 80 is taken from the photographing point CS4 where the camera Cam4 is arranged. Have been filmed.

  In this way, images of the object 80 are captured and acquired from multiple viewpoints. At this time, 3 at each of the shooting points CS1 to CS4 (hereinafter collectively referred to as “CS”) of the plurality of cameras Cam1 to Cam4. Information on the dimensional position (in FIG. 3, three-dimensional coordinates including XYZ axes) and the line-of-sight directions of the cameras Cam1 to Cam4 are also acquired. In addition, the camera lens focal length of the camera lens previously recorded in the cameras Cam1 to Cam4 and, when a CCD camera is used, structural information including internal parameters such as the size thereof are also acquired.

  Returning to FIG. The virtual viewpoint information acquisition unit 10 is a unit that acquires information on the virtual viewpoint selected by the user, and specifically acquires information including the three-dimensional position and the line-of-sight direction of the virtual viewpoint. The virtual viewpoint information acquisition unit 10 is connected to, for example, a screen controller that allows the user to input a virtual viewpoint, or an input unit such as a mouse or a pointer. Such information may be acquired.

  The three-dimensional shape generation apparatus 20 generates a three-dimensional model from an image of the object 80 obtained by photographing a plurality of cameras Cam1, Cam2,..., Camn (hereinafter collectively referred to as “Cam”). For example, it can be generated by using a known method such as a view volume intersection method or a method using a combination of a view volume intersection method and a stereo matching method as disclosed in Patent Document 1.

  Here, in the visual volume intersection method, a plurality of silhouette images obtained by extracting an object region from a captured image of an object are back-projected into the original three-dimensional space, and the intersection of the respective visual volumes is obtained, thereby obtaining the 3 of the object. This is a technique for obtaining dimensional shape data. In addition, the stereo matching method uses a set of two images taken by two cameras arranged on the left and right, and the image taken by the left camera is the image of the image taken by the right camera. This is a technique for determining whether or not to correspond to a part by calculating area correlation and estimating the three-dimensional position of each point by triangulation using the correspondence. For example, the three-dimensional shape generation apparatus of the arbitrary viewpoint video generation system according to the present embodiment may generate a three-dimensional model using such a known method. Note that the three-dimensional model may be generated using XYZ coordinates or the like.

  The visible region extraction device 30 includes a three-dimensional model of the object 80 obtained from photographing by a plurality of cameras Cam, a texture image obtained by photographing the object 80 from a plurality of photographing points CS, and a photographing point CS obtained by photographing the texture image. Based on the three-dimensional position and the line-of-sight direction and the structure information of each camera Cam that has photographed the object 80, the vertex or surface constituting the region of the visible portion of the three-dimensional model is obtained for each photographing point CS, and the texture image is obtained. On the other hand, it is means for acquiring region information (visible region information) of the visible portion of the three-dimensional model from the photographing point CS. Extraction of the visible region by the visible region extraction device 30 can be obtained by using a known method disclosed in Patent Document 2, for example.

  In FIG. 2, the visible region extraction device 30 is shown so that the extraction result is sent to both the blend texture image selection unit 40 and the blend texture image generation unit 50. It may be configured to be sent to both, or may be configured to be sent to both.

  The blended texture image selection means 40 in the arbitrary viewpoint video generation device 70 first includes virtual viewpoint information including the three-dimensional position and line-of-sight direction of the virtual viewpoint acquired by the virtual viewpoint information acquisition means 10, and a plurality of shooting points CS. This is a means for comparing the dimensional position and the line-of-sight direction and assigning priority among texture images photographed from a plurality of photographing points CS.

  As a method of assigning priorities, for example, among the texture images photographed from a plurality of photographing points CS, the priorities are given in the order of the closest distance between the three-dimensional coordinates of the photographing point CS and the three-dimensional coordinates of the virtual viewpoint. It may be. In this case, the priority order can be determined by calculating the distance between the three-dimensional coordinates of the shooting point CS and the three-dimensional coordinates of the virtual viewpoint for each shooting point CS.

  Alternatively, the line-of-sight directions of the imaging point CS may be compared, and texture mapping priorities may be given in the order of being parallel to the line-of-sight direction of the virtual viewpoint. In this case, the line-of-sight vector from the photographing point CS is obtained from the three-dimensional coordinates of the photographing point CS and a predetermined point such as the center of gravity included in the three-dimensional model, and the three-dimensional coordinates of the virtual viewpoint and the three-dimensional model are obtained. A line-of-sight vector from a virtual viewpoint may be obtained from a predetermined point such as the center of gravity included, an inner product of both line-of-sight vectors may be calculated, and a priority order may be determined in ascending order of the inner product. Further, for example, in addition to using the center of gravity, the shooting direction of each camera Cam may be calculated from the calibration, and this may be used as the line-of-sight vector. Thus, the calculation and comparison of the line-of-sight direction from the imaging point CS can be performed using various methods.

  Furthermore, the method of assigning priority by the above-described three-dimensional coordinates and the method of assigning priority by the line-of-sight direction may be combined. Further, two or more texture images with high priority are selected from the assigned texture images.

  Here, the virtual viewpoint will be described with reference to FIG. FIG. 4 is a diagram illustrating an example of the virtual viewpoint VP. FIG. 4 shows a viewpoint when the three-dimensional model 90 generated from a plurality of images obtained by photographing the object 80 in FIG. 3 is displayed on the screen of a display device such as a display. That is, the virtual viewpoint VP can be moved in any direction on the XYZ axes depending on the setting of the user at the time of reproduction.

  Returning to FIG. The blend texture image generation unit 50 calculates a plurality of texture images output from the blend texture image selection unit 40 from the three-dimensional position and line-of-sight direction of the shooting point CS and the three-dimensional position and line-of-sight direction of the virtual viewpoint VP. To generate and output a blended texture image.

  As a method for giving the blend rate, for example, there is a method in which the blend rate is given higher and the degree of influence is increased as the distance between the three-dimensional coordinate of the virtual viewpoint VP and the three-dimensional coordinate of the shooting point CS is closer.

  FIG. 5 is a diagram showing an example of a method for determining the blend rate based on the distance from the virtual viewpoint VP. For example, as shown in FIG. 5, a perpendicular line V1 is dropped from a virtual viewpoint VP to a plane P composed of the cameras Cam1 and Cam2 of the two shooting points CS1 and CS2 having a high priority and the center of gravity G of the object 80. A point a intersecting with P is calculated. In FIG. 5, a virtual camera VCam is shown corresponding to the virtual viewpoint VP. Next, a straight line L1 passing through the center of gravity G of the object 80 and the point a on the plane P is stretched, and a point b intersecting with the straight line L2 connecting the cameras Cam1 and Cam2 of the two photographing points CS1 and CS2 is calculated. The distances d1 and d2 from the cameras Cam1 and Cam2 of the two shooting points CS1 and CS2 to the point b can be calculated as the blend ratio. By blending the selected two texture images according to the following formula (1) from the blend ratios of the distances d1 and d2, a blend texture image can be acquired.

（１）式から、カメラＣａｍ１が仮想視点ＶＰに接近する程、ｄ１は小さくなるので、カメラＣａｍ１の係数（１−ｄ１／（ｄ１＋ｄ２））は大きくなり、カメラＣａｍ１によるテクスチャ画像の混合比率が高くなることが分かる。同様に、カメラＣａｍ２の方が仮想視点ＶＰに接近した場合には、ｄ１が大きくなるので、カメラＣａｍ２の係数（ｄ１／（ｄ１＋ｄ２））が大きくなり、カメラＣａｍ２によるテクスチャ画像の混合比率が高くなることが分かる。

From the equation (1), as the camera Cam1 approaches the virtual viewpoint VP, d1 decreases, so the coefficient (1-d1 / (d1 + d2)) of the camera Cam1 increases and the texture image mixing ratio by the camera Cam1 increases. I understand that Similarly, when the camera Cam2 is closer to the virtual viewpoint VP, d1 increases, so the coefficient (d1 / (d1 + d2)) of the camera Cam2 increases and the texture image mixing ratio by the camera Cam2 increases. I understand that.

  As described above, as a method of determining the blend ratio, after selecting a plurality of cameras Cam1 and Cam2 having higher priorities, the cameras at the shooting points CS1 and CS2 that are closer to the virtual viewpoint VP according to the distance from the virtual viewpoint VP. The blend ratio may be determined so that the mixing ratio of the texture images of Cam1 and Cam2 is higher. Thereby, an image closer to the actual visual recognition state from the virtual viewpoint VP can be generated.

  Next, a method for determining a blend ratio different from that in FIG. 5 will be described with reference to FIG. FIG. 6 is a diagram for explaining an example of a blend rate calculation method according to the angle formed by the virtual viewpoint direction and the shooting direction, which is different from FIG. 5.

  In FIG. 6, a perpendicular line V2 is drawn from a virtual viewpoint VP at which the virtual camera VCam is provided to a plane Q composed of the cameras Cam1 and Cam2 of the two shooting points CS1 and CS2 having high priority and the center of gravity G of the object 80. A point c that intersects with the plane Q is calculated. Next, a straight line L3 connecting the center of gravity G of the object 80 and the point c on the plane Q is considered, and an angle φ1 formed between the center of gravity G and the straight lines L4 and L5 connecting the cameras Cam1 and Cam2 of the photographing points CS1 and CS2. φ2 can be the blend rate. In mixing, a blend texture image is acquired by the following equation (2).

（２）式においても、図５の場合と同様に、仮想視点ＶＰと視線方向が平行に近く、角度φ１、φ２のより小さい方のテクスチャ画像の混合比率が高くなるような式となっている。つまり、カメラＣａｍ１の視線方向が仮想視点ＶＰの視線方法により近く、角度φ１がより小さくなる程、カメラＣａｍ１の係数（１−φ１／（φ１＋φ２））は大きくなり、カメラＣａｍ１によるテクスチャ画像の混合比率が高まる。同様に、カメラＣａｍ２の視線方向が仮想視点ＶＰにより近くなると、角度φ２が小さくなり、角度φ１が大きくなるので、カメラＣａｍ２のテクスチャ画像の混合比率がより高まることになる。

Also in the equation (2), as in the case of FIG. 5, the virtual viewpoint VP and the line-of-sight direction are nearly parallel, and the expression ratio of the texture image having the smaller angles φ1 and φ2 is increased. . That is, the closer the viewing direction of the camera Cam1 is to the viewing method of the virtual viewpoint VP and the smaller the angle φ1, the larger the coefficient of the camera Cam1 (1-φ1 / (φ1 + φ2)), and the mixing ratio of the texture image by the camera Cam1 Will increase. Similarly, when the line-of-sight direction of the camera Cam2 is closer to the virtual viewpoint VP, the angle φ2 becomes smaller and the angle φ1 becomes larger, so that the texture image mixing ratio of the camera Cam2 is further increased.

  As described above, the calculation of the blend ratio may be performed based on the relationship between the line-of-sight direction of the virtual viewpoint VP and the line-of-sight direction of the selected camera points CS1 and CS2 of the photographing points CS1 and CS2. . As a result, the mixing ratio of the cameras Cam1 and Cam2 at the photographing points CS1 and CS2 closer to the visual line direction of the virtual viewpoint VP can be increased, and a blended image closer to the viewing state from the virtual viewpoint VP can be generated. .

  5 and 6, an example in which two texture images with high priority are selected by the blend texture selecting unit 40 and the two texture images are mixed has been described. Next, referring to FIG. A case where a plurality of texture images are mixed will be described.

  FIG. 7 is a diagram for explaining an example of a blend rate calculation method when three texture images are mixed. In FIG. 7, three cameras Cam1, Cam2, and Cam3 of the shooting points CS1, CS2, and CS3 with high priority that are close to the virtual viewpoint VP or close to the line-of-sight direction are selected.

  In this blend ratio calculation method, as shown in FIG. 7, a virtual camera VCam is applied to a plane R composed of cameras Cam1, Cam2, and Cam3 of three shooting points CS1, CS2, and CS3 having a high priority. A vertical line V3 is dropped from the virtual viewpoint VP provided with the point O, and a point O that intersects the plane R is calculated.

  A perpendicular is drawn from the point O to the sides L6, L7, and L8 connecting the photographing points Cam1, Cam2, and Cam3, and the respective lengths d1, d2, and d3 can be set as the blend ratio. According to these blend ratios, a blend texture image can be acquired using the following formula (3).

（３）式においても、例えば仮想視点ＶＰが撮影点ＣＳ１のカメラＣａｍ１に接近すれば、対角のｄ１が大きくなり、撮影点ＣＳ１のカメラＣａｍ１におけるテクスチャ画像の混合比率が高くなるような式となっている。これは、図７において、仮想視点ＶＰから最も離れて距離の大きい撮影点ＣＳ２のカメラＣａｍ２の対角のｄ２が小さくなっていることからも理解できる。

In formula (3), for example, if the virtual viewpoint VP approaches the camera Cam1 at the shooting point CS1, the diagonal d1 increases, and the texture image mixing ratio at the camera Cam1 at the shooting point CS1 increases. It has become. This can also be understood from the fact that the diagonal d2 of the camera Cam2 of the photographing point CS2 that is the farthest away from the virtual viewpoint VP in FIG. 7 is small.

  As described above, the plurality of imaging points CS to be selected may be two or three as long as they are plural, and can be set to a predetermined number according to the application. In FIGS. 5 to 7, the case where the number of shooting points CS to be selected is only two and three is described. However, for example, the number may be four or four or more. Also good. In any case, if the blend rate is calculated so that the blending ratio becomes higher as the shooting point CS is closer to the virtual viewpoint VP or closer to the visual line direction, a smooth blend texture image with less flickering is generated. Can do.

  Here, by using the visible region extraction device 30 described in FIG. 2, a high-quality blend texture image can be acquired. First, N cameras Cam are selected from the virtual cameras according to the above-described priority order. Next, the top M cameras Cam to be blended for each polygon, which is the minimum structural unit of the texture image, are selected according to the priority order of the blend texture selecting means 40. However, at this time, the camera Cam that is not visible (Visible) is excluded from the polygon. The selected M texture images are mixed by the blending method described above to generate a blended texture image.

  In addition, the texture image of the camera Cam that is not visible may be excluded when the blend texture image selection unit 40 selects the blend texture image, or immediately before the blend texture image generation unit 50 generates the blend texture image. You may go. When the texture image by the camera Cam in the invisible state is excluded by the blend texture image selecting unit 40, the top M units are selected from the texture images in the visible state. In this case, the M cameras Cam are all visible, but there may be a case where the combination of cameras differs for each polygon. On the other hand, when the blended texture image generation unit 50 excludes the texture image from the invisible camera Cam, the texture image from the invisible camera is excluded from the selected texture images of the M high-priority cameras Cam. To do. Thereby, the number of texture images used for each polygon may be different, but the combination of cameras is the same. For these, an appropriate method may be adopted as appropriate according to the application.

  Here, the three-dimensional coordinates and the line-of-sight direction of the virtual viewpoint VP can be obtained using a three-dimensional programming language such as OpenGL (Open Graphics Library), for example. Further, the three-dimensional coordinates and the line-of-sight directions of the plurality of shooting points CS can be acquired by performing camera calibration or the like that takes a correspondence between the position of the real space and the position of the virtual space on the image at the time of shooting, for example. it can.

  The interpolated image generation device 60 receives the blended texture image, the three-dimensional model 90, and virtual viewpoint information including the three-dimensional coordinates and the line-of-sight direction of the virtual viewpoint VP, and texture-maps the blended texture image to the three-dimensional model 90. Then, a three-dimensional model 90 that is visible from the virtual viewpoint VP is generated. Thereby, the video from the virtual viewpoint VP in which no camera is installed can be generated with high quality.

  Next, a method for dealing with moving images will be described. Since the arbitrary viewpoint video generation device 70 described above generates an interpolated video depending on the virtual viewpoint VP, even when the virtual viewpoint VP moves or the object 80 moves, the high-definition video is generated. An interpolated video can be generated.

  In this case, the surface vertex group or the surface group also in the visible region extraction device 30 when the predetermined time has elapsed or the virtual viewpoint VP follows the movement of the virtual viewpoint VP accompanying a change of a predetermined distance or the like. Is obtained again to obtain the visible region information of the three-dimensional model 90. The blend texture image selection means 40 reselects the texture image, the blend texture image generation means 50 recalculates the blend rate, and then the blend texture image is generated. Further, the blended texture image generating means 50 newly generates a blended texture image using these pieces of information, and then the texture mapping is performed on the three-dimensional model 90 by the interpolated video generating device 60, whereby a moving image is generated. Can also respond.

  The follow-up described above is not limited to the movement of the virtual viewpoint VP, and can be similarly applied to the case where the object 80 moves, such as a person or an animal. Thereby, even when the moving object 80 such as a human being or an animal 80 is displayed as a moving image, it is possible to generate an interpolated video image with high quality.

  Alternatively, a texture image may be reselected only when the virtual viewpoint VP and the three-dimensional model 90 move relatively, and the blend rate may be updated to generate a blend texture image. In this case, as a position of the three-dimensional model 90, for example, a predetermined position such as the center of gravity may be used as a reference.

  As a result, the texture image is re-selected only when the virtual viewpoint VP and the three-dimensional model 90 are relatively moved, and the texture mapping of the blend texture is re-executed according to the new blend rate for the re-selected texture. Therefore, an efficiently interpolated video can be generated.

  Here, an execution program capable of causing a computer to execute the processing in each configuration of the interpolated video generation apparatus 60 in the present invention is generated, and the program is installed in, for example, a general-purpose personal computer, workstation, etc. The display of the interpolated image generated in the invention can be realized.

  Next, a hardware configuration example of a computer capable of executing the interpolated video generation processing of the arbitrary viewpoint video generation apparatus 70 according to the present embodiment will be described with reference to FIG. FIG. 8 is a diagram illustrating an example of a hardware configuration capable of realizing the interpolated video generation processing of the arbitrary viewpoint video generation device 70 according to the embodiment to which the present invention is applied.

  8 includes an input device 110, an output device 120, a drive device 130, an auxiliary recording device 140, a memory device 150, a CPU (Central Processing Unit) 160 that performs various controls, and a network connection device 170. These are connected to each other by a system bus 180.

  The input device 110 has a pointing device such as a keyboard and a mouse operated by a user, and inputs various operation signals such as execution of a program from the user. The output device 120 has a display for displaying various windows and data necessary for operating the computer main body for performing processing in the present embodiment, and the program execution progress and interpolation are controlled by the control program of the CPU 160. Video generation results and the like can be displayed.

  Here, the execution program installed in the computer main body in the present embodiment is provided by the recording medium 100 such as a CD-ROM, for example. The recording medium 100 on which the program is recorded can be set in the drive device 130, and an execution program included in the recording medium 100 is installed from the recording medium 100 to the auxiliary storage device 140 via the drive device 130. Therefore, the processing executed by the blend texture image selection unit 40, the blend texture image generation unit 50, and the interpolated video generation unit 60 of the arbitrary viewpoint video generation device 70 shown in FIG. 2 is recorded on the recording medium 100 as a computer program. In this case, the function of the arbitrary viewpoint video generation apparatus 70 according to the present embodiment can be executed by the computer shown in FIG. 8 as the arbitrary viewpoint video generation program according to the present embodiment. In this case, the arbitrary viewpoint video generation program constitutes a program that causes the computer to sequentially execute the blend texture image selection process, the blend texture image generation process, and the interpolated video generation process.

  The auxiliary storage device 140 is a storage unit such as a hard disk, and can store an execution program in this embodiment, a control program provided in a computer, and the like, and can perform input / output as necessary.

  Based on a control program such as an OS (Operating System) and an execution program read and stored by the memory device 150, the CPU 160 performs various operations and data input / output with each hardware component, etc. Each process in the interpolated image generation display can be realized by controlling the process. Various information necessary during the execution of the program can be acquired from the auxiliary storage device 140 and can also be stored.

  The network connection device 170 obtains an execution program from another terminal connected to the communication network by connecting to a communication network or the like, or an execution result obtained by executing the program or in this embodiment. The execution program itself can be provided to other terminals.

  The hardware configuration shown in FIG. 8 does not require a special device configuration, and can realize the generation / display processing of the interpolated video at a low cost. Further, by installing the arbitrary viewpoint video generation program, it is possible to easily realize the interpolated video generation / display processing.

  Next, the processing flow of the interpolated video generation / display process executed by the arbitrary viewpoint video generation device 70 and the arbitrary viewpoint video generation system according to the present embodiment will be described with reference to FIG. FIG. 9 is a flowchart of an example for realizing the interpolated video generation / display processing executed by the arbitrary viewpoint video generation device 70 and the arbitrary viewpoint video generation program according to the present embodiment. In addition, in the arbitrary viewpoint video generation device 70 and the arbitrary viewpoint video generation program according to the present embodiment, the process from the acquisition of the texture image to the generation of the interpolated video is performed. In the processing flow of FIG. It includes the acquisition of the texture image performed before that and the display processing of the interpolated video performed after that.

  In step 200, the texture image photographed synchronously by the plurality of cameras Cam, the three-dimensional coordinates and line-of-sight directions of the photographing point CS of the photographed camera Camn, and the three-dimensional coordinates and line-of-sight directions of the virtual viewpoint VP (virtual viewpoint information). Is entered.

  In step 210, based on each information acquired in step 200, a blend texture image is selected in the blend texture image selection means 40 or the blend texture image selection process. In step 210, in order to blend a plurality of texture images to generate a blend texture, texture image candidates to be blended are first determined. When selecting the texture image of the shooting point CS to be blended, for example, even when the priority order is given to the shooting point CS in the order that the distance of the three-dimensional coordinate of the shooting point CS is close to the three-dimensional coordinate of the virtual viewpoint VP. Good. Alternatively, the direction of the virtual line of sight may be compared with the line-of-sight direction of the shooting direction of each shooting point CS, and the priority order may be given to the shooting points CS in the order of being parallel to the direction of the virtual line of sight. Moreover, you may combine the method of giving a priority with the above-mentioned three-dimensional position, and the method of giving a priority with a gaze direction. A plurality of texture images to be blended are selected based on the priority order assigned in this way.

  When selecting, the visible region extraction device 30 calculates the visible state of each shooting point CS for each polygon, and if a method that does not select the shooting point CS in the invisible state is used as a selection target, a higher-quality interpolated image is obtained. Can be generated. When a shooting point CS in an invisible state exists in a candidate with high priority, the texture image of the shooting point CS is discarded and eliminated, and another shooting point CS with higher priority is added and selected. Therefore, when this method is used, the configuration of the texture camera Camn blended for each polygon may be different.

  In step 220, the blend texture image generation means 50 or the blend texture image generation process calculates the blend ratio (weighting) of each texture image for blending. The blend ratio is determined according to the distance between the three-dimensional coordinates of the photographing point CS and the three-dimensional coordinates of the virtual viewpoint VP. Alternatively, the direction of the visual line of sight and the line of sight of the shooting direction are compared, and the direction of the line of sight in the shooting direction is determined according to the degree of parallelism to the direction of the virtual line of sight. Moreover, you may combine the method of providing a blend rate by the above-mentioned three-dimensional position, and the method of providing a blend rate by a gaze direction. Based on the blend ratio thus given, the texture is blended to generate and output a blend texture.

  The reflection of the calculation result of the visible state by the visible region extraction device 30 performed in step 210 may be performed in step 220. That is, when the texture image selected as the blend target includes a texture image of the shooting point CS in the invisible state, at this stage, the texture image of the shooting point CS in the invisible state is discarded and removed. The blend ratio may be calculated using the remaining texture image. When this method is used, the configuration of the texture camera Camn blended for each polygon is the same. Therefore, depending on the application, the texture image of the imaging point CS in the invisible state based on the visible region information is deleted in the step 210, or the texture image of the imaging point CS in the invisible state is excluded in the step 220. It may be determined. In any case, the visible region extraction by the visible region extraction device 30 is not essential, and the visible region extraction device 30 may be provided as necessary to perform the visible region extraction.

  In step 230, in the interpolated video generation means 60 or the interpolated video generation process, texture mapping for pasting each blend texture to the three-dimensional model 90 is performed, and then the video from the virtual viewpoint VP is output. As a result, it is possible to generate and display an interpolated video with smooth image changes between textures and less flicker.

  In step 240, it is determined whether the positions of the three-dimensional model 90 and the virtual viewpoint VP have moved relative to each other by a predetermined distance. When the positions of the virtual viewpoint VP and the three-dimensional model 90 are relatively moved, the process returns to step 210 to select and generate the blend texture again, and generate and display the interpolated image after the movement.

  The determination of whether or not the positions of the three-dimensional model 90 and the virtual viewpoint VP have moved may be performed not only by the relative movement of the two but also by each movement itself. For example, when the three-dimensional model 90 is close to a fixed state and only a large movement of the virtual viewpoint VP is expected, it is determined whether or not the virtual viewpoint VP has moved a predetermined distance or more. In addition, the blended texture image may be generated again. Conversely, when there is little movement of the virtual viewpoint VP and a large movement of the three-dimensional model 90 is expected, it is determined whether or not the three-dimensional model 90 has moved by a predetermined distance and moved by a predetermined distance or more. In this case, the blend texture image may be generated again. These movement thresholds may be set for the virtual viewpoint alone, the three-dimensional model 90 alone, and all of the relative movements of both. Further, in addition to the movement of the position, the blend texture image may be reselected and regenerated when a predetermined time has elapsed. Even in the case of these movements and regeneration of blended texture images over time, it is possible to move images smoothly because all blended texture images are reproducible between images mixed at an appropriate blend rate. .

  If it is determined in step 240 that the positions of the virtual viewpoint VP and the three-dimensional model 90 have not moved relative or alone, the process proceeds to step 250.

  In step 250, it is determined whether or not to generate / display the interpolated video. If the interpolated video generation / display is not terminated, the process returns to step 240 and repeats the determination of whether the positions of the three-dimensional model 90 and the virtual viewpoint VP move relatively. When the interpolated video generation / display is to be ended, the arbitrary viewpoint video generation / display processing is ended.

  As described above, by executing the arbitrary viewpoint video generation / display processing by the arbitrary viewpoint video system or the computer, it is possible to display the arbitrary viewpoint video interpolated with high quality. Also, it is possible to realize the arbitrary viewpoint generation / display processing at a low cost by installing an execution program on a general-purpose computer and executing the above-described arbitrary viewpoint video generation / display processing without requiring a special device configuration. it can.

  As described above, according to the present embodiment, the three-dimensional model 90 texture-mapped with high quality can be displayed. This not only reduces the burden on the producer who is manually performing texture mapping, but also as a blended effect, even if there is an error in the shooting point CS or the acquired 3D model 90, There are no texture discontinuities at the boundaries. Further, even if the viewpoint is moved or the moving object is displayed, an effect that the flicker is not remarkable can be obtained.

  Specifically, in this embodiment, an object having a three-dimensional shape is photographed from a plurality of photographing points CS, a three-dimensional model 90 is created based on the photographed images, and the texture photographed from the plurality of photographing points CS. When texture mapping is performed on the created three-dimensional model 90, textures are selected depending on the virtual viewpoint VP and blended with each other, thereby generating an appropriate interpolated image corresponding to the virtual viewpoint VP. Further, by executing blend texture mapping following the virtual viewpoint VP or the like, a high-quality arbitrary viewpoint video can be output.

  In addition, in this embodiment, when performing a viewpoint-dependent surface texture mapping method that gives a texture to the model surface corresponding to the virtual viewpoint position at the time of display, a texture to be mapped is generated by blending multiple textures. By doing so, it is possible to output a high-quality arbitrary viewpoint video.

  Note that the present embodiment can be used in 3D CG (Computer Graphics) generation and display technology, and can be applied to general texture mapping technology and rendering technology.

  The preferred embodiments of the present invention have been described in detail above. However, the present invention is not limited to the above-described embodiments, and various modifications and substitutions can be made to the above-described embodiments without departing from the scope of the present invention. Can be added.

It is the figure which showed an example of the camera arrangement | positioning in the arbitrary viewpoint video generation apparatus and arbitrary viewpoint video display program which concern on a present Example. It is a figure which shows the example of a schematic structure of the arbitrary viewpoint video generation system containing the arbitrary viewpoint video generation apparatus 70 which concerns on a present Example. It is an example of the figure explaining a mode that the target object 80 is image | photographed with the some camera Cam. It is an example of the figure for demonstrating the virtual viewpoint VP. It is the figure which showed an example of the method of determining a blend rate by the distance with the virtual viewpoint VP. It is a figure for demonstrating an example of the calculation method of the blend rate according to the angle | corner which a virtual viewpoint direction and the imaging | photography direction make. It is a figure explaining an example of the calculation method of the blend rate which mixes three textures. It is the figure which showed an example of the hardware constitutions which implement | achieve the interpolation video production | generation display process which concerns on a present Example. It is an example of the flowchart which implement | achieves the interpolation video production | generation process performed with the arbitrary viewpoint video production | generation apparatus 70 and arbitrary viewpoint video production | generation program which concern on a present Example.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 Virtual viewpoint information acquisition means 20 3D shape generation apparatus 30 Visible region extraction apparatus 40 Blend texture image selection means 50 Blend texture image generation means 60 Interpolation image generation means 70 Arbitrary viewpoint image generation apparatus 80 Object 90 3D model 100 Recording Medium 110 Input device 120 Output device 130 Drive device 140 Auxiliary recording device 150 Memory device 160 CPU
170 Network Connection Device Cam Camera VCam Virtual Camera CS Shooting Point VP Virtual Viewpoint

Claims (9)

An arbitrary viewpoint video generation device that generates an image that can be seen from a specified arbitrary virtual viewpoint using texture images obtained by shooting a three-dimensional object from a plurality of shooting points,
A plurality of textures that give priority to each texture image based on the three-dimensional position and line-of-sight direction of each photographing point of the texture image and the three-dimensional position and line-of-sight direction of the virtual viewpoint and mix according to the priority order Blend texture image selection means for selecting an image;
Blending of texture images to be mixed based on the three-dimensional position and line-of-sight direction at each photographing point of the plurality of texture images selected by the blend texture image selection means and the three-dimensional position and line-of-sight direction of the virtual viewpoint Blend texture image generation means for determining a rate and mixing the plurality of texture images according to the blend rate to generate a blend texture image;
Interpolation video generation means for texture-mapping the blend texture image generated by the blend texture image generation means into a three-dimensional model and generating a two-dimensional interpolation video of the object seen from the virtual viewpoint; An arbitrary viewpoint video generation device characterized by the above.   The blend texture image selection means assigns the priorities in order of increasing distance between the three-dimensional position of each shooting point and the three-dimensional position of the virtual viewpoint, or the line-of-sight direction of each shooting point is the virtual viewpoint. The arbitrary viewpoint video generation device according to claim 1, wherein the priorities are assigned in the order from the closest to the line-of-sight direction.   The blended texture image generation means determines that the blending rate increases as the distance between the three-dimensional position of the shooting point and the three-dimensional position of the virtual viewpoint is shorter, or the line-of-sight direction of the shooting point is closer to the line-of-sight direction of the virtual viewpoint. The arbitrary viewpoint video generation device according to claim 1, wherein the arbitrary viewpoint video generation device is determined.   4. The blend texture image selection unit re-selects the blend texture image when a predetermined time elapses or the position of the virtual viewpoint moves a predetermined distance or more. The arbitrary viewpoint video generation device according to claim 1.   5. The arbitrary viewpoint video generation according to claim 1, wherein the blend texture image selection unit reselects the blend texture image when the object moves a predetermined distance or more. 6. apparatus.   The blend texture image selection unit reselects the blend texture image when the virtual viewpoint and the three-dimensional model move relative to each other by a predetermined distance or more. The arbitrary viewpoint video generation device according to the item. Visible area extraction means for acquiring a visible area from the photographing point for each texture drawing unit of the surface constituting the three-dimensional model,
The blend texture image selection means is set to select a predetermined number of texture images in order from the highest priority, and is selected from among the texture images included in the visible area acquired by the visible area extraction means. The arbitrary viewpoint video generation apparatus according to any one of claims 1 to 6, wherein the predetermined number is selected. Visible area extraction means for acquiring a visible area from the photographing point for each texture drawing unit of the surface constituting the three-dimensional model,
The blended texture image generating means excludes the texture image when the selected texture images include a texture image not within the visible area acquired by the visible area extracting means. The arbitrary viewpoint video generation apparatus according to claim 1, wherein the blend ratio is determined. Arbitrary viewpoint video generation program for causing a computer to execute processing for generating an image viewed from an arbitrary virtual viewpoint using texture images obtained by photographing a three-dimensional object from a plurality of photographing points. Because
A plurality of textures that give priority to each texture image based on the three-dimensional position and line-of-sight direction of each photographing point of the texture image and the three-dimensional position and line-of-sight direction of the virtual viewpoint and mix according to the priority order Blend texture selection process to select an image,
The blend ratio of each texture image to be mixed based on the three-dimensional position and the line-of-sight direction at each photographing point of the plurality of texture images selected by the blend texture selection process and the three-dimensional position and the line-of-sight direction of the virtual viewpoint Blend texture image generation processing for generating a blend texture image by mixing the plurality of texture images according to the blend rate;
An interpolated image generating process for texture-mapping the blended texture image generated by the blended texture image generating process into a three-dimensional model and generating a two-dimensional interpolated image of the object seen from the virtual viewpoint; Arbitrary viewpoint video generation program, characterized in that the program is executed.

Images