JP2014067372A - Motion parallax reproduction program and device and three-dimensional model generation program and device therefor - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a motion parallax reproduction program and device capable of highly accurately reproducing a motion parallax and its three-dimensional model generation program and device.SOLUTION: A distance image sensor 1 acquires the camera image and depth map of an object M with a viewpoint from a fixed position P. A polygon model generation part 3 generates a three-dimensional polygon model of the object M on the basis of the camera image and depth map. A polygon texture model generation part 4 acquires the texture of each polygon position of the polygon model from the camera image, and attaches this to each corresponding polygon of the polygon model to generate a polygon texture model. A rendering part 5 reproduces a motion parallax by rendering a two-dimensional image obtained by photographing the object M from a virtual viewpoint Pv on the basis of the image of the polygon texture model.

Description

  The present invention relates to a motion parallax reproduction program and apparatus and a three-dimensional model generation program and apparatus thereof, and in particular, a motion parallax reproduction program and apparatus for reproducing motion parallax by utilizing depth values at the same viewpoint in addition to camera images. The present invention also relates to a three-dimensional model generation program and apparatus three-dimensional display method and apparatus.

  Depth estimation based on matching of corresponding points between cameras and depth based on the premise of a dense camera arrangement, where the camera parallax is about the width of a human eye as a method to perform motion parallax for multi-view stereoscopic displays Non-Patent Document 1 discloses a method for generating an interpolated image between cameras by 3D warping using information.

  In addition, the visual volume intersection method that can construct a three-dimensional model of an object, and a simple three-dimensional model of an object from at least one camera, targeting a camera arrangement that is installed relatively widely compared to the above camera array. A possible billboard model and the like are disclosed in Patent Document 1.

Japanese Patent Application No. 2011-050081

W.R.Mark et al. "Post-Rendering 3D Warping", in Proc of Symposium on Interactive 3D Graphics, pp.7-16, 1997.

  In the proposed method of Non-Patent Document 1, depth estimation for each pixel is performed based on matching of corresponding points between cameras for a camera array arranged at an interval approximately equal to a human eye width, and between cameras by parallax compensation. A method for generating the interpolated image has been proposed. However, the problem that holes are generated in the occlusion area is unavoidable, and in particular, there is a technical problem that the combined image quality is greatly degraded when the camera interval is increased.

  In the proposed method of Patent Document 1, it is shown that a simple model of an object can be generated from at least one camera, and is effective for reproducing a virtual viewpoint in a large space such as soccer. However, since the three-dimensional shape of an object is approximated by a rectangular polygon called a “billboard”, fine irregularities on the surface of the object cannot be reproduced, which is not suitable for viewing where motion parallax reproduction is required. There was a technical problem.

  An object of the present invention is to solve the above technical problem, reproduce a motion parallax that can reproduce fine irregularities on the surface of an object, and reproduce motion parallax with little image quality degradation due to self-occlusion, and a three-dimensional model generation program thereof And providing an apparatus.

  In order to achieve the above object, the present invention is characterized by the following means in the motion parallax reproduction device that reproduces the motion parallax of a three-dimensional model simulating a subject and the three-dimensional model generation device thereof There is.

  (1) The three-dimensional model generation device of the present invention includes a means for acquiring a camera image obtained by photographing a subject from a viewpoint from a fixed position, a means for obtaining a depth map obtained by measuring the subject from a fixed position, a camera image and a depth. Means for generating a polygon model of the subject based on the map, and means for generating a three-dimensional polygon texture model by projecting a texture from each local region corresponding to the camera image onto each polygon of the polygon model.

  (2) The motion parallax reproduction device of the present invention includes a means for acquiring a camera image obtained by photographing a subject from a viewpoint from a fixed position, a means for obtaining a depth map obtained by measuring the subject from a fixed position, a camera image, and a depth map. Generating a polygon model of a subject based on the image, means for projecting a texture from each corresponding local region of the camera image to each polygon of the polygon model, generating a three-dimensional polygon texture model, and polygon texture at a fixed viewpoint Means for rendering a two-dimensional image of a virtual viewpoint based on the model.

  According to the present invention, the following effects are achieved.

  (1) Since the 3D model of the subject is constructed as a polygon texture model in which the vertex of each polygon is given in 3D coordinates based on the camera image and depth map, each local area (polygon) of the 3D model However, it is possible to define different specific directions (normals) depending on the position.

  (2) As mentioned above, a unique orientation can be defined for each local area of the 3D model according to its position, so that each billboard model is defined with the same orientation (front). Compared to the case, the motion parallax can be accurately reproduced.

  (3) As described above, since motion parallax can be accurately reproduced, deterioration of image quality in the self-occlusion area can be suppressed when a rendering image is obtained by combining a plurality of three-dimensional model images.

1 is a functional block diagram of a motion parallax reproduction device and its three-dimensional model generation device according to a first embodiment of the present invention. It is the figure which showed an example of the camera image and depth map of a to-be-photographed object. It is the figure which expressed the polygon decision method typically. It is the figure which expressed typically the generation method of a polygon model. It is the figure which expressed typically the production | generation method of a polygon texture model. It is a figure for demonstrating the effect of 1st Embodiment. It is a functional block diagram of the motion parallax reproduction device and its three-dimensional model generation device according to the second embodiment of the present invention. It is the figure which showed the camera image and depth map in 2nd Embodiment. It is the figure which showed the example of rendering in 2nd Embodiment.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. FIG. 1 is a functional block diagram showing the configuration of the main part of the motion parallax reproduction device and its three-dimensional model generation device according to the first embodiment of the present invention. Here, the configuration unnecessary for the description of the present invention is shown. The illustration is omitted.

  The distance image sensor 1 includes an image sensor such as a CCD or CMOS, and captures the subject M from a viewpoint from a fixed position P to acquire a color (RGB) camera image and the fixed position P or the vicinity thereof. A depth map creation unit 1b that creates a depth map of the subject M from the viewpoint from the position P ′ is provided.

  FIG. 2 is a diagram illustrating an example of a camera image [FIG. (A)] and a depth map [(b)] obtained and created from the subject M from the viewpoint from the fixed position P. The creation unit 1b irradiates the subject M with LED light or laser light from the light source 2, detects the reflected light with the image sensor, and creates a depth map by measuring the arrival time for each pixel.

  If the image sensor for the camera image is different from the image sensor for depth measurement and there is a difference between the positions, the depth value of each pixel on the depth map is projected onto the camera image by the central projection matrix. It is desirable to specify the corresponding pixel and adjust the displacement of each position.

  In the polygon model generation unit 3, the polygon determination unit 3 a performs the three-dimensional of each pixel for each combination of three pixels selected from each 2 × 2 = 4 pixel block of the camera image, as shown in FIG. 3. The coordinates are calculated from the two-dimensional coordinates and the depth value on the camera image, and a triangular polygon having the three-dimensional coordinates as vertex coordinates is generated. As shown in FIG. 4, the polygon model generation unit 3 generates a polygon model [(b)] by connecting all the polygons generated from the depth map [(a)]. At this time, a pattern in which the distance between the vertex coordinates is equal to or greater than the threshold is regarded as noise and eliminated.

  As shown in FIG. 5, the polygon texture model generation unit 4 applies the texture acquired by texture mapping from the corresponding position of the camera image [FIG. (A)] to each polygon position of the polygon model [FIG. (B)]. By pasting, a three-dimensional polygon texture model [FIG. (C)] is generated.

  In this embodiment, the polygon texture model generation unit 4 projects each vertex coordinate of each polygon onto a camera image using a central projection matrix, and converts each pixel in the projected camera image to each polygon. A reverse projection unit 4b for projecting is provided, and the texture of the camera image is pasted on each polygon by this reverse projection.

  The rendering unit 5 generates a rendering image obtained by capturing an image of the subject M from the virtual viewpoint Pv by applying a general CG rendering technique to the polygon texture model.

  According to the present embodiment, the 3D model of the subject M is constructed as a polygon texture model in which the vertices of each polygon are given in 3D coordinates based on the camera image and the depth map. Different local orientations (normal lines) can be defined for the local region (polygon) PG depending on its position.

  As a result, as schematically represented in FIG. 6, the angle difference θ between the normal line and the line of sight of the local region PG at the opposite position at the virtual viewpoint Pv is the same for each local region as in Patent Document 1. Compared to the case of the billboard model [Fig. (A)] defined by the orientation, the present embodiment [Fig. (B)] can reduce the size. As a result, the texture of the subject M can be expressed more accurately in each local region, so that motion parallax can be accurately reproduced.

  FIG. 7 is a functional block diagram showing a configuration of a main part of the motion parallax reproduction device and its three-dimensional model generation device according to the second embodiment of the present invention, where the same reference numerals as those described above represent the same or equivalent parts. Therefore, the explanation is omitted.

  In the present embodiment, the distance image sensor 1 is provided in at least two places P1 and P2, and the virtual viewpoint Pv is based on a plurality of polygon texture models created based on a plurality of camera images and depth map pairs with different viewpoints. A feature is that a two-dimensional image is rendered. Here, a case where two distance image sensors 1A and 1B are provided will be described as an example.

  FIG. 8 shows a camera image I1 [FIG. (A)] acquired by one distance image sensor 1A and its depth map D1 [FIG. (B)] and a camera image I2 acquired by the other distance image sensor 1B. FIG. 6C is a diagram showing an example of [FIG. (C)] and its depth map D2 [FIG. (D)].

  The rendering unit 6 determines between the fixed positions P1 and P2 based on the pair [I1, D1] and [I2, D2] of the camera images I1 and I2 and the depth maps D1 and D2 acquired at the fixed positions P1 and P2. A two-dimensional image with the virtual position Pv as the viewpoint is rendered by integrating the images of the polygon texture models at a ratio corresponding to the distance between the fixed viewpoints P1 and P2 and the virtual viewpoint Pv.

  FIG. 9 is a diagram showing an example of rendering by the rendering unit 6. The image of the polygon texture model generated from the camera image I1 acquired by one distance image sensor 1A and its depth map D1 [FIG. )] And a polygonal texture model image [b (b)] generated from the camera image I2 obtained by the other distance image sensor 1B and the depth map D2 between the fixed position P1 and the virtual position Pv. Rendered image by alpha blending according to distance difference (P1-Pv) and ratio of distance difference (P2-Pv) between fixed position P2 and virtual position Pv (P2-Pv) / (P2-Pv) (c)] is generated.

  According to the present embodiment, since the angle difference θ between the normal line and the line of sight of the local region PG at the position facing in the virtual viewpoint Pv can be reduced, when a rendering image is obtained by combining a plurality of three-dimensional models, Image quality degradation in the self-occlusion area can be suppressed.

  The object of the present invention can also be achieved by recording a program code for realizing the functions of the above-described embodiments on a recording medium, and reading and executing the program by a computer. In this case, the program code itself read from the storage medium realizes the functions of the above-described embodiments, and the program code itself and the storage medium storing the program code constitute the present invention.

DESCRIPTION OF SYMBOLS 1 ... Distance image sensor, 1a ... Imaging part, 1b ... Depth map creation part, 2 ... Light source, 3 ... Polygon model production | generation part, 3a ... Polygon determination part, 4 ... Polygon texture model production | generation part, 5, 6 ... Rendering part

Claims (10)

In the 3D model generation device of the motion parallax reproduction device that reproduces the motion parallax of the 3D model imitating the subject,
Means for acquiring a camera image obtained by photographing a subject from a fixed point of view;
Means for obtaining a depth map obtained by measuring the subject from the fixed position;
Means for generating a polygon model of the subject based on the camera image and the depth map;
3D model generation apparatus for motion parallax reproduction apparatus, comprising: means for projecting a texture from each local region corresponding to a camera image to each polygon of said polygon model to generate a 3D polygon texture model . The means for acquiring the depth map acquires a depth value for each pixel of the camera image,
The means for generating the polygon model obtains a three-dimensional coordinate based on a two-dimensional coordinate and a depth value of each pixel for each combination of three pixels selected from each 2 × 2 pixel block of the camera image, The three-dimensional model generation device for a motion parallax reproduction device according to claim 1, wherein the three-dimensional coordinates are connected to generate a triangular polygon. The means for generating the polygon texture model includes:
Means for projecting each vertex coordinate of each polygon to a camera image with a central projection matrix;
3. The three-dimensional model generation device for a motion parallax reproduction device according to claim 1, further comprising means for back-projecting each pixel in the projected camera image onto each polygon. 4.   4. The three-dimensional motion parallax reproducing device according to claim 1, wherein a corresponding pixel is specified by projecting a depth value of each pixel on the depth map onto a camera image using a central projection matrix. 5. Model generator. In the 3D model generation program of the motion parallax reproduction device that reproduces the motion parallax of the 3D model imitating the subject,
A procedure for acquiring a camera image of a subject taken from a fixed point of view;
Obtaining a depth map obtained by measuring the subject from the fixed position;
Generating a polygon model of the subject based on the camera image and the depth map;
Projecting a texture from each corresponding local region of the camera image to each polygon of the polygon model to generate a three-dimensional polygon texture model;
3D model generation program of motion parallax reproduction device for causing computer to execute. In a motion parallax reproduction device that reproduces the motion parallax of a three-dimensional model simulating a subject,
Means for acquiring a camera image obtained by photographing a subject from a fixed point of view;
Means for obtaining a depth map obtained by measuring the subject from the fixed position;
Means for generating a polygon model of the subject based on the camera image and the depth map;
Means for projecting a texture from each corresponding local region of the camera image to each polygon of the polygon model to generate a three-dimensional polygon texture model;
A motion parallax reproduction device comprising: means for rendering a two-dimensional image of a virtual viewpoint based on a polygon texture model at the fixed viewpoint. In a motion parallax reproduction device that reproduces the motion parallax of a three-dimensional model simulating a subject,
Means for acquiring a camera image obtained by photographing a subject from a plurality of different fixed viewpoints;
Means for acquiring a depth map obtained by measuring a subject from each of the fixed positions;
Means for generating a polygon model of a subject based on a camera image and a depth map acquired for each of the fixed positions;
Means for projecting a texture from each corresponding local region of each camera image to each polygon of each polygon model to generate a three-dimensional polygon texture model;
A motion parallax reproduction device comprising: means for rendering a two-dimensional image of a virtual viewpoint based on an image of a polygon texture model at each fixed viewpoint.   8. The motion parallax reproduction device according to claim 7, wherein the rendering unit integrates the polygon texture model images at each fixed viewpoint at a ratio corresponding to the distance between each fixed viewpoint and the virtual viewpoint. In a motion parallax reproduction program that reproduces the motion parallax of a three-dimensional model simulating a subject,
A procedure for acquiring a camera image of a subject taken from a fixed point of view;
Obtaining a depth map obtained by measuring the subject from the fixed position;
Generating a polygon model of the subject based on the camera image and the depth map;
Projecting a texture from each corresponding local region of the camera image to each polygon of the polygon model to generate a three-dimensional polygon texture model;
Rendering a virtual viewpoint two-dimensional image based on the fixed viewpoint polygon texture model;
Motion parallax reproduction program to make computer execute. In a motion parallax reproduction program that reproduces the motion parallax of a three-dimensional model simulating a subject,
A procedure for acquiring camera images of a subject taken from a plurality of different fixed viewpoints;
Obtaining a depth map obtained by measuring the subject from each of the fixed positions;
A procedure for generating a polygon model of a subject based on a camera image and a depth map acquired for each of the fixed positions;
A procedure of projecting a texture from each corresponding local region of each camera image to each polygon of each polygon model to generate a three-dimensional polygon texture model;
Rendering a virtual viewpoint two-dimensional image based on the polygonal texture model image of each fixed viewpoint;
Motion parallax reproduction program to make computer execute.