JP2018163467A - Method, device and program for generating and displaying free viewpoint image - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a method, device and program for generating and displaying a free viewpoint image, which can reproduce a 3D model of an object in a short period of time without damaging subjective quality even about an occlusion area.SOLUTION: When occlusion occurs between two objects Oj1, Oj2, the pixel value of a corresponding pixel of a camera image from which a rectangular projection image is extracted is acquired as texture about pixels other than an occlusion area. Since texture of a leg part of the object Oj1 cannot be acquired from a camera image of a viewpoint 2 being its extraction source camera image, a projection source voxel Bt is specified by ray search about pixels of the occlusion area (gray). Then, when the projection source voxel Bt is projected to other camera images to determine whether observation of the voxel is possible on the basis of a positional relation of three-dimensional shape models of the respective objects Oj1, Oj2, the pixel value of the corresponding pixel is acquired from an observable camera image.SELECTED DRAWING: Figure 4

Description

  The present invention relates to a free viewpoint image generation and display method, apparatus, and program, and more particularly to a free viewpoint image generation and display method suitable for shooting conditions in which a plurality of cameras cannot be closely arranged in a relatively large space such as a sporting event. , Apparatus and program.

  Free viewpoint video generation methods are roughly classified into a model-based type and an image-based type. A typical model-based method is disclosed in Non-Patent Document 1. In Non-Patent Document 1, the 3D shape of a subject is restored as a 3D model, the surface of the 3D model is divided into fine polygons, and then the texture of each polygon is acquired from a plurality of shooting camera images and appropriate according to the viewpoint position. By blending at a proper ratio, the video from the viewpoint where there is no shooting camera is synthesized.

  Although this method can theoretically reproduce the appearance from an arbitrary viewpoint smoothly, the quality of the synthesized image is 3D due to the characteristic that the final synthesized image is represented as a combination of polygons. Since it greatly depends on the accuracy of the model and camera parameters, there is a limit to the quality when the cameras cannot be arranged closely, such as in a sports scene.

  On the other hand, a typical image-based technique is disclosed in Non-Patent Document 2. Non-Patent Document 2 proposes a billboard method in which an existing area of a subject is extracted from each shooting camera video and expressed as a single rectangular polygon.

  In the billboard method, the 3D shape of the subject is not restored, but the subject is calculated by calculating the coordinates where the billboard is installed in the target 3D space and mapping the texture of the subject area obtained from the camera image. It is possible to reproduce the positional relationship between each other and the subject and the background at an arbitrary viewpoint.

  The billboard method cannot reproduce the smooth changes in appearance as in the 3D model method, but it is used as it is without processing the texture of the shooting camera image, so it has higher definition than the 3D model method. You can see how it looks.

JP-A-2015-191538

T. Kanade, P. W. Rander, and P. J. Narayanan, "Virtualized Reality: Constructing Virtual Worlds from Real Scenes," IEEE Multimedia, vol. 4, no. 1, pp. 34-47, 1997. Y. Ohta, I. Kitahara, Y. Kameda, H. Ishikawa, and T. Koyama, "Live 3D Video in Soccer Stadium," International Journal of Computer Vision (IJCV), vol. 75, no. 1, pp. 173 -187, 2007. Hiroshi Sankoh and Sei Naito, "Free-viewpoint Video Rendering in Large Outdoor Space such as Soccer Stadium based on Object Extraction and Tracking Technology," The Journal of The Institute of Image Information and Television Engineers (ITE), Vol. 68, No. 3, pp. J125-J134, 2014.

  The biggest problem common to the model base type and the image base type is occlusion caused by overlapping of subjects in the captured camera video. In particular, in the billboard method, the texture acquired from the subject area in each camera image is used as it is without being processed. Therefore, when an area that cannot be observed due to an overlap with another subject occurs, the appearance of the area is reproduced. It is impossible.

  Even in the occlusion area, it is difficult to properly separate the boundary of overlap even if the object is on the near side of the camera, so it can be said that the extraction of the appropriate object area is a difficult task. .

  In order to deal with such technical problems, a technique for detecting occlusion and interpolating between frames by assigning an individual ID to each subject and performing tracking so that the ID is held in the time direction is described in Non-Patent Document 3. And Patent Document 1.

  These prior arts propose a method of interpolating between the texture of each subject in the occlusion area and the installation coordinates of the billboard between frames by using information from a plurality of cameras. However, when the occlusion duration (the number of frames) is long, there is a problem of impairing subjective quality due to the limited interpolation performance.

  Further, when an error is included in the tracking process, the texture and position are corrected between different subjects, and there is a problem that the subjective quality of the synthesized video is significantly impaired. For the purpose of improving robustness against tracking errors, Patent Document 1 proposes a technique that enables confirmation / correction of a tracking ID by visual observation. However, there is a problem that much time is required.

  The object of the present invention is to solve the above technical problem and individually extract the existence area of each object in each camera in the composition process in the billboard system, thereby providing a unique high-definition appearance of the billboard system. An object of the present invention is to provide a free viewpoint image generation and display method, apparatus, and program capable of reproducing a 3D model of an object in a short time without sacrificing subjective quality, particularly in an occlusion area.

  In order to achieve the above object, the present invention is characterized in that a free viewpoint image generation and display method, apparatus, and program have the following configurations.

  (1) An apparatus for generating and displaying a free viewpoint image according to the present invention projects a three-dimensional shape model generated based on a plurality of camera images obtained by photographing an object from different viewpoints to each viewpoint, and a projected image of each object. Means for extracting a rectangular projection image including a portion for each viewpoint, means for obtaining a texture of the rectangular projection image at each viewpoint for each object, means for generating a billboard based on each rectangular projection image and its texture, Means for selecting a camera that displays a billboard based on information identifying a free viewpoint and displaying all billboards generated for that camera.

  (2) The computer according to the present invention generates and displays a free viewpoint image by projecting a three-dimensional shape model generated based on a plurality of camera images obtained by photographing an object from different viewpoints to each viewpoint. A rectangular projection image including a projected image portion is extracted for each viewpoint, and the texture of the rectangular projection image at each viewpoint is acquired from each camera for each object, and a billboard is generated based on each rectangular projection image and its texture. The camera that displays the billboard is selected based on the information for specifying the free viewpoint, and all the billboards generated for the camera are displayed.

  (3) A program for generating and displaying a free viewpoint image according to the present invention projects a three-dimensional shape model generated based on a plurality of camera images obtained by photographing an object from different viewpoints to each viewpoint, and projects each object. A procedure for extracting a rectangular projection image including a part for each viewpoint, a procedure for obtaining a texture of the rectangular projection image at each viewpoint from a corresponding camera image for each object, and a billboard based on each rectangular projection image and its texture And a procedure for selecting a camera that displays a billboard based on information for specifying a free viewpoint and displaying all billboards generated for the camera.

  (1) Assuming that the final display and rendering of the free viewpoint video will be carried out using the billboard method, the 3D shape model of the object is restored during the synthesis process, and the projected image of each model on each camera is used. By doing so, it becomes possible to detect the occlusion area in each camera and extract the existence area and texture for each object.

  That is, when creating a billboard, the texture of the corresponding pixel is acquired from the source image of the projection image for the non-occlusion area, while the texture is acquired from the corresponding pixel of the camera image other than the source camera for the occlusion area. can do.

  Therefore, 3D models of objects can be reproduced in a short time without impairing subjective quality, and high-quality free viewpoint images are automatically generated without requiring manual work even in scenes where frequent occlusion occurs. Can be generated.

  (2) Although there is a limit to the restoration performance of the three-dimensional shape of the subject, especially when targeting a large space, in the present invention, pay attention to the projected image on each camera of Visual Hull restored by the visual volume intersection method By doing so, it is possible to individually extract a subject existing area in each camera.

1 is a functional block diagram showing a configuration of an embodiment of a system to which a method and apparatus for generating and displaying a free viewpoint image according to the present invention are applied. It is the figure which showed the method of estimating the three-dimensional shape of an object. It is the figure which showed the method of acquiring the rectangular projection image containing the projection image for every object for every viewpoint. It is the figure which showed the acquisition method of the texture stuck on the projection image part of a rectangular projection image. It is the figure which showed the production | generation method of a billboard. It is the figure which showed the display method of a billboard.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. FIG. 1 is a functional block diagram showing a configuration of an embodiment of a free viewpoint image generation and display method, apparatus and program according to the present invention.

  Such a system may be configured by mounting an application (program) for realizing each function on a general-purpose computer or server, or a dedicated machine or a single unit in which a part of the application is implemented in hardware or ROM. You may comprise as a function.

  The multi-viewpoint image input unit 10 acquires camera images in units of frames from a plurality of cameras Ca (Ca1, Ca2, Ca3...) That capture the object Obj from different viewpoints. In the following description, each camera Ca or its camera image may be expressed as “viewpoint 1”, “viewpoint 2”. The three-dimensional shape model generation unit 20 generates a three-dimensional shape model for each object Obj on a frame basis based on the camera image acquired from each camera Ca.

  In the three-dimensional shape model generation unit 20, the camera parameter estimation unit 21 estimates the center projection matrix (camera parameter) of each camera Ca. The mask image extraction unit 22 extracts a mask image indicating the existence area of each object for each frame image. As shown in FIG. 2, the three-dimensional shape estimation unit 23 uses a visual volume intersection method based on the constraint that an object is included in the visual volume obtained by projecting the mask image extracted at each viewpoint into the real space. A common part of the visual volume corresponding to a plurality of mask images is estimated as a three-dimensional shape (three-dimensional voxel data) of the object, and a Visual Hull indicating an object existing area in the three-dimensional voxel space is restored.

  The three-dimensional shape model restoration unit 24 classifies the Visual Hull based on the geometric connectivity of the voxel space, and the size (number of voxels) and height (y coordinate) of the classified Visual Hull. Only Visual Hull satisfying the predetermined condition is restored as the three-dimensional shape model of each object.

  As shown in FIG. 3, the rectangular projection image extraction unit 30 projects the restored three-dimensional model of the object onto the camera image of each viewpoint, and includes a rectangular image including the projection image of each object (hereinafter, rectangular projection image). For each viewpoint.

  In the rectangular projection image extraction unit 30, the occlusion region specifying unit 31 performs a ray search for each pixel in the projection image portion of each rectangular projection image, thereby specifying an occlusion region (gray) for each object at each viewpoint.

  The texture acquisition unit 40 acquires the texture of the object from each viewpoint camera image and pastes it on the projection image portion of each rectangular projection image. FIG. 4 is a diagram for explaining the texture acquisition method by the texture acquisition unit 40 by paying attention to the rectangular projection image extracted from the camera image of the viewpoint 2.

  In the texture acquisition unit 40, the first texture acquisition unit 41 extracts a camera image (extraction source camera image) obtained by extracting the rectangular projection image for pixels other than the occlusion region (non-occlusion region) in the projection image portion of the rectangular projection image. ) Is acquired as a texture. The second texture acquisition unit 42 acquires, as a texture, the pixel value of the corresponding pixel of the camera image other than the extraction source camera image for the pixel of the occlusion area in the projection image portion of the rectangular projection image.

  In the example of FIG. 4, occlusion occurs between the two objects Oj1 and Oj2, and the texture of the leg of the object Oj1 cannot be acquired from the camera image of the viewpoint 2 that is the extraction source camera image.

  The second texture acquisition unit 42 specifies the projection source voxel Bt by ray search for the pixels in the occlusion area (gray) of the rectangular projection image. Then, by projecting the projection source voxel Bt onto another camera image and determining whether or not the voxel can be observed based on the positional relationship of the three-dimensional shape model of the objects Oj1 and Oj2, the corresponding pixels are obtained from the observable camera image. The pixel value of is acquired.

  When there are a plurality of cameras that can be observed, the camera nearest to the extraction source camera is specified, and the pixel value of the corresponding pixel is preferentially acquired from the nearest camera. In the illustrated example, the texture of the leg of the object Oj1 that cannot be acquired from the camera image of the viewpoint 2 can be acquired from either the viewpoint 1 or the viewpoint N, but is acquired from the camera image of the nearest viewpoint 1.

  The billboard generation unit 50 generates billboards for all objects Obj for each viewpoint. The texture image extraction unit 51 extracts the rectangular projection image including the texture for each viewpoint object.

  As shown in FIG. 5, the billboard size determination unit 52 identifies the projection source voxel of the pixel existing on the bottom side of the rectangular projection image, the distance between the projection source voxel and the target camera, and the rectangular projection image The size of each billboard is calculated based on the aspect ratio. The billboard installation unit 53 installs the generated billboard in the calculated size on the three-dimensional coordinates of each projection source voxel.

  The billboard display unit 60 estimates a camera nearest to the virtual viewpoint based on the viewpoint operation information, and selects and displays all billboards generated for the camera.

  In the billboard display unit 60, the virtual viewpoint calculation unit 61 performs a virtual viewpoint based on viewpoint operation information such as change of the gazing point acquired from each camera, movement to the photographing camera viewpoint, forward / backward movement, left / right rotation, and vertical rotation. Calculate the position and orientation of.

  As shown in FIG. 6, the billboard selection unit 62 preliminarily sets a boundary for camera selection for each adjacent camera pair (A, B), and the calculated virtual viewpoint is assigned to one camera A. If it is directed to the designated area, all billboards from which the image of the camera A is extracted are displayed. If the area assigned to the other camera B is pointed, all billboards with the camera B as an extraction source are displayed.

  DESCRIPTION OF SYMBOLS 10 ... Multi viewpoint image input part, 20 ... Three-dimensional shape model production | generation part, 21 ... Camera parameter estimation part, 22 ... Mask image extraction part, 23 ... Three-dimensional shape estimation part, 24 ... Three-dimensional shape model restoration part, 30 ... Rectangular projection image extraction unit 31... Occlusion region specifying unit 40. Texture acquisition unit 41. First texture acquisition unit 42. Second texture acquisition unit 50. ... billboard size determination unit, 53 ... billboard installation unit, 60 ... billboard display unit, 61 ... virtual viewpoint calculation unit, 62 ... billboard selection unit

Claims (11)

In an apparatus for generating and displaying a free viewpoint image from a multi-viewpoint image,
Means for individually generating a three-dimensional shape model of each object based on a plurality of camera images taken from different viewpoints;
Means for projecting each three-dimensional shape model onto each viewpoint and extracting a rectangular projection image including a projection image portion of each object for each viewpoint;
Means for acquiring the texture of the rectangular projection image at each viewpoint for each object from the corresponding camera image;
Means for generating a billboard based on each rectangular projection image and its texture;
Means for selecting a camera to display a billboard based on information identifying a free viewpoint;
A free-viewpoint image generation / display apparatus comprising: means for displaying all billboards generated with respect to the selected camera. The means for generating a three-dimensional shape model of the object includes:
Means for estimating the camera parameters of each camera;
Means for extracting a mask image of an object in each camera image;
Means for restoring Visual Hull of each object in a three-dimensional voxel space by a view volume intersection method based on the camera parameter and the mask image,
The free-viewpoint image generation and display device according to claim 1, wherein the Visual Hull is adopted as a three-dimensional shape model. Means for classifying the restored Visual Hull based on geometric connectivity in the voxel space;
3. The free viewpoint image generation and display device according to claim 2, wherein a class satisfying a predetermined condition among the classified Visual Hulls is adopted as a three-dimensional shape model.   The means for extracting the rectangular projection image for each viewpoint includes means for specifying an occlusion area for each object at each viewpoint by performing a ray search for each pixel of the projection image portion of each rectangular projection image. The free viewpoint image generation and display device according to any one of claims 1 to 3. The means for obtaining the texture comprises:
First texture acquisition means for acquiring a pixel value of a corresponding pixel of an extraction source camera image for a pixel other than an occlusion area in a projection image portion of the rectangular projection image;
2. A second texture acquisition unit configured to acquire a pixel value of a corresponding pixel of a camera image other than an extraction source camera image for a pixel of an occlusion area in the projection image portion of the rectangular projection image. Item 5. The free viewpoint image generation and display device according to any one of Items 1 to 4. The second texture acquisition means
A means for specifying a projection source voxel on a three-dimensional shape model for pixels in an occlusion area;
Means for projecting the projection source voxel onto a camera image different from the extraction source camera image and determining whether or not the voxel can be observed;
6. The apparatus for generating and displaying a free viewpoint image according to claim 5, wherein a pixel value of a corresponding pixel is acquired from an observable camera image.   When there are a plurality of observable cameras, a camera nearest to an extraction source camera is specified from the plurality of cameras, and a pixel value of a corresponding pixel of an image of the nearest camera is acquired. 7. The free viewpoint image generation and display device according to 6. The means for generating the billboard includes:
Means for extracting a rectangular projection image including a texture for each object;
Means for calculating a distance between a camera and a projection source voxel of a pixel existing on a bottom side of the rectangular projection image;
Means for determining the size of each billboard based on the distance and the aspect ratio of the rectangular projection image;
8. The apparatus for generating and displaying a free viewpoint image according to claim 1, further comprising means for setting the billboard of the determined size at a three-dimensional coordinate of the projection source voxel. The means for displaying the billboard is:
Means for calculating a virtual viewpoint based on input information related to a viewpoint operation;
9. The apparatus for generating and displaying a free viewpoint image according to claim 1, further comprising means for displaying all billboards generated with respect to the camera nearest to the virtual viewpoint. In a method in which a computer generates and displays a free viewpoint image from a multi-viewpoint image,
A 3D shape model of each object is generated individually based on multiple camera images taken from different viewpoints.
Projecting each 3D shape model to each viewpoint and extracting a rectangular projection image including the projected image portion of each object for each viewpoint,
For each object, obtain the texture of the rectangular projection image at each viewpoint from the corresponding camera image,
A billboard is generated based on each rectangular projection image and its texture,
Select the camera that displays the billboard based on the information that identifies the free viewpoint,
A method for generating and displaying a free viewpoint image, comprising displaying all billboards generated for the selected camera. In a program that generates and displays a free viewpoint image from a multi-viewpoint image,
A procedure for individually generating a three-dimensional shape model of each object based on a plurality of camera images taken from different viewpoints;
A procedure for projecting each three-dimensional shape model onto each viewpoint and extracting a rectangular projection image including a projection image portion of each object for each viewpoint;
A procedure for acquiring the texture of the rectangular projection image at each viewpoint from the corresponding camera image for each object,
A procedure for generating a billboard based on each rectangular projection image and its texture;
A procedure for selecting a camera to display a billboard based on information identifying a free viewpoint;
A program for generating and displaying a free viewpoint image in which a procedure for displaying all billboards generated for the selected camera is described so as to be executable by a computer.