KR101336955B1 - Method and system for generating multi-view image - Google Patents

Abstract

A multi-view image generating method using a multi-view image photographing camera according to the present invention comprises the steps of: generating a reference image, a depth image and a background image for a photographing object; generating a first virtual view image based on the depth image and the background image; generating a second virtual view image based on the reference image and the depth image; and generating a final virtual view image by overlapping the first virtual view image and the second virtual view image. [Reference numerals] (110) Camera unit;(120) First virtual view image generating unit;(130) Second virtual view image generating unit;(140) Image overlapping unit

Description

METHOD AND SYSTEM FOR GENERATING MULTI-VIEW IMAGE}

The present invention relates to a method and a system for generating a multiview image.

While interest in three-dimensional displays has increased greatly, the user wants a display that can feel a more realistic three-dimensional three-dimensional feeling than a sense of depth in the front. To this end, it is necessary to make the user feel a realistic three-dimensional stereoscopic feeling not only from the front but also from various viewpoints.

The simplest way to ensure high quality multiview images is to acquire more images using multiple cameras. However, this method has a problem of increasing data amount as well as a cost problem. Therefore, it should be assumed that a multi-view image is synthesized from a limited view image captured by a few cameras.

On the other hand, a multi-view image is generated by mapping a texture image at a reference viewpoint to a three-dimensional space using a depth image and then projecting to a virtual viewpoint. In this case, a problem occurs in which an excluded region is not visible in the virtual view image. The solution is to preprocess the depth image and to fill the occlusion area with the surrounding texture.

First, a method of preprocessing a depth image is to minimize a hole generated in a virtual view by smoothing the depth image through filtering. However, this method has a problem that a visually distorted result may be produced by damaging the foreground object. In particular, application is difficult when the hole area is large.

Next, the method of filling the occluded area with the surrounding texture is performed by dividing the hole area around the object and the background and filling the hole area. However, this method has a problem in that the image quality is significantly reduced when there is a background of a predetermined structure or an edge in the vertical direction.

In order to solve the above problems, various techniques such as interpolation method, in-patterning method, texture compositing method, depth image processing method and texture using method have been proposed, but there are still discontinuities in hole area and boundary area. There is a limit to improving image quality deterioration.

In this regard, Korean Patent Laid-Open Publication No. 10-2009-0062440 (name of the invention: a method and apparatus for multi-view matching using object or background separation) estimates each depth through separation of an object or background, and separates an object or background. Disclosed are a method and an apparatus for multiview matching using object or background separation using depth information in conjunction with color information.

In addition, Korean Patent Publication No. 10-2011-0071528 (name of the invention: a stereo image, a multi-view image, and a depth image acquisition camera device and its control method) adds a depth information generation function and an intermediate image generation function to a stereo camera to add stereo. A configuration for obtaining an image and a depth image is disclosed.

SUMMARY OF THE INVENTION The present invention solves the problems of the prior art described above, and some embodiments of the present invention simultaneously acquire a reference image and a background image, and convert each of them together with a depth image to convert the first virtual view image and the second virtual view. An object of the present invention is to provide a multi-view image generating method in which an image is generated and overlapped so that no occlusion area is generated.

As a technical means for achieving the above-described technical problem, a multi-view image generating method using a multi-view image shooting camera according to the first aspect of the present invention comprises the steps of generating a reference image, a depth image and a background image for the shooting target, Generating a first virtual view image based on the depth image and the background image, generating a second virtual view image based on the reference image and the depth image, and the first virtual view image and the second image And generating a final virtual view image by superimposing the virtual view images.

In addition, the multi-view image generating system according to the second aspect of the present invention is a camera unit including a reference image camera for taking a reference image, a background image camera for taking a background image and a depth camera for taking a depth image, the A first virtual viewpoint image generator for generating a first virtual viewpoint image based on a background image and the depth image, and a second virtual viewpoint image generator for generating a second virtual viewpoint image based on the reference image and the depth image. And an image overlapping unit configured to overlap the first virtual view image and the second virtual view image to generate a final virtual view image.

The multi-view image generating system according to the third aspect of the present invention includes a camera unit including a reference image camera photographing a reference image, a background image camera photographing a background image, and a depth camera photographing a depth image. A first virtual viewpoint image generator for generating a first virtual viewpoint image based on a background image and the depth image, and a second virtual viewpoint image generator for generating a second virtual viewpoint image based on the reference image and the depth image. And an image overlapping unit configured to overlap the first virtual view image and the second virtual view image to generate a final virtual view image.

According to an embodiment of the above-described problem solving means of the present invention, by generating and using the background image taken by removing the foreground separately from the reference image and the depth image at the same time it is possible to prevent the generation of occlusion area when generating the virtual viewpoint image. . Therefore, since the process involved in the occlusion area restoration process is not required, the efficiency is increased, and problems such as the unnaturalness of the image generated in the restoration process and the disconnection of the brightness value may be solved.

In addition, a high quality multi-view image may be generated in real time by configuring a data format input to a 3D TV as a reference image, a depth image and a background image. As a result, since the same process can be applied to the 3D TV terminal, it is highly useful and can automatically generate a virtual viewpoint image having no occlusion area and no interruption for various viewpoints.

1 is a diagram illustrating a multiview image generating system according to an exemplary embodiment of the present invention.
2 is a diagram illustrating a multi-view image generating method according to an embodiment of the present invention.
3 is a diagram illustrating an arrangement of a reference image photographing camera and a background image photographing camera.
4 is a diagram illustrating a reference image, a depth image, and a background image.
FIG. 5 is a diagram illustrating a first virtual view image, a second virtual view image, and a final virtual view image generated by overlapping the virtual view image.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings, which will be readily apparent to those skilled in the art. The present invention may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. In order to clearly illustrate the present invention, parts not related to the description are omitted, and similar parts are denoted by like reference characters throughout the specification.

Throughout the specification, when a part is referred to as being "connected" to another part, it includes not only "directly connected" but also "electrically connected" with another part in between . Also, when an element is referred to as "comprising ", it means that it can include other elements as well, without departing from the other elements unless specifically stated otherwise.

1 is a diagram illustrating a multiview image generating system according to an exemplary embodiment of the present invention.

The multi-view image generating system 100 according to the present invention includes a camera unit 110, a first virtual view image generating unit 120, a second virtual view image generating unit 130, and an image overlapping unit 140. .

The camera unit 110 may include a reference image camera 310 generating the reference image 370, a background image camera 320 generating the background image 360, and a depth camera generating the depth image 380. City). Meanwhile, the depth image 380 may be generated through stereo matching between the reference image photographed at not only the depth camera but also at the multiview point.

Meanwhile, the background image camera 320 may be arranged to photograph only the background area 330 at a different point of time from the reference image camera 310. In addition, the background image camera 320 may be disposed to shoot at the same point in time as the reference image camera 310. In this case, the background image camera 320 is disposed to photograph only the background area 330 in a state where the foreground 350 is removed.

The first virtual viewpoint image generator 120 generates the first virtual viewpoint image 510 based on the background image 360 generated by the background image camera 320 and the depth image 380 generated by the depth camera. do. In addition, the second virtual viewpoint image generator 130 may generate the second virtual viewpoint image 520 based on the reference image 370 generated by the reference image camera 310 and the depth image 380 generated by the depth camera. Create A detailed method of generating the first virtual viewpoint image 510 and the second virtual viewpoint image 520 will be described with reference to FIG. 2.

The image overlapping unit 140 overlaps the first virtual viewpoint image 510 and the second virtual viewpoint image 520.

 By overlapping the first virtual view image 510 and the second virtual view image 520, the brightness value of the first virtual view image 510 occupies the occlusion area 340 generated in the second virtual view image 520. Will be replaced by Therefore, the multi-view image generation system according to the present invention does not require additional work to restore the occlusion area.

2 is a diagram illustrating a method of generating a multiview image according to an exemplary embodiment of the present invention, and FIG. 3 is a diagram illustrating an arrangement of a camera for photographing a reference image and a camera for photographing a background image.

A method of generating a multiview image will be described with reference to FIG. 2. First, a reference image 370, a depth image 380, and a background image 360 are generated (S210). In this case, the reference image 370 is generated through the reference image camera 310, and the background image 360 is generated through the background image camera 320. In addition, the depth image 380 may be generated through stereo matching between the reference images 370 photographed at a multi-view, or may be generated using a depth camera (not shown).

Meanwhile, referring to FIG. 3, the arrangement of the background image camera is as follows. The background image camera 320 may be arranged to photograph only the background area 330 at a different point of time than the reference image camera 310. That is, the background image camera 320 captures only the background area 330 by setting the photographing angle so as not to overlap the foreground 350.

In addition, the background image camera 320 may photograph at the same position as the reference image camera 310, but only the background area 330 in a state in which the foreground 350 is removed.

FIG. 4 is a diagram illustrating a reference image, a depth image, and a background image, and FIG. 5 is a diagram illustrating a first virtual view image, a second virtual view image, and a final virtual view image generated by superimposing the images.

In the background image 360 captured by the background image camera 320, only the background region 330 is photographed in a state in which the foreground 350 is removed. The reference image 370 captured by the reference image camera 310 includes both the foreground 350 and the background area 330. The depth image 380 is photographed through the depth camera, and the depth of the depth image 380 may be expressed through black and white color.

Referring back to FIG. 2, after generating the reference image 370, the depth image 380, and the background image 360, the first virtual view image 510 based on the depth image 380 and the background image 360. To generate (S220).

A process of generating the first virtual view image 510 will be described below. First, a depth image coordinate with respect to a first virtual viewpoint image coordinate is calculated on a reference coordinate system. Next, the background image coordinates are calculated based on the calculated depth image coordinates, and the brightness value of the background image corresponding to the background image coordinates is obtained. Next, the brightness value of the background image is allocated as the brightness value of the first virtual view image.

As described above, when the first virtual view image coordinate is changed and repeated for all the first virtual view image coordinates to be generated, the first virtual view image 510 is changed from the background image 360 and the depth image 380. Can be generated. Since the process corresponds to one pixel of the image, the first virtual view image 510 may be generated only by repeating the above process for all pixels of the image.

Hereinafter, a process for calculating the first virtual viewpoint image coordinates will be described in detail. The position of the virtual camera for capturing the first virtual viewpoint image 510 on a reference coordinate system is (X _p , Y _p , Z _p ), the depth image coordinate is (X, Y, Z), the reference coordinate system 1, the virtual viewpoint image 510, the focal length of the virtual camera for taking said first virtual viewpoint image 510, the rotational movement matrix between the virtual camera R _p, for taking f _p, the scale factor s _p and the When the first virtual view image coordinate is (x _p , y _p ), the depth image coordinates (X, Y, Z) for the first virtual view image coordinate are calculated using the following determinant.

Meanwhile, since the first virtual view image 410 is generated based on the depth image 380 and the background image 360, the background image coordinates (x _b , y _b ) with respect to the depth image coordinates (X, Y, Z) are generated. Should be obtained. Background image coordinates are the position of the camera for capturing the background image 360 on the reference coordinate system (X _b , Y _b , Z _b ), and the rotation between the reference coordinate system and the camera coordinate system for capturing the background image 360. When the moving matrix R _b and the focal length of the camera for capturing the background image 360 are f _b and the scale constant is sb, the background image with respect to the depth image coordinates (X, Y, Z) using the following determinant. Calculate the coordinates (x _b , y _b ).

Next, referring back to FIG. 2, a second virtual view image 520 is generated based on the depth image 380 and the reference image 370 (S230).

 A process of generating the second virtual view image 520 is described below. First, a depth image coordinate of an image coordinate of the second virtual viewpoint image 520 is calculated on a reference coordinate system. Next, the reference image coordinates are calculated based on the calculated depth image coordinates, and the brightness value of the reference image corresponding to the reference image coordinates is calculated. Next, the brightness value of the reference image is allocated as the brightness value of the second virtual viewpoint image.

As described above, when the second virtual view image coordinate is changed and repeated for all coordinate values of the virtual view image to generate the process, the second virtual view image 520 from the reference image 370 and the depth image 380. Can be generated.

Hereinafter, a process for calculating the second virtual viewpoint image coordinates will be described in detail. The position of the virtual camera for capturing the second virtual viewpoint image 520 on the reference coordinate system (X _p , Y _p , Z _p ), the depth image coordinates (X, Y, Z), the reference coordinate system and the second The rotational movement matrix between the virtual cameras for capturing the virtual viewpoint image 520 is R _p , the focal length of the virtual camera for capturing the second virtual viewpoint image 520 is f _p , the scale constant is s _p, and the second 2 When the virtual view image coordinates are (x _p , y _p ), the depth image coordinates (X, Y, Z) for the second virtual view image coordinates are calculated using the following determinant.

Meanwhile, since the second virtual view image 520 is generated based on the depth image 380 and the reference image 370, the reference image coordinates (x _r , y _r ) with respect to the depth image coordinates (X, Y, Z) are generated. Should be obtained. The reference image coordinate is a rotation of the position of the camera for capturing the reference image 370 on the reference coordinate system (X _r , Y _r , Z _r ), and the rotation between the reference coordinate system and the camera coordinate system for capturing the reference image 370. When the moving matrix is R _r , the focal length of the camera for capturing the reference image 370 is f _r, and the scale constant is s _r , the depth matrix coordinates (X, Y, Z) are obtained using the following determinant. The reference image coordinates (x _r , y _r ) are calculated.

As described above, since the foreground 350 exists in the generated second virtual view image 520, the occlusion area 340 or the area in which the brightness value is distorted is generated in the portion where the occlusion occurs. This phenomenon can be eliminated by overlapping the first virtual viewpoint image 510 and the second virtual viewpoint image 520, which will be described below.

When the first virtual view image 510 and the second virtual view image 520 are generated, the final virtual view image 530 is generated by overlapping the first virtual view image 510 and the second virtual view image 520. (S240).

The final virtual view image 530 from which the occlusion area 340 is removed is generated by overlapping the first virtual view image 510 and the second virtual view image 520. Since the foreground 350 does not exist in the first virtual view image 510, the occlusion area 340 does not occur. On the other hand, since the foreground 350 of the reference image 370 is present in the second virtual view image 520, the occlusion area 340 is generated, or the brightness value is awkwardly generated. Therefore, this portion can be removed by replacing the brightness value of the first virtual view image 510 in which the occlusion area 340 is not generated.

The foregoing description of the present invention is intended for illustration, and it will be understood by those skilled in the art that the present invention may be easily modified in other specific forms without changing the technical spirit or essential features of the present invention. will be. It is therefore to be understood that the above-described embodiments are illustrative in all aspects and not restrictive. For example, each component described as a single entity may be distributed and implemented, and components described as being distributed may also be implemented in a combined form.

The scope of the present invention is defined by the appended claims rather than the detailed description and all changes or modifications derived from the meaning and scope of the claims and their equivalents are to be construed as being included within the scope of the present invention do.

100: multi-view image generation system 110: camera unit
120: first virtual viewpoint image generator 130: second virtual viewpoint image generator
140: image superimposing unit 310: camera for reference video
320: background image camera 330: background area
340: occlusion area 350: foreground
360: background video 370: reference video
380: Depth image 510: First virtual view image
520: Second virtual viewpoint image 530: Final virtual viewpoint image

Claims (10)

In the multi-view video generating method using a multi-view video camera,
Generating a reference image, a depth image, and a background image of the photographing target;
Generating a first virtual view image based on the depth image and the background image;
Generating a second virtual view image based on the reference image and the depth image; and
And generating a final virtual viewpoint image by overlapping the first virtual viewpoint image and the second virtual viewpoint image.
The method of claim 1,
Generating the first virtual view image is
Calculating depth image coordinates on a reference coordinate system;
Calculating a background image coordinate based on the depth image coordinate;
Obtaining a brightness value of the background image corresponding to the background image coordinates; and
And changing a first virtual viewpoint image coordinate by allocating a brightness value of the background image as a brightness value of a first virtual viewpoint image.
3. The method of claim 2,
Depth image coordinates with respect to the first virtual view image coordinates
The position of the virtual camera for capturing the first virtual viewpoint image on a reference coordinate system (X _p , Y _p , Z _p ), the depth image coordinates (X, Y, Z), the reference coordinate system and the first virtual viewpoint The rotational movement matrix between the virtual cameras for capturing the image is R _p , the focal length of the virtual camera for capturing the first virtual viewpoint image is f _p , the scale constant is s _p, and the first virtual viewpoint image coordinate is (x _p , y _p )

Calculated using
The background image coordinates (x _b , y _b ) with respect to the first virtual view image coordinate are
The position of the camera for capturing the background image on the reference coordinate system (X _b , Y _b , Z _b ), the rotational movement matrix R _b between the reference coordinate system and the camera coordinate system for capturing the background image, and the background image When the camera's focal length is f _b and the scale constant is s _b ,

Multi-view image generation method using a multi-view video shooting camera that is calculated by using.
The method of claim 1,
Generating the second virtual view image is
Calculating depth image coordinates on a reference coordinate system;
Calculating reference image coordinates based on the depth image coordinates;
Calculating brightness values of the reference image corresponding to the reference image coordinates; and
And changing a second virtual view image coordinate by assigning a brightness value of the reference image as a brightness value of a second virtual view image.
5. The method of claim 4,
Depth image coordinates with respect to the second virtual view image coordinates
The position of the virtual camera for capturing the second virtual viewpoint image on the reference coordinate system (X _p , Y _p , Z _p ), the depth image coordinates (X, Y, Z), the reference coordinate system and the second virtual viewpoint image The rotational movement matrix between the virtual cameras for photographing R _p , the focal length of the virtual camera for capturing the second virtual viewpoint image, f _p , the scale constant s _p, and the second virtual viewpoint image coordinates (x _p). , y _p )

Calculated using
Reference image coordinates (x _r , y _r ) with respect to the coordinates of the second virtual view image are
The position of the camera for capturing the reference image on the reference coordinate system (X _r , Y _r , Z _r ), the rotational movement matrix between the reference coordinate system and the camera coordinate system for capturing the reference image R _r , and the reference image When the focal length of the camera to shoot is f _r and the scale constant is s _r ,

A multi-view image generating method using a multi-view image capturing camera calculated using.
The method of claim 1,
The depth image is a multi-view image generating method using a multi-view image recording camera that is generated by stereo matching between the reference image taken by the depth camera or taken in a multi-view.
The method of claim 1,
The camera for capturing the background image,
It is installed to only photograph the background at a different time point than the camera for capturing the reference image,
A multi-view image generating method using a multi-view video recording camera is installed to shoot at the same point of time as the camera for taking the reference image, but only the background in the state is removed.
In the multi-view image generation system,
A camera unit including a reference image camera for taking a reference image, a background image camera for taking a background image, and a depth camera for taking a depth image;
A first virtual view image generating unit generating a first virtual view image based on the background image and the depth image;
A second virtual viewpoint image generator configured to generate a second virtual viewpoint image based on the reference image and the depth image;
And an image superimposing unit for generating a final virtual view image by overlapping the first virtual view image and the second virtual view image.
In the multi-view image generation system,
A camera unit generating a reference image, a background image, and a depth image,
A first virtual view image generating unit generating a first virtual view image based on the background image and the depth image;
A second virtual viewpoint image generator configured to generate a second virtual viewpoint image based on the reference image and the depth image;
And an image overlapping unit for generating a final virtual view image by overlapping the first virtual view image and the second virtual view image,
The reference image is photographed by a reference image camera, the background image is photographed by a background image camera, and the depth image is generated by stereo matching between reference images photographed at a multiview. system.
10. The method according to claim 8 or 9,
The background image camera,
It is installed to only photograph the background at a different point of view than the reference image camera,
A multi-view image generation system using a multi-view video recording camera to shoot at the same time as the reference image camera but only the background in the foreground is removed.

Images