KR20170100229A - System for displaying multiview image - Google Patents

Abstract

A multi-view image display system and a method thereof are disclosed. According to the present invention, the system generates a parallax which corresponds to a plurality of view points with respect to one pixel from depth information of a depth image and generates a multi-view point image with respect to one pixel from color information of a color image and the parallax which corresponds to the plurality of view points.

Description

[0001] SYSTEM FOR DISPLAYING MULTIVIEW IMAGE [0002]

The present invention relates to a multi-view image display system.

With the development of multimedia technology, there is a growing interest in realistic media. Among them, a multi-view image means a set of images obtained by using two or more cameras (multi-view cameras) having different viewpoints. Unlike conventional single-view video, multi-view video can generate three-dimensional dynamic images by shooting one scene through multiple cameras. Such a multi-view image can provide a user with a stereoscopic effect through a free view point and a wide screen.

On the other hand, in the movie industry and broadcasting stations, image synthesis technology is required for special effects. In particular, it is no exaggeration to say that it is essential to acquire excellent image synthesis technology these days when blockbuster with advanced 3-dimensional computer graphic technology is poured out.

Generally, in order to generate multi-view images, N multi-viewpoint cameras for acquiring color images are arranged, a depth camera for acquiring depth images is arranged, and a depth image and a color image of a multi- Thereby generating an image.

FIG. 1 illustrates an example in which a conventional multi-view camera and a depth camera are disposed.

A plurality of multi-viewpoint cameras 100 are composed of n cameras, and viewpoints of a plurality of cameras are different according to positions of cameras, and a plurality of images having different viewpoints are called multi-view images. The multi-view image obtained from the multi-view camera 100 includes color information per pixel on the two-dimensional image forming each image, but does not include depth information. The depth camera 200 acquires a depth map having three-dimensional depth information.

As described above, in the conventional multi-viewpoint image generating system, the multi-viewpoint cameras 100 are arranged in a row, and the depth camera 200 is disposed at the lower end or the upper end of any one of them.

However, in the conventional multi-view image generation system as described above, a camera for acquiring a plurality of color images is required, so that the cost of system configuration is high and the time and cost required for synthesizing multi- .

An object of the present invention is to provide an apparatus and method for generating a wide baseline multi-view image using one color information image and a corresponding depth information image.

According to an aspect of the present invention, there is provided a multi-view image generating apparatus comprising: a first generating unit that generates a parallax corresponding to a plurality of viewpoints of one pixel from depth information of a depth image; And a second generating unit for generating a multi-view image for the one pixel from the color information of the color image and the parallax corresponding to the plurality of viewpoints.

에 의해 생성할 수 있으며, 이때, a는 계수, D는 상기 하나의 픽셀의 깊이정보, b는 오프셋일 수 있다.In an embodiment of the present invention, the first generation unit may generate a time difference corresponding to the plurality of times

, Where a is a coefficient, D is depth information of the one pixel, and b is an offset.

In an embodiment of the present invention, the first generating unit may assign a constant magnitude change to the coefficient a of the equation for a plurality of viewpoints arranged at equidistant intervals in a wide-baseband, The time difference can be determined.

In an embodiment of the present invention, the second generating unit may determine coordinates corresponding to the plurality of viewpoints of the pixels corresponding to the parallax corresponding to the plurality of viewpoints, and assign the color information to the coordinates Thereby generating a multi-view image.

According to another aspect of the present invention, there is provided a multi-view image display system including a first generation unit generating a parallax corresponding to a plurality of viewpoints of one pixel from depth information of a depth image, part; A second generating unit for generating a multi-view image for the one pixel from the color information of the color image and the parallax corresponding to the plurality of viewpoints; And a display unit for displaying the multi-view image.

에 의해 생성할 수 있고, a는 계수, D는 상기 하나의 픽셀의 깊이정보, b는 오프셋일 수 있다.In an embodiment of the present invention, the first generation unit may generate a time difference corresponding to the plurality of times

, A may be a coefficient, D may be depth information of the one pixel, and b may be an offset.

In an embodiment of the present invention, the first generating unit may assign a constant magnitude change to the coefficient a of the equation for a plurality of viewpoints arranged at equidistant intervals in a wide-baseband, The time difference can be determined.

In an embodiment of the present invention, the second generating unit may determine coordinates corresponding to the plurality of viewpoints of the pixels corresponding to the parallax corresponding to the plurality of viewpoints, and assign the color information to the coordinates Thereby generating a multi-view image.

As described above, since it is possible to generate a multi-view image from one color image and a depth image without acquiring multi-view images by a plurality of cameras, cost and workforce Thereby reducing the consumption of the battery.

FIG. 1 illustrates an example in which a conventional multi-view camera and a depth camera are disposed.
2 is a block diagram illustrating a multi-view image display system according to an embodiment of the present invention.
FIGS. 3A through 3C illustrate an example of a multi-view image generated by a color image and a depth image in an embodiment of the present invention.
4 is a flowchart illustrating a multi-view image generating method according to an embodiment of the present invention.

While the invention is susceptible to various modifications and alternative forms, specific embodiments thereof are shown by way of example in the drawings and will herein be described in detail. It should be understood, however, that the invention is not intended to be limited to the particular embodiments, but includes all modifications, equivalents, and alternatives falling within the spirit and scope of the invention.

Hereinafter, a preferred embodiment of the present invention will be described in detail with reference to the attached drawings.

The present invention relates to an apparatus and method for generating n wide-baseline multi-view images from one color image and a depth image. The wide-baseline multi-view image is a color image shot from n cameras placed on a straight line having a large length.

One embodiment of the present invention is to create an arbitrary viewpoint on the wide-baseline from one color image and depth image. That is, it is possible to generate an arbitrary viewpoint on the wide-baseline, so that n viewpoints can be finally generated.

Referring to FIG. 1, a point P (x, y, z) inside a three-dimensional space is projected on a plurality of cameras and represented at a point (two dimensions u, v) on one plane of the camera film. In the case of the camera 100 disposed on the wide-baseline, since the y and z coordinates of each camera are identical and only the x coordinate changes, a point P (x, y, z) The coordinates (u, v) on the film of the camera 100 differ only by the u coordinate.

The difference between the different u coordinates formed on the film of the camera 100 at different positions with respect to one three-dimensional point P is referred to as parallax, and this is affected by the z-coordinate among the coordinates of the three-dimensional point P.

In particular, in the wide-baseline setting, the larger the z (the depth between the camera and the point) is, the more it changes in a certain direction, so that the parallax is proportional to z.

The present invention makes use of the characteristics of the three-dimensional space, and it is also possible to use a plurality of cameras arranged on a wide-baseline from one color image and one depth image without arranging a plurality of cameras on a wide- The same multi-view image as the obtained image can be obtained.

2 is a block diagram illustrating a multi-view image display system according to an embodiment of the present invention.

As shown in the drawing, the system of an embodiment of the present invention includes a color image acquiring unit 10, a depth image acquiring unit 20, a parallax generating unit 30, a multi-view image generating unit 40, (50).

The system of an embodiment of the present invention may be implemented in a computer system such as a personal computer (PC), or in a 3D TV, or in a smart device such as a smart phone or smart pad.

The color image acquiring unit 10 may acquire a color image for one view and the depth image acquiring unit 20 may acquire a depth image for the corresponding color image.

The image acquired by the color image acquiring unit 10 includes color information, and may include red-green (RGB) information for each pixel, for example.

In addition, the image acquired by the depth image acquiring unit 20 includes depth information including depth information on three-dimensional depth. For example, a laser or an infrared ray irradiated from a laser or an infrared ray irradiating unit (not shown) And a depth sensor (not shown) for sensing the depth information through a laser or an infrared ray.

The color image acquiring unit 10 and the depth image acquiring unit 20 may each be constituted by a camera as shown in FIG. 1 or a camera module and an image sensor (for example, a CCD (Charge-Coupled Device) Oxide Semiconductor), or the like, and may be configured in various other ways.

The parallax generator 30 can generate the parallax of the virtual multi-view camera from the depth image. In the depth image, depth information D is assumed for an arbitrary point.

In the two-dimensional image, the following relationship exists between D and time difference.

Where a may be a coefficient and b may be an offset. However, since the two are not linear as shown in Equation (1) above, the nonlinear model is used to rearrange the following equations.

Therefore, assuming c (1), c (2), c (3) ..., c (N), which are other cameras on the baseline, for virtual reference camera c The parallax between the virtual cameras can be obtained by adjusting the coefficient a.

In the general wide-baseline camera system, since the plurality of cameras c (1), c (2), c (3) ... c (N) are arranged at equal intervals from each other, the parallax generator 30 It is possible to determine the parallax at the n point by giving a constant magnitude change to the coefficient a.

2, the multi-view image generator 40 may generate a multi-view image using the time difference generated by the view generator 30. In the color image, the color information for an arbitrary point is denoted by C. The disparity is the difference in u coordinate when a point is projected on each camera as described above.

(1), P (2), P (3), ..., P (n) representing the parallax with respect to the n cameras are expressed as a time difference The coordinates of corresponding points in the n cameras are (u + P (1), v), (u + P (2), v) (n), v).

Therefore, the multi-view image generating unit 40 can generate the multi-view image by giving the color information C to the coordinates (u + P (n), v) corresponding to the image space of each camera.

The display unit 50 can display the multi-view image generated by the multi-view image generating unit 40. [

FIGS. 3A through 3C illustrate an example of a multi-view image generated by a color image and a depth image in an embodiment of the present invention.

As shown in the drawing, according to the present invention, a multi-viewpoint image as shown in FIG. 3C can be generated using the color image of FIG. 3A and the depth image of FIG. 3B.

4 is a flowchart illustrating a multi-view image generating method according to an embodiment of the present invention.

As shown in the figure, a method of an embodiment of the present invention can receive a color image from a color image acquiring unit 10 and a depth image from a depth image acquiring unit 20 (S10).

Thereafter, the parallax generator 30 generates parallax corresponding to a plurality of points of time for one pixel from the color image and the depth image (S11), and calculates a parallax corresponding to a plurality of points of time for the pixels of the entire image (S12).

There is a relationship between the depth D of an arbitrary pixel and the parallax as shown in Equation 1, which can be expressed by Equation 2 as a nonlinear model. That is, since each viewpoint (i.e., camera) of the wide-baseline is arranged at equal intervals, the parallax at the n viewpoint can be obtained by giving a constant magnitude change to the coefficient a of Equation (2).

Thereafter, the multi-view image generating unit 40 can generate a multi-view image for one pixel by using the color information of the color image and the parallax generated by the parallax generator 30 (S13) A multi-view image of a pixel of the whole image can be generated (S14).

(1), P (2), P (3), ..., P (n) representing the parallax with respect to the n cameras are expressed as a time difference The coordinates of corresponding points in the n cameras are (u + P (1), v), (u + P (2), v) (n), v). Therefore, the multi-view image generating unit 40 can generate the multi-view image by giving the color information C to the coordinates (u + P (n), v) corresponding to the image space of each camera.

When a multi-view image is generated for the entire image, the display unit 50 may display the multi-view image (S15).

It will be appreciated that the system of an embodiment of the present invention may be implemented in a computer system such as a personal computer (PC), or implemented in a 3D TV, or in a smart device such as a smart phone or smart pad same.

In an embodiment of the present invention, the color image and depth image acquisition units 10 and 20 and the display unit 50 are implemented in the same system. However, the present invention is not limited thereto, The image acquiring units 10 and 20 may be separately disposed to acquire images, and the system of the present invention may receive and display the images.

Also, in the embodiment of the present invention, the parallax generator 30, the multi-view image generator 40, and the display unit 50 are implemented in one system, but the present invention is not limited thereto The parallax generation unit 30 and the multi-view image generation unit 40 are implemented in a system separate from the display unit 50 so that the output of the multi-view image generation unit 40 is displayed on the display unit 50).

According to an embodiment of the present invention, since it is possible to generate a multi-view image from one color image and a depth image without acquiring multi-view images by a plurality of cameras, a multi-view image is produced Which can reduce the cost and manpower consumption.

While the invention has been shown and described with reference to certain preferred embodiments thereof, it will be understood by those skilled in the art that various changes and modifications may be made without departing from the spirit and scope of the invention as defined by the appended claims. Accordingly, the true scope of the present invention should be determined by the following claims.

10: Color image acquiring unit 20: Depth image acquiring unit
30: parallax generator 40: multi-view image generator
50:

Claims (8)

A first generating unit for generating, from the depth information of the depth image, a parallax corresponding to a plurality of points of time for one pixel; And
And a second generation unit for generating a multi-view image for the one pixel from the color information of the color image and the parallax corresponding to the plurality of viewpoints.
2. The apparatus according to claim 1,
And generates a parallax corresponding to the plurality of viewpoints by the following equation.

(a is a coefficient, D is depth information of the one pixel, and b is an offset)
3. The apparatus according to claim 2,
Wherein a plurality of time points arranged at equal intervals in a wide-base band are given a constant magnitude change in the coefficient a of the equation to determine a parallax corresponding to the plurality of points of time.
The apparatus according to claim 1,
Wherein the image processing apparatus determines coordinates corresponding to the plurality of viewpoints of the pixels corresponding to the parallax corresponding to the plurality of viewpoints and adds the color information to the coordinates to generate a multi- Generating device.
A first generating unit for generating, from the depth information of the depth image, a parallax corresponding to a plurality of points of time for one pixel;
A second generating unit for generating a multi-view image for the one pixel from the color information of the color image and the parallax corresponding to the plurality of viewpoints; And
And a display unit for displaying the multi-view image.
6. The apparatus according to claim 5,
And generates a parallax corresponding to the plurality of viewpoints by the following equation.

(a is a coefficient, D is depth information of the one pixel, and b is an offset)
7. The apparatus of claim 6,
Wherein a plurality of time points arranged at equal intervals in a wide-base band are given a constant magnitude variation in the coefficient a of the equation to determine a parallax corresponding to the plurality of points of time.
6. The apparatus according to claim 5,
Wherein the image processing apparatus determines coordinates corresponding to the plurality of viewpoints of the pixels corresponding to the parallax corresponding to the plurality of viewpoints and adds the color information to the coordinates to generate a multi- Display system.

Images