WO2012086912A1

WO2012086912A1 - Method for converting two-dimensional image into stereo image

Info

Publication number: WO2012086912A1
Application number: PCT/KR2011/008184
Authority: WO
Inventors: 유기령; 이광호; 김만배
Original assignee: Yoo Ki-Ryung; Lee Kwang-Ho; Kim Man-Bae
Priority date: 2010-12-24
Filing date: 2011-10-31
Publication date: 2012-06-28
Also published as: KR101207862B1; KR20120072786A

Abstract

The present invention relates to a method for converting a two-dimensional image into a stereo image. The present invention pertains to a method for converting a two-dimensional RGB image into a stereo image, which comprises: a first step of converting a two-dimensional RGB image into a two-dimensional YUV image; a second step of generating an edge map from the two-dimensional YUV image; a third step of generating each of an object depth map and a background depth map using the edge map, and generating a final depth map by combining the object depth map and the background depth map; a fourth step of acquiring a left eye image and a right eye image by horizontally moving the two-dimensional RGB image using the final depth map; and a fifth step of filling an empty hole pixel, which is generated in the fourth step, using surrounding pixels thereof.

Description

How to convert 2D image to stereo image

The present invention relates to a method for converting a 2D image into a stereo image. More specifically, an object depth map and a background depth map are separately generated and merged to generate a final depth map, and use the generated final depth map. The present invention relates to a stereo image.

Recently, as interest in 3D video (Stereoscopic Video) has been amplified, research on 3D video has been actively conducted. In general, it is known that humans feel the most three-dimensional effect by the parallax between both eyes. Thus, 3D imaging can be implemented using these characteristics of humans. For example, by distinguishing a particular subject into a left eye image seen through the viewer's left eye and a right eye image seen through the viewer's right eye, the viewer simultaneously displays the left eye image and the right eye image, thereby allowing the viewer to view the 3D image as a 3D image. I can make it visible. As a result, the 3D image may be implemented by producing a binocular image divided into a left eye image and a right eye image and displaying the same.

An object of the present invention is to create an object depth map and a background depth map separately from the two-dimensional image and merge them to create a final depth map, and to create a two-dimensional image to generate a more natural stereo image using the generated final depth map SUMMARY OF THE INVENTION An object of the present invention is to separately generate an object depth map and a background depth map from a two-dimensional image, merge them together to generate a final depth map, and use the generated final depth map. To provide a method of converting a two-dimensional image to a stereo image to create a more natural stereo image.

An object of the present invention is a method for converting a two-dimensional RGB image to a stereo image, a first step of converting a two-dimensional RGB image to a two-dimensional YUV image, and a second step of generating an edge map from the two-dimensional YUV image And a third step of generating an object depth map and a background depth map by using an edge map, and then integrating an object depth map and a background depth map to generate a final depth map, and using the final depth map. And a fifth step of filling the empty hole pixels generated in the fourth step and the fourth step of acquiring the left-eye image and the right-eye image by moving the RGB image in the horizontal direction, using the surrounding pixel values. Achievable by a method of converting a dimensional image into a stereo image.

In the present invention, the object depth map and the background depth map are separately generated from the two-dimensional image, and the final depth map is generated by merging them, and a more natural stereo image can be generated and provided using the generated final depth map.

1 is a flow chart illustrating a flow of converting a two-dimensional image to a stereo image according to the present invention.

2 is a diagram illustrating a method of generating a left and right background depth map in an edge map image having a size W × H.

FIG. 3 is a view for explaining a method of allocating depth to left and right background depth maps in an edge map image having a size W × H. FIG.

4 is a view for explaining a method of generating a vertical background depth map in an edge map image of size W x H.

FIG. 5 is a diagram for explaining a method of allocating depth to a vertical background depth map in an edge map image having a size W × H. FIG.

FIG. 6 is an explanatory diagram for explaining a method of filling a hole pixel generated according to left and right movement of an image; FIG.

7 is a flowchart for explaining a method of filling a hole pixel when a hole pixel occurs.

Hereinafter, with reference to the accompanying drawings will be described in detail a preferred embodiment, advantages and features of the present invention.

1 is an example of a flowchart illustrating a flow of converting a 2D image into a stereo image according to the present invention. As shown in FIG. 1, an input RGB image is converted into a YUV color model (ST100), and an edge map is generated by extracting an edge from the converted YUV color model (ST110). Next, an object depth map is generated using an edge map (ST120), and a left and right background depth map and an up and down background depth map are calculated using the same edge map (ST130 and ST140), and the calculated left and right background depth map and up and down are calculated. The background depth map is generated by integrating the background depth map (ST150). The final depth map is generated by integrating the generated object depth map and the background depth map (ST160), and the variation of pixels is obtained using the final depth map, and the left and right images are generated by moving the RGB image left and right by the variation (ST170). ). Next, by filling the empty hole pixels generated by the shift by the shift (ST180), it is possible to generate the final stereo image (I _R , I _L ) (ST190).

The processing flow described in FIG. 1 can be processed in hardware or software, as well as the processing techniques of the present invention in various devices. An example of a device to which the conversion technology of the present invention is applied is a device for converting a stereoscopic image into a stereo image in a television receiver receiving a 2D image and providing the same to a viewer. The television receiver receives a compressed two-dimensional image, decodes it, generates a two-dimensional RGB image, stores it in a memory, and performs the steps ST100 to ST180 of FIG. 1 using a processing processor. It generates a video for providing a video signal.

Next, each step of FIG. 1 will be described in detail.

1. 컬러모델변환1. Color Model Conversion

Given an RGB image, first convert it to the YUV color space. A YUV color image is represented by Y (luminance) brightness, U is blue-brightness, V is red-brightness, and has color difference information unlike an RGB image.

2. 에지맵 (edge map) 계산2. Edge map calculation

Next, the edge information must be obtained, and since the information obtained from a single data is uncertain, the edge data is obtained from the composite data. Edge information is applied to Y, U, and V, respectively, and the edge is extracted according to the following equation.

Variance of neighboring block pixels in each pixel as follows

Obtain First, the average value of the N × N block B including the given pixel in the Y image is calculated as in Equation 1.

[Equation 1]

Deviation

Is calculated as in Equation 2 below.

[Equation 2]

In addition, the deviation of the pixels in the U and V data is calculated as in Equation 3, Equation 4, Equation 5 and Equation 6 below.

[Equation 3]

[Equation 4]

[Equation 5]

[Equation 6]

,

Average is obtained

Is obtained using Equation 7.

[Equation 7]

Alternatively, the maximum value of the pixel can be obtained in addition to the average value.

[Equation 8]

Alternatively, the sum of the weighted products may be obtained as shown in Equation 9.

[Equation 9]

here

to be.

Once the deviation of each pixel is found,

Select one of the edge maps of the image

Obtain

In addition, edges include various edge detection methods such as Sobel edge operation and Canny edge operation. An average filter and an edge filter are combined to predict depth from a 2D image. The filter M satisfying this used the following equation (10).

[Equation 10]

F (i, j) in Equation 11 is a result obtained by convolving the image I with the filter M.

[Equation 11]

here

Is the convolution operator. The edge map is represented by the absolute value of F (i, j) as shown in Equation 12.

[Equation 12]

Finally, the final edge map EdgeMap

And the sum of the weighted products of Equation (13).

[Equation 13]

This reduces the uncertainty of the edges by using complex techniques rather than one edge extraction technique.

3. 객체 깊이맵 생성3. Create Object Depth Map

When the edge map obtained in Equation 13 is used as the depth map of the object, satisfactory depth cannot be obtained due to the loss of edge information and the strength difference between edge intensities between neighboring pixels. In this case, when viewing the left and right stereoscopic images, stereoscopic hearing deterioration such as eye fatigue occurs. To solve this problem, the edge map is processed as follows.

First, the edge map is converted to a normalized value with [0, 255]. The transformation uses a linear transformation as shown in equation (14). The maximum value of the EdgeMap

, The edge strength value E is equal to [0, 255].

Convert to a value.

[Equation 14]

Of all the pixels

Normalized Edgemap from Values

This is made. Average of edgemap pixels whose edge intensity is greater than threshold T at normalization value

Is calculated as shown in Equation 15. The threshold T here is an arbitrary number obtained empirically through various image processing.

[Equation 15]

In the next step, first, to reduce the difference in edge strength between neighboring pixels,

Same value for larger pixels

(E.g., 30) is assigned and 0 is assigned to non-pixels, so that pixels with very low intensity are ignored, and pixels that are not very strong are determined as the same edge once. The obtained thresholded edgemap is presented in equation (16).

to be.

[Equation 16]

and

Depth map of objects with proper combination of

Make

[Equation 17]

here

Has the function of adjusting the intensity of the three-dimensional effect.

4. 좌우 배경깊이맵 생성4. Create left and right background depth map

Determining the overall background composition plays an important role in conveying three-dimensional appearance. In a given image, it is necessary to determine whether the left side and the right side are in front and behind, and have different depths, which are determined by the left and right background depth maps. A background depth map is made using the edge map obtained in Equation 13.

FIG. 2 is a diagram for describing a method of generating left and right background depth maps in an edge map image having a size W × H. The vertical center line is j = W / 2, and sets v1, v2 on the left and v3, v4 on the right. Then, the edge strength of the left side is calculated by the following equation (18).

Equation 18

The edge strength on the right side is calculated in the following equation (19).

[Equation 19]

Where min (v1) = 0, max (v2) = H / 2, min (v3) = H / 2, max (v4) = W.

The depth range of the left and right background depth maps from the two edge intensities, [

] Is calculated. The minimum and maximum depth values are calculated as follows. Maximum depth value

In this case, the minimum depth value is determined by the following equation (20).

[Equation 20]

here

Can be specified by the user (eg 2.0, 3.0, etc.). ratio

Is calculated as in Equation 21 below.

[Equation 21]

If the difference between the two edge strengths is small, there is no depth difference between the two sides, and therefore, the depth difference value is reduced, and if the difference is large, the difference between the two means that the depth difference is large. Therefore, the proposed method has an advantage of assigning an appropriate depth value according to the image content. At the same time, it stores information about which side is far or near.

The range of the obtained depth values is D = [

]to be. As shown in FIG. 3, this range is assigned to a vertical line using the following equation (22). FIG. 3 is a diagram for describing a method of allocating depth to left and right background depth maps in an edge map image having a size W × H.

[Equation 22]

Where j = [0, W-1].

5. 상하 배경깊이맵 생성5. Create background depth map

In a given image, it is necessary to determine whether the upper side or the lower side is in front and behind, and the depth is different, which is determined by the upper and lower background depth map.

FIG. 4 is a diagram for describing a method of generating a vertical background depth map in an edge map image having a size W × H. The horizontal center line is a line with i = H / 2, and sets h1, h2, and h3, h4 up and down about the horizontal center line. Then, the edge strength at the top is calculated in the following equation (23).

[Equation 23]

The edge strength at the bottom is calculated in the following equation (24).

[Equation 24]

Once the two values are determined, the depth values of the upper and lower background depth maps are calculated as follows. Maximum depth value

In this case, the minimum depth value is determined by the following equation (25).

[Equation 25]

here

Can be specified by the user.

Where ratio

Is calculated as in Equation 26 below.

[Equation 26]

At the same time, either the bottom or the top is close and determines the dust.

The range of the obtained depth values is D = [

]to be. As shown in FIG. 5, this range is assigned the depth to the divided horizontal lines using the following equation (27). FIG. 5 is a diagram illustrating a method of allocating depth to upper and lower background depth maps in an edge map image having a size W × H.

[Equation 27]

6. 좌우/상하 깊이맵 통합6. Integration of left / right and up / down depth map

The background depth values obtained by the equations (22) and (27) are integrated using the following equation (28).

[Equation 28]

7. 최종 깊이맵 생성7. Create the final depth map

Finally, the depth map is obtained from the sum of the background depth map and the object depth map.

[Equation 29]

here

= [0, 1] can be adjusted to increase or decrease the three-dimensional effect of the object.

8. 좌안용 영상, 우안용 영상 생성8. Left eye image, right eye image generation

When the RGB image and the final depth map are obtained, the left eye image and the right eye image are calculated as in Equation 30 below. The disparity of the pixels is obtained and the image is moved horizontally to the left and the right to make a left eye image and a right eye image, respectively.

Equation 30

Where I _L and I _R are the left eye image and the right eye image, respectively, and the variation d is calculated from D in Equation 29 using Equation 31 below.

Equation 31

Where τ is the maximum stereo parallax. When the image is moved by the shift value, a hole occurs. These holes are filled with the surrounding pixel values.

The range of D in the depth map is typically [0, 255]. This is transformed into a variation d. Each pixel moves to the left in the left image and to the right in the right image according to the corresponding d value. Therefore, the maximum parallax that can occur in the left and right images is 2d.

9. 홀화소 채우기(Hole pixel filling)9. Hole pixel filling

When the image is moved left and right, empty hole pixels are generated. The hole pixels are filled as shown in FIG. 6, which is called hole filling. FIG. 6 is an explanatory diagram for explaining a method of filling a hole pixel generated according to left and right movement of an image. Hole filling generally uses an average value of neighboring pixels, but an improved image quality can be obtained by using an interpolation technique.

Empty hole pixels

In this case, N pixels are searched in the same scan line. When each pixel of the image is moved to the displacement value of Equation 31, a pixel having a large displacement value is relatively far away, and a hole is generated in the movement of the pixel having a small displacement value. 7 is a flowchart illustrating a method of filling a hall pixel when the hall pixel is generated. As shown in FIG. 7, when L Hall pixels are generated (ST700), deviations of the pixels are continuously calculated in a direction in which the pixel having a large disparity value exists (ST710). Search until the deviation value is less than or equal to the threshold P. Searched pixels

If the number of is N, L pixels

Fill them with interpolation. First, a scaling factor is obtained as shown in Equation 32.

Equation 32

After the expansion ratio is obtained, pixel values are obtained by using an inverse transformation of Equation 33.

[Equation 33]

10. 최종 좌안용 영상 및 최종 우안용 영상 생성10. Generating final left eye image and final right eye image

When the left eye image and the right eye image are completed, the hole filling is completed, and then converted into Top-bottom, Side by Side, Vertical Interleaving, and Interlaced formats for transmission format of 3D display.

While the preferred embodiments of the present invention have been described and illustrated using specific terms, such terms are only for clarity of the present invention, and the embodiments and the described terms of the present invention are defined and the technical spirit and scope of the following claims. It is obvious that various changes and changes can be made without departing from the scope.

Claims

In the method for converting a two-dimensional RGB image to a stereo image,

A first step of converting the two-dimensional RGB image into a two-dimensional YUV image;

A second step of generating an edge map from the two-dimensional YUV image;

A third step of generating an object depth map and a background depth map by using the edge map, and then integrating the object depth map and the background depth map to generate a final depth map;

A fourth step of obtaining a left eye image and a right eye image by moving the 2D RGB image in a horizontal direction by using the final depth map; And

And a fifth step of filling the empty hole pixels generated in the fourth step by using surrounding pixel values.
The method of claim 1,

The second step calculates respective deviations of Y, U, and V of neighboring block pixels, and uses an edge map using any one selected from the mean value, the maximum value of the deviations, and the weighted product.
Method for converting a two-dimensional image to a stereo image, characterized in that for generating a.
The method of claim 2,

The edge map
silver

(1) An edge map generated by using each selected deviation of Y, U, and V of neighboring block pixels, and using any one selected from an average value of each deviation, a maximum value of the deviations, and a weighted product.
Wow,

(2) An edge map expressed as an absolute value of the result of convolution of the two-dimensional image I (i, j) with the filter of Equation 34 below
Method for converting a two-dimensional image to a stereo image, characterized in that generated by the weighted sum of.

Equation 34
The method of claim 2 or 3,

The object depth map generation in the third step

The edge map
Normalized edge map from
Generating a 3-1 step;

Average of Edgemap Pixels Above Threshold T
Calculate,
Edge map that is thresholded by assigning the same value (K) to larger pixels and assigning zeros to equal or smaller pixels
Generating a 3-2 step; And

The normalized edgemap
And the thresholded edgemap
A method for converting a 2D image into a stereo image, characterized in that to generate an object depth map by the weighted sum of.
The method of claim 1,

In the third step, the left and right background depth maps are generated by dividing an edge map based on vertical bisectors, and then a difference value between sums of edge maps of pixels located on the left and divided edge maps of pixels located on the divided right. Method for converting a two-dimensional image to a stereo image, characterized in that for generating using.
The method of claim 1,

In the third step, the upper and lower background depth maps are generated by dividing an edge map based on horizontal bisectors, and then, a difference value between sums of edge maps of pixels located above the divided pixels and edge map sums of the pixels located below the divided pixels. Method for converting a two-dimensional image to a stereo image, characterized in that for generating using.
The method of claim 1,

The amount of shifting the two-dimensional RGB image in the horizontal direction of the fourth step is determined as a product of the maximum stereo parallax and the function of the depth map is converted into a stereo image.
The method of claim 1,

The fifth step is a method for converting a two-dimensional image to a stereo image, characterized in that performed using interpolation.