KR101302431B1 - Method for converting 2 dimensional video image into stereoscopic video - Google Patents

Abstract

The present invention relates to a method for converting a two-dimensional image to a stereo image. In the method for converting a two-dimensional RGB image to a stereo image, the present invention includes a first step of converting a two-dimensional RGB image to a two-dimensional YUV image; The second step of generating an edge map from the two-dimensional YUV image, and after generating the object depth map and the background depth map using the edge map, respectively, the final depth map by integrating the object depth map and the background depth map Pixels around the empty holes generated in the fourth and fourth steps of acquiring the left eye image and the right eye image by moving the two-dimensional RGB image in a horizontal direction using a final depth map. The present invention relates to a method for converting a 2D image into a stereo image, comprising a fifth step of filling using a value.

Description

How to convert 2D video to stereo video {METHOD FOR CONVERTING 2 DIMENSIONAL VIDEO IMAGE INTO STEREOSCOPIC VIDEO}

The present invention relates to a method for converting a 2D image into a stereo image. More specifically, an object depth map and a background depth map are separately generated and merged to generate a final depth map, and use the generated final depth map. The present invention relates to a stereo image.

Recently, as interest in 3D video (Stereoscopic Video) has been amplified, research on 3D video has been actively conducted. In general, it is known that humans feel the most three-dimensional effect by the parallax between both eyes. Thus, 3D imaging can be implemented using these characteristics of humans. For example, by distinguishing a particular subject into a left eye image seen through the viewer's left eye and a right eye image seen through the viewer's right eye, the viewer simultaneously displays the left eye image and the right eye image, thereby allowing the viewer to view the 3D image as a 3D image. I can make it visible. As a result, the 3D image may be implemented by producing a binocular image divided into a left eye image and a right eye image and displaying the same.

International Publication Number WO 1997/24000 (International Publication Date: July 3, 1997)

An object of the present invention is to create an object depth map and a background depth map separately from the two-dimensional image and merge them to create a final depth map, and to create a two-dimensional image to generate a more natural stereo image using the generated final depth map SUMMARY OF THE INVENTION An object of the present invention is to separately generate an object depth map and a background depth map from a two-dimensional image, merge them together to generate a final depth map, and use the generated final depth map. To provide a method of converting a two-dimensional image to a stereo image to create a more natural stereo image.

An object of the present invention is a method for converting a two-dimensional RGB image to a stereo image, a first step of converting a two-dimensional RGB image to a two-dimensional YUV image, and a second step of generating an edge map from the two-dimensional YUV image And a third step of generating an object depth map and a background depth map by using an edge map, and then integrating an object depth map and a background depth map to generate a final depth map, and using the final depth map. And a fifth step of filling the empty hole pixels generated in the fourth step and the fourth step of acquiring the left-eye image and the right-eye image by moving the RGB image in the horizontal direction, using the surrounding pixel values. Achievable by a method of converting a dimensional image into a stereo image.

In the present invention, the object depth map and the background depth map are separately generated from the two-dimensional image, and the final depth map is generated by merging them, and a more natural stereo image can be generated and provided using the generated final depth map.

1 is a flow chart illustrating a flow of converting a two-dimensional image to a stereo image according to the present invention.
2 is a diagram illustrating a method of generating a left and right background depth map in an edge map image having a size W × H.
FIG. 3 is a view for explaining a method of allocating depth to left and right background depth maps in an edge map image having a size W × H. FIG.
4 is a view for explaining a method of generating a vertical background depth map in an edge map image of size W x H.
FIG. 5 is a diagram for explaining a method of allocating depth to a vertical background depth map in an edge map image having a size W × H. FIG.
FIG. 6 is an explanatory diagram for explaining a method of filling a hole pixel generated according to left and right movement of an image; FIG.
7 is a flowchart for explaining a method of filling a hole pixel when a hole pixel occurs.

In the following, preferred embodiments, advantages and features of the present invention will be described in detail with reference to the accompanying drawings.

1 is an example of a flowchart illustrating a flow of converting a 2D image into a stereo image according to the present invention. As shown in FIG. 1, an input RGB image is converted into a YUV color model (ST100), and an edge map is generated by extracting an edge from the converted YUV color model (ST110). Next, an object depth map is generated using an edge map (ST120), and a left and right background depth map and an up and down background depth map are calculated using the same edge map (ST130 and ST140), and the calculated left and right background depth map and up and down are calculated. The background depth map is generated by integrating the background depth map (ST150). The final depth map is generated by integrating the generated object depth map and the background depth map (ST160), and the variation of pixels is obtained using the final depth map, and the left and right images are generated by moving the RGB image left and right by the variation (ST170). ). Next, by filling the empty hole pixels generated by the shift by the shift (ST180), it is possible to generate the final stereo image (I _R , I _L ) (ST190).

The processing flow described in FIG. 1 can be processed in hardware or software, as well as the processing techniques of the present invention in various devices. An example of a device to which the conversion technology of the present invention is applied is a device for converting a stereoscopic image into a stereo image in a television receiver receiving a 2D image and providing the same to a viewer. The television receiver receives a compressed two-dimensional image, decodes it, generates a two-dimensional RGB image, stores it in a memory, and performs the steps ST100 to ST180 of FIG. 1 using a processing processor. It generates a video for providing a video signal.

Next, each step of FIG. 1 will be described in detail.

1. Color Model Conversion

Given an RGB image, first convert it to the YUV color space. A YUV color image is represented by Y (luminance) brightness, U is blue-brightness, V is red-brightness, and has color difference information unlike an RGB image.

2. Edge map  ( edge map ) Calculation

Next, the edge information must be obtained, and since the information obtained from a single data is uncertain, the edge data is obtained from the composite data. Edge information is applied to Y, U, and V, respectively, and the edge is extracted according to the following equation.

를 구한다. 먼저 Y영상에서 주어진 화소를 포함하는 NxN 블록 B의 평균값은 수학식 1과 같이 계산된다.Variance of neighboring block pixels in each pixel as follows

. First, the average value of the N × N block B including the given pixel in the Y image is calculated as in Equation 1.

은 다음 수학식 2와 같이 계산된다.Deviation

Is calculated as in Equation 2 below.

In addition, the deviation of the pixels in the U and V data is calculated as in Equation 3, Equation 4, Equation 5 and Equation 6 below.

,

,

가 얻어지면 평균

를 수학식 7를 이용하여 구한다.

,

,

Average is obtained

Is obtained using Equation 7.

Alternatively, the maximum value of the pixel can be obtained in addition to the average value.

Alternatively, the sum of the weighted products may be obtained as shown in Equation 9.

이다.here

to be.

중에서 하나를 선택하여 영상의 에지맵

을 구한다.Once the deviation of each pixel is found,

Select one of the edge maps of the image

.

In addition, edges include various edge detection methods such as Sobel edge operation and Canny edge operation. An average filter and an edge filter are combined to predict depth from a 2D image. The filter M satisfying this used the following equation (10).

F (i, j) in Equation 11 is a result obtained by convolving the image I with the filter M.

는 콘볼루션 연산자이다. 에지맵은 수학식 12와 같이 F(i,j)의 절대값으로 표현된다.here

Is the convolution operator. The edge map is represented by the absolute value of F (i, j) as shown in Equation 12.

와 수학식 13의 가중치곱의 합으로 얻어진다.Finally, the final edge map EdgeMap

And the sum of the weighted products of Equation (13).

This reduces the uncertainty of the edges by using complex techniques rather than one edge extraction technique.

3. Object Depth map  produce

When the edge map obtained in Equation 13 is used as the depth map of the object, satisfactory depth cannot be obtained due to the loss of edge information and the strength difference between edge intensities between neighboring pixels. In this case, when viewing the left and right stereoscopic images, stereoscopic hearing deterioration such as eye fatigue occurs. To solve this problem, the edge map is processed as follows.

라고 하면, 에지강도 값 E는 [0, 255]의

값으로 변환한다.First, the edge map is converted to a normalized value with [0, 255]. The transformation uses a linear transformation as shown in equation (14). The maximum value of the EdgeMap

, The edge strength value E is equal to [0, 255].

Convert to a value.

값으로부터 정규화된 에지맵

이 만들어진다. 정규화 값에서 에지강도가 임계치 T보다 큰 에지맵 화소들의 평균

을 수학식 15와 같이 계산한다. 여기서 임계치 T는 여러 영상 처리를 통해 경험적으로 얻어지는 임의의 수이다.
Of all the pixels

Normalized Edgemap from Values

Is created. Average of edgemap pixels whose edge intensity is greater than threshold T at normalization value

Is calculated as shown in Equation 15. The threshold T here is an arbitrary number obtained empirically through various image processing.

보다 큰 화소에는 동일한 값

(예, 30) 을 할당하고, 아닌 화소에는 0을 할당하여, 강도가 매우 약한 화소는 무시하고, 그렇지 않은 화소로 일단 동일 에지로 판단한다. 얻어진 임계치화된 에지맵은 수학식 16에 제시된

이다. In the next step, first, to reduce the difference in edge strength between neighboring pixels,

Same value for larger pixels

(E.g., 30) is assigned and 0 is assigned to non-pixels, so that pixels with very low intensity are ignored, and pixels that are not very strong are determined as the same edge once. The obtained thresholded edgemap is presented in equation (16).

to be.

과

의 적절한 조합으로 객체의 깊이맵

을 만든다.Next

and

Depth map of objects with proper combination of

Lt; / RTI >

는 입체감의 강도를 조절하는 기능을 가지고 있다.
here

Has the function of adjusting the intensity of the three-dimensional effect.

4. Left and right Background Depth Map  produce

Determining the overall background composition plays an important role in conveying three-dimensional appearance. In a given image, it is necessary to determine whether the left side and the right side are in front and behind, and have different depths, which are determined by the left and right background depth maps. A background depth map is made using the edge map obtained in Equation 13.

FIG. 2 is a diagram for describing a method of generating left and right background depth maps in an edge map image having a size W × H. The vertical center line is j = W / 2, and sets v1, v2 on the left and v3, v4 on the right. Then, the edge strength of the left side is calculated by the following equation (18).

The edge strength on the right side is calculated in the following equation (19).

Where min (v1) = 0, max (v2) = H / 2, min (v3) = H / 2, max (v4) = W.

]를 계산한다. 최소 및 최대 깊이값은 다음과 같이 계산된다. 최대 깊이값이

이면, 최소 깊이값은 다음 수학식 20으로 결정한다.The depth range of the left and right background depth maps from the two edge intensities, [

]. The minimum and maximum depth values are calculated as follows. Maximum depth value

In this case, the minimum depth value is determined by the following equation (20).

은 사용자가 지정할 수 있다(예를 들어 2.0, 3.0, 등등). 비율

는 다음 수학식 21과 같이 계산된다.here

Can be specified by the user (eg 2.0, 3.0, etc.). ratio

Is calculated as in Equation 21 below.

If the difference between the two edge strengths is small, there is no depth difference between the two sides, and therefore, the depth difference value is reduced, and if the difference is large, the difference between the two means a large depth difference. Therefore, the proposed method has an advantage of assigning an appropriate depth value according to the image content. At the same time, it stores information about which side is far or near.

]이다. 이 범위를 도 3에서 보는 것처럼, 다음 수학식 22를 이용하여 수직선에 깊이를 할당한다. 도 3은 크기가 W x H 인 에지맵 영상에서 좌우 배경깊이맵에 깊이를 할당하는 방식을 설명하는 도면이다.The range of the obtained depth values is D = [

]to be. As shown in FIG. 3, this range is assigned to a vertical line using the following equation (22). FIG. 3 is a diagram for describing a method of allocating depth to left and right background depth maps in an edge map image having a size W × H.

Where j = [0, W-1].

5. Up and down Background Depth Map  produce

In a given image, it is necessary to determine whether the upper side or the lower side is in front and behind, and the depth is different, which is determined by the upper and lower background depth map.

FIG. 4 is a diagram for describing a method of generating a vertical background depth map in an edge map image having a size W × H. The horizontal center line is a line with i = H / 2, and sets h1, h2, and h3, h4 up and down about the horizontal center line. Then, the edge strength at the top is calculated in the following equation (23).

The edge strength at the bottom is calculated in the following equation (24).

이면, 최소 깊이값은 다음 수학식 25로 결정한다.Once the two values are determined, the depth values of the upper and lower background depth maps are calculated as follows. Maximum depth value

In this case, the minimum depth value is determined by the following equation (25).

은 사용자가 지정할 수 있다. here

Can be specified by the user.

는 다음 수학식 26과 같이 계산된다.Where ratio

Is calculated as in Equation 26 below.

At the same time, either the bottom or the top is close and determines the dust.

]이다. 이 범위를 도 5에서 보는 것처럼, 다음 수학식 27을 이용하여 분할된 수평선에 깊이를 할당한다. 도 5는 크기가 W x H 인 에지맵 영상에서 상하 배경깊이맵에 깊이를 할당하는 방식을 설명하는 도면이다.The range of the obtained depth values is D = [

]to be. As shown in FIG. 5, this range is assigned the depth to the divided horizontal lines using the following equation (27). FIG. 5 is a diagram illustrating a method of allocating depth to upper and lower background depth maps in an edge map image having a size W × H.

6. Left / Right / Up / Down Depth map  integrated

The background depth values obtained by the equations (22) and (27) are integrated using the following equation (28).

7. Final Depth map  produce

Finally, the depth map is obtained from the sum of the background depth map and the object depth map.

= [0, 1]을 조정하여, 객체의 입체감의 증감이 가능하다.
here

= [0, 1] can be adjusted to increase or decrease the three-dimensional effect of the object.

8. Left eye  video, Right eye  Create image

When the RGB image and the final depth map are obtained, the left eye image and the right eye image are calculated as in Equation 30 below. The disparity of the pixels is obtained and the image is moved horizontally to the left and the right to make a left eye image and a right eye image, respectively.

Where I _L and I _R are the left eye image and the right eye image, respectively, and the variation d is calculated from D in Equation 29 using Equation 31 below.

Where τ is the maximum stereo parallax. When the image is moved by the shift value, a hole occurs. These holes are filled with the surrounding pixel values.

The range of D in the depth map is typically [0, 255]. This is transformed into a variation d. Each pixel moves to the left in the left image and to the right in the right image according to the corresponding d value. Therefore, the maximum parallax that can occur in the left and right images is 2d.

9. Hall pixel  Fill ( Hole pixel filling )

When the image is moved left and right, empty hole pixels are generated. The hole pixels are filled as shown in FIG. 6, which is called hole filling. FIG. 6 is an explanatory diagram for explaining a method of filling a hole pixel generated according to left and right movement of an image. Hole filling generally uses an average value of neighboring pixels, but an improved image quality can be obtained by using an interpolation technique.

이면, 동일 스캔라인에서 N개의 화소를 탐색한다. 영상의 각 화소를 수학식 31의 변위값으로 이동하게 되면, 변위값이 큰 화소는 상대적으로 멀리 보이게 되는데, 변위값이 작은 화소의 이동에서 홀이 발생하게 된다. 도 7은 홀 화소가 발생한 경우, 홀 화소를 채우는 방법을 설명하는 흐름도이다. 도 7에 도시된 바와 같이 홀 화소가 L개 발생하면(ST700), 변이 값이 큰 화소가 있는 방향으로 연속적으로 화소들의 편차를 구한다(ST710). 구한 편차값이 임계치 P보다 작거나 동일할 때까지 탐색한다. 탐색한 화소들

의 개수가 N이면, N개의 화소로 L개의 홀 화소

들을 보간법으로 채운다. 먼저 수학식 32와 같이 신축비율(Scale factor)을 구한다.
Empty hole pixels

In this case, N pixels are searched in the same scan line. When each pixel of the image is moved to the displacement value of Equation 31, a pixel having a large displacement value is relatively far away, and a hole is generated in the movement of the pixel having a small displacement value. 7 is a flowchart illustrating a method of filling a hall pixel when the hall pixel is generated. As shown in FIG. 7, when L Hall pixels are generated (ST700), deviations of the pixels are continuously calculated in a direction in which the pixel having a large disparity value exists (ST710). Search until the deviation value is less than or equal to the threshold P. Searched pixels

If the number of is N, L pixels

Fill them with interpolation. First, a scaling factor is obtained as shown in Equation 32.

After the expansion ratio is obtained, pixel values are obtained by using an inverse transformation of Equation 33.

10. Final Left eye  Visual and final Right eye  Create image

When the left eye image and the right eye image are completed, the hole filling is completed, and then converted into Top-bottom, Side by Side, Vertical Interleaving, and Interlaced formats for transmission format of 3D display.

While the preferred embodiments of the present invention have been described and illustrated above using specific terms, such terms are used only for the purpose of clarifying the invention, and it is to be understood that the embodiment It will be obvious that various changes and modifications can be made without departing from the spirit and scope of the invention.

Claims (7)

delete delete delete In the method for converting a two-dimensional RGB image to a stereo image,
Converting the 2D RGB image into a 2D YUV image I (i, j);
Generating an edge map from the two-dimensional YUV image I (i, j);
A third step of generating an object depth map and a background depth map by using the edge map, and then integrating the object depth map and the background depth map to generate a final depth map;
A fourth step of obtaining a left eye image and a right eye image by moving the 2D RGB image in a horizontal direction by using the final depth map; And
And a fifth step of filling the empty hole pixels generated in the fourth step by using surrounding pixel values.
The second step calculates respective deviations of Y, U, and V of neighboring block pixels, and uses an edge map using any one selected from the mean value, the maximum value of the deviations, and the weighted product.

To generate a,
In the third step, the background depth map is generated by dividing the edge map based on a vertical center line, and then splitting the edge map by the vertical center line, and then adding up the sum of the intensity of the edge map with respect to the pixel located on the left side, and the vertical center line. The sum of the intensities of the edgemaps for the pixels located on the right side after being divided by, and the sum of the intensities of the edgemaps for the pixels located on the left side and the sum of the intensities of the edgemaps for the pixels located on the right side And generating a left and right background depth map using the difference.
In the method for converting a two-dimensional RGB image to a stereo image,
Converting the 2D RGB image into a 2D YUV image I (i, j);
Generating an edge map from the two-dimensional YUV image I (i, j);
A third step of generating an object depth map and a background depth map by using the edge map, and then integrating the object depth map and the background depth map to generate a final depth map;
A fourth step of obtaining a left eye image and a right eye image by moving the 2D RGB image in a horizontal direction by using the final depth map; And
And a fifth step of filling the empty hole pixels generated in the fourth step by using surrounding pixel values.
The second step calculates respective deviations of Y, U, and V of neighboring block pixels, and uses an edge map using any one selected from the mean value, the maximum value of the deviations, and the weighted product.

To generate a,
In the third step, the background depth map is generated by dividing an edge map with respect to a horizontal center line, and then splitting the edge map by the horizontal center line. Obtains the sum of the intensities of the edge maps for the pixels located below and is divided by the sum of the intensities of the edge maps for the pixels located above and the sum of the intensities of the edge maps for the pixels located below And generating a vertical background depth map by using a 2D image to a stereo image.
The method according to claim 4 or 5,
The amount of shifting the two-dimensional RGB image in the horizontal direction of the fourth step is determined as a product of the maximum stereo parallax and the function of the depth map is converted into a stereo image.
The method according to claim 4 or 5,
The fifth step is a method for converting a two-dimensional image to a stereo image, characterized in that performed using interpolation.

Images