KR20080059883A - Intermediate view reconstruction method in stereo image - Google Patents

Abstract

An IVR(Intermediate View Reconstruction) method in a stereo image is provided to process an occlusion region when reconstructing the intermediate view image effectively by using the IVR method to improve the quality of a reconstructed intermediate image. A motion predicting unit(100) obtains motion information by using a left view image and a right view image. A frame differential information generating unit(102) generates differential information about the left view image and the right view image. A region dividing unit(104) divides an image into four regions by using the motion information predicted in the motion predicting unit and the differential image information generated in the frame differential image generating unit. The four regions are divided into a stationary region, a moving region, an uncovered region, and a covered region.

Description

Intermediate viewpoint image synthesis method from stereo image {INTERMEDIATE VIEW RECONSTRUCTION METHOD IN STEREO IMAGE}

1 is an internal configuration diagram of a system for synthesizing an intermediate view image with respect to an input left view image and a right view image according to an embodiment of the present invention;

2 is a flowchart illustrating a process of preventing a blocking phenomenon when synthesizing an intermediate view image in a stereo image according to an embodiment of the present invention;

3 is an exemplary diagram illustrating left and right view images for explaining four areas according to an embodiment of the present invention;

4 is an exemplary view showing a middle view image synthesized from a left and a right view and each of a left and a right view according to an embodiment of the present invention;

FIG. 5 is an exemplary diagram for describing an occlusion region in a left and right view image according to an exemplary embodiment of the present invention. FIG.

The present invention relates to a method for preventing a blocking phenomenon occurring when synthesizing a mid-view image in a stereo image.

Recently, image technology has been actively researched for a method of realizing from 2D image to 3D image. Such a 3D image may express more realistic and realistic image information than a 2D image. By using the human visual characteristics, the left view image and the right view image are respectively scanned at the corresponding position on the existing display device, and the left view point and the right view point are separated from the user's left and right eyes to form a three-dimensional image. The method is recognized for its potential in many ways. As mentioned above, the initial model using the method of generating the 3D image has a limited number of realistic expressions by expressing the image at 2 viewpoints on the left and right images, and in order to overcome this, the 3D image is sensed by detecting the 3D image at various locations. Multi-view 3D image display devices are being introduced to further increase. In order to express a multi-view, it has all the information about the images corresponding to the minute change in the visual position of the observer, and only the image information about the left and right eye for the corresponding visual position is projected on the observer's retina. It is a way to feel the three-dimensional effect. However, such multi-view three-dimensional image generation requires an information input device such as a large number of cameras for capturing images corresponding to minute visual position changes, and requires a large channel capacity for this information transmission. In order to overcome this problem, a multiview 3D image display can be performed by using an intermediate view reconstruction (IVR) method of synthesizing an intermediate view image from an intermediate image synthesis. Such intermediate view image is obtained by uniformly sampling and transmitting image information corresponding to a certain point of time without obtaining or transmitting all image information corresponding to all necessary view points, The information may be generated by synthesizing an intermediate view image of the images of the two views using the transmitted images of the two views. Such a method has been widely used because it can generate a large amount of information required by only a small amount of information and a transmission channel.

One of the representative methods in the intermediate view image synthesis method is a method using motion prediction and motion compensation. Such a method first estimates a disparity between viewpoints of a viewpoint image and an adjacent viewpoint image. Thereafter, the intermediate image is synthesized as a whole by using a method of synthesizing a new image at a position corresponding to an intermediate disparity with respect to the estimated disparity. In this case, a block-based difference estimation method is used to estimate a disparity between images of two viewpoints.

This block-based disparity estimation (DE) is described, which is similar to block-based motion prediction. In motion prediction, an image that is a reference to two images that are input in time adjacent to each other is aread in a uniform block unit. We use the method to divide and find the motion disparity for each block in the vertical and horizontal directions in different images. In this case, a motion disparity between the reference image and the adjacent image is commonly called a motion vector. In the middle view image synthesis using the block-based difference estimation, the reference image is first divided into uniform blocks based on one of the left view image and the right view image. Thereafter, the most similar block is found in the opposite image using the reference block, and a new image is synthesized at a position corresponding to half the disparity of the corresponding disparity at that time.

Since the intermediate image synthesis method according to the prior art as described above is processed in units of blocks, blocking artifacts called mosaics in the synthesized intermediate image occur. Such blocking may cause deterioration of the image.

In addition, due to the characteristics of the stereo image, an occlusion area, which is an area existing only at one of two views, exists in the left view image and the right view image. Therefore, when synthesizing the intermediate view image using the intermediate view image synthesis method, the quality degradation of the synthesized intermediate image may be severe because the processing is performed in the same manner as the general region without exception processing for the occlusion area. have.

Accordingly, the present invention provides a method for improving the quality of the synthesized intermediate image by effectively processing the occlusion area generated when synthesizing the intermediate view image using the intermediate view image synthesis technique.

The present invention also provides a method for degrading an image due to a blocking phenomenon that occurs due to block unit processing when synthesizing an intermediate view image using the intermediate view image synthesis technique.

According to an embodiment of the present invention, a method for preventing a blocking phenomenon occurring when synthesizing an intermediate view image by using a left view image and a right view image, when the left view image and the right view image are input, the left view point Generating motion vector information through motion prediction using an image and a right view image, generating frame difference image information between the left view image and a right view image, and generating the motion vector information and frame difference image information. Partitioning the stationary region, the moving region, the uncovered region, and the covered region by using the method, and the region value of the intermediate frame for each of the divided regions. It characterized in that it comprises a process of generating and synthesizing the intermediate view image.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings. It should be noted that the same elements in the figures are represented by the same numerals wherever possible. In addition, detailed descriptions of well-known functions and configurations that may unnecessarily obscure the subject matter of the present invention will be omitted.

According to an embodiment of the present invention, a motion image and frame difference image information of a left view image and a right view image are used to cover an image of four regions: a stationary region, a moving region, an uncovered region, and a cover. This paper proposes a method of classifying a middle region into a covered region and synthesizing an intermediate image using a different processing method for each region.

First, a system for synthesizing an intermediate view image with respect to an input left view image and a right view image will be described with reference to FIG. 1. The system for synthesizing an intermediate view image according to the present invention may include a motion predictor 100, a frame difference image generator 102, an area divider 104, and an intermediate view image synthesizer 106. . Here, the motion predictor 100 obtains motion information by using a left view image and a right view image. Since a technique for obtaining motion information from each image may be obtained using a motion information extraction method in a general image processing technique, a detailed description thereof will be omitted. In addition, the frame difference image generation unit 102 generates a difference image for the left view image and the right view image. In addition, the area dividing unit 104 classifies an image into four regions by using the motion information predicted by the motion predicting unit 100 and the difference image information generated by the frame difference image generating unit 102. In this case, the four regions may be divided into a stationary region, a moving region, an uncovered region, and a covered region. Referring to FIG. 3, the four regions are the stationary region, which is a region in which both the left view image and the right view image are not moved, that is, there is no change in the image, and the moving region is a moving region. This is an area where there is movement. For example, in FIG. 3, regions 300, 301, 310, and 311 are motion regions. In addition, an uncovered region is an area in which a part where an object is hidden as a motion is shown, and a covered region is an area that is visible and hidden. For example, in FIG. 3, 320 and 330 are covered areas. That is, an uncovered region and an covered region are covered with an occlusion region. Referring to FIG. 5, referring to such occlusion regions, 500 and 502 may be occlusion regions. That is, since 500, which is an area in the left view image (a), is not included in the right view image (b), this area becomes a covered region, and 502, which is an area not included in the left view image (b), is a right view image. In (b), this area is an uncovered region.

A method of dividing into four regions as described above may be divided into regions using the following equations.

First, the definition of the variable used in the following equation will be described.

은 프레임 간 차영상의 (x,y)위치에서의 이진화 값으로, 차영상의 값이 존재하면 1, 존재하지 않으면 0을 나타낸다.

은 합성된 총 프레임 수이다. 또한,

는 i번째 합성된 프레임의 인덱스이고,

는 순방향, 역방향 수평, 수직 움직임 벡터를 나타낸다.

Is a binarization value at the (x, y) position of the difference image between frames, and indicates 1 if there is a difference image and 0 if not.

Is the total number of frames synthesized. Also,

Is the index of the i-th synthesized frame,

Denotes a forward, reverse horizontal and vertical motion vector.

Stationary region

It refers to an area where there is no movement, such as a background part, in the frame, and the area may be divided using Equation 1 below.

2. moving region

It refers to a region where there is a movement, and the region may be divided using Equations 2 and 3 below.

3. Uncovered region

Refers to an area where an object is hidden due to the movement of the object, and may be divided using Equation 4 and Equation 5 below.

4. covered region

Refers to an area that is hidden by moving the object, and can be divided using Equation 6 and Equation 7 below.

As described above, the motion vector information and the difference image information between the left view image and the right view image can be divided into four regions from each of the left view image and the right view image. As such, an example of dividing the left view image and the right view image into four regions may be illustrated in FIG. 4.

Referring to FIG. 4, since a in the left view image and A in the right view image are the same as the still region, the synthesized middle region is synthesized as one region in the same position. In addition, since the moving region is a b region in the left view image and the moving region is a C region in the right view image, the synthesized intermediate view image is synthesized into three regions which are intermediate positions. In addition, the c region, which is present in the left view image but not in the right view image, is synthesized into four regions in the synthesized intermediate view image, and the B region which is absent in the left view image but in the right view image is synthesized into two regions in the synthesized intermediate view image. do.

Meanwhile, the intermediate view image synthesis unit 106 synthesizes the intermediate images by using different methods for each of the four regions. This will be described in detail with reference to FIG. 5.

Referring to FIG. 5, an occlusion area, which is an area that exists only at one time point such as 500 and 502, exists in the left view image and the right view image due to the characteristics of the stereo image. The occlusion region 500 of the left-view image is shown as moving and occluded by using Equation 6 and Equation 7 for partitioning the covered region. can do. Also, the occlusion region 502, which is the occlusion region of the right-view image, is a region where the hidden part is moved and can be divided using Equations 4 and 5 for partitioning an uncovered region. have.

In the related art, since the intermediate images were synthesized by considering all four regions as having the same characteristics as described above, there was a problem of degrading the synthesized intermediate images such as having a blocking phenomenon. However, in the present invention, the image is divided into four regions having different characteristics as described above, and the intermediate image is synthesized by applying a different image synthesis method to each divided region so as to improve the synthesized image quality. do.

은 k번째 프레임의 (x,y) 위치에서의 화소 값이다.Next, a method of synthesizing the mid-view image by the mid-view image synthesizer 106 using another synthesis method preset for each region divided by the region divider 104 will be described. First off,

Is the pixel value at the (x, y) position of the kth frame.

Stationary region

Since there is no motion and the information on the area is in both the (k-n) th frame and the kth frame, it is generated using linear interpolation as shown in Equation (8).

2. moving region

Since the information about the region is in both the (k-n) th frame and the kth frame, and there is motion, it is generated by using bidirectional motion compensation as shown in Equation 9 below.

3. Uncovered region

Since information on the region exists only in the k-th frame, the information is generated through backward prediction as shown in Equation 10 below.

4. covered region

Since information on the region exists only in the (k-n) th frame, it is generated through forward prediction as shown in Equation 11 below.

In the above, a method of synthesizing one midpoint image has been described, but any of several intermediate images may be synthesized using the above method.

Next, a process of synthesizing an intermediate view image from a left view image and a right view image in the system configured as shown in FIG. 1 will be described with reference to FIGS. 1 and 2.

First, when the left view and right view images are input to the motion predictor 100 in step 202, the motion predictor 100 performs motion prediction using the left view and right view images in step 204. In this case, motion prediction generally uses a block matching algorithm that predicts motion on a block basis which is widely used in video compression standards. The block matching algorithm shown in FIG. 3 divides an input image into block units, compares a reference block with several blocks within a predetermined search range of a previous frame, finds the most similar block, and moves the position of the matched block with respect to the reference block. The technique is determined by. That is, the motion vector information may be derived by performing motion prediction by finding which block has moved to the predetermined block of the current frame in the search region.

In operation 206, the frame difference image generator 102 generates a frame difference between the left view image and the right view image.

In operation 208, the area divider 104 divides an image into four regions using motion vector information and frame difference image information derived through motion prediction. At this time, the division into four regions can be divided in the manner described by the region divider 104 in the above description of FIG. 1.

In operation 210, the intermediate view image synthesizer 106 generates a region value of an intermediate frame for each divided region and synthesizes the intermediate view image. In this case, the region value of the intermediate frame for each of the four divided regions may be generated by the method described by the intermediate view image synthesis unit 106 in the above description of FIG. 1.

As described above, the present invention classifies an image into four regions (stationary, moving, uncovered, and covered regions) by using motion information and frame difference image information of a left view image and a right view image, respectively, for each region. The intermediate view image is synthesized using the processing method according to the region. As a result, the quality of the synthesized intermediate image may be further improved by effectively processing a masked area that exists due to the characteristics of the stereo image. In addition, due to the pixel-by-pixel processing, there is an advantage that the quality of the image due to the blocking phenomenon caused by the conventional block-based processing can be avoided.

Claims (10)

In the method for preventing the blocking phenomenon occurring when synthesizing the intermediate view image using the left view image and the right view image, When the left view image and the right view image are input, generating motion vector information through motion prediction using the left view image and the right view image; Generating frame difference image information between the left view image and the right view image; Dividing into a stationary region, a moving region, an uncovered region, and a covered region by using the motion vector information and the frame difference image information; And synthesizing an intermediate view image by generating a region value of an intermediate frame for each of the divided regions. The method of claim 1, The stationary region is a region in which both the left view image and the right view image are motionless, the moving region is a motion region, and the uncovered region is hidden as an object moves. A method of synthesizing a mid-view image in a stereo image, wherein the portion where the portion is displayed is a region where the covered region is a visible region. The method of claim 2, The stationary region is a method of synthesizing an intermediate view image in a stereo image, wherein the region may be divided using Equation 1 below. [Equation 1]

here,

Is the binarization value at the (x, y) position of the difference image between frames, and indicates 1 if there is a difference image and 0 if not. The method of claim 2, The moving region is a method of synthesizing an intermediate view image in a stereo image, wherein the region may be divided using Equations 2 and 3 below. [Equation 2]

[Equation 3]

here,

Is a binarization value at the (x, y) position of the difference image between frames, and indicates 1 if there is a difference image and 0 if not.

Is the total number of frames synthesized,

Is the index of the i-th synthesized frame,

Represents the forward, reverse horizontal and vertical motion vectors. The method of claim 2, The uncovered region may be divided into regions using Equation 4 and Equation 5 below. [Equation 4]

[Equation 5]

here,

Is a binarization value at the (x, y) position of the difference image between frames, and indicates 1 if there is a difference image and 0 if not.

Is the total number of frames synthesized,

Is the index of the i-th synthesized frame,

Represents the forward, reverse horizontal and vertical motion vectors. The method of claim 2, The covered region is a method of synthesizing a mid-view image in a stereo image, wherein the region may be divided using Equations 6 and 7 below. [Equation 6]

[Equation 7]

here,

Is a binarization value at the (x, y) position of the difference image between frames, and indicates 1 if there is a difference image and 0 if not.

Is the total number of frames synthesized,

Is the index of the i-th synthesized frame,

Represents the forward, reverse horizontal and vertical motion vectors. The method of claim 2, A method for synthesizing an intermediate view image in a stereo image for the stationary region by generating an intermediate view image using linear interpolation as shown in Equation 8 below. [Equation 8]

At this time,

Is the pixel value at the (x, y) position of the kth frame. The method of claim 2, A method of synthesizing an intermediate view image in a stereo image with respect to the moving region by generating an intermediate view image using bidirectional motion compensation as shown in Equation 9 below. [Equation 9]

At this time,

Is the pixel value at the (x, y) position of the kth frame. The method of claim 2, The uncovered region is generated by performing backward prediction with Equation 10 below. [Equation 10]

At this time,

Is the pixel value at the (x, y) position of the kth frame. The method of claim 2, A method for synthesizing a mid-view image in a stereo image, wherein the covered region is generated through Equation 11 and forward prediction. [Equation 11]

At this time,

Is the pixel value at the (x, y) position of the kth frame.

Images