KR20030076904A - Method for Reconstructing Intermediate View Image using Adaptive Disparity Estimation - Google Patents

Abstract

PURPOSE: An intermediate view reconstruction method is provided to adaptively select the size of a matching window according to the size of a characteristic value, and to prevent the mis-matching or blocking effect. CONSTITUTION: An intermediate view reconstruction method includes the steps of detecting a block corresponding to a stereo input image by calculating a mean square error between two blocks(S100), extracting the corresponding block and a characteristic value of the image after calculating a transition value(S102), selecting a matching window to matching the transition on the basis of the extracted characteristic value(S104), and carrying out the intermediate view reconstruction by the selected matching window by the interpolation with a weighed mean value(S106).

Description

Method for Reconstructing Intermediate View Image using Adaptive Disparity Estimation}

The present invention relates to a method for synthesizing an intermediate view image, and more particularly, to a method for synthesizing a bowler image at a position of an intermediate intermediate point by adaptively selecting a matching window at the time of image registration according to a feature value. .

In general, a person can see an object three-dimensionally by binocular disparity. The two eyes observe images from different viewpoints, and the human brain synthesizes the difference between the two stereo images to recognize the three-dimensional solid object. do. In the meantime, various types of binocular three-dimensional display systems have been implemented by imitating the human visual system (HVS). However, since the stereoscopic binocular system is limited to two eyes, the practical application is limited because the observer cannot feel three-dimensional feeling when the viewer is out of focus or is out of focus, and also feels tired and dizzy eyes. Accordingly, active researches on various types of multi-view stereo 3D display systems have been conducted as approaches to solve the disadvantages of the conventional stereo system. Since the system acquires and displays multi-view images through a multi-view stereoscopic camera, the field of view is enlarged as the number of viewpoints increases and more natural 3D display is possible. However, as the number of viewpoints increases, the image data of a multi-view also increases simultaneously, which requires a real-time image processing processor and a very high speed and wideband transmission channel. Recently, an intermediate view reconstruction (IVR) technique for digitally synthesizing a plurality of arbitrary view stereo images using only a limited number of stereo images has been researched and developed as a new approach to solve this problem. . Since this mid-view image synthesis technique digitally synthesizes arbitrary multi-view stereo images from limited stereo images, it presents the possibility of realizing a natural three-dimensional stereoscopic display with expanded viewing area while solving problems of the existing three-dimensional display system. Doing.

In general, matching methods such as pixel-based, block-based, and feature-based are used as the disparity matching method in the intermediate image synthesis process. Among these, the pixel-based matching method allows fine matching but has disadvantages such as high probability of mismatching and long execution time in the matching process. In addition, in the case of the block-based matching method, the execution time is short, but fine matching is difficult, and there are disadvantages such as a blocking effect. The feature-based matching method is capable of efficient matching according to the characteristics of the input image, but has disadvantages such as incorrect dislocation allocation in the similar region having no feature value.

In the present invention, in order to solve the problems of the prior art as described above, it is proposed a method of synthesizing an intermediate view point image that can solve the misalignment and blocking phenomenon.

Another object of the present invention is to propose a method of synthesizing an intermediate viewpoint image by adaptively selecting a matching window according to a feature value.

1 is a flow chart showing the overall flow of the intermediate view image synthesis method according to a preferred embodiment of the present invention.

2 is a flowchart illustrating a process of detecting a feature value by detecting a corresponding block according to a preferred embodiment of the present invention.

3 is a flow chart showing a process for selecting a registration window according to a preferred embodiment of the present invention.

4 is a flowchart illustrating a process of synthesizing an intermediate view image using the selected registration window.

5 is a conceptual diagram for synthesizing an intermediate view image.

6 is a block search diagram illustrating a process of detecting a corresponding block of a right image based on a left image.

7 shows a test image used in a simulation.

In FIG. 8, (a) shows a disparity map of the test image when the pixel-based matching method is used, and (b) shows a disparity map of the test image when the pixel and feature-based matching method is used. c) shows the variation map of the test image when using the present invention.

In FIG. 9, (a) shows a mid-view image synthesis result by the pixel-based matching method, (b) shows a mid-view image synthesis result by the pixel-based and feature-based matching method, and (c) shows the present invention. The results of the intermediate view image synthesis according to.

In FIG. 10, (a) shows an error image with the original image when the pixel-based matching method is used, and (b) shows an error image with the original image when the pixel-based and feature-based matching method is used. (c) shows an error image from the original image when using the present invention

In order to achieve the above object, the intermediate view image synthesis method according to the present invention comprises the steps of (a) detecting a corresponding block of the stereo input image and calculating the transition between the images; (B) extracting feature values of the stereo image; (C) selecting a matching window according to the feature value extracted in the step (b) among a plurality of preset matching windows; And synthesizing an intermediate view image by the selected matching window.

The step (a) may include detecting a corresponding block of the right image based on the left image; Detecting a corresponding block of the left image based on the right image; The method may include calculating a variation using a difference in coordinate values between the right image and the left image.

The feature value of step (b) may be extracted using contour information by the Canny mask.

The registration window may be selected by the magnitude of the feature value.

In step (d), the intermediate view image synthesis may be performed by an interpolation method using a weighted average value.

Meanwhile, the present invention may further include a variation stabilization step of obtaining the average of the peripheral variation values for the region to which the variation is not assigned and replacing the variation value with the average value.

The detection of the corresponding block may be detected by calculating a mean square error (MSE) between two blocks.

Hereinafter, exemplary embodiments of a mid-view image synthesis method according to the present invention will be described in detail with reference to the accompanying drawings.

1 is a flowchart showing the overall flow of the intermediate view image synthesis method according to a preferred embodiment of the present invention.

As shown in FIG. 1, first, a corresponding block of a stereo input image is detected (S100). In general, the stereo input image is divided into a left image and a right image, and detects which part of the right image corresponds to a specific block of the left image.

According to a preferred embodiment of the present invention, it is preferable to use a bidirectional matching scheme when detecting a corresponding block.

After detecting the corresponding block, the variation value representing the block can be calculated.

After detecting the corresponding block and calculating the shift value, the feature value of the stereo input image is extracted (S102).

After the feature value is extracted, a matching window for matching the variation is selected based on the extracted feature value (S104). In the present invention, a constant matching window is defined to match the variation by adaptively selecting the matching window according to the magnitude of the feature value without matching the variation.

That is, in the present invention, it is necessary to distinguish between the case where fine matching is required and the case where it is not, to perform efficient matching.

After selecting the matching window, an image of an intermediate view is synthesized using the selected matching window (S106). According to a preferred embodiment of the present invention, it is preferable to synthesize an intermediate view image by an interpolation with weighed mean value for more natural matching.

2 is a flowchart illustrating a process of detecting a feature value by detecting a corresponding block according to a preferred embodiment of the present invention.

As shown in FIG. 2, first, a corresponding block of a right image is detected based on a left image among stereo input images (S200).

6 is a block search diagram illustrating a process of detecting a corresponding block of a right image based on a left image, and FIG. 6A shows a left image and FIG. 6B shows a right image.

As shown in FIG. 6, the left image is divided into blocks having the size of N _x and N _y horizontally and vertically, and a block existing on the same epipolar line in the right image is detected for any block.

In FIG. 6, S refers to a range searched for detecting a corresponding block in the right image. Since the left image and the right image are very similar, the block of the right image corresponding to a specific block of the left image differs only slightly from the left image block and the x coordinate value. Therefore, the search time can be reduced by presetting the search range based on the left image block. have.

S may be expressed as Equation 1 below.

S = (R (x, y) | (x _o -S _x = x <= x _o + S _x + N _x ), (y _o <= y <= y _o + N _y )}

In general, in the cognitive visual system, the vertical shift has less effect than the horizontal shift. Therefore, when the stereo cameras are configured in parallel in consideration of the characteristics of the human visual system, only the horizontal shift exists, so that the disparity vector can be found as the horizontal component as shown in FIG.

According to a preferred embodiment of the present invention, a MSE (Mean Square Root) is calculated to determine whether the found right image block corresponds to the left image block.

Equation 2 below shows an equation for calculating the MSE.

In Equation 2, N _x , N _y represent the size of the block, and I _L and I _R represent the right image block corresponding to the left image block.

That is, the block having the smallest MSE value within the search range is determined as the block of the right image corresponding to the left image using the equation (2).

After detecting the corresponding block of the right image based on the left image, the corresponding block of the left image is detected based on the right image (S202).

Since the process of detecting the left image block corresponding to the right image block is the same as the process of detecting the right image block corresponding to the left image block, a detailed description thereof will be omitted.

As described above, searching for the corresponding block twice is because it is difficult to find the exact matching point when the corresponding block is found based on only one of the left image and the right image.

After detecting the corresponding block, a shift value between the left image and the right image is calculated (S204).

The method of calculating the variation value is shown in Equation 3 below.

In Equation 3, I _cur is an image to be predicted, I _ref is a reference image, Represents the variation within the search range, and beta_i is the i th Represents a block of size.

Equation 3 is a calculation of the variation of the right image on the basis of the left image, on the contrary, it will be apparent to those skilled in the art that a value calculated from the variation of the left image on the basis of the right image can be used.

Disparity vectors obtained in Equation 3 are used to create a disparity map with values representing each block.

After calculating the variation, edge information is detected to extract feature values (S206). According to a preferred embodiment of the present invention, it is preferable to detect edge information using a Canny mask, which is a contour detection differential operator which is relatively less noise sensitive.

After detecting edge information, the feature value is extracted using the edge information (S208). According to a preferred embodiment of the present invention, the feature value is preferably extracted in units of pixels.

3 is a flowchart illustrating a process of selecting a matching window according to a preferred embodiment of the present invention.

As shown in FIG. 3, first, feature values extracted from a stereo input image are normalized (S300). According to a preferred embodiment of the present invention, the feature value is preferably calculated in units of pixels, and is preferably normalized to a value between 0 and 1.

In the present embodiment, a case in which five registration windows are selected by setting a threshold value in five steps will be described. However, the present invention is not limited to the case where the threshold value and the matching window are five, and it will be apparent to those skilled in the art that a change in the number of matching windows and the size of the threshold value falls within the scope of the present invention.

In the present embodiment, the threshold is set to 0.1, 0.3, 0.5, 0.7, 0.9, and the corresponding five size matching window is 16 16, 8 8, 4 4, 2 2, 1 Set it to 1 where 32 Since the matching efficiency is lower in the matching window larger than 32, the size of the largest matching window among the five matching windows is It is preferable to limit the setting to 16.

When the threshold value and the matching window are set as described above, a process of selecting the matching window according to the size of the extracted feature value is performed.

As shown in FIG. 3, first, it is determined whether the feature value is 0.9 or more, which is the threshold value that is set to be the largest (S302).

If the feature value is 0.9 or more, the area often has complex contours. 1 matching window is selected to perform the finest matching (S304).

If the feature value is not 0.9 or more, it is determined whether or not the second smallest threshold value is 0.7 or more (S306).

1 for feature values greater than or equal to 0.7 1 the second largest matching window 2 matching window is selected to perform a relatively fine matching (S308).

If the feature value is not 0.7 or more, it is determined whether or not the third smallest threshold value is 0.5 or more (S310).

2 for feature values greater than 0,5 2 the next matching window 4 4 size matching window is selected (S312).

If the feature value is not 0.5 or more, it is determined whether the threshold value is 0.3 or more, which is next to 0.5 (S314).

4 for feature values greater than or equal to 0.3 4 Matching window of size 8 8 registration window is selected (S316).

If the feature value is not 0.3 or more, it is determined whether the smallest threshold value is 0, 1 or more (S318). 16 registration window is selected (S318).

As described above, according to the present invention, the matching window is adaptively selected while the threshold value decreases and increases from step 5 to step 1. Therefore, a region having a small feature value, such as a background, is determined to be a similar area, and a matching is performed by selecting a matching window of a large unit, and fine matching of a pixel unit is performed for a portion having a large feature value such as an outline of an object. This reduces not only mismatches that occur frequently in small blocks but also reduces the blockage phenomenon that occurs when matching blocks.

4 is a flowchart illustrating a process of synthesizing an intermediate view image using the selected registration window, and FIG. 5 illustrates a conceptual diagram for synthesizing the intermediate view image.

In FIG. 5, when synthesizing an intermediate view image from a stereo input image, the position of the view point is represented by alpha, and alpha is set to a value between 0 and 1. FIG. In FIG. 5, an image having alpha = 0 represents a left image and an image having alpha = 1 represents a right image based on the left image.

There are two methods of synthesizing an intermediate view image, an extrapolation method and an interpolation method. FIG. 5 illustrates an interpolation method.

Referring to FIG. 4, in order to synthesize an intermediate view image, first, a view to synthesize an image is selected (S400). That is, the process of selecting the alpha value in FIG.

If a viewpoint is selected, an adaptive matching operation is performed using the matching window selected according to the feature value with respect to the selected viewpoint (S402). According to a preferred embodiment of the present invention, the matching is preferably performed by an interpolation method using a weighted average value for more natural matching.

Equation 4 below shows the case of interpolation with a weighted average value according to the position (alpha) of the viewpoint.

In general, when synthesizing an intermediate view image, an area covered with each other exists in one of a left image and a right image, which are stereo pairs. The mutation stabilization process is performed on the hidden region (S404). According to a preferred embodiment of the present invention, the average value of the peripheral variation values is obtained and the variation value is used as the variation information of the hidden region.

Hereinafter, a simulation experiment comparing the mid-view image synthesis method according to the present invention with the conventional pixel-based matching method and the block-based matching method will be described.

The test image used in the simulation is shown in FIG.

In the simulation, the feature values for the input image were normalized and the thresholds were set in five stages of 0.1, 0.3, 0.5, 0.7, and 0.9. In the initial condition, the threshold was set to 5 steps (0.9). If the condition is not satisfied, the threshold was reduced by 1 step. The initial threshold is variable and can be set arbitrarily, but in the simulation we set the initial threshold to all 5. In addition, the size of the corresponding matching window is 16 16, 8 8, 4 4, 2 2, 1 1 was set in 5 steps.

In FIG. 8, (a) shows a disparity map of the test image when the pixel-based matching method is used, and (b) shows a disparity map of the test image when the pixel and feature-based matching method is used. c) shows a variation map of the test image when the present invention is used.

Although the pixel-based method of (a) performs precise matching, it can be seen that the exact matching is not performed at a portion such as an outline of an object. In the method considering the pixel-based and feature-based of FIG. (B), the contour of an object is extracted more accurately than the pixel-based method of FIG. (A), but in the method according to the present invention, a more stable disparity vector appears in the background part. It can be seen.

In FIG. 9, (a) shows a mid-view image synthesis result by the pixel-based matching method, (b) shows a mid-view image synthesis result by the pixel-based and feature-based matching method, and (c) shows the present invention. The results of the intermediate view image synthesis according to the FIG.

In FIG. 9, the pixel-based matching method enables precise representation of the image, but the matching efficiency decreases due to excessive precision matching even in a part such as a background where there is little change, and the contour portion of the object cannot be detected in the contour portion of the object. You can see that mismatch appears in. In FIG. 9B, the pixel- and feature-based matching method performs a better synthesis of the contour portion of the object than the pixel-based method. Looking at the magnified image, it can be seen that the synthesized image according to the present invention is more accurately matched than the conventional method.

In FIG. 10, (a) shows an error image with the original image when the pixel-based matching method is used, and (b) shows an error image with the original image when the pixel-based and feature-based matching method is used. (c) shows an error image from the original image when using the present invention.

Referring to FIG. 10, it can be seen that the difference between the original image and the error image is the smallest according to the method according to the present invention.

As described above, according to the intermediate view image synthesis method according to the present invention, since the size of the matching window is adaptively selected according to the size of the feature value, a relatively large matching window is generated in a region having a small feature value such as a background. It is possible to perform block matching and fine matching by relatively small matching window in areas with large feature values, such as outlines of objects, to improve the overall matching performance, and to solve the conventional problem of mismatching or blocking. There is this.

Claims (8)

In the method for synthesizing the mid-view image, Detecting a corresponding block of the stereo input image and calculating a transition between the images; (B) extracting feature values of the stereo image; (C) selecting a matching window according to the feature value extracted in the step (b) among a plurality of preset matching windows; And And synthesizing an intermediate view image by the selected matching window. The method of claim 1, Step (a) is, Detecting a corresponding block of the right image based on the left image; Detecting a corresponding block of the left image based on the right image; Comprising the step of calculating the variation using the coordinate value difference between the right image and the left image. The method of claim 1, The feature value of step (b) is extracted by using the edge (Edge) information by the Canny mask. The method of claim 1, And the matching window is selected by a magnitude of a feature value. The method of claim 1, The step (d) the intermediate view image synthesis method is characterized in that by the interpolation method by a weighted average value. The method of claim 1, The method of claim 1, further comprising a step of stabilizing the average value of the surrounding variation values for the region to which the variation is not assigned. The method of claim 2, And detecting the corresponding block by calculating a mean square error (MSE) between two blocks. A program which is tangibly embodied in a program that can be executed by a digital processing apparatus to perform the intermediate view image synthesizing method according to claim 1, and which can be read by the digital processing apparatus.