KR102386002B1 - Method and apparatus for estimating disparity in element image array - Google Patents

Abstract

According to the present invention, a method for estimating disparity in an element image array includes: extracting a reference element image and peripheral element images located on a disparity estimation path from an input element image array; defining a first vector (u) based on pixel coordinates (r_0, c_0) corresponding to a reference position of the disparity estimation path in the reference element image; defining a second vector (v_n) based on pixel coordinates located on the disparity estimation path in each of the peripheral element images; calculating position index values (n values) representing optimal matching positions on the disparity estimation path in a process of matching the first vector and the second vector; configuring the calculated position index values (n values) as a disparity candidate set; and estimating a final disparity value among the position index values based on a frequency and distribution of the position index values included in the disparity candidate set. Accordingly, an amount of calculation and memory usage are reduced.

Description

Disparity estimation method and apparatus in element image array {Method and apparatus for estimating disparity in element image array}

The present invention relates to integral imaging technology, and more particularly, to a technology for estimating disparity in an elemental image array.

Integrated imaging technology is a technology for imaging and visualizing three-dimensional images. Compared with stereoscopy and holography technology, this technology provides full parallax and continuous viewpoints using white light without additional optical devices (Head Mounted Display, HMD). provides

Such integrated imaging technology is of great interest in various application fields, for example, 3D object visualization, 3D object recognition, driverless car system, and 3D entertainment. are receiving

1 is a view for explaining a pickup process and a restoration process performed in a conventional integrated imaging system.

Referring to FIG. 1 , the integrated imaging system performs a pickup process and a restoration (reconstruction) process.

First, in the pickup step, the light ray from the three-dimensional object (10, 3D object) is picked up by the lens array (20, Lenslet Array) is captured by the pickup device. Although the pickup device is not shown in FIG. 1 , it may be a CCD camera. In the pickup method, there is also a pickup method of a camera arrangement method instead of a lens arrangement. The images of various viewing angles obtained from the pickup device have a two-dimensional array structure, which is called an Elemental Image Array (EIA).

In the restoration step, a 3D image is restored (reconstructed) optically or digitally. The restoration methods include Optical integral imaging (OII) and computational integral imaging (CII).

In optical integral imaging (OII), the EIA is displayed on an optical display panel, and then a 3D object is observed through a lens array.

Conversely, in Computational Integral Imaging (CII), a 3D image is reconstructed by digital technology based on the EIA and the virtual pinholes array 40 . This reconstruction technique is called computational integral imaging reconstruction (CIIR).

Unlike OII, CIIR technology creates a view image regardless of the physical limitations of optical devices. Since the computer-reconstructed image can be used in application fields such as object recognition and depth estimation, CIIR technology is very practical.

The most common CIIR technique is back-projection. The back-projection method is a method in which elemental images are magnified while passing through the virtual pinhole array 40 , and these elemental images are overlapped with each other in the reconstructed image plane 50 as shown in FIG. 1 .

CIIR studies for image quality improvement have been actively discussed until recently, and among them, the pixel mapping method is a method of ray-tracing the location where each pixel of the element image reaches through the pinhole 40 and mapping it to the reconstructed image plane 50 . This is the pixel mapping method. This method reduces the amount of computation and improves image quality.

The window method interprets and defines the signal model of the reverse projection method, and improves the image quality of the reconstructed image by removing the lens phenomenon and artifacts based on this.

Finally, it is a method of applying the convolution characteristic to the component images based on the relationship between the periodic function and the component images. In this method, there is a method of adjusting the depth resolution and improving the resolution of the reconstructed image.

Analysis of existing methods for disparity extraction from elemental images

In order to extract 3D information from the element image array obtained from the lens array or the camera array, it is the most essential and basic to first extract the disparity between the element images. In particular, extracting the disparity between the upper, lower, left, and right elemental images corresponds to a core technology in the elemental image arrangement.

A known method for extracting disparity in an integrated image is as follows.

First, there is a method using stereo matching. In general, stereo matching is a method specialized for stereo images by extracting disparity between two images. But for an elemental image array,

In order to apply the stereo method, it is necessary to apply an independent stereo matching technology between the element images on the top, bottom, left, and right of the central component image, and a post-processing operation that requires processing a plurality of disparity information and extracting one disparity information from it again I need this. In addition, in the case of a component image obtained from a lens array, since the individual size of the component image is small, it is often not possible to apply a stereo image matching technique.

As another known method of extracting disparity in an integrated image, a computational integral imaging reconstruction (CIIR) technique may be utilized. This technology first acquires a three-dimensional volume image through the CIIR technology, which is a technology for reconstructing a three-dimensional volume image from an element image array, and estimates the location by searching the object image to be found in the three-dimensional space based on the three-dimensional volume image. way to do it In an elemental image array with a large number of elemental images, this method can be effective. It is especially effective when the size of each individual element image is small.

However, the number of component images may be small or may be affected by noise remaining in the 3D volume image obtained by the CIIR technique. In addition, since CIIR must be performed first, it has the disadvantage of requiring a considerable amount of computation and system memory usage.

Another method is to utilize an epipolar image (EPI). EPI is an image reconstructed by collecting epipolar lines used in stereo image matching.

There is only one isopolar for a point in two images, but when stereo images are matched from multiple images like element image array, multiple isopolars exist, and an image is obtained by simply collecting and reconstructing them. This image is called EPI ascending image.

Another type of observed straight line pattern angle in the ascending image is directly related to disparity. Using this, disparity can be estimated.

The field of using ascending images is the latest technology mainly used in light field images. The light field image obtained from the microlens array has an elemental image array form, and disparity through angle estimation in an isopolar image can utilize multiple elemental images, so the estimation efficiency is much superior to stereo image matching.

However, the lie pattern angle estimation in the isopolar image can be efficiently applied when multiple element images observe the same object. That is, when a small number of elemental images are used, it becomes difficult to reconstruct the polar image itself. In addition, there is a problem in that the memory usage is considerable in the preprocessing operation for reconstructing the ascending image.

SUMMARY OF THE INVENTION It is an object of the present invention to solve the above problems, to provide a method and apparatus for estimating disparity in an element image array capable of reducing the amount of computation and memory usage.

The above and other objects, advantages and features of the present invention, and a method for achieving them will become apparent with reference to the embodiments described below in detail in conjunction with the accompanying drawings.

A disparity estimation method according to an aspect of the present invention for achieving the above object includes extracting a reference element image and neighboring element images located on a disparity estimation path from an input element image array; defining a first vector (u) based on pixel coordinates (r ₀ , c ₀ ) corresponding to a reference position of the disparity estimation path in the reference element image; defining a second vector (v _n ) based on the coordinates of each pixel located on the disparity estimation path in each neighboring element image; calculating position index values (n values) indicating an optimal matching position on the disparity estimation path in a process of matching the first vector and the second vector; composing the calculated position index values (n values) into a disparity candidate set; and estimating a final disparity value from among the location index values based on the frequency and distribution of location index values included in the disparity candidate set.

A computing device for disparity estimation according to another aspect of the present invention includes: an extraction module for extracting a reference element image and peripheral element images located on a disparity estimation path from an input element image array; a first vector generating module that generates a first vector (u) based on pixel coordinates (r ₀ , c ₀ ) corresponding to a reference position of the disparity estimation path in the reference element image; a second vector generating module for generating a second vector (v _n ) based on pixel coordinates located on the disparity estimation path in each neighboring element image; a vector matching module for calculating position index values (n values) indicating an optimal matching position on the disparity estimation path in a process of matching the first vector and the second vector; a candidate collection module that configures the calculated position index values (n values) into a disparity candidate set; and a disparity extraction module for extracting a final disparity value from among the location index values composed of the disparity candidate set based on the frequency and distribution of the location index values.

The disparity estimation method in the element image arrangement of the present invention can reduce the amount of computation and memory usage.

1 is a view for explaining a pickup process and a restoration process performed in a general integrated imaging system.
2 is a block diagram schematically illustrating an internal configuration of an apparatus for estimating disparity in an element image arrangement according to an embodiment of the present invention.
3 is a view for explaining a process of extracting an element image array (EIA) of a 3x3 unit from the element image array (EIA) according to an embodiment of the present invention.
4 is a diagram for explaining a disparity estimation path in a 2×2 array pattern and neighboring element images selected in this case according to an embodiment of the present invention.
5 is a diagram for explaining a disparity estimation path in a 2×3 array pattern or a 3×2 array pattern and neighboring element images selected in this case, according to another embodiment of the present invention.
6 is a diagram for explaining a disparity estimation path in a 3×3 array pattern and neighboring element images selected in this case according to another embodiment of the present invention.

The aspect in which the present invention is implemented will be described with reference to each preferred embodiment below. It is apparent that the present invention is not limited to the embodiments disclosed below, but may be implemented in other various forms within the scope of the technical spirit of the present invention. The terminology used herein is also intended to describe the embodiments and is not intended to limit the present invention. In this specification, the singular also includes the plural unless specifically stated otherwise in the phrase. As used herein, “comprise” and/or “comprising” means that the stated component, step, action and/or element is the presence of one or more other components, steps, actions and/or elements. or added.

2 is a block diagram schematically illustrating an internal configuration of a computing device for implementing a method for estimating disparity in an element image arrangement according to an embodiment of the present invention.

Referring to FIG. 2 , the computing device may include a disparity estimation module 100 , a processor 200 , a memory 300 , and a system bus 500 connecting them, but is not limited thereto, and in general, computing It may be configured to further include general components included in the device, for example, a user interface, a display, a network interface, and the like.

The disparity estimation module 100 performs a number of processes for estimating disparity in the element image arrangement. The disparity estimation module 100 may be implemented as a software module (eg, an algorithm programmed with a program code), a hardware module, or a combination thereof.

The processor 200 controls the operation of the disparity estimation module 100 through the system bus 500 and processes instructions (instructions for estimating disparity) stored in the memory 300 . The processor 200 may include, for example, at least one CPU and at least one GPU.

Although the figure shows the disparity estimation module 100 and the processor 200 in a separate form, they may be integrated into one module, for example, one IC chip.

Hereinafter, the disparity estimation module 100 will be described in detail.

The disparity estimation module 100 includes a plurality of modules that can be classified according to a processing process for estimating disparity in the element image arrangement, for example, the extraction module 110 and the first vector generation module 120 . , a second vector generation module 130 , a vector matching module 140 , a candidate collection module 150 , and a disparity extraction module 160 .

The extraction module 110 extracts an elemental image array (EIA) of M×N (eg, 3×3) units from an input elemental image array (EIA), and an elemental image array of M×N units ( EIA), a reference element image (X: Reference EI) and neighboring element images (S: Neighbor EIs) are extracted, the extracted reference element image (X) is input to the first vector generation module 120, and the extracted The peripheral element images S are input to the second vector generation module 130 .

For extraction of the peripheral element images (S), the extraction module 110 sets the pixel coordinates (r ₀ , c ₀ ) of the reference element image (X). When the pixel coordinates (r ₀ , c ₀ ) of the reference element image (X) are set (set), the peripheral element images (S) according to the pixel coordinates (r ₀ , c ₀ ) of the set reference element image (X) ) is selected and extracted. Here, the selection of the neighboring element images S is determined according to the disparity estimation path starting from the pixel coordinates (r ₀ , c ₀ ). Here, the disparity estimation path includes three array patterns (eg, 2×2 array pattern, 2×3 (or 3×2) array pattern and 3×3 array pattern). A minimum of 3 to a maximum of 8 peripheral element images may be selected and extracted for each of the three arrangement patterns.

The first vector generation module 120 generates a first vector u by expressing pixel values of pixels included in the reference element image X inputted from the extraction module 110 in a one-dimensional vector form, and input to the matching module 140 . A first vector ( u ) generated from the reference element image (X) is used as a reference vector.

)로서 생성하고, 이를 벡터 정합 모듈(140)로 입력한다. 주변 요소 영상들(S)로부터 생성된 제2 벡터(

)는 n의 위치에 따라 다수의 벡터들을 포함하도록 구성된다.The second vector generation module 130 expresses the pixel values of pixels included in the peripheral element images S input from the extraction module 110 in the form of a one-dimensional vector to form a second vector (

), and input it to the vector matching module 140 . A second vector (

) is configured to contain multiple vectors according to the position of n .

) 간의 벡터 정합을 통해 최적의 정합 위치를 나타내는 n값(위치 인덱스 값)을 계산한다. n값은 주변 요소 영상들의 개수만큼 획득된다.The vector matching module 140 includes a first vector ( u ) inputted from the first vector generation module 120 and a second vector ( u ) inputted from the second vector generation module 130 .

) to calculate the value of n (position index value) representing the optimal matching position through vector registration. The n value is obtained as many as the number of neighboring element images.

The candidate collection module 150 collects n values input from the vector matching module 140 , and configures the collected n values as a disparity candidate set for pixel coordinates of pixels in the reference element image X.

The disparity extraction module 160 analyzes the frequency and/or distribution of n values (location index values) included in the disparity candidate set input from the candidate collection module 150 to obtain the highest frequency and/or the highest distribution. An n value (position index value) corresponding to the indicated mode is extracted as the final disparity.

Hereinafter, a processing process of each module included in the disparity estimation module 100 will be described in more detail.

Extraction module (110)

The extraction module 110 extracts the reference element image (X) and the peripheral element images (S) from the input element image array (EIA).

The element image array EIA input to the extraction module 110 may be obtained through a two-dimensional lens array or a two-dimensional camera array. The two-dimensional lens array may have a structure of a wide variety of patterns according to optical structures and characteristics.

The two-dimensional lens arrangement may be, for example, a two-dimensional arrangement of very small micro lenses used in a light field camera. In addition, the two-dimensional lens arrangement may be a two-dimensional arrangement of commonly used lenses having a size of several centimeters, such as spectacle lenses. In addition, the two-dimensional lens arrangement may be a two-dimensional arrangement of large-scale metric lenses.

In this embodiment, a method for estimating disparity in an elemental image array with respect to a 3x3 array, which is a lens array that can be seen as the most basic structure in a two-dimensionally arranged elemental image array (EIA), is presented. Therefore, the extraction module 110, prior to extracting the reference element image (X) and the surrounding element images (S) from the input element image array (EIA), a 3x3 array (unit) from the element image array (EIA) It precedes the process of extracting the element image array (EIA).

FIG. 3 is a diagram for explaining a process of extracting a 3x3 array (unit) of an elemental image array (EIA) from an elemental image array (EIA) according to an embodiment of the present invention.

Referring to FIG. 3 , the core structure of the 2D array element images obtained from various types of optical structures is a 3×3 structure. The 3×3 element image array (EIA) has a structure in which disparity can be obtained in diagonal directions.

Extracting an element image array (EIA) of a 3×3 array from an element image array (EIA) is based on the element image (X _ij ) corresponding to an arbitrary position (ij) of the surrounding element images (Snm: S _i- It is equivalent to extracting _1j-1 , S _i-1j , S _i-1j+1 , S _ij-1 , S _ij+1 , S _i+1j-1 , S _i+1j , S _i+1j+! ). That is, the present invention estimates the disparity from the reference element image (X _ij ) and eight 8 neighboring element images. Here, the peripheral element images S used (or selected) for disparity estimation may be determined according to three arrangement patterns that can be configured in the 3×3 element image array EIA. Here, the three arrangement patterns may include, for example, a 2x2 arrangement pattern, a 2x3 arrangement pattern (or a 3x2 arrangement pattern), and a 3x3 arrangement pattern.

4 is a diagram for explaining a disparity estimation path in a 2×2 array pattern and neighboring element images selected in this case according to an embodiment of the present invention, and FIG. 5 is a diagram for 2 according to another embodiment of the present invention. It is a diagram for explaining a disparity estimation path in a ×3 array pattern or a 3×2 array pattern and peripheral element images selected in this case, and FIG. 6 is a 3×3 array pattern according to another embodiment of the present invention. It is a diagram for explaining a disparity estimation path of , and neighboring element images selected in this case.

First, referring to FIG. 4 , in a 2×2 array pattern, the reference element image X is divided into four regions R1 , R2 , R3 and R4 divided into quarters. A disparity estimation path is determined according to which region among the regions R1 to R4 divided into quarters (r ₀ , c ₀ ) for estimating the disparity is located. As shown in FIG. 4 , when pixel coordinates (r ₀ , c ₀ ) (or disparity estimation position) for estimating disparity are located in R1, three neighboring element images S _i adjacent to R1 _-1j-1 , S _i-1j , S _ij-1 ) are selected and extracted to participate (use) in disparity estimation. If the pixel positions r ₀ , c ₀ for estimating the disparity are located in R2, three neighboring element images S _i-1j+1 , S _i-1j , S _ij+1 adjacent to R2 ) is selected and extracted to participate in disparity estimation.

Similarly, when pixel coordinates (r ₀ , c ₀ ) for estimating disparity are located in R3 or R4, three neighboring element images (S _i+1j-1 , S _ij-1 , S adjacent to R3) _i+1j ) or three neighboring element images (S _i+1j+1 , S _i+1j , S _ij+1 ) adjacent to R4 are selected and extracted to participate in disparity estimation.

In a 2x2 array pattern, a maximum of three disparities may be obtained from three neighboring element images. As will be described below, the disparity thus obtained is constituted by a set of disparity candidates for pixel coordinates (r ₀ , c ₀ ) by the candidate collection module 150 to be described below. Meanwhile, in FIG. 4 , three unidirectional arrows 41 , 42 , and 43 pointing to pixel coordinates ( r ₀ , c ₀ ) located at R1 of the reference element image indicate a disparity estimation path.

Referring to FIG. 5 , in a 2×3 array pattern or a 3×2 array pattern, the reference element image X _ij is divided into nine regions R1 to R9 divided into nine parts, and pixel coordinates r ₀ , c ₀ ) (or disparity estimation position) is selected and extracted to participate in (use) the disparity estimation according to the region in which 5 neighboring element images are located. For example, when the pixel positions r ₀ , c ₀ for disparity estimation are located in R2, the upper three neighboring element images S _i-1j-1 , S _i-1j , and S _i-1j+1 ) and the left and right two neighboring element images (S _i+1j-1 , S _ij+1 ) are selected and extracted to participate (use) in disparity estimation. Meanwhile, in FIG. 5 , five unidirectional arrows 51 , 52 , 53 , 54 and 55 pointing to the pixel positions r ₀ , c ₀ located at R2 of the reference element image indicate the disparity estimation path.

Referring to FIG. 6 , in the 3×3 array pattern, like the 2×3 array pattern or the 3×2 array pattern described in FIG. 5 , the reference element image X _ij is divided into nine regions R1 to R9 . , and when pixel coordinates (r ₀ , c ₀ ) for disparity estimation (or disparity estimation position) are located in the central region R5 among the 9 regions R1 to R9, all 8 peripheral elements Images (S _i-1j-1 , S _i-1j , S _i-1j+1 , S _ij-1 , S _ij+1 , S _i+1j-1 , S _i+1j-1 and S _i+1j+1 ) are selected and extracted to participate (use) in disparity estimation. Meanwhile, in FIG. 6 , eight unidirectional arrows 61 , 62 , 63 , 64 , 65 , 66 , 67 and 68 directed to pixel coordinates ( r ₀ , c ₀ ) located at R5 of the reference element image indicate a disparity estimation path. represent them

Based on the three arrangement patterns described with reference to FIGS. 4 to 6 , each estimated disparity (disparity data or disparity value) in all pixel coordinates of the reference element image X is configured as a candidate set for each pixel coordinate. can The size of the disparity candidate set for each pixel coordinate may vary.

Disparity estimation methods according to the three arrangement patterns are different from each other, and in the case of the central cross portion of the reference element image X, the disparity may be extracted by overlapping.

There may be two directions of the disparity estimation path. Referring to FIG. 6 as an example, when pixel coordinates (r ₀ , c ₀ ) for estimating disparity are located in the center of the reference element image (X _ij ), the start coordinate of the disparity estimation path in the peripheral element image is set equal to the pixel coordinates (r ₀ , c ₀ ) set in the reference element image (X _ij ). At this time, as shown in FIG. 6 , the arrows 61 , 62 , 63 , 64 , 65 , 66 , 67 and 68 indicating the direction of the disparity estimation path point toward the central reference element image. It is assumed that this arrow direction is not an inverted image in which the top and bottom of the element image obtained from the lens array is inverted. If the obtained elemental image is obtained from the lens array by the direct shooting method, the obtained elemental image is an upside-down inverted image. When the elemental image obtained in this way is an inverted elemental image upside down, the direction of the estimation path of the disparity (direction of arrows 61 to 68) is from the center to the outward direction, that is, from the reference element image to the peripheral element images. is the direction to If it is desired to set the direction of the disparity estimation path from the outside to the center as shown in FIG. 6 in the inverted element image in which the top and bottom are inverted, if the top and bottom of the inverted image are inverted again, as shown in FIG. It is possible to set the direction of the disparity estimation path (setting from the outside toward the center) as shown.

first vector generation module 120

The first vector generation module 120 extracts a matching vector to be matched with a vector extracted from the peripheral element image S from the reference element image X according to the disparity estimation path, and the extracted matching vector It is generated as a first vector ( u ).

First, a vector is extracted from the pixel coordinates (r ₀ , c ₀ ) set as the reference position in the reference element image (X). A block of an appropriate size including pixel coordinates (r ₀ , c ₀ ) is defined, and pixel values of pixel coordinates included in the defined block are rearranged into a one-dimensional vector. For example, when the block size defined to include the pixel coordinates (r ₀ , c ₀ ) is 5x5, a total of 25 pixel values are defined. If the reference element image is a color image, 75 (=25×3) pixel values are defined when three color components consisting of an R value, a G value, and a B value are considered. The first vector u may be generated by rearranging 25 or 75 pixel values into a one-dimensional vector. Here, the rearrangement order of the 25 or 75 pixel values is not very important. However, the rearrangement order of 25 or 75 pixel values defined in the reference element image is determined to be the same as the rearrangement order of 25 or 75 pixel values defined in the neighboring element image. Meanwhile, the block size defined to include the pixel coordinates (r ₀ , c ₀ ) is a system parameter and may affect estimation performance and operation speed.

Second vector generation module 130

)로서 생성한다.The second vector generation module 130 extracts a matching vector to be matched with the first vector ( u ) extracted from the reference element image (X) from the peripheral element image selected according to the disparity estimation path, and the extracted Match the vector to the second vector (

) is created as

)의 추출 및 생성 과정은 전술한 제1 벡터 생성 모듈(120)에 의해 수행되는 제1 벡터(u)의 추출 및 생성 과정과 유사하다. the second vector (

The extraction and generation process of ) is similar to the extraction and generation process of the first vector u performed by the above-described first vector generation module 120 .

Briefly, a block including coordinates of each pixel on the disparity estimation path is defined in the neighboring element image selected according to the arrangement pattern of FIGS. 4 to 6 . Here, the disparity estimation path is a path connecting the pixel coordinates set as the start position of the disparity estimation path in the peripheral element image and the pixel coordinates (r ₀ , c ₀ ) set as the reference position in the reference element image (X), and , the pixel coordinates set as the start position of the disparity estimation path are pixel coordinates corresponding to the pixel coordinates (r ₀ , c ₀ ) set as the reference position in the reference element image (X) in the peripheral element image. The block size is the same as the block size defined in the reference element image.

)로서 생성한다.When a block is defined, pixel values of pixel coordinates included in the defined block are rearranged into a one-dimensional vector, so that the pixel values rearranged into a one-dimensional vector are converted into a second vector (

) is created as

Vector matching module (140)

)와 상기 제1 벡터 생성 모듈(120)에 의해 기준 벡터로서 생성된(추출된 또는 정의된) 제1 벡터(u)를 정합한다. The vector matching module 140 generates (extracted or defined) from a block including the coordinates of each pixel on the disparity estimation path by the second vector generation module 130 to construct a disparity candidate set. the second vector (

) and the first vector u generated (extracted or defined) as a reference vector by the first vector generation module 120 .

The first vector obtained from the pixel coordinates (r ₀ , c ₀ ) corresponding to the reference position in the reference element image (X) is 'u', and the second vector obtained from the same position of the peripheral element image is 'v _n ' Assume that Here, the subscript n of v _n is a position index value indicating a position (or pixel coordinate) on the disparity estimation path.

For matching, a well-known appropriate price function (Cost Function) C(v _n , u) may be used. As the price function, for example, an algorithm such as mean square error (MSE), sum absolute difference (SAD), or normalized correlation coefficient (NCC) may be used.

When an appropriate price function is determined, a value at which the price function becomes the minimum value or a value at which the similarity becomes the maximum value is calculated as a location index value on the estimation path using Equation 1 below.

Candidate collection module 150

The candidate collection module 150 sets the position index values (n values) calculated in the first vector and second vector matching process performed by the vector matching module 140 to the pixel coordinates r ₀ in the reference element image X. , c ₀ ) is composed of a set of disparity candidates.

Disparity extraction module 160

The disparity extraction module 160 extracts (or estimates) a final disparity value (or disparity data) from among the position index values (n values) included in the disparity candidate set configured by the candidate collection module 150 .

In order to extract (estimate) a disparity value from the disparity candidate set, a statistical technique may be utilized. For example, an n value (a location index value) corresponding to a mode indicating the highest frequency and/or the highest distribution among location index values (n values) is extracted as the final disparity.

When choosing the mode, values of n with a difference of approximately ±1 are considered equal. When there are a plurality of n values (location index values) corresponding to the mode, the average value of the plurality of n values (location index values) corresponding to the mode is estimated and extracted as the final disparity value d. A final disparity value d based on an average of n values (position index values) may be calculated by Equation 2 below.

A method for estimating disparity in an element image arrangement according to an embodiment of the present invention will be described.

First, a first step of extracting the reference element image and the peripheral element images located on the disparity estimation path from the input element image array is performed.

Next, a second step of defining a first vector u based on pixel coordinates (r ₀ , c ₀ ) corresponding to a reference position of the disparity estimation path in the reference element image is performed.

Next, a third step of defining the second vector v _n is performed based on the coordinates of each pixel located on the disparity estimation path in each neighboring element image.

Next, in the process of matching the first vector and the second vector, a fourth step of calculating position index values (n values) indicating an optimal matching position on the disparity estimation path is performed.

Next, a fifth step of configuring the calculated position index values (n values) into a disparity candidate set is performed.

A sixth step of estimating a final disparity value among the location index values based on the frequency and distribution of location index values included in the disparity candidate set is performed.

The first step is to extract the element image array of a 3 × 3 array consisting of the reference element image and eight peripheral element images surrounding the reference element image from the input element image array and the 3 × 3 The method may include extracting neighboring element images located on the disparity estimation path from the element image arrangement of the array.

In addition, the first step is, extracting the element image array of a 3 × 3 array consisting of the reference element image and eight peripheral element images surrounding the reference element image from the input element image array and the 3 × The method may include extracting the neighboring element images located on the disparity estimation path determined according to three arrangement patterns that can be configured in a three-array element image arrangement. Here, the three arrangement patterns may include a 2×2 array pattern ( FIG. 4 ), a 2×3 or 3×2 array pattern ( FIG. 5 ), and a 3×3 array pattern ( FIG. 6 ).

The step of extracting the peripheral element images located on the disparity estimation path determined according to the 2×2 arrangement pattern includes dividing the reference element image into four regions divided into quarters; When the corresponding pixel coordinates (r ₀ , c ₀ ) are set to be located in any one of the four regions, three perimeters adjacent to the region where the pixel coordinates (r ₀ , c ₀ ) are set to be located It may include extracting the component images as the neighboring component images located on the disparity estimation path.

The extracting of the peripheral element images located on the disparity estimation path determined according to the 2×3 or 3×2 arrangement pattern may include partitioning the reference element image into nine regions divided into 9 equal parts; The pixel coordinates (r ₀ , c ₀ ) corresponding to the reference position are set in any one of the regions (R2, R4, R6, R8) located on the top, bottom, left and right of the central region (R5) among the nine regions. In this case, the method may include extracting five neighboring element images included in the 2×3 or 3×2 array pattern as neighboring element images positioned on the disparity estimation path.

The extracting of the peripheral element images located on the disparity estimation path determined according to the 3×3 arrangement pattern includes dividing the reference element image into nine regions divided into nine equal parts and at the reference position. When the corresponding pixel coordinates (r ₀ , c ₀ ) are set in the central region R5 among the 9 regions, 8 peripheral element images included in the 3×3 array pattern are displayed on the disparity estimation path. It may include the step of extracting the located surrounding element images.

In the second step, the defining of the first vector u includes defining a block including the pixel coordinates r ₀ , c ₀ , and defining pixel values of pixels included in the defined block. It may include rearranging the one-dimensional vector and defining the pixel values rearranged with the one-dimensional vector as the first vector (u).

In the third step, the step of defining the second vector (v _n ) defines a block including pixel coordinates corresponding to the same position as the pixel coordinates (r ₀ , c ₀ ) within each peripheral element image. It may include the steps of rearranging pixel values of pixels included in the defined block into a one-dimensional vector, and defining the pixel values rearranged with the one-dimensional vector as the second vector (v _n ). can

In the fourth step, the step of calculating the location index values (n values) is a value at which the price function used in the process of matching the first vector and the second vector is the minimum value of the location index values (n values) ) may be a step of calculating as

In the sixth step, the step of estimating the final disparity value includes a position index value corresponding to a mode indicating the highest frequency or highest distribution among the position index values (n values) as the final disparity value. It may be a step of estimating as

The method for estimating disparity in the element image arrangement of the present invention described above may be implemented in a computer system or recorded in a recording medium. A computer system may include at least one processor, memory, a user input device, a data communication bus, a user output device, and storage. Each of the above-described components performs data communication through a data communication bus.

With respect to the processor, the components 110 , 120 , 130 , 140 and 150 shown in FIG. 3 may be respectively implemented as functional logics and mounted in one processor, and each may be implemented as a separate processor.

The computer system may further include a network interface coupled to the network. The processor may be a central processing unit (CPU) or a semiconductor device that processes instructions stored in a memory and/or storage.

The memory and storage may include various types of volatile or non-volatile storage media. For example, memory may include ROM and RAM.

In addition, the image restoration method based on the integrated image according to the present invention may be implemented as a method executable in a computer.

The image restoration method based on the integrated image according to the present invention described above may be implemented as computer-readable codes on a computer-readable recording medium. The computer-readable recording medium includes any type of recording medium in which data that can be read by a computer system is stored. For example, there may be a read only memory (ROM), a random access memory (RAM), a magnetic tape, a magnetic disk, a flash memory, an optical data storage device, and the like.

In addition, the computer-readable recording medium may be distributed in computer systems connected through a computer communication network, and stored and executed as readable codes in a distributed manner.

So far, the present invention has been looked at focusing on examples. Those of ordinary skill in the art to which the present invention pertains will understand that the present invention may be implemented in variously changed or modified forms without departing from the essential characteristics of the present invention. Therefore, the disclosed embodiments are to be considered in terms of illustrative rather than restrictive views. The scope of the present invention is indicated in the claims rather than the foregoing description, and all differences within the scope equivalent thereto should be construed as being included in the present invention.

Claims (16)

extracting a reference element image and neighboring element images located on a disparity estimation path from the input element image array;
defining a first vector (u) based on pixel coordinates (r ₀ , c ₀ ) corresponding to a reference position of the disparity estimation path in the reference element image;
defining a second vector (v _n ) based on pixel coordinates located on the disparity estimation path in each neighboring element image;
calculating position index values (n values) indicating an optimal matching position on the disparity estimation path in a process of matching the first vector and the second vector;
composing the calculated position index values (n values) into a disparity candidate set; and
estimating a final disparity value from among the location index values based on the frequency and distribution of location index values included in the disparity candidate set,
The extracting step is
extracting an element image array of a 3×3 array consisting of the reference element image and eight peripheral element images surrounding the reference element image from the input element image array; and
extracting neighboring element images located on the disparity estimation path from the element image arrangement of the 3×3 array;
A method of estimating disparity in an elemental image array comprising delete In claim 1,
The extracting step is
extracting an element image array of a 3×3 array consisting of the reference element image and eight peripheral element images surrounding the reference element image from the input element image array; and
and extracting the neighboring element images located on the disparity estimation path determined according to three arrangement patterns that can be configured in the 3×3 element image arrangement,
The three arrangement patterns are,
A method for estimating disparity in an elemental image array comprising a 2×2 array pattern, a 2×3 or 3×2 array pattern, and a 3×3 array pattern. In claim 3,
The extracting of the neighboring element images located on the disparity estimation path determined according to the 2×2 arrangement pattern includes:
dividing the reference element image into four regions divided into quadrants;
When the pixel coordinates (r ₀ , c ₀ ) corresponding to the reference position are set to be located in any one of the four regions, the pixel coordinates (r ₀ , c ₀ ) are located in the region set to be located. extracting three adjacent component images as the neighboring component images located on the disparity estimation path;
A method for estimating disparity in an elemental image array, comprising: In claim 3,
The step of extracting the neighboring element images located on the disparity estimation path determined according to the 2×3 or 3×2 arrangement pattern includes:
dividing the reference element image into nine regions divided into nine equal parts; and
The pixel coordinates (r ₀ , c ₀ ) corresponding to the reference position are set in any one of the regions (R2, R4, R6, R8) located on the top, bottom, left and right of the central region (R5) among the nine regions. case, extracting five neighboring element images included in the 2x3 or 3x2 array pattern as neighboring element images located on the disparity estimation path
A method for estimating disparity in an elemental image array, comprising: In claim 3,
The extracting of the neighboring element images located on the disparity estimation path determined according to the 3×3 arrangement pattern comprises:
dividing the reference element image into nine regions divided into nine equal parts; and
When the pixel coordinates (r ₀ , c ₀ ) corresponding to the reference position are set in the central region R5 among the nine regions, eight peripheral element images included in the 3×3 array pattern are displayed in the disparity Extracting as peripheral element images located on the estimation path
A method for estimating disparity in an elemental image array, comprising: In claim 1,
The step of defining the first vector (u) comprises:
defining a block including the pixel coordinates (r ₀ , c ₀ );
rearranging pixel values of pixels included in the defined block into a one-dimensional vector; and
defining the pixel values rearranged in the one-dimensional vector as the first vector (u);
A method for estimating disparity in an elemental image array, comprising: In claim 1,
The step of defining the second vector (v _n ) is,
Defining a block including pixel coordinates corresponding to the same position as the pixel coordinates (r ₀ , c ₀ ) within each peripheral element image
rearranging pixel values of pixels included in the defined block into a one-dimensional vector; and
defining the pixel values rearranged in the one-dimensional vector as the second vector (v _n );
A method for estimating disparity in an elemental image array, comprising: In claim 1,
Calculating the position index values (n values) comprises:
The method of estimating disparity in an elemental image array, comprising calculating, as the position index values (n values), a value at which a price function used in the process of matching the first vector and the second vector becomes the minimum. In claim 1,
The step of estimating the final disparity value comprises:
The method of estimating a position index value corresponding to a mode indicating the highest frequency or highest distribution among the position index values (n values) as the final disparity. A computing device for estimating disparity in an element image array, the computing device comprising a disparity estimation module, a processor, a memory, and a system bus connecting them;
The disparity estimation module,
an extraction module for extracting a reference element image and neighboring element images located on a disparity estimation path from the input element image array;
a first vector generating module that generates a first vector (u) based on pixel coordinates (r ₀ , c ₀ ) corresponding to a reference position of the disparity estimation path in the reference element image;
a second vector generating module for generating a second vector (v _n ) based on pixel coordinates located on the disparity estimation path in each neighboring element image;
a vector matching module for calculating position index values (n values) indicating an optimal matching position on the disparity estimation path in a process of matching the first vector and the second vector;
a candidate collection module that configures the calculated position index values (n values) into a disparity candidate set; and
a disparity extraction module for extracting a final disparity value from among the location index values composed of the disparity candidate set based on the frequency and distribution of the location index values;
The extraction module,
After extracting a 3×3 element image array consisting of the reference element image and eight peripheral element images surrounding the reference element image from the input element image array, it is constructed in the 3×3 array element image array A computing device that extracts the neighboring element images located on the disparity estimation path determined according to three possible arrangement patterns. delete In claim 11,
The three arrangement patterns are,
A computing device comprising a 2x2 arrangement pattern, a 2x3 or 3x2 arrangement pattern, and a 3x3 arrangement pattern. In claim 13,
The extraction module,
When the disparity estimation path is determined according to a 2×2 array pattern, three neighboring element images included in the 2×2 array pattern are extracted, and the disparity estimation path is a 2×3 or 3×2 array pattern is determined according to, extracting five neighboring element images included in the 2×3 or 3×2 array pattern, and when the disparity estimation path is determined according to the 3×3 array pattern, a 3×3 array pattern A computing device that extracts eight peripheral element images included in the . In claim 11,
The vector matching module,
A computing device that calculates, as the location index values (n values), a value at which a price function used in a process of matching the first vector and the second vector becomes the minimum. In claim 11,
The disparity extraction module,
A computing device that extracts a location index value corresponding to a mode indicating the highest frequency or highest distribution among the location index values (n values) as the final disparity.

Images