KR20000058657A - Architecture for Measuring DisparIty using Memory Structual OptimIzed and Matching Pixel Count - Google Patents

Abstract

PURPOSE: An optimized memory structure and a disparity measuring architecture adapting a matching pixel count are provided for preventing an over boundary phenomenon in a stereo variation estimation using a similarity measuring method which uses a matching pixel count(MPC) having similar brightness in a matching window. CONSTITUTION: An apparatus for determining a variation value includes a strip computation unit(100) and a MPC-processor(200). In order to have the size 11x11 of a window, the values of Wx and Wy are set to 11. Sr is set to 63. The reference and search image pixel values and the threshold value Th are inputted into the apparatus. The synchronous signal Hsync and Vsync are inputted as the input values of the apparatus. The maximum MPC(x,y,d) at the pixels R(x,y) of the reference image and the variation DM(x,y) are outputted.

Description

Architecture for Measuring DisparIty using Memory Structual OptimIzed and Matching Pixel Count}

The present invention relates to digital image processing, and more particularly, to a method for measuring similarity of pixels in two different images and an architecture for efficiently implementing the same.

Stereo displacement estimation is an area where active research is being conducted in the field of three-dimensional or stereo vision. Stereo displacement refers to an offset or displacement between one pixel in a reference image and another pixel in a searched image, and each pixel corresponds to the same point in physical space. The reference and search images are images viewed by the left and right eyes, respectively. In order to determine a pixel in the search pixel that matches a given pixel in the reference image, the coordinates for the pixels in the two images must first be set. The similarity is then the similarity between the pixels in a window of constant size centered on a given pixel (i.e., the reference pixel) in the reference image and the pixels in the window of the same size centered around the pixels of the navigation image considered to be correspondence points. Calculate A pixel having the maximum similarity in the searched image is regarded as a pixel matched with the reference pixel.

For each pixel of the reference image, similarity is calculated for all candidate corresponding pixels within the search range, and a displacement map of the reference image is generated. By applying the epipolar constraint to the candidate corresponding pixels, the displacement map can be made much simpler by restricting the pixels to the same horizontal line as the reference pixel.

1 is a diagram illustrating registration by epipolar constraint. To find the corresponding point of the pixel indicated by the circle of the reference image in the search image, the similarity between the window centered on the reference pixel and the window centered on the candidate corresponding pixel in the search image is moved along the horizontal line in the search region. Calculate while doing. A candidate search pixel having a maximum similarity in the search image is regarded as a pixel matched with the reference pixel. The displacement between the matched pixel and the reference pixel is the stereo displacement with respect to the reference pixel.

Methods of calculating the similarity for determining matching pixels include Sum of Squared Differences (SSD) and Sum of Absolute Differences (SAD) methods, and Normalized Cross Correlation (NCC) methods that correlate with each other. . Similarity measurement methods, such as SSD, SAD, and NCC, depend on the brightness value of a search pixel in which one search pixel is affected in the displacement map. Therefore, the phenomenon that the boundary of displacement of the region where the brightness change between the pixels is large extends to the region where the brightness change is small "occurs, which causes an incorrect displacement value at the boundary of the object.

Accordingly, an object of the present invention is to use a similarity measurement method using a matching pixel count (MPC) of similar brightness in a matching window without directly using the brightness value of the search pixel. It is to prevent the phenomenon. Another object of the present invention is to provide an apparatus for stereo displacement estimation, which implements the method efficiently in real time by using an optimized memory structure.

One embodiment of the present invention provides a method of determining a displacement of a search image angle with respect to a reference image and its reference pixel.

Iii) a reference window comprising a set of pixels within a radius centered on a point of the reference pixel, using a matching pixel number representing the number of pixels of the reference window whose brightness of the corresponding pixels in the search window is similar, within a predetermined search range from the reference window; Calculating a similarity measure between each navigation window of a group of navigation images that are at a distance and look the same as the reference window; and

Ii) determining the displacement between the search window and the reference window in which the similarity measure is the largest as the stereo displacement with respect to the reference pixel.

Another embodiment of the present invention provides a device for determining stereo displacement between a reference image and a search image for a reference pixel thereof.

Iii) a reference window comprising a set of pixels within a radius about a point of the reference pixel and a distance within a predetermined search range from the reference window, using a matched pixel number representing the number of pixels of the reference window that is similar to the brightness of the corresponding pixels of the search window. First means for calculating a similarity measure between each navigation window of a group of navigation images of the same type as the reference window; and

Ii) a second means for determining the displacement between the search window and the reference window in which the similarity measure is greatest as the stereo displacement with respect to the reference pixel;

R (x, y) represents the reference pixel, the reference window contains Wx * Wy pixels (where Wx and Wy are already predetermined numbers) within a radius centered on a point on R (x, y), Each window has one Wx * Wy radius within a radius centered on one point on L (x + d, y), a pixel of the navigation image, where d translates from zero to a predetermined number Sr. Contains pixels,

The first means is the following formula

i 1) P (x, y, d) =

(here Wow Is an S-unit that calculates P (x, y, d) by R (x, y) and L (x + d, y), and Th is the bound

2) S-buffer for storing V (x, y, d) value from the S-unit

Ⅰ3) the following formula

Where W represents the navigation window at some distance from the reference window and on a limit on L (x + d, y) to determine the MPC (x, y, d) from d = 0 to d = Sr. Third means for

The second means is for an apparatus comprising selecting a value of d such that MPC (x, y, d) is the largest value as stereo displacement with respect to R (x, y).

1 illustrates a registration constrained by an epipolar constraint.

Fig. 2 is a diagram showing the duplication operation inherent in the MPC similarity measuring method.

3 is a diagram illustrating a method of measuring MPC similarity with duplicate calculations removed;

4 is a flowchart illustrating a method of measuring MPC similarity according to an embodiment of the present invention.

5 is an overall view for determining a displacement map D (x, y) according to an embodiment of the present invention.

Drawing showing architecture

FIG. 6 shows the internal structure of a Strip-operator of the architecture shown in FIG.

FIG. 7 shows the internal structure of the first S-unit of the architecture shown in FIG.

8 shows the internal structure of the S-unit from the second to the last of the architecture shown in FIG.

9 shows the internal structure of the S-buffer of the architecture shown in FIG.

10 shows the internal structure of the S-register of the architecture shown in FIG.

11 shows the internal structure of the MPC-unit of the architecture shown in FIG.

12 illustrates the internal structure of the MPC-buffer of the architecture shown in FIG.

FIG. 13 is a diagram showing an internal structure of an MPC updater of the architecture shown in FIG.

FIG. 14 shows the internal structure of the Max_MPC selection unit of the architecture shown in FIG.

15 illustrates the internal structure of the parallel maximum selector of the architecture shown in FIG.

16 shows the internal structure of the Comp-Unit of the architecture shown in FIG.

17 shows the internal structure of the comparison selector of the architecture shown in FIG.

<Explanation of symbols for main parts of drawing>

100: S-buffer

200 MPC-processor

210: Strip-operator

220: MPC unit

230: MPC-buffer

240: Max_MPC selection section

250 control unit

260: MPC update unit

According to the method described and claimed in the present application, the MPC is used to measure the similarity for each of the reference window in the reference image and each of the plurality of search windows in the search image. In this case, the effect of one pixel on the similarity is the same regardless of the brightness value, so that the result of the suspension can be obtained even at the boundary of the image. The MPC calculates from the number of pixels of a window in one image (pixels in which a pixel in another image corresponding to the pixel has a similar brightness value).

When the reference window and the search window satisfy the epipolar constraint, the reference window centered on the pixel R (x, y) located at (x, y) in the reference image and (x + d, y) in the search image The MPC (x, y, d) value between the search windows centering on the pixel L (x + d, y) may be represented as follows.

Denotes brightness values of pixels R (x, y) and pixels L (x + d, y), respectively, and W is based on R (x, y) of the reference image or L (x + d, y) of the search image. The size (Wx × Wy) matching window, which is a set of pixels, is shown. Th is also a predefined threshold.

P (x, y, d) has a value of 1 if the pixel brightness is similar for R (x, y) and L (x + d, y) in two images, and 0 otherwise. The displacement value for the pixel at the reference image R (x, y) position is the d value at which the maximum value is obtained after MPC (x, y, d) is obtained for all the search ranges d (d = 0 to Sr). .

2 is a diagram illustrating a redundant operation inherent in the MPC similarity measuring method. Despite the epipolar constraint, the amount of calculation for determining the MPC value for all the pixels of the reference image is proportional to (Ix × Iy) × (Wx × Wy) × Sr. Where (Ix × Iy) is the size of the image, (Wx × Wy) is the size of the window, and Sr is the search range. This large amount of computation is due to the redundant computation inherent in the MPC method.

As shown in FIG. 2, if the MPC value between two points R (x, y) and L (x + d, y) is determined, R (x + 1, y) and L (x + 1 + d, y) When determining the MPC value between), the calculations corresponding to the overlapping portions of two horizontally displaced windows will be overlapping calculations. This large amount of redundancy can be eliminated by using a buffer to store the result of the operation and reducing the overall amount of calculation to a proportional amount of (Ix × Iy) × Sr that is not affected by the size of the window.

FIG. 3 is a diagram illustrating a method of calculating an MPC value without overlapping calculations, and only the pixels of the reference image are shown. MPC (wx, y, d) is calculated by adding the S value by the following equation.

For x larger than wx in Equation 2, MPC (x, y, d) can be calculated from the MPC value MPC (x-1, y, d) calculated in the previous step as shown in the following equation.

In Equations 2 and 3, wx and wy represent distances from the center of the window to the boundary in the horizontal and vertical directions, respectively. That is as follows.

Similarly, the first S value of a column, S (x, wy, d), is calculated by adding P (x, y + i, d) for the vertical portion as follows.

Here, P (x, wy + 1, d) has a value of 1 if the values of R (x, y) and L (x + d, y) are similar as in Equation 1, and 0 otherwise. For y greater than wy, the above process is repeated by (Iy-wy).

S (x, wy, d) representing the similarity between vertical lines centered on R (x, wy) and L (x + d, wy) according to Equation 5 when starting stereo displacement estimation for the reference image and the search image. Is calculated first for x of Wx size and d value in the search interval, that is, x = 0 to Wx and d = 0 to Sr. That is, S (x, wy, 0), S (x, wy, 1), S (x, wy, 2), ..., S which are S values equal to the search range (Sr + 1) for each X value. (x, wy, Sr) is obtained. The obtained S value is stored in an S buffer of size 4 × Wy × (Sr + 1). Each size of the S buffer represents the similarity value between the vertical lines. Is a rounded up integer.

In Equation 2, MPC (wx, y, d) is the similarity between R (wx, y), which is a pixel of the wxth column of the reference image, and a registration window centered on pixel coordinate L (wx + d, y) of the searched image. It is initialized by accumulating the matching S (wx + i, y, d) between vertical lines from i = -wx to wx and storing it in the MPC buffer. This initialization of the MPC buffer is performed for the (wx) th pixel of each row.

For x values greater than wx, MPC (x, y, d) is MPC (x-1, y, which is the MPC for the previous pixel in the same row that was stored in the MPC buffer, as shown in Equation 3 and FIG. calculated by using d). Specifically, S (x + wx, y, d), the S value for the vertical line newly added to the window, is added to the previously calculated MPC value, and S (x-wx-1, the S value for the vertical line excluded from the window. can be calculated by subtracting, y, d).

Similarly, when calculating MPC (x, y, d) values for y values greater than wy, the displacement value is obtained by repeating the above process.

In summary, after initializing the S buffer with S values for the vertical lines around each pixel in the (wy) th row, and after initializing the MPC buffer with the MPC value for the (wx) th pixel in one row, MPC values can be obtained without the redundant operation by the equation (3). At this time, the search displacement of each pixel must be the same. If the pixel having the maximum value among the MPC matching results for each pixel of the searched image within the searched range for the pixel R (x, y) of the reference image is L (x + dmax, y), dmax is R (x, y Is the stereo displacement for.

4 shows a flowchart of the MPC matching algorithm for the MPC stereo displacement extraction from which the redundant calculation described above is removed. Given as the input of this algorithm, 1) the intensity of pixels R (x, y) and L (x, y) in the reference and search images (where x = 0 to Ix-1, y = 0 to Iy-1) And 2) Wx and Wy (where Wx and Wy are odd) representing the size of the registration window, and 3) search range Sr. The output of this algorithm is the displacement index DM (x, y) (where x = wx ~ Ix-wx, y = wy ~ Iy-wy) containing the displacement for each pixel.

In step S20, y is initialized to wy. Then, in step S30, an initial S value of S (x, wy, d) for x = 0 to Wx, d = 0 to Sr is calculated, and the result is stored in the S buffer. The S value stored in the S buffer indicates the similarity with respect to Wy, the size corresponding to the vertical column of the matching window, and is composed of Sr × (Wx + 1) buffers.

Steps S40, S70, S90 and S130 check the conditions for the range of x, y and d for obtaining MPC (x, y, d) values, and in step S60, x is initialized to wx, step In operation S80, initialization of d and the maximum MPC value is performed.

Steps S50, S85, and S110 represent an update process for performing an iterative process for a given range of x, y, and d values.

In steps S120, S130, S140, S150, and S160, the MPC value is initialized or updated according to the value of x, and the S value is updated. This step is performed for each value of x, y, and d (x = wx ~ Ix + wx, y = wy ~ Iy-wy-1, d = 0 ~ Sr). When x = wx, equation (2) is performed to generate the first MPC value in the row, and equations (2) and (3) are performed when the value of x is less than Ix-d-wx and greater than wx. If the value of x is greater than or equal to Ix-d-wx, the MPC value is zero.

Steps S170 and S180 determine the d value giving the maximum MPC value as the displacement with respect to the reference pixel. After all steps have been completed, a displacement map DM (x, y) for x = wx ~ Ix-wx, y = wy-Iy-wy is obtained.

5 shows an apparatus for determining a displacement value according to the invention. The device includes a strip operator 100 and an MPC-processor 200, wherein the values of Wx, Wy are set to 11 and Sr is set to 63 to size the window to 11 × 11. The threshold Th, which determines whether the two pixels are similar to the pixel values of the reference and search images, enters the input of the device, and the synchronization signals Hsync and Vsync are also given as the input of the device. As the output, the maximum MPC (x, y, d) and the displacement DM (x, y) value at the pixel R (x, y) of the reference image are obtained.

The MPC-processor 200 is composed of a strip-operator 210, an MPC-unit 220, an MPC-buffer 230, an MPC updater 260, a Max_MPC selector 240, and a control unit. The S-operator 210 has (Sr + 1) S-units 211 so that (Sr + 1) MPC values are calculated in parallel for the search range for each reference pixel R (x, y). . The pixel values R (x, y) and L (x + d, y) are input to the S-operator 210 to accumulate P (x, y, d) shown in Equation 1 for the Wy size, thereby providing MPC. The value is calculated. In the S-unit 211, (Sr + 1) pieces for a given (x, y) (Where d = 0 to Sr) values are calculated in parallel. The calculated value is stored in the S-buffer 100.

The input S (x, y, d) value is input to the MPC-unit 220 to calculate the MPC (x, y, d). After calculating the MPC (x, y, d) value, the MPC (x, y, d) value stored in the MPC-buffer 230 and updated by the MPC updater 260 is obtained and the Max_MPC selector 240 ), The maximum MPC (x, y, d) value is obtained.

The Max_MPC selector 240 receives the MPC value for the pixel R (x, y) from the MPC-unit 220 and the MPC updater, selects the largest MPC value, and calculates the displacement at that time as DM (x, y). Output to

6 shows the internal structure of the strip-operator 210. The strip-operator 210 is composed of (Sr + 1) S-units 211, 212, and P shown in Equation 1 for the pixel values R (x, y) and L (x + d, y). By accumulating (x, y, d) on Wy, the vertical size of the matching window, the MPC value is obtained and the calculated result is stored in the S-buffer.

7 and 8 show the internal structure of the S-units 211 and 212, respectively. The S-unit 211 corresponds to S30 in the flow diagram of FIG. 4 and calculates the MPC value by accumulating the calculated values for each pixel to obtain the MPC value for Wy, the vertical size of the matching window when x = 0. . Similarly, the S-unit 212 is for calculating the MPC value for (Wy + 1), the vertical size of the matching window when x = 1 to 63 in S30 in the flowchart of FIG. 4. As a result, it can be seen that the MPC value corresponding to the (Wy + 1) size is calculated by the (Sr + 1) S-units for each input pixel.

9 and 10 show the internal structure of the S-buffer 100 and the S-register. The S-buffer 100 is composed of all (Wx + 2) S-registers 110, MUX, and a counter, and is perpendicular to the matching window output from each S-unit 211 in the Strip-operator 210. MPC values for the size (Wy + 1), 64 (ie, (Sr + 1)) 4-bit values coming in in parallel, are input and stored in the S-register 110. These values are shifted in the direction of the arrow by the S-register 110 in accordance with the system clock. The size of the entire S-register 110 is 4 * (Sr + 1) * (Wy + 2), and the output value for the vertical size (Wy + 1) of the registration window is (Wx + 1) by the 6-bit counter. As many as By doing so, the MPC value for one matching window is obtained, and the MPC value can be obtained continuously.

11, 12, and 13 show the internal structure of the MPC unit 220, the MPC buffer and the MPC update unit. The MPC-unit 220 calculates the MPC (x, y, d) value of the matching window by calculating the MPC value of the vertical size (Wx + 1) of the (Wx + 1) matching windows and has a pipeline structure. By doing so, high-speed operation is possible. The MPC-unit 220 corresponds to S120 in the flowchart of FIG. 4 and calculates MPC values for (Sr + 1) d values when x = wx and outputs them to the MPC-buffer 230. The MPC-buffer 230 stores the MPC output value from the MPC-unit 220 and the output value of the MPC-updater 260 in a register through MUX, and is used as an input to the MPC-updater 260. . The MPC-update unit 260 is composed of (Sr + 1) subtractors and adders, performs an operation corresponding to Equation 3, and corresponds to S160 in the flowchart of FIG. 4. First, S (x + wx, Performs an operation on y, d) -S (x-1-wx, y, d) and adds this value to the previous MPC result. Where S (x + wx, y, d) and S (x-1-wx, y, d) are the output values from Strip-operator 210 and S-buffer 100, respectively.

14 and 15 show the internal structure of the Max_MPC selector 240 and the parallel maximum value selector 241. The Max_MPC selector is composed of (Sr + 1) MUXs and a parallel maximum value selector 241, and compares and selects the maximum MPC (x, y, d) value within the search section and DM (x), which is the position information at that time. , y) Print the value. The parallel maximum value selector 241 is composed of four Comp units 241-1 and comparators 241-2, and selects the maximum MPC value between matching windows within the search interval (Sr + 1). Outputs position information when

16 and 17 show the internal structure of the Comp unit 241-1 and the comparison selector 241-2. Comp unit 241-1 is composed of comparison selectors 241-2 and registers, and outputs maximum MPC values and position information for 16 7-bit inputs. Each comparison selector 241-2 outputs the position of the maximum matching window and the maximum MPC (x, y, d) value simultaneously for (Sr + 1) search intervals. The value output through the last comparison selector in this way is the maximum MPC value and the displacement at that time for a given pixel R (x, y).

Although the present invention has been described in terms of specific embodiments, those skilled in the art will understand that various changes and modifications can be made without departing from the spirit and scope of the invention as defined in the claims set out below.

Claims (3)

A stereo disparity determination method between a reference image and a search image for a reference pixel thereof, the method comprising: (Iii) a reference window comprising a set of pixels within a radius centered on a point of the reference pixel, using a matching pixel count representing the number of pixels of the reference window that is similar to the corresponding intensity of the search window; calculating a similarity measure between each search window of a group of search images that are at a distance within a predetermined search range from the reference window and in the same form as the reference window; and (Ii) determining the displacement between the search window and the reference window in which the similarity measure is the largest as the stereo displacement with respect to the reference pixel. (New) The apparatus of claim 1, wherein R (x, y) represents a reference pixel, and the reference window represents Wx * Wy pixels within a radius centered on a point on R (x, y), where Wx and Wy Each of the search windows is a point on L (x + d, y), which is a pixel of the search image. (Where d = 0 to d = Sr is the domain), including Wx * Wy pixels within a radius, The step (iii) is the following formula (Ⅰ1) P (x, y, d) = Calculating P (x, y, d) by (where and are the brightness values of R (x, y) and L (x + d, y), and Th is a predetermined threshold) and (Ⅰ2) : Equation 1 Where W represents the search window at some distance from the reference window and at a point on L (x + d, y) to determine the MPC (x, y, d) from d = 0 to d = Sr. Including the steps of: Step (ii) includes selecting a value of d such that MPC (x, y, d) is the largest value as the stereo displacement with respect to R (x, y) and the reference image and the reference image thereof. In the stereo displacement determination device between the search image for the pixel, (Iii) a search already determined from a reference window and a reference window comprising a set of pixels in the search window centered on a point of the reference pixel, using a matching pixel number representing the number of pixels of the reference window that is similar to the brightness of the corresponding pixels of the search window; First means for calculating a similarity measure between each search window of a group of search images of the same type as the reference window and at a distance within the range; (Ii) a second means for determining the displacement between the search window and the reference window in which the similarity measure is greatest as the stereo displacement with respect to the reference pixel, wherein R (x, y) represents the reference pixel, and the reference window represents R (x, y) includes Wx * Wy pixels (where Wx and Wy are already defined numbers) within a radius about a point on the point, each of the navigation windows being one on L (x + d, y), one pixel of the search image. Contains Wx * Wy pixels around a point (where d = 0 to d = Sr as the translation), The first means is the following formula (Ⅰ1) P (x, y, d) = (Where and denote the brightness values of R (x, y) and L (x + d, y), and Th is a predetermined threshold), and calculate P (x, y, d), Equation 2 Where wy is a strip-operator and S-unit that calculates the value of S (x, y, d) by (Wy-1) / 2, (X2) an S-buffer and an S-register for storing S (x, y, d) values from the Strip-operator, (3) : Equation 3 Where wx is (Wy-1) / 2 and S (wx + i, y, d) is the MPC value corresponding to Wy, the vertical size of the matching window) Means; And said second means selecting a value of d such that MPC (x, y, d) is the largest value as stereo displacement with respect to R (x, y). (New) The method according to claim 2, wherein each of the third means is provided for each of d values. Stereo displacement determination field containing (Sr + 1) MPC-units that determine MPC (x, y, d) And MPC means, respectively, (Where wy is the following equation using means for determining the value of S (x, y, d) by (Wy-1) / 2) and the value of S (x, y, d) (if x = wx) Equation 4 (Where wx is means for generating a value of MPC (x, y, d) by (Wx-1) / 2) and MPC (x, y, d) from the MPC determining means The apparatus for determining stereo displacements and the means for determining MPC further comprises an MPC-buffer for storing a value of and providing the stored value of MPC (x, y, d) to the MPC generating means, Equation 5 (Where wx is an MPC-update unit using an equation 4 for the value of MPC (x, y, d) updated by (Wx-1) / 2) In response to (Sr + 1) L (x + d, y) from (Sr + 1) S-units, each storing the value of L (x + d, y) for d, A stereo displacement determining apparatus comprising (Sr + 1) S-units which simultaneously provide = (Sr + 1) S (x, y, d) values.