JPH10334244A - Stereo matching device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To execute stereo matching at a high speed by compositing stereoscopic images and writing the composited image to a memory corresponding to each horizontal line to attain evaluation of coincidence of a small rectangular area for each clock with a simple circuit configuration. SOLUTION: Pixels of a left image 101 and a right image 102 are composited between corresponding pixels in a way that high-order 8 bits are pixels for the right image and low-order 8 bits are pixels for the left image as 16-bit pixels and written in memories M0 -M3 . Data 104-107 read sequentially synchronously with a clock signal are given to linear matching circuits 123-126 and a difference absolute sum of 4 pixels in the horizontal direction is calculated synchronously with the clock signal. The difference absolute sum is added by adders 128-130 and a difference absolute sum 114 of all pixels in the small rectangular area is obtained for each clock. A minimum value/parallax detection circuit 131 detects/reserves the minimum difference absolute sum and the parallax in a retrieval range and outputs a result 115 at the end of retrieval.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

[0001] 1. Field of the Invention [0002] The present invention relates to a high-speed stereo matching apparatus for performing high-speed matching (stereo matching) between stereo images.

[0002]

2. Description of the Related Art Three-dimensional measurement using stereo images (hereinafter, referred to as "stereo")
The principle of “stereo image measurement” will be described with reference to FIG. In FIG. 9, (x, y, z) is used as coordinates representing a real space, and (X, Y) is used as coordinates representing a position on an image plane (an imaging plane of a camera). However, 2
Pedestal camera 8L, to distinguish _{8R, (X L, Y L} ) position as coordinates representing the image on the surface of the left camera using a right camera (X _R position as coordinates representing the image on the surface of, Y _R )
Is used. x-axis and the X _L-axis, x-axis and X _R-axis, y-axis and the Y _L axis,
It is assumed that the y axis and the Y _R axis are parallel to each other, and the z axis is both parallel to the optical axes of the two cameras. The origin of the real space coordinate system is set at the midpoint of the projection centers of the left and right cameras, and the distance between the projection centers is called a base line length, and the length is represented by 2a. Also, the distance (focal length) between the projection center and the image plane is f
Expressed by

Now, a point p in the real space is a point P _L on the left image plane.
(X _L , Y _L ) and projected onto a point P _R (X _R , Y _R ) on the right image plane. In stereo image measurement, P _L and P _R are determined on the image plane (stereo matching),
Based on the principle of triangulation, the real space coordinates (x, y,
z). Here, since the optical axes of the two cameras are on the same plane and the x-axis and the X-axis are parallel,
Y _L and Y _R have the same value. Coordinates X _L , Y _L ,
The relationship between X _R , Y _R and the coordinates x, y, z in the real space is

(Equation 1) Or,

(Equation 2) Is required.

[0004] _{Here, d = X L -X R (} 3) represents the parallax. (2) and X _L> X _R because it is a> 0 from the _{equation, Y L = Y R (4} ) is satisfied.

[0005] This means that a point on one image plane corresponding to another point on the other image plane is on the same scanning line and in the range of X _L > X _R. Therefore, a point on the other image corresponding to one point on one image can be found by examining the similarity of the image for a certain small area along a straight line where the corresponding point may exist.

Next, a method of evaluating similarity will be described. As an example of a similarity evaluation method, a method of examining a cross-correlation value between both images is described in "Image Processing Handbook" by Morio Onoe et al. A method of checking the cross-correlation value between the two images will be described with reference to FIG.

Now, it is assumed that a point (corresponding point) in the left image corresponding to a certain pixel 903 on the right image is determined. A rectangular small area 904 having a size of n × m pixels centering on a pixel 903 on the right image for which a corresponding point is to be determined is set, and the luminance value of the pixel inside the small area is Ι _R (i, j). on the other hand,
Let the luminance value of the pixel inside the rectangular small area 905 of size n × m pixels centered on the pixel on the left image that satisfies the condition of the expression (4) be と す る_L (i, j). ΜL, μR, σL2, σR are the average and variance of the luminance value for each small area.
Assuming that 2, the cross-correlation value between these small areas is given by the following equation.

(Equation 3) This value is calculated along a straight line (in this case, a scanning line) in which there is a possibility that a corresponding point exists, and a portion where this value becomes the maximum is defined as a corresponding point.

According to this method, the corresponding point can be determined on a pixel-by-pixel basis, and if the corresponding point is determined, the parallax for each pixel can be obtained from the coordinate position of the corresponding point using equation (3). However, the determination of the corresponding point requires an extremely large amount of calculation. This is because the calculation of the above expression is performed for all the pixels for determining the corresponding points over the entire range where the corresponding points may exist.

If the size of the small area for correlation calculation is reduced, the calculation speed can be increased, but the influence of image distortion and noise is increased, and the stability of corresponding point detection is deteriorated.
Conversely, if the size of the small area is increased, not only does it require a lot of calculation time, but also the change in the correlation value becomes too gradual, and the accuracy of corresponding point detection is reduced. The size of the small area is
It is necessary to set appropriately according to the properties of the target image.

As described above, the method of determining a corresponding point for each pixel requires a huge amount of calculation. Therefore, there is a method of dividing an image into blocks each having a certain size and determining a corresponding area for each block. As a method of obtaining the corresponding area between the left and right images for each block, there is, for example, Japanese Patent Application Laid-Open No. 5-114099.

The method described in the above publication will be described with reference to FIG. Now, based on the right image 1002, the right image is divided into blocks 1004 each having a size of nxm pixels as one unit, and a corresponding area is searched from the left image 1001 for each divided block to obtain parallax. As a similarity evaluation formula for determining the corresponding area,

(Equation 4) Is used. Here, Li and Ri are the left blocks 100, respectively.
3. Luminance values at the i-th pixel in the right block 1004. Since this evaluation expression does not involve an operation such as subtracting an average value as in Expression (5), the amount of calculation is smaller than that of Expression (5).

As described above, the processing of associating a stereo image requires an enormous amount of calculation. Therefore, in practical use, hardware (stereo matching circuit) that executes these calculations at high speed is required. . The above-mentioned Japanese Patent Application Laid-Open No. 5-114099 also discloses a specific configuration of a stereo matching circuit. This is 51
An image consisting of 2 (H) × 200 (V) pixels is divided into rectangular small areas (128 horizontal, 20 vertical) each consisting of 4 × 4 pixels, and stereo matching is performed for each rectangular small area, thereby real space is obtained. (For example, distance) information is measured. In the stereo matching circuit, 2
The degree of coincidence of the rectangular small area is evaluated once every clock. That is, the configuration is such that the operation of the expression (6) is executed once every two clocks. When the search range is 100 pixels, the matching of one rectangular small area is completed in about 200 clocks. In this device, a total of 128 × 20
Approximately 0.07
It can run in 6 seconds.

[0013]

As described above, the conventional stereo matching circuit has a structure in which the degree of coincidence of a rectangular small area is evaluated once every two clocks. Match about 0.076
It can be measured in seconds.

However, in order to further reduce the processing time in the above-described circuit configuration, there is no other way but to increase the frequency of the clock, so that there is a problem that the circuit configuration becomes complicated.

The present invention has been made in view of the above situation, and has a simple circuit configuration and one clock per clock.
It is possible to evaluate the degree of coincidence of rectangular small areas at the rate of
It is an object of the present invention to provide an excellent high-speed stereo matching device capable of performing stereo matching in about 1/2 time as compared with the conventional device.

[0016]

In order to achieve the above object, the present invention takes the following measures. The invention according to claim 1, wherein a synthesizing means for synthesizing pixel data at the same pixel position for each block on the stereo image, a plurality of memories provided corresponding to each pixel position in the vertical direction of the block, A configuration is provided that includes a memory control unit that writes synthesized data at a corresponding pixel position in the vertical direction to the memory, and a matching unit that simultaneously reads the synthesized data written in each memory and performs stereo matching.

With this configuration, a stereo image is synthesized and written into the corresponding memory for each horizontal line, so that it is possible to evaluate the degree of coincidence of a rectangular small area for each clock with a simple circuit configuration. Thus, there is an effect that the stereo matching can be executed at a very high speed.

According to a second aspect of the present invention, in the stereo matching apparatus according to the first aspect, when the number of horizontal pixels of the stereo image is N _H and the number of vertical pixels of the stereo image is N _V , the memory control means includes one block. The index representing the pixel position in the horizontal direction of the composite image composed of the composite data of
x ≦ N _H −1), the index indicating the pixel position in the vertical direction is y (0 ≦ y ≦ N _V −1), and k is 0 to ((N _V / n) −1).
When a positive integer of n is a constant (n is a constant), a composite pixel value of the following index y, y = n × k is stored in memory M ₀ and a composite pixel value of y = n × k + 1 is stored in memory M ₁ :: y = n × k + A configuration is employed in which the synthesized pixel value of (n-1) is written based on the memory M _n-1 .

With this configuration, the synthesized data obtained by synthesizing the stereo image can be written into the corresponding memory for each horizontal line, and the coincidence evaluation of the rectangular small area can be evaluated for each clock with a simple circuit configuration. It becomes possible.

According to a third aspect of the present invention, in the stereo matching apparatus according to the first or second aspect, the matching data is stored in the matching means for each image and stored in synchronism with a clock. And a plurality of matching circuits for detecting a horizontal correlation value between stereo images.

With this configuration, a horizontal correlation value between stereo images can be detected with one clock.
It is possible to evaluate the degree of coincidence of a rectangular small area for each clock.

[0022]

Embodiments of the present invention will be described below in detail with reference to the drawings.

In the following description, a case will be described as an example where the stereo matching method disclosed in Japanese Patent Laid-Open No. 5-114099 cited in the prior art is realized by the high-speed stereo matching circuit of the present invention. Accordingly, as shown in FIG. 6, the size of the input image 401 (left and right images) are 512 pixels in the horizontal direction pixel number N _H, 200 pixels in the vertical direction pixel number N _V, the number of bits per pixel A Is 8 bits. It is also assumed that the input image is divided by a rectangular small area 402 composed of 4 × 4 pixels, and stereo matching processing is performed on a total of 128 × 50 rectangular small areas.

FIG. 1 is a block diagram showing a high-speed stereo matching circuit according to an embodiment of the present invention.

In this high-speed stereo matching circuit, the number of pixels in the horizontal direction is N _H (= 512) and the number of pixels in the vertical direction is N
_V (= 200) Two systems of images having a pixel value of A (= 8) bits per pixel (left image 101 and right image 1
02) is input.

In the high-speed stereo matching circuit according to the present embodiment, memories M _{0 to} M _{3 in} which a composite image of the left image 101 and the right image 102 are written in accordance with rules described later.
119 to 122, each of the memories M _{0 to} M ₃ 119 to
4 systems of data 104 to 107 read from 122
Are input to one-dimensional matching circuits 123 to 126. Further, the output 108 of the one-dimensional matching circuits 123 to 126 is provided in the high-speed stereo matching circuit.
And 109, and 110 and 111, respectively, and the outputs 11 of the adders 128 and 129.
2, 113, and a minimum value / to detect a minimum sum of absolute differences from an output 114 of the adder 130.
A parallax detection circuit 131 and a controller 127 are provided.

FIG. 2 shows a one-dimensional matching circuit (123 to 1).
26) is a functional block. The one-dimensional matching circuit includes a reference data holding block 601 and a scan data holding block 602. Reference data holding block 6
01 denotes four D flip-flops with enable 60
The scan data holding block 602 includes four D flip-flops 607.
610 are connected in series. D with enable
The flip-flops 603 to 606 are connected to the controller 127.
Is controlled by a control signal from The outputs of the D flip-flops with enable 603 to 606 and D flip-flops 607 to 610 arranged in the same stage are input to absolute difference circuits 611 to 614. The difference absolute value circuits 611 and 612 are connected to one adder 615, and the remaining difference absolute value circuits 613 and 614 are connected to the other adder 616. These two adders 615, 61
6 is connected to the adder 617.

The operation of the high-speed stereo matching configured as described above will be described. First, at the same time as input,
At each pixel position corresponding to each of the left image 101 and the right image 102, for each pixel, the upper 8 bits are 16-bit pixels each having a pixel value of one (right image) and the lower 8 bits are pixel values of the other (left image). (Hereinafter referred to as “combined pixel value”). The composite pixel value is written into n corresponding memories M _{0 to} M ₃ (n = 4) in accordance with the rules described later. “N” takes the same value as the number of pixels (= 4) in the vertical direction of the rectangular small area. Writing control to the memory is performed by a control signal 116 from the controller 127. Note that an image composed of composite pixel values will be referred to as a composite image.
12, an image having 200 pixels in the vertical direction and a pixel value of 16 bits per pixel.

<Rule> The index indicating the pixel position in the horizontal direction of the composite image is x (0 ≦ x ≦ 511), the index indicating the pixel position in the vertical direction is y (0 ≦ y ≦ 199), and k is 0 to 0. When a positive integer of 49 is used, the combined pixel value of y = 4 × k is stored in the memory M ₀ y = 4 × k + 1 The stored pixel value of the memory M ₁ y = 4 × k + 2 is stored in the memory M ₂ y = the composite pixel value of 4 × k + 3 shall be written in the memory M _3.

[0030] Therefore, as shown in FIG. 3, the vertical direction index y (0 ≦ y ≦ 199) each of the composite image, y = 0, 4, 8, ..., composite pixel value of 196 memory M ₀ y = The combined pixel values of ₁ , 5, 9,... 197 are the memory M ₁ y = 2, 6, 10,.
The combined pixel values of ₂ y = 3, 7, 11,...
Will be written to ₃ . As an example, an example of data storage memory M ₀ F shown in FIG. 4, showing an example of data storage into the memory M ₁ in FIG.

As described above, the composite image is stored in the memory M _0.
After being written to ~M _3, the control signal 116 from the controller 127, data of the same address (pixel value) at the same time from the memory M ₀ ~M ₃ in 1 clock cycle x =
Are read in the order of 0, 1, 2, 3,...

The four systems of data 104 to 107 sequentially read out in synchronization with the clock are input to one-dimensional matching circuits 123 to 126, respectively, and the sum of absolute differences of four pixels in the horizontal direction is calculated in synchronization with the clock. The sum of absolute differences of the four pixels in the horizontal direction output from the one-dimensional matching circuits 123 to 126 is further added by adders 128 to 130, and finally the difference of all the pixels (4 × 4) in the rectangular small area is obtained. The sum of absolute values, that is, the calculation result 114 of the equation (6) is obtained every clock. The minimum value / parallax detection circuit 131 detects / holds the minimum value and the parallax at that time among the sum of absolute differences obtained for each clock over 100 pixels in the search range, and when the search ends, the result 11
5 is output.

As an example, the flow of the matching process will be described using the rectangular small area 501 at the upper left of the image shown in FIG. 7 as an example. y
= 0, y = 1, y = 2, and y = 3 are respectively stored in the memories M _{0 to} M _{3 as} x = 0,
Are simultaneously read out in the order of 1, 2, 3,... And input to the one-dimensional matching circuits 123 to 126, respectively.

FIG. 8 shows the operation timing of the one-dimensional matching circuit. In the one-dimensional matching circuit 123, y = 0
Are input in the order of x = 0, 1, 2, 3,.... The input 16 bits of pixel data are separated into right image data (upper 8 bits) and left image data (lower 8 bits).
1, the right image data is the scan data holding block 60
2 is input.

In the reference data holding block 601, (x, y) =
The pixel data of (0, 0) is stored in the D flip-flop 604 with enable, and the pixel data of (x, y) = (1, 0) is stored in the D flip-flop 605 with enable (x, y).
The controller 12 controls the pixel data of (y) = (2, 0) and the pixel data of (x, y) = (3, 0) in the D flip-flop 605 with enable.
7 is controlled by the control signal 117. The D flip-flops 603 to 605 with enable control signal 1
This is a flip-flop that latches input data in synchronization with the rise of the clock only when 17 is active.

On the other hand, the scan data holding block 602
Then, the left image data input in synchronization with the clock is D
Shift register 607 composed of flip-flops
610 shifts one pixel at a time for each clock.

The pixel data held in the reference data holding block 601 (x = 0 for y = 0, 1, 2, 3 pixels) and the scan data holding block 602 are shifted and updated successively for each clock. Pixel data (4 pixels)
Are respectively calculated by the difference absolute value circuits 611 to 614, and the sum of the difference absolute values of the four pixels is calculated by the adders 615 to 617, and the sum is output from the one-dimensional matching circuit 123. Is done.

Similarly, pixel data of y = 1 in the one-dimensional matching circuit 124, y = 2 in the one-dimensional matching circuit 125, and y = 3 in the one-dimensional matching circuit 126, x = 0, 1,. Are input in the order of 2, 3,... And processed similarly.

As described above, four lines (y = 0, 1,
The sums of the absolute differences 2 and 3) are calculated in parallel, and the evaluation of the degree of coincidence of the rectangular small area 501, that is, the calculation of the expression (6) is executed once per clock. This is executed over the entire search range (for example, 100 pixels), and the minimum value and the parallax giving the minimum value are calculated by the minimum value / parallax detection circuit 131.
Are detected and held, and output as a result 115 at the end of the search.

The control signal 118 to the minimum value / parallax detection circuit 131 is the current parallax (updated every clock) and the evaluation result output timing (evaluation end timing) for each rectangular small area.

As described above, according to the embodiment of the present invention, matching of one rectangular small area can be executed in about 100 clocks when the search range is 100 pixels. Compared with the circuit configuration described in 5-114099, when the same clock frequency is used, stereo matching can be performed in about 1/2 time (0.076 / 2 = 0.036 seconds).

In the above description, Japanese Patent Application Laid-Open No. 5-114
The stereo matching method described in Japanese Patent Application No. 099 (method of using the sum of absolute differences between left and right images for matching evaluation) has been described as an example of a case where the high-speed stereo matching circuit of the present invention is used. It is not limited to the stereo matching method.

[0043]

As described above, according to the present invention,
With a simple circuit configuration, it is possible to evaluate the degree of coincidence of a small rectangular area for each clock, and this has the effect that stereo matching can be performed very quickly.

[Brief description of the drawings]

FIG. 1 is a block diagram of a high-speed stereo matching circuit according to one embodiment of the present invention.

FIG. 2 is a block diagram of a one-dimensional matching circuit in the embodiment.

FIG. 3 is a diagram illustrating a memory configuration of a high-speed stereo matching circuit according to the embodiment.

[4] memory configuration diagram showing a specific example of a method of storing data into the memory M ₀ in the above embodiment.

[5] a memory configuration diagram showing a specific example of a method of storing data into the memory M ₁ in the above embodiment.

FIG. 6 is a diagram showing an example of a method for dividing an input image into small rectangular areas according to the embodiment.

FIG. 7 is an explanatory diagram of a matching process for each rectangular small area in the embodiment.

FIG. 8 is a diagram for explaining operation timing of a one-dimensional matching circuit in the high-speed stereo matching circuit of the present invention in the above embodiment.

FIG. 9 is a diagram illustrating the principle of a three-dimensional information measurement method using stereo images.

FIG. 10 is an explanatory diagram of a stereo matching method of three-dimensional information measurement using a stereo image.

FIG. 11 is an explanatory diagram of another stereo matching method of three-dimensional information measurement using a stereo image.

[Explanation of symbols]

 Reference numeral 101: left image, 102: right image, 103: composite image, 123 to 126: one-dimensional matching circuit 127: controller, 128 to 130: adder 131: minimum value / disparity detection circuit 401: input image, 402: rectangular small Area: 501: rectangular small area, 601: reference data holding block, 602: scan data holding block, 603 to 606: D flip-flop with enable, 607 to 610: D flip-flop, 611 to 614: absolute difference circuit, ~ 617 ... adder,

Claims (3)

[Claims] 1. A synthesizing means for synthesizing pixel data at the same pixel position for each block on a stereo image, a plurality of memories provided corresponding to each pixel position in a vertical direction of the block, And a matching unit for simultaneously reading out the combined data written in each of the memories and performing stereo matching. 2. The memory control means according to claim 1, wherein when the number of pixels in the horizontal direction of the stereo image is N _H and the number of pixels in the vertical direction is N _V , an index representing a horizontal pixel position of the composite image composed of one block of composite data. Is x (0 ≦ x ≦ N _H −1), and an index representing a pixel position in the vertical direction is y (0 ≦ y ≦ N _V −1).
When k is a positive integer of 0 to ((N _V / n) −1) (n is a constant), the composite pixel value of the following index y, y = n × k is stored in the memory M ₀ y = n × k + 1 of the composite pixel value memory _{M 1:: y = n ×} k + stereo matching according to claim 1, wherein the composite pixel value of (n-1) and writes the combined data based on the memory M _n-1 apparatus. 3. The matching means includes a plurality of matching circuits for storing combined data read from corresponding memories for each image and detecting a horizontal correlation value between stereo images in synchronization with a clock. The stereo matching device according to claim 1 or 2, wherein: