DE19625727C2 - Device for determining the disparity in a stereo image pair - Google Patents

Description

The invention relates to a device for determining the disparity in a stereo image pair, comprising two movable cameras mounted in a fixed geometry, two signal converters for generating a data format compatible with the subsequent disparity determination unit, and a disparity determination unit, the disparity determination unit comprising

• - two filters
• - A phase comparator, the phase difference Φ of the two filter signals,
• - Two amplitude determination units, which the accuracy check the determined stereo disparity values and at the same time a standardization information for Supply standardization step and gate,
• - A standardization level, which fluctuations in amplitude Balances resonators, and
• - A gate with adjustable threshold, which on the one hand discard or mark those standardized phase values, for which the amplitude determination method is not justifiable inaccuracy and on the other hand the reliable values can be evaluated.

It is known for the exclusive determination of the horizontal Disparity just a lot of pixels along one  Image line and thus a one-dimensional sequence of values examine (Kost, B .: construction of intermediate views for Multi-Viewpoint-3DTV-Systems in "TELEVISION AND CINEMA TECHNOLOGY, 42. Vintage, No. 2/1988, there page 69, left column, paragraph 3 and 4 and DE 41 10 951 A1, there page 2, lines 22-52).

The invention is based on the object of a device type mentioned in the beginning so as to derive from the determined Disparity using the known geometry of the Imaging device the spatial depth of the object points to be able to determine.

We achieve this by means of a device of the type mentioned at the outset, in which the filters are resonators, which receive the signals from the transducers and react with a decaying oscillation at the resonance frequency f _x .

The use of resonators for the deep / Disparity determination in stereo image pairs is from the state of the Technology not known.

In the dependent claims, expedient configurations of the Device according to the invention have been characterized.

The invention is described below with reference to the drawings for example:

Fig. 1 shows a block diagram of the device according to the invention.

Fig. 2 and 3 show details of the Disparitätseinheit for devices according to the invention.

The invention is intended to three-dimensional recognition processes, similar to how they are carried out by the human visual system, by means of technology, e.g. B. television cameras and special electronic image analysis telogics allow to imitate. For this purpose, two image recorders 1 , 2 are used, which scan images line by line and thereby generate an analog or digital data stream. With their two-dimensional recording surfaces, these image recorders are located in the same plane and are mechanically connected to one another there. The vertical arrangement of the transducers is designed so that lines with the same line number are at the same height. In the horizontal they are identified by a fixed distance d. There is a lens in front of each of the two sensors, which provides a sharp image of the area that is relevant for depth determination in the respective application. With this arrangement, a point that is infinitely distant excites exactly the same light-sensitive positions on the image recorders. The distance between these corresponding positions between the two image recorders is then equal to d. Objects that are closer to the arrangement are now displayed with a greater distance than d. This deviation is called stereo disparity.

Most methods for depth determination work in such a way that they each view a section of the same position and the same size from the left and right image and move these two partial images horizontally until they lie optimally one above the other (US PS 5383013, DE 34 25 946 A1). Two dimensions can be used to find this optimum. On the one hand, the correlation between the shifted partial images can be formed,

or the mean square of the error.

The correlation must be maximized and the mean square of errors minimized. As is common for this purpose the derivative is set to zero. However, problems arise here: this process does take place Extremes, but these may not be global extremes. To find the global maxima or minima, additional computational effort is required. In addition, these methods are very sensitive to contrast and differences in brightness of the two images. These problems can only be solved by a complex preprocessing processing can be compensated.

There are other methods that deal with the problem of stereo depth determination busy. However, these were never implemented technically. This is mainly due to the high computing overhead wall, which prevents use in real-time systems (e.g. robots).

Newer methods move away from the direct comparison of the images. Have been in the above procedure Windows used to examine parts of the images, here the images are folded with functions, which have a high locality in both frequency and local space (Gabor functions, DOGs or Difference of Gaussians). In this way, both the amplitude value and the phase value are obtained for each pixel. This can then be compared directly with that of the other picture, which is in a direct combination (Sanger, T. D .: Stereo Disparity Computation Using Gabor Filters. Biological Cyberne tics, 59: 405-418, 1988). A disadvantage is the very computationally intensive folding that is carried out for each point got to.

In summary, it can be said that the methods that are available have shortcomings. The Correlation-based methods work very quickly, but they are also very unreliable and are very sensitive compared to differences in holiness between the two input images. The phase-based process have to perform computationally intensive folds, which are difficult to carry out in real time. The The results of the phase-based procedures are very good. With the invention presented here the benefits of phase based methods are used and the calculations in Be possible in real time.

In the invention presented here, the two cameras K1 and K2 each deliver a serial data stream, which is the represents temporal sampling of the lines. If you look at the period in which exactly one line is scanned,  one can see that the spatial x-axis and the time axis have the same meaning for this period. Corresponding points of infinitely distant objects appear at the serial scanning Outputs of the cameras at the same time, whereas closer objects at different times appear in the two data streams. This time difference directly speaks to stereo disparity. If it is possible to measure the time difference and assign it correctly, then the depth of the belonging can be derived from it determine the object point. However, you can significantly simplify the timing required if you: both signal currents on a single frequency component d. H. on two sine waves of the same frequency reduced. In this case, it is sufficient to determine the phase difference between the two sine waves, which one is directly equivalent to the time difference.

For this purpose, the outputs of the two cameras K1 and K2 are each connected to a resonator which, in a first processing step, filters out only one frequency component from the serial data stream. The resonator has two free parameters: resonance frequency f _x and quality Q, which determine the spatial resolution and the signal-to-noise ratio. Sharp edges or jumps in brightness in the images are crucial for a precise evaluation of the disparity, since they have high frequency components and are therefore easy to locate. For this reason, the best possible resonance frequency f _{x is} chosen for the resonators. Edges or jumps in brightness thus stimulate the resonators to give a strong response, the quality Q roughly indicating the number of decaying vibrations. The two resonators have the transfer function in Laplace space:

The two complex conjugate poles s _∞ and s * _∞ ensure the actual resonance effect. The zero at s = 0 causes the direct component (DC component) to be completely filtered. The quality Q and the frequency f _x are related to the pole points as follows:

The filter result of both resonators then reaches a phase comparator 7 . Since both resonators signals are essentially (damped) sinusoidal oscillations, the phase difference between them can be determined according to the mathematical theorems for trigonometric functions by multiplying the two signals. Multiplication 11 produces two signal components that differ significantly in the spectrum. On the one hand you get a frequency component at 2f _x , on the other hand there is a DC component that directly represents the phase difference. With an appropriately selected low-pass filter 12 , the component at 2f _x can now be eliminated, so that only the DC component, which represents the phase difference, is retained after the low-pass filter filter. F _x / 2 has proven to be a suitable cut-off frequency for the low-pass filter. With a phase of 0 ° the output of the low pass then delivers the highest (positive) and at 180 ° the lowest (negative) value. The phase information can easily be converted back into disparity information. A phase of 0 ° corresponds to a disparity of 0 and a phase of 180 ° corresponds to a disparity of 1 / (2f _x ). This value also represents the resolution limit. Larger disparity values cannot be determined at a given frequency f _x .

The signal of the phase comparator 7 contains the disparity information, but this is still dependent on the amplitude of the resonators 6 . The signal must therefore be standardized. For this purpose, one calculates the envelopes of the resonator signals 8 , which reflect the amplitude of the resonance, and then divides the signal of the phase comparator by the envelopes 9 . The two envelopes are determined by squaring the resonator signals 13 and then low-pass filtering 14 in the same way as for the phase comparator 14 . This procedure means that the spectral distribution is identical in both signal paths (phase comparison 7 or envelope curve determination 8 ) (for this reason: rectification with subsequent low-pass filtering is forbidden, since in this case a spectral deformation occurs that causes different group delays for phase traffic or Envelope determination has resulted, which results in an additional undesirable Pha senshift between these two signals). In order to obtain the same signal level as before the squaring, the root 15 is then drawn from both envelope signals.

If the frequency component that is filtered out by the resonator is only weakly represented in the image and thus the resonator output delivers low values, then the signal behind the phase comparator is also low. For example, the resonator signals become zero when the image sweeps over a uniformly bright area. The disparity at these pixels is undefined and cannot be determined by any of the known methods. That is, it makes sense to mark such pixels as "unreliable". In the invention presented here, this is done by a gate 10 using the envelopes of the resonator signals 8 again . If the envelopes are smaller than an adjustable threshold, the gate 10 converts the output of the normalization stage into a marking value (flag) which signals the display 4 and storage device 5 that the disparity cannot be reliably determined at this pixel. If the envelopes are above the threshold, the output value of the normalization stage 9 is passed on unchanged to the subsequent display and storage device.

As an instructive example, a rectangular jump in uniform brightness (an "edge" with brightness difference 1) should be mentioned here briefly. If the jump in brightness occurs first in the left image and then in the right image, the following signal appears at the output of the phase comparator:

With:

Here a = const. and t _r - t _l corresponds to the disparity contained in the constant K. This is present in full immediately after the second holiness leap (ie for t = t _r ) and subsides through the term L. The product of the normalization signals 8 is identical to the term L; ie after normalization 9 , ie division, the constant K is obtained without the disturbing drop in amplitude. From this, the disparity t _r - t _l can be determined after using the arccos function. The constant K and thus the determined disparity value is stable for approximately Q resonator oscillation periods at the output and, after conversion, can be adopted as a depth value in the display and storage device.

The presented invention combines the advantages of the phase-based method with the real-time properties of causal filters. The resonator here corresponds to a filter, which is both in frequency and in Location is localized, d. H. the replacement in time-critical areas is possible. The invention is used primarily in robotics. Because - with correspondingly fast digital or analog filters - it is able to provide in-depth information tion in real time from two commercially available CCD cameras, it is suitable as an inexpensive and reliable starting point for a navigation system.

Claims (3)

1.Device for determining the disparity in a stereo image pair, comprising two, fixed geometry, movable cameras ( 1 ), two signal converters ( 2 ) for generating a data format compatible with the subsequent disparity determination unit and a disparity determination unit ( 3 ), the Disparity determination unit includes

• - two filters ( 6 )
• a phase comparator ( 7 ) which determines the phase difference of the two filter signals,
• two amplitude determination units ( 8 ), which check the accuracy of the determined stereo disparity values and at the same time provide standardization information for standardization step ( 9 ) and gate ( 10 ),
• - A normalization stage ( 9 ), which compensates for fluctuations in amplitude of the resonators, and
• a gate with an adjustable threshold ( 10 ) which on the one hand rejects or marks those standardized phase values for which the amplitude determination method results in an unacceptable inaccuracy and on the other hand the reliable values are fed into an evaluation, characterized in that the filters ( 6 ) are resonators, which receive the signals from the transducers ( 2 ) and react with a decaying oscillation at the resonance frequency f _x .

2. Device according to claim 1, characterized in that the phase comparison ( 7 ) by a multiplication ( 11 ) and subsequent low-pass filtering ( 12 ) with cut-off frequency is less than or equal to f _x , so that the phase difference is obtained as a direct component behind the low-pass filtering. 3. Device according to claim 1, characterized in that the amplitude determination units ( 8 )

• - a square member ( 13 ),
• - a low-pass filter ( 14 ) with a cut-off frequency ≦ f _x ,
• - And have a square root pulling member ( 15 ),

so that the envelopes (amplitudes) of the two resonator signals are obtained.