DE19851993A1 - 3-D stereo real-time sensor system - Google Patents

Description

The present invention relates to image processing device, in particular a device for stereoscopic pischen image processing, as well as a used Ko hardness detector.

The optical detection of the three-dimensional environment has been gaining in importance for some time. In particular with automated production processes and the same robots used must have two-dimensional coor also include depth information regarding a (to be working) object are available, for example wise to be able to control a robot accordingly.

As part of the "Ident Vision" fair 1996 in Stuttgart (DE) For example, a full-body scanner was presented, the using a variety of lasers and cameras the spatial Can capture the contour of an object. However, this is sy unsuitable for real-time applications.

Previously known implementations for determining depths information can be divided into different groups, which are based on different procedures.

Triangulation methods use one or more lines, either from a laser or a projection unit are mapped to the object in question. With a or several cameras, the object is then under a slightly different from the projection direction, another point of view. Based on the past  Known geometric relationships are then carried out by means of trigonometric calculations to determine the dreidi dimensional structure of the object.

A disadvantage of these methods, however, is that under order would evaluate an image for each projected line is what is extremely time consuming, so such System is not real-time capable.

In addition, it is particularly disadvantageous that the ever because the object to be measured is to be illuminated. This does under Laboratory conditions are not a problem, but will be in practice almost impossible when in situations such as traffic or under existing in telescopes or microscopes difficult real-world situations. Furthermore, La used as line light projectors special accident prevention regulations must be observed, so with such test setups no danger for human eye exists.

In a special triangulation process, a color fan projected onto the object in question. The one Color values recorded by the camera can then be clearly assigned to one Object point are assigned, the shift being individual color dots can be evaluated as depth information is. Although this method works in real time, it brings this procedure involves the restrictive requirement that the object and the background must be white so that the The method is not universally applicable.  

In contrast, runtime methods evaluate the runtime of light, Microwaves or a sound signal as a measurement signal out to draw conclusions about the spatial environment and the To draw depth information. One room at a time direction and a distance measurement. For example, the principle of a radar device or a medical ultrasound device on such a run timing. Runtime procedures, however, require one agile signal processing and are also of their own Depending on the surrounding medium, the spread tion properties of the measurement signal concerned.

Depth of field methods are also used, the however can only be applied if the object size and the opening of a lens in a given Are related to each other, such as in the Mi microscopy. In doing so, only the narrow area of the focal plane sharp on an image sensor pictured. The areas in front and behind are more or less unfocused. The one contained in the focal plane Image information can be processed using conventional image processing procedures are evaluated. A corresponding evaluation then returns a contour of the object. By multiple Moving the focal plane and subsequent evaluation can then create a contour map of the object which, however, is correspondingly complex.

Another approach that has only been pursued for a few years is based on the evaluation of stereoscopic images for Obtaining depth information similar to the spatial one  Seeing in humans, their replication through such Process is sought.

The phenomenon of spatial vision, which is briefly described below in Relation to humans is generally based on the perception of the visible environment by means of the two human eyes, the due to the eye relief two perceived images (right eye or image, lin kes eye or picture) with a slightly different view angles are perceived. Since the visual axes of the eyes et what converge, their visual axes meet at one Point of the environment under consideration, being from a fixation this point is spoken through the eyes. The picture this point falls on a visual pit of the network skin. Each neighboring point in the visual field will then open up a location of the retina that projects somewhat from the center the sharpest sight. Generally is this distance is different in both eyes, with the Distance as a function of the spatial depth of the considered Point in the visual field relative to the fixation point vari iert. These distance deviations are called binocular dis parities, hereinafter referred to as "disparity ten "are designated.

The real problem with stereo viewing is besides that Evaluation of the given trigonometric counter to find out which pixel the rea len world or the seen object, which in the first the ste reoscopic images occurs, which pixel in the second corresponds to the stereoscopic images. Expressed differently, it is important to find out in relation to the human eye  which pixel on the left retina which pixel on the right retina.

For spatial vision or 3D stereo perception then it is necessary from the disparities to the dreidi dimensional structure of the perceived image conclude. In other words, from identified disparities can to the depths contained in the perceived image information will be closed.

So far, however, stereo methods are available stable workstation computers (PCs or so-called "Workstations") instructed, whereby to determine the ge wanted depth information from a single stereo pair of images, for example using a commercial union workstation (with a clock frequency of et wa 90 MHz), a time in the range of about 3 to 5 minutes is required.

However, if you try to include such procedures in special Re Outsourcing the chenwerke exceeds the development and Hardware expenditure very quickly economically justifiable Limits. The market launch of such systems failed So far either because the processing area is too low speed or a too high system price.

Stereo procedures strive to image each other accordingly points between a first (left) and a second (right) Find stereo image, then from the relative ver shift of such pixels with known camera position the distance of the points can be calculated.  

Among the conventional rech One can essentially implement stereo methods differentiate between so-called feature-based Stereo method, intensity based stereo method and phase-based stereo process.

In the case of feature-based stereo procedures, before the correction spondenz search for certain characteristics (e.g. edges or inten sitätsmaxima) extracted from the image data, which for stabi ler than the raw image intensities. Derar current procedures are stable if only a few stable notes male (such as object edges) can be extracted. You are too fast, since images are only processed at the locations those on which characteristics were recorded. this leads to a noticeable data reduction.

However, this can only cause disparities at the points on which the selected characteristics were recorded the. All other parts of the picture must be interpolated be what additionally time-consuming computing processes makes maneuverable. The closer the characteristics are, the more difficult It is becoming more difficult to match them up. The final Assignment can only be based on complicated Ver drive being hit, resulting in decreased processing processing speed leads.

The publication WO 94/18797 A1 describes, for example such a feature-based method and an ent speaking device.  

Intensity-based stereo processes work directly with the supplied by the stereo cameras or the image recorders image brightness. Therefore, they are related to each other appropriate stable algorithms set that require a corresponding amount of computing time. Under the intensity-based method can further differ between correlation-based methods and Dynamic programming method.

The former strive to have a small image area of the left Picture in the right picture. To do this, method to maximize correlation or Minimizing the difference used. If the picture cuts that are large enough are achieved with these Process stable and dense disparity cards, one Real time processing using special complex Hardware is conceivable. The raw disparity cards included typically, however, only shifts in the area of Ge accuracy of a picture element (pixel) so that the resolution is limited. Only by using interpolation methods or else iterative methods can increase the accuracy of the However, disparities on sub-pixel precise disparities be tested.

Such correlation-based methods are, for example in the publication DE 34 25 946 A1 and the article "A stereovision system for a planetary rover: calibration, correlation, registration, and fusion "by Z. Zhang, in: Machine Vision and Applications, No. 10, 1997, pp. 27-34 be wrote.  

When using dynamic programming, ver seeks to locally minimize the difference, this loka Then minimization to the ge using suitable procedures entire picture is expanded. Typically also calculate these procedures disparities to just one pixel, which is too imprecise for numerous applications.

In summary it can be said that intensity-based Procedure a dense disparity card (a dispari value for almost every pixel), the minimization the measure of error as in the method of minimizing the Difference, however, is time consuming and therefore not for Real-time applications is suitable. The reduction of he required computing time can be achieved by using hierarchical Procedures can be reduced, but only through use complicated and not parallelizable algorithmic Structures for which a hardware solution is at least extreme becomes complex and therefore costly.

Phase-based stereo processes are also an issue current investigations. These procedures extract from a local Fourier phase (usually through Filtering using Gabor functions), and then try either, these Fourier phase images (for the right or left picture) (e.g. described by Weng) in agreement or to use Fourier's ver shift theorem to calculate the local shift (e.g. described by Sanger, Fleet & Jepson). The ex traction of the local Fourier phase corresponds to effek tiv a local contrast compensation, which many through Variations in brightness caused errors in other stereo  procedures can be reduced. This phase-based stereo processes work with sub-pixel accuracy and are also true time applications can be implemented. However, to generate hierarchical procedures which are usually special for avoiding errors Catch exceptional situations (see Fleet). This makes the algorithms used complicated and one Hardware implementation complex and costly.

It is therefore an object of the present invention to Specify image processing device with the stereosko pical images can be processed in real time, and that can be realized with simple means and little effort ren is. It is also an object of the present invention to use a coherence detector used for this give.

According to the invention this object is achieved by a Kohä reference detector with a large number of n signal inputs which have a respective input signal value, one he Most sorting device for sorting the n fed Input signal values according to their value and parallel output of the n sorted values, a variety of n-1 subtractors for subtracting each two neighboring sorted values, a second Sor animal device for sorting the obtained n-1 differences zen regarding their amount and the issue of the small most differential amount, a selector based on of the smallest difference ascertained outputs the corresponding pair of values of the input signal values, and  an output device to which the output pair of values is performed and which outputs its mean value. Alternatively, this task is accomplished through coherence detector with a large number of n signal inputs at which a respective input signal value is present, a window limit tion device for outputting those of the n one output signals whose input signal value is within a set, finite window value range [-x,. . ., 0, . . ., + x] with m + 1 values at a distance of 2x / m, one Rounding device for rounding the window limit output device on the respective closest to the m + 1 values of the preset window values area, an addressing control device, and a folding device, the folding device one of the number of values in the window value range has a corresponding number of m + 1 signal inputs, and the respective kth input by the addressing control Erungseinrichtung those of the n signals supplied whose value is -x + (k-1) .2x / m, which is therefore a corresponding k-th of m + 1 radiation emission elements control, the radiation thus emitted by one of the number of values in the window value range speaking variety of m + 1 radiation receiving elements is detected, and an evaluation device that the address k of that of the m + 1 radiation receiving elements is determined, which detects the greatest radiation intensity, and the value k outputs this address to an output connection, the output value the result of the convolution of the convolution device supplied signals.  

The task is also solved by image processing processing device with displacement devices for the opposite side term shift of these respectively supplied image data a first picture and one to this under another Angle taken second image, the Image data in parallel on the shifting devices be tapped and in pairs from a subsequent disp Rity detection device are supplied; being the Disparity detection device for each of the supplied Image data pairs using a respective disparity element mentes a spatial depth information for each because of the disparity value representing the image data pair telt, and the determined disparity values of a coherence detector tion device feeds, the output Dispa value for each shift of the image data and the associated spatial depth information represents.

With this construction according to the invention, they can be fed th image signals are processed at speed, as they are supplied by image recorders. So that is Image data processing for obtaining depth information possible with little effort and in real time.

This high processing speed (real time) of the Sy stems is a major advantage over the state of the Technology. With a frame rate of, for example, 25 frames per second, the system on the part of the image sensor and are processed by it in real time, is the system or the method implemented with it the arrangements known from the prior art or  Procedure at least by a speed factor of 100 think.

Furthermore, the volume of the object to be measured is against no longer limited over known systems. Especially is the measuring range of the image processing according to the invention device only by limiting the used from visual optics determined, and all optical attachments such as microscopes, telephoto or wide-angle lenses can in Connection with the image sensors can be used, provided they are adapted to them. In particular, it is with him system according to the invention possible, a measurement object without Markie evaluation aids that are used in systems according to the status of technology are often required.

The possibility of a measurement object without marking aids to evaluate there is still no need for one to provide the necessary projection unit. Hence the Device according to the invention more compact and lighter, wherein handling is easier and more versatile opportunities arise.

In addition, the device according to the invention also does not apply pretreatment of the test object itself, for example covering the test object with light powder to remove it reflection and highlights on the object surface area before measuring and removing the powder after he following measurement. By saving such work gears, the device according to the invention works in time and cheaper.  

It is also advantageous with the device according to the invention fortunately possible to switch on any image signal sources conclude. That is, the device according to the invention is not limited to the area of visible light, but for example, the evaluation can also be carried out for image signals Infrared range.

It is particularly the case with the coherence detectors according to the invention special possible from a variety of input signals determine those signals in real time, their values are most similar to each other, and this "most similar" value then continue to process.

Advantageous developments of the invention are in the Un claims specified.

The present invention is hereinafter based on Aus management examples with reference to the attached drawing described in more detail. Show:

Figure 1A is a general block diagram of the inventive system SEN.

Fig. 1B is a schematic representation of a stereoscopic image capture for explanation and definition of terms;

Fig. 2 is a block diagram showing image signal shifting means and disparity determining means be details;

FIG. 3A is a functional diagram of a single gradient-based or differential Disparitätselements;

. Fig. 3B is a block diagram for the hardware implementation of Figure 3A in accordance with individual Dispari functionally described tätselements;

FIG. 4A is a block diagram of a first Ausführungsbei performance of a coherent detector;

FIG. 4B is a schematic representation of a Ausführungsva riante a coherent detector;

Fig. 4C shows a second embodiment of a coherence detector, the operation of which is based on the convolution principle;

Fig. 4D details of the implementation of the folding device in the coherence detector shown in Fig. 4C;

Figure 5 is a graph showing the line for all the picture elements of an image representing the ermittel th of different Disparitätselementen disparities, from which then the actual or true disparity is determined in the course of the image line by means of a coherent detector in real time.

Fig. 6 shows an example of two supplied as an image signal current images of a first and second (or left and right) image receiver, a presentation of the depth information obtained with the device OF INVENTION to the invention in the form of a so-called disparity map.

Fig. 1A represents the stereoscopic image processing device according to the invention for detecting three-dimensional objects in real time, a schematic block diagram illustrates.

The image processing device has a sensor head which comprises two image recorders, one image pickup  CCDL for the (first) left image and an image sensor CCDR for the (second) right picture. One of these pictures will thereby for the subsequent processing as a reference image Are defined. That is, determined disparity results to create a disparity map the image data of the Re be assigned to the reference image.

In the image sensors may be, for example, a pair of cameras with CCD imagers act to d at a predetermined known distance from each other and therefore are arranged slightly different images of the same object record (see. Fig. 1B and Fig. 6). In particular, the image recorders can record moving images and guide them to the device for subsequent real-time processing. By means of an image pickup synchronization device SYNC, the recordings of the image pickup devices CCDL, CCDR are synchronized in time, so that the left and right images are recorded synchronously with one another.

Depending on the distance of the object from the image recorders or Cameras show the recorded pixels more easily than Disparities called - shifts. This means, that the same object in different places in the first or second picture appears.

By means of a subsequently as disparity detector DD signed facility and one as disparity coherence detector KD designated device (both below are described in detail) may be actual Disparities or shifts are determined and then taken into account by trigonometric calculations the known distance d between the image recorders CCDL, CCDR is the spatial distance of the respective object are calculated, thus the depth information he is looking for will hold.

The depth in calculated from two stereo moving images formation can be called Vi and the third dimension (e.g. third image) is shown as a grayscale image or color image become. The real-time processing of the system is ins special through the serial processing the through the Image sensor CCDL, CCDR conditionally delivered image data.

With the image sensor synchronization device SYNC ver bound is a clock generator CLK, the clock signals for Control and synchronization of other components of the Sy stems generated.

The recorded image signals are used by the image recorders CCDL, CCDR are assigned to these preprocessing processes directions VVL, VVR supplied. In this preprocessing devices VVL, VVR, the brightness is compensated for two stereo images before subsequent processing. This brightness preprocessing is advantageous because of due to slightly different image recording angle of the image sensors CCDL, CCDR so-called highlights and reflexion on (reflective) surfaces of the recorded objects can occur in the subsequent processing the image data to record the disparities to errors  can lead, so that the disparity values become unstable could.

In the case of processing color images in the frame preprocessing by preprocessing directions a decomposition of the respective (first and two color image data in respective color image components (e.g. in the primary colors red, green, blue (RGB) or in Luminance and the two color difference components) It is also possible to use the first and second Image data to different preprocessing pull.

After the preprocessing of the image data by the The image data are preprocessing devices VVL, VVR of the first and second (or left and right) picture each because facilities VSEL, VSER for mutual displacement Exercise the left and right image data in relation to each other fed. More specifically, that means two moves each facilities VSEL, VSER form a displacement unit where with a shift current of image data each "Right" and "Left" per shifting unit. Depending on the application, a displacement unit is for to provide any kind of image data. That is, in the case of one Preprocessing by splitting into the primary colors R, G and B three displacement units can be provided, and one for each primary color.

The disparities of image data shifted relative to one another in this way are then for the respective pairs of image data by the Dispari DD detector determined and get a respective set  The disparity values for the image data pairs are determined by the Disparity coherence detector KD evaluated.

This coherence detection with regard to the disparities obtained is based on the selection or determination of that of the disparity values obtained which is approximately identical to at least one further of the disparity values obtained. In other words, the disparity value is determined as the actual or true disparity value, which is approximately identical to at least one further disparity value (ie lies in a predetermined interval around this) or is identical to at least one further disparity value. Alternatively, the disparity value that represents the convolution result of the disparity values obtained can be determined as the actual disparity value. Fig. 5 illustrates this principle based on data from at play, seven Disparitätselementen a disparity detector.

He on the part of the disparity coherence detector KD average actual disparity values are then from this to a facility NB for post-processing image data spent. This facility NB prepares the disparity upgrade so that this via an output connection DV egg ner optionally connectable output device such as Example of a display device (not shown) can be performed and by this as gray value or Color image can be reproduced. For better representation the disparity values are based on the image data processing also eliminating any marginal effects as well as a virtual limit.  

An external loading is not shown in the figure itself service unit for configuring the system by a Workstation computer with a suitable interface to the Sy stem can be implemented.

Fig. 1B illustratively shows schematically the principle of a ste reoskopischen image capture. CCDL, CCDR are the image recorders arranged at a distance d from one another, on the respective light-sensitive detection section of which an object to be imaged or point P1, P2 to be imaged is imaged via an upstream lens. The distance d is defined as the distance between the optical axes of the lenses, which are shown as vertical dash-dotted lines. The point P2 is further away in the direction of the optical axis than the point P1.

As can be seen from the schematic representation, the images of points P1, P2 on the acquisition the image sensor cut depending on the distance differently. Assuming that the imager best of a certain number of picture elements hen, point P1 is at one of the image recorders CCDL Image element j mapped, while point P1 in the image sensor CCDR is shown on a picture element j + y. The Image elements j and j + y thus correspond in terms of their information content. Because of this shift (Disparity) y can be from the present trigonometric Relationships to the distance of the point P1 closed become. The distance from point P1 is now a reference  distance defines what is the basic measuring range of the device can be viewed. Then point P1 is in egg ner reference plane defined as the fixation plane, for which a (relative) disparity of zero is assumed. Different expressed, is the determined disparity on the fixati related level. By the appropriate selection of a previous shift (e.g. y1, or y2) between those to be evaluated The first and second image data can thus be the fixation plane determined and the basic measuring range can be determined. The means before the image data of the first and second image the displacement devices VSEL, VSER are fed, they are already pre-shifting in relation to each other exercise amount (e.g. y2) postponed.

The fixation level in an interval of disparity values [-x,. . ., 0,. . ., + x] surrounds the so-called fusion Area. The fusion area corresponds to a "Schär fentiefbereich ". The (relative) disparity values in the Fusi ons area due to the shifting on the part directions VSEL, VSER additional conditioned shift of the image data determined. The larger the additional ver shift is evaluated, the larger the Fusion area and thus the "depth of field" around the Fixation level. The additionally achievable shift stands both in connection with the number n of displacement elements of Ver sliding devices, as well as the type of their control.

Fig. 2 of the drawing now shows details of the image data shifter VSEL for image data of the (first) left image, the image data shifter VSER for image data of the (second) right image, which together form a shift unit, and the disparity detector DD as a disparity determination means. The components according to FIG. 2 are not fixed to a specific implementation in analog or digital technology and are first described in a general form, regardless of the actual form of implementation.

The VVL, VVR delivered image data will shift the image data facilities VSEL, VSER supplied. This supply takes place as mentioned above, with a pre-shift according to the desired basic measuring range. In the shown There is only one case to simplify the illustration Shift unit for a type of image data, for example shown for one of the primary colors R, G, B.

The respective image data are fed to these shifting devices of the shifting unit in each case serially and synchronously with one another. Each of the shifting devices VSEL, VSER consists of a plurality of shifting elements ΔX _L1 to ΔX _Ln or ΔX _R1 to ΔX _Rn . These elements each cause an additional displacement of the supplied picture elements by a small distance in the x-direction, ie in the direction of the main scanning direction in line-by-line image pickups. ( Fig. 2 shows a case for n = 7.)
Starting with the first (but taking into account the pre-shift) picture element of the serial picture data streams of the left or right picture (e.g. picture element of the upper left corner of the respective picture), the picture elements (pixels) are shifted against each other. The evaluation of the picture elements with regard to the existing disparity is then carried out in pairs. More precisely, a pairwise evaluation of the disparity takes place based on the shifted image data for pairs (ΔX _L1 , ΔX _Rn ), (ΔX _L2 , ΔX _Rn-1 ),. . . , (ΔX _Ln-1 , DX _R2 ), (ΔX _Ln , ΔX _R1 ). The amount ΔX of the shift can be set to a value corresponding to the length of a picture element. If a higher reliability ("stability" of the implemented method), and therefore accuracy is desired, this shift can also be set to a value that is smaller than the length of a picture element, for example half the length of a picture element (pixel ). However, this requires a higher number of elements of the disparity detector described below if the fusion area is to remain constant.

In the case of shifting realized as an analog circuit directions VSEL, VSER each consist of a ket te or cascade of delay elements (delay elements) with appropriate signal taps. The shift amount in the x direction then corresponds to the term and can image selected element by element or in fractions of picture elements be.

In the case of shifts realized as a digital circuit facilities VSEL, VSER consists of the respective unit shift registers connected in series, to which an ent speaking clock signal from the clock generator CLK (if necessary. via an intermediary (not shown) Fre  quenz divider). But then it's him first required the analog image data using an Ana convert log / digital converter to digital image data. A such analog / digital converter can be before or after respective preprocessing device VVL, VVR in the signal be provided depending on whether the preprocessing still analog or digital.

In the case of the digital shift registers, the shifting elements ΔX _Li , ΔX _{Ri of} the respective shifting device for the shifting by picture element take place in accordance with the clock with which the digital picture data are supplied. If sub-pixel accuracy is required, the already digitized data must be fed to a special filter. For example, a shift corresponding to half a picture element is achieved by subtracting two successive picture elements by means of a corresponding filter and taking the value obtained as the signal value of the picture element at the position shifted by half a picture element. Such filtering then corresponds essentially to a dithering of adjacent picture elements.

The control can be carried out simultaneously for both Verschiebeinrichtun gene VSEL, VSER, or alternatively in phase, depending on which additional displacement range ("depth of field") is desired. Based on the example shown in FIG. 2 with n = 7, this means that with simultaneous activation, image data pairs are evaluated in each case, which are shifted by ± 6, ± 4, ± 2, and 0 against each other. With antiphase control, shifts by -7, -5, -3, are in the same example. . ., +5 as well as -6, -4, -2,. . ., +6 possible. Due to the larger additional shift, an expanded depth of field can be expected. Although FIG. 2 shows an example for n = 7, the invention is not limited to odd values for n, but an even number of shift elements can also be used.

The above-mentioned image data pairs (ΔX _L1 , ΔX _Rn ), (ΔX _L2 , ΔX _Rn-1 ),. . ., (ΔX _Ln-1 , ΔX _R2 ), (ΔX _Ln , ΔX _R1 ) are then each one of n elements EST ₁ ,. . ., EST _{n of} the disparity detector DD fed for evaluation in order to determine the corresponding disparity for the respective image data pair. It should be noted that due to the simplified representation according to FIG. 2, only one disparity detector DD is shown. However, if there are several displacement units (e.g. for R, G, B image data), each of the displacement units must be assigned a corresponding disparity detector.

Different types are conceivable for respective diparity elements EST _{i of} a disparity detector DD. For example, the evaluation can be gradient-based, tensor-based, energy-based or carried out according to Reichard's diffraction detection model. These evaluation principles are adequately described in the specialist literature (e.g. at Barron, Fleet & Jepson, Adelson & Berger) and will not be discussed in detail here.

The disparity values output by the disparity elements EST _i (1 ≦ i ≦ n) are subsequently fed to the coherence detector KD, which determines the actual disparity value from the supplied values. In particular in the case of several disparity detectors DD, all outputs of the respective disparity detectors are fed to a subsequent coherence detector KD.

Fig. 3A now shows in detail a functional diagram of a disparity element EST _i, in which the evaluation is based on the gradient-based approach.

Image data ΔX _Li , ΔX _{Rn + 1-i} (1 ≦ i ≦ n) are supplied to the respective disparity element EST _i . These are then fed to a unit performing a pixel-by-pixel (pixel-by-pixel) addition (reference symbol "+"), then to a multiplication unit designated by the reference symbol "1/2", which carries out a multiplication of the added pixel values by a factor of 1/2, so that averaging takes place with respect to the pair of image data supplied. The image data processed in this way then pass through a derivative filter DIFF. A simple derivative filter DIFF can, for example, have filter coefficients as given in Table 1 below.

Table 1

Filter coefficients derivative filter DIFF

The data values output by this filter are subjected to pixel-by-pixel squaring in a unit denoted by the reference symbol “x ² ”, the values obtained in this way then being fed to an averaging filter AVE.

On the other hand, the data output by the filter DIFF is fed to a multiplication unit (reference symbol "*"), where it is pixel-by-pixel with the difference in data ΔX _Li , ΔX _{Rn + 1-i} obtained pixel by pixel by means of a subtraction unit (reference symbol "-") be multiplied. The values obtained in this way, which are available at the output of the multiplication unit, are likewise fed to a further averaging filter AVE.

Filter coefficients can be used for both AVE averaging filters be chosen as they are for. B. in the following Table 2 are reproduced.

Table 2

Filter coefficient averaging filter AVE

The data values obtained at the outputs of both averaging filters are finally fed to a division unit (reference symbol "÷") which outputs at its output the disparity value determined in each case as the output signal of the respective disparity element EST _i .

Since for both filters only filtering in x-direction are required, which means filtering in seri corresponds to the image data course, such a dis parity element completely implemented in analog technology become. However, a digital implementation is also possible.

The disparity values determined in this way, which are output at the outputs of the disparity elements EST _i , are then fed to the disparity coherence detector KD.

FIG. 3B shows a block diagram for hardware implementation of an individual disparity element EST _i , which was described functionally above with reference to FIG. 3A. A disparity element EST consists of the synchronization of the entire circuit with a clock signal CLOCK derived from the clock generator CLK clocked shift registers SR1 to SR4 for intermediate storage of individual picture elements, and arithmetic logic circuits (mean value generator 1 : "Σ / 2", subtractor 2 : " - ", multiplication devices 4 , 5 : MUL, summers 3 , 6 , 7 :" Σ ", division device 8 : DIV). The filter designated DIFF in FIG. 3A is now implemented by the shift register SR1, a coefficient multiplication part K (K1, K2, K3) and the summer 3 . Furthermore, the filters designated AVE in FIG. 3A are now by the shift register SR3, a coefficient multiplication part JA (coefficients J1, J2, J3, J4, J5) and the summer 6 , or by the shift register SR4, a coefficient multiplication part JB (coefficients realized J1, J2, J3, J4, J5) and the summer 7 . The disparity is given at the output of the division device 8 . The circuit arrangement shown in the block diagram fully realizes the operation and mode of operation of a disparity element EST specified and described in connection with the functional circuit diagram according to FIG. 3A.

Fig. 4A shows a block diagram for a gate or Kohärenzdetek disparity coherent detector KD, being shown as an example of a coherent detector KD with only four inputs according to a first embodiment. For the case shown in FIG. 2, the circuit would have to be expanded accordingly to seven inputs or generally to n inputs. If several (z) displacement units with respectively assigned disparity detectors DD with n EST elements each are provided, the circuit would have to be expanded to zn inputs.

The data values of the determined disparities supplied by the (not shown) EST elements EST _i and applied to the inputs E1 to E4 are fed to a sorting device S1, which outputs the data values sorted at the outputs A to D according to the size of the values . In the case shown, output A has the smallest value, output D has the largest value. Such sorting devices can be implemented with logic gates consisting of comparators (comparators) and multiplexers and are not explained in detail here. Each of the outputs A to D is fed to a multiplexer MUX1, while two of the outputs are also fed to a subtractor ("-") in order to determine the differences BA, CB and DC of the corresponding values. The differences are fed to a further sorting device S2 which, in accordance with the first device S1, outputs the values of the differences in terms of amount at the output. In the present case, only the smallest result (at the output """) is to be further processed, which represents the pair of values of the disparities that are the" most "adjacent to each other, that is, the smallest deviation from each other. The result of the sorting device S2, that is to say the value at the output ““ ”, is then fed to a device that checks for equality“ = ”, which compares the result obtained with the differences fed to the sorter S2 in order to determine which of the input values represents the result. This variant represents the simpler variant in terms of circuit technology, because this could also be determined from the internal decision states of the sorter S2. The result obtained from the device checking for equality in turn is used to control the multiplexer MUX1 in order to supply the two output signals of the sorter S1 belonging to the difference to a mean value generator (reference symbol “Σ / 2”) which adds the two values for determining the mean value and then divided by two. This mean value then represents the result KE, that is to say the coherence value of the multiplicity of determined disparity values.

A variant of the circuit described above be is to determine the differences with a smolder to compare lenwert, whereby only the candidates or values  to be processed below the certain Threshold, or within a threshold certain window value range (e.g. according to the Fusion area). A downstream rear derailleur then finds the largest cluster of neighboring candidates. Both variants are possible and can be based on each respective use case the better results deliver.

FIG. 4B shows an embodiment variant of the previously beschrie surrounded KD coherent detector, wherein the coherent detector adjacent to output signals of the elements other EST data leads are fed. This is particularly useful when the image data is interlaced or "interlaced", and the entire image information of a full frame z. B. composed of two fields. Then the disparity values are determined for image data of a field and the corresponding coherence value of the disparities for image points of a field is determined. For each field, the values KE for the pixels of the field are buffered in a memory device MEM (with a storage capacity for one field), and are read out during the next field period and supplied to the coherence detector KD at an additional input terminal. The internal structure of the coherence detector KD is essentially identical to that described in connection with FIG. 4A, with the difference that it must be able to process an additional (the feedback) signal, for which different variants are possible. In particular, the feedback coherence disparity value can already be supplied to the sorter S1 and taken into account there, or can only influence the averaging later. In the latter case, it would be fed to the averager (reference symbol "Σ / 2"). The circuit shown in Fig. 4A is then to be modified accordingly.

In this way the "true" disparity value can then (Coherence disparity value) for a full frame under touch view of both fields can be obtained.

Fig. 4C shows a second embodiment of a coherence detector, the operation of which is based on the convolution principle, and Fig. 4D shows details of the realization of the convolution device in the coherence detector shown in Fig. 4C.

For a better understanding, the principle of Folding for the present case is explained in general the. That value is to be supplied with the coherence detector ter input values are determined or determined, the possible is as close as possible to one or more others, like the previous one already running. When folding, the obtained values are folded together to determine which value in the considered value range is the convolution value or convolution result and is therefore the value that is as close as possible to one or more others (the means the differences between the convolution value and the surrounding ones are less than or equal to a predetermined one Value). This can be clearly explained so that everyone of the values to be folded together to the respective value symmetrical, suitable interval is superimposed. At appropriate choice of interval (the same for each value  is), there is an overlay of some of the In tervalle. The range in the range of values under consideration, in which most of the individual intervals overlap (according to the intersection of the largest number of the individual quantities) or the area that the has the largest area share of the overlaid intervals ("largest integral") then represents the convolution result.

Fig. 4C indicates a hardware solution that performs such a folding operation, which was previously only time-consuming for example by means of digital signal processors, in real time.

In this case, the input signal values supplied by n disparity elements EST _i (1 ← i ← n) to a disparity detection device DD at respective signal inputs are fed next to a window limiting device FB. The window limiting device FB outputs those of the n input signals whose input signal value lies within a predetermined window value range. This is preferably a finite window value range in the interval [-x,. . ., 0,. . ., + x] with m + 1 values at a distance of 2x / m, the value range limits ± x being determined by the disparity values of the fusion area around the fixation plane (with a disparity of zero).

Those of the n input signals whose signal value is not in this window wet area is on the corresponding Exit the window limiting device in a state (e.g. a high impedance state) that indicates this  and further processing of the corresponding signals prevents.

Only those of the input signals with an input signal value within the window value range are further processed by a subsequent rounding device R. The rounding device R rounds the values output by the window limiting device FB to the nearest of the m + 1 values of the preset window value range. Thus, the (infinite) multiplicity of values output by the respective disparity elements EST _{i is} mapped to a finite number of m + 1 values, which are subjected to the subsequent convolution. The accuracy of the folding can be increased by increasing the number m.

So now lie after the window boundary by the Window limiting device FB and the rounding through the Rounding device R maximum n signals (if all on output signal values within the window values range) with one of m + 1 signal before.

These are via an addressing control device AA fed to a folding device F. This corresponds to corresponding to a respective value of the signals controlled signal input of the folding device.

That is, those of the (maximum n) signals supplied to the addressing control device AA, the value of which is -x + (k-1) .2x / m, thus become a corresponding k-th of m + 1 signal inputs E _{k of} the folding device F supplied, with a respective signal input E _{k of} the processing device F a radiation emission element is assigned, which is controlled by the supplied signals. The respective radiation emission elements are designed in such a way that the more radiation is emitted, the more of the supplied signals are used to control the respective radiation emission element. In a different and simplified way, a radiation emission element k emits twice as much radiation if two signals have the value -x + (k-1) .2x / m than if only one signal with the corresponding value for controlling the kth radiation mission element.

The radiation thus emitted is emitted by one of the Number of values corresponding to the window value range Variety of m + 1 radiation receiving elements detected. Dar an evaluation device AW is also provided, which is the address k of that of the m + 1 radiation receiver elements determined that the greatest radiation intensity sums up. The value k of this address is given at an output conclusion KE issued, the output value the result nis the folding of the Si fed to the folding device gnale represents.

Fig. 4D illustrates details of the folding device F. In the illustrated radiation emitting elements are, for example, light emitting diode LED _1. . ., LED _k,. . ., LED _{m + 1} . Light can be emitted in the visible range, but radiation emission elements emitting in the infrared range can also be used. The light-emitting diodes are arranged on the top of a transparent substrate SUB. This substrate can be a glass substrate, for example. Opposite and facing the diodes on the underside of the substrate are radiation receiving elements PD1,. . ., PDk,. . ., PDk + 1 arranged. These radiation receiving elements are e.g. B. to photodiodes (z. B. pn photodiodes, pin photodiodes, Schott ky photodiodes, etc.) or phototransistors. It should be noted, of course, that the radiation receiving elements used are also matched to the wavelength range of the radiation emission elements used and their working range is matched to the radiation emission elements in terms of sensitivity.

An example of how the Folding device F illustrates. It is assumed that of m + 1 radiation emission elements only the elements k-1, k, and k + 1 can be controlled, since only these correspond corresponding (rounded) values of the addressing control direction are fed. Furthermore, for this example assumed that each of the three values only once from the n Disparity elements were determined. That means that at this example, the radiation emission elements k-1, k, k + 1 are controlled identically and consequently the same Deliver the amount of radiation. The radiation emitted in each case spreads through the substrate SUB, becomes spatial superimposed and reaches the radiation receiving elements. The kth radiation receiving element detects that of k-th emission element emitted as well as that of two immediately adjacent (k-1, k + 1) emission elements emitted radiation. The two neighboring reception  elements k-1, k + 1 each receive the Radiation of the emission elements k-1, k, k + 1 and k + 1, k, k-1. Due to the spatial spread and the under however, the receiving element has a different distance k-1 the proportion of the emission element k + 1, and the receiving element ment k + 1 the proportion of the emission element k-1 stronger ge dampens than with the receiving element k. Hence the Emp capture element k capture the highest radiation intensity and the evaluation device AW determines the value k of Address as the result of the convolution and specifies it on exclude KE. In this particular example it corresponds the result of the convolution is the mean.

In one variant (not shown), the case exists device from a corresponding array of m + 1 Counters. Because of the previous rounding are each other neighboring disparity values in terms of their value rounded the same value so that the corresponding kth Counter of the array that corresponds to the corresponding disparity value is assigned, is incremented accordingly, by the number of disparity values (in the cluster of neighboring values), which gave the same rounding result remote. The following evaluation device then gives the Address k of the meter with the highest meter reading as Er result KE.

Fig. 5 illustrates the operation of the disparity-coherence detectors KD described above. In the graph of FIG. 5 are respectively applied (based on the selected reference picture first or te two image) on the abscissa, the pixel positions and pixel numbers, and the ordinate for each Figu ment which the coherent detector KD supplied n disparity values of the disparity elements EST ₁ to EST _{n are} given. In the present case, for example, n = 7 disparity elements EST _{i are provided} for a disparity detector DD. The disparity value of zero is related to the fixation level and corresponds to the distance from the fixation level. The surrounding values of the disparity in the range of z. B. -4 to +4 then represent the Fusi onsbereich and each correspond to a distance that is closer or farther than the fixation plane.

For each picture element, the present n disparity values are processed or compared in real time by means of the respectively selected coherence detector KD according to the first or second exemplary embodiment, and that disparity value is output as the actual disparity value that is approximately identical to as many further disparity values as possible. More precisely, this means that as many disparity values as possible in an interval of e.g. B. should be about 0.2 to 0.4 around the disparity value in question so that this disparity value is the actual disparity value. It is easiest if several disparity values have the same value for a certain picture element. This case corresponds to the graph in accordance with Fig. 5 an intersection of at least two of the n disparity values for a be-determined pixel. The line-by-line analysis of the n disparity values for each picture element of a relevant line then provides, for example, a course of the actual disparity for a relevant line of the picture as shown in the right part of FIG. 5, with a disparity from the totality of the image lines analyzed card can be determined in the form of a third image or can be shown (upper image on the right in Fig. 5).

By appropriately selecting the number n of disparity elements elements can determine the accuracy or reliability of the true disparity determined by the coherence detector be increased because there are more disparity values, which are evaluated for their coherence. In particular each disparity element EST calculates the disparity in the entire measuring range, i.e. completely parallel. Hence the many known from the prior art are omitted Realizations necessary iterative approximation of rough ones towards fine measuring ranges to the actual dispari to determine actual value.

In addition, based on the values obtained from the diparity elements EST of the disparity detector DD, additional information regarding the reliability of the disparity determined can be obtained. That is, if one considers the disparity values at a point on the line (cf. FIG. 5), the proximity of the neighbors around the exact disparity value is a measure of the reliability of the disparity determined. A reliability map can thus be determined over the entire image, which provides a real-time statement regarding the probability of the correctness of the determined disparity.

FIG. 6 shows a further example of images "left image" (a) and "right image" (b) of objects arranged in space, recorded by image recorders CCDL, CCDR. In both images, a first and a second object are in front of another (third) object, which can be seen in the background of the spatial scene shown. On the basis of the evaluation by the optical sensor system according to the invention for processing stereoscopic images in real time, a disparity map (c) is then obtained, which can be displayed on a screen as a gray-scale image (or as a color image). Referring to FIG. 6 an in-focus object is in the disparity map shown in light, an in-medium depth range of the object represented by a depth corresponding light to dark gray value, and the object on the screen background as the Whether ject with the greatest depth in the disparity map reproduced in approximately black. However, the form of representation for the depth information described in the example shown in FIG. 6 is not mandatory. Objects in the foreground can also be represented dark, and objects in the background can be represented light.

In addition, the data of the displayed dispari card can also be used for control purposes, in which you can use this data to control Robo, for example tern during assembly operations and a target-actual-Ver same with predefined 3-D manufacturing data and depth information determined from the disparities leads.

Unmanned vehicles can reach and / or under a predetermined minimum depth value for ver prevention of collisions by appropriate controls  to be evaded, this principle also when implementing passive and / or active distance use warning and anti-collision systems in motor vehicles is cash. Show other possible uses for the system in medical technology, for example, in three dimensions sional scanning of body contours for the classification of Dysplasia. In particular, there are possible uses in maxillofacial area or flexible or rigid Endoscopes.

Claims (17)

1. Coherence detector with
a large number of n signal inputs (EST _i (1 ← i ← n); E1, E2, E3, E4), to which a respective input signal value is present,
a first sorting device (S1) for sorting the n supplied input signal values (E1,..., E4) according to their value (A, B, C, D) and for the parallel output of the n sorted values,
a plurality of n-1 subtracting devices ("-") for subtracting two adjacent sorted values ((DC), (CB), (BA)),
a second sorting device (S2) for sorting the received n-1 differences ((DC), (CB), (BA)) with regard to their amount and for outputting the smallest difference amount ("« "),
a selection device ("=", MUX1) which outputs the value pair of the input signal values associated therewith on the basis of the smallest difference ("« ") determined, and
an output device ("Σ / 2"), to which the output pair of values is fed and which outputs its mean value (KE). 2. The detector of claim 1, wherein
the selection device ("=", MUX1) one
Comparison unit ("=") which determines the difference value which corresponds to the smallest difference amount by comparing the smallest difference amount with the n-1 difference values, and
has a multiplexer unit (MUX1) which, owing to the control by the comparison result provided by the comparison unit, outputs that pair of values of the input signal values with the smallest difference. 3. Coherence detector with
a large number of n signal inputs (EST _i ), with 1 ← i ← n, at which a respective input signal value is present,
a window limiting device (FB) for outputting those of the n input signals whose input signal value is within a preset, finite window value range [-x,. . ., 0,. . ., + x] with m + 1 values in the Ab starid of 2x / m,
a rounding device (R) for rounding the values output by the window limiting device (FB) to the nearest of the m + 1 values of the preset window value range,
an addressing control device (AA), and
a folding device (F), wherein
the folding device (F) has a number of m + 1 signal inputs ( _Ek , 1 ← k ← m + 1) corresponding to the number of values in the window value range, and the respective kth input by the addressing control device ( AA) those of the n signals are supplied whose value is -x + (k-1) .2x / m, which thus drive a corresponding k-th of m + 1 radiation emission elements, the radiation thus obtained being emitted by one of the number of Values of the window value range corresponding plurality of m + 1 radiation receiving elements is detected, and
an evaluation device (AW) which determines the address k of the one of the m + 1 radiation receiving elements which detects the greatest radiation intensity and outputs the value k of this address at an output connection (KE), the output value being the result of the convolution of the represents the folding device supplied signals. 4. Signal processing device according to claim 3, wherein the radiation emission elements of the convolution device around light emitting diodes (LED) and at the Radiation receiving elements are photodiodes (PD). 5. Signal processing device according to claim 4, wherein the Light emitting diodes (LED) on the top of a transparent substrates (SUB) are arranged, and the Photodiodes opposite the light-emitting diodes (LED) arranged on the underside of the substrate (SUB) are.   6. Signal processing device according to claim 5, wherein the transparent substrate (SUB) is a glass substrate. 7. Image processing device, with
Shifting devices (VSEL, VSER; ΔX _L1 ,..., .DELTA.X _Ln , .DELTA.X _R1 ,..., .DELTA.X _Rn ) for mutually shifting these respectively supplied image data of a first image and a second image taken at a different angle Image, the image data on the shifting devices (VSEL, VSER) are tapped in parallel and in pairs ((ΔX _L1 , ΔX _Rn ), (ΔX _L2 , ΔX _Rn-1 ),..., (ΔX _Ln-1 , ΔX _R2 ), (ΔX _Ln , ΔX _R1 )) are fed to a subsequent disparity detection device (DD; EST ₁ ,..., EST _n ); wherein the disparity detection device (DD) for each of the supplied image data pairs ((ΔX _Li , ΔX _{Rn + 1-i} )) by means of a respective disparity element (EST ₁ ,..., EST _n ) provides spatial depth information determined for the respective image data pair disparity value, and the determined disparity values a coherence detection device (KD) according to one of claims 1, 2 or 3 to 6,
the output disparity value (KE) is determined on each shift of the image data and represents the associated spatial depth information. 8. The image processing device according to claim 7, wherein
the image data of a full image are interlaced in two fields during two successive field periods,
actual disparity values determined by the coherence detection device (KD) during a first field period are temporarily stored in a memory device (MEM), and
these temporarily stored disparity values are supplied to the coherence detection device (KD) during the subsequent second field period in addition to the disparity values detected by the disparity detection device (DD) in order to determine the actual disparity values for the frame. 9. The image processing device according to claim 7, wherein the supplied image data of the first image and the second Image of a preprocessing device (VVL, VVR) be fed; and get due to preprocessing ne different types of image data of the image data of the first Picture and the second picture in each case shifting device tion (VSEL, VSER) and a disparity detection unit direction (DD) are fed, each for a respective current image data type are provided. 10. Image processing device according to one of claims 7 to 9 wherein the image data of the first and second images are se riell and synchronized to each other. 11. The image processing device according to claim 10, wherein the shifting devices (VSEL, VSER; ΔX _L1 ,..., ΔX _Ln , ΔX _R1 ,..., ΔX _Rn ) shift the image data against one another pixel by pixel. 12. Image processing device according to claim 10, wherein the shifting devices (VSEL, VSER; ΔX _L1 ,..., ΔX _Ln , ΔX _R1 ,..., ΔX _Rn ) shift the image data against one another in fractions of pixels. 13. Image processing device according to claim 11 or 12, the image data being supplied as analog image signals the, the shifting devices as analog runtime chains are executed and the mutual shift of the image elements by setting the appropriate term follows. 14. The image processing device according to claim 11, wherein the image data are supplied as digital image signals and the shifting devices as clocked shift registers are executed. 15. The image processing device according to claim 12, wherein the image data are supplied as digital image signals and the displacement devices are designed as filters, the Image data values between neighboring image elements interpo lieren. 16. The image processing device according to claim 7, wherein a respective disparity element (EST ₁ ,..., EST _i ,.., EST _n , 1 ≦ i ≦ n) determines the determined disparity value based on a gradient-based processing. 17. Image processing device according to one of claims 7 to 16, wherein a display device is provided to  the actual disparity values (KE) as the depth issue disparity card representing formation.