DE102021204411A1 - Method and apparatus for determining disparity and vehicle - Google Patents

Abstract

The invention relates to a method for determining a disparity in which a first rectified image (501) is transformed into a reference image (1101) and a second rectified image (502) is transformed into a search image (1302), a matching table (1220) for a pixel of the reference image (1101), position values (1402) for pixels of the search image (1302) determined, a frequency distribution (1400) formed for the position values (1402) and a disparity value (1528) for the pixel of the reference image (1101) from the position (1110) of the pixel in the reference image (1101) and that of the position values (1402) which has the greatest frequency in the frequency distribution (1400).

Description

State of the art

The invention is based on a device or a method according to the species of the independent claims. The subject matter of the present invention is also a computer program.

When processing a stereo image pair in the field of computer vision, a very important step is the determination of the disparity. Another important step is the determination of a surface normal. For this purpose, a surface estimation can be carried out using extracted image points and the surface normal can be determined with respect to an area resulting from the surface estimation.

the DE 196 36 028 C1 discloses an approach in which a disparity histogram is constructed using a weighted frequency increment.

Disclosure of Invention

Against this background, a method and a device for determining a disparity and a vehicle according to the main claims are presented with the approach presented here. Advantageous developments and improvements of the device specified in the independent claim are possible as a result of the measures listed in the dependent claims.

Advantageously, the disparity can be determined by means of a voting process, a so-called voting process. Optionally, in addition to the disparity, a normal vector can also be determined with the voting method.

• Einlesen eines ersten rektifizierten Bilds und eines zweiten rektifizierten Bilds;
• Transformieren des ersten rektifizierten Bilds in ein Referenzbild unter Verwendung einer Transformationsvorschrift die jedem Bildpunkt des ersten rektifizierten Bilds einen Merkmalswert aus einer vorbestimmten Menge von Merkmalswerten zuordnet, und des zweiten rektifizierten Bilds in ein Suchbild unter Verwendung der Transformationsvorschrift die jedem Bildpunkt des zweiten rektifizierten Bilds einen Merkmalswert aus der vorbestimmten Menge von Merkmalswerten zuordnet;
• Aufbauen einer Abstimmtabelle für einen Bildpunkt des Referenzbilds, indem jedem Merkmalswert aus der vorbestimmten Menge von Merkmalswerten Abstände zwischen einer Position des Bildpunkts in dem Referenzbild und zumindest sich in einer Referenzregion um die Position des Bildpunkts befindlichen Positionen des entsprechenden Merkmalswerts in dem Referenzbild zugeordnet werden;
• Bestimmen von Positionswerten für zumindest jeden sich in einer Suchregion befindlichen Bildpunkt des Suchbilds, indem für jeden sich in der Suchregion befindlichen Bildpunkt des Suchbilds der entsprechende Merkmalswert aus dem Suchbild ausgelesen wird, die dem entsprechenden Merkmalswert zugeordneten Abstände aus der Abstimmtabelle ausgelesen werden und die Positionswerte aus der Position des Bildpunkts und den Abständen ermittelt werden;
• Bilden einer Häufigkeitsverteilung für die Positionswerte der sich in der Suchregion befindlichen Bildpunkte des Suchbilds; und
• Ermitteln eines Disparitätswerts für den Bildpunkt des Referenzbilds aus der Position des Bildpunkts in dem Referenzbild und demjenigen der Positionswerte, der die größte Häufigkeit in der Häufigkeitsverteilung aufweist.

A corresponding method for determining a disparity includes the following steps:

• reading in a first rectified image and a second rectified image;
• Transforming the first rectified image into a reference image using a transformation rule which assigns a feature value from a predetermined set of feature values to each pixel of the first rectified image, and transforming the second rectified image into a search image using the transformation rule which assigns a feature value to each pixel of the second rectified image assigns from the predetermined set of feature values;
• constructing a matching table for a pixel of the reference image by associating each feature value from the predetermined set of feature values with distances between a position of the pixel in the reference image and positions of the corresponding feature value in the reference image that are at least in a reference region around the position of the pixel;
• Determination of position values for at least each pixel of the search image located in a search region by reading the corresponding feature value from the search image for each pixel of the search image located in the search region, the distances assigned to the corresponding feature value are read from the matching table and the position values are read out the position of the pixel and the distances are determined;
• forming a frequency distribution for the position values of the pixels of the search image located in the search region; and
• Determining a disparity value for the pixel of the reference image from the position of the pixel in the reference image and that of the position values that has the greatest frequency in the frequency distribution.

The method represents a method for determining the disparity in a stereo image pair using a voting method. The first rectified image can be based on a first input image captured using a first image capturing device and the second rectified image can be based on a second input image captured using a second image capturing device. The first input image and the second input image can represent a stereo image pair. The image capturing devices can be arranged on a vehicle, for example, and can be used to capture a section of the surroundings of the vehicle from different viewing angles. Geometric distortions can be eliminated by suitable rectification of the input images. The rectified images can each include a plurality of pixels. For example, each pixel can represent a gray value.

The reference region can be placed around the position of the pixel in a suitable way. All positions of the corresponding characteristic value located in the reference region can be assigned. Typically, the reference region does not include the entire reference image. However, each feature value from the predetermined set of feature values can also have distances between the position of the pixel in the reference image and all positions of the corresponding feature value in the reference image be assigned. In such a case, the reference region can encompass the entire reference image.

The size and positioning of the search region may be determined in any suitable manner. In this case, for example, empirical values can be used. For example, the line of the reference position can be used to place the search region. The search region and the reference region can be the same or different in size. Typically, the search region does not include the entire search image. However, position values for each pixel of the search image can also be determined by reading out the corresponding feature value from the search image for each pixel of the search image. The frequency distribution for the position values of the pixels of the search image can then be formed accordingly. In such a case, the search region can include the entire search image.

Optionally, the method can accordingly include a step of reading in the first input image via an interface to the first image acquisition device and a second input image via an interface to the second image acquisition device. Furthermore, the method can optionally include a step of rectifying the first input image and the second input image.

Through the step of transforming, a unique feature value can be calculated for each pixel, for example based on the pixels surrounding this pixel. Known transformation methods can be used here. In this way, the reference image and the search image can be determined in the same way. A feature value can be assigned to each pixel of the reference image and the search image as a result of the transformation.

For example, in the transformation step, the transformation rule can be used to determine a feature according to BRIEF (Binary Robust Independent Elementary Features), ORB, BRISK (Binary Robust Invariant Scalable Keypoints), or Census. Corresponding feature definitions can be found, for example, in Rublee Ethan, Orb: an efficient alternative to sift or surf., Computer Vision (ICCV) (2011), Roland Y. Siegwartleutenegger Stefan, Margarita Chli, Brisk: Binary robust invariant scalable keypoints., Computer Vision ( ICCV) (2011), Calonder Michael, Brief: Binary robust independent elementary features., Computer Vision ECCV 2010 (2010) or Wikipedia, Census-transformation, Wikipedia (2019), Available online, known.

Advantageously, the maximum number of different feature values in the reference image and the search image can be limited. For example, a descriptor can be used that comprises 12 or 14 bits and thus contains a value store of at least 4096 features.

In the construction step, a matching table can first be constructed for a selected pixel of the reference image. The matching table can represent a list in which all possible feature values are listed and in which at least one distance is noted for each feature value if the corresponding feature value is present in the reference image. If a characteristic value exists more than once, several distances are noted for this characteristic value. To determine a distance, a simple difference can be formed between the position of the pixel in the reference image and the corresponding position of the currently observed feature value in the reference image.

In the determination step, the feature values of the search image can be used as addresses and the distances entered in the matching table can be used as offsets in order to determine the position values.

In the forming step, frequencies of the position values can be accumulated in order to form the frequency distribution in the form of an accumulated histogram.

In the determination step, the maximum from the frequency distribution can be sought in order to determine the disparity value for the pixel of the reference image for which the matching table was constructed in the construction step.

In order to be able to determine a disparity value for at least one further pixel of the reference image, the steps of constructing, determining, forming and determining mentioned can be carried out repeatedly for at least one further pixel of the reference image.

The steps of constructing, determining, forming and determining can thus be carried out repeatedly for all pixels of the reference image in order to determine a disparity value for each pixel of the reference image. In this way, the disparity for the entire stereo image pair can be continuously determined.

Optionally, the method may include a step of capturing the first input image using the first image capturing device and the second input image using the second image capturing device. on In this way, input images of the image acquisition device can be processed directly after acquisition in order to obtain disparity information.

For example, one of the image capturing devices can be a front camera and the other of the image capturing devices can be a side camera of a vehicle. Such cameras typically have an overlapping detection area, but this is recorded from very different perspectives, so that a disparity determination using conventional methods is often not possible due to the strongly differing input images.

According to one embodiment, the method is supplemented by further steps which, in addition to determining the disparity, enable a normal vector to be determined.

For this purpose, according to one embodiment, the method has a step of extracting those pixels of the search image located in the search region for which that of the position values was determined in the determination step that has the greatest frequency in the frequency distribution. In a determination step, based on this extraction, a surface normal is determined from positions of the pixels of the search image extracted in the extraction step.

The mentioned steps of extracting and determining the surface normal can be carried out repeatedly for at least one further pixel of the reference image in order to determine at least one further surface normal for the at least one further pixel of the reference image.

Thus, the steps of extracting and determining as well as the steps of building up, determining, forming and ascertaining carried out to determine the disparity can be carried out repeatedly for a plurality of or also for all pixels of the reference image. In this way, both a disparity value and a surface normal can be determined for several or for each pixel of the reference image. In this way, the disparity and surface normals can be continuously determined for the entire stereo image pair.

This method can be implemented, for example, in software or hardware or in a mixed form of software and hardware, for example in a control unit.

The approach presented here also creates a device for determining a disparity, the device being designed to carry out, control or implement the steps of a variant of a method presented here in corresponding devices. The object on which the invention is based can also be achieved quickly and efficiently by this embodiment variant of the invention in the form of a device.

For this purpose, the device can have at least one computing unit for processing signals or data, at least one memory unit for storing signals or data, at least one interface to a sensor or an actuator for reading in sensor signals from the sensor or for outputting data or control signals to the Have actuator and / or at least one communication interface for reading or outputting data that are embedded in a communication protocol. The arithmetic unit can be, for example, a signal processor, a microcontroller or the like, with the memory unit being able to be a flash memory, an EEPROM or a magnetic memory unit. The communication interface can be designed to read in or output data wirelessly and/or by wire, wherein a communication interface that can read in or output wire-bound data can, for example, read this data electrically or optically from a corresponding data transmission line or can output it to a corresponding data transmission line.

The approach described can advantageously be implemented in connection with a vehicle. Such a vehicle can comprise the named image capturing device and a named device for determining a disparity.

A computer program product or computer program with program code, which can be stored on a machine-readable carrier or storage medium such as a semiconductor memory, a hard disk memory or an optical memory and for carrying out, implementing and/or controlling the steps of the method according to one of the embodiments described above, is also advantageous used, especially when the program product or program is run on a computer or device.

• 1 eine schematische Darstellung der Projektion einer beobachteten Szene;
• 2 ein Referenzbild gemäß einem Ausführungsbeispiel;
• 3 ein Suchbild gemäß einem Ausführungsbeispiel;
• 4 eine geometrische Sicht einer Bildvorverarbeitung;
• 5 ein erstes rektifiziertes Bild und ein zweites rektifiziertes Bild gemäß einem Ausführungsbeispiel;
• 6 ein Fahrzeug mit einer Vorrichtung zum Bestimmen einer Disparität gemäß einem Ausführungsbeispiel;
• 7 ein erstes Eingangsbild einer ersten Bilderfassungseinrichtung gemäß einem Ausführungsbeispiel;
• 8 ein zweites Eingangsbild einer zweiten Bilderfassungseinrichtung gemäß einem Ausführungsbeispiel;
• 9 ein erstes rektifiziertes Bild gemäß einem Ausführungsbeispiel;
• 10 ein zweites rektifiziertes Bild gemäß einem Ausführungsbeispiel;
• 11 ein Referenzbild gemäß einem Ausführungsbeispiel;
• 12 eine Abstimmtabelle gemäß einem Ausführungsbeispiel;
• 13 ein Suchbild gemäß einem Ausführungsbeispiel;
• 14 einen Ausschnitt einer Häufigkeitsverteilung gemäß einem Ausführungsbeispiel;
• 15 ein Blockschaltbild eines Verfahrens zum Bestimmen einer Disparität gemäß einem Ausführungsbeispiel;
• 16 ein Ablaufdiagramm eines Verfahrens zum Bestimmen einer Disparität gemäß einem Ausführungsbeispiel;
• 17 eine Szene mit einem erhabenen Objekt gemäß einem Ausführungsbeispiel;
• 18 ein Referenzbild gemäß einem Ausführungsbeispiel;
• 19 ein Suchbild gemäß einem Ausführungsbeispiel;
• 20 ein Referenzbild gemäß einem Ausführungsbeispiel;
• 21 ein Suchbild gemäß einem Ausführungsbeispiel;
• 22 ein Blockschaltbild eines Verfahrens zum Bestimmen einer Disparität und einer Oberflächennormalen gemäß einem Ausführungsbeispiel; und
• 23 ein Ablaufdiagramm eines Verfahrens zum Bestimmen einer Disparität und einer Oberflächennormalen gemäß einem Ausführungsbeispiel.

Exemplary embodiments of the approach presented here are shown in the drawings and explained in more detail in the following description. It shows:

• 1 a schematic representation of the projection of an observed scene;
• 2 a reference image according to an embodiment;
• 3 a search image according to one embodiment;
• 4 a geometric view of image preprocessing;
• 5 a first rectified image and a second rectified image according to an embodiment;
• 6 a vehicle having a device for determining a disparity according to an embodiment;
• 7 a first input image of a first image capturing device according to an embodiment;
• 8th a second input image of a second image capturing device according to an embodiment;
• 9 a first rectified image according to an embodiment;
• 10 a second rectified image according to an embodiment;
• 11 a reference image according to an embodiment;
• 12 a tuning table according to one embodiment;
• 13 a search image according to one embodiment;
• 14 a section of a frequency distribution according to an embodiment;
• 15 a block diagram of a method for determining a disparity according to an embodiment;
• 16 a flowchart of a method for determining a disparity according to an embodiment;
• 17 a scene with a raised object according to one embodiment;
• 18 a reference image according to an embodiment;
• 19 a search image according to one embodiment;
• 20 a reference image according to an embodiment;
• 21 a search image according to one embodiment;
• 22 a block diagram of a method for determining a disparity and a surface normal according to an embodiment; and
• 23 a flowchart of a method for determining a disparity and a surface normal according to an embodiment.

In the following description of favorable exemplary embodiments of the present invention, the same or similar reference symbols are used for the elements which are shown in the various figures and have a similar effect, with a repeated description of these elements being dispensed with.

1 shows a schematic representation of the projection of an observed scene by multiple camera projections. Real object points X _j are shown, which are imaged in a first input image 101, a second input image 102 and at least one further input image 103. For example, the first input image 101 and the second input image 102 form a stereo image pair.

When processing a stereo image pair in the field of computer vision, a very important step is the determination of the disparity.

Disparity is understood as meaning the displacement value of a small image section from the so-called reference image, for example the first input image 101, to the second image of the stereo pair. Due to the projection of the observed scene into the stereo image pair, the same object points are projected into different image regions. The object points [X _j ] are placed in the images 101, 102, 103 at the image positions [x ₁ ; j], [x ₂ ; j] and [x ₃ ; j] projected.

The shift value between [x ₁ ; j] and [x ₂ ; j] is called disparity.

If this disparity is measured from a pair of stereo images with a known extrinsic calibration [R _n |t _n ], an unknown scene can be measured by triangulation.

To determine the disparity, image sections from the reference image, here for example the first input image 101, or a transformation of the reference image into a feature image, are searched for in the search image, here for example the second input image 102 by shifting by (δ _x , δ _y ).

Advantageously, with the approach described here, it is not necessary to first set up similarity measures between a reference image region and shifted image regions in the search image in order to measure the disparity.

Such an alternative disparity measurement is based on the 2 until 4 explained.

2 shows a reference image 201, for example the one in 1 shown first input image, which is viewed as a grid of numbers. A section in the reference image 201 designated as reference region 203 is to be found in the search image.

3 shows a corresponding search image 302, for example the one in 1 shown second input image, which is viewed as a grid of numbers. the inside 2 The section shown is to be found in the search image 302. For this purpose, the most similar image section, which is also referred to as search region 303, is searched for. The displacement δ _x , δ _y from the reference position to the position found in the search image 302 is (1, 2) in this example.

im Suchbild 302 gesucht. Alle möglichen Positionen im Suchbild 302 können ebenfalls als 4-dimensionaler Vektor aufgefasst werden. So ist das Gebiet im Suchbild an der Ausgangsposition $\overrightarrow{s_{\mathrm{0,0}}}=\left(\mathrm{1,3,3,5}\right),$

die Differenz wäre $\overrightarrow{g}-\overrightarrow{s_{\mathrm{0,0}}}=\left(\mathrm{5,}-\mathrm{1,1,2}\right).$

The direct comparison of the image information includes both the reference image section and the search image as a grid of numbers instead of the visible color/grey values. In 3 the reference region 203 is exemplified from the numbers $\overrightarrow{G}=\left(\mathrm{6,2,4,7}\right)$

searched in search image 302. All possible positions in the search image 302 can also be interpreted as a 4-dimensional vector. So the area in the search image is at the starting position $\overrightarrow{s_{0.0}}=\left(\mathrm{1,3,3,5}\right),$

the difference would be $\overrightarrow{G}-\overrightarrow{s_{0.0}}=\left(\mathrm{5,}-\mathrm{1,1,2}\right).$

If you take the length as a measure of similarity, you have a value of 5.568 in the position (0; 0). The lowest value is found at a shift of (δ _x , δ _y ) = (1, 2).

Such a two-dimensional search can be restricted to a one-dimensional search by means of special image pre-processing. A search then only takes place, e.g. in the x-direction. If the X-value is found at which the selected measure of similarity finds an extremum, then the disparity is determined.

4 shows the geometric view of this simplification. An object point X is projected into the stereo image pair made up of the first input image 101 and the second input image 102 . Positions along the lines of sight can be traced back to the respective other image in a straight line on the image surface. The projection of the 3D object point X in an image, for example the first input image 101, determines the position of the projection of this 3D point in the other image, for example the second input image 102. The location can be delimited by a straight line in the other image. If these straight lines are additionally transformed in such a way that an image results that shows these straight lines exclusively in the X-direction, then the disparity can be determined very easily with a one-dimensional search. This process is called rectification.

In 4 camera projection centers 410, epipoles 412, image points 414 and epipolar lines 416 are shown for this purpose. Region 418 is referred to as the epipolar plane.

5 shows a first rectified image 501 and a second rectified image 502. Examples are those in FIGS 2 and 3 The images shown are transformed in such a way that a disparity d = 2 results. The images are transformed in such a way that the epipolar line, like the epipolar line 505 shown, propagates only in the X-direction.

A reference region 203 is placed around the object point X in the first first rectified image 501 . In the second rectified image 502 a search region 303 is shown. The reference image area can now be found again here as an example with a disparity d=2.

6 1 shows a vehicle 600 with a first image capturing device 601 in the form of a side camera and a second image capturing device 602 in the form of a front camera. Furthermore, the vehicle 600 includes a device 610 for determining a disparity. The device 610 is designed to carry out a method for determining a disparity and optionally a surface normal, as is described with reference to the further figures. The device 610 is designed to read in a first input image via an interface to the first image acquisition device 601 and a second input image via an interface to the second image acquisition device 602 and to use them to determine the disparity. For example, device 610 is designed to provide information about the disparity to a driver assistance system 615 of the vehicle.

According to one exemplary embodiment, the image acquisition devices 601, 602 are each designed as a so-called close-range camera, also referred to as an NRC camera or near-range camera. The image capturing devices 601, 602 are used to capture images of the surroundings of the vehicle 600. In the common detection area of image capturing devices 601, 602, if there is another vehicle in an adjacent lane, a fender, for example, together with a tire of the other vehicle is in the first input image of the first image capturing device 601, for example in the form of a left side mirror camera, and in the second input image of the second Image acquisition device 602, for example in the form of an image from a front camera.

7 shows a corresponding first input image 101 of FIG 6 shown first image capturing device 601, wherein in the first input image 101 a mudguard together with a tire of the other vehicle can be seen.

8th shows a corresponding second input image 102 of FIG 6 second image capturing device 602 shown, wherein the fender together with the tire of the other vehicle can also be seen in the second input image 102 .

Advantageously, with the approach described here, even with such a recording constellation, as in the 7 and 8th shown, a robust disparity can be found. This is possible, although no similarity measures can be found for such a recording constellation that could evaluate shifted image sections in such a way that reliable extremes can be found. Instead of a disparity search, trying to design a similarity measure that is robust, simple, fast, etc., and that is used to find similar image sections (be they in the original data format as a gray or color image, or through a transformation as a feature image), the approach described here uses a voting procedure that uses the features in the reference image to create a table that is used in the search area to vote directly for the disparity to be sought. No measure of similarity is determined for all possible positions, but after a run through all features in the search area, the position is selected for which the most votes from the feature voting process have voted.

This method is particularly suitable for stereo image pairs that view the scene in question from very different perspectives. Here, the projections of the scene into the image plane lead to very different images, as can be seen from 1 is described.

An application of this method can be found, for example, in the driver assistance function. Cameras are currently installed in the front area of a vehicle (rear window, radiator grille) and in the exterior mirrors for various driver assistance functions. Pictures from such a constellation are in the 7 and 8th to see.

The method described here bypasses the similarity measures, since individual features directly vote for the position in the search image.

A robust maximum within the votes can then be sought and the desired disparity can thereby be obtained.

9 shows a first rectified image 501, which, for example, consists of a rectification of a first input image in 6 shown first image acquisition device 601 has emerged. The first rectified image 501 shows similar to 7 at least one fender along with a tire of another vehicle.

10 shows a second rectified image 502, which, for example, consists of a rectification of a second input image in FIG 6 shown second image capturing device 602 has emerged. The second rectified image 502 shows similar to 8th at least the fender along with the tires of the other vehicle.

From the 7 until 10 it can be seen that the side camera and the front camera view, for example, an overtaking vehicle from very different directions. A rectification results in the 9 and 10 shown rectified images 501, 502, which can no longer be compared robustly with conventional similarity measures. The voting-based method described here can determine robust disparities here.

The method described here is therefore particularly suitable for stereo image pairs that view the scene in question from very different perspectives. Here, the projections of the scene into the image plane, i.e. the rectification, lead to very different image impressions, as can be seen from the 9 and 10 is evident.

The method described here is initially based on a transformation of the input image, after geometric rectification has taken place, into a feature image, also called feature image, for which a unique feature value, also called feature value, is calculated for each point based on the environment.

11 11 shows a reference image 1101 according to an embodiment. For example, the reference image 1101 is generated by transforming a first rectified image, as exemplified in 5 left or in 9 is shown. A transformation rule can be used for this purpose, which assigns a feature value from a predetermined set of feature values to each pixel of the first rectified image. The reference image 1101 thus represents a feature image, also called a feature image.

The reference image 1101 is composed of feature values that are in a plurality of Rows and columns are arranged. For example, the feature values 12, 08, 15, 02, 09, 14 are entered in the first line of the reference image 1101 .

If, for example, the census transformation is used as the transformation rule, the gray value of a pixel of the in 5 shown rectified first image compared individually with the gray value of the neighboring pixels and the result is stored as a feature value in the reference image 1101 at the position 1110 of the observed pixel.

For example, will the first rectified image be 5 transformed with an algorithm into the reference image 1101 with the value range of [0,...,15], this results in a voting table with a maximum of 16 addresses.

The feature calculation carried out when transforming the first rectified image can be carried out with various methods known from the literature, such as BRIEF, ORB, BRISK, Census or other feature generators and is not of interest for the basic form of the approach described here. For example, in the field of computer vision, in which matching tasks are to be solved, binary descriptors are available as features.

The basic question when determining the disparity is also with the approach described here, where in the search area is the shifted image area. In 11 a transformed pair of feature images is given as an example. The object point X in the reference image 1101 shows as in the original rectified image, e.g. shown in 5 , on (x = 1.5 I y = 1.5). Since only the x-value is of interest due to the previous rectification, the votes are then only related to the x-value of 1.5 of the image coordinates.

The various feature values are run through once and the difference between the feature's own position, i.e. the position of the corresponding feature value, and the projection of the object point to be evaluated, i.e. the position 1110 of the pixel, here x = 1.5, as a distance, also as a delta -X value, entered in the reconciliation table, as in 12 is shown.

According to one embodiment, the entire reference image and the entire search image are transformed. To determine the disparity of a point (regardless of whether the image is iterated successively or only individual points are picked out), a reference region around the point, here by way of example the object point X, in the reference image 1101 is used according to one exemplary embodiment. The tuning table is built for the reference region. The reference region is evaluated with another region called the search region in the search image. In this case, not the entire search image is evaluated, but an area located in the search image around the line of the reference position. The reference region and search region do not have to be the same size, the search region is usually larger, but the reference region is not normally the entire reference image 1101, and the search region does not include the entire search image.

12 1220 shows a tuning table 1220 according to one embodiment. All feature values 1222 from the predetermined set of feature values are entered in the first column, here the feature values 00 to 15. For each of the feature values 1222, distances 1224 are entered in the other columns of the matching table. The feature values 1222 are also referred to as features and the distances 1224 are also referred to as delta.

For example, the distance -0.5 is entered for the characteristic value 00, since the difference between the X position of the characteristic value 00 in the in 11 shown reference image 1101 and the X position of the object point X -0.5.

No distance is entered for characteristic value 01, since characteristic value 01 is in the in 11 shown reference image 1101 is not included.

A first distance +1.5, a second distance -0.5 and a third distance -1.5 are entered for the feature value 02, since the feature value 02 in the in 11 reference image 1101 shown is included three times. The differences between the X positions of the characteristic values 02 in the in 11 The reference image 1101 shown and the X position of the object point X result in the stated distances of +1.5, -0.5 and -1.5.

For the other feature values 03 to 15, distances are determined in a corresponding manner and entered into the matching table 1220.

In this way, the delta X values for projecting the object point are stored for each feature value.

The described structure of the voting table 1220 in the form of a feature delta X table serves as a preparation for a disparity voting in the subsequent voting in the search image.

Once the voting table 1220, which is also referred to as a voting table, has been set up, all table entries at each point of the feature image in the search image are interpreted as votes for the projection in the search image.

The restriction of the feature values to 16 is only chosen as an example. Typically more than 16 feature values are used, for example at least 4096 when using a descriptor with at least 12 bits. The number of characteristic values plays a subordinate role. Regardless of the coding, a feature space is used that is integer and finite or at least can be mapped to integer.

13 Figure 13 shows a search image 1302 according to one embodiment. For example, the search image 1302 is generated by transforming a second rectified image, as exemplified in 5 right or in 10 is shown. The same transformation rule that is used to determine the reference image can be used for this. The search image 1302 thus represents another feature image, also called a feature image.

The search image 1302 is accordingly composed of feature values corresponding to the reference image. A feature value is assigned to each pixel of the search image 1302 .

In 13 shows the use of the matching table built from the reference image. For this purpose, the feature values are used as an address in the search image 1302 and the distances stored in the matching table are used as an offset.

Examples are in 13 in the search screen 1302 for the characteristic values 12 and 05 in the in 12 distances entered in the matching table shown for the feature values 12 and 05 are entered as displacement vectors.

The displacement vectors point to positions that are used as position values.

For example, the feature value 14 is assigned to the first pixel arranged in the first column and first row of the search image 1302 . in the in 12 The voting table shown shows the distances -2.5 and -3.5 for the feature value 14. Two position values, namely -2 and -3, result from the X position of the first pixel, here 0.5, and the specified distances -2.5 and -3.5.

If position values have been determined for all pixels and thus all feature values in the search image 1302, a frequency distribution over all determined position values is formed according to one exemplary embodiment.

Based on 13 and the following one 14 is a table structure and subsequent voting in an accumulated histogram.

14 shows a section of a frequency distribution 1400 according to an embodiment. In the second line, those position values 1402 are entered which, as based on 13 described were determined using the search image. A frequency of the respective position values 1402 is entered in the first line.

The frequency distribution 1400 shows that based on all pixels and thus feature values of the 13 shown search image and based on 12 shown tuning table the position value 0.5 twice, the position value 1.5 eight times, the position value 2.5 seven times, the position value 3.5 twenty-four times and the position value 4.5 twice.

The distances of the matching table as offsets together with the feature positions in the search image are accumulated in a histogram.

If all voices, i.e. votes, are accumulated in a histogram, the result for this example is in 14 shown distribution. The maximum number of votes is in position 3.5. If the initial value x = 1.5 is subtracted from this, the disparity d = 2 results. The initial value is the X position of the pixel in the in 11 used for the reference image shown in 12 shown tuning table was determined.

15 shows a block diagram of a method for determining a disparity according to an embodiment. The method represents a voting method for determining a disparity image 1500.

A first image capture device 601 provides a first input image 101 and a second image capture device 602 provides a second input image 102 . The first input image 101 is rectified in a step 1510 to obtain a first rectified image 501 . Correspondingly, the second input image 102 is subjected to a rectification in a step 1512 in order to obtain a second rectified image 502 .

The first rectified image 501 is transformed into a reference image 1101 in a step 1514 and the second rectified image 502 is transformed into a search image 1302 in a step 1516 .

In a step 1520 a matching table 1220 is built for a pixel of the reference image 1101 . In a step 1522 a vote is taken carried out. For this purpose, position values 1402 are determined for each pixel of the search image 1302 using the search image 1302 and the matching table 1220 . In a step 1524, a histogram is built. For this purpose, a frequency distribution 1400 of the determined position values 1402 is formed. In a step 1526, a disparity is determined. For this purpose, using the frequency distribution 1400, a disparity value 1528 is determined for that pixel of the reference image 1101 for which the steps 1520, 1522, 1524, 1526 have just been carried out.

In a step 1529, a next pixel is selected at a further position p of the reference image 1101 and steps 1520, 1522, 1524, 1526 are repeated for this pixel in order to determine a further disparity value 1528 for this pixel.

According to an exemplary embodiment, steps 1520, 1522, 1524, 1526 are carried out for each pixel of reference image 1101, ie for each position p from (n×m), where n and m represent the number of rows and columns in reference image 1101.

The method is described in detail below using an exemplary embodiment. At the beginning, stereo image pairs are conventionally rectified 1510, 1512. From this, the method determines the disparity image 1500, which can then in turn lead to a conventional triangulation 1530 in a measured 3D world description 1532.

The block diagram shows the basic processing steps and their sequence of the procedure. After a conventional rectification 1510, 1512, the rectified pair of images is obtained from the original stereo image pair, in which the object points of the observed 3D world lie on lines in the image, as is the case, for example, using 5 is shown.

innerhalb des Kamerapaares beschreibt. Auf diesen rektifizierten Eingangsbildern wird nun in den Schritten 1514, 1516 eine Feature-Transformation durchgeführt, so dass ein Feature-Stereobildpaar erhalten wird, das sich aus dem Referenzbild 1101 und dem Suchbild 1302 zusammensetzt.This requires a camera calibration that includes the rotation R and the translation $\overrightarrow{t}$

within the camera pair describes. A feature transformation is now carried out on these rectified input images in steps 1514, 1516, so that a feature stereo image pair is obtained, which is composed of the reference image 1101 and the search image 1302.

In order to determine a disparity for each position in the feature reference image 1101 with a resolution of n*m, the following block with steps 1520, 1522, 1524, 1526 n*m is repeated.

For this purpose, as described with reference to the previous figures, the matching table 1220 is set up from at least one reference region of the reference image 1101, in which the distance of a feature value from the currently observed image position represents the offset for the later matching.

The matching is performed after the matching table 1220 for the feature search image 1302 has been successfully built. The feature values in at least one search region in the search image 1302 serve as an address in the matching table 1220 and each match the collected distances for the corresponding image position. All of these votes are accumulated in a histogram so that the maximum number of votes can then be determined. The desired disparity is the difference between the current reference position and the maximum of the voting histogram, here the frequency distribution 1400.

The entirety of all disparities is stored in the disparity image 1500 . This result is optionally entered into traditional triangulation in order to derive a 3D description of the observed world from the calibration already used.

The disparity estimation described above can facilitate the use of camera images that view a scene from very different perspectives. The approach described can be used in any multi-camera system and can be used in the event of a product change and/or update. Such multi-camera systems can be built into motor vehicles, for example. However, the method presented can also be used in other domains, in which cameras sense the environment and disparities are sought.

According to one exemplary embodiment, the method is used in connection with driver assistance systems that use a front sensor system in combination with other cameras. The approach described is also suitable for the field of near-range cameras. The approach enables the environment to be surveyed using sensors that have not previously been used in this area.

16 shows a flow chart of a method for determining a disparity according to an embodiment. The method can be carried out, for example, using an apparatus as is based on FIG 6 is shown.

In a step 1514, a first rectified image is transformed into a reference image as is for example in 11 is shown. In this case, a transformation rule is used which assigns a feature value from a predetermined set of feature values to each pixel of the first rectified image. In a step 1516, a second rectified image is transformed into a search image, such as that shown in FIG 13 is shown. The same transformation rule as in step 1514 can be used here.

In a step 1520, a matching table such as that shown in FIG 12 shown, is constructed for a pixel of the reference image. For this purpose, each possible feature value, if present, is assigned a distance or, if the feature value occurs multiple times, several distances between a position of the pixel in the reference image and all positions of the corresponding feature value in the reference image.

According to one embodiment, in step 1520 a reference region is placed around the position of the pixel in the reference image. In this case, only the reference region is considered in step 1520 to build the tuning table. To this end, each possible feature value, if present, is assigned a distance or, if the feature value occurs multiple times, several distances between a position of the pixel in the reference image and all positions of the corresponding feature value in the reference image located in the reference region.

In a step 1522, position values as defined in 14 are shown are determined 1402 for each pixel of the search image. To do this, the corresponding feature value is read from the search image for each pixel of the search image, the distances assigned to the corresponding feature value are read from the matching table, and the position values are determined from the position of the pixel and the distances.

According to one embodiment, in step 1522 a search region is specified in the search image. In this case, for each pixel of the search image located in the search region, the corresponding feature value is read from the search image, the distances assigned to the corresponding feature value are read from the matching table, and the position values are determined from the position of the pixel and the distances.

In a step 1524, a frequency distribution, such as that shown in 14 is shown, is formed for the position values determined in step 1522 .

If a search region is used in step 1522, in step 1524 the frequency distribution for the position values of the pixels of the search image located in the search region is formed.

In a step 1526, a disparity value is determined for the named pixel of the reference image. To do this, the position value that has the greatest frequency in the frequency distribution is selected from the frequency distribution. To determine the disparity value, the position, for example a coordinate value representing the position, of the named pixel is subtracted from this most frequently occurring position value.

In order to determine a disparity value for each pixel of the reference image or at least a plurality of pixels of the reference image, steps 1520, 1522, 1524, 1526 are carried out repeatedly for the corresponding further pixels. The pixels can be combined to form a disparity image, as shown in 15 is described.

The rectified images can be read in via an interface in one reading step. Additionally or alternatively, the method includes a step 1510 in which the first rectified image is determined from a first input image. Correspondingly, in this case the method comprises a step 1512 in which the second rectified image is determined from a second input image.

According to different exemplary embodiments, the input images are read in or captured in a step 1630 using at least one image capture device. If only one image capture device is used, the input images may be or may have been captured at different points in time.

The steps of the method can be implemented in a suitable device which, for example, has a transformation device for transforming the rectified images, a construction device for construction of the matching table, a determination device for determining the position values, a formation device for forming a frequency distribution and a determination device for determining the disparity value having.

According to one embodiment, the voting method described is used to enable a simultaneous determination of disparity and surface normal of the observed object section in a stereo image pair.

As already explained, under disparity the displacement value of a small image section from the so-called reference image becomes the second Image of the stereo pair understood. So is in 4 with the aid of the extrinsic calibration, the stereo image pair 101, 102 is already rectified in such a way that the search for a reference image section can be carried out one-dimensionally on a straight line in the search image without loss of generality. The determination of the disparity value is, for example, based on 15 explained by means of a voting procedure.

The successful search for the reference section in the search image, for example in 5 shown, assumes in the standard cases that there is a purely translational movement of the image sections to one another. As already explained, the voting method enables the disparity estimation to be robustened.

In many cases, the surface normal is also of interest in addition to the disparity of the observed object point, which represents a measure of the distance of the object.

This is relevant, for example, when the question arises as to whether the object point is a wall rising up in front of the camera or whether the image section belongs to the street surface.

From a geometric point of view, the disparity could be described with a translatory movement in points of a rising wall. Both the method described with reference to the previous figures and many other known methods achieve good results here.

Even if this purely translational assumption is violated, the voting procedure has an advantage if the characteristics are well chosen, also referred to as features that cast the votes.

However, all the methods mentioned so far have in common that they only determine the disparity and ignore the surface normal.

According to the exemplary embodiments described below, the disparity search and a determination of the surface normal are not carried out in two separate steps, such as first determining the disparity and then in a local neighborhood a surface estimation from which the surface normal is determined. Rather, according to the exemplary embodiments described, a simultaneous determination of disparity and surface normals is carried out.

For this purpose, the voting method described with reference to the previous figures is used, with the surface normal being determined from the features and their position in the search image that voted for the resulting disparity. This is done without examining the identified disparities in a local neighborhood in a second subsequent step.

This method is particularly suitable for applications that observe a scene in which there are preferably different surface normals.

This procedure is explained below using the 17 until 21 explained.

17 shows a corresponding scene, for example in the automotive sector, in which the surfaces of raised objects, for example vehicle surfaces of vehicles such as a truck 1701, rise up on a road surface 1703. The road surface 1703 is an example of a ground plane.

18 represents a reference image 1801 according to an embodiment, in which the already based on 17 scene shown is depicted. A reference position 1805 and a reference image section 1807 are shown, as they are shown, for example, with reference to FIG 3 are already described. The reference image section 1807 is located on a side wall of the truck 1701 and thus on a surface that is raised and angled in relation to the road surface 1703 .

The reference image section 1807, which is also referred to as an image section, is viewed at the reference position 1805 in the reference image 1801. According to one exemplary embodiment, the positions around the reference position 1805 are clearly marked in color.

19 represents a search image 1901 according to an embodiment in which the already based on 18 scene shown is depicted. With the based on the 1 until 16 A disparity was determined for the reference image detail 1807 using the voting method described for determining the disparity. According to one embodiment, the features that voted for this disparity receive the output color according to the reference position 1805.

20 sets that based on 18 described reference image 1801. A further reference position 2005 and a further reference image section 2007 are shown. The further reference image section 2007 is located on the road surface 1703.

21 represents a search image 1901 according to an embodiment in which the already based on 20 scene shown is depicted. With the based on the 1 until 16 A disparity was determined for the further reference image section 2007 using the voting method described for determining the disparity. According to one embodiment, the features that voted for this disparity receive the output color according to the reference position 2005.

In contrast to the in 19 shown result from the truck section, a parallelogram has been created here. The features are not only shifted purely translationally, but have been distorted in perspective. The amount of distortion corresponds to the surface normal.

That based on 17 until 21 The method illustrated for the temporal determination of disparity and normal vector by means of a voting method is particularly suitable for stereo image pairs such as the reference image 1801 and the search image 1901, which shows a scene with objects with different surface normals. The road surface 1703 on which the observed objects, for example the truck 1701, are placed can be seen preferably in the automotive area when sensing the vehicle surroundings.

In such a scene, it often makes sense to determine a surface normal in addition to the disparity in order to semantically separate objects from the road surface 1703 for better scene analysis.

Advantageously, these two variables can be determined in a robust manner in one step using a voting method. A second division that would otherwise be necessary, ie first determining the disparity and then calculating the surface normal in a local environment, can thus be combined in one method.

Starting from the scene in 17 two different image areas can thus be viewed as an example. On the one hand, an image section from truck 1701 can be viewed as an example, as shown in 18 is shown, on the other hand a section of the paving of the streets, or rather street surface 1703 or parking lot surface can be picked out, as it is in 20 is shown.

That based on 1 until 16 The voting method described uses features within these image sections, here the reference image section 1807 and the further reference image section 2007, and votes with the distance to the center of the image section for the disparity sought. The positions around the reference point, around which the corresponding reference image section 1807, 2007 spans, is in 18 marked in the truck cutout. The positions are marked in color as an example.

In the case of the truck cutout are in 19 the characteristics are presented that voted for the sought-after disparity. A uniformity in relation to the in 18 shown representation of the reference image detail 1807 to recognize. As a rule, the gaps cannot be avoided in real images, since not all features ideally occur in both images. The rectangular area of the reference section 1807 is also rectangular in the search image 1901 .

In the case of the road surface 1703, however, things are different. Here, too, an area was selected as an example and 21 found using the voting process in search image 1901 and marked in colour. What is striking is the parallelogram distortion of the underlying area, which reflects the position of the features voting for the disparity.

22 shows a block diagram of a method for determining a disparity according to an embodiment. The method represents a voting method for determining a disparity image 1500 and, in addition, a surface normal image 2200.

As already based on 15 described, a first image capturing device 601 is used to provide the first input image 101 and the second image capturing device 602 is used to provide the second input image 102 . The first input image 101 is rectified in step 1510 to obtain the first rectified image 501 . Correspondingly, the second input image 102 is subjected to rectification in step 1512 in order to obtain the second rectified image 502 .

The first rectified image 501 is transformed into the reference image 1101 in step 1514 and the second rectified image 502 is transformed into the search image 1302 in step 1516 .

In step 1520 the matching table 1220 is built for a pixel of the reference image 1101 . In step 1522 a vote is performed. For this purpose, position values 1402 are determined for each pixel of the search image 1302 using the search image 1302 and the matching table 1220 . In step 1524 the histogram is built. For this purpose, the frequency distribution 1400 of the determined position values 1402 educated. In step 1526, the disparity is determined. For this purpose, using the frequency distribution 1400, the disparity value 1528 is determined for that pixel of the reference image 1101 for which the steps 1520, 1522, 1524 have just been carried out.

Deviating from the description in 15 are in accordance with the basis of 22 described exemplary embodiment, a step 2230 of extracting and a step 2232 of determining a surface normal are additionally executed.

In step 2230 those pixels of the search image 1302 located in the search region are extracted for which in step 1522 that of the position values 1402 was determined that has the greatest frequency in the frequency distribution 1400 . In step 2232, the surface normal 2234 is then determined from positions of the pixels of the search image 1302 extracted in step 2230 of extraction.

Executing step 1529, as shown in FIG 15 described, a next pixel is selected at a further position p of the reference image 1101 and the steps 1520, 1522, 1524, 1526, 2230, 2232 are repeatedly carried out for this pixel in order to generate both a further disparity value 1528 and a further surface normal 2234 for this pixel to investigate.

According to an exemplary embodiment, steps 1520, 1522, 1524, 1526, 2230, 2232 are carried out for each pixel of the reference image 1101, i.e. for each position p from (n×m), where n and m represent the number of rows and columns in the reference image 1101.

Based on the surface normals 2234 determined in this way, the surface normal image 2200 of the 3D world is optionally determined.

In the 22 The basic processing steps shown in the form of a block diagram and their sequence are divided into the already based on 15 described block diagram into the voting step for each pixel. According to one exemplary embodiment, the result of the voting method is the disparity map and, at the same time, a map of the surface normals. For this purpose, after determining the disparity, the positions of the successful characteristics are determined and how using the 16 until 21 described, the surface normal 2234 is determined in the same processing step. The totality of all disparities and surface normals 2234 form the disparity image 1500 and the surface normal image 2200.

That already based on 15 For this purpose, the voting method described for determining the disparity image 1500 is expanded in accordance with the exemplary embodiment described here by the blocks which represent the steps 2230, 2232. The disparity and the normal vector of the surface reference point under consideration are simultaneously estimated from the voting procedure. The positions of the "successful" votes for the determined disparity result from the perspective distortion in the surface normal 2234 in each examined pixel.

The simultaneous estimation of disparities, also referred to as disparity values 1528, and surface normals 2234 can be used in any multi-camera system, for example, and can be used during a product change and/or update. The image capturing devices 601, 602 can be installed in motor vehicles, but can also be used in other domains, in which cameras sense the surroundings and disparities together with the associated surface normals 2234 are searched for. Thus, the approach described can be used, for example, in driver assistance systems, both for the front sensors and in combination with other vehicle cameras.

23 shows a flowchart of a method for determining a disparity and additionally a surface normal according to an embodiment. The method can be carried out, for example, using an apparatus as is based on FIG 6 is shown.

In addition to the based on 16 The required or optional steps 1510, 1512, 1514, 1516, 1520, 1522, 1524, 1526, 1630 already described in 23 shown flowchart also the already based on 22 described steps 2230, 2232.

In step 2230, the most frequent position value is first accessed, ie that position value which has the greatest frequency in the frequency distribution. Then those pixels of the search image located in the search region are extracted for which this most frequent position value was determined in step 1522 . In step 2232, the surface normal 2234 is then determined from positions of the pixels of the search image 1302 extracted in step 2230 of extraction.

QUOTES INCLUDED IN DESCRIPTION

This list of documents cited by the applicant was generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Patent Literature Cited

• DE 19636028 C1 [0003]

Claims (14)

A method of determining a disparity, the method comprising the steps of: reading in a first rectified image (501) and a second rectified image (502); Transforming (1514, 1516) the first rectified image (501) into a reference image (1101; 1801) using a transformation rule which assigns a feature value from a predetermined set of feature values (1222) to each pixel of the first rectified image (501), and des second rectified image (502) into a search image (1303; 1901) using the transformation rule which assigns a feature value from the predetermined set of feature values (1222) to each pixel of the second rectified image (502); constructing (1520) a matching table (1220) for a pixel of the reference image (1101; 1801) by assigning each feature value from the predetermined set of feature values (1222) distances (1224) between a position (1110) of the pixel in the reference image (1101; 1801) and at least positions of the corresponding feature value in the reference image (1101; 1801) located in a reference region (203) around the position (1110) of the pixel; Determination (1522) of position values (1402) for at least each pixel of the search image (1303; 1901) located in a search region (303) by for each pixel of the search image (1303; 1901) located in the search region (303) the corresponding feature value is read from the search image (1303; 1901), the distances (1224) assigned to the corresponding feature value are read from the matching table (1220) and the position values (1402) are determined from the position (1110) of the pixel and the distances (1224). will; forming (1524) a frequency distribution (1400) for the position values (1402) of the pixels of the search image (1303; 1901) located in the search region; and Determining (1526) a disparity value (1528) for the pixel of the reference image (1101; 1801) from the position (1110) of the pixel in the reference image (1101; 1801) and that of the position values (1402) which has the greatest frequency in the frequency distribution (1400). procedure according to claim 1 , with a step of reading in a first input image (101) via an interface to a first image acquisition device (601) and a second input image (102) via an interface to a second image acquisition device (602) and a step (1510, 1512) of rectifying the the first input image (101) to obtain the first rectified image (501) and the second input image (102) to obtain the second rectified image (502). Method according to one of the preceding claims, in which, in the step (1514, 1516) of transforming by the transformation rule, a feature is determined according to BRIEF (Binary Robust Independent Elementary Features), ORB, BRISK or Census. Method according to one of the preceding claims, in which, in the step (1514, 1516) of transforming, the predetermined set of feature values (1222) is restricted by the transformation rule. Method according to one of the preceding claims, in which the steps (1520, 1522, 1524, 1526) of constructing, determining, forming and determining are carried out repeatedly for at least one further pixel of the reference image (1101; 1801) in order to determine at least one further disparity value for to determine the at least one further pixel of the reference image (1101; 1801). Method according to one of the preceding claims, in which the steps (1520, 1522, 1524, 1526) of constructing, determining, forming and determining are carried out repeatedly for all pixels of the reference image (1101; 1801) in order for each pixel of the reference image (1101 ; 1801) to determine a disparity value. procedure according to claim 2 , comprising a step (1630) of capturing the first input image (101) using the first image capturing device (601) and the second input image (102) using the second image capturing device (602). procedure according to claim 7 wherein one of the image capturing devices (601, 602) is a front camera and the other of the image capturing devices (601, 602) is a side camera of a vehicle (600). Method according to one of the preceding claims, with a step (2230) of extracting those pixels of the search image (1303; 1901) located in the search region (303) for which that of the position values (1402) is determined in the determining step (1522). which has the greatest frequency in the frequency distribution (1400), and with a step (2232) of determining a surface normal (2234) from positions of the pixels of the search image (1303; 1901) extracted in step (2230) of extraction. procedure according to claim 9 , in which the steps (2230, 2232) of extracting and determining a surface normal (2234) for at least one further pixel of the reference image (1101; 1801) are carried out repeatedly in order to determine at least one further surface normal (2234) for the at least one further pixel of the to determine the reference image (1101; 1801). Device (610) for determining a disparity, wherein the device (610) is set up to carry out and/or to control the steps of the method according to one of the preceding claims in corresponding units. Vehicle (600) with a first image capturing device (601), a second image capturing device (602) and a device (610) according to claim 11 . Computer program that is set up to carry out the steps of the method according to one of Claims 1 until 10 execute and/or control. Machine-readable storage medium on which the computer program Claim 13 is saved.

Images