DE102017216854B4 - Method and device for evaluating correspondences of images, operation assistance method and operation device - Google Patents

Abstract

Method (S) for evaluating correspondences (10, 12) of images (B1, B2) from at least one camera of a vehicle, the correspondences (10, 12) being based on the concept of optical flow or reflecting a stereo disparity, with:- Providing (S1-1) the correspondences (10, 12) between given first and second images (B1, B2), - providing (S1-2) a plurality of quality measures as attributes (20, 22) for characterizing a respective correspondence ( 10, 12), the respective quality measure as an attribute (20, 22) of a correspondence (10, 12) under consideration being representative of • a measure of the uniqueness with which a respective correspondence (10, 12) was determined in a local environment, or• a measure of the consistency of a respective correspondence (10, 12) with a chronologically preceding correspondence, - evaluation (S2-1) and conditional selection (S2-2) of the correspondences (10, 12), wherein the evaluation (S2-1 ) of correspondence (10, 12) a combination of attributes (20, 22) and the selection (S2-2) of correspondence (10, 12) a result of the evaluation (S2-1) are taken as a basis, and, in which • the correspondences (10, 12), in particular in combination with the associated attributes (20, 22), are represented as an image (10, 20) or matrix (10, 20), • the image (10, 20) or the Matrix (10, 20) are covered by non-overlapping sub-regions (15), wherein a tiling with rectangular tiles with row-by-row offset joints is formed, and• for each sub-region (15) with respect to correspondences (12 ) a correspondence (12) which is best with regard to the assigned attributes (22) is selected, and providing (S2-3) selected correspondences (10, 12) as the evaluation result (100).

Description

State of the art

The present invention relates to a method and a device for evaluating correspondences of images, an operating assistance method and in particular a driving assistance method and an operating device and in particular a vehicle. The present invention further relates to a computer program and a machine-readable storage medium.

In many technical areas, image processing is used to control devices and processes, for example in the automotive industry with so-called driver assistance systems. In this context, for example, images are recorded and used as a basis for a control process. To do this, the images must be evaluated. When evaluating images, so-called correspondences are often generated and used as a basis for the evaluation. Such correspondences describe associations between coordinates in a first image and coordinates in a second image, pixel by pixel and in a temporal and/or spatial direction. A temporal relationship can involve sequentially recorded images, while a spatial relationship can involve spatially separate images that can also be recorded simultaneously, for example in stereo vision.

The paper YAMAGUCHI K. ET AL: "Vehicle ego-motion estimation and moving object detection using a monocular camera"; IEEE 2006, 18th International Conference on Pattern Recognition, IEEE Comput. soc; 20-24 SEPT. 2006 discloses a method for camera-based estimation of a vehicle's own motion.

The document U.S. 2010/0166323 A1 discloses a method for comparing images, in particular based on pixels.

The object of the present invention is to improve a method for comparing images.

Disclosure of Invention

The present object is achieved according to the invention by the independent claims 1 and 7 to 10.

1. (i) Bereitstellen von Korrespondenzen zwischen gegebenen ersten und zweiten Bildern,
2. (ii) Bereitstellen eines oder einer Mehrzahl von Qualitätsmaßen als Attribute zur Charakterisierung einer jeweiligen Korrespondenz,
3. (iii) Bewerten und bedingtes Selektieren der Korrespondenzen und
4. (iv) Bereitstellen selektierter Korrespondenzen als Bewertungsergebnis.

The method according to the invention for evaluating images with the features of claim 1 has the advantage that correspondences to a pair of images can be checked with high reliability and selected for verification with comparatively little effort. According to the invention, this is achieved with the features of claim 1 in that a method for evaluating images and in particular for evaluating correspondences of images is created, which has the following steps:

1. (i) providing correspondences between given first and second images,
2. (ii) providing one or a plurality of quality measures as attributes for characterizing a respective correspondence,
3. (iii) evaluating and conditionally screening the correspondences and
4. (iv) Provision of selected correspondence as an evaluation result.

According to the invention, the evaluation of correspondence is based on a combination of attributes and the selection of correspondence is based on a result of the evaluation. As a result of the measures provided according to the invention, a high degree of selectivity and accuracy is achieved for correspondences that have already been determined due to the use of attributes combined with one another for the correspondences when evaluating when selecting the correspondences. Due to the selection, the amount of data can be reduced without loss of accuracy during further processing or further use.

Within the meaning of the present invention, all of the correspondences can also be understood as a correspondence image, set or matrix, the individual components of which are then also referred to as elements or pixels. The individual correspondences can have one or more scalar components, for example values for different directions of movement u, v in space. They can thus be vector-valued. Furthermore, the correspondences can be based on the concept of the optical flow, also referred to as OF, and/or reflect a stereo disparity, but this is not mandatory.

In an analogous manner, the entirety of the quality measures, attributes and/or selected correspondences can also be understood as corresponding images, data sets and/or matrices with corresponding elements or pixels.

The present invention is applicable in connection with any form of cameras, surveillance cameras and/or metrology cameras, for example - but not only - on or in vehicles in general, in particular in connection with motor vehicles. This also includes applications of the invention in Related to television cameras, consumer cameras, drone cameras, man-machine interaction cameras, etc.

The dependent claims show preferred developments of the invention.

In order to implement the respective quality measures or attributes for the correspondence considered in each case, a wide variety of aspects that enable correspondence to be evaluated can be used individually or in any combination with one another.

• - für ein Maß der Eindeutigkeit, mit welcher eine jeweilige Korrespondenz in einem lokalen Umfeld ermittelt wird oder wurde, oder
• - für ein Maß der Konsistenz einer jeweiligen Korrespondenz mit einer zeitlich vorangegangenen Korrespondenz, insbesondere im Hinblick auf ein Maß von Konsistenz von Flussvektoren über die Zeit, wobei bei bestätigter Konsistenz ein zu Grunde liegendes zeitlich stabiles Verhalten insbesondere in Form eines Alters als Zahlenwert angegeben wird.

Gemäß weiteren bevorzugten Ausgestaltungsformen des erfindungsgemäßen Verfahrens ist es möglich, dass ein jeweiliges Qualitätsmaß als Attribut einer betrachteten Korrespondenz repräsentativ ist für mindestens einen der folgenden Aspekte oder einer Kombination davon, also z.B.

• - für ein Maß einer - zumindest lokalen - Ortsabhängigkeit und/oder Raumwinkelabhängigkeit der Verteilung der Korrespondenzen zumindest im Bereich einer jeweils betrachteten Korrespondenz und insbesondere für ein Maß der Gleichmäßigkeit einer ortsabhängigen und/oder raumwinkelabhängigen Verteilung der Korrespondenzen,
• - für ein Maß einer Priorität zur bevorzugten Berücksichtigung einer zu Grunde liegenden Korrespondenz,
• - für ein Maß für das Vorliegen einer korrekten Zuordnung einer jeweiligen Korrespondenz zu Bereichen der zu Grunde liegenden Bilder,
• - für ein Maß einer Eigenbewegung der mit der jeweiligen Korrespondenz im Zusammenhang stehenden Bereiche der zu Grunde liegenden Bilder, insbesondere eines damit im Zusammenhang stehenden Objekts und/oder unter Berücksichtigung nicht epipolarer konformer Bewegungen,
• - für ein Maß einer Bewegungsmöglichkeit der mit der jeweiligen Korrespondenz im Zusammenhang stehenden Bereiche der zu Grunde liegenden Bilder, insbesondere eines damit im Zusammenhang stehenden Objekts,
• - für ein Maß einer Genauigkeit, mit welcher eine jeweilige Korrespondenz bestimmt wurde,
• - für ein Maß einer bei der Ermittlung der jeweiligen Korrespondenz zu Grunde gelegten Auflösung oder Auflösungsstufe,
• - für ein Maß einer Zugehörigkeit der mit der jeweiligen Korrespondenz im Zusammenhang stehenden Bereiche der zu Grunde liegenden Bilder, insbesondere eines damit im Zusammenhang stehenden Objekts, zu einer semantischen Klasse, vorzugsweise auf der Grundlage einer videobasierten Objektklassifikation, eines Vorgangs maschinellen Lernens und/oder eines Vorgangs der semantischen Segmentierung,
• - für ein Maß einer Zugehörigkeit der mit der jeweiligen Korrespondenz im Zusammenhang stehenden Bereiche der zu Grunde liegenden Bilder, insbesondere eines damit im Zusammenhang stehenden Objekts, zu einer Klasse zur Berücksichtigung zugelassener Bereiche der zu Grunde liegenden Bilder, insbesondere im Zusammenhang mit einer Maskierung.

According to the invention, the respective quality measure as an attribute of a correspondence considered is representative of

• - for a degree of uniqueness with which a respective correspondence is or was determined in a local environment, or
• - for a measure of the consistency of a respective correspondence with a chronologically preceding correspondence, in particular with regard to a measure of the consistency of flow vectors over time, with confirmed consistency being based on a temporally stable behavior, in particular in the form of an age as a numerical value.

According to further preferred embodiments of the method according to the invention, it is possible that a respective quality measure as an attribute of a correspondence considered is representative for at least one of the following aspects or a combination thereof, ie for example

• - for a measure of an - at least local - location dependency and/or spatial angle dependency of the distribution of the correspondences at least in the area of a correspondence considered in each case and in particular for a measure of the uniformity of a location-dependent and/or spatial angle-dependent distribution of the correspondences,
• - for a measure of priority for preferential consideration of an underlying correspondence,
• - for a measure of the existence of a correct assignment of a respective correspondence to areas of the underlying images,
• - for a measure of a proper motion of the areas of the underlying images related to the respective correspondence, in particular of a related object and/or taking into account non-epipolar conformal motions,
• - for a measure of a possibility of movement of the areas of the underlying images related to the respective correspondence, in particular of an object related to it,
• - for a degree of accuracy with which a respective correspondence was determined,
• - for a measure of a resolution or resolution level used as a basis for determining the respective correspondence,
• - for a measure of an affiliation of the areas of the underlying images related to the respective correspondence, in particular a related object, to a semantic class, preferably based on a video-based object classification, a machine learning process and/or a process of semantic segmentation,
• - for a degree of affiliation of the areas of the underlying images associated with the respective correspondence, in particular an object associated therewith, to a class for considering permitted areas of the underlying images, in particular in connection with a masking.

A particularly high degree of flexibility is achieved in another advantageous embodiment of the method according to the invention in that combined quality measures are used as attributes which are derived from one or a plurality of quality measures through combination formation and/or function formation, in particular through reciprocal value formation, sign reversal, Rounding, formation of functional values from the scalar or vectorial values on which the quality measures are based.

The specific sequence of the method according to the invention can be designed particularly simply if, according to another advantageous development of the method according to the invention, a respective attribute and/or underlying quality measures and in particular their values are used in encoded form for use as an address, in particular in connection with a value of the Attributes and/or the selection table representing quality measures and in particular a lookup table.

Additionally or alternatively, a further simplification and economization of the method according to the invention occurs when, according to another embodiment, a respective correspondence is represented as a one-dimensional list or as a vector and an attribute and/or its code assigned to the correspondence tion is represented as one or more additional list elements or vector components or as parts thereof and is or are appended to the list or vector.

In specific applications, not all values of the quality measures and/or attributes to be consulted may be available at the time the correspondences are derived.

It is also advantageous in connection with such situations if attributes and/or quality measures relating to correspondence at an earlier point in time are transferred and/or extrapolated to a correspondence at a later point in time.

The method according to the invention is economical because the correspondences, in particular in combination with the assigned attributes in their entirety, are actually represented as an image, matrix or the like, the image or the matrix being represented by overlapping and/or non-overlapping partial regions—like a tessellation or tiling - are covered and for each sub-region in relation to correspondences present in the sub-region - if available - a best correspondence is selected - in particular with regard to the assigned attributes.

The present invention further relates to an operating assistance method and in particular a driving assistance method for a device and in particular for a vehicle, in which images are captured and evaluated using a method according to a method according to the invention and in which a result of the evaluation is used to control the operation of the device .

According to another aspect of the present invention, a device for evaluating images and in particular for evaluating correspondences between images is also specified, which is set up to execute an inventive operating assistance method or driving assistance method or an inventive method for evaluating images and in particular correspondences to image pairs.

The device according to the invention can be designed in particular as an ASIC, as a freely programmable digital signal processing device or as a combination thereof.

Furthermore, the present invention specifies a computer program which is set up to carry out a method according to the invention when it is run on a computer or a digital signal processing device.

Furthermore, the present invention also creates a machine-readable storage medium on which the computer program according to the invention is stored.

character list

• 1 und 2 zeigen schematisch in Form eines Flussdiagramms bzw. nach Art eines Blockdiagramms Ausführungsformen des erfindungsgemäßen Verfahrens und illustrieren die Zusammenhänge zwischen den zu Grunde liegenden Bildern und Korrespondenzen, Attributen und weiteren Informationen.
• 3 bis 5 zeigen schematisch Ergebnisse der Anwendung einer Ausführungsform des erfindungsgemäßen Verfahrens anhand entsprechender Bilder von Korrespondenzverteilungen.
• 6 zeigt schematisch einen möglichen Aufbau der Darstellung eines Flussvektors als Korrespondenz mit angehängten Attributen.
• 7 und 8 zeigen schematisch die Anwendung einer Parkettierung oder Kachelung zur Überdeckung eines Bildes von Korrespondenzen.
• 9A bis 9C beschreiben schematisch die Anwendung einer Maske und einen diesbezüglichen zeitlichen Übertrag.

Embodiments of the invention are described in detail with reference to the attached figures.

• 1 and 2 show schematically in the form of a flow chart or in the form of a block diagram embodiments of the method according to the invention and illustrate the relationships between the underlying images and correspondence, attributes and other information.
• 3 until 5 show schematic results of the application of an embodiment of the method according to the invention based on corresponding images of correspondence distributions.
• 6 shows schematically a possible structure of the representation of a flow vector as a correspondence with attached attributes.
• 7 and 8th show schematically the use of tiling to cover an image of correspondences.
• 9A until 9C describe schematically the use of a mask and a related temporal transfer.

Preferred Embodiments of the Invention

In the following, with reference to the 1 until 9C Embodiments of the invention and the technical background are described in detail. Elements and components that are the same and equivalent, and elements and components that have the same or equivalent effect, are denoted by the same reference symbols.

The detailed description of the designated elements and components is not reproduced in every case of their occurrence.

The features and other properties shown can be isolated from one another in any form and combined with one another in any way, without departing from the core of the invention.

The 1 and 2 show schematically in the form of a flow chart or in the form of a block diagram embodiments of the invention Application of the method S and illustrate the connections between the underlying images B1, B2 and correspondence 12, attributes 22 and other information 32, which here correspond to the underlying images B1, B2 as corresponding images or matrices 10, 20 or 30 are given with corresponding pixels or elements.

The images B1, B2, 10, 20, 30 can have different sizes and/or formats or else one size. In particular, attribute images can also be smaller in order to save storage space and/or bandwidth. In particular, they can be half the size of the input images horizontally and vertically and scaled up if necessary.

The embodiment of the method S according to the invention for evaluating images B1, B2 and in particular their correspondences 12 consists of a first step S1, in which in a first partial step S1-1 the correspondences 12 to the images B1, B2 are provided as such, for Example in the form of a corresponding corresponding image 10 of the correspondence 12, which can also be understood as a sentence, matrix or memory area. In this context, the individual correspondences 12 are also understood as elements or pixels.

In a second sub-step S1-2, based on an analysis of the correspondences 12, corresponding attributes 22, for example in the form of quality measures, are derived and made available. All of the attributes 22 can in turn be understood as an image 20, set, matrix or memory area, with the individual attributes 22 then in turn being contained as elements or pixels.

If necessary, additional information 32 is generated and provided either in addition to the attributes, for example contained in them, or as a separate entity in the form of an image 30, a sentence, a matrix or a memory area. However, this is not mandatory, but often advantageous.

The individual partial steps S1-1 and S1-2 of the first method step S1 of the method S can be processed in parallel and, for example, simultaneously or serially.

In a second step S2 of the method S according to the invention, the steps of evaluating S2-1 of the correspondence 12, conditionally selecting S2-2 of the correspondence 12 and providing S2-3 of the selected correspondence 12 as the evaluation result take place.

In 1 shows that the selected correspondences 12 are again provided in the form of an image 100, wherein the image 100 can also be understood as a set, matrix or memory area. However, this procedure is not mandatory.

If the selected correspondences represent only a small fraction of the input quantity of correspondences, it can be very advantageous to no longer display the selection as an image or matrix, but in a more compact form, e.g. as a list.

Step S3 of determining a relative orientation and/or a proper motion on the basis of the selected correspondences 12 follows. Due to the selection, the amount of data can be reduced and the reliability and accuracy can be increased.

Various other processes can follow on the basis of the determined relative orientation and/or the self-movement, for example a process S4 of determining the class of self-moving objects or the like, and alternatively or additionally specific applications S4, S5.

2 shows a representation in the form of a block diagram, which gives more detailed information about the data flow, in particular with regard to the influence of the data streams 35 and 36, namely the stream of correspondence 12 and the data on proper motion and/or on the relative orientation with regard to the classification self-moving objects according to the method step S4 and its influence via the feedback 34 on the second method step S2 of the method S and in particular the process 2-2 of conditional selection, specifically in the sense of supplying additional information 30.

The 3 until 5 show schematic results of the application of an embodiment of the method S according to the invention using corresponding images 10, 100 of correspondence distributions, specifically in the original form of the correspondences 12 in 3 , in a masked form of the correspondences 12 in 4 as well as after selection of correspondence 12 in 5 . This is discussed in more detail in subsequent sections below.

6 shows a possible structure of the representation of a flow vector, for example in a word with a length of 32 bits, as correspondence 12 with an attached attribute 22 as a separate code section, which is also referred to as QCode in this context and sections for 6 quality characteristics QM0 to QM05 includes. In this context, the original correspondence 12 is shown modified by appending the attribute 22 to form a modified correspondence 12'.

The 7 and 8th show schematically the application of tiling or tiling to cover an image 10 of correspondence 12.

In 7 the entirety of all provided correspondences 12 is shown in the manner of a correspondence matrix 10 and can also be understood as an image which corresponds in its structure and construction to the corresponding underlying images B1 and B2. However, this procedure is not mandatory, the matrix 10 and the images B1, B2 can have different sizes in comparison to one another. Correspondingly, corresponding elements or pixels 11 are formed in the image or in the matrix 10, which can contain one or more correspondences 12, but do not have to, ie certain pixels 11 can be empty if they do not contain any correspondence 12.

To cover the correspondence matrix 10 with the original correspondences 12, a tessellation or tiling with partial areas or partial regions 15 is generated. The individual partial areas or partial regions 15 do not overlap and in their entirety cover the entire correspondence image 10 .

Certain partial regions 16 have no correspondence 12. Certain partial regions 17 are shown in FIG 7 shaped differently and have a different extent because they are on the edge and ensure an offset of the sub-regions 15 against each other.

A preferred correspondence 13 is determined and selected for each of the partial regions 15 by a comparison of all correspondences 12 within the respective partial region 15, provided it is not empty. This is discussed in more detail in the sections below.

8th considers a single tile 15 of the coverage of the correspondence matrix 10, the path of the arrows 19 showing the path of processing the tile 15 with respect to the individual pixels 11 and the individual correspondences 12. The path of processing ensures that all pixels 11 of the tile or sub-region 15 are reached. The path of processing the tile or partial region 15 can also be selected differently, for example preferably horizontally instead of vertically, spirally or in a random order.

The 9A until 9C describe schematically the use of a mask and a related temporal transfer in the sense of a warping. These aspects are detailed in further sections below.

• Die vorliegende Erfindung betrifft unter anderem das so genannte Priority-Sampling und alternativ oder zusätzlich Aspekte der Bewertung und Selektion bevorzugter Kandidaten aus einer Menge oder einem Feld von Korrespondenzen 12 zu Bildpaaren B1, B2 oder allgemein zu einer Mehrzahl von Bildern B1, B2.

These and other features and properties of the present invention are further explained on the basis of the following explanations:

• The present invention relates, among other things, to what is known as priority sampling and alternatively or additionally to aspects of evaluating and selecting preferred candidates from a set or a field of correspondences 12 for image pairs B1, B2 or generally for a plurality of images B1, B2.

According to an alternative view of the present invention, this also relates to the evaluation and selection of correspondences 12 to image pairs B1, B2 or generally to a plurality of images B1, B2 and/or aspects of improving camera-based estimates of the relative orientation.

Furthermore, according to a further additional alternative point of view, the present invention relates to aspects of methods and devices for improving video-based self-motion estimations.

The issue of correspondence formation is encountered in the fields of machine vision, computer vision, particularly in connection with the notion of optical flow (OF), and in stereo disparity.

Correspondences in the temporal direction are formed in connection with the optical flow by assignments between coordinates in a first image B1 and coordinates in a second image B2 being determined. Such correspondence then indicates how the projection of a point in the 3D scene into the 2D image has progressed from an old coordinate to a new coordinate.

The movement in the image B1, B2 can be caused by the movement of the scene point or by the movement of the camera, or both at the same time.

In the case of stereo vision, the two images B1, B2 are recorded approximately at the same time by two cameras which are located at different locations. The relative arrangement of the cameras is usually fixed and known. Correspondence formation enables the distance to the point in the 3D scene to be determined using triangulation.

For example, “The KITTI Vision Benchmark Suite” at http://www.cvlibs.net/datasets/kitti provides an overview of the best methods in the prior art, in particular with regard to driver assistance systems.

In a camera-based determination of the relative orientation, a plurality of correspondences and in particular point correspondences between two images B1, B2 are used in order to use them to calculate the relative orientation of the camera positions in relation to the scene—often assumed to be static.

The images B1, B2 can also come from two different cameras, which can be mechanically rigidly connected to one another.

If the two images B1, B2 come from the same camera, one also speaks of a self-motion estimation. The relative movement - in particular in relation to the scene - that the camera has completed in the meantime is determined. This movement can generally be described as a combination of a 3D rotation and a 3D translation.

Within the meaning of the present invention, aspects of the estimation or determination of one's own motion S3 and/or aspects of the estimation or determination S3 of a relative orientation, which also include the multi-camera case, are treated equally and - especially if they are mentioned in isolation - in each case treated purely by way of example . All technical aspects discussed in connection with a self-motion estimation are equally advantageously applicable to technical aspects of the estimation of a relative orientation and in particular with regard to a multi-camera case.

The present invention also deals in particular with an issue of the selection S2-2 of a subset of correspondences that is as suitable and, in particular, optimal as possible from the totality of all derived correspondences.

• - Ein nachfolgender Verfahrensschritt oder Algorithmus, z.B. einer Bildbewertung und/oder Bildbearbeitung, besitzt keine ausreichende Kapazität, um die Gesamtheit aller bereitgestellten Korrespondenzen - insbesondere in Echtzeit - zu verarbeiten.
• - In der Gesamtheit aller bereitgestellten Korrespondenzen liegen ungeeignete Korrespondenzen vor, die vor der weiteren Verarbeitung besser herausgefiltert werden sollten.

Such a choice or selection can be useful or necessary, for example, if one or more of the following circumstances apply:

• - A subsequent method step or algorithm, for example an image evaluation and/or image processing, does not have sufficient capacity to process the entirety of all correspondence provided, particularly in real time.
• - In the entirety of all correspondence provided, there are unsuitable correspondences that should be better filtered out before further processing.

• - fehlerhafte Korrespondenzen, also zum Beispiel falsche Zuordnungen, und
• - Korrespondenzen zwischen Punkten der Szene, die zu eigenbewegten Objekten gehören, also nicht zu dem Teil der Szene gehören, auf den sich die Bestimmung der relativen Orientierung beziehen soll. Beispiele für eigenbewegte Objekte sind Fußgänger, fahrende Autos, ziehende Wolken, Spiegelungen auf reflektierenden Oberflächen.
• - Die Korrespondenzen wurden mit unterschiedlicher Genauigkeit ermittelt. Dann würde man die Korrespondenzen mit höherer Genauigkeit bevorzugen.
• - Die Korrespondenzen sind ungleichmäßig im Bild oder im Raum verteilt. Dann könnte die Selektion die Gleichmäßigkeit verbessern. Denn für die Bestimmung der relativen Orientierung ist es vorteilhaft, wenn die Korrespondenzen aus unterschiedlichen Raumrichtungen stammen.

Examples of this case are:

• - Incorrect correspondence, for example incorrect assignments, and
• - Correspondences between points in the scene that belong to self-moving objects, i.e. do not belong to the part of the scene to which the determination of the relative orientation should relate. Examples of self-moving objects are pedestrians, moving cars, moving clouds, reflections on reflective surfaces.
• - The correspondences were determined with varying degrees of accuracy. Then one would prefer the correspondences with higher accuracy.
• - The correspondences are distributed unevenly in the picture or in space. Then selection could improve uniformity. For the determination of the relative orientation, it is advantageous if the correspondences come from different directions in space.

According to the invention, the possibility is created to place requirements - e.g. in the sense of quality criteria - on the selection and to select those correspondences that fulfill them or even fulfill them in the best possible way.

1. (1) Zum Beispiel kann ein Maß ermittelt werden, das eine Aussage darüber ermöglicht, wie eindeutig eine Korrespondenz 12 in einem lokalen Umfeld ermittelt werden konnte. Beispielsweise kann ermittelt werden, wenn die Korrespondenz entlang einer Kante oder auf Grund periodischer Strukturen mehrdeutig sein kann. Das Maß für die Eindeutigkeit kann als Zahlenwert codiert werden.
2. (2) Alternativ oder zusätzlich kann eine Korrespondenz jeweils mit ihrer zeitlichen Vorgängerkorrespondenz verglichen und auf Konsistenz der Flussvektoren über die Zeit geprüft. Bei bestätigter Konsistenz liegt ein zeitlich stabiles Verhalten vor. Ist diese Konsistenz immer wieder erfüllt, so erhöht sich das Vertrauen, das z.B. in Form eines Alters als Zahlenwert angegeben werden kann.

Different classes of quality criteria can be used in connection with the present invention:

1. (1) For example, a measure can be determined that enables a statement to be made as to how clearly a correspondence 12 could be determined in a local environment. For example, it can be determined if the correspondence along an edge or due to periodic structures can be ambiguous. The measure of uniqueness can be encoded as a numerical value.
2. (2) Alternatively or additionally, a correspondence can be compared in each case with its chronological predecessor correspondence and checked for consistency of the flow vectors over time. If the consistency is confirmed, the behavior is stable over time. If this consistency is met again and again, the level of trust increases, which can be given as a numerical value, for example in the form of an age.

The method according to the invention can be used in existing or future generations of driver assistance systems—for example as an implementation in an ASIC - can be used to select the best candidates for a self-motion estimation from optical flow correspondences.

In addition to the two quality criteria (1) and (2) mentioned above, further quality criteria can be used, for example the quality criteria described below. All of the quality criteria can be used individually or in any combination with one another in order to select or select S2-2 from correspondences that have already been derived.

This is explained below using a specific exemplary embodiment. This exemplary embodiment corresponds to a possible implementation in a product.

First, an optical flow field is determined. Each flow vector - understood as correspondence 12 - can be assigned 6 different quality measures and appended to the data structure of the flow vector. A priority can be calculated from the quality measures, as is the case in connection with 3 is shown. In addition, further features--in the sense of additional information 30, 32--can be taken into account, for example a mask that is assigned to an earlier point in time and identifies pixels to be excluded, as is the case in connection with 4 is shown. A preferred selection of correspondence 12 can be derived and generated from all of the information, as is the case in connection with 5 is shown.

• - Eine Auswahl einer bevorzugten Untermenge von Korrespondenzen 12 aus einer Gesamtheit bereitgestellter Korrespondenzen 12 kann unter Berücksichtigung einer beliebigen Kombination der nachfolgenden Kriterien erfolgen:
• - Gleichmäßigkeit: Die Abtastpunkte sollen möglichst gleichmäßig über das Bild 10 bzw. über das Feld 10 von Korrespondenzen 12 verteilt sein.

Some core aspects and advantages of the invention are explained in more detail below:

• A selection of a preferred subset of correspondences 12 from a totality of provided correspondences 12 can be made taking into account any combination of the following criteria:
• - Uniformity: The scanning points should be distributed as evenly as possible over the image 10 or over the field 10 of correspondence 12.

Instead of uniformity in image 10, uniformity in space can also be aimed for, e.g. via a constant number of correspondences per solid angle.

• - Priorität: Es sollen bevorzugt die besten Korrespondenzen 12 gemäß einer Priorität ausgewählt werden.
• - Optionale Informationen 30, 32 können zusätzlich zum Aspekt der Priorität berücksichtigt werden oder in die Bestimmung der Priorität einfließen.
• - Qualitätsmaße: Es können unterschiedliche Qualitätsmaße für die bestehenden Korrespondenzen 12 erzeugt und genutzt werden, die bevorzugt in codierter Form vorliegen und z.B. an die jeweilige Korrespondenz angehängt oder ihr zugeordnet werden.
• - Kombination von Qualitätsmaßen: Es können mehrere bestehende Qualitätsmaße miteinander kombiniert werden, um daraus ein neues Qualitätsmaß oder eine neue Priorität zu ermitteln und einer jeweiligen Korrespondenz zuzuordnen.
• - Zusammenfassung von Qualitätsmaßen: Mehrere - insbesondere codierte - Qualitätsmaße können zusammengefasst werden zur Nutzung als Adresse einer Lookup-Tabelle (LUT) oder Auslesetabelle, die die Adresse auf eine Skala von Qualitäten und/oder Prioritäten abbildet.

Even more generally—instead of uniformity—a specific location-dependent and/or solid-angle-dependent density profile of the correspondences can be sought.

• - Priority: The best correspondence 12 should preferably be selected according to a priority.
• - Optional information 30, 32 can be taken into account in addition to the aspect of priority or incorporated into the determination of priority.
• - Quality measures: Different quality measures can be generated and used for the existing correspondence 12, which are preferably available in coded form and, for example, are appended to the respective correspondence or assigned to it.
• - Combination of quality measures: Several existing quality measures can be combined with one another in order to determine a new quality measure or a new priority and assign it to a respective correspondence.
• - Summary of quality measures: Several - in particular encoded - quality measures can be summarized for use as the address of a lookup table (LUT) or selection table, which maps the address to a scale of qualities and/or priorities.

• - Trainingsdaten: Möglich ist die Erzeugung einer solchen LUT anhand von Trainingsdaten.
• - Weitere Informationen: Denkbar ist die Einbeziehung von weiteren Informationen, wobei diese auch von anderen Systemkomponenten, Verfahren und/oder Algorithmen stammen können, z.B. von einer videobasierten Objektklassifikation oder von einer auf maschinellem Lernen basierenden semantischen Segmentierung.

The use of a LUT makes sense if there is a finite and not too large number of combinations of quality measures and/or other attributes, so that a priority can be assigned to each such combination, which can also be represented as a number. This assignment can then be saved explicitly - and usually once - as a LUT and called up again and again. This procedure can be cheaper in terms of computational effort than constantly determining the priority anew.

• - Training data: It is possible to generate such a LUT using training data.
• - Further information: The inclusion of further information is conceivable, whereby this can also originate from other system components, methods and/or algorithms, eg from a video-based object classification or from a semantic segmentation based on machine learning.

• - Informationen über die Bewegungsmöglichkeiten eines Objekts oder eine Objektklasse (Beispiel: Korrespondenzen 12 auf Fußgängern könnten vorsichtshalber grundsätzlich ausgeschlossen werden, egal ob der Fußgänger gerade in Bewegung ist oder nicht),
• - Informationen über das Vorhandensein eigenbewegter Objekte, insbesondere mit nicht epipolar konformer Bewegung, (Hintergrund: Bezieht sich die Bestimmung der relativen Orientierung auf eine als statisch angenommene Szene, so verhalten sich alle zugehörigen Korrespondenzen 12 epipolar konform, beziehen sich also auf denselben Epipol.)
• - Informationen in Form binärer Masken oder in Form von Bildern von Qualitätsmaßen oder Prioritäten und/oder
• - Informationen, die einem älteren Zeitpunkt zugeordnet sind und unter Verwendung des optischen Flusses (Korrespondenzen 12) auf einen gewünschten Zeitpunkt übertragen werden können.
• - Informationen, die einem älteren Zeitpunkt zugeordnet sind und unter Verwendung einer bekannten Eigenbewegung auf einen gewünschten Zeitpunkt übertragen werden können.

The following aspects are conceivable

• - Information about the movement options of an object or an object class (example: correspondences 12 on pedestrians could be ruled out as a precaution, regardless of whether the pedestrian is currently moving or not),
• - Information about the presence of self-moving objects, especially with non-epi polar conform movement, (Background: If the determination of the relative orientation refers to a scene that is assumed to be static, then all associated correspondences 12 behave epipolar conform, i.e. they refer to the same epipole.)
• - information in the form of binary masks or in the form of images of quality measures or priorities and/or
• - Information associated with an older point in time that can be transferred to a desired point in time using the optical flow (correspondence 12).
• - Information associated with an earlier point in time that can be transferred to a desired point in time using known proper motion.

According to the invention, these aspects can be used to be used in a method for selecting a preferred subset of correspondences 12 from a totality 10 of correspondences 12 provided, and in order to use this to estimate the camera's own movement with respect to its environment.

Quality measures and mapping on a priority

In order to be able to determine a preferred subset of correspondences 12, criteria or measures must be available, based on which z. B. a priority order can be determined.

If only a single quality measure is available, the definition is trivial, because the quality measure or an image of it, e.g. B. reciprocal, sign reversal, rounding, function, etc., can be used immediately as a priority.

If there are several quality measures, e.g. a number n, it makes sense to combine them in a suitable way and, if necessary, to map an n-dimensional space onto the one-dimensional scale of priorities.

As already mentioned above, the quality measures (1) and (2) can be appended individually to each flow vector as additional information, e.g. with 2 bits per quality measure.

A further quality measure could, for example, contain the information in which pyramid level of a resolution pyramid the respective correspondence was determined.

For example, in one embodiment of the present invention, four additional quality measures may be appended, each of 1 bit or 2 bits. A total of 6 quality measures, denoted for example by QM0 to QM5, would therefore be present in this exemplary embodiment, represented by 10 bits. It is advantageous to encode different quality aspects in these quality measures, which are as statistically independent as possible, in order to make the entire information content that can be represented therein as large as possible.

6 represents the binary layout for the coding of a flow vector or correspondence 12 and its quality measures. The total of 32 bits are divided into 22 bits for the actual flow vector or correspondence 12 itself and 10 bits for the quality measures QM0 to QM5.

It is advantageous to store all quality measures in consecutive bits. This facilitates direct access to subgroups or to the entirety of the quality measures, which are denoted here with "QCode".

The quality metrics can be combined, e.g. weighted added, multiplied, logically combined, etc., to produce a combined quality metric. The specification for the combination can depend on the application. For example, a first application could consider all quality measures, a second application only a part of them, a third application different weights.

However, constructing a suitable arithmetic link can be difficult, especially if the priority generated by the link is to use a value range similar to that of the entirety of the original quality measures, which here have 10 bits and thus a value range of 0-1023.

An alternative is therefore mapping with a lookup table (LUT). The summary of the quality measures in one word, referred to here as QCode, can be understood as an address in the LUT. Priority values that can be read are then stored at these LUT addresses.

A number of such LUTs can be provided in a corresponding ASIC: the LUT for priority sampling maps 10 bits to 10 bits. More LUTs map 10 bits to fewer bits.

Instead of using an ASIC, freely programmable structures are alternatively or additionally in combination, for example in a CPU or in a general digital signal processing device, conceivable.

• - sind die vorgesehenen Prioritätswerte der LUT alle unterschiedlich, mit dem Vorteil einer umkehrbar eindeutigen Abbildung, d.h. eine Abbildung ohne Informationsverlust, und
• - decken insgesamt denselben Wertebereich ab wie die Adressen, und zwar mit dem Vorteil, dass die binäre Wortlänge so gering wie möglich bleibt.

In a particularly advantageous embodiment

• - the intended priority values of the LUT are all different, with the advantage of a one-to-one mapping, ie a mapping without loss of information, and
• - cover the same range of values as the addresses, with the advantage that the binary word length remains as small as possible.

In the exemplary embodiment, a mapping is provided in which the 10-bit address is mapped to a 10-bit priority using the LUT.

This approach also opens up the possibility of performing more complex procedures offline to find an optimal LUT, e.g. using an automated training procedure based on large amounts of data.

The result of applying the LUT to the summary of quality measures, referred to as QCode, for a flow field or set of correspondences is in 3 shown: The darker, more heavily hatched or contrasted a respective pixel 11 is, the higher the priority of the flow vector ending at this point or of the correspondence 12 assigned to this pixel. If there is no flow vector or no correspondence 12, the priority is 0, corresponding to a white one pixels in the display.

that in 3 The central area of the image is darker than the areas on the left and right is due to the fact that the central area was processed with a higher spatial resolution, which is also reflected in the quality measures and was taken into account when determining the priority.

Equal sampling of high-priority correspondence

For many applications it is advantageous to select the subset of correspondences 12 distributed as evenly as possible over the image. This applies in particular to the estimation of the relative orientation.

To do this, it would initially be obvious to place a suitably scaled, uniform grid over the image, e.g. a rectangular grid or hexagonal grid, and to carry out the sampling at the grid points, or alternatively to generate the sampling points with a quasi-random generator that ensures good uniform distribution .

However, such an approach would not ensure that high-priority correspondence would be given priority.

A method is therefore sought that simultaneously ensures both, ie the best possible uniform distribution and the highest possible priority of the scanned correspondences.

The method according to the invention therefore provides for the image 10 to be broken down into non-overlapping partial regions 15, ie to form a tiling or a tiling, and for example to select the first best correspondence 12 from each tile 15. 7 shows a preferred embodiment with rectangular tiles 15 with line-by-line staggered joints as in masonry. The tiles 15 here are 8 pixels wide and 4 pixels high and are offset by half a tile width in tile rows.

There are also smaller tiles 17, e.g. at the edges of the image. The tile shapes and sizes can also be variable, e.g. smaller in the center of the image if more samples are to be drawn there than in the outer image areas.

The tile shapes can be polygonal (polygons), for example. Non-overlapping tiling is preferred because it ensures that a correspondence cannot be selected more than once. However, overlapping tiling, e.g. of rectangles of variable size, is not excluded, since it has other advantages: For example, determining the assignment, i.e. in which tiles a correspondence lies, is particularly easy with rectangular shapes.

One or more correspondences 12 can be stored at each pixel 11, each of which is provided with a priority.

The pixels 11 of a tile 15 are preferably processed in a predetermined order, for example in columns from top to bottom and then from left to right, as is shown in 8th is represented by the course of the arrows 19. The correspondence 13 with the highest priority within the tile is determined. If several correspondences 13 have the highest priority in the tile 15, a predetermined rule is followed, for example the first or last correspondence with the highest priority is selected.

This results in the desired behavior, that is to say good equal distribution and high priority of the scanned correspondences 12 are ensured. At the same time, the computational effort for this is very low.

consideration of other features

In addition to the quality measures attached to the correspondence 12, other features may also be considered.

These can be fed in as one or more masks, for example, where a mask can have the same size as the field 10 of correspondence 12 . Such a mask is in 4 shown. The content of the mask can be binary or have a larger range of values.

The content of the masks can in turn be combined with the priority described above, e.g. weighted, added, multiplied, logically linked, etc., or included in the calculation of the priority from the outset, if necessary using a LUT with a correspondingly larger value range.

It can be particularly advantageous to proceed in two steps, e.g. first carrying out the above-described uniform sampling and then, for example, taking into account one or more binary masks in order to decide whether the candidate selected during the sampling is accepted or rejected. This procedure reduces the effort, since the information from the masks only has to be taken into account for the smaller subset of the scanned correspondences 12. The advantage of the lower outlay is particularly evident in the exemplary embodiment shown, because the computationally intensive warping described in detail below remains limited to this smaller subset.

In the exemplary embodiment shown, the mask identifies potentially self-moving image regions that should better be excluded if a self-motion estimation is to be carried out on the basis of the correspondences, because the self-motion to be estimated here is intended to indicate the relative motion between the camera and the static world and from others independently moving objects remain unaffected.

It is therefore advantageous to recognize such independently moving objects in advance and to mark them in a mask, for example.

This detection can be accomplished by detecting non-epipolar conformal motion as shown 2 with connections 34 to 36. The estimated relative orientation is used, from which the epipole or the epipolar directions can be determined. It can thus be determined for each correspondence whether it is epipolar conform. If this is not the case, this can be marked accordingly using the mask, for example. The mask formed in this way can then preferably be used in a next time step or even immediately.

To combine the priorities according to 3 with the mask in accordance 4 the following solution was used in the exemplary embodiment: the selected correspondence is discarded if the underlying pixel is marked as potentially moving independently, otherwise it is kept. Other combinations are also conceivable.

• - Denkbar ist die Nutzung einer semantischen Segmentierung, die z.B. für jeden Bildpunkt angibt, um welche Objektklasse es sich handelt, ggf. mit Angabe von Wahrscheinlichkeiten. Diese Information kann hier sinnvoll genutzt werden, z.B. um daraus ein Maß für die „Immobilität“ daraus abzuleiten. Je größer die Immobilität, umso besser ist der Szenenpunkt zur Eigenbewegungsschätzung geeignet:
  • - Z.B. wäre die Immobilität besonders hoch bei den Objektklassen Straße, Bordstein, Leitplanke, Verkehrsschild, Baum, Mauer, Gebäude.
  • - Besonders niedrig wäre die Immobilität z.B. bei den Objektklassen Zweiradfahrer, Fußgänger, Glasoberfläche (wegen möglicher Spiegelungen), Schneeflocke.
• - Ferner kann die Maskierung von Bildteilen genutzt werden, die von der Selektion ausgeschlossen werden sollen: Beispielsweise ist es bei einer nach vorne ausgerichteten Fahrerassistenzkamera sinnvoll, den Bereich der eigenen Motorhaube auszuschließen, wenn dieser von der Kamera erfasst wird. Denn es wäre von erheblichem Nachteil, Korrespondenz aus dem Bereich der Motorhaube mit in die Eigenbewegungsschätzung einzubeziehen, da diese ja starr mit der Kamera verbunden ist und eine Nullbewegung, also maximale Immobilität zu erwarten ist, oder Spiegelungen von Szenenpunkten zeigt, z.B. Bewegungen, die die Eigenbewegungsschätzung irritieren könnten.
• - Denkbar ist zusätzlich oder alternativ auch die Nutzung einer Maskierung von Bildteilen, die auf Grund von Beschränkungen des optischen Pfades entstehen. Z.B. kann es bei einer Weitwinkeloptik in den Bildecken Bereiche geben, die von der Optik zum Beispiel wegen zu kleinem Bildkreis nicht oder nur oder unzureichend abgedeckt werden, zum Beispiel auch wegen Bildfehlern, starken Aberrationen oder dergleichen, oder in denen Störungen durch Spiegelungen und Streulicht zu erwarten sind und die daher besser ausmaskiert werden sollten.

Other examples of information that can be considered in the form of such masks are listed below:

• - It is conceivable to use a semantic segmentation which, for example, indicates for each pixel which object class it is, possibly with an indication of probabilities. This information can be used sensibly here, for example to derive a measure of “immobility” from it. The greater the immobility, the better suited the scene point is for estimating self-motion:
  • - For example, the immobility would be particularly high for the object classes street, curb, crash barrier, traffic sign, tree, wall, building.
  • - The immobility would be particularly low, for example, for the object classes cyclists, pedestrians, glass surface (because of possible reflections), snowflake.
• - Furthermore, the masking of image parts that are to be excluded from the selection can be used: For example, with a driver assistance camera facing forward, it makes sense to exclude the area of your own bonnet if this is captured by the camera. Because it would be of considerable disadvantage to include correspondence from the area of the bonnet in the self-motion estimation, since this is rigidly connected to the camera and a zero movement, i.e. maximum immobility, is to be expected, or shows reflections of scene points, e.g. movements that the Own motion estimation could irritate.
• - It is also conceivable, in addition or as an alternative, to use a masking of parts of the image due to limitations of the optical path arise. For example, in the case of wide-angle optics, there may be areas in the corners of the image that are not covered or only covered insufficiently by the optics, for example because the image circle is too small, for example also because of image errors, strong aberrations or the like, or in which interference from reflections and scattered light occurs are to be expected and which should therefore be better masked out.

Temporal warping of the other features by optical flow

In practice, and especially in a real-time system, one can be confronted with the problem that required information - such as the masks mentioned above - is not yet available for the current point in time, but only in an older version for an earlier point in time.

Waiting is sometimes not possible because otherwise the latency and thus the reaction time would be too great or because causality reasons speak against it: In the exemplary embodiment shown, the determination of the independently moving objects depends on the result of the self-motion estimation. So there is feedback. This is also in 2 represented by the arrow 34.

In such cases, it is advantageous to transfer the information from the old mask to the desired point in time. This process is also known as warping. This is of particular advantage because the correspondences are already available and the flow vectors indicate how scene points have moved further in the image. Accordingly, how the attributes associated with the scene points have progressed can be indicated.

• - t_M ist ein Zeitpunkt, für den die Information z.B. in Form einer Maske vorliegt (M : Maske),
• - t_P ist ein früherer Zeitpunkt für die Bestimmung der Korrespondenzen 12 (P : Previous),
• - tc ist ein aktueller Zeitpunkt für die Bestimmung der Korrespondenzen 12 (C : Current).

In the following, 3 points in time t _M < t _P < t _C are assumed:

• - t _M is a point in time for which the information is available, e.g. in the form of a mask (M : mask),
• - t _P is an earlier point in time for determining the correspondences 12 (P : Previous),
• - tc is a current time for determining the correspondences 12 (C: Current).

The correspondences 12 between the points in time t _P and t _C should already be available, for example as backward flow, ie in the more recent coordinates to tc, and sampled.

If tp is identical to _tM , i.e. the mask for time _tP is available, then warping is particularly simple: for each sampled coordinate (x, y) at time tc there is a flow vector (u, v) as correspondence 12, so that the correspondence 12 with the structure (x - u, y - v) can be determined from this. The mask can be read out accordingly at this image position.

While (x,y) are usually integer pixel coordinates, the correspondence 12 with the correspondence vector (x - u, y - v) is generally not (sub-pixel part of the motion), here must i.e. suitably rounded or interpolated. Interpolation can also be understood as a majority decision, for example based on the values of the mask in a small neighborhood. The case with t _M < t _P < tc, which is particularly important in practice, is illustrated 9 . Shown by arrows 91 are the reverse flow vectors ending at the integer pixel coordinates at tc and beginning at the non-integer pixel coordinates at _tp . Since the mask belongs to an earlier point in time t _M <t _P , this additional time interval should be suitably compensated for. This can be done at least approximately by lengthening the respective flow vector backwards as correspondence 12, specifically by a factor $$\text{s}=\left({\text{t}}_{\text{C}}-{\text{t}}_{\text{M}}\right)/\left({\text{t}}_{\text{C}}-{\text{t}}_{\text{P}}\right)$$

The mask should therefore be accessed accordingly at the point (x-s*u, y-s*v), in which case suitable rounding or interpolation may again be necessary here. The extension is in 9 each indicated by the arrow 92.

The approach is based on the assumption that the direction and speed of movement are approximately constant, at least for a short time (inertia assumption), which is usually sufficiently fulfilled in practice.

The exemplary embodiment described here relates to a (sampled) reverse flow between the times tp and _tc .

Other variants are also conceivable, for example the additional use of a reverse flow between times t _M and t _P , if present. This approach is more accurate and the assumption of inertia is not necessary, but the effort is higher because the information contained in the mask has to be transported in two steps: First, a position in the image t _P is determined with the newer flow vector between t _P and t _C . The older flow vector between t _P and t _M is read out there—possibly after appropriate rounding or interpolation. A position in the mask is thus determined, which is finally read out, in which case suitable rounding or interpolation may again be necessary here.

Variants with forward flow are also conceivable, as well as variants in which both forward flow and backward flow can be used.

bottom line and usage

The end result for the embodiment is in 5 to see. The sampling positions are shown here. These scanning positions are evenly distributed over the image 10. The offset of the rows of tiles and the aspect ratio of the tiles 15 result in an approximately hexagonal scanning pattern. Nevertheless, the priorities are always maximum in the respective tile. Potentially independently moving objects are excluded according to the mask propagated by means of warping, which can be seen from the punched-out pedestrians. The correspondences 12 selected in this way are therefore particularly well suited for an estimation of one's own motion. Optionally, the priority values can be passed on to the algorithm for self-motion estimation. This can take the priorities into account, e.g. B. into weights, with which the respective correspondences are included in the estimation.

2 shows a block diagram for a possible process. The first and second images B1 and B2 can come from different cameras or from the same camera. Dashed lines represent optional portions. The connections 34 to 36 stand for the described acquisition and consideration of information about self-moving objects in order to exclude their correspondences when estimating the relative orientation as far as possible.

3 shows that each flow vector or each correspondence 12 is assigned a priority. The priorities are represented by different dot densities, the higher the dot density, the higher the priority.

In 3 there are also pixels or areas for which no correspondence can be determined. These pixels or areas are then displayed in white, which corresponds to priority 0 or invalid.

4 makes it clear that, in addition to the priority, further features can be taken into account, eg a mask for a point in time t _M which characterizes potentially non-static regions, eg due to self-moving objects, which are shown in white in the center here.

5 shows a result of the priority sampling according to the invention, taking into account potentially non-static regions. Shown in black are the selected sample positions where the preferred correspondences are located.

6 shows an example of the bit layout of a flow vector and its quality characteristics. Of a total of 32 bits, 22 bits are provided for the flow vector or for the correspondence 12 itself and 10 bits for additional information, which contains individual quality features of 1 bit or 2 bits each (QM0 to QM5), which can also be viewed together ( QCode), e.g. B. as an address for a lookup table.

7 shows an example of the breakdown of an image 10 with pixels 11 and, if necessary, correspondence 12 into tiles 15. Here the tiles 15 are eight pixels wide and four pixels high. The rows of tiles are offset horizontally to one another, like in masonry. If there are 15 flow vectors in the tile, e.g. B. a preferred flow vector or a preferred correspondence 13 per tile 15 is selected, marked here by a circle. There are also tiles 15 in which there is no flow vector 11, namely in accordance with the shaded tiles 15, which are also labeled 16.

In 8th It is shown that in a tile 15 the pixels 11, each of which can be assigned one or more correspondences, are processed in a predetermined order, eg starting at the top left, preferably in columns, then from left to right.

In 9 it is shown that normally the mask - here MOM, Moving Objects Mask - is assigned to an earlier point in time t _M < t _P < t _C than the two images between which the optical flow is determined (points in time t _P and t _C ). However, this time offset between t _M and t _P can be compensated for, for example by backward extension 92 of the flow vector 91 by the factor s=(t _C -t _M )/(t _C -t _P ) and reading out the mask at the appropriate point.

The binary format of a flow vector 12 and its quality characteristics 22 can be defined as shown in 6 be constructed. The format can be openly visible and constant transfers of fields of correspondences can occur between an underlying ASIC and memory. To map the quality features into a priority using a 10-bit QCode, the user can program a LUT or readout table, for example, which is stored in registers via a driver. Binary masks can also be kept in a memory and read in at the right moment and are then possibly correspondingly visible. The result of the priority sampling can in turn be saved as a list and also be visible.

A corresponding ASIC can be developed for driver assistance systems, for example. Alternatively or additionally, the functions described according to the invention or parts thereof can be implemented in a freely programmable manner, for example in connection with a CPU or a general digital signal processor.

Claims (10)

Method (S) for evaluating correspondences (10, 12) of images (B1, B2) from at least one camera of a vehicle, the correspondences (10, 12) being based on the concept of optical flow or reflecting a stereo disparity, with: - providing (S1-1) the correspondences (10, 12) between given first and second images (B1, B2), - Providing (S1-2) a plurality of quality measures as attributes (20, 22) for characterizing a respective correspondence (10, 12), the respective quality measure as an attribute (20, 22) being representative of a correspondence (10, 12) under consideration for • a measure of the uniqueness with which a respective correspondence (10, 12) was determined in a local environment, or • a measure of the consistency of a respective correspondence (10, 12) with a chronologically preceding correspondence, - Evaluation (S2-1) and conditional selection (S2-2) of the correspondences (10, 12), whereby the evaluation (S2-1) of correspondences (10, 12) involves a combination of attributes (20, 22) and the selection (S2-2) of correspondences (10, 12) are based on a result of the evaluation (S2-1), and, in which • the correspondences (10, 12), in particular in combination with the associated attributes (20, 22), are represented as an image (10, 20) or matrix (10, 20), • the image (10, 20) or the matrix (10, 20) is covered by non-overlapping partial regions (15), wherein a tiling with rectangular tiles with joints offset in rows is formed, and • a correspondence (12) which is best in terms of the associated attributes (22) is selected for each partial region (15) in relation to correspondences (12) present in the partial region (15), and - Providing (S2-3) selected correspondence (10, 12) as an evaluation result (100). Procedure (S) according to claim 1 , in which a respective quality measure as an attribute (20, 22) of a correspondence (10, 12) under consideration is representative of at least one of the following aspects or a combination thereof: - for a measure of - at least local - spatial dependency and/or solid angle dependency of the distribution of the correspondences at least in the area of a respectively considered correspondence (10, 12) and in particular for a measure of the uniformity of a location-dependent and/or spatial angle-dependent distribution of the correspondences, - for a measure of a priority for preferential consideration of an underlying correspondence (10, 12) - for a measure of the existence of a correct assignment of a respective correspondence (10, 12) to areas of the underlying images (B1, B2), - for a measure of a proper movement of the respective correspondence (10, 12) in connection Areas of the underlying images (B1, B2), in particular of an object related thereto and/or taking into account non-epipolar conformal movements, - for a degree of movement possibility of the areas associated with the respective correspondence (10, 12) of the underlying images (B1, B2), in particular of an object associated therewith, - for a degree of accuracy with which a respective correspondence (10, 12) was determined, - for a degree of accuracy when determining the respective correspondence ( 10, 12) underlying resolution or resolution level, - for a measure of an affiliation of the areas of the underlying images (B1, B2) associated with the respective correspondence (10, 12), in particular an object associated therewith, to a semantic class, preferably on the basis of a video-based object classification, a process of machine learning and/or a process of semantic segmentation, - for a degree of affiliation of the areas associated with the respective correspondence (10, 12) of the underlying Images (B1, B2), in particular of a related object, to a class for considering permitted areas of the underlying images (B1, B2), in particular in connection with masking, and/or - for a measure of the consistency of a respective correspondence (10, 12) with a chronologically preceding correspondence with regard to a degree of consistency of flow vectors over time, with confirmed consistency being based on a temporally stable behavior, in particular in the form of an age as a numerical value. Method (S) according to one of the preceding claims, in which combined quality measures are used as attributes (20, 22) which are derived from one or a plurality of quality measures by combination formation and/or Function formation, in particular by reciprocal value formation, sign reversal, rounding, formation of function values from the scalar or vectorial values on which the quality measures are based. Method (S) according to one of the preceding claims, in which a respective attribute (20, 22) and/or the underlying quality measures and in particular their values are used in encoded form for use as an address, in particular in connection with a value of the attributes (20 , 22) and/or the selection table representing quality measures. Method (S) according to any one of the preceding claims, in which - a respective correspondence (10, 12) is represented as a one-dimensional list or as a vector and - An attribute (20, 22) assigned to the correspondence (10, 12) and/or its coding is represented as one or more additional list elements or vector components or as parts thereof and is or are attached to the list or the vector. Method (S) according to one of the preceding claims, in which attributes (20, 22) and/or quality measures for correspondence (10, 12) at an earlier point in time are transferred and/or extrapolated to a correspondence (10, 12) at a later point in time . Operating assistance method and in particular driving assistance method for a device and in particular for a vehicle, in which - images (B1, B2) recorded and with a method according to one of Claims 1 until 6 are evaluated and - a result of the evaluation is used in controlling the operation of the device. Device for evaluating correspondences (10, 12) of images (B1, B2), - which is set up, a method according to one of Claims 1 until 6 to be carried out and - which is designed in particular as an ASIC, as a freely programmable digital signal processing device or as a combination thereof. Computer program that is set up, a method (S) according to one of Claims 1 until 7 to be performed when running on a computer or digital signal processing device. Machine-readable storage medium on which the computer program claim 9 is saved.

Images