DE102018113325A1 - A method and computerized location system for determining the location of at least one structure by means of an imaging sensor - Google Patents

Abstract

The invention relates to a method for determining the location of at least one structure (24) by means of an imaging sensor (18), in particular an imaging sensor (18) mounted on a vehicle (10), wherein the location of the individual structure (26) consists of images ( 24) generated by the imaging sensor (18) from at least two different sensor positions (P1, P2), the sensor positions (P1, P2) having a distance (D) greater than or equal to a minimum distance (D) for determining the location of the structure (24). The imaging sensor (18) generates a series of successive images (26) during movement, wherein the distance (D) of the sensor positions (P1, P2) is detected by recognizing the structure (26) in the images (24) and determining the location of the structure (26) relative to the sensor position (P1, P2) and to an orientation of the imaging sensor (18) for each of the images (24). The invention further relates to a corresponding computer program product and a corresponding computerized location determining system (14).

Description

The invention relates to a method for determining the location of at least one structure by means of an imaging sensor, in particular a vehicle-mounted imaging sensor, wherein the location of the individual structure is determined from images generated by the imaging sensor from at least two different sensor positions, wherein the sensor positions are at a distance greater than or equal to a minimum distance to determine the location of the structure.

The invention further relates to a corresponding computer program product and a corresponding computer-aided location system.

The determination of the location of "Structure from Motion" and / or a 3D reconstruction using an imaging sensor, such as a camera, requires the movement of the imaging sensor before an estimate of a structure in the environment is possible. Therefore, in an application that requires structure from motion or 3D reconstruction, the application must allow sufficient movement of the imaging sensor before starting 3D reconstruction. Typically, this is done by forcing a minimum baseline (minimum distance Dth) between the two sensor positions, i.e., before triggering the 3D reconstruction. by saying that the imaging sensor must perform a minimum translational motion prior to performing the reconstruction. This approach, associated with the term "degeneracy", appears in the textbook: Hartley, R. and Zisserman, A .: "Multiple View Geometry in Computer Vision, Second Edition"; Cambridge University Press, March 25, 2004, discussed in more detail.

The invention has for its object to provide a correspondingly improved method for determining the location of at least one structure by means of a moving imaging sensor, a correspondingly improved computer program product and a correspondingly improved computer-aided positioning system.

The solution of this object is achieved by a method, a computer program product and a corresponding computer-aided location system with the features of the respective independent claims. Advantageous embodiments of the invention are subject of the dependent claims, the description and the figures.

According to the method according to the invention for determining the location of at least one structure by means of an imaging sensor, wherein the location is determined from images generated by the imaging sensor from at least two different sensor positions at a distance greater than or equal to a minimum distance for determining the Once the structure is in place, the moving imaging sensor generates a series of successive images, verifying the distance of the sensor positions by detecting the structure in the images and determining the location of the structure relative to the position and orientation of the imaging sensor for each of the images becomes. The orientation of the imaging sensor is e.g. given by a viewing direction of the sensor. In a preferred application, the imaging sensor is mounted in / on a vehicle and are the structures in an environment of the vehicle. The method according to the invention has the advantage that a single moving imaging sensor is sufficient to determine the location of the structure (s), wherein the provision of a sufficient distance of the sensor positions where the images were generated from the images themselves determined / verified can be.

The imaging sensor is preferably a camera, an imaging ultrasound sensor, an imaging radar sensor or an imaging lidar sensor. All of these types of imaging sensors are known from various automotive applications.

in Beziehung stehen, größer ist als ein Schwellenwert θ_th. Der Referenzvektor $\overrightarrow{n},$

der die jeweilige Orientierung des bildgebenden Sensors anzeigt, ist beispielsweise durch eine Blickrichtung des Sensors gegeben.According to a preferred embodiment of the invention, a reference vector indicates the respective orientation of the imaging sensor, a direction vector indicates the location of the structure relative to the position of the imaging sensor and the presence of a sufficient distance between the sensor positions results from the fact that an angular difference of respective direction vectors s, s' associated with the reference vector $\overrightarrow{n}$

is greater than a threshold θ _th . The reference vector $\overrightarrow{n}.$

which indicates the respective orientation of the imaging sensor, is given for example by a viewing direction of the sensor.

In many cases, the orientation of the imaging sensor between the two sensor positions does not change significantly. In these cases, the distance of the sensor positions can only be determined using the direction vectors.

According to a further preferred embodiment of the invention, the images are frames (frames) of a video sequence which are recorded by the moving imaging sensor, in particular a (video) camera. Preferably, a pair of frames, i. two consecutive frames used to determine the spacing of the sensor positions.

According to another preferred embodiment of the invention, the angular difference θ is determined by calculating the scalar product of the direction vectors s, s'. Each of the directional vectors s, s' is preferably a unit vector of length 1. This yields the following equation: $$\text{s}\cdot \text{s'}<\text{cos}\left({\mathrm{\theta }}_{\text{th}}\right)$$

Preferably, the location is determined by triangulation. Triangulation is a process of determining the position of a point such that only angles to it are measured from known points at each end of a fixed baseline, rather than measuring the distances to the point directly, as in a trilateration. The point can then be defined as the third point of a triangle with a known side and two known angles.

According to yet another preferred embodiment of the invention, the location determination is part of a 3D reconstruction of multiple structures. In other words, the method is a method for 3D reconstruction. 3D reconstruction is a process for capturing the shape and appearance of real objects. This process can be implemented by either active or passive methods. If the shape of the model is allowed to change over time, this is called non-rigid or spatial-temporal reconstruction.

The computer program product according to the invention comprises computer executable program code portions having program code instructions configured to execute the above method when e.g. is loaded into a processor of a computerized location system.

In accordance with the computerized location system for determining the location of at least one structure by means of an imaging sensor to produce a series of successive images, the system is configured to determine the location of the individual structure from images generated by the imaging sensor from at least two different sensor locations having a distance greater than or equal to a minimum distance to determine the location of the structure, the system being arranged to adjust the distance of the sensor positions by detecting the structure in the images and determining the location of the structure relative to the position and to verify an orientation of the imaging sensor for each of the images.

According to a preferred embodiment of the computer-assisted positioning system according to the invention, the location determining system is adapted to carry out the above method by means of at least one imaging sensor.

In general, the computerized location system includes a computer unit having a processor and memory. Preferably, the location system further comprises the imaging sensor.

Further features of the invention will become apparent from the claims, the figure and the description of the figures. All of the features and feature combinations mentioned above in the description as well as the features and feature combinations mentioned below in the description of the figures and / or shown only in the figure are usable not only in the respectively indicated combination but also in other combinations or independently.

The invention is explained below with reference to a preferred embodiment and with reference to the accompanying drawings.

• 1 eine Draufsicht eines Fahrzeugs auf einer Straße, das Strukturen eines Objekts unter Verwendung eines bildgebenden Sensors erfasst.

It shows

• 1 a plan view of a vehicle on a road that detects structures of an object using an imaging sensor.

1 shows a plan view of a scene with a vehicle 10 that on a street 12 moves. This vehicle 10 is a motor vehicle, to be more specific: a passenger car. The vehicle 10 has a computerized location system 14 with a computer unit 16 and one (or more) imaging sensor (s) 18 on. The imaging sensor used in the example 18 is a camera 20 located in the front of the vehicle 10 located. Other possible imaging sensors 18 For this purpose ultrasound, radar, lidar etc. are based on the detection range of the imaging sensor 18 is an appropriate environment area 22 in front of the vehicle 10 ,

The vehicle 12 drives on the street 12 at the speed v, and the mounted imaging sensor 18 creates a series of consecutive images during the movement 24 , Two pictures 24 are on the left side of 1 shown. The lower first picture 24 is an image of the imaging sensor 18 that at the position P1 is generated, and the upper second image 24 is an image of the imaging sensor 18 that at the position P2 is produced. The two positions P1 . P2 have a distance D. At the roadside of the road 12 in the detection range of the imaging sensor 18 (in the surrounding area 22 ) there is a visible structure 26 , eg of an object, such as a road sign, a parked car, etc.

gibt die jeweilige Orientierung des bildgebenden Sensors 18 an. Ein Richtungsvektor s, s' gibt den Ort der Struktur 26 relativ zur Sensorposition P1, P2 an, und das Vorhandensein eines ausreichend großen Abstands D zwischen den Sensorpositionen P1 an P2 ergibt sich aus der Tatsache, dass eine Winkeldifferenz θ der jeweiligen Richtungsvektoren s, s' bezogen auf den Referenzvektor $\overrightarrow{n}$

größer ist als ein Schwellenwert θ_th (in 1 nicht dargestellt). Die Orientierung des bildgebenden Sensors 18 ändert sich zwischen den beiden Sensorpositionen P1, P2 nicht wesentlich, so dass nur ein Referenzvektor verwendet wird. In diesen Fällen kann der Abstand D der Sensorpositionen P1, P2 unter Verwendung nur der Richtungsvektoren s, s' bestimmt werden.Thus, this structure is 26 (next to the street 12 ) in both pictures 24 visible. The distance D of the sensor positions P1 . P2 is by recognizing the structure 26 in the pictures 24 verified, and the place of the structure 26 relative to the position P1 . P2 with respect to an orientation of the imaging sensor 18 for each of the pictures 24 is determined. A reference vector $\overrightarrow{n}$

gives the respective orientation of the imaging sensor 18 on. A direction vector s, s' gives the location of the structure 26 relative to the sensor position P1 . P2 and the presence of a sufficiently large distance D between the sensor positions P1 at P2 results from the fact that an angular difference θ of the respective direction vectors s, s' with respect to the reference vector $\overrightarrow{n}$

is greater than a threshold value θ _th (in 1 not shown). The orientation of the imaging sensor 18 changes between the two sensor positions P1 . P2 not essential, so only one reference vector is used. In these cases, the distance D of the sensor positions P1 . P2 be determined using only the direction vectors s, s'.

By examining each pair of vectors s, s' corresponding to a coincident point in adjacent video images 24 and observing whether the angle θ therebetween is "large enough", there is no need for a general camera baseline. This has the advantage of having structures 26 closer objects are detected earlier and the accuracy of capturing the structures 26 of distant objects (objects that are farther away) is larger.

In addition, one can say that a particular set of features is "unsharp", indicating that an object has the structure 26 exists, but with low accuracy. This is very useful for a fusion system (eg ultrasound) where you want the vehicle 10 responds to more than one source (to avoid false responses), but also desires to be able to react quickly before accurate observations are made.

Features are mapped between two processed video images, for example, using optical flow or feature matching. Each feature may be calibrated using a beam from the imaging sensor 18 (eg a camera 20 ) being transformed. For each feature, to reconstruct the position of this feature in 3D space, the rays must be triangulated. That is, the rays must have some convergence. However, this is not the only way to reconstruct a feature, since performing a direct ray convergence in R ³ can be prone to error. Other methods involve the use of a re-projection error to estimate the depth.

wobei s der Vektor ist, der einem Strahl entspricht, der von dem aktuellen Einzelbild projiziert wird, s' das Äquivalent eines vorhergehenden Einzelbildes ist und θ_th der Winkelschwellenwert ist (und · sich auf das Skalarprodukt von zwei Vektoren in R³ bezieht). Es wird darauf hingewiesen, dass s und s' im gleichen Koordinatensystem liegen müssen, daher muss die Odometrie (die als Rotationsmatrix R und Translationsvektor t dargestellt werden kann) der Kamera 20 bei der Formulierung von s und s' berücksichtigt werden. Es wird auch darauf hingewiesen, dass der Kosinus eines kleinen Winkels nahe bei Eins liegt und der Kosinus eines großen Winkels nahe bei null liegt. Daher die Umkehrung des Ungleichheitszeichens.To ensure this convergence, existing solutions require the restriction that the camera 20 must move over a certain minimum baseline distance before processing to estimate the 3D point is performed. However, we propose to ensure convergence by examining the projected beams and to ensure that the angle formed by the two beams is greater than a threshold. That is, for a feature to be added to the internal buffer for processing, the following must be true: $$\text{s}\cdot \text{s'}<\text{cos}\left({\mathrm{\theta }}_{\text{th}}\right).$$

where s is the vector corresponding to a ray projected from the current frame, s' is the equivalent of a previous frame, and θ _{th is} the angle threshold (and · refers to the dot product of two vectors in R ³ ). It should be noted that s and s' must be in the same coordinate system, so the odometry (which can be represented as rotation matrix R and translation vector t) of the camera must be 20 be taken into account in the formulation of s and s'. It should also be noted that the cosine of a small angle is close to unity and the cosine of a large angle is close to zero. Hence the reversal of the inequality sign.

Sometimes it is useful to track a feature across more than one frame, for example, when using three or more samples to create the 3D point in the reconstruction. In this case, the internal buffer for the features is added only if all new features meet the criteria mentioned above.

It should be noted that the reason for using a minimum convergence angle threshold is that rays converging at 0 or at a very small angle have a high numerical error and are very susceptible to noise in the signal, so the estimated 3D Point for this Strahlentupel will be afflicted with a high error.

For some applications, however, a higher error is acceptable, which is why we introduce the concept of the "fuzzy" feature. A "blurred" feature is a feature that does not necessarily meet the criteria mentioned above. For example, the following criteria can be applied to a beam that is to be considered blurred: $$\text{cos}\left({\mathrm{\theta }}_{\text{unsharp}}\right)>\text{s}\cdot \text{s'}>\text{cos}\left({\mathrm{\theta }}_{\text{th}}\right).$$

That is, the angle between the beams must be greater than a threshold to be out of focus (again, it should be noted that the cosine values result in an inverted inequality sign). Below this threshold the features are completely ignored, they are not considered at all. Above this threshold, but below the minimum angle for a "normal" feature, the features are considered out of focus and can be used at locations where the application permits (i.e., where an accurate 3D position is not strictly required). As usual, features having an angle greater than the threshold discussed above are considered "normal" features for reconstruction.

A similar approach can be used in bundle balancing. Bundle Adjustment (BA) is a numerical method for refining the position of the camera 20 and the location of the points in R ³ being reconstructed, typically using a re-projection error as an error function. As with many numerical methods, individual samples may be weighted according to certain criteria. Given a point in R ³ that is reconstructed with rays that are parallel or nearly parallel, we can add a weighting to the beam equalization so that the influence of this feature on the overall beam balance result is minimal. The reason for this is that if the rays for the feature are nearly parallel (or exactly parallel), the reprojection error will always be small, even if sometimes there is a very large error in R ³ .

As already mentioned, beam balancing typically uses the reprojection error as an error function. The reprojection error is a common mechanism by which the estimated point in R ^{3 is} re-projected into the image space and the distance to the image sampling point is measured as an error. In addition, the beam-compensation error function can be designed such that such parallel beams always reflect a high error, and in this way features designed to have a high error can be assigned a lower reliability.

LIST OF REFERENCE NUMBERS

vehicle 10 Street 12 Ortbestimmungssystem 14 computer unit 16 Imaging sensor 18 camera 20 detection range 22 picture 24 structure 26 reference vector

$\overrightarrow{n}$

direction vector s direction vector s' speed v distance D first position P1 second position P2

Claims (10)

Method for determining the location of at least one structure (26) by means of an imaging sensor (18), in particular an imaging sensor (18) mounted on a vehicle (10), the location of the individual structure (26) being determined by the imaging sensor (18 ) are determined from at least two different sensor positions (P1, P2), wherein the sensor positions (P1, P2) have a distance (D) which is greater than or equal to a minimum distance ( _Dth ) for determining the location of Structure (26), characterized in that the imaging sensor (18) generates a series of successive images (24) during the movement, wherein the distance (D) of the sensor positions (P1, P2) by detecting the structure (26) in the Images (24) and determining the location of the structure (26) relative to the sensor position (P1, P2) and to an orientation of the imaging sensor (18) for each of the images (24). Method according to Claim 1 , characterized in that a reference vector $\left(\overrightarrow{n}\right)$

indicates the respective orientation of the imaging sensor (18), a direction vector (s, s') indicates the location of the structure (26) relative to the sensor position (P1, P2), and the presence of a sufficiently large distance (D) between the sensor positions ( P1, P2) results from the fact that an angular difference (θ) of the respective direction vectors (s, s') with respect to the reference vector $\left(\overrightarrow{n}\right)$

is greater than a threshold (θ _th ). Method according to Claim 1 or 2 , characterized in that the images (24) are frames of a video sequence taken by the moving image sensor (18). Method according to one of Claims 1 to 3 , characterized in that the angular difference (θ) is determined by calculating the scalar product of the direction vectors (s, s'). Method according to one of Claims 1 to 4 , characterized in that the location of the structure (26) is determined by triangulation. Method according to Claim 4 or 5 , characterized in that the location determination is part of a 3D reconstruction of a plurality of structures (26). A computer program product having computer executable program code portions having program code instructions configured to perform the method of any one of Claims 1 to 6 perform. A computerized location determining system (14) for determining the location of at least one structure (24) by means of an imaging sensor (18) for generating a series of successive images (24), in particular an imaging sensor (18) mounted on a vehicle (10) (14) is adapted to determine the location of the individual structure (26) from images (24) generated by the imaging sensor (18) from at least two different sensor positions (P1, P2), the sensor positions (P1, P2) have a distance (D) greater than or equal to a minimum distance (D _th ) for determining the location of the structure (26), the system (14) being arranged to adjust the distance of the sensor positions (P1, P2) by verifying the structure (24) in the images (24) and determining the location of the structure (26) relative to the sensor position (P1, P2) and to an orientation of the imaging sensor (18) for each of the images (24). System after Claim 8 , characterized in that the system (14) is adapted to perform the method according to any one of Claims 1 to 6 perform. System after Claim 8 or 9 characterized by a computer unit (16) having a processor and a memory.

Images