DE102010013093A1 - Method for creating model of surrounding area of motor vehicle i.e. car, involves determining whether card cells are loaded with object represented by three- dimensional structures - Google Patents

Abstract

The method involves detecting surrounding area of a car (10) in travel direction of the car by utilizing a camera i.e. mono camera, and producing multiple images of the surrounding area. Three-dimensional structures and position of the car in the surrounding area are detected through the images. A movement of the structures within an image sequence is tracked. Information is transmitted via the structures into a two-dimensional cell-based card. The information allocated to cells of the card is determined whether the cells are loaded with an object (1) represented by the structures. An independent claim is also included for a system for creating a model of a surrounding area of a car.

Description

The present invention relates to a method and a system for vehicle environment reconstruction by means of a camera as well as a correspondingly configured vehicle.

According to the prior art, a method is known for detecting traffic signs from a moving vehicle by means of a camera; see DE 10 2007 021 576 A1 or EP 2 048 618 A1 , In addition, it is from the DE 10 2006 039 104 A1 is known to make a distance measurement of objects based on image data of a camera system. The DE 10 2006 055 908 A1 discloses a method for automatic high beam control of automotive headlights based on a camera sensor with which the vehicle environment is monitored.

The object of the present invention is to create a model of an environment of a vehicle, which describes this environment in more detail than is possible according to the prior art.

According to the invention, this object is achieved by a method for creating a model of an environment of a vehicle according to claim 1, by a system for creating a model of an environment of a vehicle according to claim 11 and by a vehicle according to claim 13. The dependent claims define preferred and advantageous embodiments of the present invention.

In the context of the present invention, a method for creating a model of an environment of a vehicle is provided. In this case, with the help of a camera, in particular a monocamera, the environment in the direction of travel of the vehicle is detected and several images of this environment generated by means of the camera. These images are used to determine three-dimensional structures and their position within the environment, for example by using image processing algorithms to track the movement of these structures or of the pixels forming these structures within an image sequence which consists of the multiple images. In this case, information about these three-dimensional structures or the 3D points describing these structures is entered in a two-dimensional occupancy map or environment map subdivided into cells, this occupancy map representing a two-dimensional discretization of the vehicle surroundings. In this case, each cell of the occupancy card is assigned an information as to whether the corresponding cell is occupied by an object represented by the structures. This can be done, for example, via probability values which are assigned to these cells and which indicate a value for a probability with which the corresponding cell is occupied by an object.

In other words, in a first step of the method according to the invention, for example by means of an approach such as "structure from motion" certain structures and thereby determined objects including their position in the environment of the vehicle detected, while in a second step information about these structures or objects in one Map are transferred, in which for the plane on which the vehicle moves, free and occupied areas are available. In this case, an occupied surface or cell is understood to mean an area or cell occupied by an object, while a free area or cell is not occupied by any object.

The structures and objects determined for example by means of "structure from motion" can also be stored in a three-dimensional map or a three-dimensional data record, from which the corresponding information is then extracted and transmitted to the two-dimensional map. The storage in the three-dimensional map has the advantage that here also a height information is stored.

With the method according to the invention, advantageously a collision of the vehicle with objects in the environment of the vehicle can be avoided.

To carry out the method according to the invention, the camera is thereby aligned, in particular in the direction of travel of the vehicle.

By being oriented in the direction of travel of the vehicle, the camera advantageously detects precisely the area of the surroundings of the vehicle which the vehicle will soon pass through, which is therefore most important for collision avoidance.

According to a preferred embodiment of the invention, the method according to the invention is supplemented by a surface segmentation. In this case, image areas within the images with a similar color or structure are detected with the aid of image processing means and thus the corresponding image is subdivided or segmented into clusters or area segments with a similar color or structure. Subsequently, the free cells (ie, cells in which the probability of occupancy with an object is below a predetermined threshold) are assigned to the respective area segments of the map. Surfaces that include a predetermined percentage of free cells become a free area segment considered and thus entered as a contiguous free (ie occupied by any object) area back into the occupancy map.

The use of area segmentation in the context of the present invention has the advantage that individual false 3D measurements are reliably filtered out.

In addition, it has proved to be advantageous if, from the plurality of images captured by the camera, those images are selected in which the camera has moved by more than a predetermined distance between two successive of these selected images when the corresponding image was taken. Using triangulation, a three-dimensional localization of the structures within the environment is then carried out for each of these selected images.

Since it is ensured that the camera moves between the point at which the camera has acquired a specific one of these selected or evaluated images, and the point at which the camera has taken the image to be evaluated following the specific image, at least by the predetermined distance has the distance of the camera positions with respect to two temporally successive images to be evaluated sufficiently large so that by means of triangulation correspondingly accurate measurement results can be achieved.

In particular, a movement of the camera is detected as accurately as possible in order to improve the results of the method according to the invention. For this purpose, the movement of the camera on the one hand based on vehicle data, eg. B. on the basis of data from a Wegimpulsgebers, a yaw rate sensor or steering wheel sensor of the vehicle detected. On the other hand, the movement of the camera can be reconstructed on the basis of the optical flow within the image sequence taking into account corresponding restrictions which are given by the epipolar geometry with regard to the movement of the structures within the image sequence. From the thus determined movement of the camera as accurate as possible respective position of the camera is determined for each image. The triangulation is performed depending on the thus determined positions of the camera (exact position of the camera relative to the position of the camera in the previous image) for locating the structures within the images.

By such a determination of the movement of the camera, corresponding pitching and rolling movements of the vehicle (and thus of the camera) can be taken into account accordingly, so that the results of the method according to the invention are advantageously not distorted by these pitching and rolling movements.

The optical flow is determined in particular via a so-called dense process in which a velocity vector is determined for each pixel of a specific image, which indicates in which direction and at what speed the respective pixel from the particular image to a corresponding pixel in a temporally following picture moves. In other words, an optical flow in image processing is understood to mean a vector field which indicates the direction of movement and the speed for each pixel of an image sequence. The optical flow can therefore be understood as the velocity vectors of visible objects projected onto the image plane.

The use of a dense method for determining the optical flow increases the data density and thus the robustness of the method.

In addition, it is possible that one or more additional cameras are used to detect the surroundings of the vehicle. These one or more cameras are advantageously attached to the vehicle at a location other than the aforementioned camera and, in a manner similar to the aforementioned camera, produce a plurality of images of the environment. By means of these images, for example, the movement of the aforementioned camera can be estimated, so that the determination of the movement of the camera by means of the one or more further cameras can be further improved.

The present invention may also employ a so-called top-view camera configuration in which one camera is directed forward, backward, rightward, and rearward (with respect to the vehicle). This allows the environment around the vehicle to be captured.

By using such a multi-camera configuration (ie, a configuration that includes one or more additional cameras in addition to the aforementioned camera), the movement of the aforementioned camera can be estimated more robustly and accurately, thereby improving the quality of creating the surrounding model can be.

A further extension according to the invention is to detect information about objects in the surroundings of the vehicle by means of ultrasonic wave sensors and to improve the information content of the occupancy map on the basis of this information.

By virtue of this coupling of the method according to the invention with information acquired by ultrasonic sensors, the robustness of the method according to the invention can advantageously be increased.

The method according to the invention can be carried out, for example, automatically in a collision monitoring system of the vehicle. In this context, a collision monitoring system is understood to mean on the one hand a parking assistance (eg in partially or fully automatic parking processes, eg in a garage) and on the other extensions of a lane assist (heading control, lane assist), for example in the construction site area, by detecting the guardrail , In addition, a replacement of the PDC sensors (Park Distance Control) by cameras by means of the present invention conceivable, although according to the invention, multiple cameras should be used to ensure the most accurate detection of the environment even when the vehicle is stationary.

In the context of the present invention, a system is also provided for creating a model of an environment of a vehicle. The system includes a camera for detecting the environment of the vehicle in the direction of travel and a controller. The controller controls the camera in such a way that the camera generates several images of the surroundings of the vehicle. The controller is designed in such a way that the controller, on the basis of these images, detects three-dimensional structures and their position in the environment by the controller tracking a movement of these structures within an image sequence consisting of the multiple images. The controller transfers information about the structures into a two-dimensional cell-based map of the environment, the controller assigning information to each cell of the map as to whether that cell is occupied by an object represented by the structures or not.

The advantages of the system according to the invention essentially correspond to the advantages of the method according to the invention, which have been carried out in detail in advance, so that a repetition is dispensed with here.

Finally, in the context of the present invention, a vehicle is provided with a system according to the invention.

The present invention is particularly suitable for collision avoidance for a vehicle. Of course, the present invention is not limited to this preferred application, since the present invention could be applied to ships, airplanes, or track-bound vehicles.

In the following, the present invention will be explained in detail by means of embodiments according to the invention with reference to the figures.

With 1 it shows how the method "Structure from Motion" works.

In 2 an example of a movement detected by the optical flow is shown.

With 3 It shows which images of a sequence are selected for a three-dimensional reconstruction.

In 4 an example of an occupancy map compared to the real environment is shown.

In 5 the method according to the invention is shown schematically.

6 schematically represents a vehicle according to the invention with a system according to the invention.

As in 1 is shown, visual rays, which come from two different camera positions or from two different positions of a vehicle meet 10 the same pixel of an object 1 aim, at a certain angle. With knowledge of the exact camera position in both shots, the three-dimensional position of the targeted pixel and thus of the object can be determined 1 calculate what is also called triangulation.

• 1. Während einer Fahrt des Fahrzeugs 10 werden aufeinanderfolgende Bilder 5 mit einer in Fahrtrichtung des Fahrzeugs 10 ausgerichteten Monokamera (Frontkamera) 4 aufgenommen.
• 2. Mit Bildverarbeitungsalgorithmen wird eine Bewegung von einzelnen Pixeln oder Bildbereichen innerhalb einer aus diesen Bildern 5 bestehenden Bildsequenz verfolgt. Diese Verfolgung der Bewegung wird über den optischen Fluss realisiert (siehe 2, in welcher der optische Fluss für bestimmte Bildpunkte dargestellt ist). Dabei werden so genannte Tracks (Trajektorien auf der Bildebene) von den einzelnen Pixeln (auch als Feature benannt) gebildet.
• 3. Die Bewegung des Fahrzeugs 10 wird mit Hilfe von üblicherweise im Fahrzeug 10 vorhandenen Sensoren (z. B. Wegimpulsgeber, Gierratensensor, Lenkradsensor) erfasst, um daraus die Bewegung der Kamera 4 und schließlich die Position der Kamera 4 bei der Aufnahme eines bestimmten Bildes zu erfassen.
• 4. Für jedes Bild A werden bestimmte Bildpunktpaare aus den Feature-Tracks extrahiert, wobei der eine Bildpunkt dem Feature in dem Bild A und der andere Bildpunkt des Bildpunktpaars dem entsprechenden Bildpunkt des Features in dem Bild B, welches vor dem Bild A aufgenommen worden ist, entspricht. Dabei werden die Bilder A und B derart aus den mit der Kamera 4 erfassten Bildern ausgewählt, dass die Fahrzeugbewegung und damit die Kamerabewegung zwischen den Bildern A und B signifikant (z. B. > 10 cm) ist. In 3 sind die von der Monokamera 4 über der Zeit aufgenommenen Bilder 5 dargestellt. Der optische Fluss (OF) wird jeweils zwischen zwei zeitlich benachbarten Bildern 5 berechnet und in Tracks verkettet. Für die dreidimensionale Rekonstruktion (”Structure from Motion” (SfM)) werden zwei zeitlich aufeinanderfolgende Bilder derart ausgewählt, dass die entsprechende Bewegung der Kamera 4 zwischen diesen beiden Bildern signifikant (z. B. > 10 cm) ist.
• 5. Die im Schritt 3 bestimmte Bewegung des Fahrzeugs 10 bzw. der Kamera 4 zwischen den Bildern A und B wird mit Hilfe von Kalibrierdaten (Einbauort der Kamera und Orientierung der Kamera) in die Bewegung der Kamera 4 umgerechnet, um so die Position der Kamera 4 bei der Aufnahme des entsprechenden Bildes 5 zu bestimmen.
• 6. Die Rekonstruktion der Kamerabewegung kann durch die Schätzung der so genannten epipolaren Geometrie (relative Lage der Kamera zum Zeitpunkt der Aufnahme bestimmter Bilder) unter Berücksichtigung des optischen Flusses verbessert werden. Dadurch kann auch eine Nick- und Wankbewegung des Fahrzeugs 10 und damit der Kamera 4 besser berücksichtigt werden.
• 7. Die möglichst genaue Position der Kamera bei der Aufnahme des Bildes A relativ zu der möglichst genauen Position der Kamera bei der Aufnahme des Bildes B wird genutzt, um für jedes Bildpunktpaar A-B den Schnittpunkt der Strahlen aus Bild A und aus Bild B zu ermitteln. Dieses Verfahren wird in der Bildverarbeitung Triangulation genannt, wobei der Schnittpunkt der Strahlen die Position des jeweiligen Bildpunkts in dem dreidimensionalen Raum bestimmt.
• 8. Strukturen bzw. Objekte, welche durch die derart erfassten Bildpunkte genau im dreidimensionalen Raum lokalisiert werden können, werden in eine zellenbasierte Belegungskarte 2 oder Umfeldkarte eingetragen, so dass diese Belegungskarte 2 eine zweidimensionale Diskretisierung des Fahrzeugumfelds darstellt. Dabei wird jeder Zelle dieser Belegungskarte 2 ein Wert zugewiesen, welcher die Wahrscheinlichkeit einer Belegung der entsprechenden Zelle mit einem Objekt beschreibt. Mit anderen Worten besitzt jede Zelle einen Wahrscheinlichkeitswert, welcher der Wahrscheinlichkeit entspricht, dass die entsprechende Zelle mit einem Kollisionsobjekt belegt ist. Aus dieser Wahrscheinlichkeit lassen sich die Zellen der Belegungskarte 2 in freie (von keinem Objekt belegte) und belegte Zellen unterteilen.
• 9. Die Wahrscheinlichkeitswerte der Zellen werden ständig aktualisiert, wodurch eine Filterung der Daten oder Wahrscheinlichkeitswerte über der Zeit möglich ist.
• 10. Auf der Grundlage der Informationen aus der Belegungskarte 2 kann der Freiraum 3 vor und um das Fahrzeug 10 herum bestimmt werden. In 4b ist ein Bild der Kamera 4 dargestellt. Die 4a stellt die zu diesem Bild zugehörige Belegungskarte des Fahrzeugumfelds dar, wobei freie Bereiche 3 und Hindernisse 7 aus der Belegungskarte 2 ersichtlich sind.

In the following, a preferred embodiment of the method according to the invention will be described step by step. The specified sequence of the method is not to be understood as a chronological order.

• 1. During a drive of the vehicle 10 become successive pictures 5 with one in the direction of travel of the vehicle 10 aligned mono camera (front camera) 4 added.
• 2. With image processing algorithms, a movement of individual pixels or image areas within one of these images 5 tracked existing image sequence. This tracking of the movement is realized via the optical flow (see 2 in which the optical flux for certain pixels is shown). In the process, so-called tracks (trajectories on the image plane) are formed by the individual pixels (also named as feature).
• 3. The movement of the vehicle 10 is done with the help of usually in the vehicle 10 existing sensors (eg Wegimpulsgeber, yaw rate sensor, steering wheel sensor) is detected in order to determine the movement of the camera 4 and finally the position the camera 4 when capturing a particular image.
• 4. For each image A, certain pixel pairs are extracted from the feature tracks, with one pixel being the feature in the image A and the other pixel of the pixel pair being the corresponding pixel of the feature in the image B taken before the image A. , corresponds. The images A and B are so out of the camera 4 Captured images selected that the vehicle movement and thus the camera movement between the images A and B is significant (eg> 10 cm). In 3 are those of the mono camera 4 pictures taken over time 5 shown. The optical flow (OF) is in each case between two temporally adjacent images 5 calculated and chained in tracks. For three-dimensional reconstruction ("Structure from Motion" (SfM)) two temporally successive images are selected such that the corresponding movement of the camera 4 between these two images is significant (eg> 10 cm).
• 5. The in step 3 certain movement of the vehicle 10 or the camera 4 between the images A and B is in the movement of the camera with the help of calibration data (location of the camera and orientation of the camera) 4 Converted to the position of the camera 4 when taking the corresponding picture 5 to determine.
• 6. Reconstruction of the camera movement can be improved by estimating the so-called epipolar geometry (relative position of the camera at the time of taking certain pictures) taking into account the optical flow. This can also be a pitching and rolling motion of the vehicle 10 and with it the camera 4 better taken into account.
• 7. The exact position of the camera when taking the image A relative to the position of the camera as accurate as possible when taking the image B is used to determine the intersection of the rays from image A and image B for each pixel pair AB. This method is called triangulation in image processing, where the intersection of the rays determines the position of the respective pixel in the three-dimensional space.
• 8. Structures or objects which can be localized precisely in three-dimensional space by the pixels thus detected, are transformed into a cell-based occupancy map 2 or environment map entered, so this occupancy card 2 represents a two-dimensional discretization of the vehicle environment. Each cell of this occupancy card becomes 2 assigned a value which describes the probability of occupying the corresponding cell with an object. In other words, each cell has a probability value corresponding to the probability that the corresponding cell is occupied by a collision object. From this probability, the cells of the occupancy map can be 2 subdivide into free (occupied by no object) and occupied cells.
• 9. The probability values of the cells are constantly updated, allowing filtering of the data or probability values over time.
• 10. Based on the information from the occupancy card 2 can the free space 3 in front of and around the vehicle 10 be determined around. In 4b is a picture of the camera 4 shown. The 4a represents the occupancy map of the vehicle environment associated with this image, with blank areas 3 and obstacles 7 from the occupancy card 2 can be seen.

In 5 the method according to the invention is shown schematically. Input data of the method are from the camera 4 taken pictures 5 and by vehicle odometry (position determination of the vehicle 10 based on the data of his propulsion system, z. B. counting the Radumdrehungen) determined data. About the optical flow, a determination of the movement of the camera (ego motion estimation) and through the triangulation are collision objects in the environment of the vehicle 10 determined and corresponding to the occupancy card 2 entered.

In 6 schematically is a vehicle according to the invention 10 with a system according to the invention 20 for creating a model of an environment of the vehicle 10 shown. The system 20 includes besides a controller 6 two cameras 4 and 8th , with which pictures 5 the environment of the vehicle 10 be recorded.

QUOTES INCLUDE IN THE DESCRIPTION

This list of the documents listed by the applicant has been generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Cited patent literature

• DE 102007021576 A1 [0002]
• EP 2048618 A1 [0002]
• DE 102006039104 A1 [0002]
• DE 102006055908 A1 [0002]

Claims (13)

Method for creating a model of an environment of a vehicle ( 10 ), using a camera ( 4 ) the environment in the direction of travel of the vehicle ( 10 ) and several images ( 5 ) of the environment, whereby by means of these images three-dimensional structures and their position in the environment are detected by moving these structures within one of the several images ( 5 ) and wherein information about the structures in a two-dimensional cell-based map ( 2 ) of the environment, each cell of the card ( 2 ) is assigned an information as to whether this cell has an object represented by the structures ( 1 ) is occupied. Method according to claim 1, characterized in that the camera is a monocamera ( 4 ). Method according to claim 1 or 2, characterized in that the camera ( 4 ) in the direction of travel of the vehicle ( 10 ) is aligned. Method according to one of the preceding claims, characterized in that for certain of the several images ( 5 ) a surface segmentation, in which the respective image is subdivided into surface segments of similar color or structure, and that surface segments which contain a predetermined amount of free cells, their probability of occupancy with an object ( 1 ) are below a predetermined value include, as object-free area segments ( 3 ). Method according to one of the preceding claims, characterized in that from the plurality of images ( 5 ) certain images are selected in which the camera ( 4 ) is moved by more than a predetermined distance between two successive of these particular images, and that by means of triangulation for each of the specific images a localization of the structures within the environment is determined. Method according to claim 5, characterized in that a movement of the camera ( 4 ) is determined by means of the restrictions imposed by the epipolar geometry on the movement of the structures in the image sequence and by detecting an optical flow (OF) within the image sequence, and that the movement of the camera ( 4 ) a respective position of the camera ( 4 ) is determined for each of the particular images, and that the triangulation depends on the position of the camera ( 4 ) is performed on the location of the structures. A method according to claim 6, characterized in that the optical flow (OF) is determined by a method with which for each pixel of a given image (OF) 5 ) depending on neighboring images ( 5 ) of the particular image ( 5 ) a velocity vector is calculated. Method according to claim 6 or 7, characterized in that at least one further camera ( 8th ) for detecting the environment of the vehicle ( 10 ), which is at a location other than the camera ( 4 ) on the vehicle ( 10 ) and with which several images ( 5 ) of the environment that by means of the images ( 5 ) of the at least one further camera ( 8th ) the movement of the camera ( 4 ) and that the determination of the movement of the camera ( 4 ) based on the estimated motion of the camera ( 4 ) he follows. Method according to one of the preceding claims, characterized in that by means of ultrasonic waves information about the objects ( 1 ) are detected in the environment and that, based on this information, the cell-based map ( 2 ) of the environment is generated. Method according to one of the preceding claims, characterized in that the method is automated in a collision monitoring system of the vehicle ( 10 ) is carried out. System for creating a model of an environment of a vehicle ( 10 ), whereby the system ( 20 ) a camera ( 4 ), which the environment in the direction of travel of the vehicle ( 10 ), and a controller ( 6 ), wherein the controller ( 6 ) for controlling the camera ( 4 ) is designed to take multiple pictures ( 5 ) of the environment of the vehicle ( 10 ), whereby the controller ( 6 ) is configured such that the controller ( 6 ) by means of these images ( 5 ) three-dimensional structures and their position in the environment detected by the controller ( 6 ) a movement of these structures within one of the multiple images ( 5 ), and that the controller ( 6 ) Information about the structures in a two-dimensional cell-based map ( 2 ) of the environment, whereby the controller ( 6 ) each cell of the card ( 2 ) assigns an information as to whether this cell has an object represented by the structures ( 1 ) is occupied. System according to claim 11, characterized in that the system ( 20 ) is configured for carrying out the method according to one of claims 1-10. Vehicle with a system ( 20 ) according to claim 11 or 12.

Images