DE102016124888A1 - A method for assisting a driver of a motor vehicle when parking using a driver assistance device, corresponding computer program product and driver assistance device - Google Patents

Abstract

The invention relates to a method for assisting a driver of a motor vehicle (12) when parking in a parking space (P) using a driver assistance device of the motor vehicle (12), the method having the steps: in a training mode (TS1 - TS4) of the driver assistance device: Recording and storing reference data via a trajectory (14) to the parking lot (P) while the motor vehicle (12) is controlled by the driver on the trajectory (14), at least some of said reference data having image-based reference information using at least one of Vehicle (12) mounted camera is generated; and in a subsequent reproduction mode (RS1-RS3) other than the training mode: recording image-based information using the at least one camera and comparing the image-based information with the image-based reference information by the driver assistance device, the driver assistance device determining a current position (FIG. 22) of the motor vehicle (12) with respect to the trajectory (14) determined. The invention further relates to a corresponding computer program product and a corresponding driver assistance device.

Description

The invention relates to a method for assisting a driver of a motor vehicle when parking in a parking space using a driver assistance device of the motor vehicle.

The invention further relates to a corresponding computer program product and a driver assistance device for assisting a driver of a motor vehicle when parking in a parking space.

Automatic parking systems are already on the market. A new category of automatic parking systems is the trained parking system. Modern semi-autonomous vehicles are configured to self-park. For this they need to know the geometry of their environment. Trained parking systems use various sensors to record information from the environment (landmarks) corresponding to a trained ("trained") trajectory, and in subsequent "playback" relate the newly acquired information to previously stored information determine its own position with respect to the stored trajectory, which is then used to make decisions about how to maneuver the vehicle until it finally parks at the stored parking lot position.

The document DE 10 2013 015 348 A1 describes a method for assisting a driver of a motor vehicle when parking in a parking space using a driver assistance device of the motor vehicle. The method includes a training (or learning) mode of the driver assistance device and an operating mode of the driver assistance device. In both modes, ie in the training mode and in the operating mode, the motor vehicle must travel from a starting point along a trajectory (of a path) to an end position of the parking operation in the parking space, ie to the parking position.

It is an object of the invention to provide a method for assisting a driver of a motor vehicle when parking in a parking space using a driver assistance device of the motor vehicle, a corresponding computer program product and a corresponding driver assistance device, which are cost-efficient and suitable for many applications.

This object is achieved by a method, a computer program product and a driver assistance device having the features according to the respective independent claims. Advantageous embodiments of the invention are subject of the dependent claims, the description and the figures.

The inventive method for assisting a driver of a motor vehicle when parking in a parking space using a driver assistance device of the motor vehicle comprises: (i) in a training mode of the driver assistance device: recording and storing reference data via a trajectory (path) to the parking lot while the motor vehicle is controlled by the driver along the trajectory, at least some of said reference data having image-based reference information generated using at least one vehicle-mounted camera; and (ii) in a subsequent playback mode other than the training mode: recording image-based information using the at least one camera by the driver assistance device and comparing the image-based information with the image-based reference information, the driver assistance device relating a current position of the motor vehicle with respect to a result of the comparison determines the trained trajectory. The method may further comprise a step of determining the remaining portion of the trajectory from the current position to the parking lot and identifying the parking space using knowledge of the determined current position and the trained trajectory.

This method enables trained parking assistance using cameras that provide the only sensor signal. The core idea is that stored information extracted from camera images can be used to locate the vehicle position at a later time.

According to a preferred embodiment of the invention, the image-based reference information and / or the image-based information has characteristic features that are captured in a frame captured by the camera, each feature being associated with a corresponding image descriptor. The characteristic features are related to objects in the vicinity of the vehicle (landmarks) when traveling along the trajectory. The descriptors are such that they uniquely define a feature, and they may be associated with features that are extracted in the subsequent rendering. The image-based reference information and / or the image-based information is preferably a description of a characteristic 3D feature in the vicinity of the trajectory. This description is based on the 2D representation of the 3D feature in the image. The core idea is that stored characteristic Features extracted from camera images can be used to locate the vehicle position at a later time.

• Schritt 1: Erfassen der charakteristischen Merkmale in den Einzelbildern der Kamera;
• Schritt 2: Anpassen der charakteristischen Merkmale an die Einzelbilder;
• Schritt 3: 3D-Rekonstruktion der Positionen der charakteristischen Merkmale; und
• Schritt 4: Bündelausgleichung von 3D-Positionen der charakteristischen Merkmale und der Fahrzeugposition.

According to another preferred embodiment of the invention, the training mode comprises the following steps:

• Step 1: Detecting the characteristic features in the individual images of the camera;
• Step 2: Match the characteristic features to the frames;
• Step 3: 3D reconstruction of the positions of the characteristic features; and
• Step 4: Bundle equalization of 3D positions of the characteristic features and the vehicle position.

According to a particularly preferred embodiment of the invention, in the 3D reconstruction step (step 3) an estimation of the vehicle movement is also carried out by an estimate of the corresponding essential / fundamental matrix.

According to a particularly preferred embodiment of the invention, the bundle adjustment step (step 4) comprises a non-linear optimization in which the reprojection error of the 3D features is minimized.

• Schritt 1: Erfassen der charakteristischen Merkmale in den Einzelbildern der Kamera;
• Schritt 2: Zuordnen der charakteristischen Merkmale zu den im Trainingsmodus erfassten charakteristischen Merkmalen; und
• Schritt 3: Bündelausgleichung der Fahrzeugposition unter Verwendung der 3D-Positionen der charakteristischen Merkmale zur Fahrzeuglageoptimierung.

According to another preferred embodiment of the invention, the playback mode comprises the following steps:

• Step 1: Detecting the characteristic features in the individual images of the camera;
• Step 2: associating the characteristic features with the characteristic features acquired in the training mode; and
• Step 3: Bundle adjustment of the vehicle position using the 3D positions of the characteristic features for vehicle attitude optimization.

According to another preferred embodiment of the invention, there are two types of frames, namely key frames and normal frames, wherein in training both types of frames are processed, but only the trained reference points of the trajectory (trajectory points) and trained features associated with the key frames are incorporated in a trained card and used for playback. Key frames are selected based on the distance traveled by the vehicle, the number of features adapted to the previous key frames, or a combination thereof.

According to yet another preferred embodiment of the invention, information of a navigation satellite system is stored for at least some of the individual images, in particular the key single images. This information can be GPS data.

The computer program product of the invention comprises computer executable program code portions having program code instructions configured to execute the above method.

1. (i) Referenzdaten über eine Trajektorie zu dem Parkplatz in einem Trainingsmodus aufzuzeichnen und zu speichern, während das Kraftfahrzeug durch den Fahrer auf der Trajektorie gesteuert wird, wobei mindestens einige dieser Referenzdaten bildbasierte Referenzinformation enthalten, die durch die Verwendung mindestens einer am Kraftfahrzeug montierten Kamera erzeugt wird;
2. (ii) bildbasierte Information unter Verwendung der mindestens einen Kamera in einem vom Trainingsmodus verschiedenen nachfolgenden Wiedergabemodus aufzuzeichnen; und
3. (iii) die bildbasierte Information mit der bildbasierten Referenzinformation im Wiedergabemodus zu vergleichen, wobei die Fahrerassistenzvorrichtung in Abhängigkeit von einem Ergebnis des Vergleichs eine aktuelle Position des Kraftfahrzeugs bezüglich der Trajektorie bestimmt. Ferner kann die Fahrerassistenzvorrichtung dafür konfiguriert sein, den verbleibenden Abschnitt der Trajektorie von der aktuellen Position zum Parkplatz zu bestimmen und den Parkplatz unter Verwendung der Kenntnis über die erfasste aktuelle Position und die trainierte Trajektorie zu identifizieren. Die Fahrerassistenzvorrichtung ist vorzugsweise eine computerbasierte Vorrichtung mit einem Prozessor und einem Datenspeicher.

The invention further relates to a driver assistance device for assisting a driver of a motor vehicle when parking in a parking space, wherein the driver assistance device is configured for:

1. (i) record and store reference data about a trajectory to the parking lot in a training mode while the motor vehicle is being controlled by the driver on the trajectory, at least some of said reference data including image-based reference information generated by the use of at least one camera mounted on the motor vehicle becomes;
2. (ii) record image-based information using the at least one camera in a subsequent playback mode other than the training mode; and
3. (iii) comparing the image-based information with the image-based reference information in the playback mode, wherein the driver assistance device determines a current position of the motor vehicle with respect to the trajectory in dependence on a result of the comparison. Further, the driver assistance device may be configured to determine the remaining portion of the trajectory from the current position to the parking lot and to identify the parking lot using knowledge of the detected current position and the trained trajectory. The driver assistance device is preferably a computer-based device with a processor and a data memory.

According to a preferred embodiment of this driver assistance device, the driver assistance device is configured to execute the aforementioned method.

Further features of the invention will become apparent from the claims, the figures and the description of the figures. All mentioned in the above description features and feature combinations and those mentioned in the following description of the figures and / or only in The features and feature combinations shown in the figures are usable not only in the combination specified, but also in other combinations or independently.

• 1 eine im Trainingsmodus erzeugte Karte einer Parkzone mit einer trainierten Trajektorie eines Parkvorgangs;
• 2 ein Ablaufdiagramm eines Trainingsmodus einer Fahrerassistenzvorrichtung zum Unterstützen eines Fahrers eines Kraftfahrzeugs beim Einparken in einen Parkplatz einer Parkzone;
• 3 ein Einzelbild und eine Karte der Parkzone mit einer Wiedergabetrajektorie eines Parkvorgangs, die beide im Wiedergabemodus erzeugt werden; und
• 4 ein Ablaufdiagramm eines Wiedergabemodus der Fahrerassistenzvorrichtung zum Unterstützen des Fahrers beim Einparken in einen Parkplatz.

The invention will be explained in more detail below based on a preferred embodiment and with reference to the accompanying drawings; show it:

• 1 a training map generated map of a parking zone with a trained trajectory of a parking operation;
• 2 a flowchart of a training mode of a driver assistance device for assisting a driver of a motor vehicle when parking in a parking lot of a parking zone;
• 3 a frame and a map of the parking zone with a playback trajectory of a parking operation, both of which are generated in the playback mode; and
• 4 a flowchart of a playback mode of the driver assistance device for assisting the driver when parking in a parking lot.

The following is an example of a trained parking parking aid described: The driver "trains" the vehicle by entering a parking lot P (Pitch) drives and selects to save the trajectory. In "playback" mode, the vehicle recognizes the scene and repeats the trained trajectory.

This requires two operating modes: "Training" and "Playback", whereby in the "Training" mode a card 10 the scene is generated while a motor vehicle 12 by the driver on a trajectory 14 is controlled, the trajectory 14 , along the vehicle 12 drives, is recorded. In "Playback" mode, the vehicle must 12 recognize the scene and its position within the scene.

training phase

1 shows the corresponding map 10 the scene, for example, a parking zone, wherein the motor vehicle 12 travels over this zone during training mode. The car 12 is for example a passenger car. The car 12 has a driver assistance device with a plurality of cameras (not shown). The cameras are eg in the front area and in the rear area of the vehicle 12 arranged. The cameras detect front and rear surrounding areas of the motor vehicle 12 ,

The vehicle 12 travels along a trajectory 14 over the parking zone from a reference home position (not shown) to a parking lot P the parking area, the parking lot P the reference target position of the trajectory 14 is. The trajectory 14 has a straight section followed by a curved section and is defined by several reference points 16 (Trajectory points) on the trajectory 14 sufficiently described. There are several characteristic features 18 . 20 in an area of the trajectory detected by the driver assistance device 14 in frames taken by the camera.

2 FIG. 12 shows a type of a flowchart of a training mode of a driver assistance device for assisting a driver of a motor vehicle when parking in a parking lot. FIG P , This training mode has four training steps TS1 to TS4 on. The goal of the training phase is to create a data-poor consistent map 10 to provide trained trajectory points and trained traits. step TS1 is a step for feature detection; step TS2 is a feature adaptation step; step TS3 is a step to 3D reconstruction and step TS4 is a step to bundle adjustment of 3D positions / points and vehicle position.

Like in 1 is the target of the training phase, a data-poor consistent map 10 trained trajectory points 16 and trained features. Trained trajectory points 16 are key positions in the 3D world that collectively identify the path (the trajectory) 14 to which the vehicle 12 in the training phase follows.

Trained features are visual features with a 3D "in the world" position and an associated visual descriptor. The descriptors are such that they uniquely define a feature, and they may be associated with features that are extracted in a subsequent rendering. The driver assistance device is configured to work with any descriptor feature, but a sufficiently robust descriptor is recommended. AKAZE is the feature descriptor proposed here for this solution. Feature descriptor storage during training and customization during playback is the key that gives it to the vehicle 12 allows itself to locate (associate) itself with the training data during playback.

There are two types of frames, namely key frames and normal frames, where both frame types are processed during training, but only the trained trajectory points and trained features associated with key frames are stored in the trained card and used for playback. The The concept of these "key frames" and "normal frames" is important to this solution.

Frames are bundled in windows with a key frame at the beginning, followed by normal frames. Key frames are dynamically selected based on a distance traveled distance and the number of features matched to a previous key frame.

In the training phase, a set of visual features are extracted from each live camera frame, and a visual descriptor is stored for each feature. The number of extracted features is limited to a fixed number to ensure that efficient run time is achieved.

Features are fit to subsequent frames according to their descriptor, and using this fit, a 3D reconstruction is performed to obtain an estimate for each feature position in a 3D world. The 3D Reconstruction also provides an estimate of the vehicle motion via the estimate of the essential / fundamental matrix.

Each window, consisting of a key frame and subsequent normal frames, undergoes a burst equalization to obtain the optimal trajectory positions and 3D feature positions. The bundle equalization step involves non-linear optimization in which the reprojection error of the 3D features is minimized. In addition, GPS information for each key frame in the trajectory 14 Obviously, this position is not accurate enough for playback, but it can be used for an absolute position reference.

Playback phase

3 shows a map 10 the scene in the playback phase and a corresponding image I , which is recorded by the camera at the current vehicle position. The point 22 in the map 10 is the estimate of the current vehicle position on the data-poor map. The big points 24 are the trained features of the fully trained trajectory. The smaller points 26 are those trained features that match the current playback frame. The characteristic features 24 . 26 are with the different objects 28 connected in the scene.

In the playback phase, a new set of features 24 . 26 captured, and every feature 24 . 26 gets a visual descriptor. Using these visual descriptors will become the new features 24 . 26 to the trained features 18 . 20 customized.

The initial re-localization becomes a subset of the new features 24 . 26 selected, and the full trained trajectory 14 is searched to find matching descriptors. The key match most match image is selected as the "closest key frame" and then attempts to match the complete set of new features to the features in the "closest key frame". With GPS information for the current position and key frames in the stored trajectory 14 For example, a refined search space may be used for the feature descriptor matches to reduce run time and ensure a more robust fit.

When a sufficient number of features are matched, the current vehicle position becomes the trained trajectory 14 determined using a bunching technique. The adapted trained features 24 are reprojected into the current live frame, and the reprojection error is calculated as the difference between the reprojected position and the observed position in the current frame. This error is made by selecting the optimum vehicle position with respect to the trained trajectory 14 minimized. A strict filtering of outliers ensures a reliable position.

In subsequent frames, the search is restricted to the "closest key frame" and its neighbors. The "closest key frame" is reset to one of these groups as required. If there is not a sufficient number of matches within this sentence, the entire trajectory is searched again.

4 FIG. 12 shows a type of a flowchart of a reproduction mode of the driver assistance device for assisting a driver when parking in a parking space P. This training mode has three reproduction steps RS1 to RS3 on. step RS1 is a step for feature detection; step RS2 is a feature mapping step to trained features; and step RS3 is a bundle adjustment step for vehicle position optimization.

LIST OF REFERENCE NUMBERS

10

map

12

vehicle

14

trained trajectory

16

Trajectory points (trained trajectory)

18

3D features (large circles)

20

other 3D features (small circles)

22

position

24

trained characteristics

26

recorded characteristics

28

Object in the environment

P

parking spot

I

image

TS1

first training step

TS2

second training step

TS3

third training step

TS4

fourth training step

RS1

first reproduction step

RS2

second reproduction step

RS3

third reproduction step

QUOTES INCLUDE IN THE DESCRIPTION

This list of the documents listed by the applicant has been generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Cited patent literature

• DE 102013015348 A1 [0004]

Claims (11)

A method of assisting a driver of a motor vehicle (12) when parking in a parking lot (P) using a driver assistance device of the motor vehicle (12), the method comprising the steps of: in a training mode (TS1 - TS4) of the driver assistance apparatus: recording and storing reference data via a trajectory (14) to the parking lot (P) while the motor vehicle (12) is controlled by the driver on the trajectory (14), at least some of which Reference data comprising image-based reference information generated using at least one camera mounted on the vehicle (12); and in a subsequent reproduction mode (RS1-RS3) other than the training mode: recording image-based information using the at least one camera and comparing the image-based information with the image-based reference information by the driver assistance device, the driver assistance device determining a current position (22 ) of the motor vehicle (12) with respect to the trajectory (14). Method according to Claim 1 characterized in that the image-based reference information and / or the image-based information comprises characteristic features (18, 20, 24, 26) captured in a frame of an image captured by the camera, this feature being associated with a corresponding image descriptor , Method according to Claim 2 characterized by the following steps of the training mode: detecting the characteristic features (18, 20) in the frames of the camera (TS1); Adapting the characteristic features (18, 20) to individual images (TS2); 3D reconstruction of the positions of the characteristic features (18, 20) (TS3); and beam equalization of 3D positions of the characteristic features (18, 20) and the vehicle position (TS4). Method according to Claim 3 , characterized in that in the 3D reconstruction step (TS3) also carried out an estimation of the vehicle trajectory. Method according to Claim 3 or 4 , characterized in that the beam equalization step (TS4) includes non-linear optimization in which the re-projection error of the 3D features is minimized. Method according to one of Claims 3 to 5 Characterized by the steps of the playback mode: detecting said characteristics (24, 26) in the individual images from the camera (RS1); Associating the characteristic features (24, 26) with the features (18, 20) (RS2) detected in the training mode; and beam adjustment of the vehicle position using the 3D positions of the characteristic features (18, 20) and the vehicle position for vehicle attitude optimization (RS3). Method according to one of Claims 2 to 6 , characterized by two types of frames, namely key frames and normal frames, wherein both single-frame types are processed during training, but only the trained reference points of the trajectory and trained features associated with key frames are stored in a trained map and for the Playback to be used. Method according to one of Claims 2 to 7 , characterized in that for at least some of the frames, in particular the key frames, information of a navigation satellite system is stored. A computer program product comprising computer executable program code portions having program code instructions for carrying out the method of any one of Claims 1 to 8th , A driver assistance device for assisting a driver of a motor vehicle (12) when parking in a parking space (P), the driver assistance device being configured to: in a training mode, recording and storing reference data about a trajectory (14) to the parking lot (P) while the motor vehicle (12) is being controlled by the driver on the trajectory (14), at least some of said reference data having image-based reference information generated by the vehicle Use of at least one camera mounted on the vehicle; record image-based information using the at least one camera in a subsequent reproduction mode other than the training mode; and comparing the image-based information with the image-based reference information in the playback mode, wherein the driver assistance device determines a current position (22) of the motor vehicle (12) with respect to the trajectory (14) in dependence on a result of the comparison. Device after Claim 10 , characterized in that the driver assistance device is configured to perform the method according to one of Claims 1 to 8th perform.

Images