DE102015203016A1 - Method and device for optical self-localization of a motor vehicle in an environment - Google Patents

Abstract

A method for locating a motor vehicle in an environment, comprising the steps of: in a first method step, executing a learn run (100) comprising: receiving odometry data of the motor vehicle via an interface of a camera (101) capturing at least two images with the camera while driving at different positions of the motor vehicle (102), extracting features in the at least two images (103), finding pairs of identical features in the at least two images (104), stereoscopically determining a three-dimensional position of the features using the odometry data (105 ), Creating a three-dimensional map (106) and entering the features at their three-dimensional positions (107), then, in one or more second process sections, executing a follow-up run (200) comprising: acquiring at least one further image with the camera (10); 202), find me at least one further mappings (203), determining relative orientations of the plurality of features to the motor vehicle from the mapping position (204), locating the plurality of features in the map (205), triangulating based on the features and relative orientations, and inferring localization information of the motor vehicle in the map (206), outputting of the localization information at the interface (207), wherein all method steps of the two procedural sections are executed in the camera. Furthermore, the invention relates to associated device (1).

Description

The invention relates to a method and a device for locating a motor vehicle in an environment.

Modern motor vehicles include a number of assistance systems that provide the driver with extensive additional information while on the move. This information can be, for example, navigation instructions, traffic jam messages or information about traffic signs for the current roadway area. Also known to the driver when parking assist parking assistance, which detect obstacles in the immediate vehicle environment and, for example, as a graphical representation or as an acoustic signal to notify the driver information about their distance or their position.

Localization methods are known from the prior art in which a motor vehicle generates an environmental map in an unknown environment with the aid of ultrasonic or radar measurements. For this purpose, relative distances to objects in the surroundings of the motor vehicle are determined on the basis of the ultrasonic or radar measurements and the odometry data of the motor vehicle. Subsequently, the motor vehicle can locate itself in already mapped areas of the map. These methods are also referred to as Simultaneous Localization and Mapping (SLAM).

Also known are methods in which images of the vehicle environment are detected by means of cameras and evaluated computer-aided. In this way, the environment visible to the camera can be analyzed and their information content can be utilized for assistance systems.

Several variants of assistance systems are known in which a front camera detects an environment in front of the motor vehicle. Illustrations are recorded and stored in a memory with the odometrically determined position of the motor vehicle.

In this way, extensive maps can be created. If the motor vehicle later returns to an already mapped location, the assistance system can autonomously locate the motor vehicle and notify the driver of the current position of the motor vehicle in the map.

Out DE 10 2008 002 598 A1 a device is known which assists the driver when driving off a previously learned route. For this purpose, images of the vehicle surroundings at the start position and optionally at further positions of the learned route are recorded with a camera and stored in a memory. If the start position of such a learned route is approached again, the starting point and optionally further positions are detected by taking further pictures and comparing them with the stored pictures. Subsequently, the learned route can be traversed, the driver gets current information about the deviation to the desired position of the learned route. For this purpose, the device evaluates the odometry data of the motor vehicle and compares the current with the stored images.

A disadvantage of the aforementioned devices and methods is that they must hold large amounts of data for a plurality of images in order to capture the environment of the motor vehicle at various positions with sufficient accuracy. This increases the technical requirements: There must be a memory with sufficient capacity and the capacity of the computer must be large enough to process the data in real time. Overall, this increases the effort and costs.

A disadvantage of methods and devices that locate the motor vehicle mainly based on odometry data is that the odometry data is too inaccurate to always accurately locate the motor vehicle. For example, unknown deviations of the paths of the wheels due to slippage or different abrasions can occur. This then falsifies the result when locating the motor vehicle or reduces its accuracy.

It is therefore desirable to have available a method and a device in which localization of a motor vehicle on the one hand is possible with the aid of a much smaller amount of data stored and on the other hand high accuracy is achieved in locating the motor vehicle in an already passed through range ,

The invention is therefore based on the problem to provide a method and a device that allow a better localization of a motor vehicle relative to objects in an environment. In particular, a better localization for recurring trips should be possible, such as parking in a narrow garage or parking space.

The technical problem is solved by a method with the features of claim 1 and a device having the features of claim 10. advantageous Embodiments of the invention will become apparent from the dependent claims.

The invention is based on the idea to define a desired trajectory during a learning trip and to capture images of an environment of a motor vehicle with a camera, so that a monoscopic image sequence arises. In each of the illustrations, graphical features, which can originate, for example, from objects in the environment, are recognized and extracted. Such graphic features in the figures may be, for example, geometric shapes, spatial frequencies, color gradients, contrasts or pixel patterns. The positions of the recognized features are then determined stereoscopically from the monoscopic image sequence using the odometry data of the motor vehicle and entered into a three-dimensional map. The captured images themselves are then discarded, so that only very small amounts of data for the card incurred. After the learning journey, a follow-up journey can be carried out. In this case, further images of the environment are captured with the camera and features are recognized in the other images, as well as their relative orientation, that is, directions of the characteristics in space relative to the camera, d. h to the motor vehicle or in which the camera is fixed, determines the positions of the features in the other figures. The recognized features and their respective relative orientation to the camera, d. H. to the motor vehicle, are then used to locate the motor vehicle in the map and output a localization information, which includes, for example, a deviation of an actual trajectory of the desired trajectory. In this case, all method steps in a single camera module, hereinafter referred to as camera, executed.

definitions

A position denotes a punctiform location in a three-dimensional coordinate system. The coordinate system can be designed, for example, Cartesian.

An orientation of a motor vehicle refers to an alignment of the center axes of the motor vehicle with respect to the axes in the coordinate system. The orientation may include a roll, pitch and yaw angle. The orientation can be detected or determined both by the camera and by vehicle odometry. For example, a pitch angle of the motor vehicle can be measured with additional loading of the vehicle odometry in the chassis and be taken into account later in the analysis of the images captured by the camera for correction purposes.

A feature refers to a property or set of properties associated or attributable to an area in a map or position in the map based on graphic image information of the map or a portion of the map. The subarea can be identical to the area to which the property or totality of the properties and thus the characteristic is assigned. Features may be, for example, horizontal or vertical lines or their crossing points in the images, but also contrasts, color gradients or spatial frequencies or local frequency patterns after a Fourier or wavelet analysis. Also, pixel patterns or particular arrangements of pixel patterns or the like may represent features. As a rule, features can not be directly assigned to a physical object.

A map is a three-dimensional non-physical scaffold with a coordinate system. The map may be a representation of three-dimensional space formed from a grid. For example, in memory, the map may be constructed in the form of vectors that designate the three-dimensional locations of features. As a reference point, for example, serve a very first specific position of the motor vehicle. Mapping then refers to the entry of features and their position in the three-dimensional map.

A trajectory denotes a set of temporally and spatially linked positions and possibly orientations of the motor vehicle. The trajectory has a start position and possibly a start orientation and an end position and optionally an end orientation, which define the beginning or the end of the trajectory.

An actual trajectory designates the quantity of the positions determined for a currently performed journey and possibly orientations of the motor vehicle.

A desired trajectory designates the quantity of the positions predefined for a currently performed journey and possibly orientations of the motor vehicle.

The difference position and the difference orientation denote the deviations of the corresponding desired sizes and actual sizes from each other.

Location information comprises a set or a subset of the currently measured or determined values for the following quantities: the actual Position, the actual orientation, the difference position, the difference orientation.

A stereoscopic position determination refers to a position determination of a feature from a monoscopic image sequence, in which at least two images, which were acquired from a camera moving in an environment from different surrounding positions, and a known distance between the two different surrounding positions and possibly an additional orientation change Camera between the two different surrounding positions, a three-dimensional position of the feature is determined. For example, a structure-from-motion method can be used for this purpose (see, for example, US Pat. RC Bolles et al., Int. Journal of Computer Vision, 1, 7-55 [1987] ).

Preferred embodiments

In particular, there is thus provided a method for locating a motor vehicle in an environment, comprising the steps of: in a first method step, performing a learning run comprising: receiving odometry data of the motor vehicle via an interface of a camera, capturing at least two images with the camera during driving at different positions of the camera relative to the environment, the camera being fixed to the motor vehicle, extracting features in the at least two images, finding pairs of identical features in the at least two images, stereoscopically determining a three-dimensional position of the features using the odometry data Creating a three-dimensional map and plotting the features at their three-dimensional positions, then, in one or more second method sections, executing a sequential trip, comprising: detecting at least one more n imaging with the camera, determining a plurality of features in the at least one further image, determining relative orientations of the plurality of features to the motor vehicle based on the imaging position of the features, locating the plurality of features in the map, triangulating based on the features and relative orientations, and inferring localization information of the motor vehicle in the map, output of the localization information at the interface, wherein all process steps of the two process sections are executed in the camera.

Furthermore, a device is advantageously provided for locating a motor vehicle in an environment comprising a camera for detecting the environment of the motor vehicle, which is fixed to the motor vehicle, wherein the camera comprises a control device which comprises a memory and an interface, wherein the control device is configured to receive odometry data of the motor vehicle during a learning run via the interface of the camera, to acquire at least two images while driving at different positions of the camera relative to the surroundings with the camera, to determine features in the at least two images, Identify pairwise identical features in the at least two images and stereoscopically determine a three-dimensional position using the odometry data for the features, a three-dimensional map of the features and their three-dimensional positions to create and store in the memory, then, in a second process section during one or more subsequent trips to capture at least one more image with the camera to determine several features in the at least one other image, the relative orientation of the plurality of features to the motor vehicle based determine the imaging position, locate the plurality of features in the map, locate the motor vehicle in the map by means of triangulation and output localization information derived therefrom at the interface.

The advantage of the invention is that not more complete images and thus large amounts of data must be stored and kept, but only a lot of features and their positions in a three-dimensional map. This results in a drastic reduction of data compared to the above-mentioned prior art methods. The data reduction also means that a much smaller amount of data has to be evaluated when searching for features found in the current images in the map.

In addition, the position and possibly additionally the orientation of the motor vehicle are determined on the basis of the features recognized in the environment, which can be done much more accurately than with the methods known from the prior art based on the odometry data.

Furthermore, with the method according to the invention, for example, the localization of the motor vehicle can be designed to be location-specific variably in terms of accuracy. Thus, determining the position and optionally additionally the orientation of the motor vehicle on a wide highway lane requires less accuracy than with a narrow garage entrance.

All functions are performed by a camera, with the camera receiving odometry data at the interface and Output localization information, for example via a Controller Area Network (CAN) bus interface, which is widely used in motor vehicles. Retrofitting of motor vehicles with such a camera is therefore inexpensive possible. Since already in the camera a strong information compression is made, only very few data must be transmitted via the interface, so that the camera can be integrated into existing vehicle infrastructure.

A particularly advantageous embodiment provides that the localization information that is output by the camera, a position and optionally additionally include an orientation of the motor vehicle. The advantage is that it can be used to output an absolute position in the map. If, in addition, the orientation of the motor vehicle with respect to a reference direction is output, then the state of the motor vehicle with respect to its surroundings is completely known.

Particularly preferred is an embodiment in which several positions and possibly additionally orientations of the motor vehicle are stored in their chronological order as desired positions and optionally additionally desired orientations during the learning run in the first method section and linked to a desired trajectory. The desired trajectory can then be traversed later, in the second process section as part of a follow-up run and can thus serve as a reference trajectory. For example, during a parking operation, the actual positions and possibly also the actual orientations are entered into the three-dimensional map as desired trajectory. During the subsequent follow-up runs, the setpoint trajectory is selected as the reference and thus facilitates parking. During the travel of the desired trajectory localization information is provided continuously, which support the driver when driving the motor vehicle.

The invention makes it possible for a person other than the later driver to make a first-time parking and to learn the desired trajectory and then to store it. In this way, for example, a skilled driver take over the first time parking. Another, for example, inexperienced, driver can later, if he is in the vicinity of the desired trajectory, resort to the desired trajectory. During the travel of the desired trajectory he receives then continuously localization information of the motor vehicle, which support him when parking, for example by providing current position and possibly orientations of the motor vehicle.

In some situations it is necessary to additionally store information about the condition of the motor vehicle. Therefore, a particularly preferred embodiment of the invention provides that in addition to the positions and, where appropriate, the orientations of the motor vehicle also steering angles, directions of movement, torques and / or speeds are stored in their time sequence and with the respective position at which they were recorded during the Lernfahrt be linked. If necessary, a position can additionally be characterized by an orientation. This offers the advantage that additional information about the way in which a desired trajectory has been traversed or should be traversed is provided. For example, it makes little sense to park in a garage entrance at a speed of 50 km / h or to park in the garage when the motor vehicle is at an angle of 90 ° to the garage entrance. The recording of steering angles, directions of movement and speeds for the positions and possibly orientations on a desired trajectory makes sense, for example, when parking in a parallel parking space at the roadside, where in addition to the direction of movement above all the steering angle, linked to the current position on the target Trajectory that decides the success or failure of a parking operation. Thus, the parking for the driver is facilitated because he receives ongoing localization information about the motor vehicle and its actions.

The invention makes it possible to operate an automatic parking, lane or driving assistant based on the location information. Thus, for example, a parking assistance system can take over the parking of the motor vehicle fully automatically based on a desired trajectory. It is even possible that the driver steps out of the vehicle before parking and the parking assistance system assumes parking completely autonomously.

This increases the comfort for the driver and the stress due to complicated parking maneuvers with little space on busy roads can be minimized. In addition, parking spaces and car parks could be made more compact, so that there is an area savings, since the driver does not need more space for getting in and out, but always in the parked state in the vehicle on and off.

There are situations in which it is desirable to have several different target trajectories available for the same environment, for example, when there are multiple drivers who have different wishes or there are, for example, several possible parking spaces in a parking garage. Therefore, ideally, an embodiment is realized in which multiple target trajectories for areas in the map by further executing the first method section can be stored and then selected by an operator as the active setpoint trajectory. This is advantageous when multiple trajectories are needed for an area in the map. For example, so double garages can be comfortably approached from the same position. The driver drives onto the driveway and as soon as the motor vehicle has located itself in his 3D map, an assistance system offers both available target trajectories. The driver then selects there the target trajectory, for example, for the left of the two garages from. From now on, he receives localization information of the motor vehicle with respect to the desired trajectory for parking in the left-hand garage. Another time, because the left-hand garage is occupied, he has to park in the right-hand garage. He selects the target trajectory for the right-hand garage on the driveway and henceforth receives localization information relating to the target trajectory for parking in the right-hand garage. Also possible are desired trajectories for different positions and orientations of the motor vehicle in the garage, as well as different access routes, for example, if the entry into the driveway is already part of the desired trajectory.

In order to increase the convenience of operation, the simplest possible selection of a desired trajectory is desirable. Therefore, an embodiment is particularly advantageous in which one or more target trajectories stored for the current environment of the motor vehicle are recognized as lying in the surrounding area and are made available to the driver for selection as an active desired trajectory. For example, the driver drives to the entrance to his property and the assistance system recognizes that the motor vehicle is now in an area of the map for which there are two desired trajectories, one for the left and one for the right garage. It then alerts the driver, for example by outputting a signal that target trajectories are present, and offers the driver both target trajectories for selection.

The driver can now decide whether to select one of the two target trajectories or whether to do without a selection. Thus, for a position and, if appropriate, orientation in a simple manner that is comfortable for the driver, a plurality of desired trajectories can be kept available and, if necessary, provided in a user-friendly manner.

One embodiment provides for loading at least one area of a map with a desired trajectory via the interface into the camera. For example, an unknown environment, for example a parking garage of a shopping center, can be safely passed through by a driver. For example, the following scenario is possible: The driver is automatically assigned the parking space X when entering the car park. The motor vehicle is then automatically assigned by a corresponding system of the parking garage an area of the map for the parking garage and a target trajectory for the parking lot X, which are dubbed over the interface to the camera. From now on, the area of the map and the desired trajectory are available for parking lot X and can be selected by the driver. A purposeful parking or driving in unknown environments can thus be facilitated or even automated.

Particularly advantageous is an embodiment of the invention, in which the localization information comprises a difference between the desired and actual position and, where appropriate, desired and actual orientation. Thus, the driver of the motor vehicle continuously information and instructions are provided to assist him in driving the motor vehicle. The differences can be output, for example, at the interface and then subsequently displayed on a display device, for example a head-up display, or further processed by a vehicle control.

It is also possible that a fully automatic system for guiding the motor vehicle is operated on the basis of the determined differences between the desired and actual position and optionally additional orientation.

Ideally, an embodiment is realized in which the current location information is used to restrict the area of the map used for comparison with the features determined from the further map to a closer environment. For example, it is possible to limit the area to just a few meters in the immediate vicinity of the motor vehicle, for example when the motor vehicle is already located in the map.

This reduces the number of features in the map that need to be processed or compared. As a result, the computational effort in the control device decreases and the comparison can be performed faster and more time-saving, so that the localization information can be provided faster and thus more often.

It is desirable to assign an absolute, unique, possibly global, coordinate to the relative positions of the map. An embodiment of the invention therefore provides that positions in the map are assigned a global position coordinate. For example, a coarse positioning of the motor vehicle by means of a global navigation system, such as the Global Positioning System (GPS), performed and the means of GPS certain positions of the motor vehicle with those stored in the map positions and possibly linked additional orientations. This has the advantage that the motor vehicle can be coarsely localized even in the case of a card that has not yet been created or is still insufficient. Even with an existing card, a first coarse localization can be made in the card, so that the range for which the comparison is made between the features recognized in the current image and the features stored in the card can be restricted. The GPS data can be transmitted via the interface into the camera, for example.

In addition, by including global position data, it is possible to more quickly assign multiple different target trajectories to the same area of a map. For example, so differently executed bending operations in a driveway of a property can be immediately recognized as belonging to the same area of a map. This speeds up the construction of the map and reduces errors in the evaluation of the features in the map, which could be caused, for example, by other features being visible when turning from the left into the driveway than when turning from the right into the driveway.

Preferably, the camera is designed as an automotive front camera, i. H. firmly integrated into the motor vehicle. One advantage is that this camera can also be used for other assistance systems that detect and evaluate an environment in front of the vehicle. For example, a track recognition assistance system can be additionally realized with the camera.

A particularly advantageous arrangement provides that the automotive front camera is arranged behind or in the region of the windshield of the motor vehicle. This guarantees a good field of view of the camera and ensures adequate protection against damage.

Particularly preferably, the front camera is arranged in a roof node behind an interior mirror or designed as a roof node camera. In this installation position or design variant, a field of vision of a driver by the front camera is not or only slightly affected.

In one embodiment, it is provided that the camera is a rear view camera. In other embodiments, a rear view camera sensing the environment behind the vehicle may also be used in addition to the front camera to enhance and accelerate localization. The illustrations of the rear view camera are the same as with the images of the front camera. The determined features are entered and located in the same card.

The abovementioned embodiments represent examples for carrying out the invention. However, other embodiments are also possible which, above all, comprise different combinations of several features of the subclaims.

The invention will be explained in more detail with reference to preferred embodiments with reference to the figures. Hereby show:

1 a schematic representation of an embodiment for locating a motor vehicle in an environment with learning and departing a desired trajectory;

2a . 2 B both schematic diagrams for explaining the capture of an image with the camera and the following feature extraction;

3a . 3b . 3c schematic representations for explaining the determination of the positions of the features in a map by evaluating pairwise identical features in two different images at different positions of the motor vehicle in an environment;

4a . 4b . 4c schematic representations for explaining the emergence of the image by projection of a section of a front-side environment of a motor vehicle at different positions of the motor vehicle in the environment;

5a . 5b schematic representations for explaining the three-dimensional position determination of the feature from the figures in the 4b and 4c ;

6a . 6b . 6c schematic representations for explaining the determination of the actual position of the motor vehicle in the map using a single figure;

7a . 7b . 7c schematic representations for examples of multiple target trajectories in an area of the map, which represents an example of a garage entrance;

8th a schematic representation of a desired and an actual trajectory and exemplified two difference vectors;

9 a schematic flow diagram of the method.

In 1 is an embodiment of the device 1 for locating a motor vehicle 50 shown in an environment with learning and traversing a desired trajectory. In the motor vehicle 50 is, for example, the front side, the device 1 including the camera 2 , built-in. The camera 2 is preferably arranged as a front camera in a roof node, viewed from the driver behind an interior mirror. The camera 2 includes an image capture device 3 For example, a Charged Coupled Devices (CCD) sensor with associated optics, a control device 4 , an interface 5 and a memory 6 , Not to the device 1 include an illustrative vehicle odometry 7 and a vehicle controller 8th which via a data bus 9 and the interface 5 to the device 1 are connected.

The method will be simplified in the following for explanation.

The car 50 is located in an environment in which a learning journey is to be carried out. In the first part of the procedure, during a learning journey, the camera detects 2 via the image capture device 3 at least two images of the front-end environment at different positions of the motor vehicle in the environment, wherein the detection of the control device 4 is triggered and controlled. At the same time, the control device receives 4 over the interface 5 Odometry data of the motor vehicle 50 Which of the vehicle odometry 7 on the data bus 9 be sent and there via the interface 5 are available. In the at least two captured images, the controller extracts 4 then graphic features. Such features do not have to come from physically formed objects. However, different physically trained objects may have similar features in the graphical representation in the images. Features can be, for example, horizontal or vertical lines or their crossing points in the images, but also contrasts, color gradients or spatial frequencies after Fourier or wavelet analysis. Also, pixel patterns or particular arrays of pixel patterns may represent features. A person skilled in the art knows several pattern recognition methods which can extract characteristic properties, for example geometric or color features, from images. The at least two mappings then search for pairwise identical features, for example the same pixel pattern. After finding pairwise identical features, the three-dimensional position is determined by means of stereoscopic methods using the odometry data. This can be done, for example, with the known structure-from-motion (SfM) method (see, for example, US Pat. RC Bolles et al., Int. Journal of Computer Vision, 1, 7-55 [1987] ).

The control device 4 creates a map of the environment and then enters the three-dimensional position of the feature in the map. The data of the card is stored in memory 6 stored, which of the control device 4 managed and maintained. During the learning journey, further images are repeatedly recorded, features extracted and entered into the map. In this way, a map is produced for the area of the surroundings of the motor vehicle passed through during the learning journey 50 , If the features are extracted from the images and noted with their positions in the map, the images themselves are discarded.

For example, to facilitate a complicated parking operation, several temporally successive positions and possibly additional orientations can be linked to a desired trajectory and in memory 6 be stored.

If the first process section ends with the learning run, then one or more subsequent trips can be carried out. In this case, a desired trajectory previously stored during a learning run can be selected as the active desired trajectory, so that the following travel relates to this active desired trajectory. During the following drive, the camera detects 2 via the image capture device 3 at least one more illustration of the frontal environment. In the at least one further image are then by the control device 4 extracted several features. In each case, the relative orientation, that is to say the directions of the features in the space relative to the camera, becomes one of these features 2 ie to the motor vehicle 50 , determines what in known characteristics of the image capture device 3 , such as the aperture angle, the angle of view, the distortion and the number of pixels of the sensor, on the basis of the positions of the features in the at least one further image is possible. Subsequently, the several features are searched in the map. In most cases, the features identified in the figures are so different that each feature is assigned exactly one position in the map, so that a clear assignment to a position in the map is always ensured during subsequent searches. When finding several features, the accuracy can be increased. After finding the features is a triangulation based on the multiple features and the relative orientations to the motor vehicle 50 possible, so that the motor vehicle 50 can be located in the map. The location information includes, for example, the position and possibly additionally the orientation of the motor vehicle 50 in the card and is over the interface 5 on the data bus 9 of the motor vehicle 50 output.

At the same time or subsequently, the control device 4 use the localization information to determine a difference between an actual trajectory and the previously activated desired trajectory and the driver of the motor vehicle 50 to communicate the difference, for example by taking the difference as coded data over the interface 5 on the data bus 9 of the motor vehicle is output and is displayed from there via a display device of the motor vehicle.

It is also possible that the localization information and possibly additionally the difference from the vehicle control 8th over the data bus 9 be tapped and for automatic or semi-automatic control of the motor vehicle 50 be used. For example, a vehicle controller 8th the localization information and optionally additionally use the difference between the desired and actual trajectory, the steering angle or the direction of movement of the motor vehicle 50 to control or regulate.

An advantage of the invention is the implementation of all steps in the camera 2 which only have a narrowband interface 5 , For example, the usual in motor vehicles CAN bus communicates. Communication via the interface 5 covers only very small amounts of data. This is a simple removal or retrofitting of motor vehicles 50 inexpensive possible.

In the 2a and 2 B schematically the implementation of the image acquisition and the feature extraction during a learning trip in the first process section are shown. The device 1 is installed in a motor vehicle, which is not shown here for reasons of clarity, so that the camera 2 the frontal environment 10 of the motor vehicle within an angle of view 11 detected. Coinciding with capturing the picture 17 receives the control device of the camera 2 Odometry data of vehicle odometry via the interface from the data bus of the motor vehicle. Within the picture angle 11 are schematically a plan view of a building 12 and a carriageway 13 a lane boundary by curbs 14 and a lane marker 15 in the middle of the roadway 13 includes shown. Also shown is a tree 16 at the right edge of the picture angle 11 , The from the camera 2 captured picture 17 is in 2 B shown schematically. There are the building 12 , the roadway 13 and the tree 16 visible in a front view. The control device of the camera 2 then leads in this picture 17 an image analysis by, in the features 18 . 19 . 20 be searched for and identified. For example, here are the corners 21 and edges 22 the graphical representation of the building 12 recognized and with a corresponding feature 18 connected. For the roadway 13 become, for example, the graphical representations of the curbs 14 and the lane marking 15 each with its own characteristics 19 . 20 Mistake. The graphical representation of the tree is not marked with features in this example.

To create a detailed map of the surroundings of the motor vehicle, it is necessary to acquire several images of the surroundings at different positions of the motor vehicle.

The next steps are schematic in the 3a . 3b and 3c shown. In the 3a illustrated illustration 17 with the features 18 . 19 . 20 is identical to the picture 17 of the 2 B , The illustration 17 was recorded at time t = t1. The control device now determines the relative orientation of the features in the next step 18 . 19 . 20 to the motor vehicle.

This is possible because the properties of the image capture device in the camera 2 , So opening angle, angle of view and number of pixels are known. In this way, for each feature is an angle with respect to the center axis of the motor vehicle, which, for example, with the optical axis of the camera 2 determines which one in the picture 17 schematically as one of the imaging center 23 outgoing angle vector 24 for the upper right corner of the building 12 is shown.

The camera 2 detects at a later time t2 another figure 25 the frontal environment, which is schematically in 3b is shown. Since the motor vehicle is moving at a speed v, it has traveled a distance s = v · dt in the time dt = t2-t1, so that the front-side environment is opposite the first image 17 has changed and the building 12 , the roadway 13 and the tree 16 at other positions of the picture 25 are located. The control device of the camera 2 seek and extract features again 18 . 19 . 20 in the picture 25 and determines their relative orientations to the motor vehicle. Since the motor vehicle has moved by the distance s, identical features are in the two figures 16 , Illustration 25 now visible at a different angle to the vehicle, in 3b illustrated by another angle vector 26 for the feature 18 , which is the upper right corner of the building 12 marked. From the different angles or angle vectors 24 . 26 in the two illustration 17 , Illustration 25 The position of the feature can be determined by stereoscopic methods, for example by the known structure-from-motion (Sfm) methods (see e.g. RC Bolles et al., Int. Journal of Computer Vision, 1, 7-55 [1987] ), using the route information.

The image captured in this way from at least two different positions 17 , Illustration 25 certain positions of the characteristics 18 . 19 . 20 are then entered in the next step in a three-dimensional map. A schematic section 27 such a three-dimensional map is in 3c shown. In further steps during the learning run in the first method section, further images can be acquired and further features contained in them can be extracted and analyzed with regard to their position in the same way. The resulting map is stored in the memory of the camera 2 stored. Since in each case the positions of the features in the environment of the motor vehicle relative to the motor vehicle itself are determined, the position of the motor vehicle in the generated three-dimensional map can be determined after a single determination of a reference position, for example, the very first position of the motor vehicle. Several such positions then become a desired trajectory during the learning journey 28 linked, which also schematically in the section 27 the map is shown.

In the second process section, during one or more subsequent trips through the same environment, the motor vehicle can then be located in the map and a comparison between the desired trajectory 28 and an actual trajectory 29 the following trip take place. For this purpose, the camera captures at least one additional image of the environment. From the figure, the controller extracts a plurality of features and determines the relative orientation of the plurality of features to the motor vehicle. Subsequently, the control device searches for the plurality of features in the map and determines after finding by triangulating and inference the actual position and optionally additionally the actual orientation of the motor vehicle. If several such actual positions and possibly also the actual orientations are linked to one another, then the actual trajectory can be determined 29 describe and represent.

The 4a . 4b and 4c schematically illustrate the determination of the three-dimensional position of a feature during the learning run in the first method section. The 4a shows an example of a projection of the generated by an object in the frontal environment of the motor vehicle features in the figure 61 the camera. The camera captures a certain angle, which is assumed to be known for the captured image. Examples become points 75 which align with the corners of the object 60 , correspond to a viewing point 62 projected into the camera, thereby increasing the positions of the graphical representation of the points 75 in the captured picture 61 be determined.

In 4b showing only a single feature for the sake of simplicity, a situation is shown at time t = t1 with the motor vehicle at the viewpoint 62 located. From the viewpoint 62 off, the camera captures a first image 64 on which an example point 76 of the object 60 at a certain position 65 is shown. The position 65 can be described two-dimensionally by the coordinates y _Abb and z _Abb . The graphic representation of the example point 76 at the position 65 is used by the camera as a graphical feature 63 detected, for example, because the corner of the object 60 as a special feature 63 discriminate.

4c shows a situation at a later time t = t2. In the time dt = t2 - t1 has the motor vehicle and thus the viewpoint 62 a route 66 for simplicity's sake, as along the x-coordinate of the coordinate system 68 the map is accepted, and which is determined from the odometry data of the motor vehicle, which are received via the interface of the camera.

The object 60 now appears from the viewpoint 62 seen from another angle, so that the coordinates of the example point 76 in a second map made at time t2 67 have changed. This will be a position 74 also in the second illustration 67 recognized feature 63 , for example, closer to the edge of a left upper corner of the figure 67 discovered.

From the first picture 64 and the second picture 67 can now change the position of the feature 63 in the coordinate system 68 of the card. This is schematic in the 5a and 5b shown. 5a shows the determination of the x and y coordinates. From the first picture 64 can be closed in one direction from the viewpoint using the characteristics of the camera's image acquisition system, such as the viewing angle and the number of pixels. This direction forms a set of xy coordinate tuples in the coordinate system of the map, which is shown schematically as a vector 69 is shown, which starts from the position of the reference point. From the second picture 67 this results in another set of xy-coordinate tuples in the coordinate system of the map, which is also a vector 70 is shown. However, since the vehicle is in the time dt = t2 - t1 by the distance 66 has moved is the vector 70 opposite the vector 69 around the route 66 moved on the x-axis. The path 66 is determined from the odometry data. If one now transfers the sets of xy coordinate tuples into a common graph, an intersection point can be found 71 of the two vectors 69 . 70 determine. The point of intersection 71 then uniquely determines the position with respect to the x and y coordinates of the feature in the coordinate system of the map.

In the same way, the xz-coordinate in the coordinate system of the map is determined. This is schematically in shown. Again let the first 64 and the second 67 Figure two sets of xz coordinate tuples at the two times t1 and t2 determine which as two vectors 69 . 70 are shown. The vector 70 is again about the route 66 opposite the vector 69 postponed. From the intersection 70 of the two vectors 69 . 70 then the xz-coordinate of the feature results in the coordinate system of the map. Together with the xy coordinate, all three location coordinates are thus known, so that a three-dimensional position can be assigned to the feature.

Subsequently, the specific position for the feature is entered and stored in a three-dimensional map. It should be noted that the three-dimensional positions of the features are all determined with respect to the same reference position in the map.

This can be, for example, a very first viewpoint of the motor vehicle. All other positions are then always determined with respect to this very first viewpoint.

The 6a and 6b schematically show the location of the motor vehicle during a subsequent trip in the second process section. It will be at a current actual position 73 of the motor vehicle from the camera a single picture 72 of, for example, front side, environment detected, which schematically in 6a is shown. In this picture 72 features are searched and found, here for example three features 18 passing through the corners of an object 60 be caused. From the properties of the image acquisition system can be the directions with respect to the viewing point 62 determine which schematically as vectors 77 ' . 77 '' . 77 ''' . 78 ' . 78 '' . 78 ''' in a relative coordinate system for the three features 18 ' . 18 '' . 18 ' in 6b are shown. The found features 18 ' . 18 '' . 18 ' are then searched in the map, here they are for example in a section 27 the map, presented in 6c , found. Because the three-dimensional positions of the features 18 ' . 18 '' . 18 ' stored in the map, can be derived from the directions of the characteristics 18 ' . 18 '' . 18 ' with regard to the viewpoint 62 , the actual position 73 and optionally additionally determine the actual orientation of the motor vehicle clearly. When determining the actual position 73 It is also possible to take account of odometry data received, for example, via the data bus from vehicle odometry, for example, when a pitch or roll angle must be corrected. In this way it is possible only with the single picture 72 to locate the motor vehicle in the map.

The 7a . 7b and 7c show several examples of target trajectories, as they are desirable for example in everyday life. The environment in which the motor vehicle 50 is to be located and in which the comparison between a desired trajectory and the actual trajectory arising during driving is to take place is a road 30 from which a driveway 31 branches off and onto a double garage 32a . 32b leads. In 7a are two possible target trajectories 33 . 34 shown which of the current in the 7a position shown 51 of the motor vehicle 50 can be activated, as they are recognized by the camera as lying in the immediate vicinity and the driver as lying in the environment and activated for the current position 51 are displayed. One of the target trajectories 33 then leads directly to the end of the right garage 32a while the other of the target trajectories 34 right to the end of the left garage 32b leads. Thus, the driver receives when approaching in the environment of the target trajectories 33 . 34 a message from the camera that targets trajectories 33 . 34 Are available. The message is preferably output at the interface on the data bus and can be picked up there by a display device and / or acoustic signal generator for further processing. He can now choose one of the target trajectories 33 . 34 select and then receives after the selection when parking in one of the garages 32a . 32b the double garage continuously localization information, such as a difference between one of the desired trajectories 33 . 34 and an actual trajectory of the motor vehicle 50 ,

Another embodiment provides, the motor vehicle 50 with the help of the selected target trajectory 33 . 34 automatically in the double garage 32a . 32b for example, by using the location information to provide data to a vehicle controller.

Two more possible target trajectories 35 . 36 are in 7b shown. In this case, the motor vehicle 50 during the learning journey on the opposite side of the road and came from the other direction. Here are again two target trajectories available, which are recognized by the camera as lying in the immediate vicinity and can be selected by the driver as the active target trajectory.

Two more possible target trajectories 37 . 38 are in 7c shown. This time the car became 50 during the learning journey backwards into each of the garages 32a . 32b the double garage parked. It is particularly advantageous here, if to the positions and optionally additionally the orientations of the motor vehicle 50 also steering angles, directions of movement and / or torques were detected during the learning drive and with the corresponding positions on the target trajectory 37 . 38 were linked. This allows the driver in the form of localization information precise instructions when moving off the target trajectory 37 . 38 to give. An automatic parking assistant also makes this possible for complicated parking and maneuvering maneuvers.

The target trajectories have been shown in different figures only for simplicity and better representability. Of course, embodiments are possible in which more than two desired trajectories are stored and can currently be selected. For example, all desired trajectories could be derived from the 7a and 7c be selectable, so for example, from the position 51 of the motor vehicle 50 a total of four possible target trajectories would be available.

8th schematically shows the comparison between a desired trajectory 40 and an actual trajectory 41 , There are also the concrete target positions 42 or actual positions 43 of the motor vehicle. The trajectories 40 . 41 start from a common starting position 44 at the positions 42 . 43 of desired trajectory 40 and actual trajectory 41 are still identical. The driver now drives off and deviates during the drive from the target trajectory 41 so that there is a difference between the target positions 42 and the current actual positions 43 which gives an example here for a position 47 as a difference vector 45 is shown. Another vector 46 may for example describe the desired orientation of the motor vehicle. The camera is now available for every actual position 43 on the actual trajectory 41 both a localization information of the motor vehicle in the map and the current difference between the target trajectory 40 and actual trajectory 41 at the interface, for example, on the CAN bus of the motor vehicle. The location information and the difference can then be displayed on a display device in the motor vehicle or tapped and processed by the vehicle control. Thus, assistance systems are possible, which support the driver while driving. Also possible are fully automatic systems, the motor vehicle completely autonomous along the target trajectory 40 Taxes.

9 shows a schematic flow diagram of the method for locating a motor vehicle in an environment with training and departure of a desired trajectory. In a first process section, a learning run is performed 100 , This learning trip includes the following steps: Receive odometry data through an interface of the camera 101 , As well as the detection of at least two images of, for example, the front side environment of the motor vehicle with the camera at different positions 102 , In the two figures, features are searched for and extracted 103 , In the at least two figures pairs identical features are then found and their position determined by stereoscopic evaluation using the Odometriedaten 105 , A three-dimensional map is created 106 and the features and the associated three-dimensional positions are entered in the three-dimensional map 107 , Optionally, a desired trajectory from a plurality of positions and, if appropriate, additional orientations can now be stored 108 , Also optionally, steering angles, directions of movement, torques and / or speeds at the corresponding positions and possibly additionally orientations are linked to the desired trajectory and stored 109 ,

In one or more second process sections, one or more subsequent trips are performed 200 , The following drive comprises the following steps: Optionally, a selection of a desired trajectory for a current environment 201 , Subsequently, at least one further image of the environment, for example at the front, is captured with the camera 202 , In the at least one further illustration, several features are determined 203 and the relative orientations of the plurality of features, that is, their directions in space relative to the camera, ie the motor vehicle to which the camera is fixed determined 204 , In the next step, the several features in the map are searched for and found 205 , so that subsequently can be triangulated and can be deduced on a localization information of the motor vehicle in the map 206 , The determined localization information, for example, the actual position and possibly additionally the actual orientation of the motor vehicle in the map, is then output to the interface 207 , Optionally, a difference between the actual and the desired trajectory can be determined and output 208 ,

LIST OF REFERENCE NUMBERS

1

contraption

2

camera

3

Image capture device

4

control device

5

interface

6

Storage

7

Fahrzeugodometrie

8th

vehicle control

9

bus

10

environment

11

angle of view

12

building

13

roadway

14

curbs

15

road marking

16

tree

17

Illustration

18

feature

18 '

feature

18 ''

feature

18 '

feature

19

feature

20

feature

21

corner

22

edge

23

Figure center

24

angle vector

25

further illustration

26

another angle vector

27

Detail of the map

28

Nominal trajectory

29

Actual trajectory

30

Street

31

entrance

32a, 32b

double garage

33

Nominal trajectory

34

Nominal trajectory

35

Nominal trajectory

36

Nominal trajectory

37

Nominal trajectory

38

Nominal trajectory

40

Nominal trajectory

41

Actual trajectory

42

Nominal position

43

Actual position

44

starting position

45

difference vector

46

another vector

47

example position

50

motor vehicle

51

Position of the motor vehicle

60

object

61

Illustration

62

viewpoint

63

feature

64

first picture

65

Position in the picture

66

path

67

second picture

68

Coordinate system of the map

69

Verktor

70

vector

71

intersection

72

single picture

73

Actual position

74

Position in the picture

75

example point

76

example point

77 '

vector

77 ''

vector

77 '' '

vector

78 '

vector

78 ''

vector

78 '' '

vector

100-109

Process steps in the first part of the process

200-208

Process steps in the second process section

QUOTES INCLUDE IN THE DESCRIPTION

This list of the documents listed by the applicant has been generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Cited patent literature

• DE 102008002598 A1 [0007]

Cited non-patent literature

• RC Bolles et al., Int. Journal of Computer Vision, 1, 7-55 [1987] [0023]
• RC Bolles et al., Int. Journal of Computer Vision, 1, 7-55 [1987] [0063]
• RC Bolles et al., Int. Journal of Computer Vision, 1, 7-55 [1987] [0074]

Claims (13)

Method for locating a motor vehicle in an environment, comprising the following steps: in a first method section, performing a learning journey ( 100 ), which comprises: receiving odometry data of the motor vehicle via an interface of a camera ( 101 ), Capture at least two images with the camera while driving at different positions of the camera relative to the environment ( 102 ), with the camera fixed to the motor vehicle, extracting features in the at least two images ( 103 ), Finding pairwise identical features in the at least two figure ( 104 ), stereoscopically determining a three-dimensional position of the features using the odometry data ( 105 ), Creating a three-dimensional map ( 106 ) and entering the features at their three-dimensional positions ( 107 ), then, in one or more second method sections, executing a follow-up run ( 200 ), which comprises: capturing at least one further image with the camera ( 202 ), Determining a plurality of features in the at least one further mapping ( 203 ), Determining relative orientations of the plurality of features to the motor vehicle based on the imaging position ( 204 ), Finding the several features in the map ( 205 ), Triangulating on the basis of the features and the relative orientations and inferring on localization information of the motor vehicle in the map ( 206 ), Output of the localization information at the interface ( 207 ), wherein all process steps of the two process sections are performed in the camera. A method according to claim 1, characterized in that the location information comprises a position and optionally additionally an orientation of the motor vehicle. Method according to one of Claims 1 or 2, characterized in that, during the learning run, a plurality of positions and optionally additionally a plurality of orientations of the motor vehicle are stored in their chronological order as desired positions and possibly additionally desired orientations and linked to a desired trajectory ( 108 ). A method according to claim 3, characterized in that stored during a learning drive steering angles, directions of movement, torques and / or speeds in their temporal order and with the respective positions, which are optionally additionally characterized by orientations, a desired trajectory ( 109 ). A method according to claim 4, characterized in that for a position and optionally additionally an orientation a plurality of desired trajectories are stored by further execution of the first method section and can be selected by the operator as currently active setpoint trajectory ( 201 ). Method according to one of claims 1 to 5, characterized in that a stored for a current environment of the motor vehicle target trajectory is recognized as lying in the environment and the driver is provided for selection as an active target trajectory. Method according to one of claims 1 to 6, characterized in that the localization information comprises a difference between a desired and actual position and optionally additionally an orientation ( 208 ). Method according to one of claims 1 to 7, characterized in that the current localization information is used to limit the area in the map, which is used for a comparison with the features determined from the further image, to a closer environment. Method according to one of claims 1 to 8, characterized in that positions and optionally additionally orientations in the map is assigned a global position coordinate. Contraption ( 1 ) for locating a motor vehicle ( 50 ) in an environment ( 10 ), comprising: a camera ( 2 ) for recording the environment ( 10 ) of the motor vehicle ( 50 ) fixed to the motor vehicle, the camera having a control device ( 4 ) comprising a memory ( 6 ) and an interface ( 5 ), characterized in that the control device ( 4 ) is configured in a first method section during a learning run via the interface ( 5 ) the camera ( 2 ) Odometry data of the motor vehicle ( 50 ) to receive at least two Illustrations ( 64 . 67 ) while driving at different positions of the camera ( 2 ) relative to the environment ( 10 ) with the camera ( 2 ), characteristics ( 18 . 19 . 20 ) in the at least two figures ( 64 . 67 ), in the at least two figures ( 64 . 67 ) pairwise identical characteristics ( 63 ) and stereoscopically determine a three-dimensional position of the features using the odometry data, create a three-dimensional map of the features and their three-dimensional positions, and store them in memory ( 5 ), then, in a second process section during one or more subsequent runs, at least one further image with the camera ( 2 ), several characteristics ( 18 ' . 18 '' . 18 ' ) in the at least one further illustration ( 72 ) to determine the relative orientation of the plurality of features to the motor vehicle ( 50 ) based on the imaging position, the multiple features ( 18 ' . 18 '' . 18 ' ) in the map, the motor vehicle ( 50 ) in the map by means of triangulation and localization information derived therefrom at the interface ( 5 ). Contraption ( 1 ) according to claim 10, characterized in that the camera ( 2 ) is an automotive front camera. Contraption ( 1 ) according to claim 11, characterized in that the automotive front camera behind or in the region of the windshield of the motor vehicle ( 50 ) is arranged. Contraption ( 1 ) according to claim 10, characterized in that the camera ( 2 ) is an automotive rear view camera.

Images