DE102016220079B4 - Method for determining distance data - Google Patents

Abstract

Method for determining distance data describing a distance to at least one object captured by a camera (6), at least one image of the object being captured by the camera (6), after which the distance data are determined as a function of the image data of the image, the image data are two-dimensional image data, the distance data being determined exclusively from the image data of exactly one image from the camera (6), the distance data being determined by applying a determination function to the image data or to processing data determined from the image data, the determination function being parameterized by a number of processing parameters , which are determined by the determination function being trained by a plurality of training data sets (1) as part of machine learning, the camera (6) being arranged on a motor vehicle (5), characterized in that depending on the distance data, a vehicle device (8) of the Motor vehicle is controlled to output driving instructions to a driver and / or to carry out a driving intervention, in which case the distance data are merged with data from other information sources or validated by them.

Description

The invention relates to a method according to the preamble of claim 1. The invention also relates to a motor vehicle.

A large number of driver assistance systems can be provided in motor vehicles, which support a driver in driving the motor vehicle by giving him instructions or by intervening in the driving operation. For the functionality of these driver assistance systems and for assisted or automated driving, the importance of which is likely to increase in the future, it is essential to collect information about the vehicle environment that is as detailed as possible. At the same time, this information should be recorded with as little technical effort as possible. Cameras can be used for this purpose, for example, which record the surroundings of the motor vehicle.

The disadvantage of using cameras to capture the surroundings is that conventional cameras do not provide any information about the distance to the captured objects. However, the distance to objects, for example to other road users, can be highly relevant for driver assistance systems or systems for automated driving of motor vehicles.

Various approaches are known for obtaining corresponding distance information. It is thus possible, for example, to supplement the cameras that record the surroundings with additional sensors, for example ultrasonic sensors and/or radar sensors, which can record distances to objects. In addition or as an alternative, it is possible to use image-capturing systems instead of conventional cameras, which at the same time provide distance information, for example imaging laser scanners or time-of-flight cameras. However, if a high angular resolution is to be achieved, the corresponding sensor systems are technically complex.

Another approach to capturing distance information is to use multiple cameras with overlapping recording areas and to determine distances to the captured objects with the help of a stereoscopic system. Such an approach is, for example, in the publication DE 43 08 776 A1 used. Here, a fixed region outside of a vehicle is imaged using a stereoscopic optical system, and distance distributions over the entire image are calculated by an image processing device.

The publication also teaches a similar procedure DE 10 2008 061 749 A1 . In this publication, the surrounding area to be optically recorded is recorded using more than one camera. A stereoscopic evaluation method is used in an overlapping area.

The problem with such stereo-based image processing methods is that the digital cameras used have a limited resolution, with which a sufficiently accurate distance estimation can typically only be achieved for a close range of up to a maximum of 50 meters. If the method is to be used for distance estimation in motor vehicles, a corresponding evaluation is typically only possible up to a maximum speed of 60 to 70 km/h due to the required braking distance. In addition, when using stereo cameras, it is disadvantageous that they require pixel-precise calibration, which can hardly be kept stable over long periods of time due to vibrations and temperature fluctuations.

Another approach to obtaining distance information are so-called "structure from motion" techniques, in which two-dimensional views are recorded in succession over time, from which a 3D model is calculated. The document discloses such an approach, for example DE 10 2009 012 435 A1 . Image sequences are captured with a single rearview fisheye camera and 3D information is obtained using point correspondences between these images. The disadvantage of this approach is that the 3D modulation range is limited to a few tens of meters due to the limited camera resolution. In addition, it is often not possible to determine reliable distance information for moving objects, for example other road users.

Furthermore, a method for determining depth information from two-dimensional images using a neural network is known from the article F. Liu, C. Shen, G. Lin, I. Reid: Learning Depth from Single Monocular Images Using Deep Convolutional Neural Fields. arXiv: 1502.07411 v6 [cs.CV], 25 Nov 2015. Preprint Document Server Cornell University Library.

A method for detecting obstacles for autonomous systems that move at relatively high speeds, in which a distance is determined using a neural network and using an image from a camera, is known from the article M. Mancini, G. Constante, P. Valigi, T.A. Ciarfuglia: Fast Robust Monocular Depth Estimation for Obstacle Detection with Fully Convolutional Networks. arXiv: 1607.06349v1 [cs.RO], 21 Jul 2016. Preprint Document Server Cornell University Library.

The invention is therefore based on the object of specifying a method for determining distance data, which also enables distance estimates for distant and/or moving objects even with simpler sensor technology.

The object is achieved by a method having the features of claim 1.

According to the invention, it is proposed to determine the distance to at least one object captured by the camera using the determination function that was trained by machine learning. For example, a neural network can be used. The processing parameters thus act as a cognitive memory of the trained determination function. A major advantage of using machine learning methods is that it is not necessary to manually specify certain processing rules in order to determine one or more distances from the image data. Machine learning methods, especially neural networks, can achieve results similar to those of the human brain in the context of image processing. It is known here that human observers can also estimate relative and absolute distances to recognized objects from two-dimensional images or when using only one eye. Here, relative sizes, occlusions, shadows and the like are evaluated intuitively.

The invention is based on the idea that a similar “intuition” for a determination function can be trained by machine learning. Corresponding processing parameters, which enable a distance estimation from two-dimensional image data, can be learned automatically within the framework of machine learning. The proposed method for determining distance data can be used in a large number of application areas. However, it can be used particularly advantageously to provide data for driver assistance systems or for systems for carrying out autonomous driving maneuvers in motor vehicles.

The distance data can describe a respective distance to a respective object imaged by the pixel or the group for each pixel of the image data or for a plurality of groups of pixels. A similar distance image can thus ultimately be provided, as would be possible when using more complex sensors, for example laser scanners or time-of-flight cameras. The groups of pixels can in particular include neighboring pixels, for example a rectangular group of pixels.

The training data sets can each include an input image and a distance reference image, with machine learning minimizing a measure of the deviation of processing images, which are determined by applying the determination function to the input image of a respective training data set, from the respective distance reference images. A cost function can be defined as a measure of the deviation between the respective processing image and the distance reference image. This can depend on a number of individual deviations that are calculated for the individual pixels or for groups of these pixels of the individual processing images. The processing image and the reference image preferably have the same image resolution, with which the individual deviations can each be calculated based on pixels. The corresponding individual deviation can indicate the difference between the distance stored for the respective pixel in the distance reference image and the distance determined for this pixel in the processing image. Depending on these individual deviations, the cost function can be calculated in the form of a metric or standard. For example, a 1-norm, a so-called Manhattan norm, or a 2-norm, a so-called Euclidean norm, can be used. With a 1-norm, the individual deviations are added to arrive at the cost function. With a 2-norm, the squares of the individual deviations are summed and the cost function is calculated as the square root of this sum.

The cost function can be minimized by varying the processing parameters. This approach is known from the field of neural networks and is called error feedback or backpropagation of error there. In general, gradient methods can be used, in which a derivation of the cost function or of the measure for the deviation with regard to the processing parameters is calculated in order to determine a direction of variation for the processing parameters. This is a well-known optimization approach that will not be explained in detail.

In particular, training data sets can be used as training data sets that were recorded by an imaging sensor system that supplies additional distance data for each pixel in addition to image data. For example, at least parts of the training data sets can be captured by time-of-flight cameras, with the input image including the respective intensities for all pixels and the distance reference image including the distances. In addition or as an alternative, corresponding training data records can be obtained by a laser scan be provided. It is also possible to record at least parts of the training data sets using combined sensor devices, which include, for example, a radar sensor system with high angular resolution and at least one camera, with the individual pixels of the camera being assigned to individual detection angles of the radar sensor by a calibration process, or the like. In addition or as an alternative, synthetic training datasets can also be used, in which a 3D model is used as the starting point and an input image and a distance reference image for the corresponding 3D scene are calculated from this.

The training of the determination function using such training data sets makes it possible to transfer properties of the relatively complex sensors used to determine the training data sets to a technically very simple determination of the distances using a camera and the determination function. If, as part of the determination of the training data sets, a sensor system is used that enables a very precise distance determination or a distance determination over large ranges, the advantages of this sensor system can also be used at least partially through the training of the determination function if the actual data acquisition is carried out by a relatively simple camera done.

In the context of the invention, it was found that it is advantageous to use a processing algorithm that has a plurality of processing layers arranged in a processing sequence, with at least one sub-function being executed in each of these processing layers, which is parameterized by at least one of the processing parameters, with the at least a partial function of the first processing layer depends on at least a respective part of the image data, wherein the respective at least one partial function of the processing layers following in the processing sequence depends on the at least one processing result of the at least one partial function in the respective immediately preceding processing layer. Each of the processing layers preferably has a plurality of the sub-algorithms. Preferably, at least one of the sub-algorithms, preferably a plurality of sub-algorithms, of the first processing layer depends on each pixel of the image data or the processing data. The sub-algorithms of a last processing layer can each output one or more pixels of the distance data as an output value.

At least parts of the sub-algorithms can be non-linear functions, it being possible in particular to use a weighted sum of a plurality of processing results from the immediately preceding processing layer as the input value of the non-linear function. For example, the processing algorithm may have at least three, five, or seven processing layers. For example, so-called “deep learning” algorithms can be used as the processing algorithm. These use a deep graph with multiple processing layers, each composed of multiple linear or non-linear transformations.

An artificial neural network, for example a convolutional neural network, is preferably used as the processing algorithm. A corresponding neural network is preferably not used in the method according to the invention in order to determine a single classification result, but instead a large number of output values are generated, namely different distances for different pixels or image areas. The neural network thus acts as a kind of filter on the image data or the processing data in order to generate a result image.

Different types of machine learning use different types of knowledge representation. In so-called symbolic systems, the knowledge learned is explicitly recognizable in the form of rules. In the case of sub-symbolic systems, for example in the case of neural networks, a specific behavior of the determination function is trained, but the corresponding knowledge in the form of the processing parameters is usually not available in such a way that it can be easily interpreted. This means, among other things, that once a determination function has been trained, it always requires data in the same input format. It is therefore possible in the method according to the invention not to apply the determination function directly to image data, but rather to processing data that were determined from them. In this way, for example, an abstraction layer can be implemented in order to use a determination function that has been trained once, ie a specific set of processing parameters, for different cameras, for example black-and-white and color cameras and/or cameras with different image resolutions. The pre-processing for processing data can only be used for format conversion, but in principle more complex pre-processing, for example vectorization of the image data, is also possible.

The method according to the invention provides for the image data to be two-dimensional image data, with the distance data being determined exclusively from the image data of the camera. According to the invention, the distance data are determined exclusively from image data of exactly one image from the camera. The inventive use of a determination function parameterized by machine learning can thus determine distances to objects very quickly and with simple sensors.

The method according to the invention can be used with particular advantage in motor vehicles. It is provided according to the invention that the camera is arranged on a motor vehicle, with a vehicle device of the motor vehicle being controlled to output driving instructions to a driver and/or to carry out a driving intervention depending on the distance data. In particular, the vehicle device can be activated exclusively when a distance condition that evaluates the distance data is met. In addition, it is possible for the activation to depend on an object classification of the objects detected in the motor vehicle environment.

However, it is also possible to control at least one vehicle device in at least one operating mode of the motor vehicle continuously or repeatedly in a way that depends on the distance data. For example, a distance to specific objects in the area surrounding the motor vehicle can be output to a driver optically or acoustically, longitudinal guidance, for example by a distance assistant, can depend on the distance data or the like. In this case, it is possible for the vehicle device to be activated solely as a function of the distance data. According to the invention, however, the distance data is merged with data from other information sources or validated by them. Other sensors of the motor vehicle and/or data received via radio connections, for example car-2-car communication, can serve as sources of information.

In addition to the method according to the invention, the invention relates to a motor vehicle with a camera and a processing device, with the processing device being able to evaluate image data from the camera in accordance with the method according to the invention, with a vehicle device of the motor vehicle for outputting driving instructions to a driver and/or for Carrying out a driving intervention can be controlled. The processing device can also implement one or more driver assistance systems and/or be designed for autonomous guidance of the motor vehicle. For this purpose, at least one vehicle device of the motor vehicle can be controlled by the processing device, for example a display device, an output device for acoustic information and/or an actuator for driving interventions.

• 1 ein Ablaufdiagramm eines Ausführungsbeispiels des erfindungsgemäßen Verfahrens, und
• 2 ein Ausführungsbeispiel eines erfindungsgemäßen Kraftfahrzeugs.

Further advantages and details of the invention result from the following exemplary embodiments and the associated drawings. Here show:

• 1 a flowchart of an embodiment of the method according to the invention, and
• 2 an embodiment of a motor vehicle according to the invention.

1 shows a flowchart of a method for determining distance data describing a distance to at least one object captured by a camera. In the method, image data of an image are captured by a camera. This is done by a camera 6, which as in 2 is shown, is arranged on a motor vehicle 5 . By using a determination function, distance data are determined, which preferably describe a distance of the detected object for each pixel. The determination function is parameterized by a large number of processing parameters, which were learned by a machine learning method using training data sets.

This in 1 The process presented thus ultimately breaks down into two process sections. In the first method step from step S1 to step S7, the determination function is parameterized and the correspondingly parameterized determination function or the parameters for an already existing determination function are stored in the processing device 7 of the motor vehicle 5 . Steps S8 to S10, in which this parameterized determination function can be used to determine the distance as part of the driving operation of the motor vehicle 5, can then be carried out in the motor vehicle 5.

In order to train the determination function, a large number of training data records 1 are initially provided in step S1. These each include an input image 2 and a distance reference image 3. The training data sets 1 can be determined in a variety of ways. For example, a time-of-flight camera can be used, which provides information about a detected light intensity and about a light propagation time for a large number of pixels. The light intensities can be used as image data of the input image 2. The transit times or distances determined from the transit times can be used as image data of the distance reference image 3 .

Alternatively or additionally, training data records 1 can also be recorded, for example by imaging laser scanners. The procedure here corresponds to the procedure for a time-of-flight camera. It is also possible to acquire at least parts of the training data sets 1 using sensor devices that fuse sensor data from different sensors in order to provide the input image and the distance reference image. For example, the range data of the range reference image can be detected by an angle-resolving radar detector. In addition, a camera can be used whose detection area is calibrated relative to the detection area of the radar detector in order to determine the associated input image. In this case, too, individual pixels of the input image can be assigned defined distances of the distance reference image.

Ultimately, it is also possible to generate at least parts of the training data sets 1 synthetically. For this purpose, 3D models of a specific environment can be provided, for example, and at least one virtual data acquisition can be carried out in this model, by means of which image data for the input image and distances for the distance reference image can be acquired.

In step S2, the determination function is applied to the input image 2 for each of the training data sets 1. The determination function can be, for example, a neural network or statistical data modeling that is to be trained. Essentially any initial values for the processing parameters of the determination function can be used at the beginning of the training. Iteratively adjusted processing parameters are used in later iterations, which adjustment will be discussed later.

By applying the determination function to the input images, a respective processing image 4 is available for each of the training data sets 1 in step S3. This is compared in step S4 with the distance reference image 3 of the associated training data set 1 in order to determine a measure of the differences between the respective processing image 4 and the respective distance reference image 3, ie a cost function. In this case, the measure of the difference can evaluate differences in the distance for the respective pixels as difference variables. The various difference sizes can be viewed as deviations for which a measure of the total deviation is calculated. This can take the form of a 1-norm, also known as the Manhattan norm, i.e. a sum of the differences, or a 2-norm, also known as the Euclidean norm, in which a square root of the sum of the squares of the differences is calculated. In particular, the summation takes place over the pixels of all training data sets.

In step S5 it is determined whether the parameter optimization is converging. For this purpose, it can be checked, for example, whether the difference between the cost functions determined in successive runs in step S4 falls below a limit value. If this is not the case, the processing parameters are adjusted in step S6 and steps S2 to S5 are then repeated with the adjusted processing parameters.

A gradient method can be used to adapt the processing parameters. For this purpose, a derivation of the cost function calculated in step S4 can be calculated according to the processing parameters in order to determine a direction of variation for which the cost function decreases, preferably the direction of the steepest descent.

If it was determined in step S5 that the optimization method has already converged, the processing parameters last used in step S2 are defined in step S7 as learned or final processing parameters. If the processing algorithm is to be used in a large number of motor vehicles, for example, steps S1 to S6 can be carried out jointly at least for all those motor vehicles that have cameras 6 that can be used interchangeably. In step S7, the determination function for all of these motor vehicles can be parameterized by the determined processing parameters. The correspondingly parameterized determination function is used in steps S8 to S10. These are described below for an example in which these steps are carried out in motor vehicle 5 . Of course, other applications are also possible.

In step S7, an image of the area around the front of the motor vehicle 5 is captured by the camera 6 of the motor vehicle 5 . In step S9, the determination function parameterized by the previously determined processing parameters is then applied to the image data of this image. In this way, a distance image is calculated which, as distance data for each pixel of the original image data, describes a respective distance to a respective object imaged by this pixel. Since the determination function was specifically trained by the training data sets 1 to determine distances from image data from a camera, corresponding distances can be determined with a high level of robustness and accuracy given sufficient training. In further developments of the method explained, it would be possible to verify the distances determined by the determination function using sensor data recorded via additional sensors of motor vehicle 5 or other data received from motor vehicle 5 .

The distance data is then further processed in step S10, for example to determine distances and/or movements relative to other road users, to generate a three-dimensional environment model for motor vehicle 5 or the like. Depending on the processing result, a vehicle device 8 is activated in order to give the driver driving instructions and/or to intervene in the driving operation. For example, distance information to other road users can be displayed on a display device, in particular in the form of a head-up display, or at least one actuator can be controlled as vehicle device 8 in order to accelerate or brake motor vehicle 5 as part of automatic distance control.

Claims (5)

Method for determining distance data describing a distance to at least one object captured by a camera (6), at least one image of the object being captured by the camera (6), after which the distance data are determined as a function of the image data of the image, the image data are two-dimensional image data, the distance data being determined exclusively from the image data of exactly one image from the camera (6), the distance data being determined by applying a determination function to the image data or to processing data determined from the image data, the determination function being parameterized by a number of processing parameters that are determined by the determination function being trained by a plurality of training data sets (1) as part of machine learning, the camera (6) being arranged on a motor vehicle (5), characterized in that depending on the distance data, a vehicle device (8) of the Motor vehicle is controlled to output driving instructions to a driver and / or to carry out a driving intervention, in which case the distance data are merged with data from other information sources or validated by them. procedure after claim 1 , characterized in that the distance data for each pixel of the image data or for a plurality of groups of pixels describes a respective distance to a respective object imaged by the pixel or the group. procedure after claim 1 or 2 , characterized in that the training data sets (1) each comprise an input image (2) and a distance reference image (3), with a measure for the deviation of processing images (4), which are generated by applying the determination function to the input image (2nd ) of a respective training data set (1) are determined, is minimized by the respective distance reference images (3). Method according to one of the preceding claims, characterized in that the determination function has a plurality of processing layers arranged in a processing sequence, in each of these processing layers at least one sub-function is executed, which is parameterized by at least one of the processing parameters, the at least one sub-function of the first processing layer depends on at least a respective part of the image data, wherein the respective at least one partial function of the processing layers following in the processing sequence depends on the at least one processing result of the at least one partial function in the respective immediately preceding processing layer. Motor vehicle (5) with a camera (6) and a processing device (7), characterized in that the processing device (7) can evaluate image data from the camera (6) according to the method according to one of the preceding claims, with a Vehicle device (8) of the motor vehicle (5) can be controlled to issue driving instructions to a driver and/or to carry out a driving intervention, the distance data being fused with data from other information sources or validated by them.

Images