DE102018128535A1 - Training an artificial neural network with data captured by sensors of a vehicle - Google Patents

Abstract

The invention relates to a method for training an artificial neural network, comprising the steps of: providing a vehicle (1) with at least one ultrasound sensor (2) and another sensor (3) with less uncertainty in the distance measurement than the ultrasound sensor (2) , Providing an x, y coordinate system (5) relative to the vehicle (1), emitting an ultrasound signal by the ultrasound sensor (2), receiving an ultrasound signal reflected by an object (6, 7) in the vicinity of the vehicle (1) Ultrasonic sensor (2), determining the transit time of the reflected ultrasonic signal and the distance of the object (6, 7) from the ultrasonic sensor (2), measuring a position of the object (6, 7) by the other sensor (3), providing an artificial neural network , Feeding the artificial neural network with the position of the object (6, 7) measured by the other sensor (3), the transit time of the reflected ul trasound signal, the distance of the object (6, 7) from the ultrasonic sensor (2), the position of the ultrasonic sensor (2) and the orientation of the ultrasonic sensor (2), and outputting data to represent the uncertainties in the position when determining the position of the Object (6, 7) in the vicinity of the vehicle (1) on the basis of the reflected ultrasound signal.

Description

The invention relates to a method for training an artificial neural network with data recorded by sensors of a vehicle, in particular for automatic parking.

Automatic parking is an autonomous vehicle maneuvering system that moves a vehicle from a lane into a parking lot to perform a parallel, vertical, or oblique parking process. The automatic parking system aims to improve the comfort and safety of a driving operation in tight spaces where a lot of attention and experience is required to drive the vehicle. The parking maneuver is implemented through a coordinated control of the steering angle and the speed, taking into account the actual situation in the environment in order to ensure collision-free movement within the available space. The parking maneuver is performed as a result of controlled movements by the vehicle's servo systems using sensor data and distance measurements with respect to the environment. The steering and speed controls are calculated and executed in real time. This approach leads to various path types that are required to perform parking maneuvers.

Sensors used for automatic parking include optical and ultrasonic sensors. Such sensors do not provide precise distance information, but rather distance information with a certain degree of uncertainty. In conventional parking assistance systems, the uncertainty regarding the position of detected objects is generated due to the uncertainty in the sensor model and the systematic error in the system model equations. It is known to use a Kalman filter and some mathematical approaches to merge the data and update the position data and its uncertainty.

It is also known to use machine learning algorithms to find solutions to various problems that arise in the development of driver assistance systems. Artificial neural networks are computing systems that are separated from the biological neural networks, i.e. human brains to be inspired. Such systems learn to perform tasks by looking at examples, generally without being programmed with any task-specific rules, and identifying features from the learning material they process. An artificial neural network is based on a system of connected units or nodes, called artificial neurons, that model the neurons in a biological brain. Each connection can transmit a signal from one artificial neuron to another. An artificial neuron that receives a signal can process it and then signal other connected artificial neurons.

With regard to the estimation of the uncertainty regarding the position of detected objects, there is a hitherto unfulfilled requirement to make this estimation of the uncertainty more reliable.

This object is achieved by the subject matter of the independent claims. Advantageous refinements are described in the subclaims.

• - Bereitstellen eines Fahrzeugs mit mindestens einem Ultraschallsensor und einem anderen Sensor, der bei der Abstandsmessung eine geringere Unsicherheit hat als der Ultraschallsensor,
• - Bereitstellen eines x,y-Koordinatensystems relativ zum Fahrzeug zum Anzeigen einer jeweiligen x-Position und einer jeweiligen y-Position von Einrichtungen des Fahrzeugs oder von Objekten in der Nähe des Fahrzeugs, wobei der Ultraschallsensor eine vordefinierte x-Position, eine vordefinierte y-Position und eine vordefinierte Orientierung im Koordinatensystem hat,
• - Emittieren eines Ultraschallsignals durch den Ultraschallsensor,
• - Empfangen eines Ultraschallsignals, das durch ein Objekt in der Nähe des Fahrzeugs reflektiert worden ist, an dem Ultraschallsensor,
• - Bestimmen der Laufzeit des reflektierten Ultraschallsignals und des Abstands des Objekts vom Ultraschallsensor,
• - Messen einer x-Position und einer y-Position des Objekts im Koordinatensystem durch den anderen Sensor,
• - Bereitstellen eines künstlichen neuronalen Netzwerks mit mindestens sieben Eingangsknoten und mindestens zwei Ausgangsknoten,
• - Speisen der mindestens sieben Eingangsknoten des künstlichen neuronalen Netzwerks mit
  1. a) der durch den anderen Sensor gemessenen x-Position des Objekts,
  2. b) der durch den anderen Sensor gemessenen y-Position des Objekts,
  3. c) der Laufzeit des reflektierten Ultraschallsignals,
  4. d) dem Abstand des Objekts vom Ultraschallsensor,
  5. e) der x-Position des Ultraschallsensors,
  6. f) der y-Position des Ultraschallsensors, und
  7. g) der Orientierung des Ultraschallsensors, und
• - Ausgeben von Daten, die die Unsicherheiten in der x-Position und in der y-Position bei einer Bestimmung der x-Position und der y-Position des Objekts in der Nähe des Fahrzeugs auf der Basis des reflektierten Ultraschallsignals darstellen, durch das künstliche neuronale Netzwerk.

The invention therefore relates to a method for training an artificial neural network, the method comprising the following method steps:

• Providing a vehicle with at least one ultrasonic sensor and another sensor which has less uncertainty in the distance measurement than the ultrasonic sensor,
• Providing an x, y coordinate system relative to the vehicle for displaying a respective x position and a respective y position of devices of the vehicle or of objects in the vicinity of the vehicle, the ultrasound sensor having a predefined x position, a predefined y Position and a predefined orientation in the coordinate system,
• Emitting an ultrasound signal by the ultrasound sensor,
• Receiving an ultrasound signal, which has been reflected by an object in the vicinity of the vehicle, at the ultrasound sensor,
• Determining the transit time of the reflected ultrasound signal and the distance of the object from the ultrasound sensor,
• Measuring an x position and a y position of the object in the coordinate system by the other sensor,
• Providing an artificial neural network with at least seven input nodes and at least two output nodes,
• - Feed the at least seven input nodes of the artificial neural network
  1. a) the x position of the object measured by the other sensor,
  2. b) the y-position of the object measured by the other sensor,
  3. c) the transit time of the reflected ultrasound signal,
  4. d) the distance of the object from the ultrasonic sensor,
  5. e) the x position of the ultrasonic sensor,
  6. f) the y position of the ultrasonic sensor, and
  7. g) the orientation of the ultrasonic sensor, and
• - Output by the artificial neural data representing the uncertainties in the x-position and in the y-position when determining the x-position and the y-position of the object in the vicinity of the vehicle on the basis of the reflected ultrasound signal Network.

It is therefore an essential aspect of the invention that instead of the uncertainty calculated on the basis of the model type of the ultrasonic sensor, the uncertainty is determined by an artificial neural network. Since the uncertainty can change with different situations, this means that the uncertainty is estimated in real time depending on parameters of the real situation, in which the uncertainty can be influenced by other factors besides the model type of the sensor. Due to the use of an artificial neural network, these factors do not need to be identified in detail as they are automatically taken into account during the training. In order to provide a reference for the data generated by an ultrasonic sensor, instead of the previously determined measurement data, data are used which are generated by another sensor which has less uncertainty in real-time distance measurement than the ultrasonic sensor.

While the positions of the ultrasonic sensors and the other sensor are explicitly given by the x and y coordinates, the orientation of the ultrasonic sensors can be given by the axis of their respective ultrasonic emission cone. This axis can be specified as a vector in the predefined x, y coordinate system.

In general, any type of sensor can be used for the other sensor, which has less uncertainty in the distance measurement than the ultrasonic sensor. According to a preferred embodiment of the invention, however, the other sensor is an optical range finder. The optical sensor is even more preferably a LIDAR sensor. Such LIDAR sensors are reliable and easily available and are therefore often already part of vehicles with systems for driver-assisted driving.

The artificial neural network can be provided at various locations, and in particular the artificial neural network can generally also be held in a processor outside the vehicle. According to a preferred embodiment of the invention, however, the vehicle has a processor in which the artificial neural network is held. In this way, reliable use of the artificial neural network can be ensured even if the vehicle has no wireless connection to an external processor with the artificial neural network.

Various types of artificial neural networks can be used for the invention. According to a preferred embodiment of the invention, the artificial neural network is a recurrent neural network. A recurrent neural network is a class of artificial neural networks in which connections between nodes form a directed graph along a sequence. This enables him to show a dynamic temporal behavior for a time sequence. In contrast to neural feedforward networks, recurrent neural networks can use their internal state (memory) to process input sequences. In this way, the movement of the vehicle can be taken into account. It is further preferred if the artificial network uses supervised learning. According to a preferred embodiment of the invention, the artificial network also uses an activation function with hidden ReLU layers with a Tanh output layer. It is also preferred if the artificial network uses AdaGrad for training optimization.

The invention also relates to the use of a method described above in a vehicle traveling on a road.

Furthermore, the invention also relates to a non-volatile computer-readable medium which has instructions stored thereon which, when executed on a processor, cause the method described above to be carried out.

These and other aspects of the invention will become apparent and explained from the embodiments described below. Individual features specified in the embodiments may form an aspect of the present invention by themselves or in combination. Features of the various embodiments can be transferred from one embodiment to another embodiment.

• 1 schematisch ein Fahrzeug 1 zur Verwendung mit dem Verfahren gemäß einer bevorzugten Ausführungsform der Erfindung;
• 2 schematisch eine Darstellung der Umgebung des Fahrzeugs, die durch Abtasten der Umgebung des Fahrzeugs durch einen LIDAR-Sensor des Fahrzeugs empfangen worden ist, gemäß der bevorzugten Ausführungsform der Erfindung;
• 3 schematisch die Kennzeichnung erfasster Objekte mit einem jeweiligen Rahmen und geschlossenen Konturlinien um die jeweiligen Objekte herum, die die Kontur des jeweiligen Objekts beschreiben, gemäß der bevorzugten Ausführungsform der Erfindung; und
• 4 schematisch die Bestimmung der Unsicherheit eines Punkts eines Objekts gemäß der bevorzugten Ausführungsform der Erfindung.

Show it:

• 1 schematically a vehicle 1 for use with the method according to a preferred embodiment of the invention;
• 2nd schematically a representation of the surroundings of the vehicle, which has been received by scanning the surroundings of the vehicle by a LIDAR sensor of the vehicle, according to the preferred embodiment of the invention;
• 3rd schematically the identification of detected objects with a respective frame and closed contour lines around the respective objects, which describe the contour of the respective object, according to the preferred embodiment of the invention; and
• 4th schematically the determination of the uncertainty of a point of an object according to the preferred embodiment of the invention.

The preferred embodiment described herein relates to automatic parking of a vehicle 1 . This automatic parking is based on an autonomous vehicle maneuver during which the vehicle 1 is moved from a lane into a parking lot in order to carry out a parallel, vertical or inclined parking process. This automatic parking improves the comfort and safety of a driving operation in tight spaces, where to steer the vehicle 1 much attention and experience is required. The parking maneuver is implemented through a coordinated control of the steering angle and the speed, taking into account the actual situation in the environment to ensure a collision-free movement within the available space. In the present embodiment, the parking maneuver is performed as a sequence of controlled movements using sensor data from sensors 2nd , 3rd of the vehicle 1 executed, the steering and speed control are calculated and executed in real time. In this regard, automatic parking relates to conventional automatic parking methods known from the prior art.

• Für das Verfahren gemäß einer bevorzugten Ausführungsform der Erfindung weist das Fahrzeug 1 einen Satz Ultraschallsensoren 2 und einem anderen Sensor auf, der eine geringere Unsicherheit bei der Abstandsmessung hat als die Ultraschallsensoren, wie in 1 schematisch dargestellt ist. Als der andere Sensor wird ein LIDAR-Sensor 3 verwendet, der eine Abstandsmessung mit einer sehr geringen Unsicherheit ermöglicht. Ferner weist das Fahrzeug 1 einen Prozessor 4 auf, in dem ein künstliches neuronales Netzwerk vorgehalten wird.

According to the preferred embodiment of the invention described here, ultrasonic sensors 2nd used as main sensors for automatic parking. Such ultrasonic sensors 2nd do not provide exact distance information, but rather distance information with a certain uncertainty. This is due to the uncertainty in the sensor model of the ultrasonic sensors 2nd and attribute the systematic error in the system model equations. According to conventional techniques, a Kalman filter and some mathematical approaches can be used to merge the data and update the position data and its uncertainty. In comparison, the preferred embodiment described herein offers the following improvement:

• For the method according to a preferred embodiment of the invention, the vehicle 1 a set of ultrasonic sensors 2nd and another sensor that has less uncertainty in the distance measurement than the ultrasonic sensors, as in 1 is shown schematically. As the other sensor is a lidar sensor 3rd used, which enables a distance measurement with a very low uncertainty. The vehicle also points 1 a processor 4th in which an artificial neural network is maintained.

An x, y coordinate system 5 is relative to the vehicle 1 provided to a respective x position and a respective y position of devices of the vehicle 1 display, such as the ultrasonic sensors 2nd and the LIDAR sensor 3rd , or objects near the vehicle 1 . In this way it is possible for each of the ultrasonic sensors 2nd a predefined x position, a predefined y position and a predefined orientation in the coordinate system 5 define. The orientation of the ultrasonic sensors 2nd is given by its emission direction, which in 1 is indicated by dashed lines. The LIDAR sensor has a similar design 3rd a predefined x position and a predefined y position in the coordinate system 5 . This is also the relative position of the LIDAR sensor 3rd relative to the ultrasonic sensors 2nd Are defined.

• Ultraschallsignale werden von den Ultraschallsensoren 2 emittiert, und Ultraschallsignale, die von Objekten in der Umgebung des Fahrzeugs 1 reflektiert worden sind, werden durch die jeweiligen Ultraschallsensoren 2 empfangen. Ferner werden die Laufzeit der reflektierten Ultraschallsignale und der Abstand der Objekte von den Ultraschallsensoren 2 bestimmt. Diese Erfassung der Laufzeit der reflektierten Ultraschallsignale und des Abstands der Objekte von den Ultraschallsensoren 2 kann gemäß herkömmlichen Verfahren erfolgen, die dem Fachmann bekannt sind.

According to the preferred embodiment of the invention described here, the method for training the artificial neural network has the following method steps:

• Ultrasonic signals are from the ultrasonic sensors 2nd emits, and ultrasound signals from objects in the area surrounding the vehicle 1 have been reflected by the respective ultrasonic sensors 2nd receive. Furthermore, the transit time of the reflected ultrasonic signals and the distance of the objects from the ultrasonic sensors 2nd certainly. This detection of the transit time of the reflected ultrasound signals and the distance of the objects from the ultrasound sensors 2nd can be done according to conventional methods known to those skilled in the art.

In addition, the LIDAR sensor 3rd an x position and a y position of the objects in the coordinate system 5 measured. This has the advantage of being through the LIDAR sensor 3rd measured data as a reference for determining the uncertainties in the x-position and in the y-position when determining the x-position and the y- Position of objects around the vehicle 1 can be used based on the reflected ultrasound signals, as described below. By determining these uncertainties in the x-position and in the y-position on the basis of the reflected ultrasound signals, a real-time evaluation of the measurement uncertainty is made possible without the factors that cause these uncertainties having to be known in detail, as will be explained below.

The artificial neural network that is in the processor 4th in the vehicle 1 has seven input nodes and two output nodes. This artificial neural network is a deep model that has five additional layers in addition to the input layer and the output layer. The artificial neural network is also a recurrent neural network that enables it to show dynamic temporal behavior for a time sequence around the movement of the vehicle 1 to consider when driving.

1. a) der durch den LIDAR-Sensor 3 gemessenen x-Position des Objekts,
2. b) der durch den LIDAR-Sensor 3 gemessenen y-Position des Objekts,
3. c) der Laufzeit des reflektierten Ultraschallsignals,
4. d) dem Abstand des Objekts vom Ultraschallsensor 2,
5. e) der x-Position des Ultraschallsensors 2,
6. f) der y-Position des Ultraschallsensors 2 und
7. g) der Orientierung des Ultraschallsensors 2.

For every ultrasonic sensor 2nd the seven input nodes of the artificial neural network are fed with:

1. a) by the LIDAR sensor 3rd measured x-position of the object,
2. b) by the LIDAR sensor 3rd measured y-position of the object,
3. c) the transit time of the reflected ultrasound signal,
4. d) the distance of the object from the ultrasonic sensor 2nd ,
5. e) the x position of the ultrasonic sensor 2nd ,
6. f) the y position of the ultrasonic sensor 2nd and
7. g) the orientation of the ultrasonic sensor 2nd .

The artificial neural network uses Kullback-Leibler divergence with the back propagation algorithm to cause the network to learn about the uncertainties in the x-position and in the y-position when determining the x-positions and the y- Positions of objects near the vehicle 1 learns based on the reflected ultrasound signals, as in Blundell, C., Cornebise, J., Kavukcuoglu, K. and Wierstra, D. Weight “Uncertainty in Neural Networks” Proceedings of the 32nd International Conference on Machine Learning, Lille, France, 2015, JMLR: W&CP Volume 37 is described. The artificial network uses supervised learning and an activation function with hidden ReLU layers with a Tanh output layer. AdaGrad is used for training optimization.

• Mit Hilfe des LIDAR-Sensors 3 wird die Umgebung des Fahrzeugs 1 abgetastet, was zu einer Darstellung der Umgebung des Fahrzeugs 1 führt, wie in 2 schematisch dargestellt ist. Diese Darstellung besteht aus Abtastpunkten, die durch die LIDAR-Messung empfangen werden, während das Fahrzeug 1 in dieser Darstellung nur zur Veranschaulichung dargestellt ist. Die Fahrtrichtung des Fahrzeugs 1 ist durch einen Pfeil angegeben, und eine mögliche Trajektorie, entlang der das Fahrzeug 1 fahren soll, ist als gestrichelte Linie dargestellt.

To output data indicating the uncertainties in the x-position and in the y-position when determining the x-positions and the y-positions of objects in the vicinity of the vehicle 1 based on the reflected ultrasound signal, the following procedure is used:

• With the help of the LIDAR sensor 3rd becomes the environment of the vehicle 1 scanned, resulting in a representation of the surroundings of the vehicle 1 leads as in 2nd is shown schematically. This representation consists of sample points that are received by the LIDAR measurement while the vehicle 1 this illustration is shown for illustration only. The direction of travel of the vehicle 1 is indicated by an arrow, and a possible trajectory along which the vehicle 1 should drive is shown as a dashed line.

Objects such as other vehicles become due to the sampling points 6 and road boundaries 7 , visible. As in 3rd is shown, these are objects 6 , 7 through a respective frame 6a , 7a marked and have a closed contour line 6b , 7b around the respective object 6 , 7 around that the contour of each object 6 , 7 describes. While for the road limits 7 the frame 7a has essentially the same shape as the contour line 7b that the contour of the road boundaries 7 describes, the shape of the contour line differs 6b that the contour of the vehicle 6 describes from the frame 6a who the vehicle 6 indicates due to round corners of the vehicle 6 .

Then the uncertainty of a point on an object 6 , 7 based on the smallest distance from the contour lines 6b , 7b based on the assumption that the particular point is on the contour line 6b , 7b should be. This is in 4th shown where the ellipse 8th the uncertainty of the x position and the y position of the point of the object 7 displays in two dimensions while the arch 9 the ultrasound bow of an ultrasound sensor 2nd of the vehicle 1 is. That bow 9 is based on ultrasound data and results from the fact that an ultrasound measurement provides the distance to an object, but no direction to the object. Hence the distance through the arch 9 shown, the center of which is on the ultrasonic sensor 2nd is the radius of the arc 9 by the distance of the object from the sensor 2nd is defined.

In this way, the training data for the artificial neural network are generated. If the artificial neural network has been trained sufficiently as described above, it can be used to estimate the uncertainties in the x-position and in the y-position when determining the x-position and the y-position of an object 6 , 7 in the Near the vehicle 1 can be used in regular operation on the basis of the reflected ultrasound signal. This leads to better accuracy than when this uncertainty is based solely on the model of the ultrasonic sensors 2nd is appreciated.

Reference symbol list

1

vehicle

2nd

Ultrasonic sensors

3rd

LIDAR sensor

4th

processor

5

Coordinate system

6

other vehicles

6a

frame

6b

Contour line

7

Road boundaries

7a

frame

7b

Contour line

8th

ellipse

9

arc

QUOTES INCLUDE IN THE DESCRIPTION

This list of documents listed by the applicant has been generated automatically and is only included for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Non-patent literature cited

• Blundell, C., Cornebise, J., Kavukcuoglu, K. and Wierstra, D. Weight “Uncertainty in Neural Networks” Proceedings of the 32nd International Conference on Machine Learning, Lille, France, 2015, JMLR: W&CP Volume 37 [0024]

Claims (10)

Method for training an artificial neural network, the method comprising the following method steps: - Providing a vehicle (1) with at least one ultrasonic sensor (2) and another sensor (3), which has less uncertainty in the distance measurement than the ultrasonic sensor (2), - Providing an x, y coordinate system (5) relative to the vehicle (1) for displaying a respective x position and a respective y position of devices of the vehicle (1) or objects (6, 7) in the vicinity of the vehicle (1), the ultrasonic sensor (2) having a predefined x position, a predefined y position and a predefined orientation in the coordinate system (5), - emitting an ultrasound signal by the ultrasound sensor (2), Receiving an ultrasound signal, which has been reflected by an object (6, 7) in the vicinity of the vehicle (1), at the ultrasound sensor (2), - determining the transit time of the reflected ultrasound signal and the distance of the object (6, 7) from the ultrasound sensor (2), - Measuring an x position and a y position of the object (6, 7) in the coordinate system (5) by the other sensor (3), Providing an artificial neural network with at least seven input nodes and at least two output nodes, - Feed the at least seven input nodes of the artificial neural network a) the x position of the object (6, 7) measured by the other sensor (3), b) the y position of the object (6, 7) measured by the other sensor (3), c) the transit time of the reflected ultrasound signal, d) the distance of the object (6, 7) from the ultrasonic sensor (2), e) the x position of the ultrasonic sensor (2), f) the y position of the ultrasonic sensor (2), and g) the orientation of the ultrasonic sensor (2), and - Outputting data indicating the uncertainties in the x position and in the y position when determining the x position and the y position of the object (6, 7) in the vicinity of the vehicle (1) on the basis of the reflect reflected ultrasound signal through the artificial neural network. Procedure according to Claim 1 , the other sensor (3) being an optical distance sensor. Procedure according to Claim 2 , wherein the optical sensor (3) is a LIDAR sensor. Method according to one of the preceding claims, with the method step: - Providing a processor (4) in which the artificial neural network is kept in the vehicle (1). Method according to one of the preceding claims, wherein the artificial neural network is a recurrent neural network. A method according to any preceding claim, wherein the artificial network uses supervised learning. Method according to one of the preceding claims, wherein the artificial network uses an activation function with hidden ReLU layers with a Tanh output layer. Method according to one of the preceding claims, wherein the artificial network uses AdaGrad for training optimization. Use of a method according to one of the preceding claims in a vehicle (1) traveling on a road. Non-volatile computer readable medium containing instructions stored thereon which, when executed on a processor, cause a method according to one of the Claims 1 to 9 is performed.

Images