WO2018202552A1

WO2018202552A1 - Method and device for classifying objects in the environment of a motor vehicle

Info

Publication number: WO2018202552A1
Application number: PCT/EP2018/060788
Authority: WO
Inventors: Marc-Michael Meinecke; Michael Heuer; Mikael Johansson; Volker Schomerus; Tom NYSTRÖM
Original assignee: Volkswagen Ag; Scania Cv Ab
Priority date: 2017-05-03
Filing date: 2018-04-26
Publication date: 2018-11-08
Also published as: DE102017207442A1

Abstract

The invention relates to a method for classifying objects in the environment of a motor vehicle, comprising the following method steps: sensing an environment of the motor vehicle by means of at least one radar sensor (2), radar echos (6) of a radar pulse that are caused by object in the environment being sensed; evaluating a radar signal (7) provided by the radar sensor (2) on the basis of the sensed radar echos (6) by means of a radar evaluation apparatus (3), a signal section (8) corresponding to an object being identified in the radar signal (7), and a distance (9), a speed (10) and/or an azimuth angle (11) corresponding to the object being determined; classifying the signal section (8) by means of a classifying apparatus (4), the signal section (8) being classified by means of a deep neural network (12) by assigning at least one object class (13) to the signal section (8), and a class probability measure (14) being determined by means of the deep neural network (12) for each object class (13) assigned to the signal section (8); outputting the at least one assigned object class (13) and the associated class probability measures (14) by means of an outputting apparatus (5). The invention further relates to an associated device (1).

Description

description

Method and device for classifying objects in the environment of a motor vehicle

The invention relates to a method and a device for classifying objects in the environment of a motor vehicle.

Modern motor vehicles are equipped with a variety of sensor systems for environment detection. For example, radar sensors are capable of determining a distance, a speed and an azimuth angle of objects in the surroundings of the motor vehicle. Such radar sensors already form the basis for diverse

Driver assistance systems and can also be used in combination with other sensors.

Furthermore, methods for image analysis based on artificial neural networks are known. Such a process is described for example in EP 1 674 883 A2. A special class of artificial neural network optimization methods are Deep Neural Networks. These deep neural networks provide a more stable learning experience due to an extensive internal structure.

The invention has for its object to provide a method and an apparatus for

To classify objects in the environment of a motor vehicle, in which the classification of the objects is improved.

The object is achieved by a method having the features of patent claim 1 and a device having the features of patent claim 7. Advantageous embodiments will be apparent from the dependent claims.

In particular, a method for classifying objects in the environment of a

Motor vehicle provided, comprising the following method steps: detecting an environment of the motor vehicle by means of at least one radar sensor, being detected by objects in the environment caused radar echoes of a radar pulse; Evaluating one of the radar sensor based on the detected radar returns

provided radar signal by means of a radar evaluation device, one with a Object corresponding signal portion is identified in the radar signal, and wherein a distance corresponding to the object, a speed and / or an azimuth angle are determined; Classifying the signal section by means of a classification device, wherein the signal section is classified by means of a deep neural network by assigning at least one object class to the signal section, and by means of the deep neural network for each of the signal section associated object class

Class probability measure is determined; Outputting the at least one associated object class and the respectively associated class probability measures by means of an output device.

Furthermore, an apparatus for classifying objects in the vicinity of a motor vehicle is provided, comprising at least one radar sensor for detecting an environment of the motor vehicle, wherein the radar sensor caused by objects in the environment

Radar echoes detected radar pulse, a radar evaluation device for evaluating a provided by the radar sensor based on the detected radar echoes

Radar signal, wherein the radar evaluation means is adapted to identify a signal portion corresponding to an object in the radar signal, and to determine a distance corresponding to the object, a speed and / or an azimuth angle; a classifier, wherein the classifier is adapted to classify the signal portion by means of a deep neural network by assigning at least one class of objects to the signal portion, and by means of the deep neural network for each class of the object associated with the signal portion

To determine class likelihood measure; and an output device configured to output the at least one associated object class and the respective associated class probability measures.

The basic idea of the invention is objects in the environment of a motor vehicle by means of a

To detect radar sensor and to evaluate a radar signal provided by the radar sensor in detail. For this purpose, a radar evaluation device determines from the radar signal a distance, a speed and / or a distance corresponding to the object

Azimutwinkel, wherein the distance, the speed and / or the azimuth angle are respectively determined relative to the radar sensor or the motor vehicle. In addition, the radar evaluation device identifies a signal section corresponding to the object in the radar signal. This signal section is fed to a classification device which, by means of a deep neural network, classifies according to various

Makes object classes. The Deep Neural Network also provides a signal section to each or the corresponding object associated object class

Klassenwahrscheinlichkeitsmaß. An object is thus assigned at least one object class with an associated class probability measure.

The class probability measure here forms a confidence value or a

Confidence interval, which is a measure of a plausibility or a reliability of the respective assignment of an object class to an object.

A signal section can be understood here as a radar signature of an associated object. In this case, such a radar signature corresponds to the time profile of the radar echo of the radar pulse caused by the object. Alternatively, the radar signature can be taken from the radar power spectrum. Such a radar signature consists of all of the object reflected by individual radar echo centers

Radar echoes together. Depending on size, shape, surface finish and / or

The surface finish of the object thus results in radar signatures characteristic of different objects. For example, the differences between

Radar signatures of pedestrians clearly from those of a motor vehicle or a

Truck. These differences are recognized by the Deep Neural Network and evaluated for classification.

In the simplest case, the class probability measure indicates with which probability a detected object belongs to a certain object class. Since an assignment of an object class will generally not be unambiguous, the method and the apparatus for a detected object will yield a set consisting of object classes and respective associated probability values. For example, with 90% probability, a detected object can have the object class "motor vehicle" and with 5% each

Probability associated with the object classes "tree" and "truck".

In other words, in this case, the Deep Neural Network provides for each signal portion the probabilities p ,, that this signal portion was caused by a particular object class i of object:

Object class 1

Object class 2

Object class np _n ,

The advantage of the method and the device is that by means of a radar sensor not only information about the distance, the speed and / or the azimuth angle of a

Object in the environment of the motor vehicle can be obtained, but also statements are made about what an object (eg, vehicle, cyclist, pedestrian, tree, etc.) is concerned. This allows a variety of applications, for example, by making this information in addition in driver assistance systems advantage. For example, a parking assistant can detect whether an object in the vicinity of the motor vehicle is another motor vehicle, a curb or a bollard, and make use of this information during automatic parking.

After performing the method, the output device thus provides an assignment to at least one object class as well as a distance, a speed and / or an azimuth angle for each object detected in the surroundings.

In one embodiment, it is provided that an object is detected several times in the environment, evaluated and classified accordingly and results of these classifications are summarized on the basis of the associated class probability measures. This allows the multiple detection of an object, for example, from different distances, perspectives and / or detection angles. In each individual detected radar echo of the object, the signal section corresponding to the object is identified and assigned to it by the Deep Neural Network at least one object class. The class likelihood measures for the different object classes, which are determined after each individual acquisition, are then summarized, that is, for example, summed up and renormalized. This has the advantage that objects can be captured more than once and in this way a more accurate image of the environment is provided.

In the device, it is correspondingly provided that the at least one radar sensor, the radar evaluation device and the classification device are designed to detect, evaluate and classify an object in the environment several times, and to combine results of these classifications on the basis of the associated class probability measures. In some embodiments, it is further provided that, after summarizing the individual results, a decision is made as to which of the object classes from the combined result is assigned to an object. In other words, after summarizing the results, it is decided, for example by maximum value recognition within the various class probability measures of the object classes, which object class is finally selected or uniquely assigned to an object. For example, if the class probability measure is a simple probability with which the individual object classes are present, the object class with the highest probability can be selected as the only and final object class that is assigned to this object.

In a further development, it is further provided that the class probability measure is an evidence value and the summarization by means of the Dempster-Shafer method

is carried out. Evidence is a two-dimensional measure of probability: It is composed of the degree of belief or the degree of trust that the statement of a source applies (English: "degree of belief"), and the plausibility of the event or from a In this embodiment, the Deep Neural Network provides an associated Evidence value to a signal portion then to each of the object classes By means of the Dempster-Shafer method, multiple-detected objects can be classified much better by adding the individual measurements summarized in an overall statement.

In a further embodiment, it is provided that the deep neural network is trained by means of training data of a camera-based reference system. For this purpose, images of the surroundings captured by the camera-based reference system are classified by the reference system and the objects assigned to surrounding objects are assumed to be reference classes ("ground truth") .These reference classes are then used as training data when training the deep neural network to evaluate the radar sensor provided It may be provided here that camera data are previously evaluated manually, so that the reference classes are assigned manually to the corresponding objects.

It may be provided here to use camera data captured during a journey in an unknown environment. Alternatively, however, it can also be provided that camera data is used which has been acquired in an environment with known objects. The corresponding embodiment of the device accordingly provides a camera-based reference system, wherein the deep neural network is adapted to be trained by means of training data provided by the camera-based reference system.

In a further embodiment, it is provided that the signal section corresponding to the object is determined by means of a threshold value detection. For this purpose, it is provided that signal components in the radar signal which are below a certain threshold value are not taken into account. In this way, noise components in the radar signal can be completely suppressed or at least significantly reduced. The consequence is that one

Success rate in assigning an object class to an object can be significantly improved by means of the Deep Neural Network.

The device according to this embodiment provides that the

Radar evaluation device is designed such that to determine the signal portion corresponding to the object by means of a threshold value detection.

In a further development, it is further provided that the threshold value detection is carried out by means of the ordered-constant-false-alarm-rate method (OS-CFAR).

The classifier and the Deep Neural Network can be implemented, for example, by means of a Graphics Processing Unit (GPU or GPGPU).

The device may be formed for example in a motor vehicle. In this way, the classification can be carried out directly in the motor vehicle itself and is available while driving through an environment.

The invention will be explained in more detail with reference to preferred embodiments with reference to the figures. Hereby show:

Fig. 1 is a schematic representation of an embodiment of the device for

Classifying objects in the environment of a motor vehicle;

Fig. 2 is a schematic representation of another embodiment of the device for

Classifying objects in the environment of a motor vehicle; Fig. 3 is a schematic representation of various radar echo centers at a rear of a motor vehicle to illustrate the formation of a radar echo;

4a shows a schematic radar signature of a pedestrian;

4b shows a schematic radar signature of a motor vehicle;

4c shows a schematic radar signature of a truck;

Fig. 5 is a schematic representation for explaining the threshold detection.

FIG. 1 shows a schematic illustration of an embodiment of the device 1 for classifying objects in the environment of a motor vehicle. The device 1 comprises a radar sensor 2, a radar evaluation device 3, a classification device 4 and an output device 5. The radar sensor 2 detects the surroundings of the motor vehicle in which this radar echo 6 of a radar pulse caused by objects in this environment detects. The radar sensor 2 provides a radar signal 7 on the basis of the detected radar echoes 6 and supplies this to the radar evaluation device 3. The

Radar evaluation device 3 is designed such, one with an object

identify corresponding signal section 8 in the radar signal 7 and to determine a distance 9 corresponding to the object, a speed 10 and / or an azimuth angle 1 1. The signal section 8 identified by the radar evaluation device 3 in the radar signal 7 is fed to the classifier 4. The

Classifier 4 is designed to classify the signal section 8 by means of a deep neural network 12 by assigning at least one object class 13 to the signal section 8, and to determine a class probability measure 14 by means of the deep neural network 12 for each object class 13 assigned to the signal section 8. The

Output device 5 gives the associated object class 13 or the associated

Object classes 13 and the associated class probability measures 14. Furthermore, the output device 5 also outputs the distance 9, the speed 10 and / or the

Azimuth angle 1 1 off. The output device 5 can output the abovementioned values, for example, as a data telegram in digital form or as analog voltage values.

It may be provided in some embodiments that an object in the environment is detected several times, evaluated and classified accordingly and results of this

Classifications based on the associated class likelihood measures 14 be summarized. For this purpose, the device 1, for example, in addition to an averaging device (not shown), which summarize the individual

Performs results. Furthermore, the summarizing also by means of

Output device 5 are performed, which for this purpose, for example, a suitably trained averaging module (not shown).

In further embodiments, it may further be provided that the

Class Probability measure is an Evidence value and the summarization is performed by means of the Dempster-Shafer method.

FIG. 2 shows a schematic representation of a further embodiment of the device 1 for classifying objects in the environment of a motor vehicle. The device 1 substantially corresponds to the embodiment shown in FIG. Same

Reference numerals denote the same features. In addition, the embodiment shown in FIG. 2 comprises a camera-based reference system 15, wherein the camera-based reference system 15 comprises a camera 16 and a reference classifier 17. The camera 16 captures images 18 of the surroundings of the motor vehicle and supplies the acquired images 18 to the reference classification device 17. The

Reference classification device 17 recognizes 18 objects in these figures and assigns reference objects 19 to these objects. These reference classes 19 then form training data 20 to train the deep neural network 12 of the classifier 4. The reference classes 19 classified by the reference system 15 and assigned to the objects in the environment are hereby assumed to be "ground truth." It can be provided here that the images 18 are previously evaluated manually, so that an assignment of the reference classes 19 to the corresponding Objects in Figures 18 is done manually.

In Fig. 3 is a schematic representation of the evoked signal strength of

various Radarechozentren 31 at a rear 30 of a motor vehicle for

Clarification of the emergence of a radar echo shown. That from the stern 30

thrown back radar echo of a radar pulse reflected at the rear 30 then sits down together from a superposition of the individual radar echo centers 31. From the superimposed radar echo detected by a radar sensor, the radar signal is subsequently formed.

Depending on the size, shape, surface texture and / or material of an object on which a radar echo is generated, a radar signature of the corresponding object differs. FIGS. 4 a to 4 c show three different radar signatures 32 of different objects, wherein in each case the signal profile of the radar echo of these objects detected by a radar sensor is shown over the time axis. In Fig. 4a is the

Radar signature 32 of a pedestrian shown. FIG. 4b shows the radar signature 32 of a motor vehicle, and FIG. 4c shows the radar signature 32 of a lorry. It can be seen that the different radar signatures 32 are clear

differ from each other. This difference is made use of in the described method and in the device described when assigning object classes to the objects by means of the deep neural network.

FIG. 5 is a schematic diagram for explaining the threshold detection used in some embodiments. In this case, a radar signal 7 via a

Threshold detection evaluated, with a certain threshold 33 is specified. Signal components which are below the threshold value 33 are neglected.

Signal portions above the threshold 33 are cut out, and the cut-out period is supplied as a signal section 8 to the deep neural network as an input vector. In this way, a noise component in the radar signal 7 can be suppressed or at least minimized before classification.

In one embodiment, threshold detection is performed using the Ordered-Statistics Constant-False-Alarm-Rate (OS-CFAR) method.

LIST OF REFERENCE NUMBERS

contraption

radar sensor

Radar evaluation device

classifier

output device

radar echo

radar signal

signal section

distance

speed

azimuth angle

Deep Neural Network

object class

Klassenwahrscheinlichkeitsmaß

Camera-based reference system

camera

Reference classifier

Illustration

reference class

training data

Rear

Radar echo center

radar signature

threshold

Claims

claims

Method for classifying objects in the environment of a motor vehicle, comprising the following method steps:

Detecting an environment of the motor vehicle by means of at least one radar sensor (2), wherein radar returns (6) of a radar pulse caused by objects in the environment are detected;

Evaluate one of the radar sensor

(2) radar signal (7) provided on the basis of the detected radar echoes (6) by means of a radar evaluation device

(3), wherein a signal portion (8) corresponding to an object is identified in the radar signal (7), and wherein a distance (9) corresponding to the object, a

Speed (10) and / or an azimuth angle (1 1) can be determined;

Classifying the signal section (8) by means of a classification device

(4), wherein the signal portion (8) is classified by means of a deep neural network (12) by assigning at least one object class (13) to the signal portion (8), and by means of the deep neural network (12) for each the signal portion (12). 8th)

assigned object class (13) a class probability measure (14) is determined; Outputting the at least one associated object class (13) and the respectively associated class probability measures (14) by means of an output device (5).

A method according to claim 1, characterized in that an object is detected, evaluated and classified several times in the environment and results of this

Classifications based on the associated class likelihood measures (14).

A method according to claim 2, characterized in that the

Class Probability Measure (14) is an Evidence Value and summarization is performed by the Dempster-Shafer method.

Method according to one of the preceding claims, characterized in that the deep neural network (12) by means of training data (20) of a camera-based

Reference system (15) is trained.

5. The method according to any one of the preceding claims, characterized in that the corresponding signal with the object portion (8) is determined by means of a threshold detection.

6. The method according to claim 5, characterized in that the threshold detection is performed by means of the Ordered-Statistics Constant-False-Alarm-Rate-method.

7. Device (1) for classifying objects in the environment of a motor vehicle,

full:

at least one radar sensor (2) for detecting an environment of the motor vehicle, wherein the radar sensor (2) detects radar returns (6) of a radar pulse caused by objects in the surroundings,

a radar evaluation device (3) for evaluating a radar signal (7) provided by the radar sensor (2) on the basis of the detected radar returns (6), wherein the radar evaluation device (3) is designed such that it has a signal section (8) in the radar signal corresponding to an object ( 7), and to determine a distance (9) corresponding to the object, a speed (10) and / or an azimuth angle (1 1);

a classification device (4), wherein the classification device (4) is designed to assign the signal section (8) by means of a deep neural network (12) by assigning at least one object class (13) to the signal section (8)

classify and determine a class probability measure (14) for each object class (13) associated with the signal section (8) by means of the deep neural network (12); an output device (5) which is designed such that the at least one associated object class (13) and the respective associated

To output class likelihood measures (14).

8. Device (1) according to claim 7, characterized in that the at least one radar sensor (2), the radar evaluation device (3) and the

Classifying means (4) are adapted to multiple times to capture, evaluate and classify an object in the environment, and summarize results of these classifications based on the associated class probability measures (14).

9. Device (1) according to one of claims 7 to 8, characterized by a

camera-based reference system (15), the Deep Neural Network (12) being such is designed to be trained by means of training data (20) provided by the camera-based reference system (15).

10. Device (1) according to one of claims 7 to 9, characterized in that the radar evaluation device (3) is designed in such a way, with the object

corresponding signal section (8) to be determined by means of a threshold detection.