DE102020203828A1 - Method for recognizing the surroundings of a vehicle - Google Patents

Abstract

The present invention relates to a method (400) for detecting the surroundings of a vehicle (100), the vehicle (100) having a sensor system with at least two sensor units (104, 106, 108) for detecting the surroundings of the vehicle (100) and an evaluation unit (110) for evaluating sensor data (112) of the sensor system. The method (400) comprises the following steps: receiving (410) sensor data (112) generated by the at least two sensor units (104, 106, 108) in the evaluation unit (110); Inputting (420) the sensor data (112) into a fusion algorithm which is configured to determine probability distributions with respect to features of objects (113) in the vicinity of the vehicle (100) based on the sensor data (112) and based on the probability distributions to generate and output first object hypotheses (204) assigned to the objects (113); Inputting (420) the sensor data (112) into a machine learning algorithm which has been trained to generate and output second object hypotheses (206) assigned to the objects (113) based on the sensor data (112); Deciding (430) whether or not to reject the first object hypotheses (204); if the first object hypotheses (204) are not to be discarded: using (440) the first object hypotheses (204) to update an environment model representing the environment of the vehicle (100); if the first object hypotheses (204) are to be rejected: using (450) the second object hypotheses (206) to update the environment model.

Description

Field of invention

The invention relates to a method for recognizing the surroundings of a vehicle. The invention also relates to an evaluation unit, a computer program and a computer-readable medium for carrying out the method, as well as a corresponding vehicle system.

State of the art

In order to identify the surroundings of a vehicle, sensor data from different sensors of the vehicle can be merged by means of suitable algorithms in a common representation of the surroundings, which is also referred to as sensor data fusion. The aim of the sensor data fusion is to merge the respective sensor data in such a way that respective strengths of the sensors are profitably combined with one another or respective weaknesses of the sensors are reduced. In such a sensor data fusion, for example, a Kalman filter can be used, the detection performance of which can be restricted in certain situations.

Disclosure of the invention

Against this background, the approach presented here presents a method, an evaluation unit, a computer program and a computer-readable medium according to the independent claims. Advantageous developments and improvements of the approach presented here emerge from the description and are described in the dependent claims.

Advantages of the invention

Embodiments of the present invention advantageously make it possible to increase the robustness of a sensor data fusion by combining a fusion algorithm based on a probabilistic model with a machine learning algorithm for pattern recognition. In this way, for example, false positive or false negative recognition results can be avoided.

Another advantage is that the machine learning algorithm can only be used, for example, when the detection performance of the fusion algorithm reaches its limits, i.e. that is, if this produces uncertain, ambiguous or contradicting results. In this case, the machine learning algorithm only needs to be trained with a comparatively small set of training data.

A first aspect of the invention relates to a computer-implemented method for recognizing the surroundings of a vehicle, the vehicle having a sensor system with at least two sensor units for detecting the surroundings of the vehicle and an evaluation unit for evaluating sensor data from the sensor system. The method comprises the following steps, which can in particular be carried out in the specified sequence: receiving sensor data generated by the at least two sensor units in the evaluation unit; Inputting the sensor data into a fusion algorithm which is configured to determine probability distributions with respect to features of objects in the vicinity of the vehicle based on the sensor data and to generate and output first object hypotheses assigned to the objects based on the probability distributions; Inputting the sensor data into a machine learning algorithm which has been trained to generate and output second object hypotheses assigned to the objects based on the sensor data; Deciding whether or not to reject the first object hypotheses; if the first object hypotheses are not to be discarded: using the first object hypotheses to update an environment model representing the environment of the vehicle; if the first object hypotheses are to be rejected: using the second object hypotheses to update the environment model.

A vehicle can generally be understood as a partially or fully automated moving machine. For example, the vehicle can be a car, truck, bus, motorcycle, robot or the like.

The vehicle can include a vehicle system that is designed to control the vehicle in a partially or fully automated manner. For this purpose, the vehicle system can control corresponding actuators in the vehicle, such as steering or brake actuators or an engine control unit.

A sensor unit can be, for example, a radar, lidar, ultrasonic sensor or a camera. The sensor system can include similar (for example redundant) or different (for example complementary) sensor units. For example, a combination of a radar sensor with a camera or a combination of several radar sensors with different detection directions is conceivable. For example, the sensor system can also include three or more sensor units or sensor types.

The sensor data can recognize features such as position, speed, acceleration, expansion by the respective sensor unit or object category of objects in the vicinity of the vehicle. The features can have been extracted from raw data of the respective sensor unit.

The sensor data can be merged with one another to identify the surroundings of the vehicle. Such a sensor data fusion can be understood as a process in which information from different sensor units or sensor types is used to detect and classify objects in the vicinity of the vehicle (also called object discrimination) and to estimate the respective states of the objects, i.e. H. predict with a certain probability (also called track estimation).

An object hypothesis can be understood to be a model of a real object located in the vicinity of the vehicle, for example an observed vehicle, a road marking, a pedestrian or the like. The object hypotheses can be stored in an environment model representing the environment of the vehicle. The environment model can be continuously updated based on the sensor data by comparing the predicted states of the objects with current measurements.

The fusion algorithm can be, for example, a Bayesian filter, Kalman filter or particle filter or a combination of at least two of the examples mentioned. For example, the fusion algorithm can be configured to generate a plurality of assignment matrices that describe and weight possible assignments between features, more precisely feature hypotheses, and object hypotheses, and to select a suitable assignment matrix for updating the environment model from these assignment matrices (also called multi-hypothesis approach).

The machine learning algorithm can include a classifier into which the sensor data can be entered in order to assign features of the objects to specific object classes.

The machine learning algorithm can be an artificial neural network such as a multilayer perceptron, a recurrent neural network, a neural network with long short-term memory or a convolutional neural network. However, it can also be a Bayesian classifier, a support vector machine, a k-nearest neighbor algorithm, a decision tree, a random forest or a combination of at least two of the examples mentioned.

For example, the machine learning algorithm can only be activated if the fusion algorithm delivers uncertain, ambiguous or contradicting results.

In other words, the fusion algorithm can be used for a large number of situations to be recognized, while the machine learning algorithm can only be used in special cases, for example in situations that the fusion algorithm cannot clearly recognize. This has the advantage that the machine learning algorithm can be trained with relatively little effort. In addition, the system can be easily expanded as a result.

It is possible for the machine learning algorithm to influence the fusion algorithm in a corresponding manner, for example by selecting or changing object hypotheses that are generated by the fusion algorithm, or by adding new object hypotheses.

It is also possible for the sensor data to be entered into at least two different machine learning algorithms. The at least two machine learning algorithms can have been trained with training data that differ from one another.

Whether or not the first object hypotheses should be discarded can be determined, for example, on the basis of the respective trustworthiness of the first object hypotheses. The trustworthiness can be determined as a function of statistical parameters such as, for example, a covariance, a level of confidence or a (average) probability of existence, or also on whether the sensor units provide results that match or differ from one another. In principle, the trustworthiness can be understood as a measure of a match between the respective first object hypothesis and a real object.

For example, the first object hypotheses can be compared with the second object hypotheses for this purpose.

If the first object hypotheses are to be discarded, the second object hypotheses can be used, in particular with the exclusion of the first object hypotheses, to update the environment model. Conversely, it is possible for the first object hypotheses to be used to update the environment model, excluding the second object hypotheses, if the first object hypotheses are not to be discarded.

A second aspect of the invention relates to an evaluation unit which is configured to carry out the method as described above and below. Features of this method can also be features of the evaluation unit and vice versa.

A third aspect of the invention relates to a vehicle system that is configured to carry out the method as described above and below. Features of this method can also be features of the vehicle system and vice versa.

Further aspects of the invention relate to a computer program which, when executed by a computer such as the evaluation unit mentioned above, executes the method as described above and below, and a computer-readable medium on which such a Computer program is stored.

The computer readable medium can be volatile or non-volatile data storage. For example, the computer-readable medium can be a hard disk, a USB storage device, a RAM, ROM, EPROM or flash memory. The computer-readable medium can also be a data communication network that enables a program code to be downloaded, such as the Internet or a data cloud.

Features of the method, as described above and below, can also be features of the computer program and / or the computer-readable medium, and vice versa.

Ideas for embodiments of the present invention can be viewed, inter alia, as being based on the thoughts and findings described below.

According to one embodiment, at least one further object hypothesis is generated based on the first object hypotheses and the second object hypotheses. The at least one further object hypothesis is used to update the environment model.

In other words, the respective outputs of the fusion algorithm and of the machine learning algorithm can be merged with one another in order to provide suitable update data for updating the environment model. The at least one further object hypothesis can be generated, for example, by a further fusion algorithm and / or a further machine learning algorithm.

The at least one further object hypothesis can then be used, for example, to update the environment model if the first object hypotheses are to be discarded. In particular, the at least one further object hypothesis can be used to update the environment model, excluding the first object hypotheses.

According to one embodiment, it is determined by comparing the first object hypotheses with the second object hypotheses whether the first object hypotheses are false or not. If the first object hypotheses are wrong, it is decided that the first object hypotheses should be rejected. Additionally or alternatively, it is decided that the first object hypotheses should not be discarded if the first object hypotheses are not false.

For example, it can be determined on the basis of the second object hypotheses whether the first object hypotheses represent a false positive or false negative result.

According to one embodiment, the fusion algorithm is configured to weight the first object hypotheses. In this case, a weighting of at least one of the first object hypotheses is changed based on the second object hypotheses.

In other words, the second object hypotheses can be used to influence the fusion algorithm in a suitable manner. This can improve the detection accuracy of the fusion algorithm.

According to one embodiment, the machine learning algorithm was trained with training data which at least for the most part comprise features of objects that are difficult to recognize for the fusion algorithm.

In other words, the machine learning algorithm may have been trained to only recognize a certain type of situation in which the vehicle can find itself. In particular, these can be situations in which the fusion of the sensor data by means of the fusion algorithm would produce an uncertain, ambiguous or contradicting result. A possible example is a manhole cover from which steam rises. In this case, for example, the manhole cover could be falsely recognized as an object (potentially dangerous for the vehicle) based on the sensor data of a radar sensor, the radar beams of which are reflected by the manhole cover (false positive recognition). Using the sensor data of a lidar sensor, the laser beams of which are reflected by the steam, could the existence of the object can also be deduced. On the other hand, the sensor data of a camera could correctly identify that no object (potentially dangerous for the vehicle) is present. The contradicting detections can lead to the fusion algorithm not correctly detecting the situation.

According to one embodiment, the fusion algorithm comprises a Kalman filter. Additionally or alternatively, the machine learning algorithm includes a multilayer perceptron, for example en convolutional neural network or the like.

Figure list

• 1 zeigt ein Fahrzeug mit einem Fahrzeugsystem gemäß einem Ausführungsbeispiel der Erfindung.
• 2 zeigt ein Blockschaltbild einer Auswerteeinheit aus 1.
• 3 zeigt ein Blockschaltbild einer Auswerteeinheit gemäß einem weiteren Ausführungsbeispiel der Erfindung.
• 4 zeigt ein Ablaufdiagramm eines Verfahrens gemäß einem Ausführungsbeispiel der Erfindung.

Embodiments of the invention are described below with reference to the accompanying drawings, wherein neither the drawings nor the description are to be interpreted as restricting the invention.

• 1 shows a vehicle with a vehicle system according to an embodiment of the invention.
• 2 shows a block diagram of an evaluation unit from 1 .
• 3 shows a block diagram of an evaluation unit according to a further embodiment of the invention.
• 4th shows a flow chart of a method according to an embodiment of the invention.

The figures are only schematic and not true to scale. In the figures, the same reference symbols denote the same or equivalent features.

Embodiments of the invention

1 shows a vehicle 100 with a vehicle system 102 , a sensor system with a first sensor unit 104 , here a radar sensor, a second sensor unit 106 , here a lidar sensor, and a third sensor unit 108 , here a camera, for capturing objects in the vicinity of the vehicle 100 as well as an evaluation unit 110 for evaluating the respective sensor data 112 of the three sensor units 104 , 106 , 108 includes. The sensors in 1 a vehicle in front 113 .

In addition, the vehicle system 102 an actuator system 114 include, for example, steering or brake actuators or an engine control of the vehicle 100 . The evaluation unit 110 can the actuators 114 based on the sensor data 112 drive in a suitable manner, for example around the vehicle 100 fully automated control.

To the vehicle in front 113 to be recognized as an object, the sensor data 112 of the various sensor units 104 , 106 , 108 in the evaluation unit 110 suitably fused together. The recognized object is saved in an environment model and based on the sensor data 112 continuously updated, which is also known as tracking. Here, future states of the recognized object are estimated in the environment model in each time step and with the respective current sensor data 112 matched.

2 shows the evaluation unit 110 the end 1 as a block diagram. The modules described below can each be hardware and / or software modules.

The evaluation unit 110 includes a fusion module 200 and a machine learning module 202 , in each of which the sensor data 112 of the three sensor units 104 , 106 , 108 can be entered. The fusion module 200 executes a fusion algorithm, such as a Kalman filter, based on the sensor data 112 Probability distributions in relation to features of objects in the vicinity of the vehicle 100 , for example in relation to the vehicle in front 113 , determines and depending on the first object hypotheses 204 generated. One of the first object hypotheses 204 represents, for example, the vehicle in front 113 . Depending on the situation, the first object hypotheses 204 be more or less trustworthy.

For example, the fusion algorithm can determine several alternative assignment variants in the form of assignment matrices, with exactly one object hypothesis being assigned to a feature, more precisely a feature hypothesis, in each assignment matrix, and each of these assignments being weighted. From the set of assignment matrices, depending on the weighting, a suitable assignment matrix for updating the environment model can now be selected.

The environment model can, for example, be in a planner module 208 be implemented that is a trajectory of the vehicle 100 based on digital map data and a current geographical position of the vehicle 100 plans taking into account the environmental model.

The features can, for example, as part of a preprocessing of raw sensor data by the respective sensor unit 104 , 106 respectively. 108 have been extracted and thus already in the sensor data 112 be included. Alternatively or additionally, the preprocessing of the Raw sensor data through the fusion unit 200 and / or the machine learning module 202 take place.

The machine learning module 202 executes a machine learning algorithm, such as a convolutional neural network or a support vector machine. The machine learning algorithm was trained to identify certain features or combinations of features that are contained in the sensor data 112 are included to be assigned to certain object classes, here among other things an object class "vehicle in front". Based on this assignment, the machine learning algorithm generates corresponding second object hypotheses 206 each with a detected object in the vicinity of the vehicle 100 represent. One of the second object hypotheses 206 represents, for example, the vehicle in front 113 .

A monitoring module 210 is configured to be a result of the fusion module 200 to evaluate the executed sensor data fusion with regard to its trustworthiness. For example, the result can be classified as not very trustworthy if it is uncertain, ie has a low covariance or a low level of confidence, is ambiguous, ie if at least one of the sensor units 104 , 106 , 108 delivers a result that differs from the other two sensor units, or an (average) probability of existence with regard to the detected objects is too low.

It is possible that the trustworthiness is additionally or alternatively in the machine learning module 202 Is evaluated. The machine learning module 202 In this case, a corresponding evaluation information can be sent to the monitoring module 210 send.

The first object hypotheses 204 and the second object hypotheses 206 can also be used in a further fusion module 212 can be entered, which executes a further fusion algorithm, such as another Kalman filter. The further fusion algorithm is configured by fusing the first object hypotheses 204 with the second object hypotheses 206 one or more further object hypotheses 214 to create. The other object hypotheses 214 can be more trustworthy than the object hypotheses 204 , 206 exhibit. One of the other object hypotheses 214 can for example be the vehicle in front 113 represent.

Provides the monitoring module 210 If the trustworthiness is too low, the first object hypotheses become 204 discarded. In this case the monitoring module prevents 210 that the first object hypotheses 204 to the planner module 208 are forwarded, and causes the further object hypotheses instead 214 into the planner module 208 can be entered. Alternatively, the monitoring module initiates 210 that instead of the first object hypotheses 204 the second object hypotheses 206 into the planner module 208 can be entered as it is in 3 is shown.

In addition, the monitoring module 210 information about its respective decision to a security module 216 send. The security module 216 can, for example, depend on a decision by the monitoring module 210 the planner module 208 influence, for example, to the extent that if the trustworthiness is too low, a trajectory of the vehicle 100 is changed accordingly or the vehicle 100 braked or stopped.

For example, the monitoring module 210 the first object hypotheses 204 and the second object hypotheses 206 Compare with each other to see if they are the first object hypotheses 204 whether it is false positives or false negatives. If a false positive or false negative detection is determined, the corresponding first object hypotheses are made 204 discarded. Otherwise the first object hypotheses 204 to the planner module 208 forwarded.

The machine learning module 202 can be configured, for example, to include weightings of individual ones of the first object hypotheses 204 to change and thus to increase the trustworthiness of the sensor data fusion.

3 shows a possible variant of the evaluation unit 110 without the further fusion module 212 . Either the first object hypotheses 204 or the second object hypotheses 206 into the planner module 208 can be entered.

The decision as to whether the machine learning algorithm should be used or not can be made, for example, by another machine learning algorithm. This can, for example, in the monitoring module 210 or in the machine learning module 202 be implemented. For example, the further machine learning algorithm can have been trained on this, based on the sensor data 112 to recognize certain key situations that are not clearly recognizable for the fusion algorithm. If such a key situation is recognized, an input of the planner module, for example 208 automatically to an output of the machine learning module 202 or the further fusion module 212 be switched.

The machine learning algorithm can, for example, have been trained with training data that predominantly or exclusively contain such key situations.

It is possible that the evaluation unit 110 in addition to the machine learning module 202 has at least one further machine learning module that, for example, has different situations than the machine learning module 202 can recognize. A decision about which of the respective outputs of the machine learning modules is to be used to update the environment model can be made, for example, depending on the respective trustworthiness of the outputs.

4th shows a flow chart of a method 400 as it is from the evaluation unit 110 from the 1 until 3 can be executed.

In a first step 410 are the through the three sensor units 104 , 106 , 108 generated sensor data 112 in the evaluation unit 110 receive.

In a second step 420 are the sensor data 112 into the fusion algorithm of the fusion module 200 entered based on the sensor data 112 the first object hypotheses 204 generates and outputs. Furthermore, the sensor data 112 into the machine learning algorithm of the machine learning module 202 entered based on the sensor data 112 the second object hypotheses 206 generates and outputs.

In a third step 430 it is decided whether the first object hypotheses 204 should be discarded or not.

If so, be in one step 440 the second object hypotheses 206 or the other object hypotheses 214 in the planner module 208 entered. If not, be in one step 450 the first object hypotheses 204 into the planner module 208 entered.

Finally, it is pointed out that terms such as “having”, “comprising” etc. do not exclude any other elements or steps and that terms such as “a” or “an” do not exclude a plurality. Reference signs in the claims are not to be regarded as a restriction.

Claims (10)

Method (400) for recognizing the surroundings of a vehicle (100), wherein the vehicle (100) has a sensor system with at least two sensor units (104, 106, 108) for detecting the surroundings of the vehicle (100) and an evaluation unit (110) for evaluation of sensor data (112) of the sensor system, the method (400) comprising: Receiving (410) in the evaluation unit (110) sensor data (112) generated by the at least two sensor units (104, 106, 108); Inputting (420) the sensor data (112) into a fusion algorithm which is configured to determine probability distributions with respect to features of objects (113) in the vicinity of the vehicle (100) based on the sensor data (112) and based on the probability distributions to generate and output first object hypotheses (204) assigned to the objects (113); Inputting (420) the sensor data (112) into a machine learning algorithm which has been trained to generate and output second object hypotheses (206) assigned to the objects (113) based on the sensor data (112); Deciding (430) whether or not to reject the first object hypotheses (204); if the first object hypotheses (204) are not to be discarded: using (440) the first object hypotheses (204) to update an environment model representing the environment of the vehicle (100); if the first object hypotheses (204) are to be rejected: using (450) the second object hypotheses (206) to update the environment model. Method (400) according to Claim 1 wherein at least one further object hypothesis (214) is generated based on the first object hypotheses (204) and the second object hypotheses (206); wherein the at least one further object hypothesis (214) is used to update the environment model. Method (400) according to one of the preceding claims, wherein it is determined by comparing the first object hypotheses (204) with the second object hypotheses (206) whether the first object hypotheses (204) are false or not; if the first object hypotheses (204) are false: deciding that the first object hypotheses (204) should be discarded; and / or if the first object hypotheses (204) are not false: deciding that the first object hypotheses (204) should not be rejected. The method (400) according to any one of the preceding claims, wherein the fusion algorithm is configured to weight the first object hypotheses (204); wherein a weighting of at least one of the first object hypotheses (204) is changed based on the second object hypotheses (206). Method (400) according to one of the preceding claims, wherein the machine learning algorithm was trained with training data which at least for the most part comprise features of objects that are difficult to recognize for the fusion algorithm. Method (400) according to one of the preceding claims, wherein the fusion algorithm comprises a Kalman filter; and or wherein the machine learning algorithm comprises a multilayer perceptron. Evaluation unit (110) which is configured to carry out the method (400) according to one of the preceding claims. Vehicle system (102), comprising: a sensor system with at least two sensor units (104, 106, 108) for detecting the surroundings of a vehicle (100); and the evaluation unit (110) according to Claim 7 . Computer program, comprising instructions which, when the computer program is executed by a computer, cause the computer to carry out the method (400) according to one of the Claims 1 until 6th to execute. Computer-readable medium on which the computer program is based Claim 9 is stored.

Images