DE102022204767A1 - Method and device for processing sensor data from a sensor system of a vehicle - Google Patents

Abstract

A method for processing sensor data (13, 15, 29, 33) of a sensor system (5) for detecting an environment of a vehicle (1) comprises: receiving first sensor data (13) generated by a first sensor (9), and second sensor data (15), which were generated by a second sensor (11) which differs in type from the first sensor; Entering the first sensor data into a first neural network (35) that has been trained to convert the first sensor data into first object data (39), and entering the second sensor data into a second neural network (37) that has been trained to convert the second Converting sensor data into second object data (41), the object data defining parameters of object models (43) assigned to objects in the environment, the parameters comprising, in addition to an object class, a position and/or orientation of the objects, the object models of the first object data in their type match the object models of the second object data; Updating an environment model (44) representing the environment with the object data.

Description

Field of invention

The invention relates to a computer-implemented method for processing sensor data from a sensor system of a vehicle. The invention also relates to a data processing device and a computer program for carrying out the method, as well as a computer-readable medium on which the computer program is stored.

State of the art

A vehicle such as a car or truck can be equipped with a driver assistance system that enables partially or fully automated control of the vehicle. For this purpose, the driver assistance system can, for example, use suitable sensors to detect positions, orientations and/or types of objects in the area surrounding the vehicle and steer, brake and/or accelerate the vehicle taking these objects into account.

For example, the vehicle's surroundings can be detected simultaneously by several sensors of different types. In this case, the respective sensor and/or object data can be fused together using a suitable fusion algorithm, i.e. H. be linked, for example to estimate current and/or future states of the detected objects and/or the vehicle in a digital map. Depending on the number of different sensor types, such data fusion can be very complex and require correspondingly powerful hardware components.

Disclosure of the invention

Against this background, a computer-implemented method for processing sensor data from a sensor system of a vehicle, a corresponding data processing device, a corresponding computer program and a corresponding computer-readable medium according to the independent claims are presented below. Advantageous developments and improvements to the approach presented here result from the description and are described in the dependent claims.

Advantages of the invention

Embodiments of the present invention enable improved data fusion of sensor data generated by various sensor modalities of a sensor system for detecting a vehicle's surroundings. For example, the robustness of data fusion against failure or malfunction of one or more sensor modalities can be improved, which is particularly important in the context of autonomous driving. In the best case scenario, data fusion works so well that certain sensor modalities, such as a lidar sensor, can be dispensed with entirely, which can make development and production significantly cheaper.

A first aspect of the invention relates to a computer-implemented method for processing sensor data from a sensor system of a vehicle. The sensor system includes at least a first sensor and a second sensor, which differs in type from the first sensor, for detecting an environment of the vehicle. The method includes at least the following steps (which may be performed continuously during operation of the vehicle): receiving first sensor data generated by the first sensor and second sensor data generated by the second sensor; inputting the first sensor data into a first artificial neural network trained to convert the first sensor data into first object data, and inputting the second sensor data into a second artificial neural network trained to convert the second sensor data into second object data, wherein the first object data and the second object data each define parameters of object models that are assigned to objects in the environment of the vehicle, wherein the parameters of each object model include, in addition to an object class, a position and / or orientation of the object assigned to the object model, wherein the first Object models defined by object data correspond in their type, in particular in their (geometric) representation form in a coordinate system assigned to the environment of the vehicle, to the object models defined by the second object data; Updating an environment model representing the environment of the vehicle with the first object data and the second object data.

In other words, each sensor modality (e.g. camera, lidar or radar sensor) provides the same environmental model regarding the vehicle's surroundings, where the environmental model can include all necessary information for safe and efficient operation of the vehicle. Weaknesses in the individual sensor modalities can be compensated for by processing their sensor data using a suitable algorithm. This has the effect of allowing the sensor modalities to operate independently of each other with sufficient accuracy so that in the event of a failure or malfunction of one or even more sensors, the system sensor modalities can continue to work on the basis of the remaining sensor modality(s).

For example, executing the method can cause a 3D model of the environment including static objects (e.g. buildings, vegetation or road surface) to be created and dynamic objects (e.g. cars, bicycles, pedestrians or animals) to be tracked over time in this 3D model . For this purpose, the sensor data from two or more sensors of different types (for example a camera in combination with a radar sensor and/or a lidar sensor) can be combined with one another.

Different types of sensors usually have different advantages and disadvantages. Cameras, for example, enable precise semantic segmentation by assigning different sections of the captured image, such as each pixel of the image, to one or more classes. Lidar sensors, on the other hand, provide high-precision 3D point clouds, but this can make classification difficult. Radar data is typically less detailed than lidar data. In addition, radar data can be corrupted by reflections and/or noise. Radar sensors can also be used in fog and rain. Radar sensors also provide very precise speed values. The combination of these different sensors generally serves to compensate for the weaknesses of one sensor with the strengths of the other sensor. In contrast to conventional systems, in which the performance is severely impaired or even the entire system can fail if one of the sensors fails, the approach presented here offers the advantage that the performance of the system can be maintained in the event of such disruptions or only to a very limited extent to a small, negligible extent.

The method can be carried out automatically by a processor, for example a control unit of the vehicle.

“Vehicle” can be understood, for example, as a car, a truck, a bus, a motorcycle or an autonomously moving robot.

The sensor system can include, for example, a camera, a radar, lidar or ultrasonic sensor or a combination of at least two of these examples as the first or second sensor. In addition to such environmental sensors, the sensor system can also include a driving dynamics sensor for detecting driving dynamics of the vehicle and/or a location sensor for determining geographical coordinates of the vehicle using a global navigation satellite system such as GPS, GLONASS or similar. include. Such a driving dynamics sensor can be, for example, an acceleration, wheel speed, steering wheel angle, steering torque, brake pressure or brake pedal travel sensor or a combination of at least two of these examples.

The driving dynamics data generated by the driving dynamics sensor can be used in addition to the object data to update the environmental model, in particular to determine a (current and/or future) position and/or orientation of the vehicle.

The first and second sensors can, for example, be aligned so that their detection areas at least partially overlap. This means that objects in the area surrounding the vehicle can be detected simultaneously by different types of sensors. This can improve the accuracy of the procedure.

An object class can be semantic information such as "car", "truck", "bus", "motorcycle", "bicycle", "pedestrian", "building", "tree" or "road sign".

Several object classes can be assigned to one and the same object model at the same time.

“Artificial neural network” can, for example, include a (multi-layer) perceptron, a convolutional neural network, a recurrent neural network; e.g. as long short-term memory) or a combination of different network types be understood.

Such a network can generally be viewed as a function with machine-trainable parameters for converting input data into output data, for example through classification and/or regression. To train the parameters (also called weights), real (ie measured by a sensor) and/or simulated (ie calculated by a computer) example values can be entered into the network and the resulting output values can be compared with predefined target values (also called labels). A deviation between the output and target values can be quantified using a suitable cost and/or benefit function. The parameters can then be achieved in a suitable optimization method, for example in a gradient method using backpropagation, depending on the deviation quantified by the cost or benefit function chen of a local or global optimum can be adjusted iteratively. This can also be called supervised learning. Theoretically, however, the network can also be trained using unsupervised and/or reinforcement learning.

The first and second networks can differ in type or match one another.

Alternatively, other machine learning algorithms can also be used to process the first and second sensor data, for example a decision tree, a random forest, a k-nearest neighbors algorithm, a support vector machine, a Bayes classifier, a k -Means algorithm, a genetic algorithm, a kernel regression algorithm, a discriminant analysis algorithm, or a combination of at least two of these examples.

The environment model can be configured to store parameters of objects in the environment of the vehicle associated with object models, for example in a digital map, and to continuously update them. The environment model may also be configured to estimate current and/or future states of the objects and/or the vehicle based on the first and second object data. As mentioned above, the environmental model can additionally evaluate driving dynamics data regarding the vehicle to estimate a position and/or orientation of the vehicle relative to the objects.

To update the environment model, the first and second object data may, for example, be input into a fusion module that executes a fusion algorithm to merge or link the first and second object data (for example, in a digital map). Generally speaking, in the fusion module the first object data can be processed taking into account the second object data and/or the second object data can be processed taking into account the first object data. Such a fusion algorithm may also include one or more state estimators for estimating states of the objects and/or the vehicle, for example an (extended) Kalman filter or a particle filter.

The data resulting from this data fusion can be used in particular to control an actuator system of the vehicle. Such actuators can include, for example, a brake actuator, a steering actuator, an engine control unit, an electric drive motor or a combination of at least two of these examples.

For example, a driver assistance system can run on the control unit of the vehicle, which detects objects in sensor data using the method described here and automatically controls the vehicle, i.e. steers, accelerates and/or brakes, by appropriately controlling the actuators depending on the detected objects.

A second aspect of the invention relates to a data processing device comprising a processor configured to carry out the method described above and below. The data processing device can include hardware and/or software modules. In addition to the processor, the data processing device may include a memory and data communication interfaces for data communication with peripheral devices. The data processing device can be, for example, a control device of a vehicle or a robot, a PC, a server, a laptop, a tablet or a smartphone.

Further aspects of the invention relate to a computer program and a computer-readable medium on which the computer program is stored.

The computer program includes instructions that cause a processor to carry out the method described above and below when the computer program is executed by the processor.

The computer-readable medium may be a volatile or non-volatile data storage device. For example, the computer-readable medium may be a hard drive, a USB storage device, a RAM, a ROM, an EPROM, a flash memory, or a combination of at least two of these examples. The computer-readable medium can also be a data communication network that enables a download of program code, such as the Internet or a data cloud.

Features of the method described above and below can also be understood as features of the data processing device, the computer program and/or the computer-readable medium (and vice versa).

Embodiments of the invention may be considered based on the ideas and findings described below, without limiting the invention.

According to one embodiment, the object models of the same type can be three-dimensional geometric models be. This allows the real (three-dimensional) objects to be depicted in object models as realistically as possible.

Alternatively, the object models of the same type can be two-dimensional geometric models, for example rectangles.

In addition to the object models, the environment model can include a (two- or three-dimensional) vehicle model assigned to the vehicle.

Other possible parameters of the object models, especially in the form of three-dimensional geometric models, are the position of the center of gravity, length, width, height, roll, pitch, yaw angle and values derived therefrom such as speed, acceleration or jerk.

According to one embodiment, the three-dimensional geometric models can be cuboids. This enables accurate and computationally efficient modeling of the objects.

According to one embodiment, the first artificial neural network may include a first module and a second module. In this case, the first module may have been trained to classify the first sensor data, while the second module may have been trained to classify input data comprising outputs of the first module and/or data generated from outputs of the first module into the first Convert object data. Additionally or alternatively, the second artificial neural network may include a third module and a fourth module. In this case, the third module may have been trained to classify the second sensor data, while the fourth module may have been trained to integrate input data comprising outputs of the third module and/or data generated from outputs of the third module into the second Convert object data. This can simplify the implementation of the first or second network.

In particular, the classification of the first or second sensor data in a module that can be trained independently of the rest of the respective artificial neural network can reduce the training effort. For example, when updating with new training data for classification, only part of the respective artificial neural network (and not the entire network) needs to be retrained.

It is possible that the outputs of the first and third modules are converted in an intermediate step before they are input into the second or fourth module. This intermediate step can, for example, be carried out by an external conversion module that is not part of the respective artificial neural network. This means that the respective artificial neural network can be constructed more simply overall. This can further reduce the training effort.

The second or fourth module may, for example, have been trained to extract the three-dimensional geometric models from the respective input data.

The first and second modules may have been trained at least partially with the same training data. Additionally or alternatively, the third and fourth modules may have been trained at least partially with the same training data. As a result, the effort required to train the first or second network can be greatly reduced.

According to one embodiment, the input data for the second module may include the first sensor data underlying the outputs of the first module. Additionally or alternatively, the input data for the fourth module can include the second sensor data underlying the outputs of the third module. In other words, the second or fourth module can be configured to compare the respective (estimated) input data and the (measured) sensor data corresponding to these input data with one another. This improves the estimation of the object models.

According to one embodiment, the input data for the second module may be related to a first coordinate system assigned to the environment of the vehicle, while the first object data may be related to a second coordinate system assigned to the environment of the vehicle, which deviates from the first coordinate system. Additionally or alternatively, the input data for the third module can be related to a third coordinate system assigned to the surroundings of the vehicle, while the second object data can be related to a fourth coordinate system assigned to the surroundings of the vehicle, which deviates from the third coordinate system. In other words, the generation of the first or second object data may include one or more coordinate transformations (which can be carried out by the second or fourth module).

It is possible for the first coordinate system to have one or more common planes with the second coordinate system and/or for the third coordinate system to have one or more in common has same planes with the fourth coordinate system.

According to one embodiment, the second coordinate system can differ from the first coordinate system in the number of its dimensions. Additionally or alternatively, the fourth coordinate system can differ from the third coordinate system in the number of its dimensions. In particular, the second coordinate system can include more dimensions than the first coordinate system and/or the fourth coordinate system can include more dimensions than the third coordinate system.

According to one embodiment, the first coordinate system may be two-dimensional (e.g. an xy coordinate system) and the second coordinate system may be three-dimensional (e.g. an xyz coordinate system). Additionally or alternatively, the third coordinate system may be two-dimensional (e.g. an xy coordinate system) and the fourth coordinate system may be three-dimensional (e.g. an xyz coordinate system).

According to one embodiment, the first coordinate system and the second coordinate system can be assigned to different perspectives from which the surroundings of the vehicle are viewed. Additionally or alternatively, the third coordinate system and the fourth coordinate system can be assigned to different perspectives from which the surroundings of the vehicle are viewed. “Perspective” can be understood as meaning a predefined direction and/or a predefined angular range in which a real or virtual viewer views the surroundings of the vehicle.

According to one embodiment, the first coordinate system can be assigned to a bird's eye view from which the surroundings of the vehicle, for example including the vehicle, are looked down (vertically). Additionally or alternatively, the second coordinate system can be assigned to a driver's perspective from which the surroundings of the vehicle, for example including the vehicle, are viewed in a current direction of travel of the vehicle. Additionally or alternatively, the third coordinate system can be assigned to a bird's eye view from which the surroundings of the vehicle, for example including the vehicle, are looked down (vertically). Additionally or alternatively, the fourth coordinate system can be assigned to a driver's perspective from which the surroundings of the vehicle, for example including the vehicle, are viewed in a current direction of travel of the vehicle.

“Current direction of travel” can also be a predefined, i.e. H. horizontally and / or vertically limited angular range can be understood, which can correspond to a current field of vision of the driver of the vehicle and / or a current field of vision of another (virtual) driver of a (virtual) vehicle (immediately) following the vehicle. For example, the view of the vehicle's surroundings from a bird's eye view may be two-dimensional, while the view of the vehicle's environment from the driver's perspective may be three-dimensional. But the opposite case is also possible.

According to one embodiment, the third coordinate system may be the first coordinate system. Additionally or alternatively, the fourth coordinate system can be the second coordinate system. In other words, the input data for the second module may be referenced to the same (first) coordinate system as the input data for the fourth module. In this case, the first object data can be related to the same (second) coordinate system as the second object data. This means that the object models of the first object data can not only match the object models of the second object data in their (geometric) representation form, but can also be related to the same coordinate system as the object models of the second object data.

According to one embodiment, the first sensor data may include camera data generated by a camera as the first sensor and the second sensor data may not include camera data. For example, in this case, the second sensor data may include radar data generated by a radar sensor as the second sensor and/or lidar data generated by a lidar sensor as the second sensor.

Additionally or alternatively, the first sensor data may include radar data generated by a radar sensor as the first sensor and the second sensor data may not include radar data. For example, in this case, the second sensor data may include camera data generated by a camera as the second sensor and/or lidar data generated by a lidar sensor as the second sensor.

Additionally or alternatively, the first sensor data may include lidar data generated by a lidar sensor as the first sensor and the second sensor data may not include lidar data. For example, the second sensor data in this case can be camera data generated by a camera as the second sensor and/or by a radar sensor as the second sensor include radar data generated.

Brief description of the drawings

• 1 zeigt ein Fahrzeug mit einem Fahrzeugsystem, das eine Sensorik zum Erfassen einer Umgebung des Fahrzeugs und eine Datenverarbeitungsvorrichtung gemäß einer Ausführungsform der Erfindung umfasst.
• 2 zeigt ein Blockschaltbild des Fahrzeugsystems.
• 3 zeigt einen Abschnitt eines dreidimensionalen Umgebungsmodells, das eine Umgebung des Fahrzeugs aus einer Fahrerperspektive darstellt.
• 4 zeigt einen Abschnitt eines zweidimensionalen Umgebungsmodells, das eine Umgebung des Fahrzeugs aus der Vogelperspektive darstellt.

Embodiments of the invention are described below with reference to the accompanying drawings. Neither the description nor the drawings are to be construed as limiting the invention.

• 1 shows a vehicle with a vehicle system that includes a sensor system for detecting an environment of the vehicle and a data processing device according to an embodiment of the invention.
• 2 shows a block diagram of the vehicle system.
• 3 shows a section of a three-dimensional environment model that represents an environment of the vehicle from a driver's perspective.
• 4 shows a section of a two-dimensional environment model that represents a bird's eye view of the vehicle's surroundings.

The figures are only schematic and not to scale. The same reference numerals in the figures indicate the same features or features that have the same effect.

Embodiments of the invention

1 shows a vehicle 1 with a vehicle system 3, which includes a sensor system 5 and a control device 7 as a data processing device 7. In this example, the sensor system 5 includes a camera 9 as a first sensor 9 for detecting the surroundings of the vehicle 1 and a radar sensor 11 as a second sensor 11, which differs in type from the first sensor 9, for detecting the surroundings of the vehicle 1.

The camera 9 generates camera data 13 as first sensor data 13, while the radar sensor 11 generates radar data 15 as second sensor data 15.

The control device 7 is configured to evaluate the sensor data 13, 15 in a method described in more detail below. The controller 7 may include a processor 17 and a memory 19, wherein the processor 17 may be configured to execute the method by executing instructions of a computer program that may be stored in the memory 19.

For example, the control unit 7 can be configured to automatically control an actuator system 21 of the vehicle 1 depending on the results of this evaluation.

2 shows possible modules of the control device 7, here a perception module 23 and a fusion module 25 connected downstream of the perception module 23. The modules 23, 25 can be implemented as hardware and/or software.

How out 2 As can be seen, the sensor system 5 can additionally include a third sensor 27 for detecting the surroundings of the vehicle 1 (here a lidar sensor 27). The third sensor data 29 (ie lidar data 29) generated by the third sensor 27 can also be evaluated by the control device 7, for example in order to control the actuator system 21.

In addition, the sensor system 5 can include a driving dynamics sensor 31 for detecting driving dynamics of the vehicle 1. The driving dynamics data 33 generated by the driving dynamics sensor 31 can, for example, be entered directly into the fusion module 25 and processed there together with outputs from the perception module 23.

Processing the sensor data 13, 15 can include the following steps, which are carried out here by the perception module 23 (the third sensor data 29 can be processed in the same or similar way as the sensor data 13 and 15, respectively).

In a first step, the sensor data 13, 15 are received in the control device 7 (here in the perception module 23). The sensor data 13, 15 can, for example, be received and processed continuously in several successive time steps during operation of the vehicle system 3.

In a second step, the camera data 13 is entered into a first artificial neural network 35 and the radar data 15 is entered into a second artificial neural network 37 (independent of the first network 35).

The first network 35 was trained (for example in a supervised learning process with exemplary camera data and reference data, also called labels, assigned to the exemplary camera data) in order to convert the camera data 13 into first object data 39.

The second network 37 was trained (for example in a supervised learning process with exemplary radar data and reference data associated with the exemplary radar data) to convert the radar data 15 into second object data 41.

The object data 39, 41 each define parameters of object models 43 in an environmental model 44 representing the environment of the vehicle 1 (see 3 ). The object models 43 are assigned to objects in the surroundings of the vehicle 1 (detected by the respective sensor 9 or 11).

As in 3 To recognize, the object models 43 can be three-dimensional geometric models in an xyz coordinate system 45 assigned to the environment of the vehicle 1, here in the form of cuboids, the other vehicles on a road 47 or objects on the edge of the road 47, such as such as trees, traffic signs, pedestrians or buildings.

The parameters of each object model 43 may include at least one class value indicating one of several predefined object classes (e.g., “tree,” “stop sign,” “pedestrian,” “building”), as well as values for a position, orientation and/or size of the object Object model 43 associated object (and/or values derived therefrom). In this case, the position, orientation and/or size of the object can be predetermined by a position, orientation and/or size of the (geometric) object model 43 in the coordinate system 45.

For example, the class value can be a Boolean value or a probability for one of the object classes in percent.

In order to enable simpler and more robust data processing in the downstream fusion module 25, the two networks 35, 37 were trained in such a way that the object models 43 defined by the first object data 39 correspond in their type, in particular in their geometric representation form in the coordinate system 45, to those defined by the second object data 41 defined object models 43 match.

In this example, all object models 43 are configured as cuboid grids that surround the respective objects in a box-like manner. However, other forms of representation are also possible, for example round forms of representation, angular forms of representation with more than four corners or two-dimensional, i.e. H. flat forms of representation.

In a third step, the environment model 44 is updated with the object data 39, 41. The update is carried out here by the fusion module 25.

Additionally, in the second step, the lidar data 29 can be input into a third artificial neural network 49 that has been trained (for example in a supervised learning process with exemplary lidar data and reference data associated with the exemplary lidar data) in order to convert the lidar data 29 into third object data 51, which in their type matches the other object data 39, 41.

In the third step, the environment model 44 can then additionally be updated with the third object data 51.

Each network 35, 37, 49 can be a single network. However, networks are also possible, each of which is made up of several independently trainable modules in the form of subnets. On the one hand, this can simplify the training of the networks; on the other hand, simpler network architectures can be used for the smaller modules than for a larger individual network.

In this example, the first network 35 is made up of a first module 35a and a second module 35b, the second network 37 is made up of a third module 37a and a fourth module 37b and the third network 49 is made up of a fifth module 49a and a sixth module 49b.

The first module 35a was trained to classify the camera data 13. The second module 35b has been trained to convert input data 53, which includes outputs of the first module 35a and/or data generated from outputs of the first module 35a, into the first object data 39.

The third module 37a was trained to classify the radar data 15. The fourth module 37b was trained to convert input data 53, which includes outputs of the third module 37a and/or data generated from outputs of the third module 37a, into the second object data 41.

The fifth module 49a was trained to classify the lidar data 29. The sixth module 49b was trained to convert input data 53, which includes outputs of the fifth module 49a and/or data generated from outputs of the fifth module 49a, into the third object data 51.

In addition, the input data 53 for the fourth module 37b can include the radar data 15 underlying the outputs of the third module 37a and the input data 53 for the sixth module 49b can include the lidar data 29 underlying the outputs of the fifth module 49a.

In particular, the input data 53 for one or more of the downstream modules 35b, 37b, 49b can refer to a first coordinate system 55 (see 4 ) relate that of one two th coordinate system 45 (see 3 ), to which the object data 39, 41, 51 refer, may differ.

In this example, the first coordinate system 55 is a two-dimensional xy coordinate system that represents the surroundings of the vehicle 1 from a bird's eye view, whereas the second coordinate system 45 is a three-dimensional xyz coordinate system that represents the surroundings of the vehicle 1 from a driver's perspective, i.e. H. the way the driver of vehicle 1 or another driver of a vehicle (immediately) following vehicle 1 would see the surroundings.

It is possible for a coordinate system 55, to which the input data 53 for one of the downstream modules 35b, 37b, 49b relate, to be different from a coordinate system 55, to which the input data 53 for another downstream module 35b, 37b, 49b relate, differs. Alternatively, the input data 53 can refer to the same coordinate system 55 for all downstream modules 35b, 37b, 49b.

This can apply in a corresponding manner to the coordinate system 45 to which the object data 39, 41, 51 relate.

As shown here, the fusion module 25 can also include several individual modules, for example a pose estimator 57 for estimating a pose of the vehicle 1, a state estimator 59 for estimating current and/or future states of the objects, more precisely the object models 43, and a map module 61, which stores the objects in a digital map of the surroundings of the vehicle 1 (e.g. a raster map).

Further details are described below.

The lidar data 29 can be raw data from the lidar sensor 27, which were generated by a perspective scan of a partial area of the surroundings of the vehicle 1. This raw data can be used by the fifth module 49a for semantic segmentation, whereby the fifth module 49a can output a semantic class for each 3D point in the raw data. The resulting semantic data can then be combined with the point coordinates of the respective 3D points and projected into the coordinate system 55, ie into a bird's eye view representation (see 4 ).

The bird's eye view images resulting from this projection can be used as inputs for the sixth module 49b, which can then recognize object models 43 in these images in the form of 3D boxes for relevant objects such as cars, bicycles, pedestrians, etc.

The first module 35a can be, for example, a multitasking network that outputs a depth value and semantic information for each pixel of a camera image.

The camera images can be transformed into 3D point clouds using the estimated depth values (D) and a mathematical model of the camera 9, which includes an intrinsic calibration matrix (K) and the pixel coordinates: $$POintClOud=D\ast camera\_mOdel\left(K,pixel\_cOOrdinates\right)$$

The 3D point clouds can then be combined with the semantic information and stored in the coordinate system 55, i.e. H. projected into the bird's eye view. Further processing can take place as described above using the example of lidar data 29.

The radar data 15 can already be 2D images in the coordinate system 55, i.e. H. Bird's-eye images include. These radar images can be used as inputs to the third module 37a, which outputs one or more semantic classes for each pixel. The radar images can then be combined with the semantic information and used as input for the fourth module 37b.

For example, to train the semantic segmentation (semseg) modules, the softmax function can be used as a loss function (L): $$L_{\text{semseg}}=sOftmax\left(l,\widehat{l}\right)$$

l stands for the network's output and l̂ for target values regarding these outputs (ground truth).

For example, for the multitasking network, the following weighted multitasking loss function can be used: $$L_{\text{multitask}}=w\ast L_{\text{semseg}}+\left(1-w\right)\ast L_{\text{depth}}$$

Where w stands for a weight, L _semseg for the loss in terms of semantic segmentation (see above) and L _depth for the loss in estimating the depth values.

für die Zielwerte bezüglich der Tiefenwerte): $$L_1\left(depth,\widehat{depth}\right)=\left|depth-\widehat{depth}\right|$$

L _depth can also be formulated as L ₁ loss if labels are available for a supervised learning process (with depth for the estimated ones Depth values and $\widehat{deptH}$

for the target values regarding the depth values): $$L_1\left(deptH,\widehat{deptH}\right)=\left|deptH-\widehat{deptH}\right|$$

The projection from the point cloud display into the bird's eye view display can be done, for example, according to this formula: $$POintClOudTOp=POintClOud\cdot tOp\_view\_T\_sensOr$$

Top_view_T_sensor stands for a 4 × 4 transformation matrix for the transformation from the respective sensor coordinate system into the coordinate system 55 of the bird's-eye view representation.

For example, to obtain corresponding coordinates of a pixel in the bird's eye view image from the coordinates of each 3D point, the following formula can be used: $$bev\_cOOrd=POintClOudTOp/pixels\_si\mathrm{e.g}e\_meters$$

Pixels_size_meters stands for the area in meters that is projected onto a single pixel.

The fusion module 25 serves to combine the information obtained from sensor data from different sensors in a common environment model 44. Since each sensor modality provides the same complete representation of the environment, this simplifies the fusion process and makes the system robust against failure or malfunction of individual sensors or individual sensor modalities.

The pose estimator 57 can, for example, determine an estimated vehicle position and/or orientation and their uncertainty(s) from the object data 37, 39 and/or 51 (with additional use of the vehicle dynamics data 33).

In the state estimator 59, e.g. B. an (extended) Kalman or a particle filter, for example the parameters of the object models 43 and the vehicle dynamics data 33 can be entered.

For example, statistical compatibility tests can be performed to find a connection between an object i from a list of predicted objects and observations from sensors j. For example, given v _ij -innovation and S _ij -innovation covariance of the pairing ij, the Mahalonobis distance can be calculated: $$d_{ij}^2=v_{ij}^T{S}_{ij}^{-1}{v}_{ij}$$

This can be compared to the threshold from a cumulative χ ² distribution and should not be greater than this threshold (with n being the number of degrees of freedom and a being the significance level, which should be between 0.95 and 0.99): $$d_{ij}^2\le {\mathrm{\chi }}_{n,a}^2$$

The states of each dynamic object can be predicted using the extended Kalman filter. The advantage of uniform state representation for multiple sensor modalities, e.g. B. in the form of 3D boxes, is that the same data association and motion models can be applied to the dynamic objects detected by the different sensor modalities. This simplifies the algorithm and improves accuracy.

The estimate of a probabilistic occupancy grid m (ie an occupancy of a grid map representing the surroundings of the vehicle 1) can be formulated as follows: $$p\left(m|{\mathrm{e.g}}_{1:t},x_{1:t}\right)={\displaystyle \prod _{i}p\left(m_i|{\mathrm{e.g}}_{1:t},x_{1:t}\right)}$$

Here p(m|z _1:t ,x _1:t ) is the probabilistic state of the map m for all observations z _1:t of the environment and all positions of the ego vehicle x _1:t up to time t.

The map module 61 can combine the object data 37, 39 and/or 51 (with additional use of the vehicle dynamics data 33) into a probabilistic occupancy grid map with semantic information.

The probabilistic occupancy map discretizes the space above the grid cells and estimates the probability of grid cell occupancy for a given observation. Each grid cell can also be assigned one or more object classes and/or (if applicable) an object height.

Finally, it is noted that terms such as "comprising", "comprising", etc. do not exclude other elements or steps, and indefinite articles such as "a" or "an" do not exclude a plurality. Reference symbols in the claims are not to be viewed as a limitation.

Claims (15)

Computer-implemented method for processing sensor data (13, 15, 29, 33) of a sensor system (5) of a vehicle (1), the sensor system (5) having a first sensor (9) and a sensor of its type second sensor (11), which deviates from the first sensor (9), for detecting an environment of the vehicle (1), the method comprising: receiving first sensor data (13) generated by the first sensor (9) and second sensor data (15 ), which were generated by the second sensor (11); Entering the first sensor data (13) into a first artificial neural network (35) that has been trained to convert the first sensor data (13) into first object data (39), and entering the second sensor data (15) into a second artificial neural network (37), which was trained to convert the second sensor data (15) into second object data (41), the first object data (39) and the second object data (41) each defining parameters of object models (43) that correspond to objects in the Surroundings of the vehicle (1), the parameters of each object model (43) comprising, in addition to an object class, a position and/or orientation of the object assigned to the object model (43), the object models (43) defined by the first object data (39). ) in their type, in particular in their form of representation in a coordinate system (45) assigned to the environment of the vehicle (1), match the object models (43) defined by the second object data (41); Updating an environmental model (44) representing the environment of the vehicle (1) with the first object data (39) and the second object data (41). Procedure according to Claim 1 , whereby the object models (43), which are of the same type, are three-dimensional geometric models. Procedure according to Claim 2 , where the three-dimensional geometric models are cuboids. Method according to one of the preceding claims, wherein the first artificial neural network (35) comprises a first module (35a) and a second module (35b), the first module (35a) being trained to classify the first sensor data (13), and the second module (35b ) has been trained to convert input data (53), which includes outputs of the first module (35a) and/or data generated from outputs of the first module (35a), into the first object data (39); and or wherein the second artificial neural network (37) comprises a third module (37a) and a fourth module (37b), the third module (37a) being trained to classify the second sensor data (15), and the fourth module (37b ) was trained to convert input data (53), which includes outputs of the third module (37a) and / or data generated from outputs of the third module (37a), into the second object data (41). Procedure according to Claim 4 , wherein the input data (53) for the second module (35b) comprises the first sensor data (13) on which the outputs of the first module (35a) are based; and/or wherein the input data (53) for the fourth module (37b) comprises the second sensor data (15) on which the outputs of the third module (37a) are based. Procedure according to Claim 4 or 5 , wherein the input data (53) for the second module (35b) is related to a first coordinate system (55) assigned to the environment of the vehicle (1) and the first object data (39) to a second coordinate system assigned to the environment of the vehicle (1). (45), which deviates from the first coordinate system (55), are related; and/or wherein the input data (53) for the third module (37a) is related to a third coordinate system (55) assigned to the surroundings of the vehicle (1) and the second object data (41) to one assigned to the surroundings of the vehicle (1). fourth coordinate system (45), which deviates from the third coordinate system (55), are related. Procedure according to Claim 6 , wherein the second coordinate system (45) differs from the first coordinate system (55) in the number of its dimensions; and/or wherein the fourth coordinate system (45) differs from the third coordinate system (55) in the number of its dimensions. Procedure according to Claim 7 , wherein the first coordinate system (55) is two-dimensional and the second coordinate system (45) is three-dimensional; and/or wherein the third coordinate system (55) is two-dimensional and the fourth coordinate system (45) is three-dimensional. Procedure according to one of the Claims 6 until 8th , wherein the first coordinate system (55) and the second coordinate system (45) are assigned to different perspectives from which the surroundings of the vehicle (1) are viewed; and/or wherein the third coordinate system (55) and the fourth coordinate system (45) are assigned to different perspectives from which the surroundings of the vehicle (1) are viewed. Procedure according to one of the Claims 6 until 9 , wherein the first coordinate system (55) is assigned to a bird's eye view from which the The surroundings of the vehicle (1) are looked down; and/or wherein the second coordinate system (45) is assigned to a driver's perspective from which the surroundings of the vehicle (1) are viewed in a current direction of travel of the vehicle (1); and/or wherein the third coordinate system (55) is assigned to a bird's eye view from which the surroundings of the vehicle (1) are viewed; and/or wherein the fourth coordinate system (45) is assigned to a driver's perspective from which the surroundings of the vehicle (1) are viewed in a current direction of travel of the vehicle (1). Procedure according to one of the Claims 6 until 10 , wherein the third coordinate system (55) is the first coordinate system (55); and/or wherein the fourth coordinate system (45) is the second coordinate system (45). Method according to one of the preceding claims, wherein the first sensor data (13) includes camera data and the second sensor data (15) does not include camera data; and or wherein the first sensor data (13) includes radar data and the second sensor data (15) does not include radar data; and or wherein the first sensor data (13) includes lidar data and the second sensor data (15) does not include any lidar data. Data processing device (7) comprising a processor (17) configured to carry out the method according to one of the preceding claims. Computer program, comprising instructions that cause a processor (17) to carry out the method according to one of the processors (17) when the computer program is executed by the processor (17). Claims 1 until 11 to carry out. Computer-readable medium on which the computer program is written Claim 14 is stored.

Images