WO2023247550A1

WO2023247550A1 - Method and network for sensor data fusion

Info

Publication number: WO2023247550A1
Application number: PCT/EP2023/066657
Authority: WO
Inventors: Evren Divrikli; Federico Ignacio Sanchez Pinzon
Original assignee: Robert Bosch Gmbh
Priority date: 2022-06-23
Filing date: 2023-06-20
Publication date: 2023-12-28
Also published as: DE102022206345A1

Abstract

The present invention relates to a method for sensor data fusion, wherein object lists (108) containing features of captured objects (106) are read in from at least two sensors (102), kinematic features (112) and additional features (114) of the captured objects (106) are each extracted from the object lists (108), and a common identifier (116) is allocated to each of the kinematic features (112) and additional features (114) captured together in terms of time, wherein the kinematic features (112) captured by the various sensors (102) for an object (106) are combined using a kinematic fusion to form a fusion object (120), wherein the allocated identifiers (116) of the kinematic features (112) are obtained during the kinematic fusion, wherein the additional features (114) with the corresponding identifiers (116) are added to the fusion object (120) using the identifiers (116) of the kinematic features (112) which have been combined to form the fusion object (120).

Description

Method and network for sensor data fusion

Field of invention

The invention relates to a method for sensor data fusion, a corresponding network, and a corresponding computer program product.

State of the art

Sensor data fusion brings together data from different sensors. For sensor data fusion, objects depicted in the different data are tracked and associated. For this purpose, objects located in a similar place at a similar time are associated with similar kinematics.

Disclosure of the invention

Against this background, the approach presented here presents a method for sensor data fusion, a corresponding network, and a corresponding computer program product according to the independent claims. Advantageous developments and improvements to the approach presented here result from the description and are described in the dependent claims.

Advantages of the invention

In the approach presented here, objects detected by different sensors are associated and brought together based on their kinematics. In addition, others not used for kinematic association are used Features cached and added to the merged objects after association.

The approach presented here allows features that are not used for kinematic association to be passed on to subsequent steps. These features can no longer be lost.

A method for sensor data fusion is proposed, in which object lists with features of detected objects are read in by at least two sensors, kinematic features and additional features of the detected objects are extracted from the object lists, and a common identifier is assigned to kinematic features and additional features recorded together at the same time, whereby the of The kinematics features detected by the various sensors to form an object are merged into a fusion object using a kinematics fusion, the assigned identifiers of the kinematics features being obtained during the kinematics fusion, wherein using the identifiers of the kinematics features merged to form the fusion object, the additional features with the corresponding identifiers for the fusion object to be added.

Ideas for embodiments of the present invention may be considered based on, among other things, the ideas and findings described below.

Embodiments of the sensor data fusion method described herein and the network used to implement it can, for example, be used to enable a vehicle to evaluate sensor data received using a plurality of sensors in an advantageous manner in order to ultimately generate information about the vehicle's surroundings. The information generated can be used, for example, to control the vehicle or individual functions thereof and/or to support a driver of the vehicle in controlling the vehicle. The network can be part of a perception sensor system or communicate with it. In particular, the network can be used in a vehicle assistance system.

A sensor can detect objects in a detection area, detect the objects and features of the objects, and describe the features of the detected objects as Output object list. An object list can contain different characteristics. Different object lists can be provided by different sensors. The sensors can have at least partially overlapping detection areas. This means that one and the same object can be detected by multiple sensors, features of the object can be recognized by multiple sensors, and the features can be mapped in multiple object lists.

Sensor data fusion can combine the features represented in multiple object lists. For this purpose, the kinematics or movement of the object can be observed. The kinematics can be represented by kinematics features of the object in the object list. Kinematic features can, for example, represent a speed of the object, an acceleration of the object, a position of the object and/or an orientation of the object. If the same object is detected by multiple sensors, the same kinematics of the object are also detected and the kinematics features from different object lists represent the same kinematics. These kinematics features can be brought together through kinematics fusion.

Additional features can represent information about the object that has no relation to the kinematics of the object. The additional features can, for example, represent a color of the object, a size of the object, a mass of the object, a material of the object and/or a type or class of the object.

An identifier can, for example, represent a detection time of the feature. The identifier can be unique. The identifier can identify the detecting sensor. The identifier can uniquely identify the features recorded together.

A fusion object can be an object detected by at least two sensors. The fusion object can have features detected by the different sensors.

The additional features with their identifiers can be stored in a feature buffer. The additional features can be cached while the kinematics fusion is running. The feature buffer can for example, be a ring buffer memory. The additional features can be read from the feature buffer after the kinematics function. The additional features can be saved for each sensor in its own feature buffer. The additional features of several object lists can also be saved together. The identifiers can define a chronological order of the additional features of the various sensors.

The kinematics features with their identifiers can be saved in a kinematics buffer. The kinematics buffer can be, for example, a ring buffer memory. The kinematics features can be saved in a separate kinematics buffer for each sensor. The kinematics features of several object lists can also be saved together. The identifiers can define a chronological order of the kinematic features. The kinematics features of the different sensors can be merged into the fusion object using the chronological order during kinematics fusion. Kinematic features read with a time delay can be arranged in chronological order using the identifiers. If kinematics features are read with a time delay, for example due to signal transit times and/or previous processing steps, the kinematics fusion can be run through again.

The additional features in the feature buffer can be deleted after a predefined storage period. If the additional features are not read within the storage period, the additional features can be deleted. The additional features can quickly become outdated. Likewise, kinematics features not used in kinematics fusion can be deleted after a predefined storage period. The kinematics features can quickly become outdated.

The fusion object can be checked for plausibility using the additional features added. In particular, additional features of different sensors can be used to check plausibility.

For example, the fusion object can be rated with reduced reliability if an inconsistency in the additional features is detected during the plausibility check. An inconsistency can be detected, for example, when one sensor detects a first additional feature while the other sensor detects it but detects a contradictory other additional feature, for example the first sensor detects a first object type, but the other sensor detects a different object type. Likewise, an inconsistency can be recognized if at least one additional feature changes in the course of the kinematics under consideration, for example if a color of the object is suddenly recognized as having changed.

Kinematic features whose additional features show an inconsistency can be discarded. These kinematic features may potentially belong to another object. Kinematics fusion can be performed again without the discarded kinematics features. Another kinematics fusion can be performed using the discarded kinematics features to recognize the other object as another fusion object.

The method is preferably computer-implemented and can be implemented, for example, in software or hardware or in a mixed form of software and hardware, for example in a control device.

The approach presented here also creates a network that is designed to carry out, control or implement the steps of a variant of the method presented here in appropriate facilities.

The network can be an electrical device with at least one computing unit for processing signals or data, at least one storage unit for storing signals or data, and at least one interface and/or a communication interface for reading in or outputting data that is embedded in a communication protocol, be. The computing unit can be, for example, a signal processor, a so-called system ASIC or a microcontroller for processing sensor signals and outputting data signals depending on the sensor signals. The storage unit can be, for example, a flash memory, an EPROM or a magnetic storage unit. The interface can be designed as a sensor interface for reading in the sensor signals from a sensor and/or as an actuator interface for outputting the data signals and/or control signals to an actuator. The communication interface can be designed to read or output the data wirelessly and/or by wire. The interfaces can also be software modules that are present, for example, on a microcontroller alongside other software modules. Also advantageous is a computer program product or computer program with program code, which can be stored on a machine-readable carrier or storage medium such as a semiconductor memory, a hard drive memory or an optical memory and for carrying out, implementing and / or controlling the steps of the method according to one of the embodiments described above is used, particularly if the program product or program is executed on a computer or device.

It should be noted that some of the possible features and advantages of the invention are described herein with reference to various embodiments. A person skilled in the art will recognize that the features of the control device and the method can be suitably combined, adapted or exchanged to arrive at further embodiments of the invention.

Short description of the drawing

Embodiments of the invention are described below with reference to the accompanying drawing, whereby neither the drawing nor the description are to be construed as limiting the invention.

1 shows a representation of a network for sensor data fusion according to an exemplary embodiment.

The figure is only schematic and not to scale. The same reference numerals denote the same or identical features.

Embodiments of the invention

1 shows a representation of a network 100 for sensor data fusion according to an exemplary embodiment. The sensor data to be merged is provided by at least two sensors 102. Here, for example, two sensors 102 provide sensor data. The sensors 102 can be similar or different. For example, one sensor 102 may be a radar sensor while the other sensor 102 is a camera. The sensors 102 each detect a detection area 104. The detection areas 104 of at least two of the sensors 102 overlap at least partially. The sensors 102 detect objects 106 within the detection areas 104 and map features of the objects 106 in object lists 108. The sensor data to be merged is therefore available here in the form of object lists 108. The sensor data can also be present as a preliminary stage of the object lists 108, for example as point clouds. Then an intermediate object recognition is required.

If an object is detected by multiple sensors 102, features of the object 106 are mapped in the object lists 108 of all detecting sensors 102. Since the object 106 is detected from a perspective of the detecting sensor 102, the features depicted in the object lists 108 differ at least slightly. Different sensors 102 can also detect different features of the same object 106 due to their different sensor technology.

The network 100 reads the object lists 108 from the sensors 102. In an extraction device 110 of the network 100, kinematic features 112 of the detected objects 106 and additional features 114 of the detected objects 106 are extracted from the object lists.

The kinematic features 112 represent kinematic properties of the objects 106, such as a speed, an acceleration and/or a position of the respective object 106. The kinematic properties are used in a subsequent kinematics fusion.

The additional features 114 represent properties of the objects 106 that are not used in kinematics fusion, such as a color, a size, a mass, a material and/or a type of the respective object 106.

During the extraction in the extraction device 110, the kinematics features 112 and the additional features 114 are assigned identifiers 116. The identifiers 116 are the same for features recorded at the same time. The identifiers 116 identify associated kinematics features 112 and additional features 114. The kinematics features 112 from the different object lists 108 are fused in a fusion device 118 of the network 100 by kinematics fusion. The kinematic features 112 of an object 106 detected by several sensors 102 are combined to form a fusion object 120. The identifiers 116 of the fused kinematics features 112 are retained and are assigned to the fusion object 120.

After the kinematics fusion, the additional features 114 with the corresponding identifiers 116 are added to the at least one fusion object 120 in a merging device 122 of the network 100.

In one exemplary embodiment, the kinematic features 112 and additional features 114 with their identifiers 116 are cached in ring buffers 124 of the network 100. The kinematic features 112 and additional features 114 are stored chronologically in the ring buffers 124. Memory locations of the ring buffers 124 are used one after the other. When saving, the storage space with the oldest content is overwritten. If the sensor data of one or more of the sensors 102 are read in with a time offset, the time offset can be compensated for by buffering. If sensor data is read in with the time offset, the kinematics fusion can be repeated with the subsequently extracted kinematics features 112. In this way, kinematic features 112 that are recorded at the same time can be used for the kinematics fusion and the accuracy of the kinematics fusion can be increased. The additional features 114 remain in the ring buffer 124 at least until they can be assigned to the appropriate fusion object 120.

In one exemplary embodiment, the kinematic features 112 and additional features 114 are temporarily stored for each sensor 102 in their own ring buffers 124. In this way, a mixing of the features can be prevented if identifiers 116 happen to have the same name.

In one exemplary embodiment, the fusion object 120 is checked for plausibility using the additional features 114 in a plausibility check device 126 of the network 100. The plausibility check device 126 assigns a reliability value 128 to the fusion object 120. For example, at an inconsistency can be detected in a fusion object 120 if an additional feature 114 suddenly changes. The fusion object 120 can be assigned a reduced reliability value 128 in the event of an inconsistency. The reliability value 128 can be taken into account in subsequent processing steps.

For example, an inconsistency may be detected when the type of fusion object 120, the color of the fusion object 120, the size of the fusion object 120, the mass of the fusion object 120, and/or the material of the fusion object 120 suddenly changes.

In one embodiment, the kinematics features 112 from the sensor data of the sensor 102 that detected the changed additional feature 114 are excluded from the kinematics fusion and the kinematics fusion is repeated using the remaining kinematics features 112 to obtain an improved fusion object 120.

Possible embodiments of the invention are summarized again below or presented with a slightly different choice of words.

A fusion improvement is presented.

A computer-controlled car requires a perception system. Since there are many different parameters that need to be captured by the sensors, and also for better performance in precision and reliability, it is common to include all sensors in a so-called "fusion" which attempts to match the inputs from all sensors and one to output a consolidated image of the environment.

Fusion refers in particular to object fusion, in contrast to other types that are possible.

Typical implementations of such fusion systems include an association algorithm that attempts to establish matches between already tracked objects and new incoming objects. The association is based on a type of "distance measurement" that assigns a value to the similarity between the already tracked objects and the newly incoming objects. This matching is typically done on the basis kinematic features of the objects based on position, speed, acceleration, etc. This matching rarely depends on other features of the objects, such as color, as only the camera can determine the color.

Some fusion implementations use a ring buffer for the incoming measurements to handle out-of-order measurement information, for example from a source such as a roadside camera unit that may send old information due to processing and communication delays.

Valuable available information, such as the color of the vehicle (from the camera), the metal mass (from the radar), and others, is typically lost during this fusion process and/or not taken into account for the association process. Also, only the tracked information, which is a merged information of all inputs up to that point, can be taken into account when viewing.

A fusion is presented here in which at least a first ring buffer is used to deal with out-of-order information, additionally a second ring buffer is used that contains all the information available from each measurement. In particular, all information that is not used in the association/fusion process is stored in the second ring buffer in parallel to the first ring buffer, which contains the information relevant to the algorithms. In particular, for each tracked object, not only the relevant tracking information is tracked, but also a list of references to the original data in the ring buffers.

The availability of this information makes it possible to improve and/or enrich the output of the merger with all collected data that does not play a role in the merger. Furthermore, this available information can be used for better fusion. For example, if a tracked object is determined to be a product of measurements where, for example, the color changes from one measurement to another measurement, deviant measurements can be eliminated (e.g. using an algorithm in the RANSAC family).

The incoming measurements are converted into relevant information for association/fusion/tracking algorithms and "additional information divided up. Both information is stored in ring buffers that are independent for each sensor.

The association/fusion path corresponds to that of a state-of-the-art fusion system. The only difference is that for each tracked object a list of pointers/references to the original incoming information is stored. Each time measurement information is used to update the tracked object, a pointer is added to the list.

At the end of the fusion algorithms, the list of original information is scanned and all "not relevant to the algorithms" information is added to the output. Plausibility checks are carried out to ensure that the information used for the object is consistent. For example, if the object has been recognized once as a bicycle and once as a truck in history, the information can be classified as "less reliable" by, for example, increasing the variance or decreasing the confidence rating. Outliers can be filtered out using RANSAC or other similar algorithms.

Finally, it should be noted that terms such as "comprising", "comprising", etc. do not exclude other elements or steps, and terms such as "a" or "an" do not exclude a plurality. Reference symbols in the claims are not to be viewed as a limitation.

Claims

Expectations

1. Method for sensor data fusion, wherein object lists (108) with features of detected objects (106) are read in from at least two sensors (102), from the object lists (108) each kinematics features (112) and additional features (114) of the detected objects (106) are extracted and a common identifier (116) is assigned to kinematic features (112) and additional features (114) recorded together in time, the kinematic features (112) recorded by the various sensors (102) for an object (106) using kinematic fusion a fusion object (120), the assigned identifiers (116) of the kinematics features (112) being obtained during the kinematics fusion, the additional features (114) being obtained using the identifiers (116) of the kinematics features (112) merged to form the fusion object (120). ) can be added to the fusion object (120) with the corresponding identifiers (116).

2. Method according to claim 1, in which the additional features (114) are stored with their identifiers (116) in a feature buffer.

3. The method according to claim 2, in which the additional features (114) in the feature buffer are deleted after a predefined storage period.

4. Method according to one of the preceding claims, in which the fusion object (120) is checked for plausibility using the added additional features (114).

5. The method according to claim 4, in which the fusion object (120) is rated with reduced reliability if an inconsistency in the additional features (114) is detected during plausibility checking.

6. Method according to one of claims 4 to 5, in which kinematic features (112) are discarded whose additional features (114) have an inconsistency.

7. Method according to one of the preceding claims, in which the kinematics features (112) are stored with their identifiers (116) in a kinematics buffer, the identifiers (116) defining a chronological order of the kinematics features (116), the kinematics features (112) the various sensors (102) are merged into the fusion object (120) during the kinematics fusion using the chronological order.

8. The method according to claim 7, in which kinematic features (112) read with a time delay are arranged in the chronological order using the identifiers (116).

9. Network (100) for sensor data fusion, wherein the network (100) is configured to execute, implement and / or control the method according to one of the preceding claims in corresponding devices.

10. Computer program product which is designed to instruct a processor when executing the computer program product to execute, implement and/or control the method according to one of claims 1 to 8.

11. Machine-readable storage medium on which the

Computer program product is stored according to claim 10.