CN116226782A

CN116226782A - Sensor data fusion method, device, equipment and storage medium

Info

Publication number: CN116226782A
Application number: CN202310001855.5A
Authority: CN
Inventors: 李俊慧
Original assignee: Apollo Intelligent Connectivity Beijing Technology Co Ltd
Current assignee: Apollo Intelligent Connectivity Beijing Technology Co Ltd
Priority date: 2022-12-22
Filing date: 2023-01-03
Publication date: 2023-06-06

Abstract

The disclosure provides a sensor data fusion method, device, equipment and storage medium, relates to the technical field of data processing, and particularly relates to the technical field of sensing and automatic driving. The specific implementation scheme is as follows: performing association fusion processing on the target data of the first sensor and the fusion data of the first sensor to obtain fusion target data of the first sensor; and carrying out association fusion processing on the fusion target data of the first sensor and the global fusion data to obtain updated global fusion data. According to the embodiment of the disclosure, the data of the plurality of sensors are associated and fused, so that the detection performance of the plurality of sensors can be fully utilized, and the method is beneficial to being suitable for more application scenes.

Description

Sensor data fusion method, device, equipment and storage medium

Technical Field

The invention relates to the technical field of data processing, in particular to the technical field of perception and automatic driving.

Background

An important premise for an automated or unmanned vehicle to be able to run safely and comfortably is accurate and reliable environmental awareness. The context awareness system is a sensor based system. The environmental awareness system is required to have high accuracy and high reliability in order to correctly detect various traffic participants at front-rear, left-right, adjacent, etc. positions of the vehicle, and obstacles such as stationary backgrounds. Thus, many environmental awareness sensors need to be used, covering all possible angles. Because of different application requirements, the type, number, installation location, etc. of sensors that are installed by design choice may be different, and many different sensors may collect a large amount of data.

Disclosure of Invention

The disclosure provides a sensor data fusion method, device, equipment and storage medium.

According to an aspect of the present disclosure, there is provided a sensor data fusion method including:

performing association fusion processing on the target data of the first sensor and the fusion data of the first sensor to obtain fusion target data of the first sensor;

and carrying out association fusion processing on the fusion target data of the first sensor and the global fusion data to obtain updated global fusion data.

According to another aspect of the present disclosure, there is provided a sensor data fusion apparatus including:

the first processing module is used for carrying out association fusion processing on the target data of the first sensor and the fusion data of the first sensor to obtain the fusion target data of the first sensor;

and the second processing module is used for carrying out association fusion processing on the fusion target data of the first sensor and the global fusion data to obtain updated global fusion data.

According to another aspect of the present disclosure, there is provided an electronic device including:

at least one processor; and

a memory communicatively coupled to the at least one processor; wherein,,

The memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of the embodiments of the present disclosure.

According to another aspect of the present disclosure, there is provided a non-transitory computer-readable storage medium storing computer instructions for causing the computer to perform a method according to any one of the embodiments of the present disclosure.

According to another aspect of the present disclosure, there is provided a computer program product comprising a computer program which, when executed by a processor, implements a method according to any of the embodiments of the present disclosure.

According to the embodiment of the disclosure, the data of the plurality of sensors are associated and fused, so that the detection performance of the plurality of sensors can be fully utilized, and the method is beneficial to being suitable for more application scenes.

It should be understood that the description in this section is not intended to identify key or critical features of the embodiments of the disclosure, nor is it intended to be used to limit the scope of the disclosure. Other features of the present disclosure will become apparent from the following specification.

Drawings

The drawings are for a better understanding of the present solution and are not to be construed as limiting the present disclosure. Wherein:

FIG. 1 is a flow diagram of a sensor data fusion method according to an embodiment of the present disclosure;

FIG. 2 is a flow chart diagram of a sensor data fusion method according to another embodiment of the present disclosure;

FIG. 3 is a flow chart diagram of a sensor data fusion method according to another embodiment of the present disclosure;

FIG. 4 is a flow chart diagram of a sensor data fusion method according to another embodiment of the present disclosure;

FIG. 5 is an architectural diagram of a multi-sensor fusion of an embodiment of the present disclosure

FIG. 6 is a diagram of a single sensor perception process of an embodiment of the present disclosure;

FIG. 7 is a flow chart of a process of raw data-based perceptual fusion between sensors in an embodiment of the present disclosure;

FIG. 8 is a flowchart of an original data-based associative fusion process in accordance with an embodiment of the present disclosure;

FIG. 9 is a schematic diagram of track relevance determination in accordance with an embodiment of the present disclosure;

FIG. 10 is a schematic structural view of a sensor data fusion device according to an embodiment of the present disclosure;

FIG. 11 is a schematic structural view of a sensor data fusion device according to another embodiment of the present disclosure;

fig. 12 is a block diagram of an electronic device used to implement an embodiment of the present disclosure.

Detailed Description

Exemplary embodiments of the present disclosure are described below in conjunction with the accompanying drawings, which include various details of the embodiments of the present disclosure to facilitate understanding, and should be considered as merely exemplary. Accordingly, one of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope of the present disclosure. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.

FIG. 1 is a flow diagram of a method of sensor data fusion according to an embodiment of the present disclosure, which may include:

s101, performing association fusion processing on target data of a first sensor and fusion data of the first sensor to obtain fusion target data of the first sensor;

s102, performing association fusion processing on fusion target data of the first sensor and global fusion data to obtain updated global fusion data.

In embodiments of the present disclosure, many sensors may be included in an environmental awareness system of a vehicle or the like. For example, various sensors at different mounting locations or different locations may collect a large amount of data. The types of sensors may include, for example, vision sensors, lidar, millimeter wave radar, and the like. A vision sensor such as a camera may collect images, a lidar may collect point clouds, and a millimeter wave radar may collect point clouds. Each sensor can perform sensing processing independently. For example, a perception process flow may include: target detection, target classification, scene semantic segmentation, target dynamic and static identification and the like; performing association tracking on the target; estimating the existence of the target; filtering the targets according to target information such as existence or confidence; outputting target data according to a target protocol; the module output is then output according to the frame protocol.

In the embodiment of the disclosure, the target data of the sensor can be obtained by performing sensing processing based on the original data of the sensor. The object data of the sensor may include attribute information of one or more objects perceived by the sensor, such as position, shape, orientation, speed, acceleration, class, etc. of the object. The fused data of the first sensor may also include attribute information of one or more targets perceived by the first sensor and other sensors.

And carrying out association fusion processing by utilizing the original data and the fusion data of the single sensor to obtain fusion target data of the single sensor. Global fusion data can then be obtained using the fusion target data of the plurality of sensors. The global fusion data not only can comprise fusion target data of a plurality of sensors, but also can comprise fusion track data of a plurality of sensors. And then carrying out association fusion processing according to the fusion target data of the single sensor and the global fusion data, and updating the global fusion data. Therefore, the data of the plurality of sensors are associated and fused, so that the detection performance of the plurality of sensors can be fully utilized, and the method is beneficial to being suitable for more application scenes.

FIG. 2 is a flow diagram of a method of sensor data fusion according to another embodiment of the present disclosure, which may include one or more features of the sensor data fusion method of the above embodiments, in one implementation, the method further includes:

s201, fusing the target data of the first sensor and the target data of at least one second sensor to obtain fused data of the first sensor.

In the embodiment of the disclosure, the fusion data of the first sensor can be obtained based on the perception processing after the fusion of the original data of the first sensor and the original data of other sensors. Alternatively, the fusion data of the first sensor may be obtained based on the fusion of the target data of the first sensor with the target data of other sensors. Take laser radar, vision sensor, millimeter wave radar, etc. as examples. The laser radar sensor is a first sensor, and the vision sensor and the millimeter wave radar are a second sensor. The target data of the laser radar sensor is { L }, the target data of the vision sensor is { C }, and the target data of the millimeter wave radar sensor is { R }. The fusion data obtained by fusion processing of the laser radar sensor and the vision sensor is { LC }, and the fusion data obtained by fusion processing of the laser radar sensor and the millimeter wave radar is { LR }. The target data of different sensors are fused, so that the overall detection performance can be improved, and the applicable scene is enlarged.

In one embodiment, S201 performs fusion processing on the target data of the first sensor and the target data of at least one second sensor to obtain fused data of the first sensor, including:

and fusing the target data of the first sensor and the target data of the plurality of second sensors in pairs after extracting features by obtaining expression modes based on projection relations, so as to obtain a plurality of fusion data of the first sensor.

In the embodiment of the disclosure, the sensors can be fused with each other based on the original data. The two-to-two fusion mode can comprise: and extracting features by obtaining expression modes based on projection relations or respectively extracting features from the original data and fusing the features together. Because of different sensors and different detection performances, applicable scenes are different, and the performance can be improved and the applicable scene range can be enlarged by fusion processing of the two sensors based on the original data.

Taking laser radar, vision sensor and millimeter wave radar as examples, the two can be mutually fused based on the original data. The laser radar point cloud and the visual image are fused, so that the class precision of the target can be improved compared with single laser radar perception processing; compared with the single vision sensor sensing processing, the method can improve the position accuracy of the target. Compared with single-laser radar sensing processing, the laser radar point cloud and millimeter wave radar point cloud fusion can improve the detection performance of targets in special weather such as rain, fog, dust and the like; compared with single millimeter wave radar sensing processing, the method can improve the position accuracy of the target. Compared with single vision sensor fusion processing, the vision image and millimeter wave radar point cloud fusion processing can improve the position accuracy of a target and the detection performance in environments such as weak illumination or strong illumination; compared with single millimeter wave radar sensing processing, the method can improve the class precision of the target. After the original data between the sensors are fused, target detection, target classification, target dynamic and static identification and the like can be performed, the subsequent processing steps are similar to the perception processing of a single sensor, and finally the module output can be output according to a frame protocol.

In one embodiment, S101 performs a correlation fusion process on the target data of the first sensor and the fusion data of the first sensor to obtain fused target data of the first sensor, where the method includes:

comparing the target data of the first sensor with different types of fusion data according to a correlation mode to determine whether an expected target exists;

under the condition that a plurality of expected targets exist in a correlation mode, fusing the plurality of expected targets in the correlation mode to obtain fused targets;

and obtaining fusion target data of the first sensor according to the expected targets or the fusion targets corresponding to different association modes.

In the disclosed embodiments, the fusion of a first sensor with a second, different sensor may result in different types of fusion data. For example, the target data of the laser radar sensor is { L }, the fusion data obtained by fusion processing of the laser radar sensor and the vision sensor is { LC }, and the fusion data obtained by fusion processing of the laser radar sensor and the millimeter wave radar is { LR }. There may be a variety of ways of correlation between { L }, { LC }, and { LR }. One or more desired targets may or may not be obtained based on one manner of association. If multiple desired targets are obtained based on a correlation manner, the desired targets can be fused to obtain a fused target. And then, summarizing expected targets or fusion targets corresponding to each association mode, and obtaining the fusion target data of the first sensor. The fusion target data may include a fusion target list of the first sensor. The list may include information such as identification, attributes, etc. of the fusion target.

In the embodiment of the disclosure, the expected targets can be associated and matched from the target data and the fusion data of the first sensor in an association mode, and a plurality of expected targets in the same association mode are fused, so that the fusion target data can be obtained, the overall detection performance is improved, and the target detection scene is enlarged.

In one embodiment, the association means includes at least one of:

the target data of the first sensor is not associated with all of the fused data of the first sensor;

the first fusion data is not related to the target data of the first sensor and the target of the second fusion data;

the second fusion data is not related to the target data of the first sensor and the target of the first fusion data;

the target data of the first sensor has a target associated with the first fused data but not with the second fused data;

the target data of the first sensor has a target associated with the second fused data but not with the first fused data;

the first fused data has an object associated with the second fused data but not with the object data of the first sensor;

the target data of the first sensor having an associated target with all of the fused data of the first sensor;

The first fusion data and the second fusion data are fusion data obtained by fusing the first sensor and a different second sensor.

For example, the target data of the laser radar sensor is { L }, the fusion data obtained by fusion processing of the laser radar sensor and the vision sensor is { LC }, and the fusion data obtained by fusion processing of the laser radar sensor and the millimeter wave radar is { LR }. The association is exemplified as follows:

mode one: no object in { L } that is associated with any of { LC } and { LR };

mode two: no object in { LC } that is associated with either of { L } and { LR };

mode three: no object in { LR } that is associated with either of { L } and { LC };

mode four: objects in { L } and { LC } associated with objects in { LR } but not with objects in { LR };

mode five: targets in { L } and { LR } that are associated with targets in { LC };

mode six: targets in { LC } and { LR } associated with targets in { L };

mode seven: targets associated in both { L }, { LC }, and { LR }.

In the embodiment of the disclosure, the association manner of the target data of the first sensor with the first fusion data and the second fusion data is merely an example and not a limitation, and in an actual application scenario, the target data of the first sensor may be associated with more fusion data. The manner of association may be deduced in terms of permutation and combination, and is not exhaustive herein. Through various association modes, expected targets can be associated and matched from the target data of the first sensor and the fusion data, so that the fusion target data is obtained, the overall detection performance can be improved, and the applicable scene is enlarged.

FIG. 3 is a flow diagram of a method of sensor data fusion according to another embodiment of the present disclosure, which may include one or more features of the sensor data fusion method of the above embodiments, in one implementation, the method further includes:

s301, obtaining fusion track data of the first sensor according to fusion target data of the first sensor of a plurality of frames;

s302, obtaining global fusion track data according to the global fusion target data of the multi-frame.

In the embodiment of the disclosure, the fusion target data of a single frame may include tracking information of each fusion target, for example, the acquisition time and position of the fusion target in the current frame, and the like. And obtaining the fusion track data according to the multi-frame fusion target data. Global fusion target data can be obtained according to the fusion target data of the plurality of sensors. According to the global fusion target data of the multi-frame, global fusion track data can be obtained. Based on the relation between the target and the track, the association fusion tracking processing can be performed. For example, fusion target lists based on different sensors are fused together through the association results of targets and tracks or tracks and tracks, a global fusion target list is generated, and track management is carried out on global fusion targets in the global fusion target list.

Fusion results based on different sensor raw data may be asynchronous and may be unordered. The target and track or track and track association fusion tracking algorithm used can maintain a stable global track over time and maintain consistency even if one target moves in the field of view of multiple sensors. The target or track based associative fusion tracking process may include: associating the fusion target or track with a global fusion target or track, and determining which targets derived from different sensor raw data are likely to represent the same target in reality; according to the association processing result, carrying out fusion processing of various attention attributes on the associated targets; and finally, tracking the fusion target.

In one embodiment, S102 performs an association fusion process on the fusion target data of the first sensor and the global fusion data to obtain updated global fusion data, including:

obtaining an incidence matrix according to the fusion target data and the global fusion data of the first sensor;

and checking and updating the association matrix by using at least one of tracking information, association information between sensors, category information, dynamic and static properties and speed information in the fusion target data.

In an embodiment of the present disclosure, the element in the association matrix may represent whether a certain target or track of the fused target data of the first sensor has an association relationship with a certain target or track in the global fused data. For example, if the fused target data of the first sensor includes M targets and the global fused data includes N targets, the correlation matrix may include m×n elements. A variety of different sensors may result in different correlation matrices. For example, millimeter wave radar corresponds to an M1 x N correlation matrix, camera corresponds to an M2 x N correlation matrix, and lidar M3 x N correlation matrix. After the incidence matrix is established, the accuracy of elements in the incidence matrix can be checked by utilizing the original data of the sensor or various attribute information in the target data, so that more accurate incidence relation between targets or tracks is obtained. Furthermore, the global fusion data can be updated more accurately.

In one embodiment, the elements in the correlation matrix are used to represent at least one of:

the correlation distance between the target in the fusion target data of the first sensor and the target in the global fusion data;

the correlation distance between the target in the fusion target data of the first sensor and the track in the global fusion data;

The associated distance between the track in the fusion track data of the first sensor and the track in the global fusion data;

the associated distance of the track in the fused track data of the first sensor and the target in the global fused data.

In the embodiment of the present disclosure, an element in the initially constructed association matrix may represent an association distance between two objects. The correlation distance can be the correlation distance between the targets or the correlation matrix between the targets and the track. In general, the closer the association distance, the greater the likelihood of belonging to the same target. Therefore, the association matrix is constructed based on the association distance, and the association relation between the targets or tracks can be accurately reflected.

In one embodiment, when the association distance between two objects represented by the elements in the association matrix is greater than or equal to a threshold value, the corresponding element in the association matrix is a first value; and when the association distance is smaller than the threshold value, the corresponding element in the association matrix is equal to the association distance divided by the threshold value.

For example, the association distance L1 between the target in the fused target data of the first sensor and the target in the global fused data is greater than the first threshold T1, and the elements corresponding to the two are 0. Otherwise, the value of the element is equal to L1/T1.

For another example, the association distance L2 between the target in the fused target data of the first sensor and the track in the global fused data is greater than the second threshold T2, and the elements corresponding to the two are 0. Otherwise, the value of the element is equal to L2/T2.

For another example, the correlation distance L3 between the track in the fused track data of the first sensor and the track in the global fused data is greater than the third threshold T3, and the elements corresponding to the track and the track are 0. Otherwise, the value of the element is equal to L3/T3.

For another example, the association distance L4 between the track in the fused track data of the first sensor and the target in the global fused data is greater than the fourth threshold T4, and the elements corresponding to the track and the target in the global fused data are 0. Otherwise, the value of the element is equal to L4/T4.

In the embodiment of the disclosure, the first value may be 0, or may be other values, such as 1, 2, etc. The value of the element of the incidence matrix is determined based on the incidence distance, and the established incidence matrix can accurately reflect the incidence relation between targets or tracks.

In one embodiment, S102 performs an association fusion process on the fusion target data of the first sensor and the global fusion data to obtain updated global fusion data, and further includes:

performing attribute fusion processing on the fusion target data of the single sensor and the global fusion target data according to the association pair obtained by the updated association matrix;

And updating the global fusion track data according to the global fusion target data after the attribute fusion processing.

In the embodiment of the disclosure, the elements of the updated association matrix may represent which targets or tracks have an association relationship therebetween. Targets or tracks having an association relationship may be referred to as an association pair. For example, if the value of an element is not 0, the target A1 in the fused target data of the first sensor represented by the element has an association relationship with the target A2 in the global fused target data. The attribute fusion process may be performed on the target A1 and the target A2. For another example, if the value of an element is not 0, the target A1 in the fused target data of the first sensor represented by the element has an association relationship with the track B2 in the global fused target data. The attribute fusion process may be performed on the attributes of the target A2 included in the target A1 and the track B2 to update the track of the target A2. For example, track B2 includes tracks for target A2 at times t1 to t2, and the attribute at time t3 represented by target A1 may be added to track B2, with the updated tracks including data at times t1 to t 3.

In the embodiment of the disclosure, if the fusion target of a certain sensor is not matched with the associated track or target in the global fusion data, one track or target can be newly established for subsequent association and tracking. By means of attribute fusion and track updating, target tracking can be performed more rapidly and accurately.

In one embodiment, the attribute fusion process includes at least one of: fusing position and shape; orientation fusion; category fusion; motion fusion; presence fusion; semantic fusion. Each attribute can be fused separately, and different association pairs can be different for the attributes to be fused. The associated pairs corresponding to different sensors may also differ in the properties that need to be fused. The fusion of more types of sensors can be supported through the fusion of multiple attributes, and the target detection performance is improved.

In one embodiment, the location shape fusion comprises: filtering or smoothing the position observed values from different sensors, and performing time sequence estimation on the position observed values with low confidence coefficient to obtain the position of the fusion target; alternatively, a high-precision shape observation value is selected from shape observations from different sensors, and the high-precision shape observation value is corrected by using other shape observations to obtain the shape of the fusion target. For example, according to the fact that a plurality of sensors obtain that a certain target includes P1, P2 and P3 according to time sequence, the confidence of P1 and P3 is high, the confidence of P2 is low, P1 and P3 can be used for estimating P2, and the value of P2 can be updated. For another example, if the accuracy of the shape observation value detected by the camera is highest, the shape observation value detected by the camera may be selected such as length, width, height, etc., and then the shape observation value detected by the millimeter wave radar may be used to correct the shape observation value detected by the camera such as width.

In one embodiment, the orientation fusion comprises: and obtaining the orientation of the fusion target according to the statistical information of the orientation observation value. For example, the statistics may include accuracy statistics, forward and reverse hopping statistics, and the like. And selecting the orientation detected by the sensor with higher precision or more accurate forward and reverse jump as the orientation of the fusion target according to the requirements.

In one embodiment, the class fusion includes: and obtaining the classification probability corresponding to the classification vector according to the perception processing result of the original target data, and obtaining the category of the fusion target according to the classification probability. For example, the fusion targets may be classified using machine learning or data mining methods, etc., and the classification of the fusion targets may be determined, for example, pedestrians, vehicles, animals, ground, guardrails, green plants, etc.

In one embodiment, the motion fusion comprises: and estimating the motion state quantity of the fusion target according to the motion model and a Kalman filtering algorithm. For example, the motion model of the fusion target may include motion state quantities such as position, velocity, acceleration, direction angle, angular velocity, and the like. The Kalman filtering algorithm can be used for estimating the motion state quantity of the fusion target such as the position, the speed, the acceleration, the direction angle, the angular speed and the like at a certain moment based on the motion model.

In one embodiment, the presence fusion comprises: and calculating the existence probability of the fusion target. The existence probabilities can be calculated for the same target based on the data of the sensors from different sources, and then the existence probabilities of the fusion targets are calculated comprehensively. The existence probability of the fusion target can also be directly calculated based on the fusion target data of the sensors with different sources.

In one embodiment, the semantic fusion includes: and obtaining a dynamic and static shielding region according to the original target data, and providing vanishing semantics for the fusion target entering the shielding region. For example, an area behind the vehicle may not be included in the point cloud of the millimeter wave radar in front of the vehicle, which belongs to the shielding area. If a fusion target enters the shielding area, the semantics can be disappeared for the fusion target.

By fusing various attributes of the fusion targets in the association pair, data of more types of sensors can be comprehensively utilized, accuracy of target detection performance is improved, and applicable scenes of target detection are expanded.

FIG. 4 is a flow diagram of a method of sensor data fusion according to another embodiment of the present disclosure, which may include one or more features of the sensor data fusion method of the above embodiments, in one implementation, the method further includes:

S401, performing target filtering according to the associated information, attribute information and tracking information of the fusion target; the fusion target comprises a fusion target in the updated global fusion data.

In embodiments of the present disclosure, the association information of the fusion target may include which sensors the fusion information is associated with, and so on. If a fusion target is associated with fewer sensors or less accurate, the fusion target may be filtered out. The attribute information of the fusion target may include the attribute information fused according to the above attribute fusion manner. If the fused attribute information is obviously unreasonable, for example, the speed is obviously beyond a reasonable range, the fused target can be filtered out. The tracking information of the fusion target may include tracking information of each frame. If the tracking information is significantly unreasonable, e.g., the jump in position between two or more frames is significantly out of a reasonable range, the fusion target may be filtered out.

In the embodiment of the disclosure, the specific operation of filtering out the fusion target may include deleting the fusion target from the fusion target data or the global fusion data of the first sensor, or may include not deleting the fusion target but not pushing the fusion target to a specific application program. The targets which do not meet the requirements can be deleted by filtering the targets, so that interference targets are reduced, and the accuracy of the overall detection result is improved.

In one embodiment, as shown in fig. 4, further comprising:

s402, outputting a concerned fusion target in the application program according to the data source of the application program and the updated global fusion data.

In the embodiment of the disclosure, the data sources of the application program may include data of a host vehicle, a high-precision map, road boundaries, an occupied grid map (dynamic or static), and the like, and according to the requirements of a specific application scene, the fusion target focused by the application program may be output according to the updated global fusion data. For example, the position of an obstacle detected by a host vehicle multisensor, a tracking track, and the like are displayed in a high-precision map. By combining with the data source of the application program, the application scene which is richer can be expanded based on the multi-sensor fusion data.

Since there are often cases where multiple sensors are not synchronized with each other, significant post-processing and communication delays may be required. The sensor fusion method provided by the embodiment of the disclosure is a general multi-sensor asynchronous fusion method, has modularity, practicability and expandability, can enable different application requirements or different sensor configurations to be applicable to the same fusion architecture, and is easy to realize in automobile application. The method may be performed based on a flow as shown in fig. 5.

First, a frame protocol and a target protocol used by the method are described.

The frame protocol, mainly comprising the following examples: timestamp information, coordinate system conversion information, raw data information, raw quality information, raw data semantic information, object list information, association information of fusion objects between objects and sensors, and information associated with global fusion objects.

The targets in the target list can be obtained according to a target protocol, and the content included in the target protocol is exemplified as follows: target information (target position, size, shape, etc.), target tracking information (tracking number, state quantity, covariance matrix, etc.), target category information (category, classification probability corresponding to the classification vector), confidence information of the target, target presence information (presence probability), dynamic and static information of the target, raw data information corresponding to the target, etc.

As shown in fig. 5, the multi-sensor asynchronous fusion method may include the steps of:

s1, perception processing based on original data. Two main categories are known:

(1) Each single sensor senses, i.e., each sensor independently senses, as shown in fig. 6. The data output by the common sensor mainly comprises data of a primary level or data of a target level. If the output is raw level data, then the target level data needs to be perceptually processed, as shown in FIG. 6. For example, the laser radar sensor outputs original point cloud data generally, and target data is required to be obtained through processing; millimeter wave radar, different manufacturers provide different outputs, typically providing source point cloud data and target data; the visual sensor, typically image data, needs to be perceptually processed to obtain the target data. The conventional perception processing flow mainly comprises the following steps: target detection, target classification, scene semantic segmentation, target dynamic and static identification and the like; performing association tracking on the target; estimating the existence of the target; filtering the targets according to the existence or confidence; outputting target data according to a target protocol; the perception processing module output may be output in accordance with a frame protocol.

(2) The sensors are fused based on the original data two by two, as shown in fig. 7. Raw data fusion between different sensors is commonly used to include two types: the expression mode extraction features are obtained based on the projection relation or the features are respectively extracted from the original data and fused together. After the original data among the sensors are fused, sensing processing such as target detection, target classification, target dynamic and static identification and the like is performed. These sensing process steps are similar to the single sensor sensing process, and the final sensing process module output may be output according to a frame protocol.

S2, target association fusion processing based on the original data.

As shown in fig. 8, the original data corresponding to the object oi is included in the object list of the frame information obtained by the sensor ni sensing process, and the original data corresponding to the object oj is included in the object list of the frame information obtained by the sensor ni and the sensor nj sensing fusion process based on the original data. The association fusion processing based on the sensor ni raw data comprises the following steps: if the target oi is identical to the original data corresponding to target oj by a certain ratio, and the correlation properties of target oi and target oj may have consistency, then target oi and target oj are associated. Fusion is performed from the object oi and the object oj to obtain a fused object oij, and semantic information (e.g., point category: object, ground, guardrail, green plant, noise, etc.) based on the raw data of the sensor ni and the sensor nj may be updated, respectively. And finally, outputting the fusion target based on the original data of the sensor ni by the target association fusion processing module according to a frame protocol. In addition, frame information is obtained based on sensing processing of the sensor nj, frame information is obtained based on sensing fusion processing of the sensor ni and the sensor nj based on original data, and association fusion processing based on the original data of the sensor nj can be performed. The fusion target based on the sensor nj raw data may then be output in accordance with the frame protocol. In the case of three or more types of sensors, see FIG. 8 and so on.

Taking laser radar, a vision sensor and millimeter wave radar as examples, the three types of sensors are used for describing the correlation fusion result based on the original laser radar point cloud data. The single sensors respectively perform sensing processing to obtain target data which is laser { L }, camera { C }, and radar { R }. The laser radar sensor and the vision sensor are fused to obtain target data { LC }, and the laser radar sensor and the millimeter wave radar are fused to obtain target data { LR }. And (3) association result description: the number 1 after the letter indicates an association, and 0 indicates no association. { L }, { LC } and { LR }, obtaining { L1 LC0 LR0, L0 LC1 LR0, L0 LC0 LR1, L1 LC1 LR0, L0 LC1 LR1, L1 LC1 LR1}, based on the original laser radar point cloud data correlation matching process.

Wherein, each association mode represents the following targets:

l1×lc0×lr0 denotes a target in { L } which is not associated with a target in both { LC } and { LR };

l0×lc1×lr0 denotes a target in { LC } which is not associated with a target in both { L } and { LR };

l0.lc0.lr1 represents a target in { LR } that is not associated with a target in both { L } and { LC };

l1×lc1×lr0 represents a target in { L } and { LC } associated with a target in { LR } but not associated with a target in { LR };

l1×lc0×lr1 denotes a target in { L } and { LR } associated with a target in { LC };

L0×lc1×lr1 denotes a target in { LC } and { LR } associated with a target in { L } but not associated with a target in { L };

l1×lc1×lr1 denotes an object associated with each of { L }, { LC }, and { LR }.

A specific example is as follows:

there are 4 targets in { L }: l1, l2, l3, l4;

there are 4 targets in { LC }: lc1, lc2, lc3, lc4;

there are 4 targets in LR: lr1, lr2, lr3, lr4;

{ L } and { LC } are related to obtain a related pair (L2, LC 1) (L3, LC 4);

{ L } and { LR } are related to obtain a related pair (L2, LR 3), (L4, LR 2);

{ LC } and { LR } are related to obtain a related pair (LC 1, LR 3), (LC 2, LR 4);

correlation results: (l 1), (lc 3), (lr 1), (l 3, lc 4), (l 4, lr 2), (lc 2, lr 4), (l 2, lc1, lr 3);

the corresponding relation between each association mode and the association result is as follows:

L1*LC0*LR0：(l1)；

L0*LC1*LR0：(lc3)；

L0*LC0*LR1：(lr1)；

L1*LC1*LR0：(l3,lc4)；

L1*LC0*LR1：(l4,lr2)；

L0*LC1*LR1：(lc2,lr4)；

L1*LC1*LR1：(l2,lc1,lr3)；

the last four types (i.e. on the object association matching) of the association result respectively comprise a plurality of objects, and fusion processing can be performed respectively to obtain fusion objects.

S3, performing association fusion tracking processing based on the target or the track.

In the step, the fusion target list obtained based on the original data of different sensors is fused together through the association results of the targets and the tracks, or through the association results of the tracks and the tracks, so as to generate global fusion target list information. And performing track management on the global fusion target. And managing the scene, wherein the scene comprises the contents such as global fusion targets, original data, semantic information of the original data and the like.

(1) And (3) performing association matching processing based on the targets or tracks.

The correlation distance threshold is initially set by calculating the correlation distance between the fusion target or track obtained based on the original data of different sensors and the global fusion target or track, the correlation is judged in the threshold, the value is the correlation distance divided by the distance threshold, and the value is 0 otherwise, so that the correlation matrix is obtained. In the association process, the attribute is used to check the association accuracy. Such as whether the original tracking information of the fusion target is consistent; fusing the association information between the original sensors of the targets, and judging whether the association information is consistent; category information, calculating the similarity between two category vectors, namely obtaining a similarity measure, and applying the similarity measure to an association matrix as a certain weighting value; such as dynamic and static properties or speed information, whether the dynamic and static states of the target have consistency, checking association accuracy and the like. And the correlation matrix is matched optimally one-to-one according to a common matching algorithm.

As shown in fig. 9, it is possible to compare whether the track of the established target track (global) and the track of the new frame of target data (single frame) are associated with the target. If so, filtering the new frame of target data, and updating the track to obtain the target track. Otherwise, unassociated track maintenance may be performed, creating a new track based on the unassociated new targets. Then, whether to delete the track is judged based on the tracking loss times. If the number of times of loss is excessive, the new track can be deleted to continuously compare with other target tracks, otherwise, the new track can be reserved as the target track to participate in the subsequent comparison.

(2) Fusion processing based on targets or tracks.

And carrying out fusion processing on the targets on the association according to the association processing result. The fusion process can be divided into: fusing position and shape; orientation fusion; category fusion; motion fusion, including velocity, acceleration, dynamic and static fusion; presence fusion, including presence probability fusion; semantic fusion, including scene semantic, occlusion semantic, and vanishing semantic fusion; fusion of other attributes of interest.

Position and shape fusion: for the position, based on the results of the source-different processing, as the observed value, the low-confidence position is subjected to timing estimation by using a filtering processing or a smoothing processing. For the shape, the shape accuracy varies depending on the result from the different sensors, and the result with high accuracy is selected from the observed values and corrected using other observed values.

Orientation fusion: and comprehensively considering the orientation of the fusion target according to the statistical information of the observed value. For example, based on the target of the laser radar original data, the orientation information precision is higher, but forward and reverse jump is easy; based on the target of the vision camera, the whole forward and reverse accuracy of the orientation information is high, and the accuracy is poor.

Category fusion: based on the perception processing result of the original data, the classification probability corresponding to the classification vector can be obtained. Some form of machine learning or data mining method is typically used to classify the target data.

Motion fusion, namely constructing a motion module, and estimating acceleration and speed by adopting a common Kalman filter;

presence fusion: the presence of a target is measured by the probability of presence of the target. The presence probability is used to describe the likelihood that the target is actually a true, detectable target, rather than false detection. By estimating the existence probability of the target, the target management and the target selection can be greatly improved, so that the true detection probability is improved, and the false detection probability is reduced. For example, based on Dempster-Shafer evidence theory, fusion target existence probabilities at sensor uncertainty can be modeled.

In the semantic fusion, the semantics are blocked, and the vanishing semantics are provided for the targets entering the blocking area according to the dynamic and static blocking area obtained based on the original data.

(3) And (5) fusing target tracking processing.

And updating the track according to the fusion result.

S4, filtering the fusion target.

And filtering the fusion target, namely filtering out the target which does not accord with the expectations mainly according to the associated information, the fusion attribute information and the tracking information of the fusion target by comprehensively considering the information. For example, if the position of the fusion target is a stable detection overlapping area of a plurality of sensors, but only one sensor is used for processing, the possibility of whether the target is a real target is low; the existence probability of the fusion target obtained by fusing the existence of the fusion target is very low; short tracking time in the target tracking information, and the like.

Examples: in the 30m right ahead of the main vehicle, in a non-blind area, three types of sensors should be detected stably in theory, and the associated information display target of the fusion target is obtained by laser radar perception processing; the perceived processing of the other modules results in a target that is not associated with it, which may be a false detection target.

S5, outputting the attention target and the information according to the requirements by combining other data sources.

Other commonly used data sources include data of a host vehicle, high-precision maps, lane lines, road boundaries, occupied grid maps (dynamic or static), and the like, and output attention targets and information according to application requirements. For example, in public road environments, high-precision maps are often used to match objects to the map, which objects are located in that lane, and which objects are focused on the host vehicle's lane and adjacent lanes.

FIG. 10 is a schematic structural view of a sensor data fusion device according to an embodiment of the present disclosure, which may include:

the first processing module 1001 is configured to perform association fusion processing on target data of a first sensor and fusion data of the first sensor, so as to obtain fusion target data of the first sensor;

And a second processing module 1002, configured to perform association fusion processing on the fusion target data of the first sensor and the global fusion data, so as to obtain updated global fusion data.

Fig. 11 is a schematic structural diagram of a sensor data fusion device according to another embodiment of the present disclosure, which may include one or more features of the obstacle detection device of the above embodiment, and in one possible implementation, the device further includes:

and a third processing module 1003, configured to perform fusion processing on the first sensor target data and the target data of at least one second sensor, to obtain fusion data of the first sensor.

In one possible implementation, as shown in fig. 11, the third processing module 1003 includes:

and the feature fusion submodule 1102 is used for fusing the target data of the first sensor and the target data of the plurality of second sensors in pairs after extracting features based on the projection relation to obtain the expression mode, so as to obtain a plurality of fusion data of the first sensor.

In one possible implementation, as shown in fig. 11, the first processing module 1001 includes:

a target association submodule 1103, configured to compare, in an association manner, whether a desired target exists between target data of the first sensor and different types of fusion data;

The target fusion submodule 1104 is used for fusing a plurality of expected targets of a correlation mode to obtain a fusion target under the condition that the plurality of expected targets exist in the correlation mode;

the data processing sub-module 1105 is configured to obtain the fusion target data of the first sensor according to the desired targets or the fusion targets corresponding to the different association modes.

In one possible embodiment, the association means includes at least one of:

In one possible embodiment, as shown in fig. 11, the apparatus further includes:

the sensor track module 1004 is configured to obtain fused track data of the first sensor according to the fused target data of the first sensor for a plurality of frames;

the global track module 1005 is configured to obtain global fusion track data according to the global fusion target data of the multiple frames.

In one possible implementation, as shown in fig. 11, the second processing module 1002 includes:

a matrix building sub-module 1108, configured to obtain an association matrix according to the fusion target data and the global fusion data of the first sensor;

a matrix updating sub-module 1109 is configured to check and update the correlation matrix using at least one of tracking information, correlation information between sensors, category information, dynamic and static properties, and speed information in the fusion target data.

In one possible implementation, the elements in the correlation matrix are used to represent at least one of:

In one possible implementation manner, when the association distance between two objects represented by the elements in the association matrix is greater than or equal to a threshold value, the corresponding element in the association matrix is a first value; and when the association distance is smaller than the threshold value, the corresponding element in the association matrix is equal to the association distance divided by the threshold value.

In one possible implementation, as shown in fig. 11, the second processing module 1002 further includes:

the attribute fusion submodule 1110 is used for carrying out attribute fusion processing on fusion target data of a single sensor and global fusion target data according to an association pair obtained by the updated association matrix;

the global updating submodule 1111 is configured to update global fusion track data according to the global fusion target data after the attribute fusion processing.

In one possible implementation, the attribute fusion process includes at least one of: fusing position and shape; orientation fusion; category fusion; motion fusion; presence fusion; semantic fusion.

In one possible embodiment, the location shape fusion comprises: filtering or smoothing the position observed values from different sensors, and performing time sequence estimation on the position observed values with low confidence coefficient to obtain the position of the fusion target; or selecting a high-precision shape observation value from shape observation values from different sensors, and correcting the high-precision shape observation value by using other shape observation values to obtain the shape of the fusion target;

the orientation fusion includes: according to the statistical information of the orientation observation value, the orientation of the fusion target is obtained;

the category fusion includes: obtaining a classification probability corresponding to the classification vector according to the perception processing result of the original target data, and obtaining the category of the fusion target according to the classification probability;

the motion fusion includes: estimating the motion state quantity of the fusion target according to the motion model and a Kalman filtering algorithm;

the presence fusion includes: calculating the existence probability of the fusion target;

The semantic fusion includes: and obtaining a dynamic and static shielding region according to the original target data, and providing vanishing semantics for the fusion target entering the shielding region.

the filtering module 1006 is configured to perform target filtering according to the association information, the attribute information, and the tracking information of the fusion target; the fusion target comprises a fusion target in the updated global fusion data.

an output module 1007 is configured to output a fusion target of interest in the application according to the data source of the application and the updated global fusion data.

For descriptions of specific functions and examples of each module and sub-module of the apparatus in the embodiments of the present disclosure, reference may be made to the related descriptions of corresponding steps in the foregoing method embodiments, which are not repeated herein.

In the technical scheme of the disclosure, the acquisition, storage, application and the like of the related user personal information all conform to the regulations of related laws and regulations, and the public sequence is not violated.

According to embodiments of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

Fig. 12 shows a schematic block diagram of an example electronic device 1200 that can be used to implement embodiments of the present disclosure. Electronic devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device may also represent various forms of mobile apparatuses, such as personal digital assistants, cellular telephones, smartphones, wearable devices, and other similar computing apparatuses. The components shown herein, their connections and relationships, and their functions, are meant to be exemplary only, and are not meant to limit implementations of the disclosure described and/or claimed herein.

As shown in fig. 12, the apparatus 1200 includes a computing unit 1201, which may perform various appropriate actions and processes according to a computer program stored in a Read Only Memory (ROM) 1202 or a computer program loaded from a storage unit 1208 into a Random Access Memory (RAM) 1203. In the RAM 1203, various programs and data required for the operation of the device 1200 may also be stored. The computing unit 1201, the ROM 1202, and the RAM 1203 are connected to each other via a bus 1204. An input/output (I/O) interface 1205 is also connected to the bus 1204.

Various components in device 1200 are connected to I/O interface 1205, including: an input unit 1206 such as a keyboard, mouse, etc.; an output unit 1207 such as various types of displays, speakers, and the like; a storage unit 1208 such as a magnetic disk, an optical disk, or the like; and a communication unit 1209, such as a network card, modem, wireless communication transceiver, etc. The communication unit 1209 allows the device 1200 to exchange information/data with other devices via a computer network, such as the internet, and/or various telecommunications networks.

The computing unit 1201 may be a variety of general and/or special purpose processing components having processing and computing capabilities. Some examples of computing unit 1201 include, but are not limited to, a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), various specialized Artificial Intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital Signal Processors (DSPs), and any suitable processor, controller, microcontroller, etc. The computing unit 1201 performs the various methods and processes described above, such as the sensor data fusion method. For example, in some embodiments, the sensor data fusion method may be implemented as a computer software program tangibly embodied on a machine-readable medium, such as storage unit 1208. In some embodiments, part or all of the computer program may be loaded and/or installed onto device 1200 via ROM 1202 and/or communication unit 1209. When a computer program is loaded into RAM 1203 and executed by computing unit 1201, one or more steps of the sensor data fusion method described above may be performed. Alternatively, in other embodiments, the computing unit 1201 may be configured to perform the sensor data fusion method by any other suitable means (e.g., by means of firmware).

Various implementations of the systems and techniques described here above may be implemented in digital electronic circuitry, integrated circuit systems, field Programmable Gate Arrays (FPGAs), application Specific Integrated Circuits (ASICs), application Specific Standard Products (ASSPs), systems On Chip (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs, the one or more computer programs may be executed and/or interpreted on a programmable system including at least one programmable processor, which may be a special purpose or general-purpose programmable processor, that may receive data and instructions from, and transmit data and instructions to, a storage system, at least one input device, and at least one output device.

Program code for carrying out methods of the present disclosure may be written in any combination of one or more programming languages. These program code may be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus such that the program code, when executed by the processor or controller, causes the functions/operations specified in the flowchart and/or block diagram to be implemented. The program code may execute entirely on the machine, partly on the machine, as a stand-alone software package, partly on the machine and partly on a remote machine or entirely on the remote machine or server.

In the context of this disclosure, a machine-readable medium may be a tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. The machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and pointing device (e.g., a mouse or trackball) by which a user can provide input to the computer. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic input, speech input, or tactile input.

The systems and techniques described here can be implemented in a computing system that includes a background component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such background, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), wide Area Networks (WANs), and the internet.

The computer system may include a client and a server. The client and server are typically remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server may be a cloud server, a server of a distributed system, or a server incorporating a blockchain.

It should be appreciated that various forms of the flows shown above may be used to reorder, add, or delete steps. For example, the steps recited in the present disclosure may be performed in parallel, sequentially, or in a different order, provided that the desired results of the disclosed aspects are achieved, and are not limited herein.

The above detailed description should not be taken as limiting the scope of the present disclosure. It will be apparent to those skilled in the art that various modifications, combinations, sub-combinations and alternatives are possible, depending on design requirements and other factors. Any modifications, equivalent substitutions, improvements, etc. that are within the principles of the present disclosure are intended to be included within the scope of the present disclosure.

Claims

1. A method of sensor data fusion, comprising:

performing association fusion processing on target data of a first sensor and fusion data of the first sensor to obtain fusion target data of the first sensor;

2. The method of claim 1, further comprising:

And carrying out fusion processing on the target data of the first sensor and the target data of at least one second sensor to obtain fusion data of the first sensor.

3. The method of claim 2, wherein the fusing the first sensor target data and the target data of the at least one second sensor to obtain the fused data of the first sensor comprises:

4. A method according to any one of claims 1 to 3, wherein performing a correlation fusion process on the target data of the first sensor and the fusion data of the first sensor to obtain the fusion target data of the first sensor comprises:

5. The method of claim 4, wherein the association means comprises at least one of:

the target data of the first sensor and all fusion data of the first sensor are not associated targets;

a target to which the first fused data is not associated with the target data of the first sensor and the second fused data;

the first fused data has an associated target with the second fused data but not with the target data of the first sensor;

the target data of the first sensor has an associated target with all of the fused data of the first sensor;

the first fusion data and the second fusion data are fusion data obtained by fusing the first sensor and a second sensor which is different.

6. The method of any one of claims 1 to 5, further comprising:

Obtaining fusion track data of the first sensor according to fusion target data of the first sensor of a plurality of frames;

and obtaining global fusion track data according to the global fusion target data of the multiple frames.

7. The method according to any one of claims 1 to 6, wherein performing an associative fusion process on the fusion target data of the first sensor and global fusion data to obtain updated global fusion data, includes:

8. The method of claim 7, wherein elements in the correlation matrix are used to represent at least one of:

the correlation distance between the track in the fusion track data of the first sensor and the track in the global fusion data;

And the correlation distance between the track in the fusion track data of the first sensor and the target in the global fusion data.

9. The method according to claim 7 or 8, wherein, in the case that the association distance between two objects represented by elements in the association matrix is greater than or equal to a threshold value, the corresponding element in the association matrix is a first value; and under the condition that the association distance is smaller than the threshold value, the corresponding element in the association matrix is equal to the association distance divided by the threshold value.

10. The method according to any one of claims 7 to 9, wherein the fusion target data of the first sensor and global fusion data are subjected to an associative fusion process to obtain updated global fusion data, and at least one of the following is included:

11. The method of claim 10, wherein the attribute fusion process comprises at least one of: fusing position and shape; orientation fusion; category fusion; motion fusion; presence fusion; semantic fusion.

12. The method of claim 11, wherein the location shape fusion comprises: filtering or smoothing the position observed values from different sensors, and performing time sequence estimation on the position observed values with low confidence coefficient to obtain the position of the fusion target; or selecting a high-precision shape observation value from shape observation values from different sensors, and correcting the high-precision shape observation value by using other shape observation values to obtain the shape of the fusion target;

the category fusion includes: obtaining classification probability corresponding to the classification vector according to the perception processing result of the original target data, and obtaining the category of the fusion target according to the classification probability;

the motion fusion comprises: estimating the motion state quantity of the fusion target according to a motion model and a Kalman filtering algorithm;

13. The method of any one of claims 1 to 12, further comprising:

performing target filtering according to the associated information, attribute information and tracking information of the fusion target; the fusion targets include fusion targets in updated global fusion data.

14. The method of any one of claims 1 to 13, further comprising:

and outputting the concerned fusion target in the application program according to the data source of the application program and the updated global fusion data.

15. A sensor data fusion device comprising:

16. The apparatus of claim 15, further comprising:

and the third processing module is used for carrying out fusion processing on the target data of the first sensor and the target data of at least one second sensor to obtain fusion data of the first sensor.

17. The apparatus of claim 16, the third processing module comprising:

and the feature fusion sub-module is used for fusing the target data of the first sensor and the target data of the plurality of second sensors in pairs after extracting features based on the projection relation to obtain the expression mode, so as to obtain the plurality of fusion data of the first sensor.

18. The apparatus of any of claims 15 to 17, the first processing module comprising:

the target association sub-module is used for comparing whether expected targets exist between target data of the first sensor and fusion data of different types in an association mode;

the target fusion sub-module is used for fusing a plurality of expected targets of a correlation mode under the condition that the plurality of expected targets exist in the correlation mode, so as to obtain fusion targets;

and the data processing sub-module is used for obtaining the fusion target data of the first sensor according to the expected targets or the fusion targets corresponding to different association modes.

19. The apparatus of claim 18, wherein the association means comprises at least one of:

20. The apparatus of any of claims 15 to 19, further comprising:

the sensor track module is used for obtaining the fusion track data of the first sensor according to the fusion target data of the first sensor of a plurality of frames;

And the global track module is used for obtaining global fusion track data according to the multi-frame global fusion target data.

21. The apparatus of any of claims 15 to 20, wherein the second processing module comprises:

the matrix building sub-module is used for obtaining an incidence matrix according to the fusion target data and the global fusion data of the first sensor;

and the matrix updating sub-module is used for checking and updating the association matrix by using at least one of tracking information, association information between sensors, category information, dynamic and static properties and speed information in the fusion target data.

22. The apparatus of claim 21, wherein elements in the correlation matrix are used to represent at least one of:

23. The apparatus according to claim 21 or 22, wherein, in a case where an association distance of two objects represented by elements in the association matrix is greater than or equal to a threshold value, a corresponding element in the association matrix is a first value; and under the condition that the association distance is smaller than the threshold value, the corresponding element in the association matrix is equal to the association distance divided by the threshold value.

24. The apparatus of any of claims 21 to 23, wherein the second processing module further comprises:

the attribute fusion sub-module is used for carrying out attribute fusion processing on the fusion target data of the single sensor and the global fusion target data according to the association pair obtained by the updated association matrix;

and the global updating sub-module is used for updating the global fusion track data according to the global fusion target data after the attribute fusion processing.

25. The apparatus of claim 24, wherein the attribute fusion process comprises at least one of: fusing position and shape; orientation fusion; category fusion; motion fusion; presence fusion; semantic fusion.

26. The apparatus of claim 25, wherein the location shape fusion comprises: filtering or smoothing the position observed values from different sensors, and performing time sequence estimation on the position observed values with low confidence coefficient to obtain the position of the fusion target; or selecting a high-precision shape observation value from shape observation values from different sensors, and correcting the high-precision shape observation value by using other shape observation values to obtain the shape of the fusion target;

27. The apparatus of any one of claims 15 to 26, further comprising:

the filtering module is used for filtering the targets according to the associated information, the attribute information and the tracking information of the fusion targets; the fusion targets include fusion targets in updated global fusion data.

28. The apparatus of any of claims 15 to 27, further comprising:

and the output module is used for outputting the concerned fusion target in the application program according to the data source of the application program and the updated global fusion data.

29. An electronic device, comprising:

at least one processor; and

a memory communicatively coupled to the at least one processor; wherein,,

the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-14.

30. A non-transitory computer readable storage medium storing computer instructions for causing the computer to perform the method of any one of claims 1-14.

31. A computer program product comprising a computer program which, when executed by a processor, implements the method according to any of claims 1-14.