CN108828527B

CN108828527B - Multi-sensor data fusion method and device, vehicle-mounted equipment and storage medium

Info

Publication number: CN108828527B
Application number: CN201810632459.1A
Authority: CN
Inventors: 王子涵; 刘洋
Original assignee: Uisee Shanghai Automotive Technologies Ltd
Current assignee: Uisee Shanghai Automotive Technologies Ltd
Priority date: 2018-06-19
Filing date: 2018-06-19
Publication date: 2021-04-16
Anticipated expiration: 2038-06-19
Also published as: CN108828527A

Abstract

The embodiment of the invention relates to a multi-sensor data fusion method, a device, a vehicle-mounted device and a storage medium, wherein original data are internally processed by a sensor through a distributed data fusion mode to obtain a measured track of an obstacle and a corresponding measuring time, the measured track and the corresponding measuring time are output to the vehicle-mounted device, and then a predicted track corresponding to a historical track of the obstacle is obtained by the vehicle-mounted device based on the obtained measuring time, so that the vehicle-mounted device can determine whether the predicted track of the obstacle and the measured track of the obstacle correspond to the same obstacle or not, further, the vehicle-mounted device can correlate the measured track of the obstacle and the historical track of the obstacle to realize track correlation of the same obstacle at different times, and finally, the historical track of the obstacle is updated based on a correlation relationship, so that multi-sensor data fusion is realized.

Description

Multi-sensor data fusion method and device, vehicle-mounted equipment and storage medium

Technical Field

The embodiment of the invention relates to the technical field of vehicle driving, in particular to a multi-sensor data fusion method and device, vehicle-mounted equipment and a storage medium.

Background

With the rapid development of vehicle auxiliary driving technology and unmanned driving technology, the perception requirement on the surrounding environment of the vehicle is higher and higher, and a data fusion scheme with low cost, high reliability and high adaptability becomes the root of the rapid landing of the vehicle auxiliary driving technology and the unmanned driving technology.

The perception sensors mainly used in current vehicle-assisted driving and unmanned driving include: image sensor and radar, wherein, radar includes again: millimeter wave radar, ultrasonic radar, laser radar, and the like. Different perception sensors have different characteristics and different perception ranges, so that the fusion of different perception sensor data becomes an important technical component in vehicle auxiliary driving and unmanned driving, and the environment perception system with larger detection range, more accurate measurement precision, less false reports and less false reports can be realized by fusing the perception sensor data with different characteristics and different perception ranges.

Therefore, a multi-sensor data fusion scheme is needed to realize sensing of obstacles around the vehicle and provide sensing information for planning and controlling vehicle-assisted driving and unmanned driving.

Disclosure of Invention

In order to solve the problems in the prior art, at least one embodiment of the invention provides a multi-sensor data fusion method and device, an on-board device and a storage medium.

In a first aspect, an embodiment of the present invention provides a multi-sensor data fusion method, including:

acquiring first measurement tracks of a plurality of track measurement sensors and first measurement time corresponding to each first measurement track;

unifying the data format of each first measurement track to obtain a second measurement track corresponding to each first measurement track;

based on each first measurement time, performing time synchronization on each stored system track to obtain a predicted track corresponding to each system track;

determining an association relationship between each of the second measured tracks and each of the system tracks based on each of the predicted tracks;

and updating the stored system track based on the incidence relation.

In some embodiments, said determining an association between each of said second measured tracks and each of said system tracks based on each of said predicted tracks comprises:

searching a second measurement track matched with the predicted track from all the second measurement tracks;

and associating the system track corresponding to the predicted track with the searched second measured track.

In some embodiments, the finding a second measured trajectory that matches the predicted trajectory comprises:

searching whether at least one second measurement track exists in a preset range around the predicted track by taking the predicted track as a center;

and if so, matching a second measured track closest to the predicted track with the predicted track.

In some embodiments, after determining the correlation between each of the second measured tracks and each of the system tracks, the method further comprises:

storing the incidence relation between the second measurement track and the system track;

correspondingly, after the time synchronization is performed on each stored system track based on each first measurement time to obtain the predicted track corresponding to each system track, the method further includes:

searching whether an incidence relation between the second measurement track and the system track is stored;

and if not, executing the step of determining the association relationship between each second measured track and each system track based on each predicted track.

In some embodiments, the method further comprises:

if the incidence relation between the second measurement track and the system track is stored, judging whether the second measurement track and the system track in the incidence relation meet a preset matching condition or not;

if so, maintaining the association relationship;

In some embodiments, said updating the stored system track based on said incidence relation comprises:

judging whether each stored system track is the system track in the association relationship;

if so, updating the system track by using a second measurement track corresponding to the system track in the incidence relation;

and if not, updating the system track by using the predicted track corresponding to the system track.

In some embodiments, after updating the stored system track, the method further comprises:

and managing the reliability of the updated system track based on the updating mode of the system track.

In some embodiments, the managing the reliability of the updated system track based on the updating manner of the system track includes:

and if the updating mode of the system track is to update by using the second measured track in the incidence relation, the reliability of the system track is increased.

if the updating mode of the system track is to use the predicted track for updating, acquiring a track measuring sensor corresponding to the updating mode;

judging whether the system track is in the sensing range of the track measuring sensor corresponding to the updating mode, and if so, reducing the reliability of the system track; and if not, maintaining the reliability of the system track.

In some embodiments, after managing the trustworthiness of the updated system track, the method further comprises:

judging whether the reliability of the managed system flight path reaches a preset output threshold or not to obtain the system flight path reaching the preset output threshold;

and outputting the system track reaching the preset output threshold.

In some embodiments, after the reducing the confidence level of the system track, the method further comprises:

judging whether the reliability of the system track is lower than a preset elimination threshold or not;

and if the current position is lower than the preset value, eliminating the system track.

and initializing second measurement tracks which are not used by the association relation in second measurement tracks corresponding to the first measurement tracks.

In some embodiments, the initialization process includes:

obtaining the state information of the vehicle;

estimating a vehicle trajectory based on the vehicle state information;

judging whether the second measurement track which is not used by the incidence relation is far away from the vehicle track;

if so, eliminating the second measurement track which is not used by the incidence relation;

if not, initializing the second measurement track which is not used by the incidence relation into a system track.

In some embodiments, the initialization process further includes:

and giving confidence to a second measurement track initialized to the system track.

In some embodiments, the initialization process further includes:

acquiring judgment information of a plurality of obstacle judgment sensors;

determining whether an obstacle exists according to the judgment information;

if so, judging whether the barrier and the second measurement track initialized to the system track are positioned on the same side of the vehicle; if so, giving the reliability of a second measurement track initialized to the system track as a first reliability; if not, giving the reliability of a second measurement track initialized to the system track as a second reliability;

if no obstacle exists, giving the reliability of a second measurement track initialized to the system track as a second reliability;

wherein the first confidence level is greater than the second confidence level.

In some embodiments, the plurality of track measurement sensors comprises: a forward image sensor, a forward radar, a left front side radar, and a right front side radar.

In some embodiments, the plurality of obstacle determination sensors includes: a left rear radar and a right rear radar.

In a second aspect, an embodiment of the present invention further provides a multi-sensor data fusion apparatus, where the apparatus includes:

the system comprises an acquisition unit, a processing unit and a processing unit, wherein the acquisition unit is used for acquiring first measurement tracks of a plurality of track measurement sensors and first measurement time corresponding to each first measurement track;

the unifying unit is used for unifying the data format of each first measuring track to obtain a second measuring track corresponding to each first measuring track;

the synchronization unit is used for carrying out time synchronization on each stored system track based on each first measurement time to obtain a predicted track corresponding to each system track;

a determining unit, configured to determine, based on each of the predicted tracks, an association relationship between each of the second measured tracks and each of the system tracks;

and the updating unit is used for updating the stored system track based on the incidence relation.

In some embodiments, the determining unit includes:

the searching subunit is used for searching a second measured track matched with the predicted track from all the second measured tracks;

and the association subunit is used for associating the system track corresponding to the predicted track with the searched second measured track.

In some embodiments, the lookup subunit is configured to:

In some embodiments, the apparatus further comprises:

the storage unit is used for storing the incidence relation between the second measurement flight path and the system flight path after the determining unit determines the incidence relation between each second measurement flight path and each system flight path;

the searching unit is used for searching whether the incidence relation between the second measurement track and the system track is stored or not after the predicted track corresponding to each system track is obtained by the synchronizing unit;

accordingly, the determining unit is configured to determine, based on each of the predicted tracks, an association relationship between each of the second measured tracks and each of the system tracks after the finding unit determines that the second measured tracks are not stored.

In some embodiments, the apparatus further comprises:

the first judgment unit is used for judging whether the second measurement track and the system track in the association relation meet the preset matching condition after the searching unit determines that the association relation between the second measurement track and the system track is stored;

a maintaining unit, configured to maintain the association relationship after the first determining unit determines that a preset matching condition is satisfied;

the determining unit is configured to determine, after the first determining unit determines that the preset matching condition is not satisfied, an association relationship between each second measured track and each system track based on each predicted track.

In some embodiments, the update unit includes:

the judging subunit is used for judging whether each stored system track is the system track in the association relationship;

the first updating subunit is used for updating the system track by using a second measured track corresponding to the system track in the incidence relation after the judging subunit judges that the system track is the system track in the incidence relation;

and the second updating subunit is used for updating the system track by using the predicted track corresponding to the system track after the judging subunit judges that the system track is not the system track in the incidence relation.

In some embodiments, the apparatus further comprises:

and the management unit is used for managing the credibility of the updated system track based on the updating mode of the system track after the updating unit updates the stored system track.

In some embodiments, the management unit is configured to:

In some embodiments, the apparatus further comprises:

the second judging unit is used for judging whether the reliability of the managed system flight path reaches a preset output threshold after the management unit manages the reliability of the updated system flight path, so as to obtain the system flight path reaching the preset output threshold;

and the output unit is used for outputting the system track reaching the preset output threshold.

In some embodiments, the management unit is further configured to:

after the reliability of the system track is reduced, judging whether the reliability of the system track is lower than a preset elimination threshold or not;

and if the number of the system tracks is lower than a preset elimination threshold, eliminating the system tracks.

In some embodiments, the apparatus further comprises:

and the initialization unit is used for initializing second measurement tracks which are not used by the association relation in second measurement tracks corresponding to each first measurement track after the management unit manages the reliability of the updated system tracks.

In some embodiments, the initialization unit performs an initialization process including:

obtaining the state information of the vehicle;

estimating a vehicle trajectory based on the vehicle state information;

In some embodiments, the initialization unit performs initialization processing, and further includes:

acquiring judgment information of a plurality of obstacle judgment sensors;

determining whether an obstacle exists according to the judgment information;

wherein the first confidence level is greater than the second confidence level.

In a third aspect, an embodiment of the present invention further provides an on-board device, including:

a processor, memory, a network interface, and a user interface;

the processor, memory, network interface and user interface are coupled together by a bus system;

the processor is adapted to perform the steps of the method according to the first aspect by calling a program or instructions stored by the memory.

In a fourth aspect, an embodiment of the present invention also provides a non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform the steps of the method according to the first aspect.

In at least one embodiment of the invention, the sensor performs internal processing on the original data in a distributed data fusion mode to obtain the measurement track of the obstacle and the corresponding measurement time, the measurement track and the corresponding measurement time are output to the vehicle-mounted equipment, and the vehicle-mounted equipment obtains the predicted track corresponding to the historical track of the obstacle based on the obtained measurement time, so that the vehicle-mounted equipment can determine whether the predicted track of the obstacle and the measurement track of the obstacle correspond to the same obstacle. And then the vehicle-mounted equipment can correlate the measured flight path of the obstacle with the historical flight path of the obstacle, so that the flight path correlation of the same obstacle at different moments is realized, and finally, the historical flight path of the obstacle is updated based on the correlation relation, so that the multi-sensor data fusion is realized.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required to be used in the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art that other drawings can be obtained according to these drawings without inventive labor.

Fig. 0 is a structural diagram of a vehicle-mounted device according to an embodiment of the present invention;

fig. 1 is a flowchart of a multi-sensor data fusion method according to an embodiment of the present invention;

FIG. 2 is a flow chart of another multi-sensor data fusion method provided by the embodiment of the invention;

FIG. 3 is a flow chart of another multi-sensor data fusion method provided by the embodiment of the invention;

FIG. 4 is a flow chart of another multi-sensor data fusion method provided by the embodiment of the invention;

FIG. 5 is a flow chart of another multi-sensor data fusion method provided by the embodiment of the invention;

FIG. 6 is a flow chart of another multi-sensor data fusion method provided by the embodiment of the invention;

FIG. 7 is a flow chart of another multi-sensor data fusion method provided by the embodiment of the invention;

FIG. 8 is a flow chart of another multi-sensor data fusion method provided by an embodiment of the present invention;

FIG. 9 is a flow chart of another multi-sensor data fusion method provided by the embodiment of the invention;

FIG. 10 is a flow chart of yet another multi-sensor data fusion method provided by an embodiment of the present invention;

FIG. 11 is a flow chart of another multi-sensor data fusion method provided by the embodiments of the present invention;

FIG. 12 is a flowchart of an initialization process according to an embodiment of the present invention;

FIG. 12' is a flowchart of another initialization process provided by embodiments of the present invention;

FIG. 13 is a flowchart of another initialization process provided by an embodiment of the invention;

FIG. 14 is a flowchart of another initialization process provided by an embodiment of the invention;

FIG. 15 is a schematic view of a sensor mounted on a vehicle according to an embodiment of the present invention;

FIG. 16 is a schematic structural diagram of a system including a multi-sensor data fusion function according to an embodiment of the present invention;

fig. 17 is a block diagram of a multi-sensor data fusion apparatus according to an embodiment of the present invention;

FIG. 18 is a block diagram of another multi-sensor data fusion apparatus according to an embodiment of the present invention;

FIG. 19 is a block diagram of another multi-sensor data fusion apparatus according to an embodiment of the present invention;

FIG. 20 is a block diagram of another multi-sensor data fusion apparatus according to an embodiment of the present invention;

FIG. 21 is a block diagram of another multi-sensor data fusion apparatus according to an embodiment of the present invention;

FIG. 22 is a block diagram of another multi-sensor data fusion apparatus according to an embodiment of the present invention;

FIG. 23 is a block diagram of another multi-sensor data fusion apparatus according to an embodiment of the present invention;

fig. 24 is a block diagram of another multi-sensor data fusion apparatus according to an embodiment of the present invention.

Detailed Description

In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, but not all, embodiments of the present invention. All other embodiments, which can be obtained by a person skilled in the art without any inventive step based on the embodiments of the present invention, are within the scope of the present invention.

It is noted that, in this document, relational terms such as "first" and "second," and the like, may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions.

Currently, the development of functions for vehicle driving assistance includes: forward functional development and multidirectional functional development, wherein the forward functional development includes, for example: an AEB (automatic Emergency Braking) system, an ACC (Adaptive Cruise Control) system, an FCW (Forward Collision Warning) system, an LKA (Lane Keeping Aid) system, and the like; multidirectional functional development includes, for example: automatic lane system trades.

It can be seen that, with the development of the function of vehicle driving assistance, the configuration scheme of the perception sensor is also developed from a single forward image sensor and a single forward radar into a scheme of combining the forward image sensor and the forward radar, the forward image sensor is installed at the front of the vehicle and is used for collecting images and videos right in front of the vehicle, and the forward radar is installed at the front of the vehicle and is used for detecting obstacles right in front of the vehicle. For the multi-directional function development, it is also necessary to fuse the data of the lateral sensors, which can be understood as sensing sensors installed on both sides of the vehicle for detecting obstacles around both sides of the vehicle.

Fig. 0 is a schematic structural diagram of an in-vehicle device provided in an embodiment of the present invention. The vehicle-mounted device shown in fig. 0 includes: at least one processor 901, at least one memory 902, at least one network interface 904, and other user interfaces 903. Various components in the in-vehicle device are coupled together by a bus system 905. It is understood that the bus system 905 is used to enable communications among the components. The bus system 905 includes a power bus, a control bus, and a status signal bus, in addition to a data bus. For clarity of illustration, however, the various buses are labeled in fig. 9 as bus system 905.

The user interface 903 may include, among other things, a display, a keyboard, or a pointing device (e.g., a mouse, trackball, or touch pad, among others.

It will be appreciated that the memory 902 in the subject embodiment can be either volatile memory or nonvolatile memory, or can include both volatile and nonvolatile memory. The non-volatile memory may be a Read-only memory (ROM), a programmable Read-only memory (PROM), an erasable programmable Read-only memory (erasabprom, EPROM), an electrically erasable programmable Read-only memory (EEPROM), or a flash memory. The volatile memory may be a Random Access Memory (RAM) which functions as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (staticiram, SRAM), dynamic random access memory (dynamic RAM, DRAM), synchronous dynamic random access memory (syncronous DRAM, SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM ), Enhanced Synchronous DRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), and direct memory bus RAM (DRRAM). The memory 902 described herein is intended to comprise, without being limited to, these and any other suitable types of memory.

In some embodiments, memory 902 stores the following elements, executable units or data structures, or a subset thereof, or an expanded set thereof: an operating system 9021 and application programs 9022.

The operating system 9021 includes various system programs, such as a framework layer, a core library layer, a driver layer, and the like, and is configured to implement various basic services and process hardware-based tasks. The application 9022 includes various applications, such as a media player (MediaPlayer), a Browser (Browser), and the like, for implementing various application services. A program implementing the method of an embodiment of the present invention may be included in application 9022.

In this embodiment of the present invention, the processor 901 is configured to execute the method steps provided by the method embodiments by calling a program or an instruction stored in the memory 902, specifically, a program or an instruction stored in the application 9022, where the method steps include:

and updating the stored system track based on the incidence relation.

The method disclosed in the above embodiments of the present invention may be applied to the processor 901, or implemented by the processor 901. The processor 901 may be an integrated circuit chip having signal processing capabilities. In implementation, the steps of the above method may be implemented by integrated logic circuits of hardware or instructions in the form of software in the processor 901. The processor 901 may be a general-purpose processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), an off-the-shelf programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic device, or discrete hardware components. The various methods, steps and logic blocks disclosed in the embodiments of the present invention may be implemented or performed. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The steps of the method disclosed in connection with the embodiments of the present invention may be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software elements in the decoding processor. The software elements may be located in ram, flash, rom, prom, or eprom, registers, among other storage media that are well known in the art. The storage medium is located in the memory 902, and the processor 901 reads the information in the memory 902, and completes the steps of the above method in combination with the hardware thereof.

It is to be understood that the embodiments described herein may be implemented in hardware, software, firmware, middleware, microcode, or any combination thereof. For a hardware implementation, the processing units may be implemented within one or more Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), Digital Signal Processing Devices (DSPDs), Programmable Logic Devices (PLDs), Field Programmable Gate Arrays (FPGAs), general purpose processors, controllers, micro-controllers, microprocessors, other electronic units designed to perform the functions described herein, or a combination thereof.

For a software implementation, the techniques described herein may be implemented by means of units performing the functions described herein. The software codes may be stored in a memory and executed by a processor. The memory may be implemented within the processor or external to the processor.

Those of ordinary skill in the art will appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware or combinations of computer software and electronic hardware. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.

It is clear to those skilled in the art that, for convenience and brevity of description, the specific working processes of the above-described systems, apparatuses and units may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.

In the embodiments provided in the present application, it should be understood that the execution sequence of the steps of the method embodiments can be arbitrarily adjusted unless there is an explicit precedence sequence. The disclosed apparatus and method may be implemented in other ways. For example, the above-described apparatus embodiments are merely illustrative, and for example, the division of the units is only one logical division, and other divisions may be realized in practice, for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may be in an electrical, mechanical or other form.

The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.

In addition, functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit.

The functions, if implemented in the form of software functional units and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the embodiments of the present invention may be essentially implemented or make a contribution to the prior art, or may be implemented in the form of a software product stored in a storage medium and including instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the methods described in the embodiments of the present invention. And the aforementioned storage medium includes: various media capable of storing program codes, such as a U disk, a removable hard disk, a ROM, a RAM, a magnetic disk, or an optical disk.

As shown in fig. 1, the present embodiment discloses a multi-sensor data fusion method, in which a plurality of sensors are mounted on a vehicle, the method is executed by an in-vehicle device, and the in-vehicle device is in communication connection with each sensor, and the method may include the following steps 101 to 105:

101. first measurement tracks of the plurality of track measurement sensors and first measurement time corresponding to each first measurement track are obtained.

102. Unifying the data format of each first measurement track to obtain a second measurement track corresponding to each first measurement track.

103. And carrying out time synchronization on each stored system track based on each first measurement moment to obtain a predicted track corresponding to each system track.

104. Based on each predicted track, an association between each second measured track and each system track is determined.

105. And updating the stored system track based on the incidence relation.

In this embodiment, the track measuring sensor belongs to an intelligent sensor, and has functions of obstacle detection and data processing. Specifically, the track measurement sensor may collect obstacle data within a detection range, and the detected obstacle data is referred to as raw data. The track measuring sensor can process the original data to obtain a track corresponding to the obstacle in the detection range, and the track is called as a first measuring track. The first measurement track includes: the motion state, the position, the distance and the like of the obstacle, wherein the motion state comprises the following steps: one or more of a speed of movement and a direction of movement.

In this embodiment, the track measurement sensor may output the obtained first measurement track to the vehicle-mounted device. The way of outputting the first measured track by the track measuring sensor can be active or passive. The active type, namely the track measuring sensor actively outputs the first measured track to the vehicle-mounted equipment after obtaining the first measured track. The passive track measuring sensor outputs a first measuring track in response to a data request command after receiving the data request command sent by the vehicle-mounted equipment.

In this embodiment, the first measurement time corresponding to the first measurement track may be understood as a time when the track measurement sensor outputs the first measurement track, and may also be understood as a time when the track measurement sensor obtains the first measurement track. The vehicle-mounted equipment acquires the first measurement time and simultaneously records the identification of the track measuring sensor which sends the first measurement time, namely records which track measuring sensor sends the first measurement time.

In this embodiment, in consideration of different formats of data output by different track measurement sensors, in order to facilitate subsequent data fusion, data formats need to be unified, so that the data formats of each first measurement track are unified, a unified manner may follow the prior art, for example, a unified coordinate system is defined, and each first measurement track is converted into data in the unified coordinate system, which is not described herein again in detail.

In this embodiment, after unifying the data format of each first measurement track, a second measurement track corresponding to each first measurement track can be obtained, and the data formats of all the second measurement tracks are the same.

In this embodiment, each stored system track may be understood as an obstacle history track stored in the vehicle-mounted device. If the vehicle-mounted device acquires a new obstacle track, namely a first measurement track, in order to determine whether the first measurement track is a track after the obstacle moves and corresponding to the system track, the system track needs to be associated with the first measurement track, and if the first measurement track is associated with the first measurement track, the first measurement track and the system track correspond to the same obstacle.

In this embodiment, to implement the association between the system track and the first measurement track, first, data format conversion is performed on the first measurement track to obtain a second measurement track. And then, based on each first measurement time, carrying out time synchronization on each stored system track to obtain a predicted track corresponding to each system track, so that the predicted track and the second measured track are data at the same time and have comparability. The difference between the predicted track and the second measured track is that the predicted track is data calculated by the vehicle-mounted device and is virtual data, and the first measured track is data processed by the track measuring sensor according to detected raw data and is actual data. And finally, determining the association relationship between the second measurement track and the system track based on the predicted track, which is equivalent to the association between the system track and the first measurement track. To reduce computational load, only one system track is associated with each second measured track.

In this embodiment, the time synchronization mode may follow the existing mode, and this embodiment is not described again.

In this embodiment, after the incidence relation between each second measurement track and each system track is determined, the system tracks can be updated through a kalman filtering algorithm based on the incidence relation, so as to realize multi-sensor data fusion. The method for updating the system track through the Kalman filtering algorithm belongs to a mature technology, and is not described herein again.

The existing data fusion mode is generally centralized data fusion, each sensor sends the original data to a fusion center of a vehicle (namely, a processor with a data fusion function), and the fusion center realizes data fusion, so that the original data is less lost, but the data information amount is large. The multi-sensor data fusion method disclosed in this embodiment is a distributed data fusion method, in which a sensor performs internal processing on raw data to obtain a measurement track (i.e., a first measurement track) of an obstacle, and sends the measurement track of the obstacle to a vehicle-mounted device, and the vehicle-mounted device fuses the measurement tracks of the obstacle. Compared with centralized data fusion, the distributed data fusion has the advantages of low transmission load, high redundancy, less computing resources and high adaptation flexibility although part of original data is lost.

As can be seen, in the multi-sensor data fusion method disclosed in this embodiment, the sensor performs internal processing on the raw data in a distributed data fusion manner to obtain the measurement track of the obstacle and the corresponding measurement time (i.e., the first measurement time), the vehicle-mounted device obtains the measurement track of the obstacle and the corresponding measurement time, and the vehicle-mounted device obtains the predicted track corresponding to the historical track of the obstacle (i.e., the system track stored by the vehicle-mounted device) based on the obtained measurement time, so that the vehicle-mounted device can determine whether the predicted track of the obstacle and the measurement track of the obstacle correspond to the same obstacle. And then the vehicle-mounted equipment can correlate the measured flight path of the obstacle with the stored system flight path, so that the flight path correlation of the same obstacle at different moments is realized, and finally, the historical flight path of the obstacle is updated based on the correlation relation, so that the multi-sensor data fusion is realized.

In some embodiments, as shown in FIG. 2, in addition to comprising

steps

101, 102, 103, and 105 shown in FIG. 1, FIG. 2 focuses on a specific implementation of step 104 in FIG. 1, i.e., on how to determine the correlation between the second measured track and the system track. Step 104, determining the association relationship between each second measured track and each system track based on each predicted track, may include the following steps 1041 and 1042:

1041. and searching for a second measured track matched with the predicted track from all the second measured tracks.

1042. And associating the system track corresponding to the predicted track with the searched second measured track.

In this embodiment, an association relationship table between the second measured flight path and the system flight path may be established, the second measured flight path matched with the predicted flight path is searched from all the second measured flight paths, and if the second measured flight path and the predicted flight path are matched, it is indicated that the searched second measured flight path and the predicted flight path correspond to the same obstacle. And associating the system track corresponding to the predicted track with the searched second measured track, so that the system track and the associated second measured track correspond to the same barrier, and the track change of the barrier at different moments is reflected.

Based on the previous embodiment, as shown in fig. 3, the difference from fig. 2 is that: in this embodiment, a specific implementation of step 1041 is described with emphasis. The step 1041 of searching for a second measured track matching the predicted track includes the following steps 1041' and 1041 ":

1041' taking the predicted flight path as a center, searching whether at least one second measured flight path exists in a preset range around the predicted flight path; if so, perform step 1041 ";

1041 ", the second measured track closest to the predicted track matches the predicted track.

In this embodiment, it is considered that the predicted flight path and the second measured flight path of the same obstacle have errors, and are not completely the same, and therefore, by reasonably setting the preset range, the errors of the predicted flight path and the second measured flight path within the preset range can be ignored, and the predicted flight path and the second measured flight path are considered to both correspond to the same obstacle.

In this embodiment, the preset range may be set according to an actual situation, and the specific value of the preset range is not limited in this embodiment.

In this embodiment, considering that a plurality of second measured tracks may exist in a preset range around the predicted track, according to the nearest principle, the second measured track closest to the predicted track in the preset range around the predicted track is matched with the predicted track, and the matching can be understood as corresponding to the same obstacle.

In some embodiments, as shown in fig. 4, in addition to the steps shown in fig. 1, after determining the association relationship between each second measured track and each system track in step 104, the following steps 105 are further included:

106. and storing the association relationship between the second measurement track and the system track.

Correspondingly, after the step 103 of performing time synchronization on each stored system track based on each first measurement time to obtain a predicted track corresponding to each system track, the method may further include the following step 107:

107. and searching whether the association relationship between the second measured flight path and the system flight path is stored, and if not, executing step 104 to determine the association relationship between each second measured flight path and each system flight path based on each predicted flight path.

In this embodiment, if the data fusion is not performed for the first time, the association relationship determined when the data fusion is performed before the data fusion is performed for the present time is stored in the vehicle-mounted device. Therefore, before determining the association relationship between the second measurement track and the system track in the data fusion process, whether the association relationship between the second measurement track and the system track is stored should be searched. And when the correlation is not found, namely the correlation is not stored in the vehicle-mounted equipment, determining the correlation between each second measured track and each system track.

Based on the previous embodiment, as shown in fig. 5, in addition to the steps shown in fig. 4, the method further includes step 108, specifically, step 107 is to find whether an association relationship between the second measured track and the system track is stored, and if the association relationship between the second measured track and the system track is stored, the following step 108 is executed:

108. judging whether a second measurement track and a system track in the association relation meet a preset matching condition or not; if so, maintaining the association relationship; if not, the method proceeds to step 104, where an association relationship between each second measured track and each system track is determined based on each predicted track.

In this embodiment, if the data fusion is not performed for the first time, the association relationship determined when the data fusion is performed before the data fusion is performed for the present time is stored in the vehicle-mounted device. Therefore, before determining the association relationship between the second measurement track and the system track in the data fusion process, whether the association relationship between the second measurement track and the system track is stored should be searched. When the correlation is found, namely the vehicle-mounted equipment stores the correlation, whether the second measurement track and the system track in the correlation meet a preset matching condition is judged. The matching conditions include, for example: one or more of whether the motion state of the second measured trajectory in the association matches the motion state of the system trajectory and whether the motion state of the second measured trajectory in the association matches the position of the system trajectory.

In some embodiments, as shown in fig. 6, in addition to

steps

101, 102, 103 and 104 shown in fig. 1, fig. 6 mainly describes a specific implementation of step 105 in fig. 1, and the step 105 of updating the stored system track based on the association relationship may include the following steps 1051 to 1053:

1051. judging whether each stored system track is the system track in the association relationship; if yes, go to step 1052; if not, go to step 1053.

1052. And updating the system track by using the second measurement track corresponding to the system track in the incidence relation.

1053. And updating the system track by using the predicted track corresponding to the system track.

In this embodiment, if the stored system track is a system track in the association relationship, it indicates that the obstacle corresponding to the stored system track is still within the detection range of the track measurement sensor of the vehicle during the data fusion process, and the new track of the obstacle during the data fusion process is the second measurement track corresponding to the system track in the association relationship, so that the system track is updated by using the second measurement track corresponding to the system track in the association relationship, thereby implementing track update of the obstacle.

In this embodiment, if the stored system track is not the system track in the association relationship, it is described that the obstacle corresponding to the stored system track is not within the detection range of the track measurement sensor of the vehicle in the current data fusion process, and therefore, the predicted track corresponding to the system track is used to update the system track, and the track update of the obstacle is implemented.

In some embodiments, as shown in FIG. 7, in addition to the steps shown in FIG. 1, the step of updating the stored system track further comprises the following step 109:

109. and managing the reliability of the updated system track based on the updating mode of the system track.

In this embodiment, after the system track is updated in different updating manners, the new system track has a deviation from the real track of the obstacle, the system track with a large deviation may be an error track, and the system track is less reliable when the deviation is larger. Therefore, in order to show whether the system track is credible or not, a credibility concept is introduced, and the credibility of the updated system track is managed, so that whether the updated system track is an error track or not is determined according to the credibility.

Based on the previous embodiment, as shown in fig. 8, the difference from fig. 7 is that the implementation of step 109 is described with emphasis, and the updating manner based on the system track in step 109 manages the credibility of the updated system track, including the following steps 1091 to 1095:

1091. and if the updating mode of the system track is to update by using the second measured track in the incidence relation, the reliability of the system track is increased.

1092. And if the updating mode of the system track is to use the predicted track for updating, acquiring a track measuring sensor corresponding to the updating mode.

1093. Judging whether the system track is in the sensing range of the track measuring sensor corresponding to the updating mode, if so, executing a step 1094; if not, go to step 1095.

1094. The reliability of the system track is reduced.

1095. And maintaining the reliability of the system track.

In this embodiment, if the system track is updated by using the second measured track in the association relationship, it is described that the updated system track is updated based on the data of the track measurement sensor, and the possibility of error is low, so that the reliability of the system track can be increased.

In this embodiment, if the update mode of the system track is to update using the predicted track, it indicates that the updated system track is the calculated track, the possibility of an error is high, and the reliability of the system track may be reduced or kept unchanged.

In this embodiment, for the update mode that the predicted track is used to update the system track, whether the reliability of the system track is reduced or kept unchanged can be determined by determining whether the system track is in the sensing range of the track measuring sensor corresponding to the update mode, if so, it indicates that the track measuring sensor should be able to detect the track, but the track measuring sensor does not actually detect the track, so the reliability of the system track should be reduced; if the system is not in the normal state, and the reliability can be kept unchanged temporarily.

In this embodiment, the step 1092 of acquiring the track measurement sensor corresponding to the update mode specifically includes: the vehicle-mounted equipment records the track measuring sensor which transmits the first measuring time, so that after the first measuring time is determined, the track measuring sensor corresponding to the updating mode can be determined as the track measuring sensor which transmits the first measuring time.

In some embodiments, as shown in FIG. 9, in addition to including the method steps shown in FIG. 7, after managing the trustworthiness of the updated system track as described in step 109, the following

steps

110 and 111 are included:

110. and judging whether the reliability of the managed system flight path reaches a preset output threshold or not to obtain the system flight path reaching the preset output threshold.

111. And outputting the system track reaching the preset output threshold.

In this embodiment, the purpose of the credibility management is to determine which system tracks are credible, so as to be convenient for use by subsequent different functional systems (for example, one or more systems of an AEB system, an ACC system, an LKA system, an FCW system, an automatic lane change system, and the like).

In this embodiment, the setting of the preset output threshold is convenient to determine which system tracks are credible, when the credibility of the system tracks reaches the preset output threshold, the system tracks are credible, the system tracks can be added into the output list, and after the credibility of all the system tracks is executed in step 110, the final output list is obtained.

In this embodiment, the preset output threshold may be determined according to an actual situation, and the specific value of the preset output threshold is not limited in this embodiment.

In some embodiments, as shown in FIG. 10, in addition to the steps shown in FIG. 8, after reducing the confidence level of the system track in step 1094, step 109 further includes the following steps 1096 and 1097:

1096. judging whether the reliability of the system track is lower than a preset elimination threshold or not; if so, go to step 1097; if not, steps 110 and 111 shown in FIG. 9 may be performed.

1097. And eliminating the system track.

In this embodiment, after the confidence level of the system track is maintained in step 1095,

steps

110 and 111 shown in fig. 9 may be executed.

In this embodiment, after the reliability of the system track is reduced to a certain degree, the system track is not trusted and is a wrong track. The false track may not be the track of an obstacle, and a fixed position object, such as a tree that may be on the side of a horse, relative to the track of the host vehicle should be eliminated.

In this embodiment, a preset elimination threshold is set, and when the reliability of the system track is lower than the preset elimination threshold, the system track is not trusted. The preset elimination threshold may be determined according to actual conditions, and the specific value of the preset elimination threshold is not limited in this embodiment.

In the embodiment, the system track with the reliability lower than the preset elimination threshold is eliminated, the error probability of judging the obstacle can be reduced, meanwhile, the calculation load of subsequent data fusion can be reduced, and the real-time performance of the data fusion is improved.

In some embodiments, as shown in FIG. 11, in addition to the steps shown in FIG. 7, after managing the trustworthiness of the updated system track as described in step 109, the following step 112 may be included:

112. and initializing second measurement tracks which are not used by the association relation in the second measurement tracks corresponding to each first measurement track.

In this embodiment, the second measured track that is not used by the association relationship may be a track of a new obstacle appearing around the vehicle, and the vehicle-mounted system may record the track of the new obstacle through initialization processing, so as to facilitate subsequent data fusion.

In some embodiments, as shown in fig. 12, the initialization process described in step 112 may include the following

steps

1121, 1122', 1125, and 1126:

1121. and acquiring the state information of the vehicle.

1122. Based on the own-vehicle state information, the own-vehicle trajectory is estimated.

1122' determining whether a second measured trajectory not used by the correlation is away from the vehicle trajectory; if yes, go to step 1125; if not, go to step 1126.

1125. Eliminating a second measured trajectory that is not used by the correlation.

1126. Initializing a second measured track that is not used by the incidence to a system track.

In this embodiment, the specific implementation of step 1122' includes, for example, the following first and second modes:

the first method is as follows: and judging whether the extension line of the motion direction of the second measurement track which is not used by the association relation is intersected with the extension line of the motion direction of the vehicle track, and if the extension lines are not intersected, judging that the vehicle track is far away from the vehicle track.

The second method comprises the following steps: as shown in fig. 12 ', step 1122' includes

steps

1123 and 1124 as follows.

1123. Determining a lateral distance between a second measured trajectory that is not used by the correlation and the trajectory of the host vehicle; the lateral distance is a projection distance relative to the movement direction of the vehicle.

1124. Judging whether the lateral distance is greater than a preset distance threshold value or not; if so, go to step 1125; if not, go to step 1126.

In the second mode, if the lateral distance is greater than the preset distance threshold, it is determined that the vehicle is far away from the vehicle track. The preset distance threshold may be determined according to actual conditions, and the specific value of the preset distance threshold is not limited in this embodiment.

Based on the above example, as shown in fig. 13, in addition to the initialization processing in step 112 including steps 1121 through 1126 shown in fig. 12, step 112 may further include the following step 1127:

1127. and giving confidence to a second measurement track initialized to the system track.

In this embodiment, the second measurement track initialized to the system track may be given the same preset confidence level. The second measured track for different track measurement sensors may be given different degrees of confidence, e.g., the second measured track for image sensors may be given a higher degree of confidence and the second measured track for radar may be given a lower degree of confidence. By giving credibility to the second measurement track initialized to the system track, the credibility of the updated system track is convenient to manage in the next data fusion process.

In some embodiments, as shown in fig. 14, in addition to the initialization process described in step 112 including steps 1121 through 1126 shown in fig. 12, step 112 may further include steps 1128 through 1132 of:

1128. acquiring judgment information of a plurality of obstacle judgment sensors;

1129. determining whether an obstacle exists according to the judgment information; if an obstacle exists, go to step 1130; if no obstacle exists, go to step 1131;

1130. judging whether the barrier and a second measurement track initialized to the system track are positioned on the same side of the vehicle; if yes, go to step 1132; if not, go to step 1131;

1131. giving the reliability of a second measurement track initialized to the system track as a second reliability;

1132: giving the reliability of a second measurement track initialized to the system track as a first reliability;

wherein the first confidence level is greater than the second confidence level.

In this embodiment, the obstacle determination sensor belongs to an intelligent sensor, and has obstacle detection and data processing functions. Specifically, the obstacle determination sensor may collect obstacle data within a detection range, and the detected obstacle data is referred to as raw data. The obstacle judgment sensor can process the original data, judge whether an obstacle exists in the detection range, and obtain judgment information. The judgment information is used to indicate whether an obstacle exists.

In this embodiment, if the obstacle determining sensor determines that an obstacle exists, it is further determined whether the obstacle and the second measured trajectory initialized to the system trajectory are located on the same side of the vehicle. If so, the possibility that the second measurement track initialized to the system track is the track of the obstacle is higher, so the reliability of the second measurement track is higher, namely the first reliability; if not, it is determined that the second measured trajectory initialized to the system trajectory is less likely to be the trajectory of the obstacle, and therefore the reliability of the second measured trajectory is given a lower reliability, that is, the second reliability.

In this embodiment, if the obstacle determination sensor determines that there is no obstacle, it is described that the possibility that the second measurement trajectory initialized to the system trajectory is the trajectory of the obstacle is low, and therefore, the reliability of the second measurement trajectory is given as a low reliability, that is, the second reliability.

The difference from the embodiment shown in fig. 13 is that in this embodiment, whether an obstacle exists is determined by the obstacle determination sensor, and different degrees of reliability are given to the second measurement track initialized to the system track, so that the updated reliability of the system track is managed in the next data fusion process, and the system track to be eliminated is determined more quickly.

In some embodiments, as shown in fig. 15, compared with the prior art, the number and positions of the sensors mounted on the vehicle are adjusted in the present embodiment, which is specifically described as follows:

a Front Long Range Radar (Front Long Range Radar) mounted at the Front of the vehicle for detecting obstacles in a first sector area directly in Front of the vehicle;

a forward camera mounted at the front of the vehicle for capturing images and video of a second sector area directly in front of the vehicle;

a Left Front Side Radar (Left Front Side Radar) and a Right Front Side Radar (Right Front Side Radar) are respectively arranged at the Left Front Side and the Right Front Side of the vehicle and are used for respectively detecting obstacles at the Left Front Side and the Right Front Side of the vehicle;

a Left Rear Radar (Left read Side Radar) and a Right Rear Radar (Right read Side Radar) are respectively installed at the Left Rear Side and the Right Rear Side of the vehicle, and are used for respectively detecting obstacles at the Left Rear Side and the Right Rear Side of the vehicle.

Based on the above detailed description, the forward long-distance radar, the forward camera, the left front side radar, and the right front side radar are all track measurement sensors, and the output is track measurement data (i.e. the aforementioned first measurement track). For example, the left front radar detects an obstacle in front of the left of the vehicle, processes the detected obstacle data to obtain track measurement data corresponding to the obstacle, and outputs the track measurement data. In this embodiment, the radar is a millimeter wave radar.

Based on the above detailed description, the left rear side radar and the right rear side radar are both obstacle determination sensors, and output determination information indicating whether an obstacle exists. For example, the left rear radar detects an obstacle in the left rear of the vehicle, processes the detected obstacle data, determines whether there is an obstacle in the left rear of the vehicle, obtains determination information, and outputs the determination information.

In this embodiment, as can be seen from fig. 15, the sensing ranges of adjacent sensors have a certain overlap, which can ensure the improvement of the sensing reliability of the overlapped region on the one hand, and ensure the data consistency when the obstacle passes through the sensing ranges of different sensors on the other hand. It can be seen from fig. 15 that to ensure that the most important vehicle front has high reliability, forward remote radar, forward camera, left front side radar and right front side radar have more overlapping perception regions, which is beneficial to reducing false detection and missing detection, and simultaneously the left front side radar and the right front side radar have the ability of perceiving the larger range of the left front and the right front of the vehicle. In addition, the left rear side radar and the right rear side radar have a small range blind area, but for an obstacle with a certain relative speed, the tracking can still be continuously performed through track keeping and fusion.

Based on the vehicle-mounted sensors shown in fig. 15, the multi-sensor data fusion flow is described as follows (1) to (7):

(1) the vehicle-mounted device acquires the track measurement data output by the forward long-distance radar, the forward camera, the left front side radar and the right front side radar and the measurement time (namely the first measurement time) corresponding to the track measurement data, and acquires the judgment information output by the left rear side radar and the right rear side radar.

(2) And the vehicle-mounted equipment unifies the data format of the acquired track measurement data. Because the formats of the track measurement data output by different sensors are different, in order to facilitate subsequent data fusion, the data formats need to be unified, and the unified mode can continue to use the prior art, which is not described herein again.

(3) And the vehicle-mounted equipment carries out time synchronization on the stored system flight path based on the measuring time corresponding to the flight path measuring data to obtain the predicted flight path corresponding to each system flight path. The system path may be understood as the historical path of the obstacle.

(4) And matching the predicted track with the track measurement data by the vehicle-mounted equipment, wherein if the predicted track and the track measurement data are matched, the predicted track and the track measurement data are the track of the same obstacle at the same moment, and the system track corresponding to the predicted track and the track measurement data are the track of the same obstacle at different moments. Thus, if there is a match, the system track is associated with the track measurement data.

(5) The vehicle-mounted equipment updates the associated system track by using the track measurement data and updates the unassociated system track by using the predicted track. Since the vehicle is moving and the obstacle is moving relative to the vehicle, the system track needs to be updated in order to represent the track of the obstacle at different times.

(6) And after all the system tracks are updated, the vehicle-mounted equipment manages the reliability of the updated system tracks so as to determine whether the updated system tracks and the system tracks before updating aim at the same obstacle.

(7) And after managing the reliability of the updated system track, the vehicle-mounted equipment outputs the system track with higher reliability, maintains a system track list, and gives the reliability based on the acquired judgment information output by the left rear side radar and the right rear side radar.

In this embodiment, specific details of each step may refer to the embodiments shown in fig. 1 to 14, which are not described herein again.

In some embodiments, as shown in fig. 16, a system including a multi-sensor data fusion function, the system being a soft-hard combined system, hereinafter referred to as a fusion system, includes the following components: sensing (persistence), State estimation (State Estimate), Fusion (Fusion), Situation assessment (situational assessment), and Feature Function (Feature Function). Wherein, except for sensing (permission), the execution subject of other components may be the vehicle-mounted device. The components are specifically described as follows:

the sensing (perceivion) includes: a MAP (MAP) module, a forward Camera (Front Camera), a forward Long Range Radar (Front Long Range), a Vehicle System (Vehicle System), a Left Front Side Radar (Left Front Side Radar), a Right Front Side Radar (Right Front Side Radar), a Left Rear Side Radar (Left Rear Side Radar), and a Right Rear Side Radar (Right Rear Side Radar). The MAP (MAP) module may provide MAP information of the environment where the vehicle is currently located, and may also provide a positioning function, such as a GPS function. Vehicle systems (Vehicle systems) may provide Vehicle status information, including Vehicle speed and Vehicle steering angle, etc.

The State estimation (State Estimate) includes: the Vehicle location (Ego Vehicle location) and the Vehicle State Estimate (Vehicle State Estimate). Wherein the positioning of the host vehicle may be understood as determining the location of the host vehicle based on data from a MAP (MAP) module, a forward-facing camera, a forward-facing remote radar, and a vehicle system. Vehicle state estimation may be understood as estimating the state of the host vehicle from vehicle state information provided by a vehicle system.

Fusion (Fusion) includes: forward Fusion (Front Fusion) and Global Fusion (Global Fusion). Wherein, forward fusion can be understood as forward data fusion according to data of a forward camera, a forward long-distance radar and vehicle state estimation. Global fusion may be understood as global data fusion based on the results of forward fusion, left front radar, right front radar, left rear radar, right rear radar, and vehicle state estimation data.

Situation assessment (Situation Assss) includes: forward Object Selection (Front Object Selection) and backward or Side Object Selection (reader or Side Object Selection). Wherein the target is understood to be an obstacle. The forward target selection can be understood as selecting the barrier corresponding to the system track with the highest reliability as the forward target according to the data of the forward fusion and the global fusion, and meanwhile, the forward target selection can assist in selecting the forward target according to the data of the vehicle positioning and the vehicle state estimation, and the forward target which is obviously not the barrier is excluded. Backward or lateral target selection can be understood as selecting a backward obstacle or a lateral obstacle corresponding to the system track with the highest reliability as a backward or lateral target according to the globally fused data, and meanwhile, the backward or lateral target selection can assist in selecting the backward or lateral target according to the data of the vehicle positioning and the vehicle state estimation, and the backward or lateral target which is obviously not an obstacle is excluded.

The main functions (Feature functions) include: an Automatic Emergency Braking (AEB) system, an Adaptive Cruise Control (ACC) system, a Forward Collision Warning (FCW) system, and an automatic lane changing system. And each functional system in the main functions acquires data selected by the forward target and the backward or lateral target to execute corresponding functions.

In this embodiment, forward fusion can be used as an independent module to fuse data of a forward remote radar and a forward camera, and simultaneously, a fusion result is sent to global fusion as output, and secondary data fusion is performed by global fusion. The fusion system can eliminate some targets which are not possible to exist through data of a MAP (MAP) module, reduces calculation and errors, estimates the track of the vehicle by receiving vehicle state information provided by a vehicle system, including vehicle information such as vehicle speed, vehicle steering angle and the like, eliminates targets which do not need to be calculated, and further reduces calculation load.

The system with the multi-sensor data fusion function provided by the embodiment is convenient for module segmentation, improves the adaptive capacity, and can be adapted to auxiliary driving systems and unmanned driving systems with different requirements.

In addition, the system including the multi-sensor data fusion function provided in the present embodiment may be adapted to a fusion system having only a sensor for presence detection, or may be adapted to a fusion system having a sensor for obstacle detection.

As shown in fig. 17, the present embodiment discloses a multi-sensor data fusion apparatus, which may include the following units: the device comprises an acquisition unit 1, a unification unit 2, a synchronization unit 3, a determination unit 4 and an update unit 5. The units are specifically described as follows:

the system comprises an acquisition unit 1, a tracking unit and a tracking unit, wherein the acquisition unit is used for acquiring first measurement tracks of a plurality of track measurement sensors and first measurement time corresponding to each first measurement track;

the unifying unit 2 is configured to unify a data format of each first measurement track to obtain a second measurement track corresponding to each first measurement track;

the synchronization unit 3 is configured to perform time synchronization on each stored system track based on each first measurement time to obtain a predicted track corresponding to each system track;

a determining unit 4, configured to determine, based on each of the predicted tracks, an association relationship between each of the second measured tracks and each of the system tracks;

and the updating unit 5 is used for updating the stored system track based on the incidence relation.

The multi-sensor data fusion device disclosed in this embodiment corresponds to the multi-sensor data fusion method shown in fig. 1, and for specific description and effects, reference may be made to the method embodiment shown in fig. 1, and in order to avoid repetition, details are not repeated here.

In some embodiments, as shown in fig. 18, in addition to including the units shown in fig. 17, focusing on the specific implementation of the determining unit 4, the determining unit 4 may include a finding subunit 41 and an associating subunit 42. The concrete description is as follows:

a searching subunit 41, configured to search, from all the second measured trajectories, a second measured trajectory that matches the predicted trajectory;

and the associating subunit 42 is configured to associate the system track corresponding to the predicted track with the found second measured track.

The multi-sensor data fusion device disclosed in this embodiment corresponds to the multi-sensor data fusion method shown in fig. 2, and for specific description and effects, reference may be made to the method embodiment shown in fig. 2, and in order to avoid repetition, details are not repeated here.

Based on the previous example, the embodiment of the lookup subunit 41 is described in this embodiment with emphasis, and the lookup subunit 41 is specifically configured to perform the following steps a and B:

A. searching whether at least one second measurement track exists in a preset range around the predicted track by taking the predicted track as a center; if yes, executing step B;

B. a second measured track closest to the predicted track matches the predicted track.

The multi-sensor data fusion device disclosed in this embodiment corresponds to the multi-sensor data fusion method shown in fig. 3, and for specific description and effects, reference may be made to the method embodiment shown in fig. 3, and in order to avoid repetition, details are not repeated here.

In some embodiments, as shown in FIG. 19, in addition to including the units shown in FIG. 17, a storage unit 6 and a lookup unit 7 may be included. The concrete description is as follows:

a storage unit 6, configured to store the association relationship between the second measured flight path and the system flight path after the determining unit 4 determines the association relationship between each second measured flight path and each system flight path;

the searching unit 7 is used for searching whether the incidence relation between the second measurement track and the system track is stored or not after the synchronization unit 3 obtains the predicted track corresponding to each system track;

accordingly, the determining unit 4 is configured to determine, based on each of the predicted tracks, an association relationship between each of the second measured tracks and each of the system tracks after the searching unit 7 determines that the second measured tracks are not stored.

The multi-sensor data fusion device disclosed in this embodiment corresponds to the multi-sensor data fusion method shown in fig. 4, and for specific description and effects, reference may be made to the method embodiment shown in fig. 4, and in order to avoid repetition, details are not repeated here.

Based on the above example, as shown in fig. 20, in addition to the units shown in fig. 19, the first judgment unit 8 and the maintenance unit 9 may be further included. The concrete description is as follows:

the first judging unit 8 is used for judging whether the second measurement track and the system track in the association relation meet the preset matching condition after the searching unit 7 determines that the association relation between the second measurement track and the system track is stored;

a maintaining unit 9, configured to maintain the association relationship after the first determining unit 8 determines that a preset matching condition is satisfied;

the determining unit 4 is configured to determine, based on each predicted flight path, an association relationship between each second measured flight path and each system flight path after the first determining unit 8 determines that the preset matching condition is not satisfied.

The multi-sensor data fusion device disclosed in this embodiment corresponds to the multi-sensor data fusion method shown in fig. 5, and for specific description and effects, reference may be made to the method embodiment shown in fig. 5, and in order to avoid repetition, details are not repeated here.

In some embodiments, as shown in fig. 21, in addition to including the units shown in fig. 17, the description focuses on a specific implementation of the updating unit 5, and the updating unit 5 includes: a judgment subunit 51, a first update subunit 52 and a second update subunit 53. The concrete description is as follows:

a judging subunit 51, configured to judge whether each stored system track is a system track in the association relationship;

a first updating subunit 52, configured to update the system track by using a second measured track corresponding to the system track in the association relationship after the determining subunit 51 determines that the system track is the system track in the association relationship;

and a second updating subunit 53, configured to update the system track by using the predicted track corresponding to the system track after the determining subunit 52 determines that the system track is not the system track in the association relationship.

The multi-sensor data fusion device disclosed in this embodiment corresponds to the multi-sensor data fusion method shown in fig. 6, and for specific description and effects, reference may be made to the method embodiment shown in fig. 6, and in order to avoid repetition, details are not repeated here.

In some embodiments, as shown in fig. 22, in addition to the elements shown in fig. 17, a management unit 10 may be included:

and the management unit 10 is configured to manage the reliability of the updated system track based on an update mode of the system track after the update unit 5 updates the stored system track.

The multi-sensor data fusion device disclosed in this embodiment corresponds to the multi-sensor data fusion method shown in fig. 7, and for specific description and effects, reference may be made to the method embodiment shown in fig. 7, and in order to avoid repetition, details are not repeated here.

Based on the above example, the embodiment of the management unit 10 is described in the present embodiment with emphasis, and the management unit 10 is specifically configured to execute the following steps C, D and E:

C. and if the updating mode of the system track is to update by using the second measured track in the incidence relation, the reliability of the system track is increased.

D. And if the updating mode of the system track is to use the predicted track for updating, acquiring a track measuring sensor corresponding to the updating mode.

E. After the track measuring sensor corresponding to the updating mode is obtained in the step D, whether the system track is in the sensing range of the track measuring sensor corresponding to the updating mode or not is judged, and if the system track is in the sensing range, the reliability of the system track is reduced; and if not, maintaining the reliability of the system track.

The multi-sensor data fusion device disclosed in this embodiment corresponds to the multi-sensor data fusion method shown in fig. 8, and for specific description and effects, reference may be made to the method embodiment shown in fig. 8, and in order to avoid repetition, details are not repeated here.

Based on the above example, the management unit 10 is further configured to perform the following steps F and G:

f: after the reliability of the system track is reduced in the step E, judging whether the reliability of the system track is lower than a preset elimination threshold or not; if the value is lower than the preset elimination threshold, executing the step G;

g: and eliminating the system track.

The multi-sensor data fusion device disclosed in this embodiment corresponds to the multi-sensor data fusion method shown in fig. 10, and for specific description and effects, reference may be made to the method embodiment shown in fig. 10, and for avoiding repetition, details are not repeated here.

In some embodiments, as shown in fig. 23, in addition to the units shown in fig. 22, a second determination unit 11 and an output unit 12 may be included. The concrete description is as follows:

a second judging unit 11, configured to judge, after the management unit 10 manages the reliability of the updated system track, whether the reliability of the managed system track reaches a preset output threshold, so as to obtain a system track that reaches the preset output threshold;

and the output unit 12 is used for outputting the system track reaching the preset output threshold.

The multi-sensor data fusion device disclosed in this embodiment corresponds to the multi-sensor data fusion method shown in fig. 9, and for specific description and effects, reference may be made to the method embodiment shown in fig. 9, and in order to avoid repetition, details are not repeated here.

In some embodiments, as shown in fig. 24, in addition to the units shown in fig. 22, an initialization unit 13 may be included:

an initializing unit 13, configured to initialize a second measurement track that is not used by the association relationship in a second measurement track corresponding to each first measurement track after the managing unit 10 manages the reliability of the updated system track.

The multi-sensor data fusion device disclosed in this embodiment corresponds to the multi-sensor data fusion method shown in fig. 11, and for specific description and effects, reference may be made to the method embodiment shown in fig. 11, and in order to avoid repetition, details are not repeated here.

Based on the above example, the initialization unit 13 performs the initialization process, specifically including the following steps 1121 to 1126:

1121. and acquiring the state information of the vehicle.

The multi-sensor data fusion device disclosed in this embodiment corresponds to the multi-sensor data fusion method shown in fig. 12, and for specific description and effects, reference may be made to the method embodiment shown in fig. 12, and for avoiding repetition, details are not repeated here.

Based on the above example, the initialization unit 13 performs the initialization process, and may include, in addition to steps 1121 through 1126, the following step 1127:

The multi-sensor data fusion device disclosed in this embodiment corresponds to the multi-sensor data fusion method shown in fig. 13, and for specific description and effects, reference may be made to the method embodiment shown in fig. 13, and in order to avoid repetition, details are not repeated here.

In some embodiments, the initialization unit 13 performs the initialization process, and may include the following steps 1128 to 1132 in addition to the steps 1121 to 1126:

wherein the first confidence level is greater than the second confidence level.

The multi-sensor data fusion device disclosed in this embodiment corresponds to the multi-sensor data fusion method shown in fig. 14, and for specific description and effects, reference may be made to the method embodiment shown in fig. 14, and for avoiding repetition, details are not repeated here.

An embodiment of the present invention further provides a non-transitory computer-readable storage medium, which stores computer instructions, where the computer instructions cause the computer to perform the method steps provided in the method embodiments of the first aspect, for example, including:

and updating the stored system track based on the incidence relation.

The embodiment of the invention also provides:

a1, a multi-sensor data fusion method, the method comprising:

and updating the stored system track based on the incidence relation.

A2, the method according to A1, wherein the determining the association between each of the second measured tracks and each of the system tracks based on each of the predicted tracks comprises:

A3, the finding a second measured track matching the predicted track according to the method of A2, comprising:

A4, after the determining the association between each of the second measured tracks and each of the system tracks according to the method of A1, the method further comprising:

A5, the method of A4, the method further comprising:

if so, maintaining the association relationship;

A6, the method according to A1, wherein the updating the stored system track based on the incidence relation includes:

A7, after the updating the stored system track according to the method of A1, the method further comprising:

A8, according to the method in A7, the method for managing the credibility of the updated system track based on the updating mode of the system track comprises the following steps:

A9, according to the method in A8, the method for managing the credibility of the updated system track based on the updating mode of the system track comprises the following steps:

A10, after managing the credibility of the updated system track according to the method of A7, the method further comprising:

and outputting the system track reaching the preset output threshold.

A11, after the reducing the confidence level of the system track according to the method of A9, the method further comprises:

A12, after managing the credibility of the updated system track according to the method of A7, the method further comprising:

A13, the method of A12, the initializing process comprising:

obtaining the state information of the vehicle;

estimating a vehicle trajectory based on the vehicle state information;

A14, the method of A13, the initialization process further comprising:

A15, the method of A13, the initialization process further comprising:

acquiring judgment information of a plurality of obstacle judgment sensors;

determining whether an obstacle exists according to the judgment information;

wherein the first confidence level is greater than the second confidence level.

A16, the method of any one of A1 to A15, the plurality of track measurement sensors comprising: a forward image sensor, a forward radar, a left front side radar, and a right front side radar.

A17, the method of a15, the plurality of obstacle determination sensors comprising: a left rear radar and a right rear radar.

A18, a multi-sensor data fusion device, the device comprising:

A19, the apparatus of A18, the determining unit comprising:

A20, the apparatus of A19, the lookup subunit to:

A21, the apparatus of A18, further comprising:

A22, the apparatus of A21, further comprising:

A23, the apparatus of A18, the update unit comprising:

A24, the apparatus of A18, further comprising:

A25, the apparatus of A24, the management unit to:

A26, the apparatus of A25, the management unit to:

A27, the apparatus of A24, further comprising:

A28, the apparatus of A26, the management unit further to:

A29, the apparatus of A24, further comprising:

A30, the apparatus of A29, the initialization unit performing initialization processing, comprising:

obtaining the state information of the vehicle;

estimating a vehicle trajectory based on the vehicle state information;

A31, the apparatus of a30, the initialization unit performing initialization processing, further comprising:

A32, the apparatus of a30, the initialization unit performing initialization processing, further comprising:

acquiring judgment information of a plurality of obstacle judgment sensors;

determining whether an obstacle exists according to the judgment information;

wherein the first confidence level is greater than the second confidence level.

A33, the device of any one of A18 to A32, the plurality of track measurement sensors comprising: a forward image sensor, a forward radar, a left front side radar, and a right front side radar.

A34, the apparatus of a32, the plurality of obstacle determination sensors comprising: a left rear radar and a right rear radar.

A35, an in-vehicle apparatus, comprising:

a processor, memory, a network interface, and a user interface;

the processor is operable to perform the steps of the method of any one of A1 to A17 by calling a program or instructions stored in the memory.

A36, a non-transitory computer readable storage medium storing computer instructions for causing the computer to perform the steps of the method of any one of a1 to a 17.

It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.

Those skilled in the art will appreciate that although some embodiments described herein include some features included in other embodiments instead of others, combinations of features of different embodiments are meant to be within the scope of the invention and form different embodiments.

Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the invention, and such modifications and variations fall within the scope defined by the appended claims.

Claims

1. A multi-sensor data fusion method, the method comprising:

acquiring first measurement tracks of a plurality of track measurement sensors and first measurement time corresponding to each first measurement track; the flight path measuring sensor acquires barrier data in a detection range;

based on each first measurement time, performing time synchronization on each stored system track to obtain a predicted track corresponding to each system track; the system track is an obstacle historical track;

updating the stored system flight path based on the incidence relation;

managing the reliability of the updated system track based on the updating mode of the system track;

wherein the managing the reliability of the updated system track comprises: and if the updating mode of the system track is to update by using the second measured track in the incidence relation, the reliability of the system track is increased.

2. The method of claim 1, wherein said determining an association between each of said second measured trajectories and each of said system trajectories based on each of said predicted trajectories comprises:

3. The method of claim 2, wherein finding the second measured trajectory that matches the predicted trajectory comprises:

4. The method of claim 1, wherein after determining the correlation between each of the second measured tracks and each of the system tracks, the method further comprises:

5. The method of claim 4, further comprising:

if so, maintaining the association relationship;

6. The method of claim 1, wherein updating the stored system track based on the incidence relation comprises:

7. The method according to claim 1, wherein managing the credibility of the updated system track based on the updating manner of the system track comprises:

8. The method of claim 1, wherein after managing the trustworthiness of the updated system track, the method further comprises:

and outputting the system track reaching the preset output threshold.

9. The method of claim 7, wherein after the reducing the confidence level of the system track, the method further comprises:

10. The method of claim 1, wherein after managing the trustworthiness of the updated system track, the method further comprises:

11. The method of claim 10, wherein the initialization process comprises:

obtaining the state information of the vehicle;

estimating a vehicle trajectory based on the vehicle state information;

12. The method of claim 11, wherein the initialization process further comprises:

13. The method of claim 11, wherein the initialization process further comprises:

acquiring judgment information of a plurality of obstacle judgment sensors;

determining whether an obstacle exists according to the judgment information;

wherein the first confidence level is greater than the second confidence level.

14. The method of any one of claims 1 to 13, wherein the plurality of track measurement sensors comprises: a forward image sensor, a forward radar, a left front side radar, and a right front side radar.

15. The method according to claim 13, wherein the plurality of obstacle determination sensors include: a left rear radar and a right rear radar.

16. A multi-sensor data fusion apparatus, the apparatus comprising:

the system comprises an acquisition unit, a processing unit and a processing unit, wherein the acquisition unit is used for acquiring first measurement tracks of a plurality of track measurement sensors and first measurement time corresponding to each first measurement track; the flight path measuring sensor acquires barrier data in a detection range;

the synchronization unit is used for carrying out time synchronization on each stored system track based on each first measurement time to obtain a predicted track corresponding to each system track; the system track is an obstacle historical track;

the updating unit is used for updating the stored system track based on the incidence relation;

the management unit is used for managing the credibility of the updated system track based on the updating mode of the system track after the updating unit updates the stored system track;

and the management unit is used for increasing the reliability of the system track if the updating mode of the system track is to use the second measurement track in the incidence relation for updating.

17. The apparatus of claim 16, wherein the determining unit comprises:

18. The apparatus of claim 17, wherein the lookup subunit is configured to:

19. The apparatus of claim 16, further comprising:

20. The apparatus of claim 19, further comprising:

21. The apparatus of claim 16, wherein the updating unit comprises:

22. The apparatus of claim 16, wherein the management unit is configured to:

23. The apparatus of claim 16, further comprising:

24. The apparatus of claim 22, wherein the management unit is further configured to:

25. The apparatus of claim 16, further comprising:

26. The apparatus according to claim 25, wherein the initialization unit performs an initialization process including:

obtaining the state information of the vehicle;

estimating a vehicle trajectory based on the vehicle state information;

27. The apparatus according to claim 26, wherein the initialization unit performs initialization processing, and further comprises:

28. The apparatus according to claim 26, wherein the initialization unit performs initialization processing, and further comprises:

acquiring judgment information of a plurality of obstacle judgment sensors;

determining whether an obstacle exists according to the judgment information;

wherein the first confidence level is greater than the second confidence level.

29. The apparatus of any one of claims 16 to 28, wherein the plurality of track measurement sensors comprises: a forward image sensor, a forward radar, a left front side radar, and a right front side radar.

30. The apparatus according to claim 28, wherein the plurality of obstacle judging sensors comprise: a left rear radar and a right rear radar.

31. An in-vehicle apparatus, characterized by comprising:

a processor, memory, a network interface, and a user interface;

the processor is adapted to perform the steps of the method of any one of claims 1 to 15 by calling a program or instructions stored in the memory.

32. A non-transitory computer readable storage medium storing computer instructions for causing a computer to perform the steps of the method according to any one of claims 1 to 15.