CN111860551A - Multi-sensor data fusion method and device and vehicle-mounted terminal - Google Patents
Multi-sensor data fusion method and device and vehicle-mounted terminal Download PDFInfo
- Publication number
- CN111860551A CN111860551A CN201910346798.8A CN201910346798A CN111860551A CN 111860551 A CN111860551 A CN 111860551A CN 201910346798 A CN201910346798 A CN 201910346798A CN 111860551 A CN111860551 A CN 111860551A
- Authority
- CN
- China
- Prior art keywords
- sensor
- data
- sensor data
- storage space
- timestamp
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000007500 overflow downdraw method Methods 0.000 title claims abstract description 11
- 230000004927 fusion Effects 0.000 claims abstract description 76
- 238000000034 method Methods 0.000 claims abstract description 32
- 238000012795 verification Methods 0.000 claims description 28
- 238000012217 deletion Methods 0.000 claims description 10
- 230000037430 deletion Effects 0.000 claims description 10
- 238000004364 calculation method Methods 0.000 claims description 7
- 238000005259 measurement Methods 0.000 claims description 7
- 238000010586 diagram Methods 0.000 description 8
- 238000013480 data collection Methods 0.000 description 6
- 238000012545 processing Methods 0.000 description 3
- 238000001514 detection method Methods 0.000 description 2
- 230000008447 perception Effects 0.000 description 2
- 230000001960 triggered effect Effects 0.000 description 2
- 238000013500 data storage Methods 0.000 description 1
- 230000003247 decreasing effect Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 230000002035 prolonged effect Effects 0.000 description 1
- 238000012163 sequencing technique Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/25—Fusion techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/901—Indexing; Data structures therefor; Storage structures
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- Databases & Information Systems (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Evolutionary Computation (AREA)
- Evolutionary Biology (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Bioinformatics & Computational Biology (AREA)
- Artificial Intelligence (AREA)
- Software Systems (AREA)
- Life Sciences & Earth Sciences (AREA)
- Navigation (AREA)
Abstract
The embodiment of the invention discloses a multi-sensor data fusion method and device and a vehicle-mounted terminal. The method comprises the following steps: determining a first timestamp with a latest moment in a plurality of first sensor data in a storage space; subtracting a preset maximum time delay from the first time stamp to obtain a second time stamp earlier than the first time stamp; deleting the first sensor data and the second sensor data with the time stamp earlier than the second time stamp from the storage space; performing data fusion according to the remaining first sensor data and second sensor data in the storage space; in the data acquisition process, the first sensor and the second sensor can acquire data within the maximum time delay; the stability when the first sensor collects data is greater than the stability when the second sensor collects data. By applying the scheme provided by the embodiment of the invention, the data of each sensor used for data fusion has timeliness and comprehensive data, and the stability of the data can be considered.
Description
Technical Field
The invention relates to the technical field of intelligent driving, in particular to a multi-sensor data fusion method and device and a vehicle-mounted terminal.
Background
In the driving process of the intelligent driving vehicle, a control platform in the intelligent vehicle can collect data of various sensors mounted on the intelligent vehicle, fuse the data of the various sensors, and obtain a high-precision positioning result or detection result according to the fusion result, so that the control platform can better guide the control of the intelligent vehicle according to the positioning result or the detection result.
When various sensor data are fused, the various sensor data need to be acquired at the same time. Because the frequency of each sensor when gathering data is different, and each kind of sensor data is gathered and stored constantly, need to carry out the replacement processing to the data of storage. How to enable various sensor data used for data fusion after replacement processing to have timeliness and contain various sensor data is a problem to be solved in data fusion.
Disclosure of Invention
The invention provides a multi-sensor data fusion method and device and a vehicle-mounted terminal, so that each sensor data used for data fusion has timeliness and comprehensive data, and the stability of the data can be considered. The specific technical scheme is as follows.
In a first aspect, an embodiment of the present invention discloses a vehicle-mounted terminal, including: a processor, a first sensor and a second sensor; the processor includes: the device comprises a storage module, a determination module, a calculation module, a deletion module and a fusion module; the data acquisition frequency of the first sensor is different from that of the second sensor, and the stability of the first sensor in data acquisition is greater than that of the second sensor in data acquisition;
the storage module is used for storing the acquired first sensor data acquired by the first sensor into a storage space and storing the acquired second sensor data acquired by the second sensor into the storage space;
the determining module is configured to determine a latest first timestamp in the plurality of first sensor data in the storage space;
the calculation module is used for subtracting a preset maximum time delay from the first time stamp to obtain a second time stamp which is earlier than the first time stamp; in the data acquisition process, the first sensor and the second sensor can acquire data within the maximum time delay;
the deleting module is used for deleting the first sensor data and the second sensor data with the time stamps earlier than the second time stamps from the storage space;
And the fusion module is used for carrying out data fusion according to the remaining first sensor data and the second sensor data in the storage space.
Optionally, the determining module is specifically configured to:
when it is detected that the latest first sensor data is stored in the storage space, the timestamp of the latest first sensor data is determined as the latest first timestamp among the plurality of first sensor data in the storage space.
Optionally, the fusion module is specifically configured to:
and when the multi-sensor data fusion is determined to be needed, performing data fusion according to the data with the earliest time stamp in the first sensor data and the second sensor data which are left in the storage space.
Optionally, when the storage module stores the acquired second sensor data acquired by the second sensor to the storage space, the storage module includes:
adjusting the time stamp of the second sensor data according to the predetermined time difference information between the system clock of the second sensor and the system clock of the first sensor, and storing the second sensor data with the time stamp adjusted in the storage space;
The deletion module is specifically configured to:
deleting first sensor data with a timestamp earlier than the second timestamp and deleting second sensor data with an adjusted timestamp earlier than the second timestamp from the storage space.
Optionally, when the storage module stores the second sensor data with the adjusted timestamp in the storage space, the storage module includes:
verifying the second sensor data after the timestamp is adjusted according to the attribute information of the second sensor data after the timestamp is adjusted to obtain a verification result; wherein the attribute information includes time information and/or data amount information of the second sensor data;
and when the second sensor data with the timestamp adjusted passes the verification according to the verification result, storing the second sensor data with the timestamp adjusted into the storage space.
Optionally, the first sensor is an inertial measurement unit IMU, and the second sensor includes at least one of the following sensors: image sensor, global navigation satellite system, odometer.
In a second aspect, an embodiment of the present invention provides a multi-sensor data fusion method, applied to a processor, including:
Storing acquired first sensor data acquired by a first sensor into a storage space, and storing acquired second sensor data acquired by a second sensor into the storage space; the data acquisition frequency of the first sensor is different from that of the second sensor, and the stability of the first sensor in data acquisition is greater than that of the second sensor in data acquisition;
determining a first timestamp of the first plurality of sensor data in the storage space that is latest in time;
subtracting a preset maximum time delay from the first time stamp to obtain a second time stamp earlier than the first time stamp; in the data acquisition process, the first sensor and the second sensor can acquire data within the maximum time delay;
deleting the first sensor data and the second sensor data with the time stamps earlier than the second time stamps from the storage space;
and performing data fusion according to the remaining first sensor data and the second sensor data in the storage space.
Optionally, the step of determining a latest first time stamp in the plurality of first sensor data in the storage space includes:
When it is detected that the latest first sensor data is stored in the storage space, the timestamp of the latest first sensor data is determined as the latest first timestamp among the plurality of first sensor data in the storage space.
Optionally, the step of performing data fusion according to the remaining first sensor data and second sensor data in the storage space includes:
and when the multi-sensor data fusion is determined to be needed, performing data fusion according to the data with the earliest time stamp in the first sensor data and the second sensor data which are left in the storage space.
Optionally, the step of storing the acquired second sensor data acquired by the second sensor in the storage space includes:
adjusting the time stamp of the second sensor data according to the predetermined time difference information between the system clock of the second sensor and the system clock of the first sensor, and storing the second sensor data with the time stamp adjusted in the storage space;
the step of deleting the first sensor data and the second sensor data having the time stamps earlier than the second time stamp from the storage space includes:
Deleting first sensor data with a timestamp earlier than the second timestamp and deleting second sensor data with an adjusted timestamp earlier than the second timestamp from the storage space.
Optionally, the step of storing the second sensor data after adjusting the timestamp in the storage space includes:
verifying the second sensor data after the timestamp is adjusted according to the attribute information of the second sensor data after the timestamp is adjusted to obtain a verification result; wherein the attribute information includes time information and/or data amount information of the second sensor data;
and when the second sensor data with the timestamp adjusted passes the verification according to the verification result, storing the second sensor data with the timestamp adjusted into the storage space.
Optionally, the first sensor is an inertial measurement unit IMU, and the second sensor includes at least one of the following sensors: image sensor, global navigation satellite system, odometer.
In a third aspect, an embodiment of the present invention discloses a multi-sensor data fusion apparatus, applied to a processor, including:
the storage module is configured to store the acquired first sensor data acquired by the first sensor into a storage space and store the acquired second sensor data acquired by the second sensor into the storage space; the data acquisition frequency of the first sensor is different from that of the second sensor, and the stability of the first sensor in data acquisition is greater than that of the second sensor in data acquisition;
A determining module configured to determine a latest first timestamp of a time in the plurality of first sensor data in the storage space;
a calculation module configured to subtract a preset maximum delay from the first timestamp to obtain a second timestamp earlier than the first timestamp; in the data acquisition process, the first sensor and the second sensor can acquire data within the maximum time delay;
a deletion module configured to delete the first sensor data and the second sensor data having a timestamp earlier than the second timestamp from the storage space;
and the fusion module is configured to perform data fusion according to the first sensor data and the second sensor data which are remained in the storage space.
Optionally, the determining module is specifically configured to:
when it is detected that the latest first sensor data is stored in the storage space, the timestamp of the latest first sensor data is determined as the latest first timestamp among the plurality of first sensor data in the storage space.
Optionally, the fusion module is specifically configured to:
and when the multi-sensor data fusion is determined to be needed, performing data fusion according to the data with the earliest time stamp in the first sensor data and the second sensor data which are left in the storage space.
Optionally, when the storage module stores the acquired second sensor data acquired by the second sensor to the storage space, the storage module includes:
adjusting the time stamp of the second sensor data according to the predetermined time difference information between the system clock of the second sensor and the system clock of the first sensor, and storing the second sensor data with the time stamp adjusted in the storage space;
the deletion module is specifically configured to:
deleting first sensor data with a timestamp earlier than the second timestamp and deleting second sensor data with an adjusted timestamp earlier than the second timestamp from the storage space.
Optionally, when the storage module stores the second sensor data with the adjusted timestamp in the storage space, the storage module includes:
verifying the second sensor data after the timestamp is adjusted according to the attribute information of the second sensor data after the timestamp is adjusted to obtain a verification result; wherein the attribute information includes time information and/or data amount information of the second sensor data;
and when the second sensor data with the timestamp adjusted passes the verification according to the verification result, storing the second sensor data with the timestamp adjusted into the storage space.
Optionally, the first sensor is an inertial measurement unit IMU, and the second sensor includes at least one of the following sensors: image sensor, global navigation satellite system, odometer.
As can be seen from the above, in the multi-sensor data fusion method and apparatus and the in-vehicle terminal provided in the embodiments of the present invention, the first sensor data and the second sensor data having the timestamps earlier than the second timestamp are deleted from the storage space, so that the remaining first sensor data and second sensor data in the storage space are updated data; the second timestamp is obtained by subtracting the maximum time delay from the first timestamp, and the maximum time delay enables the first sensor data and the second sensor data to exist in the storage space between the second timestamp and the first timestamp, so that each sensor data in the storage space can be more comprehensive; the stability of the first sensor data is larger when the first sensor collects data, the stability of the collected first sensor data is also larger, the time period when the data is deleted is determined according to the time stamp of the first sensor data, and the stability of the data in the storage space can be considered. Therefore, the embodiment of the invention can ensure that the data of each sensor used for data fusion not only has timeliness and comprehensive data, but also can give consideration to the stability of the data. Of course, not all of the advantages described above need to be achieved at the same time in the practice of any one product or method of the invention.
The innovation points of the embodiment of the invention comprise:
1. the latest data collected by the first sensor with higher stability is used as the first timestamp, the maximum time delay is subtracted from the first timestamp to obtain the second timestamp, the data earlier than the second timestamp are deleted from the storage space, the data left in the storage space can contain the data of each sensor, the data stability in the storage space is better, and meanwhile, the data in the storage space is updated.
2. The time stamps of the second sensor data are adjusted based on the time difference information between the system clock of the second sensor and the system clock of the first sensor, the time stamps of the sensor data can be unified under the system clock of the first sensor, the time synchronism of the data for data fusion can be improved, and the accuracy of the data fusion is improved.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below. It is to be understood that the drawings in the following description are merely exemplary of some embodiments of the invention. For a person skilled in the art, without inventive effort, further figures can be obtained from these figures.
Fig. 1 is a schematic structural diagram of a vehicle-mounted terminal according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of the relationship between the first and second timestamps and the respective sensor data of FIG. 1;
FIG. 3 is a schematic diagram of a fixed time offset between system clocks of different sensors;
FIG. 4 is a flowchart illustrating processing of threads in a processor according to an embodiment of the present invention;
FIG. 5 is a schematic flow chart of a multi-sensor data fusion method according to an embodiment of the present invention;
fig. 6 is a schematic structural diagram of a multi-sensor data fusion apparatus according to an embodiment of the present invention.
Detailed Description
The technical solution in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. It is to be understood that the described embodiments are merely a few embodiments of the invention, and not all embodiments. All other embodiments, which can be obtained by a person skilled in the art without inventive effort based on the embodiments of the present invention, are within the scope of the present invention.
It is to be noted that the terms "comprises" and "comprising" and any variations thereof in the embodiments and drawings of the present invention are intended to cover non-exclusive inclusions. For example, a process, method, system, article, or apparatus that comprises a list of steps or elements is not limited to only those steps or elements listed, but may alternatively include other steps or elements not listed, or inherent to such process, method, article, or apparatus.
The embodiment of the invention discloses a multi-sensor data fusion method and device and a vehicle-mounted terminal, which can enable each sensor data used for data fusion to have timeliness and comprehensive data and also give consideration to the stability of the data. The following provides a detailed description of embodiments of the invention.
Fig. 1 is a schematic structural diagram of a vehicle-mounted terminal according to an embodiment of the present invention. The vehicle-mounted terminal includes: a processor 110, a first sensor 120, and a second sensor 130. The processor 110 includes: the device comprises a storage module 11, a determination module 12, a calculation module 13, a deletion module 14 and a fusion module 15. The data collection frequency of the first sensor 120 is different from the data collection frequency of the second sensor 130, and the stability of the data collection of the first sensor 120 is greater than that of the data collection of the second sensor 130.
Wherein, the sensor can include: an image sensor, an Inertial Measurement Unit (IMU), an odometer (Odograph), a Global Navigation satellite system (Global Navigation satellite system), and the like. The processor may be a CPU or the like.
The data acquisition frequency is understood to be the number of data frames acquired in a unit time, and for example, Hz may be used as the unit of the data acquisition frequency.
The stability of the sensor in data acquisition can be obtained by monitoring the data acquisition process of the sensor in advance. This stability may be reflected in the number of dropped frames during a period of time that the sensor is collecting data, and the uniformity of the spacing of the collected frames of data. The smaller the number of lost frames is, the more uniform the interval of each frame data is, and the better the stability of the sensor in data acquisition is considered.
The storage module 11 is configured to store the acquired first sensor data acquired by the first sensor 120 in a storage space, and store the acquired second sensor data acquired by the second sensor 130 in the storage space.
In another embodiment, the processor 110 may further include a receiving module for receiving the first sensor data collected by the first sensor 120 and receiving the second sensor data collected by the second sensor 130. The receiving module is connected to the memory module 11.
The storage space can be located inside the vehicle-mounted terminal or outside the vehicle-mounted terminal. The storage space component may be a storage space in a hard disk or a storage space in a cache. The first sensor data and the second sensor data may be stored at different locations in the storage space, respectively.
The determining module 12 is configured to determine a latest first timestamp in the plurality of first sensor data in the storage space.
Each sensor data may include data content and a timestamp. The time stamp is used to indicate the time of acquisition of the data content. And sequencing and storing the first sensor data in the storage space according to the sequence of the timestamps, and determining the latest first timestamp at the moment according to the sequenced first sensor data.
And the calculating module 13 is configured to subtract the preset maximum time delay from the first timestamp to obtain a second timestamp earlier than the first timestamp.
During the data collection process, the first sensor 120 and the second sensor 130 can both collect data within a maximum time delay. The maximum time delay may be understood as a time period during which both the first sensor 120 and the second sensor 130 can acquire data. The data acquisition frequencies of the first sensor 120 and the second sensor 130 are different, and the maximum time delay is larger than the acquisition time interval between two adjacent frames of data of the sensor with the minimum data acquisition frequency.
For example, the data acquisition frequencies of the IMU, the GNSS, the ODO and the image sensor are 200Hz, 100Hz and 10Hz, respectively, where 10Hz is the minimum data acquisition frequency, the acquisition time interval between two corresponding adjacent frames of data is 100ms, and the maximum time delay may be a duration greater than 100 ms.
The maximum time delay is not suitable to be too large, so that the real-time performance of data in the storage space is not facilitated, the data storage time of the storage space is prolonged, and the utilization rate of the storage space is low. The maximum delay can be determined empirically.
For another example, when the first timestamp is 2:00:00:000 and the maximum latency is 200ms, the second timestamp obtained by subtracting 200ms from 2:00:00:000 may be 1:59:59: 800.
And a deleting module 14 for deleting the first sensor data and the second sensor data with the timestamp earlier than the second timestamp from the storage space.
And the fusion module 15 is configured to perform data fusion according to the remaining first sensor data and second sensor data in the storage space.
In an embodiment, the fusion module 15 may be specifically configured to perform data fusion according to the remaining first sensor data and second sensor data in the storage space when it is determined that multi-sensor data fusion is required.
In this embodiment, there may be one first sensor and one or more second sensors.
In one embodiment, the first sensor may be an IMU and the second sensor may include at least one of the following sensors: image sensor, global navigation satellite system, odometer.
Referring to fig. 2, vertically upward is a time axis t. At the same time, the IMU collects the most frames of data, followed by GNSS and ODO, and the least frames of data collected are the cameras. And the data acquired by each sensor are stored in sequence. When the first time stamp t1 and the second time stamp t2 are determined and the data earlier than t2 are deleted, the data between the horizontal line of t1 and the horizontal line of t2 can be obtained.
As is apparent from the above, in the present embodiment, the first sensor data and the second sensor data having the timestamps earlier than the second timestamp are deleted from the storage space, so that the remaining first sensor data and second sensor data in the storage space can be updated data; the second timestamp is obtained by subtracting the maximum time delay from the first timestamp, and the maximum time delay enables the first sensor data and the second sensor data to exist in the storage space between the second timestamp and the first timestamp, so that each sensor data in the storage space can be more comprehensive; the stability of the first sensor data is larger when the first sensor collects data, the stability of the collected first sensor data is also larger, the time period when the data is deleted is determined according to the time stamp of the first sensor data, and the stability of the data in the storage space can be considered. Therefore, the embodiment can enable the data of each sensor used for data fusion to have timeliness and comprehensive data and also have data stability.
In another embodiment of the present invention, based on the embodiment shown in fig. 1, the determining module 12 is specifically configured to:
when it is detected that the latest first sensor data is stored in the storage space, the timestamp of the latest first sensor data is determined as the latest first timestamp among the plurality of first sensor data in the storage space.
The latest first sensor data may be understood as the sensor data with the latest timestamp. The first sensor 120 may transmit one first sensor data at a time or may transmit a plurality of first sensor data at a time when transmitting data to the processor 110.
In this embodiment, when it is detected that the latest first sensor data is stored in the storage space, the determining module 12, the calculating module 13, and the deleting module 14 may be triggered, that is, the first timestamp and the second timestamp are triggered to be updated, so as to execute an operation of deleting the data in the storage space. Due to the fact that the stability of the first sensor in data collection is better, the data in the storage space can be updated more stably by adopting the triggering mode.
In another embodiment of the present invention, based on the embodiment shown in fig. 1, the fusion module 15 is specifically configured to:
And when the multi-sensor data fusion is determined to be needed, performing data fusion according to the data with the earliest time stamp in the first sensor data and the second sensor data which are left in the storage space.
Wherein, the frequency of data fusion can be different from the frequency of updating the data in the storage space.
In one embodiment, the need for multi-sensor data fusion may be determined when: when an instruction for data fusion is received; or when a preset time period comes.
When data fusion is performed, data with the earliest timestamp can be extracted from the first sensor data and the second sensor data which are left in the storage space, and the data fusion is performed. In extracting data, the respective sensor data may be extracted in different orders. The operation of data fusion may include an operation of correcting the positioning result of the smart vehicle according to the data of each sensor.
In this embodiment, data fusion is performed according to the oldest remaining data of the timestamps in the storage space, so that the timestamps of the sensor data used for data fusion are closer to the second timestamp, the difference between the acquisition times of the sensor data is not too large, and the time consistency of the data during data fusion is improved.
In another embodiment of the present invention, based on the embodiment shown in fig. 1, when the storage module 11 stores the acquired second sensor data collected by the second sensor into the storage space, the method may include:
and adjusting the time stamp of the second sensor data according to the predetermined time difference information between the system clock of the second sensor and the system clock of the first sensor, and storing the second sensor data with the time stamp adjusted in the storage space.
The system clock of the first sensor can be understood as the reference clock. The time stamp of each sensor data is determined by the corresponding sensor according to the system clock, and when the system clocks of the sensors are different, the determined time stamps of the sensor data are not synchronous. For example, the timestamps of data collected by different sensors may be different at the same time.
Referring to fig. 3, fig. 3 is a diagram illustrating a fixed time offset between an absolute time t in an IMU time axis and an absolute time t in a GNSS time axis.
When the time stamp of the second sensor data is adjusted, the time stamp of the second sensor data may be increased or decreased according to the time difference information. For example, when the time difference information indicates that the system clock of the second sensor is N milliseconds faster than the system clock of the first sensor, the timestamp of the second sensor data may be subtracted by N milliseconds. When the time difference information indicates that the system clock of the second sensor is N milliseconds slower than the system clock of the first sensor, the timestamp of the second sensor data may be added to the N milliseconds.
In this embodiment, the deleting module 14 is specifically configured to:
first sensor data with a timestamp earlier than the second timestamp is deleted from the storage space, and second sensor data with an adjusted timestamp earlier than the second timestamp is deleted.
There is a time difference between different clock hardware. Although these time differences are very small, perhaps on the order of milliseconds, in positioning systems where high accuracy is required, these time differences can cause large errors in the algorithm results. Therefore, the time systems of the data of the sensors are unified, and the accuracy of fusion can be improved.
In summary, in this embodiment, the time stamps of the second sensor data are adjusted based on the time difference information between the system clock of the second sensor and the system clock of the first sensor, so that the time stamps of the sensor data can be unified under the system clock of the first sensor, the time synchronization of the data for data fusion can be improved, and the accuracy of the data fusion can be improved. The stability of the system clock adopting the first sensor is better, and the accuracy of the data timestamp can be improved by using the system clock as a reference clock.
In another embodiment of the present invention, based on the embodiment shown in fig. 1, when the storage module 11 stores the second sensor data after adjusting the timestamp into the storage space, the method may include:
Verifying the second sensor data after the timestamp is adjusted according to the attribute information of the second sensor data after the timestamp is adjusted to obtain a verification result;
and when the second sensor data with the timestamp adjusted passes the verification according to the verification result, storing the second sensor data with the timestamp adjusted to a storage space.
Wherein the attribute information includes time information and/or data amount information of the second sensor data.
When the attribute information is time information of the second sensor data, and the verifying the second sensor data after the timestamp is adjusted, the method may include: judging whether the difference of the timestamps of the second sensor data after the timestamps are adjusted is consistent with the period of the second sensor data acquired in advance or not according to the second sensor data after the timestamps are adjusted, and if so, determining that the verification is passed; if not, there may be a frame loss, and the check is determined not to pass. In such an embodiment, the second sensor data after adjusting the time stamp includes at least two.
When the attribute information is data amount information of the second sensor data, and the verifying the second sensor data after the timestamp is adjusted, the method may include: judging whether the data volume of the second sensor data after the timestamp is adjusted is within a preset data volume range, and if so, determining that the verification is passed; if not, the verification is determined not to pass.
In another embodiment, it may be further verified whether the data format of the second sensor data is a preset format.
And when the second sensor data with the timestamp adjusted is determined not to pass the verification according to the verification result, deleting the second sensor data with the timestamp adjusted. And when the number of the second sensor data which are continuously deleted is determined to be larger than the preset number, alarming can be carried out.
In conclusion, in this embodiment, the stability of the second sensor when collecting data is lower, and the data of the second sensor can be verified, so that invalid data can be filtered out, and the validity of the data can be improved.
In another embodiment of the present invention, when the storage module 11 stores the first sensor data in the storage space, the verification method may also be adopted, and when the verification passes, the first sensor data is stored in the storage space, so as to improve the validity of the data.
The above embodiments will be described with reference to specific examples.
Referring to fig. 4, wherein the processor starts 4 data receiving threads for IMU, GNSS, ODO and image sensor, and one data fusion thread, respectively S1. And the data receiving thread is used for receiving the image and a perception result of the image by the perception thread. And S2, when each data receiving thread receives the data frame, adjusting the time stamp of the data frame and checking. S3, when the check is passed, the data receiving thread stores the data frame into the corresponding buffer (buffer). And S4, the data fusion thread reads the timestamp of the latest IMU data in the buffer as a first timestamp, and a second timestamp is obtained according to the first timestamp and the maximum time delay. And S5, the data fusion thread releases the data in each buffer earlier than the second timestamp, and performs data fusion on the data with the earliest timestamp in the rest data.
Fig. 5 is a schematic flowchart of a multi-sensor data fusion method according to an embodiment of the present invention. The method is applied to a processor. The method corresponds to fig. 1, and comprises the following steps:
s510: and storing the acquired first sensor data acquired by the first sensor into a storage space, and storing the acquired second sensor data acquired by the second sensor into the storage space.
The data acquisition frequency of the first sensor is different from that of the second sensor, and the stability of the first sensor in data acquisition is greater than that of the second sensor in data acquisition.
S520: a first timestamp of the first plurality of sensor data in the storage space that is latest in time is determined.
S530: and subtracting the preset maximum time delay from the first time stamp to obtain a second time stamp earlier than the first time stamp.
In the data acquisition process, the first sensor and the second sensor can acquire data within the maximum time delay.
S540: the first sensor data and the second sensor data having a timestamp earlier than the second timestamp are deleted from the storage space.
S550: and performing data fusion according to the remaining first sensor data and the second sensor data in the storage space.
In another embodiment of the present invention, based on the embodiment shown in fig. 5, the step S520 of determining the latest first timestamp in the plurality of first sensor data in the storage space may include:
when it is detected that the latest first sensor data is stored in the storage space, the timestamp of the latest first sensor data is determined as the latest first timestamp among the plurality of first sensor data in the storage space.
In another embodiment of the present invention, based on the embodiment shown in fig. 5, in step S550, performing data fusion according to the remaining first sensor data and second sensor data in the storage space may include:
and when the multi-sensor data fusion is determined to be needed, performing data fusion according to the data with the earliest time stamp in the first sensor data and the second sensor data which are left in the storage space.
In another embodiment of the present invention, based on the embodiment shown in fig. 5, the step of storing the acquired second sensor data collected by the second sensor in the storage space in step S510 may include:
adjusting the time stamp of the second sensor data according to the predetermined time difference information between the system clock of the second sensor and the system clock of the first sensor, and storing the second sensor data with the time stamp adjusted to a storage space;
Step S540 of deleting the first sensor data and the second sensor data having the timestamp earlier than the second timestamp from the storage space may include:
first sensor data with a timestamp earlier than the second timestamp is deleted from the storage space, and second sensor data with an adjusted timestamp earlier than the second timestamp is deleted.
In another embodiment of the present invention, based on the above embodiments, the step of storing the second sensor data after adjusting the timestamp in a storage space may include:
verifying the second sensor data after the timestamp is adjusted according to the attribute information of the second sensor data after the timestamp is adjusted to obtain a verification result; wherein the attribute information includes time information and/or data amount information of the second sensor data;
and when the second sensor data with the timestamp adjusted passes the verification according to the verification result, storing the second sensor data with the timestamp adjusted to a storage space.
In another embodiment of the invention, based on the embodiment shown in fig. 5, the first sensor is an inertial measurement unit IMU and the second sensor comprises at least one of the following sensors: image sensor, global navigation satellite system, odometer.
Fig. 6 is a schematic structural diagram of a multi-sensor data fusion apparatus according to an embodiment of the present invention. The apparatus is applied to a processor, and this embodiment corresponds to the method embodiment shown in fig. 5. The device includes:
the storage module 610 is configured to store the acquired first sensor data acquired by the first sensor into a storage space, and store the acquired second sensor data acquired by the second sensor into the storage space; the data acquisition frequency of the first sensor is different from that of the second sensor, and the stability of the first sensor in data acquisition is greater than that of the second sensor in data acquisition;
a determining module 620 configured to determine a latest first timestamp of the plurality of first sensor data in the storage space;
a calculating module 630 configured to subtract the preset maximum delay from the first timestamp to obtain a second timestamp earlier than the first timestamp; in the data acquisition process, the first sensor and the second sensor can acquire data within the maximum time delay;
a deletion module 640 configured to delete the first sensor data and the second sensor data having the time stamp earlier than the second time stamp from the storage space;
And a fusion module 650 configured to perform data fusion according to the remaining first sensor data and second sensor data in the storage space.
In another embodiment of the present invention, based on the embodiment shown in fig. 6, the determining module 620 is specifically configured to:
when it is detected that the latest first sensor data is stored in the storage space, the timestamp of the latest first sensor data is determined as the latest first timestamp among the plurality of first sensor data in the storage space.
In another embodiment of the present invention, based on the embodiment shown in fig. 6, the fusion module 650 is specifically configured to:
and when the multi-sensor data fusion is determined to be needed, performing data fusion according to the data with the earliest time stamp in the first sensor data and the second sensor data which are left in the storage space.
In another embodiment of the present invention, based on the embodiment shown in fig. 6, when the storage module 610 stores the acquired second sensor data collected by the second sensor into the storage space, the method includes:
adjusting the time stamp of the second sensor data according to the predetermined time difference information between the system clock of the second sensor and the system clock of the first sensor, and storing the second sensor data with the time stamp adjusted to a storage space;
The deletion module 640 is specifically configured to:
first sensor data with a timestamp earlier than the second timestamp is deleted from the storage space, and second sensor data with an adjusted timestamp earlier than the second timestamp is deleted.
In another embodiment of the present invention, based on the embodiment shown in fig. 6, when the storage module 610 stores the second sensor data after adjusting the timestamp into the storage space, the method includes:
verifying the second sensor data after the timestamp is adjusted according to the attribute information of the second sensor data after the timestamp is adjusted to obtain a verification result; wherein the attribute information includes time information and/or data amount information of the second sensor data;
and when the second sensor data with the timestamp adjusted passes the verification according to the verification result, storing the second sensor data with the timestamp adjusted to a storage space.
In another embodiment of the invention, based on the embodiment shown in fig. 6, the first sensor is an inertial measurement unit IMU, and the second sensor comprises at least one of the following sensors: image sensor, global navigation satellite system, odometer.
The device embodiment and the method embodiment correspond to the system embodiment, have the same technical effects as the system embodiment, and refer to the system embodiment for specific description. The device embodiment and the method embodiment are obtained based on the system embodiment, and specific description may refer to the system embodiment section, which is not described herein again.
Those of ordinary skill in the art will understand that: the figures are merely schematic representations of one embodiment, and the blocks or flow diagrams in the figures are not necessarily required to practice the present invention.
Those of ordinary skill in the art will understand that: modules in the devices in the embodiments may be distributed in the devices in the embodiments according to the description of the embodiments, or may be located in one or more devices different from the embodiments with corresponding changes. The modules of the above embodiments may be combined into one module, or further split into multiple sub-modules.
Finally, it should be noted that: the above examples are only intended to illustrate the technical solution of the present invention, but not to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.
Claims (10)
1. A vehicle-mounted terminal characterized by comprising: a processor, a first sensor and a second sensor; the processor includes: the device comprises a storage module, a determination module, a calculation module, a deletion module and a fusion module; the data acquisition frequency of the first sensor is different from that of the second sensor, and the stability of the first sensor in data acquisition is greater than that of the second sensor in data acquisition;
The storage module is used for storing the acquired first sensor data acquired by the first sensor into a storage space and storing the acquired second sensor data acquired by the second sensor into the storage space;
the determining module is configured to determine a latest first timestamp in the plurality of first sensor data in the storage space;
the calculation module is used for subtracting a preset maximum time delay from the first time stamp to obtain a second time stamp which is earlier than the first time stamp; in the data acquisition process, the first sensor and the second sensor can acquire data within the maximum time delay;
the deleting module is used for deleting the first sensor data and the second sensor data with the time stamps earlier than the second time stamps from the storage space;
and the fusion module is used for carrying out data fusion according to the remaining first sensor data and the second sensor data in the storage space.
2. The vehicle-mounted terminal according to claim 1, wherein the determining module is specifically configured to:
when it is detected that the latest first sensor data is stored in the storage space, the timestamp of the latest first sensor data is determined as the latest first timestamp among the plurality of first sensor data in the storage space.
3. The vehicle-mounted terminal according to claim 1, wherein the fusion module is specifically configured to:
and when the multi-sensor data fusion is determined to be needed, performing data fusion according to the data with the earliest time stamp in the first sensor data and the second sensor data which are left in the storage space.
4. The vehicle-mounted terminal according to claim 1, wherein the storage module, when storing the acquired second sensor data collected by the second sensor into the storage space, comprises:
adjusting the time stamp of the second sensor data according to the predetermined time difference information between the system clock of the second sensor and the system clock of the first sensor, and storing the second sensor data with the time stamp adjusted in the storage space;
the deletion module is specifically configured to:
deleting first sensor data with a timestamp earlier than the second timestamp and deleting second sensor data with an adjusted timestamp earlier than the second timestamp from the storage space.
5. The vehicle-mounted terminal according to claim 4, wherein the storage module, when storing the second sensor data with the timestamp adjusted in the storage space, comprises:
Verifying the second sensor data after the timestamp is adjusted according to the attribute information of the second sensor data after the timestamp is adjusted to obtain a verification result; wherein the attribute information includes time information and/or data amount information of the second sensor data;
and when the second sensor data with the timestamp adjusted passes the verification according to the verification result, storing the second sensor data with the timestamp adjusted into the storage space.
6. The vehicle terminal of claim 1, wherein the first sensor is an Inertial Measurement Unit (IMU), and the second sensor comprises at least one of: image sensor, global navigation satellite system, odometer.
7. A multi-sensor data fusion method applied to a processor comprises the following steps:
storing acquired first sensor data acquired by a first sensor into a storage space, and storing acquired second sensor data acquired by a second sensor into the storage space; the data acquisition frequency of the first sensor is different from that of the second sensor, and the stability of the first sensor in data acquisition is greater than that of the second sensor in data acquisition;
Determining a first timestamp of the first plurality of sensor data in the storage space that is latest in time;
subtracting a preset maximum time delay from the first time stamp to obtain a second time stamp earlier than the first time stamp; in the data acquisition process, the first sensor and the second sensor can acquire data within the maximum time delay;
deleting the first sensor data and the second sensor data with the time stamps earlier than the second time stamps from the storage space;
and performing data fusion according to the remaining first sensor data and the second sensor data in the storage space.
8. The method of claim 7, wherein the step of performing data fusion based on the remaining first sensor data and second sensor data in the storage space comprises:
and when the multi-sensor data fusion is determined to be needed, performing data fusion according to the data with the earliest time stamp in the first sensor data and the second sensor data which are left in the storage space.
9. The method of claim 7, wherein the step of storing the acquired second sensor data collected by the second sensor to the storage space comprises:
Adjusting the time stamp of the second sensor data according to the predetermined time difference information between the system clock of the second sensor and the system clock of the first sensor, and storing the second sensor data with the time stamp adjusted in the storage space;
the step of deleting the first sensor data and the second sensor data having the time stamps earlier than the second time stamp from the storage space includes:
deleting first sensor data with a timestamp earlier than the second timestamp and deleting second sensor data with an adjusted timestamp earlier than the second timestamp from the storage space.
10. A multi-sensor data fusion device applied to a processor comprises:
the storage module is configured to store the acquired first sensor data acquired by the first sensor into a storage space and store the acquired second sensor data acquired by the second sensor into the storage space; the data acquisition frequency of the first sensor is different from that of the second sensor, and the stability of the first sensor in data acquisition is greater than that of the second sensor in data acquisition;
A determining module configured to determine a latest first timestamp of a time in the plurality of first sensor data in the storage space;
a calculation module configured to subtract a preset maximum delay from the first timestamp to obtain a second timestamp earlier than the first timestamp; in the data acquisition process, the first sensor and the second sensor can acquire data within the maximum time delay;
a deletion module configured to delete the first sensor data and the second sensor data having a timestamp earlier than the second timestamp from the storage space;
and the fusion module is configured to perform data fusion according to the first sensor data and the second sensor data which are remained in the storage space.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910346798.8A CN111860551B (en) | 2019-04-27 | 2019-04-27 | Multi-sensor data fusion method and device and vehicle-mounted terminal |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910346798.8A CN111860551B (en) | 2019-04-27 | 2019-04-27 | Multi-sensor data fusion method and device and vehicle-mounted terminal |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111860551A true CN111860551A (en) | 2020-10-30 |
| CN111860551B CN111860551B (en) | 2024-02-06 |
Family
ID=72952460
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910346798.8A Active CN111860551B (en) | 2019-04-27 | 2019-04-27 | Multi-sensor data fusion method and device and vehicle-mounted terminal |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111860551B (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113473118A (en) * | 2021-08-23 | 2021-10-01 | 追觅创新科技(苏州)有限公司 | Data timestamp alignment method, device, equipment and storage medium |
| CN114205175A (en) * | 2022-02-19 | 2022-03-18 | 深圳市三江电气有限公司 | Self-checking modification system in multi-sensor sensing data transmission process |
| CN114413890A (en) * | 2022-01-14 | 2022-04-29 | 广州小鹏自动驾驶科技有限公司 | Vehicle track generation method, vehicle track generation device, electronic device, and storage medium |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105682222A (en) * | 2016-03-01 | 2016-06-15 | 西安电子科技大学 | Vehicle location positioning information fusion method based on vehicular ad hoc network |
| CN105711597A (en) * | 2016-02-25 | 2016-06-29 | 江苏大学 | System and method for sensing local driving environment in front |
| CN108769786A (en) * | 2018-05-25 | 2018-11-06 | 网宿科技股份有限公司 | A kind of method and apparatus of synthesis audio and video data streams |
-
2019
- 2019-04-27 CN CN201910346798.8A patent/CN111860551B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105711597A (en) * | 2016-02-25 | 2016-06-29 | 江苏大学 | System and method for sensing local driving environment in front |
| CN105682222A (en) * | 2016-03-01 | 2016-06-15 | 西安电子科技大学 | Vehicle location positioning information fusion method based on vehicular ad hoc network |
| CN108769786A (en) * | 2018-05-25 | 2018-11-06 | 网宿科技股份有限公司 | A kind of method and apparatus of synthesis audio and video data streams |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113473118A (en) * | 2021-08-23 | 2021-10-01 | 追觅创新科技(苏州)有限公司 | Data timestamp alignment method, device, equipment and storage medium |
| CN113473118B (en) * | 2021-08-23 | 2024-05-14 | 追觅创新科技(苏州)有限公司 | Data timestamp alignment method, device, equipment and storage medium |
| CN114413890A (en) * | 2022-01-14 | 2022-04-29 | 广州小鹏自动驾驶科技有限公司 | Vehicle track generation method, vehicle track generation device, electronic device, and storage medium |
| CN114205175A (en) * | 2022-02-19 | 2022-03-18 | 深圳市三江电气有限公司 | Self-checking modification system in multi-sensor sensing data transmission process |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111860551B (en) | 2024-02-06 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111860551B (en) | Multi-sensor data fusion method and device and vehicle-mounted terminal | |
| CN102707300B (en) | Method, device and system for optimizing GPS track | |
| US11994400B2 (en) | Systems and methods for assessment of a rideshare trip | |
| US20220163348A1 (en) | Positioning method and electronic device | |
| US9519306B2 (en) | Distribution device, distribution system, and distribution method | |
| US9749802B2 (en) | Method, apparatus, and terminal device for determining user activity range | |
| CN109905194A (en) | A kind of vehicle-mounted terminal system and synchronization data obtaining method, device | |
| CN112541527A (en) | Multi-sensor synchronization method and device, electronic equipment and storage medium | |
| CN111123336B (en) | Water conservancy patrol positioning and offline track optimization method | |
| CN112154614B (en) | Sensing system, sensing apparatus, control method thereof, movable platform, and storage medium | |
| US11189162B2 (en) | Information processing system, program, and information processing method | |
| CN111182226A (en) | Method, device, medium and electronic equipment for synchronous working of multiple cameras | |
| CN110737006A (en) | Track denoising processing method and device and electronic equipment | |
| JP6141148B2 (en) | Time series recording data analyzer | |
| JP2019100914A (en) | Vibration monitoring system | |
| CN113918652A (en) | Data synchronization method and device and readable storage medium | |
| CN107392983B (en) | Method and system for recording animation | |
| CN113610136A (en) | Sensor data synchronization method and device, computer equipment and storage medium | |
| US11988521B2 (en) | Navigation system, navigation display method, and navigation display program | |
| CN112817301B (en) | Fusion method, device and system of multi-sensor data | |
| CN107241624B (en) | The method and device of information processing | |
| CN111563546A (en) | Sensor data matching method, device, equipment and storage medium | |
| CN113411368A (en) | Data synchronization method and device, storage medium and electronic equipment | |
| CN113049001B (en) | Evaluation system and method for crowdsourcing map construction | |
| CN118157803A (en) | Data processing method, device, equipment, medium and vehicle-mounted data acquisition equipment and system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20220303 Address after: 100083 unit 501, block AB, Dongsheng building, No. 8, Zhongguancun East Road, Haidian District, Beijing Applicant after: BEIJING MOMENTA TECHNOLOGY Co.,Ltd. Address before: Room 28, 4 / F, block a, Dongsheng building, No. 8, Zhongguancun East Road, Haidian District, Beijing 100089 Applicant before: BEIJING CHUSUDU TECHNOLOGY Co.,Ltd. |
|
| TA01 | Transfer of patent application right | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |