CN109729277B - Multi-sensor acquisition timestamp synchronization device - Google Patents
Multi-sensor acquisition timestamp synchronization device Download PDFInfo
- Publication number
- CN109729277B CN109729277B CN201811378292.7A CN201811378292A CN109729277B CN 109729277 B CN109729277 B CN 109729277B CN 201811378292 A CN201811378292 A CN 201811378292A CN 109729277 B CN109729277 B CN 109729277B
- Authority
- CN
- China
- Prior art keywords
- data
- time
- channel
- image data
- rate
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Abstract
The invention discloses a multi-sensor acquisition timestamp synchronization device, which is applied to a sensing signal acquisition integrated end and comprises: a sensing data acquisition module configured to acquire a plurality of sensing data and time service information, wherein the plurality of sensors comprise rolling shutter image data, IMU data, position data and wheel speed data; the time stamp synchronization module is configured to perform time stamp synchronization on the plurality of sensing data according to the time service information to obtain time stamps corresponding to the plurality of sensing data respectively; the transmission module is configured to send the image data to a computing platform through a first channel, and send timestamps corresponding to the image data, other sensing data except the image data in the plurality of sensing data and corresponding timestamps to the computing platform through a second channel, wherein a transmission rate of the first channel is greater than a transmission rate of the second channel.
Description
Technical Field
The invention relates to the field of intelligent driving, in particular to a remote multi-sensing signal synchronous acquisition device.
Background
At present, acquisition equipment for performing SLAM (Simultaneous Localization and Mapping, instant positioning and map construction) and inertial navigation at home and abroad is mostly based on a Global Shutter binocular camera + IMU (inertial measurement Unit) instead of Rolling Shutter, because the exposure time of all pixels of an image acquired by the Global Shutter is the same, and the Shutter speed is fast), the exposure time of different rows of pixels of the acquired image is different; for a scene using a rolling shutter, how to perform time stamp synchronization on a plurality of collected sensing signals becomes an urgent problem to be solved in the industry.
Disclosure of Invention
The invention provides a method and a device for synchronously acquiring and receiving remote multi-sensing signals, which are used for overcoming at least one problem in the prior art.
According to a first aspect of the present invention, a remote multi-sensor signal synchronous acquisition method is provided, which is applied to a sensor signal acquisition integrated terminal, and includes the following steps:
acquiring a plurality of sensing data and time service information, wherein the sensing data comprises rolling shutter image data, IMU data, position data and wheel speed data;
carrying out time stamp synchronization on the plurality of sensing data according to the time service information to obtain time stamps corresponding to the plurality of sensing data respectively;
and sending the image data to a computing platform through a first channel, and sending timestamps corresponding to the image data, other sensing data except the image data in the sensing data and corresponding timestamps to the computing platform through a second channel, wherein the transmission rate of the first channel is greater than that of the second channel.
Optionally, for the image data, the time stamp synchronization of the multiple sensing data according to the time service information to obtain the time stamps corresponding to the multiple sensing data includes:
and obtaining a time stamp corresponding to the image data according to the acquired time for reading the first row of the photosensitive chip for the first time and the time service information.
Optionally, for the IMU data, the time stamp synchronizing the multiple sensing data according to the time service information to obtain the time stamps corresponding to the multiple sensing data includes:
and obtaining a time stamp corresponding to the IMU data according to the obtained time of the interrupt signal uploaded by the IMU and the time service information, wherein the IMU is configured with interrupt in advance, and the IMU generates the interrupt signal while generating the IMU data.
Optionally, for the wheel speed data, the time stamp synchronization of the multiple sensing data according to the time service information to obtain the time stamps corresponding to the multiple sensing data includes:
and obtaining a time stamp corresponding to the wheel speed data according to the acquired time of the UART data and the time service information.
Optionally, the time service information is acquired through a GPS module.
Optionally, the sensing signal acquisition integration terminal acquires the plurality of sensing data and synchronously adds corresponding timestamps to the plurality of sensing data respectively by using an MCU.
Optionally, the first channel is a GMSL channel, and the second channel is an I2C channel.
According to a second aspect of the present invention, there is provided a remote multi-sensor signal receiving method applied to a computing platform, including the following steps:
receiving image data sent by a sensing signal acquisition integration end through a first channel, and receiving a timestamp corresponding to the image data sent by the sensing signal acquisition integration end, other sensing data except the image data in a plurality of sensing data and a corresponding timestamp through a second channel, wherein the transmission rate of the first channel is greater than that of the second channel;
determining a timestamp corresponding to each frame of image in the image data according to a transmission mapping relation between the image data and the timestamp corresponding to the image data, wherein the transmission mapping relation is acquired in advance;
and correcting the corresponding time stamp according to the exposure central point of each frame of image in the image data.
Optionally, the method further includes the following steps:
and receiving time service information corresponding to the sensing signal acquisition integrated end so as to correct the system time of the computing platform end.
Optionally, the transmission mapping relationship is obtained by the following method:
setting the transmission frame rate of the first channel as a first rate, and sending image data from the sensing signal acquisition integrated end to the computing platform end through the first channel according to the first rate;
after the first channel sends stable image data according to the first transmission rate, updating the transmission frame rate of the first channel to a second rate, wherein the second rate is smaller than the first rate;
and respectively detecting the corresponding timestamp change relation between two frames of image data in the change process of the transmission frame rate at the sensing signal acquisition integrated end and the computing platform end to obtain the transmission mapping relation.
According to a third aspect of the present invention, there is provided a remote multi-sensor signal synchronous acquisition apparatus, applied to a sensor signal acquisition integrated terminal, including:
a sensing data acquisition module configured to acquire a plurality of sensing data and time service information, wherein the plurality of sensors comprise rolling shutter image data, IMU data, position data and wheel speed data;
the time stamp synchronization module is configured to perform time stamp synchronization on the plurality of sensing data according to the time service information to obtain time stamps corresponding to the plurality of sensing data respectively;
the transmission module is configured to send the image data to a computing platform through a first channel, and send timestamps corresponding to the image data, other sensing data except the image data in the plurality of sensing data and corresponding timestamps to the computing platform through a second channel, wherein a transmission rate of the first channel is greater than a transmission rate of the second channel.
Optionally, the timestamp synchronization module includes an image timestamp unit, and the image timestamp unit includes:
an absolute exposure time calculation subunit configured to calculate an absolute exposure time of the current frame image from a single-line data read time of the image output first line data acquired from the field synchronization signal, a current exposure time acquired from the image processing unit, and a time interval of all single-line data reads calculated by the effective number of lines and the pixel clock;
and the time stamp calculating subunit is configured to obtain a time stamp corresponding to the current frame image according to the single-line data reading time of the first line of data, the absolute exposure time and the time service information.
Optionally, the timestamp synchronization module includes:
and the IMU time stamp unit is configured to obtain a time stamp corresponding to the IMU data according to the acquired time of the interrupt signal uploaded by the IMU and the time service information, wherein the IMU is configured with interrupt in advance, and generates the interrupt signal while generating the IMU data.
Optionally, the timestamp synchronization module includes:
and the wheel speed timestamp unit is configured to obtain a timestamp corresponding to the wheel speed data according to the acquired time of the UART data and the time service information.
According to a fourth aspect of the present invention, there is provided a remote multi-sensor signal receiving apparatus applied to a computing platform, including:
the sensing data receiving module is configured to receive image data sent by a sensing signal acquisition integrated terminal through a first channel, and receive a timestamp corresponding to the image data sent by the sensing signal acquisition integrated terminal, other sensing data except the image data in a plurality of sensing data and a corresponding timestamp through a second channel, wherein the transmission rate of the first channel is greater than that of the second channel;
the timestamp corresponding module is configured to determine a timestamp corresponding to each frame of image in the image data according to a transmission mapping relation between the image data and the timestamp corresponding to the image data, wherein the transmission mapping relation is acquired in advance;
and the time stamp correction module is configured to correct the corresponding time stamp according to the exposure center point of each frame of image in the image data.
Optionally, the apparatus further comprises:
and the system time correction module is configured to receive time service information corresponding to the sensing signal acquisition integrated terminal so as to correct the system time of the computing platform terminal.
Optionally, the apparatus further includes a transmission mapping relationship obtaining module, where the transmission mapping relationship obtaining module includes:
a first rate setting unit configured to set a transmission frame rate of the first channel to a first rate, and to transmit image data from the sensing signal acquisition integration side to the computation platform side through the first channel at the first rate;
a second rate setting unit configured to update a transmission frame rate of the first channel to a second rate after the first channel transmits stabilized image data at the first transmission rate, wherein the second rate is smaller than the first rate;
and the mapping relation detection unit is configured to detect a corresponding timestamp change relation between two frames of image data in a transmission frame rate change process at the sensing signal acquisition integration end and the computing platform end respectively to obtain the transmission mapping relation.
The invention achieves the following beneficial effects:
according to the embodiment of the invention, based on the communication scheme of the sensing signal acquisition integrated end serving as the far end and the computing platform serving as the near end, the Rolling Shutter camera is used for acquiring image data, the sensing signal function module is integrated with the Rolling Shutter camera, the image data, the IMU data, the wheel speed data and the position data are acquired, and meanwhile, the time service information is acquired to the equipment, so that the accurate data acquisition of the image data, the IMU data, the wheel speed data and the position data in the same time dimension is realized, the precision of the time stamp of each sensing data is improved, and a better technical effect is obtained.
The invention comprises the following steps:
1. the method adopts a Rolling Shutter image sensor to acquire image data, but because the exposure time of the Rolling Shutter is not fixed, the dynamic calculation of the time stamp corresponding to the acquired image data, including the synchronization of the time stamp, is one of the invention points of the invention;
2. the method and the device have the advantages that a plurality of sensing data such as image data, IMU data, wheel speed data and position data are collected, time service is carried out on the device through the obtained time service information, and timestamp synchronization of the image data, the IMU data, the wheel speed data and the position data in the same time dimension is achieved.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.
FIG. 1 is a flow chart of a remote multi-sensor signal synchronous acquisition method according to an embodiment of the present invention;
FIG. 2 is a flow chart of a remote multi-sensor signal receiving method according to an embodiment of the present invention;
FIG. 3 is a hardware schematic of one embodiment of the present invention;
FIG. 4 is a flowchart of a method for processing sensor data acquisition and reception according to an embodiment of the present invention;
FIG. 5 is a schematic illustration of an exposure using a rolling shutter image sensor OV2718 in one embodiment of the present invention;
FIG. 6 is a block diagram of a remote multi-sensor signal synchronous acquisition device according to an embodiment of the present invention;
fig. 7 is a block diagram of a multi-sensor signal receiving apparatus according to an embodiment of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be obtained by a person skilled in the art without inventive effort based on the embodiments of the present invention, are within the scope of the present invention.
Fig. 1 is a flowchart of a remote multi-sensor signal synchronous acquisition method according to an embodiment of the present invention, and as shown in fig. 1, the method is applied to a sensor signal acquisition integrated terminal, and includes the following steps:
and S110, acquiring a plurality of sensing data and time service information, wherein the plurality of sensors comprise rolling shutter image data, IMU data, position data and wheel speed data.
In a specific implementation manner, the time service information is acquired through a GPS module.
In a specific implementation manner, the sensing signal acquisition integration terminal acquires the plurality of sensing data and synchronously adds corresponding timestamps to the plurality of sensing data respectively by using an MCU.
And S120, carrying out time stamp synchronization on the plurality of sensing data according to the time service information to obtain time stamps corresponding to the plurality of sensing data respectively.
In one specific implementation manner, for the image data, the time stamp synchronizing the plurality of sensing data according to the time service information to obtain the time stamps corresponding to the plurality of sensing data includes:
and obtaining a time stamp corresponding to the image data according to the acquired time for reading the first row of the photosensitive chip for the first time and the time service information.
Correspondingly, at the computing platform, the correcting the timestamp corresponding to the image data includes: calculating the absolute exposure time of the current frame image according to the single-line data reading time of the first line data output by the image acquired from the field synchronizing signal, the current exposure time acquired from the image processing unit and the time interval of reading all the single-line data calculated by the effective line number and the pixel clock;
and correcting the timestamp corresponding to the current frame image according to the single-line data reading time and the absolute exposure time of the first line of data.
In a specific implementation manner, for the IMU data, the time stamp synchronizing the multiple sensing data according to the time service information, and obtaining the time stamps corresponding to the multiple sensing data includes:
and obtaining a time stamp corresponding to the IMU data according to the obtained time of the interrupt signal uploaded by the IMU and the time service information, wherein the IMU is configured with interrupt in advance, and the IMU generates the interrupt signal while generating the IMU data.
In one specific implementation manner, for the wheel speed data, the time stamp synchronization of the plurality of sensing data according to the time service information to obtain the time stamps corresponding to the plurality of sensing data includes:
and obtaining a time stamp corresponding to the wheel speed data according to the acquired time of the UART data and the time service information.
S130, sending the image data to a computing platform through a first channel, and sending the image data
And sending the X and the corresponding time stamp to the computing platform through a second channel, wherein the transmission rate of the first channel is greater than that of the second channel.
In a specific implementation manner, the first channel is a GMSL channel, and the second channel is an I2C channel.
FIG. 2 is a flow chart of a remote multi-sensor signal receiving method according to an embodiment of the present invention; as shown in the figure, the method is applied to a computing platform end and comprises the following steps:
s210, receiving image data sent by a sensing signal acquisition integration end through a first channel, and receiving a timestamp corresponding to the image data sent by the sensing signal acquisition integration end, other sensing data except the image data in a plurality of sensing data and a corresponding timestamp through a second channel, wherein the transmission rate of the first channel is greater than that of the second channel.
S220, determining the time stamp corresponding to each frame of image in the image data according to the transmission mapping relation between the image data and the time stamp corresponding to the image data, wherein the transmission mapping relation is acquired in advance.
And S230, correcting the corresponding time stamp according to the exposure center point of each frame of image in the image data.
In a specific implementation manner, in order to ensure that the computing platform end and the sensing signal acquisition integrated end are in the same time dimension, the method further includes the following steps:
and receiving time service information corresponding to the sensing signal acquisition integrated end so as to correct the system time of the computing platform end.
In a specific implementation manner, the transmission mapping relationship is obtained by:
setting the transmission frame rate of the first channel as a first rate, and sending image data from the sensing signal acquisition integrated end to the computing platform end through the first channel according to the first rate; setting the transmission rate of the second channel as a third rate, and sending a timestamp corresponding to the image data from the sensing signal acquisition integration end to the computing platform end through the second channel according to the third rate;
after the image data sent by the first channel according to the first transmission rate is stable, updating the transmission frame rate of the first channel to a second rate, and updating the transmission rate of the second channel to a fourth rate, wherein the second rate is different from the first rate, the fourth rate is different from the third rate, preferably, the second rate is less than the first rate, and the fourth rate is less than the third rate;
and respectively detecting the corresponding timestamp change relation between two frames of image data in the change process of the transmission frame rate at the sensing signal acquisition integrated end and the computing platform end to obtain the transmission mapping relation.
In the above embodiment, in the process of updating the transmission rate of the first channel from the first rate to the second rate, the first frame of image data received in the first channel corresponds to the first image timestamp, and by analogy, the mapping relationship between the image data transmitted in the first channel and the image timestamp transmitted in the second channel is obtained according to the timestamp change relationship between the two frames of images, so that the image data and the image timestamp are conveniently corresponding at the computing platform end according to the mapping relationship.
In a preferred embodiment of the present invention, the sensing signal acquisition integration module employs an MCU (an ARM Comtex-M series single chip microcomputer), the computing platform employs an NVIIDA PX2 (an artificial intelligence computing platform manufactured by NVIDIA corporation), and the sensing signal acquisition integration module and the computing platform communicate with each other by using a GMSL (gigabit multimedia serial transmission technology, which is suitable for remote multi-channel transmission of media stream data, where in the transmission protocol, there are high-channel image data transmission and low-channel I2C protocol transmission).
FIG. 3 is a hardware schematic of one embodiment of the present invention; as shown in fig. 3, the NVIDIA PX2 terminal and the GMSL module integrated terminal are included. The main function of the NVIDIA PX2 end is to drive each hardware peripheral and manage the hardware peripheral; the method comprises the steps that communication is carried out between an NVIDIA PX2 end and a GMSL module end through a user-defined transmission protocol, and image data, IMU data, CAN wheel speed data and a timestamp are obtained; the GPS data is received through a USB to UART (Universal Asynchronous Receiver/Transmitter) device. The GMSL module end is responsible for GPS time service, processing the work in the aspect of timestamp synchronization, receiving IMU data, CAN wheel speed data and all external interrupts.
In this embodiment, based on the NVIDIA PX2 platform, a GMSL communication scheme is utilized, a Rolling Shutter camera is used, a high-performance and low-cost MCU is added to be integrated with the Rolling Shutter camera, image data, 400Hz IMU data, 100Hz wheel speed data, and 10Hz GPS data are collected, and at the same time, a PPS (pulse per second) of a GPS is used to provide time for the device, so that accurate collection of the image data, the IMU data, the wheel speed data, and the GPS data in the same time dimension is achieved, a timestamp of each sensing data reaches 100 us-level accuracy, and a better technical effect is achieved.
FIG. 4 is a flowchart of a method for processing sensor data acquisition and reception according to an embodiment of the present invention; the principle of timestamp calibration according to the present invention is explained below with reference to fig. 4:
image timestamp calibration principle:
for the Rolling Shutter camera, the image exposure timestamp is determined from three time stamps:
(1) acquiring a time when first line data Readout (one-line data read) of an image is output from a Vsync (field sync signal) signal output from an ISP (image processing unit);
(2) acquiring the current exposure time from the ISP through I2C;
(3) calculating a total Readout time interval through the effective line number and PCLK (pixel clock);
the absolute exposure time of the current frame can be calculated according to the data.
FIG. 5 is an exposure diagram of a rolling shutter image sensor OV2718 according to an embodiment of the present invention, in which the horizontal axis t is a time axis, the vertical axis ra is a row address axis, A-B is a single-row exposure time t1, B is a time t0 of the first Readout (reading the first row of photo-sensing chips), and B-C is a time t2 of all Readout (reading all photo-sensing chips).
T2 represents the time for reading all lines of data of the image, such as 1280 × 720 resolution, i.e., the time for reading all 720 lines of data. T3 in fig. 5 is the blanking time (Tv-blanking) of the Vsync signal, which has a short high level at the beginning of a frame of image, and is the Vsync signal, which represents the beginning of the image data transmission, and then remains low during the transmission, i.e. the blanking time. This time is not used in the calculation of the time stamp.
The time stamp of the image data sent from the sensing signal acquisition integration end (far end) (i.e. time t0 in fig. 5) is corrected at the computing platform end (near end), and the calculation formula for the correction of the exposure center point Tn is as follows:
Tn＝B+(CB-BA)/2＝t0+(t2-t1)/2。
description of exposure center and exposure time:
the exposure center point is the position of the frame image from the beginning to the end of the exposure at the time when the exposure center is located, that is, the exposure center point is the center time point of the frame image from the beginning to the end of the exposure. The exposure time is a time with a constant size for the Rolling Shutter, and the exposure position of each line is sequentially increased, and the exposure time is a concept with a constant size for the Global Shutter, and the exposure position of each line is the same.
In the above embodiment, for the time stamp of the image data acquired by the rolling shutter image sensor, the time stamp corresponding to the acquired image data is dynamically corrected at the computing platform according to the deviation of the line exposure time center, so as to improve the accuracy of the exposure center point of the acquired image
Since the image and the image timestamp are from two channels (GMSL (gigabit multimedia serial link) high speed channel and I2C low speed channel) to NVIDIA PX2 side, mapping of data of the two channels is required. Firstly, setting the frame rate to be 30fps, modifying the frame rate to be 25Hz after stabilization, and respectively detecting the frame rate change at the MCU end and the PX2 end, wherein the detection principle is that the time stamp between two frames is changed from 33ms to 40 ms. For OV2718, the frame rate will change gradually, 30fps- >27fps- >25fps, and the specific implementation can be confirmed by oscilloscope measurement.
IMU timestamp calibration principle:
for an IMU, configuring an IMU updating frequency as a demand frequency; after the IMU is configured to be interrupted, interruption can be generated when IMU data are generated, the MCU captures the interruption of the IMU, then the timestamp is calibrated, and then the data are read and packaged and reported.
CAN wheel speed calibration principle:
CAN wheel speed is through CAN commentaries on classics UART equipment, output 100hz data, and MCU marks timestamp information through calibration after DMA receives UART data, then encapsulates, reports wheel speed data.
GPS timestamp calibration principle:
the GPS module may use UM4B0-BOX2, which includes self differential signal calibration and data output, and outputs 3 UART interfaces, UARTA and UARTB are serial data output, and UARTC is PPS signal output, in this embodiment, PPS is output to the MCU through UARTB output data and UARTC, time service and time calibration are performed on the MCU terminal, data required by the algorithm is directly imported to the PX2 terminal through UARTA, and data reading and packaging is performed by PX 2.
The MCU analyzes the year, month, day, hour, minute and second information contained in the $ GPRMC data in the UARTB data; converting the information into the seconds from 1970-01-0100: 00:00 to the present; and then updating the second according to the PPS interruption, simultaneously resetting the MCU timer, and timing ns within the second, so that the time generated by the MCU end is ensured to be absolute time corresponding to the time output to PX2 by UARTA.
In a specific implementation, in order to implement a requirement of high real-time performance on the MCU side, the real-time performance required for the MCU to process all transactions is high, and there is no delay, which determines that an event cannot be processed in the main function, all that depends on an interrupt, where table 1 shows an interrupt list of an embodiment, where event priorities are defined to ensure timely data response.
TABLE 1 MCU interrupt priority List
I2C is used as a slave device to respond NVIDIA PX2 communication, and the priority must be set highest to ensure stable communication. The MCU side interrupts this much, ensuring shared data FIFO protection by: first, for the IMU, the data update frequency is the highest, and merge method is invoked to update data into the FIFO. In order to prevent the image timestamp data and the CAN data from frequently calling the merge method, the image timestamp data and the CAN data are marked, the image timestamp data and the CAN data are added into the merge in a buffer (cache) of the IMU when the IMU is updated, and the merge times are reduced. The I2C and IMU share FIFO is to keep the data through the double FIFO mechanism, if I2C takes the merge method in extreme cases, when IMU updates the data, it opens a Cache (fast buffer storage area) to keep the data, then updates the data into the double FIFO, and guarantees the FIFO protection of the shared data.
FIG. 6 is a block diagram of a remote multi-sensor signal synchronous acquisition device in accordance with an embodiment of the present invention; as shown, the apparatus 600 is applied to a sensing signal acquisition integrated terminal, and includes:
a sensing data acquisition module 610 configured to acquire a plurality of sensing data and time service information, wherein the plurality of sensors include rolling shutter image data, IMU data, position data, and wheel speed data;
a timestamp synchronization module 620, configured to perform timestamp synchronization on the multiple sensing data according to the time service information, so as to obtain timestamps corresponding to the multiple sensing data respectively;
a transmission module 630, configured to send the image data to a computing platform through a first channel, and send the timestamp corresponding to the image data, the other sensing data except the image data in the plurality of sensing data, and the corresponding timestamp to the computing platform through a second channel, where a transmission rate of the first channel is greater than a transmission rate of the second channel.
In one specific implementation, the timestamp synchronization module includes an image timestamp unit, and the image timestamp unit includes:
an absolute exposure time calculation subunit configured to calculate an absolute exposure time of the current frame image from a single-line data read time of the image output first line data acquired from the field synchronization signal, a current exposure time acquired from the image processing unit, and a time interval of all single-line data reads calculated by the effective number of lines and the pixel clock;
and the time stamp calculating subunit is configured to obtain a time stamp corresponding to the current frame image according to the single-line data reading time of the first line of data, the absolute exposure time and the time service information.
In a specific implementation manner, the timestamp synchronization module includes:
and the IMU time stamp unit is configured to obtain a time stamp corresponding to the IMU data according to the acquired time of the interrupt signal uploaded by the IMU and the time service information, wherein the IMU is configured with interrupt in advance, and generates the interrupt signal while generating the IMU data.
In a specific implementation manner, the timestamp synchronization module includes:
and the wheel speed timestamp unit is configured to obtain a timestamp corresponding to the wheel speed data according to the acquired time of the UART data and the time service information.
Fig. 7 is a block diagram of a multi-sensor signal receiving apparatus according to an embodiment of the present invention. As shown, the apparatus 700 is applied to a computing platform side, and includes:
a sensing data receiving module 710 configured to receive, through a first channel, image data sent by a sensing signal acquisition integration end, and receive, through a second channel, a timestamp corresponding to the image data sent by the sensing signal acquisition integration end, other sensing data except the image data in a plurality of sensing data, and a corresponding timestamp, where a transmission rate of the first channel is greater than a transmission rate of the second channel;
a timestamp corresponding module 720, configured to determine, according to a transmission mapping relationship between the image data obtained in advance and a timestamp corresponding to the image data, a timestamp corresponding to each frame of image in the image data;
and the timestamp correction module 730 is configured to correct the timestamp corresponding to each frame of image in the image data according to the exposure center point of each frame of image in the image data.
In a specific implementation manner, the apparatus further includes:
and the system time correction module is configured to receive time service information corresponding to the sensing signal acquisition integrated terminal so as to correct the system time of the computing platform terminal.
Optionally, the apparatus further includes a transmission mapping relationship obtaining module, where the transmission mapping relationship obtaining module includes:
a first rate setting unit configured to set a transmission frame rate of the first channel to a first rate, and to transmit image data from the sensing signal acquisition integration side to the computation platform side through the first channel at the first rate;
a second rate setting unit configured to update a transmission frame rate of the first channel to a second rate after the first channel transmits stabilized image data at the first transmission rate, wherein the second rate is smaller than the first rate;
and the mapping relation detection unit is configured to detect a corresponding timestamp change relation between two frames of image data in a transmission frame rate change process at the sensing signal acquisition integration end and the computing platform end respectively to obtain the transmission mapping relation.
The foregoing description is intended to be illustrative rather than limiting, and it will be appreciated by those skilled in the art that many modifications, variations or equivalents may be made without departing from the spirit and scope of the invention as defined in the appended claims.
Claims (10)
1. The utility model provides a long-range many sensing signal synchronous acquisition device, is applied to the sensing signal and obtains integrated end, its characterized in that includes:
a sensing data acquisition module configured to acquire a plurality of sensing data and time service information, wherein the plurality of sensors comprise rolling shutter image data, IMU data, position data and wheel speed data;
the time stamp synchronization module is configured to perform time stamp synchronization on the plurality of sensing data according to the time service information to obtain time stamps corresponding to the plurality of sensing data respectively;
the transmission module is configured to transmit the image data to a computing platform through a first channel, and transmit a timestamp corresponding to the image data, other sensing data except the image data in the plurality of sensing data and a corresponding timestamp to the computing platform through a second channel, wherein a transmission rate of the first channel is greater than a transmission rate of the second channel;
for image data, the computing platform takes an exposure central point of each frame of image in the image data as a corresponding timestamp, wherein the exposure central point is a central time point from exposure start to exposure end of one frame of image; for each frame of image, the calculation formula of the exposure center point is as follows: tn ═ t0+ (t2-t 1)/2;
where Tn is an exposure center point of the frame image, t0 is a time stamp of the frame image data, t1 is a single line exposure time of the frame image, and t2 is a time for reading all line data of the frame image.
2. The remote multi-sensor signal synchronous acquisition device according to claim 1, wherein the timestamp synchronization module comprises:
and the image time stamp unit is configured to obtain a time stamp corresponding to the image data according to the acquired time for reading the first row of the photosensitive chip for the first time and the time service information.
3. The remote multi-sensor signal synchronous acquisition device according to claim 1, wherein the timestamp synchronization module comprises:
and the IMU time stamp unit is configured to obtain a time stamp corresponding to the IMU data according to the acquired time of the interrupt signal uploaded by the IMU and the time service information, wherein the IMU is configured with interrupt in advance, and generates the interrupt signal while generating the IMU data.
4. The remote multi-sensor signal synchronous acquisition device according to claim 1, wherein the timestamp synchronization module comprises:
and the wheel speed timestamp unit is configured to obtain a timestamp corresponding to the wheel speed data according to the acquired time of the UART data and the time service information.
5. The device for synchronously acquiring remote multi-sensor signals according to claim 1, wherein the time service information and the position data are acquired through a GPS module.
6. The remote multi-sensor signal synchronous acquisition device according to claim 1,
and the sensing signal acquisition integrated terminal acquires the plurality of sensing data by adopting the MCU and synchronously adds corresponding timestamps for the plurality of sensing data respectively.
7. The remote multi-sensor signal synchronous acquisition device according to claim 1, wherein the first channel is a GMSL channel, and the second channel is an I2C channel.
8. The device for synchronously acquiring remote multi-sensor signals according to claim 1, further comprising:
and the mapping relation acquisition module is configured to acquire a transmission mapping relation between the image data and the time stamp corresponding to the image data.
9. The device for synchronously acquiring the remote multi-sensor signals according to claim 8, wherein the mapping relation obtaining module comprises:
a first rate setting unit configured to set a transmission frame rate of the first channel to a first rate, and to transmit image data from the sensing signal acquisition integration side to the computation platform side through the first channel at the first rate;
a second rate setting unit configured to update a transmission frame rate of the first channel to a second rate after the first channel stabilizes to transmit image data at the first rate, wherein the second rate is different from the first rate;
and the mapping relation detection unit is configured to detect a corresponding timestamp change relation between two frames of image data in a transmission frame rate change process at the sensing signal acquisition integration end and the computing platform end respectively to obtain the transmission mapping relation.
10. The remote multi-sensor signal synchronous acquisition device of claim 9, wherein the second rate is less than the first rate.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811378292.7A CN109729277B (en) | 2018-11-19 | 2018-11-19 | Multi-sensor acquisition timestamp synchronization device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811378292.7A CN109729277B (en) | 2018-11-19 | 2018-11-19 | Multi-sensor acquisition timestamp synchronization device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109729277A CN109729277A (en) | 2019-05-07 |
| CN109729277B true CN109729277B (en) | 2021-10-01 |
Family
ID=66295081
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811378292.7A Active CN109729277B (en) | 2018-11-19 | 2018-11-19 | Multi-sensor acquisition timestamp synchronization device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109729277B (en) |
Families Citing this family (15)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110275496A (en) * | 2019-05-14 | 2019-09-24 | 清华大学 | A kind of method and apparatus of more time series timestamp alignment |
| CN110198415B (en) * | 2019-05-26 | 2021-08-24 | 初速度（苏州）科技有限公司 | Method and device for determining image timestamp |
| CN110234029B (en) * | 2019-07-31 | 2021-12-17 | 上海商汤临港智能科技有限公司 | Playing processing method, device, equipment and storage medium of multi-sensor data |
| WO2021035645A1 (en) * | 2019-08-29 | 2021-03-04 | 上海飞来信息科技有限公司 | Sensing system, sensing device and control method therefor, movable platform, and storage medium |
| CN113765611B (en) * | 2020-06-03 | 2023-04-14 | 杭州海康威视数字技术股份有限公司 | Time stamp determination method and related equipment |
| CN112218115B (en) * | 2020-09-25 | 2022-07-29 | 深圳市捷视飞通科技股份有限公司 | Control method and device for streaming media audio and video synchronization and computer equipment |
| CN112577517A (en) * | 2020-11-13 | 2021-03-30 | 上汽大众汽车有限公司 | Multi-element positioning sensor combined calibration method and system |
| CN112532262B (en) * | 2020-11-18 | 2023-05-26 | 深圳市哈博森科技有限公司 | SDR-based unmanned aerial vehicle communication, GPS and ads-b integration method and device |
| CN112945228B (en) * | 2021-02-04 | 2023-04-07 | 北京眸星科技有限公司 | Multi-sensor time synchronization method and synchronization device |
| CN113885020A (en) * | 2021-09-15 | 2022-01-04 | 万赛智能科技(苏州)有限公司 | Automatic driving data acquisition and fusion system and method |
| CN113820965A (en) * | 2021-09-26 | 2021-12-21 | 上汽通用五菱汽车股份有限公司 | IMU signal simulation method, system, device and computer readable storage medium |
| CN114268706B (en) * | 2021-12-13 | 2024-02-02 | 北京元客视界科技有限公司 | Time timing method and device for camera |
| CN113945229A (en) * | 2021-12-20 | 2022-01-18 | 北京北斗华大科技有限公司 | Embedded INS measuring time calibration system and method based on PPS |
| CN114859071A (en) * | 2022-06-14 | 2022-08-05 | 北京斯年智驾科技有限公司 | Wheel speed acquisition module, device and method |
| CN117714620A (en) * | 2024-02-05 | 2024-03-15 | 中国科学院自动化研究所 | Method and system for realizing synchronous acquisition of multiple types of sensors |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101949715A (en) * | 2010-08-10 | 2011-01-19 | 武汉武大卓越科技有限责任公司 | Multi-sensor integrated synchronous control method and system for high-precision time-space data acquisition |
| CN103716544A (en) * | 2013-12-27 | 2014-04-09 | 豪威科技(上海)有限公司 | Rapid and continuous focusing method and system for high-resolution module, |
| CN103963593A (en) * | 2013-01-08 | 2014-08-06 | 福特全球技术公司 | Adaptive active suspension system with road preview |
| CN104112363A (en) * | 2014-07-04 | 2014-10-22 | 西安交通大学 | Multi-sensing-data space-time synchronization method and road multi-sensing-data vehicle-mounted acquisition system |
| CN105940390A (en) * | 2013-12-30 | 2016-09-14 | 谷歌技术控股有限责任公司 | Methods and Systems for Synchronizing Data Received from Multiple Sensors of a Device |
| CN106162038A (en) * | 2015-03-25 | 2016-11-23 | 中兴通讯股份有限公司 | A kind of audio frequency sending method and device |
| CN107204820A (en) * | 2012-02-22 | 2017-09-26 | 日本电信电话株式会社 | Multichannel conveyer and multichannel transfer approach |
| CN107885772A (en) * | 2017-09-29 | 2018-04-06 | 北京空间机电研究所 | A kind of dynamically changeable multiple data format satellite assistance data processing system |
Family Cites Families (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| TW445432B (en) * | 1999-09-15 | 2001-07-11 | Avision Inc | Optical scanner to adjust the image sensing voltage by controlling the exposure time of the sensor cell |
| CN102035794A (en) * | 2009-09-29 | 2011-04-27 | 中兴通讯股份有限公司 | Method and system for transmitting and receiving synchronizing sequence of multimedia broadcast multicast service |
| US9462210B2 (en) * | 2011-11-04 | 2016-10-04 | Remote TelePointer, LLC | Method and system for user interface for interactive devices using a mobile device |
| DE102015205937A1 (en) * | 2015-04-01 | 2016-10-06 | Siemens Healthcare Gmbh | Acquisition of physiological signals and image data |
| CN104935584B (en) * | 2015-06-01 | 2018-06-15 | 广州广哈通信股份有限公司 | A kind of time service method and apparatus and system based on circuit transmission mode |
| US9654672B1 (en) * | 2015-11-06 | 2017-05-16 | Intel Corporation | Synchronized capture of image and non-image sensor data |
| CN109983414A (en) * | 2016-12-07 | 2019-07-05 | 深圳市大疆创新科技有限公司 | System and method for supporting to synchronize in moveable platform |
| CN106850115B (en) * | 2017-03-07 | 2019-02-05 | 中国地质大学（武汉） | A kind of Multi-channel data acquisition synchronization system and method |
| CN107231533B (en) * | 2017-06-12 | 2019-12-13 | 深圳市瑞立视多媒体科技有限公司 | synchronous exposure method and device and terminal equipment |
| US10812711B2 (en) * | 2018-05-18 | 2020-10-20 | Samsung Electronics Co., Ltd. | Semantic mapping for low-power augmented reality using dynamic vision sensor |
| CN108768576B (en) * | 2018-06-29 | 2020-11-17 | 广东星舆科技有限公司 | Method and device for realizing high-precision time synchronization data of sensor based on single chip microcomputer |
-
2018
- 2018-11-19 CN CN201811378292.7A patent/CN109729277B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101949715A (en) * | 2010-08-10 | 2011-01-19 | 武汉武大卓越科技有限责任公司 | Multi-sensor integrated synchronous control method and system for high-precision time-space data acquisition |
| CN107204820A (en) * | 2012-02-22 | 2017-09-26 | 日本电信电话株式会社 | Multichannel conveyer and multichannel transfer approach |
| CN103963593A (en) * | 2013-01-08 | 2014-08-06 | 福特全球技术公司 | Adaptive active suspension system with road preview |
| CN103716544A (en) * | 2013-12-27 | 2014-04-09 | 豪威科技(上海)有限公司 | Rapid and continuous focusing method and system for high-resolution module, |
| CN105940390A (en) * | 2013-12-30 | 2016-09-14 | 谷歌技术控股有限责任公司 | Methods and Systems for Synchronizing Data Received from Multiple Sensors of a Device |
| CN104112363A (en) * | 2014-07-04 | 2014-10-22 | 西安交通大学 | Multi-sensing-data space-time synchronization method and road multi-sensing-data vehicle-mounted acquisition system |
| CN106162038A (en) * | 2015-03-25 | 2016-11-23 | 中兴通讯股份有限公司 | A kind of audio frequency sending method and device |
| CN107885772A (en) * | 2017-09-29 | 2018-04-06 | 北京空间机电研究所 | A kind of dynamically changeable multiple data format satellite assistance data processing system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN109729277A (en) | 2019-05-07 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109729277B (en) | Multi-sensor acquisition timestamp synchronization device | |
| CN109729278B (en) | Remote multi-sensor synchronous receiving device capable of setting transmission rate of each channel | |
| CN110139041B (en) | Remote multi-sensing signal synchronous acquisition method | |
| KR101722068B1 (en) | Methods and systems for synchronizing data received from multiple sensors of a device | |
| CN111200698B (en) | Remote multi-sensor multi-channel receiving method | |
| CN110139066B (en) | Sensor data transmission system, method and device | |
| WO2021031604A1 (en) | Method and device for hardware time synchronization between multi-channel imus and cameras of bionic eye | |
| CN109104259B (en) | Multi-sensor time synchronization system and method | |
| CN109922260B (en) | Data synchronization method and synchronization device for image sensor and inertial sensor | |
| KR101753316B1 (en) | Methods and systems for providing sensor data and image data to an application processor in a digital image format | |
| US9485366B2 (en) | Methods and systems for communicating sensor data on a mobile device | |
| CN102306160B (en) | Assistant data processing method for improving image positioning precision of stereoscopic plotting camera | |
| US9185269B2 (en) | Imaging device, information processing device, information processing method, and method for synchronizing frame data output | |
| CN112822480B (en) | VR system and positioning tracking method thereof | |
| JP6383439B2 (en) | Method and system for calibrating a sensor using a recognized object | |
| US10051158B2 (en) | Control apparatus for image pickup apparatus | |
| CN114553351A (en) | Time synchronization method, control device, equipment and storage medium | |
| CN111405139B (en) | Time synchronization method, system, visual mileage system and storage medium | |
| CN112606000A (en) | Method for automatically calibrating robot sensor parameters, calibration room, equipment and computer medium | |
| CN111343379A (en) | FPGA-based high-speed data acquisition device and method | |
| CN112398559B (en) | Clock synchronization method, device, system and storage medium | |
| CN115729743A (en) | Sensing system test data recharging device and method and readable storage medium | |
| CN212012915U (en) | Binocular image and IMU data high-speed acquisition device | |
| CN113624254A (en) | Time synchronization calibration device and method for image sensor and gyroscope | |
| CN111355913A (en) | POS data and video data synchronization method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |