CN111813716A - Multi-sensor data synchronization, electronic device, and storage medium - Google Patents
Multi-sensor data synchronization, electronic device, and storage medium Download PDFInfo
- Publication number
- CN111813716A CN111813716A CN201911095346.3A CN201911095346A CN111813716A CN 111813716 A CN111813716 A CN 111813716A CN 201911095346 A CN201911095346 A CN 201911095346A CN 111813716 A CN111813716 A CN 111813716A
- Authority
- CN
- China
- Prior art keywords
- sensor
- acquisition
- signal
- data
- sending
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F13/00—Interconnection of, or transfer of information or other signals between, memories, input/output devices or central processing units
- G06F13/14—Handling requests for interconnection or transfer
- G06F13/16—Handling requests for interconnection or transfer for access to memory bus
- G06F13/1605—Handling requests for interconnection or transfer for access to memory bus based on arbitration
- G06F13/161—Handling requests for interconnection or transfer for access to memory bus based on arbitration with latency improvement
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F13/00—Interconnection of, or transfer of information or other signals between, memories, input/output devices or central processing units
- G06F13/14—Handling requests for interconnection or transfer
- G06F13/16—Handling requests for interconnection or transfer for access to memory bus
- G06F13/1668—Details of memory controller
- G06F13/1689—Synchronisation and timing concerns
Abstract
The invention discloses a multi-sensor data synchronization method, electronic equipment and a storage medium, wherein the method comprises the following steps: in response to the interrupt signal, sending a first acquisition signal to the first sensor, the first acquisition signal indicating that the first sensor outputs first acquisition data; after sending a first acquisition signal to the first sensor, sending a second acquisition signal to the second sensor through a first time delay; and after sending a first acquisition signal to the first sensor, sending a receiving signal to the cooperative processor through a second time delay, wherein the receiving signal indicates the cooperative processor to receive and fuse the first acquisition data sent by the first sensor and the second acquisition data sent by the second sensor. According to the invention, the problem of asynchronous sampling time among various sensors is solved by adopting a delayed triggering mode for the sensors, and synchronous sampling among multiple sensors is realized.
Description
Technical Field
The invention relates to the technical field of automobile correlation, in particular to multi-sensor data synchronization, electronic equipment and a storage medium.
Background
In view of the actual requirement of high-precision map acquisition, it is necessary to implement fusion of Inertial Measurement Unit (IMU)/Global Positioning System (GPS)/CAMERA shooting (CAMERA) data. One of the most important items is time alignment. The current requirement is that the relative error of time is not more than 1ms when 3 persons perform synchronous data acquisition. To achieve this precision, it is impossible to simply acquire the data of the 3 persons by software and then synchronize according to the time stamp. Even if the alignment accuracy can be achieved at 1ms, the output frequency is very low, even 0. To actually achieve this 1ms accuracy error, software and hardware cooperation is required to achieve this goal.
The prior art uses IMU/GPS integrated devices such as u-blob M8U. The advantage of using the IMU/GPS integrated device is that the time error between the IMU/GPS can be considered as 0, and the output precision is higher, which can be generally 20HZ to 100 HZ. Taking M8U as an example, the output frequency can reach 30 HZ. If the CMOS is controlled to output pulses of this frequency as a VSync signal, image data synchronized at 30fps can be theoretically obtained. However, since CMOS image output has an inherent delay, it is theoretically impossible to make a one-to-one correspondence with the output of M8U.
Second is the alignment of the samples. Alignment means that the inherent delay of Camera sampling and the indeterminate delay due to software processing need to be eliminated. If Camera sampling is notified by IMU/GPS and the CPU is notified to generate an interrupt to receive image data, from a software perspective, it is known that only one interrupt can receive one frame of data, but there is no other information associated with it (e.g., Camera has a hardware counter that can be used to correlate IMU/GPS time), and thus there is no way to correlate image data with IMU/GPS data.
Disclosure of Invention
In view of the above, it is necessary to provide a multi-sensor data synchronization method, an electronic device, and a storage medium for solving the technical problem in the prior art that accurate synchronization between multi-sensor data cannot be achieved.
The invention provides a multi-sensor data synchronization method, which comprises the following steps:
responding to an interrupt signal, and sending a first acquisition signal to a first sensor, wherein the first acquisition signal indicates the first sensor to output first acquisition data, and the first sensor starts to acquire the first acquisition data after a first time delay;
after a first acquisition signal is sent to a first sensor, a second acquisition signal is sent to a second sensor through a first time delay, the second acquisition signal indicates the second sensor to output second acquisition data, and the second sensor acquires the second acquisition data in real time;
and after sending a first acquisition signal to the first sensor, sending a receiving signal to the coprocessor through a second time delay, wherein the receiving signal indicates the coprocessor to receive and fuse first acquisition data sent by the first sensor and second acquisition data sent by the second sensor.
Further, the first sensor is a camera sensor, and the first time delay is a total time length from when the camera sensor receives the first acquisition signal to when the camera sensor starts to acquire the effective pixels.
Further, the time interval of the image frames set by the image pickup sensor is greater than or equal to the data calculation output time of the second sensor.
Further, the second time delay is the time of acquiring one frame of image of the camera sensor.
Further, the sending a first acquisition signal to the first sensor in response to the interrupt signal specifically includes:
and responding to the interrupt signal, and sending a first acquisition signal to the first sensor at a preset frequency.
The invention provides a multi-sensor data synchronization electronic device, comprising:
at least one processor; and the number of the first and second groups,
a memory communicatively coupled to the at least one processor; wherein the content of the first and second substances,
the memory stores instructions executable by the at least one processor to enable the at least one processor to:
responding to an interrupt signal, and sending a first acquisition signal to a first sensor, wherein the first acquisition signal indicates the first sensor to output first acquisition data, and the first sensor starts to acquire the first acquisition data after a first time delay;
after a first acquisition signal is sent to a first sensor, a second acquisition signal is sent to a second sensor through a first time delay, the second acquisition signal indicates the second sensor to output second acquisition data, and the second sensor acquires the second acquisition data in real time;
and after sending a first acquisition signal to the first sensor, sending a receiving signal to the coprocessor through a second time delay, wherein the receiving signal indicates the coprocessor to receive and fuse first acquisition data sent by the first sensor and second acquisition data sent by the second sensor.
Further, the first sensor is a camera sensor, and the first time delay is a total time length from when the camera sensor receives the first acquisition signal to when the camera sensor starts to acquire the effective pixels.
Further, the time interval of the image frames set by the image pickup sensor is greater than or equal to the data calculation output time of the second sensor.
Further, the second time delay is the time of acquiring one frame of image of the camera sensor.
Further, the sending a first acquisition signal to the first sensor in response to the interrupt signal specifically includes:
and responding to the interrupt signal, and sending a first acquisition signal to the first sensor at a preset frequency.
The invention provides a sensor receiving control synchronization method, which comprises the following steps:
sending an interrupt signal to the multi-sensor data synchronization electronic device;
receiving a second acquisition signal which is sent by the multi-sensor data synchronization electronic equipment after a first acquisition signal is sent to a camera sensor through a first time delay, wherein the first time delay is the total time length of the camera sensor before the first acquisition signal is received and the effective pixel acquisition is started;
and outputting the collected data to the coprocessor.
Further, the sending an interrupt signal to the multi-sensor data synchronization electronic device specifically includes:
and sending an interrupt signal to the multi-sensor data synchronization electronic equipment at regular time.
The invention provides a sensor receiving control synchronous electronic device, comprising:
at least one processor; and the number of the first and second groups,
a memory communicatively coupled to the at least one processor; wherein the content of the first and second substances,
the memory stores instructions executable by the at least one processor to enable the at least one processor to:
sending an interrupt signal to the multi-sensor data synchronization electronic device;
receiving a second acquisition signal which is sent by the multi-sensor data synchronization electronic equipment after a first acquisition signal is sent to a camera sensor through a first time delay, wherein the first time delay is the total time length of the camera sensor before the first acquisition signal is received and the effective pixel acquisition is started;
and outputting the collected data to the coprocessor.
Further, the sending an interrupt signal to the multi-sensor data synchronization electronic device specifically includes:
and sending an interrupt signal to the multi-sensor data synchronization electronic equipment at regular time.
The present invention provides a storage medium storing computer instructions for performing all the steps of the multi-sensor data synchronization method as described above when executed by a computer.
The present invention provides a storage medium storing computer instructions for performing all the steps of the sensor reception control synchronization method as described above when the computer executes the computer instructions.
According to the invention, the problem of asynchronous sampling time among various sensors is solved by adopting a delayed triggering mode for the sensors, and synchronous sampling among multiple sensors is realized.
Drawings
FIG. 1 is a flow chart of the operation of a multi-sensor data synchronization method of the present invention;
FIG. 2 is a flowchart illustrating a multi-sensor data synchronization method according to a second embodiment of the present invention;
FIG. 3 is a timing diagram of the output of a camera sensor using CMOS;
FIG. 4 is a schematic diagram of a hardware configuration of a multi-sensor data synchronization electronic device according to the present invention;
FIG. 5 is a flowchart illustrating a method for synchronizing sensor reception control according to the present invention;
FIG. 6 is a flowchart illustrating a method for synchronizing reception control of a sensor according to a sixth embodiment of the present invention;
FIG. 7 is a schematic diagram of an interrupt signal;
FIG. 8 is a diagram of a hardware configuration of a sensor receiving and controlling synchronous electronic device according to the present invention;
FIG. 9 is a system schematic of the preferred embodiment of the present invention;
FIG. 10 is a timing diagram illustrating the operation of the preferred embodiment of the present invention.
Detailed Description
The invention is described in further detail below with reference to the figures and specific examples.
Example one
Fig. 1 is a flowchart illustrating a multi-sensor data synchronization method according to the present invention, which includes:
step S101, responding to an interrupt signal, sending a first acquisition signal to a first sensor, wherein the first acquisition signal indicates the first sensor to output first acquisition data, and the first sensor starts to acquire the first acquisition data after a first time delay;
step S102, after a first acquisition signal is sent to a first sensor, a second acquisition signal is sent to a second sensor through a first time delay, the second acquisition signal indicates the second sensor to output second acquisition data, and the second sensor immediately acquires the second acquisition data;
and step S103, after sending a first acquisition signal to the first sensor, sending a receiving signal to the cooperative processor through a second time delay, wherein the receiving signal indicates the cooperative processor to receive and fuse the first acquisition data sent by the first sensor and receive second acquisition data sent by the second sensor.
Specifically, the invention is applied to the multi-sensor data synchronization electronic device, and when an interrupt signal is received, the step S101 is triggered to respond. Then, step S102 first sends a first collecting signal to the first sensor, and instructs the first sensor to collect data and output the data to the co-processor, where the first sensor may be any one of the existing sensors, and the first sensor starts to collect the first collecting data after a first time delay after receiving the first collecting signal. Therefore, step S103 sends a second acquisition signal to the second sensor after the first time delay, and the second sensor acquires the second acquisition data in real time, thereby ensuring synchronous acquisition of the first acquisition data and the second acquisition data. Finally, step S104 sends a receiving signal to the co-processor, and since the first collected data and the second collected data are collected synchronously, the co-processor can directly fuse the first collected data and the second collected data without considering synchronization methods such as time stamps.
According to the invention, the problem of asynchronous sampling time among various sensors is solved by adopting a delayed triggering mode for the sensors, and synchronous sampling among multiple sensors is realized.
Example two
Fig. 2 is a flowchart illustrating a multi-sensor data synchronization method according to a second embodiment of the present invention, including:
step S201, responding to an interrupt signal, sending a first acquisition signal to a first sensor at a preset frequency, wherein the first acquisition signal indicates the first sensor to output first acquisition data, the first sensor starts to acquire the first acquisition data after a first time delay, the first sensor is a camera sensor, the first time delay is the total time length from the moment the camera sensor receives the first acquisition signal to the moment the camera sensor starts to acquire effective pixels, and the time interval of image frames set by the camera sensor is greater than or equal to the data resolving output time of a second sensor;
step S202, after sending a first acquisition signal to a first sensor, sending a second acquisition signal to a second sensor through a first time delay, wherein the second acquisition signal indicates the second sensor to output second acquisition data, and the second sensor immediately acquires the second acquisition data;
step S203, after sending the first acquisition signal to the first sensor, sending a reception signal to the coprocessor through a second time delay, where the reception signal indicates that the coprocessor receives and fuses the first acquisition data sent by the first sensor and receives the second acquisition data sent by the second sensor, and the second time delay is the time for the camera sensor to acquire one frame of image.
Specifically, the first sensor is a Camera sensor. The second sensor is preferably an IMU/GPS integrated device such as u-blob M8U.
One of the core problems of time synchronization is the alignment of the samples. Alignment means that the inherent delay of Camera sampling and the indeterminate delay due to software processing need to be eliminated. If the Camera is notified of sampling by a second sensor, such as IMU/GPS, and the CPU is notified of an interrupt to receive image data, from a software perspective, it is only known that an interrupt can receive a frame of data, but there is no other information associated with it (e.g., Camera has a hardware counter that can be used to correlate IMU/GPS time), and thus there is no way to correlate image data with IMU/GPS data. Therefore, the final solution is preferably to adopt a Camera module with a hardware counter or capable of fusing time information in an image, or to solve the problem of insufficient IMU/GPS output frequency in a hardware frequency-up mode, so as to completely eliminate the influence caused by software processing.
The present embodiment adopts a hardware frequency-up method to solve the problem. Step S201 is to send a first collecting signal to the first sensor at a preset frequency in response to the interrupt signal, so as to implement frequency boosting. By sending the first acquisition signal to the first sensor at a predetermined frequency at a certain interval, and then sequentially performing steps S202 to S203, a plurality of synchronized first acquisition data and second acquisition data can be obtained between two interrupt signals.
Step S202 sends a second acquisition signal to the second sensor through a first time delay after sending the first acquisition signal to the first sensor. Fig. 3 is a timing diagram of an output of a camera sensor using CMOS, which explains the inherent delay of CMOS image output, wherein:
t1 is the time of one frame (40 ms if 25 fps). T2 is Blank Time. I.e., the time VSync is inactive. T3 is the time VSync to HREF. I.e., the Time before VSync falls until the Start of the acquisition of a valid pixel (Start Time). This time is used to reset the potential wells and also includes the time for the acquisition of invalid lines. T4 is the time to acquire an entire row. Where T6 is the acquisition time of the active pixels in a row and T7 is the blank time (horizontal blanking) after a row has been acquired. There is a time T5, HREF to VSync, after the last row acquisition is completed. T5+ T7 is commonly referred to as an End Time (End Time).
The second sensor is preferably an IMU/GPS. As can be seen from fig. 1, in order to align the time of image acquisition and IMU/GPS time exactly, it is necessary to use the time of the start of exposure, rather than the time of the start of frame, so this fixed time delay, i.e. T2+ T3, needs to be considered. This period of time needs to be determined according to the register setting of the CMOS actually used, or by making an accurate measurement. The time length of T2+ T3 is the total time length of the image sensor before the first acquisition signal is received and the acquisition of the effective pixels is started, i.e. the first time delay.
In addition, since a part of the second sensor, for example, IMU/GPS, needs time for resolving data, the frame rate of the first sensor Camera cannot be too large, and the time between two adjacent frames covers the data resolving time of the second sensor. For example, if the second sensor is M8U, the time required for IMU/GPS resolution is about 75ms, and therefore the Camera frame rate cannot be too large, at least a 75ms delay needs to be covered to ensure that Camera takes a picture at the same time as the IMU/GPS output signal. Such as 10 fps. Such that there is a 100ms interval between each frame, 100ms being sufficient time to output synchronized IMU/GPS data and image data, as long as synchronous trigger sampling is ensured.
Finally, since the output frame of the first sensor is sometimes long, the fusion of the co-processor needs to wait until the image data is completely output before the fusion is started, and therefore, the receiving signal sent to the co-processor waits for a second time delay, that is, the time for the camera sensor to acquire one frame of image.
In the embodiment, the synchronization of data acquisition, namely fusion, is ensured by setting different time delays, and the data acquisition is increased by increasing the frequency, so that the influence caused by software processing is thoroughly eliminated.
EXAMPLE III
Fig. 4 is a schematic diagram of a hardware structure of a multi-sensor data synchronization electronic device according to the present invention, which includes:
at least one processor 401; and the number of the first and second groups,
a memory 402 communicatively coupled to the at least one processor 401; wherein the content of the first and second substances,
the memory 402 stores instructions executable by the one processor to cause the at least one processor to:
responding to an interrupt signal, and sending a first acquisition signal to a first sensor, wherein the first acquisition signal indicates the first sensor to output first acquisition data, and the first sensor starts to acquire the first acquisition data after a first time delay;
after a first acquisition signal is sent to a first sensor, a second acquisition signal is sent to a second sensor through a first time delay, the second acquisition signal indicates the second sensor to output second acquisition data, and the second sensor acquires the second acquisition data in real time;
and after sending a first acquisition signal to the first sensor, sending a receiving signal to the coprocessor through a second time delay, wherein the receiving signal indicates the coprocessor to receive and fuse first acquisition data sent by the first sensor and second acquisition data sent by the second sensor.
Further, the first sensor is a camera sensor, and the first time delay is a total time length from when the camera sensor receives the first acquisition signal to when the camera sensor starts to acquire the effective pixels.
Further, the time interval of the image frames set by the image pickup sensor is greater than or equal to the data calculation output time of the second sensor.
The electronic device is preferably a microcontroller, such as an STM32 microcontroller. In fig. 4, one processor 401 is taken as an example.
The electronic device may further include: an input device 403 and a display device 404.
The processor 401, the memory 402, the input device 403, and the display device 404 may be connected by a bus or other means, and are illustrated as being connected by a bus.
The memory 402, which is a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as program instructions/modules corresponding to the multi-sensor data synchronization method in the embodiments of the present application, for example, the method flow shown in fig. 1. The processor 401 executes various functional applications and data processing, i.e., implements the multi-sensor data synchronization method in the above-described embodiments, by executing nonvolatile software programs, instructions, and modules stored in the memory 402.
The memory 402 may include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required for at least one function; the storage data area may store data created according to the use of the multi-sensor data synchronization method, and the like. Further, the memory 402 may include high speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other non-volatile solid state storage device. In some embodiments, memory 402 optionally includes memory located remotely from processor 401, which may be connected over a network to a device that performs the multi-sensor data synchronization method. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
The input device 403 may receive input of user clicks and generate signal inputs related to user settings and function control of the multi-sensor data synchronization method. The display device 404 may include a display screen or the like.
The multi-sensor data synchronization method in any of the method embodiments described above is performed when the one or more modules are stored in the memory 402 and executed by the one or more processors 401.
According to the invention, the problem of asynchronous sampling time among various sensors is solved by adopting a delayed triggering mode for the sensors, and synchronous sampling among multiple sensors is realized.
Example four
A fourth embodiment of the present invention provides a multi-sensor data synchronization electronic device, including:
at least one processor;
a memory communicatively coupled to the at least one processor; wherein the content of the first and second substances,
the memory stores instructions executable by the one processor to cause the at least one processor to:
responding to an interrupt signal, sending a first acquisition signal to a first sensor at a preset frequency, wherein the first acquisition signal indicates the first sensor to output first acquisition data, the first sensor starts to acquire the first acquisition data after a first time delay, the first sensor is a camera sensor, the first time delay is the total time length of the camera sensor before the first acquisition signal is received and an effective pixel starts to be acquired, and the time interval of an image frame set by the camera sensor is greater than or equal to the data resolving output time of a second sensor;
after a first acquisition signal is sent to a first sensor, a second acquisition signal is sent to a second sensor through a first time delay, the second acquisition signal indicates the second sensor to output second acquisition data, and the second sensor acquires the second acquisition data in real time;
after a first acquisition signal is sent to a first sensor, a receiving signal is sent to a coprocessor through a second time delay, the receiving signal indicates that the coprocessor receives and fuses first acquisition data sent by the first sensor and second acquisition data sent by a second sensor, and the second time delay is the time for acquiring a frame of image of the camera sensor.
In the embodiment, the synchronization of data acquisition, namely fusion, is ensured by setting different time delays, and the data acquisition is increased by increasing the frequency, so that the influence caused by software processing is thoroughly eliminated.
EXAMPLE five
Fig. 5 is a flowchart illustrating a method for synchronizing sensor reception control according to the present invention, which includes:
step S501, sending an interrupt signal to multi-sensor data synchronization electronic equipment;
step S502, receiving a second acquisition signal which is sent by the multi-sensor data synchronization electronic equipment after a first acquisition signal is sent to a camera sensor through a first time delay, wherein the first time delay is the total time length from the moment the camera sensor receives the first acquisition signal to the moment the effective pixel acquisition is started;
step S503, outputting the collected data to the co-processor.
The invention is applicable to sensor electronics, such as IMU/GPS integrated devices, e.g., u-blox M8U. The sensor electronic device executes step S501 to send an interrupt signal to the multi-sensor data synchronization electronic device, and then step S502 receives a second acquisition signal sent by the multi-sensor data synchronization electronic device after a first acquisition signal is sent to the camera sensor and a first time delay is passed, and triggers step S503 to output the acquisition data to the coprocessor.
According to the invention, the problem of asynchronous sampling time among various sensors is solved by adopting a delayed triggering mode for the sensors, and synchronous sampling among multiple sensors is realized.
EXAMPLE six
Fig. 6 is a flowchart illustrating a method for synchronizing sensor reception control according to a sixth embodiment of the present invention, including:
step S601, sending an interrupt signal to the multi-sensor data synchronization electronic equipment at regular time;
step S602, receiving a second acquisition signal sent by the multi-sensor data synchronization electronic device after sending a first acquisition signal to a camera sensor through a first time delay, wherein the first time delay is the total time length from the moment the camera sensor receives the first acquisition signal to the moment the effective pixel acquisition is started;
step S603, outputting the collected data to the co-processor.
One of the core problems of time synchronization is the time alignment of the starts. The embodiment is carried out by using the sensor, such as IMU/GPS to trigger sampling of another sensor, such as Camera sensor, and simultaneously eliminating the accumulated error by means of timing trigger. For example, a Pulse of 1 Pulse Per Second (pps) is used to control Camera imaging, and the Pulse may be raised to a higher frequency, allowing Camera imaging to be 5HZ/10HZ, and so on. Example as shown in fig. 7, one interrupt signal timer is issued every 1000 ms.
The embodiment eliminates the accumulated error by sending the interrupt signal at regular time.
EXAMPLE seven
Fig. 8 is a schematic diagram of a hardware structure of a sensor receiving control synchronization electronic device according to the present invention, including:
at least one processor 801; and the number of the first and second groups,
a memory 802 communicatively coupled to the at least one processor 801; wherein the content of the first and second substances,
the memory 802 stores instructions executable by the one processor to cause the at least one processor to:
sending an interrupt signal to the multi-sensor data synchronization electronic device;
receiving a second acquisition signal which is sent by the multi-sensor data synchronization electronic equipment after a first acquisition signal is sent to a camera sensor through a first time delay, wherein the first time delay is the total time length of the camera sensor before the first acquisition signal is received and the effective pixel acquisition is started;
and outputting the collected data to the coprocessor.
The electronic device is preferably a sensor, for example an IMU/GPS integrated device such as u-blob M8U. Fig. 8 illustrates an example of a processor 801.
The electronic device may further include: an input device 803 and a display device 804.
The processor 801, the memory 802, the input device 803, and the display device 804 may be connected by a bus or other means, and are illustrated as being connected by a bus.
The memory 802, which is a non-volatile computer-readable storage medium, may be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as program instructions/modules corresponding to the multi-sensor data synchronization method in the embodiments of the present application, for example, the method flow shown in fig. 5. The processor 801 executes various functional applications and data processing, that is, implements the sensor reception control synchronization method in the above-described embodiments, by executing nonvolatile software programs, instructions, and modules stored in the memory 802.
The memory 802 may include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required for at least one function; the storage data area may store data created according to the use of the sensor reception control synchronization method, and the like. Further, the memory 802 may include high speed random access memory and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other non-volatile solid state storage device. In some embodiments, the memory 802 optionally includes memory located remotely from the processor 801, which may be connected over a network to a device that performs the sensor reception control synchronization method. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
The input device 803 may receive an input of a user click and generate signal inputs related to the sensor receiving user settings and function control that control the synchronization method. The display device 804 may include a display screen or the like.
The sensor reception control synchronization method in any of the method embodiments described above is performed when the one or more modules are stored in the memory 802 and executed by the one or more processors 801.
According to the invention, the problem of asynchronous sampling time among various sensors is solved by adopting a delayed triggering mode for the sensors, and synchronous sampling among multiple sensors is realized.
Example eight
An eighth embodiment of the present invention provides a sensor reception control synchronization electronic device, including:
at least one processor; and the number of the first and second groups,
a memory communicatively coupled to the at least one processor; wherein the content of the first and second substances,
the memory stores instructions executable by the one processor to cause the at least one processor to:
sending an interrupt signal to the multi-sensor data synchronization electronic equipment at regular time;
receiving a second acquisition signal which is sent by the multi-sensor data synchronization electronic equipment after a first acquisition signal is sent to a camera sensor through a first time delay, wherein the first time delay is the total time length of the camera sensor before the first acquisition signal is received and the effective pixel acquisition is started;
and outputting the collected data to the coprocessor.
The embodiment eliminates the accumulated error by sending the interrupt signal at regular time.
Example nine
Fig. 9 is a schematic diagram of a system according to a preferred embodiment of the present invention, which includes a camera sensor 1, an IMU/GPS integrated device 2, an STM32 microcontroller 3, and an SoC co-processor 4, wherein:
the IMU/GPS integrated equipment 2 sends IMU/GPS data to an SoC coprocessor 4 through a serial port at a rate of 10 HZ;
2. meanwhile, the TimePulse pin on the IMU/GPS integrated device 2 outputs a trigger signal of 1pps using a clock source that is homologous to the IMU/GPS. This signal is used as a reference signal to calibrate time per second
The STM32 microcontroller 3 up-converts the 1HZ signal output by the TimePulse into a 10HZ signal, and the output signal is divided into 2 paths, which are delayed appropriately to trigger the external interrupts (ExtInt) of the Camera sensor 1 and the IMU/GPS integrated device 2, respectively. It should be noted that since there will be some delay between the time when the Camera exposure starts and the time when the IMU/GPS is resolving and outputting data, the delay duration of STM32 needs to be adjusted precisely to ensure that the start time of the Camera exposure is synchronized with the signals output by the IMU/GPS. It should be noted that if M8U is selected, the time required for IMU/GPS resolution is about 75ms, so the frame rate of Camera cannot be too large, and at least 75ms of delay needs to be covered to ensure that Camera photographing time can be synchronized with the IMU/GPS output signal. Such as 10 fps. Such that there is a 100ms interval between each frame, 100ms being sufficient time to output synchronized IMU/GPS data and image data, as long as synchronous trigger sampling is ensured. Further, if an output such as BT656 (a video signal transmission standard) is employed between Camera and SoC, serial-to-parallel conversion is inevitably involved. Thus, if the frame rate of the output is low, PCLK (pixel clock) required is low, and it is easy to fail to match the operating frequency of the serial-parallel converter, and thus close attention is required. For example, if a serial-to-parallel conversion device such as TI933 is adopted, PCLK thereof is required to be between 37.5MHz and 100 MHz. The pixel clock required for 720P output was 1650 × 750 × 10 × 2 — 24.75Mhz, so that the synchronized photographed picture could not be output at a frequency of 10 HZ. The overall operational timing diagram is shown in FIG. 10 (non-precision timing, only schematic; emphasis is placed on illustrating that Camera and IMU/GPS can achieve precision synchronization of the output data under the control of TimePulse/STM 32).
Test verification
The purpose of the test is to verify the accuracy of the synchronization that can ultimately be achieved. The method adopted is to carry out verification by shooting oscilloscope waveforms by Camera. The triggering mode of the oscilloscope is performed by the pulse generated by the IMU/GPS device. The oscilloscope carries out time delay according to the method of the invention, and the time length of the square wave generated by the oscilloscope is the exposure time length. And the synchronous precision can be accurately judged according to the shot starting position of the square wave of the oscilloscope. Then the time delay between IMU/GPS and Camera shots is interpreted as 0 if the position where the shot grabbed to start is the trigger position, and half the time duration of the peak if the position where the shot grabbed to start is the middle of the peak.
A tenth embodiment of the present invention provides a storage medium storing computer instructions for performing all the steps of the multi-sensor data synchronization method as described above when executed by a computer.
An eleventh embodiment of the present invention provides a storage medium storing computer instructions for executing all the steps of the sensor reception control synchronization method as described above, when the computer executes the computer instructions.
The above-mentioned embodiments only express several embodiments of the present invention, and the description thereof is more specific and detailed, but not construed as limiting the scope of the present invention. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the inventive concept, which falls within the scope of the present invention. Therefore, the protection scope of the present patent shall be subject to the appended claims.
Claims (16)
1. A multi-sensor data synchronization method, comprising:
responding to an interrupt signal, and sending a first acquisition signal to a first sensor, wherein the first acquisition signal indicates the first sensor to output first acquisition data, and the first sensor starts to acquire the first acquisition data after a first time delay;
after a first acquisition signal is sent to a first sensor, a second acquisition signal is sent to a second sensor through a first time delay, the second acquisition signal indicates the second sensor to output second acquisition data, and the second sensor acquires the second acquisition data in real time;
and after sending a first acquisition signal to the first sensor, sending a receiving signal to the coprocessor through a second time delay, wherein the receiving signal indicates the coprocessor to receive and fuse first acquisition data sent by the first sensor and second acquisition data sent by the second sensor.
2. The multi-sensor data synchronization method according to claim 1, wherein the first sensor is a camera sensor, and the first time delay is a total time length of the camera sensor before the first acquisition signal is received until the effective pixel acquisition is started.
3. The multi-sensor data synchronization method according to claim 2, wherein a time interval of image frames set by the camera sensor is greater than or equal to a data calculation output time of the second sensor.
4. The multi-sensor data synchronization method according to claim 2, wherein the second time delay is a time of acquiring one frame of image of the camera sensor.
5. The multi-sensor data synchronization method according to claim 1, wherein the sending a first acquisition signal to a first sensor in response to the interrupt signal specifically comprises:
and responding to the interrupt signal, and sending a first acquisition signal to the first sensor at a preset frequency.
6. A multi-sensor data synchronization electronic device, comprising:
at least one processor; and the number of the first and second groups,
a memory communicatively coupled to the at least one processor; wherein the content of the first and second substances,
the memory stores instructions executable by the at least one processor to enable the at least one processor to:
responding to an interrupt signal, and sending a first acquisition signal to a first sensor, wherein the first acquisition signal indicates the first sensor to output first acquisition data, and the first sensor starts to acquire the first acquisition data after a first time delay;
after a first acquisition signal is sent to a first sensor, a second acquisition signal is sent to a second sensor through a first time delay, the second acquisition signal indicates the second sensor to output second acquisition data, and the second sensor acquires the second acquisition data in real time;
and after sending a first acquisition signal to the first sensor, sending a receiving signal to the coprocessor through a second time delay, wherein the receiving signal indicates the coprocessor to receive and fuse first acquisition data sent by the first sensor and second acquisition data sent by the second sensor.
7. The multi-sensor data synchronization electronic device of claim 6, wherein the first sensor is a camera sensor, and wherein the first time delay is a total duration of the camera sensor before the first capture signal is received until the valid pixel starts to be captured.
8. The multi-sensor data synchronization electronic device according to claim 7, wherein a time interval of image frames set by the camera sensor is greater than or equal to a data calculation output time of the second sensor.
9. The multi-sensor data synchronization electronic device of claim 7, wherein the second time delay is a time of the camera sensor to capture one frame of image.
10. The multi-sensor data synchronization electronic device of claim 6, wherein said sending a first acquisition signal to a first sensor in response to an interrupt signal specifically comprises:
and responding to the interrupt signal, and sending a first acquisition signal to the first sensor at a preset frequency.
11. A sensor reception control synchronization method, comprising:
sending an interrupt signal to the multi-sensor data synchronization electronic device;
receiving a second acquisition signal which is sent by the multi-sensor data synchronization electronic equipment after a first acquisition signal is sent to a camera sensor through a first time delay, wherein the first time delay is the total time length of the camera sensor before the first acquisition signal is received and the effective pixel acquisition is started;
and outputting the collected data to the coprocessor.
12. The sensor reception control synchronization method according to claim 11, wherein the sending of the interrupt signal to the multi-sensor data synchronization electronic device specifically includes:
and sending an interrupt signal to the multi-sensor data synchronization electronic equipment at regular time.
13. A sensor reception control synchronization electronic device, comprising:
at least one processor; and the number of the first and second groups,
a memory communicatively coupled to the at least one processor; wherein the content of the first and second substances,
the memory stores instructions executable by the at least one processor to enable the at least one processor to:
sending an interrupt signal to the multi-sensor data synchronization electronic device;
receiving a second acquisition signal which is sent by the multi-sensor data synchronization electronic equipment after a first acquisition signal is sent to a camera sensor through a first time delay, wherein the first time delay is the total time length of the camera sensor before the first acquisition signal is received and the effective pixel acquisition is started;
and outputting the collected data to the coprocessor.
14. The sensor reception control synchronization electronic device of claim 13, wherein the sending of the interrupt signal to the multi-sensor data synchronization electronic device specifically comprises:
and sending an interrupt signal to the multi-sensor data synchronization electronic equipment at regular time.
15. A storage medium storing computer instructions for performing all the steps of the multi-sensor data synchronization method according to any one of claims 1 to 5 when executed by a computer.
16. A storage medium storing computer instructions for performing all the steps of the sensor reception control synchronization method according to any one of claims 11 to 12 when the computer instructions are executed by a computer.
Priority Applications (3)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911095346.3A CN111813716A (en) | 2019-11-11 | 2019-11-11 | Multi-sensor data synchronization, electronic device, and storage medium |
| PCT/CN2020/124262 WO2021093586A1 (en) | 2019-11-11 | 2020-10-28 | Data synchronization system, and method |
| US17/662,837 US11902378B2 (en) | 2019-11-11 | 2022-05-10 | Systems and methods for data synchronization |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911095346.3A CN111813716A (en) | 2019-11-11 | 2019-11-11 | Multi-sensor data synchronization, electronic device, and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111813716A true CN111813716A (en) | 2020-10-23 |
Family
ID=72843699
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911095346.3A Pending CN111813716A (en) | 2019-11-11 | 2019-11-11 | Multi-sensor data synchronization, electronic device, and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111813716A (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112362088A (en) * | 2020-12-11 | 2021-02-12 | 中国石油大学(华东) | Synchronous acquisition method and system for multi-grating data |
| WO2021093586A1 (en) * | 2019-11-11 | 2021-05-20 | 北京嘀嘀无限科技发展有限公司 | Data synchronization system, and method |
| CN114413890A (en) * | 2022-01-14 | 2022-04-29 | 广州小鹏自动驾驶科技有限公司 | Vehicle track generation method, vehicle track generation device, electronic device, and storage medium |
| WO2022226978A1 (en) * | 2021-04-30 | 2022-11-03 | 深圳市大疆创新科技有限公司 | Data processing method and device |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP1575058A1 (en) * | 2004-03-08 | 2005-09-14 | Consejo Superior De Investigaciones Cientificas | System and method for detecting the displacement of a plurality of micro- and nanomechanical elements, such as micro-cantilevers |
| CN107172320A (en) * | 2017-06-21 | 2017-09-15 | 成都理想境界科技有限公司 | Method of data synchronization and picture pick-up device |
| CN108645402A (en) * | 2018-03-30 | 2018-10-12 | 深圳清创新科技有限公司 | Camera shooting and inertia measurement sensing device, scene cut and pose computing system |
| CN108988974A (en) * | 2018-06-19 | 2018-12-11 | 远形时空科技（北京）有限公司 | Measurement method, device and the system to electronic equipment time synchronization of time delays |
| CN209297128U (en) * | 2018-12-31 | 2019-08-23 | 南京茂森电子技术有限公司 | A kind of multisensor multi channel signals synchronous |
-
2019
- 2019-11-11 CN CN201911095346.3A patent/CN111813716A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP1575058A1 (en) * | 2004-03-08 | 2005-09-14 | Consejo Superior De Investigaciones Cientificas | System and method for detecting the displacement of a plurality of micro- and nanomechanical elements, such as micro-cantilevers |
| CN107172320A (en) * | 2017-06-21 | 2017-09-15 | 成都理想境界科技有限公司 | Method of data synchronization and picture pick-up device |
| CN108645402A (en) * | 2018-03-30 | 2018-10-12 | 深圳清创新科技有限公司 | Camera shooting and inertia measurement sensing device, scene cut and pose computing system |
| CN108988974A (en) * | 2018-06-19 | 2018-12-11 | 远形时空科技（北京）有限公司 | Measurement method, device and the system to electronic equipment time synchronization of time delays |
| CN209297128U (en) * | 2018-12-31 | 2019-08-23 | 南京茂森电子技术有限公司 | A kind of multisensor multi channel signals synchronous |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2021093586A1 (en) * | 2019-11-11 | 2021-05-20 | 北京嘀嘀无限科技发展有限公司 | Data synchronization system, and method |
| US11902378B2 (en) | 2019-11-11 | 2024-02-13 | Beijing Didi Infinity Technology And Development Co., Ltd. | Systems and methods for data synchronization |
| CN112362088A (en) * | 2020-12-11 | 2021-02-12 | 中国石油大学(华东) | Synchronous acquisition method and system for multi-grating data |
| WO2022226978A1 (en) * | 2021-04-30 | 2022-11-03 | 深圳市大疆创新科技有限公司 | Data processing method and device |
| CN114413890A (en) * | 2022-01-14 | 2022-04-29 | 广州小鹏自动驾驶科技有限公司 | Vehicle track generation method, vehicle track generation device, electronic device, and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111813716A (en) | Multi-sensor data synchronization, electronic device, and storage medium | |
| CN109104259B (en) | Multi-sensor time synchronization system and method | |
| US9654672B1 (en) | Synchronized capture of image and non-image sensor data | |
| CN107277385B (en) | Multi-camera system synchronous exposure control method and device and terminal equipment | |
| CN112383675B (en) | Time synchronization method and device and terminal equipment | |
| CN111556224B (en) | Multi-camera synchronous calibration method, device and system | |
| CN107231533B (en) | synchronous exposure method and device and terminal equipment | |
| CN211308445U (en) | Multi-sensor data synchronization system and automobile | |
| US20150271469A1 (en) | Image synchronization method for cameras and electronic apparatus with cameras | |
| KR101389789B1 (en) | Image pickup apparatus, image pickup system, image pickup method and computer readable non-transitory recording medium | |
| CN107948463B (en) | Camera synchronization method, device and system | |
| CN110198415B (en) | Method and device for determining image timestamp | |
| CN104349163A (en) | Measurement display device and measurement system for end-to-end video transmission delay | |
| CN108781259A (en) | A kind of control method of image taking, control device and control system | |
| CN104202534A (en) | Multi-camera synchronous control device based on GPS and pulse generator and method | |
| CN109862281B (en) | Camera Link imaging system with adjustable exposure time in global shutter mode | |
| CN106165394B (en) | The control device of photographic device | |
| CN110139066A (en) | A kind of Transmission system of sensing data, method and apparatus | |
| KR20190118548A (en) | Image sensor, and control system | |
| CN114025055A (en) | Data processing method, device, system, equipment and storage medium | |
| JP5553409B2 (en) | Total delay time measuring system and total delay time measuring method | |
| US7929856B2 (en) | Imaging system and method for capturing still image | |
| CN106068642A (en) | Imaging device, image processing apparatus, display control unit and imaging display apparatus | |
| CN111371982B (en) | Data synchronization method, processor and vehicle-mounted terminal | |
| CN112995524A (en) | High-precision acquisition vehicle, and photo exposure information generation system, method and synchronization device thereof |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20201023 |
|
| RJ01 | Rejection of invention patent application after publication |