WO2022205734A1

WO2022205734A1 - Image time synchronization method and apparatus, and electronic device and storage medium

Info

Publication number: WO2022205734A1
Application number: PCT/CN2021/112178
Authority: WO
Inventors: 陈士荣; 苗占东; 蔡军; 欧启琛
Original assignee: 亿咖通（湖北）科技有限公司
Priority date: 2021-04-02
Filing date: 2021-08-12
Publication date: 2022-10-06
Also published as: CN112751983B; CN112751983A

Abstract

The present disclosure provides an image time synchronization method and apparatus, and an electronic device and a storage medium. The method comprises: acquiring a timestamp sequence of the exposure of images, and acquiring a system time of each timestamp from the timestamp sequence; acquiring an image sequence, and determining a system time of each image from the image sequence; on the basis of a predetermined image generation duration, performing first alignment on a timestamp in the timestamp sequence and an image in the image sequence; calculating a standard frame number difference between a first-type frame number of the timestamp which has been subjected to first alignment and a second-type frame number of the image which has been subjected to first alignment; and on the basis of the standard frame number difference, performing matching between each timestamp in the timestamp sequence and each image in the image sequence, and determining a timestamp, which matches an image, to be an exposure timestamp of the image.

Description

Image time synchronization method, device, electronic device and storage medium

technical field

The present application relates to the technical field of intelligent vehicles, and in particular, to an image time synchronization method, device, electronic device and storage medium.

Background technique

The traditional method of obtaining the time stamp of the image captured by the camera is: after the system side of the industrial computer obtains the image through the driver, the current time of the system is used as the time stamp of the image. But this is not accurate enough. First, the system clock is not a high-precision clock, and secondly, the time to acquire the image from the driver is not the image exposure time. Therefore, the above method is usually applied to some systems with weak real-time requirements.

For in-vehicle intelligent driving or high-precision map acquisition operation systems with high real-time requirements, a more accurate high-precision clock is required for camera time synchronization.

In response to this problem, some autonomous driving companies have applied time synchronizers in smart vehicles. The principle is to provide timing for each sensor through the pulse generator that comes with the time synchronizer, thereby providing high-precision time. When taking an image, the synchronizer is controlled to generate a pulse signal, and the exposure of the vehicle-mounted camera is controlled by the pulse signal, and the time stamp obtained from the serial port of the synchronizer is the high-precision exposure time of the vehicle-mounted camera.

However, due to the instability of the system, the serial port of the time synchronizer and the camera driver may have the problem of data frame loss, which will cause the timestamp data and image data to be unable to accurately correspond. Exposure timestamps are also not accurate enough.

SUMMARY OF THE INVENTION

One or more embodiments of the present application provide an image time synchronization method, the method comprising:

Obtain a timestamp sequence of image exposure, and obtain the system time of each frame timestamp from the timestamp sequence;

Acquire an image sequence, and determine the system time of each frame of image in the image sequence;

Based on a predetermined image generation duration, the timestamps in the timestamp sequence and the images in the image sequence are aligned for the first time, wherein the difference between the system time of the timestamps aligned for the first time and the system time of the images aligned for the first time satisfies all requirements the image generation time;

Calculate the standard frame number difference between the first type of frame number of the time stamp of the first alignment and the second type of frame number of the first aligned image;

Based on the standard frame number difference, each frame time stamp in the time stamp sequence is matched with each frame image in the image sequence, and the time stamp matched with the image is determined as the exposure time stamp of the frame image.

According to one or more embodiments, the step of aligning the timestamps in the timestamp sequence and the images in the image sequence for the first time based on a predetermined image generation duration includes:

discarding the first several frames of data of the image sequence;

Calculate the system time difference between the system time of the queue head image of the image queue after discarding the first several frames of data and the system time of the queue head timestamp in the timestamp sequence;

Determine whether the system time difference satisfies the image generation duration:

If so, then it is determined that after discarding the first several frames of data, the head image and the head timestamp of the image queue are respectively the first aligned image and the first aligned timestamp;

If not, discard the head-of-line timestamp, and return the system time difference between the system time of the head-of-line image of the image queue after performing the calculation and discarding the first several frames of data and the system time of the head-of-line timestamp in the timestamp sequence step.

According to one or more embodiments, based on the standard frame number difference, the time stamp of each frame in the time stamp sequence is matched with each frame of the image in the image sequence, and the time stamp matched with the image is determined. The steps for the exposure time stamp for the frame image include:

Calculate the frame number difference between the first type of frame number of the line head timestamp in the time stamp sequence and the second type of frame number of the line head image in the image sequence, and determine whether the frame number difference is equal to the Describe the standard frame number difference:

If so, determine the team head timestamp as the exposure timestamp of the team head image, update the team head timestamp of the timestamp sequence and the team head image of the image sequence, and return to calculate The step of the frame number difference between the first type frame number of the team head timestamp in the time stamp sequence and the second type frame number of the team head image in the image sequence;

If not, update the team head timestamp of the time stamp sequence or the team head image of the image sequence, and return the frame number of the first type for calculating the team head timestamp in the time stamp sequence and the team head image of the image sequence. The step of the frame number difference between the second type of frame numbers of the image.

According to one or more embodiments, the step of updating the head-of-line timestamp of the time-stamp sequence and the head-of-line image of the image sequence includes:

Discarding the team head timestamp in the time stamp sequence, and determining the next time stamp of the discarded team head timestamp as a new team head timestamp in the timestamp sequence;

The head image in the image sequence is discarded, and the next image of the discarded head image is determined as a new head image in the image sequence.

According to one or more embodiments, the step of updating the head-of-line timestamp of the sequence of time stamps or the head-of-line image of the sequence of images includes:

If the frame number difference is greater than the standard frame number difference, discard the head image in the image sequence, and determine the next image of the discarded head image as the new head image in the image sequence image;

If the frame number difference is smaller than the standard frame number difference, discard the line head timestamp in the time stamp sequence, and determine the next time stamp of the discarded line head time stamp as the time stamp sequence The new team leader timestamp.

According to one or more embodiments, the step of acquiring a sequence of time stamps for image exposure includes:

The time stamp sequence of image exposure is obtained from the serial port of the synchronizer, wherein each frame of time stamp represents the time when the pulse signal generated by the synchronizer triggers the exposure of the camera.

One or more embodiments of the present application also provide an image time synchronization apparatus, the apparatus comprising:

a first acquisition module, configured to acquire a timestamp sequence of image exposure, and acquire the system time of each frame timestamp from the timestamp sequence;

a second acquisition module, configured to acquire an image sequence, and determine the system time of each frame of image in the image sequence;

an alignment module for aligning the timestamps in the timestamp sequence and the images in the image sequence for the first time based on a predetermined image generation duration, wherein the system time of the timestamps aligned for the first time and the system time of the aligned images for the first time The time difference satisfies the image generation duration;

A calculation module for calculating the standard frame number difference between the first type of frame number of the time stamp of the first alignment and the second type of frame number of the first aligned image;

The matching module is configured to match each frame time stamp in the time stamp sequence with each frame image in the image sequence based on the standard frame number difference, and determine the time stamp matched with the image as the frame of the image. Exposure timestamp.

According to one or more embodiments, the alignment module is specifically used for:

discarding the first several frames of data of the image sequence;

According to one or more embodiments, the matching module is specifically used for:

If not, update the team head timestamp of the timestamp sequence or the team head image of the image sequence, and return the first type of frame number and the first type of frame number for calculating the team head timestamp in the timestamp sequence The step of the frame number difference between the second type of frame numbers of the head of line images in the sequence of images.

According to one or more embodiments, when performing the updating of the team head timestamp of the timestamp sequence and the team head image of the image sequence, the matching module is specifically configured to:

discarding the team head timestamp in the timestamp sequence, and determining the next timestamp of the discarded team head timestamp as a new team head timestamp in the timestamp sequence;

According to one or more embodiments, when performing the updating of the team head timestamp of the timestamp sequence or the team head image of the image sequence, the matching module is specifically configured to:

According to one or more embodiments, when executing the time stamp sequence for acquiring image exposure, the first acquiring module is specifically configured to:

One or more embodiments of the present application also provide an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus. Memory is used to store computer programs. When the processor is configured to execute the program stored in the memory, the method described in any of the foregoing embodiments is implemented.

One or more embodiments of the present application further provide a computer-readable storage medium, where a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, the computer program described in any of the foregoing embodiments is implemented. Methods.

Description of drawings

In order to illustrate the embodiments of the present application more clearly, the accompanying drawings will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and for those of ordinary skill in the art, other embodiments can also be obtained according to these drawings without creative effort.

1 is a schematic flowchart of an image time synchronization method according to one or more embodiments of the present invention;

2 is a schematic flowchart of first alignment of timestamps and images according to one or more embodiments of the present invention;

3 is a schematic flowchart of matching a timestamp sequence and an image sequence according to one or more embodiments of the present invention;

4 is a schematic structural diagram of an image time synchronization apparatus according to one or more embodiments of the present invention;

FIG. 5 is a schematic structural diagram of an electronic device according to one or more embodiments of the present invention.

Detailed ways

The embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, but not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art based on the present application fall within the protection scope of the present application.

For ease of understanding, the application scenarios of the present application are briefly introduced first.

The image time synchronization method according to one or more embodiments of the present invention can be applied to scenarios with high real-time requirements, such as vehicle intelligent driving, high-precision map acquisition, etc. The exact exposure time of the image.

In order to provide high-precision time, a time synchronizer can be installed in the vehicle system. The principle is to use the second pulse generator to provide timing for each sensor. In turn, the synchronizer generates a pulse signal, which further controls the triggering of each sensor.

For the vehicle-mounted camera, the synchronizer can generate a pulse signal to trigger the vehicle-mounted camera exposure, and the time stamp when the synchronizer triggers the vehicle-mounted camera exposure is the high-precision time of the vehicle-mounted camera exposure.

However, the time stamp signal that triggers the exposure of the vehicle camera must be read through the serial port of the synchronizer, but the serial port data of the synchronizer may lose frames or images, so that the image captured by the vehicle camera exposure and the time stamp that triggers the exposure of the vehicle camera cannot be exposed. Accurate correspondence, which in turn leads to the inaccurate exposure time of the final image.

One or more embodiments of the present application provide an image time synchronization method, apparatus, electronic device, and storage medium. Referring to Figure 1, the method may include the following steps:

S101: Acquire a time stamp sequence of image exposure, and obtain the system time of each time stamp from the time stamp sequence.

According to one or more embodiments, a sequence of time stamps of image exposure can be obtained from the serial port of the synchronizer, wherein each time stamp represents the time when the pulse signal generated by the synchronizer triggers the exposure of the camera.

In addition, the time of the system when each time stamp is obtained from the serial port of the synchronizer can be recorded as the system time of each time stamp, wherein the system can be an industrial control system.

According to one or more embodiments, the acquired timestamps are sequentially arranged and combined into a timestamp sequence, and each timestamp corresponds to a system time.

In order to ensure that the serial port data of the synchronizer is real and effective, the data check digit can be set. After obtaining the data from the serial port of the synchronizer, the data is checked first according to the check digit. When the check is correct, the serial port data of the synchronizer is considered to be credible, and then the data is obtained from the serial port of the synchronizer. Parse out timestamp data. If it is not credible, data repair can be performed based on the frequency of the on-board camera and the timestamp data of the previous frame.

As an example, the obtained timestamp sequence contains 10 timestamps, which are represented as a1, a2, a3, a4, a5, a6, a7, a8, a9, and a10 in sequence, and the corresponding system times are 10, 60, and 110 in sequence. , 160, 210, 260, 310, 360, 410, and 460 in milliseconds.

S102: Acquire an image sequence, and determine the system time of each frame of image in the image sequence.

According to one or more embodiments, a captured image may be acquired from a vehicle-mounted camera driver, and the acquired image is an image generated by a synchronizer triggering exposure of the vehicle-mounted camera.

In addition, the system time when each image is acquired from the camera driver can be recorded as the system time of each image.

The acquired images are arranged and combined in sequence to form an image sequence, and each image corresponds to a system time.

As an example, the acquired image sequence contains 10 images, which are denoted as b1, b2, b3, b4, b5, b6, b7, b8, b9, and b10 in sequence, and the corresponding system times are 70, 120, 170, and 220 in sequence. , 270, 320, 370, 420, 470, and 560 in milliseconds.

S103: Based on the predetermined image generation duration, align the timestamps in the timestamp sequence with the images in the image sequence for the first time, wherein the difference between the system time of the timestamps aligned for the first time and the system time of the images aligned for the first time satisfies the image generation duration .

The image generation duration can be understood as the duration from when the synchronizer triggers the exposure of the camera to when the camera drives to generate the image. According to one or more embodiments, the image generation duration can be predetermined.

As an example, the image generation time is 60ms.

According to one or more embodiments, the time stamps in the time stamp sequence and the images in the image sequence may be first aligned according to the image generation duration.

Since the time stamp of the serial port of the synchronizer may not correspond to the image driven by the camera when the industrial control system is started, it cannot be considered that the obtained time stamp sequence and the image sequence exactly match. Therefore, to complete the first alignment of the time stamp and the image, then When the data is not lost, the timestamp after the first alignment is in a one-to-one correspondence with the image.

FIG. 2 is a schematic flowchart of the first alignment of time stamps and images according to one or more embodiments of the present invention. As shown in FIG. 2, step S103 may include the following refinement steps:

S201: Discard the first several frames of data of the image sequence.

S202 : Calculate the system time difference between the system time of the line head image of the image queue after discarding several frames of data and the system time of the line head timestamp in the timestamp sequence.

As an example, the first two frames of images b1 and b2 in the image sequence may be discarded, and the head image of the image sequence after the data is discarded is b3. Then, the difference between the system time of b3 and the system time of the queue head timestamp a1 in the timestamp sequence is calculated, that is, 170-10=160.

S203: Determine whether the system time difference satisfies the image generation duration, if yes, execute S204; if not, execute S205.

S204: Determine that the head image and the head timestamp of the image queue after discarding several frames of data are the first aligned image and the first aligned timestamp, respectively.

S205: Discard the queue head timestamp, and return to the step of calculating the system time difference between the system time of the queue head image of the image queue after discarding several frames of data and the system time of the queue head timestamp in the timestamp sequence.

It is easy to understand that as long as the calculated system time difference is not much different from the image generation time, it can be considered that the system time difference meets the image generation time. Image generation time.

Following the above example, the system acquisition time of image b3 is 170, the system acquisition time of the first timestamp a1 in the timestamp sequence and the system acquisition time of timestamp a1 are 10, and the system time difference between the two is 170-10=160, The image generation time is roughly 60 milliseconds, so it does not meet the image generation time.

Furthermore, compared with the next time stamp of a1, that is, a2, the calculated system time difference is 170-60=110, which still does not satisfy the image generation duration.

Continue to compare with the system acquisition time of timestamp a3, and the calculated system time difference is 170-110=60, which satisfies the image generation time. Then, the timestamp a3 and the image b3 are determined as the timestamp of the first alignment and the image aligned for the first time, respectively.

According to one or more embodiments, after the first-aligned timestamp and the first-aligned image are determined, the timestamp after the first-aligned timestamp is the same as the first-aligned timestamp without considering the data loss and image loss of the serial port timestamp of the synchronizer. The images after the aligned images are in a one-to-one correspondence.

S104: Calculate the standard frame number difference between the first-type frame number of the time stamp of the first alignment and the second-type frame number of the first-aligned image.

According to one or more embodiments, the time stamp sequence further includes the frame number of each time stamp, which represents the generation sequence of the time stamps, which can be recorded as the first type of frame number; the image sequence also includes the frame number of each image, Indicates the generation order of the image, which can be recorded as the second type of frame number.

In this step, the standard frame number difference between the first-type frame number of the first-aligned timestamp and the second-type frame number of the first-aligned image can be calculated.

As an example, the first type frame numbers of a1-a10 in the above timestamp sequence are 21, 22, 23, 24, 25, 26, 27, 28, 29, and 30, respectively;

The second type frame numbers of b1-b10 in the above image sequence are 11, 12, 13, 14, 15, 16, 17, 18, 19, and 20, respectively.

Then, it can be determined that the difference between the first-type frame number of the time stamp a3 aligned for the first time and the second-type frame number of the image b3 aligned for the first time is 23-13=10, which is taken as the standard frame number difference.

S105: Based on the standard frame number difference, match the timestamp of each frame in the timestamp sequence with each frame of the image in the image sequence, and determine the timestamp matched with the image as the exposure timestamp of the frame of image.

According to one or more embodiments, after the standard frame number difference is determined, there is no need to perform matching between the time stamp and the system time of the image, and only the standard frame number difference can match the time stamp and the image.

According to an embodiment, if the frame number difference between the first type of frame number of a certain timestamp in the timestamp sequence and the second type of frame number of a certain frame image in the image sequence is equal to the standard frame number difference, the timestamp can be determined match this image.

According to one or more embodiments, during the matching process, each frame of image in the image sequence may be sequentially read, and for each frame of image, the second type of frame number of the frame of image may be determined, and the sequence of time stamps may be searched for the same The time stamp of the frame image matching, the frame number difference between the first type frame number of the time stamp and the second type frame number of the frame image is equal to the standard frame number difference. Further, the time stamp can be assigned to the frame image, that is, the time stamp can be determined as the exposure time stamp of the frame image.

In addition, if the time stamp that matches the image cannot be found from the time stamp sequence, indicating that the time stamp data of the serial port of the synchronizer is lost, this frame of image can be discarded and the next frame of image can be read.

Applying the image time synchronization method according to one or more embodiments of the present invention, first aligning the time stamp and the image for the first time, and calculating the difference between the first type frame number of the time stamp aligned for the first time and the second type frame number of the image aligned for the first time. According to the standard frame number difference between the two, the time stamp in the timestamp sequence and the image in the image sequence can be matched according to the standard frame number difference, and it can ensure that the time stamp and the image whose frame number difference is equal to the standard frame number difference are exactly matched. , and then assign the timestamp to the matching image, which can determine the high-precision exposure time of the image.

FIG. 3 is a schematic flowchart of matching a timestamp sequence and an image sequence according to one or more embodiments of the present invention. In order to reduce the computational complexity and further improve the matching efficiency between the timestamp and the image, as shown in FIG. 3 , the above step S105 may specifically include the following steps:

S301: Calculate the frame number difference between the first type frame number of the queue head timestamp in the timestamp sequence and the second type frame number of the queue head image in the image sequence.

In the embodiment shown in FIG. 3 , the sequence of time stamps may be the sequence of time stamps after the time stamp of the first alignment, and the sequence of images may be the sequence of images after the first aligned image.

According to one or more embodiments, considering that the serial port timestamp data loss and image loss of the synchronizer may occur, the first type of frame number of the first timestamp in the timestamp sequence and the second frame number of the first image in the image sequence may be based on The class frame number is matched, and after each match, regardless of whether the match is successful or not, the timestamp sequence and/or the image sequence are updated, specifically deleting the team head timestamp and/or the team head image.

According to one embodiment, both the timestamp sequence and the image sequence may be stored in a data format in the form of a queue to follow a first-in, first-out principle.

S302: Determine whether the frame number difference is equal to the standard frame number difference, if so, execute S303; if not, execute S304.

S303: Determine the team head timestamp as the exposure timestamp of the team head image, update the team head timestamp of the timestamp sequence and the team head image of the image sequence, and return to the step of S301.

According to one embodiment, when the first type frame number of the team head timestamp and the second type frame number of the team head image are equal to the standard frame number difference, indicating that the team head timestamp matches the team head image, the team head timestamp Determine the exposure timestamp of the team head image, that is, assign the team head timestamp to the team head image.

According to an embodiment, in the case of successful matching, updating the team head timestamp of the timestamp sequence and the team head image of the image sequence, specifically:

Discard the team head timestamp in the timestamp sequence, and determine the next timestamp of the discarded team head timestamp as the new team head timestamp in the timestamp sequence; discard the team head image in the image sequence, and use the The next image of the discarded head image is determined to be the new head image in the image sequence.

After updating the team head timestamp of the time stamp sequence and the team head image of the image sequence, return to step S301 to re-match the new team head timestamp and the new team head image.

S304: Update the team head timestamp of the timestamp sequence or the team head image of the image sequence, and return to the step of S301.

When the first type frame number of the team head timestamp and the second type frame number of the team head image are not equal to the standard frame number difference, it means that the team head timestamp does not match the team head image, and there may be time stamp loss or image loss. Case.

According to an embodiment, if the frame number difference is greater than the standard frame number difference, indicating that the time stamp is lost, the head image in the image sequence can be discarded, and the next image of the discarded head image is determined as the image The new team head image in the sequence is returned to the step of S301, and the new team head image and the team head timestamp are re-matched.

If the frame number difference is less than the standard frame number difference, it means that the image is lost, the queue head timestamp in the timestamp sequence can be discarded, and the next timestamp of the discarded queue head timestamp can be determined as the timestamp sequence and return to step S301 to re-match the new team head timestamp and the team head image.

For ease of understanding, the following description is given with reference to specific examples.

Assuming that the timestamp a5 is lost due to the instability of the serial port of the synchronizer, the sequence of timestamps to be matched is a4, a6, a7, a8, a9, a10; assuming that the camera driver is unstable and the image b8 is lost, the sequence of images to be matched is b4, b5, b6, b7, b9, b10. The first type of frame number of each timestamp and the second type of frame number of each image continue the above example.

Then the frame number difference between the first type frame number of the team head timestamp and the second type frame number of the team head image can be calculated, that is, the first type frame number of the timestamp a4 and the second type frame number of the image b4. The frame number difference between them, specifically: 24-14=10, is equivalent to the standard frame number difference, so the timestamp a4 and the image b4 are matched, and the timestamp a4 is determined as the exposure timestamp of the image b4. And discard the team head timestamp a4, and lose the team head image b4.

Because the time stamp a5 is lost, the new team head time stamp becomes the time stamp a6, and its first type frame number is 26; the new team head image becomes image b5, and its second type frame number is 15.

Recalculate the frame number difference between the first type frame number of the team head timestamp and the second type frame number of the team head image, and obtain 26-15=11, which is greater than the standard frame number difference.

According to one or more embodiments, the calculated frame number difference is greater than the standard frame number difference, indicating that the time stamp is lost, and therefore the precise exposure time of the corresponding image cannot be obtained. The step of the frame number difference between the first-class frame number and the second-class frame number of the head image.

Continuing the above example, discard the image b5, and calculate the frame number difference between the first type frame number of the current team head timestamp a6 and the second type frame number of the team head image b6, which is equivalent to the standard frame number difference, so the timestamp a6 is Exposure timestamp of image b6, and then delete timestamp a6 and image b6, respectively, to obtain a new team head timestamp and team head image.

In the same way, the timestamp a7 matches the image b7, delete the timestamp a7 and the image b7 respectively, the new team head timestamp is a8, but the image b8 is lost due to the unstable camera driver, the new team head image is b9, calculate The frame number difference between the first type frame number of a8 and the second type frame number of b9 is 28-19=9, which is smaller than the standard frame number difference.

According to one or more embodiments, the calculated frame number difference is smaller than the standard frame number difference, indicating that the image is lost, so the head-of-line timestamp can be deleted, and the difference between the first type of frame number used for the calculation of the head-of-group timestamp and the head image of the lineup is returned. The step of the frame number difference between the second type of frame numbers.

Continue the above example, delete the timestamp a8, continue to calculate the frame number difference between the first type frame number of the team head timestamp and the second type frame number of the team head image, until the timestamp sequence does not contain timestamps and/or images The sequence does not contain images.

After the above calculation, it can be obtained: the time stamp a4 matches the image b4, that is, the exposure time stamp of the image b4 is a4; the time stamp a6 matches the image b6, that is, the exposure time stamp of the image b6 is a6; the time stamp a7 is the same as the image b7 matches, that is, the exposure timestamp of image b7 is a7; timestamp a9 matches image b9, that is, the exposure timestamp of image b9 is a9; timestamp a10 matches image b10, that is, the exposure timestamp of image b10 is a10.

It can be seen that, according to one or more embodiments, the timestamp sequence and the image sequence are matched based on the standard frame number difference, and in each matching, only the first type of frame number of the team head timestamp and the second frame number of the team head image need to be calculated. The frame number difference of the class frame number can be accurately matched according to the size relationship between the frame number difference and the standard frame number difference, and the accurate exposure timestamp of the image can be obtained. The above matching process has very low computational complexity.

Moreover, it can take into account the loss of the serial port timestamp of the synchronizer and the loss of the camera-driven image. Even if the above situation occurs, it can be handled well without affecting the subsequent matching, and has strong robustness and practicability.

According to one or more embodiments, after the precise exposure time of the image is determined, the image data and its corresponding precise exposure data can be sent to the application layer through interfaces such as ROS, CyberRT, Socket, etc., for data fusion or mapping.

Corresponding to the image time synchronization method embodiments according to one or more embodiments of the present invention, the embodiments of the present application further provide an image time synchronization apparatus. Referring to Figure 4, the apparatus may include the following modules:

The first acquisition module 401 is used for acquiring the time stamp sequence of image exposure, and acquiring the system time of each time stamp from the time stamp sequence;

A second acquisition module 402, configured to acquire an image sequence, and determine the system time of each frame of image in the image sequence;

The alignment module 403 is configured to align the timestamps in the timestamp sequence and the images in the image sequence for the first time based on the predetermined image generation duration, wherein the difference between the system time of the timestamps aligned for the first time and the system time of the images aligned for the first time Satisfy the image generation time;

Calculation module 404, for calculating the standard frame number difference between the first type frame number of the time stamp aligned for the first time and the second type frame number of the image aligned for the first time;

The matching module 405 is configured to match each frame time stamp in the time stamp sequence with each frame image in the image sequence based on the standard frame number difference, and determine the time stamp matched with the image as the exposure time stamp of the frame image.

According to one embodiment, the matching model 405 can be specifically used for:

Calculate the frame number difference between the first type frame number of the queue head timestamp in the time stamp sequence and the second type frame number of the queue head image in the image sequence, and determine whether the frame number difference is equal to the standard frame number If it is bad, if so, determine the team head timestamp as the exposure timestamp of the team head image, update the team head timestamp of the timestamp sequence and the team head image of the image sequence, and return to calculate the The step of the frame number difference between the first type frame number of the queue head timestamp in the time stamp sequence and the second type frame number of the queue head image in the image sequence;

If the frame number difference is not equal to the standard frame number difference, update the line head timestamp of the time stamp sequence or the line head image of the image sequence, and return the result of calculating the line head time stamp in the time stamp sequence. The step of the frame number difference between the frame number of the first type and the frame number of the second type of the first image in the sequence of images.

Applying the image time synchronization apparatus according to one or more embodiments of the present invention, firstly align the timestamp and the image for the first time, and calculate the difference between the first-type frame number of the first-aligned timestamp and the second-type frame number of the first-aligned image. According to the standard frame number difference between the two, the time stamp in the timestamp sequence and the image in the image sequence can be matched according to the standard frame number difference, and it can ensure that the time stamp and the image whose frame number difference is equal to the standard frame number difference are exactly matched. , and then assign the timestamp to the matching image, which can determine the high-precision exposure time of the image.

The method and the device are conceived based on the same application. Since the principles of the method and the device for solving the problem are similar, the implementation of the device and the method can be referred to each other, and repeated descriptions will not be repeated here.

An embodiment of the present application further provides an electronic device, as shown in FIG. 5 , including a processor 501 , a communication interface 502 , a memory 503 and a communication bus 504 , wherein the processor 501 , the communication interface 502 , and the memory 503 pass through the communication bus 504 complete communication with each other,

a memory 503 for storing computer programs;

When the processor 501 is used to execute the program stored in the memory 503, the following steps are implemented:

Get a sequence of timestamps for image exposure, and get the system time for each timestamp from the sequence of timestamps;

Acquire an image sequence and determine the system time of each frame in the image sequence;

Based on the predetermined image generation duration, the timestamps in the timestamp sequence and the images in the image sequence are aligned for the first time, wherein the difference between the system time of the timestamps aligned for the first time and the system time of the images aligned for the first time satisfies the image generation duration;

Calculate the standard frame number difference between the first-type frame number of the first-aligned timestamp and the second-type frame number of the first-aligned image;

Based on the standard frame number difference, the time stamp of each frame in the time stamp sequence is matched with each frame of the image in the image sequence, and the time stamp matching the image is determined as the exposure time stamp of the frame image.

The communication bus mentioned in the above electronic device may be a peripheral component interconnect standard (Peripheral Component Interconnect, PCI) bus or an Extended Industry Standard Architecture (Extended Industry Standard Architecture, EISA) bus or the like. The communication bus can be divided into an address bus, a data bus, a control bus, and the like. For ease of presentation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

The communication interface is used for communication between the above electronic device and other devices.

The memory may include random access memory (Random Access Memory, RAM), and may also include non-volatile memory (Non-Volatile Memory, NVM), such as at least one disk storage. According to one embodiment, the memory may also be at least one storage device located remotely from the aforementioned processor.

The above-mentioned processor can be a general-purpose processor, including a central processing unit (Central Processing Unit, CPU), a network processor (Network Processor, NP), etc.; it can also be a digital signal processor (Digital Signal Processing, DSP), dedicated integrated Circuit (Application Specific Integrated Circuit, ASIC), Field-Programmable Gate Array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

Applying the electronic device according to one or more embodiments of the present invention, firstly align the timestamp and the image for the first time, and calculate the difference between the first-type frame number of the first-aligned timestamp and the second-type frame number of the first-aligned image. Standard frame number difference, according to the standard frame number difference, the time stamp in the time stamp sequence and the image in the image sequence can be matched, and it can ensure that the time stamp and the image with the frame number difference equal to the standard frame number difference are exactly matched, and then Assigning timestamps to matching images enables highly accurate exposure times to be determined for the images.

In another embodiment provided by the present application, a computer-readable storage medium is also provided, and a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, any one of the above image time synchronization methods is implemented. step.

In yet another embodiment provided by the present application, there is also provided a computer program product including instructions, which, when running on a computer, causes the computer to execute the steps of any of the image time synchronization methods in the above-mentioned embodiments.

In the above-mentioned embodiments, it may be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented in software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present application are generated. The computer may be a general purpose computer, special purpose computer, computer network, or other programmable device. The computer instructions may be stored in or transmitted from one computer readable storage medium to another computer readable storage medium, for example, the computer instructions may be downloaded from a website site, computer, server or data center Transmission to another website site, computer, server, or data center is by wire (eg, coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (eg, infrared, wireless, microwave, etc.). The computer-readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. that includes an integration of one or more available media. The usable media may be magnetic media (eg, floppy disks, hard disks, magnetic tapes), optical media (eg, DVD), or semiconductor media (eg, Solid State Disk (SSD)), and the like.

It should be noted that, in this document, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any relationship between these entities or operations. any such actual relationship or sequence exists. Moreover, the terms "comprising", "comprising" or any other variation thereof are intended to encompass a non-exclusive inclusion such that a process, method, article or device that includes a list of elements includes not only those elements, but also includes not explicitly listed or other elements inherent to such a process, method, article or apparatus. Without further limitation, an element defined by the phrase "comprising a..." does not preclude the presence of additional identical elements in a process, method, article or apparatus that includes the element.

Each embodiment in this specification is described in a related manner, and the same and similar parts between the various embodiments may be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the embodiments of the image exposure time determination apparatus, electronic equipment, computer-readable storage medium and computer program product, since they are basically similar to the image exposure time determination method embodiments, the description is relatively simple. Part of the description of the embodiment of the exposure time determination method is sufficient.

One or more embodiments of the present application may have one or more of the following beneficial effects:

Applying the image time synchronization method, device, electronic device, and storage medium provided by one or more embodiments of the present application, firstly align the timestamp and the image for the first time, and calculate the first-type frame number of the first-aligned timestamp and the first-aligned frame number. The standard frame number difference between the second type of frame numbers of the image, according to the standard frame number difference, the time stamp in the time stamp sequence and the image in the image sequence can be matched, and the frame number difference can be guaranteed to be equal to the standard frame number difference. Timestamps and images are precisely matched, and by assigning timestamps to matching images, the exposure time of the images can be determined with high precision.

While the present invention has been described with respect to a limited number of embodiments, those skilled in the art having the benefit of this disclosure will appreciate that other embodiments can be devised without departing from the scope of the invention disclosed herein. Accordingly, the scope of the present invention should be limited only by the appended claims.

Claims

An image time synchronization method, comprising:

Obtain a timestamp sequence of image exposure, and obtain the system time of each frame timestamp from the timestamp sequence;

Acquire an image sequence, and determine the system time of each frame of image in the image sequence;

Based on a predetermined image generation duration, the timestamps in the timestamp sequence and the images in the image sequence are aligned for the first time, wherein the difference between the system time of the timestamps aligned for the first time and the system time of the images aligned for the first time satisfies all requirements the image generation time;

Calculate the standard frame number difference between the first type of frame number of the time stamp of the first alignment and the second type of frame number of the first aligned image;

Based on the standard frame number difference, each frame time stamp in the time stamp sequence is matched with each frame image in the image sequence, and the time stamp matched with the image is determined as the exposure time stamp of the frame image.
The method of claim 1, wherein the step of first aligning the time stamps in the time stamp sequence and the images in the image sequence based on a predetermined image generation duration comprises:

discarding the first several frames of data of the image sequence;

Calculate the system time difference between the system time of the queue head image of the image queue after discarding the first several frames of data and the system time of the queue head timestamp in the timestamp sequence;

Determine whether the system time difference satisfies the image generation duration:

If so, then it is determined that after discarding the first several frames of data, the head image and the head timestamp of the image queue are respectively the first aligned image and the first aligned timestamp;

If not, discard the head-of-line timestamp, and return the system time difference between the system time of the head-of-line image of the image queue after performing the calculation and discarding the first several frames of data and the system time of the head-of-line timestamp in the timestamp sequence step.
The method according to claim 1, wherein, based on the standard frame number difference, matching each frame time stamp in the time stamp sequence with each frame image in the image sequence, the matching image The steps of determining the time stamp as the exposure time stamp of the frame image include:

Calculate the frame number difference between the first type of frame number of the line head timestamp in the time stamp sequence and the second type of frame number of the line head image in the image sequence, and determine whether the frame number difference is equal to The standard frame number difference:

If so, determine the team head timestamp as the exposure timestamp of the team head image, update the team head timestamp of the timestamp sequence and the team head image of the image sequence, and return to calculate The step of the frame number difference between the first type frame number of the team head timestamp in the time stamp sequence and the second type frame number of the team head image in the image sequence;

If not, update the team head timestamp of the timestamp sequence or the team head image of the image sequence, and return the first type of frame number and the first type of frame number for calculating the team head timestamp in the timestamp sequence The step of the frame number difference between the second type of frame numbers of the head of line images in the sequence of images.
The method of claim 3, wherein the step of updating the head-of-line timestamp of the time-stamp sequence and the head-of-line image of the image sequence comprises:

discarding the team head timestamp in the timestamp sequence, and determining the next timestamp of the discarded team head timestamp as a new team head timestamp in the timestamp sequence;

The head image in the image sequence is discarded, and the next image of the discarded head image is determined as a new head image in the image sequence.
The method according to claim 3, wherein the step of updating the head-of-line timestamp of the time-stamp sequence or the head-of-line image of the image sequence comprises:

If the frame number difference is greater than the standard frame number difference, discard the head image in the image sequence, and determine the next image of the discarded head image as the new head image in the image sequence image;

If the frame number difference is smaller than the standard frame number difference, discard the line head timestamp in the time stamp sequence, and determine the next time stamp of the discarded line head time stamp as the time stamp sequence The new team leader timestamp.
The method according to claim 1, wherein the step of obtaining a timestamp sequence of image exposures comprises:

The time stamp sequence of image exposure is obtained from the serial port of the synchronizer, wherein each frame of time stamp represents the time when the pulse signal generated by the synchronizer triggers the exposure of the camera.
An image time synchronization device, comprising:

a first acquisition module, configured to acquire a timestamp sequence of image exposure, and acquire the system time of each frame timestamp from the timestamp sequence;

a second acquisition module, configured to acquire an image sequence, and determine the system time of each frame of image in the image sequence;

an alignment module for aligning the timestamps in the timestamp sequence and the images in the image sequence for the first time based on a predetermined image generation duration, wherein the system time of the timestamps aligned for the first time and the system time of the aligned images for the first time The time difference satisfies the image generation duration;

A calculation module for calculating the standard frame number difference between the first type frame number of the time stamp of the first alignment and the second type frame number of the image aligned for the first time;

The matching module is configured to match each frame time stamp in the time stamp sequence with each frame image in the image sequence based on the standard frame number difference, and determine the time stamp matched with the image as the frame of the image. Exposure timestamp.
The apparatus of claim 7, wherein the alignment module is specifically configured to:

discarding the first several frames of data of the image sequence;

Calculate the system time difference between the system time of the queue head image of the image queue after discarding the first several frames of data and the system time of the queue head timestamp in the timestamp sequence;

Determine whether the system time difference satisfies the image generation duration:

If so, then it is determined that after discarding the first several frames of data, the head image and the head timestamp of the image queue are respectively the first aligned image and the first aligned timestamp;

If not, discard the head-of-line timestamp, and return the system time difference between the system time of the head-of-line image of the image queue after performing the calculation and discarding the first several frames of data and the system time of the head-of-line timestamp in the timestamp sequence step.
The device according to claim 7, wherein the matching module is specifically used for:

Calculate the frame number difference between the first type of frame number of the line head timestamp in the time stamp sequence and the second type of frame number of the line head image in the image sequence, and determine whether the frame number difference is equal to The standard frame number difference:

If so, determine the team head timestamp as the exposure timestamp of the team head image, update the team head timestamp of the timestamp sequence and the team head image of the image sequence, and return to calculate The step of the frame number difference between the first type frame number of the team head timestamp in the time stamp sequence and the second type frame number of the team head image in the image sequence;

If not, update the team head timestamp of the timestamp sequence or the team head image of the image sequence, and return the first type of frame number and the first type of frame number for calculating the team head timestamp in the timestamp sequence The step of the frame number difference between the second type of frame numbers of the head of line images in the sequence of images.
The apparatus according to claim 9, wherein, when performing the updating of the team head timestamp of the timestamp sequence and the team head image of the image sequence, the matching module is specifically configured to:

discarding the team head timestamp in the timestamp sequence, and determining the next timestamp of the discarded team head timestamp as a new team head timestamp in the timestamp sequence;

The head image in the image sequence is discarded, and the next image of the discarded head image is determined as a new head image in the image sequence.
The apparatus according to claim 9, wherein, when performing the updating of the team head timestamp of the timestamp sequence or the team head image of the image sequence, the matching module is specifically configured to:

If the frame number difference is greater than the standard frame number difference, discard the head image in the image sequence, and determine the next image of the discarded head image as the new head image in the image sequence image;

If the frame number difference is smaller than the standard frame number difference, discard the line head timestamp in the time stamp sequence, and determine the next time stamp of the discarded line head time stamp as the time stamp sequence The new team leader timestamp.
The apparatus according to claim 7, wherein, when executing the time stamp sequence for obtaining image exposure, the first obtaining module is specifically configured to:

The time stamp sequence of image exposure is obtained from the serial port of the synchronizer, wherein each frame of time stamp represents the time when the pulse signal generated by the synchronizer triggers the exposure of the camera.
An electronic device comprising:

Communication Interface;

memory for storing computer programs;

communication bus;

The processor, when executing the program stored on the memory, realizes the method according to any one of claims 1-6,

Wherein, the processor, the communication interface, and the memory communicate with each other through the communication bus.
A computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, the method according to any one of claims 1-6 is implemented.