WO2024011408A1

WO2024011408A1 - Method and apparatus for synchronously collecting data, synchronization determination method and apparatus, and autonomous vehicle

Info

Publication number: WO2024011408A1
Application number: PCT/CN2022/105187
Authority: WO
Inventors: 李贤飞; 黄自瑞; 张满江
Original assignee: 阿波罗智能技术(北京)有限公司; 百度（美国）有限责任公司
Priority date: 2022-07-12
Filing date: 2022-07-12
Publication date: 2024-01-18

Abstract

A method (200, 400, 600) and apparatus (1300, 1400, 1500) for synchronously collecting data, a synchronization determination method (800) and apparatus (1600), a vehicle (110, 1100), and an electronic device (1700), which relate to the field of artificial intelligence, and in particular to the technical field of autonomous driving, computer vision and cloud computing. The specific implementation solution of the method (200) for synchronously collecting data is: in response to receiving a data packet for a predetermined angle (302) from a radar sensor (111, 601), determining first time information (301) of when the radar sensor (111, 601) collects point cloud data at the predetermined angle (302) (S210), wherein the predetermined angle (302) is within an angle-of-view range of an image sensor (112, 602); according to the first time information (301), determining delay information (306) for the image sensor (112, 602) (S220); and sending the delay information (306) to a controller (1000) (S230), such that the controller (1000) controls the image sensor (112, 602) to synchronously collect data with the radar sensor (111, 601) which is rotated to the predetermined angle (302), wherein the data packet comprises the point cloud data collected by the radar sensor (111, 601) at the predetermined angle (302).

Description

Synchronous data collection method and synchronization determination method, device, and autonomous vehicle

Technical field

The present disclosure relates to the field of artificial intelligence, specifically to technical fields such as autonomous driving, computer vision, and cloud computing, and in particular to a method of synchronously collecting data, a synchronization determination method, a device, a vehicle-mounted terminal, a controller, a vehicle, a cloud system, and an electronic device and readable storage media.

Background technique

With the development of computer technology and network technology, autonomous driving technology has developed rapidly. In autonomous driving technology, it is usually necessary to rely on image data collected by image sensors and point cloud data collected by radar sensors to perceive the environmental information around the vehicle, and determine the autonomous driving strategy based on the sensing results.

Contents of the invention

The present disclosure aims to provide a method for synchronously collecting data, a synchronization determination method, a device, a vehicle-mounted terminal, a controller, a vehicle, a cloud system, an electronic device and a readable storage medium that are conducive to improving data alignment accuracy.

According to an aspect of the present disclosure, a method for synchronously collecting data is provided, including: in response to receiving a data packet from a radar sensor for collection at a predetermined angle, determining first time information when the radar sensor collects point cloud data at a predetermined angle. ; wherein the predetermined angle is within the viewing angle range of the image sensor; determining delay information for the image sensor according to the first time information; and sending the delay information to the controller so that the controller controls the image sensor to synchronize with the radar sensor rotated to the predetermined angle The data is collected on the ground, where the data package includes point cloud data collected by the radar sensor at a predetermined angle.

According to another aspect of the present disclosure, a method for synchronously collecting data is provided, including: in response to receiving delay information for the image sensor, adjusting the trigger moment of the image sensor according to the delay information; in response to reaching the trigger moment, sending a signal to the image sensor sending a trigger signal; and in response to receiving the image data collected by the image sensor, adding time information to the image data so as to utilize the time information to align the image data with the point cloud data collected by the radar sensor, wherein the delay information is predetermined based on the radar sensor The angle is determined by the time information of collecting point cloud data, and the predetermined angle is within the viewing angle range of the image sensor.

According to another aspect of the present disclosure, a method for synchronously collecting data is provided, including: in response to receiving a data packet from a radar sensor for a predetermined angle, determining first time information when the radar sensor collects point cloud data at a predetermined angle. ; Wherein, the predetermined angle is within the viewing angle range of the image sensor; according to the first time information, delay information for the image sensor is determined; and the triggering time of the image sensor is adjusted according to the delay information, so that the image sensor and the radar sensor rotate to the predetermined angle are synchronized. Collecting data; in response to reaching the trigger moment, sending a trigger signal to the image sensor; and in response to receiving the image data collected by the image sensor, adding second time information to the image data to utilize the second time information to combine the image data with the radar sensor collection Point cloud data alignment.

According to another aspect of the present disclosure, a synchronization determination method is provided, including: acquiring a data packet sequence collected by a radar sensor and an image data sequence collected by an image sensor; for each data packet in the data packet sequence, according to each data The time information of the packet and the time information of each image data in the image data sequence are used to determine the image data matching each data packet; based on the difference and difference threshold in the time information between each data packet and the matching image data, each data packet is determined. the synchronization relationship between each data packet and the matching image data; and determining the synchronization relationship between the radar sensor and the image sensor based on the synchronization relationship between each of the multiple data packets in the data packet sequence and the matching image data.

According to another aspect of the present disclosure, a vehicle-mounted terminal is provided, the vehicle-mounted terminal being configured to: in response to receiving a data packet from a radar sensor for a predetermined angle, determine the first time the radar sensor collects point cloud data at a predetermined angle. time information; wherein the predetermined angle is within the viewing angle range of the image sensor; determining delay information for the image sensor according to the first time information; and sending the delay information to the controller so that the controller controls the image sensor and the radar that rotates to the predetermined angle The sensors collect data synchronously, where the data packets include point cloud data collected by the radar sensor at a predetermined angle.

According to another aspect of the present disclosure, a controller is provided, configured to: in response to receiving delay information for the image sensor, adjust a trigger moment of the image sensor according to the delay information; in response to reaching the trigger moment, send a signal to the image sensor a trigger signal; and in response to receiving the image data collected by the image sensor, adding time information to the image data so as to utilize the time information to align the image data with the point cloud data collected by the radar sensor, wherein the delay information is at a predetermined angle according to the radar sensor The time information for collecting point cloud data is determined, and the predetermined angle is within the viewing angle range of the image sensor.

According to another aspect of the present disclosure, an autonomous vehicle is provided, including a vehicle-mounted terminal, a controller, a radar sensor, and an image sensor, wherein the vehicle-mounted terminal is configured to: respond to receiving data for a predetermined angle from the radar sensor The package determines the first time information that the radar sensor collects point cloud data at a predetermined angle; wherein the predetermined angle is within the viewing angle range of the image sensor; the data package includes the point cloud data collected by the radar sensor at a predetermined angle; according to the first time information, Determine delay information for the image sensor; and send the delay information to the controller; wherein the controller is configured to: in response to receiving the delay information, adjust the trigger moment of the image sensor according to the delay information; in response to reaching the trigger moment, send the delay information to the image sensor sending a trigger signal; and in response to receiving the image data collected by the image sensor, adding second time information to the image data so as to utilize the second time information to align the image data with the point cloud data collected by the radar sensor.

According to another aspect of the present disclosure, a cloud system is provided, configured to: obtain a data packet sequence collected by a radar sensor and an image data sequence collected by an image sensor; for each data packet in the data packet sequence, according to each The time information of the data packet and the time information of each image data in the image data sequence are used to determine the image data matching each data packet; based on the difference in time information and the difference threshold between each data packet and the matching image data, the image data is determined. the synchronization relationship between each data packet and the matching image data; and determining the synchronization relationship between the radar sensor and the image sensor based on the synchronization relationship between each of multiple data packets in the data packet sequence and the matching image data.

According to another aspect of the present disclosure, a device for synchronously collecting data is provided. The device includes: a time information determination module, configured to determine whether the radar sensor is at a predetermined angle in response to receiving a data packet from a radar sensor for a predetermined angle. Collect first time information of point cloud data; wherein the predetermined angle is within the viewing angle range of the image sensor; a delay information determination module for determining delay information for the image sensor based on the first time information; and an information sending module for Delay information is sent to the controller so that the controller controls the image sensor to collect data synchronously with the radar sensor rotated to a predetermined angle, wherein the data packet includes point cloud data collected by the radar sensor at the predetermined angle.

According to another aspect of the present disclosure, a device for synchronously collecting data is provided, including: a time adjustment module, configured to adjust the triggering time of the image sensor according to the delay information in response to receiving delay information for the image sensor; a signal sending module , used to send a trigger signal to the image sensor in response to reaching the trigger moment; and a time adding module, used to add time information to the image data in response to receiving the image data collected by the image sensor, so as to use the time information to combine the image data with the radar The point cloud data collected by the sensor are aligned, wherein the delay information is determined based on the time information when the radar sensor collects the point cloud data at a predetermined angle, and the predetermined angle is within the viewing angle range of the image sensor.

According to another aspect of the present disclosure, a device for synchronously collecting data is provided, including: a time information determination module, configured to determine that the radar sensor collects points at a predetermined angle in response to receiving a data packet from a radar sensor for a predetermined angle. The first time information of the cloud data; wherein the predetermined angle is within the viewing angle range of the image sensor; the delay information determination module is used to determine the delay information for the image sensor based on the first time information; the time adjustment module is used to determine the delay information based on the delay The information adjusts the trigger moment of the image sensor so that the image sensor collects data synchronously with the radar sensor rotated to a predetermined angle; a signal sending module for sending a trigger signal to the image sensor in response to reaching the trigger moment; and a time adding module for In response to receiving the image data collected by the image sensor, second time information is added to the image data to align the image data with the point cloud data collected by the radar sensor using the second time information.

According to another aspect of the present disclosure, a synchronization determination device is provided, including: a data acquisition module for acquiring a data packet sequence collected by a radar sensor and an image data sequence collected by an image sensor; a data matching module for targeting data packets For each data packet in the sequence, the image data matching each data packet is determined based on the time information of each data packet and the time information of each image data in the image data sequence; the data relationship determination module is used to determine the image data matching each data packet according to the time information of each data packet. The difference in time information between the packet and the matching image data and the difference threshold determine the synchronization relationship between each data packet and the matching image data; and a synchronization relationship determination module is used to determine the synchronization relationship between multiple data packets in the data packet sequence. The synchronization relationship between each and the matching image data determines the synchronization relationship between the radar sensor and the image sensor.

According to another aspect of the present disclosure, an electronic device is provided, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are At least one processor executes, so that at least one processor can execute the method for synchronously collecting data or the method for synchronization determination provided by the present disclosure.

According to another aspect of the present disclosure, there is provided a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause the computer to execute the method of synchronously collecting data or the method of synchronization determination provided by the present disclosure.

According to another aspect of the present disclosure, a computer program product is provided, including a computer program/instruction that, when executed by a processor, implements the method for synchronously collecting data or the method for synchronization determination provided by the present disclosure.

It should be understood that what is described in this section is not intended to identify key or important features of the embodiments of the disclosure, nor is it intended to limit the scope of the disclosure. Other features of the present disclosure will become readily understood from the following description.

Description of drawings

The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure. in:

Figure 1 is a schematic diagram of an application scenario of a method for synchronously collecting data, a method for determining synchronization, and a device according to an embodiment of the present disclosure;

Figure 2 is a schematic flowchart of a method for synchronously collecting data according to an embodiment of the present disclosure;

Figure 3 is a schematic diagram of the principle of determining delay information for an image sensor according to an embodiment of the present disclosure;

Figure 4 is a schematic flowchart of a method for synchronously collecting data according to another embodiment of the present disclosure;

Figure 5 is a schematic flowchart of adjusting the triggering time of an image sensor according to an embodiment of the present disclosure;

Figure 6 is a schematic diagram of the principle of a method for synchronously collecting data according to an embodiment of the present disclosure;

Figure 7 is a schematic diagram of the principle of synchronous data collection according to an embodiment of the present disclosure;

Figure 8 is a schematic flowchart of a synchronization determination method according to an embodiment of the present disclosure;

Figure 9 is a schematic diagram of a vehicle-mounted terminal according to an embodiment of the present disclosure;

Figure 10 is a schematic diagram of a controller according to an embodiment of the present disclosure;

Figure 11 is a schematic diagram of an autonomous vehicle according to an embodiment of the present disclosure;

Figure 12 is a schematic diagram of a cloud system according to an embodiment of the present disclosure;

Figure 13 is a structural block diagram of a device for synchronously collecting data according to an embodiment of the present disclosure;

Figure 14 is a structural block diagram of a device for synchronously collecting data according to another embodiment of the present disclosure;

Figure 15 is a structural block diagram of a device for synchronously collecting data according to another embodiment of the present disclosure;

Figure 16 is a structural block diagram of a synchronization determination device according to an embodiment of the present disclosure; and

FIG. 17 is a block diagram of an electronic device used to implement the method of synchronously collecting data or the method of synchronization determination according to an embodiment of the present disclosure.

Detailed ways

Exemplary embodiments of the present disclosure are described below with reference to the accompanying drawings, in which various details of the embodiments of the present disclosure are included to facilitate understanding and should be considered to be exemplary only. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the disclosure. Also, descriptions of well-known functions and constructions are omitted from the following description for clarity and conciseness.

During the process of autonomous driving, autonomous vehicles need to sense rich environmental information around them and take timely safety measures when there are safety hazards in the surrounding environment. Among them, in order to sense the rich surrounding environmental information, autonomous vehicles are usually equipped with radar sensors and image sensors. Image sensors are used to collect image data to obtain rich texture and color information in the environment, and radar sensors are used to collect point cloud data to obtain distance information of objects in the environment. During the autonomous driving process, reconstructed three-dimensional environmental information can be obtained by fusing image data and point cloud data.

If the collection time of image data and point cloud data is not synchronized, the environmental information obtained by fusing the image data and point cloud data will deviate from the actual environmental information. This deviation will be amplified as the speed of the autonomous vehicle increases, which will affect the safe driving of the autonomous vehicle.

Based on this, the present disclosure provides a method and device for synchronous data collection that improves the collection synchronization of image data and point cloud data, and also provides a synchronization determination that efficiently and accurately evaluates the synchronization of data collection. Methods and apparatus.

The application scenarios of the method and device provided by the present disclosure will be described below with reference to FIG. 1 .

Figure 1 is a schematic diagram of application scenarios of a method for synchronously collecting data, a method for determining synchronization, and a device according to an embodiment of the present disclosure.

As shown in FIG. 1 , the application scenario 100 of this embodiment may include an autonomous vehicle 110 , a road 120 and a cloud system 130 .

Among them, the autonomous vehicle 110 may be integrated with a vehicle-mounted terminal, a radar sensor 111 and an image sensor 112. The vehicle-mounted terminal can be connected to the radar sensor 111 and the image sensor 112 through communication cables. While the autonomous vehicle 110 is driving on the road 120, the vehicle-mounted terminal can obtain the point cloud data collected by the radar sensor 111 and the image data collected by the image sensor 112 through the communication cable. For example, the vehicle-mounted terminal can fuse image data and point cloud data in real time, decide the driving strategy of the vehicle based on the fusion results, and send control signals to the power system of the autonomous vehicle 110 according to the driving strategy to achieve autonomous driving.

The radar sensor 111 may be, for example, a laser radar, a millimeter wave radar, or other sensor with a mechanical rotation function. The radar sensor 111 may be, for example, installed on the roof of an autonomous vehicle. For example, the radar sensor 111 can rotate 360° under the control of pulse signals to collect all-round point cloud data around the autonomous vehicle. Taking lidar as an example, the radar sensor can obtain the position point information of objects in the environment around the autonomous vehicle by emitting and receiving laser beams, and performs three-dimensional modeling based on the position point information to obtain point cloud data.

The image sensor 112 may be, for example, any one of the following cameras: a front-view camera installed on the self-driving vehicle 110 , a surround-view camera installed in any direction of the self-driving vehicle 110 , a rear-view camera, a side-view camera, etc. .

In one embodiment, the autonomous vehicle 110 may also be integrated with a positioning device. The positioning device may be composed of a Global Positioning System (GPS) and a Geographic Information System (Geographic Information System, GIS). Realize tracking and positioning of autonomous vehicles. In this embodiment, the vehicle-mounted terminal can, for example, use the positioning device time of the positioning device as the reference time to control the timing of data collection by the radar sensor and the image sensor, so that the radar sensor and the image sensor collect data synchronously.

In one embodiment, the vehicle-mounted terminal may, for example, control the triggering of the image acquisition device based on the time information of the radar sensor collecting point cloud data, so that the radar sensor and the image sensor collect data synchronously.

In one embodiment, the autonomous vehicle 110 may also be integrated with a controller that controls the triggering of the image sensor 112 , and the controller may be an artificial intelligence chip, for example. In this embodiment, the vehicle-mounted terminal can be communicatively connected with the controller to send delay information determined based on the time information of the radar sensor collecting point cloud data to the controller, so that the controller controls the triggering of the image sensor based on the delay information.

As shown in Figure 1, in this application scenario 100, the vehicle-mounted terminal can also communicate with the cloud system 130 through a wireless communication link. The cloud system 130 can monitor the driving of the autonomous vehicle based on the data uploaded by the vehicle terminal. For example, the cloud system 130 can obtain the point cloud data and image data collected by the autonomous vehicle within a predetermined period from the vehicle terminal, and collect the point cloud data and images of the autonomous vehicle based on the fusion result of the point cloud data and image data. The synchronicity of the data is evaluated to provide a reference for adjusting the synchronous collection strategy of point cloud data and image data.

It should be noted that the method for synchronously collecting data provided by the present disclosure can be executed by the autonomous vehicle 110 . Specifically, some operations can be executed by the vehicle-mounted terminal in the autonomous vehicle 110 and some operations can be executed by the controller. Accordingly, the device for synchronously collecting data provided by the present disclosure can be provided in the autonomous vehicle 110. Specifically, some modules can be provided in the vehicle-mounted terminal and some modules can be provided in the controller. The synchronization determination method provided by the present disclosure can be executed by the cloud system 130 . Correspondingly, the synchronization determining device provided by the present disclosure can be provided in the cloud system 130 .

It should be understood that the structures and types of the autonomous vehicle 110, radar sensor 111, image sensor 112 and cloud system 130 in Figure 1 are only schematic. Depending on implementation requirements, the autonomous vehicle 110, radar sensor 111, image sensor 112 and cloud system 130 may have any structure and type.

The method for synchronously collecting data provided by the present disclosure will be described in detail below with reference to FIGS. 2 to 6 .

Figure 2 is a schematic flowchart of a method for synchronously collecting data according to an embodiment of the present disclosure.

As shown in FIG. 2 , the method 200 for synchronously collecting data in this embodiment may include operations S210 to S230. The method 200 may be executed, for example, by a vehicle-mounted terminal in an autonomous vehicle.

In operation S210, in response to receiving a data packet for a predetermined angle from the radar sensor, first time information at which the radar sensor collects point cloud data at a predetermined angle is determined.

In operation S220, delay information for the image sensor is determined according to the first time information.

In operation S230, delay information is sent to the controller.

According to embodiments of the present disclosure, the vehicle-mounted terminal may receive a data packet sent by a radar sensor, where the data packet includes point cloud data collected by the radar sensor. The vehicle-mounted terminal can use the timestamp of the data packet as the time information for the radar sensor to collect point cloud data. Among them, the timestamp can be added by the radar sensor when encapsulating the collected point cloud data to obtain the data package.

In one embodiment, the radar sensor may send data packets using UDP/IP protocol. The data packet includes Ethernet Header and User Data Packet Protocol Data (UDP Data). Among them, the user data packet protocol data may include ranging data and additional information. Among them, the ranging data consists of multiple data blocks, and each data block includes an azimuth angle value, a distance value, etc. Additional information may include motor speed and time data that drives the radar sensor to rotate.

In this embodiment, the vehicle-mounted terminal can parse the received data packet and determine whether a data packet for a predetermined angle is received based on the parsing result. For example, a data packet whose azimuth angle value is a predetermined angle is a data packet for the predetermined angle. This embodiment can obtain the first time information of the radar sensor collecting point cloud data at a predetermined angle based on the time data in the data packet for the predetermined angle. The predetermined angle is an angle within the viewing angle range of the image sensor. For example, the predetermined angle can be determined based on the angle between the main optical axis of the image sensor and the horizontal plane. For example, the predetermined angle can be the value of the included angle. The angle at which the radar sensor collects point cloud data can be represented by the angle between the laser beam emitted by the radar sensor when collecting point cloud data and the horizontal plane.

For example, the time data in the data packet can include standard time (such as Universal Time Coordinated, UTC) and the periodic encapsulation time of the data packet. Among them, the value range of the periodic encapsulation time of the data packet can be [0μs, 1s]. Standard time can represent the year, month, day, hour, minute and second of the collected data. In this embodiment, the standard time and the periodic encapsulation time can be added to obtain the time information of the ranging data in the collected data packet. In this embodiment, the data packet whose azimuth angle value is a predetermined angle can be used as the target data packet, and the time information for collecting the ranging data in the target data packet can be used as the time information determined in operation S210. It can be understood that the combination of ranging data and azimuth angle values can represent point cloud data.

According to an embodiment of the present disclosure, while obtaining the first time information T of point cloud data collected at a predetermined angle, this embodiment can also calculate the radar sensor based on the time data in the data packet with an azimuth angle value of 0°. Time information for collecting point cloud data at a predetermined angle. For example, the time information determined based on the time data in the data packet with an azimuth angle value of 0° is T ₀ , the predetermined angle is angle, and the rotation period of the radar sensor is LIDAR_ROT_INTERVAL. The following formula (1) can be used to calculate the radar sensor to the predetermined angle. Time information T' for collecting point cloud data at angle:

T'=T ₀ +angle*LIDAR_ROT_INTERVAL/360. Formula 1)

The delay information for the image sensor can be determined based on the difference between the time information T and the time information T'.

In one embodiment, after obtaining the difference between the time information T and the time information T', the difference can also be used to take the remainder of the time interval at which the image sensor collects images, and the remainder can be used as the delay information. In this way, the value of the delay information can be limited to the time interval during which the image sensor collects images. When controlling the acquisition time of the image sensor acquisition according to the delay information, the adjustment amount of the acquisition time can be reduced. The time interval at which the image sensor collects images may, for example, be positively correlated according to the reciprocal of the acquisition frame rate of the image sensor. The unit of the acquisition frame rate may be, for example, fps, that is, the acquisition frame rate is the number of image data collected by the image sensor per second. The reason why this embodiment can limit the value of the delay information to the time interval during which the image sensor collects images is because the image sensor periodically collects image data according to the collection frame rate, and the synchronization of data collection only needs to ensure that the image sensor collects The image data only needs to include data collected simultaneously with the point cloud data collected by the radar sensor turned to a predetermined angle.

After obtaining the delay information, the delay information can be sent to the controller that controls the triggering timing of the image sensor, so that the triggering timing of the image sensor is controlled by the controller, that is, the acquisition time of the image sensor is controlled, so that the image sensor can communicate with the image sensor. Radar sensors rotated to a predetermined angle collect data synchronously. That is, at the moment when the radar sensor rotated to a predetermined angle collects point cloud data, the image sensor collects image data synchronously.

This embodiment determines the delay information based on the time information when the radar sensor collects point cloud data, and sends the delay information to the controller, so that the time when the image sensor collects image data is consistent with the time when the radar sensor collects point clouds within the viewing angle range of the image sensor. The time alignment of data enables the collected image data and point cloud data to express the environmental information at the same time, which can improve the alignment accuracy of the collected image data and point cloud data, help autonomous vehicles make correct driving strategies, and improve autonomous driving. Vehicle driving safety.

It can be understood that, compared with the technical solution in the prior art in which the image sensor and the radar sensor only collect data based on the same reference time, the technical solution of this embodiment can improve the synchronization of the data collected by the two sensors and ensure that the image sensor The acquired image data includes data aligned with the point cloud data collected by the radar sensor.

It can be understood that the collection angle data and time data in the data packet can be obtained by parsing the data packet from the radar sensor. For example, for the Pandar 40P model lidar, parse the data packet from the lidar, and use the value of the analyzed Azlmuth Angle parameter as the acquisition angle data, that is, as the azimuth angle value. The parsed UTC parameter value and the GPS Timestamp parameter value can be used as time data. This embodiment can use any parsing tool that parses UDP packets to parse the data packets from the radar sensor. It can be understood that the above-mentioned radar sensor models and analysis tools are only used as examples to facilitate understanding of the present disclosure, and the present disclosure does not limit this. For example, for different types of radar sensors, the values of the corresponding parameters obtained by analysis can be used as the collection angle data and time data. For data packets transmitted using communication protocols other than UDP, corresponding parsing tools can be used to parse the data packets.

In one embodiment, the reference time described above may be aligned with the time of the positioning device of the positioning device in the vehicle where the radar sensor is located, for example. Specifically, when the vehicle is started, the positioning device can send a recommended positioning information (Recommended Minimum Specific GPS/TRANSIT Data, GPRMC) data packet to the radar sensor. The radar sensor can use the UTC time in the GPRMC packet as the initial time. Subsequently, during the process of collecting point cloud data by the radar sensor, the sum of the time difference between the time when the point cloud data is collected and the time when the GPRMC data packet is received and the initial time can be used as the reference time. Among them, the UTC time in the GPRMC data packet is usually in seconds. In this way, the data packets of point cloud data collected by the radar sensor can have high-precision timestamps, which is conducive to achieving high-precision control of the image sensor.

FIG. 3 is a schematic diagram of the principle of determining delay information for an image sensor according to an embodiment of the present disclosure.

According to embodiments of the present disclosure, when determining the delay information, for example, the time information of the radar sensor when the rotation angle is 0° can be calculated based on the time information of point cloud data collected at a predetermined angle. And based on the calculated time information when the rotation angle is 0° and the rotation period of the radar sensor, the rotation deviation value of the radar sensor is determined. Delay information is then determined based on this rotational deviation value.

Specifically, as shown in Figure 3, when determining the delay information in this embodiment 300, the first time information 301 of the point cloud data collected by the radar sensor at a predetermined angle, the predetermined angle 302 and the rotation period 303 of the radar sensor can be calculated. The second time information 304 of the radar sensor collecting point cloud data at 0° is obtained. For example, set the first time information as T, the predetermined angle as angle, and the rotation period of the radar sensor as LIDAR_ROT_INTERVAL, then the second time information T ₀ ' can be calculated using the following formula (2):

T ₀ '=T-angle*LIDAR_ROT_INTERVAL/360. Formula (2)

In one embodiment, the minimum angle resolution of the radar sensor is set to 0.01°, and the azimuth angle Azlmuth Angle in the data packet is measured in units of 0.01°, then the predetermined angle in this embodiment should also be measured in units of 0.01°. Measured in units of 0.01°. In this case, 360 in formula (2) should be replaced by 36000.

In this embodiment, after obtaining the second time information 304, the second time information 304 can be modulated by the rotation period 303, and the rotation deviation value 305 of the radar sensor can be determined based on the remainder. This is because the initial angle of the radar sensor is 0°, and the starting time for the radar sensor to collect point cloud data is usually in seconds, and the rotation period of the radar sensor is usually in the order of ms. If there is no deviation in the rotation of the radar sensor, the time information of the radar sensor collecting point cloud data at an angle of 0° should be an integer multiple of the rotation period. Therefore, the remainder obtained by taking the remainder can be used to represent the rotational deviation value. It can be understood that the difference between T and T' mentioned above can be understood as the rotation deviation value determined in this embodiment.

After obtaining the rotational deviation value 305, this embodiment may use the rotational deviation value 305 as the delay information of the image sensor. A method similar to the method described above can also be used, in which the rotation deviation value 305 is modulated by the time interval at which the image sensor collects images, and the remainder value is used as the delay information.

In one embodiment, after the rotation deviation value 305 is obtained, the delay information may also be determined based on the sum of the rotation deviation value 305 and the rotation period 303, for example. In this way, it can be avoided that when the first time information T is small, T ₀ ′ obtained by the above formula (2) is a negative number and cannot be modulated by the time interval during which the image sensor collects images. For example, in this embodiment, the sum of the rotation deviation value 305 and the rotation period 303 may be modulated by the time interval for image sensor acquisition, and the remainder may be used as the delay information 306 of the image sensor.

In an embodiment, when determining the delay information, a predetermined error value 307 determined based on the optical center position of the image sensor and the position of the target object within the viewing angle range of the image sensor may also be considered. The target object may be the ground, for example, and may be determined based on the angle between a point on the central axis of the vehicle at a predetermined distance from the vehicle and a straight line where the optical center of the image sensor is located and the main optical axis of the image sensor. The predetermined error value is a value greater than 0. The predetermined error value 307 may be an empirical value. For example, the predetermined error value 307 may be 5 ms.

For example, this embodiment may determine the delay information of the image sensor based on the sum of the predetermined error value 307 and the rotational deviation value 305 . In this way, the accuracy of the determined delay information can be improved. This is because in the image data collected by the image sensor, the data that can reflect the objects on the road that affect the driving of the vehicle are the lower image pixels. Therefore, when aligning data, it is usually preferable to focus on the lower image pixels in the image data collected by the image sensor. The angle at which the radar sensor can collect objects on the road usually deviates from the predetermined angle, and this deviation will cause a deviation in determining the acquisition time of the image sensor. This embodiment can compensate for this deviation by setting a predetermined error value 307. In this way, controlling the time at which the image sensor collects image data based on the delay information determined in this embodiment can enable the image sensor and the radar sensor to synchronously collect objects on the road, thereby improving the synchronization accuracy of data collection.

It can be understood that, in order to avoid the occurrence of negative numbers, this embodiment may also determine the delay information of the image sensor based on the sum of the predetermined error value 307, the rotation deviation value 305 and the rotation period 303.

In one embodiment, the controller that controls the image sensor can be, for example, an artificial intelligence chip, such as a field programmable gate array chip (Field Programmable Gate Array, FPGA), which is suitable for processing data based on delay information at a high sampling frequency. The image sensor with high sampling frequency adjusts the acquisition time. It can be applied to scenarios where data is collected synchronously in real time. Accordingly, the synchronous data collection of this embodiment can be performed periodically.

After the controller receives the delay information, for example, the triggering time of the image sensor can be adjusted based on the delay information, so that the image sensor collects data synchronously with the radar sensor that rotates to a predetermined angle. The method of synchronously collecting data executed by the controller will be described in detail below with reference to Figures 4 to 5 .

Figure 4 is a schematic flowchart of a method for synchronously collecting data according to an embodiment of the present disclosure.

As shown in Figure 4, the method 400 of synchronously collecting data in this embodiment may include operations S410 to S430. The method 400 can be executed by a controller, for example, it can be executed by an artificial intelligence chip (such as FPGA).

In operation S410, in response to receiving the delay information for the image sensor, the triggering timing of the image sensor is adjusted according to the delay information.

In operation S420, in response to reaching the trigger moment, a trigger signal is sent to the image sensor.

In operation S430, in response to receiving the image data collected by the image sensor, time information is added to the image data.

According to embodiments of the present disclosure, the system time of the controller may, for example, be aligned with the positioning device time of the positioning device in the vehicle where the controller is located. For example, when the vehicle is started, the positioning device can send a GPRMC data packet to the controller. The controller can use the UTC time in the GPRMC packet as the initial time. For example, the controller can control the image sensor to collect image data at an acquisition frame rate of 30fps starting from the initial time.

The delay information is sent by the terminal device to the controller through operation S230 described above. After receiving the delay information, the controller can adjust the time at which the image sensor collects image data after the current time according to the delay information. For example, the adjusted trigger time can be obtained by adding the trigger time after the current time and the delay duration indicated by the delay information.

Alternatively, when adjusting the triggering moment, the time interval between two adjacent triggering moments of the image sensor can be considered. For example, this embodiment can use the remainder obtained by taking the delay length indicated by the delay information as the remainder of the time interval as the adjustment amount, and add the trigger time after the current time to the adjustment amount to obtain the adjusted trigger time. Through this method, the degree of adjustment of the trigger time can be reduced while ensuring synchronous data collection, and frame loss caused by adjusting the trigger time can be avoided.

After adjusting the trigger time, the controller can control the trigger of the image sensor according to the adjusted trigger time, so that the image sensor collects data synchronously with the radar sensor rotated to a predetermined angle. It can be understood that the trigger signal sent to the image sensor is any type of trigger signal that the image sensor can recognize and receive, and the present disclosure does not limit this. The image sensor is triggered in response to the trigger signal and begins collecting image data. After collecting the image data, the image sensor can send the collected image data to the controller. After receiving the image data, the controller can add time information to the image data according to the time information of the current moment.

In this way, the point cloud data and image data collected by the radar sensor in the present disclosure all have time information. The vehicle-mounted terminal can timely align the two data based on the time information of the two data, and fuse the two aligned data to reconstruct the environmental information.

It can be understood that since the radar sensor rotates periodically, the data packets for the predetermined angle and the determined delay information received by the vehicle-mounted controller are both periodic. In the embodiment of the present disclosure, the controller rotates periodically according to the periodicity. The delay information is used to adjust the triggering moment of the image sensor, which can avoid inaccurate data alignment caused by the rotation deviation of the radar sensor during vehicle operation. This will help improve the safety of autonomous vehicle driving.

FIG. 5 is a schematic flowchart of adjusting the triggering time of an image sensor according to an embodiment of the present disclosure.

According to embodiments of the present disclosure, when adjusting the triggering time of the image sensor, if the delay time indicated by the delay information is long, the triggering time can be adjusted step by step to avoid causing the image sensor to be damaged due to a large degree of single adjustment. The collected images lose frames relative to the radar sensor, affecting the autonomous driving of the vehicle.

In one embodiment, the number of times to adjust the trigger moment and the step size for each adjustment may be determined based on the acquisition frame rate and delay duration of the image sensor. Then adjust the triggering time of the image sensor according to the number of times and step size.

For example, the time interval for the image sensor to collect image data can be first determined based on the collection frame rate, and the time interval can be the reciprocal of the collection frame rate. Then, divide the delay duration by the time interval, round up, and use the rounded value as the number of times to adjust the trigger moment. Set the number of times to n, then for the 1st to (n-1)th time, the adjustment step is the value of the time interval. For the nth time, the adjustment step is the delay length divided by the time interval. the remainder. Or, for the first time, the adjustment step is the remainder obtained by dividing the delay length by the time interval, and for subsequent times, the adjustment step is the value of the time interval. Alternatively, the delay duration can be equally divided into n parts, and the step size of each adjustment in n adjustments is the length of one part of the delay duration.

As shown in FIG. 5 , in embodiment 500 , the value interval 502 of the adjustment step size at the triggering moment can be determined first based on the acquisition frame rate 501 of the image sensor. Then, the number of adjustments 503 and the step size of each adjustment 504 are determined based on the value interval 502 and the delay length.

For example, the collection frame rate 501 may be negatively correlated with the length of the value interval 502 . For example, the length of the value interval 502 may be proportional to the length of the time interval for collecting image data. For example, the upper limit of the value in the value interval 502 can be set to the value of the time interval, or the upper limit of the value can be set to 0.5 times the time interval, etc. This disclosure does not limit this. After determining the value interval 502, this embodiment may divide the delay duration by the value of the value interval 502, round up, and use the value obtained by rounding up as the number of adjustments.

In one embodiment, the controller may, for example, maintain the delay information received in the past. After receiving the latest delay information, the controller may, for example, first find the delay information adjacent to the latest delay information at the reception time from the maintained accepted delay information, and then determine whether the latest delay information is relative to Adjacent delay information changes. If changes occur, adjust the triggering time of the image sensor. Specifically, the triggering time of the image sensor can be adjusted according to the change value of the delay information. In this way, the iterative adjustment of the image sensor can be realized, the degree of adjustment of the triggering moment of the image sensor within a single adjustment cycle can be reduced, and the stability of the image data collected by the image sensor can be improved.

For example, as shown in Figure 5, if the latest delay information received by the controller is set to the m-th delay information 505, then this embodiment can calculate the delay duration indicated by the m-th delay information 505 and the (m-1)-th delay. The difference 507 between the delay durations indicated by the information 506. Subsequently, based on the difference 507 and the value interval 502, the number of adjustments 503 and the step size 504 of each adjustment are determined.

For example, when adjusting the triggering moment of the image sensor according to the change value of the delay information, considering that the difference between the delay lengths indicated by two adjacent delay information may be a negative value, in this embodiment, according to the acquisition frame rate The lower limit of the value interval 502 determined in step 501 may be, for example, a value less than 0. For example, the absolute value of the difference between the lower limit value and 0 may be smaller than the absolute value of the difference between the upper limit value of the value interval and 0. This is due to the limitation of the acquisition frame rate of the image sensor. The difference between this lower limit value and 0 is inversely related to the acquisition frame rate.

For example, the acquisition frame rate of the image sensor is set to 30 fps, and the value interval can be set to [-200 μm, 32 ms], for example. If the acquisition frame rate is 15fps, the value interval can be set to [-400μm, 64ms], for example.

It can be understood that, after determining the number of times to adjust the triggering moment and the step size of each adjustment, the most recent one or multiple consecutive triggering moments can be adjusted based on the number of times and the step size. For example, if the rotation period of the radar sensor is set to 100ms and the acquisition frame rate of the image sensor is 30fps, the image data that the image sensor can collect in one rotation period is 3 frames. In order to reduce the delay of signal transmission, the controller may, for example, send a trigger signal every two frames. After receiving the trigger signal, the image sensor continuously collects three frames of image data. The differences between the collection time of the second frame of image data, the third frame of image data and the collection time of the first frame of image data are 33.3ms and 66.7ms respectively.

For example, when the determined number of adjustments is once, the controller may only adjust the sending timing of the trigger signal sent every two frames, thereby adjusting the triggering moment of the image sensor. If the number of adjustments is multiple, the controller can adjust the signal sending mechanism that sends the trigger signal every two frames to the signal sending mechanism that sends the trigger signal every frame. After completing the adjustment of the trigger time and sending the trigger signal to the image sensor according to the adjusted trigger time, the controller can adjust the signal sending mechanism back to the mechanism of sending the trigger signal every two frames.

According to an embodiment of the present disclosure, if within one rotation cycle of the radar sensor, the image data that the image sensor can collect is p frames, and the number of adjustments determined based on the delay information is greater than p times, this embodiment can only adjust the most recent Adjustment is made for p consecutive triggering moments. The adjustment to the latest p+1th trigger moment can be determined based on the delay information received in the next cycle.

The above-mentioned controller and vehicle-mounted terminal can both be integrated into the automatic driving system of the vehicle. For the automatic driving system, the image sensor and the radar sensor can collect data synchronously by executing the synchronous data collection methods of the above embodiments.

Figure 6 is a schematic diagram of the principle of a method for synchronously collecting data according to an embodiment of the present disclosure.

As shown in FIG. 6 , in this embodiment, the execution subject of the method 600 for synchronously collecting data is the automatic driving system of the automatic driving vehicle. The automatic driving system includes a vehicle-mounted terminal 610 and a controller 620. The controller 620 may specifically be an FPGA chip. The autonomous vehicle may also be provided with a radar sensor 601 and an image sensor 602.

In the method 600, the radar sensor 601 may encapsulate the point cloud data collected from various angles into data packets as data packets for each angle. And the data packets of each angle are sent to the vehicle-mounted terminal 610.

The vehicle-mounted terminal 610 can analyze the received data packet and determine the angle targeted by the data packet. And after it is determined that the data packet for the predetermined angle is received, the first time information for the radar sensor to collect point cloud data at the predetermined angle is determined. The predetermined angle is within the viewing angle range of the image sensor. It can be understood that the implementation manner of determining the first time information is similar to the implementation manner of operation S210 described above, and will not be described again here.

Subsequently, the vehicle-mounted terminal 610 can determine the delay information for the image sensor 602 based on the first time information, and send the delay information to the FPGA 620. It can be understood that the implementation of determining the delay information may be similar to the implementation of operation S220 described above, and will not be described again here.

After receiving the delay information, the FPGA chip 620 can adjust the triggering time of the image sensor 602 according to the delay information, so that the image sensor 602 and the radar sensor 601 can collect data synchronously. The implementation of adjusting the triggering time is similar to the implementation of operation S410 described above. After adjusting the trigger time, the FPGA chip 620 may send a trigger signal to the image sensor 602 when the adjusted trigger time is reached. Image sensor 602 may begin collecting image data in response to the trigger signal.

For example, the image sensor 602 may send the collected image data to the FPGA chip 620, so that the FPGA chip 620 adds second time information to the received image data. Specifically, a timestamp can be added to the image data to obtain image data 631 with a timestamp.

In this embodiment, since the time at which the image sensor collects image data is adjusted based on the time information at which the radar sensor collects point cloud data, the obtained series of image data 631 includes points that are aligned with the point cloud data subsequently collected by the radar sensor 601 image data. Specifically, the image data and the point cloud data can be aligned according to the second time information of the image data and the time information of the point cloud data subsequently collected by the radar sensor. For example, point cloud data and image data with the same time information can be used as an aligned pair of data, or point cloud data and image data with time information that differs less than a threshold can be used as an aligned pair of data.

Figure 7 is a schematic diagram of the principle of synchronous data collection according to an embodiment of the present disclosure.

As shown in Figure 7, in embodiment 700, the reference time based on which the FPGA adds time information to image data and the reference time used by the radar sensor to collect point cloud data are both set to the GPS clock. Set the positioning period of the GPS to 1s, the point cloud data collection period of the radar sensor to 100ms, and the collection frame rate of the image sensor (such as a camera) to 30fps.

For example, if the GPS clock is used as the reference, the predetermined angle is 0°, and the starting time is 0, then the time when the radar sensor collects point cloud data at 0° should be 0.1s, 0.9s, 1s, 1.1s..., and the camera The time for collecting image data should be 33.3ms, 66.7ms, 0.1s,..., 0.8667s, 0.9s, 0.9333s.... In this embodiment 700, the time at which the radar sensor collects point cloud data is actually 0.92s, 1.02s,...2.01s. The time when the camera collects image data is actually 0.89s, 0.9233s, ....

Using the method of synchronously collecting data provided by this disclosure, the trigger time of the camera can be adjusted from 0.89s to 0.92s, and the delay length based on the adjustment is t1, which is 30ms. By analogy, the trigger time of the camera is adjusted from 1.99s to 2.01s, and the delay length based on the adjustment is t2, which is 20ms. Through adjustment, the image data collected by the camera and the point cloud data collected by the radar are temporally aligned.

The present disclosure also provides a synchronization determination method for evaluating the synchronization of an image sensor and a radar sensor. The synchronization determination method will be described in detail below with reference to FIG. 8 .

Figure 8 is a schematic flowchart of a synchronization determination method according to an embodiment of the present disclosure.

As shown in FIG. 8 , the synchronization determination method 800 of this embodiment may include operations S810 to S840. The synchronization determination method 800 can be executed by a cloud system.

In operation S810, a data packet sequence collected by the radar sensor and an image data sequence collected by the image sensor are obtained.

According to embodiments of the present disclosure, the cloud system can obtain the data packet sequence and image data sequence collected by the autonomous vehicle within the unit time on a daily basis. The cloud system may obtain the sequence of data packets and the sequence of image data via a wireless communication link, for example.

In operation S820, for each data packet in the data packet sequence, image data matching each data packet is determined based on the time information of each data packet and the time information of each image data in the image data sequence.

According to embodiments of the present disclosure, the cloud system can parse each data packet in the data packet sequence and determine the first time information of the point cloud data in the data packet collected by the radar sensor. Then, the image data whose time information is added to the image data sequence is the same as or similar to the first time information is used as the image data matching each data packet. For example, if the time difference between the second time information of the image data and the first time information of the point cloud data is less than ±3 ms, it can be determined that the image data matches the point cloud data. For data packets for which matching image data is not determined, the data packet can be discarded.

In operation S830, a synchronization relationship between each data packet and the matching image data is determined based on the difference in time information between each data packet and the matching image data and the difference threshold.

In one embodiment, the difference threshold may be set based on experience, for example, the difference threshold may be set on the premise that it does not affect the safe driving of the vehicle. For example, the difference threshold may be 15 ms, 7 ms, etc., which is not limited in this disclosure.

In one embodiment, the difference threshold may be determined based on the fusion result of historical image data and point cloud data in the historical data package. If the positional deviation between the pixels projected from the point cloud data into the image coordinate system in the fusion result and the corresponding pixels in the historical image data reaches a critical value (such as a predetermined deviation value), the time information of the historical image data can be combined with the historical data. The difference in packet time information is used as the difference threshold. In this way, the accuracy of the set difference threshold and therefore the evaluation of the synchronization relationship can be increased.

In this embodiment, if the difference in time information between each data packet and the matching image data is greater than the difference threshold, it may be determined that the synchronization relationship between each data packet and the matching image data is asynchronous. If the difference is less than or equal to the difference threshold, it can be determined that the synchronization relationship between each data packet and the matching image data is synchronization.

In operation S840, the synchronization relationship between the radar sensor and the image sensor is determined according to the synchronization relationship between each of the plurality of data packets in the data packet sequence and the matching image data.

According to embodiments of the present disclosure, for example, the synchronization relationship between each data packet and the matching image data in the data packet sequence can be counted. If the proportion of data packets synchronized with the matching image data is greater than or equal to the predetermined proportion threshold, it can be determined that the synchronization relationship between the radar sensor and the image sensor is synchronous, otherwise it is determined that the synchronization relationship is out of synchronization. The predetermined ratio threshold may be, for example, a value less than 1 but close to 1 such as 0.8, which is not limited in this disclosure.

Embodiments of the present disclosure can improve the efficiency of determining the synchronization relationship by setting a difference threshold and using the difference threshold as a basis for synchronization between the data packets collected by the radar sensor and the image data collected by the image sensor. Compared with the technical solution of fusion of image data and point cloud data in the data package, and determining the synchronization relationship based on the fusion results, it can reduce human investment, reduce the consumption of computing resources, and improve parallel processing capabilities.

According to embodiments of the present disclosure, a predetermined containerization tool may be used to fuse the point cloud data and historical image data in the historical data package. By using the predetermined container flower tool, it can be compatible with different platforms installed on different cloud systems, which is beneficial to improving the applicability of the synchronization determination method provided by the embodiments of the present disclosure.

Among them, the predetermined containerization tool can, for example, fuse the point cloud data and the image data based on the following principle: align and fuse the point cloud data and the image data according to the external parameters of the radar sensor and the image sensor. Specifically, the point cloud data can be projected into the camera coordinate system to obtain the pixels projected from the point cloud data into the camera coordinate system. The pixels are superimposed on the corresponding pixels of the image data to obtain the fused data.

According to an embodiment of the present disclosure, when it is determined that the synchronization relationship between the radar sensor and the image sensor is asynchronous, this embodiment may also send prompt information to the target object. For example, a reminder message can be sent to the target communication account. The target communication account can be an email account or a mobile communication account, etc., and this disclosure does not limit this. By sending prompt information, monitoring personnel can be informed of the out-of-synchronization between the image sensor and the radar sensor in time, so that relevant strategies can be adopted to solve the out-of-synchronization problem and reduce the chance of unsafe driving of autonomous vehicles due to out-of-synchronization.

Based on the method for synchronous data collection performed by a vehicle-mounted terminal provided by the disclosure, the disclosure also provides a vehicle-mounted terminal.

Figure 9 is a schematic diagram of a vehicle-mounted terminal according to an embodiment of the present disclosure.

As shown in FIG. 9 , this embodiment provides a vehicle-mounted terminal 900 , which can be integrated into any vehicle such as an autonomous vehicle. The vehicle-mounted terminal 900 may, for example, be configured to perform the method of synchronously collecting data performed by the vehicle-mounted terminal described above.

Specifically, the vehicle-mounted terminal 900 may be configured to determine first time information when the radar sensor collects point cloud data at a predetermined angle in response to receiving a data packet from the radar sensor for a predetermined angle. The predetermined angle is within the viewing angle range of the image sensor in the vehicle equipped with the vehicle-mounted terminal, and the data package includes point cloud data collected by the radar sensor at the predetermined angle. The implementation manner of determining the first time information in this embodiment is similar to the implementation manner of operation S210 described above, and will not be described again here.

The vehicle-mounted terminal 900 may also be configured to determine delay information for the image sensor based on the first time information, and send the delay information to the controller that controls the image sensor, so that the controller controls the image sensor and rotates to the predetermined position. The radar sensors collect data simultaneously at angles.

In one embodiment, the vehicle-mounted terminal 900 may, for example, use the principle described above with respect to FIG. 3 to determine the delay information, which is not limited in this disclosure. It can be understood that the structure of the vehicle-mounted terminal 900 in FIG. 9 is only used as a diagram to facilitate understanding of the present disclosure, and the present disclosure does not limit it.

Based on the synchronous data collection method executed by the controller provided by the disclosure, the disclosure also provides a controller.

Figure 10 is a schematic diagram of a controller according to an embodiment of the present disclosure.

As shown in FIG. 10 , this embodiment provides a controller 1000 that can be integrated into any vehicle such as an autonomous vehicle, and the controller 1000 can also be communicatively connected with a vehicle-mounted terminal in the vehicle. The controller 1000 may, for example, be configured to perform the method of synchronously collecting data performed by the controller described above.

Specifically, the controller 1000 may be configured to, in response to receiving delay information for the image sensor, adjust the triggering moment of the image sensor according to the delay information. The principle of adjusting the triggering time in this embodiment may be similar to the adjustment principle in operation S410 described above, and will not be described again here. It can be understood that the delay information may be sent by a vehicle-mounted terminal that is communicatively connected to the controller.

The controller 1000 may also be configured to send a trigger signal to the image bed Angel in the vehicle in response to reaching the trigger moment, and in response to receiving the image data collected by the image sensor, add time information to the image data so that the vehicle can The terminal uses time information to align the image data with the point cloud data collected by the radar sensor.

In one embodiment, the controller 1000 may, for example, use the principle described above with respect to FIG. 5 to determine the number of times and steps to adjust the triggering moment, and adjust the triggering moment according to the determined number of times and the step size. This disclosure does not apply to this. Make limitations. It can be understood that the structure of the controller 1000 in FIG. 10 is only used as a diagram to facilitate understanding of the present disclosure, and the present disclosure does not limit this.

Based on the vehicle-mounted terminal and controller provided by the present disclosure, the present disclosure also provides an autonomous driving vehicle.

Figure 11 is a schematic diagram of an autonomous vehicle according to an embodiment of the present disclosure.

As shown in FIG. 11 , this embodiment provides an autonomous vehicle 1100 in which a vehicle-mounted terminal, a controller, a radar sensor, and an image sensor can be integrated.

The vehicle-mounted terminal may be the vehicle-mounted terminal 900 described above, and the controller may be the controller 1000 described above.

During the process of autonomous driving, the image sensor can periodically collect image data, and the radar sensor can periodically collect point cloud data, and encapsulate the point cloud data to obtain a data package. The vehicle-mounted terminal in the autonomous vehicle may determine the first time information that the radar sensor collects point cloud data at a predetermined angle in response to receiving a data packet from the radar sensor for a predetermined angle. The predetermined angle is within the viewing angle range of the image sensor, and the data package includes point cloud data collected by the radar sensor at the predetermined angle. After obtaining the first time information, the vehicle-mounted terminal can also determine the delay information for the image sensor based on the first time information, and send the determined delay information to the controller. It can be understood that the vehicle-mounted terminal determines the delay information in a manner similar to the implementation principle of the method 200 for synchronously collecting data described above, which will not be described again here.

After receiving the delay information, the controller can adjust the trigger time of the image sensor according to the delay information, and when it is determined that the trigger time of the image sensor is reached, send a trigger signal to the image sensor. The controller can also add second time information to the image data after receiving the image data collected by the image sensor, so that the vehicle-mounted terminal uses the second time information to align the image data with the point cloud data collected by the radar sensor.

Based on the synchronization determination method provided by the present disclosure, the present disclosure also provides a cloud system for executing the synchronization determination method.

Figure 12 is a schematic diagram of a cloud system according to an embodiment of the present disclosure.

As shown in Figure 12, the present disclosure provides a cloud system 1200. The cloud system 1200 can communicate with the self-driving vehicle provided by the present disclosure through a communication link to obtain data collected by sensors from the self-driving vehicle, or receive automatic Data sent by driving a vehicle.

Specifically, the cloud system 1200 may be configured to obtain the data packet sequence collected by the radar sensor and the image data sequence collected by the image sensor in the autonomous vehicle. After acquiring the data packet sequence and the image data sequence, the cloud system 1200 can determine, for each data packet in the data packet sequence, the time information associated with each data packet based on the time information of each data packet and the time information of each image data in the image data sequence. packets of matching image data. Subsequently, the synchronization relationship between each data packet and the matching image data is determined based on the difference in time information between each data packet and the matching image data and the difference threshold. Finally, the cloud system 1200 can determine the synchronization relationship between the radar sensor and the image sensor based on the synchronization relationship between each of the multiple data packets in the data packet sequence and the matching image data.

In one embodiment, the cloud system 1200 can be used to perform the synchronization determination method described above to determine the synchronization relationship between the radar sensor and the image sensor, which is not limited by the present disclosure.

Based on the method for synchronously collecting data performed by a vehicle-mounted terminal provided by the disclosure, the disclosure also provides a device for synchronously collecting data, which device can be integrated into the vehicle-mounted terminal.

Figure 13 is a structural block diagram of a device for synchronously collecting data according to an embodiment of the present disclosure.

As shown in Figure 13, the device 1300 for synchronously collecting data in this embodiment can be integrated into a vehicle-mounted terminal. The device 1300 for synchronously collecting data can include a time information determination module 1310, a delay information determination module 1320 and an information sending module 1330.

The time information determination module 1310 is configured to determine first time information that the radar sensor collects point cloud data at a predetermined angle in response to receiving a data packet from the radar sensor for a predetermined angle; wherein the predetermined angle is within the viewing angle range of the image sensor. Among them, the data package includes point cloud data collected by the radar sensor at a predetermined angle. In an embodiment, the time information determination module 1310 may be configured to perform the above-described operation S210, which will not be described again here.

The delay information determining module 1320 is configured to determine delay information for the image sensor according to the first time information. In an embodiment, the delay information determination module 1320 may be configured to perform the above-described operation S220, which will not be described again here.

The information sending module 1330 is used to send delay information to the controller, so that the controller controls the image sensor to collect data synchronously with the radar sensor rotated to a predetermined angle. In one embodiment, the information sending module 1330 may be configured to perform the above-described operation S230, which will not be described again here.

According to an embodiment of the present disclosure, the time information determination module 1310 may include a parsing sub-module and a first determination sub-module. The parsing sub-module is used to parse the data packet and obtain the time data in the data packet; the time data includes the base time and the periodic encapsulation time. The first determination sub-module is used to determine the first time information for the radar sensor to collect point cloud data at a predetermined angle based on the reference time and periodic packaging time. The reference time is aligned with the time of the positioning device of the positioning device in the vehicle where the radar sensor is located.

According to an embodiment of the present disclosure, the delay information determination module 1320 may include a second determination sub-module, a deviation value determination sub-module and a delay determination sub-module. The second determination sub-module is used to determine the second time information for the radar sensor to collect point cloud data at 0° based on the first time information, the rotation period of the radar sensor and the predetermined angle. The deviation value determination sub-module is used to determine the rotation deviation value of the radar sensor based on the remainder obtained by taking the remainder of the rotation period based on the second time information. The delay determination sub-module is used to determine delay information for the image sensor according to the rotation deviation value.

According to an embodiment of the present disclosure, the delay determination sub-module may be specifically configured to determine the delay information of the image sensor based on the sum of the predetermined error value and the rotation deviation value. The predetermined error value is greater than 0, and the predetermined error value is determined based on the optical center position of the image sensor and the position of the target object within the viewing angle range of the image sensor.

According to an embodiment of the present disclosure, the controller includes an artificial intelligence chip.

Based on the method for synchronously collecting data executed by the controller provided by the present disclosure, the present disclosure also provides a device for synchronously collecting data, which device can be integrated into the controller.

Figure 14 is a structural block diagram of a device for synchronously collecting data according to another embodiment of the present disclosure.

As shown in Figure 14, the device 1400 for synchronously collecting data in this embodiment may include a time adjustment module 1410, a signal sending module 1420 and a time adding module 1430.

The timing adjustment module 1410 is configured to respond to receiving delay information for the image sensor and adjust the triggering timing of the image sensor according to the delay information. Among them, the delay information is determined based on the time information of the radar sensor collecting point cloud data at a predetermined angle, and the predetermined angle is within the viewing angle range of the image sensor. In an embodiment, the time adjustment module 1410 may be used to perform the above-described operation S410, which will not be described again here.

The signal sending module 1420 is configured to send a trigger signal to the image sensor in response to reaching the trigger moment. In an embodiment, the signal sending module 1420 may be configured to perform the above-described operation S420, which will not be described again here.

The time adding module 1430 is configured to add time information to the image data in response to receiving the image data collected by the image sensor, so as to use the time information to align the image data with the point cloud data collected by the radar sensor. In one embodiment, the time adding module 1430 may be used to perform the operation S430 described above, which will not be described again.

According to an embodiment of the present disclosure, the delay information indicates the delay duration, and the above-mentioned time adjustment module 1410 may include an adjustment information determination sub-module and an adjustment sub-module. The adjustment information determination submodule is used to determine the number of adjustment trigger moments and the step size for each adjustment based on the acquisition frame rate and delay time of the image sensor. The adjustment sub-module is used to adjust the triggering moment of the image sensor based on the number of times and step size.

According to embodiments of the present disclosure, the adjustment information determination sub-module may include an adjustment interval determination unit and an information determination unit. The adjustment interval determination unit is used to determine the value interval of the adjustment step at the triggering moment based on the acquisition frame rate of the image sensor. The information determination unit is used to determine the number of times and step size based on the value interval and delay length. Among them, the acquisition frame rate is negatively related to the length of the value interval.

According to an embodiment of the present disclosure, the above-mentioned time adjustment module 1410 is specifically configured to respond to receiving delay information and the delay information changes relative to adjacent delay information in the received delay information, adjusting the triggering of the image sensor according to the change value of the delay information. time.

Based on the method for synchronously collecting data performed by an autonomous driving vehicle provided by the present disclosure, the present disclosure also provides a device for synchronously collecting data, which device can be integrated into the autonomous driving vehicle.

Figure 15 is a structural block diagram of a device for synchronously collecting data according to another embodiment of the present disclosure.

As shown in Figure 15, the device 1500 for synchronously collecting data in this embodiment may include a time information determination module 1510, a delay information determination module 1520, a time adjustment module 1530, a signal sending module 1540 and a time adding module 1550.

The time information determination module 1510 is configured to determine first time information that the radar sensor collects point cloud data at a predetermined angle in response to receiving a data packet from the radar sensor for a predetermined angle; wherein the predetermined angle is within the viewing angle range of the image sensor. The data package includes point cloud data collected by the radar sensor at predetermined angles. In an embodiment, the time information determination module 1510 may be used to perform the operation S210 described above, which will not be described again.

The delay information determination module 1520 is configured to determine delay information for the image sensor according to the first time information. In an embodiment, the delay information determination module 1520 may be configured to perform the operation S220 described above, which will not be described again.

The time adjustment module 1530 is used to adjust the triggering time of the image sensor according to the delay information, so that the image sensor collects data synchronously with the radar sensor rotated to a predetermined angle. In one embodiment, the time adjustment module 1530 may be used to perform the operation S410 described above, which will not be described again.

The signal sending module 1540 is configured to send a trigger signal to the image sensor in response to reaching the trigger moment. In an embodiment, the signal sending module 1540 may be used to perform the operation S420 described above, which will not be described again here.

The time adding module 1550 is configured to add second time information to the image data in response to receiving the image data collected by the image sensor, so as to utilize the second time information to align the image data with the point cloud data collected by the radar sensor. In one embodiment, the time adding module 1550 may be used to perform the above-described operation S430, which will not be described again here.

Based on the synchronization determination method provided by the present disclosure, the present disclosure also provides a synchronization determination device, which can be integrated in a cloud system.

Figure 16 is a structural block diagram of a synchronization determination device according to an embodiment of the present disclosure.

As shown in Figure 16, the synchronization determination device 1600 of this embodiment may include a data acquisition module 1610, a data matching module 1620, a data relationship determination module 1630 and a synchronization relationship determination module 1640.

The data acquisition module 1610 is used to acquire the data packet sequence collected by the radar sensor and the image data sequence collected by the image sensor. In an embodiment, the data acquisition module 1610 may be used to perform the above-described operation S810, which will not be described again here.

The data matching module 1620 is configured to determine, for each data packet in the data packet sequence, image data matching each data packet based on the time information of each data packet and the time information of each image data in the image data sequence. In one embodiment, the data matching module 1620 may be used to perform the above-described operation S820, which will not be described again here.

The data relationship determination module 1630 is configured to determine a synchronization relationship between each data packet and the matching image data based on the difference in time information between each data packet and the matching image data and the difference threshold. In one embodiment, the data relationship determination module 1630 may be used to perform the above-described operation S830, which will not be described again here.

The synchronization relationship determination module 1640 is used to determine the synchronization relationship between the radar sensor and the image sensor based on the synchronization relationship between each of the multiple data packets in the data packet sequence and the matching image data. In an embodiment, the synchronization relationship determination module 1640 may be used to perform the above-described operation S840, which will not be described again here.

According to an embodiment of the present disclosure, the above-mentioned synchronization determining device may further include a prompt information sending module, configured to send prompt information to the target object in response to the synchronization relationship between the radar sensor and the image sensor indicating out-of-synchronization.

According to an embodiment of the present disclosure, the above-mentioned synchronization determination device may further include a threshold determination module for determining a difference threshold. The threshold determination module may include a data fusion sub-module and a threshold determination sub-module. The data fusion sub-module is used to obtain fusion data based on the historical data packets collected by the radar sensor and the historical image data matching the historical data packets; the fusion data represents the projection of the point cloud data in the historical data packets to the pixels in the image coordinate system , and the position deviation value between pixels in historical image data. The threshold determination sub-module is used to determine the difference in time information between the historical data packet and the historical image data as a difference threshold when the position deviation value reaches a predetermined deviation value.

According to embodiments of the present disclosure, the above-mentioned data fusion sub-module is specifically used to use predetermined containerization tools to fuse point cloud data and historical image data in historical data packages to obtain fused data.

It should be noted that in the technical solution of this disclosure, the collection, storage, use, processing, transmission, provision, disclosure and application of user personal information are all in compliance with relevant laws and regulations, and necessary confidentiality measures have been taken. , and does not violate public order and good customs. In the technical solution of the present disclosure, the user's authorization or consent is obtained before obtaining or collecting the user's personal information.

According to embodiments of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

FIG. 17 shows a schematic block diagram of an example electronic device 1700 that can be used to implement the method of synchronously collecting data or the method of synchronization determination according to embodiments of the present disclosure. Electronic devices are intended to refer to various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic devices may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are examples only and are not intended to limit implementations of the disclosure described and/or claimed herein.

As shown in Figure 17, the device 1700 includes a computing unit 1701 that can execute according to a computer program stored in a read-only memory (ROM) 1702 or loaded from a storage unit 1708 into a random access memory (RAM) 1703. Various appropriate actions and treatments. In the RAM 1703, various programs and data required for the operation of the device 1700 can also be stored. Computing unit 1701, ROM 1702 and RAM 1703 are connected to each other via bus 1704. Input/output (I/O) interface 1705 is also connected to bus 1704.

Multiple components in device 1700 are connected to I/O interface 1705, including: input unit 1706, such as keyboard, mouse, etc.; output unit 1707, such as various types of displays, speakers, etc.; storage unit 1708, such as magnetic disk, optical disk, etc. ; and communication unit 1709, such as a network card, modem, wireless communication transceiver, etc. The communication unit 1709 allows the device 1700 to exchange information/data with other devices through computer networks such as the Internet and/or various telecommunications networks.

Computing unit 1701 may be a variety of general and/or special purpose processing components having processing and computing capabilities. Some examples of the computing unit 1701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, digital signal processing processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 1701 performs various methods and processes described above, such as the method of synchronously collecting data or the method of synchronization determination. For example, in some embodiments, the synchronization data collection method or the synchronization determination method may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as the storage unit 1708. In some embodiments, part or all of the computer program may be loaded and/or installed onto device 1700 via ROM 1702 and/or communication unit 1709. When the computer program is loaded into the RAM 1703 and executed by the computing unit 1701, one or more steps of the above-described method of synchronously collecting data or the synchronously determining method may be performed. Alternatively, in other embodiments, the computing unit 1701 may be configured to perform the method of synchronously collecting data or the method of synchronously determining in any other suitable manner (eg, by means of firmware).

Various implementations of the systems and techniques described above may be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on a chip implemented in a system (SOC), complex programmable logic device (CPLD), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include implementation in one or more computer programs executable and/or interpreted on a programmable system including at least one programmable processor, the programmable processor The processor, which may be a special purpose or general purpose programmable processor, may receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device. An output device.

Program code for implementing the methods of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing device, such that the program codes, when executed by the processor or controller, cause the functions specified in the flowcharts and/or block diagrams/ The operation is implemented. The program code may execute entirely on the machine, partly on the machine, as a stand-alone software package, partly on the machine and partly on a remote machine or entirely on the remote machine or server.

In the context of this disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media may include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media would include one or more wire-based electrical connections, laptop disks, hard drives, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination of the above.

To provide interaction with a user, the systems and techniques described herein may be implemented on a computer having a display device (eg, a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user ); and a keyboard and pointing device (eg, a mouse or a trackball) through which a user can provide input to the computer. Other kinds of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and may be provided in any form, including Acoustic input, voice input or tactile input) to receive input from the user.

The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., A user's computer having a graphical user interface or web browser through which the user can interact with implementations of the systems and technologies described herein), or including such backend components, middleware components, or any combination of front-end components in a computing system. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communications network). Examples of communication networks include: local area network (LAN), wide area network (WAN), and the Internet.

Computer systems may include clients and servers. Clients and servers are generally remote from each other and typically interact over a communications network. The relationship of client and server is created by computer programs running on corresponding computers and having a client-server relationship with each other. Among them, the server can be a cloud server, also known as cloud computing server or cloud host. It is a host product in the cloud computing service system to solve the problem of traditional physical host and VPS service ("Virtual Private Server", or "VPS" for short). "), there are defects such as difficult management and weak business scalability. The server can also be a distributed system server or a server combined with a blockchain.

It should be understood that various forms of the process shown above may be used, with steps reordered, added or deleted. For example, each step described in the present disclosure can be executed in parallel, sequentially, or in a different order. As long as the desired results of the technical solution disclosed in the present disclosure can be achieved, there is no limitation here.

The above-mentioned specific embodiments do not constitute a limitation on the scope of the present disclosure. It will be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions are possible depending on design requirements and other factors. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of this disclosure shall be included in the protection scope of this disclosure.

Claims

A method for synchronous data collection, including:

In response to receiving a data packet for a predetermined angle from a radar sensor, determining first time information at which the radar sensor collects point cloud data at the predetermined angle; wherein the predetermined angle is within a viewing angle range of the image sensor;

determining delay information for the image sensor based on the first time information; and

sending the delay information to the controller so that the controller controls the image sensor to collect data in synchronization with the radar sensor rotated to the predetermined angle,

Wherein, the data package includes point cloud data collected by the radar sensor at the predetermined angle.
The method of claim 1, wherein determining, in response to receiving a data packet from a radar sensor for a predetermined angle, the first time information at which the radar sensor collects point cloud data at the predetermined angle includes:

Parse the data packet to obtain the time data in the data packet; wherein the time data includes a reference time and a periodic encapsulation time; and

Determine the first time information for the radar sensor to collect point cloud data at the predetermined angle according to the reference time and the periodic packaging time,

Wherein, the reference time is aligned with the positioning device time of the positioning device in the vehicle where the radar sensor is located.
The method of claim 1, wherein determining delay information for the image sensor according to the first time information includes:

According to the first time information, the rotation period of the radar sensor and the predetermined angle, determine the second time information for the radar sensor to collect point cloud data at 0°;

Determine the rotation deviation value of the radar sensor based on the remainder obtained by taking the remainder of the rotation period based on the second time information; and

Delay information for the image sensor is determined based on the rotational deviation value.
The method of claim 3, wherein determining delay information for the image sensor according to the rotational deviation value includes:

Determine the delay information of the image sensor according to the sum of the predetermined error value and the rotation deviation value,

Wherein, the value of the predetermined error value is greater than 0, and the predetermined error value is determined based on the optical center position of the image sensor and the position of the target object within the viewing angle range of the image sensor.
The method of claim 1, wherein the controller includes an artificial intelligence chip.
A method for synchronous data collection, including:

In response to receiving delay information for the image sensor, adjusting a triggering moment of the image sensor based on the delay information;

In response to reaching the trigger moment, sending a trigger signal to the image sensor; and

In response to receiving the image data collected by the image sensor, adding time information to the image data to utilize the time information to align the image data with the point cloud data collected by the radar sensor,

Wherein, the delay information is determined based on the time information of the radar sensor collecting point cloud data at a predetermined angle, and the predetermined angle is within the viewing angle range of the image sensor.
The method according to claim 6, wherein the delay information indicates a delay duration; and adjusting the triggering moment of the image sensor according to the delay information includes:

Determine the number of times to adjust the trigger moment and the step size for each adjustment according to the acquisition frame rate of the image sensor and the delay duration; and

The triggering moment of the image sensor is adjusted according to the number of times and the step size.
The method according to claim 7, wherein determining the number of times to adjust the trigger moment and the step size for each adjustment according to the acquisition frame rate of the image sensor and the delay duration includes:

Determine the value interval of the adjustment step size of the trigger moment according to the acquisition frame rate of the image sensor; and

Determine the number of times and the step size based on the value interval and the delay length,

Wherein, the collection frame rate is negatively correlated with the length of the value interval.
The method of claim 6, wherein in response to receiving delay information for the image sensor, adjusting the triggering moment of the image sensor according to the delay information includes:

In response to receiving the delay information and the delay information changing relative to adjacent delay information in the received delay information, adjusting the triggering moment of the image sensor according to the change value of the delay information.
A method for synchronous data collection, including:

In response to receiving a data packet for a predetermined angle from a radar sensor, determining first time information at which the radar sensor collects point cloud data at the predetermined angle; wherein the predetermined angle is within a viewing angle range of the image sensor;

determining delay information for the image sensor based on the first time information;

Adjust the triggering time of the image sensor according to the delay information, so that the image sensor collects data synchronously with the radar sensor rotated to the predetermined angle;

In response to reaching the trigger moment, sending a trigger signal to the image sensor; and

In response to receiving the image data collected by the image sensor, second time information is added to the image data to align the image data with the point cloud data collected by the radar sensor using the second time information.
A synchronization determination method, including:

Obtain the data packet sequence collected by the radar sensor and the image data sequence collected by the image sensor;

For each data packet in the data packet sequence, determine the image data matching each data packet according to the time information of each data packet and the time information of each image data in the image data sequence;

Determine the synchronization relationship between each data packet and the matching image data according to the difference in time information and the difference threshold between each data packet and the matching image data; and

The synchronization relationship between the radar sensor and the image sensor is determined according to the synchronization relationship between each of the plurality of data packets in the data packet sequence and the matching image data.
The method of claim 11, further comprising:

In response to the synchronization relationship between the radar sensor and the image sensor indicating out-of-synchronization, prompt information is sent to the target object.
The method of claim 11, further comprising determining the difference threshold by:

Fusion data is obtained based on the historical data packets collected by the radar sensor and the historical image data matching the historical data packets; the fusion data represents the projection of the point cloud data in the historical data packets to the pixels in the image coordinate system , and the position deviation value between pixels in the historical image data; and

When the position deviation value reaches a predetermined deviation value, the difference in time information between the historical data packet and the historical image data is determined to be the difference threshold.
The method according to claim 13, wherein obtaining the fusion data based on the historical data packets collected by the radar sensor and the historical image data matching the historical data packets includes:

A predetermined containerization tool is used to fuse the point cloud data in the historical data package and the historical image data to obtain the fused data.
A vehicle-mounted terminal configured as:

In response to receiving a data packet for a predetermined angle from a radar sensor, determining first time information at which the radar sensor collects point cloud data at the predetermined angle; wherein the predetermined angle is within a viewing angle range of the image sensor;

determining delay information for the image sensor based on the first time information; and

sending the delay information to the controller so that the controller controls the image sensor to collect data in synchronization with the radar sensor rotated to the predetermined angle,

Wherein, the data package includes point cloud data collected by the radar sensor at the predetermined angle.
A controller configured to:

In response to receiving delay information for the image sensor, adjusting a triggering moment of the image sensor based on the delay information;

In response to reaching the trigger moment, sending a trigger signal to the image sensor; and

In response to receiving the image data collected by the image sensor, adding time information to the image data to utilize the time information to align the image data with the point cloud data collected by the radar sensor,

Wherein, the delay information is determined based on the time information of the radar sensor collecting point cloud data at a predetermined angle, and the predetermined angle is within the viewing angle range of the image sensor.
An autonomous vehicle, including a vehicle-mounted terminal, a controller, a radar sensor and an image sensor, wherein:

The vehicle-mounted terminal is configured as:

In response to receiving a data packet for a predetermined angle from a radar sensor, determining first time information for the radar sensor to collect point cloud data at the predetermined angle; wherein the predetermined angle is within a viewing angle range of the image sensor; The data package includes point cloud data collected by the radar sensor at the predetermined angle;

determining delay information for the image sensor based on the first time information; and

sending the delay information to the controller;

The controller is configured to:

In response to receiving the delay information, adjusting a triggering moment of the image sensor based on the delay information;

In response to reaching the trigger moment, sending a trigger signal to the image sensor; and

In response to receiving the image data collected by the image sensor, second time information is added to the image data to align the image data with the point cloud data collected by the radar sensor using the second time information.
A cloud system configured to:

Obtain the data packet sequence collected by the radar sensor and the image data sequence collected by the image sensor;

For each data packet in the data packet sequence, determine the image data matching each data packet according to the time information of each data packet and the time information of each image data in the image data sequence;

Determine the synchronization relationship between each data packet and the matching image data according to the difference in time information and the difference threshold between each data packet and the matching image data; and

The synchronization relationship between the radar sensor and the image sensor is determined according to the synchronization relationship between each of the plurality of data packets in the data packet sequence and the matching image data.
A device for synchronously collecting data, including:

A time information determination module, configured to determine the first time information of the radar sensor collecting point cloud data at the predetermined angle in response to receiving a data packet from the radar sensor for a predetermined angle; wherein the predetermined angle is in the image Within the viewing angle of the sensor;

a delay information determination module, configured to determine delay information for the image sensor according to the first time information; and

an information sending module, configured to send the delay information to the controller so that the controller controls the image sensor to collect data synchronously with the radar sensor rotated to the predetermined angle,

Wherein, the data package includes point cloud data collected by the radar sensor at the predetermined angle.
The device according to claim 19, wherein the time information determination module includes:

A parsing sub-module, used to parse the data packet to obtain the time data in the data packet; wherein the time data includes a reference time and a periodic encapsulation time; and

A first determination sub-module, configured to determine the first time information of the radar sensor collecting point cloud data at the predetermined angle based on the reference time and the periodic packaging time,

Wherein, the reference time is aligned with the positioning device time of the positioning device in the vehicle where the radar sensor is located.
The device according to claim 19, wherein the delay information determining module includes:

The second determination sub-module is used to determine the second time information for the radar sensor to collect point cloud data at 0° based on the first time information, the rotation period of the radar sensor and the predetermined angle;

A deviation value determination submodule, configured to determine the rotation deviation value of the radar sensor based on the remainder obtained by taking the remainder of the rotation period based on the second time information; and

A delay determination submodule, configured to determine delay information for the image sensor according to the rotation deviation value.
The device according to claim 21, wherein the delay determination sub-module is used to:

Determine the delay information of the image sensor according to the sum of the predetermined error value and the rotation deviation value,

Wherein, the value of the predetermined error value is greater than 0, and the predetermined error value is determined based on the optical center position of the image sensor and the position of the target object within the viewing angle range of the image sensor.
The apparatus of claim 19, wherein the controller includes an artificial intelligence chip.
A device for synchronous data collection, including:

a time adjustment module, configured to respond to receiving delay information for the image sensor and adjust the triggering time of the image sensor according to the delay information;

A signal sending module, configured to send a trigger signal to the image sensor in response to reaching the trigger moment; and

a time adding module, configured to add time information to the image data in response to receiving the image data collected by the image sensor, so as to utilize the time information to align the image data with the point cloud data collected by the radar sensor,

Wherein, the delay information is determined based on the time information of the radar sensor collecting point cloud data at a predetermined angle, and the predetermined angle is within the viewing angle range of the image sensor.
The device according to claim 24, wherein the delay information indicates a delay duration; the time adjustment module includes:

The adjustment information determination submodule is used to determine the number of times to adjust the triggering moment and the step size for each adjustment based on the acquisition frame rate of the image sensor and the delay length; and

The adjustment submodule is used to adjust the triggering moment of the image sensor according to the number of times and the step size.
The device according to claim 25, wherein the adjustment information determination sub-module includes:

an adjustment interval determination unit, configured to determine the value interval of the adjustment step at the triggering moment according to the acquisition frame rate of the image sensor; and

an information determination unit, configured to determine the number of times and the step size according to the value interval and the delay length,

Wherein, the collection frame rate is negatively correlated with the length of the value interval.
The device according to claim 24, wherein the time adjustment module is used for:

In response to receiving the delay information and the delay information changing relative to adjacent delay information in the received delay information, adjusting the triggering moment of the image sensor according to the change value of the delay information.
A device for synchronous data collection, including:

A time information determination module, configured to determine the first time information of the radar sensor collecting point cloud data at the predetermined angle in response to receiving a data packet from the radar sensor for a predetermined angle; wherein the predetermined angle is in the image Within the viewing angle of the sensor;

A delay information determination module, configured to determine delay information for the image sensor according to the first time information;

A time adjustment module, configured to adjust the triggering time of the image sensor according to the delay information, so that the image sensor collects data synchronously with the radar sensor rotated to the predetermined angle;

a signal sending module, configured to send a trigger signal to the image sensor in response to reaching the trigger moment; and

A time adding module, configured to add second time information to the image data in response to receiving the image data collected by the image sensor, so as to use the second time information to combine the image data with the image data collected by the radar sensor. Point cloud data alignment,

Wherein, the data package includes point cloud data collected by the radar sensor at the predetermined angle.
A synchronization determination device, including:

The data acquisition module is used to obtain the data packet sequence collected by the radar sensor and the image data sequence collected by the image sensor;

A data matching module, configured to determine, for each data packet in the data packet sequence, the time information associated with each data packet according to the time information of each data packet and the time information of each image data in the image data sequence. Package matching image data;

A data relationship determination module, configured to determine a synchronization relationship between each data packet and the matching image data based on the difference in time information between each data packet and the matching image data and a difference threshold; and

A synchronization relationship determination module is configured to determine a synchronization relationship between the radar sensor and the image sensor based on the synchronization relationship between each of the plurality of data packets in the data packet sequence and the matching image data.
The device of claim 29, further comprising:

A prompt information sending module, configured to send prompt information to the target object in response to the synchronization relationship between the radar sensor and the image sensor indicating out-of-synchronization.
The device according to claim 29, further comprising a threshold determination module for determining the difference threshold; the threshold determination module includes:

The data fusion submodule is used to obtain fusion data based on the historical data packets collected by the radar sensor and the historical image data matching the historical data packets; the fusion data represents the point cloud data in the historical data packets projected to The position deviation value between the pixel point in the image coordinate system and the pixel point in the historical image data; and

A threshold determination submodule, configured to determine the difference in time information between the historical data packet and the historical image data as the difference threshold when the position deviation value reaches a predetermined deviation value.
The device according to claim 31, wherein the data fusion sub-module is used for:

A predetermined containerization tool is used to fuse the point cloud data in the historical data package and the historical image data to obtain the fused data.
An electronic device including:

at least one processor; and

a memory communicatively connected to the at least one processor; wherein,

The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor, so that the at least one processor can execute any one of claims 1 to 14. Methods.
A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 14.
A computer program product, including a computer program, the computer program is stored on at least one of a readable storage medium and an electronic device, and when executed by a processor, the computer program implements the method according to any one of claims 1 to 14 method described in the item.