WO2020258901A1

WO2020258901A1 - Method and apparatus for processing data of sensor, electronic device, and system

Info

Publication number: WO2020258901A1
Application number: PCT/CN2020/076813
Authority: WO
Inventors: 邓龙
Original assignee: 上海商汤智能科技有限公司
Priority date: 2019-06-25
Filing date: 2020-02-26
Publication date: 2020-12-30
Also published as: JP7164721B2; CN112214009B; CN112214009A; KR20210087495A; JP2022513780A

Abstract

A method and apparatus for processing data of a sensor, an electronic device, and a system. The method comprises: obtaining first target data of a first sensor of a smart device at a first moment (S201); obtaining second target data of a second sensor of the smart device at a second moment, the first moment and the second moment being different moments on a clock of the smart device (S202); obtaining a first pose at the first moment and a second pose at the second moment of the smart device, the first pose and the second pose being different (S203); and performing compensation processing on the first target data according to the first pose and the second pose to obtain compensation data of the first sensor at the second moment (S204).

Description

Sensor data processing method, device, electronic equipment and system

Cross references to related applications

This application is based on a Chinese patent application with an application number of 201910556258.2 and an application date of June 25, 2019, and claims the priority of the Chinese patent application. The entire content of the Chinese patent application is hereby incorporated into this application by way of introduction.

Technical field

The embodiments of the present disclosure relate to smart driving technology, and in particular to a sensor data processing method, device, electronic equipment and system.

Background technique

Assisted driving and automatic driving are two important technologies in the field of intelligent driving. Through assisted driving or automatic driving, the occurrence of traffic accidents can be reduced, so they play an important role in the field of intelligent driving. The implementation of assisted driving technology and autonomous driving technology requires the cooperation of multiple sensors. A variety of sensors are set at different positions of the vehicle, and real-time collection of road images, vehicle operation data, etc., and the auxiliary driving system or automatic driving system performs path planning and other control operations based on the data collected by each sensor. Since the trigger time and trigger source of the sensors installed on the vehicle may be different, there may be a problem of out-of-synchronization between multiple sensors and multiple sensors in each sensor.

Summary of the invention

The embodiments of the present disclosure provide a sensor data processing method, device, electronic equipment and system.

In a first aspect, an embodiment of the present disclosure provides a sensor data processing method, including: acquiring first target data of a first sensor of a smart device at a first time; acquiring a second sensor of the smart device at a second time Two target data, the first moment and the second moment are different moments under the clock of the smart device; acquiring the first pose of the smart device at the first moment and the second moment In the second pose, the first pose and the second pose are different; according to the first pose and the second pose, the first target data is compensated to obtain the Compensation data of the first sensor at the second moment.

In a second aspect, an embodiment of the present disclosure further provides a sensor data processing device, including: a first acquisition module for acquiring first target data of a first sensor of a smart device at a first moment; a second acquisition module for Acquire the second target data of the second sensor of the smart device at the second time, the first time and the second time are different time under the clock of the smart device; the third acquisition module is used to acquire The first pose of the smart device at the first moment and the second pose at the second moment, the first pose and the second pose are different; the compensation module is used for In the first pose and the second pose, compensation processing is performed on the first target data to obtain the compensation data of the first sensor at the second moment.

In a third aspect, embodiments of the present disclosure also provide an intelligent driving control method, including: an intelligent driving control device acquires detection data of a sensor provided on a smart device, and the detection data uses the sensor data described in the first aspect above The processing method is obtained; the smart driving control device performs smart driving control on the smart device according to the detection data.

In a fourth aspect, an embodiment of the present disclosure further provides an intelligent driving control device, including: an acquisition module, configured to acquire detection data of a sensor set on a smart device, and the detection data uses the sensor described in the first aspect. The data processing method is obtained; the intelligent driving control module is used for intelligent driving control of the smart device according to the detection data.

In a fifth aspect, embodiments of the present disclosure further provide an electronic device, including: a memory, configured to store program instructions; a processor, configured to call and execute the program instructions in the memory, and execute the method described in the first aspect above step.

In a sixth aspect, embodiments of the present disclosure also provide an intelligent driving system, including: a sensor connected in communication, the electronic device described in the fifth aspect, and the intelligent driving control device in the fourth aspect.

In a seventh aspect, the embodiments of the present disclosure also provide a readable storage medium in which a computer program is stored, and the computer program is used to execute the method steps described in the first aspect; or, the The computer program is used to execute the method steps described in the third aspect.

According to the sensor data processing method, device, electronic device and system provided by the embodiments of the present disclosure, the first target data of the first sensor at the first time and the second target data of the second sensor at the second time are acquired according to the smart device The pose information at the first moment and the pose information of the smart device at the second moment are compensated for the first target data to obtain the compensation data of the first sensor at the second moment. Since the second target data of the second sensor is also the monitoring data at the second time, that is, by performing compensation processing on the data of the first sensor, the data of the first sensor at the corresponding time of the second sensor can be obtained, that is, the obtained first sensor The first target data of the second sensor and the second target data of the second sensor are data at the same time, so as to achieve synchronization of the first sensor and the second sensor. The method realizes the synchronization of the sensors through software, therefore, there is no need to additionally deploy dedicated hardware for synchronous triggering of multiple sensors, which reduces the hardware cost required for data synchronization of multiple sensors.

Description of the drawings

In order to more clearly explain the technical solutions in the present disclosure or the prior art, the following will briefly introduce the drawings that need to be used in the embodiment or the prior art description. Obviously, the drawings in the following description are the present For some of the disclosed embodiments, for those of ordinary skill in the art, other drawings can be obtained from these drawings without creative labor.

FIG. 1 is a schematic diagram of an application scenario of a sensor data processing method provided by an embodiment of the disclosure;

FIG. 2 is a first flowchart of a sensor data processing method provided by an embodiment of the disclosure;

3 is a schematic diagram of data synchronization between the first sensor and the second sensor through the above process;

FIG. 4 is a second schematic flowchart of a sensor data processing method provided by an embodiment of the disclosure;

FIG. 5 is a third schematic flowchart of a sensor data processing method provided by an embodiment of the disclosure;

6 is a fourth schematic flowchart of a sensor data processing method provided by an embodiment of the disclosure;

FIG. 7 is a fifth schematic flowchart of a sensor data processing method provided by an embodiment of the disclosure;

FIG. 8 is a sixth schematic flowchart of a sensor data processing method provided by an embodiment of the disclosure;

FIG. 9 is an example diagram of performing intra-frame data synchronization on the first original data to obtain the first target data;

FIG. 10 is a seventh schematic flowchart of a sensor data processing method provided by an embodiment of the disclosure;

Figure 11 is an example diagram of synchronizing sensors of the same type;

FIG. 12 is a first module structure diagram of a sensor data processing device provided by an embodiment of the disclosure;

FIG. 13 is a second module structure diagram of a sensor data processing device provided by an embodiment of the disclosure;

FIG. 14 is the third module structure diagram of the sensor data processing device provided by an embodiment of the disclosure;

FIG. 15 is a fourth module structure diagram of a sensor data processing device provided by an embodiment of the disclosure;

FIG. 16 is the fifth module structure diagram of the sensor data processing device provided by the embodiments of the disclosure;

FIG. 17 is a sixth module structure diagram of a sensor data processing device provided by an embodiment of the disclosure;

18 is a schematic structural diagram of an electronic device provided by an embodiment of the disclosure;

FIG. 19 is a schematic flowchart of an intelligent driving control method provided by an embodiment of the disclosure;

20 is a schematic structural diagram of an intelligent driving control device provided by an embodiment of the disclosure;

FIG. 21 is a schematic diagram of a smart driving system provided by an embodiment of the disclosure.

Detailed ways

In order to make the objectives, technical solutions and advantages of the present invention clearer, the technical solutions in the embodiments of the present disclosure will be described clearly and completely in conjunction with the accompanying drawings in the embodiments of the present disclosure. Obviously, the described embodiments are the present invention. Invented some embodiments, but not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative work shall fall within the protection scope of the present invention.

FIG. 1 is a schematic diagram of an application scenario of a sensor data processing method provided by an embodiment of the disclosure. As shown in FIG. 1, the method can be applied to smart devices such as vehicles, robots, and blind guide devices that are installed with sensors. The types of sensors installed on smart devices can include cameras, LiDAR (Light Detection And Ranging, LiDAR), Millimeter Wave Radar (Radio Detection And Ranging, RADAR), high-precision inertial navigation, Controller Area Net-work At least one sensor such as Bus, CANBUS, etc. The number of sensors of the same type set on the smart device may be one or more, for example, one or more cameras, one or more lidars, etc. may be set. Different sensors can be set in different positions of the smart device. Figure 1 shows a camera and LiDAR as an example. Among them, the camera can capture images of the surrounding environment of the smart device in real time, and report the captured images to the smart driving system of the smart device. LiDAR can obtain three-dimensional point coordinates around the smart device by emitting and receiving laser pulses, forming point cloud data, and reporting the point cloud data to the smart driving system of the smart device. RADAR uses electromagnetic waves to detect the ground, vehicles, trees and other objects around the smart device and receive their echoes to obtain the object's position, height, distance and other information, and report it to the smart device's smart driving system. CANBUS transmits the operating parameters of the smart device, such as the vehicle's accelerator operating parameters, steering wheel operating parameters, wheel speed, etc., to the smart driving system of the smart device using serial data transmission. The intelligent driving system performs intelligent driving control based on the data reported by each sensor, such as vehicle positioning, route planning, route deviation warning, and traffic flow analysis.

For ease of description, the following embodiments of the present disclosure refer to the smart driving system of the smart device as a "system" for short.

FIG. 2 is a schematic flowchart 1 of a sensor data processing method provided by an embodiment of the disclosure. As shown in FIG. 2, the method includes:

S201. Acquire first target data of the first sensor of the smart device at the first moment.

S202. Acquire second target data of a second sensor of the smart device at a second time, where the first time and the foregoing second time are different moments under the clock of the smart device.

Optionally, the system may send a data report instruction to the first sensor and the second sensor, and the first sensor and the second sensor detect data after receiving the data report instruction, and send the detected data to the system. Alternatively, the first sensor and the second sensor may also automatically detect data according to a preset cycle, and send the detected data to the system.

S203. Acquire a first pose of the smart device at the first moment and a second pose at the second moment, where the first pose and the second pose are different.

S204: Perform compensation processing on the first target data according to the first pose and the second pose to obtain compensation data of the first sensor at the second moment.

Wherein, the first sensor and the second sensor may be any two sensors on the smart device. One or more sensors of the same type may be provided on the smart device, and the first sensor and the second sensor may be two sensors of different types, or two sensors of the same type.

In an example, the first sensor may be LiDAR, and the second sensor may be a camera.

In the case that the first sensor is LiDAR and the second sensor is a camera, since the data detected by the camera is two-dimensional information, and the data detected by LiDAR is three-dimensional information, the data needs to be rotated and translated in the subsequent compensation processing. Therefore, the second sensor that uses the camera that detects two-dimensional information as a reference to perform motion compensation operations such as rotation and translation on the three-dimensional data detected by LiDAR that detects three-dimensional information can ensure that no additional depth is introduced during the compensation process. error.

In another example, the first sensor and the second sensor may both be LiDAR.

In the embodiments of the present disclosure, taking the smart device as a vehicle as an example, each sensor may use its own independent clock, or may use the same clock as the intelligent driving system of the vehicle. The first time and the second time mentioned in this embodiment both refer to the time under the clock of the intelligent driving system of the vehicle. Each sensor may report the detection time of the data under the sensor's clock to the system by carrying time information in the reported data. In the embodiments of the present disclosure, the time when each sensor sends data is equal to or approximate to the time when the system receives the data. If the clock of the first sensor is the same as the clock of the system, the above-mentioned first time is the time when the first sensor sends data, the time under the clock of the first sensor itself, and the time under the system clock when the system receives data. If the clock of the first sensor itself is different from the clock of the system, the system needs to obtain the first time according to the data sending time under the clock of the first sensor and the clock difference between the first sensor and the system. The process will be implemented as follows Detailed description in the example. The processing method for the above second moment is similar to the above first moment, and will not be repeated here.

The first sensor and the second sensor may be out of synchronization due to factors such as different trigger sources and different trigger moments. For example, the first sensor is LiDAR, and the second sensor is a camera. The LiDAR reports one frame of data to the system every time it rotates. The camera can report data to the system according to its own shooting cycle. Therefore, even if the LiDAR and the camera start working at the same time, they report to the system The data may not be the data at the same time. For example, LiDAR detects a person 100 meters in front of the vehicle, and the camera may capture a person 120 meters in front of the vehicle. Based on this problem, in this embodiment, under the premise that the first time and the second time are both times under the clock of the system, the second sensor is used as the reference sensor of the first sensor, and the system receives the second sensor of the second sensor. The second moment of the second target data is used as the reference moment of the first moment when the first sensor sends data. According to the first pose of the vehicle at the first moment and the second pose of the vehicle at the second moment, the first target data Compensation processing obtains the compensation data of the first sensor at the second moment. Since the second target data of the second sensor is also the detection data at the second time, that is, by performing compensation processing on the data of the first sensor, the data of the first sensor at the corresponding time of the second sensor can be obtained, that is, the obtained first sensor The data of and the data of the second sensor are data at the same time, so as to realize the synchronization of the first sensor and the second sensor. In this way, the synchronization of the sensors is realized through software, so there is no need to deploy additional dedicated hardware for triggering synchronization of multiple sensors, which reduces the hardware cost required for data synchronization of multiple sensors.

Figure 3 is a schematic diagram of data synchronization between the first sensor and the second sensor through the above process. As shown in Figure 3, taking the first sensor as LiDAR and the second sensor as the camera as an example, the first sensor and the second sensor are The data frame reported by LiDAR may be called a radar frame (or LiDAR frame), and the data frame reported by the camera is called a camera frame (or data frame). In a cycle, the system receives a radar frame at time Tl (ie, the first time), and a camera frame at time Tc (ie, the second time), using the camera as a reference sensor, and using the above process to get Tl time The data of the radar frame at time Tc is equivalent to acquiring radar data and camera data at the same time at Tc time, thus realizing the synchronization of LiDAR and camera.

In the above example, each sensor sends detected data to the system in real time, and the data is used to synchronize sensor data. In another scenario, you can also record the data detected by each sensor in advance, and perform data playback on the data recorded by each sensor. During data playback, the data of each sensor needs to be synchronized.

Figure 4 is the second schematic diagram of the synchronization processing of the recorded data of each sensor during data playback. As shown in Figure 4, the detection data of the camera is recorded in advance to obtain a series of camera frames, and each camera frame records the time of detection Pre-record the detection data of CANBUS/high-precision inertial navigation to obtain a series of vehicle operating data, and pre-record the detection data of LiDAR to obtain a series of radar frames. Each radar frame records the time stamp of the detection. During data playback, read a camera frame and a radar frame. At the same time, get the vehicle's pose queue according to the recorded CANBUS/high-precision inertial navigation detection data, and then get the detection according to the time stamp of the camera frame. The pose of the vehicle at the time of the camera frame, the pose when the radar frame was detected according to the time stamp of the radar frame, and the radar frame is compensated to the detection of the camera frame according to the obtained two poses At the moment, the camera frame and radar frame are synchronized, and operations such as driving control can be performed based on the synchronized camera frame and radar frame.

In the specific implementation process, for the multiple sensors on the smart device, one of them can be selected as the second sensor, that is, the reference sensor, and the other sensors are synchronized with the reference sensor, so as to realize the synchronization of the sensors on the smart device.

In this embodiment, after obtaining the first target data of the first sensor at the first moment and the second target data of the second sensor at the second moment, respectively, according to the first pose of the smart device at the first moment and the smart device At the second posture at the second moment, the first target data is compensated to obtain the compensation data of the first sensor at the second moment. Since the second target data of the second sensor is also the monitoring data at the second time, by performing compensation processing on the data of the first sensor, the data of the first sensor at the corresponding time of the second sensor can be obtained, that is, the obtained data of the first sensor The first target data and the second target data of the second sensor are both data at the same time, so as to achieve synchronization of the first sensor and the second sensor. The method realizes the synchronization of the sensors through software, therefore, there is no need to additionally deploy dedicated hardware for synchronous triggering of multiple sensors, which reduces the hardware cost required for data synchronization of multiple sensors.

FIG. 5 is a schematic diagram of the third flow of a sensor data processing method provided by an embodiment of the disclosure. As shown in FIG. 5, the first target data is compensated according to the first pose and the second pose in step S204. Alternative methods include:

S501: Determine the first coordinate system of the first sensor at the first moment according to the first pose;

S502: Determine the second coordinate system of the first sensor at the second moment according to the second pose;

S503: Perform compensation processing on the first target data according to the first coordinate system and the second coordinate system to obtain compensation data of the first sensor at the second time.

Optionally, taking the smart device as a vehicle as an example, the vehicle has a corresponding pose at each moment in the running state, and the pose at different moments may change, and the pose change may include rotation and translation. Based on the pose of the vehicle at each moment, that is, rotation and translation, the pose of the sensor can be obtained. Each pose of the sensor corresponds to a coordinate system, and the point in the world coordinate system detected by the sensor in each pose is a point in the coordinate system, that is, the coordinate value of the detected point is the coordinate The coordinate value under the system. In this embodiment, the vehicle has a corresponding pose at the first moment and the second moment, and further has a first coordinate system corresponding to the first pose and a second coordinate system corresponding to the second pose. For a point in the coordinate system, based on the transformation of the first coordinate system and the second coordinate system, the coordinate value of the point in the second coordinate system can be derived, that is, the coordinate value at the second moment, and the value detected by the first sensor Each point in the data performs the above processing, and the detection data of the data detected by the first sensor at the second time can be obtained.

The following illustrates the process of performing compensation processing on the first target data of the first sensor based on the first coordinate system and the second coordinate system through examples.

Suppose there is a point X in the world coordinate system, the first time is t0, the second time is tn, the pose of the vehicle at t0 is P0, and the coordinate system corresponding to P0 is the first coordinate system. The coordinate data of point X obtained in the first coordinate system at time is x0, and the relationship between x0 and X satisfies the following formula (1):

X=P0*x0 (1)

Assuming that the second time is tn, the pose of the vehicle is Pn, the coordinate system corresponding to Pn is the second coordinate system, and the coordinate data of point X obtained in the second coordinate system at the time tn is xn, then xn and The relationship of X satisfies the following formula (2):

X=Pn*xn (2)

According to the above formula (1) and formula (2), the relationship between x0 and xn satisfies the following formula (3):

P0*x0=Pn*xn (3)

By transforming the above formula (3), xn can be obtained, and xn can be expressed by the following formula (4):

xn=Pn ^-1 *P0*x0 (4)

In the above process, for a point in the world coordinate system, the first sensor at the first time t0 detects the coordinates of the point in the first coordinate system corresponding to the pose at t0, and t0 and tn respectively. When the first sensor detects the point at the second time tn, the coordinates of the point in the second coordinate system corresponding to the pose at the time tn can be derived. The above processing is performed on each point corresponding to the first target data, and then the detection data of the first sensor at time tn can be obtained.

In the above process, the compensation process is performed based on the first pose of the vehicle at the first moment and the second pose of the vehicle at the second moment. Therefore, the first position of the vehicle at the first moment can be obtained before the compensation process. And the second pose of the vehicle at the second moment.

In an alternative embodiment, the pose queue of the smart device may be generated first, and then based on the pose queue of the smart device, the first pose of the smart device at the first moment and the second position of the smart device at the second moment are obtained. posture.

FIG. 6 is a fourth flowchart of a sensor data processing method provided by an embodiment of the present disclosure. As shown in FIG. 6, the pose queue of the smart device is generated, and the first pose and the second pose are determined based on the pose queue of the smart device The process includes:

S601. Obtain respectively the pose detection data detected at multiple times by a sensor with a pose detection function set on the smart device, and each of the multiple times is a time under the clock of the smart device.

S602: Generate a pose queue of the smart device according to the above-mentioned pose detection data.

Optionally, taking the smart device as a vehicle as an example, the sensors with a pose detection function provided on the vehicle may include CANBUS, high-precision inertial navigation and other sensors. The system can receive real-time pose detection data reported by CANBUS, high-precision inertial navigation sensors, such as vehicle wheel speed, steering wheel and other operating data. Based on these pose detection data, the system can calculate the vehicle's pose at multiple moments, and then Build a pose queue.

S603: Determine the first pose and the second pose according to the pose queue of the smart device, the first moment and the second moment.

Optionally, the pose queue of the smart device is composed of poses at each moment. In a specific implementation process, in one case, the first moment may be a moment corresponding to a certain pose in the pose pair, that is, the mapping relationship between the first moment and the pose information directly exists in the pose queue. In this case, the pose information corresponding to the first moment can be directly obtained from the pose queue. The same processing is performed for the pose information sampling at the second moment, that is, if the mapping relationship between the second moment and the pose information directly exists in the pose queue, the pose information corresponding to the second moment can be directly obtained from the pose queue.

In another case, in response to the situation that the pose queue does not include the pose at the first moment, the pose queue of the smart device is compensated according to the first moment to obtain The first pose; and/or, in response to the situation that the pose at the second moment is not included in the pose queue, perform compensation processing on the pose queue of the smart device according to the second moment , To get the second pose.

In this embodiment, the pose information corresponding to the first moment may not exist in the pose queue, and the pose queue of the smart device can be compensated according to the first moment to obtain the first moment of the smart device. Posture. Among them, the compensation processing for the pose queue may be interpolation processing, for example. Exemplarily, the first moment is t3, and there is no pose information corresponding to t3 in the pose queue of the smart device, you can find the two adjacent moments closest to t3 in the pose queue, for example Find time t4 and t5, t4 and t5 are adjacent, at the same time, t3 is the time between t4 and t5. Using the pose information at time t4 and the pose information at time t5, perform interpolation processing to obtain the pose information corresponding to time t3. For the second moment, the above process can also be processed, which will not be repeated here.

In the foregoing embodiment, in the process of performing compensation processing on the first target data of the first sensor according to the first pose and the second pose, the data used is the foregoing first target data. The above-mentioned first target data may refer to unprocessed data directly detected by the first sensor, or the above-mentioned first target data may also be data obtained by pre-synchronizing raw data detected by the first sensor. In some scenarios, for example, when the first sensor is LiDAR, the LiDAR rotates one circle and reports one frame of data to the system after detecting one circle of data. Since LiDAR rotates one circle for a certain period of time, the LiDAR In a LiDAR frame data reported to the system, the actual detection time of each sub-data is different. Among them, one LiDAR frame data may include multiple data packets, and each data packet is a sub-data. In a scenario such as this, before performing compensation processing on the data of the first sensor to achieve data synchronization with the second sensor as the reference sensor, the sub-data in the data sent by the first sensor can be synchronized first to Make each frame of data sent by the first sensor achieve intra-frame synchronization. Wherein, synchronizing the above-mentioned sub-data refers to taking the transmission time of one sub-data in each sub-data as a reference time, and performing compensation processing on the remaining sub-data to obtain sub-data of the remaining sub-data at the reference time.

In an optional manner, the third time point carried when the first sensor reports the first raw data corresponding to the first target data, and the difference information between the clock of the first sensor and the clock of the smart device may first determine the The first moment.

In this embodiment, the first raw data refers to data that has not undergone intra-frame synchronization processing reported by the first sensor to the system, and the first target data refers to data that has undergone intra-frame synchronization processing.

When the first sensor reports the first raw data, the time of detecting the data may be carried in the first raw data, that is, the aforementioned third time.

Then, before acquiring the first target data of the first sensor of the smart device at the first moment, after the system receives the first raw data carrying the third moment, according to the difference information between the clock of the first sensor and the clock of the smart device, It can be determined that the third time is the first time under the clock of the smart device, the third time is used to identify the time when the first sensor detects the first raw data corresponding to the first target data, and the third time is the first sensor Time under the clock.

Optionally, the difference information between the clock of the first sensor and the clock of the smart device may be obtained in advance through a specific means. Exemplarily, when the first sensor is based on the Global Positioning System (Global Positioning System, GPS), the difference information between the clock of the first sensor and the clock of the smart device may be determined according to the error between the GPS clock and the clock of the smart device.

In addition, for the above second time, before acquiring the second target data of the second sensor of the smart device at the second time, the second sensor may also be determined according to the difference information between the clock of the second sensor and the clock of the smart device. The fourth time carried by the second sensor when reporting the second target data is the second time under the smart device clock, and the fourth time is used to identify the detection time of the second target data. The fourth time It is the time under the clock of the second sensor.

In an example, if the above-mentioned second sensor is a sensor capable of detecting video frames, such as a camera, the second sensor can be used to shoot multiple video frames of the stopwatch of the smart device, and the time information of each video frame and each The time information displayed by the stopwatch corresponding to the video frame is compared and analyzed to obtain the difference information between the clock of the second sensor and the clock of the smart device.

The following describes the process of performing intra-frame data synchronization based on the third moment carried when the first sensor reports the first original data.

FIG. 7 is a schematic flow chart 5 of the sensor data processing method provided by an embodiment of the disclosure. As shown in FIG. 7, the process of performing intra-frame data synchronization on the first raw data to obtain the first target data includes:

S701. Receive first raw data reported by the first sensor, where the first raw data includes multiple sub-data, and each sub-data has a corresponding detection time. The third time is the detection of multiple data included in the first raw data. The reference moment of the moment.

Optionally, the first raw data reported by the first sensor includes multiple sub-data, and carries the detection time of each sub-data at the same time, that is, each sub-data has a corresponding detection time. The system can select one of these detection moments as the reference moment as the third moment; respectively compensate the sub-data at other moments to the reference moment, so as to obtain all the sub-data at the reference moment, and these sub-data are combined into The data is the first target data, so as to realize the intra-frame data synchronization of the first sensor. Exemplarily, the latest one of the multiple detection moments may be selected as the third moment.

S702. Obtain the first target data of the first sensor at the first time according to the pose of the smart device at the detection time of each sub-data and each sub-data.

In this embodiment, after receiving the first raw data reported by the first sensor, based on the pose of each sub-data that constitutes the first raw data at the detection time, the frame of multiple sub-data in the first raw data can be completed Internal synchronization, thereby further improving the accuracy of sensor synchronization.

FIG. 8 is a schematic flow chart 6 of a sensor data processing method provided by an embodiment of the disclosure. As shown in FIG. 8, an optional implementation manner of the foregoing step S702 includes:

S801: Determine the coordinate system of the first sensor at the detection time of each sub-data according to the pose of the smart device at the detection time of each sub-data.

The process of determining the coordinate system at each detection moment is the same as the process of determining the first coordinate system at the first moment described above, and reference may be made to the foregoing embodiment, which will not be repeated here.

S802. According to the coordinate system of the first sensor at the detection time of each sub-data except the third time and the coordinate system of the first sensor at the third time, respectively determine that each sub-data except the third time is The sub-data corresponding to the third moment above.

In this step, the third time is used as the reference time, and the sub-data of other detection time is compensated to the third time to realize the synchronization of the sub-data.

Among them, the process of determining the sub-data corresponding to the sub-data at the third time according to the coordinate system at the detection time of the sub-data and the coordinate system at the third time is consistent with the processing in the above step S503. You can refer to the above step S503 and formula (1 The description from) to (4) will not be repeated here.

S803: Perform integration processing on the sub-data corresponding to each sub-data at the third time except the third time to obtain the first target data of the first sensor at the first time.

Optionally, the sub-data corresponding to each sub-data at the third time can be sorted and combined according to the original detection time to obtain the above-mentioned first target data. All sub-data in the first target data are corresponding to the third time. The data in the coordinate system, that is, all the sub-data in the first target data are synchronous data. Under the clock of the first sensor, the first target data is the data in the coordinate system corresponding to the third time. As can be seen from the foregoing, the time at the third time under the clock of the smart device is the first time. Therefore, in the smart device Under the clock of, the above-mentioned first target data is the detection data at the first moment.

Figure 9 is an example diagram of performing intra-frame data synchronization on the first raw data to obtain the first target data. As shown in Figure 9, taking the first sensor as LiDAR as an example, the LiDAR includes n in each frame of data reported. +1 data packet, each data packet corresponds to a detection time, you can use the detection time of the nth data packet (data packet n) as the reference time to compensate data packets 0 to n-1 to the data packets respectively n corresponds to the time, so as to realize the intra-frame synchronization of one frame of data.

The above describes the process of performing intra-frame synchronization on the first original data composed of multiple sub-data to obtain the first target data. In the specific implementation process, if the smart device contains multiple sensors of the same type, and the detection data of the same type of sensor is in the above-mentioned form including multiple sub-data, the first sensor is one of multiple sensors of the same type. , Before synchronizing the first sensor with the second sensor, the detection data of multiple sensors of the same type may be synchronized based on the multiple sub-data synchronization method described above.

FIG. 10 is a schematic flow diagram 7 of a sensor data processing method provided by an embodiment of the disclosure. As shown in FIG. 10, the process of synchronizing detection data of multiple sensors of the same type includes:

S1001. Receive second raw data reported by a third sensor. The type of the third sensor is the same as that of the first sensor. The second raw data includes multiple sub-data. Each sub-data has a corresponding detection time. The third time is the reference time of the detection time of the multiple data included in the second original data;

S1002, according to the pose of the smart device at the detection time of each sub-data of the second raw data and each sub-data of the second raw data, obtain the third target data of the third sensor at the first time.

In this embodiment, one of the detection moments of the first sensor, that is, the third moment, is used as the reference moment when the third sensor is synchronized. According to the pose of each sub-data in the second raw data reported by the third sensor, the Each sub-data in the second original data is compensated to the third moment. After this processing, all the sub-data in the second original data are synchronized to obtain the synchronized third target data. At the same time, due to the third sensor The third time of the first sensor is used as the reference time. Therefore, the third target data synchronized by the third sensor and the second target data synchronized by the first sensor are synchronized.

Figure 11 is an example diagram of synchronizing sensors of the same type. As shown in Figure 11, taking the first sensor as LiDAR, the third sensor and the fourth sensor as LiDAR as an example, each frame of data reported by each LiDAR , Including n+1 data packets, each data packet corresponds to a detection time. Then, the detection time of the nth data packet (data packet n) of the first sensor can be used as the reference time, and the data packet 0 to data packet n-1 of the first sensor can be compensated to the corresponding time of data packet n, and, Compensate data packet 0 to data packet n of the third sensor to the corresponding time of data packet n of the first sensor, and respectively compensate data packet 0 to data packet n of the fourth sensor to data packet n of the first sensor At the corresponding time, the intra-frame synchronization of the first sensor, the third sensor, and the fourth sensor, and the inter-frame synchronization between the first sensor, the third sensor, and the fourth sensor are realized.

In this embodiment, when there are multiple sensors of the same type on the smart device, and the detection data reported by this type of sensor includes multiple sub-data, the detection time of one sub-data in the detection data of one sensor can be used as a reference At time, for each of the remaining sensors of the same type, all sub-data of the sensor are compensated to the reference time. After this processing, not only the intra-frame synchronization of each sensor can be realized, but also the same time can be realized. Interframe synchronization between types of sensors.

In some optional embodiments, the compensation processing is performed on the first target data according to the first pose and the second pose to obtain the data of the first sensor at the second moment The compensation data includes: performing compensation processing on the first target data and the third target data according to the first pose and the second pose to obtain that the first sensor is at the second moment And the compensation data of the third sensor at the second moment.

After the synchronization between the third sensor and the first sensor of the same type is achieved through the above process, while the first target data is compensated according to the first pose and the second pose, it can also be based on the first pose and second pose. The pose performs compensation processing on the third target data to obtain the compensation data of the first sensor at the second moment and the compensation data of the third sensor at the second moment. Since the first sensor and the third sensor are the same type of sensors, synchronization has been achieved, and on this basis, synchronization with the second sensor can further improve the accuracy of sensor synchronization.

FIG. 12 is the first module structure diagram of a sensor data processing device provided by an embodiment of the disclosure. As shown in FIG. 12, the device includes:

The first acquiring module 1201 is configured to acquire the first target data of the first sensor of the smart device at the first moment;

The second acquisition module 1202 is configured to acquire second target data of the second sensor of the smart device at a second time, where the first time and the second time are different time under the clock of the smart device;

The third acquiring module 1203 is configured to acquire the first pose of the smart device at the first moment and the second pose at the second moment, the first pose and the second pose different;

The compensation module 1204 is configured to perform compensation processing on the first target data according to the first pose and the second pose to obtain compensation data of the first sensor at the second time.

The device is used to implement the foregoing method embodiments, and its implementation principles and technical effects are similar, and will not be repeated here.

In another embodiment, the compensation module 1204 is configured to determine the first coordinate system of the first sensor at the first moment according to the first pose; determine the first coordinate system according to the second pose The second coordinate system of the first sensor at the second time; according to the first coordinate system and the second coordinate system, the first target data is compensated to obtain the first sensor at the Describe the compensation data at the second moment.

In another embodiment, the third acquisition module 1203 is configured to respectively acquire the pose detection data of the smart device detected at multiple times by the sensor with the pose detection function set on the smart device. Each of the two moments is a moment under the clock of the smart device; according to the pose detection data of the smart device detected at multiple moments by the sensor with pose detection function set on the smart device, generate The pose queue of the smart device; the first pose and the second pose are determined according to the pose queue of the smart device, the first moment and the second moment.

In another embodiment, the third acquiring module 1203 is configured to respond to the situation that the pose at the first moment is not included in the pose queue, and perform an assessment of the pose of the smart device according to the first moment. The queue performs compensation processing to obtain the first pose; and/or, in response to the situation that the pose at the second moment is not included in the pose queue, the smart device is processed according to the second moment Perform compensation processing on the pose queue to obtain the second pose.

FIG. 13 is the second module structure diagram of the sensor data processing device provided by the embodiment of the disclosure. As shown in FIG. 13, the device further includes: a first determining module 1205, which is used to determine the relationship between the clock of the first sensor and the smart device. The difference information of the clock of the device determines that the third time is the first time under the smart device clock, and the third time is used to identify the first original data corresponding to the first target data detected by the first sensor. The time of the data, the third time is the time under the clock of the first sensor.

In another embodiment, the first acquisition module 1201 is configured to receive the first raw data reported by the first sensor, the first raw data includes multiple sub-data, and each sub-data has a corresponding detection time, The third time is the reference time of the detection time of the multiple sub-data included in the first raw data; the first sensor is acquired according to the pose of the smart device at the detection time of each sub-data and each sub-data The first target data at the first moment.

In another embodiment, the first acquisition module 1201 is configured to determine the coordinate system of the first sensor at the detection time of each sub-data according to the pose of the smart device at the detection time of each sub-data; The coordinate system of the first sensor at the detection time of each sub-data except the third time and the coordinate system of the first sensor at the third time, respectively determine each sub-data except the third time The sub-data corresponding to the third time; the sub-data corresponding to each sub-data at the third time except the third time is integrated to obtain the data of the first sensor at the first time The first target data.

FIG. 14 is the third module structure diagram of the sensor data processing device provided by the embodiment of the disclosure. As shown in FIG. 14, the device further includes: a receiving module 1206 for receiving the second raw data reported by the third sensor. The type of the three sensors is the same as the type of the first sensor, the second raw data includes multiple sub-data, each sub-data has a corresponding detection time, and the third time is the multiple included in the second raw data. The reference time of the detection time of the sub-data;

The fourth acquisition module 1207 is configured to acquire the location of the third sensor at the time of detection of each sub-data of the second raw data and each sub-data of the second raw data according to the smart device. The third target data at the first moment;

The compensation module 1204 is configured to perform compensation processing on the first target data and the third target data according to the first pose and the second pose, to obtain that the first sensor is in the second The compensation data at the time and the compensation data of the third sensor at the second time.

15 is the fourth module structure diagram of the sensor data processing device provided by the embodiments of the disclosure. As shown in FIG. 15, the device further includes: a fifth acquisition module 1208, which is used by the first determination module 1205 according to the first The difference information between the clock of the sensor and the clock of the smart device determines that the third time is before the first time under the smart device clock, and obtains the first time according to the clock error between the GPS clock and the smart device Difference information between the clock of the sensor and the clock of the smart device.

FIG. 16 is the fifth module structure diagram of the sensor data processing device provided by the embodiments of the disclosure. As shown in FIG. 16, the device further includes: a second determining module 1209, which is used for the second acquiring module 1202 to acquire the first determination of the smart device Before the second target data at the second time, the second sensor determines the fourth carried when the second sensor reports the second target data according to the difference information between the clock of the second sensor and the clock of the smart device. The time is the second time under the smart device clock, the fourth time is used to identify the detection time of the second target data, and the fourth time is the time under the clock of the second sensor.

FIG. 17 is a module structure diagram 6 of the sensor data processing device provided by an embodiment of the disclosure. As shown in FIG. 17, the device further includes: a photographing module 1210 for photographing the stopwatch of the smart device using the second sensor Multiple video frames;

The analysis module 1211 is used to compare and analyze the time information of each video frame and the time information displayed by the stopwatch corresponding to each video frame to obtain the clock of the second sensor and the smart Difference information of the device's clock.

In another embodiment, the second sensor is a camera, and the first sensor is a lidar or millimeter wave radar.

It should be noted that it should be understood that the division of the various modules of the above device is only a division of logical functions, and may be fully or partially integrated into a physical entity in actual implementation, or may be physically separated. And these modules can all be implemented in the form of software called by processing elements; they can also be implemented in the form of hardware; some modules can be implemented in the form of calling software by processing elements, and some of the modules can be implemented in the form of hardware. For example, the determining module may be a separately established processing element, or it may be integrated into a certain chip of the above-mentioned device for implementation. In addition, it may also be stored in the memory of the above-mentioned device in the form of program code, and a certain processing element of the above-mentioned device Call and execute the functions of the above-identified module. The implementation of other modules is similar. In addition, all or part of these modules can be integrated together or implemented independently. The processing element described here may be an integrated circuit with signal processing capability. In the implementation process, each step of the above method or each of the above modules can be completed by hardware integrated logic circuits in the processor element or instructions in the form of software.

For example, the above modules may be one or more integrated circuits configured to implement the above methods, such as: one or more application specific integrated circuits (ASIC), or one or more microprocessors (Digital Signal Processor, DSP), or, one or more Field Programmable Gate Array (Field Programmable Gate Array, FPGA), etc. For another example, when one of the above modules is implemented in the form of processing element scheduling program code, the processing element may be a general-purpose processor, such as a central processing unit (CPU) or other processors that can call program codes. For another example, these modules can be integrated together and implemented in the form of a System-On-a-Chip (SOC).

In the above embodiments, it may be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in the form of a computer program product in whole or in part. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on the computer, the procedures or functions described in the embodiments of the present disclosure are generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website, computer, server, or data center. Transmission to another website site, computer, server or data center via wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.). The computer-readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server or data center integrated with one or more available media. The usable medium may be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a solid state disk (SSD)), etc.

FIG. 18 is a schematic structural diagram of an electronic device provided by an embodiment of the disclosure. As shown in FIG. 18, the electronic device 1800 may include: a processor 181 and a memory 182; the memory 182 is used to store computer instructions, and when the processor 181 executes the computer instructions, the implementation is as shown in FIGS. 1 to 10 above. Show the scheme of the embodiment.

Optionally, the electronic device 1800 may further include a communication interface 183 for communicating with other devices. It can be understood that the electronic device 1800 may further include a system bus 184, and the system bus 184 is used to implement connection and communication between these components.

Among them, the system bus 184 may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The system bus 184 can be divided into an address bus, a data bus, a control bus, and the like. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus. The communication interface 183 is used to implement communication between the database access device and other devices (for example, a client, a read-write library, and a read-only library). The memory 182 may include a random access memory (Random Access Memory, RAM), and may also include a non-volatile memory (Non-Volatile Memory), such as at least one disk memory.

The aforementioned processor 181 may be a general-purpose processor, including a CPU, a network processor (Network Processor, NP), etc.; it may also be a DSP, ASIC, FPGA or other programmable logic device, discrete gate or transistor logic device, discrete hardware component .

FIG. 19 is a schematic flow chart of the intelligent driving control method provided by the embodiments of the present disclosure. On the basis of the above-mentioned embodiments, the embodiments of the present disclosure also provide an intelligent driving control method, including:

S1901, respectively obtain detection data of a first sensor and a second sensor set on the smart device, and the detection data is obtained by using the sensor data processing method provided in the embodiment of the present disclosure;

S1902 Perform intelligent driving control on the smart device according to the detection data.

The execution subject of this embodiment may be a smart driving control device. The smart driving control device of this embodiment and the electronic equipment described in the above embodiments may be located in the same device or separate devices in different devices. Among them, the intelligent driving control device of this embodiment is in communication connection with the above-mentioned electronic equipment.

Wherein, the detection data of the first sensor and the second sensor are obtained by the method of the above-mentioned embodiment, and the specific process is referred to the description of the above-mentioned embodiment, which will not be repeated here.

Specifically, the electronic device executes the above-mentioned sensor data processing method, obtains detection data of the first sensor and the second sensor set on the smart device, and outputs the detection data of the first sensor and the second sensor set on the smart device . The intelligent driving control device obtains the detection data of the first sensor and the second sensor, and performs intelligent driving control on the smart device according to the detection data.

The smart driving in this embodiment includes assisted driving, automatic driving, and/or driving mode switching between auxiliary driving and automatic driving, and the like. The above-mentioned intelligent driving control may include: braking, changing driving speed, changing driving direction, maintaining lane line, changing the state of lights, switching driving mode, etc., wherein the driving mode switching may be a switching between assisted driving and automatic driving, for example , Switch assisted driving to automatic driving through intelligent driving control.

In the smart driving control method provided in this embodiment, the smart driving control device obtains the detection data of the sensor set on the smart device, and performs smart driving control based on the detection data of the sensor set on the smart device, thereby improving the smart driving performance. Safety and reliability.

FIG. 20 is a schematic structural diagram of an intelligent driving control device provided by an embodiment of the disclosure. As shown in FIG. 20, on the basis of the above embodiment, the intelligent driving control device 2000 of the embodiment of the present disclosure includes: an acquisition module 2001 for Acquiring detection data of the first sensor and the second sensor set on the smart device respectively, and the detection data is obtained by using the aforementioned sensor data processing method;

The intelligent driving control module 2002 is used to perform intelligent driving control on the smart device according to the detection data.

The intelligent driving control device of the embodiment of the present disclosure may be used to execute the technical solution of the method embodiment shown above, and its implementation principle and technical effect are similar, and will not be repeated here.

FIG. 21 is a schematic diagram of a smart driving system provided by an embodiment of the disclosure. As shown in FIG. 21, the smart driving system 2100 of this embodiment includes: a sensor 2101, an electronic device 1800, and a smart driving control device 2000 connected in communication, wherein the electronic device 1800 is shown in FIG. 18, and the intelligent driving control device 2000 is shown in FIG. 20. The sensor 2101 may include at least one of the cameras, LiDAR, RADAR, high-precision inertial navigation and other sensors described in the foregoing embodiments.

Specifically, as shown in FIG. 21, in actual use, the sensor 2101 detects the surrounding environment of the smart device to obtain original detection data, and sends these detection data to the electronic device 1800. After the electronic device 1800 receives the original detection data , Perform data synchronization according to the above-mentioned sensor data processing method to obtain synchronized detection data. The electronic device 1800 sends the synchronized detection data to the smart driving control device 2000, and the smart driving control device 2000 performs smart driving control on the smart device according to the synchronized detection data.

The embodiment of the present application also provides a storage medium, the storage medium stores instructions, which when run on a computer, causes the computer to execute the method of any one of the embodiments shown in FIGS. 1 to 10; or The computer executes the method of the embodiment shown in FIG. 19 above.

An embodiment of the present application also provides a chip for executing instructions. The chip is used to execute the method of any one of the embodiments shown in FIG. 1 to FIG. 10; or, the chip is used to execute the method of the embodiment shown in FIG. method.

An embodiment of the present application further provides a program product, the program product includes a computer program, the computer program is stored in a storage medium, at least one processor can read the computer program from the storage medium, and the at least one When the processor executes the computer program, the method of any one of the embodiments shown in FIGS. 1 to 10 may be implemented; or, when the at least one processor executes the computer program, the method of the embodiment shown in FIG. 19 may be implemented .

In the embodiments of the present disclosure, "at least one" refers to one or more, and "multiple" refers to two or more. "And/or" describes the association relationship of the associated objects, indicating that there can be three relationships, for example, A and/or B, which can mean: A alone exists, both A and B exist, and B exists alone, where A, B can be singular or plural. The character "/" generally indicates that the associated objects before and after are in an "or" relationship; in the formula, the character "/" indicates that the associated objects before and after are in a "division" relationship. "The following at least one item (a)" or similar expressions refers to any combination of these items, including any combination of a single item (a) or plural items (a). For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple A.

It can be understood that the various numerical numbers involved in the embodiments of the present disclosure are only for easy distinction for description and are not used to limit the scope of the embodiments of the present disclosure.

It can be understood that, in the embodiments of the present disclosure, the size of the sequence numbers of the foregoing processes does not mean the order of execution. The execution order of the processes should be determined by their functions and internal logic, and should not correspond to the embodiments of the present disclosure. The implementation process constitutes any limitation.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the embodiments of the present disclosure, not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand : It can still modify the technical solutions described in the foregoing embodiments, or equivalently replace some or all of the technical features; these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the technology of the embodiments of the present disclosure The scope of the program.

Claims

A sensor data processing method, including:

Acquiring the first target data of the first sensor of the smart device at the first moment;

Acquiring second target data of a second sensor of the smart device at a second moment, where the first moment and the second moment are different moments under a clock of the smart device;

Acquiring a first pose of the smart device at the first moment and a second pose at the second moment, where the first pose and the second pose are different;

According to the first pose and the second pose, performing compensation processing on the first target data to obtain compensation data of the first sensor at the second moment.
The method according to claim 1, wherein the compensation process is performed on the first target data according to the first pose and the second pose to obtain that the first sensor is in the second Compensation data at the moment, including:

Determine the first coordinate system of the first sensor at the first moment according to the first pose;

Determine the second coordinate system of the first sensor at the second moment according to the second pose;

According to the first coordinate system and the second coordinate system, performing compensation processing on the first target data to obtain compensation data of the first sensor at the second time.
The method according to claim 2, wherein said acquiring the first pose of the smart device at the first moment and the second pose at the second moment comprises:

Acquiring respectively the pose detection data detected by a sensor with a pose detection function set on the smart device at multiple moments, each of the multiple moments being a moment under the clock of the smart device;

Generating a pose queue of the smart device according to the pose detection data;

The first pose and the second pose are determined according to the pose queue of the smart device, the first moment and the second moment.
The method according to claim 3, wherein the determining the first pose and the second pose according to the pose queue of the smart device, the first moment and the second moment, include:

In response to the situation that the pose queue at the first moment is not included in the pose queue, perform compensation processing on the pose queue of the smart device according to the first moment to obtain the first pose; and /or,

In response to the situation that the pose at the second moment is not included in the pose queue, compensation processing is performed on the pose queue of the smart device according to the second moment to obtain the second pose.
The method according to any one of claims 1 to 4, wherein before the acquiring the first target data of the first sensor of the smart device at the first moment, the method further comprises:

According to the difference information between the clock of the first sensor and the clock of the smart device, it is determined that the third time is the first time under the clock of the smart device, and the third time is used to identify the first sensor The time when the first raw data corresponding to the first target data is detected, and the third time is a time under the clock of the first sensor.
The method according to claim 5, wherein said acquiring the first target data of the first sensor of the smart device at the first moment comprises:

Receive the first raw data reported by the first sensor, the first raw data includes multiple sub-data, each sub-data has a corresponding detection time, and the third time is included in the first raw data The reference time of the detection time of multiple sub-data;

The first target data of the first sensor at the first time is acquired according to the pose of the smart device at the detection time of each sub-data and each sub-data.
The method of claim 6, wherein the first target of the first sensor at the first time is obtained according to the pose of the smart device at the detection time of each sub-data and each sub-data Data, including:

Determine the coordinate system of the first sensor at the detection time of each sub-data according to the pose of the smart device at the detection time of each sub-data;

According to the coordinate system of the first sensor at the detection time of each sub-data except the third time and the coordinate system of the first sensor at the third time, determine the coordinates other than the third time. The sub-data corresponding to each sub-data at the third moment;

Perform integration processing on the sub-data corresponding to each sub-data at the third time except the third time, to obtain the first target data of the first sensor at the first time.
7. The method according to any one of claims 5-7, wherein the compensation processing is performed on the first target data according to the first pose and the second pose to obtain the first sensor Before the compensation data at the second moment, the method further includes:

Receive the second raw data reported by the third sensor, the type of the third sensor is the same as the type of the first sensor, the second raw data includes multiple sub-data, each sub-data has a corresponding detection time, so The third time is a reference time of detection time of the multiple sub-data included in the second original data;

Obtain the third target of the third sensor at the first time according to the pose of the smart device at the detection time of each sub-data of the second raw data and each sub-data of the second raw data data;

The performing compensation processing on the first target data according to the first pose and the second pose to obtain the compensation data of the first sensor at the second moment includes:

According to the first pose and the second pose, perform compensation processing on the first target data and the third target data to obtain the compensation data of the first sensor at the second moment, and Compensation data of the third sensor at the second moment.
8. The method according to any one of claims 5-8, wherein the determination of the third moment in the clock of the smart device according to the difference information between the clock of the first sensor and the clock of the smart device Before the first moment, the method further includes:

Determine the difference information between the clock of the first sensor and the clock of the smart device according to the error between the GPS clock of the global positioning system and the clock of the smart device.
The method according to any one of claims 1-9, wherein before the acquiring the second target data of the second sensor of the smart device at the second moment, the method further comprises:

According to the difference information between the clock of the second sensor and the clock of the smart device, it is determined that the fourth time carried when the second sensor reports the second target data is in the second time under the smart device clock. Time, the fourth time is used to identify the detection time of the second target data, and the fourth time is the time under the clock of the second sensor.
The method according to claim 10, wherein said determining the fourth sensor carried when the second sensor reports the second target data according to the difference information between the clock of the second sensor and the clock of the smart device. The time is before the second time under the smart device clock, and the method further includes:

Using the second sensor to shoot multiple video frames of the stopwatch of the smart device;

Compare and analyze the time information of shooting each of the video frames with the time information displayed by the stopwatch corresponding to each of the video frames to obtain the difference information between the clock of the second sensor and the clock of the smart device .
The method according to any one of claims 1-11, wherein the second sensor is a camera, and the first sensor is a lidar or millimeter wave radar.
A sensor data processing device includes:

The first acquisition module is used to acquire the first target data of the first sensor of the smart device at the first moment;

A second acquisition module, configured to acquire second target data of a second sensor of the smart device at a second time, where the first time and the second time are different times under the clock of the smart device;

The third acquisition module is used to acquire the first pose of the smart device at the first moment and the second pose at the second moment, where the first pose and the second pose are different ；

The compensation module is configured to perform compensation processing on the first target data according to the first pose and the second pose to obtain the compensation data of the first sensor at the second moment.
The device according to claim 13, wherein the compensation module is configured to determine the first coordinate system of the first sensor at the first moment according to the first pose; according to the second position Pose, determine the second coordinate system of the first sensor at the second moment; perform compensation processing on the first target data according to the first coordinate system and the second coordinate system to obtain the first Compensation data of a sensor at the second moment.
The device according to claim 14, wherein the third acquisition module is configured to acquire the pose detection data detected at multiple times by the sensor with the pose detection function set on the smart device, and the Each of the multiple times is a time under the clock of the smart device; generates the pose queue of the smart device according to the pose detection data; according to the pose queue of the smart device, the The first moment and the second moment determine the first pose and the second pose.
The device according to claim 15, wherein the third acquiring module is configured to respond to the situation that the pose at the first moment is not included in the pose queue, and perform a check on all the poses according to the first moment. The pose queue of the smart device performs compensation processing to obtain the first pose; and/or, in response to the situation that the pose queue does not include the pose at the second moment, according to the second moment , Performing compensation processing on the pose queue of the smart device to obtain the second pose.
The device according to any one of claims 13-16, wherein the device further comprises:

The first determining module is configured to determine, according to the difference information between the clock of the first sensor and the clock of the smart device, that the third time is the first time under the smart device clock, and the third time is used To identify the time when the first sensor detects the first raw data corresponding to the first target data, the third time is a time under the clock of the first sensor.
The apparatus according to claim 17, wherein the first acquisition module is configured to receive the first raw data reported by the first sensor, the first raw data includes a plurality of sub-data, and each sub-data has A corresponding detection time, the third time is the reference time of the detection time of the multiple sub-data included in the first original data; according to the pose of the smart device at the detection time of each sub-data and each sub-data, Acquiring the first target data of the first sensor at the first moment.
The apparatus of claim 18, wherein the first acquisition module is configured to determine the coordinates of the first sensor at the detection time of each sub-data according to the pose of the smart device at the detection time of each sub-data System; according to the first sensor at the coordinate system of the detection time of each sub-data except the third time and the coordinate system of the first sensor at the third time, respectively determine except for the third time The sub-data corresponding to each sub-data except the third time at the third time; the sub-data corresponding to each sub-data at the third time except the third time is integrated to obtain the first sensor at the The first target data at the first moment.
The device according to any one of claims 17-19, wherein the device further comprises:

The receiving module is configured to receive the second raw data reported by the third sensor, the type of the third sensor is the same as the type of the first sensor, and the second raw data includes multiple sub-data, and each sub-data has a corresponding The detection time of the third time is a reference time of the detection time of the multiple sub-data included in the second original data;

The fourth acquisition module is configured to acquire the position of the third sensor at the detection time of each sub-data of the second raw data and each sub-data of the second raw data according to the smart device The third target data at the first moment;

The compensation module is configured to perform compensation processing on the first target data and the third target data according to the first pose and the second pose, to obtain that the first sensor is in the first position The compensation data at the second time and the compensation data of the third sensor at the second time.
The device according to any one of claims 17-20, wherein the device further comprises:

The fifth acquiring module is used for the first determining module to determine that the third time is before the first time under the smart device clock according to the difference information between the clock of the first sensor and the clock of the smart device According to the error between the GPS clock of the global positioning system and the clock of the smart device, the difference information between the clock of the first sensor and the clock of the smart device is obtained.
The device according to any one of claims 13-21, wherein the device further comprises:

The second determining module is configured to, before the second acquiring module acquires the second target data of the second sensor of the smart device at the second moment, according to the difference information between the clock of the second sensor and the clock of the smart device, It is determined that the fourth time carried when the second sensor reports the second target data is the second time under the smart device clock, and the fourth time is used to identify the detection time of the second target data , The fourth time is a time under the clock of the second sensor.
The device according to claim 22, wherein the device further comprises:

A photographing module, configured to use the second sensor to photograph multiple video frames of the stopwatch of the smart device;

The analysis module is used to compare and analyze the time information of each video frame and the time information displayed by the stopwatch corresponding to each video frame to obtain the clock of the second sensor and the smart device The difference information of the clock.
The device according to any one of claims 13-23, wherein the second sensor is a camera, and the first sensor is a lidar or millimeter wave radar.
An intelligent driving control method, including:

Acquiring detection data of the first sensor and the second sensor set on the smart device respectively, and the detection data is obtained using the sensor data processing method according to any one of claims 1-12;

Perform intelligent driving control on the smart device according to the detection data.
An intelligent driving control device, including:

The acquisition module is configured to separately acquire detection data of the first sensor and the second sensor set on the smart device, and the detection data is obtained by using the sensor data processing method according to any one of claims 1-12;

The intelligent driving control module is used to perform intelligent driving control on the smart device according to the detection data.
An electronic device including:

Memory, used to store computer instructions;

The processor is configured to call and execute computer instructions in the memory, and execute the method steps of any one of claims 1-12.
An intelligent driving system, comprising: a sensor connected in communication, the electronic device according to claim 27, and the intelligent driving control device according to claim 26.
A readable storage medium in which a computer program is stored, and the computer program is used to execute the method steps of any one of claims 1-12; or, the computer program is used to execute rights Method steps described in claim 25.