WO2023143132A1 - Sensor data calibration - Google Patents

Abstract

During a sensor data calibration process, image data, radar data and inertial data at multiple moments during a movement process of a test device are acquired (S100); a motion trajectory of the test device is determined according to the inertial data and the image data, and the movement velocity of the test device under an image reference frame is determined according to the motion trajectory (S102); and the Doppler velocity in the movement process of the test device is determined according to the radar data, and the Doppler velocity and the movement velocity of the test device under the image reference frame are registered so as to calibrate sensor data (S104).

Description

Calibration of sensor data

This application claims the priority of the Chinese patent application with the application number 202210112012.8 and the application name "A Calibration Method and Device for Sensor Data" filed on January 29, 2022, the entire contents of which are incorporated in this application by reference.

technical field

The present application relates to the field of computer technology, in particular to calibration of sensor data.

Background technique

At present, with the development of unmanned driving technology, the driving safety of unmanned driving equipment is becoming more and more important. The obstacle detection and classification method based on the fusion of image data and radar data is widely used in obstacle detection and classification scenarios due to its relatively accurate detection results and classification results. The premise of image data and radar data fusion is the calibration of radar sensor and image sensor.

Contents of the invention

This application provides calibration of sensor data. The application adopts the following technical solutions:

This application provides a calibration method for sensor data, including:

Acquiring sensor data at multiple moments during the movement of the test equipment, the sensor data at least including image data, radar data and inertial data;

determining the motion trajectory of the testing device according to the inertial data and the image data, and determining the moving speed of the testing device in the image reference system according to the motion trajectory;

According to the radar data, determine the Doppler velocity during the moving process of the test equipment, and register the Doppler velocity and the moving velocity of the test equipment in the image reference system, so as to Calibrate the above sensor data.

In some embodiments, according to the inertial data and the image data, determining the motion trajectory of the testing device includes:

According to the inertial data and the image data, determine the angular velocity, acceleration, and observation position of the image marker respectively corresponding to the multiple moments of the test equipment under the inertial reference system;

According to the angular velocities, accelerations and observed positions of the image markers respectively corresponding to the multiple moments, the motion trajectory of the test equipment in the world reference system is solved.

In some implementations, according to the angular velocity, acceleration, and observed position of the image marker corresponding to the multiple moments, the motion trajectory of the test device in the world reference system is solved, including:

According to the motion trajectory to be solved of the test equipment in the world reference system, determine the first parameters to be solved respectively corresponding to the multiple moments, and the first parameters are used to solve the motion trajectory;

For each of the plurality of moments, according to the first parameter to be solved corresponding to the moment, determine the conversion relationship between the world reference frame and the inertial reference frame, and determine the test equipment in the inertial Estimated angular velocity, estimated acceleration, and estimated position of image markers in the frame of reference;

The angular velocity corresponding to this moment is the same as the estimated angular velocity, the acceleration is the same as the estimated acceleration, and the observed position of the image marker is the same as the estimated position, constructing constraint conditions for the motion Find the trajectory untie.

In some implementations, according to the first parameter to be solved corresponding to this moment, the conversion relationship between the world reference frame and the inertial reference frame is determined, and the estimated angular velocity, estimated Estimated positions of acceleration and image markers, including:

According to the pose of the test device in the first parameter to be solved corresponding to this moment, determine the conversion relationship between the world reference frame and the inertial reference frame to be solved at this moment, wherein the first parameter includes The pose, acceleration offset, angular velocity offset and observation position of the image marker of the test equipment;

According to the transformation relationship to be solved, the angular velocity offset to be solved in the first parameter, the acceleration offset to be solved, and the observed position of the image marker to be solved, respectively determine the estimated value of the test device Acceleration, estimated angular velocity, and estimated position of image markers.

In some implementations, determining the moving speed of the test device under the image reference system according to the motion track includes:

According to the movement trajectory, determining the movement speeds of the collection device corresponding to the multiple moments in the world reference system;

According to the poses and postures of the acquisition device at the multiple moments, determine the conversion relationship between the world reference frame and the inertial reference frame respectively corresponding to the multiple moments;

According to the moving speed of the acquisition device at the multiple moments in the world reference system, the conversion relationship between the world reference system and the inertial reference system corresponding to the multiple moments, and the preset inertial reference system The conversion relationship between the image reference system and the image reference system is used to determine the moving speed of the test equipment at the multiple moments in the image reference system.

In some embodiments, registering the Doppler velocity and the moving velocity of the test device in an image reference system to calibrate the sensor data includes:

For each of the plurality of moments, according to the acquired radar data and the preset directional components, determine the Doppler velocity in each directional component at that moment;

According to the moving speed of the test equipment in the image reference system at this moment, the calibration relationship between the radar reference system and the image reference system to be solved, and the preset components in each direction, it is determined that the test equipment is to be solved in each direction component moving speed;

The moving velocity to be solved in each direction component is registered with the Doppler velocity in each direction component, and the calibration relationship is solved, so as to calibrate the sensor data.

In some implementations, according to the moving speed of the test device under the image reference system at this moment, the calibration relationship to be solved between the radar reference system and the image reference system, and the preset direction components, it is determined that the test device is in The moving speed to be solved on each direction component, including:

According to the time difference to be solved between the internal clock of the radar sensor set on the test device and the image sensor, determine the moving speed to be solved of the test device at this moment under the image reference system, and the moving speed to be solved includes The time difference to be resolved;

According to the moving speed of the test equipment to be solved in the image reference system at this moment, the conversion relationship to be solved between the radar reference system and the image reference system, determine the moving speed of the test equipment in the radar reference system to be solved ;

According to each preset direction component, determine the moving speed of the test equipment to be solved on each direction component;

Register the moving speed to be solved on each direction component and the Doppler speed on each direction component, and solve the calibration relationship, including:

The moving velocity to be solved in each direction component is registered with the Doppler velocity in each direction component, and the conversion relationship and the time difference are calculated as the calibration relationship.

In some embodiments, the method also includes:

According to the determined calibration relationship and the collected sensor data, determine the pose gap of the test equipment corresponding to each moment in the image reference frame and the radar reference frame, and judge whether the pose gap is greater than Preset error thresholds;

If the pose difference is greater than the error threshold, it is determined that the sensor data needs to be calibrated, and the sensor data is stored;

If the pose difference is not greater than the error threshold, it is determined that sensor data does not need to be calibrated.

The application provides a calibration device for sensor data, including:

An acquisition module, configured to acquire sensor data at multiple moments during the movement of the test equipment, the sensor data at least including image data, radar data and inertial data;

A trajectory determination module, configured to determine the motion trajectory of the test device according to the inertial data and the image data, and determine the moving speed of the test device in the image reference system according to the motion trajectory;

a calibration module, configured to determine the Doppler velocity during the moving process of the test equipment according to the radar data, and register the Doppler velocity with the moving velocity of the test equipment in the image reference system, to calibrate the sensor data.

In some implementations, the trajectory determination module is configured to determine the angular velocity, acceleration and image markers of the test equipment corresponding to the multiple moments in the inertial reference system according to the inertial data and the image data According to the observation positions of the angular velocities, accelerations, and image markers corresponding to the multiple moments, the motion trajectory of the test equipment in the world reference system is solved.

In some implementations, the trajectory determination module is configured to determine the first parameters to be solved respectively corresponding to the multiple moments according to the motion trajectory to be solved of the test equipment in the world reference system, the first The parameters are used to solve the motion trajectory, and for each of the multiple moments, according to the first parameter to be solved corresponding to the moment, determine the conversion relationship between the world reference frame and the inertial reference frame, and Determine the estimated angular velocity, estimated acceleration, and estimated position of the image marker of the test device in the inertial reference system, and the angular velocity corresponding to this moment is the same as the estimated angular velocity, and the acceleration and The estimated acceleration is the same, the observed position of the image marker is the same as the estimated position, a constraint condition is constructed, and the motion trajectory is solved.

In some implementations, the trajectory determination module is configured to determine the world reference frame and the inertial reference frame to be solved at this moment according to the pose of the test device in the first parameter to be solved corresponding to the moment The conversion relationship between, wherein, the first parameter includes the posture, acceleration offset, angular velocity offset and the observation position of the image marker of the test equipment; according to the conversion relationship to be solved, the first Among the parameters, the angular velocity offset to be solved, the acceleration offset to be solved, and the observed position of the image marker to be solved are determined respectively to determine the estimated acceleration, estimated angular velocity, and estimated position of the image marker of the test device .

In some implementations, the trajectory determination module is configured to determine the moving speeds of the test equipment corresponding to the multiple moments in the world reference system according to the motion trajectory, and the test equipment according to the multiple moments The pose of the device, determine the conversion relationship between the world reference system and the inertial reference system corresponding to the multiple times, according to the moving speed of the test equipment at the multiple times in the world reference system, the The conversion relationship between the world reference system and the inertial reference system corresponding to multiple times and the preset conversion relationship between the inertial reference system and the image reference system determine the position of the test equipment in the image reference system The movement speed at various times described below.

In some implementations, the calibration module is configured to, for each of the multiple moments, determine the Doppler in each direction component at that moment according to the acquired radar data and preset each direction component speed, according to the moment The moving speed of the test equipment in the image reference system, the calibration relationship to be solved between the radar reference system and the image reference system, and the preset components in each direction determine the moving speed of the test equipment in each direction component to be solved. The moving velocity to be solved in each direction component is registered with the Doppler velocity in each direction component, and the calibration relationship is solved to calibrate the sensor data.

In some embodiments, the calibration module is configured to determine the waiting time of the testing equipment under the image reference system at this moment according to the waiting time difference between the internal clock of the radar sensor and the image sensor set on the testing equipment. The moving speed to be solved, the moving speed to be solved includes the time difference to be solved, according to the moving speed to be solved of the test equipment in the image reference system at this moment, the radar reference system and the image reference system to be solved The conversion relationship of the test equipment to be solved in the radar reference system is determined. According to the preset components in each direction, the moving speed of the test equipment to be solved in each direction component is determined. The moving velocity of the solution is registered with the Doppler velocity in each direction component, and the conversion relationship and the time difference are calculated as the calibration relationship.

In some implementations, the calibration module is further configured to determine, according to the determined calibration relationship and the collected sensor data, that the test equipment corresponding to each moment is in the image reference system and the radar reference system pose difference, and judge whether the pose difference is greater than the preset error threshold, if the pose difference is greater than the error threshold, then determine that the sensor data needs to be calibrated, and store the sensor data, if the If the pose difference is not greater than the error threshold, it is determined that the sensor data does not need to be calibrated.

The present application provides an electronic device, which includes a memory, a processor, and a computer program stored on the memory and operable on the processor. The processor implements the above sensor data calibration method when executing the program.

The present application provides a non-transitory computer-readable storage medium, the non-transitory computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the above sensor data calibration method is realized.

The present application provides a computer program product, the computer program product includes a computer program or an instruction, and the computer program or instruction is executed by a processor, so that the computer implements the above sensor data calibration method.

The above at least one technical solution adopted in the present application can achieve the following beneficial effects:

In the sensor data calibration method provided in the present application, the image data, radar data and inertial data at multiple moments during the movement of the test equipment are obtained, and the motion trajectory of the test equipment is determined according to the inertial data and image data. , determine the moving speed of the test equipment under the image reference system, and then determine the Doppler speed during the moving process of the test equipment according to the radar data, and compare the Doppler speed with the moving speed of the test equipment under the image reference system Registration to calibrate sensor data.

It can be seen from the above method that this scheme can also be applied when the acquisition ranges of the radar sensor and the image sensor do not overlap, and does not require calibration objects moving within the overlapping field of view, the calibration process is more convenient, and the accuracy of determining the sensor data is improved. calibration efficiency.

Description of drawings

The drawings described here are used to provide a further understanding of the application and constitute a part of the application. The schematic embodiments and descriptions of the application are used to explain the application and do not constitute an improper limitation to the application. In the attached picture:

Fig. 1 is the schematic flow chart of the calibration method of the sensor data provided by the present application;

FIG. 2 is a schematic diagram of a sensor calibration scenario provided by the present application;

Fig. 3 is the schematic diagram of the calibration device of the sensor data provided by the present application;

FIG. 4 is a schematic diagram of an electronic device corresponding to FIG. 1 provided in the present application.

Detailed ways

In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in conjunction with specific embodiments of the present application and corresponding drawings. Apparently, the described embodiments are only some of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in this application, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the scope of protection of this application.

In the field of sensor data correction, the commonly used sensor data calibration method is based on the overlapping areas of the acquisition ranges of image sensors and radar sensors.

Exemplarily, the radar sensor and the image sensor can be controlled to be stationary, and the calibration object can be controlled to move. Then, according to the radar data collected during the movement of the calibration object, the first position of the calibration object at each time is determined, and according to the collected image data, the second position of the calibration object at each time is determined. After that, for each moment, according to the constraint condition that the first position and the second position are the same, the calibration parameters of the radar sensor and the image sensor are determined.

However, the related technology is based on the fact that there is an overlapping area between the acquisition ranges of the image sensor and the radar sensor. The calibration parameters of the sensor are determined, so that the calibration efficiency of the related technology is poor.

Based on this, a new calibration method for sensor data is urgently needed.

The technical solutions provided by various embodiments of the present application will be described in detail below in conjunction with the accompanying drawings.

Fig. 1 is a schematic flow chart of the calibration method of sensor data provided by the present application, including the following steps:

S100: Acquire sensor data at multiple moments during the movement of the testing device, where the sensor data at least includes image data, radar data and inertial data.

Different from the method in the related art based on the overlapping areas of the acquisition ranges of the image sensor and the radar sensor, a moving calibration object is set in the overlapping area to calibrate the image sensor and the radar sensor. This application provides a new sensor data calibration method, so that there is no need to set the overlapping area between the image sensor and the radar sensor, but the radar sensor and the image sensor are set in the test equipment, and the mobile test equipment is used to determine the corresponding points at multiple times. imagery data, radar data and inertial data. The calibration relationship between the sensor data is determined based on the sensor data respectively corresponding to a plurality of time points.

Based on this, sensor data at multiple moments during the moving process of the test equipment can be obtained, wherein the sensor data includes: image data, radar data and inertial data.

In one or more embodiments provided by the present application, during the driving process of the unmanned equipment, the unmanned equipment can acquire sensor data according to a preset frequency, and the sensor data is required for calibrating the relationship of the sensor data The sensor data includes at least: image data, inertial data and radar data. certainly. The test equipment can also send the collected sensor data to the server, and the server performs subsequent steps to determine the calibration relationship between the radar sensor and the image sensor. For the convenience of description, the calibration process of the sensor data performed by the test equipment is taken as an example for description in the following.

Exemplarily, the testing device can acquire image data, inertial data and radar data collected by itself. Wherein, the test equipment can be unmanned equipment, manned equipment, or handheld equipment, which can control the movement of the test equipment, or hold the test equipment to move, and collect sensor data during the movement of the test equipment .

In addition, in this application, the sensor data calibration method is applied to placing calibration objects on the ground, controlling the movement of test equipment, collecting the position of calibration objects by image sensors, determining inertial data at multiple times by inertial sensors, and using radar A scenario where the sensor determines the Doppler velocity at multiple instants. as shown in picture 2.

FIG. 2 is a schematic diagram of a sensor calibration scenario provided by the present application. In the picture, the white cube is the test equipment, and the three gray cubes set on the white cube are radar sensor, inertial sensor and image sensor respectively, fixed on the ground There are image markers, the image sensor can collect image data containing markers, the radar sensor can collect Doppler velocity in various direction components through the Doppler effect of the wall, and the inertial sensor can collect inertial data corresponding to multiple moments. Among them, the image markers can be two-dimensional codes, checkerboards, etc., and the test equipment, radar sensors, image sensors, and inertial sensors are all in simplified form. The specific form and fixing method can be set according to needs, and this application does not make any limit.

It should be noted that the above-mentioned multiple moments in this application are multiple consecutive moments, so as to ensure that the motion track of the test equipment can be determined according to the acquired inertial data and image data, without obtaining the test equipment Sensor data at every moment during the movement.

S102: Determine a motion track of the test device according to the inertial data and the image data, and determine a moving speed of the test device in an image reference system according to the motion track.

In one or more embodiments provided by the present application, during the moving process of the test equipment, the motion trajectory of the test equipment under the image reference system and the motion trajectory under the radar reference system are actually the same trajectory. Therefore, if the test equipment The calibration relationship between the image reference system and the radar reference system can be determined by registering the movement trajectory of the device under the image reference system and the movement trajectory under the radar reference system.

Based on this, the test device can determine its own motion trajectory according to the acquired inertial data and image data.

Wherein, the acquired inertial data and image data correspond to multiple time points respectively. Its purpose is to determine the trajectory of the test equipment in the world reference system based on the inertial data and image data corresponding to multiple moments. Wherein, the motion trajectory corresponds to the inertia data and image data corresponding to multiple moments during the moving process of the test equipment, that is to say, the determined motion trajectory is not the entire trajectory corresponding to the moving process of the testing equipment, but the movement of the testing equipment The continuous trajectory determined by the sensor data corresponding to multiple moments acquired in the process.

Further, due to the large amount of calculation and difficulty required to determine the trajectory of the test equipment in the radar reference system based on the radar data, the Doppler velocity of the test equipment can be determined according to the Doppler effect of the acquired radar data. The calculation amount and calculation difficulty are small, therefore, the test equipment can determine the Doppler velocity during its own movement, as well as its own moving velocity in the image reference system, and register the Doppler velocity and the moving velocity, that is, A more accurate calibration relationship can be obtained.

Based on this, the test device can determine the moving speed of the test device in the image reference system according to the determined motion track.

Exemplarily, the test device can determine the corresponding displacement between each adjacent time in multiple moments of the test device according to the determined motion trajectory, and then determine the corresponding displacements of the test device at multiple times according to the displacement. movement speed.

Furthermore, the above-mentioned steps of determining the motion trajectory of the test equipment can be shown in the following method:

Exemplarily, the test device can determine the angular velocity, acceleration and observation position of the image marker corresponding to the test device at multiple moments in the inertial reference system according to the acquired inertial data and image data.

Then, the testing equipment can solve the motion trajectory of the testing equipment in the world reference system according to the angular velocity, acceleration and observed position of the image marker corresponding to multiple moments.

After that, according to the motion trajectory obtained by the solution, the test device can determine its own motion trajectory in the world reference system.

In addition, since the motion trajectory of the object can be determined by the corresponding poses of the object at multiple times, the test device can determine the motion trajectory of the test device itself based on the poses corresponding to the multiple times.

Exemplarily, since the poses corresponding to the test equipment at multiple moments are unknown quantities, the test equipment can be based on the poses corresponding to itself at multiple moments and the motion trajectory to be solved in the world reference system , to determine the first parameters to be solved respectively corresponding to multiple moments for determining the motion trajectory.

Then, for each of the multiple moments, the test device can determine the conversion relationship between the world reference frame and the inertial reference frame according to the first parameter to be solved corresponding to the moment, and determine the time when the test device is in the inertial reference frame. Department of Estimates Angular velocity, estimated acceleration, and estimated position of image markers.

Afterwards, the test equipment can establish constraint conditions based on the angular velocity corresponding to the moment is the same as the estimated angular velocity, the acceleration is the same as the estimated acceleration, and the observed position of the image marker is the same as the estimated position. solve.

Furthermore, since the estimated angular velocity, estimated acceleration, and estimated position of the image marker are all in the inertial reference frame, the test equipment needs to determine the respective The corresponding estimated angular velocity, estimated acceleration, and estimated position of the image marker.

Exemplarily, the test device can determine the conversion relationship between the world reference frame and the inertial reference frame to be solved at the moment according to the position and attitude of the test device in the first parameter to be solved corresponding to the moment. Then, based on the conversion relationship to be solved, the angular velocity offset to be solved in the first parameter, the acceleration offset to be solved, the observation position of the image marker to be solved, etc., respectively determine the expected position of the test equipment in the inertial reference system. estimated acceleration, estimated angular velocity, and estimated position of image markers. Wherein, the position and attitude of the test equipment may be referred to as the pose of the test equipment.

In addition, the above-mentioned steps of calculating the motion trajectory of the test equipment can be determined in the following manner:

Taking the motion trajectory of the above-mentioned construction test equipment modeled by the B-spline algorithm as an example, the first parameter can be constructed: x=[x _q ^T , x _p ^T , b _a ^T , b _w ^T , l ^T ]. Among them, for each of the multiple moments, x _q is the attitude of the test device in the world reference system at that moment, x _p is the position of the test device in the world reference system at this moment, b _a is the acceleration corresponding to this moment offset, b _w is the angular velocity offset corresponding to this moment, l is the observation position of the image marker at this moment, where the observation position of the image marker can be characterized by the position of each image feature point in the image marker, for example, the image The position of the central point of the marker, the position of the edge point of the image marker, etc. The image marker can be of various types such as checkerboard, two-dimensional code, etc., and the shape of the marker can be various shapes such as triangle, rectangle, circle, polygon, etc., and the shape and type of the image marker can be set according to needs. T represents the transpose of the matrix.

Then, the estimated acceleration can be determined as Among them, p is the estimate, k is the kth moment, is the estimated acceleration corresponding to time k, is the rotation matrix from the world reference system to the inertial reference system at time k, is the two derivations of the translation matrix from the world reference system to the inertial reference system at time k, that is, the acceleration of the test equipment in the world reference system at time k, and g _w is the gravitational acceleration at the current time. Since its direction is downward, negative The sign indicates the influence of its direction, and b _a is the bias of acceleration. Wherein, the rotation matrix from the world reference system to the inertial reference system and the translation matrix from the world reference system to the inertial reference system are determined based on the conversion relationship between the world reference system and the inertial reference system.

Similarly, the estimated angular velocity can be determined in, is the estimated angular velocity corresponding to time k, is the primary derivative of the rotation matrix from the world reference system to the inertial reference system at time k, and the estimated angular velocity can be derived according to the derivation formula of the rotation matrix derivative: Determination, through the characteristics of the anti-symmetric matrix, the determination formula of the above-mentioned estimated angular velocity can be determined.

Likewise, the test equipment can also determine the estimated position of image markers Among them, j is the moment when the image sensor collects the image, p _j represents the image collected at the jth moment, and for each image collection moment, is the translation matrix between the world reference system and the image reference system at this moment, is the rotation matrix between the world reference system and the image reference system at this moment, l is the observation position of the image marker in the world reference system at this moment, ∏() means to normalize the contents in brackets, w[ ,ζ] Indicates that the observed position of the image marker in the image reference system is transformed into the pixel reference system, and the position of the pixel where the image marker is located is determined. in, and It can be determined according to the conversion relationship between the world reference frame and the inertial reference frame at time j, and the predetermined conversion relationship between the inertial reference frame and the image reference frame.

Therefore, according to the estimated accelerations, estimated angular velocities, and estimated positions of the image markers corresponding to the multiple moments determined above, and the accelerations corresponding to the multiple moments in the inertial reference frame angular velocity and the observation position of the image marker Constraints can be constructed in,

That is to say, the test equipment can be based on the minimum difference between the estimated acceleration and acceleration, estimated angular velocity and angular velocity, and the observed position and estimated position of the image marker corresponding to multiple times respectively. One parameter is calculated to determine the motion trajectory of the test equipment.

It should be noted that the conversion relationship between the inertial reference frame and the image reference frame is predetermined. The rotation matrix and translation matrix between the world reference system and the inertial reference system can be determined according to the poses of the test equipment in the world reference frame at multiple moments. This application determines the rotation between the world reference frame and the inertial reference frame according to the pose The matrix and the way of translating the matrix are not limited.

In addition, the method of modeling the motion trajectory in this application can also use various modeling methods such as Bezier curves, and which type of motion trajectory construction method is used to solve the motion trajectory according to the inertial data and image data corresponding to multiple moments. It can be set as required, which is not limited in this application.

S104: According to the radar data, determine the Doppler velocity during the moving process of the test equipment, and register the Doppler velocity with the moving velocity of the test equipment in the image reference system, so as to Calibrate the above sensor data.

In one or more embodiments provided in the present application, as mentioned above, the test device can solve the conversion relationship between the image reference frame and the radar reference frame based on the Doppler velocity and the moving velocity. Thus, the test equipment can determine the Doppler velocity in each direction component.

Exemplarily, the test device can determine the Doppler velocity of the test device in each direction component according to the acquired radar data and the preset radar direction components for each time point.

Wherein, for each direction component, the direction component is composed of a pitch angle and an azimuth angle, and the Doppler velocity on the direction component is the Doppler velocity corresponding to the pitch angle and the azimuth angle. For example, assuming that the direction component is composed of a pitch angle of 30° and an azimuth angle of 60°, the Doppler velocity on the direction component is the velocity component corresponding to a pitch angle of 30° and an azimuth angle of 60°.

It should be noted that the above-mentioned Doppler velocity is determined by the radar sensor according to the Doppler effect, that is, for each direction component, the Doppler velocity on the direction component can be measured.

Further, since the Doppler velocity is the moving velocity of the stationary object relative to the test equipment, the Doppler velocity and the moving velocity of the test equipment are opposite in direction and equal in magnitude, therefore, the Doppler velocity is determined Under certain circumstances, the test equipment can register the Doppler velocity and the moving speed of the test equipment under the image reference system to determine the calibration relationship between the radar reference system and the image reference system.

Exemplarily, the testing equipment can determine the Doppler velocity in each direction component as the Doppler velocity in the same direction as the testing equipment and the moving velocity according to the direction of the moving velocity at multiple moments, and then the Doppler velocity Registering with the moving speed to determine the calibration relationship between the radar reference frame and the image reference frame.

Therefore, after determining the calibration relationship between the image sensor and the radar sensor, the test device can calibrate the acquired sensor data according to the determined calibration relationship. For example, according to the calibration relationship, the radar data in the acquired sensor data is converted into the image reference system, the target recognition and other steps are performed, and then the motion strategy of the test equipment is determined according to the recognition results.

Of course, after the calibration relationship is determined, the calibration relationship can also be used in various scenarios such as obstacle detection and obstacle classification, and the application of the calibration relationship can be set according to needs, which is not limited in this application.

Exemplarily, the calibrated sensor data may be sensor data collected during this movement of the test equipment, or may be sensor data collected during a subsequent movement of the test equipment. Of course, it can also be sensor data collected by an unmanned device similar in structure to the test device during delivery tasks and the like. The sensor data to be calibrated can be set according to needs, which is not limited in this application.

Furthermore, since the calibration relationship between the radar reference system and the image reference system is to be solved, according to the direction of the moving speed of the test equipment in the image reference system, the determined test equipment is in the radar reference system. There may be errors in the direction of the moving speed.

Based on this, the testing equipment can determine the moving speed of the testing equipment in each moving direction component based on the various direction components of the Doppler velocity, and then use the moving speed and The difference between the Doppler velocities in each direction component is the smallest, and the calibration relationship is solved.

Exemplarily, the test device may determine the moving speed of the test device in the image reference system at multiple moments according to the motion trajectory of the test device in the world reference system determined in step S102.

Then, for each of the plurality of times, the test device can determine each of the radar reference frames based on the unresolved calibration relationship between the image reference frame and the radar reference frame and the moving speed of the test device in the image reference frame. The speed of movement to be solved for in the direction component. Wherein, the moving speed to be solved includes a calibration relationship to be solved. The calibration relationship at least includes a rotation matrix and a translation matrix between the image reference frame and the radar reference frame.

Afterwards, the test device can register the moving velocity to be solved in each direction component and the Doppler velocity in each direction component, so as to solve the calibration relationship. Wherein, the registration method may take the minimum difference between the moving speed to be solved and the Doppler speed in each direction component as the optimization goal, and solve the optimization goal.

In addition, the above-mentioned steps of calculating the moving velocity and Doppler velocity to be solved by the test equipment can be determined in the following manner:

Exemplary, by The speed of movement of the test equipment in the frame of reference of the image can be determined. in, is the moving speed of the test device in the image reference system at time i, is the rotation matrix between the world reference system and the image reference system at time i, v ^w (t _i ) is the moving speed of the test equipment in the world reference system at time i, which can pass the first derivative of the displacement change of the test equipment at this time Sure. It can be determined according to the conversion relationship between the world reference system and the inertial reference system at time i, and the predetermined conversion relationship between the inertial reference system and the image reference system.

Then according to the calibration relationship between the image reference system to be solved and the radar reference system, the moving speed to be solved in the radar reference system at this moment can be determined Among them, v ^r (t _i ) is the moving speed of the test equipment in the radar reference frame at time i, is the rotation matrix between the radar reference frame and the image reference frame at time i, w _c (t _i ) identifies the angular velocity of the test equipment in the image reference frame at time i, represents an antisymmetric matrix, is the translation matrix between the image reference frame and the radar reference frame at time i.

Then, the test equipment can determine the moving speed to be solved on each direction component in the radar reference system according to the preset each direction component Then according to the Doppler velocity determined in step S104, the following cost function can be determined: in, is the pitch angle, and θ is the azimuth angle.

The goal is to minimize the cost function, that is, to solve the conversion relationship between the radar reference frame and the image reference frame. Afterwards, the conversion relationship obtained by solving is used as a calibration relationship between the radar reference system and the image reference system.

Further, since the radar sensor and the image sensor have their own internal clock systems, when determining the calibration relationship, the time difference between the internal clock system of the radar sensor and the internal clock system of the image sensor can also be determined. That is to say, the calibration relationship includes the conversion relationship between the image reference frame and the radar reference frame and the time difference between the internal clock system of the radar sensor and the internal clock system of the image sensor.

Exemplarily, for each time t _s of the multiple times of the test device under the radar reference system, according to the time difference t _d to be resolved between the internal clock of the radar sensor and the internal clock of the image sensor set on the test device , to determine the moving speed of the test device in the image reference system at this moment, that is, the moving speed at the moment t _i =t _s −t _d .

Secondly, the testing device can determine the moving speed of the testing device in the radar reference system according to the moving speed of the testing device in the image reference system at that moment, the conversion relationship between the radar reference system and the image reference system to be solved

Then, the test device can determine the moving speed of the test device to be resolved in each direction component according to the preset direction components.

Afterwards, register the moving velocity to be solved in each direction component and the Doppler velocity in each direction component, solve the conversion relationship and the time difference, and use the conversion relationship and the time difference obtained from the solution as the calibration relation. Among them, t _d is the time difference between the internal clock system of the radar sensor and the internal clock system of the image sensor.

Among them, the moving speed of the test equipment under the image reference system includes the time difference to be solved. If there is a time difference, the moving speed of the test equipment under the radar reference system includes the time difference to be solved and the image reference to be solved The rotation matrix and translation matrix to be solved between the radar frame and the radar reference frame.

Based on the calibration method of the sensor data provided in Fig. 1, image data, radar data and inertial data at multiple moments in the moving process of the test equipment are obtained, and according to the inertial data and image data, the motion track of the test equipment is determined, and according to the motion track, Determine the moving speed of the test equipment under the image reference system, and then determine the Doppler speed during the moving process of the test equipment according to the radar data, and match the Doppler speed with the moving speed of the test equipment under the image reference system standard to calibrate sensor data. This solution can also be applied when the acquisition ranges of the radar sensor and the image sensor do not overlap, and does not require a calibration object moving within the overlapping field of view, the calibration process is more convenient, and the calibration efficiency of determining sensor data is improved.

In addition, due to factors such as shaking of the testing equipment, the positional relationship between the image sensor and the radar sensor may change. Therefore, the testing equipment can also determine whether the calibration relationship needs to be re-determined during the moving process.

Exemplarily, the test device can determine the pose gap between the image reference system and the radar reference frame of the test device corresponding to multiple times according to the determined calibration relationship and the collected sensor data, and judge the pose gap Whether it is greater than the preset error threshold. If the pose difference is greater than the error threshold, the testing device may determine that the sensor data needs to be calibrated, and store the sensor data. If the pose difference is not greater than an error threshold, the testing device may determine that sensor data does not require calibration.

Of course, the test equipment can also store the sensor data within a preset time period, and determine the calibration relationship according to the pre-stored sensor data of a preset time length when the pose difference is greater than the preset error threshold, so as to avoid a large gap Sometimes it is still necessary to collect sensor data for a period of time before calibration, which makes the calibration efficiency low. The preset duration can be set as required, which is not limited in this application.

Furthermore, during the moving process of the testing equipment, there may also be situations such as shaking of the testing equipment, which may lead to a relatively large difference in determined poses. Usually, the jitter of the test equipment will disappear in a short period of time. Therefore, the test device can also record the number of times the pose difference is greater than the error threshold, so as to improve the accuracy of the judgment result. And when the pose difference is less than the error When the threshold value is reached, the calibration relationship is considered to be still correct. Then, when the number of times the pose difference is greater than the error threshold reaches the preset number threshold, it is considered that the determined calibration relationship is no longer reliable.

In addition, after determining the calibration relationship, the test equipment can determine the point cloud data and the projection of the point cloud data in the image reference system according to the acquired sensor data and the calibration relationship, and then the projection and the sensor data Fusion of the image data in the image, determine the fusion result, and perform obstacle detection on the fusion result, and determine the position of the obstacle.

Of course, the above method of merging image data and point cloud data to determine the position of obstacles is only one of the uses of the calibration relationship. After the calibration relationship is determined, the calibration relationship can also be used for obstacle detection, obstacle classification, etc. In this scenario, the application of the calibration relationship can be set as required, which is not limited in this application.

The above is the sensor data calibration method provided by one or more embodiments of the present application. Based on the same idea, the present application also provides a corresponding sensor data calibration device, as shown in FIG. 3 .

Fig. 3 is the calibration device of the sensor data provided by the application, including:

The acquiring module 200 is configured to acquire sensor data at multiple moments during the moving process of the testing device, the sensor data at least including image data, radar data and inertial data.

The trajectory determination module 202 is configured to determine the motion trajectory of the test device according to the inertial data and the image data, and determine the moving speed of the test device in the image reference system according to the motion trajectory.

A calibration module 204, configured to determine the Doppler velocity during the moving process of the test equipment according to the radar data, and register the Doppler velocity with the moving velocity of the test equipment in the image reference system , determining a calibration relationship between the radar reference frame and the image reference frame, so as to calibrate the sensor data.

In some implementations, the trajectory determination module 202 is configured to determine the angular velocity, acceleration and image markers of the test equipment corresponding to the multiple moments in the inertial reference system according to the inertial data and the image data According to the observed position of the object, according to the angular velocity, acceleration and observed position of the image marker corresponding to the multiple times, the motion track of the test equipment in the world reference system is solved.

In some implementations, the trajectory determination module 202 is configured to determine the first parameters to be solved respectively corresponding to the multiple moments according to the motion trajectory to be solved of the test equipment in the world reference system, the first parameter A parameter is used to solve the motion trajectory, and for each of the multiple moments, according to the first parameter to be solved corresponding to the moment, determine the conversion relationship between the world reference frame and the inertial reference frame, And determine the estimated angular velocity, estimated acceleration, and estimated position of the image marker of the test device under the inertial reference system, and the angular velocity corresponding to this moment is the same as the estimated angular velocity, and the acceleration The same as the estimated acceleration, the observed position of the image marker is the same as the estimated position, a constraint condition is constructed, and the motion track is solved.

In some implementations, the trajectory determination module 202 is configured to determine the world reference frame and inertial reference to be solved at this moment according to the pose of the test device in the first parameter to be solved corresponding to this moment The conversion relationship between systems, wherein, the first parameter includes the posture, acceleration offset, angular velocity offset and the observation position of the image marker of the test equipment; according to the conversion relationship to be solved, the second In one parameter, the angular velocity offset to be solved, the acceleration offset to be solved, and the observed position of the image marker to be solved are determined respectively to determine the estimated acceleration, the estimated angular velocity, and the estimated image marker of the test device Location.

In some implementations, the trajectory determination module 202 is configured to determine, according to the motion trajectory, the moving speeds of the test equipment corresponding to the multiple moments in the world reference system, and according to the The pose of the test equipment, determine the conversion relationship between the world reference frame and the inertial reference system corresponding to the multiple moments, according to the moving speed of the test equipment at the multiple moments in the world reference system, the The conversion relationship between the world reference system and the inertial reference system corresponding to the multiple times and the preset conversion relationship between the inertial reference system and the image reference system, A velocity of movement of the test device at the plurality of time instants in the image reference frame is determined.

In some implementations, the calibration module 204 is configured to, for each of the multiple moments, determine the Doppler in each direction component at that moment according to the acquired radar data and the preset each direction component. Le velocity, according to the moving speed of the test equipment in the image reference system at this moment, the calibration relationship to be solved between the radar reference system and the image reference system, and the preset components in each direction, determine the components of the test equipment in each direction The moving speed to be resolved is registered, the moving speed to be solved in each direction component is registered with the Doppler velocity in each direction component, and the calibration relationship is solved to calibrate the sensor data.

In some implementations, the calibration module 204 is configured to determine, according to the time difference to be resolved between the internal clocks of the radar sensor and the image sensor set on the test device, to determine the value of the test device under the image reference system at this moment. The moving speed to be solved, the moving speed to be solved includes the time difference to be solved, according to the moving speed to be solved of the test equipment in the image reference system at this moment, the waiting time between the radar reference system and the image reference system The conversion relationship of the solution is to determine the moving speed of the test equipment to be solved in the radar reference system, and to determine the moving speed of the test equipment to be solved in each direction component according to the preset components in each direction. The moving velocity to be solved is registered with the Doppler velocity in each direction component, and the conversion relationship and the time difference are calculated as the calibration relationship.

In some implementations, the calibration module 204 is further configured to determine, according to the determined calibration relationship and the collected sensor data, the test equipment corresponding to each moment in the image reference system and the radar The pose difference in the reference system, and judge whether the pose difference is greater than a preset error threshold, if the pose difference is greater than the error threshold, then determine that the sensor data needs to be calibrated, and store the sensor data , if the pose difference is not greater than the error threshold, it is determined that the sensor data does not need to be calibrated.

The present application also provides a non-transitory computer-readable storage medium, the non-transitory computer-readable storage medium stores a computer program, and the computer program can be used to execute the sensor data calibration method provided in FIG. 1 above.

The present application also provides a computer program product, the computer program product includes a computer program or an instruction, and the computer program or instruction is executed by a processor, so that the computer implements the sensor data calibration method provided in FIG. 1 above.

The present application also provides a schematic structural diagram of the electronic device shown in FIG. 4 . As shown in FIG. 4 , at the hardware level, the electronic device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory, and of course may also include hardware required by other services. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it, so as to realize the sensor data calibration method described in FIG. 1 above. Of course, in addition to the software implementation, this application does not exclude other implementations, such as logic devices or the combination of software and hardware, etc., that is to say, the execution subject of the following processing flow is not limited to each logic unit, and can also be hardware or logic device.

In the 1990s, the improvement of a technology can be clearly distinguished as an improvement in hardware (for example, improvements in circuit structures such as diodes, transistors, and switches) or improvements in software (improvement in method flow). However, with the development of technology, the improvement of many current method flows can be regarded as the direct improvement of the hardware circuit structure. Designers almost always get the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that the improvement of a method flow cannot be realized by hardware physical modules. For example, a programmable logic device (Programmable Logic Device, PLD) (such as a Field Programmable Gate Array (Field Programmable Gate Array, FPGA)) is such an integrated circuit, and its logic function is determined by programming the device by a user. It is programmed by the designer to "integrate" a digital system on a PLD, instead of asking a chip manufacturer to design and make a dedicated integrated circuit chip. Moreover, nowadays, instead of making integrated circuit chips by hand, this kind of programming is mostly realized by "logic compiler (logic compiler)" software, which is similar to the software compiler used when writing programs. The original code must also be written in a specific programming language, which is called a hardware description language (Hardware Description) Language, HDL), and there is not only one kind of HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL ( Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. VHDL (Very-High-Speed Integrated Circuit Hardware Description Language) and Verilog are currently the most commonly used. It should also be clear to those skilled in the art that only a little logical programming of the method flow in the above-mentioned hardware description languages and programming into an integrated circuit can easily obtain a hardware circuit for realizing the logic method flow.

The controller may be implemented in any suitable way, for example the controller may take the form of a microprocessor or processor and a computer readable medium storing computer readable program code (such as software or firmware) executable by the (micro)processor , logic gates, switches, application specific integrated circuits (Application Specific Integrated Circuit, ASIC), programmable logic controllers and embedded microcontrollers, examples of controllers include but are not limited to the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20 and Silicone Labs C8051F320, the memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art also know that, in addition to realizing the controller in a purely computer-readable program code mode, it is entirely possible to make the controller use logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded The same function can be realized in the form of a microcontroller or the like. Therefore, such a controller can be regarded as a hardware component, and the devices included in it for realizing various functions can also be regarded as structures within the hardware component. Or even, means for realizing various functions can be regarded as a structure within both a software module realizing a method and a hardware component.

The systems, devices, modules or units described in the above embodiments may be realized by computer chips or entities, or by products with certain functions. A typical implementing device is a computer. Exemplary, a computer may be, for example, a personal computer, laptop computer, cellular phone, camera phone, smart phone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device Or a combination of any of these devices.

For the convenience of description, when describing the above devices, functions are divided into various units and described separately. Of course, when implementing the present application, the functions of each unit can be implemented in one or more pieces of software and/or hardware.

Those skilled in the art should understand that the embodiments of the present application may be provided as methods, systems, or computer program products. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.

The present application is described with reference to flowcharts and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present application. It should be understood that each procedure and/or block in the flowchart and/or block diagram, and a combination of procedures and/or blocks in the flowchart and/or block diagram can be realized by computer program instructions. These computer program instructions may be provided to a general purpose computer, special purpose computer, embedded processor, or processor of other programmable data processing equipment to produce a machine such that the instructions executed by the processor of the computer or other programmable data processing equipment produce a An apparatus for realizing the functions specified in one or more procedures of the flowchart and/or one or more blocks of the block diagram.

These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing apparatus to operate in a specific manner, such that the instructions stored in the computer-readable memory produce an article of manufacture comprising instruction means, the instructions The device realizes the function specified in one or more procedures of the flowchart and/or one or more blocks of the block diagram.

These computer program instructions can also be loaded onto a computer or other programmable data processing device, causing a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process, thereby The instructions provide steps for implementing the functions specified in the flow chart or blocks of the flowchart and/or the block or blocks of the block diagrams.

In a typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.

Memory may include non-permanent storage in computer readable media, in the form of random access memory (RAM) and/or nonvolatile memory such as read-only memory (ROM) or flash RAM. Memory is an example of computer readable media.

Computer-readable media, including both permanent and non-permanent, removable and non-removable media, can be implemented by any method or technology for storage of information. Information may be computer readable instructions, data structures, modules of a program, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read only memory (ROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Flash memory or other memory technology, Compact Disc Read-Only Memory (CD-ROM), Digital Versatile Disc (DVD) or other optical storage, Magnetic tape cartridge, tape magnetic disk storage or other magnetic storage device or any other non-transmission medium that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media excludes transitory computer-readable media, such as modulated data signals and carrier waves.

It should also be noted that the term "comprises", "comprises" or any other variation thereof is intended to cover a non-exclusive inclusion such that a process, method, article, or apparatus comprising a set of elements includes not only those elements, but also includes Other elements not expressly listed, or elements inherent in the process, method, commodity, or apparatus are also included. Without further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article or apparatus comprising said element.

Those skilled in the art should understand that the embodiments of the present application may be provided as methods, systems or computer program products. Accordingly, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.

This application may be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. The application may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media including storage devices.

Each embodiment in the present application is described in a progressive manner, the same and similar parts of each embodiment can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and for relevant parts, refer to part of the description of the method embodiment.

The above descriptions are only examples of the present application, and are not intended to limit the present application. For those skilled in the art, various modifications and changes may occur in this application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included within the scope of the claims of the present application.

Claims (19)

1. A calibration method for sensor data, wherein the method includes:

   Acquiring sensor data at multiple moments during the movement of the test equipment, the sensor data at least including image data, radar data and inertial data;

   determining the motion trajectory of the testing device according to the inertial data and the image data, and determining the moving speed of the testing device in the image reference system according to the motion trajectory;

   According to the radar data, determine the Doppler velocity during the moving process of the test equipment, and register the Doppler velocity and the moving velocity of the test equipment in the image reference system, so as to Calibrate the above sensor data.
2. The method according to claim 1, wherein, according to the inertial data and the image data, determining the motion trajectory of the testing device comprises:

   According to the inertial data and the image data, determine the angular velocity, acceleration, and observation position of the image marker respectively corresponding to the multiple moments of the test equipment under the inertial reference system;

   According to the angular velocities, accelerations and observed positions of the image markers respectively corresponding to the multiple moments, the motion trajectory of the test equipment in the world reference system is solved.
3. The method according to claim 2, wherein, according to the angular velocities, accelerations, and observation positions of the image markers corresponding to the plurality of moments, solving the motion trajectory of the test equipment in the world reference system includes:

   According to the motion trajectory to be solved of the test equipment in the world reference system, determine the first parameters to be solved respectively corresponding to the multiple moments, and the first parameters are used to solve the motion trajectory;

   For each of the plurality of moments, according to the first parameter to be solved corresponding to the moment, determine the conversion relationship between the world reference frame and the inertial reference frame, and determine the test equipment in the inertial Estimated angular velocity, estimated acceleration, and estimated position of image markers in the frame of reference;

   The angular velocity corresponding to this moment is the same as the estimated angular velocity, the acceleration is the same as the estimated acceleration, and the observed position of the image marker is the same as the estimated position, constructing constraint conditions for the motion track solution.
4. The method according to claim 3, wherein, according to the first parameter to be solved corresponding to the moment, determine the conversion relationship between the world reference frame and the inertial reference frame, and determine the The estimated angular velocity, estimated acceleration, and estimated position of image markers under the system, including:

   According to the pose of the test device in the first parameter to be solved corresponding to this moment, determine the conversion relationship between the world reference frame and the inertial reference frame to be solved at this moment, wherein the first parameter includes The pose, acceleration offset, angular velocity offset and observation position of the image marker of the test equipment;

   According to the transformation relationship to be solved, the angular velocity offset to be solved in the first parameter, the acceleration offset to be solved, and the observed position of the image marker to be solved, respectively determine the estimated value of the test device Acceleration, estimated angular velocity, and estimated position of image markers.
5. The method according to any one of claims 1-4, wherein, according to the motion trajectory, determining the moving speed of the test device under the image reference system comprises:

   According to the motion trajectory, determine the moving speeds of the test equipment corresponding to the multiple moments in the world reference system;

   According to the poses and postures of the test equipment at the multiple moments, determine the conversion relationship between the world reference frame and the inertial reference frame respectively corresponding to the multiple moments;

   According to the moving speed of the test equipment at the multiple moments in the world reference system, the conversion relationship between the world reference system and the inertial reference system corresponding to the multiple moments, and the preset inertial reference system The conversion relationship between the image reference system and the image reference system is used to determine the moving speed of the test equipment at the multiple moments in the image reference system.
6. The method according to any one of claims 1-4, wherein the Doppler velocity is registered with the moving velocity of the test device in the image reference system to calibrate the sensor data, include:

   For each of the plurality of moments, according to the acquired radar data and the preset directional components, determine the Doppler velocity in each directional component at that moment;

   According to the moving speed of the test equipment in the image reference system at this moment, the calibration relationship between the radar reference system and the image reference system to be solved, and the preset components in each direction, it is determined that the test equipment is to be solved in each direction component moving speed;

   The moving velocity to be solved in each direction component is registered with the Doppler velocity in each direction component, and the calibration relationship is solved, so as to calibrate the sensor data.
7. The method according to claim 6, wherein, according to the moving speed of the test equipment under the image reference system at this moment, the calibration relationship to be solved between the radar reference system and the image reference system, and the preset direction components, determine The moving speed of the test equipment to be solved on each direction component includes:

   According to the time difference to be solved between the internal clock of the radar sensor set on the test device and the image sensor, determine the moving speed to be solved of the test device at this moment under the image reference system, and the moving speed to be solved includes The time difference to be resolved;

   According to the moving speed of the test equipment to be solved in the image reference system at this moment, the conversion relationship to be solved between the radar reference system and the image reference system, determine the moving speed of the test equipment in the radar reference system to be solved ;

   According to each preset direction component, determine the moving speed of the test equipment to be solved on each direction component;

   Register the moving speed to be solved on each direction component and the Doppler speed on each direction component, and solve the calibration relationship, including:

   The moving velocity to be solved in each direction component is registered with the Doppler velocity in each direction component, and the conversion relationship and the time difference are calculated as the calibration relationship.
8. The method according to any one of claims 1-4, wherein the method further comprises:

   According to the determined calibration relationship and the collected sensor data, determine the pose gap of the test equipment corresponding to each moment in the image reference frame and the radar reference frame, and judge whether the pose gap is greater than Preset error thresholds;

   If the pose difference is greater than the error threshold, it is determined that the sensor data needs to be calibrated, and the sensor data is stored;

   If the pose difference is not greater than the error threshold, it is determined that sensor data does not need to be calibrated.
9. A calibration device for sensor data, wherein the device includes:

   An acquisition module, configured to acquire sensor data at multiple moments during the movement of the test equipment, the sensor data at least including image data, radar data and inertial data;

   A trajectory determination module, configured to determine the motion trajectory of the test device according to the inertial data and the image data, and determine the moving speed of the test device in the image reference system according to the motion trajectory;

   a calibration module, configured to determine the Doppler velocity during the moving process of the test equipment according to the radar data, and register the Doppler velocity with the moving velocity of the test equipment in the image reference system, to calibrate the sensor data.
10. The device according to claim 9, wherein said trajectory determining module is configured to use said inertial data and said Image data, determine the angular velocity, acceleration and observation position of the image marker corresponding to the multiple moments of the test equipment under the inertial reference system; Position, solve the motion track of the test equipment in the world reference system.
11. The device according to claim 10, wherein the trajectory determination module is configured to determine the first parameters to be solved respectively corresponding to the multiple times according to the motion trajectory to be solved of the test equipment in the world reference system , the first parameter is used to solve the motion trajectory; for each of the multiple moments, according to the first parameter to be solved corresponding to the moment, determine the distance between the world reference frame and the inertial reference frame conversion relationship, and determine the estimated angular velocity, estimated acceleration, and estimated position of the image marker of the test device under the inertial reference system; the angular velocity corresponding to this moment is the same as the estimated angular velocity , the acceleration is the same as the estimated acceleration, the observed position of the image marker is the same as the estimated position, constructing constraint conditions, and solving the motion trajectory.
12. The device according to claim 11, wherein the trajectory determining module is configured to determine the world reference to be solved at this moment according to the pose of the test device in the first parameter to be solved corresponding to the moment The conversion relationship between the frame and the inertial reference frame, wherein the first parameter includes the pose, acceleration offset, angular velocity offset, and observation position of the image marker of the test device; according to the conversion relationship to be solved , the angular velocity offset to be resolved, the acceleration offset to be resolved, and the observed position of the image marker to be resolved in the first parameter, respectively determine the estimated acceleration, estimated angular velocity, and image marker of the test device the estimated location of the object.
13. The device according to any one of claims 9-12, wherein the trajectory determination module is configured to determine the moving speeds of the test equipment corresponding to the multiple moments in the world reference system according to the movement trajectory; According to the poses of the test equipment at the multiple moments, determine the conversion relationship between the world reference frame and the inertial reference frame corresponding to the multiple moments; according to the test equipment in the world reference system, The moving speed at multiple times, the conversion relationship between the world reference frame and the inertial reference system corresponding to the multiple times, and the preset conversion relationship between the inertial reference system and the image reference system are determined. The moving speed of the test equipment at the multiple moments in the image reference system.
14. The device according to any one of claims 9-12, wherein the calibration module is configured to, for each of the multiple moments, determine the The Doppler velocity on each direction component at the moment; according to the moving speed of the test equipment under the image reference system at this moment, the calibration relationship to be solved between the radar reference system and the image reference system, and the preset components in each direction, determine The moving speed of the test equipment to be solved in each direction component; the moving speed to be solved in each direction component and the Doppler speed in each direction component are registered, and the calibration relationship is solved, so that the Calibration of sensor data.
15. The device according to claim 14, wherein the calibration module is configured to determine the time difference between the radar sensor and the internal clock of the image sensor set on the test equipment to determine the The moving speed to be solved in the reference system, the moving speed to be solved includes the time difference to be solved; according to the moving speed to be solved in the image reference system of the test equipment at this moment, the radar reference system and the image reference The conversion relationship to be solved between the systems is used to determine the moving speed of the test equipment to be solved under the radar reference system; according to the preset components in each direction, the moving speed of the test equipment to be solved in each direction component is determined; The moving velocity to be resolved in each direction component is registered with the Doppler velocity in each direction component, and the conversion relationship and the time difference are calculated as the calibration relationship.
16. The device according to any one of claims 9-12, wherein the calibration module is further configured to determine the corresponding time points of the test equipment in the image according to the determined calibration relationship and the collected sensor data. The pose gap between the reference frame and the radar reference frame, and determine whether the pose gap is greater than a preset error threshold; if the pose gap is greater than the error threshold, it is determined that the sensor data needs to be calibrated, and The sensor data is stored; if the pose difference is not greater than the error threshold, it is determined that the sensor data does not need to be calibrated.
17. A non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in any one of claims 1-8 is implemented. method.
18. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and operable on the processor, wherein, when the processor executes the program, it implements the method described in any one of claims 1-8 method.
19. A computer program product, wherein the computer program product includes a computer program or instruction, and the computer program or instruction is executed by a processor, so that the computer implements the method described in any one of claims 1-8 above.