JP2022510418A - Time synchronization processing method, electronic devices and storage media - Google Patents

Abstract

It is a time synchronization processing method, and the method is to obtain angular velocity information collected by two different sensors, respectively (S101), and the angular velocity information is angular velocity information when an electronic device performs a rotational motion. Both of the two different sensors are provided in the electronic device and are fixedly connected (S101), and the two different angular velocity information obtained are aligned and the delay between the two different sensors is performed. It includes determining the time information (S102) and performing time synchronization processing on the measurement results of the two different sensors based on the delay time information (S103). Further discloses time synchronization processing devices, electronic devices and computer readable storage media.

Description

(Mutual reference of related applications)
This application claims priority based on the Chinese patent application filed June 21, 2019, with application number 201910545218.8, the entire contents of which Chinese patent application is incorporated herein by reference.

The present application relates to the field of computer vision, and specifically to a time synchronization processing method, an electronic device and a storage medium.

Visual inertia odometry has become the focus of today's computer vision field and is widely applied in the navigation and entertainment of electronic devices. The main principle is to estimate the position and orientation of the camera itself in the movement process by fusing the visual and inertial sensors to obtain accurate positioning information, which belongs to autonomous navigation.

Due to the different trigger and transmission delays present in different sensors, the electronic device needs to determine the delay time information between the different sensors when fusing the measurement results of the different sensors. Currently, in general, the delay time information between different sensors is calibrated by the movement in the three-dimensional space, or the time delay between different sensors is treated as a constant. However, since the delay time between different sensors changes in the movement process of the electronic device, there is a problem that the time synchronization accuracy is low.

It is desirable that the embodiments of the present application provide a time synchronization processing method, an electronic device and a storage medium.

According to the first aspect, the embodiments of the present application provide a time synchronization processing method. The method is to obtain the angular velocity information collected by two different sensors, respectively. The angular velocity information is the angular velocity information when the electronic device performs a rotational motion, and the two different sensors both have the angular velocity information. It is provided in the electronic device and is fixedly connected, alignment processing is performed on the obtained two angular velocity information, and delay time information between the two different sensors is determined, and the delay is described. It includes performing time synchronization processing on the measurement results of each of the two different sensors based on the time information.

According to the second aspect, the embodiment of the present application provides a time synchronization processing apparatus. The apparatus includes a first acquisition module, an alignment module, and a synchronization module, and the first acquisition module is configured to obtain angular velocity information collected by two different sensors, respectively. , Angular velocity information when the electronic device performs a rotational motion, both of the two different sensors are provided in the electronic device and are fixedly connected, and the alignment module is attached to the obtained two angular velocity information. It is configured to perform alignment processing on the object and determine the delay time information between the two different sensors, and the synchronization module is based on the delay time information with respect to the measurement result of each of the two different sensors. It is configured to perform time synchronization processing.

According to the third aspect, the embodiment of the present application provides an electronic device. The electronic device comprises at least a processor and a memory configured to store a computer program that can be executed by the processor, and when the processor executes the computer program, the step in the time synchronization processing method is performed. Configured to run.

According to a fourth aspect, the embodiments of the present application provide a computer-readable storage medium. When the computer program is stored in the computer-readable storage medium and the computer program is executed by the processor, the steps in the time synchronization processing method are realized.

The embodiments of the present application provide time synchronization processing methods, electronic devices and storage media. Two different sensors provided in the electronic device and fixedly connected obtain angular velocity information collected by the two different sensors, perform alignment processing on the obtained two angular velocity information, and perform two different angular velocity information. The delay time information between the sensors is determined, and based on the delay time information, time synchronization processing is performed on the measurement results of the two different sensors. This makes full use of the principle that the angular velocity information is the same by being fixedly connected, and realizes time synchronization of different sensors.

It is 1st schematic diagram which shows the system structure applied to the time synchronization processing method by an Example of this application. 2 is a second schematic diagram showing a system architecture applied to the time synchronization processing method according to the embodiment of the present application. FIG. 3 is a third schematic diagram showing a system architecture applied to the time synchronization processing method according to the embodiment of the present application. It is 1st schematic diagram which shows the realization flow of the time synchronization processing method by the Example of this application. FIG. 3 is a schematic diagram showing a case where there is a time delay between the angular velocity of an exemplary visual sensor and the angular velocity of an inertial sensor according to an embodiment of the present application. It is a schematic diagram which shows that the angular velocity of an exemplary visual sensor and the angular velocity of an inertial sensor are synchronized according to the embodiment of the present application. It is a 2nd schematic diagram which shows the realization flow of the time synchronization processing method by the Example of this application. It is a 3rd schematic diagram which shows the realization flow of the time synchronization processing method by the Example of this application. It is a 4th schematic diagram which shows the realization flow of the time synchronization processing method by the Example of this application. It is the first schematic diagram which shows that the position-posture rotation matrix information measured by the position-posture type sensor is represented by the quaternion according to the embodiment of the present application. It is a 2nd schematic diagram which shows that the position-posture rotation matrix information measured by the position-posture type sensor is represented by the quaternion according to the embodiment of the present application. It is a schematic diagram which shows the structure of the time synchronization processing apparatus according to the Example of this application. It is a schematic diagram which shows the structure of the electronic device by the Example of this application.

In order to further clarify the object, technical solution and advantage of the present application, the technical solution in the embodiment of the present application will be clearly and completely described below with reference to the drawings in the embodiment of the present application. The embodiments described should not be acknowledged as limiting the embodiments of the present application, and all other embodiments obtained by those skilled in the art without creative effort are within the scope of the invention.

FIG. 1 is a first schematic diagram showing a system structure applied to the time synchronization processing method according to the embodiment of the present application. As shown in FIG. 1, the system may include a processor 11, two different sensors 12, and a memory 13. The two different sensors 12 transmit the acquired information to the processor 11 for processing. One of the two different sensors 12 may be a sensor that directly measures the angular velocity, and the other sensor may be a sensor that indirectly measures the angular velocity. The sensor that indirectly measures the angular velocity is a sensor that can independently estimate the rotational motion of the sensor itself (that is, acquire position-attitude rotation matrix information). The two different sensors 12 may be sensors of other structures, and the embodiments of the present application do not limit the structure of the two different sensors.

Here, two of the two different sensors 12 are rigidly fixed. That is, the two different sensors 12 of the electronic device are relatively fixed. In the process of movement, the angular velocities measured by different sensors at the same time are the same. Each of the two different sensors 12 inputs the acquired information to the processor 11. The processor 11 performs the time synchronization process by executing the method provided in the embodiment of the present application.

In some embodiments, the electronic device may include a time synchronization processor and two different sensors 12. The time synchronization processing device may include the processor 11 and the memory 13. By providing the above two different sensors 12 in the electronic device, the angular velocity information collected by the two different sensors can be obtained respectively. Each of the two different sensors 12 inputs the obtained information to the processor 11. The processor 11 performs the time synchronization process by executing the method provided in the embodiment of the present application.

Further, in some embodiments, the system architecture of the time synchronization processing method may be as shown in FIGS. 2A and 2B. The system includes a time synchronization processing device 20, a first sensor 22a and a second sensor 22b. The first sensor 22a and the second sensor 22b are provided in, for example, an electronic device of a vehicle. The time synchronization processing device 20 may be an in-vehicle device in the vehicle. The first sensor 22a and the second sensor 22b can transmit the acquired information to the time synchronization processing device 20 by an electrical connection or a wireless communication connection. The time synchronization processing device 20 includes a processor and a memory, and the first sensor 22a and the second sensor 22b transmit the obtained information to the time synchronization processing device 20 and perform processing from the processor, and carry out the present application. It should be understood to implement the time synchronization processing method provided in the example.

As shown in FIGS. 2A and 2B, the first sensor 22a and the second sensor 22b may be provided at different positions in the vehicle or may be provided at the same position in the vehicle. For example, as shown in FIG. 2B, both the first sensor 22a and the second sensor 22b may be provided on the tire portion of the vehicle. Further, for example, as shown in FIG. 2A, the first sensor 22a may be provided on the tire portion of the vehicle, and the second sensor 22b may be provided on another portion of the vehicle. For example, it is provided in an in-vehicle device unit. The embodiments of the present application do not limit the arrangement positions of the first sensor 22a and the second sensor 22b.

Different processors process information from different sensors in different ways. For example, if the sensor is a sensor that directly measures the angular velocity, the processor can directly process the obtained angular velocity information. When the sensor is a sensor that indirectly measures the angular velocity, the processor first obtains the position / orientation rotation matrix information, obtains the angular velocity information based on the position / orientation rotation matrix information, and then obtains the acquired angular velocity information. To process.

In general, the time synchronization processing device can be applied to various types of electronic devices having information processing capability in the execution process. The time synchronization processing apparatus can acquire data collected by two different sensors in real time in an online manner and perform time synchronization processing by the technical solution means of the embodiment of the present application. The time synchronization processing apparatus can also acquire data collected by two different sensors in an offline manner and perform time synchronization processing by the technical solution means of the embodiment of the present application. Here, to acquire the data collected by two different sensors in an offline method stores the data collected by the two different sensors, and if necessary, derives the stored data, and of the different sensors. It may include performing synchronous processing for each measurement result. Acquiring the data collected by two different sensors in an online manner involves acquiring the data collected by two different sensors in real time and performing synchronous processing on the measurement results of each of the different sensors. But it may be.

Here, the time synchronization processing device may be provided inside the electronic device or may be provided outside the electronic device. Two different sensors are provided in the electronic device. Here, the electronic device may be, for example, a device such as a mobile robot device, an unmanned driving device, or various mobile terminals. Here, the unmanned driving equipment may include, but is not limited to, a vehicle, an airplane, or a ship. The embodiments of the present application are not limited here.

According to the above architecture, the embodiments of the present application provide a time synchronization processing method that can solve the problem that the calibration of the time delay between different sensors is inaccurate and complicated. The function realized by the time synchronization processing method may be realized by calling an instruction which can be executed by a processor in the time synchronization processing apparatus. Of course, the executable instruction may be stored in the storage medium of the memory. Therefore, the time synchronization processing device includes at least a processor and a storage medium.

FIG. 3 is a first schematic diagram showing a flow for realizing the time synchronization processing method according to the embodiment of the present application. The method is applied to a time synchronization processing device. As shown in FIG. 3, the time synchronization processing method includes the following.

In S101, the angular velocity information collected by two different sensors is obtained, and the angular velocity information is the angular velocity information when the electronic device performs a rotational motion, and both of the two different sensors are attached to the electronic device. Provided and fixedly connected.

In the embodiments of the present application, two different sensors are fixedly connected. That is, the two different sensors are relatively fixed. In the process of rotating the electronic device, the angular velocities represented by the angular velocity information acquired by two different sensors at the same time are the same.

The two different sensors in the embodiments of the present application include sensors having different structures, or sensors having the same structure but different placement positions.

For example, if the two different sensors are sensors with different structures, the two different sensors may include any two structures of a visual sensor, an inertial sensor, a magnetic force sensor, a laser radar sensor and a wheel sensor. , The embodiment of the present application is not limited here.

Further, for example, when the two different sensors are sensors having the same structure but different positions, the two different sensors may be visual sensors or magnetic sensors arranged at different positions in an electronic device. , The embodiments of the present application are not limited here.

In the embodiment of the present application, the angular velocity information may be angular velocity information when the electronic device performs a rotational motion within a predetermined period acquired by two different sensors, respectively. That is, when the electronic device rotates, it rotates at a constant angle, and further, it is possible to acquire the angular velocity information collected by different sensors within a predetermined period.

The predetermined period may be set by the user according to an actual situation, and the predetermined period may be, for example, half an hour or 15 minutes, and the embodiments of the present application limit this. do not do.

The above two different sensors can be classified according to an acquisition method such as a method of directly acquiring angular velocity information or a method of indirectly acquiring angular velocity information. Illustratively, one type of sensor is a gyro sensor that can directly measure angular velocity information, and another type of sensor is a position and attitude sensor that can indirectly measure angular velocity information. Alternatively, the above two different sensors can be classified based on the sensor structure. Illustratively, a gyro sensor and a position / attitude sensor are sensors having different structures.

In some selectable embodiments in the embodiments of the present application, at least one of the two different sensors is a position-orientation sensor, and obtaining angular velocity information collected by the two different sensors, respectively. It includes obtaining the position / attitude rotation matrix information collected by the position / attitude sensor and determining the angular velocity information when the electronic device performs a rotational motion based on the position / attitude rotation matrix information.

In some selectable embodiments in the embodiments of the present application, at least one of the two different sensors is a gyro sensor, and obtaining angular velocity information collected by each of the two different sensors is a gyro. Includes obtaining angular velocity information collected by the sensor.

In the embodiment of the present application, obtaining angular velocity information when an electronic device performs a rotational motion by two different sensors includes the following plurality of application scenes.

In scene 1, the two different sensors are both gyro sensors, and the positions of the two gyro sensors are different. Two gyro sensors directly collect angular velocity information when an electronic device makes a rotational motion.

In the scene 2, the two different sensors are both position / attitude sensors, and the positions of the two position / attitude sensors are different. The two position / orientation sensors obtain position / attitude rotation matrix information when the electronic device performs rotational movement, and the position / attitude rotation matrix information further calculates angular velocity information when the electronic device performs rotational movement. ..

In Scene 3, one of the two different sensors is the position-attitude sensor and the other is the gyro sensor. The position / attitude sensor can obtain the position / attitude rotation matrix information when the electronic device performs the rotational movement, and further, the position / attitude rotation matrix information can be used to calculate the angular velocity information when the electronic device performs the rotational movement. On the other hand, the gyro sensor can directly collect the angular velocity information when the electronic device performs a rotational motion.

Illustratively, a gyro sensor includes any one of an inertial sensor, a magnetic force sensor and a wheel odometry sensor, and a position / orientation sensor includes any one of a visual sensor and a laser radar sensor. ..

In S102, alignment processing is performed on the two obtained angular velocity information, and the delay time information between the two different sensors is determined.

In the embodiment of the present application, since the data acquisition frequencies of different sensors are different, it is necessary to obtain the two angular velocity information and then perform the alignment process on the two angular velocity information to determine the delay time information between the two different sensors. There is.

In the embodiment of the present application, it is an object of performing the alignment process on the two angular velocity information to match the angular velocity information acquired by the two different sensors with frequency. This makes it possible to determine the delay time information between two different sensors based on the two aligned angular velocity information.

In the embodiment of the present application, the two angular velocity information can be aligned by performing the interpolation processing on the two angular velocities.

Illustratively, the interpolation processing can use any one of linear interpolation, cubic spline interpolation, and spherical interpolation processing method, and the embodiment of the present application is not limited here.

In S103, time synchronization processing is performed on the measurement results of the two different sensors based on the delay time information.

In the embodiment of the present application, after the delay time information between the two different sensors is determined, the time synchronization processing can be performed on the measurement results of the two different sensors based on the delay time information. Performing time synchronization processing on the measurement results of the two different sensors is to determine the measurement results of the two different sensors at the same time. This solves the problem of low time synchronization accuracy due to trigger delay and transmission delay between different sensors.

In the embodiment of the present application, the measurement result of each of the two different sensors is the measurement result obtained by measuring the rotational motion of the electronic device by the two different sensors when the electronic device performs the rotational motion. ..

Illustratively, the measurement results obtained by each of the two different sensors may include at least one of angular velocity information, directional position angle information and acceleration information. For example, the measurement result of the inertial sensor may include the angular velocity information and the acceleration information when the electronic device performs a rotary motion, and the measurement result of the magnetic force sensor includes the direction position angle information when the electronic device performs a rotary motion. But it may be.

In one embodiment, the electronic device may be provided with at least three sensors, the at least three sensors being fixedly connected. The angular velocity information collected by the at least three sensors is obtained respectively. Alignment processing is performed on the obtained three angular velocity information, and the delay time information between at least three sensors is determined. Based on the delay time information, time synchronization processing is performed on the measurement results of each of the at least three sensors.

When at least three sensors are provided in an electronic device, at least two sensors are combined to acquire delay time information between any two sensors, and the delay time information is used to correspond at the same time. Time synchronization processing is performed on the measurement results of each of the two sensors, and time synchronization processing of at least three sensors is realized.

Illustratively, when two different sensors provided in an electronic device are a visual sensor and an inertial sensor, respectively, the measurement result and inertia of the visual sensor are based on the acquired delay time information between the visual sensor and the inertial sensor. It is possible to perform time synchronization processing on the measurement result of the sensor and realize time synchronization between the measurement result of the visual sensor and the measurement result of the inertial sensor.

Illustratively, when the three sensors provided in the electronic device are a visual sensor, an inertial sensor and a laser radar sensor, respectively, first, the delay time information between the visual sensor and the inertial sensor is determined, and the delay time information is used. Based on this, time synchronization processing is performed on the measurement result of the visual sensor and the measurement result of the inertial sensor, and the time synchronization between the measurement result of the visual sensor and the measurement result of the inertial sensor is realized. Subsequently, the delay time information between the visual sensor and the laser radar sensor is determined, and based on the delay time information, the measurement result of the visual sensor and the measurement result of the laser radar sensor are subjected to time synchronization processing to perform visual observation. Realizes time synchronization between the measurement result of the sensor and the measurement result of the laser radar sensor. In this way, the time synchronization of the measurement results between the three sensors, the visual sensor, the inertial sensor, and the laser radar sensor, is realized by the two time synchronization processes.

In order to more clearly express the difference between the measurement results of the two different sensors before and after synchronization, one of the two different sensors is the visual sensor and the other sensor is the inertial sensor. The time synchronization process in the embodiment of the present application will be described by taking as an example that the measurement result is the angular velocity information.

FIG. 4 is a schematic diagram showing a case where there is a time delay between the angular velocity of the exemplary visual sensor and the angular velocity of the inertial sensor according to the embodiment of the present application. FIG. 5 is a schematic diagram showing that the angular velocity of the exemplary visual sensor and the angular velocity of the inertial sensor according to the embodiment of the present application are synchronized. In FIGS. 4 and 5, the broken line represents the angular velocity of the visual line sensor, and the solid line represents the angular velocity of the inertial sensor.

As can be seen from FIG. 4, the angular velocity curve of the visual sensor and the angular velocity curve of the inertial sensor are out of alignment, and the angular velocity curve of the visual sensor has a time lag. After synchronizing with the angular velocity of the visual sensor by the delay time information between the visual sensor and the inertial sensor, as can be seen from FIG. 5, the angular velocity curve of the visual sensor and the angular velocity curve of the inertial sensor are aligned. And it is not out of alignment. Thereby, it is possible to realize the time synchronization processing of the angular velocity information of the visual sensor and the inertial sensor based on the determined delay time information.

In the embodiment of the present application, after the delay time information between the two different sensors is determined, the time synchronization processing can be performed on the measurement results of the two different sensors based on the direct delay time information.

In one embodiment, it is also possible to perform an alignment process on the two obtained angular velocity information, determine the delay time information between the two different sensors, and then store the delay time information. When the electronic device is in a non-moving state, time synchronization processing is performed on the measurement results of each of the two different sensors based on the stored delay time information.

The technical solution of the embodiment of the present application can not only realize the online time synchronization process but also the offline time synchronization process, and the time synchronization process is more flexible. At the same time, the time synchronization process in the embodiment of the present application is more convenient, simpler, and more adaptable without relying on a calibration reference such as a grid image.

In the embodiment of the present application, the electronic device is required to be able to perform a rotary motion. That is, the electronic device can calibrate the delay time information between different sensors in the electronic device as long as it rotates at least around the axis of rotation. It does not have to rotate around multiple axes, reducing the time synchronization complexity of different sensors and adapting to the demands of different scenes. Also, in the embodiments of the present application, the delay time can be calibrated based on the rotational motion generated by the rotation of the electronic device around a plurality of axes. As a result, richer rotation information can be obtained and the accuracy of time synchronization can be improved. In the embodiment of the present application, it is required that the sensor can independently acquire the angular velocity information. This makes it possible to determine the delay time information between different sensors. It is widely applicable to time synchronization between multiple sensors and has general-purpose adaptability. In the embodiment of the present application, it is a method of realizing time synchronization based on software, and it is not necessary to separately arrange dedicated hardware for time synchronization. In the embodiment of the present application, time synchronization can be performed and time synchronization processing can be performed online based on the delay time information acquired in real time. In the embodiment of the present application, the delay time information is determined based on the angular velocity information acquired when the electronic device performs the rotational motion. Improves the accuracy of time synchronization without processing the delay time information as a constant.

FIG. 6 is a second schematic diagram showing a flow for realizing the time synchronization processing method according to the embodiment of the present application. As shown in FIG. 6, in the embodiment of the present application, step S102 of determining the delay time information between two different sensors by performing an alignment process on the angular velocity information acquired by each of the two different sensors is S102a and S102b may be included as follows.

In S102a, interpolation processing is performed on at least one of the two angular velocity information, and the two angular velocities are aligned.

In the embodiment of the present application, the interpolation processing of the sensors having different structures may be different from the processing method of the sensors having the same structure. Therefore, based on the different structures of the two sensors, interpolation processing is performed on at least one of the two angular velocity information, and the two angular velocity information are aligned.

For example, when the two different sensors are both gyro sensors, the two different angular velocity information are aligned by performing interpolation processing on at least one of the angular velocity information collected by each of the obtained two gyro sensors. ..

Further, for example, when the two different sensors are both position / attitude sensors, at least one of the position / attitude rotation matrix information collected by each of the two position / attitude sensors is subjected to interpolation processing, and the two angular velocities are performed. Align the information.

Further, for example, when the two different sensors are the position / attitude sensor and the gyro sensor, the position / attitude rotation matrix information collected by the position / attitude sensor is subjected to interpolation processing to align the two angular velocity information.

To perform interpolation processing on at least one of the two angular velocity information is to perform interpolation processing on both of the two angular velocity information and to perform interpolation processing on any one of the two angular velocity information. Includes performing interpolation processing.

In one embodiment, performing interpolation processing on two angular velocity information selects one of the data acquisition frequencies as the standard data acquisition frequency, and based on the standard data acquisition frequency, the two angular velocity information are obtained. It includes performing interpolation processing for the data. Here, the standard data acquisition frequency may be between the data acquisition frequencies of the two sensors, or may be higher than the data acquisition frequency of the two sensors.

Illustratively, when the data acquisition frequencies of the two sensors are 10 Hz and 15 Hz, respectively, a data acquisition frequency higher than that of the two sensors can be selected as the standard data acquisition frequency. For example, the standard data acquisition frequency is 17 Hz. Of course, the frequency between the data acquisition frequencies of the two sensors can also be selected as the standard data acquisition frequency. For example, the standard data acquisition frequency is 13 Hz.

When the selected standard data acquisition frequency is 13 Hz, the angular velocity information acquired by the two sensors is subjected to interpolation processing based on the standard data acquisition frequency, and the angular velocity acquisition frequency of the first sensor and the angular velocity acquisition frequency of the second sensor are performed. The angular velocity acquisition frequency is set to be the same as the standard data acquisition frequency, and both are set to 13 Hz. As described above, the problem that the data processing has a deviation can be solved by the fact that the data acquisition frequencies of different sensors do not match. It contributes to obtaining more accurate delay time information based on the aligned angular velocity.

In one embodiment, if two different sensors are referred to as a first sensor and a second sensor, respectively, and the data acquisition frequency of the first sensor is higher than the data acquisition frequency of the second sensor, the two sensors are included. Performing the interpolation processing for one sensor includes performing the interpolation processing for the angular velocity information acquired by the second sensor based on the data acquisition frequency of the first sensor.

Illustratively, when the data acquisition frequency of the first sensor is 15 Hz and the data acquisition frequency of the second sensor is 10 Hz, the angular velocity information acquired by the second sensor is based on the data acquisition frequency of the first sensor of 15 Hz. Is subjected to interpolation processing so that the angular velocity acquisition frequency of the first sensor and the angular velocity acquisition frequency of the second sensor are the same, and both are set to 15 Hz. As described above, the problem that the data processing has a deviation can be solved by the fact that the data acquisition frequencies of different sensors do not match. It contributes to obtaining more accurate delay time information based on the aligned angular velocity.

In one embodiment, when two different sensors in the electronic device are the position / attitude sensor and the gyro sensor, respectively, the position / attitude rotation matrix information collected by the position / attitude sensor is subjected to interpolation processing. Align two angular velocity information. Specifically, S01, S02 and S03 may be included as follows.

In S01, the adjacent position / orientation rotation matrix information is acquired from the plurality of position / attitude rotation matrix information.

In the embodiment of the present application, after acquiring a plurality of position / orientation rotation matrix information, it is possible to acquire adjacent position / orientation rotation matrix information from the plurality of position / attitude rotation matrix information. The position / attitude rotation matrix information collected by the position / attitude sensor within a predetermined period is a plurality of position / attitude rotation matrix information.

Illustratively, when the position / orientation sensor is a visual sensor, the position / orientation rotation matrix information of adjacent frame images can be acquired. When the position / attitude sensor is a laser radar sensor, the position / attitude rotation matrix information within a predetermined period can be independently acquired by the laser radar sensor and used for estimating the rotational motion of the electronic device.

S02, Based on the frequency of the angular velocity information acquired by the gyro sensor, the adjacent position / attitude rotation matrix information is subjected to the interpolation processing, and the rotation matrix information after the interpolation is obtained.

In the embodiment of the present application, after the adjacent position / orientation rotation matrix information is acquired, the rotation matrix information after interpolation can be acquired based on the adjacent position / orientation rotation matrix information.

In the process of acquiring the rotation matrix information after interpolation based on the adjacent position / orientation rotation matrix information, an interpolation model is constructed, and further, the rotation matrix information after interpolation is obtained by the interpolation model and the adjacent position / orientation rotation matrix information. Can be obtained.

Interpolation models are used to infer other adjacent position-orientation rotation information between adjacent position-orientation rotation information by values at a limited number of adjacent position-orientation rotation information.

Illustratively, the interpolation model may include a spherical linear interpolation model, a cubic spline interpolation model, and a nearest neighbor interpolation model, and the examples of the present application do not limit the interpolation model.

In S03, differential processing is performed on the rotation matrix information after interpolation, angular velocity information corresponding to the position / attitude sensor is obtained, and the two angular velocity information acquired by each of the position / attitude sensor and the gyro sensor are aligned.

In the embodiment of the present application, after the rotation matrix information after interpolation is acquired, the rotation matrix information after interpolation is subjected to differential processing by a geometric method, and the angular velocity information corresponding to the position / orientation sensor can be acquired.

A differential model can be constructed in the process of performing differential processing on the rotation matrix information after interpolation and acquiring angular velocity information corresponding to the position / attitude sensor. The angular velocity information corresponding to the position / attitude sensor is acquired from the constructed differential model and the rotation matrix information after interpolation.

（１） Illustratively, the derivative model satisfies Eq. (1).

(1)

は、微分位置姿勢回転行列情報であり、

は、第２サブ角速度情報であり、

は、補間後の位置姿勢回転行列情報である。 here,

Is the differential position / attitude rotation matrix information,

Is the second sub-angular velocity information,

Is the position / orientation rotation matrix information after interpolation.

The differential model in the embodiment of the present application is a model for expressing angular velocity information. In some possible implementations, the derivative model may be a quaternion derivative model or a model of other mathematical representations, and the embodiments of the present application are not limited herein.

In S102b, the delay time information between the two different sensors is determined based on the two angular velocity information after the alignment.

In the embodiment of the present application, since the angular velocity information acquired by different sensors at the same time is the same in the process of rotating the electronic device, the delay between the two different sensors is based on the two angular velocity information after alignment. Time information can be determined.

Before determining the delay time information between two different sensors based on the two post-aligned angular velocity information, first build an error model, then based on the two post-aligned angular velocity information and the error model. It is possible to determine the delay time information between two different sensors.

In the embodiment of the present application, in the process of constructing the error model, different sensors correspond to different coordinate axes, and when comparing the angular velocity information of different sensors, it is necessary to perform the error with the coordinate axes corresponding to the same coordinate system. When constructing the model, the angular velocity information of the two different sensors at the same time is set to the same axis by introducing the rotation parameter between the different axes corresponding to each of the two different sensors.

In addition, since there is a deviation in the angular velocity information acquired by different sensors, an error parameter between the two sensors is introduced when constructing the error model, and the delay time information determined by the error model is made more accurate.

（２） Illustratively, the error model is as shown in Eq. (2).

(2)

は、２つの異なるセンサのそれぞれに対応する異なる座標軸の間の回転パラメータであり、

は、２つの異なるセンサ間の遅延時間情報であり、

は、２つの異なるセンサの間の誤差パラメータであり、

及び

はそれぞれ、同一の時刻ｔで、２つの異なるセンサのそれぞれにより取得された角速度情報であり、

は、角速度間の誤差である。 here,

Is the rotation parameter between the different axes corresponding to each of the two different sensors.

Is the delay time information between two different sensors,

Is the error parameter between two different sensors

as well as

Is the angular velocity information acquired by each of the two different sensors at the same time t, respectively.

Is the error between the angular velocities.

In one embodiment, after performing alignment processing on the acquired two angular velocity information and determining delay time information between two different sensors, the method performs alignment processing on the two angular velocity information. By determining the external parameters between the two different sensors, the external parameters are the rotation parameter between the different axes corresponding to each of the two different sensors and the error parameter between the two different sensors. Including, further including.

In the embodiment of the present application, not only the delay time information but also the rotation parameter and the error parameter between the two different sensors can be acquired by the two angular velocity information after the alignment, and the function of the electronic device can be further enriched.

FIG. 7 is a third schematic diagram showing a flow for realizing the time synchronization processing method according to the embodiment of the present application. As shown in FIG. 7, step S102b for determining the delay time information between two different sensors based on the two angular velocity information after alignment may include S102b1, S102b2 and S102b3 as follows.

In S102b1, the sub-error equations corresponding to different times of the two different sensors are determined based on the two angular velocity information after the alignment.

In the embodiment of the present application, after aligning the two angular velocity information, the sub-error equation corresponding to the different time of the two different sensors is obtained in the process of the electronic device performing the rotational motion based on the two angular velocity information after the alignment. Can be decided.

The different time n within a predetermined period may be preset according to the actual situation. For example, within a predetermined period, 15 times or 20 times can be set, and a sub-error equation corresponding to each time can be obtained.

及び

であり、時刻２に対応する異なるセンサにより収集された角速度情報がそれぞれ

及び

であり、このように類推して、時刻ｎに対応する異なるセンサにより収集された角速度情報がそれぞれ

及び

であると、誤差モデルが式（２）を満たす場合、その異なる時刻ｎに対応するサブ誤差方程式は、以下の通りである。 Illustratively, the angular velocity information collected by different sensors corresponding to time 1 is each.

as well as

The angular velocity information collected by the different sensors corresponding to time 2 is

as well as

By analogy with this, the angular velocity information collected by different sensors corresponding to the time n is each.

as well as

Then, when the error model satisfies the equation (2), the sub-error equations corresponding to the different time n are as follows.

（３） The sub-error equation corresponding to time 1 satisfies equation (3).

(3)

（４） The sub-error equation corresponding to time 2 satisfies equation (4).

(4)

（５） By analogy with this, the sub-error equation corresponding to the time n satisfies the equation (5).

(5)

In S102b2, the sum of the sub-error equations corresponding to different times is calculated to obtain the final error equation.

In the embodiment of the present application, after the sub-error equations corresponding to each time are determined, the sum of the sub-error equations corresponding to different times is calculated to obtain the final error equation.

In the embodiment of the present application, the delay time information is acquired by obtaining the cumulative error at different times, and the sum of the sub-error equations corresponding to the different times is calculated to obtain the final error equation.

（６） Illustratively, when the sub-error equations corresponding to different times are the above equations (3), (4) and (5), respectively, the corresponding final error equations satisfy equation (6).

(6)

In S102b3, the minimum value processing is performed on the final error equation, and the delay time information is obtained.

In the embodiment of the present application, after the final error equation is obtained, the final error equation can be subjected to the minimum value processing to obtain the delay time information.

The minimum value processing is for minimizing the value of the final error equation and further estimating the delay time information, the rotation parameter, and the error parameter between the sensors in the error model.

Illustratively, when the minimum value processing is performed on the final error equation, the minimum value processing can be performed on the final error equation by a nonlinear model or an iterative closest point model.

In the embodiment of the present application, in the process of performing the minimum value processing on the final error equation and obtaining the delay time information, the iterative closest point processing is performed on the final error equation, and the second minimization equation is obtained. Obtaining and solving the second minimization equation until the solution satisfies the predetermined second threshold, and after the solution of the second minimization equation satisfies the predetermined second threshold, the delay time information in the second minimization equation. And may include.

When the final error equation is repeatedly subjected to the closest point processing to obtain the second minimization equation, the delay time can be set within a predetermined period. For example, one delay time is selected by the golden section method and further substituted into the error model. In this case, the obtained error terms are only two unknowns, the rotation parameter and the error parameter between the sensors. Here, the rotation parameter and the error parameter between the sensors can be acquired by performing the iterative closest point processing again.

の間にあり、黄金分割法で選択される遅延時間が0.3であるとすれば、得られた第２最小化方程式は、式（７）を満たす。

（７） Illustratively, the delay time information

If the delay time selected by the golden section method is 0.3, the obtained second minimization equation satisfies Eq. (7).

(7)

は、角速度間の誤差であり、ｎは、最近傍点対の数であり、

は、２つのセンサ間の回転行列であり、

は、２つのセンサ間の誤差パラメータであり、

及び

はそれぞれ、２つのセンサにそれぞれ対応する角速度ポイントクラウドにおける１つのポイントである。 here,

Is the error between the angular velocities, n is the number of pairs of nearest emphasis marks, and

Is the rotation matrix between the two sensors

Is the error parameter between the two sensors

as well as

Is one point in the angular velocity point cloud corresponding to each of the two sensors.

In the embodiment of the present application, the process of performing the minimum value processing on the final error equation and obtaining the delay time information performs the nonlinear optimization processing on the final error equation to obtain the first minimization equation. That, the first minimization equation is solved until the solution satisfies the predetermined first threshold, and after the solution of the first minimization equation satisfies the predetermined first threshold, the delay time information in the first minimization equation is obtained. It may further include to obtain.

Since the final error equation is a non-linear function, it is necessary to perform a Taylor expansion on it, and by performing a minimization iteration on the final error equation, the delay time corresponding to when the predetermined first threshold is satisfied. It looks for information, thereby lowering the final error equation to the minimum.

In the embodiment of the present application, a nonlinear optimization model can be constructed in the process of performing the nonlinear optimization process on the final error equation and obtaining the first minimization equation. The first minimization equation is determined based on the final error equation and the nonlinear optimization model.

Illustratively, a given nonlinear optimization model may include a Gauss-Newton algorithm model or a Levenberg-Marquardt algorithm model, where the first threshold depends on the user's actual needs. It may be set, for example, 0.1 or 0.01, and the embodiment of the present application is not limited here.

In the embodiment of the present application, the first minimization equation is solved until the solution satisfies the predetermined first threshold, and after the solution of the first minimization equation satisfies the predetermined first threshold, the delay in the first minimization equation is reached. When it is determined that acquiring the time information is the determination of the delay time information for the first time, the current variable value is acquired based on the predetermined initial variable value and the predetermined nonlinear optimization model, and the initial variable is acquired. The delay time information in the first minimization equation is determined when the value of the current solution of the minimization equation is determined based on the value, the current variable value and the first minimization equation, and the value of the solution satisfies a predetermined first threshold. To get and include.

（８） The first equation consists of the final error equations corresponding to adjacent variable values. For example, the first minimization equation may be Eq. (8).

(8)

は、現在変数値であり、

は、初期変数値であり、

は、最小化方程式の現在の解の値である。 here,

Is the current variable value,

Is the initial variable value,

Is the value of the current solution of the minimization equation.

If it is determined that the delay time information before the first time is not determined, the previous variable value is acquired. Get the current variable value based on the previous variable value and a given nonlinear optimization model. The value of the current solution of the minimization equation is determined based on the previous variable value, the current variable value, and the first minimization equation, and if the value of the current solution satisfies a predetermined first threshold value, the first minimization Get the delay time information in the equation.

If the value of the current solution does not meet the predetermined first threshold, it is necessary to obtain the next variable value based on the current variable value and the predetermined nonlinear optimization model. Based on the next variable value, the current variable value, and the first minimization equation, the value of the next solution of the minimization equation is determined, and whether the value of the next solution satisfies the predetermined first threshold value is repeated and sequentially sequentially. The determination is made, the iteration is terminated, and the delay time information in the first minimization equation is determined until the value of the next solution satisfies the predetermined first threshold value.

In the embodiment of the present application, the final error equation can be minimized to obtain the delay time information, and at the same time, the rotation parameter and the sensor deviation parameter can be obtained. In this way, the measurement information of different coordinate systems can be converted into the same coordinate system by the obtained rotation parameters.

In one embodiment, based on the delay time information, time synchronization processing is performed on the measurement results of each of the two different sensors, and then fusion processing is performed on the measurement results after synchronization. It may further include performing at least one of positioning processing, distance measurement processing, target detection in the scene where the electronic device is located, and map generation or update based on the measurement result after the fusion processing.

In the embodiment of the present application, performing fusion processing on two measurement results after synchronization means that analysis and integration are performed on two measurement results at the same time to obtain a reliable fusion processing result. include.

Illustratively, by applying the fusion processing result to the positioning processing process, accurate positioning for an electronic device can be realized. By applying the fusion processing result to the distance measurement processing process, the measurement accuracy can be improved. By applying the fusion processing result to the target detection process in the scene where the electronic device is located, an accurate target detection result can be obtained. Accurate maps can be obtained by applying the fusion processing results to the map generation or update process.

In the embodiment of the present application, a fusion algorithm model can be constructed and an accurate fusion processing result can be obtained in the process of performing fusion with respect to the measurement result after synchronization.

Illustratively, the fusion algorithm model may include a Kalman filtering fusion algorithm model and a clustering analysis recognition algorithm model, and the embodiments of the present application are not limited herein.

In one embodiment, using the Kalman filtering fusion algorithm model, performing fusion processing on the measured results after synchronization uses the measurement results of two different sensors provided in the electronic device to perform state propagation. The position and orientation of the current time are estimated from the position and orientation estimation of the first measurement result of the previous time, and the second measurement result of the current time is used as observation information. It is to correct the primary estimation to obtain the optimum estimation of the position and orientation at the current time, and the optimum estimation includes the fusion processing result.

The time synchronization processing method in the embodiment of the present application is similarly applicable to three or more sensors. When the electronic device is provided with at least three sensors, the time-synchronized measurement results of the two sensors are acquired, the time-synchronized measurement results are fused, and the fusion-processed results are obtained. A more accurate measurement result can be obtained by performing the fusion process on the measurement results corresponding to at least three sensors as compared with the fusion process on the measurement results corresponding to the two sensors.

FIG. 8 is a fourth schematic diagram showing a flow for realizing the time synchronization processing method according to the embodiment of the present application. As shown in FIG. 8, in the embodiment of the present application, the two different sensors provided in the electronic device are a visual sensor and an inertial sensor, respectively, and the time synchronization processing method of the present embodiment may include the following steps.

In S201, the inertial sensor acquires the first angular velocity information when the electronic device performs a rotational motion.

In the embodiment of the present application, the inertial sensor in the electronic device can independently acquire the angular velocity information. Since the gyro unit is provided in the inertial sensor, the gyro unit can directly acquire the angular velocity information when the electronic device performs a rotational motion.

The inertial sensor is a sensor that measures triaxial attitude angle (or angular velocity) and acceleration, and the inertial sensor may further include an acceleration unit in addition to the gyro unit, where the acceleration unit is of its coordinates. Acceleration information of electronic devices on three axes can be detected.

In S202, the position-posture rotation matrix information when the electronic device performs a rotational movement is acquired by a visual sensor.

In the embodiment of the present application, the visual sensor in the electronic device can independently acquire the position / orientation rotation matrix information. A position / attitude estimation unit is provided in the visual sensor, and the position / attitude estimation unit can acquire information on the position / attitude rotation matrix when the electronic device performs a rotational movement.

The visual sensor mainly consists of one or two graphic sensors. It is also necessary to place a light projector and other auxiliary equipment. This is to obtain the position / orientation rotation matrix information by acquiring the original image within a predetermined period and comparing the acquired image with the reference in the memory, and further, the position / orientation rotation matrix information of the electronic device is obtained. Can be solved.

Illustratively, a quaternion can represent the rotational motion of an electronic device. Further, the rotational motion of the electronic device can be represented by the three-dimensional rotation group, and the embodiments of the present application do not limit this.

In S203, the second angular velocity information is determined based on the position / attitude rotation matrix information, and the second angular velocity information and the first angular velocity information are aligned.

In the embodiment of the present application, the image frequency of the visual sensor is generally 10 Hz, the frequency of the inertial sensor is generally 100 Hz, and approximately 10 inertial sensors are used between the two frames of the image by the visual sensor. Since there is the measured first angular velocity information, the position / orientation rotation matrix information of the visual sensor is interpolated, the frequencies of the visual sensor and the inertial sensor are matched, and the position / orientation rotation matrix information after the interpolation is used for acquisition. The generated second angular velocity information can be aligned with the first angular velocity information.

In this embodiment, the interpolation process can be performed on at least one of the second angular velocity information and the first angular velocity information by the interpolation model. The interpolation model used also differs depending on the various forms of representation of the rotational motion of the electronic device. For example, for a rotational motion represented by a quaternion, interpolation processing can be performed on the position-posture rotation matrix by a spherical linear interpolation model. For the rotational motion represented by the three-dimensional rotation group, interpolation processing can be performed on the position-orientation rotation matrix information by the cubic spline interpolation model and the nearest neighbor interpolation model.

For rotational motion represented by a quaternion, it is required to represent three-dimensional rotation with four parameters. For the rotational motion represented by the three-dimensional rotation group, it is required to represent the three-dimensional rotation with nine parameters.

Hereinafter, a process of performing interpolation processing on the position / orientation rotation matrix information of the visual sensor will be exemplified by taking a spherical linear interpolation model as an example for the rotational motion represented by a quaternion.

（９） Illustratively, a quaternion consists of a real number and an imaginary number, as shown in equation (9).

(9)

、

、

、

は、実数であり、i、j、kは、相互直交する虚数単位であり、ｑは、四元数で表される位置姿勢回転行列情報である。 here,

,

,

,

Is a real number, i, j, and k are imaginary units that are orthogonal to each other, and q is position-orientation rotation matrix information represented by a quaternion.

The method for expressing the position-posture rotation matrix information by a quaternion includes a triangular expression and an exponential expression, and the exponential expression expression is as shown in the equation (10).

（１０）

(10)

here,

Is the angular velocity, t is the time, and q is the position-posture rotation matrix information represented by the exponential mapping of the quaternion.

（１１） Illustratively, the constructed spherical interpolation model is as shown in Eq. (11).

(11)

及び

はそれぞれ、隣接する姿勢回転行列であり、

は、姿勢回転行列に対して球面補間処理を行うことで得られた補間後の位置姿勢回転行列情報である。 here,

as well as

Are adjacent attitude rotation matrices, respectively.

Is the position / orientation rotation matrix information after interpolation obtained by performing the spherical interpolation processing on the attitude rotation matrix.

、

及び球面補間モデルにより、補間後の位置姿勢回転行列情報を取得することができる。 Posture rotation matrix information

,

The position / orientation rotation matrix information after interpolation can be acquired by the spherical interpolation model.

Illustratively, FIG. 9 is a first schematic diagram showing that the quaternion according to the embodiment of the present application represents the position / attitude rotation matrix information measured by the position / attitude type sensor. As shown in FIG. 9, the electronic device continuously moves and rotates from the previous time to the current time. That is, it rotates from a solid line circle to a dotted line circle. FIG. 10 is a second schematic diagram showing that the quaternion according to the embodiment of the present application represents the position / attitude rotation matrix information measured by the position / attitude type sensor. As shown in FIG. 10, it is a plan view of the dotted line frame taken out from FIG.

及び

はそれぞれ、隣接する姿勢回転行列であり、

は、姿勢回転行列に対して球面補間処理を行うことで得られた補間後の位置姿勢回転行列情報であり、回転角度は、

であり、ｔは時間であり、０≦ｔ≦１である。

as well as

Are adjacent attitude rotation matrices, respectively.

Is the position / attitude rotation matrix information after interpolation obtained by performing spherical interpolation processing on the attitude rotation matrix, and the rotation angle is

T is time, and 0 ≦ t ≦ 1.

In the embodiment of the present application, in the process of acquiring the second angular velocity information based on the position / orientation rotation matrix information after interpolation, the angular velocity information corresponding to the visual sensor is based on the position / orientation rotation matrix information after interpolation and the differential model. Can be determined. The differential model is as shown in Eq. (1).

The second angular velocity information can be indirectly acquired by first performing interpolation processing on the position / attitude rotation matrix information and then performing differential processing. In this way, time synchronization between many sensors is performed. It should be understood that it is adaptable and versatile.

In S204, the delay time information between the visual sensor and the inertial sensor is determined.

In the embodiment of the present application, after the alignment processing is performed on the first angular velocity information and the second angular velocity information, the time delay information between the visual sensor and the inertial sensor is based on the first angular velocity information and the second angular velocity information. Can also be determined.

In another embodiment, the first angular velocity information and the second angular velocity information can be used not only to solve the delay time information between the visual sensor and the inertial sensor, but also to solve the error parameter between the rotation parameter and the sensor. You can also.

In S205, time synchronization processing is performed on the measurement result of the visual sensor and the measurement result of the inertial sensor based on the delay time information.

The time synchronization processing method provided in the embodiment of the present application is applicable to unmanned driving or mobile robot navigation. In the unmanned driving or mobile robot navigation process, the unmanned driving electronic device or the mobile robot electronic device can realize accurate positioning by the time synchronization processing method provided in the embodiment of the present application.

In the embodiment of the present application, only the process of the time synchronization processing method by the electronic device will be described by taking the visual sensor and the inertial sensor as an example, and the time synchronization processing method provided in the embodiment of the present application is the visual sensor and the time synchronization processing method. It is not limited to the two sensors called the inertia sensor, and can be applied to other sensors. Other sensors may be able to independently acquire angular velocity information and position / attitude rotation matrix information. At the same time, the embodiments of the present application are also applicable to electronic devices including three or more sensors, that is, the time synchronization processing method is similarly applicable to electronic devices including three or more sensors. ..

Based on the above time synchronization processing method, the embodiment of the present application provides a time synchronization processing apparatus. FIG. 11 is a schematic view showing the structure of the time synchronization processing apparatus according to the embodiment of the present application. As shown in FIG. 11, the time synchronization processing device 300 includes a first acquisition module 301, an alignment module 302, and a synchronization module 303.
The first acquisition module 301 is configured to acquire angular velocity information collected by two different sensors, respectively, and the angular velocity information is angular velocity information when an electronic device performs a rotational motion, and the two different sensors are Both are provided in the electronic device and are fixedly connected.
The alignment module 302 is configured to perform an alignment process on the obtained two angular velocity information and determine the delay time information between the two different sensors.
The synchronization module 303 is configured to perform time synchronization processing on the measurement results of the two different sensors based on the delay time information.

The time synchronization processing device of the embodiment of the present application is required to be capable of performing a rotary motion of an electronic device. That is, the electronic device can calibrate the delay time information between different sensors in the electronic device if it performs at least rotation about the axis of rotation. It does not require movement around multiple axes, reduces the time synchronization complexity of different sensors, and can adapt to the demands of different scenes. Also, in the embodiments of the present application, the delay time can be calibrated based on the rotational motion generated by the rotation of the electronic device around a plurality of axes. As a result, richer rotation information can be obtained and the accuracy of time synchronization can be improved. In the embodiment of the present application, it is required that the sensor can acquire the angular velocity information independently. This makes it possible to determine the delay time information between different sensors. It is widely applicable to time synchronization between multiple sensors and has general-purpose adaptability. In the embodiment of the present application, it is a method of realizing time synchronization based on software, and it is not necessary to separately arrange dedicated hardware for time synchronization. In the embodiment of the present application, time synchronization can be performed and time synchronization processing can be performed online based on the delay time information acquired in real time. In the embodiment of the present application, the delay time information is determined based on the angular velocity information acquired when the electronic device performs the rotational motion. Improves the accuracy of time synchronization without processing the delay time information as a constant.

In another embodiment, the first acquisition module 301 acquires the position / orientation rotation matrix information collected by the position / attitude sensor, and based on the position / attitude rotation matrix information, the angular velocity information when the electronic device performs a rotational motion. At least one of the two different sensors is a position-attitude sensor.

In another embodiment, the first acquisition module 301 is configured to acquire the angular velocity information collected by the gyro sensor, and at least one of the two different sensors is a gyro sensor.

In another embodiment, the alignment module 302 performs an interpolation process on at least one of the two angular velocity information, aligns the two angular velocity information, and is based on the two post-alignment angular velocity information. , It is configured to determine the delay time information between the two different sensors.

In another embodiment, the alignment module 302 is attached to at least one of the acquired angular velocity information collected by each of the two gyro sensors when the two different sensors are both gyro sensors. It is configured to perform interpolation processing on the surface and align the two angular velocity information.

In another embodiment, the alignment module 302 has the position / orientation rotation matrix information acquired by each of the two position / attitude sensors when the two different sensors are both position / attitude sensors. It is configured to perform an interpolation process on at least one of the two angular velocity information to align the two angular velocity information.

In another embodiment, the alignment module 302 interpolates with the acquired position-attitude rotation matrix information collected by the position-attitude sensor when the two different sensors are a position-attitude sensor and a gyro sensor, respectively. It is configured to perform processing and align the two angular velocity information.

In another embodiment, the time synchronization processing device 300 is configured to perform alignment processing on the two angular velocity information by the alignment module 302 and determine an external parameter between the two different sensors. Further comprising an acquisition module 304, said external parameters include rotation parameters between different axes corresponding to each of the two different sensors and error parameters between the two different sensors.

In another embodiment, the time synchronization processing device 300 further comprises a storage module 305 configured to store the delay time information.
When the electronic device is in a non-moving state, the synchronization module performs time synchronization processing on the measurement results of the two different sensors based on the delay time information stored by the storage module 305. It is configured as follows.

In another embodiment, the alignment module 302 determines sub-error equations corresponding to different times of the two different sensors based on the two angular velocity information after the alignment, and the sub-errors corresponding to the different times. It is configured to calculate the sum of the equations, obtain the final error equation, perform the minimum value processing on the final error equation, and obtain the delay time information.

In another embodiment, the alignment module 302 performs non-linear processing on the final error equation to obtain a first minimization equation and the first minimization equation until the solution meets a predetermined first threshold. Is solved, and after the solution of the first minimization equation satisfies a predetermined first threshold value, the delay time information in the first minimization equation is acquired.

In another embodiment, the alignment module 302 performs iterative closest point processing on the final error equation to obtain a second minimization equation and the second until the solution meets a predetermined second threshold. It is configured to solve the minimization equation and acquire the delay time information in the second minimization equation after the solution of the second minimization equation satisfies a predetermined second threshold value.

In another embodiment, the time synchronization processing device 300 is
A fusion module 307 configured to perform fusion processing on the measured results after synchronization,
With an execution module 308 configured to perform at least one of positioning processing, distance measurement processing, target detection in the scene where the electronic device is located, map generation or update, based on the measurement results after fusion processing. , Are further provided.

When the time synchronization processing apparatus provided in the above embodiment performs the time synchronization processing, only the division of each of the above program modules will be described as an example, and in the actual application, the above processing will be performed by different program modules as needed. Can be completed. That is, the internal structure of the device is divided into different program modules, and all or part of the above-mentioned processing is completed. The time synchronization processing apparatus provided in the above embodiment belongs to the same concept as the time synchronization processing method, and for the specific realization process, refer to the example of the method. A detailed description will be omitted here.

The embodiments of the present application further provide electronic devices. FIG. 12 is a schematic view showing the structure of an electronic device according to an embodiment of the present application. As shown in FIG. 12, the electronic device comprises at least a processor 21 and a memory 23 configured to store a computer program that can be executed by the processor, when the processor 21 executes the computer program. It is configured to perform a step in the time synchronization processing method provided in the above embodiment.

The electronic device for performing the steps in the time synchronization processing method provided in the above embodiment in this embodiment may be the same as or different from the electronic device provided with the two different sensors.

Optionally, the electronic device may further include a communication interface 24, which is configured to acquire angular velocity information and position / attitude rotation matrix information. Each component in an electronic device is coupled via a bus system 25. It should be understood that the bus system 25 is intended to provide connectivity and communication between these components. The bus system 25 includes a power bus, a control bus, and a status signal bus, in addition to including a data bus. However, for the sake of clarity, all buses are referred to as bus system 25 in FIG.

It should be understood that the memory 23 may be a volatile memory or a non-volatile memory, or may be both a volatile memory and a non-volatile memory. Here, the non-volatile memory includes a read-only memory (ROM: Read Only Memory), a programmable read-only memory (PROM: Programmable Read-Only Memory), and an erasable programmable read-only memory (EPROM: Erasable Programmable Read-Only Memory). , Electrically erasable programmable read-only memory (EEPROM: Electrically Erasable Read-Only Memory), Magnetic Random Access Memory (FRAM: ferromagnetic random access memory), Flash memory (Flash memory), Flash memory (Flash memory) It may be a dedicated optical disk (CD-ROM: Compact Disc Read-Only Memory). The magnetic surface memory may be a magnetic disk memory or a magnetic tape memory. The volatile memory may be a random access memory (RAM: Random Access Memory) used as an external cache. As a non-limiting example, the RAM includes a static random access memory (SRAM: Static Random Access Memory), a synchronous static random access memory (SSRAM: Synchronous Static Access Memory), and a dynamic random access memory (DRAM: Dynamic Random Access Memory). , Synchronous Dynamic Random Access Memory (SDRAM: Synchronous Dynamic Random Access Memory), Double Data Rate Synchronous Dynamic Random Access Memory (DDR DRAM: Double Data Synchronous Dynamic Random Memory Synchronous Dynamic Random Access Memory) Synchronous Dynamic Random Access Memory (Synchronous Dynamic Random Access Memory) Use in many forms such as Access Memory), Synchronic Dynamic Random Access Memory (SLRAM: Synchlink Dynamic Random Access Memory) and Direct Rambus Random Access Memory (DRRAM: Direct Rambus Random Access Memory). The memory 23 described in the embodiments of the present invention includes, but is not limited to, these and any other suitable form of memory.

The method disclosed in the embodiment of the present invention is applied to or realized by the processor 21. The processor 21 can be an integrated circuit chip capable of processing signals. Each step of the above method can be completed by a command in the form of a hardware integrated logic circuit or software in the processor 21 in the process of realization. The processor 21 may be a general purpose processor, DSP, or other programmable logic device, discrete gate or transistor logic device, discrete hardware component, or the like. The processor 21 can realize or execute each method, step and logical block diagram disclosed in the embodiments of the present invention. The general-purpose processor may be a microprocessor, and the processor may be any conventional processor or the like. In accordance with the steps of the method disclosed in the examples of the present invention, it is shown to be executed by a hardware decoding processor or executed by a combination of hardware and software modules in the decoding processor to be completed. The software module may reside in the storage medium. The storage medium is located in memory 23 and the processor 21 reads the information in memory 23 and completes the steps of the above method with its hardware.

Any combination of the features disclosed in the embodiments of the plurality of methods or devices provided in the embodiments of the present application, as long as no logic error is generated or inconsistent, is an embodiment of the new method or an embodiment of the device. Can be obtained.

Based on the embodiments described above, the embodiments of the present application provide a computer-readable storage medium. When a computer program is stored in the computer-readable storage medium and the computer program is executed by the processor, the steps of the time synchronization processing method in the above embodiment are realized.

It should be understood that in some of the embodiments provided herein, the disclosed devices and methods can be implemented by other methods. For example, the embodiment of the device described above is merely an example. For example, the division of the unit is merely a division of a logic function, and when it is actually realized, another division method may be used. For example, a plurality of units or assemblies may be combined or incorporated into another system. Alternatively, some features may or may not be implemented. Also, the mutual or direct coupling or communication connection shown or considered may be an indirect coupling or communication connection by some interface, device or unit, electrical, mechanical or other. It may be in the form of.

The unit described as a separating member may or may not be physically separate. The member shown as a unit may or may not be a physical unit. That is, it may be located at the same position or may be distributed over a plurality of networks. The objectives of the measures of this embodiment can be achieved by some or all of the units depending on the actual demand.

Further, each functional unit in each embodiment of the present application may be integrated in one processing unit, each unit may exist as physically separate units, or two or more units may be one unit. It may be accumulated in. The integrated unit may be realized as hardware, or may be realized by a combination of hardware and a software function unit.

All or part of the steps of the embodiment according to each of the above methods are realized by the hardware related to the program instruction, and the program may be stored in a computer-readable storage medium, and when the program is executed, the above-mentioned program is executed. It is possible for those skilled in the art to carry out the steps in the embodiment of the method and that the storage medium includes various media capable of storing the program code, such as a portable storage device, ROM, RAM, magnetic disk or optical disk. Should be understood.

Alternatively, when the integrated unit of the present application is realized in the form of a software functional unit and sold or used as an independent product, it may be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the present application essentially, or parts that have contributed to the prior art, or parts of the technical solutions, are embodied in the form of software products. Such computer software products may be stored in storage media and may be stored in computer equipment (personal computers, servers, network devices, etc.) in all or one of the methods described in each embodiment of the present application. Includes some instructions to execute the steps of the part. The storage medium includes various media capable of storing a program code, such as a portable storage device, a ROM, a RAM, a magnetic disk, or an optical disk.

The above are merely embodiments of the present application, and the scope of protection of the present application is not limited thereto, and any changes or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present application are described in the present application. Should be included within the scope of protection of. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims (28)

It is a time synchronization processing method
Obtaining angular velocity information collected by two different sensors, the angular velocity information is angular velocity information when an electronic device performs a rotational motion, and both of the two different sensors are attached to the electronic device. Being provided and fixedly connected,
Alignment processing is performed on the two obtained angular velocity information to determine the delay time information between the two different sensors.
A time synchronization processing method including performing time synchronization processing on the measurement results of each of the two different sensors based on the delay time information. At least one of the two different sensors is a position-attitude sensor, and obtaining angular velocity information collected by the two different sensors can be obtained.
It is characterized by acquiring the position / orientation rotation matrix information collected by the position / attitude sensor and determining the angular velocity information when the electronic device performs a rotational motion based on the position / attitude rotation matrix information. The method according to claim 1. At least one of the two different sensors is a gyro sensor, and obtaining angular velocity information collected by the two different sensors can be obtained.
The method according to claim 1 or 2, wherein the method comprises acquiring the angular velocity information collected by the gyro sensor. Alignment processing is performed on the two obtained angular velocity information to determine the delay time information between the two different sensors.
Interpolation processing is performed on at least one of the two angular velocity information to align the two angular velocity information.
The method according to claim 1, wherein the delay time information between the two different sensors is determined based on the two angular velocity information after the alignment. Performing interpolation processing on at least one of the two angular velocity information is possible.
When both of the two different sensors are gyro sensors, it is characterized by including performing interpolation processing on at least one of the acquired angular velocity information collected by each of the two gyro sensors. The method according to claim 4. Performing interpolation processing on at least one of the two angular velocity information is possible.
When both of the two different sensors are position / attitude sensors, the interpolation process is performed on at least one of the acquired position / attitude rotation matrix information collected by each of the two position / attitude sensors. The method according to claim 4, wherein the method comprises. Performing interpolation processing on at least one of the two angular velocity information is possible.
When the two different sensors are a position / attitude sensor and a gyro sensor, respectively, the claim comprises performing an interpolation process on the acquired position / attitude rotation matrix information collected by the position / attitude sensor. Item 4. The method according to Item 4. After performing an alignment process on the obtained two angular velocity information and determining the delay time information between the two different sensors, the method is performed.
By performing alignment processing on the two angular velocity information, an external parameter between the two different sensors is determined, and the external parameter is a different coordinate axis corresponding to each of the two different sensors. The method of any one of claims 1-7, further comprising: including a rotation parameter between and an error parameter between the two different sensors. After performing an alignment process on the obtained two angular velocity information and determining the delay time information between the two different sensors, the method is performed.
To store the delay time information and
When the electronic device is in a non-moving state, it is characterized by further including performing time synchronization processing on the measurement results of each of the two different sensors based on the stored delay time information. The method according to any one of claims 1-8. Determining the delay time information between the two different sensors based on the two angular velocity information after alignment can be done.
To determine the sub-error equations corresponding to different times of the two different sensors based on the two angular velocity information after the alignment.
To obtain the final error equation by calculating the sum of the sub-error equations corresponding to the different times,
The method according to claim 4, further comprising performing minimum value processing on the final error equation to obtain the delay time information. Performing minimum value processing on the final error equation and obtaining the delay time information is not possible.
Non-linear processing is performed on the final error equation to obtain the first minimization equation.
The delay time information in the first minimization equation is solved after the first minimization equation is solved until the solution satisfies the predetermined first threshold value and the solution of the first minimization equation satisfies the predetermined first threshold value. 10. The method of claim 10, wherein the method comprises: Performing minimum value processing on the final error equation and obtaining the delay time information is not possible.
Iterative closest point processing is performed on the final error equation to obtain the second minimization equation.
The second minimization equation is solved until the solution satisfies the predetermined second threshold value, and after the solution of the second minimization equation satisfies the predetermined second threshold value, the delay time information in the second minimization equation is satisfied. 10. The method of claim 10, wherein the method comprises: After performing time synchronization processing on the measurement results of each of the two different sensors based on the delay time information, the method is:
Performing fusion processing on the measurement results after synchronization,
It is characterized by further including performing at least one of positioning processing, distance measurement processing, target detection in the scene where the electronic device is located, and map generation or update based on the measurement result after the fusion processing. The method according to any one of claims 1-12. A time synchronization processing device, the device including a first acquisition module, an alignment module, and a synchronization module.
The first acquisition module is configured to acquire angular velocity information collected by two different sensors, respectively, and the angular velocity information is angular velocity information when an electronic device performs a rotational motion, and the two different sensors Both are provided in the electronic device and are fixedly connected.
The alignment module is configured to perform an alignment process on the obtained two angular velocity information and determine the delay time information between the two different sensors.
The synchronization module is a time synchronization processing device configured to perform time synchronization processing on the measurement results of each of the two different sensors based on the delay time information. The first acquisition module acquires the position / orientation rotation matrix information collected by the position / attitude sensor, and determines the angular velocity information when the electronic device performs a rotational motion based on the position / attitude rotation matrix information. The device of claim 14, wherein at least one of the two different sensors configured is a position-attitude sensor. The first acquisition module is configured to acquire the angular velocity information collected by the gyro sensor, and claim 14 or 15 is characterized in that at least one of the two different sensors is a gyro sensor. The device described in. The alignment module performs interpolation processing on at least one of the two angular velocity information, aligns the two angular velocity information, and based on the two post-aligned angular velocity information, between the two different sensors. 14. The apparatus of claim 14, wherein the device is configured to determine the delay time information of. When the two different sensors are both gyro sensors, the alignment module performs interpolation processing on at least one of the acquired angular velocity information collected by each of the two gyro sensors. 17. The apparatus according to claim 17, wherein the two angular velocity information are configured to be aligned. The alignment module provides for at least one of the acquired position / attitude rotation matrix information collected by each of the two position / attitude sensors when the two different sensors are both position / attitude sensors. The apparatus according to claim 17, wherein the apparatus is configured to perform an interpolation process and align the two angular velocity information. When the two different sensors are a position / attitude sensor and a gyro sensor, the alignment module performs interpolation processing on the acquired position / attitude rotation matrix information collected by the position / attitude sensor, and performs interpolation processing on the two different sensors. 17. The apparatus of claim 17, wherein the apparatus is configured to align angular velocity information. The device is
A second acquisition module configured to perform alignment processing on the two angular velocity information by the alignment module and determine an external parameter between the two different sensors is further provided, and the external parameter is the two. The apparatus according to any one of claims 14-20, comprising: a rotation parameter between different axes corresponding to each of the different sensors and an error parameter between the two different sensors. The device is
A storage module configured to store the delay time information is further provided.
When the electronic device is in a non-moving state, the synchronization module performs time synchronization processing on the measurement results of the two different sensors based on the delay time information stored by the storage module. The apparatus according to any one of claims 14-21. The alignment module determines the sub-error equations corresponding to different times of the two different sensors based on the two angular velocity information after the alignment, and calculates the sum of the sub-error equations corresponding to the different times. The apparatus according to claim 17, wherein the apparatus is configured to obtain a final error equation, perform minimum value processing on the final error equation, and obtain the delay time information. The alignment module performs non-linear processing on the final error equation, obtains a first minimization equation, solves the first minimization equation until the solution satisfies a predetermined first threshold value, and the first minimum. 23. The apparatus of claim 23, wherein the delay time information in the first minimization equation is acquired after the solution of the equation of equations satisfies a predetermined first threshold value. The alignment module performs iterative closest point processing on the final error equation, obtains a second minimization equation, solves the second minimization equation until the solution satisfies a predetermined second threshold value, and then performs the second minimization equation. 23. The second aspect of claim 23 is characterized in that the delay time information in the second minimization equation is acquired after the solution of the second minimization equation satisfies a predetermined second threshold value. Device. The device is
A fusion module configured to perform fusion processing on the measured results after synchronization,
An execution module configured to perform at least one of positioning processing, distance measurement processing, target detection in a scene where an electronic device is located, map generation or update, based on the measurement results after fusion processing. The apparatus according to any one of claims 14-25, further comprising: An electronic device, wherein the electronic device comprises at least a processor and a memory configured to store a computer program that can be executed by the processor, and the processor is claimed when the computer program is executed. An electronic device configured to perform the method according to any one of 1 to 13. A computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, the method according to any one of claims 1 to 13 is realized. , Computer readable storage medium.

Images