CN113556366A

CN113556366A - Multi-sensor data synchronization method and system and electronic equipment

Info

Publication number: CN113556366A
Application number: CN202010262813.3A
Authority: CN
Inventors: 徐文杰; 王城特; 张本好; 胡增新
Original assignee: Sunny Optical Zhejiang Research Institute Co Ltd
Current assignee: Sunny Optical Zhejiang Research Institute Co Ltd
Priority date: 2020-04-07
Filing date: 2020-04-07
Publication date: 2021-10-26
Anticipated expiration: 2040-04-07
Also published as: CN113556366B

Abstract

A multi-sensor data synchronization method, a system and an electronic device thereof are provided. The multi-sensor data synchronization method comprises the following steps: respectively storing multi-frame data acquired by sampling of a plurality of sensors to corresponding data cache units to obtain multi-frame cache data corresponding to the plurality of sensors one by one; and performing data synchronization on the cache data corresponding to the sensor to be synchronized in the plurality of sensors by respectively delaying for a preset time by taking the cache data corresponding to the reference sensor in the plurality of sensors as a reference so as to obtain corresponding synchronization frame cache data, wherein the frame rate of the reference sensor in the plurality of sensors is minimum.

Description

Multi-sensor data synchronization method and system and electronic equipment

Technical Field

The invention relates to the technical field of multiple sensors, in particular to a multi-sensor data synchronization method, a multi-sensor data synchronization system and electronic equipment.

Background

In the artificial intelligence and robotics industries, slam (simultaneous Localization and mapping) is used as a basic technology, mainly for the instant Localization and mapping in the face of unknown environments. In the practical application of SLAM, whether to quickly and accurately acquire data of multiple sensors (such as a vision sensor, a depth sensor, or an IMU) at the same time point directly affects the accuracy and efficiency of positioning and map building.

Currently, the existing multi-sensor synchronization is generally synchronized through hardware, that is, a plurality of sensors are simultaneously started through hardware setting, and data is reported simultaneously. However, although the hardware synchronization scheme can directly acquire the synchronization data of multiple sensors, the implementation difficulty and the cost are high, so that the hardware synchronization scheme is difficult to popularize and apply in a wide range. In particular, different sensors often have different sampling frame rates, i.e., it is difficult for multiple sensors to sample data simultaneously, which results in the inability to quickly and reliably acquire synchronized multiple sensor data even if the multiple sensors are activated in synchronization.

Disclosure of Invention

An advantage of the present invention is to provide a multi-sensor data synchronization method, a system and an electronic device thereof, which can reduce the cost of hardware implementation and facilitate the popularization and application of multi-sensor data synchronization technology.

Another advantage of the present invention is to provide a multi-sensor data synchronization method, a system and an electronic device thereof, wherein in an embodiment of the present invention, the multi-sensor data synchronization method does not need to perform hardware synchronization on multiple sensors, and can implement data synchronization of multiple sensors only through software synchronization, so as to reduce difficulty in implementing data synchronization.

Another advantage of the present invention is to provide a multi-sensor data synchronization method, a system and an electronic device thereof, wherein in an embodiment of the present invention, the multi-sensor data synchronization method can start a plurality of sensors in sequence and then perform data synchronization, which is helpful to greatly reduce the cost required for data synchronization.

Another advantage of the present invention is to provide a multi-sensor data synchronization method, a system and an electronic device thereof, wherein in an embodiment of the present invention, the multi-sensor data synchronization method can efficiently and reliably acquire synchronization data of multiple sensors, which is beneficial to application and popularization of SLAM technology.

Another advantage of the present invention is to provide a multi-sensor data synchronization method, a system and an electronic device thereof, wherein in an embodiment of the present invention, the multi-sensor data synchronization method can use a circular buffer mechanism to implement data caching sampled by each sensor, so as to avoid data movement of the remaining sampling caches after new sampled data is acquired, thereby facilitating implementation of efficient sampling data caching and reducing overhead when a CPU operates.

Another advantage of the present invention is to provide a multi-sensor data synchronization method, a system and an electronic device thereof, wherein in an embodiment of the present invention, the multi-sensor data synchronization method can perform synchronization from an oldest reference frame, which helps to ensure that the reference frame can find a nearest neighboring frame to be synchronized, so as to improve reliability of synchronization data.

Another advantage of the present invention is to provide a multi-sensor data synchronization method, a system and an electronic device thereof, wherein in an embodiment of the present invention, the multi-sensor data synchronization method can quickly and reliably acquire synchronization data of multiple sensors when the frame rates of the multiple sensors or the sensors are not stable.

Another advantage of the present invention is to provide a multi-sensor data synchronization method, a system and an electronic device thereof, wherein the method does not require complex structure and huge calculation amount, and has low requirements on software and hardware. Therefore, the present invention successfully and effectively provides a solution to not only provide a multi-sensor data synchronization method and system thereof, and an electronic device, but also increase the practicality and reliability of the multi-sensor data synchronization method and system thereof, and the electronic device.

To achieve at least one of the above advantages or other advantages and objects, the present invention provides a multi-sensor data synchronization method, including the steps of:

respectively storing multi-frame data acquired by sampling of a plurality of sensors to corresponding data cache units to obtain multi-frame cache data corresponding to the plurality of sensors one by one; and

and performing data synchronization on the cache data corresponding to the sensor to be synchronized in the plurality of sensors by respectively delaying for a preset time by taking the cache data corresponding to a reference sensor in the plurality of sensors as a reference so as to obtain corresponding synchronization frame cache data, wherein the frame rate of the reference sensor in the plurality of sensors is the minimum.

In an embodiment of the present invention, the predetermined time is greater than or equal to a sampling interval time of the sensor to be synchronized.

In an embodiment of the invention, the predetermined time is equal to the sampling interval time of the reference sensor.

In an embodiment of the present invention, before the step of respectively storing the multiple frames of data sampled and acquired by the multiple sensors into the corresponding data buffer units to obtain the multiple frames of buffer data corresponding to the multiple sensors one to one, the method for synchronizing data of multiple sensors further includes the steps of:

and respectively allocating the storage sizes of the data cache units corresponding to the plurality of sensors so that the cache data corresponding to the reference sensor and the cache data corresponding to the sensor to be synchronized have time stamps overlapped.

In an embodiment of the present invention, in the step of allocating the storage sizes of the data cache units corresponding to the plurality of sensors respectively:

and according to the frame rate of the plurality of sensors, respectively carrying out proportional allocation on the storage sizes of the data cache unit corresponding to the reference sensor and the data cache unit corresponding to the sensor to be synchronized.

In an embodiment of the present invention, the data caching unit stores data by using a circular cache region mechanism.

the read sensors are activated for sampling in a predetermined sequence, wherein the reference sensor is activated last.

In an embodiment of the present invention, after all the sensors to be synchronized are activated, the predetermined time is delayed, and then the reference sensor is activated.

In an embodiment of the present invention, the step of performing data synchronization on the buffered data corresponding to the sensor to be synchronized among the plurality of sensors by delaying for a predetermined time respectively with reference to the buffered data corresponding to the reference sensor among the plurality of sensors to obtain corresponding synchronized frame buffered data, where a frame rate of the reference sensor among the plurality of sensors is the minimum, includes the steps of:

when the current reference frame cache data is stored in the reference frame data cache unit, respectively delaying the preset time to select the frame cache data to be synchronized closest to the time stamp of the current reference frame cache data from the frame data cache units to be synchronized, so as to serve as the matched frame cache data corresponding to the current reference frame cache data;

judging whether the time interval of the timestamp between the matched frame cache data and the current reference frame cache data is less than or equal to a preset interval threshold value;

in response to the timestamp interval time being less than or equal to the predetermined interval threshold, determining the matching frame buffer data as the synchronized frame buffer data corresponding to the current reference frame buffer data; and

and skipping the current reference frame buffer data to perform data synchronization on the next reference frame buffer data in response to the time stamp interval time being greater than the predetermined interval threshold.

In an embodiment of the invention, the predetermined interval threshold is half of the sampling interval time of the sensor to be synchronized or the sampling interval time of the reference sensor.

In an embodiment of the present invention, the multi-sensor data synchronization method further includes the steps of:

and reporting the synchronous frame cache data and the corresponding reference frame cache data to an SLAM algorithm processing unit for synchronous positioning and map construction.

According to another aspect of the present invention, the present invention further provides a multi-sensor data synchronization system, comprising:

the data storage module is used for respectively storing the multi-frame data acquired by sampling of the plurality of sensors into the corresponding data cache units so as to obtain the multi-frame cache data corresponding to the plurality of sensors one by one; and

and the data synchronization module is used for delaying preset time respectively to perform data synchronization on the cache data corresponding to the sensor to be synchronized in the plurality of sensors by taking the cache data corresponding to the reference sensor in the plurality of sensors as a reference so as to obtain corresponding synchronous frame cache data, wherein the frame rate of the reference sensor in the plurality of sensors is the minimum.

In an embodiment of the present invention, the multi-sensor data synchronization system further includes an allocation module, where the storage allocation module is configured to allocate storage sizes of the data cache units corresponding to the multiple sensors, respectively, so that time stamps of the cache data corresponding to the reference sensor and the cache data corresponding to the sensor to be synchronized overlap each other.

In an embodiment of the present invention, the multi-sensor data synchronization system further includes an activation module, wherein the activation module is configured to activate the reading sensors to perform sampling according to a predetermined sequence, and wherein the reference sensor is activated last.

In an embodiment of the present invention, the data synchronization module includes:

a selecting module, configured to delay the predetermined time to select, from the frame-to-be-synchronized data caching unit, the frame-to-be-synchronized data that is closest to the timestamp of the current reference frame cache data, as the matching frame cache data corresponding to the current reference frame cache data, when the current reference frame cache data is stored in the reference frame data caching unit;

a judging module for judging whether the time interval of the timestamp between the matching frame buffer data and the current reference frame buffer data is less than or equal to a predetermined interval threshold;

a determining module, configured to determine the matching frame buffer data as the synchronous frame buffer data corresponding to the current reference frame buffer data in response to the timestamp interval time being less than or equal to the predetermined interval threshold; and

and the skipping module is used for skipping the current reference frame cache data in response to the time stamp interval time being greater than the preset interval threshold value so as to perform data synchronization on the next reference frame cache data.

In an embodiment of the present invention, the multi-sensor data synchronization system further includes a reporting module, configured to report the synchronization frame buffer data and the corresponding reference frame buffer data to the SLAM algorithm processing unit for performing synchronous positioning and mapping.

According to another aspect of the present invention, the present invention further provides an electronic device comprising:

at least one processor configured to execute instructions; and

a memory communicatively coupled to the at least one processor, wherein the memory has at least one instruction, wherein the instruction is executable by the at least one processor to cause the at least one processor to perform some or all of the steps of a multi-sensor data synchronization method, wherein the multi-sensor data synchronization method comprises the steps of:

an electronic device body; and

at least one multi-sensor data synchronization system, wherein the multi-sensor data synchronization system is configured on the electronic device body, and the multi-sensor data synchronization system comprises:

Further objects and advantages of the invention will be fully apparent from the ensuing description and drawings.

These and other objects, features and advantages of the present invention will become more fully apparent from the following detailed description, the accompanying drawings and the claims.

Drawings

FIG. 1 is a flow diagram of a multi-sensor data synchronization method according to an embodiment of the invention.

Fig. 2 shows a flow chart of one of the steps of the multi-sensor data synchronization method according to the above-described embodiment of the present invention.

Fig. 3 shows an example of a data sampling process in the multi-sensor data synchronization method according to the above-described embodiment of the present invention.

Fig. 4 shows an application example of the multi-sensor data synchronization method according to the above-described embodiment of the present invention.

FIG. 5 illustrates a block diagram schematic of a multi-sensor data synchronization system in accordance with an embodiment of the present invention.

FIG. 6 shows a block diagram schematic of an electronic device according to an embodiment of the invention.

Fig. 7 shows a schematic structural diagram of another electronic device according to an embodiment of the invention.

Detailed Description

The following description is presented to disclose the invention so as to enable any person skilled in the art to practice the invention. The preferred embodiments in the following description are given by way of example only, and other obvious variations will occur to those skilled in the art. The basic principles of the invention, as defined in the following description, may be applied to other embodiments, variations, modifications, equivalents, and other technical solutions without departing from the spirit and scope of the invention.

In the present invention, the terms "a" and "an" in the claims and the description should be understood as meaning "one or more", that is, one element may be one in number in one embodiment, and the element may be more than one in number in another embodiment. The terms "a" and "an" should not be construed as limiting the number unless the number of such elements is explicitly recited as one in the present disclosure, but rather the terms "a" and "an" should not be construed as being limited to only one of the number.

In the description of the present invention, it is to be understood that the terms "first," "second," and the like are used for descriptive purposes only and are not to be construed as indicating or implying relative importance. In the description of the present invention, it should be noted that, unless explicitly stated or limited otherwise, the terms "connected" and "connected" are to be interpreted broadly, and may be, for example, fixedly connected, detachably connected, or integrally connected; can be mechanically or electrically connected; they may be directly connected or indirectly connected through an intermediate. The specific meanings of the above terms in the present invention can be understood by those skilled in the art according to specific situations.

In the description herein, references to the description of the term "one embodiment," "some embodiments," "an example," "a specific example," or "some examples," etc., mean that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the invention. In this specification, the schematic representations of the terms used above are not necessarily intended to refer to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. Furthermore, various embodiments or examples and features of different embodiments or examples described in this specification can be combined and combined by one skilled in the art without contradiction.

Illustrative method

Referring to fig. 1 and 2 of the drawings, a multi-sensor data synchronization method according to an embodiment of the present invention is illustrated. Specifically, as shown in fig. 1, the multi-sensor data synchronization method includes the steps of:

s300: respectively storing multi-frame data acquired by sampling of a plurality of sensors into corresponding data cache units to obtain multi-frame cache data corresponding to the plurality of sensors one by one; and

s400: and performing data synchronization on the cache data corresponding to the other sensors in the plurality of sensors by respectively delaying for a predetermined time based on the cache data corresponding to the sensor with the smallest frame rate in the plurality of sensors to obtain corresponding synchronous frame cache data.

In particular, in this embodiment of the present invention, for convenience of description, a sensor with the smallest frame rate among the plurality of sensors is used as a reference sensor, a data buffer unit corresponding to the reference sensor is defined as a reference frame data buffer unit, and data stored in the reference frame data buffer unit is defined as reference frame buffer data; similarly, the other sensors in the plurality of sensors are used as the sensors to be synchronized, the data cache unit corresponding to the sensor to be synchronized is defined as a frame data cache unit to be synchronized, and the data stored in the sensor to be synchronized is defined as frame cache data to be synchronized.

It should be noted that, since the frame rate of the reference sensor is the minimum, that is, the sampling time interval of the reference sensor is greater than the sampling time interval of the sensor to be synchronized, when data synchronization is performed with reference to the reference buffer data, the corresponding synchronous frame buffer data can be obtained from the frame buffer data to be synchronized corresponding to each sensor to be synchronized to the maximum extent, so as to ensure the efficiency and validity of data synchronization. Meanwhile, when the data sampled and acquired by the reference sensor is stored in the corresponding data cache unit, the multi-sensor data synchronization method of the invention delays the preset time to ensure that the data sampled and acquired subsequently by the sensor to be synchronized can be stored in the corresponding data cache unit, thereby ensuring that the cache data sampled and acquired by the sensor to be synchronized also exists after the timestamp of the reference frame cache data, and being beneficial to improving the accuracy of the data synchronization result.

It is to be understood that the number of the sensors to be synchronized in the plurality of sensors of the present invention may be one or more, and the present invention does not limit the number of the sensors to be synchronized. In other words, the number of the plurality of sensors of the present invention may be two or more. Further, the plurality of sensors of the present invention may be implemented as, but not limited to, visual sensors such as RGB cameras, depth sensors such as structured light cameras or TOF cameras, and Inertial Measurement Units (IMUs), among others.

Preferably, the predetermined time is greater than or equal to the sampling interval time of the sensor to be synchronized, so as to ensure that when data synchronization is performed with reference to the reference frame buffer data, buffer data sampled and acquired by the sensor to be synchronized also exists after the timestamp of the reference frame buffer data (that is, the data sampled and acquired by the sensor to be synchronized has a buffer after the timestamp of the reference frame buffer data), which is convenient for comprehensively analyzing the buffer data sampled and acquired by the sensor to be synchronized, and further, more accurate synchronous frame buffer data corresponding to the reference frame buffer data is obtained.

More preferably, the predetermined time of the present invention is equal to a sampling interval time of the reference sensor. In this way, when the predetermined time is delayed to perform data synchronization on the current reference frame buffer data, the next reference frame buffer data sampled and acquired by the reference sensor is also stored in the data buffer unit, and all the sensors to be synchronized must sample and acquire corresponding data and store the corresponding data in the corresponding data buffer unit due to the large frame frequency in the predetermined time, so that smooth data synchronization is ensured. In other words, when the next reference frame buffer data sampled and acquired by the reference sensor is stored in the data buffer unit, the multi-sensor data synchronization method of the present invention starts data synchronization with reference to the current reference frame buffer data. Because the frame rate of the reference sensor is the minimum, all the sensors to be synchronized must have sampled and acquired at least one frame buffer data to be synchronized between two adjacent reference frame buffer data sampled and acquired by the reference sensor, thereby ensuring that the most adjacent frame buffer data to be synchronized can be found, and improving the reliability of the synchronized data.

Of course, in other examples of the present invention, the predetermined time required to delay may also be different for different sensors to be synchronized. For example, the predetermined time is exactly equal to the sampling interval time of the sensor to be synchronized, so that the predetermined times corresponding to different sensors to be synchronized are different due to different frame rates. In this way, after the timestamp of the reference frame buffer data and before the reference frame buffer data is used as a reference for data synchronization, the corresponding sensor to be synchronized just samples and acquires one frame buffer data to be synchronized, so that the number of the frame buffer data to be synchronized can be reduced as much as possible while the most adjacent frame buffer data to be synchronized can be found, the calculation amount of data synchronization is reduced, and the efficiency of data synchronization is improved.

It is worth mentioning that, in order to further ensure that the frame to be synchronized cache data exists before and after the timestamp of the reference frame cache data, that is, timestamp overlapping exists between the reference frame cache data and the frame to be synchronized cache data, so as to facilitate subsequent data synchronization. The multi-sensor data synchronization method of the present invention needs to control the storage size of the data buffer units corresponding to the plurality of sensors in addition to delaying the predetermined time. Specifically, as shown in fig. 1, the multi-sensor data synchronization method of the present invention, before the step S300, further includes the steps of:

s100: and respectively allocating the storage sizes of the data cache units corresponding to the plurality of sensors so that the reference frame cache data and the frame cache data to be synchronized can have time stamp overlapping.

Preferably, the storage size of the reference frame data buffer unit is allocated to store at least one reference frame buffer data; accordingly, the storage size of the frame data buffer unit to be synchronized is allocated to store at least two frame buffer data to be synchronized. At this time, the predetermined time can only be implemented to be equal to the sampling time interval of the sensor to be synchronized, in case that the predetermined time is too large, the sensor to be synchronized samples and acquires more frame buffer data to be synchronized after the time stamp of the reference frame buffer data, so that the frame data to be synchronized before the time stamp of the reference frame buffer data is replaced and lost.

More preferably, according to the frame rate of the plurality of sensors, the storage sizes of the reference frame data buffer unit and the to-be-synchronized frame data buffer unit are respectively allocated proportionally, so that the ratio of the number of frames stored by the to-be-synchronized frame data buffer unit to the frame rate of the corresponding to-be-synchronized sensor is equal to twice the ratio of the number of frames stored by the reference frame data buffer unit to the frame rate of the corresponding reference sensor. In other words, when the reference frame data buffering unit can store all reference frame buffer data sampled and acquired via the reference sensor at a predetermined sampling time period, the to-be-synchronized frame data buffering unit can store all to-be-synchronized frame buffer data sampled and acquired via the to-be-synchronized sensor at twice the predetermined sampling time period. It is understood that the storage size of the data buffer unit is related to not only the number of frames of stored data, but also the size of each frame of data itself.

Most preferably, the data buffering unit (including the reference frame data buffering unit and the frame data to be synchronized buffering unit) stores data by using a circular buffer mechanism, so as to avoid that the overhead of the CPU in operation is increased due to the shift of the previously acquired buffered data after acquiring new sampled data, which is beneficial to realizing efficient sampled data buffering. In other words, when the circular buffer (i.e., the data cache unit) is full, the oldest cache data is automatically replaced without moving other cache data.

It is understood that the circular buffer is a circular table of the FIFO, and the actual ending position in the memory points to the actual starting position of the memory, which is suitable for the situation that the maximum capacity of the buffer is determined in advance. Of course, in other examples of the present invention, the data caching unit may also be implemented as a memory that adopts other storage mechanisms, such as a conventional buffering mechanism, and the present invention is not described in detail herein again.

It should be noted that, in order to ensure that when data synchronization is performed based on the first reference frame buffer data sampled and acquired by the reference sensor, the frame buffer to be synchronized unit stores the frame buffer data to be synchronized before and after the timestamp of the first reference frame buffer data, the multi-sensor data synchronization method of the present invention further needs to control the starting sequence of the plurality of sensors to ensure that the first reference frame buffer data overlaps with the timestamp of the frame buffer data to be synchronized.

Specifically, as shown in fig. 1, the multi-sensor data synchronization method of the present invention, before the step S300, further includes the steps of:

s200: activating the plurality of sensors for a sampling operation in a predetermined sequence, wherein the reference sensor of the plurality of sensors is activated last.

Preferably, in the step S200, after all the sensors to be synchronized are started, the predetermined time is delayed, and then the reference sensor is started, so as to ensure that the frame buffer data to be synchronized exists between the timestamps of the first reference frame buffer data.

It should be noted that, since the multi-sensor data synchronization method is often applied to a scene or a field where synchronization data needs to be obtained in real time, such as SLAM, in step S300 of the multi-sensor data synchronization method of the present invention, it is necessary to respectively store multi-frame data sampled and obtained by a plurality of sensors in a corresponding data buffer unit in real time, so as to ensure that data synchronization work is performed in real time while the multi-frame data is sampled and obtained by the plurality of sensors, which is helpful for improving real-time performance of the multi-sensor data synchronization method, and is convenient for application and popularization in the technical field or the scene, such as SLAM.

According to the above embodiment of the present invention, as shown in fig. 2, the step S400 of the multi-sensor data synchronization method may include the steps of:

s410: when the current reference frame cache data is stored in the reference frame data cache unit, delaying the preset time respectively to select the frame cache data to be synchronized closest to the time stamp of the current reference frame cache data from the frame data cache unit to be synchronized, so as to serve as the matched frame cache data corresponding to the current reference frame cache data;

s420: judging whether the time interval of the timestamp between the matched frame cache data and the current reference frame cache data is less than or equal to a preset interval threshold value or not;

s430: in response to the timestamp interval time being less than or equal to the predetermined interval threshold, determining the matching frame buffer data as the synchronization frame buffer data corresponding to the current reference frame buffer data;

s440: and skipping the current reference frame buffer data in response to the time stamp interval time being larger than the preset interval threshold value so as to perform data synchronization on the next reference frame buffer data.

Preferably, the predetermined interval threshold of the present invention may be implemented as half of a sampling interval time of the sensor to be synchronized.

It is to be noted that, when the sampling frame rate of the sensor to be synchronized is stable, the matching frame buffer data and the current reference frame buffer data are necessarily less than or equal to half of the sampling interval time of the sensor to be synchronized. If the sampling frame rate of the sensor to be synchronized is unstable due to equipment heating and the like, the situation that the matching frame cache data and the current reference frame cache data are more than half of the sampling interval time of the sensor to be synchronized occurs, and at the moment, the synchronization failure of the current reference frame cache data and the corresponding frame cache data to be synchronized can be determined, and the current reference frame cache data can be skipped over to perform data synchronization on the next reference frame cache data.

However, in fact, in a specific application scenario (such as SLAM) of the synchronized data, the loss of the synchronized data has a larger impact on the specific application than the high error of the synchronized data, so the multi-sensor data synchronization method of the present invention should ensure the integrity of the synchronized data as much as possible. Therefore, in other examples of the present invention, the predetermined interval threshold is preferably implemented as a sampling interval time of the reference sensor, that is, as long as a timestamp of the matching frame buffer data is between a timestamp of a previous reference frame buffer data and a timestamp of a next reference frame buffer data, the matching frame buffer data may be determined as the synchronization frame buffer data corresponding to the current reference frame buffer data, so as to avoid data loss as much as possible.

In addition, after the data sampled and acquired by all the sensors to be synchronized are synchronized through the above steps, the synchronization frame buffer data and the corresponding reference frame buffer data are transmitted to a subsequent processing unit such as a SLAM. For example, as shown in fig. 1, the multi-sensor data synchronization method of the present invention may further include, after the step S400, the steps of:

s500: and reporting the synchronous frame cache data and the corresponding reference frame cache data to an SLAM algorithm processing unit for synchronous positioning and map construction.

In this way, by repeating the steps S300 to S500, it is possible to perform SLAM mapping and robot synchronization positioning in real time by continuously acquiring reliable sensor synchronization data (i.e., the synchronization frame buffer data and the corresponding reference frame buffer data).

It should be noted that, although in the above-mentioned embodiment of the present invention, the multi-sensor data synchronization method uses the nearest neighbor principle to find out the frame buffer data to be synchronized closest to the reference frame buffer data from each frame buffer unit to be synchronized as the matching frame buffer data, it is only an example. Of course, in other examples of the present invention, the multi-sensor data synchronization method may also directly fit the synchronization frame data that is most matched with the reference frame buffer data from the to-be-synchronized frame buffer data in the to-be-synchronized frame data buffer unit by using a linear fitting or non-linear fitting manner, which is not described in detail herein.

Illustratively, as shown in fig. 3, the advantage of the multi-sensor data synchronization method of the present invention is illustrated by taking the plurality of sensors including sensor 1, sensor 2, and sensor 3 as an example. Specifically, as shown in fig. 3 and 4, the multi-sensor data synchronization method includes the steps of:

(A) and preprocessing a data caching unit. The data buffer unit with fixed size is distributed for each sensor, wherein the data buffer unit of the sensor to be synchronized is proportionally distributed according to the frame rate of the sensor, the data buffer unit of the reference sensor is set to be two frames or more, and the overlapping of the buffer data time stamps of the reference frame and the synchronous frame must be ensured so as to be beneficial to the synchronization of the data synchronization unit. It is to be understood that the timestamp of the present invention refers to the system time at which the sensor was sampled; the overlapping of the timestamps means that the frame data to be synchronized in the data caching unit are cached before and after the timestamp of the current reference frame caching data; the data cache unit is preferably realized by adopting a circular buffer zone mechanism, so that the movement of the rest of the sampling cache data after the new sampling data is acquired is avoided, and the high-efficiency sampling data cache is realized.

(B) A multi-sensor start-up phase. The sensors are started in a certain sequence, wherein the reference sensor is configured by starting and setting the sensor with the lowest sampling frequency (i.e. the lowest frame rate) as the reference sensor. As shown in fig. 3, the sensor 2 is a reference sensor (sampling times T1 to T6, which is also a time reference for acquiring multi-sensor data), and the other sensors (the sensor 1 and the sensor 3) are all sensors to be synchronized. At this time, the starting sequence of the sensor is that the sensor 3 is started first, then the sensor 1 is started, and finally the sensor 2 is started, so that the overlapping of the time stamps always exists between the reference frame data buffer unit and the frame data buffer unit to be synchronized. It can be understood that, the sensor to be synchronized (i.e. the sensor 3) with the lower frame rate is started first, and then the sensor to be synchronized (i.e. the sensor 1) with the higher frame rate is started, so that it can be avoided that the sensor to be synchronized with the higher frame rate acquires, stores and replaces more cache data when waiting for the start of other sensors, so as to reduce the overhead of the CPU during operation to the maximum extent. Of course, in other examples of the present invention, the present invention does not limit the starting sequence among the plurality of sensors to be synchronized, as long as it is ensured that the reference sensor is started last.

(C) And (3) a sensor data acquisition stage. And each sensor respectively samples data and carries and stores the data to the corresponding data cache unit. For example, the sampled data of n sensors are stored in corresponding circular buffer units, respectively. It can be understood that the purpose of the invention adopting the circular cache region is to avoid moving the rest of the sampling cache data after acquiring the new sampling data, realize the high-efficiency sampling data cache, and reduce the expenditure of the CPU in operation.

(D) Waiting for the synchronization phase. In order to ensure that the time stamps of the reference frame data and the synchronous frame data of the data buffer unit overlap, it is necessary to ensure that the reference frame data buffer unit is filled with data before data synchronization can be started. The reference frame data buffer unit only needs to buffer two frames, so the multi-sensor data synchronization method of the invention starts data synchronization from the sampling to the second frame of reference frame data. In other words, in this example of the present invention, when reference frame buffer data is stored to the reference frame data buffer unit, the sampling interval time of the reference sensor is delayed and then data synchronization is performed based on the reference frame buffer data.

(E) A data synchronization phase of the synchronization frame. And the oldest reference frame buffer data is taken out from the reference frame data buffer unit as a reference to carry out data synchronization, so that the frame buffer data to be synchronized which is most adjacent to the reference frame buffer data can be found, and the reliability of the synchronization data is improved.

For example, the data synchronization strategy adopted by the present invention is as follows:

strategy 1, the reference frame buffer data finds out the closest matching frame buffer data from each circular buffer area of the frame to be synchronized according to the nearest rule.

And strategy 2, judging the time stamp interval between the reference frame cache data and the matched frame cache data, and if the time stamp interval is less than half of the smaller interval (namely the sampling time interval of the sensor to be synchronized) in the sampling time stamp intervals of the two sensors, determining that the reference frame cache data and the matched frame cache data are synchronous frame data.

Policy 3, in case of the policy 2 condition not being satisfied (possibly due to unstable sensor sampling frame rate, which may be caused by heat generated by the device, etc.), the case 1 may occur: the timestamp interval between reference frame buffer data and matching frame buffer data is greater than half of the timestamp interval between adjacent sampled frames of the reference sensor, or case 2: a condition that the timestamp of the reference frame buffer data does not fall within the frame buffer to be synchronized. At this time, if case 1 occurs and case 2 does not occur, the reference frame buffer data and the matching frame buffer data may be considered as synchronized video. If the situation 2 occurs, after the latest frame buffer data to be synchronized arrives, the circular buffer area of the reference frame is synchronized by taking the latest frame buffer data to be synchronized as the reference.

(F) And a sensor data updating stage. Similar to step 3, when the circular buffer is full, the oldest sample data is automatically replaced (based on the circular buffer data replacement mechanism). And the data caching unit caches the latest sensor sampling data in real time.

(G) And a synchronous multi-sensor data reporting stage. And after the synchronization of all the sensor data is finished, the data synchronization unit reports all the synchronization frames and the corresponding reference frames to the SLAM algorithm processing unit.

(H) SLAM Algorithm processing phase. And the SLAM algorithm processing unit is used for carrying out map construction and synchronous positioning by utilizing synchronous multi-sensor data.

(I) And (E) repeating the steps (E) to (H), continuously acquiring reliable synchronous data of the sensor, and performing SLAM map construction and robot synchronous positioning.

Therefore, the multi-sensor data synchronization method of the invention firstly adopts the circular cache region mentioned in the step (A) to perform fast cache on the data sampled and acquired by each sensor, and then adopts the synchronization strategy mentioned in the step (E) to select reliable multi-sensor synchronous data, thereby realizing fast data cache and data synchronization and simultaneously ensuring the reliability of the synchronous data.

Illustrative System

Referring to FIG. 5 of the drawings, a multi-sensor data synchronization system in accordance with an embodiment of the present invention is illustrated. Specifically, as shown in FIG. 5, the multi-sensor data synchronization system 400 includes a data storage module 410 and a data synchronization module 420 communicatively coupled to each other. The data storage module 410 is configured to store the multiple frames of data sampled and acquired by the multiple sensors into corresponding data cache units, respectively, so as to obtain multiple frames of cache data corresponding to the multiple sensors one to one. The data synchronization module 420 is configured to perform data synchronization on the buffered data corresponding to the sensor to be synchronized among the plurality of sensors by delaying for a predetermined time respectively with reference to the buffered data corresponding to the reference sensor among the plurality of sensors, so as to obtain corresponding synchronized frame buffered data, where a frame rate of the reference sensor among the plurality of sensors is the minimum.

It should be noted that, in the above embodiment of the present invention, as shown in fig. 5, the multi-sensor data synchronization system 400 may further include an allocating module 430, where the storing allocating module 430 is configured to allocate the storage sizes of the data buffer units corresponding to the multiple sensors, respectively, so that there is a time stamp overlap between the buffer data corresponding to the reference sensor and the buffer data corresponding to the sensor to be synchronized.

In addition, as shown in fig. 5, the multi-sensor data synchronization system 400 may further include an activation module 440, wherein the activation module 440 is configured to activate the read sensors for sampling according to a predetermined sequence, wherein the reference sensor is activated last.

In an example of the present invention, as shown in fig. 5, the data synchronization module 420 includes: a selecting module 421, configured to delay the predetermined time to select, from the frame-to-be-synchronized data buffer units, the frame-to-be-synchronized data that is closest to the timestamp of the current reference frame buffer data, as the matching frame buffer data corresponding to the current reference frame buffer data when the current reference frame buffer data is stored in the reference frame data buffer units; a determining module 422, configured to determine whether a timestamp interval between the matching frame buffer data and the current reference frame buffer data is less than or equal to a predetermined interval threshold; a determining module 423 for determining the matching frame buffer data as the synchronous frame buffer data corresponding to the current reference frame buffer data in response to the timestamp interval time being less than or equal to the predetermined interval threshold; and a skipping module 424, for skipping the current reference frame buffer data in response to the timestamp interval time being greater than the predetermined interval threshold, so as to perform data synchronization on the next reference frame buffer data.

It should be noted that, in an example of the present invention, as shown in fig. 5, the multi-sensor data synchronization system 400 may also further include a reporting module 450, configured to report the synchronization frame buffer data and the corresponding reference frame buffer data to the SLAM algorithm processing unit for performing synchronous positioning and mapping.

Illustrative electronic device

Next, an electronic apparatus according to an embodiment of the present invention is described with reference to fig. 6. As shown in fig. 6, the electronic device 90 includes one or more processors 91 and memory 92.

The processor 91 may be a Central Processing Unit (CPU) or other form of processing unit having data processing capabilities and/or instruction execution capabilities, and may control other components in the electronic device 90 to perform desired functions. In other words, the processor 91 comprises one or more physical devices configured to execute instructions. For example, the processor 91 may be configured to execute instructions that are part of: one or more applications, services, programs, routines, libraries, objects, components, data structures, or other logical constructs. Such instructions may be implemented to perform a task, implement a data type, transform the state of one or more components, implement a technical effect, or otherwise arrive at a desired result.

The processor 91 may include one or more processors configured to execute software instructions. Additionally or alternatively, the processor 91 may include one or more hardware or firmware logic machines configured to execute hardware or firmware instructions. The processors of the processor 91 may be single core or multicore, and the instructions executed thereon may be configured for serial, parallel, and/or distributed processing. The various components of the processor 91 may optionally be distributed over two or more separate devices, which may be remotely located and/or configured for coordinated processing. Aspects of the processor 91 may be virtualized and executed by remotely accessible networked computing devices configured in a cloud computing configuration.

The memory 92 may include one or more computer program products that may include various forms of computer-readable storage media, such as volatile memory and/or non-volatile memory. The volatile memory may include, for example, Random Access Memory (RAM), cache memory (cache), and/or the like. The non-volatile memory may include, for example, Read Only Memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer readable storage medium and executed by the processor 11 to implement some or all of the steps of the above-described exemplary methods of the present invention described above, and/or other desired functions.

In other words, the memory 92 comprises one or more physical devices configured to hold machine-readable instructions executable by the processor 91 to implement the methods and processes described herein. In implementing these methods and processes, the state of the memory 92 may be transformed (e.g., to hold different data). The memory 92 may include removable and/or built-in devices. The memory 92 may include optical memory (e.g., CD, DVD, HD-DVD, blu-ray disc, etc.), semiconductor memory (e.g., RAM, EPROM, EEPROM, etc.), and/or magnetic memory (e.g., hard-disk drive, floppy-disk drive, tape drive, MRAM, etc.), among others. The memory 92 may include volatile, nonvolatile, dynamic, static, read/write, read-only, random-access, sequential-access, location-addressable, file-addressable, and/or content-addressable devices.

It is understood that the memory 92 comprises one or more physical devices. However, aspects of the instructions described herein may alternatively be propagated by a communication medium (e.g., an electromagnetic signal, an optical signal, etc.) that is not held by a physical device for a limited period of time. Aspects of the processor 91 and the memory 92 may be integrated together into one or more hardware logic components. These hardware logic components may include, for example, Field Programmable Gate Arrays (FPGAs), program and application specific integrated circuits (PASIC/ASIC), program and application specific standard products (PSSP/ASSP), system on a chip (SOC), and Complex Programmable Logic Devices (CPLDs).

In one example, as shown in FIG. 6, the electronic device 90 may also include an input device 93 and an output device 94, which may be interconnected via a bus system and/or other form of connection mechanism (not shown). The input device 93 may be, for example, a camera module or the like for capturing image data or video data. As another example, the input device 93 may include or interface with one or more user input devices such as a keyboard, mouse, touch screen, or game controller. In some embodiments, the input device 93 may include or interface with a selected Natural User Input (NUI) component. Such component parts may be integrated or peripheral and the transduction and/or processing of input actions may be processed on-board or off-board. Example NUI components may include a microphone for speech and/or voice recognition; infrared, color, stereo display and/or depth camera for machine vision and/or gesture recognition; a head tracker, eye tracker, accelerometer and/or gyroscope for motion detection and/or intent recognition; and an electric field sensing component for assessing brain activity and/or body movement; and/or any other suitable sensor.

The output device 94 may output various information including the classification result and the like to the outside. The output devices 94 may include, for example, a display, speakers, a printer, and a communication network and its connected remote output devices, among others.

Of course, the electronic device 90 may further comprise the communication means, wherein the communication means may be configured to communicatively couple the electronic device 90 with one or more other computer devices. The communication means may comprise wired and/or wireless communication devices compatible with one or more different communication protocols. As a non-limiting example, the communication subsystem may be configured for communication via a wireless telephone network or a wired or wireless local or wide area network. In some embodiments, the communications device may allow the electronic device 90 to send and/or receive messages to and/or from other devices via a network such as the internet.

It will be appreciated that the configurations and/or approaches described herein are exemplary in nature, and that these specific embodiments or examples are not to be considered in a limiting sense, because numerous variations are possible. The specific routines or methods described herein may represent one or more of any number of processing strategies. As such, various acts illustrated and/or described may be performed in the sequence illustrated and/or described, in other sequences, in parallel, or omitted. Also, the order of the above-described processes may be changed.

Of course, for simplicity, only some of the components of the electronic device 90 relevant to the present invention are shown in fig. 6, omitting components such as buses, input/output interfaces, and the like. In addition, the electronic device 90 may include any other suitable components, depending on the particular application.

According to another aspect of the present invention, an embodiment of the present invention further provides another electronic device. Illustratively, as shown in fig. 7, the electronic device includes an electronic device body 800 and at least one multi-sensor data synchronization system 400, wherein the multi-sensor data synchronization system 400 is configured on the electronic device body 800, and the multi-sensor data synchronization system 400 includes: the data storage module is used for respectively storing the multi-frame data acquired by sampling of the plurality of sensors into the corresponding data cache units so as to obtain the multi-frame cache data corresponding to the plurality of sensors one by one; and the data synchronization module is used for delaying preset time respectively to perform data synchronization on the cache data corresponding to the sensor to be synchronized in the plurality of sensors by taking the cache data corresponding to the reference sensor in the plurality of sensors as a reference so as to obtain corresponding synchronous frame cache data, wherein the frame rate of the reference sensor in the plurality of sensors is minimum.

It is noted that the electronic device body 800 can be any device or system capable of being configured with the depth gesture recognition system 400, such as glasses, a head-mounted display device, an augmented reality device, a virtual reality device, a smart phone, or a mixed reality device. It will be understood by those skilled in the art that although the electronic device body 800 is implemented as AR glasses in fig. 7, it does not limit the content and scope of the present invention.

It should also be noted that in the apparatus, devices and methods of the present invention, the components or steps may be broken down and/or re-combined. These decompositions and/or recombinations are to be regarded as equivalents of the present invention.

The previous description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present invention. Various modifications to these aspects will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other aspects without departing from the scope of the invention. Thus, the present invention is not intended to be limited to the aspects shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

It will be appreciated by persons skilled in the art that the embodiments of the invention described above and shown in the drawings are given by way of example only and are not limiting of the invention. The objects of the invention have been fully and effectively accomplished. The functional and structural principles of the present invention have been shown and described in the examples, and any variations or modifications of the embodiments of the present invention may be made without departing from the principles.

Claims

1. A multi-sensor data synchronization method, comprising the steps of:

2. The multi-sensor data synchronization method of claim 1, wherein the predetermined time is equal to or greater than a sampling interval time of the sensor to be synchronized.

3. The multi-sensor data synchronization method of claim 2, wherein the predetermined time is equal to a sampling interval time of the reference sensor.

4. The multi-sensor data synchronization method according to any one of claims 1 to 3, further comprising, before the step of respectively storing the multi-frame data acquired by sampling the plurality of sensors into the corresponding data buffer units to obtain the multi-frame buffer data in one-to-one correspondence with the plurality of sensors:

5. The multi-sensor data synchronization method according to claim 4, wherein in the step of allocating the storage sizes of the data buffer units corresponding to the plurality of sensors, respectively:

6. The multi-sensor data synchronization method of claim 5, wherein the data cache unit employs a circular buffer mechanism for storage.

7. The multi-sensor data synchronization method according to claim 4, further comprising, before the step of respectively storing the multi-frame data sampled and acquired by the plurality of sensors into the corresponding data buffer units to obtain the multi-frame buffer data in one-to-one correspondence with the plurality of sensors, the steps of:

8. The multi-sensor data synchronization method of claim 7, wherein after all the sensors to be synchronized are activated, the predetermined time is delayed and then the reference sensor is activated.

9. The multi-sensor data synchronization method according to any one of claims 1 to 3, wherein the step of delaying the buffer data corresponding to the sensor to be synchronized among the plurality of sensors by a predetermined time respectively with reference to the buffer data corresponding to the reference sensor among the plurality of sensors to obtain the corresponding synchronized frame buffer data, wherein the frame rate of the reference sensor among the plurality of sensors is minimized, comprises the steps of:

10. The multi-sensor data synchronization method of claim 9, wherein the predetermined interval threshold is half of a sampling interval time of the sensor to be synchronized or a sampling interval time of the reference sensor.

11. The multi-sensor data synchronization method of claim 10, further comprising the steps of:

12. A multi-sensor data synchronization system, comprising communicatively coupled to each other:

13. The multi-sensor data synchronization system of claim 12, further comprising an allocation module, wherein the storage allocation module is configured to allocate storage sizes of the data buffer units corresponding to the plurality of sensors, respectively, so that there is an overlap between the time stamps of the buffer data corresponding to the reference sensor and the buffer data corresponding to the sensor to be synchronized.

14. The multi-sensor data synchronization system of claim 13, further comprising an activation module, wherein the activation module is configured to activate the read sensors for sampling operations in a predetermined order, wherein the reference sensor is activated last.

15. The multi-sensor data synchronization system of any of claims 12 to 14, wherein the data synchronization module comprises communicatively coupled to each other:

16. The multi-sensor data synchronization system of claim 15, further comprising a reporting module for reporting the synchronization frame buffer data and the corresponding reference frame buffer data to a SLAM algorithm processing unit for synchronous positioning and mapping.

17. An electronic device, comprising:

at least one processor configured to execute instructions; and

18. An electronic device, comprising:

an electronic device body; and