CN110782492B - Pose tracking method and device - Google Patents

Abstract

Provided are a pose tracking method and device. The pose tracking method includes: acquiring an image of a tracking object, wherein the tracking object is provided with a marker that flickers at a specific frequency; acquiring pixels with brightness changes from the acquired image; and calculating a 6-degree-of-freedom attitude of the tracking object based on the acquired pixels , thereby reducing the dependence of pose tracking on the specific layout of LED markers, reducing the latency of pose tracking, and improving the accuracy and efficiency of pose tracking.

Description

Posture tracking method and device

Technical Field

The present disclosure relates to the field of computer vision technology. More specifically, the present disclosure relates to a posture tracking method and device.

Background Art

In recent years, a variety of 6-DOF pose estimation methods have been proposed and widely used in fields such as robot grasping, virtual reality/augmented reality, and human-computer interaction. Virtual reality and augmented reality have high requirements for system latency. If the system is slow to respond to head movements, it will cause users to feel dizzy and nauseous. The delay of Valve's system is 7-15 milliseconds. The lowest delay of current commercial virtual reality (VR) tracking products is 15 milliseconds, which still cannot provide users with a perfect immersive experience.

Many current optical tracking systems are based on complementary metal oxide semiconductor (CMOS) cameras. However, the latency of CMOS cameras in consumer devices is generally greater than 16.7 milliseconds (60FPS). Due to hardware limitations, these methods cannot display the user's action input on the screen in a timely manner, and cannot meet the low latency requirements of VR.

Some schemes also use active light-emitting diode (LED) markers to recover the 6-DOF posture of the object. However, these methods have some limitations. Either the number of LED lights must be four and coplanar, or the computational effort exceeds the preset deviation threshold and cannot be applied to real-time systems. Having only four LED lights will affect the accuracy of pose tracking and the robustness of the system, because if one of the lights is not detected, the pose solution will fail. In addition, brute force solution of the correspondence between 2D/3D point sets is very time-consuming and cannot be applied to cases with a large number of LED lights and real-time systems.

Summary of the invention

An exemplary embodiment of the present disclosure is to provide a posture tracking method and device to reduce the dependence of posture tracking on the specific layout of LED markers, while reducing the latency of posture tracking and improving the accuracy and efficiency of posture tracking.

According to an exemplary embodiment of the present disclosure, there is provided a posture tracking method, comprising: acquiring an image of a tracked object, wherein a marker flashing at a specific frequency is provided on the tracked object; acquiring pixels with brightness changes from the acquired image; and calculating a 6-DOF posture of the tracked object based on the acquired pixels, thereby reducing the dependence of posture tracking on a specific layout of an LED marker, while reducing the latency of posture tracking, and improving the accuracy and efficiency of posture tracking.

Optionally, the step of calculating the 6-degree-of-freedom posture of the tracked object based on the acquired pixels may include: acquiring inertial measurement unit data of the tracked object, and estimating the 3-degree-of-freedom posture of the tracked object based on the acquired inertial measurement unit data, wherein the 3-degree-of-freedom posture is a posture rotating around the three coordinate axes x, y, and z in the body coordinate system of the tracked object; and calculating the 6-degree-of-freedom posture of the tracked object based on the 3-degree-of-freedom posture and the acquired pixels, wherein the 6-degree-of-freedom posture is a posture along the three coordinate axes x, y, and z and a posture rotating around the three rectangular coordinate axes x, y, and z in the body coordinate system of the tracked object, thereby improving the accuracy and efficiency of posture tracking.

Optionally, the step of calculating the 6-degree-of-freedom posture of the tracked object based on the 3-degree-of-freedom posture and the acquired pixels may include: based on the 3-degree-of-freedom posture and the acquired pixels, solving the correspondence between the 2D point set and the 3D point set of the mark, and obtaining matching pairs of the 2D point set and the 3D point set of the mark, wherein the 2D point set includes the pixel coordinates of the mark, and the 3D point set includes the coordinates of the mark in the body coordinate system of the tracked object; based on the matching pairs, calculating the 6-degree-of-freedom posture of the tracked object, thereby improving the accuracy and efficiency of posture tracking.

Optionally, the step of calculating the 6-degree-of-freedom posture of the tracked object may include: removing pixels whose reprojection deviation exceeds a preset deviation threshold from the matching pair, and calculating the 6-degree-of-freedom posture based on the remaining pixels after removal; performing a reprojection error minimization operation on the calculated 6-degree-of-freedom posture to obtain the 6-degree-of-freedom posture of the tracked object, thereby improving the accuracy and efficiency of posture tracking.

Optionally, the step of calculating the 6-degree-of-freedom posture of the tracked object may include: removing pixels whose reprojection deviation exceeds a preset deviation threshold from the matching pair, and calculating the 6-degree-of-freedom posture based on the remaining pixels after removal; performing a reprojection error minimization operation on the calculated 6-degree-of-freedom posture; and optimizing the 6-degree-of-freedom posture after the reprojection error minimization operation based on the 3-degree-of-freedom posture to obtain the 6-degree-of-freedom posture of the tracked object, thereby improving the accuracy and efficiency of posture tracking.

Optionally, after obtaining the 6-DOF posture of the tracked object, the posture tracking method may further include: re-matching pixels in the matching pair whose reprojection error exceeds a preset deviation threshold according to the 6-DOF posture of the tracked object for subsequent posture tracking.

According to an exemplary embodiment of the present disclosure, there is provided a posture tracking device, comprising: an image acquisition unit, configured to acquire an image of a tracked object, wherein a mark flashing at a specific frequency is provided on the tracked object; a pixel acquisition unit, configured to acquire pixels with brightness changes from the acquired image; and a posture calculation unit, configured to calculate the 6-DOF posture of the tracked object based on the acquired pixels, thereby reducing the dependence of posture tracking on a specific layout of LED marks, while reducing the delay of posture tracking, and improving the accuracy and efficiency of posture tracking.

Optionally, the posture calculation unit can be configured to: obtain inertial measurement unit data of the tracked object, and estimate the 3-degree-of-freedom posture of the tracked object based on the acquired inertial measurement unit data, wherein the 3-degree-of-freedom posture is a posture rotating around the three coordinate axes x, y, and z in the body coordinate system of the tracked object; calculate the 6-degree-of-freedom posture of the tracked object based on the 3-degree-of-freedom posture and the acquired pixels, wherein the 6-degree-of-freedom posture is a posture along the three coordinate axes x, y, and z and a posture rotating around the three rectangular coordinate axes x, y, and z in the body coordinate system of the tracked object, thereby improving the accuracy and efficiency of posture tracking.

Optionally, the posture calculation unit can also be configured to: based on the 3-degree-of-freedom posture and the acquired pixels, solve the correspondence between the 2D point set and the 3D point set of the mark, and obtain matching pairs of the 2D point set and the 3D point set of the mark, wherein the 2D point set includes the pixel coordinates of the mark, and the 3D point set includes the coordinates of the mark in the body coordinate system of the tracked object; based on the matching pairs, calculate the 6-degree-of-freedom posture of the tracked object, thereby improving the accuracy and efficiency of posture tracking.

Optionally, the posture calculation unit can also be configured to: remove pixels whose reprojection deviation exceeds a preset deviation threshold from the matching pair, and calculate the 6-degree-of-freedom posture based on the remaining pixels after removal; perform a reprojection error minimization operation on the calculated 6-degree-of-freedom posture to obtain the 6-degree-of-freedom posture of the tracked object, thereby improving the accuracy and efficiency of posture tracking.

Optionally, the posture calculation unit can also be configured to: remove pixels whose reprojection deviation exceeds a preset deviation threshold from the matching pair, and calculate the 6-degree-of-freedom posture based on the remaining pixels after removal; perform a reprojection error minimization operation on the calculated 6-degree-of-freedom posture; and fuse the 3-degree-of-freedom posture with the 6-degree-of-freedom posture after the reprojection error minimization operation to obtain the 6-degree-of-freedom posture of the tracked object, thereby improving the accuracy and efficiency of posture tracking.

Optionally, the posture tracking device may also include: a re-matching unit, which is configured to re-match pixels whose reprojection errors exceed a preset deviation threshold according to the 6-degree-of-freedom posture of the tracked object after obtaining the 6-degree-of-freedom posture of the tracked object, for subsequent posture tracking.

According to an exemplary embodiment of the present disclosure, there is provided an electronic device, comprising: a camera, for acquiring an image of a tracked object, and acquiring pixels of brightness changes from the acquired image, wherein a mark flashing at a specific frequency is provided on the tracked object; and a processor, for calculating a 6-DOF posture of the tracked object based on the pixels acquired by the camera, thereby reducing the dependence of posture tracking on a specific layout of LED marks, while reducing the delay of posture tracking, and improving the accuracy and efficiency of posture tracking.

According to an exemplary embodiment of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the posture tracking method according to the exemplary embodiment of the present disclosure is implemented.

According to an exemplary embodiment of the present disclosure, there is provided a computing device, including: a processor; and a memory storing a computer program, wherein when the computer program is executed by the processor, a posture tracking method according to an exemplary embodiment of the present disclosure is implemented.

According to the posture tracking method and device of the exemplary embodiments of the present disclosure, by acquiring an image of the tracked object, pixels with brightness changes are acquired from the acquired image, and the 6-DOF posture of the tracked object is calculated based on the acquired pixels, thereby reducing the dependence of posture tracking on the specific layout of the LED marker, while reducing the latency of posture tracking and improving the accuracy and efficiency of posture tracking.

Additional aspects and/or advantages of the present general inventive concept will be set forth in part in the following description and in part will be clear from the description or may be learned through practice of the present general inventive concept.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other objects and features of the exemplary embodiments of the present disclosure will become more apparent through the following description in conjunction with the accompanying drawings which exemplarily illustrate the embodiments, in which:

FIG1 shows a flow chart of a posture tracking method according to an exemplary embodiment of the present disclosure;

FIG2 shows the 2D active LED marker tracking result of the tracked object;

FIG3 shows a block diagram of a posture tracking device according to an exemplary embodiment of the present disclosure;

FIG. 4 illustrates a schematic diagram of an electronic device according to an exemplary embodiment of the present disclosure; and

FIG. 5 shows a schematic diagram of a computing device according to an exemplary embodiment of the present disclosure.

DETAILED DESCRIPTION

Reference will now be made in detail to exemplary embodiments of the present disclosure, examples of which are shown in the accompanying drawings, wherein like reference numerals refer to like parts throughout. The embodiments will be described below with reference to the drawings in order to explain the present disclosure.

FIG1 shows a flow chart of a posture tracking method according to an exemplary embodiment of the present disclosure. The posture tracking method shown in FIG1 is applicable to a tracking object provided with a plurality of markers flashing at a specific frequency, wherein the marker flashing at a specific frequency may be, for example, an active LED. In the following description, an LED lamp is used as an example of a marker, but it should be understood that the present invention is not limited thereto. Those skilled in the art may adopt other forms of markers according to the needs of the embodiment.

1 , in step S101 , an image of a tracked object is acquired.

In an exemplary embodiment of the present disclosure, an image of a tracked object may be acquired by a DVS camera. The pose tracking method shown in FIG1 may be applicable to an electronic device having a camera capable of acquiring pixels of brightness changes and a host capable of performing calculations, or may be applicable to a system consisting of a camera capable of acquiring pixels of brightness changes and a host capable of performing calculations.

In step S102, pixels with brightness changes are obtained from the acquired image.

In an exemplary embodiment of the present disclosure, after the DVS camera obtains the image in step S101, it does not directly transmit the image to the host, but obtains pixels with brightness changes from the obtained image in step S102, and then transmits the obtained pixels to the host for posture tracking, thereby reducing the amount of data transmitted and the amount of data used for calculation, and reducing the delay of posture tracking.

Specifically, when a marker (e.g., an active LED light) flashes, DVS can generate corresponding on and off events. These events can be easily distinguished from other events by the frequency of the LED flashing. These filtered events can be divided into different clusters, each cluster representing an LED light. Then, a region growing-based clustering algorithm and a lightweight voting algorithm can be used to process these filtered events, and the point with the highest density in the cluster is identified as the center of the marker. In addition, a global nearest neighbor tracking method and a Kalman filter based on a uniform velocity model can be used to track multiple markers, thereby reducing the probability of missed detection and false detection of markers.

In step S103, the 6-DOF posture of the tracked object is calculated based on the acquired pixels. Here, the 6-DOF posture represents the posture along the three coordinate axes x, y, and z and the posture rotated around the three rectangular coordinate axes x, y, and z in the body coordinate system of the tracked object.

In an exemplary embodiment of the present disclosure, only those pixels whose brightness changes due to motion are used to calculate the 6-DOF pose of the tracked object, thereby reducing the latency of pose tracking.

In an exemplary embodiment of the present disclosure, when calculating the 6-DOF posture of a tracked object based on acquired pixels, the inertial measurement unit (IMU) data of the tracked object can be first acquired, and the 3-DOF posture of the tracked object (i.e., the posture rotating around the three coordinate axes x, y, and z in the body coordinate system of the tracked object) can be estimated based on the acquired inertial measurement unit (IMU) data, and then the 6-DOF posture of the tracked object is calculated based on the 3-DOF posture and the acquired pixels, thereby improving the accuracy and efficiency of posture tracking.

In an exemplary embodiment of the present disclosure, when calculating the 6-DOF posture of a tracked object based on the 3-DOF posture and the acquired pixels, the corresponding relationship between the 2D point set and the 3D point set of the marker can be solved based on the 3-DOF posture and the acquired pixels, and the matching pairs of the 2D point set and the 3D point set of the marker can be obtained, and then the 6-DOF posture of the tracked object can be calculated based on the matching pairs. Here, the 2D point set includes the pixel coordinates of each marker, and the 3D point set includes the coordinates of each marker in the body coordinate system of the tracked object.

Specifically, when calculating the 6-DOF posture of the tracked object based on the 3-DOF posture and the acquired pixels, p _IA and p _IB can be defined as the pixel coordinates of the two LED lights (LED light A and LED light B) that have been dedistorted on the image, and p _A and p _B are the coordinates of LED light A and LED light B in the object's body coordinate system. The 3-DOF posture matrix R = [r ₁ , r ₂ , r ₃ ] ^T of the tracked object can be estimated by the attitude reference system through IMU data, and r ₁ , r ₂ and r ₃ represent the row vectors of the first, second and third rows of R, respectively. t = [t _x , _ty , t _z ] ^T is an unknown translation vector, where t _x , _ty , t _z represent the displacements along the three coordinate axes of x, y and z, respectively. Through the pinhole imaging principle, the following equation can be obtained:

Among them, x _A and y _A represent the x-coordinate and y-coordinate of LED light A on the image. x _B and y _B represent the x-coordinate and y-coordinate of LED light B on the image. Only t is unknown in the above equations. There are 4 equations and 3 unknown quantities. Solving the equations yields:

t＝ _{AzpIA - RpA} (3)

in,

Through DVS camera detection and clustering algorithm, we can obtain the pixel coordinate point set O of the LED light in the image and the coordinate point set L of the known LED light in the object body coordinate system. We can select any two points (o, l) from the point sets O and L for pairing, and then we can calculate the translation t by formula (3). Through the above operation, we can get a list of possible translation vectors T. Some invalid translation vectors (the elements of each vector are too large or t _z is negative) can be deleted, and some approximately equal translation vectors can be merged. For any valid translation vector t _valid in T, we can determine the point set L _v of the visible LED light corresponding to it by the Moller-Trumbore ray intersection algorithm: if the first point where the ray determined by the camera intersects with the object is this LED light, then this LED light is visible, otherwise this LED light is invisible in this posture. Determining the visible LED light point set can reduce the amount of calculation and the situation of mismatching for multiple LED lights. Then the visible point set L _v is projected onto the image plane P using the corresponding 6-DOF posture and camera intrinsic parameters. In this way, the Kuhn-Munkres algorithm can be used to solve the best match and matching error between the visible point set L _v and the observation point set O. Traversing the possible translation vector list T, the set of matches with the smallest matching error is the correct matching pair of the 2D point set and the 3D point set.

In an exemplary embodiment of the present disclosure, when calculating the 6-DOF posture of a tracked object, pixels whose reprojection deviation exceeds a preset deviation threshold can be first removed from the matching pair, and the 6-DOF posture can be calculated based on the remaining pixels after removal. The calculated 6-DOF posture is then subjected to a reprojection error minimization operation to obtain the 6-DOF posture of the tracked object, thereby further improving the accuracy and efficiency of posture tracking.

In an exemplary embodiment of the present disclosure, when calculating the 6-DOF posture of a tracked object, pixels whose reprojection deviation exceeds a preset deviation threshold can be first removed from the matching pair, and the 6-DOF posture can be calculated based on the remaining pixels after the removal. The calculated 6-DOF posture is then subjected to a reprojection error minimization operation, and then the 6-DOF posture after the reprojection error minimization operation is optimized based on the 3-DOF posture to obtain the 6-DOF posture of the tracked object, thereby further improving the accuracy and efficiency of posture tracking.

In an exemplary embodiment of the present disclosure, after obtaining the 6-DOF posture of the tracked object, the pixels in the matching pair whose reprojection error exceeds a preset deviation threshold may be re-matched according to the 6-DOF posture of the tracked object. In addition, the correspondence between the 2D point set and the 3D point set of the newly observed marker (e.g., active LED light) or the matching pair of the 2D point set and the 3D point set may be updated, thereby further improving the accuracy and efficiency of posture tracking.

Specifically, after obtaining the correspondence between the 2D point set and the 3D point set, the Random Sample Consensus (RANSAC) algorithm can be used to remove the points in the matching pair whose reprojection deviation exceeds the preset deviation threshold, and then the Efficient Per Efficient Perspective-n-Pointspective-n-Point (EPnP) algorithm is used to solve the 6-DOF pose. Then, the rough pose obtained by the EPnP algorithm is further optimized using the Bundle Adjustment (BA) algorithm, so that a more accurate pose can be obtained. This accurate pose can be used to re-match the points in the matching pair whose reprojection error exceeds the preset deviation threshold and update the matching relationship of the newly observed LED. Finally, the sensor fusion algorithm based on the extended Kalman filter can be used to fuse the above-obtained 6-DOF pose with the 3-DOF pose in the IMU to obtain a smoother and more consistent 6-DOF pose (i.e., the fused 6-DOF pose). In addition, the fused 6-DOF pose can be used to rematch points whose reprojection errors exceed the preset deviation threshold and update the newly observed LED matching relationship.

According to the pose tracking method of the exemplary embodiment of the present disclosure, the dependence of pose tracking on the specific layout of LED markers is reduced, while the latency of pose tracking is reduced and the accuracy and efficiency of pose tracking are improved.

The DVS camera used to obtain pixels that cause brightness changes due to the movement of the tracking object can be, for example, a third-generation VGA device from Samsung, which has a resolution of 640*480 and can be connected to the host via USB3.0. The DVS camera can generate events for pixels with changes in relative light intensity. Each event is represented by a tuple <t,x,y,p>, where t is the timestamp of the event (with microsecond resolution), x,y are the pixel coordinates corresponding to the event, and p∈{0,1} is the polarity of the event, where when the LED light is on, an event of <t,x,y,1> is generated, and when the LED light is off, an event of <t,x,y,0> is generated.

The pose tracking method requires the application of some fixed parameters (camera intrinsic parameters) and transfer matrices (IMU to handle body, DVS camera to helmet, etc.). In the exemplary embodiment of the present disclosure, the OptiTrack optical motion tracking system is used to calibrate these fixed parameters and transfer matrices. At the same time, the OptiTrack optical motion tracking system is also used to provide the true value of the handle pose to evaluate the accuracy of pose tracking. After the calibration is completed, the OptiTrack optical motion tracking system is not needed in the actual use of the user.

是手柄模型坐标系到相机坐标系的旋转矩阵，^CP_M是模型坐标系原点在相机坐标系下的表示。需要提前标定DVS相机的内参和一些固定的转移矩阵

There are multiple coordinate systems in the OptiTrack optical motion tracking system, such as the camera (C) coordinate system, the helmet (H) coordinate system, the world (W) coordinate system, the IMU (I) coordinate system, the handle body (B) coordinate system, the handle model (M) coordinate system, and the OptiTrack (O) coordinate system. In order to simplify the OptiTrack optical motion tracking system, the handle body coordinate system and the handle model coordinate system can be aligned, and the world coordinate system and the OptiTrtack coordinate system can be aligned. Therefore, the 6-DOF pose of the motion handle to be solved can be expressed as

is the rotation matrix from the handle model coordinate system to the camera coordinate system, _and ^CPM is the representation of the origin of the model coordinate system in the camera coordinate system. The intrinsic parameters of the DVS camera and some fixed transfer matrices need to be calibrated in advance.

Since the optical properties of DVS are the same as those of ordinary CMOS cameras, the standard pinhole imaging model can be used to determine the camera internal parameters (for example, focal length, projection center and distortion parameters). The difference between DVS cameras and ordinary cameras is that DVS cameras cannot see things without lighting changes, so a flashing chessboard can be used to calibrate DVS.

也可以通过那些特定模式的闪烁LED来确定：

其中，

和

都由OptiTrack光学运动跟踪系统来提供，可以通过用EPnP算法计算得到

(特定模式的点集匹配关系是固定的)。计算得到的两个矩阵例如可如下所示：The OptiTrack ball can be used to calibrate some fixed transfer matrices. In the 3D model, the origin of the handle model is the center of the large circle at the top of the handle. The OptiTrack ball can be fixed to the circumference of the large circle, and the directions of the X, Y, and Z axes in the marked model can be marked accordingly. This way, the model coordinate system of the motion handle can be determined in the OptiTrack optical motion tracking system. Fix the OptiTrack ball to the IMU chip, and use the IMU readings to determine the directions of the X, Y, and Z axes. This way, the IMU coordinate system is determined in the OptiTrack optical motion tracking system. The OptiTrack optical motion tracking system can record the position and posture of the IMU coordinate system and the model coordinate system in real time, so the conversion relationship between the model coordinate system of the motion handle and the IMU coordinate system can be calculated. Moreover, the transformation matrix from the helmet to the camera

It can also be determined by the LEDs flashing in a specific pattern:

in,

and

All are provided by the OptiTrack optical motion tracking system and can be calculated using the EPnP algorithm

(The point set matching relationship of a specific pattern is fixed.) The two calculated matrices can be shown as follows, for example:

The flashing interval of the active LED marker can be determined by the on-off-on, off-on-off events. If the flashing interval of a pixel is in the interval [800μs, 1200μs], it can be concluded that the event is caused by the LED flashing. Then the region growing algorithm and voting method are used to determine the center position of the LED. Multiple LED lights can be tracked using global nearest neighbor and Kalman filtering. Figure 2 shows the 2D active LED marker tracking results of the tracked object. In Figure 2, each continuous line represents the motion trajectory of a marker (e.g., LED), the small hollow circle represents the starting point of the motion trajectory, and the large circle with a solid circle inside represents the end point of the motion trajectory.

In addition, in order to improve the performance of the BA algorithm, the optimization window of the BA algorithm can be set to 10, and the optimization is performed once every 4 frames. The matching relationship does not need to be updated every time. The matching relationship is updated only when the ratio of the number of matched LED lights to the number of all observed LED lights is less than a preset value, such as 0.6, thereby improving the efficiency of the BA algorithm. After the startup phase is initialized, the entire processing flow only takes 1.23 milliseconds. Therefore, adding the screening time of the LED light flashing event (1 millisecond), the total delay is 2.23 milliseconds.

The posture tracking method according to the exemplary embodiment of the present disclosure has been described above in conjunction with Figures 1 to 2. Hereinafter, the posture tracking device and its units according to the exemplary embodiment of the present disclosure will be described with reference to Figure 3.

FIG. 3 shows a block diagram of a posture tracking apparatus according to an exemplary embodiment of the present disclosure.

3 , the posture tracking device includes an image acquisition unit 31 , a pixel acquisition unit 32 , and a posture calculation unit 33 .

The image acquisition unit 31 is configured to acquire an image of a tracked object, wherein a marker flashing at a specific frequency is provided on the tracked object.

The pixel acquisition unit 32 is configured to acquire pixels of brightness change from the acquired image.

The posture calculation unit 33 is configured to calculate the 6-DOF posture of the tracked object based on the acquired pixels.

In an exemplary embodiment of the present disclosure, the posture calculation unit 33 can be configured to: acquire inertial measurement unit data of the tracked object, and estimate the 3-degree-of-freedom posture of the tracked object based on the acquired inertial measurement unit data, where the 3-degree-of-freedom posture is a posture rotating around the three coordinate axes x, y, and z in the body coordinate system of the tracked object; calculate the 6-degree-of-freedom posture of the tracked object based on the 3-degree-of-freedom posture and the acquired pixels, where the 6-degree-of-freedom posture is a posture along the three coordinate axes x, y, and z and a posture rotating around the three rectangular coordinate axes x, y, and z in the body coordinate system of the tracked object.

In an exemplary embodiment of the present disclosure, the posture calculation unit 33 can also be configured to: based on the 3-degree-of-freedom posture and the acquired pixels, solve the correspondence between the marked 2D point set and the 3D point set, and obtain the matching pairs of the marked 2D point set and the 3D point set, where the 2D point set includes the pixel coordinates of each marker, and the 3D point set includes the coordinates of each marker in the body coordinate system of the tracked object; based on the matching pairs, calculate the 6-degree-of-freedom posture of the tracked object.

In an exemplary embodiment of the present disclosure, the posture calculation unit 33 can also be configured to: remove pixels whose reprojection deviation exceeds a preset deviation threshold from the matching pair, and calculate the 6-degree-of-freedom posture based on the remaining pixels after removal; perform a reprojection error minimization operation on the calculated 6-degree-of-freedom posture to obtain the 6-degree-of-freedom posture of the tracked object.

In an exemplary embodiment of the present disclosure, the posture calculation unit 33 can also be configured to: remove pixels whose reprojection deviation exceeds a preset deviation threshold from the matching pair, and calculate the 6-degree-of-freedom posture based on the remaining pixels after removal; perform a reprojection error minimization operation on the calculated 6-degree-of-freedom posture; optimize the 6-degree-of-freedom posture after the reprojection error minimization operation according to the 3-degree-of-freedom posture to obtain the 6-degree-of-freedom posture of the tracked object.

In an exemplary embodiment of the present disclosure, the posture tracking device may further include: a re-matching unit, which is configured to re-match pixels whose reprojection errors exceed a preset deviation threshold according to the 6-degree-of-freedom posture of the tracked object after obtaining the 6-degree-of-freedom posture of the tracked object.

FIG. 4 is a schematic diagram illustrating an electronic device according to an exemplary embodiment of the present disclosure.

4 , the electronic device 4 includes a camera 41 and a processor 42 .

The camera 41 is used to obtain an image of the tracked object and obtain pixels with brightness changes from the obtained image, wherein a mark flashing at a specific frequency is set on the tracked object. The processor 42 is used to calculate the 6-DOF posture of the tracked object based on the pixels obtained by the camera.

In an exemplary embodiment of the present disclosure, the processor 42 can be used to acquire inertial measurement unit data of the tracked object, and estimate the 3-degree-of-freedom posture of the tracked object based on the acquired inertial measurement unit data, where the 3-degree-of-freedom posture is a posture rotating around the three coordinate axes x, y, and z in the body coordinate system of the tracked object; and calculate the 6-degree-of-freedom posture of the tracked object based on the 3-degree-of-freedom posture and the acquired pixels, where the 6-degree-of-freedom posture is a posture along the three coordinate axes x, y, and z and a posture rotating around the three rectangular coordinate axes x, y, and z in the body coordinate system of the tracked object.

In an exemplary embodiment of the present disclosure, the processor 42 can be used to solve the correspondence between the marked 2D point set and the 3D point set based on the 3-DOF posture and the acquired pixels, and obtain matching pairs of the marked 2D point set and the 3D point set, where the 2D point set includes the pixel coordinates of each marker, and the 3D point set includes the coordinates of each marker in the body coordinate system of the tracked object; based on the matching pairs, the 6-DOF posture of the tracked object is calculated.

In an exemplary embodiment of the present disclosure, the processor 42 can be used to remove pixels whose reprojection deviation exceeds a preset deviation threshold from the matching pair, and calculate the 6-degree-of-freedom pose based on the remaining pixels after removal; perform a reprojection error minimization operation on the calculated 6-degree-of-freedom pose to obtain the 6-degree-of-freedom posture of the tracked object.

In an exemplary embodiment of the present disclosure, the processor 42 can be used to remove pixels whose reprojection deviation exceeds a preset deviation threshold from the matching pair, and calculate the 6-degree-of-freedom pose based on the remaining pixels after the removal; perform a reprojection error minimization operation on the calculated 6-degree-of-freedom pose; and optimize the 6-degree-of-freedom pose after the reprojection error minimization operation based on the 3-degree-of-freedom pose to obtain the 6-degree-of-freedom pose of the tracked object.

In an exemplary embodiment of the present disclosure, the processor 42 may be configured to re-match pixels having a reprojection error exceeding a preset deviation threshold according to the 6-DOF posture of the tracked object after obtaining the 6-DOF posture of the tracked object.

In addition, according to an exemplary embodiment of the present disclosure, a computer-readable storage medium is also provided, on which a computer program is stored. When the computer program is executed, the posture tracking method according to the exemplary embodiment of the present disclosure is implemented.

As an example, the computer-readable storage medium may carry one or more programs, and when the computer program is executed, the following steps may be implemented: acquiring an image of a tracked object, wherein a mark flashing at a specific frequency is provided on the tracked object; acquiring pixels with brightness changes from the acquired image; and calculating a 6-DOF posture of the tracked object based on the acquired pixels.

The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In an embodiment of the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a computer program that may be used by or in conjunction with an instruction execution system, device or device. The computer program contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination thereof. The computer-readable storage medium may be contained in any device; it may also exist alone without being assembled into the device.

The posture tracking device and the electronic device according to the exemplary embodiment of the present disclosure have been described above in conjunction with Figures 3 and 4. Next, a computing device according to the exemplary embodiment of the present disclosure will be described in conjunction with Figure 5.

FIG. 5 shows a schematic diagram of a computing device according to an exemplary embodiment of the present disclosure.

5 , a computing device 5 according to an exemplary embodiment of the present disclosure includes a memory 51 and a processor 52 , wherein the memory 51 stores a computer program, and when the computer program is executed by the processor 52 , the posture tracking method according to the exemplary embodiment of the present disclosure is implemented.

As an example, when the computer program is executed by the processor 52, the following steps can be implemented: acquiring an image of a tracked object, wherein a mark flashing at a specific frequency is provided on the tracked object; acquiring pixels with brightness changes from the acquired image; and calculating a 6-DOF posture of the tracked object based on the acquired pixels.

The computing device shown in FIG. 5 is merely an example and should not limit the functionality and scope of use of the embodiments of the present disclosure.

The posture tracking method and device according to the exemplary embodiments of the present disclosure have been described above with reference to Figures 1 to 5. However, it should be understood that the posture tracking device and its units shown in Figure 3 can be respectively configured as software, hardware, firmware or any combination of the above items to perform specific functions, and the computing device shown in Figure 5 is not limited to including the components shown above, but some components can be added or deleted as needed, and the above components can also be combined.

According to the posture tracking method and device of the exemplary embodiments of the present disclosure, by acquiring an image of the tracked object, pixels with brightness changes are acquired from the acquired image, and the 6-DOF posture of the tracked object is calculated based on the acquired pixels, thereby reducing the dependence of posture tracking on the specific layout of the LED marker, while reducing the latency of posture tracking and improving the accuracy and efficiency of posture tracking.

While the present disclosure has been particularly shown and described with reference to exemplary embodiments thereof, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the present disclosure as defined by the following claims.

Claims (11)

1. A pose tracking method, comprising:

acquiring an image of a tracking object, wherein a mark which flickers at a specific frequency is arranged on the tracking object;

acquiring pixels with changed brightness from the acquired image;

calculating a 6 degree-of-freedom pose of the tracked object based on the acquired pixels,

wherein the step of calculating the 6 degree-of-freedom pose of the tracked object based on the acquired pixels comprises:

acquiring inertial measurement unit data of the tracked object, and estimating a 3-degree-of-freedom attitude of the tracked object based on the acquired inertial measurement unit data, wherein the 3-degree-of-freedom attitude is an attitude rotating around three coordinate axes of x, y and z in a body coordinate system of the tracked object;

based on the 3-degree-of-freedom posture and the obtained pixels, solving a corresponding relation between a 2D point set and a 3D point set of the marker to obtain a matching pair of the 2D point set and the 3D point set of the marker, wherein the 2D point set comprises pixel coordinates of the marker, and the 3D point set comprises coordinates of the marker in a body coordinate system of the tracked object;

and calculating the 6-degree-of-freedom posture of the tracked object based on the matching pair, wherein the 6-degree-of-freedom posture is a posture in the directions of three coordinate axes of x, y and z and a posture rotating around three rectangular coordinate axes of x, y and z under the body coordinate system of the tracked object.

2. The pose tracking method of claim 1, wherein the step of calculating a 6 degree of freedom pose of the tracked object comprises:

removing pixels with the reprojection deviation exceeding a preset deviation threshold value from the matching pair, and calculating a pose with 6 degrees of freedom according to the remaining pixels after removal;

and carrying out minimum reprojection error operation on the calculated pose with 6 degrees of freedom to obtain the pose with 6 degrees of freedom of the tracked object.

3. The pose tracking method of claim 1, wherein the step of calculating a 6 degree of freedom pose of the tracked object comprises:

removing pixels with the reprojection deviation exceeding a preset deviation threshold value from the matching pair, and calculating a pose with 6 degrees of freedom according to the remaining pixels after removal;

performing minimum reprojection error operation on the pose with 6 degrees of freedom obtained by calculation;

and optimizing the pose of 6 degrees of freedom after the operation of minimizing the reprojection error according to the pose of 3 degrees of freedom to obtain the pose of 6 degrees of freedom of the tracked object.

4. The pose tracking method according to claim 2 or 3, further comprising, after obtaining the 6 degree-of-freedom pose of the tracked object:

and carrying out re-matching on the pixels of which the re-projection errors in the matching pairs exceed a preset deviation threshold according to the 6-degree-of-freedom posture of the tracked object.

5. A pose tracking apparatus, comprising:

an image acquisition unit configured to acquire an image of a tracking object on which a marker that blinks at a specific frequency is provided;

a pixel acquisition unit configured to acquire a pixel of which luminance varies from the acquired image; and

a pose calculation unit configured to calculate a 6-degree-of-freedom pose of the tracking object based on the acquired pixels,

wherein the attitude calculation unit is configured to:

acquiring inertial measurement unit data of the tracked object, and estimating a 3-degree-of-freedom attitude of the tracked object based on the acquired inertial measurement unit data, wherein the 3-degree-of-freedom attitude is an attitude rotating around three coordinate axes of x, y and z under a body coordinate system of the tracked object;

based on the 3-degree-of-freedom posture and the obtained pixels, solving a corresponding relation between a 2D point set and a 3D point set of the marker to obtain a matching pair of the 2D point set and the 3D point set of the marker, wherein the 2D point set comprises pixel coordinates of the marker, and the 3D point set comprises coordinates of the marker in a body coordinate system of the tracked object;

and calculating the 6-degree-of-freedom posture of the tracked object based on the matching pair, wherein the 6-degree-of-freedom posture is a posture in the directions of three coordinate axes of x, y and z and a posture rotating around three rectangular coordinate axes of x, y and z under the body coordinate system of the tracked object.

6. The pose tracking apparatus of claim 5, wherein the pose calculation unit is further configured to:

removing pixels with the reprojection deviation exceeding a preset deviation threshold value from the matching pair, and calculating a pose with 6 degrees of freedom according to the remaining pixels after removal;

and carrying out minimum reprojection error operation on the calculated pose with 6 degrees of freedom to obtain the pose with 6 degrees of freedom of the tracked object.

7. The pose tracking apparatus of claim 6, wherein the pose calculation unit is further configured to:

removing pixels with the reprojection deviation exceeding a preset deviation threshold value from the matching pair, and calculating a pose with 6 degrees of freedom according to the remaining pixels after removal;

carrying out minimum reprojection error operation on the pose with 6 degrees of freedom obtained by calculation;

and optimizing the pose of 6 degrees of freedom after the operation of minimizing the reprojection error according to the pose of 3 degrees of freedom to obtain the pose of 6 degrees of freedom of the tracked object.

8. The pose tracking apparatus of claim 6 or 7, further comprising:

a re-matching unit configured to re-match pixels, whose re-projection errors exceed a preset deviation threshold, according to the 6-degree-of-freedom posture of the tracked object after the 6-degree-of-freedom posture of the tracked object is obtained.

9. An electronic device, comprising:

a camera for acquiring an image of a tracking object on which a marker blinking at a specific frequency is provided, and acquiring pixels of which luminance varies from the acquired image, and

a processor for acquiring inertial measurement unit data of the tracked object, and estimating a 3-degree-of-freedom attitude of the tracked object based on the acquired inertial measurement unit data, wherein the 3-degree-of-freedom attitude is an attitude rotating around three coordinate axes of x, y, and z in a body coordinate system of the tracked object; based on the 3-degree-of-freedom posture and the obtained pixels, solving a corresponding relation between a 2D point set and a 3D point set of the marker to obtain a matching pair of the 2D point set and the 3D point set of the marker, wherein the 2D point set comprises pixel coordinates of the marker, and the 3D point set comprises coordinates of the marker in a body coordinate system of the tracked object; and calculating the 6-degree-of-freedom posture of the tracked object based on the matching pair, wherein the 6-degree-of-freedom posture is a posture in the directions of three coordinate axes of x, y and z and a posture rotating around three rectangular coordinate axes of x, y and z under the body coordinate system of the tracked object.

10. A computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the pose tracking method of any one of claims 1 to 4.

11. A computing device, comprising:

a processor;

a memory storing a computer program that, when executed by the processor, implements the pose tracking method of any one of claims 1 to 4.

Images