TW202203644A - Method and system for simultaneously tracking 6 dof poses of movable object and movable camera - Google Patents

Abstract

A method and a system for simultaneously tracking several 6 DoF poses of a movable object and a movable camera are provided. The method includes the following steps: A series of images are captured by a movable camera. Several environmental feature points are extracted from the images. The environmental feature points are matched to calculate several camera matrixes of the movable camera, and then the 6 DoF poses of the movable camera are calculated by the camera matrixes. At the same time, several feature points of the movable object are inferred from the images captured by the movable camera. The coordinates of the feature points of the movable object are corrected through the camera matrixes corresponding to the images, as well as the predefined geometric and temporal constraints. Then, the 6 DoF poses of the movable object are calculated.

Description

Method and system for simultaneously tracking the 6-DOF orientation of a movable object and a movable camera

The present disclosure relates to a method and system for simultaneously tracking the 6DOF orientation of a movable object and a movable camera.

Among existing tracking technologies, such as Simultaneous Localization And Mapping (SLAM), which can track the 6-DOF orientation of movable cameras, it cannot track movable objects at the same time. The reason is that the movable camera needs stable environment feature points for localization, and the feature points of movable objects are unstable and are usually discarded and cannot be used for tracking.

On the other hand, techniques used to track moving objects ignore environmental features to avoid interference, so none of these techniques can track moving cameras.

Most of the features learned by neural networks are used to distinguish the type of objects, not to calculate the six-degree-of-freedom orientation of the object. Some neural networks for recognizing poses or gestures can only output the 2D coordinates (x, y) of the skeletal joints in the image plane. Even if the distance between the joint and the camera is estimated by depth sensing technology, it is not the real space. 3D coordinates, and it is impossible to calculate the six-degree-of-freedom orientation in space.

In a motion capture system, multiple fixed cameras are used to track joint positions, and marks are generally attached to the joints to reduce errors, and the 6-DOF orientation of movable cameras is not tracked.

Therefore, as far as the currently known technology is concerned, there is no technology that can simultaneously track a movable object and a movable camera.

The rapid development of mixed reality (MR) has prompted researchers to develop technologies that can simultaneously track the orientation of moveable cameras and moveable objects in six degrees of freedom. In mixed reality applications, since the camera mounted on the MR glasses moves with the head, it is necessary to know the 6-DOF orientation of the camera to know the user's position and orientation. Objects that the user interacts with also move, so knowledge of the object's 6DOF orientation is also required to display virtual content in the proper position and orientation. Users wearing MR glasses may move freely indoors or outdoors, making it difficult to place markers in the environment. Moreover, in order to have a better user experience, in addition to the characteristics of the object itself, no special marks will be attached to the object.

While these situations increase the difficulty of tracking 6DOF orientations, we have developed techniques that can track both moving objects and moving cameras simultaneously to address these issues and satisfy more applications.

The technology proposed in the present disclosure can be applied to, for example, when a user wears MR glasses, one or more virtual screens can be displayed next to the real screen of a handheld device, such as a mobile phone, according to the six characteristics of the camera on the mobile phone and MR glasses. DOF Orientation sets the default position, orientation and size of the virtual screen. And, through 6-DOF azimuth tracking, the virtual screen can be automatically controlled to rotate and move so that it is consistent with the viewing direction. The disclosed technology can provide users with the following benefits: (1) expand a small physical screen to a large virtual screen; (2) increase a single physical screen to multiple virtual screens to view more applications at the same time; (3) ) The content of the virtual screen will not be spied on by others.

According to an embodiment of the present disclosure, a method for simultaneously tracking 6 DoF poses of a movable object and a movable camera is proposed, which includes the following steps: capturing a series of images with the movable camera, and from the images Extract several environmental feature points, match these environmental feature points to calculate several camera matrices of the movable camera, and then calculate the six-degree-of-freedom orientation of the movable camera from these camera matrices; and at the same time calculate from these images captured by the movable camera Several feature points of the movable object, using the corresponding camera matrices of these images, as well as the pre-defined geometric and time constraints, correct the coordinates of these feature points of the movable object, and then use the corrected coordinates of the feature points and their The corresponding camera matrix, calculates the six-degree-of-freedom orientation of the movable object.

According to another embodiment of the present disclosure, a system for simultaneously tracking the 6-DOF orientation of a movable object and a movable camera is provided, including a movable camera, a movable camera 6-DOF orientation calculation unit, and a movable object 6-DOF orientation computing unit. The movable camera is used to capture a series of images. The 6-DOF orientation calculation unit of the movable camera is used to extract several environmental feature points from these images, match these environmental feature points to calculate several camera matrices of the movable camera, and then calculate the 6-DOF of the movable camera from these camera matrices position. The six-degree-of-freedom orientation calculation unit of the movable object is used to calculate several feature points of the movable object from the images captured by the movable camera, through the corresponding camera matrices of these images, as well as the predefined geometric constraints, and time Restriction, correct the coordinates of these feature points of the movable object, and then calculate the six-degree-of-freedom orientation of the movable object based on the corrected coordinates of the feature points and the corresponding camera matrices.

In order to have a better understanding of the above-mentioned and other aspects of the present disclosure, the following embodiments are given and described in detail with the accompanying drawings as follows:

Please refer to FIGS. 1A and 1B , which illustrate the application of the technology of simultaneously tracking a movable object and a movable camera of the present disclosure compared with the prior art. For example, the technology proposed in the present disclosure can be applied to: as shown in FIG. 1A , when the user wears the MR glasses G1 (the movable camera 110 is arranged on the MR glasses G1 ), the user can use a handheld device, such as the mobile phone P1 (ie, the mobile phone P1 ) One or more virtual screens are displayed next to the real screen of the movable object 900), and the preset positions, directions and sizes of the virtual screens D2 and D3 are set according to the 6-DOF orientation of the movable camera 110 on the mobile phone P1 and the MR glasses G1. "Moveable" of the movable camera 110 means relative to a stationary object in three-dimensional space. Moreover, through the 6-DOF azimuth tracking, the rotation and movement of the virtual screens D2 and D3 can be automatically controlled to be consistent with the viewing direction (as shown in Figure 1B). Users can also adjust these virtual screens D2 according to their own preferences. , the position and angle of D3. The virtual screen displayed by the conventional technology will move with the MR glasses G1, and will not move with the 6-DOF orientation of the object. The disclosed technology can provide users with the following benefits: (1) expand a small physical screen D1 to a large virtual screen D2; (2) increase a single physical screen D1 to multiple virtual screens D2 and D3 to view more Multi-application; (3) The content of virtual screens D2 and D3 will not be spied on by others. The above technology can also be applied to a tablet or laptop, where a virtual screen is placed alongside its physical screen. In addition to the physical screen, the movable object 900 can also be other objects that can define features, such as: cars, bicycles, pedestrians, and the like. The movable camera 110 is not limited to a camera on the MR glasses G1, but can also be a camera on an autonomous mobile robot and a vehicle.

Please refer to FIG. 2A , which illustrates a system 100 and method for simultaneously tracking the 6DOF orientation of a movable object 900 (labeled in FIG. 1A ) and a movable camera 110 according to one embodiment. The movable object 900 is, for example, the mobile phone P1 in FIG. 1A ; the movable camera 110 is, for example, the camera on the MR glasses G1 in FIG. 1A . The system 100 for simultaneously tracking the 6DOF orientation of the movable object 900 and the movable camera 110 includes the movable camera 110 , the movable camera 6DOF orientation calculation unit 120 and the movable object 6DOF orientation calculation unit 130 . The movable camera 110 is used to capture a series of images IM. The movable camera 110 can be installed on a head-mounted stereoscopic display, a mobile device, a computer or a robot. The movable camera 6-DOF orientation calculation unit 120 and/or the movable object 6-DOF orientation calculation unit 130 is, for example, a circuit, a chip, a circuit board, a code, or a storage device for storing the code.

The six-degree-of-freedom orientation calculation unit 120 of the movable camera includes an environmental feature extraction unit 121, a camera matrix calculation unit 122 and a camera orientation calculation unit 123, the implementations of which are, for example, circuits, chips, circuit boards, code, or stored code. storage device. The environmental feature extraction unit 121 is used for extracting several environmental feature points EF from the images IM. The camera matrix calculation unit 122 calculates several camera matrices CM of the movable camera 110 by matching these environmental feature points EF. The camera orientation calculation unit 123 then calculates the six-degree-of-freedom orientation CD of the movable camera 110 from the camera matrix CM.

The six-degree-of-freedom orientation calculation unit 130 of the movable object includes an object feature coordinate calculation unit 131, an object feature coordinate correction unit 132, and an object orientation calculation unit 133, which are implemented by, for example, a circuit, a chip, a circuit board, a code, or a stored code. storage device. The object feature coordinate estimation unit 131 is used to estimate several feature points OF of the movable object 900 from the images IM captured by the movable camera 110 . These feature points OF are predefined, and the images captured by the movable camera 110 IM makes a comparison to calculate the coordinates of these feature points OF. The movable object 900 is a rigid object.

Referring to another embodiment shown in FIG. 2B , the method for simultaneously tracking the 6DOF orientation of the movable object 900 and the movable camera 110 includes a training stage ST1 and a tracking stage ST2 . The object feature coordinate estimation unit 131 uses the neural network inference model MD to estimate the coordinates of the feature point OF of the movable object 900 from the images IM captured by the movable camera 110 . The neural network inference model MD is pre-trained and trained. Data were obtained by manual or automatic labeling, and geometry-limited GC and time-limited TC were added during training.

The object feature coordinate correction unit 132 corrects the coordinates of these feature points OF of the movable object 900 by using the camera matrix CM corresponding to each of the images IM, and the predefined geometric constraints GC and time constraints TC. Wherein, the object feature coordinate correction unit 132 uses these camera matrices CM to project the two-dimensional coordinates of these feature points OF to the corresponding three-dimensional coordinates, and delete the feature points OF whose three-dimensional coordinate deviation is greater than a predetermined value according to the geometric limit GC, or use the corresponding The coordinates of the adjacent feature points OF are supplemented by the coordinates of the undetected feature points OF according to the geometric constraints GC. In addition, the object feature coordinate correction unit 132 compares the coordinate changes of the feature points OF in the plurality of continuous images IM according to the time limit TC, and then corrects the coordinate changes based on the coordinates of the corresponding feature points OF in the continuous images IM greater than The coordinates of the feature points OF with predetermined values are obtained to obtain the coordinates of these feature points OF' after correction.

Please refer to FIG. 3A , which illustrates the corresponding relationship between the environmental feature points and the movable object feature points in a series of images captured by the movable camera. For non-planar objects, the orientation and position can be defined by the centroids of several selected feature points OF. Please refer to Figure 3B, which illustrates the position and orientation of objects in space. The feature point OF fits the best plane PL, the center point C of the best plane PL can represent the position (x, y, x) of the object in the three-dimensional space, and the normal vector N of the best plane PL can represent the object's position. direction.

The geometric constraint GC is defined in three-dimensional space, and for rigid objects, the distance between feature points OF should be fixed. After the camera matrix is projected to the two-dimensional image plane, the positions of all feature points OF must be limited to a reasonable range.

Please refer to FIGS. 4A to 4B, which illustrate the coordinates of the corrected feature point OF as an example. The camera matrix CM can not only be used to calculate the six-degree-of-freedom orientation of the movable camera 110 and the movable object 900, but also apply the three-dimensional geometric constraints GC to correct the coordinates of the feature point OF* projected to the two-dimensional image plane (as shown in Figure 4A). shown) or add the missing feature point OF** coordinates (as shown in Figure 4B).

The object orientation calculation unit 133 then calculates the six-degree-of-freedom orientation OD of the movable object 900 based on the corrected coordinates of the feature points OF' and the corresponding camera matrices CM. For plane movable objects, use these feature points OF to calculate the best fit plane. The six-degree-of-freedom orientation OD of the movable object 900 is defined by the center point and normal vector of the plane. For a non-planar movable object, the six-degree-of-freedom orientation OD of the movable object 900 is defined by the centroid of the three-dimensional coordinates of these feature points OF'.

As shown in FIG. 2B , the training stage ST1 of the system 100 for simultaneously tracking the 6-DOF orientation of the movable object 900 and the movable camera 110 includes a training data generating unit 140 and a neural network training unit 150 . The method is, for example, a circuit, a chip, a circuit board, a code, or a storage device that stores the code.

、座標

、座標

、座標

。即使將實體螢幕D4屏幕旋轉到橫向，特徵點OF的順序也保持不變（如第5B圖所示）。在某些情況中，並不是所有的特徵點OF都能被拍到。因此，訓練資料需要包含一些類似第5C圖或第5D圖這種缺漏一些特徵點OF的影像。如第5A圖與第5D圖所示，特徵點標記的動作可以分辨出手機的正面（即螢幕）與背面，而僅在正面進行標記。為了獲得較高的精準度，在標記特徵點OF時放大每張影像，直到清楚地看到每個像素。由於手動標記的動作非常耗時，因此需要自動擴充才能將訓練資料擴展到百萬張數等級。對手動標記的影像進行自動擴充的方法包含：按比例縮放與旋轉、以透視投影法進行映射、轉換到不同的顏色、調整其亮度和對比度、添加移動模糊和雜訊、加上其他物體遮蓋某些特徵點(如第5C圖與第5D圖所示)、變更螢幕顯示的內容、或者替換背景等等。再將這些手動標記之特徵點OF的位置按照轉換關係重新計算在自動擴充的影像中的位置。The neural network training unit 150 is used for training the neural network inference model MD. The neural network inference model MD is used to infer the position and sequence of the feature points OF of the movable object 900 . In the training data generating unit 140, the training data may be images of manually marked positions and sequences of feature points, or images that have been automatically augmented. Please refer to Figures 5A to 5D, which illustrate various training materials using a mobile phone as an example. In these figures, the feature point OF is defined by the four inner corners of the solid screen D4. When the physical screen D4 is placed in the vertical direction, the order of the four feature points OF is designated in the clockwise direction from the upper left corner to the lower left corner. As shown in Fig. 5A, the four feature points OF have coordinates in sequence

,coordinate

,coordinate

,coordinate

. Even if the solid screen D4 screen is rotated to landscape orientation, the order of feature points OF remains unchanged (as shown in Fig. 5B). In some cases, not all feature points OF can be captured. Therefore, the training data needs to include some images like the 5C or 5D images that lack some feature points OF. As shown in Figures 5A and 5D, the action of feature point marking can distinguish the front (ie, screen) and back of the mobile phone, and only mark the front. For better accuracy, zoom in on each image while marking the feature point OF until each pixel is clearly seen. Since manually labeled actions are time-consuming, automatic augmentation is required to scale the training data to the millions of sheets. Methods for automatic augmentation of manually labeled images include: scaling and rotating, mapping in perspective projection, converting to different colors, adjusting their brightness and contrast, adding motion blur and noise, and adding other objects to obscure certain objects. Some feature points (as shown in Figure 5C and Figure 5D), change the content displayed on the screen, or replace the background and so on. Then, the positions of these manually marked feature points OF are recalculated in the automatically expanded image according to the conversion relationship.

表示、前一張影像之特徵點OF的座標以

表示）。除了特徵點預測層之外，本實施例還加上幾何限制層GCL和時間限制層TCL以減少錯誤的預測。在訓練階段，每一層會根據損失函數計算出預測值與真值的損失值LV，然後將這些損失值及其各自的權重進行累加以獲得總損失值OLV。Please refer to Figure 6, which illustrates that the main structure of the neural network in the training phase includes feature extraction and feature point coordinate prediction. The feature extractor ET can use a deep residual network such as ResNet or other networks with similar functions. The extracted feature vector FV is sent to the feature point coordinate prediction layer FL, and the coordinates of the feature point OF are estimated (for example, the coordinates of the feature point OF of the current image are

Indicates that the coordinates of the feature point OF of the previous image are given by

Express). In addition to the feature point prediction layer, this embodiment also adds a geometric constraint layer GCL and a temporal constraint layer TCL to reduce erroneous predictions. In the training phase, each layer calculates the loss value LV of the predicted value and the true value according to the loss function, and then accumulates these loss values and their respective weights to obtain the total loss value OLV.

，同一特徵點OF在前一張影像中的座標為

，其間的位移定為

。Please refer to FIG. 7 , which illustrates the calculation method of the displacement of feature points between two adjacent images. The coordinates of the feature point OF in the current image are

, the coordinates of the same feature point OF in the previous image are

, the displacement between them is set as

.

進行限制。懲罰值

例如是按照下式（1）進行計算。

………………………………………………..（1）   Unreasonable displacement with penalty value

limit. penalty value

For example, it is calculated according to the following formula (1).

………………………………………………..(1)

）。請參照第8圖，其示例說明時間限制TC、懲罰值

的計算及判定方法。圓的中心代表在前一張影像中特徵點OF的座標

，圓的面積代表在目前影像中特徵點OF可接受的位移。如果在目前影像中，特徵點OF的預測座標

在圓內（位移d'≤m），則懲罰值

為零。如果在目前影像中特徵點OF的預測座標

在圓外（位移d"＞m），則懲罰值

為

。位移超出圓的半徑(即m)越多，在訓練過程中將會得到較大的懲罰值

和較大的損失值，以此限制特徵點OF的座標在合理範圍內。where m is the average displacement calculated by all training data for each feature point OF, s is the standard deviation of the displacement, and d is the displacement of the same feature point OF between the previous image and the current image. When d≤m, the displacement falls within the acceptable range without penalty (ie

). Please refer to Figure 8 which illustrates the time limit TC, penalty value

calculation and determination method. The center of the circle represents the coordinates of the feature point OF in the previous image

, the area of the circle represents the acceptable displacement of the feature point OF in the current image. If in the current image, the predicted coordinates of the feature point OF

In the circle (displacement d'≤m), then the penalty value

zero. If the predicted coordinates of the feature point OF in the current image

Outside the circle (displacement d">m), the penalty value

for

. The more the displacement exceeds the radius of the circle (i.e. m), the larger the penalty value will be during the training process

and a larger loss value to limit the coordinates of the feature point OF within a reasonable range.

之特徵點OF。但在目前影像中，從反光成像中辨識出具有座標

之特徵點OF，座標

與座標

之間的位移大於時間限制TC所設定的範圍，故可以判定座標

不正確。Please refer to Fig. 9, which illustrates the case of incorrect displacement due to lack of time limit TC. The left side of Figure 9 is the previous image, and the right side is the current image. In the previous image, it is recognized that there are coordinates

The feature point OF. However, in the current image, it is recognized from the reflection image that there are coordinates

The feature point OF, the coordinates

with coordinates

The displacement between them is greater than the range set by the time limit TC, so the coordinates can be determined.

Incorrect.

As shown in FIG. 2B, in the tracking stage ST2, the movable camera 110 captures a series of images IM. Several environmental feature points EF are extracted from these images, which are then used to calculate the corresponding camera matrix CM and six-degree-of-freedom orientation CD of the movable camera 110 . At the same time, the coordinates of the feature point OF of the movable object 900 are also calculated by the neural network inference model MD, and converted and corrected by the camera matrix CM to obtain the six-degree-of-freedom orientation OD of the movable object 900 .

Please refer to FIG. 10, which illustrates a system 200 and method for simultaneously tracking the 6DOF orientation of a movable object 900 (marked in FIG. 1A ) and a movable camera 110 while adding an incremental learning stage ST3, Including: the automatic amplification unit 260 and the weight adjustment unit 270, the implementations of which are, for example, a circuit, a chip, a circuit board, a code, or a storage device for storing the code.

In the embodiment of FIG. 10, the neural network inference model MD is in the training phase, and its training data is composed of manual labeling and automatic expansion; and in the incremental learning phase, its training data is composed of automatic labeling and automatic expansion.

While tracking the movable object 900, the neural network inference model MD performs incremental learning in the background. The training data for incremental learning includes: the image IM captured by the movable camera 110 and the image IM' automatically augmented by the automatic augmentation unit 260 according to the image IM. The automatic augmentation unit 260 replaces the manual mark with the coordinates of the corrected feature point OF corresponding to the images IM and IM' as the true value of the feature point coordinates. The weight adjustment unit 270 adjusts the weights in the neural network inference model MD to update the neural network inference model MD', thereby adapting to the usage situation to accurately track the six-degree-of-freedom orientation OD of the movable object 900 .

In addition, please refer to FIG. 11, which shows a system 300 and a method for simultaneously tracking the 6-DOF orientation of the movable object 900 and the movable camera 110 when applied to MR glasses, including an orientation correction unit 310, an orientation stabilization unit 320, The viewing axis calculation unit 330 , the screen orientation calculation unit 340 and the stereoscopic image generation unit 350 are implemented by, for example, a circuit, a chip, a circuit board, a code, or a storage device for storing the code. The orientation correction unit 310 includes a cross-comparison unit 311 and a correction unit 312, which are implemented by, for example, a circuit, a chip, a circuit board, a code, or a storage device for storing the code. The stereoscopic image generating unit 350 includes an image generating unit 351 and an imaging unit 352, which are implemented by, for example, a circuit, a chip, a circuit board, a code, or a storage device for storing the code.

As the movable camera 110 and the movable object 900 move, their six-degree-of-freedom orientations CD and OD need to be cross-compared and corrected (as shown in FIG. 8 ). The cross-comparison unit 311 of the orientation correction unit 310 is used for cross-comparison of the six-degree-of-freedom orientation OD of the movable object 900 and the six-degree-of-freedom orientation CD of the movable camera 110 . The correction unit 312 is used for correcting the 6-DOF orientation OD of the movable object 900 and the 6-DOF orientation CD of the movable camera 110 .

In order to reduce the unintentional slight shaking of the head, the 6-DOF orientations of the movable camera and the movable object are recalculated, causing the virtual screen D2 (shown in Figure 1A) to sway and cause dizziness. The orientation stabilization unit 320 is used to determine that when the change of the six-degree-of-freedom orientation OD of the movable object 900 or the six-degree-of-freedom orientation CD of the movable camera 110 is less than a preset value, the six-degree-of-freedom orientation OD of the movable object 900 and the six-degree-of-freedom orientation OD of the movable object 900 are not changed. A six-degree-of-freedom orientation CD of the movable camera 110 .

The viewing axis calculating unit 330 is used for calculating the viewing axis of the user's eyes according to the six-degree-of-freedom orientation CD of the movable camera 110 .

The screen orientation calculation unit 340 is used for calculating the six-degree-of-freedom orientation DD of the virtual screen D2 according to the six-degree-of-freedom orientation OD of the movable object 900 and the six-degree-of-freedom orientation CD of the movable camera 110 , so that the virtual screen D2 follows the movable object 900 move together (as shown in FIG. 1B ), or change the displayed viewing angle of the virtual screen D2 with the 6-DOF orientation of the movable camera 110 .

The image generating unit 351 of the stereoscopic image generating unit 350 is used for generating the left-eye image and the right-eye image of the virtual screen D2 according to the 6-DOF orientation DD of the virtual screen D2 and the optical parameters of the stereoscopic display (for example, the MR glasses G1 in FIG. 1A ). image. The imaging unit 352 of the stereoscopic image generating unit 350 is used for displaying the stereoscopic image of the virtual screen D2 on the stereoscopic display (eg, the MR glasses G1 in FIG. 1A ).

The imaging unit 352 of the stereoscopic image generating unit 350 can display the virtual screen D2 at a specific position around the movable object 900 according to user settings.

To sum up, although the present disclosure has been disclosed above with embodiments, it is not intended to limit the present disclosure. Those with ordinary knowledge in the technical field to which the present disclosure pertains can make various changes and modifications without departing from the spirit and scope of the present disclosure. Therefore, the scope of protection of the present disclosure should be determined by the scope of the appended patent application.

,

,

,

,

,

,

,

,

,

,:座標

:位移

:懲罰值 PL:最佳平面 C:中心點 N:法向量100, 200, 300: a system for simultaneously tracking the 6-DOF orientation of a movable object and a movable camera 110: movable camera 120: movable camera 6-DOF orientation calculation unit 121: environmental feature extraction unit 122: camera matrix calculation Unit 123: camera orientation calculation unit 130: movable object six degrees of freedom orientation calculation unit 131: object feature coordinate calculation unit 132: object feature coordinate correction unit 133: object orientation calculation unit 140: training data generation unit 150: neural network training unit 260: automatic augmentation unit 270: weight adjustment unit 310: orientation correction unit 311: cross comparison unit 312: correction unit 320: orientation stabilization unit 330: boresight calculation unit 340: screen orientation calculation unit 350: stereoscopic image generation unit 351: Image generating unit 352: Imaging unit 900: Movable object CD: Six degrees of freedom orientation of movable camera CM: Camera matrix d, d', d": Displacement D1, D4: Physical screen D2, D3: Virtual screen DD : 6-DOF orientation of virtual screen EF: Environmental feature point ET: Feature extractor FL: Feature point coordinate prediction layer FV: Feature vector G1: MR glasses GC: Geometric constraint GCL: Geometric constraint layer IM, IM': Image LV: Loss value MD: Neural network inference model m: Displacement mean OD: Six degrees of freedom orientation of movable object OF, OF', OF*, OF**: Feature point OLV: Total loss value P1: Mobile phone s: Displacement standard Difference ST1: Training Phase ST2: Tracking Phase ST3: Incremental Learning Phase TC: Time Constrained TCL: Time Constrained Layer

,

,

,

,

,

,

,

,

,

,:coordinate

: displacement

: Penalty value PL: Best plane C: Center point N: Normal vector

FIGS. 1A and 1B illustrate the application of the technology of simultaneously tracking a movable object and a movable camera of the present disclosure compared with the prior art. FIG. 2A illustrates a system and method for simultaneously tracking the 6DOF orientation of a movable object and a movable camera, according to one embodiment. Figure 2B illustrates a system and method for simultaneously tracking the 6DOF orientation of a movable object and a movable camera while adding a training phase. FIG. 3A shows the corresponding relationship between the environmental feature points and the movable object feature points in a series of images captured by the movable camera. Figure 3B illustrates the position and orientation of objects in space. 4A to 4B illustrate the feature points of the repaired movable object. Figures 5A to 5D illustrate the definition of feature points and various training data for a mobile phone as an example. Figure 6 shows the structure of the neural network in the training phase. FIG. 7 shows the calculation method of the displacement of feature points between two adjacent images. FIG. 8 shows the calculation and determination method of the time limit. Figure 9 shows the case of incorrect displacement due to lack of time constraints. Figure 10 illustrates a system and method for simultaneously tracking the 6DOF orientation of a movable object and a movable camera while adding incremental learning. FIG. 11 illustrates a system and method for simultaneously tracking the 6-DOF orientation of a movable object and a movable camera when applied to MR glasses.

100: A 6DOF Orientation System for Simultaneous Tracking of Movable Objects and Movable Cameras

110: removable camera

120: A six-degree-of-freedom orientation calculation unit for a movable camera

121: Environmental feature extraction unit

122: Camera matrix calculation unit

123: Camera orientation calculation unit

130: Six-degree-of-freedom orientation calculation unit for movable objects

131: Object feature coordinate calculation unit

132: Object feature coordinate correction unit

133: Object orientation calculation unit

CD: Six Degrees of Freedom Orientation for Movable Cameras

CM: Camera Matrix

EF: Environmental Feature Point

GC: geometric constraints

IM: Video

MD: Neural Network Inference Models

OD: six degrees of freedom orientation of movable objects

OF,OF': feature points of movable objects

ST1: Training Phase

ST2: Tracking Phase

TC: time limit

Claims (20)

A method of simultaneously tracking a plurality of 6 DoF poses of a movable object and a movable camera, including: Capture a plurality of images with the movable camera, extract a plurality of environmental feature points from the images, match the environmental feature points to calculate a plurality of camera matrices of the movable camera, and then use the camera matrices to calculate the movable camera matrix those 6DOF orientations of the camera; and Calculate a plurality of feature points of the movable object from the images captured by the movable camera, and correct the characteristics of the movable object through the camera matrices corresponding to the images and the predefined geometric and time constraints. The coordinates of the feature points, and the corrected coordinates of the feature points and the corresponding camera matrices are used to calculate the 6-DOF orientations of the movable object. The method for simultaneously tracking the 6-DOF orientations of a movable object and a movable camera as described in claim 1, wherein the feature points of the movable object are estimated from the images captured by the movable camera as It is pre-defined that the coordinates of the feature points are calculated by comparing with the images captured by the movable camera. The method for simultaneously tracking the 6-DOF orientations of a movable object and a movable camera as described in claim 1, wherein the feature points of the movable object are estimated from the images captured by the movable camera, The coordinates of the feature points are estimated by a neural network inference model, which is pre-trained, and the training data is composed of manual marking and automatic expansion, and the geometric constraints and the time constraints are added in the training process. The method of simultaneously tracking the 6DOF orientations of a movable object and a movable camera as described in claim 3, wherein while tracking the movable object, the neural network inference model performs incremental learning in the background, the incremental learning The training data for quantitative learning includes: the images captured by the movable camera and the images automatically augmented by the images, and the coordinates of the feature points after the correction corresponding to the images are used to replace the manual markers, and the adjustment of the The weight in the neural network inference model, and the neural network inference model is updated to accurately estimate the coordinates of the feature points of the movable object. The method for simultaneously tracking the 6-DOF orientations of the movable object and the movable camera as described in claim 1, further comprising: Cross-comparing the 6DOF orientations of the movable object and the 6DOF orientations of the movable camera to correct the 6DOF orientations of the movable object and the 6DOF orientations of the movable camera degree orientation; When the 6DOF orientations of the movable object or the 6DOF orientations of the movable camera change less than a preset value, the 6DOF orientations of the movable object and the movable camera are not changed the 6DOF orientations of the camera; calculating the visual axis of the user's eyes according to the 6DOF orientations of the movable camera; Calculate the 6DOF orientations of the virtual screen based on the 6DOF orientations of the movable object and the 6DOF orientations of the movable camera; and The left-eye image and the right-eye image of the virtual screen are generated according to the 6-DOF orientations of the virtual screen and the optical parameters of the stereoscopic display, so as to display the stereoscopic image of the virtual screen on the stereoscopic display. The method for simultaneously tracking the 6-DOF orientations of a movable object and a movable camera as described in claim 5, wherein the virtual screen is set by the user to be displayed at a specific position around the movable object, the virtual screen move with the movable object. The method for simultaneously tracking the six-degree-of-freedom orientations of a movable object and a movable camera as described in claim 1, wherein the step of correcting the coordinates of the feature points of the movable object comprises: Using the camera matrices, project the two-dimensional coordinates of the feature points to the corresponding three-dimensional coordinates; According to the geometric restriction, delete the feature points whose three-dimensional coordinate deviation is greater than a predetermined value, or supplement the coordinates of the undetected feature points with the coordinates of the adjacent feature points according to the geometric restriction; and According to the time limit, compare the coordinate changes of the feature points in the consecutive images, and then use the coordinates of the corresponding feature points in the consecutive images to correct the features whose coordinate changes are greater than a set value coordinates of the point. The method for simultaneously tracking the 6-DOF orientations of a movable object and a movable camera as described in claim 1, wherein in the step of calculating the 6-DOF orientations of the movable object, For the movable object in the plane, use the feature points to calculate the fitting plane, and the 6-DOF orientations of the movable object are defined by the center point and normal vector of the plane; For the non-planar movable object, the 6DOF orientations of the movable object are defined by the centroids of the three-dimensional coordinates of the feature points. The method for simultaneously tracking the six-degree-of-freedom orientations of a movable object and a movable camera as described in claim 1, wherein the movable object is a rigid object, and the movable camera is disposed on a head-mounted stereoscopic display, a mobile device, on a computer or robot. A system for simultaneously tracking multiple 6DOF orientations of a movable object and a movable camera, comprising: The movable camera is used to capture a plurality of images; A six-degree-of-freedom orientation calculation unit for a movable camera is used to extract a plurality of environmental feature points from the images, match the environmental feature points to calculate a plurality of camera matrices of the movable camera, and then calculate the movable camera matrix from the camera matrices. these 6DOF orientations of the mobile camera; and A six-degree-of-freedom orientation calculation unit for a movable object is used to calculate a plurality of feature points of the movable object from the images captured by the movable camera, through the camera matrices corresponding to the images, and a predefined Geometric constraints and time constraints, modify the coordinates of the feature points of the movable object, and then calculate the six-freedom of the movable object based on the corrected coordinates of the feature points and the corresponding camera matrices degree orientation. The system for simultaneously tracking the 6-DOF orientations of a movable object and a movable camera as described in claim 10, wherein the movable camera 6-DOF orientation calculation unit comprises: an environmental feature extraction unit for extracting the environmental feature points from the images; a camera matrix calculation unit for calculating the camera matrices of the movable camera by matching the environmental feature points; and The camera orientation calculation unit uses the camera matrices to calculate the six-degree-of-freedom orientations of the movable camera. The system for simultaneously tracking the 6-DOF orientations of a movable object and a movable camera as described in claim 10, wherein the movable object 6-DOF orientation calculation unit comprises: an object feature estimating unit for estimating the feature points of the movable object from the images captured by the movable camera; an object feature coordinate correction unit, configured to correct the coordinates of the feature points of the movable object through the camera matrices corresponding to the images, and the predefined geometric constraints and the time constraints; and The object orientation calculation unit calculates the six-degree-of-freedom orientations of the movable object based on the corrected coordinates of the feature points and the corresponding camera matrices. The system for simultaneously tracking the 6-DOF orientations of a movable object and a movable camera as described in claim 12, wherein the object feature estimation unit estimates the position of the movable object from the images captured by the movable camera The feature points are predefined, and the coordinates of the feature points are calculated by comparing with the images captured by the movable camera. The system for simultaneously tracking the 6-DOF orientations of a movable object and a movable camera as described in claim 12, wherein the object feature estimation unit estimates the position of the movable object from the images captured by the movable camera For these feature points, the coordinates of these feature points are calculated by a neural network inference model. The neural network inference model is pre-trained, and the training data is composed of manual marking and automatic expansion. In the training process, the geometric constraints and the time limit. A system for simultaneously tracking the 6DOF orientations of a movable object and a movable camera as described in claim 14, wherein while tracking the movable object, the neural network inference model performs incremental learning in the background, the incremental learning The training data for quantitative learning includes: the images captured by the movable camera and the images automatically augmented by the images, and the coordinates of the feature points after the correction corresponding to the images are used to replace the manual markers, and the adjustment of the The weight in the neural network inference model, and the neural network inference model is updated to accurately estimate the coordinates of the feature points of the movable object. The system for simultaneously tracking the six-degree-of-freedom orientations of a movable object and a movable camera as described in claim 10, further comprising: an orientation correction unit for cross-comparing the 6-DOF orientations of the movable object and the 6-DOF orientations of the movable camera to correct the 6-DOF orientations of the movable object and the movable the 6DOF orientations of the camera; The azimuth stabilization unit does not change the 6-DOF azimuths of the movable object when the change of the 6-DOF azimuths of the movable object or the 6-DOF azimuths of the movable camera is less than a preset value the 6DOF orientations with the movable camera; a viewing axis calculating unit for calculating the viewing axis of the user's eyes according to the six-degree-of-freedom orientations of the movable camera; a screen orientation calculation unit for calculating a plurality of 6-DOF orientations of the virtual screen according to the 6-DOF orientations of the movable object and the 6-DOF orientations of the movable camera; and The stereoscopic image generating unit is used for generating the left-eye image and the right-eye image of the virtual screen according to the 6-DOF orientations of the virtual screen and the optical parameters of the stereoscopic display, so as to display the stereoscopic image of the virtual screen on the stereoscopic display. The system for simultaneously tracking the 6DOF orientations of a movable object and a movable camera as described in claim 16, wherein the virtual screen is set by the user to be displayed at a specific position around the movable object, the virtual screen move with the movable object. The system for simultaneously tracking the six-degree-of-freedom orientations of a movable object and a movable camera as described in claim 12, wherein the object feature coordinate correction unit Using the camera matrices, project the two-dimensional coordinates of the feature points to the corresponding three-dimensional coordinates; and According to the geometric restriction, delete the feature points whose three-dimensional coordinate deviation is greater than a predetermined value, or supplement the coordinates of the undetected feature points with the coordinates of the adjacent feature points according to the geometric restriction; and According to the time limit, compare the coordinate changes of the feature points in the consecutive images, and then use the coordinates of the corresponding feature points in the consecutive images to correct the features whose coordinate changes are greater than a set value coordinates of the point. The system for simultaneously tracking the 6DOF orientations of a movable object and a movable camera as described in claim 12, wherein the object orientation calculation unit For the movable object in the plane, use the feature points to calculate the fitting plane, and the 6-DOF orientations of the movable object are defined by the center point and normal vector of the plane; For the non-planar movable object, the 6DOF orientations of the movable object are defined by the centroids of the three-dimensional coordinates of the feature points. The system for simultaneously tracking the six-degree-of-freedom orientations of a movable object and a movable camera as described in claim 10, wherein the movable object is a rigid object, and the movable camera is disposed on a head-mounted stereoscopic display, a mobile device, on a computer or robot.

Images