TWM594152U - Planar dynamic detection system - Google Patents

Abstract

A plane dynamic detection system, an inertial sensor can continuously obtain an inertial data, a depth camera can continuously obtain a depth image of a physical object (such as a plane or the ground) within a viewing range, and a computing device is assembled The state can continuously determine whether the acceleration and angular velocity obtained by the inertial sensor exceed a threshold to determine the motion state of the inertial sensor itself or the device mounted on it. Among them, the computing device can be based on the acceleration, depth image coordinates, depth value and Internal parameter matrix, when initializing or continuously updating the inertial sensor in a stable state, a plane equation of the physical object in the camera coordinate system can also obtain the pose information of the depth camera through the VIO algorithm to continuously correct the inertial sensor in Plane equations when moving fast.

Description

Planar motion detection system

This creation involves computer vision technology, especially a "plane motion detection system" that can refer to depth images, color images, and inertial data to accurately detect the relative position of the plane and dynamically update the plane in three-dimensional space.

In order to provide more realistic interactive effects in applications that require 3D information (such as AR/VR services), it is critical to detect planes in reality. If the plane that belongs to the ground is targeted, the method of detecting the ground can be For: (a) Assuming that the ground is the largest plane and use the RANSAC (Random Sample Consensus, random sampling) algorithm, or use the Hough Transform (Hough Transform) algorithm to find the largest plane in three-dimensional space, and define it as the ground ; (B) Assuming that the ground has the largest Z value on each scan line in the image, and after correcting the camera rotation (roll rotation), the pixel with the largest Z value in the image and conforming to the C curve (fit curve C) Set, define it as the ground.

However, in many cases, the maximum plane assumed by the aforementioned (a) method is often not the ground (for example, the maximum plane in the image may be the wall surface of the corridor), and a RANSAC or Hough Transform algorithm judgment error may occur, and, The RANSAC algorithm has at least 50% of the correct data (inliers), and the Hough Transform algorithm is also quite time-consuming; the aforementioned (b) method may also produce a pixel set with the largest Z value in the image and conform to the C curve, which is not This is the situation on the ground.

，其中，

為點雲座標系中的三維座標，

為深度值，

為相機投影反矩陣，而

通常為一內部參數（內部參數為深度感測器的固有性質參數，主要有關於相機座標與影像座標間的轉換關係），

為深度影像於中每個像素的影像座標(其處於影像座標系)；其後，再令此些三維座標的特徵點集合以點雲的型態呈現，接著，再以前述(a)或(b)等方法偵測點雲影像中的平面，但前述對每個像素均作矩陣乘法的方式，計算量相當龐大而有不佳的計算效能。 Furthermore, no matter what method is used to detect the plane in the image, after acquiring the depth image by the depth sensor (such as a depth camera), according to the conventional practice of the Point Cloud Library (PCL), all Each pixel obtained by the depth sensor is successively matrix multiplied with an inverse camera matrix and a depth value to convert into multiple 3D points in the point cloud coordinate system Coordinates, as shown in this relation:

,among them,

Is a three-dimensional coordinate in the point cloud coordinate system,

Is the depth value,

Project the inverse matrix for the camera, and

It is usually an internal parameter (internal parameter is the inherent property parameter of the depth sensor, mainly about the conversion relationship between the camera coordinate and the image coordinate),

Is the image coordinate of each pixel in the depth image (which is in the image coordinate system); thereafter, the feature point sets of these three-dimensional coordinates are presented in the form of a point cloud, and then, the above (a) or ( b) and other methods to detect the plane in the point cloud image, but the aforementioned matrix multiplication method for each pixel has a very large amount of calculation and has poor calculation performance.

In summary, the conventional method of detecting planes in three-dimensional space requires strong assumptions for different plane types (such as ground, wall, etc.), which may cause the problem of misjudgment of plane types, and also have poor computing performance. Therefore, how to propose a "plane detection system and detection method" that can detect planes more accurately and save computing resources is a problem to be solved.

To achieve the above purpose, the author proposes a planar motion detection system, including: an inertial sensor, a depth camera and a computing device, wherein the inertial sensor includes an accelerometer and a gyroscope; the depth camera is sustainable Capture a depth image to continuously input a depth image coordinate and a depth value for one or more physical objects within a viewing range of the depth camera; the computing device is respectively coupled to the inertial sensor and the depth camera, the computing device It has a motion state judgment unit and a plane detection unit. The motion state judgment unit is used for continuously judging whether the acceleration information and the angular velocity information obtained by the inertial sensor exceed a threshold, and if the threshold is not exceeded, the plane detection The unit can calculate a normal vector and a distance constant based on acceleration information, depth image coordinates, depth values and an internal parameter matrix, and use the normal vector and the distance constant to initialize or continuously update the inertial sensor when it is in a stable state. A plane equation in a camera coordinate system; conversely, if the threshold has been exceeded, the plane detection unit can execute a visual inertial odometry algorithm based on a gravitational acceleration of the acceleration information to obtain the position information of the depth camera , And based on a rotation matrix and a displacement information of the pose information, continuously modify the plane equation of the inertial sensor during rapid movement, and the meaning of the plane equation is that any point on a plane and the plane perpendicular to the plane The normal can uniquely define the plane in three-dimensional space.

In order to enable your reviewing committee to clearly understand the purpose, technical features and effects of this creation, the following description is accompanied by illustrations, please refer to it.

Please refer to "Figure 1", which is a system architecture diagram of this creation. This creation proposes a planar motion detection system 1, which mainly includes an inertial sensor 10, a depth camera 20, and a computing device 30, in which: (1) Inertial Measurement Unit (IMU) 10 includes an accelerometer (G-Seosor) 101 and a gyro (Gyroscope) 102, which can continuously obtain acceleration information and angular velocity information; (2) The depth camera 20 can continuously capture a depth image to continuously input a depth image coordinate and a depth value for one or more physical objects within a viewing range of the depth camera 20, and the depth camera 20 can be A depth sensor configured to measure the depth of the aforementioned physical objects using a Time of Flight (TOF) scheme, a Structured Light or a Stereo Visual scheme, Among them, the time-of-flight method means that the depth camera 20 can be used as a ToF camera, and uses the light emitting diode (LED) or laser diode (Laser Diode, LD) to emit infrared light, when the object is irradiated to the physical object After the light on the surface is reflected back, since the speed of light is known, an infrared image sensor can be used to measure the time for the physical object to reflect the light at different depths, and then the physical object can be calculated Depth images of different positions and depth of physical objects; structured light scheme means that the depth camera 20 can use laser diode (LD) or digital light source processor (DLP) to produce different light patterns and pass through specific gratings It is diffracted onto the surface of the physical object to form a spot pattern. Since the spot pattern reflected by the physical object at different depths will be distorted, when the reflected light enters the infrared image sensor Afterwards, the three-dimensional structure and depth image of the physical object can be reversed; the binocular vision scheme means that the depth camera 20 can be used as a stereo camera, and at least two camera lenses are used to shoot the physical object and the depth camera 20. The generated disparity (disparity), through the principle of triangulation (Triangulation) to measure the three-dimensional information (depth image) of the physical object; (3) The computing device 30 is respectively coupled to the inertial sensor 10 and the depth camera 20, and has a motion state judgment unit 301 and a plane detection unit 302, and the motion state judgment unit 301 and the plane detection unit 302 are communicatively connected, The motion state judging unit 301 is configured to continuously judge whether the acceleration information and the angular velocity information obtained by the inertial sensor 10 exceed a threshold to judge the motion state of the inertial sensor 10 itself or the device mounted thereon, It is worth noting that the computing device 30 may have at least one processor (not shown in the figure, such as CPU and MCU), which is used to run the computing device 30, and has logic operations, temporary storage of operation results, and storage of execution instruction positions, etc. In addition, the motion state determination unit 301 and the plane detection unit 302 can run on a plane dynamic device (not shown in the figure, such as a head-mounted display, and the head-mounted display can be a VR helmet, MR helmet, etc. Head-mounted display), a host (Host), a physical server or a virtualized server (VM) computing device 30, but not limited to this; (4) According to the above, if the current threshold is not exceeded, the plane detection unit 302 is configured to be based on acceleration information, depth image coordinates (Pixel Domain), depth value (depth value), and an intrinsic parameter matrix Calculate a normal vector (normal vector) and a distance constant (d value), and use the normal vector and distance constant (which is in the image coordinate system) to initialize or continuously update the inertial sensor 10 in a stable state, the physical object 3D plane equation in a camera coordinate system, and the meaning of the plane equation, that is, any point on a plane and the normal normal to the plane, can uniquely define the three-dimensional The plane in space; (5) Conversely, if the current threshold has been exceeded, the plane detection unit 302 is configured to perform a filter-based or optimization-based vision based on a gravitational acceleration in the acceleration information Inertial odometry (VIO) algorithm to obtain the one-position information of the depth camera 20, and based on a rotation matrix (orientation matrix) and a displacement information (translation) of the position and orientation information, continuously correct the inertia The plane equation of the detector 10 during rapid movement; (6) In addition, the aforementioned image coordinates are introduced to describe the projection transmission relationship of the physical object from the camera coordinate system to the image coordinate system during the imaging process, which is where we actually read the image from the depth camera 20. The coordinate system is in pixels, and the aforementioned camera coordinate is a coordinate system established with the depth camera 20 as the origin, and is defined to describe the position of the object from the perspective of the depth camera 20.

Please continue to refer to "Figure 1". In a preferred embodiment of the present invention, the plane detection unit 302 of the computing device 30 can also perform an inner product operation on the depth image coordinates and depth values of the physical object to continuously generate the entity The object is in a three-dimensional coordinate of an image coordinate system, and the plane equation is calculated by the aforementioned three-dimensional coordinate and internal parameter matrix.

Please continue to refer to "Figure 1". In this preferred embodiment, the plane detection unit 302 of the computing device 30 may also perform an iterative optimization algorithm or an iterative optimization algorithm on the aforementioned normal vector. The Gauss Newton algorithm finds an optimal normal vector and its corresponding distance constant (d value), and replaces the foregoing normal vector with the optimal normal vector to calculate a more accurate plane equation.

Please refer to "Picture 2" to "Picture 3", which are flow charts (1) and (2) of the plane motion detection method of this creation, and please refer to "Picture 1" for the creation. The motion detection method S may include the following steps: (1) Step of capturing an image (step S10): a depth camera 20 continuously captures a depth image to continuously input the depth camera 20 within one viewing range for one or more A depth image coordinate and a depth value of each physical object; (2) Step of detecting inertial data (step S20): an inertial sensor 10 continuously obtains inertial data such as acceleration information and angular velocity information; (3) judging motion State step (step S30): an arithmetic device 30 continuously determines whether the acceleration information and the angular velocity information obtained by the inertial sensor 10 exceed a threshold to determine the motion state of the inertial sensor 10 itself or the device mounted on it; (4) The first step of updating the plane equation (step S40): following step S30, if the threshold is not exceeded, the computing device 30 can calculate a normal vector and a distance based on the acceleration information, depth image coordinates, depth value and an internal parameter matrix Constant (which corresponds to the image coordinate system), and the normal vector and the distance constant are used to initialize or continuously update a plane equation of the physical object in a camera coordinate system when the inertial sensor 10 is in a stable state; (5) Second Step of updating the plane equation (step S50): following step S30, if the threshold has been exceeded, the computing device 30 executes a visual inertial odometer algorithm based on a gravitational acceleration of the acceleration information to obtain the position information of the depth camera 20, Based on a rotation matrix and displacement information of the pose information, the plane equations of the inertial sensor 10 during rapid movement are continuously corrected.

(3)依此，深度影像中的實體物件(地面)於影像座標下的法向量

可表示為：

Continuing on, please continue to refer to "Picture 2" to "Picture 3", and please refer to "Picture 1", when step S40 is executed, if the type of plane to be detected is ground as an example, and the inertial sensing If the inertial data of the sensor 10 does not exceed the threshold value, that is, when the inertial sensor 10 itself or its mounted device is in a stable state (for example, at rest), the inertial sensor 10 will only read the static acceleration value g (gravity force direction ), and the opposite direction is the normal equation n of the plane equation of the solid object at the camera coordinate, the relationship can refer to the following: (1) The static acceleration value of the inertial sensor 10: g=9.8m/s ² or 10m/s ² (2) The normal vector of the plane equation to the camera coordinates n=-g=

(3) According to this, the normal vector of the physical object (ground) in the depth image under the image coordinates

Can be expressed as:

，並假設更新前的平面方程式為

，則之後的平面方程式得依以下關係式更新，但以下僅為舉例，並不以此為限：

Inherit, please continue to refer to "Picture 2" to "Picture 3", and please refer to "Picture 1", when step S50 is executed, if the plane type to be detected is taken as an example, due to inertia When the detector 10 is in a violent or fast motion, the normal vector of the plane equation cannot be estimated by the reading of the accelerometer 101, so when the foregoing step S50 is executed, for example, a VIO algorithm based on filtering or optimization can be used to Update the plane equation of the solid object (ground), assuming that the relative pose (Relative Pose Motion) of the depth camera 20 estimated by the VIO is

And assume that the plane equation before the update is

, Then the following plane equations must be updated according to the following relationship, but the following is only an example, not limited to this:

及其對應的距離常數(

值)，並以最佳法向量

取代法向量

而演算出平面方程式，更具體而言，運算裝置30的平面偵測單元302演算最佳法向量

的公式可參照如下，但以下僅為舉例，並不以此為限： (1)首先，將深度影像中的深度值超過一定數值

的像素予以排除，再以前述提及的法向量

(此處暫稱法向量

，其對應於影像座標系)排除後的n個深度影像座標，算出對應的

值，如下關係式所示：

(2)接著，假設實體物件(地面)的

值，是在所有深度影像中法向量為

之實體物件(其它平面)中最小的，因為地面應為距離深度相機20最遠的平面，所以得依以下關係式，算出距離深度相機20最遠平面之對應的

值：

(3)其後，平面偵測單元302進一步對法向量執行一疊代最佳化演算法或一高斯牛頓演算法，求得誤差函數(Error Function，亦可稱評價函數)最小的一最佳法向量

，在此之前需先定義一誤差函數E(

)及一閾值

，如下所示：

In addition, please continue to refer to "Picture 2" to "Picture 3", and please refer to "Picture 1". In a preferred embodiment of this creation, if the system targets the plane type to be detected as the ground Since step S40 is executed, even if the motion state judging unit 301 judges that the inertial sensor 10 is in a stable state, the inertial sensor 10 itself or the device mounted thereon may not be completely still, and there are also some physical objects (the ground itself) The tilted condition, so when the aforementioned step S40 is executed, the computing device 30 can further execute an iteration optimization algorithm or a Gaussian Newton algorithm (eg, gauss newton least square) on the normal vector to obtain an optimal Normal vector

And its corresponding distance constant (

Value) and the best normal vector

Replace normal vector

The plane equation is calculated, more specifically, the plane detection unit 302 of the computing device 30 calculates the best normal vector

The formula of can refer to the following, but the following is only an example, not limited to this: (1) First, the depth value in the depth image exceeds a certain value

Pixels are excluded, and the normal vector mentioned above is used

(Temporarily called normal vector here

, Which corresponds to the image coordinate system) n depth image coordinates after elimination, and calculate the corresponding

Value, as shown in the following relationship:

(2) Next, assume that the physical object (ground)

The value is the normal vector in all depth images

The smallest of the solid objects (other planes), because the ground should be the plane farthest from the depth camera 20, so the corresponding relation to the plane farthest from the depth camera 20 must be calculated according to the following relationship

value:

(3) Thereafter, the plane detection unit 302 further executes an iterative optimization algorithm or a Gaussian Newton algorithm on the normal vector to obtain an optimal function with the smallest error function (error function, also called evaluation function) Normal vector

, An error function E(

) And a threshold

,As follows:

B.假設於深度影像中的一像素點座標為(

，則：

C.第i個點於前述兩個不同座標系之三維座標的Z值相同，前述兩個三維座標於相機座標系與影像座標系的轉換關係如下：

D.所以相機座標系與影像座標系的三維影像座標，係可透過深度相機20之內部參數矩陣K相關聯，而展開上述公式可得出，第i個點於影像座標系中的深度影像座標的x、y值分別為：

E.依據平面方程式的定義，並假設實體物件所處的平面上有前述的第i個點，可知以處於相機座標系的

演算出的平面方程式為：

F.承上，相機座標系的法向量

G.依據平面方程式的定義，並假設實體物件所處的平面上有前述的第i個點，可知以處於影像座標系的

演算出的平面方程式為：

H.承上，影像座標系的法向量

I.接著，演算處於相機座標系中實體物件(平面)的法向量，假設

兩個點都在該平面上的話，會符合前述第G點的平面方程式，代入後的平面方程式分別如下：

J.將上述兩個平面方程式相減後，可得出：

K.接著，將前述第D點的第i個點於影像座標系中的深度影像座標的x、y值代入第J點的方程式可得出：

L.所以，實體物件對應到相機座標系的平面方程式的法向量為：

Please continue to refer to "Picture 2" to "Picture 3", and please refer to "Picture 1", if the plane type to be detected is the ground, the plane detection unit 302 of the computing device 30 calculates the foregoing The calculation formula of the normal vector can be referred to as follows, but it is not limited to this. Chen Ming first: A. Assume that there are N pixels belonging to the ground part in the depth image;

B. Assume that the coordinate of a pixel in the depth image is (

,then:

C. The i-th point has the same Z value in the three-dimensional coordinates of the two different coordinate systems. The conversion relationship between the two three-dimensional coordinates in the camera coordinate system and the image coordinate system is as follows:

D. Therefore, the three-dimensional image coordinates of the camera coordinate system and the image coordinate system can be related through the internal parameter matrix K of the depth camera 20, and the above formula can be obtained by expanding the above formula, the depth image coordinate of the i-th point in the image coordinate system The x and y values are:

E. According to the definition of the plane equation, and assuming the i-th point on the plane where the solid object is located, it can be known that it is in the camera coordinate system

The calculated plane equation is:

F. Continued, the normal vector of the camera coordinate system

G. According to the definition of the plane equation, and assuming that the i-th point on the plane where the solid object is located, it can be known that it is in the image coordinate system

The calculated plane equation is:

H. Continued, the normal vector of the image coordinate system

I. Next, calculate the normal vector of the solid object (plane) in the camera coordinate system, assuming

If both points are on the plane, it will conform to the aforementioned plane equation at point G. The substituted plane equations are as follows:

J. After subtracting the above two plane equations, we can get:

K. Next, the x and y values of the depth image coordinates of the ith point of the aforementioned D point in the image coordinate system are substituted into the equation of the J point to obtain:

L. Therefore, the normal vector of the plane equation of the solid object corresponding to the camera coordinate system is:

N.將處於影像座標系的像素點

代入前述第G點的平面方程式可得出：

O.將前述第D點「第i個點於影像座標系中的深度影像座標的x、y值」代入前述第N點的平面方程式可得出：

P.對前述第O點之平面方程式的等式兩側均除以c：

Q.於此，可得出實體物件於相機座標中的平面方程式的d值為：

To continue, please continue to refer to "Picture 2" to "Picture 3", and please refer to "Picture 1", when the computing device 30 calculates the normal vector n of the plane equation corresponding to the camera coordinate system of the physical object, The calculation formula for successively calculating the d value can be referred to as follows, but it is not limited to this. Chen Ming first: M. First, let a constant

N. The pixels that will be in the image coordinate system

Substituting the above-mentioned plane equation at point G gives:

O. Substituting the aforementioned D point "x, y value of the depth image coordinate of the i-th point in the image coordinate system" into the aforementioned plane equation of the N-th point can be obtained:

P. Divide both sides of the equation of the aforementioned plane equation at point O by c:

Q. Here, the d value of the plane equation of the physical object in the camera coordinate can be obtained as:

，其中，

為處於影像座標系的三維座標，Z為深度值，

則為深度影像座標(處於影像座標系)，藉此，相較於習知點雲庫(PCL)皆需將深度相機20所取得的每個像素，先後與一相機投影反矩陣(即前述的K)及一深度值作矩陣乘法運算，以轉換成點雲座標系中的多個三維座標的作法，本實施例可省去像素、深度值與相機投影反矩陣作矩陣運算的步驟，而直接以前述的三維座標進行實體物件(平面)的偵測，而能達成節省運算量的有益功效，同時能省去從深度影像轉換至點雲的轉換時間。 In addition, please continue to refer to "Picture 2" to "Picture 3", and please refer to "Picture 1". In a preferred embodiment of this creation, before the aforementioned step S30 is executed, an acquisition may be performed first. Three-dimensional coordinate step (step S25): The computing device 30 performs an inner product operation on the depth image coordinates and depth value of the physical object to continuously generate a three-dimensional coordinate of the physical object in an image coordinate system. Accordingly, it can be performed in step S40 or When step S50 is executed, the aforementioned normal vector and distance constant are calculated using the aforementioned three-dimensional coordinates, internal parameter matrix and acceleration information, and then the plane equation of the physical object is calculated. More specifically, the calculation formula for generating the aforementioned three-dimensional coordinates can be referred to :

,among them,

Is the three-dimensional coordinate in the image coordinate system, Z is the depth value,

It is the depth image coordinate (in the image coordinate system). Therefore, compared to the conventional point cloud library (PCL), each pixel obtained by the depth camera 20 must be projected with a camera inverse matrix (that is, the aforementioned K) and a depth value for matrix multiplication operation to convert into multiple three-dimensional coordinates in the point cloud coordinate system. In this embodiment, the step of performing matrix operation for the pixel, depth value and camera projection inverse matrix can be omitted, and the direct The detection of physical objects (planes) with the aforementioned three-dimensional coordinates can achieve the beneficial effect of saving calculation amount, and at the same time can save the conversion time from the depth image to the point cloud.

Please refer to "Figure 4", which is a system architecture diagram of another preferred embodiment created. This embodiment is similar to the technology disclosed in "Figure 1" to "Figure 3", the main difference is that The planar motion detection system 1 of the embodiment may further include a color camera 40 (for example, an RGB camera), which is coupled to the depth camera 20 and the computing device 30, respectively, for continuously capturing a color image of the physical object for The computing device 30 can establish the correspondence between the depth image coordinate of the physical object and a color image coordinate when step S10 (capture image step) is executed to improve the accuracy of the plane detection. In addition, the color of this embodiment The camera 40 and the depth camera 20 can also form an RGB-D camera, as shown in this figure, and the depth camera 20 in this embodiment can be a binocular camera, but not limited to this.

In summary, after the implementation of this creation, the problem of misjudgment of planes that must be made as a strong assumption for different plane types when conventional detection of planes in three-dimensional space can be solved, and the conventional plane detection method can be improved. The disadvantage of poor computing performance, and can achieve a more accurate detection of planes, and save the beneficial effects of computing resources.

The above are only the preferred embodiments of this creation, and are not intended to limit the scope of the implementation of this creation; anyone who is familiar with this skill will make equal changes and modifications without departing from the spirit and scope of this creation. It should be covered by the patent scope of this creation.

In summary, this creative department has patent requirements such as "industrial utility", "novelty" and "progressiveness"; the applicant has filed an application for a new type of patent with the Bureau of Law in accordance with the provisions of the Patent Law.

1: Planar motion detection system 10: Inertial sensor 101: accelerometer 102: Gyroscope 20: depth camera 30: computing device 301: Motion state judgment unit 302: plane detection unit 40: Color camera S: Planar motion detection method S10: Steps to capture images S20: Steps to detect inertial data S25: Steps to obtain 3D coordinates S30: Steps for judging exercise status S40: The first step of updating the plane equation S50: Second step of updating the plane equation

Figure 1 is a system architecture diagram for this creation. Figure 2 is a flow chart of the plane detection method for this creation (1). Figure 3 is a flow chart of the plane detection method for this creation (2). Figure 4 is a system architecture diagram created in another preferred embodiment.

1: Planar motion detection system

10: Inertial sensor

101: accelerometer

102: Gyroscope

20: depth camera

30: computing device

301: Motion state judgment unit

302: plane detection unit

Claims (7)

A planar motion detection system, including: An inertial sensor including an accelerometer and a gyroscope; A depth camera for continuously capturing a depth image to continuously input a depth image coordinate and a depth value for one or more physical objects within a viewing range of the depth camera; A computing device has at least a processor, the computing device is coupled to the inertial sensor and the depth camera, the computing device has a motion state judgment unit and a plane detection unit, the motion state judgment unit and the The plane detection unit is in communication connection, and the motion state judgment unit is used for continuously judging whether the acceleration information and the angular velocity information obtained by the inertial sensor exceed a threshold; If the threshold is not exceeded, the plane detection unit is configured to calculate a normal vector and a distance constant based on the acceleration information, the depth image coordinates, the depth value and an internal parameter matrix, and use the normal vector and the Distance constant, a plane equation of the physical object in a camera coordinate system when the inertial sensor is initialized or continuously updated; and If the threshold has been exceeded, the plane detection unit is configured to execute a visual inertial odometry algorithm based on a gravitational acceleration of the acceleration information to obtain the pose information of the depth camera and based on the pose A rotation matrix of information and a displacement information continuously correct the plane equation of the inertial sensor during rapid movement. For example, the planar motion detection system of the first patent application, wherein the planar detection unit is also configured to perform an inner product operation on the depth image coordinates and the depth value, and continuously generate the physical object in an image A three-dimensional coordinate of the coordinate system, and the plane equation is calculated using the three-dimensional coordinate, the internal parameter matrix, and the acceleration information. For example, the planar motion detection system of patent application item 1 or item 2, in which the computing device is also used to execute an iterative optimization algorithm or a Gauss-Newton algorithm on the normal vector to obtain an optimal Normal vector, and replace the normal vector with the best normal vector to calculate the plane equation. For example, the planar motion detection system according to item 1 or 2 of the patent application scope further includes a color camera, which is coupled to the depth camera and the computing device, respectively, for continuously capturing a color image of the physical object, For the computing device to establish the correspondence between the depth image coordinate of the physical object and a color image coordinate. For example, the plane motion detection system of the first or second patent application scope, wherein the motion state judgment unit and the plane detection unit operate on a host, a physical server, a virtualized server or a headset Computing device of the digital display. For example, the planar motion detection system according to item 1 or 2 of the patent application, wherein the depth camera is configured as a depth sensor based on a time-of-flight method scheme, a structured light scheme, or a binocular vision scheme. For example, the planar motion detection system of claim 4 of the patent application, wherein the color camera is an RGB camera, and the RGB camera and the depth camera group form an RGB-D camera.

Images