TWI730482B - Plane dynamic detection system and detection method - Google Patents

Abstract

The present invention discloses a plane motion detection system and detection method. An inertial sensor can continuously obtain inertial data, and a depth camera can continuously obtain a depth image of a physical object (such as a plane or ground) within a viewing range. , And an arithmetic device is configured to continuously determine whether the acceleration and angular velocity obtained by the inertial sensor exceed a threshold to determine the motion state of the inertial sensor itself or the device it is equipped with. The arithmetic device can be based on acceleration, Depth image coordinates, depth values and internal parameter matrix. When the inertial sensor is initialized or continuously updated, a plane equation of the physical object in the camera coordinate system can also be obtained through the VIO algorithm to obtain the position and attitude information of the depth camera. Continuously modify the plane equation of the inertial sensor when it moves fast.

Description

Plane dynamic detection system and detection method

The present invention relates to computer vision technology, in particular to a "plane motion detection system and detection method" that can refer to depth images, color images, and inertial data to accurately detect a plane and dynamically update the relative position of the plane in three-dimensional space .

In order to provide more realistic interactive effects in applications that require 3D information (such as AR/VR services), it is very important to detect the plane in reality. If the target is to detect a plane that belongs to the ground, the method of detecting the ground can be It is: (a) Assume that the ground is the largest plane and use the RANSAC (Random Sample Consensus) algorithm, or use the Hough Transform (Hough Transform) algorithm to find the largest plane in the three-dimensional space, and define it as the ground (B) Assuming that the Z value of the ground on each scan line in the image is the largest, and after correcting the camera posture (roll rotation), the pixel with the largest Z value in the image and conforming to the fit curve C (fit curve C) is used Set, define it as the ground.

However, in many cases, the maximum plane assumed by the aforementioned method (a) is often not the ground (for example, the maximum plane in the image may be the wall of the corridor), and errors in the judgment of the RANSAC or Hough Transform algorithm may occur, and, The RANSAC algorithm has the limitation that correct data (inliers) need to account for at least 50%, and the Hough Transform algorithm is also quite time-consuming; the aforementioned method (b) may also occur in the image with the largest Z value and the pixel set that conforms to the C curve, which is not It's the situation on the ground.

，其中，

為點雲座標系中的三維座標，

為深度值，

為相機投影反矩陣，而

通常為一內部參數（內部參數為深度感測器的固有性質參數，主要有關於相機座標與影像座標間的轉換關係），

為深度影像於中每個像素的影像座標(其處於影像座標系)；其後，再令此些三維座標的特徵點集合以點雲的型態呈現，接著，再以前述(a)或(b)等方法偵測點雲影像中的平面，但前述對每個像素均作矩陣乘法的方式，計算量相當龐大而有不佳的計算效能。 Furthermore, no matter what method is used to detect the plane in the image, after the depth sensor (such as a depth camera) captures the depth image, according to the conventional practice of the Point Cloud Library (PCL), it is necessary to Each pixel (pixel) obtained by the depth sensor is sequentially multiplied with an inverse camera matrix and a depth value to be converted into multiple three dimensions in the point cloud coordinate system Coordinates, as shown in this relationship:

,among them,

Is the three-dimensional coordinate in the point cloud coordinate system,

Is the depth value,

For the camera projection inverse matrix, and

Usually an internal parameter (the internal parameter is the inherent property parameter of the depth sensor, mainly related to the conversion relationship between the camera coordinate and the image coordinate),

Is the image coordinate of each pixel in the depth image (which is in the image coordinate system); then, the feature point set of these three-dimensional coordinates is presented in the form of a point cloud, and then the above (a) or ( b) Other methods to detect the plane in the point cloud image, but the aforementioned method of matrix multiplication for each pixel requires a huge amount of calculation and poor calculation performance.

In summary, the conventional method of detecting planes in three-dimensional space requires strong assumptions for different plane types (such as ground, wall, etc.), which may lead to misjudgment of plane types and poor calculation performance. Based on this, how to propose a "plane detection system and detection method" that can detect planes more accurately and save computing resources is a problem to be solved.

To achieve the above objective, the present invention proposes a planar motion detection system, which includes: an inertial sensor, a depth camera, and an arithmetic device, wherein the inertial sensor includes an accelerometer and a gyroscope; the depth camera is sustainable Capture a depth image to continuously input a depth image coordinate and a depth value of one or more physical objects within a viewing range of the depth camera; the computing device is respectively coupled to the inertial sensor and the depth camera, and the computing device It has a motion state determination unit and a plane detection unit. The motion state determination unit is used to continuously determine whether an acceleration information and an angular velocity information obtained by the inertial sensor exceed a threshold value, and if the threshold value is not exceeded, plane detection The unit can calculate a normal vector and a distance constant based on acceleration information, depth image coordinates, depth values and an internal parameter matrix, and use the normal vector and distance constant to initialize or continuously update the physical object when the inertial sensor is in a stable state A plane equation in a camera coordinate system; otherwise, if the threshold is exceeded, the plane detection unit can execute a visual inertial odometry algorithm based on a gravitational acceleration of the acceleration information to obtain the position information of the depth camera , And based on a rotation matrix and a displacement information of the pose information, continue to modify the plane equation of the inertial sensor during rapid movement. The meaning of the plane equation is that any point on a plane and the perpendicular to the plane The normal can uniquely define the plane in the three-dimensional space.

To achieve the above objective, the present invention also provides a planar motion detection method, including: (1) A step of detecting inertial data: an inertial sensor continuously obtains inertial data such as acceleration information and angular velocity information; (2) A step of judging the movement state: an arithmetic device continuously judges whether an acceleration information and an angular velocity information obtained by the inertial sensor exceed a threshold to judge the movement state of the inertial sensor; (3) A first step of updating the plane equation: if the threshold is not exceeded, the computing device calculates a normal vector and a distance constant based on the acceleration information, depth image coordinates, depth value and an internal parameter matrix, and uses the normal vector and distance constant , To initialize or continuously update a plane equation of the physical object in a camera coordinate system when the inertial sensor is in a stable state; and (4) A second step of updating the plane equation: if the threshold is exceeded, the computing device executes a visual inertial odometry algorithm based on a gravitational acceleration of the acceleration information to obtain the position information of the depth camera based on the position and position A rotation matrix and a displacement information of the information continuously modify the plane equation of the inertial sensor when it moves quickly.

In order for your reviewer to have a clear understanding of the purpose, technical features and effects of the present invention after implementation, the following descriptions and illustrations are used for illustration, please refer to it.

Please refer to "Figure 1", which is a system architecture diagram of the present invention. The present invention proposes a planar motion detection system 1, which mainly includes an inertial sensor 10, a depth camera 20, and a computing device 30, in which: (1) The Inertial Measurement Unit (IMU) 10 includes an accelerometer (G-Seosor) 101 and a gyroscope (Gyroscope) 102, which can continuously obtain acceleration information and angular velocity information; (2) The depth camera 20 can continuously capture a depth image, so as to continuously input a depth image coordinate and a depth value of the depth camera 20 for one or more physical objects within a viewing range, and the depth camera 20 can be It is configured as a depth sensor that uses a Time of Flight (TOF), a Structured Light (Structured Light) or a Stereo Visual (Stereo Visual) scheme to measure the depth of the aforementioned physical objects, Among them, the time-of-flight method means that the depth camera 20 can be used as a ToF camera, and uses a light-emitting diode (LED) or a laser diode (LD) to emit infrared light when it is irradiated to a physical object. After the light on the surface is reflected back, since the speed of light is known, an infrared light image sensor can be used to measure the time for the physical object to reflect the light back at different depths, and then the physical object can be calculated. The depth images of different positions and physical objects; the structured light scheme means that the depth camera 20 can use a laser diode (LD) or a digital light source processor (DLP) to produce different light patterns and pass through a specific grating It is diffracted on the surface of the physical object to form a pattern of light spots. Since the light spot pattern reflected by the physical object at different depths will be distorted, when the reflected light enters the infrared image sensor Then, the three-dimensional structure of the physical object and its depth image can be reversed; the binocular vision solution means that the depth camera 20 can be used as a stereo camera, and at least two camera lenses are used to photograph the physical object and the depth camera 20. The generated disparity is used to measure the three-dimensional information (depth image) of the physical object through the principle of triangulation; (3) The computing device 30 is respectively coupled to the inertial sensor 10 and the depth camera 20, and has a motion state determination unit 301 and a plane detection unit 302, and the motion state determination unit 301 and the plane detection unit 302 are in communication connection. The motion state determination unit 301 is configured to continuously determine whether the acceleration information and angular velocity information obtained by the inertial sensor 10 exceed a threshold, so as to determine the motion state of the inertial sensor 10 itself or the device on which it is mounted. It is worth noting that the computing device 30 may have at least one processor (not shown in the figure, such as CPU, MCU), which is used to run the computing device 30, and is equipped with logical operations, temporary storage of computing results, storage of execution instruction positions, etc. In addition, the motion state determination unit 301 and the plane detection unit 302 themselves can run on a plane dynamic device (not shown in the figure, such as a head-mounted display, and the head-mounted display can be a VR helmet, an MR helmet, etc. Head-mounted display), a host (Host), a physical server or a virtualized server (VM) computing device 30, but they are not limited to this; (4) In addition, if the current threshold is not exceeded, the plane detection unit 302 is configured to be based on acceleration information, depth image coordinates (Pixel Domain), depth value (depth value) and an internal parameter matrix (intrinsic parameter matrix) Calculate a normal vector (normal vector) and a distance constant (d value), and use the normal vector and the distance constant (its position is in the image coordinate system) to initialize or continuously update the inertial sensor 10 in a stable state, the physical object A 3D plane equation in a camera coordinate system. The meaning of the plane equation is that any point on a plane and the normal line perpendicular to the plane can uniquely define the three-dimensional This plane in space; (5) Conversely, if the current threshold is exceeded, the plane detection unit 302 is configured to perform a filter-based or optimization-based vision based on a gravitational acceleration in the acceleration information. The visual inertial odometry (VIO) algorithm is used to obtain the position information of the depth camera 20, and based on a rotation matrix (orientation matrix) and a displacement information (translation) of the position and attitude information, the inertial sensibility is continuously corrected The plane equation of the detector 10 when it is moving fast; (6) In addition, the aforementioned image coordinates are introduced to describe the projection and transmission relationship of the physical object from the camera coordinate system to the image coordinate system during the imaging process. It is where the image we actually read from the depth camera 20 is located. The coordinate system, the unit is pixel, and the aforementioned camera coordinate is the coordinate system established with the depth camera 20 as the origin, and is defined for describing the position of the object from the perspective of the depth camera 20.

Please continue to refer to "Figure 1". In a preferred embodiment of the present invention, the plane detection unit 302 of the computing device 30 can also perform an inner product operation on the depth image coordinates and depth values of the physical object to continuously generate the physical object. The object is in a three-dimensional coordinate system of an image coordinate system, and a plane equation is calculated using the aforementioned three-dimensional coordinate and internal parameter matrix.

Please continue to refer to "Figure 1". In a preferred embodiment of the present invention, the plane detection unit 302 of the computing device 30 can also perform an iterative optimization algorithm or an iterative optimization algorithm on the aforementioned normal vector. The Gauss Newton algorithm obtains an optimal normal vector and its corresponding distance constant (d value), and replaces the aforementioned normal vector with the optimal normal vector to calculate a more accurate plane equation.

Please refer to "Figure 2" to "Figure 3", which are the flow diagrams (1) and (2) of the planar motion detection method of the present invention. Please also refer to "Figure 1". The present invention proposes a planar motion detection method. The motion detection method S may include the following steps: (1) Image capturing step (Step S10): A depth camera 20 continuously captures a depth image, so as to continuously input the depth camera 20 for one or more images within a viewing range. A depth image coordinate and a depth value of a physical object; (2) Step of detecting inertial data (step S20): An inertial sensor 10 continuously obtains inertial data such as acceleration information and angular velocity information; (3) Judging movement State step (step S30): an arithmetic device 30 continues to determine whether the acceleration information and angular velocity information obtained by the inertial sensor 10 exceed a threshold, to determine the motion state of the inertial sensor 10 itself or the device on which it is mounted; (4) The first step of updating the plane equation (step S40): following step S30, if the threshold is not exceeded, the computing device 30 can calculate a normal vector and a distance based on acceleration information, depth image coordinates, depth values and an internal parameter matrix Constants (which correspond to the image coordinate system), and use normal vectors and distance constants to initialize or continuously update a plane equation of the physical object in a camera coordinate system when the inertial sensor 10 is in a stable state; (5) Second Update the plane equation step (step S50): Step S30, if the threshold is exceeded, the computing device 30 executes a visual inertial odometry algorithm according to a gravitational acceleration of the acceleration information to obtain the position information of the depth camera 20, And based on a rotation matrix of the pose information and a displacement information, the plane equation of the inertial sensor 10 when it moves quickly is continuously corrected.

(3)依此，深度影像中的實體物件(地面)於影像座標下的法向量

可表示為：

Continuing, please continue to refer to "Picture 2" to "Picture 3", and please refer to "Picture 1" for collocation. When step S40 is executed, if the type of plane to be detected is the ground as an example, and the inertial sensing The inertial data of the sensor 10 does not exceed the threshold, that is, when the inertial sensor 10 itself or its carrying device is in a stable state (for example, at rest), the inertial sensor 10 will only read the static acceleration value g (gravity force direction ), and the opposite direction is the normal vector n of the plane equation of the physical object to the camera coordinates. The relational expression can be referred to as follows: (1) The static acceleration value of the inertial sensor 10: g=9.8m/s ² or 10m/s ² (2) The normal vector of the plane equation to the camera coordinates n=-g=

(3) According to this, the normal vector of the physical object (ground) in the depth image under the image coordinates

Can be expressed as:

，並假設更新前的平面方程式為

，則之後的平面方程式得依以下關係式更新，但以下僅為舉例，並不以此為限：

Continuing from above, please continue to refer to "Picture 2" to "Picture 3", and please refer to "Picture 1" in conjunction. When step S50 is executed, if the type of plane to be detected is the ground as an example, because it is used to be sexy The measuring device 10 is in a situation of violent or fast movement, and it is no longer possible to estimate the normal vector of the plane equation based on the reading of the accelerometer 101. Therefore, the aforementioned step S50 can be performed by using, for example, a filtering or optimization-based VIO algorithm. Update the plane equation of the physical object (ground), assuming that the relative pose (Relative Pose Motion) of the depth camera 20 estimated by VIO is

, And assume that the plane equation before the update is

, The subsequent plane equations have to be updated according to the following relational expressions, but the following are only examples and not limited to this:

及其對應的距離常數(

值)，並以最佳法向量

取代法向量

而演算出平面方程式，更具體而言，運算裝置30的平面偵測單元302演算最佳法向量

的公式可參照如下，但以下僅為舉例，並不以此為限： (1)首先，將深度影像中的深度值超過一定數值

的像素予以排除，再以前述提及的法向量

(此處暫稱法向量

，其對應於影像座標系)排除後的n個深度影像座標，算出對應的

值，如下關係式所示：

(2)接著，假設實體物件(地面)的

值，是在所有深度影像中法向量為

之實體物件(其它平面)中最小的，因為地面應為距離深度相機20最遠的平面，所以得依以下關係式，算出距離深度相機20最遠平面之對應的

值：

(3)其後，平面偵測單元302進一步對法向量執行一疊代最佳化演算法或一高斯牛頓演算法，求得誤差函數(Error Function，亦可稱評價函數)最小的一最佳法向量

，在此之前需先定義一誤差函數E(

)及一閾值

，如下所示：

In addition, please continue to refer to "Figure 2" to "Figure 3", and please refer to "Figure 1" together. In a preferred embodiment of the present invention, if the system targets the surface type to be detected as the ground Since step S40 is executed, even if the motion state determination unit 301 determines that the inertial sensor 10 is in a stable state, the inertial sensor 10 itself or the device it carries may not be completely static. In addition, there are some physical objects (the ground itself). In the inclined state, when the aforementioned step S40 is executed, the computing device 30 may further execute an iterative optimization algorithm or a Gauss Newton algorithm (for example, Gauss Newton least square) on the normal vector to obtain an optimum Normal vector

And its corresponding distance constant (

Value), and the best normal vector

Replace normal vector

The plane equation is calculated. More specifically, the plane detection unit 302 of the arithmetic device 30 calculates the best normal vector

The formula for can refer to the following, but the following is only an example, and not limited to this: (1) First, the depth value in the depth image exceeds a certain value

Pixels to be excluded, and then use the aforementioned normal vector

(Here tentatively called normal vector

, Which corresponds to the image coordinate system) excluded n depth image coordinates, and calculate the corresponding

Value, as shown in the following relation:

(2) Next, suppose that the physical object (ground)

Value, the normal vector in all depth images is

The smallest of the physical objects (other planes), because the ground should be the plane farthest from the depth camera 20, so according to the following relational formula, calculate the corresponding to the farthest plane from the depth camera 20

value:

(3) Thereafter, the plane detection unit 302 further executes an iterative optimization algorithm or a Gauss-Newton algorithm on the normal vector, and obtains an optimal error function (Error Function, also called an evaluation function). Normal vector

, Before this, it is necessary to define an error function E(

) And a threshold

,As follows:

B.假設於深度影像中的一像素點座標為(

，則：

C.第i個點於前述兩個不同座標系之三維座標的Z值相同，前述兩個三維座標於相機座標系與影像座標系的轉換關係如下：

D.所以相機座標系與影像座標系的三維影像座標，係可透過深度相機20之內部參數矩陣K相關聯，而展開上述公式可得出，第i個點於影像座標系中的深度影像座標的x、y值分別為：

E.依據平面方程式的定義，並假設實體物件所處的平面上有前述的第i個點，可知以處於相機座標系的

演算出的平面方程式為：

F.承上，相機座標系的法向量

G.依據平面方程式的定義，並假設實體物件所處的平面上有前述的第i個點，可知以處於影像座標系的

演算出的平面方程式為：

H.承上，影像座標系的法向量

I.接著，演算處於相機座標系中實體物件(平面)的法向量，假設

兩個點都在該平面上的話，會符合前述第G點的平面方程式，代入後的平面方程式分別如下：

J.將上述兩個平面方程式相減後，可得出：

K.接著，將前述第D點的第i個點於影像座標系中的深度影像座標的x、y值代入第J點的方程式可得出：

L.所以，實體物件對應到相機座標系的平面方程式的法向量為：

Please continue to refer to "Figure 2" to "Figure 3", and also refer to "Figure 1". If the type of plane to be detected is the ground as an example, the plane detection unit 302 of the computing device 30 calculates the aforementioned The calculation formula of the normal vector can be referred to as follows, but it is not limited to this, and it is first stated: A. Assume that there are N pixels belonging to the ground part in the depth image;

B. Assume that the coordinate of a pixel in the depth image is (

,then:

C. The Z value of the ith point in the three-dimensional coordinates of the aforementioned two different coordinate systems is the same, and the conversion relationship between the aforementioned two three-dimensional coordinates in the camera coordinate system and the image coordinate system is as follows:

D. Therefore, the camera coordinate system and the three-dimensional image coordinate of the image coordinate system can be related through the internal parameter matrix K of the depth camera 20, and the above formula can be expanded to obtain the depth image coordinate of the i-th point in the image coordinate system The x and y values are:

E. According to the definition of the plane equation, and assuming that there is the aforementioned i-th point on the plane where the physical object is located, it can be known that it is in the camera coordinate system

The calculated plane equation is:

F. Continuing, the normal vector of the camera coordinate system

G. According to the definition of the plane equation, and assuming that there is the aforementioned i-th point on the plane where the physical object is located, it can be known that it is in the image coordinate system

The calculated plane equation is:

H. Continuing, the normal vector of the image coordinate system

I. Next, calculate the normal vector of the physical object (plane) in the camera coordinate system, assuming

If both points are on the plane, they will conform to the plane equation of the G-th point mentioned above. The substituted plane equations are as follows:

J. After subtracting the above two plane equations, we can get:

K. Then, substituting the x and y values of the depth image coordinates of the ith point of the aforementioned D-th point in the image coordinate system into the equation for the J-th point, we can get:

L. Therefore, the normal vector of the plane equation corresponding to the physical object to the camera coordinate system is:

N.將處於影像座標系的像素點

代入前述第G點的平面方程式可得出：

O.將前述第D點「第i個點於影像座標系中的深度影像座標的x、y值」代入前述第N點的平面方程式可得出：

P.對前述第O點之平面方程式的等式兩側均除以c：

Q.於此，可得出實體物件於相機座標中的平面方程式的d值為：

Continuing, please continue to refer to "Figure 2" to "Figure 3", and please refer to "Figure 1" together, when the computing device 30 calculates the normal vector n of the plane equation corresponding to the physical object to the camera coordinate system, The calculation formula for the subsequent calculation of the value of d can be referred to as follows, but it is not limited to this, and it is first stated: M. First, let a constant

N. Pixels that will be in the image coordinate system

Substituting into the plane equation of the aforementioned G-th point, we can get:

O. Substituting the aforementioned D-th point "the x and y values of the depth image coordinates of the ith point in the image coordinate system" into the plane equation of the aforementioned Nth point, we can get:

P. Divide both sides of the equation of the plane equation at point O by c:

Q. Here, the d value of the plane equation of the physical object in the camera coordinates can be obtained:

，其中，

為處於影像座標系的三維座標，Z為深度值，

則為深度影像座標(處於影像座標系)，藉此，相較於習知點雲庫(PCL)皆需將深度相機20所取得的每個像素，先後與一相機投影反矩陣(即前述的K)及一深度值作矩陣乘法運算，以轉換成點雲座標系中的多個三維座標的作法，本實施例可省去像素、深度值與相機投影反矩陣作矩陣運算的步驟，而直接以前述的三維座標進行實體物件(平面)的偵測，而能達成節省運算量的有益功效，同時能省去從深度影像轉換至點雲的轉換時間。 In addition, please continue to refer to "Figure 2" to "Figure 3", and please refer to "Figure 1" in conjunction. In a preferred embodiment of the present invention, before the aforementioned step S30 is executed, a first execution can be obtained. Three-dimensional coordinate step (step S25): the computing device 30 performs an inner product operation on the depth image coordinates and the depth value of the physical object to continuously generate a three-dimensional coordinate of the physical object in an image coordinate system. Accordingly, it can be performed in step S40 or When step S50 is executed, the aforementioned normal vector and distance constant are calculated using the aforementioned three-dimensional coordinates, internal parameter matrix and acceleration information, and then the plane equation of the physical object is calculated. More specifically, the calculation formula for generating the aforementioned three-dimensional coordinates can be referred to :

,among them,

Is the three-dimensional coordinate in the image coordinate system, Z is the depth value,

It is the depth image coordinates (in the image coordinate system). Therefore, compared with the conventional point cloud library (PCL), each pixel obtained by the depth camera 20 is sequentially inversely projected by a camera (that is, the aforementioned K) and a depth value is subjected to a matrix multiplication operation to convert it into multiple three-dimensional coordinates in the point cloud coordinate system. This embodiment can omit the steps of matrix operation for pixels, depth values and the inverse matrix of the camera projection, and directly The aforementioned three-dimensional coordinates are used to detect physical objects (planes), which can achieve the beneficial effect of saving calculations and save the conversion time from depth images to point clouds.

Please refer to "Figure 4", which is a system architecture diagram of another preferred embodiment of the present invention. This embodiment is similar to the technology disclosed in "Figure 1" to "Figure 3". The main difference lies in the The planar motion detection system 1 of the embodiment may further include a color camera 40 (such as an RGB camera), which is respectively coupled to the depth camera 20 and the computing device 30 for continuously capturing a color image of the physical object for computing When the device 30 executes step S10 (image capturing step), it can establish the correspondence between the depth image coordinates of the physical object and a color image coordinate, so as to improve the accuracy of plane detection. In addition, the color camera of this embodiment 40 can also form an RGB-D camera with the depth camera 20, as shown in this figure, and the depth camera 20 of this embodiment can be a binocular camera, but it is not limited to this.

In summary, after the present invention is implemented, it can solve the problem that when conventionally detecting planes in three-dimensional space, strong assumptions must be made for different plane types, which may cause plane misjudgment, and at the same time, it can improve the conventional plane detection method. The disadvantage of poor computing performance can achieve the beneficial effects of more accurate detection of the plane and more saving of computing resources.

The above are only preferred embodiments of the present invention, and are not intended to limit the scope of implementation of the present invention; anyone who is familiar with this technique can make equal changes and modifications without departing from the spirit and scope of the present invention. Should be covered within the scope of the patent of the present invention.

To sum up, the present invention has patent requirements such as "industrial applicability", "novelty" and "advancedness"; the applicant filed an application for a patent for invention with the Bureau in accordance with the provisions of the Patent Law.

1 Plane motion detection system 10 Inertial sensor 101 Accelerometer 102 Gyroscope 20 Depth Camera 30 Computing device 301 Motion state judgment unit 302 Plane detection unit 40 Color camera S Planar motion detection method S10 Steps to capture images S20 Steps to detect inertial data S25 Steps to obtain 3D coordinates S30 Steps for judging the movement status S40 The first step to update the plane equation S50 The second update of the plane equation step

Figure 1 is a system architecture diagram of the present invention. Figure 2 is a flow chart (1) of the plane detection method of the present invention. Figure 3 is a flowchart (2) of the plane detection method of the present invention. Figure 4 is a system architecture diagram of another preferred embodiment of the present invention.

1 Plane motion detection system 10 Inertial sensor 101 Accelerometer 102 Gyroscope 20 Depth Camera 30 Computing device 301 Motion state judgment unit 302 Plane detection unit

Claims (5)

A plane motion detection system includes: an inertial sensor including an accelerometer and a gyroscope; a depth camera for continuously capturing a depth image to continuously input the depth camera in a viewing range For a depth image coordinate and a depth value of one or more physical objects; an arithmetic device, the arithmetic device is respectively coupled to the inertial sensor and the depth camera, the arithmetic device has a motion state determination unit and a plane The detection unit, the motion state determination unit and the plane detection unit are communicatively connected, and the motion state determination unit is used to continuously determine whether an acceleration information and an angular velocity information obtained by the inertial sensor exceeds a threshold; if it does not exceed For the threshold, the plane detection unit is configured to calculate a normal vector and a distance constant based on the acceleration information, the depth image coordinates, the depth value, and an internal parameter matrix, and use the normal vector and the distance constant, Initialize or continuously update a plane equation of the physical object in a camera coordinate system when the inertial sensor is in a stable state; and if the threshold value has been exceeded, the plane detection unit is configured as a basis for the acceleration information Gravity acceleration, execute a visual inertial odometry algorithm to obtain the position information of the depth camera, and based on a rotation matrix of the position and position information and a displacement information, continuously correct the inertial sensor when it is moving quickly Of the plane equation. A plane motion detection method, comprising: an image capturing step: a depth camera continuously captures a depth image to continuously input a depth image coordinate of one or more physical objects within a viewing range of the depth camera and A depth value; A step of detecting inertial data: an inertial sensor continuously obtains an acceleration information and an angular velocity information; a step of determining a motion state: an arithmetic device continuously determines whether the acceleration information and the angular velocity information obtained by the inertial sensor exceed A threshold to determine the motion state of the inertial sensor; a first step of updating the plane equation: if the threshold is not exceeded, the computing device calculates according to the acceleration information, the depth image coordinates, the depth value and an internal parameter matrix Out a normal vector and a distance constant, and use the normal vector and the distance constant to initialize or continuously update a plane equation of the physical object in a camera coordinate system when the inertial sensor is in a stable state; and 2. The step of updating the plane equation: if the threshold is exceeded, the computing device executes a visual inertial odometry algorithm based on a gravitational acceleration of the acceleration information to obtain the position information of the depth camera, and based on the position and position A rotation matrix and a displacement information of the information continuously modify the plane equation of the inertial sensor when it is moving rapidly. For example, the planar motion detection method of item 2 of the scope of the patent application further includes a step of obtaining three-dimensional coordinates before the step of determining the motion state: the computing device performs an internal operation on the depth image coordinates and the depth value of the physical object. Product operation to continuously generate a three-dimensional coordinate of the physical object in an image coordinate system to use the three-dimensional coordinate, the internal parameter matrix and the The acceleration information calculates the normal vector and the distance constant. For example, the plane motion detection method of item 2 or item 3 of the scope of patent application, wherein, when the first step of updating the plane equation is executed, the arithmetic device also executes an iterative optimization algorithm or a Gaussian on the normal vector Newton's algorithm is to obtain an optimal normal vector, and replace the normal vector with the optimal normal vector to calculate the plane equation. For example, the planar motion detection method of item 2 or item 3 of the scope of patent application, wherein, when the image capturing step is executed, it further includes a color camera continuing a color image of the physical object to continuously input the color camera to the A color image coordinate of the physical object.

Images