KR20220052784A - Depth Image based Real-time ground detection method - Google Patents

Abstract

The present invention relates to a depth image-based real-time bottom detection method to be performed by a small amount of computation. According to the present invention, a method for detecting the bottom in real-time on the basis of a depth image detects the bottom in a region of interest (ROI) area from a time of flight (TOF) camera while the TOF camera is mounted to photograph objects located within a predetermined space. The method comprises the following steps: dividing a depth image into cells after setting a bottom search area; determining whether an image cell is flat by using an eigen value after normal distribution transformation (NDT) is applied to the image cell; extracting only cells close to a bottom criterion as candidate cells; determining whether the number of cells is greater than the minimum number of candidate cells; determining that there is no bottom plane when the number of cells is less than the minimum number of candidate cells in the determination step and performing floor modeling with cell-based RANSAC when the number of cells is greater than the minimum number of candidate cells in the determination step; and adding a distance criterion inlier except for the candidate cell.

Description

Depth Image based Real-time ground detection method

The present invention relates to a depth image-based real-time floor detection method, and more particularly, to a depth image-based real-time floor detection method that is robust against noise based on a depth image and has a small amount of computation to be executed in real time.

Object information in a mono-camera color image cannot obtain accurate distance information. This is because, in the image obtained when the light reflected from the background and object of the 3D real world collides with the camera image sensor, the dimension called depth is reduced by one, and a lot of information is lost.

Therefore, in order to obtain distance information from an image, camera calibration should be preceded. Since the scale factor cannot be obtained even with calibration to know whether a point in the real world corresponds to the x and y coordinates of the image obtained by refracting through the camera lens and colliding with the sensor, another specific condition must be satisfied. can be obtained It is information such as whether the object is attached to the ground and the ground is flat, or the actual size of the object is known.

Sensors that obtain depth information include lidar, image registration using stereo cameras, and TOF cameras. A Time Of Flight (TOF) camera has a pixel count of m x n, similar to a normal camera, and each pixel has a distance from the camera, not a color or brightness. In order to detect an object and obtain information, it will be necessary to detect and group the remaining distance data after first detecting and removing the floor area.

The data of 3D point cloud is data with values of x, y, and z when converted to the 3D real world coordinate system. need.

In addition, it is difficult to apply the algorithm because it is naturally affected by the noise of 3D point clouds.

Image-based distance detection

The exact distance cannot be known from a single monocamera image. In general, a distance can be measured using an image simultaneously photographed using a stereo camera and a geometry of the stereo camera. This method has a cut-off blind spot in registration, and the accuracy decreases as the distance increases. However, it has the advantage of being able to measure the distance only with an image. If the actual size of the object is known, it can be obtained by a simple proportional expression when there is no camera distortion.

A depth image obtained by image-based distance measurement and a depth image obtained from a TOF camera have the same meaning in principle. In both cases, the pixel has the same depth value, not color and brightness information, and there is only a difference in performance and characteristics of the technology used to obtain this value.

Depth image-based floor detection

Definition of depth image

A depth image refers to an image in which a pixel in a camera image has a depth value rather than color and brightness information. In general, the depth value is expressed like a rainbow color, so it can look like color information. In a depth image, one pixel corresponds to a distance, and it is difficult to simply calculate it using only the x and y positions and distance values of the depth image. Therefore, it is necessary to convert the distance value into coordinates in the real 3D real world. Since the TOF camera also follows the camera characteristics, it can be used to convert the distance value into a real 3D real world coordinate system.

Research on 3D point cloud

Since not all points in the 3D point cloud are components of the floor, it is necessary to detect the floor taking this into account. In addition, the 3D point cloud has a large number of points, and the real world data is divided into 3D grids rather than using each point itself, and the data in the real world is divided into 3D grids and the A method for efficiently managing the distribution of 3D points by analyzing and expressing them in voxel is being studied. In order to divide the real world space where the 3D point cloud exists, it is appropriate to apply it to a map in which multiple depth information is accumulated and matched using an algorithm such as SLAM (Simultaneous Localization and Mapping) instead of one TOF depth image.

The floor plane can be modeled using 3D point cloud and RANSAC (RANdomSAmple Consensus) algorithm. The floor plane is modeled by extracting 3 points and calculating the normal vector of the plane using the equation of the plane using 3 points. Then, by counting the consensus of other points, the modeling and consensus are repeated in the direction of updating the maximum value. In the process of checking the consensus, distance information from the plane is used, but the amount of computation is high because the number of points is large. And when modeling, there is a problem even when selecting 3 points. The information of 3d point cloud generally contains noise, and this noise is said to be unpredictable. If the floor is slightly tilted when extracting and extracting three points containing this noise, the actual inlier is often judged as an outlier and excluded when calculating consensus. Incorrect modeling due to noise causes the need for many iterations, leading to an increase in the amount of computation.

camera calibration

It is a geometric description of a point in the coordinate system of the 3D real world being transformed into u and v of the image space as it passes through the camera lens and is projected on the image sensor.

RANSAC is

It is a method of predicting model parameters from a dataset containing outliers such as noise. It is a method to obtain a model with high support by making a model by randomly sampling a part of data and counting the number of data supporting this model. When the ratio of inlier and outlier is known by repeatedly performing this operation, the probability of success of the algorithm can be known.

The advantage of RANSAC is that it is theoretically possible to obtain a parametric model consisting of inliers excluding outliers.

The disadvantage of RANSAC lies in the number of iterations and there is a possibility that the results obtained may not be optimal. The more outliers there are, the more iterations are needed. Only one model can be obtained.

NDT

Paper: Remote Sens. 2017,9,433;doi:10.3390/rs9050433 www.mdpi.com/journal/remotesensing, Article: An Improved RANSAC for 3D Cloud Plane Segmentation Based on Normal Transform Cells 3D Point Clould Plane Segmentation Based on Normal Distribution Transformation Cells))

In the above paper, the 3D point cloud map is spatially partitioned in the 3D real world and NDT is applied to determine the planar cell. Fit the planar model through optimization.

NDT is an abbreviation of Normal Distribution Transformation and can be interpreted as Normal Distribution Transformation. Covariance can be obtained by performing NDT transformation of 3D real world data. Covariance shows the distribution and correlation of data. The covariance in 2D has the shape of an ellipse, and in 3D, the covariance has the shape of an ellipsoid ( FIG. 1 is a diagram showing the covariance, eigen vector, and eigen value).

[Prior art literature]

1. Paper: Remote Sens. 2017,9,433;doi:10.3390/rs9050433 www.mdpi.com/journal/remotesensing, Article: An Improved RANSAC (An Improved for 3D Cloud Plane Segmentation Based on Normal Distribution Cells) RANSAC for 3D Point Clould Plane Segmentation Based on Normal Distribution Transformation Cells))

2. Republic of Korea Patent Publication No. 10-2015-0109868 (published on October 2, 2015) (Title of the invention: a method for processing a floor area using depth information, a processing device for the same, and a computer-readable recording medium recording a program)

It is an object of the present invention to provide a depth image-based real-time floor detection method, which has been proposed in view of the conventional situation as described above, and has a small amount of computation to be executed in real time while being robust to depth image-based noise.

According to a preferred embodiment of the present invention for achieving the above object,

A real-time floor detection method based on a depth image in an ROI area from a TOF camera image in a state where a TOF (Time Of Flight) camera is mounted so that objects located within a certain space can be photographed,

After setting the bottom search area, dividing the depth image into cells,

After applying NDT (Normal Distribution Transformation) to an image cell, determining whether it is flat using an eigen value;

Extracting only cells close to the bottom criterion as candidate cells;

Determining whether the cell (cell) is greater than the minimum number of candidate cells (cell),

If, in the determination step, the number of cells is smaller than the minimum number of candidate cells, it is determined that there is no floor plane,

If, in the determination step, the cell (cell) is greater than the minimum number of candidate cells (cell), the step of modeling the floor with a cell (cell) based RANSAC, and

There is provided a depth image-based real-time floor detection method including adding a distance-based inlier other than a candidate cell (cell).

In addition, the step of modeling the floor with the cell-based RANSAC is

3 cell (cell) random selection as the mean (mean) value of the candidate cell (cell) and floor plane modeling,

Determining an inlier cell condition after calculating the plane angle and distance with the candidate cells;

Thereafter, determining whether the consensus (support) count is greater than the maximum count (MAX COUNT);

In the determination step, if the consensus (support) count is greater than the maximum count (MAX COUNT),

Determining the best model (Best model), updating the inlier cell (inlier cell),

Determining whether the inlier ratio of the best model is 95% or more;

In the above step, if the inlier ratio of the best model is 95% or more, escaping the RANSAC iteration;

In the above step, if the inlier ratio of the best model is not more than 95%, determining whether it is the maximum (MAX) iteration;

In the above step, if the maximum (MAX) iteration (Iteration), characterized in that performing the step of escaping the RANSAC iteration (Iteration).

In addition, by calculating the covariance of the set of point data of the 3D real world divided into cells in the image space and included in one cell, it is determined whether the shape in which the points are spread is the shape of a plane or not. characterized by judging.

In addition, in 2D, the eigen vector of the covariance of the data cluster has the smallest distribution and the direction of the largest 2 axes, and the eigen value indicates the size of the axis, the 2 axes are orthogonal, and the data in 3D The eigen vector of the covariance of the cluster represents the three-axis direction of the ellipsoid, and the eigen value represents the size of each axis, and the three axes are orthogonal to each other. It is characterized in that it is possible to distinguish whether the distribution corresponds to a plane or a line or not.

According to another aspect of the present invention, a real-time floor detection method based on a depth image in an ROI region from a TOF camera image in a state in which a TOF (Time Of Flight) camera is mounted so as to photograph objects located in a certain space,

A first step of receiving a depth image from a TOF camera,

A second step of setting the ROI area of the lower part in an image of a certain size,

A third step of dividing the ROI area of a certain size into a square pixel grid of a certain size;

For each cell that has passed the above steps, the covariance is calculated using NDT (Normal Distribution Transformation), and the eigen value and eigen vector of the ellipsoid-shaped covariance matrix are calculated. Step 4,

A fifth step of determining whether there are enough cells for the candidate group planar cells that have passed the above procedure to calculate a plane,

A sixth step of extracting three from a set of floor plan candidate cells;

The seventh step of making three points using the average value of point sets in this cell, modeling the plane using them, and calculating the normal vector;

The equation of the plane becomes a model in RANSAC and examines the floor plan candidate cells, but the angle between the floor model and the cell must be within ±10 degrees, and the distance between the two must be less than 5 cm. 8th step of adding to the Inlier,

After the 8th step, finally, all cells other than the inlier cells of the current floor model in the ROI are checked based on the distance, and if they have a distance within 5 cm, they are also converted to the floor as inliers. There is provided a depth image-based real-time floor detection method further comprising a ninth step of adding.

In addition, in the fourth step, it is characterized in that the eigen value of each cell is sorted in order of size and used.

In addition, if any one of the two conditions in the eighth step is deviated, it is excluded from the inlier, and if the current floor model has a higher consensus (support) than the existing floor model that has the most support, the best model (best model) is replaced with the current model, the inlier list is also updated, this is repeated up to the maximum number of iterations, and the consensus of the bottom plane candidate cells for the model during iteration ( If the consensus) (support) ratio exceeds 95%, it is characterized in that the RANSAC algorithm is terminated without further repetition.

In addition, the depth image cell (cell) is characterized in that it is clustered based on the depth image cell (cell) among the data in the 3D real world space,

It is characterized by determining the plane after NDT algorithm processing based on image cell and modeling with RANSAC in a small candidate group.

As described above, according to the depth image-based real-time floor detection method according to the present invention, there is an effect of having a small amount of computation to be executed in real time while being robust to the depth image-based noise.

In addition, according to the present invention, if an approach is not performed in a pixel unit rather than a cell unit in this part, even a more detailed part can be detected as the bottom area. Since the pixels were examined in the form of an image cell-based NDT plane, data corresponding to most planes is included. That is, the number of points to be inspected itself is small. In addition, since only the adjacent area near the cell included in the floor plan model is checked, the burden of computation is small. Since only the remaining point pixels on the edge of the floor plane are added, it can be added using the distance from the best model plane.

1 is a diagram showing covariance, eigen vector, and eigen value, in which covariance in two dimensions has an ellipse shape and in 3D it has an ellipsoid shape.
2 is a flowchart sequentially illustrating an example of a floor detection process in a depth image-based real-time floor detection method according to the present invention.
3 and 4 are diagrams showing result screens of a general point-based RANSAC floor detection method.
5 is a diagram illustrating a depth video image.
6 is a diagram illustrating an example of an initial image for executing the depth image-based real-time floor detection method according to the present invention.
7 is a diagram illustrating an example of an execution image of a depth image-based real-time floor detection method according to the present invention.

Hereinafter, a depth image-based real-time floor detection method according to the present invention will be described in detail with reference to the accompanying drawings.

2 is a flowchart sequentially showing an example of a floor detection process in a depth image-based real-time floor detection method according to the present invention, and FIGS. 3 and 4 are diagrams showing result screens of a general point-based RANSAC floor detection method, FIG. 5 is a view showing a depth image image, FIG. 6 is a view showing an example of an initial image for executing a real-time floor detection method based on a depth image according to the present invention, and FIG. 7 is a view showing a real-time floor detection based on a depth image according to the present invention It is a diagram showing an example of an execution image of the method.

According to the depth image-based real-time floor detection method according to the present invention,

By calculating the covariance of the set of point data of the 3D real world divided into cells in the image space and included in one cell, it is determined whether the shape in which the points are spread is the shape of a plane or not. will judge In addition, an eigen vector of covariance of a data cluster in two dimensions indicates the direction of two axes having the smallest and largest distribution, and an eigen value indicates the size of the axis. The two axes are orthogonal. In 3D, the eigen vector of the covariance of the data cluster indicates the three-axis direction of the ellipsoid, the eigen value indicates the size of each axis, and the three axes are orthogonal to each other. By using the direction and size of this axis, it is possible to distinguish whether the distribution of data corresponds to a plane or a line.

According to the present invention, the goal is

It is used as a still shot of a 320x240 depth image obtained from a TOF camera.

It uses one still shot, not map-type data that accumulates 3D point cloud data.

The installation position and posture are roughly known (because it is floor detection, not plane finding).

Real-time & Robust Algorithm

Real-time--> From having to calculate distance information between points, it is a point-packed cell unit calculation, so when using the RANSAC algorithm, the maximum number of repetitions is reduced, and the number of comparisons for voting judgment is reduced, so real-time processing is possible.

Robust--> 3D point cloud contains unpredictable noise, so when modeling using point data as it is, results with errors appear. Cells are grouped together, so it is judged whether the cells form a planar shape or not, and non-planar cells are excluded from the material for floor modeling. When points are placed on a nearby object, the number is greater than that of a distant object, and it is possible to solve the problem that a model rather than the actual floor is detected because it is greatly influenced by simple point distance-based voting.

In the present invention, a cell-based RANSAC algorithm using a depth image is used, and a near-planar inlier is added to the extracted model.

2 is a flowchart sequentially showing an example of a floor detection process in a depth image-based real-time floor detection method according to the present invention;

Referring to FIG. 2 , an algorithm process will be briefly described.

In the present invention, in a state where the TOF camera is mounted in a certain space, the floor is detected in real time based on a depth image (or called an image) in the ROI (interest) area set from the image of the TOF camera.

First, after setting the bottom search area, the depth image is divided into cells (S2).

Thereafter, it is determined whether the image cell is flat using an eigen value after applying the NDT algorithm (S4). Meanwhile, the NDT algorithm is proposed based on a map for spatial division. For example, lidar is being introduced as a road map-based technology.

Thereafter, only cells close to the bottom criterion are extracted as candidate cells (S6).

Thereafter, it is determined whether the number of cells is greater than the minimum number of candidate cells (S8).

If, in step S8, the number of cells is smaller than the minimum number of candidate cells, it is determined that there is no floor plane (S7).

If, in step S8, the number of cells is greater than the minimum number of candidate cells,

The floor is modeled with a cell-based RANSAC (algorithm) (S10). If it proceeds to step S10, an image is displayed as shown in FIG.

Thereafter, a distance-based inlier other than the candidate cell is added (S12).

In the depth image-based real-time floor detection program storage (not shown) for the processing of the above steps,

The image segmentation processing unit (not shown) divides the depth image into cells after setting the floor search area.

In the plane determination unit (not shown), it is determined whether the image cell is flat using an eigen value after applying the NDT algorithm.

In the candidate group extraction unit (not shown), only cells close to the floor criterion are extracted as candidate cells.

A cell determination unit (not shown) determines whether the number of cells is greater than the minimum number of candidate cells.

If the cell determination unit is smaller than the minimum number of candidate cells in step S8, it is determined that there is no floor plan.

In addition, if the cell determination unit is greater than the minimum number of candidate cells in step S8,

The floor is modeled with cell-based RANSAC (algorithm).

The inlier adding unit (not shown) adds a distance-based inlier other than the candidate cell.

Meanwhile, the cell-based RANSAC floor modeling process is performed by a cell-based RANSAC floor modeling program (not shown). That is, after random selection of three cells as the mean value of the candidate cells, the floor plane is modeled,

After calculating the plane angle and distance with the candidate cells, the condition of the inlier cell is determined,

Thereafter, it is determined whether the Consensus count is greater than the maximum count (MAX COUNT),

If the consensus (support) count is greater than the maximum count (MAX COUNT), control to update the best model (Best model), inlier cell (inlier cell),

After that, it is determined whether the inlier ratio of the best model is 95% or more,

If the inlier ratio of the best model is more than 95%, control to escape the RANSAC iteration,

In the above step, if the inlier ratio of the Best model is not more than 95%, it is determined whether it is MAX Iteration,

In the above step, if the MAX iteration (Iteration), control to escape the RANSAC iteration (Iteration).

3 and 4 are diagrams showing a general point-based RANSAC floor detection method. Referring to FIG. 3, it is a method of calculating only the distance between the plane equation modeled by the RANSAC and the point, and thus a vertical partition view It works as an inlier of RANSAC modeling.

Referring to FIG. 4 , it is similar to the case of FIG. 3 , but shows a case in which an actual floor area is detected (because the floor area is wider than the desk area). 4 shows a case in which the bottom part was detected well.

The problem with RANSAC is that RANSAC has the advantage of being able to create a robust model excluding outliers, but RANSAC-based floor detection using commonly used 3d point cloud data may produce incorrect results as shown in FIG. 3, and vertical wall outliers There is a problem in that it cannot filter out outliers. In addition, there is a disadvantage that real-time processing is not possible because it requires a large number of iterations.

5 shows a general depth image photograph received by a TOF camera.

6 is a diagram illustrating an example of an initial image for executing the depth image-based real-time floor detection method according to the present invention.

Referring to FIG. 6 , an ROI area considering the installation location among 320*240 images and a state divided into 10 by 10 pixels within the ROI area are shown.

In the present invention, data can be obtained and used in real time.

7 is a view showing an example of an execution image of the depth image-based real-time floor detection method according to the present invention.

Referring to FIG. 7 , after NDT transformation in the ROI region, the RANSAC floor plane inlier is displayed. The white part is the inlier bottom detection part, and the gray part is the bottom plane candidate group added at the end. It is an inlier that was not (see step S12),

The light blue part represents three cells selected during the RANSAC best model (the number of light blue parts changes every time), and the small circle represents a non-planar area.

When the floor plane is displayed in the step S12, this floor display data can be usefully used. This is because, in order to reliably recognize an object, it is necessary to remove the floor data because an object (eg, a standing person) is important in the entire image.

On the other hand, an inlier can be said to be data that matches a model to be actually detected, and an outlier refers to data that should not be reflected during detection.

According to the present invention, a method for increasing the success rate while reducing the number of iterations, in particular, a method capable of processing in real time is proposed.

In the present invention, the amount of computation is reduced through steps S2 and S4.

In addition, floor modeling is performed with cell-based RANSAC (S112, S114, S116, S118, S120, S122, S124).

Meanwhile, step S114 is a process of filtering out an image portion that may not be flat, an L-shape, or other noise.

In the following description, in the process of storing the best model, only the indexes of those that have been supported (consensus) are stored.

Features of the present invention

According to the present invention, first, the approximate installation position of the camera is known. For example, if it is installed on the rear of an autonomous vehicle, the installation location is installed at approximately 1m and the horizontal line is installed so that the central axis of the camera faces, but the posture may change due to various factors. The reason is that the ground is tilted, the posture when installing is slightly distorted, or the rear wheel of the vehicle has fallen slightly due to lack of wind. At this time, the installation angle will be the standard when finding the floor plane.

- Receive depth image from TOF camera. The depth image has a size of 320*240 and has 76800 pixels. If the size of the camera changes, the parameters can also be changed accordingly.

-Set the ROI area in the lower part of the image of -320*240 size (this can be adjusted according to the case where the floor is seen more or less depending on the installation location).

- Divide the 320*120 ROI area into a grid of 10 by 10 pixels (size division can be changed, but if it is too small, planar features are not well extracted due to noise, so it is good if it is large enough. If you use it, you can change the size accordingly). In addition, only cells with more than 50% of data among 10 by 10 pixels proceed with the following procedure. Cells that do not exceed the size are considered to be too small or noise as noise.

-For each cell that has passed the above procedure, find the covariance using Normal Distribution Transformation. Then, the eigen value and eigen vector of the covariance matrix in the form of an ellipsoid are calculated.

- Use the eigen value of each cell in order of size (λ1>λ2>λ3). If λ1 and λ2 are more than 5 times greater than λ3, it is calculated as a flat surface (this may vary depending on the accuracy of the sensor. The greater the accuracy of λ, the less noise, so the difference in magnitude of λ is large). After that, if the angle with the floor reference plane according to the installation location is within ±20 degrees, it remains as a floor plane candidate cell. For the rest, only the NDT remains calculated (if the performance of the 20-degree camera is good, the size can be reduced).

- It is judged whether there are enough cells to calculate the plane of the candidate group planar cells that have passed the above procedure. In situations where an object is too close or looking at a wall, it is difficult to see the floor area. At this time, it concludes that “there is no floor plane” and terminates the algorithm. If the minimum number of bottom cells is exceeded, proceed as follows.

- Extract 3 cells from the floor plan candidate cell set. Create 3 points using the average value of point sets in the cell, model the plane using them, and calculate the normal vector. The plane equation is modeled in RANSAC to examine the bottom plane candidate cells. The protractor between the floor model and the cell must be within ±10 degrees (not limited thereto, it can be ±5 degrees or ±20 degrees), and the distance between the two must be 5 cm or less (but not limited to this) Add to Inlier as consensus count . If either of these conditions is exceeded, it is excluded from the inlier. If the current floor model has a higher consensus than the existing floor model with the most support, the best model is replaced with the current model, and the inlier list is also updated. Repeat this up to the maximum number of iterations. If the consensus ratio of the floor plane candidate cells for the model exceeds 95% during iteration, the RANSAC algorithm is terminated without repeating any more.

-Finally, all cells other than the inlier cell of the current floor model in the ROI are checked based on the distance, and if they have a distance within 5 cm, they are also added as inliers as the floor.

Hereinafter, the method according to the present invention will be described in more detail.

Camera installation position, reference plane initialization

Mainly, floor detection is performed before object detection and is used to separate and process objects. This patent aims to find one floor. And it assumes that you roughly know the installation angle of the camera.

The required information is the height of the camera from the floor and the equation of the reference floor plane with the camera's attitude value applied. For example, about 1m, the camera was installed to face the front. Depending on the environment in which the TOF camera is used, the standard parameters of the camera may be determined according to the mounting object or location.

The equation of the plane is ax+by+cz+d=0, and (a, b, c) represents the normal vector to the plane. Before applying the above conditions, if we explain the x, y, and z coordinate axes of the real world, the x-axis is the same as and parallel to the u-axis in the image plane, and the y-axis is parallel to the v-axis. The z-axis is parallel to the central axis of the camera, and the viewing direction is the same as the central axis of the camera. According to this condition, 1 m of the floor plane, the central axis of the camera should be installed so that it is horizontal to the floor plane, and the u-axis should be installed so that the u-axis is horizontal to the horizon on the image. Then, it can be expressed using the equation of the floor plane, where (a,b,c) has a value of (0,1,0). The formula is simplified to 1y+d=0. Since it is installed at a height of 1 m here, if (0, 1, 0) is substituted into x, y, z, d = -1. Using this part, the distance between the reference floor plane and the candidate plane obtained by RANSAC (the distance calculated based on the camera position) can be used, but the position and angle of the camera may be shaken, so it cannot be fixed. In addition, since this patent takes into account that the installation position and angle of the camera may not be accurate, the part using the distance from the reference plane equation is excluded. If the actual installation location is changed or the reference floor plane is changed when you come back, you can find the normal vector of the reference floor plane and apply it. Even if the installation height is changed, if the angle is the same, the values of a, b, and c do not change. Therefore, when the reference plane and the floor candidate cell plane are used, when the reference plane and the floor model plane extracted by RANSAC are used, the value does not change when calculating the plane angle with the plane. This is because only a, b, and c are used in the calculation.

If the camera is installed based on the above criteria, if the distance can be measured to infinity, the bottom area occupies half of the v-axis in the TOF image area, so the Region of Interest is given a size of 320*120. This value may vary depending on the installation location and direction. According to the conditions described above, the lower half of the 320*240 image was set as an ROI, and it was divided into 10 by 10 pixel cells. In general, there is noise in the data of 3D Point Cloud, but if the cells are divided into too small cells, even if these cells are subjected to NDT transformation, it is difficult to obtain planar characteristics due to noise. If the size of the noise is very small due to the good sensor performance, good performance can be achieved even if the cell size is small. This should be adjusted according to the specification and performance of the sensor. If it is set too large, since the curved data is represented by one cell, there may be a difference from the actual floor area, so the division size must be adjusted according to the size of the image received from the TOF sensor and the accuracy of the data.

There are 100 3D point data in a cell with a size of 10 by 10 pixels. In TOF depth images, not all pixels have data. Data may not be available because it is too close or too far away, because of limited sensor performance, light is not reflected back on a smooth surface, is absorbed, etc. Therefore, data is often empty among cells. In the absence of such data, it is difficult to calculate the covariance of the cell in a flat form during NDT transformation. NDT is calculated only when there are more than 3/4 of data in the cell. The 3/4 criterion can be set differently depending on the performance of the sensor. Based on this criterion, cells for NDT calculation are filtered out. If the value of one pixel of the depth image is simply expressed only by the distance from the camera, it is a 3D real world value using the camera calibration information, and after converting it into X, Y, Z, which is a 3-axis Cartesian coordinate system, NDT is performed. NDT is an abbreviation of Normal Distribution Transformation and can be translated to the extent of Normal Distribution Transformation. The position average value m of the three-dimensional point set and the covariance matrix in the form of 3 by 3 are obtained. An eigen value and an eigen vector are obtained for this covariance matrix. There are three eigenvectors, and these three eigenvectors are principal component vectors perpendicular to each other. And it has three axes. The eigenvalue tells the size of each eigenvector. With the size of the eigenvalues, it can be determined whether the covariance of the data has a disk shape, and the angle with the reference floor plane can be calculated using the equation of the reference floor plane as the value of the eigenvector. λ1, λ2, and λ3, which are the eigen values of each cell, are sorted in the order of size to make λ1 > λ2 > λ3. When λ1>λ3 ×5 and λ2>λ3 ×5 are satisfied, it is determined that the form of covariance corresponds to a plane. In this part, if NDT is applied by spatially dividing the 3D Point Cloud data of the real world into square-shaped cells, it will come out in the form of a disk, not an ellipse, on the plane. A lot of spatial division is required depending on the size of the map, and it is an effective method when there is a lot of data, but real world spatial division is not appropriate for a single depth image obtained from a TOF camera. There are many empty spaces, and if the distance increases, the number of data in the space decreases rapidly, so even if the covariance is calculated, the accuracy is inevitably reduced. Image cell-based NDT guarantees the number of data in a cell. However, as the distance increases, the shape of an ellipse, not a disk, comes out, so there is only a comparison between λ1 and λ3, and λ2 and λ3, and there is no comparison between λ1 and λ2.

The cell determined to be flat calculates the angle with the reference floor plane. When calculating the angle of two planes, the angle is obtained as the dot product of the vector using normal vectors. When the size of this angle is less than +20 degrees, it is added to the list of cell candidates for the floor plane. If this condition is not satisfied, the cell corresponds to the plane, but it cannot enter the floor candidate plane. For example, the exterior of a wall or box. A comparison of distance values is not performed here. The reason is that if the installation angle is slightly distorted, the distance between the cell and the reference floor plane increases as the distance increases, so it cannot be adjusted with a single threshold. As a result of NDT, if the number of selected floor plan candidate cells is less than 20, it is said that there is no floor and the algorithm ends. This condition is optional, and if the number is too large, the floor detection algorithm will not work, and if it is set too low, the detection performance of the floor plane modeling by RANSAC is low.

If the minimum number of candidate cells is passed, floor plane detection by RANSAC is applied. RANSAC is a method that selects a model with a high support by selecting data randomly from a data cluster, creating a model, checking the support for this model, and repeatedly applying it. Although the model can be estimated robustly by reducing the influence of the data of the outliers in the data including outliers, it has the disadvantage that it does not guarantee the optimal model generation. By using the RANSAC method, three cells are randomly selected from among the floor plan candidate cells so that they do not overlap. Calculate the equation of a general plane using the average values of the extracted cells. If these three points are Q, R, S, the vectors QR and QS are calculated, and the values of a, b, c, and d are calculated at ax+by+cz+d=0. Calculate the distance using the calculated formula and the average value of all floor plane candidate cells, and calculate the angle between the model plane and the cell plane. If the distance has a value of 5 cm or less and the angle has a value of 10 degrees or less, it is added to the number of supports and classified as an inlier. If the number of supports is greater than the count number of the previous best model, the current model, number of supports, and Inlier list are updated to best. If the consensus ratio of the best model exceeds 95%, which is the RANSAC iteration escape condition, the RANSAC algorithm is terminated. Parameters such as angles, minimum number of cells, and escape conditions used here can be changed according to the environment. The formula used in the process of detecting the floor with RANSAC is as follows.

equation of the plane

Find the equation of a plane with three points

Find the angle of two planes

Find the distance between a plane and a point

After obtaining the best floor model extracted with RANSAC in the form of a plane equation, all cells in the ROI area are checked and added to the Inlier. This is to add cells that appear in a non-planar shape with NDT based on the distance. For example, the space where a small boulder is placed is not calculated as a flat surface, but it is too small to be classified as an object, so it is added as a floor. Alternatively, a structure in which the wall meets the floor and has a right angle is classified into cells, so this part can also be added as a floor.

In this part, if the approach is not done in units of pixels, not in units of cells, even more detailed parts can be detected as the bottom area. Since the pixels were examined in the form of an image cell-based NDT plane, data corresponding to most planes is included. That is, the number of points to be inspected itself is small. In addition, since only the adjacent area near the cell included in the floor plan model is checked, the burden of computation is small. Since only the remaining point pixels on the edge of the floor plane are added, it can be added using the distance from the best model plane.

S2: Step of dividing the depth image into cells after setting the bottom search area
S4: Step of determining whether a cell is flat using an eigen value after applying NDT
S6: Step of extracting only cells close to the bottom criterion as candidate cells
S8: determining whether the cell (cell) is greater than the minimum number of candidate cells (cell)
S10: Step of floor modeling with cell-based RANSAC
S12: adding a distance-based inlier other than the candidate cell (cell)

Claims (8)

A real-time floor detection method based on a depth image in an ROI area from a TOF camera image in a state where a TOF (Time Of Flight) camera is mounted so that objects located within a certain space can be photographed,
After setting the bottom search area, dividing the depth image into cells,
After applying NDT (Normal Distribution Transformation) to an image cell, determining whether it is flat using an eigen value;
Extracting only cells close to the bottom criterion as candidate cells;
Determining whether the cell (cell) is greater than the minimum number of candidate cells (cell),
If, in the determination step, the number of cells is smaller than the minimum number of candidate cells, it is determined that there is no floor plane,
If, in the determination step, the cell (cell) is greater than the minimum number of candidate cells (cell), the step of modeling the floor with a cell (cell) based RANSAC, and
A depth image-based real-time floor detection method comprising adding a distance-based inlier other than a candidate cell. The method of claim 1,
The step of modeling the floor with the cell-based RANSAC is
3 cell (cell) random selection as the mean (mean) value of the candidate cell (cell) and floor plane modeling,
Determining an inlier cell condition after calculating the plane angle and distance with the candidate cells;
Thereafter, determining whether the consensus (support) count is greater than the maximum count (MAX COUNT);
In the determination step, if the consensus (support) count is greater than the maximum count (MAX COUNT),
Determining the best model (Best model), updating the inlier cell (inlier cell),
Determining whether the inlier ratio of the best model is 95% or more;
In the above step, if the inlier ratio of the best model is 95% or more, escaping the RANSAC iteration;
In the above step, if the inlier ratio of the best model is not more than 95%, determining whether it is the maximum (MAX) iteration;
In the above step, if the maximum (MAX) iteration (Iteration), depth image-based real-time floor detection method, characterized in that performing the step of escaping the RANSAC iteration (Iteration). According to claim 1, wherein the covariance of a set of point data of the 3D real world divided into cells in the image space and included in one cell is calculated, so that the shape in which the points are spread is a plane Depth image-based real-time floor detection method, characterized in that it is determined whether the shape is or not. The method according to claim 1, wherein the eigen vector of the covariance of the data cluster in two dimensions indicates the direction of the two axes having the smallest and largest distribution, and the eigen value indicates the size of the axis, and the two axes are orthogonal; In 3D, the eigen vector of the covariance of the data cluster indicates the three-axis direction of the ellipsoid, and the eigen value indicates the size of each axis, and the three axes are orthogonal to each other. Depth image-based real-time floor detection method, characterized in that by using it, it is possible to distinguish whether the distribution of data corresponds to a plane, a line, or not. A real-time floor detection method based on a depth image in an ROI area from a TOF camera image in a state where a TOF (Time Of Flight) camera is mounted so that objects located within a certain space can be photographed,
A first step of receiving a depth image from a TOF camera,
A second step of setting the ROI area of the lower part in an image of a certain size,
A third step of dividing the ROI area of a certain size into a square pixel grid of a certain size;
For each cell that has passed the above steps, the covariance is calculated using NDT (Normal Distribution Transformation), and the eigen value and eigen vector of the ellipsoid-shaped covariance matrix are calculated. Step 4,
A fifth step of determining whether there are enough cells for the candidate group planar cells that have passed the above procedure to calculate a plane,
A sixth step of extracting three from a set of floor plan candidate cells;
The seventh step of making three points using the average value of point sets in this cell, modeling the plane using them, and calculating the normal vector;
The equation of the plane becomes a model in RANSAC and examines the floor plan candidate cells, but the angle between the floor model and the cell must be within ±10 degrees, and the distance between the two must be less than 5 cm. 8th step of adding to the Inlier,
After the 8th step, finally, all cells other than the inlier cells of the current floor model in the ROI are checked based on the distance, and if they have a distance within 5 cm, they are also referred to as inliers to the floor. Depth image-based real-time floor detection method further comprising a ninth step of adding. [Claim 6] The method of claim 5, wherein, in the fourth step, the eigen value of each cell is arranged in order of size and used. The method of claim 5, wherein if any one of the two conditions in the eighth step is deviated, the current floor model has a higher consensus (support) than the existing floor model that has obtained the most support, except for an inlier. If it is high, it replaces the best model with the current model, updates the inlier list, iterates up to the maximum number of iterations, and during iteration, the bottom plane candidate cell for the model ), if the consensus (support) ratio exceeds 95%, the depth image-based real-time floor detection method, characterized in that the RANSAC algorithm is terminated without further repetition. 6. The method of claim 5,
The depth image cell is characterized in that it is clustered based on a depth image cell among data in a 3D real world space,
Depth image-based real-time floor detection method, characterized in that the plane is determined after NDT based on an image cell and modeled with RANSAC in a small candidate group.

Images