WO2019056306A1

WO2019056306A1 - Robust depth information-based plane detection method and system

Info

Publication number: WO2019056306A1
Application number: PCT/CN2017/102947
Authority: WO
Inventors: 金枝; 罗海丽; 周长源; 邹文斌; 李霞
Original assignee: 深圳大学
Priority date: 2017-09-22
Filing date: 2017-09-22
Publication date: 2019-03-28

Abstract

The present invention is applicable to image processing. Provided is a robust depth information-based plane detection method, comprising: receiving a depth image, and extracting, from the depth image, a plurality of valid seed blocks; performing a region growing operation according to the valid seed block so as to obtain a grown plane of the valid seed block; performing overgrowth correction or undergrowth correction on the grown plane of the valid seed block so as to obtain a valid grown plane of the valid seed block; and outputting a depth image detection plane containing the valid grown plane of the valid seed blocks. In an embodiment of the present invention, overgrowth correction or undergrowth correction is performed on a grown plane obtained by performing a region growing operation on a valid seed block, thereby increasing accuracy and robustness of a plane detection method.

Description

Robust depth information based plane detection method and system

Technical field

The invention belongs to the field of image processing and computer vision, and in particular relates to a robust depth information based plane detection method and system.

Background technique

Since the plane carries the direction and size information of the object in the 3D scene, the plane detection technique can be used for 3D reconstruction. 3D reconstruction can be simply summarized as the process of plane detection of indoor and outdoor scenes and the establishment of a segmentation plane model. In addition, planar inspection technology is also widely used for object detection in robot navigation systems and computer vision.

In the early plane inspection work, the texture information of the plane was mainly used, but when the plane color or texture is inconsistent, it will bring great challenges to this method. In the prior art, the distance information of the depth map is used to solve the above problem, and the result also proves that the method can effectively cope with the complicated situation. The depth map can be generated directly by a depth camera (such as SwissRanger SR40001, Microsoft Kinect) or synthesized by software. The different values in the graph reflect the distance information of the object in the scene relative to the camera. Since the depth map represents spatial information for each point in the scene, points from the same plane will have similar spatial characteristics, such as gradients and normals. Based on this, some work proposes to use a local gradient to segment the plane of significant objects in the scene, and then use the Random Sample Consensus (RANSAC) to segment the ground. There are also work to extract real-time plane detection by extracting the three components of the normal vector of each point and clustering points with similar directions, but their accuracy and robustness are poor.

According to the working principle, the plane detection methods can be divided into three categories: iterative plane fitting method, based on Hough transform method and region growing method. The iterative plane fitting method is a commonly used method for plane detection, and its typical representative is RANSAC, in which the fitting model is initialized according to several randomly selected points in RANSAC. The method has better effect in detecting large planes and is robust to noise, but the calculation amount is too large, and Complex planes are oversimplified during the calculation process. The Hough transform method is often used for parameterized target detection, especially for lines and circles in 2D planes. In order to make this type of method available in 3D space and reduce computational cost, a variety of Hough transform-based derivative algorithms have emerged. For example, the 3D Hough transform method uses the slope of the plane in the x-axis and y-axis directions and its distance from the origin of the coordinate to represent the plane, but it also has a higher computational cost when looking for the parameters of the fitted model, especially when This problem becomes more pronounced when the input data is large or the accumulator is sensitive. The Random Hough Transform (RHT) uses a probabilistic model to calculate parameters to avoid high computational costs when finding optimal parameters. The main idea of the regional growth method is to use the correlation between adjacent points to construct the plane. The work proposes an algorithm based on the growth of two seed points. The plane parameters are gradually updated by the centroid and covariance matrix of the grown region, but the calculation is too big. Another work proposes the Cached-Octree Region-Growing (CORG) algorithm to divide the point cloud into several planar regions, but the planar overgrowth problem may occur in the intersecting planar region. In another work, two growth strategies are proposed: a sub-window growth algorithm and a hybrid growth algorithm in an unstructured environment in a point cloud of a structured environment. This algorithm is faster than point-based growth when the window size is set appropriately. In addition, there is work to propose the Robust Principle Component Analysis (RPCA), which divides all 3D points into boundary line points, corner points and bins. The process of plane growth begins with a bin, and if the angle between the growing bin and its neighboring bin is less than a certain threshold, the neighboring bin is included in the currently growing plane. The method works normally when the angle between adjacent bins is an acute angle, but the method may fail when the angle between adjacent bins is an obtuse angle.

Summary of the invention

The technical problem to be solved by the present invention is to provide a robust depth information based plane detection method and system, aiming at solving the problem that the existing plane detection algorithm has poor accuracy and robustness in complex scenarios.

The present invention is implemented in such a manner that a robust depth information based plane detection method includes:

Receiving a depth map, extracting a number of valid seed blocks in the depth map;

Performing region growing according to the effective seed block to obtain a generating plane of the effective seed block;

Performing growth correction or undergrowth correction on the generation plane of the effective seed block to obtain an effective growth plane of the effective seed block;

A depth map detection plane containing the effective growth plane of the valid seed block is output.

The invention also provides a robust depth information based plane detection system, comprising:

An extracting unit, configured to receive a depth map, and extract a number of valid seed blocks in the depth map;

a growth unit for performing region growth according to the effective seed block to obtain a generation plane of the effective seed block;

a correcting unit, configured to perform growth correction or undergrowth correction on a generating plane of the effective seed block to obtain an effective growth plane of the effective seed block;

And an output unit, configured to output a depth map detection plane including an effective growth plane of the effective seed block.

Compared with the prior art, the present invention has the beneficial effects that the embodiment of the present invention obtains the growth plane of each effective seed block by extracting the effective seed block of the depth map and performing region growing according to the extracted effective seed block, for each effective seed. The growth plane of the block undergoes growth correction or undergrowth correction to obtain an effective growth plane, and the depth map is detected according to the effective growth plane. The embodiment of the invention improves the accuracy and robustness of the plane detection method by performing growth correction or undergrowth correction on the growth plane obtained by growing the effective seed block region.

DRAWINGS

1 is a flowchart of a robust depth information based plane detection method according to an embodiment of the present invention;

2 is a schematic diagram of an adjacent point set according to an embodiment of the present invention;

3 is a schematic diagram of adjacent points of an effective seed block according to an embodiment of the present invention;

4 is a flowchart of an over-growth correction method according to an embodiment of the present invention;

FIG. 5 is a flowchart of an undergrowth correction method according to an embodiment of the present invention; FIG.

FIG. 6 is a schematic structural diagram of a robust depth information based plane detection system according to an embodiment of the present invention.

Detailed ways

The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It is understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

FIG. 1 shows a robust depth information based plane detection method provided by an embodiment of the present invention, including:

S101. Receive a depth map, and extract a number of valid seed blocks in the depth map.

S102: Perform area growing according to the effective seed block to obtain a generating plane of the effective seed block.

S103, performing growth correction or undergrowth correction on a generating plane of the effective seed block to obtain an effective growth plane of the effective seed block;

S104. Output a depth map detection plane including an effective growth plane of the effective seed block.

In order to solve the problem that the existing plane detection algorithm is inferior in accuracy and robustness in a complex scenario, the embodiment of the present invention provides a robust Depth-driven Plane Detection (DPD) method. That is, the depth detection is used for plane detection. The plane detection method includes two parts: plane detection based on seed block growth and post-processing process to further enhance the robustness of the algorithm. The plane detection method starts from the seed block with the highest smoothness, and uses the plane equation of the fitting plane of the growth plane and the dynamic threshold function to guide the growth process. Under the action of this mechanism, when the seed block grows to the maximum extent, the next seed block begins to grow, and the growth process is iterated until all the planes are detected. The accuracy and robustness of the proposed method can be improved by using the dynamic threshold function and the post-processing process of the enhancement mechanism. The latter is proposed for the planar overgrowth and undergrowth problems that are easy to occur based on the growth-based planar detection method.

Specifically, the plane equation of the fitted plane refers to the plane equation obtained by the linear least squares plane fitting method of the growth plane, assuming that the plane equation of the fitting plane is z=p ₀₀ +p ₁₀ x+p ₀₁ y, p ₀₀ +p ₁₀ x+p ₀₁ yz=0. For n pixel points on the plane: (x _i , y _i , z _i ), i=1, 2...n, according to the linear least squares plane fitting method, the n pixel points are used to fit the above plane equation, then Minimize the value of the following formula:

That should be satisfied

j=00,10,01, find the coefficients p ₀₀ , p ₁₀ , p ₀₁ , so the plane equation z is obtained, the normal vector n _i = (p ₁₀ , p ₀₁ , -1), d _i = p ₀₀ .

The embodiments of the present invention are further described below:

Generate a valid seed block:

Affected by the shooting scene and the camera, there are sometimes holes in the depth map acquired by the depth camera, that is, pixels with no depth value. If the growing seed block is randomly selected, the problem of model fitting failure and high computational cost is prone to occur. Therefore, a key step in region growth is to select growing seed pieces, ie, to select areas without holes.

Therefore, in this step, on the input depth map, an L×L rectangular window is slid one pixel at a time in a raster scan manner to traverse the depth map, and the rectangular window is checked at the position of each pixel point. Whether all the points are hole points, when there is no hole point in the rectangular window, the point set in the rectangular window is regarded as a valid seed block, and is calculated by linear least square plane fitting method (Linear Least Squares, LLS) The plane equation of the effective seed block and the corresponding smoothness, the linear least squares plane fitting method is the best matching function for finding the data by minimizing the sum of the squares of the errors. Through such continuous scanning, all valid seed blocks in the input depth map are extracted.

In 3D space, the plane can be represented by the usage vector n and the distance d from the origin of the coordinate system. The fitting plane of the effective seed block can be obtained by linear least squares plane fitting method. The plane of the effective seed block is assumed to be S, and the plane equation of the effective seed block is f(t), with S=f(t) , that is, the plane equation is mathematically used to represent the plane of fit. Assuming that p is any point on the i-th valid seed block, the fitting error of the fit plane of the i-th effective seed block to point p is defined as:

e _i (p)=|n _i ·p+d _i |;

Where n _i represents the normal vector of the fitting plane of the i-th effective seed block, and d _i represents the distance between the fitting plane of the i-th effective seed block and the origin of the coordinate system.

The root mean square fitting error δ _{i is} defined as:

Where P represents the set of points of all pixels in the valid seed block, and |P| represents the number of pixels in the valid seed block.

In the present invention, the root mean square fitting error δ _{i is} used to represent the smoothness of the effective seed block, so a smaller root mean square fitting error δ _i means higher smoothness. Since the planar growth process starts with the most smooth effective seed block, accurate growth of the plane can be ensured, so in this embodiment, all the effective seed blocks are sorted from small to large according to the root mean square fitting error δ _i .

Regional growth process: Regional growth is an iterative process that generates planes from an effective seed block. In the specific growth process, not all effective seed blocks will eventually have the opportunity to grow, because the regional growth begins with the smoothest effective seed block, and the growth of the effective seed block used for growth will result in the inclusion of growth planes. With some valid seed blocks that have not been used, these included effective seed blocks do not have the opportunity to grow.

The key to the region-based growth method is to distinguish the inner point and the outer point of the current growth plane, and the judgment is based on the threshold T, wherein the threshold T is the output of the dynamic threshold function proposed by the embodiment of the present invention. When it is detected that the adjacent point set of the growth plane is an empty set, or there is no point in the adjacent point set suitable for the current growth plane, it is determined that the current plane growth reaches the maximum range, and the current plane ends the growth. As shown in FIG. 2, the adjacent point set refers to a set of all pixel points adjacent to the current growth plane, that is, points 1 to 16 constitute a set of adjacent points of the current growth plane.

Specifically, the main contents of the growth process include:

In the first stage, all adjacent points of the effective seed block with the highest smoothness are detected. In this step, as shown in FIG. 3, the adjacent side of the effective seed block is taken as the center, and the adjacent side of the 3×3 rectangular window is taken. Point, a total of 8 adjacent points, point 1 to point 8. When it is detected that the adjacent point is not the hole point and does not belong to all the previously detected planes, the adjacent point is substituted into the plane equation of the current growth plane by fitting the error formula e _i (p)=|n _i · The corresponding fitting error is obtained by p+d _i |, and the adjacent point is substituted into the fitting error formula, which means that the point coordinate of the adjacent point is replaced by the point p, and the fitting error is calculated. When the fitting error is less than the threshold T, the adjacent point is merged into the current growth plane, and conversely, the adjacent point is regarded as the outer point of the current growth plane. In this phase, a fitting error needs to be calculated for each adjacent point.

In the second stage, the linear least squares plane fitting method LLS is used to update the parameters of the plane equation, and the root mean square fitting error is also updated. Specifically, at the first stage, a growth plane has been fitted and the fitting error is calculated. When new adjacent points are merged into the current growth plane, it is an iterative process to re-use LLS to fit the new growth plane and calculate a new fitting error. The planar plane can be optimally adjusted again and again during the growth process. In this phase, the plane equation is updated once after each growth, and a root mean square fitting error is also calculated for all points on the plane.

In the prior art, most plane detection algorithms use a fixed value as a threshold value, and thus easily cause planar overgrowth or undergrowth problems. Later, an algorithm proposed a threshold function based on the depth map to capture the camera noise model, but the thresholds proposed in these methods will increase monotonically as the depth value increases. However, this is not completely consistent with the actual situation, and the following three situations occur: 1) when the threshold is small, the growth plane is prone to undergrowth; 2) when the threshold is large, the distal facet is prone to overgrowth; 3) when growing When the plane is parallel to the camera plane, the threshold becomes a fixed value because the depth of the plane does not change.

To overcome the above disadvantages, the embodiment of the present invention designs a dynamic threshold function based on a noise model and a plane size.

The dynamic threshold function is defined as follows:

Wherein, I represents the depth map, and I _d represents a depth value of a point on the depth map that is substituted into a plane equation using a fitting error formula to calculate a fitting error. In this embodiment, the point is the adjacent of the current growth plane. Point, τ represents the maximum allowable roughness of the growth plane, λ determines the growth rate of the threshold, H and W represent the height and width of the depth map, respectively, α and k are constants, j represents the number of iterations of the plane during the growth process, initialization order j =1, the maximum threshold T of the output is determined by I _d .

The dynamic threshold function can solve the problem based on the noise model well. For example, when the growth plane is the distal facet, the overgrowth caused by the large threshold can be avoided because the growth plane is considered; when the growth plane is parallel to the camera In the plane, the threshold will not be a fixed value due to the accumulation of noise taking into account the growth process.

Post-processing of the strengthening mechanism:

Specifically, the post-processing process of the mandatory mechanism includes an over-growth correction process and an under-growth correction process, and the over-growth correction process and the under-growth correction process are respectively described below:

1. Over-growth correction process:

During planar growth, one of the growth planes preferentially grows to its intersection with another growth plane. Overgrowth refers to the phenomenon that the current growth plane erroneously grows onto the intersecting plane if the fitting error between the adjacent pixel points of the plane intersection line and the current plane is smaller than the current threshold. Among them, the growth direction can be divided into longitudinal growth and lateral growth.

Using the plane equations of the two planes, the intersection line can be expressed as a parametric equation:

Where × represents the cross product of the two vectors,

Represents the normal vector of the detection plane S _i ,

Represents the normal vector of the largest plane S _u , t represents the unknown parameter,

Indicates the direction vector of the intersection line, p ₀ is a point on the intersection line, defined as:

among them,

Indicates the distance between S _i and the origin of the coordinate system,

Indicates the distance between S _i and the origin of the coordinate system.

The main contents of the correction process include:

First, the overgrowth region S _{0 is} accurately detected.

The degree of overgrowth depends mainly on the angle of the intersection plane, the accuracy of the depth data, and the threshold T, where the angle θ is the inverse cosine of the two-plane normal vector point multiplication, and the threshold T is the output of the dynamic threshold function. Maximum value. The theoretical width w of the overgrowth region is equal to the ratio of the threshold value T to the sine value of the angle θ. To ensure that all pixels in the overgrowth region are within the scan range, the embodiment of the present invention takes the actual width as [w] (1+). ε), where [w] represents a larger integer close to w, ε>0, which ensures that the actual width is greater than the theoretical width.

Second, redistribute.

Redistribution refers to the plane equation that substitutes the point on the farthest boundary of the overgrowth region S ₀ into two planes, and calculates the fitting error of the point to the plane respectively, compares the two fitting error magnitudes, and merges the overgrowth region S ₀ To the plane with a small fitting error.

Finally, the plane of erroneous growth is corrected and its plane equation is updated. Again, this correction process is used for other detected planes.

The process of the overgrowth correction step is shown in Figure 4, including:

(1) finding the adjacent plane of the detection plane S _i with the growth plane of the effective seed block currently corrected as the detection plane S _i ;

(2) Find the intersection line of the detection plane S _i and the adjacent plane by the parametric equation of the intersection line;

(3) judging whether the intersection line exists in the 2D 2D range of the current depth map, if present, setting the width of the detection scan bar; otherwise, moving to the next adjacent plane, and then finding the intersection line and determining whether it exists currently Depth map 2D;

(4) sliding the detection scan bar along the intersection line to scan the pixel points to obtain the overgrowth region S _o ;

(5) It is judged whether or not the current detection plane S _i is overgrowth according to the overgrowth region S ₀ . If over growth, the overgrowth region S ₀ is merged with the adjacent plane, and conversely, the overgrowth region S _{0 is} merged with the detection plane S _i . In this step, if the point on the overgrowth region S _o is substituted into the adjacent plane, it is found that the fitting error is smaller than the fitting error substituted into the current detection plane, indicating that the overgrowth region should belong to the adjacent plane, so the current A growth problem has occurred in the detection plane.

(6) During the merging process, the overgrowth area S _o may be broken into small isolated blocks to detect and redistribute these small isolated blocks. Specifically, a primary, secondary overgrowth region is created in the detection of the previous step (5), and after the main region has been reallocated, the secondary region may become an isolated region. Redistribution is based on substituting points in isolated regions into adjacent plane equations. The plane with the smallest root mean square fitting error is the plane to which the isolated region belongs.

(7) Updating the plane equation of the current detection plane S _i and its adjacent plane, respectively. Since each plane has its own plane equation, it is necessary to update the plane equations of both in this step.

(8) Iterate the above correction process until the growth planes of all the valid seed blocks are detected, and the growth correction is completed.

2. Under-growth correction process:

In order to solve the undergrowth of the growth plane, the embodiment of the present invention proposes a method of plane merging, that is, when the two growth planes satisfy the three conditions of parallel, coplanar, and adjacent, they can be merged into one larger plane. These relationships are detected in sequence in the method of under-growth correction.

Parallel: Since the error and depth noise of the fitting equation of the growth plane should be considered in the actual situation, when the angle between the two growth planes is smaller than the angle between the normal vector of the current plane and the maximum error estimation plane normal vector, the two growth planes are determined to be parallel.

Coplanar: Calculate the fitting error of each point on the two growth planes to the detection plane, that is, the fitting error of the point of the smaller growth plane in the two growth planes to the plane of the larger growth, and the larger growth plane From the point above to the fitting error of the self, when the calculated Hellinger distance of the two fitting errors is less than the threshold, the two growth planes are discriminated. The Hellinger distance is usually used to measure the similarity of two probability distributions. However, if the size of the two growth planes is too different, even if the two growth planes are coplanar in real conditions, a large Hellinger distance can be obtained, so for the two growth planes with large differences in size, the judgment threshold _{Tm is} used. Determine whether it is coplanar.

Adjacent: After expanding the current growth plane by one pixel, it is detected that its edge has an intersection with the edge of another growth plane, and the two growth planes are adjacent planes.

The process of the undergrowth correction step is shown in Figure 5, including:

(1) Using all the growth planes as the detection plane, all the detection planes are arranged and stored in descending order according to the area size, and the detection plane S _{u having the} largest area is found, wherein the area of the detection plane after the descending arrangement is saved in the form of a list. ;

(2) Calculate the angle θ _ij of all the two detection planes, and put θ _ij into an upper triangular matrix;

(3) Using the upper triangular matrix, find all detection planes parallel to the detection plane S _u . In this step, whether the angle between the two detection planes is smaller than the angle between the detection plane S _u normal vector and the maximum error estimation plane normal vector determines whether the plane is parallel. If it is smaller, the two detection planes are parallel. Otherwise, the two detection planes are not parallel;

(4) The detection planes parallel to the detection plane S _{u are} arranged in descending order according to the size of the area. Specifically, in over-growth correction and under-growth correction, the principle of correction is to start from a plane with a large area. The plane equation is more accurate due to the larger area.

(5) S _k denotes any detection plane parallel to S _u , assuming that the plane pairs S _k and S _{u are} parallel, respectively calculating the fitting error of each pixel point on the detection plane S _k and the detection plane S _u _Ku , and the fitting error e _uu of each pixel point on the detection plane S _u ;

(6) determining whether the detection plane S _u is a large plane, and the larger plane refers to a plane whose size exceeds one sixth of the depth map;

(7) If the detection plane S _u is not a large plane, the two fitting errors in the step (5) are represented by a histogram, and the Hellinger distance of the two curves in the histogram is calculated. Determining whether the hellinger distance is smaller than a hellinger distance determination threshold T _h , and when it is smaller than a hellinger distance determination threshold T _h , then S _k is coplanar with S _u , and if not smaller, S _k and S _{u are} not coplanar, the hellinger The distance judgment threshold T _h is a constant;

(8) If the detection plane S _u is a large plane, the fitting error e _ku of each pixel point on the detection plane S _k and the detection plane S _u is calculated. When the fitting error e _{ku is} smaller than the judgment threshold T _m , then S _k is coplanar with S _u , and if not smaller, S _k and S _{u are} not coplanar, and the judgment threshold T _m is constant.

(9) If S _k and S _{u are} coplanar, it is further determined whether S _k and S _u are adjacent, and adjacent common faces can be merged;

(10) If a total of S _k is adjacent to S _u , the detection plane S _{k is} merged with the detection plane S _u to update the plane equation of S _u ;

(11) If S _k is parallel to S _u , the parallel plane of S _k is also necessarily parallel to the detection plane S _u , so the parallel plane of _Sk is added to the parallel group of detection planes S _u ;

(12) Whether all parallel planes of S _u are detected, if the detection is completed, the undergrowth correction step for the detection plane S _u ends, otherwise, moves to the next parallel plane, and starts looping from step (5) until All parallel planes are detected.

(13) If the undergrowth correction for the detection plane S _u is completed, the next detection plane is detected according to the order of the area of all the detection planes until all the detection planes are corrected.

The above embodiment provided by the present invention uses a depth information to drive a plane detection model to detect a depth map generated by a depth camera using a seed growth method, and post-processes a plane on which the seed block is grown, so that each Plane detection of indoor scenes and depth noise is more robust, improving the accuracy of plane detection. In addition, when adjacent planes require precise boundaries, the post-processing of the enhancement mechanism can re-allocate the correct boundary points to the plane.

The above embodiments provided by the present invention can be applied to fields such as robust planar detection (wall surface, desktop, etc.), indoor scene reconstruction and position recognition, robot navigation systems, and object recognition in the field of computer vision.

FIG. 6 shows a robust depth information based plane detection system provided by an embodiment of the present invention, including:

An extracting unit 601, configured to receive a depth map, and extract a number of valid seed blocks in the depth map;

a growing unit 602, configured to perform region growing according to the effective seed block, to obtain a generating plane of the effective seed block;

a correcting unit 603, configured to perform a growth correction or an undergrowth correction on a generating plane of the effective seed block to obtain an effective growth plane of the effective seed block;

The output unit 604 is configured to output a depth map detection plane including an effective growth plane of the effective seed block.

Further, the extracting unit 601 is specifically configured to:

Centering on each pixel of the depth map, traversing the depth map in a raster scan manner with a preset size rectangular window;

When traversing each pixel of the depth map, checking whether all points in the rectangular window are hole points;

If there is no hole point in the rectangular window, the point set in the rectangular window is used as a valid seed block;

The plane equation and smoothness of each of the effective seed blocks are calculated by a linear least squares plane fitting method.

Where the mathematical plane equation is used to express the fitting plane, the extraction unit is also used to:

Obtaining a fitting plane of the effective seed block by using a linear least squares plane fitting method, where the fitting plane usage vector n and the distance d from the origin of the coordinate system are represented;

Calculating a fitting error of a fitted plane of the effective seed block and a point on the effective seed block;

Let p denote any point on the i-th effective seed block, n _i denote the normal vector of the fitting plane of the i-th effective seed block, and d _i denote the distance between the fitting plane of the i-th effective seed block and the origin of the coordinate system , e _i (p) represents the fitting error of the fitting plane of the i-th effective seed block and the point p on the i-th effective seed block, then e _i (p)=|n _i ·p+d _i |

Calculating a root mean square fitting error according to the fitting error, and indicating a smoothness of the effective seed block by the root mean square fitting error;

Calculating the error according to the root mean square with δ _i

Where P represents the set of points of all pixels within the valid seed block, and |P| represents the number of pixels within the valid seed block;

According to the root mean square fitting error of the effective seed block, the order is sorted in order from small to large.

Further, the growth unit 602 is specifically configured to:

Detecting all adjacent points of the most smooth effective seed block;

If the currently detected adjacent points of the valid seed block are not holes and do not belong to the growth plane of other valid seed blocks, then the adjacent points of the currently detected effective seed block are substituted into the plane equation of the effective seed block. The corresponding fitting error is calculated by fitting the error formula e _i (p)=|n _i ·p+d _i |

Determining whether the fitting error is less than a threshold T, if less than, merging adjacent points of the currently detected effective seed block into a growth plane in which the effective seed block is currently growing, and if larger, the phase is The neighbor is considered to be the extraneous point of the growth plane of the current growth;

Updating the parameters of the equation of the effective seed block by using a linear least squares plane fitting method, and updating the root mean square fitting error accordingly;

When it is determined that the adjacent point set of the growth plane of the effective seed block is an empty set, or there is no adjacent point in the adjacent point set that is suitable for the currently growing growth plane, the growth is stopped, and a growth plane of the effective seed block is obtained;

Wherein the threshold T is an output of a dynamic threshold function, and the dynamic threshold function is:

I denotes the depth map, I _d denotes a depth value of a point on the depth map that is substituted into a plane equation using a fitting error formula to calculate a fitting error, τ denotes an allowable maximum roughness of the growth plane, and λ determines a growth rate of the threshold, H and W represent the height and width of the depth map, respectively, α and k are constants, j represents the number of iterations of the growth plane during growth, the initialization order j=1, and the maximum threshold T of the output is determined by I _d .

Further, the step of correcting the growth plane of the effective seed block by the correcting unit 603 includes:

Finding an adjacent plane of the detection plane by using a growth plane of a valid seed block that is currently corrected as a detection plane;

Finding a line of intersection of the detection plane and the currently detected adjacent plane by a parametric equation of the intersection line;

Determining whether the intersection line exists in a two-dimensional range of the depth map, and if not, detecting a next adjacent plane, and finding a line intersecting the detection plane with the currently detected adjacent plane;

If yes, setting a width of the detection scan strip, and scanning the detection scan strip along the intersection line to obtain a super-growth region;

Determining, according to the overgrowth region, whether the detection plane is overgrowth, and if it is determined that the detection plane is overgrowth, combining the overgrowth region with the adjacent plane, if it is determined that the detection plane has not grown, Combining the overgrowth region with the detection plane to obtain an effective growth plane; wherein Detecting and reallocating isolated blocks resulting from fragmentation during the merging process;

Updating a plane equation of the detection plane and the adjacent plane respectively;

Iterating through the above-mentioned overgrowth correction steps until all growth planes have completed growth detection and correction;

The step of correcting the under-growth correction of the generating plane of the effective seed block by the correcting unit 603 includes:

Taking all the growth planes as the detection plane, all the detection planes are arranged in descending order according to the size of the area, and the detection plane S _{u having the} largest area is found;

Calculate the angle θ _ij of all the two detection planes, and put all the calculated angles θ _ij into the upper triangular matrix;

Using the upper triangular matrix, finding a detection plane parallel to the detection plane S _u ;

Detecting planes parallel to the detection plane S _u in descending order according to the size of the area;

To S _k and S _u represents any of a parallel detection plane, respectively, calculated for each pixel on each pixel on the S _k and S _u fitting error e _ku, S _u and the fitting error itself e _uu ;

Determining whether the detection plane S _u is a larger plane whose area size exceeds one sixth of an area of the depth map;

If the detection plane S _u is not a large plane, the fitting error e _ku and the fitting error e _uu are represented by a histogram, and the hellinger distance of the two curves in the histogram is calculated;

Determining whether the hellinger distance is smaller than a hellinger distance determination threshold T _h ; if less than, S _k is coplanar with S _u , if not less, then S _k and S _{u are} not coplanar, and the hellinger distance determination threshold T _h is constant ;

If the detection plane S _u is a large plane, it is determined whether the fitting error e _ku is smaller than the determination threshold T _m , and if less than, S _k is coplanar with S _u , and if not smaller, S _k is not shared with S _u The judgment threshold T _m is a constant;

If S _k is coplanar with S _u , it is necessary to further determine whether S _k and S _u are adjacent. If adjacent, then S _k and S _{u are} combined to obtain an effective growth plane, and the plane equation of S _u is updated, and The parallel plane of S _k is added to the parallel group of S _u ;

Determining whether all parallel planes of the detection plane S _u are detected, and if so, ending the undergrowth correction step for the detection plane S _u , if not, detecting the next parallel plane until all parallel planes are detected;

If the undergrowth of the detection plane S _u is corrected, the next detection plane is detected according to the order of the area of all the detection planes until all the detection planes are corrected.

The above is only the preferred embodiment of the present invention, and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection of the present invention. Within the scope.

Claims

A robust depth information based plane detection method, comprising:

Receiving a depth map, extracting a number of valid seed blocks in the depth map;

Performing region growing according to the effective seed block to obtain a generating plane of the effective seed block;

Performing growth correction or undergrowth correction on the generation plane of the effective seed block to obtain an effective growth plane of the effective seed block;

A depth map detection plane containing the effective growth plane of the valid seed block is output.
The method of detecting a plane according to claim 1, wherein the extracting the plurality of valid seed blocks in the depth map comprises:

Centering on each pixel of the depth map, traversing the depth map in a raster scan manner with a preset size rectangular window;

When traversing each pixel of the depth map, checking whether all points in the rectangular window are hole points;

If there is no hole point in the rectangular window, the point set in the rectangular window is used as a valid seed block;

The plane equation and smoothness of each of the effective seed blocks are calculated by a linear least squares plane fitting method.
The method of detecting a plane according to claim 2, wherein the plane of fit is expressed by a mathematical plane equation, and then the plane equation of each of the effective seed blocks is calculated by a linear least squares plane fitting method And smoothness includes:

Obtaining a fitting plane of the effective seed block by using a linear least squares plane fitting method, where the fitting plane usage vector n and the distance d from the origin of the coordinate system are represented;

Calculating a fitting error of a fitted plane of the effective seed block and a point on the effective seed block;

Let p denote any point on the i-th effective seed block, n i denote the normal vector of the fitting plane of the i-th effective seed block, and d i denote the distance between the fitting plane of the i-th effective seed block and the origin of the coordinate system , e i (p) represents the fitting error of the fitting plane of the i-th effective seed block and the point p on the i-th effective seed block, then e i (p)=|n i ·p+d i |

Calculating a root mean square fitting error according to the fitting error, and indicating a smoothness of the effective seed block by the root mean square fitting error;

Calculating the error according to the root mean square with δ i
Where P represents the set of points of all pixels in the valid seed block, and |P| represents the number of pixels in the valid seed block;

The valid seed blocks are ordered in ascending order of the root mean square fitting error of the effective seed blocks.
The method for detecting a plane according to claim 3, wherein the generating a region according to the effective seed block to obtain the effective seed block comprises:

Detecting all adjacent points of the most smooth effective seed block;

If the currently detected adjacent points of the valid seed block are not holes and do not belong to the growth plane of other valid seed blocks, then the adjacent points of the currently detected effective seed block are substituted into the plane equation of the effective seed block. The corresponding fitting error is calculated by fitting the error formula e i (p)=|n i ·p+d i |

Determining whether the fitting error is less than a threshold T. If less, merging adjacent points of the currently detected effective seed block into a growth plane in which the effective seed block is currently growing, if greater than, An outlier that is considered to be the growth plane of the current growth;

Updating the parameters of the equation of the effective seed block by using a least squares plane fitting method, and updating the root mean square fitting error accordingly;

When it is determined that the adjacent point set of the growth plane of the effective seed block is an empty set, or there is no adjacent point in the adjacent point set that is suitable for the currently growing growth plane, the growth is stopped, and a growth plane of the effective seed block is obtained;

Wherein the threshold T is an output of a dynamic threshold function, and the dynamic threshold function is:

I denotes the depth map, I d denotes a depth value of a point on the depth map that is substituted into a plane equation using a fitting error formula to calculate a fitting error, τ denotes an allowable maximum roughness of the growth plane, and λ determines an increase of the threshold value Speed, H and W represent the height and width of the depth map, respectively, α and k are constants, j represents the number of iterations of the growth plane during growth, the initialization order j = 1, and the maximum threshold T of the output is determined by I d .
The plane detecting method according to claim 1, wherein the step of performing growth correction on the generating plane of the effective seed block comprises:

Finding an adjacent plane of the detection plane by using a growth plane of a valid seed block that is currently corrected as a detection plane;

Finding a line of intersection of the detection plane and the currently detected adjacent plane by a parametric equation of the intersection line;

Determining whether the intersection line exists in a two-dimensional range of the depth map, and if not, detecting a next adjacent plane, and finding a line intersecting the detection plane with the currently detected adjacent plane;

If yes, setting a width of the detection scan strip, and scanning the detection scan strip along the intersection line to obtain a super-growth region;

Determining, according to the overgrowth region, whether the detection plane is overgrowth, and if it is determined that the detection plane is overgrowth, combining the overgrowth region with the adjacent plane, if it is determined that the detection plane has not grown, Combining the overgrowth region with the detection plane to obtain an effective growth plane; wherein, during the merging process, the isolated block generated by the fragmentation is detected and redistributed;

Updating a plane equation of the detection plane and the adjacent plane respectively;

The above-described overgrowth correction steps are iterated until all growth planes have completed growth detection and correction.
The method of detecting a plane according to claim 1, wherein the step of performing undergrowth correction on a generation plane of the effective seed block comprises:

Taking all the growth planes as the detection plane, all the detection planes are arranged in descending order according to the size of the area, and the detection plane S u having the largest area is found;

Calculate the angle θ ij of all the two detection planes, and put all the calculated angles θ ij into the upper triangular matrix;

Using the upper triangular matrix, finding a detection plane parallel to the detection plane S u ;

Detecting planes parallel to the detection plane S u in descending order according to the size of the area;

To S k and S u represents any of a parallel detection plane, respectively, calculated for each pixel on each pixel on the S k and S u fitting error e ku, S u and the fitting error itself e uu ;

Determining whether the detection plane S u is a larger plane whose area size exceeds one sixth of an area of the depth map;

If the detection plane S u is not a large plane, the fitting error e ku and the fitting error e uu are represented by a histogram, and the hellinger distance of the two curves in the histogram is calculated;

Determining whether the hellinger distance is smaller than a hellinger distance determination threshold T h ; if less than, S k is coplanar with S u , if not less, then S k and S u are not coplanar, and the hellinger distance determination threshold T h is constant ;

If the detection plane S u is a large plane, it is only judged whether the fitting error e ku is smaller than the determination threshold T m , and if it is smaller, S k is coplanar with S u , and if not smaller, S k and S u are not Coplanar, the judgment threshold T m is a constant;

If S k is coplanar with S u , it is necessary to further determine whether S k and S u are adjacent. If adjacent, then S k and S u are combined to obtain an effective growth plane, and the plane equation of S u is updated, and a plane parallel to S k is added to the parallel group of S u ;

It is judged whether all parallel planes of the detection plane S u are detected, and if so, the undergrowth correction step for the detection plane S u is ended, and if not, the next parallel plane is detected until all parallel planes are detected.

If the undergrowth of the detection plane S u is corrected, the next detection plane is detected according to the order of the area of all the detection planes until all the detection planes are corrected.
A robust depth information based plane detection system, comprising:

An extracting unit, configured to receive a depth map, and extract a number of valid seed blocks in the depth map;

a growing unit, configured to perform region growing according to the effective seed block, to obtain a generating plane of the effective seed block;

a correcting unit, configured to perform growth correction or undergrowth correction on a generating plane of the effective seed block Positive, obtaining an effective growth plane of the effective seed mass;

And an output unit, configured to output a depth map detection plane including an effective growth plane of the effective seed block.
The plane detecting system according to claim 7, wherein the extracting unit is specifically configured to:

Centering on each pixel of the depth map, traversing the depth map in a raster scan manner with a preset size rectangular window;

When traversing each pixel of the depth map, checking whether all points in the rectangular window are hole points;

If there is no hole point in the rectangular window, the point set in the rectangular window is used as a valid seed block;

The plane equation and smoothness of each of the effective seed blocks are calculated by a linear least squares plane fitting method.

Where the mathematical plane equation is used to express the fitting plane, the extraction unit is also used to:

Obtaining a fitting plane of the effective seed block by using a linear least squares plane fitting method, where the fitting plane usage vector n and the distance d from the origin of the coordinate system are represented;

Calculating a fitting error of a fitted plane of the effective seed block and a point on the effective seed block;

Let p denote any point on the i-th effective seed block, n i denote the normal vector of the fitting plane of the i-th effective seed block, and d i denote the distance between the fitting plane of the i-th effective seed block and the origin of the coordinate system , e i (p) represents the fitting error of the fitting plane of the i-th effective seed block and the point p on the i-th effective seed block, then e i (p)=|n i ·p+d i |

Calculating a root mean square fitting error according to the fitting error, and indicating a smoothness of the effective seed block by the root mean square fitting error;

Calculating the error according to the root mean square with δ i
Where P represents the set of points of all pixels in the valid seed block, and |P| represents the number of pixels in the valid seed block;

According to the order of the root mean square fitting error of the effective seed block, the effective seed block is entered in the order of small to large Row sorting.
The plane detecting system according to claim 8, wherein said growth unit is specifically configured to:

Detecting all adjacent points of the most smooth effective seed block;

If the currently detected adjacent points of the valid seed block are not holes and do not belong to the growth plane of other valid seed blocks, then the adjacent points of the currently detected effective seed block are substituted into the plane equation of the effective seed block. The corresponding fitting error is calculated by fitting the error formula e i (p)=|n i ·p+d i |

Determining whether the fitting error is less than a threshold T. If less, merging adjacent points of the currently detected effective seed block into a growth plane in which the effective seed block is currently growing, if greater than, An outlier that is considered to be the growth plane of the current growth;

Updating the parameters of the equation of the effective seed block by using a linear least squares plane fitting method, and updating the root mean square fitting error accordingly;

When it is determined that the adjacent point set of the growth plane of the effective seed block is an empty set, or there is no adjacent point in the adjacent point set that is suitable for the currently growing growth plane, the growth is stopped, and a growth plane of the effective seed block is obtained;

Wherein the threshold T is an output of a dynamic threshold function, and the dynamic threshold function is:

I denotes the depth map, I d denotes a depth value of a point on the depth map that is substituted into a plane equation using a fitting error formula to calculate a fitting error, τ denotes an allowable maximum roughness of the growth plane, and λ determines a growth rate of the threshold, H and W represent the height and width of the depth map, respectively, α and k are constants, j represents the number of iterations of the growth plane during growth, the initialization order j=1, and the maximum threshold T of the output is determined by I d .
The plane detecting system according to claim 7, wherein the step of the correction unit performing the growth correction on the generating plane of the effective seed block comprises:

Finding an adjacent plane of the detection plane by using a growth plane of a valid seed block that is currently corrected as a detection plane;

Finding a line of intersection of the detection plane and the currently detected adjacent plane by a parametric equation of the intersection line;

Determining whether the intersection line exists in a two-dimensional range of the depth map, and if not, detecting a next adjacent plane, and finding a line intersecting the detection plane with the currently detected adjacent plane;

If yes, setting a width of the detection scan strip, and scanning the detection scan strip along the intersection line to obtain a super-growth region;

Determining, according to the overgrowth region, whether the detection plane is overgrowth, and if it is determined that the detection plane is overgrowth, combining the overgrowth region with the adjacent plane, if it is determined that the detection plane has not grown, Combining the overgrowth region with the detection plane to obtain an effective growth plane; wherein, during the merging process, the isolated block generated by the fragmentation is detected and redistributed;

Updating a plane equation of the detection plane and the adjacent plane respectively;

Iterating through the above-mentioned overgrowth correction steps until all growth planes have completed growth detection and correction;

The step of the correcting unit performing the undergrowth correction on the generating plane of the effective seed block includes:

Taking all the growth planes as the detection plane, all the detection planes are arranged in descending order according to the size of the area, and the detection plane S u having the largest area is found;

Calculate the angle θ ij of all the two detection planes, and put all the calculated angles θ ij into the upper triangular matrix;

Using the upper triangular matrix, finding a detection plane parallel to the detection plane S u ;

Detecting planes parallel to the detection plane S u in descending order according to the size of the area;

To S k and S u represents any of a parallel detection plane, respectively, calculated for each pixel on each pixel on the S k and S u fitting error e ku, S u and the fitting error itself e uu ;

Determining whether the detection plane S u is a larger plane whose area size exceeds one sixth of an area of the depth map;

If the detection plane S u is not a large plane, the fitting error e ku and the fitting error e uu are represented by a histogram, and the hellinger distance of the two curves in the histogram is calculated;

Determining whether the hellinger distance is smaller than a hellinger distance determination threshold T h ; if less than, S k is coplanar with S u , if not less, then S k and S u are not coplanar, and the hellinger distance determination threshold T h is constant ;

If the detection plane S u is a large plane, it is only judged whether the fitting error e ku is smaller than the determination threshold T m , and if it is smaller, S k is coplanar with S u , and if not smaller, S k and S u are not Coplanar, the judgment threshold T m is a constant;

If S k is coplanar with S u , it is necessary to further determine whether S k and S u are adjacent. If adjacent, then S k and S u are combined to obtain an effective growth plane, and the plane equation of S u is updated, and The parallel plane of S k is added to the parallel group of S u ;

Determining whether all parallel planes of the detection plane S u are detected, and if so, ending the undergrowth correction step for the detection plane S u , if not, detecting the next parallel plane until all parallel planes are detected;

If the undergrowth of the detection plane S u is corrected, the next detection plane is detected according to the order of the area of all the detection planes until all the detection planes are corrected.