CN110675360B

CN110675360B - Real-time plane detection and extraction method based on depth image

Info

Publication number: CN110675360B
Application number: CN201910712025.7A
Authority: CN
Inventors: 颜成钢; 韩顾稳; 徐俊; 姚霆; 梅涛; 孙垚棋; 张继勇; 张勇东
Original assignee: Hangzhou Dianzi University
Current assignee: Hangzhou Dianzi University
Priority date: 2019-08-02
Filing date: 2019-08-02
Publication date: 2022-04-01
Anticipated expiration: 2039-08-02
Also published as: CN110675360A

Abstract

The invention discloses a real-time plane detection and extraction method based on a depth image. The invention comprises the following steps: step 1: preprocessing the depth image, and extracting block information from the depth image; step 2: performing dimensionality reduction statistics on the geometric parameter distribution of the block; and step 3: searching a parallel plane for each estimated plane normal vector; and 4, step 4: and (5) optimizing plane parameters. According to the imaging principle of the depth image and the spatial characteristics of the plane structure, the computer can acquire the plane structure information in the environment in real time through the depth image, so that the three-dimensional reconstruction result is more accurate. Therefore, the plane structure information can be accurately and efficiently extracted from the depth image.

Description

Real-time plane detection and extraction method based on depth image

Technical Field

The invention relates to the fields of robot positioning and three-dimensional reconstruction, in particular to a plane structure detection and segmentation method of a depth image.

Background

At the earliest 2010, Microsoft corporation developed a low-cost structured light depth camera that was able to acquire accurate depth images in a small range of scenes in the room, thereby enabling reconstruction of scenes in the room. Microsoft shows a Kinectfusion real-time reconstruction algorithm in a SIGGRAPH (Special Interest Group on Graphics and Interactive technologies, short for Graphics and inter-technology Special fan Group) meeting in 2011, Newcomb is the main responsible person of the algorithm project, and the algorithm is the first algorithm for realizing real-time dense scene reconstruction. And Izadi completes the dynamic interaction of the scene and the human on the basis of the above.

In 2016, Whelan proposed a method for detecting map loops using color image features, which simultaneously was able to detect the position of a single point source in space. This article is also representative of the binning expression model (Surfel Representation).

The technical method in the field of off-line (non-real-time) dense reconstruction also has a very profound influence on real-time dense reconstruction, and the Zhou Q Y team makes a huge contribution in the field, and in 2012, the team proposes a dense reconstruction method based on interest points.

However, at present, because GPS signals cannot be received indoors, there is no drift-free indoor positioning and reconstruction algorithm suitable for real scenes, and in recent research on drift-free indoor positioning and reconstruction algorithms in the field, the core lies in detecting and dividing planes shot by a depth camera as reference points to improve positioning accuracy. Heretofore, the method capable of extracting planes from a depth map in real time is based only on the assumption of manhattan planes, that is, assuming that all planes in space are orthogonal to each other, and there is no concern about the presence of non-orthogonal planes.

Disclosure of Invention

The invention provides a real-time and parallelizable depth image-based plane detection and segmentation method.

A real-time plane detection and extraction method based on a depth image can accurately extract plane information from a depth image with a non-orthogonal plane in a scene, and comprises the following steps:

step 1: preprocessing the depth image and extracting block information from the depth image, which comprises the following steps:

1-1, performing filtering processing on the depth image by using a bilateral filtering algorithm;

1-2, dividing the depth image into a plurality of blocks with equal-size rectangles;

1-3, as shown in fig. 2, the pixels in the rectangular block are back projected to a three-dimensional space to obtain a three-dimensional point cloud, and the tangent plane information of the block is extracted, wherein the tangent plane information comprises the space coordinate of the block, the normal vector of the tangent plane, the distance between the tangent plane and the origin, and the block confidence coefficient. Note that the (u, v) th block is patch (u, v).

Step 2: performing dimensionality reduction statistics on the geometric parameter distribution of the block;

2-1, as shown in FIG. 3, performing spherical polar mapping on the normal vector of each block from the three-dimensional space nx-ny-nz to the two-dimensional plane Px-Py;

2-2, carrying out grid division on the normal vector points mapped on the two-dimensional plane Px-Py, and carrying out histogram statistics on the normal vector points on the two-dimensional plane Px-Py in a Gaussian weighting mode to obtain a Px-Py histogram;

and 2-3, searching 8-domain maximum values for the Px-Py histogram, and performing ball polar inverse mapping on the Px-Py coordinates of the corresponding points to obtain estimated plane normal vector parameters.

And step 3: searching a parallel plane for each estimated plane normal vector;

3-1, performing inner product on the space coordinates of all blocks and the estimated plane normal vector, namely projecting the space coordinates of the blocks to the normal vector direction;

3-2, carrying out histogram statistics on the block distribution obtained by projection, and searching a maximum value to obtain the distance between each parallel plane on the normal vector and the origin;

and 4, step 4: optimizing plane parameters, and recording the ith plane parameter as P_iThe normal vector of the plane and the distance between the plane and the origin are included;

4-1. optimization of P Using MeanShift_iAnd eliminating the grid error of the histogram statistics, wherein the single iteration process comprises the following steps: searching all blocks in the mean radius and calculating the mean value to obtain a new P_i。

4-2, using K-means to inquire the nearest plane parameter for each block, and optimizing P by the inquiry result_i。

The invention has the following beneficial effects:

according to the imaging principle of the depth image and the spatial characteristics of the plane structure, the computer can acquire the plane structure information in the environment in real time through the depth image, so that the three-dimensional reconstruction result is more accurate. Therefore, the plane structure information can be accurately and efficiently extracted from the depth image.

The traditional method for extracting the plane from the depth map is only based on the assumption of Manhattan planes, that is, all planes in the space are assumed to be mutually orthogonal, and the situation that non-orthogonal planes exist is not involved. The condition that the planes are orthogonal to each other has an influence on the depth map plane extraction algorithm mainly in real time and accuracy. The invention provides a real-time, parallel and efficient depth map plane detection and extraction algorithm, and a plurality of planes can be in a non-orthogonal geometric relationship.

In a traditional method for extracting a plane from a depth map, main plane information needs to be extracted, that is, the largest plane in the depth map is extracted first, and then the remaining planes are extracted through a priori condition that all planes in a space are mutually orthogonal. While in reality, the planes in space may be non-orthogonal to each other. Therefore, the method and the device can extract the geometric information of all planes in the scene from the depth map, and the algorithm result is more accurate and comprehensive.

Drawings

FIG. 1 is a flow chart of planar normal vector detection;

FIG. 2 is a block partitioning and block information diagram;

FIG. 3 is a ball polar map;

fig. 4 is a two-dimensional statistical histogram.

FIG. 5 is a parallel plane query flow diagram.

Fig. 6 is a diagram of the effect of planar normal vector optimization.

Fig. 7 is a rendering diagram of a plane segmentation result.

Detailed Description

The objects and effects of the present invention will become more apparent from the following detailed description of the present invention with reference to the accompanying drawings.

Fig. 1 is a flow chart of plane normal vector detection, and fig. 2 to 4 are views of the visualization process of fig. 1. The part is a part for detecting a plane normal vector by the algorithm, and mainly describes how to detect the normal vector of a plane in a scene from an input depth image. The following is a detailed description of fig. 1 to 4:

first, as shown in FIG. 2, the input depth image is preprocessed and bilateral filtering is used to initially filter out the sensorThe depth image is divided into several blocks of equal size, here, 16 × 16 blocks are taken as an example. Each block contains a normal vector (n)_x,n_y,n_z) The average positions (x, y, z) of all the points in the block can be calculated by back projecting all the pixels of each block into a three-dimensional space to form a point cloud, and the tangent plane normal vector (n) of the block can be calculated by fitting a plane to all the points in the block_x,n_y,n_z) And the number of vertices in the block that fit the tangent plane distribution, i.e., the number of significant points c, can be found to be equivalent to the confidence of the planarity property of the block.

Secondly, as shown in fig. 3, a spherical pole mapping is performed on the normal vector of each block, and the mapping relationship is as follows:

wherein p is_x,p_yIs the Px-Py coordinate after mapping. The spherical surface in the figure is a unit spherical surface, the blue color point on the spherical surface is the normal vector distribution point of each block, and the red color point on the Px-Py is the point of the normal vector point mapped to the Px-Py plane. The red bold line is the connecting line from the highest point of the sphere to a certain normal vector point and the mapping point.

Thirdly, as shown in fig. 4, the result mapped to Px-Py in the second step is rasterized and segmented, histogram statistics is performed in a gaussian weighting mode to obtain two-dimensional histogram distribution in fig. 4, and the maximum value in the two-dimensional histogram and the coordinate of the maximum value in the Px-Py coordinate system are searched in the 8-field to obtain the coordinate (p) of the plane in the depth image scene, which is mapped to the Px-Py coordinate system_x,p_y). By calculating the inverse sphere polar mapping of the coordinates, the normal vector (n) of the plane can be calculated_x,n_y,n_z)。

As shown in fig. 5, the normal vector of the plane can be obtained by the first three steps of calculation, each block is sequentially projected to the direction of the normal vector to obtain the distance from the block to the origin in the normal vector direction, the histogram distribution of each block in each normal vector direction is obtained by discretizing the distance from each block to the origin and performing gaussian weighted histogram statistics, and the maximum value of the histogram is searched to obtain the distance between each parallel plane and the origin in a certain normal vector direction.

Through the steps, the parameters P of each plane in the actual scene corresponding to the depth image are obtained_iUsing MeanShift algorithm on P_iAnd optimizing the data of the block to eliminate errors caused by rasterization on plane parameters. Reuse of K-means algorithm with threshold value with P_iFor the initial seed point, P is marked while the corresponding plane parameters are marked for each block_iAnd (6) optimizing.

The normal vector optimization effect is shown in fig. 6, in the diagram, white point cloud is scene point cloud corresponding to a depth image, blue thin lines and red thin lines represent tangent plane directions of each block, yellow long lines are normal vector directions of a previous plane after optimization, pink thick lines are normal vector directions of the plane after optimization, and it can be observed that a result before optimization is not completely perpendicular to a plane in an actual scene, and a result after optimization is perpendicular to a plane corresponding to the actual scene.

The plane segmentation result is shown in fig. 7, the right image is a scene shot by a color camera, the left image is a real-time segmentation point cloud rendering result of the depth map by the algorithm, a red area represents a non-planar area, and other color areas are different planes detected and segmented.

The running time of the algorithm on a Titan X (Pascal) GPU is 1.90 milliseconds under the parallel standard of a CUDA 3.5 computational force architecture, and very high operational efficiency is achieved.

Claims

1. A real-time plane detection and extraction method based on depth images is characterized by comprising the following steps:

step 1: preprocessing the depth image, and extracting block information from the depth image;

and step 3: searching a parallel plane for each estimated plane normal vector;

and 4, step 4: optimizing plane parameters;

the step 1 is specifically realized as follows:

1-3, back projecting pixels in the rectangular block to a three-dimensional space to obtain three-dimensional point cloud, and extracting tangent plane information of the block, wherein the tangent plane information comprises space coordinates of the block, a normal vector of the tangent plane, a distance between the tangent plane and an origin point and block confidence; marking the (u, v) th block as patch (u, v);

the step 2 is realized as follows:

2-1, performing ball polar mapping on the normal vector of each block from a three-dimensional space nx-ny-nz to a two-dimensional plane Px-Py;

2-3, searching an 8-adjacent domain maximum value for the Px-Py histogram, and performing ball polar inverse mapping on the Px-Py coordinates of the corresponding points to obtain estimated plane normal vector parameters;

the step 3 is realized as follows:

the step 4 is realized as follows:

let the ith plane parameter be P_iThe normal vector of the plane and the distance between the plane and the origin are included;

p optimization Using Meanshift_iAnd eliminating the grid error of the histogram statistics, wherein the single iteration process comprises the following steps: searching all blocks in the mean radius and calculating the mean value to obtain a new P_i；

Then using K-means to query each blockApproximate plane parameters and optimize P by query result_i。

2. The method for real-time plane detection and extraction based on depth image as claimed in claim 1, wherein step 2-1 is implemented as follows:

the spherical pole mapping is carried out on the normal vector of each block, and the mapping relation is as follows:

wherein p is_x,p_yFor the mapped Px-Py coordinates, the normal vector of the plane is (n)_x,n_y,n_z)。