CN110675360B - Real-time plane detection and extraction method based on depth image - Google Patents
Real-time plane detection and extraction method based on depth image Download PDFInfo
- Publication number
- CN110675360B CN110675360B CN201910712025.7A CN201910712025A CN110675360B CN 110675360 B CN110675360 B CN 110675360B CN 201910712025 A CN201910712025 A CN 201910712025A CN 110675360 B CN110675360 B CN 110675360B
- Authority
- CN
- China
- Prior art keywords
- plane
- normal vector
- block
- depth image
- dimensional
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/0002—Inspection of images, e.g. flaw detection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/11—Region-based segmentation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/136—Segmentation; Edge detection involving thresholding
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/50—Depth or shape recovery
- G06T7/55—Depth or shape recovery from multiple images
- G06T7/593—Depth or shape recovery from multiple images from stereo images
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10028—Range image; Depth image; 3D point clouds
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20021—Dividing image into blocks, subimages or windows
Landscapes
- Engineering & Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Quality & Reliability (AREA)
- Image Analysis (AREA)
- Image Processing (AREA)
Abstract
The invention discloses a real-time plane detection and extraction method based on a depth image. The invention comprises the following steps: step 1: preprocessing the depth image, and extracting block information from the depth image; step 2: performing dimensionality reduction statistics on the geometric parameter distribution of the block; and step 3: searching a parallel plane for each estimated plane normal vector; and 4, step 4: and (5) optimizing plane parameters. According to the imaging principle of the depth image and the spatial characteristics of the plane structure, the computer can acquire the plane structure information in the environment in real time through the depth image, so that the three-dimensional reconstruction result is more accurate. Therefore, the plane structure information can be accurately and efficiently extracted from the depth image.
Description
Technical Field
The invention relates to the fields of robot positioning and three-dimensional reconstruction, in particular to a plane structure detection and segmentation method of a depth image.
Background
At the earliest 2010, Microsoft corporation developed a low-cost structured light depth camera that was able to acquire accurate depth images in a small range of scenes in the room, thereby enabling reconstruction of scenes in the room. Microsoft shows a Kinectfusion real-time reconstruction algorithm in a SIGGRAPH (Special Interest Group on Graphics and Interactive technologies, short for Graphics and inter-technology Special fan Group) meeting in 2011, Newcomb is the main responsible person of the algorithm project, and the algorithm is the first algorithm for realizing real-time dense scene reconstruction. And Izadi completes the dynamic interaction of the scene and the human on the basis of the above.
In 2016, Whelan proposed a method for detecting map loops using color image features, which simultaneously was able to detect the position of a single point source in space. This article is also representative of the binning expression model (Surfel Representation).
The technical method in the field of off-line (non-real-time) dense reconstruction also has a very profound influence on real-time dense reconstruction, and the Zhou Q Y team makes a huge contribution in the field, and in 2012, the team proposes a dense reconstruction method based on interest points.
However, at present, because GPS signals cannot be received indoors, there is no drift-free indoor positioning and reconstruction algorithm suitable for real scenes, and in recent research on drift-free indoor positioning and reconstruction algorithms in the field, the core lies in detecting and dividing planes shot by a depth camera as reference points to improve positioning accuracy. Heretofore, the method capable of extracting planes from a depth map in real time is based only on the assumption of manhattan planes, that is, assuming that all planes in space are orthogonal to each other, and there is no concern about the presence of non-orthogonal planes.
Disclosure of Invention
The invention provides a real-time and parallelizable depth image-based plane detection and segmentation method.
A real-time plane detection and extraction method based on a depth image can accurately extract plane information from a depth image with a non-orthogonal plane in a scene, and comprises the following steps:
step 1: preprocessing the depth image and extracting block information from the depth image, which comprises the following steps:
1-1, performing filtering processing on the depth image by using a bilateral filtering algorithm;
1-2, dividing the depth image into a plurality of blocks with equal-size rectangles;
1-3, as shown in fig. 2, the pixels in the rectangular block are back projected to a three-dimensional space to obtain a three-dimensional point cloud, and the tangent plane information of the block is extracted, wherein the tangent plane information comprises the space coordinate of the block, the normal vector of the tangent plane, the distance between the tangent plane and the origin, and the block confidence coefficient. Note that the (u, v) th block is patch (u, v).
Step 2: performing dimensionality reduction statistics on the geometric parameter distribution of the block;
2-1, as shown in FIG. 3, performing spherical polar mapping on the normal vector of each block from the three-dimensional space nx-ny-nz to the two-dimensional plane Px-Py;
2-2, carrying out grid division on the normal vector points mapped on the two-dimensional plane Px-Py, and carrying out histogram statistics on the normal vector points on the two-dimensional plane Px-Py in a Gaussian weighting mode to obtain a Px-Py histogram;
and 2-3, searching 8-domain maximum values for the Px-Py histogram, and performing ball polar inverse mapping on the Px-Py coordinates of the corresponding points to obtain estimated plane normal vector parameters.
And step 3: searching a parallel plane for each estimated plane normal vector;
3-1, performing inner product on the space coordinates of all blocks and the estimated plane normal vector, namely projecting the space coordinates of the blocks to the normal vector direction;
3-2, carrying out histogram statistics on the block distribution obtained by projection, and searching a maximum value to obtain the distance between each parallel plane on the normal vector and the origin;
and 4, step 4: optimizing plane parameters, and recording the ith plane parameter as PiThe normal vector of the plane and the distance between the plane and the origin are included;
4-1. optimization of P Using MeanShiftiAnd eliminating the grid error of the histogram statistics, wherein the single iteration process comprises the following steps: searching all blocks in the mean radius and calculating the mean value to obtain a new Pi。
4-2, using K-means to inquire the nearest plane parameter for each block, and optimizing P by the inquiry resulti。
The invention has the following beneficial effects:
according to the imaging principle of the depth image and the spatial characteristics of the plane structure, the computer can acquire the plane structure information in the environment in real time through the depth image, so that the three-dimensional reconstruction result is more accurate. Therefore, the plane structure information can be accurately and efficiently extracted from the depth image.
The traditional method for extracting the plane from the depth map is only based on the assumption of Manhattan planes, that is, all planes in the space are assumed to be mutually orthogonal, and the situation that non-orthogonal planes exist is not involved. The condition that the planes are orthogonal to each other has an influence on the depth map plane extraction algorithm mainly in real time and accuracy. The invention provides a real-time, parallel and efficient depth map plane detection and extraction algorithm, and a plurality of planes can be in a non-orthogonal geometric relationship.
In a traditional method for extracting a plane from a depth map, main plane information needs to be extracted, that is, the largest plane in the depth map is extracted first, and then the remaining planes are extracted through a priori condition that all planes in a space are mutually orthogonal. While in reality, the planes in space may be non-orthogonal to each other. Therefore, the method and the device can extract the geometric information of all planes in the scene from the depth map, and the algorithm result is more accurate and comprehensive.
Drawings
FIG. 1 is a flow chart of planar normal vector detection;
FIG. 2 is a block partitioning and block information diagram;
FIG. 3 is a ball polar map;
fig. 4 is a two-dimensional statistical histogram.
FIG. 5 is a parallel plane query flow diagram.
Fig. 6 is a diagram of the effect of planar normal vector optimization.
Fig. 7 is a rendering diagram of a plane segmentation result.
Detailed Description
The objects and effects of the present invention will become more apparent from the following detailed description of the present invention with reference to the accompanying drawings.
Fig. 1 is a flow chart of plane normal vector detection, and fig. 2 to 4 are views of the visualization process of fig. 1. The part is a part for detecting a plane normal vector by the algorithm, and mainly describes how to detect the normal vector of a plane in a scene from an input depth image. The following is a detailed description of fig. 1 to 4:
first, as shown in FIG. 2, the input depth image is preprocessed and bilateral filtering is used to initially filter out the sensorThe depth image is divided into several blocks of equal size, here, 16 × 16 blocks are taken as an example. Each block contains a normal vector (n)x,ny,nz) The average positions (x, y, z) of all the points in the block can be calculated by back projecting all the pixels of each block into a three-dimensional space to form a point cloud, and the tangent plane normal vector (n) of the block can be calculated by fitting a plane to all the points in the blockx,ny,nz) And the number of vertices in the block that fit the tangent plane distribution, i.e., the number of significant points c, can be found to be equivalent to the confidence of the planarity property of the block.
Secondly, as shown in fig. 3, a spherical pole mapping is performed on the normal vector of each block, and the mapping relationship is as follows:
wherein p isx,pyIs the Px-Py coordinate after mapping. The spherical surface in the figure is a unit spherical surface, the blue color point on the spherical surface is the normal vector distribution point of each block, and the red color point on the Px-Py is the point of the normal vector point mapped to the Px-Py plane. The red bold line is the connecting line from the highest point of the sphere to a certain normal vector point and the mapping point.
Thirdly, as shown in fig. 4, the result mapped to Px-Py in the second step is rasterized and segmented, histogram statistics is performed in a gaussian weighting mode to obtain two-dimensional histogram distribution in fig. 4, and the maximum value in the two-dimensional histogram and the coordinate of the maximum value in the Px-Py coordinate system are searched in the 8-field to obtain the coordinate (p) of the plane in the depth image scene, which is mapped to the Px-Py coordinate systemx,py). By calculating the inverse sphere polar mapping of the coordinates, the normal vector (n) of the plane can be calculatedx,ny,nz)。
As shown in fig. 5, the normal vector of the plane can be obtained by the first three steps of calculation, each block is sequentially projected to the direction of the normal vector to obtain the distance from the block to the origin in the normal vector direction, the histogram distribution of each block in each normal vector direction is obtained by discretizing the distance from each block to the origin and performing gaussian weighted histogram statistics, and the maximum value of the histogram is searched to obtain the distance between each parallel plane and the origin in a certain normal vector direction.
Through the steps, the parameters P of each plane in the actual scene corresponding to the depth image are obtainediUsing MeanShift algorithm on PiAnd optimizing the data of the block to eliminate errors caused by rasterization on plane parameters. Reuse of K-means algorithm with threshold value with PiFor the initial seed point, P is marked while the corresponding plane parameters are marked for each blockiAnd (6) optimizing.
The normal vector optimization effect is shown in fig. 6, in the diagram, white point cloud is scene point cloud corresponding to a depth image, blue thin lines and red thin lines represent tangent plane directions of each block, yellow long lines are normal vector directions of a previous plane after optimization, pink thick lines are normal vector directions of the plane after optimization, and it can be observed that a result before optimization is not completely perpendicular to a plane in an actual scene, and a result after optimization is perpendicular to a plane corresponding to the actual scene.
The plane segmentation result is shown in fig. 7, the right image is a scene shot by a color camera, the left image is a real-time segmentation point cloud rendering result of the depth map by the algorithm, a red area represents a non-planar area, and other color areas are different planes detected and segmented.
The running time of the algorithm on a Titan X (Pascal) GPU is 1.90 milliseconds under the parallel standard of a CUDA 3.5 computational force architecture, and very high operational efficiency is achieved.
Claims (2)
1. A real-time plane detection and extraction method based on depth images is characterized by comprising the following steps:
step 1: preprocessing the depth image, and extracting block information from the depth image;
step 2: performing dimensionality reduction statistics on the geometric parameter distribution of the block;
and step 3: searching a parallel plane for each estimated plane normal vector;
and 4, step 4: optimizing plane parameters;
the step 1 is specifically realized as follows:
1-1, performing filtering processing on the depth image by using a bilateral filtering algorithm;
1-2, dividing the depth image into a plurality of blocks with equal-size rectangles;
1-3, back projecting pixels in the rectangular block to a three-dimensional space to obtain three-dimensional point cloud, and extracting tangent plane information of the block, wherein the tangent plane information comprises space coordinates of the block, a normal vector of the tangent plane, a distance between the tangent plane and an origin point and block confidence; marking the (u, v) th block as patch (u, v);
the step 2 is realized as follows:
2-1, performing ball polar mapping on the normal vector of each block from a three-dimensional space nx-ny-nz to a two-dimensional plane Px-Py;
2-2, carrying out grid division on the normal vector points mapped on the two-dimensional plane Px-Py, and carrying out histogram statistics on the normal vector points on the two-dimensional plane Px-Py in a Gaussian weighting mode to obtain a Px-Py histogram;
2-3, searching an 8-adjacent domain maximum value for the Px-Py histogram, and performing ball polar inverse mapping on the Px-Py coordinates of the corresponding points to obtain estimated plane normal vector parameters;
the step 3 is realized as follows:
3-1, performing inner product on the space coordinates of all blocks and the estimated plane normal vector, namely projecting the space coordinates of the blocks to the normal vector direction;
3-2, carrying out histogram statistics on the block distribution obtained by projection, and searching a maximum value to obtain the distance between each parallel plane on the normal vector and the origin;
the step 4 is realized as follows:
let the ith plane parameter be PiThe normal vector of the plane and the distance between the plane and the origin are included;
p optimization Using MeanshiftiAnd eliminating the grid error of the histogram statistics, wherein the single iteration process comprises the following steps: searching all blocks in the mean radius and calculating the mean value to obtain a new Pi;
Then using K-means to query each blockApproximate plane parameters and optimize P by query resulti。
2. The method for real-time plane detection and extraction based on depth image as claimed in claim 1, wherein step 2-1 is implemented as follows:
the spherical pole mapping is carried out on the normal vector of each block, and the mapping relation is as follows:
wherein p isx,pyFor the mapped Px-Py coordinates, the normal vector of the plane is (n)x,ny,nz)。
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201910712025.7A CN110675360B (en) | 2019-08-02 | 2019-08-02 | Real-time plane detection and extraction method based on depth image |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201910712025.7A CN110675360B (en) | 2019-08-02 | 2019-08-02 | Real-time plane detection and extraction method based on depth image |
Publications (2)
Publication Number | Publication Date |
---|---|
CN110675360A CN110675360A (en) | 2020-01-10 |
CN110675360B true CN110675360B (en) | 2022-04-01 |
Family
ID=69068679
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN201910712025.7A Active CN110675360B (en) | 2019-08-02 | 2019-08-02 | Real-time plane detection and extraction method based on depth image |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN110675360B (en) |
Families Citing this family (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN111862185A (en) * | 2020-07-24 | 2020-10-30 | 唯羲科技有限公司 | Method for extracting plane from image |
Family Cites Families (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN103106651B (en) * | 2012-07-16 | 2015-06-24 | 清华大学深圳研究生院 | Method for obtaining parallax error plane based on three-dimensional hough |
JP2016038886A (en) * | 2014-08-11 | 2016-03-22 | ソニー株式会社 | Information processing apparatus and information processing method |
CN105825499A (en) * | 2016-03-09 | 2016-08-03 | 京东方科技集团股份有限公司 | Reference plane determination method and determination system |
CN109544677B (en) * | 2018-10-30 | 2020-12-25 | 山东大学 | Indoor scene main structure reconstruction method and system based on depth image key frame |
CN110070570B (en) * | 2019-03-20 | 2023-05-26 | 重庆邮电大学 | Obstacle detection system and method based on depth information |
-
2019
- 2019-08-02 CN CN201910712025.7A patent/CN110675360B/en active Active
Also Published As
Publication number | Publication date |
---|---|
CN110675360A (en) | 2020-01-10 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US10977818B2 (en) | Machine learning based model localization system | |
US11727661B2 (en) | Method and system for determining at least one property related to at least part of a real environment | |
AU2018292610B2 (en) | Method and system for performing simultaneous localization and mapping using convolutional image transformation | |
KR102647351B1 (en) | Modeling method and modeling apparatus using 3d point cloud | |
US9053571B2 (en) | Generating computer models of 3D objects | |
CN111986307A (en) | 3D object reconstruction using photometric grid representation | |
CN110633628B (en) | RGB image scene three-dimensional model reconstruction method based on artificial neural network | |
CN103414861A (en) | Method for self-adaptation geometric correction of projector picture | |
EP3422709A1 (en) | Stereo rendering | |
CN112614173A (en) | Real-time plane detection and extraction method based on depth image | |
Boom et al. | Interactive light source position estimation for augmented reality with an RGB‐D camera | |
CN110675360B (en) | Real-time plane detection and extraction method based on depth image | |
WO2018140223A1 (en) | Stereo rendering | |
CN107274477B (en) | Background modeling method based on three-dimensional space surface layer | |
Lee et al. | Estimation of illuminants for plausible lighting in augmented reality | |
US20220392121A1 (en) | Method for Improved Handling of Texture Data For Texturing and Other Image Processing Tasks | |
KR102129060B1 (en) | Content-based 3d model retrieval method using a single depth image, 3d model retrieval server for performing the methods and computer readable recording medium thereof | |
Kunert et al. | Efficient point cloud rasterization for real time volumetric integration in mixed reality applications | |
Villa-Uriol et al. | Automatic creation of three-dimensional avatars | |
Yoshimoto et al. | Cubistic representation for real-time 3D shape and pose estimation of unknown rigid object | |
KR101179969B1 (en) | Device and method for detecting marker | |
Ru et al. | Action recognition based on binocular vision. | |
Wu et al. | A novel augmented reality framework based on monocular semi‐dense simultaneous localization and mapping | |
Petit et al. | Deformed reality | |
Chenwei | Target Workpiece Recognition and Localization Based on Monocular Vision |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication | ||
SE01 | Entry into force of request for substantive examination | ||
SE01 | Entry into force of request for substantive examination | ||
GR01 | Patent grant | ||
GR01 | Patent grant |