WO2021147548A1

WO2021147548A1 - Three-dimensional reconstruction method, detection method and system for small obstacle, and robot and medium

Info

Publication number: WO2021147548A1
Application number: PCT/CN2020/135028
Authority: WO
Inventors: 刘勇; 朱俊安; 黄寅; 郭璁
Original assignee: 深圳市普渡科技有限公司
Priority date: 2020-01-20
Filing date: 2020-12-09
Publication date: 2021-07-29
Also published as: CN111260773B; CN111260773A

Abstract

The present invention provides a three-dimensional reconstruction method, detection method and detection system for a small obstacle. The method comprises: obtaining ground images respectively by means of a binocular camera and a structured light depth camera; performing dense reconstruction on backgrounds occupying main parts of the ground images, and performing, using the binocular camera, sparse feature reconstruction on image positions having a large gradient; extracting a three-dimensional point cloud by means of visual processing technology, and performing, using a "background subtraction" detection method, separation and detection on the point cloud of a small obstacle; mapping the point cloud of the small obstacle to an image, and performing image segmentation to obtain a target image; and performing three-dimensional reconstruction on the target image using a fusion scheme to obtain a complete dense point cloud. According to the three-dimensional reconstruction method, detection method and detection system for a small obstacle provided by the present invention, accurate three-dimensional reconstruction can be achieved, and therefore, the method provides a guarantee for the accuracy of small obstacle detection.

Description

Three-dimensional reconstruction method, detection method and system, robot and medium of small obstacle

This application is based on the Chinese invention patent application filed on January 20, 2020 with the application number 202010064033.8, titled "Three-dimensional reconstruction method, detection method and detection system for small obstacles", and claims priority.

Technical field

The invention relates to the technical field of robots, in particular to a three-dimensional reconstruction method, a detection method and a system, a robot and a medium for small obstacles.

Background technique

Obstacle detection is an important part of autonomous robot navigation and has been extensively studied in the field of robotics and vision, and has been applied in some consumer-grade products. If small obstacles cannot be accurately detected, the safety of the robot's movement will be affected. However, in the actual environment, due to the various types and small size of small obstacles, the detection of small obstacles is still a challenge.

Summary of the invention

Obstacle detection in the robot scene generally obtains the three-dimensional information (such as three-dimensional position, contour) of the target through distance acquisition sensors or algorithms. On the one hand, due to its small size, small obstacles need to obtain more accurate three-dimensional information during detection, which requires higher measurement accuracy and resolution of sensors and algorithms. Active sensors such as radar or sonar have high measurement accuracy, but the resolution is low; depth cameras based on infrared structured light can achieve higher resolution, but they are susceptible to sunlight interference. When the interference is large, the imaging has holes and imaging There is insufficient robustness when the target is small. Passive sensors, such as binocular stereo matching or sequential image stereo matching, if indiscriminate dense reconstruction is used, there will be a large amount of calculation, and it will be difficult to reconstruct small targets, especially when there is a lot of background noise. On the other hand, there are many types of small obstacles, and detection schemes and algorithms are required to be applicable to various types of small obstacles. If the obstacle is directly detected, then the detection object needs to be defined in advance, and the robustness has certain limitations.

In view of this, the purpose of the present invention is to provide a three-dimensional reconstruction method, detection method and system, robot and medium for small obstacles, which improve the robustness and accuracy of the scheme and algorithm, thereby improving the detection of various small obstacles. accuracy.

In order to achieve the foregoing objectives, the embodiments of the present invention provide the following technical solutions:

The present invention provides a method for three-dimensional reconstruction of small obstacles. The method is used to perform three-dimensional reconstruction of small obstacles on the ground. The method includes:

Obtain ground images through binocular camera and structured light depth camera,

Performing dense reconstruction on the background of the main body of the ground image, and using the binocular camera to perform sparse feature reconstruction on image positions with larger gradients;

Extracting a three-dimensional point cloud through visual processing technology, and using a "background reduction" detection method to separate and detect the point cloud of the small obstacle;

Map the point cloud of the small obstacle to an image, and perform image segmentation to obtain a target image;

A fusion scheme is used to perform three-dimensional reconstruction on the target image to obtain a complete dense point cloud.

In this case, by performing dense reconstruction on the background of the main body of the ground image, and using the "background reduction" detection method to separate and detect the point cloud of the small obstacle, the overall robustness of the method is improved And by mapping small obstacles to the image and performing image segmentation, small obstacles can be completely segmented to achieve accurate three-dimensional reconstruction. Therefore, this method provides a guarantee for the accuracy of small obstacle detection.

Wherein, the binocular camera includes a left camera and a right camera, the ground image acquired by the binocular camera includes a left image and a right image, and the ground image acquired by the structured light depth camera is a structured light depth map.

Wherein, the three-dimensional reconstruction of the target image by the fusion scheme to obtain a complete dense point cloud specifically includes:

Sensor calibration includes internal parameter distortion calibration, external parameter calibration of the binocular camera, and external parameter calibration of the binocular camera and the structured light depth camera;

Distortion and epipolar correction, performing distortion and epipolar correction on the left and right images;

Data alignment, using the external parameters of the structured light depth camera to align the structured light depth map to the coordinate system of the left image and the right image to obtain a binocular depth map;

Sparse stereo matching, performing sparse stereo matching on the hollow part of the structured light depth map and obtaining disparity, converting the disparity into depth, using the depth and fusing the structured light depth map and the binocular depth map, Rebuild a robust depth map.

Therefore, there is no need to match the full image stereo, but only the sparse stereo matching of the hollow part of the structured light depth map, which significantly reduces the calculation amount of 3D reconstruction as a whole, and improves the robustness of 3D reconstruction of small obstacles. Awesome.

Wherein, the operation of sparse stereo matching specifically includes:

The hole mask is extracted, sparse stereo matching is performed on the image in the hole mask, and the parallax is obtained.

Wherein, the distortion and epipolar correction operations specifically include:

The matching points used for sparse stereo matching are constrained to be aligned on a horizontal straight line.

In this case, the time for subsequent stereo matching is significantly reduced and the accuracy is greatly improved.

Wherein, the operation of sparse stereo matching further includes:

The robust depth map is divided into blocks, the depth map in each block is converted into the point cloud, and the point cloud is ground-fitted with a plane model. If the point cloud in the block is If the plane assumption is not satisfied, remove the point cloud in the block, otherwise keep the point cloud in the block;

The reserved blocks will be checked twice through the deep neural network, and the blocks that have passed the second check will be grown based on the plane normal and the center of gravity, and the three-dimensional plane equations and boundaries of a large piece of ground will be segmented Point cloud

Obtain the distances from all the point clouds in the block that have not passed the secondary verification to the ground to which they belong, and if the distance is greater than a threshold, segment them to obtain suspected obstacles;

Map the point cloud of the suspected obstacle to the image as a seed point for region segmentation, grow the seed point, and extract a complete obstacle region;

The obstacle area is mapped into a complete point cloud to complete the three-dimensional detection of the small obstacle.

In this case, the depth map in the block that conforms to the plane model but is not on the ground can be excluded through the secondary verification, thereby improving the accuracy of the detection of small obstacles as a whole.

The present invention also provides a method for detecting small obstacles, which includes the three-dimensional reconstruction method for small obstacles as described above.

The present invention also provides a detection system for small obstacles, characterized in that the detection system includes the above-mentioned three-dimensional reconstruction method for small obstacles.

Wherein, it further includes a robot body, the binocular camera and the structured light depth camera are arranged on the robot body, and the binocular camera and the structured light depth camera are arranged obliquely toward the direction of the ground.

A robot includes a processor and a memory, and a computer program is stored in the memory, and the processor is configured to execute the computer program to implement the above-mentioned three-dimensional reconstruction method for small obstacles.

A computer storage medium, the computer storage medium stores a computer program, and when the computer program is executed, the method for realizing the three-dimensional reconstruction of small obstacles as described above is executed.

In this case, the coverage area of the image collected by the binocular camera and the structured light depth camera on the ground is improved, and the completeness of the recognition of small obstacles is significantly improved.

According to the three-dimensional reconstruction method, detection method, and detection system of small obstacles provided by the present invention, dense reconstruction is performed on the background of the main body of the ground image, and the detection method of "background reduction" is used to detect the small obstacles. The point cloud is separated and detected, which improves the robustness of the method as a whole. By mapping small obstacles to the image and performing image segmentation, the small obstacles can be completely segmented to achieve accurate three-dimensional reconstruction. Therefore, this method Provide a guarantee for the accuracy of small obstacle detection.

Description of the drawings

FIG. 1 shows a schematic flowchart of a method for 3D reconstruction of small obstacles according to an embodiment of the present invention;

2 shows a schematic flowchart of a depth map processing method of a three-dimensional reconstruction method for small obstacles according to an embodiment of the present invention;

Fig. 3 shows a schematic flow chart of sparse stereo matching of the three-dimensional reconstruction method for small obstacles according to the embodiment of the present invention.

Detailed ways

Hereinafter, the preferred embodiments of the present invention will be described in detail with reference to the drawings. In the following description, the same symbols are assigned to the same components, and repeated descriptions are omitted. In addition, the drawings are only schematic diagrams, and the ratio of dimensions between components or the shapes of components may be different from actual ones.

The embodiments of the present invention relate to a three-dimensional reconstruction method, a detection method and a detection system of small obstacles.

As shown in FIG. 1, the method 200 for three-dimensional reconstruction of small obstacles involved in this embodiment is used to perform three-dimensional reconstruction of small obstacles on the ground. The method specifically includes:

201. Obtain ground images through the binocular camera and the structured light depth camera respectively;

202. Perform dense reconstruction on the background of the main body of the ground image, and use the binocular camera to perform sparse feature reconstruction on image positions with larger gradients;

203. Extract a three-dimensional point cloud through a visual processing technology, and use a "background reduction" detection method to separate and detect the point cloud of the small obstacle;

204. Map the point cloud of the small obstacle to an image, and perform image segmentation to obtain a target image;

205. Use a fusion solution to perform three-dimensional reconstruction on the target image to obtain a complete dense point cloud.

In this embodiment, the binocular camera includes a left camera and a right camera, the ground image acquired by the binocular camera includes a left image and a right image, and the ground image acquired by the structured light depth camera is a structure Light depth map.

As shown in FIG. 2, in this embodiment, the three-dimensional reconstruction of the target image by the fusion scheme to obtain a complete dense point cloud specifically includes:

2051. Sensor calibration, including internal parameter distortion calibration and external parameter calibration of the binocular camera, and external parameter calibration of the binocular camera and the structured light depth camera;

2052. Distortion and epipolar correction, performing distortion and epipolar correction on the left image and the right image;

2053. Data alignment, using the external parameters of the structured light depth camera to align the structured light depth map to the coordinate system of the left image and the right image to obtain a binocular depth map;

2054. Sparse stereo matching, performing sparse stereo matching on the hollow part of the structured light depth map and obtaining disparity, converting the disparity into depth, using the depth and fusing the structured light depth map and the binocular depth Figure, reconstruct a robust depth map.

In this embodiment, the operation of the sparse stereo matching (2054) specifically includes:

In this embodiment, the operation of the distortion and epipolar correction (2052) specifically includes:

As shown in FIG. 3, in this embodiment, the operation of the sparse stereo matching (2054) further includes:

2055. Divide the robust depth map into blocks, convert the depth map in each block into the point cloud, and perform ground fitting on the point cloud using a plane model. If the point cloud does not satisfy the plane assumption, remove the point cloud in the block, otherwise keep the point cloud in the block;

2056. Perform a secondary check on the reserved block through a deep neural network, and perform regional growth on the block that has passed the secondary check based on the plane normal and the center of gravity, so as to segment the three-dimensional plane equation of a large piece of ground And boundary point cloud;

2057. Obtain the distances from all the point clouds in the blocks that have not passed the secondary verification to the ground to which they belong, and if the distance is greater than a threshold, segment them to obtain suspected obstacles;

2058. Map the point cloud of the suspected obstacle to the image as a seed point for region segmentation, perform the seed point growth, and extract a complete obstacle region;

2059. Map the obstacle area to a complete point cloud, and complete the three-dimensional detection of the small obstacle.

The embodiment of the present invention also relates to a method for detecting small obstacles. The detection method includes the three-dimensional reconstruction method of small obstacles as described above. Regarding the three-dimensional reconstruction method of small obstacles, I will not go into details here.

The embodiment of the present invention also relates to a detection system for small obstacles. The detection system includes the three-dimensional reconstruction method of small obstacles as described above. Regarding the three-dimensional reconstruction method of small obstacles, I will not go into details here.

In this embodiment, the detection system for small obstacles further includes a robot body, the binocular camera and the structured light depth camera are arranged on the robot body, and the binocular camera and the structured light depth camera face the direction of the ground. Tilt setting.

Optionally, an embodiment of the present invention further provides a robot, which includes a processor, a memory, a network interface, and a database connected through a system bus. Among them, the robot's processor is used to provide calculation and control capabilities. The robot's memory includes non-volatile storage media and internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The robot's network interface is used to communicate with external terminals through a network connection. The computer program is executed by the processor to realize a three-dimensional reconstruction method of small obstacles.

The embodiment of the present invention also provides a computer storage medium, the computer storage medium stores a computer program, and when the computer program is executed, it executes the three-dimensional reconstruction method for realizing the small obstacle as described above.

A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage. In the medium, when the computer program is executed, it may include the processes of the above-mentioned method embodiments. Wherein, any reference to memory, storage, database or other media used in the various embodiments provided by the present invention may include non-volatile and/or volatile memory. Non-volatile memory may include read only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not a limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous chain Channel (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

The above-mentioned embodiments do not constitute a limitation on the protection scope of the technical solution. Any modifications, equivalent replacements and improvements made within the spirit and principles of the above implementation manners should be included in the protection scope of the technical solution.

Claims

A method for three-dimensional reconstruction of small obstacles, characterized in that the method is used to perform three-dimensional reconstruction of small obstacles on the ground, and the method includes:

Obtain ground images through binocular camera and structured light depth camera;

Performing dense reconstruction on the background of the main body of the ground image, and using the binocular camera to perform sparse feature reconstruction on image positions with larger gradients;

Extracting a three-dimensional point cloud through visual processing technology, and using a "background reduction" detection method to separate and detect the point cloud of the small obstacle;

Map the point cloud of the small obstacle to an image, and perform image segmentation to obtain a target image;

A fusion scheme is used to perform three-dimensional reconstruction on the target image to obtain a complete dense point cloud.
The method for 3D reconstruction of a small obstacle according to claim 1, wherein the binocular camera includes a left camera and a right camera, and the ground image acquired by the binocular camera includes a left image and a right image. The ground image acquired by the structured light depth camera is a structured light depth map.
The method for 3D reconstruction of small obstacles according to claim 2, characterized in that said adopting a fusion scheme to perform 3D reconstruction on said target image to obtain a complete dense point cloud specifically comprises:

Sensor calibration includes internal parameter distortion calibration, external parameter calibration of the binocular camera, and external parameter calibration of the binocular camera and the structured light depth camera;

Distortion and epipolar correction, performing distortion and epipolar correction on the left and right images;

Data alignment, using the external parameters of the structured light depth camera to align the structured light depth map to the coordinate system of the left image and the right image to obtain a binocular depth map;

Sparse stereo matching, performing sparse stereo matching on the hollow part of the structured light depth map and obtaining disparity, converting the disparity into depth, using the depth and fusing the structured light depth map and the binocular depth map, Rebuild a robust depth map.
The method for 3D reconstruction of small obstacles according to claim 3, wherein the operation of sparse stereo matching specifically includes:

The hole mask is extracted, sparse stereo matching is performed on the image in the hole mask, and the parallax is obtained.
The method for three-dimensional reconstruction of small obstacles according to claim 3, wherein the operation of distortion and epipolar correction specifically includes:

The matching points used for sparse stereo matching are constrained to be aligned on a horizontal straight line.
The method for 3D reconstruction of small obstacles according to claim 3, wherein the operation of sparse stereo matching further comprises:

The robust depth map is divided into blocks, the depth map in each block is converted into the point cloud, and the point cloud is ground-fitted with a plane model. If the point cloud in the block is If the plane assumption is not satisfied, remove the point cloud in the block, otherwise keep the point cloud in the block;

The reserved blocks will be checked twice through the deep neural network, and the blocks that have passed the second check will be grown based on the plane normal and the center of gravity, and the three-dimensional plane equations and boundaries of a large piece of ground will be segmented Point cloud

Obtain the distances from all the point clouds in the block that have not passed the secondary verification to the ground to which they belong, and if the distance is greater than a threshold, segment them to obtain suspected obstacles;

Map the point cloud of the suspected obstacle to the image as a seed point for region segmentation, grow the seed point, and extract a complete obstacle region;

The obstacle area is mapped into a complete point cloud to complete the three-dimensional detection of the small obstacle.
A method for detecting small obstacles, characterized in that the detecting method comprises the three-dimensional reconstruction method for small obstacles according to any one of claims 1-6.
A detection system for small obstacles, characterized in that the detection system comprises the three-dimensional reconstruction method for small obstacles according to any one of claims 1-6.
The small obstacle detection system of claim 8, further comprising a robot body, the binocular camera and the structured light depth camera are provided on the robot body, the binocular camera and the structured light depth camera It is arranged obliquely towards the direction of the ground.
A robot, characterized in that the robot comprises a processor and a memory, and a computer program is stored in the memory, and the processor is configured to execute the computer program to implement the computer program according to any one of claims 1 to 6 Three-dimensional reconstruction method of small obstacles.
A computer storage medium, wherein the computer storage medium stores a computer program, and when the computer program is executed, the method for realizing the three-dimensional reconstruction of a small obstacle according to any one of claims 1 to 6 is executed.