CN103955920A - Binocular vision obstacle detection method based on three-dimensional point cloud segmentation - Google Patents

Abstract

The invention provides a binocular visual obstacle detection method based on three-dimensional point cloud segmentation, which includes synchronously collecting two camera images of the same specification; performing calibration correction on the binocular camera and calculating the three-dimensional point cloud segmentation threshold; using a stereo matching algorithm and 3D reconstruction calculations to obtain 3D point clouds, image segmentation of the reference map to obtain tiles; automatically detect the road height of the 3D point cloud, use the 3D point cloud segmentation threshold to segment the road surface point cloud, obstacle point cloud at different positions and unknown areas Point cloud: use the segmented point cloud combined with the segmented tiles to determine the correctness of obstacles and road surfaces, and determine the location range of obstacles, road surfaces, and unknown areas. The present invention can still detect the height of the camera and the road surface and automatically estimate the threshold of three-dimensional segmentation in a more complex environment, and segment out obstacle point clouds, road surface point clouds and unknown area point clouds; end the color image segmentation technology, and integrate color information to judge obstacles The correctness of objects and road surfaces, determine the location range of obstacles, road surfaces and unknown areas, and achieve highly robust obstacle detection, with higher reliability and practicability.

Description

Binocular Vision Obstacle Detection Method Based on 3D Point Cloud Segmentation

technical field

The invention relates to the field of automatic detection of obstacles based on binocular stereo vision, such as autonomous mobile robots and automatic driving, in particular to a binocular vision obstacle detection method based on three-dimensional point cloud segmentation.

Background technique

Binocular stereo vision is an important branch of computer vision. This vision directly simulates the way human eyes process scenes. It is simple and reliable, and has great application value in many fields, such as robot navigation and aerial survey, three-dimensional measurement, intelligent transportation and virtual reality. wait. Binocular stereo vision is to shoot the same scene by two or one camera in different positions after moving or rotating, and obtain the three-dimensional coordinate value of the point by calculating the parallax of the spatial point in the two images. In the research of autonomous mobile robots and autonomous driving, the most basic condition is to require robots or autonomous vehicles to be equipped with vision devices. The vision system should be able to complete the basic functions similar to the human eye, and provide reliable visual information for path planning. Object detection of obstacles and acquisition of distance information are key parts in autonomous mobile robots and autonomous driving. The key to obstacle detection based on binocular stereo vision is: (1) Stereo matching of binocular image pairs to calculate 3D point cloud; (2) Obstacle target extraction, that is, to identify the position and size of obstacles. Typically applications such as autonomous mobile robots and autonomous driving have complex environments affected by optical distortion and noise, specular reflections from smooth surfaces, projection reduction, perspective distortion, low texture, repetitive texture, transparent objects, and overlapping and non-contiguous areas, and cannot be guaranteed A dense disparity map can be obtained through stereo matching calculation. In addition, in a complex road environment, only relying on road color experience or road edge detection to detect roads in a single image is not accurate enough and cannot be directly applied to practical situations.

With the continuous and in-depth research on machine vision, many stereo matching algorithms that are efficient and can obtain denser disparity maps have been applied to actual scenes, such as 3D reconstruction, intelligent transportation, and intelligent detection. Image segmentation is the technology and process of dividing an image into several specific regions with unique properties and proposing objects of interest. It is a key step from image processing to image analysis. Researchers have continuously improved the original color image segmentation methods and applied some new theories and methods from other disciplines to image segmentation, and proposed many new segmentation methods suitable for specific occasions.

Contents of the invention

The purpose of the present invention is to provide a binocular vision obstacle detection method based on three-dimensional point cloud segmentation, the method preliminarily segment the road surface point cloud and the obstacle point cloud above the ground on the three-dimensional point cloud, and then combine the color image Segmentation algorithms fuse decisions to finally identify obstacles.

The technical scheme that realizes the object of the present invention is:

A method for automatic detection of obstacles based on binocular stereo vision based on three-dimensional point cloud segmentation, comprising the steps of:

(1) Obtain two color images through two cameras in different positions, use the stereo calibration method to calibrate the binocular camera, calculate the internal and external parameters and relative positional relationship of the two cameras, and eliminate the distortion of the two cameras according to these parameters Align with the row (or column) so that the imaging origin coordinates of the two color images are consistent, and a corrected binocular color view is obtained. Obtain or predetermine the pitch angle and height of the camera relative to the road surface by the sensor;

(2) For the binocular color views obtained after step (1) correction, select one of the views as the reference image. Use the stereo matching algorithm to perform matching calculation on the reference image to obtain the preliminary disparity image. Carry out color image segmentation on the reference image to obtain blocks with similar colors. According to the position relationship data of the camera relative to the road surface obtained in step (1), calculate the threshold of point cloud segmentation, including the shortest distance, the furthest distance, the maximum height and the maximum width of the obstacle that can be detected with a certain accuracy.

(3) After the disparity map calculated in step (2), use the relative positional relationship data (reprojection matrix) of the two cameras calculated in step (1) to perform 3D reconstruction of the disparity map to calculate a 3D point cloud.

(4) According to the position data of the camera relative to the ground obtained in step (1), correct the 3D point cloud calculated by step (3), so that the height of the road surface point cloud relative to the camera is consistent.

(5) Detect the road surface height (the height of the road surface relative to the camera) from the 3D point cloud corrected in step (4), and segment obstacles, road surfaces and unknown point clouds by the segmentation threshold calculated in step (2).

(6) Fusion the point cloud segmented in step (5) and the segmented block of the reference map image in step (2), determine obstacles, road surfaces and unknown areas, and mark them out.

Throughout the above process, the two cameras mentioned above always maintain the same specifications, that is, the performance of pixels, focal lengths, apertures, etc. are the same or close. Considering the factors that affect the stereo matching calculation, try to choose a high-definition camera. The relative position and focal length of the two cameras are fixed, that is, once calibrated, the relative position and focal length of the two cameras will not be changed. The pitch angle and height of the two cameras relative to the road surface can be fixed or variable. In step (1), the resolutions of the two cameras are determined based on the computing performance of the device, and usually a resolution of 640x480 or 1027x768 can be used. The calibration camera can be manually calibrated, plane checkerboard calibration or circle (plane circle or parallel circle) calibration, etc., to calculate tangential distortion, radial distortion, focal length, imaging origin, correction mapping matrix and reprojection matrix. The correction mapping is used to correct the color views of the two cameras, so that the coordinates of the origin of the two images are consistent, the optical axes of the two cameras are parallel, the left and right imaging planes are coplanar, and the epipolar lines are aligned. The position of the camera relative to the road surface includes a pitch angle and a height, where the height is optional.

The reference image selected in step (2) is the main image that determines the reference for subsequent calculations. At present, the stereo matching algorithm is a difficult and hot spot in computer vision. There are many algorithms. The general steps include matching cost calculation, matching cost superposition, parallax acquisition, and parallax refinement (sub-pixel level). The matching strategies include local algorithms and global algorithm. Considering the computing performance and accuracy requirements of the comprehensive equipment, an appropriate stereo matching algorithm is selected to efficiently and quickly calculate a denser disparity map. Color image segmentation is to segment the color image according to the color information. According to the relative distance of the two cameras and the focal length of the reference camera, the shortest distance from the camera, the furthest distance from the camera, the maximum height and the maximum width of the obstacle detectable with acceptable accuracy can be determined.

The 3D point cloud described in step (3) is obtained by calculating the 3D reconstruction of the disparity map using the reprojection matrix calculated in step (1). The disparity map is based on any image in the image pair, its size is the size of the reference image, and the element value is an image of the disparity value, which contains the distance information of the scene.

In step (4), the three-dimensional point cloud is rotated according to the pitch angle of the camera relative to the ground, so that the height of the road surface point cloud relative to the camera is consistent.

In step (5), segment the 3D point cloud according to the maximum height detected in step (2) or the height obtained in step (1) to obtain the road surface point cloud; Long-distance, segment the 3D point cloud to obtain obstacle point cloud and unknown area point cloud at different distances from the camera; segment the obstacle point cloud at different distances from the camera according to the maximum width detected in step (2), and obtain obstacles at different positions object point cloud. This step is to segment the obstacle point cloud, the road surface point cloud and the unknown area point cloud. The non-area point cloud is composed of some point clouds with wrong matching calculations and point clouds with poor accuracy.

Step (6) The fusion of the point cloud segmented in step (5) and the segmented block of the reference map image in step (2), that is, the image block obtained by segmenting the color image in step (2) with the obstacle point cloud and the road surface point cloud Compare and judge the correctness and area range of obstacles and road surfaces.

The binocular camera calibration: calibration includes camera calibration and stereo calibration. Camera calibration mainly calculates tangential distortion, radial distortion, focal length and optical center. Stereo calibration is the process of calculating the geometric relationship between two cameras in space, which depends on finding the rotation matrix and translation vector between the two cameras; the binocular correction: the camera needs correction and stereo correction, and correction is to eliminate tangential and radial distortion. Stereo correction is because it is almost impossible for two cameras to have accurate coplanar and line-aligned imaging planes, and it is necessary to calibrate the rotation matrix and translation vector correction between the two cameras through stereo;

The binocular matching and disparity calculation: the stereo matching is mainly to find out the corresponding relationship between each pair of images, and obtain the disparity map according to the principle of triangulation; after obtaining the disparity information, the original image can be easily obtained according to the projection model depth information and three-dimensional information. Stereo matching technology is generally considered to be the most difficult and critical problem in stereo vision. At present, stereo matching algorithm is a difficult and hot spot in computer vision. There are many algorithms. The general calculation steps are: matching cost calculation, matching cost superposition, parallax Acquisition, parallax refinement;

The automatic estimation segmentation threshold: within the allowable minimum depth accuracy range, calculate the shortest distance, the farthest distance, and the maximum height of the obstacle from the camera according to the distance between the two cameras and the focal length obtained by the camera resolution and stereo calibration and max width. Then, within the allowable accuracy range, the distance between the front, rear and left and right of the obstacle, and the error range of the height of the obstacle are calculated.

The three-dimensional point cloud road height detection: the road height detection is performed on the three-dimensional point cloud obtained by three-dimensional reconstruction, and the height satisfying the set error is regarded as the road height. The error may be large when the road height is detected for the first time. After multiple iterative detections, the detection accuracy of the road height tends to be stable, and it will be updated with the recent height detection value updates.

The 3D point cloud segmentation: after 3D reconstruction obtains a relatively dense 3D point cloud, the 3D point cloud is rotated according to the pitch angle of the camera relative to the ground, so that the road surface point cloud on the 3D point cloud is at the same height as the camera. According to the shortest distance from the obstacle to the camera, the farthest distance from the camera, the maximum height, the maximum width, the height of the road point cloud and the height error of the obstacle, the 3D point cloud is divided to obtain the road point cloud, obstacle point cloud and unknown area point cloud; The obstacle point cloud is obtained by segmenting the obstacle point cloud from the distance between the front, rear and left and right of the obstacle; the point cloud of the unknown area is composed of some point clouds with incorrect matching calculations and point clouds with poor accuracy.

The 3D point cloud fusion color information detects obstacles: the 3D point cloud fusion color information is to segment the 3D point cloud to obtain obstacle point clouds, road surface point clouds and unknown area point clouds at different positions, and perform color-based image analysis on the color reference map. The segmented image blocks are obtained, and the segmented point cloud is fused and compared with the segmented image blocks to judge the correctness of obstacles and road surfaces, determine the location range of obstacles, road surfaces, and unknown areas, and achieve highly reliable obstacle detection.

The advantage of the present invention is: compared with the prior art, the present invention reconstructs a denser 3D point cloud in a more complex environment, and can still detect and automatically estimate the height of the camera and the road when the camera and the road form a certain pitch angle Threshold of 3D segmentation, segment out obstacle point cloud, road surface point cloud and unknown area point cloud; end color image segmentation technology, fuse color information to judge the correctness of obstacles and road surface, and determine the position range of obstacles, road surface and unknown area ; Compared with the method of single image relying on experience to detect road obstacles, it has higher reliability and practicability.

Description of drawings

Figure 1 is a schematic diagram of a binocular visual obstacle detection method based on 3D point cloud segmentation.

Figure 2 is a schematic diagram of binocular stereo imaging.

Detailed ways

An automatic obstacle detection method based on 3D point cloud segmentation and fusion of color information, as shown in Figure 1, includes the following steps:

Step 1: Obtain two color images through two cameras at different positions, use the stereo calibration method to calibrate the binocular camera, calculate the internal and external parameters and relative positional relationship of the two cameras, and eliminate the distortion of the two cameras according to these parameters Align with the row (or column) so that the imaging origin coordinates of the two color images are consistent, and a corrected binocular color view is obtained. The pitch angle and height of the camera relative to the road surface are acquired or predetermined by the sensor. The relative position and focal length of the two cameras are fixed, that is, once calibrated, the relative position and focal length of the two cameras will not be changed. The pitch angle and height of the two cameras relative to the road surface can be fixed or variable.

The resolutions of the two cameras are determined based on the computing performance of the device, and resolutions such as 640x480 or 1027x768 can usually be used. The calibration camera can be manually calibrated, plane checkerboard calibration or circle (plane circle or parallel circle) calibration, etc., to calculate tangential distortion, radial distortion, focal length, imaging origin, correction mapping matrix and reprojection matrix. The correction mapping is used to correct the color views of the two cameras, so that the coordinates of the origin of the two images are consistent, the optical axes of the two cameras are parallel, the left and right imaging planes are coplanar, and the epipolar lines are aligned. The position of the camera relative to the road surface includes a pitch angle and a height, where the height is optional.

In the present invention, the two cameras always maintain the same specifications, that is, the performance of pixels, focal lengths, apertures, etc. is the same or close. Considering the factors affecting the calculation of stereo matching, a high-definition camera should be selected as much as possible.

Here is an example of using the plane checkerboard calibration method to calibrate the left and right horizontally placed binocular cameras. After adjusting the binocular camera and collecting two RGB images, use n rows and m columns of black and white checkerboard to calibrate it (n and m are greater than 4). Due to the characteristics of the optical lens, the imaging of the camera has radial distortion:

x _corrected =x(1+k ₁ r ² +k ₂ r ⁴ +k ₃ r ⁶ ) (1)

y _corrected =y(1+k ₁ r ² +k ₂ r ⁴ +k ₃ r ⁶ )

In the formula, (x, y) is the obtained original image pixel, (x _corrected , y _corrected ) is the image pixel corrected for radial distortion, and the parameters k ₁ , k ₂ , k ₃ are the radial distortion parameters;

Due to the error in assembling the imaging sensor and lens in the process, the sensor and the optical lens are not completely parallel, so there is tangential distortion in the imaging:

x _corrected =x+[2p ₁ y+p ₂ (r ² +2x ² )] (2)

y _corrected =y+[p ₁ (r ² +2y ² )+2p ₂ x]

In the formula (x, y) is the acquired original image pixel, (x _corrected , y _corrected ) is the image pixel corrected for tangential distortion, and the parameters p ₁ and p ₂ are tangential distortion parameters; the calibration of a single camera is mainly to calculate Intrinsic and extrinsic parameters of the camera. Stereo camera calibration not only needs to obtain the internal parameters of each camera, but also needs to measure the relative position (3D translation vector and rotation matrix) between the two cameras through stereo calibration.

Binocular correction is based on the monocular internal reference data (focal length, imaging origin, distortion coefficient) and binocular relative position relationship (rotation matrix and translation vector) obtained after camera calibration, to eliminate distortion and line alignment of the left and right views respectively. The imaging origin coordinates of the left and right views are consistent, the optical axes of the two cameras are parallel, the left and right imaging planes are coplanar, and the epipolar lines are aligned, which facilitates the calculation of stereo matching.

Step 2: For the binocular color views obtained after the correction in Step 1, select one of the views as the reference image. Use the stereo matching algorithm to perform matching calculation on the reference image to obtain the preliminary disparity image. Carry out color image segmentation on the reference image to obtain blocks with similar colors. According to the position relationship data of the camera relative to the road surface obtained in step 1, calculate the threshold of point cloud segmentation, including the shortest distance, the longest distance, the maximum height and the maximum width that can be detected with acceptable accuracy.

Now use the SGBM stereo matching algorithm proposed by Heiko Hirschmüller and optimize it, use the left view as the base image for matching, and calculate the preliminary disparity map. The stereo matching algorithm includes the following steps: first, the matching cost is calculated,

C(p,d)=C(x,y,d)=C _BT (p _Li ,p _Li +d,I′ _L ,I′ _R )

=max{0,I _L (p _Li )-I′ _max ,I′ _min -I _L (p _Li )} (3)

In equation (3), d is the parallax.

The matching cost is superimposed as follows:

$\begin{array}{c}\mathrm{<span\; class="notranslate">\; E.</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; D.</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; p</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}\mathrm{<span\; class="notranslate">\; C</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; p</span>}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; D.</span>}}_{\mathrm{<span\; class="notranslate">\; p</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; +</span>\underset{\mathrm{<span\; class="notranslate">\; q</span>}<span\; class="notranslate">\; \&Element;</span>{\mathrm{<span\; class="notranslate">\; N</span>}}_{\mathrm{<span\; class="notranslate">\; p</span>}}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{\mathrm{<span\; class="notranslate">\; P</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}\mathrm{<span\; class="notranslate">\; T</span>}<span\; class="notranslate">\; [</span><span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; D.</span>}}_{\mathrm{<span\; class="notranslate">\; p</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; D.</span>}}_{\mathrm{<span\; class="notranslate">\; q</span>}}<span\; class="notranslate">\; |</span><span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; ]</span>\\ <span\; class="notranslate">\; +</span>\underset{\mathrm{<span\; class="notranslate">\; q</span>}<span\; class="notranslate">\; \&Element;</span>{\mathrm{<span\; class="notranslate">\; N</span>}}_{\mathrm{<span\; class="notranslate">\; p</span>}}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{\mathrm{<span\; class="notranslate">\; P</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}\mathrm{<span\; class="notranslate">\; T</span>}<span\; class="notranslate">\; [</span><span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; D.</span>}}_{\mathrm{<span\; class="notranslate">\; p</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; D.</span>}}_{\mathrm{<span\; class="notranslate">\; q</span>}}<span\; class="notranslate">\; |</span><span\; class="notranslate">\; ></span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; ]</span>\end{array}<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 4</span>}<span\; class="notranslate">\; )</span>$

In formula (4), D is the disparity map, and the first term on the right represents the sum of matching costs of all pixels.

There are many image segmentation methods, and the color-based mean shift clustering similarity algorithm is used here. Here, the mean shift segmentation can find the peak of the color distribution in space. By giving a set of multiple data whose dimensions are (x, y, blue, green, red), the mean shift can use a window to scan the space to find the data The densest "clumps". Set a space radius, set a color radius according to the color variable. When the mean shift window moves, all points that converge to the peak of the data after window transformation will be connected and belong to the peak. This affiliation, radiating from the dense spikes, forms the segmentation of the image. Segmentation is done by proportional pyramids, where color clusters higher up in the pyramid have their own boundaries defined precisely in the lower levels of the pyramid.

According to the camera resolution, use the following formula to calculate the shortest distance, the longest distance, the maximum height, the maximum width and determine the error range of the obstacle height:

$\mathrm{<span\; class="notranslate">\; \&Delta;Z</span>}<span\; class="notranslate">\; =</span>\frac{{\mathrm{<span\; class="notranslate">\; Z</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}}{\mathrm{<span\; class="notranslate">\; f</span>}}\mathrm{<span\; class="notranslate">\; Vd</span>}<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 5</span>}<span\; class="notranslate">\; )</span>$

Among them, Vz is the accuracy of the minimum trial range obtained, z is the distance between the obstacle and the camera, f is the focal length of the camera, T is the distance between the two cameras, and Vd is the disparity value.

Step 3: After the disparity map calculated in step 2, use the relative position relationship data (reprojection matrix) of the two cameras calculated in step 1 to perform 3D reconstruction of the disparity map to calculate a 3D point cloud.

According to the principle of binocular stereo imaging, as shown in Figure 2:

The baseline distance T is the distance between the projection centers of the two cameras; the focal length of the camera is f.

After stereo correction, the images of the two cameras are on the same plane, the imaging center of the reference image is (c _x , _cy ), the x coordinate of this point on the right image is c′ _x , and the distance between the two cameras in the x direction is T _x , just reproject the matrix under the post:

$\mathrm{<span\; class="notranslate">\; Q</span>}<span\; class="notranslate">\; =</span>\left[\begin{array}{cccc}\mathrm{<span\; class="notranslate">\; 1</span>}& \mathrm{<span\; class="notranslate">\; 0</span>}& \mathrm{<span\; class="notranslate">\; 0</span>}& <span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; c</span>}}_{\mathrm{<span\; class="notranslate">\; x</span>}}\\ \mathrm{<span\; class="notranslate">\; 0</span>}& \mathrm{<span\; class="notranslate">\; 1</span>}& \mathrm{<span\; class="notranslate">\; 0</span>}& {<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; c</span>}}_{\mathrm{<span\; class="notranslate">\; the\; y</span>}}\\ \mathrm{<span\; class="notranslate">\; 0</span>}& \mathrm{<span\; class="notranslate">\; 0</span>}& \mathrm{<span\; class="notranslate">\; 0</span>}& \mathrm{<span\; class="notranslate">\; f</span>}\\ \mathrm{<span\; class="notranslate">\; 0</span>}& \mathrm{<span\; class="notranslate">\; 0</span>}& <span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; /</span>{\mathrm{<span\; class="notranslate">\; T</span>}}_{\mathrm{<span\; class="notranslate">\; x</span>}}& <span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; c</span>}}_{\mathrm{<span\; class="notranslate">\; x</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; c</span>}}_{\mathrm{<span\; class="notranslate">\; x</span>}}^{<span\; class="notranslate">\; \&prime;</span>}{\mathrm{<span\; class="notranslate">\; T</span>}}_{\mathrm{<span\; class="notranslate">\; x</span>}}<span\; class="notranslate">\; )</span>\end{array}\right]<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 6</span>}<span\; class="notranslate">\; )</span>$

Given a 2D homogeneous point and its associated disparity d, the point can be projected into 3D:

$\mathrm{<span\; class="notranslate">\; Q</span>}\left[\begin{array}{c}\mathrm{<span\; class="notranslate">\; x</span>}\\ \mathrm{<span\; class="notranslate">\; the\; y</span>}\\ \mathrm{<span\; class="notranslate">\; d</span>}\\ \mathrm{<span\; class="notranslate">\; 1</span>}\end{array}\right]<span\; class="notranslate">\; =</span>\left[\begin{array}{c}\mathrm{<span\; class="notranslate">\; x</span>}\\ \mathrm{<span\; class="notranslate">\; Y</span>}\\ \mathrm{<span\; class="notranslate">\; Z</span>}\\ \mathrm{<span\; class="notranslate">\; W</span>}\end{array}\right]<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 7</span>}<span\; class="notranslate">\; )</span>$

Therefore, (X/W, Y/W, Z/W) is the three-dimensional coordinates corresponding to the parallax point d.

Step 4: According to the position data of the camera relative to the ground obtained in step 1, correct the 3D point cloud calculated in step 3, so that the height of the road surface point cloud in the 3D point cloud is consistent with that of the camera. After obtaining the pitch angle of the camera relative to the ground, the 3D point cloud can be rotated to make the road height values of the 3D point cloud consistent. Here the binocular camera is set parallel to the road surface.

Step 5: Detect the road surface height (the height of the road surface relative to the camera) from the 3D point cloud corrected in step 4, and segment obstacles, road surfaces and unknown point clouds by the segmentation threshold calculated in step 2.

It is assumed that the road surface is not concave and the objects higher than the road plane are obstacles. By finding the hourly value of the 3D point cloud corrected in step 4, take the average value of all values within the error range of the obstacle height, which is the currently detected road surface height, which is calculated by weighting with the previously detected road surface height The current final road height. After detecting the road surface height several times, the accuracy of the road surface height value is improved.

Step 6: Fusion the point cloud segmented in step 5 and the segmented blocks of the reference map image in step 2, determine obstacles, road surfaces and unknown areas, and mark them out.

The 3D point cloud fusion color information is to segment the 3D point cloud to obtain obstacle point cloud, road surface point cloud and unknown area point cloud at different positions, segment the color reference image by color image to obtain image blocks, and combine the segmented point cloud with The image blocks obtained by segmentation are fused and compared. After the proportion of the block reaches a certain value, it is determined whether the block is an obstacle or a road surface, and the location range of the obstacle, road surface, and unknown area is determined.

The automatic obstacle detection method based on 3D point cloud segmentation and fusion of color information designed by the above method, as shown in Figure 1, includes binocular camera calibration, binocular correction, binocular matching and disparity calculation, and automatic estimation of segmentation threshold. 3D point cloud road height detection, 3D point cloud segmentation, 3D point cloud fusion color information to detect obstacles. in

Binocular camera calibration: Calibration includes camera calibration and stereo calibration. Camera calibration mainly calculates tangential distortion, radial distortion, focal length and optical center. Stereo calibration is the process of calculating the geometric relationship between two cameras in space, relying on finding the rotation matrix and translation vector between the two cameras.

Binocular correction: The camera needs correction and stereo correction, correction is to eliminate tangential and radial distortion. Stereo correction is because it is almost impossible for two cameras to have accurate coplanar and line-aligned imaging planes, and it is necessary to calibrate the rotation matrix and translation vector correction between the two cameras through stereo;

Binocular matching and disparity calculation: Stereo matching is mainly to find out the corresponding relationship between each pair of images, and obtain the disparity map according to the principle of triangulation; after obtaining the disparity information, the depth of the original image can be easily obtained according to the projection model information and three-dimensional information. Stereo matching technology is generally considered to be the most difficult and critical problem in stereo vision. At present, stereo matching algorithm is a difficult and hot spot in computer vision. There are many algorithms, but the general steps are: matching cost calculation, matching cost superposition, parallax Acquisition, parallax refinement;

Automatically estimate the segmentation threshold: within the allowable minimum depth range accuracy, calculate the shortest distance from the camera, the furthest distance from the camera, the maximum height and the maximum width according to the distance between the two cameras and the focal length obtained by the camera resolution and stereo calibration. Then, within the allowable accuracy range, the distance between the front, rear and left and right of the obstacle, and the error range of the height of the obstacle are calculated.

3D point cloud road height detection: The road height detection is performed on the 3D point cloud obtained by 3D reconstruction, and the height that meets the set error is regarded as the road height. The error may be large when the road height is detected for the first time. After multiple iterative detections, the detection accuracy of the road height tends to be stable, and it will be updated with the recent height detection value updates.

3D point cloud segmentation: After 3D reconstruction to obtain a dense 3D point cloud, rotate the 3D point cloud according to the pitch angle of the camera relative to the ground, so that the road point cloud on the 3D point cloud is at the same height as the camera. According to the shortest distance from the camera, the farthest distance from the camera, the maximum height, the maximum width of the road surface, the height of the point cloud and the error of the obstacle height, the 3D point cloud is divided to obtain the point cloud of the road surface, the point cloud of the obstacle and the point cloud of the unknown area; The distance between the front and back and the left and right of the object is divided, and the obstacle point cloud of each position is obtained by segmenting the obstacle point cloud; the point cloud of the unknown area is composed of some point clouds with wrong matching calculations and point clouds with poor accuracy.

3D point cloud fusion color information to detect obstacles: 3D point cloud fusion color information is to segment the 3D point cloud to obtain obstacle point cloud, road surface point cloud and unknown area point cloud at different positions, and segment the color reference image by color image to obtain Image block, fused and compared the segmented point cloud with the segmented image block to judge the correctness of obstacles and road surfaces, determine the position range of obstacles, road surfaces and unknown areas, and achieve highly reliable obstacle detection.

Claims (10)

1. The binocular vision obstacle detection method based on three-dimensional point cloud segmentation is characterized in that: comprise the steps: (1) Obtain two color images through two cameras in different positions, use the stereo calibration method to calibrate the binocular camera, calculate the internal and external parameters and relative positional relationship of the two cameras, and eliminate the distortion of the two cameras according to these parameters Align with the row or column so that the imaging origin coordinates of the two color images are consistent, and the corrected binocular color view is obtained, and the pitch angle and height of the camera relative to the road surface are acquired or predetermined by the sensor; (2) For the binocular color view obtained after the correction in step (1), select one of the views as the reference image, and use the stereo matching algorithm to perform matching calculations on the reference image to obtain a preliminary disparity map; perform color image segmentation on the reference image , to get blocks with similar colors; according to the position relationship data of the camera relative to the road surface obtained in step (1), calculate the threshold for point cloud segmentation, including the shortest distance, the furthest distance, the maximum height and the maximum width; (3) After the disparity map calculated in step (2), use the relative positional relationship data (reprojection matrix) of the two cameras calculated in step (1) to perform 3D reconstruction of the disparity map to calculate a 3D point cloud; (4) According to the position data of the camera relative to the ground obtained in step (1), correct the 3D point cloud calculated by step (3), so that the road surface point cloud in the 3D point cloud is at the same height relative to the camera; (5) Detect the road surface height (the height of the road surface relative to the camera) on the 3D point cloud corrected in step (4), and segment obstacles, road surfaces and unknown point clouds by the segmentation threshold calculated in step (2); (6) Fusion the point cloud segmented in step (5) and the segmented block of the reference map image in step (2), determine obstacles, road surfaces and unknown areas, and mark them out. 2. the binocular visual obstacle detection method based on three-dimensional point cloud segmentation according to claim 1 is characterized in that: the binocular camera is a two-way video device combined by two cameras of the same specification, Different picture pixels can be obtained according to needs, and the processing method is the original RGB three-channel color picture. 3. according to the described binocular vision obstacle detection method based on three-dimensional point cloud segmentation of claim 1 or 2, it is characterized in that: the described automatic detection three-dimensional segmentation threshold method is based on the resolution of the camera and the stereo calibration data, calculates The shortest distance, furthest distance, maximum height, maximum width and segmentation threshold of obstacles can be detected with a certain accuracy. 4. according to the described binocular vision obstacle detection method based on three-dimensional point cloud segmentation of claim 1 and 3, it is characterized in that: described utilize three-dimensional point cloud to detect road surface height, is to utilize the method for loop iteration to improve detection road surface height At the same time, under the assumption that the object higher than the road surface is an obstacle, the obstacle point cloud, road surface point cloud and unknown area point cloud of different positions are segmented by the road surface height and the three-dimensional point cloud segmentation threshold. 5. According to the binocular vision obstacle detection method based on 3D point cloud segmentation according to claims 1 and 4, it is characterized in that: the acquisition of the obstacle, road surface and unknown area is obtained by segmenting in step (5) After the image segmentation of the 3D point cloud, the image blocks are determined, and the markers are displayed. 6. The binocular visual obstacle detection method based on three-dimensional point cloud segmentation according to claim 1, characterized in that: said binocular camera calibration: calibration includes camera calibration and stereo calibration; Camera calibration mainly calculates tangential distortion, radial distortion, focal length and optical center; Stereo calibration is the process of calculating the geometric relationship between two cameras in space, which depends on finding the rotation matrix and translation vector between the two cameras; Binocular correction: the camera needs correction and stereo correction, correction is to eliminate tangential and radial distortion; Stereo correction is because it is almost impossible for two cameras to have accurate coplanar and line-aligned imaging planes, and it is necessary to correct the rotation matrix and translation vector between the two cameras through stereo calibration. 7. The binocular visual obstacle detection method based on three-dimensional point cloud segmentation according to claim 1, characterized in that: said binocular matching and parallax calculation: stereo matching is mainly by finding out the corresponding relationship between each pair of images , according to the principle of triangulation, the disparity map is obtained; after obtaining the disparity information, the depth information and three-dimensional information of the original image can be easily obtained according to the projection model; Stereo matching technology is generally considered to be the most difficult and critical problem in stereo vision. At present, stereo matching algorithm is a difficult and hot spot in computer vision. There are many algorithms, but the general steps are: matching cost calculation, matching cost superposition, parallax Get, parallax refinement. 8. The binocular vision obstacle detection method based on three-dimensional point cloud segmentation according to claim 1, characterized in that: the automatic estimation of the segmentation threshold: within the allowable minimum depth range accuracy, according to the resolution of the camera and the stereo The distance and focal length between the two cameras obtained from the calibration calculate the shortest distance from the camera, the farthest distance from the camera, the maximum height and the maximum width, and then calculate the distance between the front, rear and left and right of the obstacle, and the obstacle within the allowable accuracy range height error range. 9. The binocular visual obstacle detection method based on three-dimensional point cloud segmentation according to claim 1, characterized in that: said three-dimensional point cloud road height detection: the three-dimensional point cloud obtained by three-dimensional reconstruction is used for road height detection, The height that satisfies the set error is regarded as the height of the road surface; the error may be large when the road height is detected for the first time. The value is updated while updating. 10. the binocular vision obstacle detection method based on three-dimensional point cloud segmentation according to claim 1, is characterized in that: The three-dimensional point cloud segmentation: after the denser three-dimensional point cloud is obtained by three-dimensional reconstruction, the three-dimensional point cloud is rotated according to the pitch angle of the camera relative to the ground, so that the road surface point cloud on the three-dimensional point cloud is at the same height as the camera; According to the shortest distance from the obstacle to the camera, the farthest distance from the camera, the maximum height, the maximum width of the road surface, point cloud height and obstacle height error, the 3D point cloud is divided to obtain road point cloud, obstacle point cloud and unknown area point cloud; The obstacle point cloud is obtained by segmenting the obstacle point cloud from the distance between the front and rear and left and right of the obstacle; the point cloud of the unknown area is composed of some point clouds with wrong matching calculations and point clouds with poor accuracy; 3D point cloud fusion color information to detect obstacles: 3D point cloud fusion color information is to segment the 3D point cloud to obtain obstacle point cloud, road surface point cloud and unknown area point cloud at different positions, and segment the color reference image by color image to obtain Image block, fused and compared the segmented point cloud with the segmented image block to judge the correctness of obstacles and road surfaces, determine the position range of obstacles, road surfaces and unknown areas, and achieve highly reliable obstacle detection.