CN114119700B - Obstacle ranging method based on U-V disparity map - Google Patents

Obstacle ranging method based on U-V disparity map Download PDF

Info

Publication number
CN114119700B
CN114119700B CN202111423802.XA CN202111423802A CN114119700B CN 114119700 B CN114119700 B CN 114119700B CN 202111423802 A CN202111423802 A CN 202111423802A CN 114119700 B CN114119700 B CN 114119700B
Authority
CN
China
Prior art keywords
parallax
obstacle
image
map
value
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202111423802.XA
Other languages
Chinese (zh)
Other versions
CN114119700A (en
Inventor
王智慧
李致远
张亚娟
张泺陶
曹越
郭晨璐
崔宾阁
于建志
路燕
包永堂
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Shandong University of Science and Technology
Original Assignee
Shandong University of Science and Technology
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Shandong University of Science and Technology filed Critical Shandong University of Science and Technology
Priority to CN202111423802.XA priority Critical patent/CN114119700B/en
Publication of CN114119700A publication Critical patent/CN114119700A/en
Application granted granted Critical
Publication of CN114119700B publication Critical patent/CN114119700B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/50—Depth or shape recovery
    • G06T7/55—Depth or shape recovery from multiple images
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
    • G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
    • Y02T10/00—Road transport of goods or passengers
    • Y02T10/10—Internal combustion engine [ICE] based vehicles
    • Y02T10/40—Engine management systems

Abstract

The invention discloses an obstacle ranging method based on a U-V parallax map, which comprises the steps of collecting a picture to be measured through a binocular camera, inputting the picture to be measured into a trained YOLO-V3 model, and obtaining category and position coordinate information of an obstacle in the picture to be measured; processing the picture to be detected through an SSD algorithm to obtain a whole picture parallax map; calculating to obtain rough distances of the obstacles, and sequencing the rough distances of the obstacles from small to large to obtain a rough distance sequencing table of the obstacles; constructing a U disparity map and a V disparity map of an obstacle with the smallest value in the rough distance sorting table of the obstacle; and obtaining three-dimensional scene position information of the obstacle according to the U parallax map and the V parallax map of the obstacle, and finally obtaining the three-dimensional scene position information of all the obstacles in the image to be detected through iterating the whole-map parallax map. The method and the device can accurately extract the parallax information of the obstacle, and can effectively avoid the situation that the background parallax is used as the parallax of the obstacle, thereby reducing the error of the distance of the obstacle and improving the accuracy of the distance information.

Description

Obstacle ranging method based on U-V disparity map
Technical Field
The invention relates to the technical field of binocular stereoscopic vision ranging, in particular to an obstacle ranging method based on a U-V parallax map.
Background
Compared with monocular vision ranging, binocular stereoscopic vision ranging has the characteristics of high precision and strong robustness, and has become a mainstream method in machine vision ranging gradually. Binocular stereoscopic vision ranging imitates the function of sensing the depth of surrounding environment space by human eyes, the same target object is shot from different positions by using a binocular camera, the shot binocular images are subjected to stereoscopic matching by various algorithms, and then the actual distance measurement is realized by using a triangulation principle. The existing binocular stereoscopic vision ranging technology is based on estimation of binocular image matching point parallax, but the background and the foreground cannot be separated, so that the distance error is large.
Disclosure of Invention
Aiming at the problem that the background and the foreground can not be separated to cause large distance error in the existing binocular stereoscopic vision distance measurement technology, the invention provides an obstacle distance measurement method based on a U-V parallax map.
The invention adopts the following technical scheme:
an obstacle ranging method based on a U-V disparity map comprises the following steps:
step 1: training a YOLO-V3 model by using a data set, acquiring a picture to be detected by using a binocular camera, and inputting the picture to be detected into the trained YOLO-V3 model to obtain the category and position coordinate information of the obstacle in the picture to be detected;
step 2: processing the picture to be detected through an SSD algorithm to obtain a whole picture parallax map;
step 3: calculating to obtain rough obstacle distances according to the whole-image parallax map obtained in the step 2 and the obstacle position coordinate information obtained in the step 1, and sorting the rough obstacle distances from small to large to obtain a rough obstacle distance sorting table;
step 4: constructing a U disparity map and a V disparity map of the obstacle with the minimum value in the rough obstacle distance sorting table according to the whole map disparity map obtained in the step 2, the rough obstacle distance sorting table obtained in the step 3 and the position coordinate information of the obstacle obtained in the step 1;
step 5: reducing and fusing the effective areas in the U disparity map and the V disparity map obtained in the step 4 to obtain an obstacle accurate disparity map, converting the obstacle accurate disparity map into an obstacle depth map, obtaining the position information of the obstacle three-dimensional scene through the obstacle depth map, subtracting the obstacle accurate disparity map from the whole disparity map, and deleting the obstacle rough distance with the minimum number in the obstacle rough distance sorting table;
step 6: and (5) repeating the steps 4 to 5 to finally obtain the three-dimensional scene position information of all the obstacles in the picture to be detected.
Preferably, step 2 specifically comprises the following steps:
step 2.1: dividing a picture to be detected into a left image and a right image, converting the left image and the right image into gray images, setting the size of a window win in an SSD algorithm as kernel, and searching for the maximum parallax max_disp as 256;
step 2.2: taking a Gaussian convolution kernel of 3*3 for the left image and the right image respectively, and performing Gaussian blur operation on the two images respectively by using Gaussian convolution;
step 2.3: sobel filtering is respectively carried out on the left image in the x direction and the y direction to obtain gradient matrixes in the x direction and the y direction of the left image, and then the gradient matrixes in the two directions are added according to weights of 0.5 respectively to obtain a total gradient matrix dst of the left image;
step 2.4: traversing the value of a gradient matrix dst corresponding to each pixel point in the left image in each thread, judging whether the value of the gradient matrix corresponding to the current pixel point is larger than a threshold value k, if not, not calculating the parallax value of the pixel point and setting the parallax value as 0; if the pixel point is larger than k, the pixel point is considered to be the edge of a certain object in the image, and the parallax value of the pixel point is calculated, and the specific method is as follows:
for each parallax value d of the current pixel point, calculating an average value of pixel gray values in the left and right images win, and then subtracting the average value from the pixel gray values in the left and right images win respectively, so as to calculate the sum of square differences in win, wherein the formula is as follows:
SSD(u,v)＝sum{[Left′(u,v)-Right′(u,v)] 2 }
then the right image window win moves leftwards by one pixel, the moved SSD is calculated, the right image window moves max_disp times altogether, and the parallax value d corresponding to the minimum value of the SSD (u, v) is taken as the gray value of the parallax image corresponding point finally output; the parallax value d is the distance between the left image window and the right image window of the minimum value SSD (u, v), and the value range is [0, max_disp ];
is the average value of the gray values of the pixels within the left image win +.>Is the average value of the pixel gray values within the right image win;
left '(u, v) represents the pixel gray value minus the average value in the Left image win, right' (u, v) represents the pixel gray value minus the average value in the Right image win;
u and v are relative coordinates in the window win with respect to the intermediate point, i.e., u and v belong to (-Kernel, kernel);
SSD (u, v) is the gap between the left image window and the right image window;
and then carrying out confidence judgment: if the 'minimum SSD (u, v) value'/'next-smallest SSD (u, v) value' of the current pixel is smaller than the confidence threshold value uniquenessRatio, reserving a calculation result; if the parallax is greater than or equal to the confidence threshold, setting the parallax of the current pixel to 0;
step 2.5: and (3) repeating the step (2.4) to obtain parallax values of all pixel points in the left image, and finally forming the whole-image parallax map.
Preferably, the step 3 specifically includes the following steps:
step 3.1: let the coordinates (x) of the upper left and lower right corners of the frame in the obstacle information 1 ，y 1 )，(x 2 ，y 2 ) Extracting a line range as [ x ] from a two-dimensional array of the whole-image parallax 1 ,x 2 ]The column range is [ y ] 1 ,y 2 ]Obtaining a local parallax map of the obstacle;
step 3.2: converting the local parallax image into a local depth image by using a parallax depth conversion formula, and setting f as the focal length of the binocular camera, tx as the baseline distance of the binocular camera, d as the parallax value and z as the depth, wherein the formula is as follows:
z＝f*Tx/d；
step 3.3: obtaining rough distance information expected to be used as an obstacle by utilizing Gaussian distribution on the local depth map;
step 3.4: and 3.1 to 3.3 are executed for each obstacle, and the rough distance information of the obstacles is ordered from small to large, so that a rough distance ordered list of the obstacles is obtained.
Preferably, the step 4 specifically includes the following steps:
step 4.1: taking an obstacle local disparity map of an obstacle corresponding to the obstacle rough distance with the smallest numerical value in the obstacle rough distance sorting table as an original disparity map I 0 (i, j, d), wherein d is a parallax value corresponding to the pixel point (i, j); obtaining the parallax maximum d max ；
Step 4.2: according to the maximum value d of parallax in step 4.1 max Constructing the number of rows and the number of columns of the U parallax map, wherein the number of rows of the U parallax map is the parallax maximum value d max Adding 1, wherein the column number of the U disparity map is the column number of the original disparity map;
step 4.3: after the number of rows and columns of the U parallax map are determined, parallax mapping is carried out, and finally the U parallax map is obtained;
step 4.4: according to the maximum value d of parallax in step 4.1 max Constructing the row number and the column number of the V parallax map, wherein the row number of the V parallax map is the row number of the original parallax map, and the column number of the V parallax map is the maximum parallax value plus 1;
step 4.5: determining the number of rows and columns of the V disparity map, and performing disparity mapping to finally obtain the V disparity map;
step 4.6: performing convolution enhancement processing on the U parallax image and the V parallax image, performing image filtering on the U parallax image and the V parallax image, and processing the U parallax image and the V parallax image after image filtering to extract effective areas in the U parallax image and the V parallax image;
step 4.7: and selecting a parallax maximum value or a parallax average value according to different scenes to filter the effective area.
Preferably, the step 5 specifically includes the following steps:
step 5.1: taking an effective area, setting coordinates of two points (a 1, b 1) and (a 2, b 2) at the upper left and lower right of the effective area, taking columns b1 to b2 of an original parallax image as traversal images for the U parallax image, judging whether each pixel point in the traversal images is in the effective area in the U parallax image or not, and storing all the pixel points in the effective area; for the V parallax image, taking a 1-a 2 columns of the original parallax image as a traversing image, judging whether each pixel point in the traversing image is in an effective area in the V parallax image, and storing all the pixel points in the effective area;
step 5.2: traversing all the effective areas by utilizing the step 5.1, and finally obtaining a restored U parallax map and a restored V parallax map;
step 5.3: taking intersection of the restored U disparity map and the V disparity map as an accurate disparity map of the obstacle;
step 5.4: converting the accurate parallax image of the obstacle into an obstacle depth image by utilizing a parallax depth conversion formula;
step 5.5: obtaining Gaussian distribution of all points with pixel values different from 0 in the depth map of the obstacle, and taking the expected Gaussian distribution as the distance of the obstacle;
step 5.6: subtracting the accurate parallax map of the obstacle obtained in the step 5.3 from the full-map parallax map, and deleting the rough distance of the obstacle with the smallest numerical value in the rough distance sorting table of the obstacle.
The invention has the beneficial effects that:
according to the obstacle ranging method based on the U-V parallax map, a YOLO-V3 model is trained by utilizing a data set, a picture to be measured is acquired through a binocular camera, the picture to be measured is input into the trained YOLO-V3 model, and the type and position coordinate information of an obstacle in the picture to be measured are obtained; the method improves the SSD algorithm, and compared with the traditional SSD algorithm, the method increases the step of averaging, so that the result is more accurate; in addition, compared with a dense disparity map, the sparse disparity map obtained by the SSD algorithm is small in calculated amount and high in speed. Calculating to obtain rough obstacle distances according to the obtained whole-image disparity map and the obstacle position coordinate information, and sorting the rough obstacle distances from small to large to obtain a rough obstacle distance sorting table; constructing a U disparity map and a V disparity map of an obstacle with the minimum value in the rough obstacle distance sorting table according to the whole map disparity map, the rough obstacle distance sorting table and the category and position coordinate information of the obstacle; reducing and fusing the obtained effective areas in the U disparity map and the V disparity map to obtain depth map and three-dimensional scene position information of the obstacle, and deleting the rough obstacle distance with the minimum value in the rough obstacle distance sorting table; and repeating the process to finally obtain the three-dimensional scene position information of all the obstacles in the picture to be detected.
The invention can accurately extract the parallax information of the obstacle, and is different from the whole U-V parallax image, the invention adopts the local U-V parallax image, and can effectively avoid the condition of taking the background parallax as the parallax of the obstacle, thereby reducing the error of the distance of the obstacle and improving the accuracy of the distance information. The iterative thought solves the problem of large error caused by mutual shielding of a plurality of obstacles in one graph by adopting the iterative thought of firstly calculating the obstacles with short distance and then calculating the obstacles with long distance.
Drawings
Fig. 1 is a flowchart of an obstacle ranging method based on a U-V disparity map.
Fig. 2 is a picture to be measured of example 1.
Fig. 3 is an overall view parallax map of embodiment 1.
Fig. 4 is a partial parallax map of an obstacle in embodiment 1.
Fig. 5 is a U-disparity map of the barrier of fig. 4.
Fig. 6 is a V disparity map of the barrier of fig. 4.
Fig. 7 is an accurate parallax map of the obstacle after the reduction fusion of fig. 5 and 6.
Fig. 8 is three-dimensional position information of the picture to be measured in embodiment 1.
Detailed Description
The following description of the embodiments of the invention will be given with reference to the accompanying drawings and examples:
example 1
Referring to fig. 1 to 8, an obstacle ranging method based on a U-V disparity map includes the steps of:
step 1: training a YOLO-V3 model by using the existing data set, acquiring a picture to be tested by using a binocular camera, and inputting the picture to be tested into the trained YOLO-V3 model to obtain the category and position coordinate information of the obstacle in the picture to be tested; wherein the existing data set is available over a network. The picture to be measured is shown in fig. 2.
Step 2: and processing the picture to be detected through an SSD algorithm to obtain an overall picture parallax map, as shown in FIG. 3.
The method specifically comprises the following steps:
step 2.1: dividing a picture to be detected into a left image and a right image, converting the left image and the right image into gray images, setting the size of a window win in an SSD algorithm as kernel, and searching for the maximum parallax max_disp as 256;
step 2.2: taking a Gaussian convolution kernel of 3*3 for the left image and the right image respectively, and performing Gaussian blur operation on the two images respectively by using Gaussian convolution;
step 2.3: sobel filtering is respectively carried out on the left image in the x direction and the y direction to obtain gradient matrixes in the x direction and the y direction of the left image, and then the gradient matrixes in the two directions are added according to weights of 0.5 respectively to obtain a total gradient matrix dst of the left image;
step 2.4: traversing the value of a gradient matrix dst corresponding to each pixel point in the left image in each thread, judging whether the value of the gradient matrix corresponding to the current pixel point is larger than a threshold value k, if not, not calculating the parallax value of the pixel point and setting the parallax value as 0; if the pixel point is larger than k, the pixel point is considered to be the edge of a certain object in the image, and the parallax value of the pixel point is calculated, and the specific method is as follows:
for each parallax value d of the current pixel point, calculating an average value of pixel gray values in the left and right images win, and then subtracting the average value from the pixel gray values in the left and right images win respectively, so as to calculate the sum of square differences in win, wherein the formula is as follows:
SSD(u,v)＝sum{[Left′(u,v)-Right′(u,v)] 2 }
then the right image window win moves leftwards by one pixel, the moved SSD is calculated, the right image window moves max_disp times altogether, and the parallax value d corresponding to the minimum value of the SSD (u, v) is taken as the gray value of the parallax image corresponding point finally output; the parallax value d is the distance between the left image window and the right image window of the minimum value SSD (u, v), and the value range is [0, max_disp ];
is the average value of the gray values of the pixels within the left image win +.>Is the average value of the pixel gray values within the right image win;
left '(u, v) represents the pixel gray value minus the average value in the Left image win, right' (u, v) represents the pixel gray value minus the average value in the Right image win;
u and v are relative coordinates in the window win with respect to the intermediate point, i.e., u and v belong to (-kernel, kernel);
SSD (u, v) is the gap between the left image window and the right image window;
and then carrying out confidence judgment: if the 'minimum SSD (u, v) value'/'next-smallest SSD (u, v) value' of the current pixel is smaller than the confidence threshold value uniquenessRatio, reserving a calculation result; if the parallax is greater than or equal to the confidence threshold, setting the parallax of the current pixel to 0;
step 2.5: and (3) repeating the step (2.4) to obtain parallax values of all pixel points in the left image, and finally forming the whole-image parallax map.
Compared with the traditional SSD algorithm, the SSD algorithm is improved, and the step of averaging is added, so that the result is more accurate; in addition, compared with a dense disparity map, the sparse disparity map obtained by the SSD algorithm is small in calculated amount and high in speed.
Step 3: and (3) calculating to obtain rough obstacle distances according to the whole-image parallax map obtained in the step (2) and the obstacle position coordinate information obtained in the step (1), and sorting the rough obstacle distances from small to large to obtain a rough obstacle distance sorting table.
The method specifically comprises the following steps:
step 3.1: let the coordinates (x) of the upper left and lower right corners of the frame in the obstacle information 1 ，y 1 )，(x 2 ，y 2 ) Extracting a line range as [ x ] from a two-dimensional array of the whole-image parallax 1 ,x 2 ]The column range is [ y ] 1 ,y 2 ]Obtaining a local parallax map of the obstacle; as in fig. 4.
Step 3.2: converting the local parallax image into a local depth image by using a parallax depth conversion formula, and setting f as the focal length of the binocular camera, tx as the baseline distance of the binocular camera, d as the parallax value and z as the depth, wherein the formula is as follows:
z＝f*Tx/d；
step 3.3: obtaining rough distance information expected to be used as an obstacle by utilizing Gaussian distribution on the local depth map;
step 3.4: and 3.1 to 3.3 are executed for each obstacle, and the rough distance information of the obstacles is ordered from small to large, so that a rough distance ordered list of the obstacles is obtained.
Step 4: and constructing a U disparity map and a V disparity map of the obstacle with the smallest number in the rough distance sorting table of the obstacle according to the whole map disparity map obtained in the step 2, the rough distance of the obstacle obtained in the step 3 and the category and position coordinate information of the obstacle obtained in the step 1.
The method specifically comprises the following steps:
step 4.1: taking an obstacle local disparity map of an obstacle corresponding to the obstacle rough distance with the smallest numerical value in the obstacle rough distance sorting table as an original disparity map I 0 (i, j, d), wherein d is a parallax value corresponding to the pixel point (i, j); obtaining the parallax maximum d max ；
Step 4.2: according to the maximum value d of parallax in step 4.1 max Constructing the number of rows and the number of columns of the U parallax map, wherein the number of rows of the U parallax map is the parallax maximum value d max Adding 1, wherein the column number of the U disparity map is the column number of the original disparity map;
step 4.3: after the number of rows and columns of the U parallax map are determined, parallax mapping is carried out, and finally the U parallax map is obtained; as in fig. 5.
The method comprises the following steps: in the original disparity map I 0 (i, j, d) based on the same disparity value d for each column u The number of pixels of (a)d u_count Accumulate and use (d) u J) as new pixel point coordinates, d u_count As the value of the pixel point, a U-disparity map is obtained, and the process formula is as follows:
step 4.4: according to the maximum value d of parallax in step 4.1 max Constructing the row number and the column number of the V parallax map, wherein the row number of the V parallax map is the row number of the original parallax map, and the column number of the V parallax map is the maximum parallax value plus 1;
step 4.5: determining the number of rows and columns of the V disparity map, and performing disparity mapping to finally obtain the V disparity map; as in fig. 6.
The method comprises the following steps: in the original disparity map I 0 (i, j, d) on the basis of the same disparity value d for each row v The number d of the pixel points of (2) v_count Accumulate and use (i, d) v ) D as a new pixel point coordinate v_count As the value of the pixel point, a V disparity map is obtained, and the process formula is as follows:
step 4.6: carrying out convolution enhancement processing on the U parallax image and the V parallax image, wherein the convolution kernel window is 3 multiplied by 3 pixels, and the convolution weight matrix is as follows:
and then carrying out image filtering on the U parallax image and the V parallax image, wherein the image filtering method is a Gaussian filtering method with good effect.
The basic idea of gaussian filtering is to convolve the gaussian kernel function with the original signal to obtain the filtered output signal. Setting sigma as standard deviation; k is a gaussian kernel, (x, y) is the pixel coordinates of the image, and the gaussian filters are generally expressed as:
and processing the U disparity map and the V disparity map after image filtering to extract effective areas in the U disparity map and the V disparity map.
The method comprises the following steps: binarization is carried out on the U parallax map and the V parallax map after image filtering by adopting an OTSU algorithm, and the principle is as follows: the assumption of the OTSU algorithm is that there is a threshold TH that divides all pixels of the image into two classes C1 (less than TH) and C2 (greater than TH), and the respective average of these two classes of pixels is m1, m2, and the global average of the image is mG. While the probability of a pixel being classified into C1 and C2 is p1, p2, respectively. Thus there is:
p1*m1+p2*m2＝mG
p1+p2＝1
based on the concept of variance, the inter-class variance sigma 2 The expression is:
σ 2 ＝p1(m1-mG) 2 +p2(m2-mG) 2
the above formula is simplified, and the formula (1) is substituted into the formula (3) to obtain:
σ 2 ＝p1p2(m1-m2) 2
wherein:
the gray level k capable of maximizing the above formula is calculated as an OTSU threshold, G (x, y) is set as a pixel value of coordinates as points (x, y) on an image, min is set as a minimum threshold, max is set as a maximum threshold, and the threshold is brought into the following formula:
the U parallax image and the V parallax image after binarization processing are subjected to expansion operation by selecting proper structural elements, the hollow part of the obtained image is filled, and then the opening operation is performed to eliminate noise points in the image; and finding the outline by using the cv2.findcontours () function of OpenCV, and obtaining the maximum and minimum values of the horizontal and vertical coordinates of each outline pixel point as an effective area.
Step 4.7: and selecting a parallax maximum value or a parallax average value according to different scenes to filter the effective area. For the vehicle collision early warning, an effective area with the largest parallax should be selected, for the vehicle speed detection, a parallax average value should be used, for the pedestrian abnormal behavior detection, a parallax average value should be used, and the like.
Step 5: and (3) reducing and fusing the effective areas in the U disparity map and the V disparity map obtained in the step (4) to obtain an obstacle accurate disparity map, converting the obstacle accurate disparity map into an obstacle depth map, obtaining the position information of the obstacle three-dimensional scene through the obstacle depth map, subtracting the obstacle accurate disparity map from the whole disparity map, and deleting the obstacle rough distance with the minimum value in the obstacle rough distance sorting table.
The method specifically comprises the following steps:
step 5.1: taking an effective area, setting coordinates of two points (a 1, b 1) and (a 2, b 2) at the upper left and lower right of the effective area, taking columns b1 to b2 of an original parallax image as traversal images for the U parallax image, judging whether each pixel point in the traversal images is in the effective area in the U parallax image or not, and storing all the pixel points in the effective area; for the V parallax image, taking a 1-a 2 columns of the original parallax image as a traversing image, judging whether each pixel point in the traversing image is in an effective area in the V parallax image, and storing all the pixel points in the effective area;
step 5.2: traversing all the effective areas by utilizing the step 5.1, and finally obtaining a restored U parallax map and a restored V parallax map;
step 5.3: taking intersection of the restored U disparity map and the V disparity map as an accurate disparity map of the obstacle; as shown in fig. 7.
Step 5.4: converting the accurate parallax image of the obstacle into an obstacle depth image by utilizing a parallax depth conversion formula;
step 5.5: the Gaussian distribution of all points with the pixel value not being 0 in the depth map of the obstacle is obtained, and the expected Gaussian distribution is taken as the distance of the obstacle.
Step 5.6: subtracting the accurate parallax map of the obstacle obtained in the step 5.3 from the full-map parallax map, and deleting the rough distance of the obstacle with the smallest numerical value in the rough distance sorting table of the obstacle.
Step 6: and (5) repeating the steps 4 to 5 to finally obtain the three-dimensional scene position information of all the obstacles in the picture to be detected. As in fig. 8.
It should be understood that the above description is not intended to limit the invention to the particular embodiments disclosed, but to limit the invention to the particular embodiments disclosed, and that the invention is not limited to the particular embodiments disclosed, but is intended to cover modifications, adaptations, additions and alternatives falling within the spirit and scope of the invention.

Claims (5)

1. The obstacle ranging method based on the U-V disparity map is characterized by comprising the following steps of:
step 1: training a YOLO-V3 model by using a data set, acquiring a picture to be detected by using a binocular camera, and inputting the picture to be detected into the trained YOLO-V3 model to obtain the category and position coordinate information of the obstacle in the picture to be detected;
step 2: processing the picture to be detected through an SSD algorithm to obtain a whole picture parallax map;
step 3: calculating to obtain rough obstacle distances according to the whole-image parallax map obtained in the step 2 and the obstacle position coordinate information obtained in the step 1, and sorting the rough obstacle distances from small to large to obtain a rough obstacle distance sorting table;
step 4: constructing a U disparity map and a V disparity map of the obstacle with the minimum value in the rough obstacle distance sorting table according to the whole map disparity map obtained in the step 2, the rough obstacle distance sorting table obtained in the step 3 and the position coordinate information of the obstacle obtained in the step 1;
step 5: reducing and fusing the effective areas in the U disparity map and the V disparity map obtained in the step 4 to obtain an obstacle accurate disparity map, converting the obstacle accurate disparity map into an obstacle depth map, obtaining the position information of the obstacle three-dimensional scene through the obstacle depth map, subtracting the obstacle accurate disparity map from the whole disparity map, and deleting the obstacle rough distance with the minimum number in the obstacle rough distance sorting table;
step 6: and (5) repeating the steps 4 to 5 to finally obtain the three-dimensional scene position information of all the obstacles in the picture to be detected.
2. The obstacle ranging method based on the U-V disparity map according to claim 1, wherein the step 2 specifically includes the steps of:
step 2.1: dividing a picture to be detected into a left image and a right image, converting the left image and the right image into gray images, setting the size of a window win in an SSD algorithm as kernel, and searching for the maximum parallax max_disp as 256;
step 2.2: taking a Gaussian convolution kernel of 3*3 for the left image and the right image respectively, and performing Gaussian blur operation on the two images respectively by using Gaussian convolution;
step 2.3: sobel filtering is respectively carried out on the left image in the x direction and the y direction to obtain gradient matrixes in the x direction and the y direction of the left image, and then the gradient matrixes in the two directions are added according to weights of 0.5 respectively to obtain a total gradient matrix dst of the left image;
step 2.4: traversing the value of a gradient matrix dst corresponding to each pixel point in the left image in each thread, judging whether the value of the gradient matrix corresponding to the current pixel point is larger than a threshold value k, if not, not calculating the parallax value of the pixel point and setting the parallax value as 0; if the pixel point is larger than k, the pixel point is considered to be the edge of a certain object in the image, and the parallax value of the pixel point is calculated, and the specific method is as follows:
for each parallax value d of the current pixel point, calculating an average value of pixel gray values in the left and right images win, and then subtracting the average value from the pixel gray values in the left and right images win respectively, so as to calculate the sum of square differences in win, wherein the formula is as follows:
then the right image window win moves leftwards by one pixel, the moved SSD is calculated, the right image window moves max_disp times altogether, and the parallax value d corresponding to the minimum value of the SSD (u, v) is taken as the gray value of the parallax image corresponding point finally output; the parallax value d is the distance between the left image window and the right image window of the minimum value SSD (u, v), and the value range is [0, max_disp ];
is the average value of the gray values of the pixels within the left image win +.>Is the average value of the pixel gray values within the right image win;
left '(u, v) represents the pixel gray value minus the average value in the Left image win, right' (u, v) represents the pixel gray value minus the average value in the Right image win;
u and v are relative coordinates in the window win with respect to the intermediate point, i.e., u and v belong to (-ker nel, ker nel);
SSD (u, v) is the gap between the left image window and the right image window;
and then carrying out confidence judgment: if the 'minimum SSD (u, v) value'/'next-smallest SSD (u, v) value' of the current pixel is smaller than the confidence threshold value uniquenessRatio, reserving a calculation result; if the parallax is greater than or equal to the confidence threshold, setting the parallax of the current pixel to 0;
step 2.5: and (3) repeating the step (2.4) to obtain parallax values of all pixel points in the left image, and finally forming the whole-image parallax map.
3. The obstacle ranging method based on the U-V disparity map according to claim 1, wherein the step 3 specifically includes the steps of:
step 3.1: let the coordinates (x) of the upper left and lower right corners of the frame in the obstacle information 1 ，y 1 )，(x 2 ，y 2 ) Extracting a line range as [ x ] from a two-dimensional array of the whole-image parallax 1 ,x 2 ]The column range is [ y ] 1 ,y 2 ]Obtaining a local parallax map of the obstacle;
step 3.2: converting the local parallax image into a local depth image by using a parallax depth conversion formula, and setting f as the focal length of the binocular camera, tx as the baseline distance of the binocular camera, d as the parallax value and z as the depth, wherein the formula is as follows:
z＝f*Tx/d；
step 3.3: obtaining rough distance information expected to be used as an obstacle by utilizing Gaussian distribution on the local depth map;
step 3.4: and 3.1 to 3.3 are executed for each obstacle, and the rough distance information of the obstacles is ordered from small to large, so that a rough distance ordered list of the obstacles is obtained.
4. The obstacle ranging method based on the U-V disparity map according to claim 3, wherein the step 4 specifically includes the steps of:
step 4.1: will beThe obstacle local disparity map of the obstacle corresponding to the obstacle rough distance with the smallest number of the obstacle rough distance sorting table is taken as an original disparity map I 0 (i, j, d), wherein d is a parallax value corresponding to the pixel point (i, j); obtaining the parallax maximum d max ；
Step 4.2: according to the maximum value d of parallax in step 4.1 max Constructing the number of rows and the number of columns of the U parallax map, wherein the number of rows of the U parallax map is the parallax maximum value d max Adding 1, wherein the column number of the U disparity map is the column number of the original disparity map;
step 4.3: after the number of rows and columns of the U parallax map are determined, parallax mapping is carried out, and finally the U parallax map is obtained;
step 4.4: according to the maximum value d of parallax in step 4.1 max Constructing the row number and the column number of the V parallax map, wherein the row number of the V parallax map is the row number of the original parallax map, and the column number of the V parallax map is the maximum parallax value plus 1;
step 4.5: determining the number of rows and columns of the V disparity map, and performing disparity mapping to finally obtain the V disparity map;
step 4.6: performing convolution enhancement processing on the U parallax image and the V parallax image, performing image filtering on the U parallax image and the V parallax image, and processing the U parallax image and the V parallax image after image filtering to extract effective areas in the U parallax image and the V parallax image;
step 4.7: and selecting a parallax maximum value or a parallax average value according to different scenes to filter the effective area.
5. The obstacle ranging method based on the U-V disparity map according to claim 4, wherein the step 5 specifically includes the steps of:
step 5.1: taking an effective area, setting coordinates of two points (a 1, b 1) and (a 2, b 2) at the upper left and lower right of the effective area, taking columns b1 to b2 of an original parallax image as traversal images for the U parallax image, judging whether each pixel point in the traversal images is in the effective area in the U parallax image or not, and storing all the pixel points in the effective area; for the V parallax image, taking a 1-a 2 columns of the original parallax image as a traversing image, judging whether each pixel point in the traversing image is in an effective area in the V parallax image, and storing all the pixel points in the effective area;
step 5.2: traversing all the effective areas by utilizing the step 5.1, and finally obtaining a restored U parallax map and a restored V parallax map;
step 5.3: taking intersection of the restored U disparity map and the V disparity map as an accurate disparity map of the obstacle;
step 5.4: converting the accurate parallax image of the obstacle into an obstacle depth image by utilizing a parallax depth conversion formula;
step 5.5: obtaining Gaussian distribution of all points with pixel values different from 0 in the depth map of the obstacle, and taking the expected Gaussian distribution as the distance of the obstacle;
step 5.6: subtracting the accurate parallax map of the obstacle obtained in the step 5.3 from the full-map parallax map, and deleting the rough distance of the obstacle with the smallest numerical value in the rough distance sorting table of the obstacle.
CN202111423802.XA 2021-11-26 2021-11-26 Obstacle ranging method based on U-V disparity map Active CN114119700B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202111423802.XA CN114119700B (en) 2021-11-26 2021-11-26 Obstacle ranging method based on U-V disparity map

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202111423802.XA CN114119700B (en) 2021-11-26 2021-11-26 Obstacle ranging method based on U-V disparity map

Publications (2)

Publication Number Publication Date
CN114119700A CN114119700A (en) 2022-03-01
CN114119700B true CN114119700B (en) 2024-03-29

Family

ID=80370332

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202111423802.XA Active CN114119700B (en) 2021-11-26 2021-11-26 Obstacle ranging method based on U-V disparity map

Country Status (1)

Country Link
CN (1) CN114119700B (en)

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105550665A (en) * 2016-01-15 2016-05-04 北京理工大学 Method for detecting pilotless automobile through area based on binocular vision
CN105740802A (en) * 2016-01-28 2016-07-06 北京中科慧眼科技有限公司 Disparity map-based obstacle detection method and device as well as automobile driving assistance system
CN107169418A (en) * 2017-04-18 2017-09-15 海信集团有限公司 A kind of obstacle detection method and device
WO2018058356A1 (en) * 2016-09-28 2018-04-05 驭势科技（北京）有限公司 Method and system for vehicle anti-collision pre-warning based on binocular stereo vision
CN108230392A (en) * 2018-01-23 2018-06-29 北京易智能科技有限公司 A kind of dysopia analyte detection false-alarm elimination method based on IMU
CN111476762A (en) * 2020-03-26 2020-07-31 南方电网科学研究院有限责任公司 Obstacle detection method and device of inspection equipment and inspection equipment

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112149458A (en) * 2019-06-27 2020-12-29 商汤集团有限公司 Obstacle detection method, intelligent driving control method, device, medium, and apparatus

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105550665A (en) * 2016-01-15 2016-05-04 北京理工大学 Method for detecting pilotless automobile through area based on binocular vision
CN105740802A (en) * 2016-01-28 2016-07-06 北京中科慧眼科技有限公司 Disparity map-based obstacle detection method and device as well as automobile driving assistance system
WO2018058356A1 (en) * 2016-09-28 2018-04-05 驭势科技（北京）有限公司 Method and system for vehicle anti-collision pre-warning based on binocular stereo vision
CN107169418A (en) * 2017-04-18 2017-09-15 海信集团有限公司 A kind of obstacle detection method and device
CN108230392A (en) * 2018-01-23 2018-06-29 北京易智能科技有限公司 A kind of dysopia analyte detection false-alarm elimination method based on IMU
CN111476762A (en) * 2020-03-26 2020-07-31 南方电网科学研究院有限责任公司 Obstacle detection method and device of inspection equipment and inspection equipment

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
基于U-V视差算法的障碍物识别技术研究;上官;;兰州工业高等专科学校学报;20111231(05);18-23 *
基于立体视觉与纹理特征相结合的障碍物探测;赵申;;软件导刊;20180515(05);121-124 *

Also Published As

Publication number Publication date
CN114119700A (en) 2022-03-01

Similar Documents

Publication Publication Date Title
CN110569704B (en) Multi-strategy self-adaptive lane line detection method based on stereoscopic vision
CN110310320B (en) Binocular vision matching cost aggregation optimization method
JP7134012B2 (en) Parallax estimation device and method
CN111553252B (en) Road pedestrian automatic identification and positioning method based on deep learning and U-V parallax algorithm
CN110942449A (en) Vehicle detection method based on laser and vision fusion
CN104574393B (en) A kind of three-dimensional pavement crack pattern picture generates system and method
CN106960449B (en) Heterogeneous registration method based on multi-feature constraint
CN108235774B (en) Information processing method and device, cloud processing equipment and computer program product
CN109410264B (en) Front vehicle distance measuring method based on laser point cloud and image fusion
CN105335955A (en) Object detection method and object detection apparatus
CN110243390B (en) Pose determination method and device and odometer
CN112184792B (en) Road gradient calculation method and device based on vision
CN111178193A (en) Lane line detection method, lane line detection device and computer-readable storage medium
CN113205604A (en) Feasible region detection method based on camera and laser radar
CN108596032B (en) Detection method, device, equipment and medium for fighting behavior in video
Lion et al. Smart speed bump detection and estimation with kinect
CN113744315A (en) Semi-direct vision odometer based on binocular vision
CN109886064A (en) Determination can driving space boundary method
CN114119700B (en) Obstacle ranging method based on U-V disparity map
JP2001116513A (en) Distance image calculating device
CN107958226B (en) Road curve detection method, device and terminal
CN113450335B (en) Road edge detection method, road edge detection device and road surface construction vehicle
CN116052120A (en) Excavator night object detection method based on image enhancement and multi-sensor fusion
Pfeiffer et al. Ground truth evaluation of the Stixel representation using laser scanners
KR20160063039A (en) Method of Road Recognition using 3D Data

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant