WO2018058356A1 - Method and system for vehicle anti-collision pre-warning based on binocular stereo vision - Google Patents

Abstract

A method and system for vehicle anti-collision pre-warning based on a binocular stereo vision, comprising: acquiring left and right images by means of a binocular camera mounted on a vehicle body, and obtaining a disparity image based on the left and right images (S210); converting the disparity image to obtain a V-disparity image (S220); binarizing the V-disparity image (S230); fitting to obtain a segment straight line from points of the binarized V-disparity image using a RANSAC method (S240); smoothing and filtering the straight line according to multiple frames of images (S250); obtaining an accessible region in an original grayscale image by means of the extracted straight line (S260); according to the original image and the disparity image, calculating a three-dimensional coordinate of a point on the ground in a real-world coordinate system, and assuming that the ground is a planar model, fitting the plane using RANSAC so as to obtain a ground model (S270); converting the entire scene in the original grayscale image from a camera coordinate into a world coordinate while generating a plane graph, and obtaining an occupied map from the plane graph (S280); dividing the occupied map by means of a connected domain labeling detection algorithm to obtain the position of each obstacle, converting same into the original image for marking, and calculating the distance from the obstacle to this vehicle through the disparity image (S290); and when a current vehicle distance is less than a certain threshold value, giving an alarm or introducing a decision-making module for participation in decision-making (S2901). The method is adapted to various pavements and road conditions, has a low requirement for disparity image precision, does not rely on data, and is not affected by an artificial design.

Description

Automobile anti-collision warning method and system based on binocular stereo vision Technical field

The present invention relates generally to automotive autonomous driving techniques, and more particularly to automotive anti-collision techniques based on binocular stereo vision.

Background technique

Accurate and real-time anti-collision warning has important application significance, especially in the safety control of assisted driving and the automatic control of automatic driving. For example, in automatic driving, anti-collision warning can reduce accidents as much as possible to avoid personal injury. And property damage; in autonomous driving, the more accurate the anti-collision warning, the higher the safety.

At present, there are mainly anti-collision warning methods. First, based on laser radar sensors or millimeter-wave radars, the calibration is first performed. The area below a certain threshold is judged as the ground. The laser radar required for this method is very expensive and difficult to popularize. The millimeter-wave accuracy is far less than that of the laser radar; the second is to use the monocular color camera to detect obstacles in front by machine learning and computer vision. This method relies heavily on the training samples and the artificially designed features, and the driving area varies widely. If the training sample does not exist, the detection will not be detected, and the scalability and versatility are not strong. On the other hand, the monocular camera cannot accurately obtain the depth information, and the obtained result often does not conform to the real scene. Finally, the method is real-time. Sex is also difficult to guarantee.

In recent years, some safe driving techniques based on machine vision (including monocular vision and stereo vision) have been proposed.

Patent document 1CN101135558B discloses a machine vision-based automobile anti-collision warning technology, in which a machine vision method is used to collect the front vehicle license plate features and lane line information, and according to the size of the projected image pixel points of the vehicle license plate in front of the machine vision. Calculate the distance from the vehicle in front, and calculate the driving state of the preceding vehicle based on the state information of the vehicle speed and steering. According to the relative distance between the vehicle and the boundary of the lane line, it is judged whether or not the vehicle is traveling within a safe lane range.

Patent Document 1 uses machine learning methods for detection, which relies heavily on trained samples and artificially designed features. The scene regions encountered during driving vary widely, and cannot be detected in the absence of training samples. Sexuality is not strong; on the other hand, monocular cameras cannot accurately obtain depth information and speed information, and the results obtained often do not conform to the real scene; finally, the real-time nature of the method is difficult to guarantee.

Patent Document 2CN102685516A discloses an active safety assisted driving method based on stereo vision technology. The active safety assisted driving system comprehensively utilizes opto-mechanical information technology, consisting of a stereo vision subsystem, an image rapid processing subsystem and a safety assisted driving subsystem, including two high-resolution CCD cameras, ambient light level sensors, and dual-channel video capture. Card, synchronous controller, data transmission circuit, power supply circuit, image fast processing algorithm library, voice reminder module, screen display module and active safe driving control module. In various weather conditions, real-time identification of the relative distance, relative speed and relative acceleration of dangerous targets such as lane separation, vehicles ahead, bicycles, pedestrians, etc., through voice prompting the driver to take countermeasures, automatic deceleration in critical situations And emergency braking, all-weather to ensure safe driving.

Patent Document 2 also uses a method based on machine learning recognition to detect, which relies heavily on trained samples and artificially designed features. The scene regions encountered during driving vary widely, and cannot be detected in the absence of training samples. Scalability and versatility are not strong, but more accurate results can be achieved by binoculars in distance calculation.

A machine vision-based automotive anti-collision warning technology that is more versatile, more real-time, and less dependent on training data and manual design features.

Summary of the invention

The present invention has been made in view of the above circumstances.

According to an aspect of the present invention, an automobile anti-collision warning method based on binocular stereo vision is provided, which may include: capturing, by a binocular camera mounted on an automobile body, two left and right gray scales in front of a car in a traveling direction of the automobile. Image, calculated disparity map; converted from disparity map to V disparity map; binarization of V disparity map; using RANSAC method to fit segmentation line from points of binarized V disparity map;

Smoothing the line according to the multi-frame image; and obtaining the travelable area in the original gray image by the extracted straight line; calculating the three-dimensional coordinates of the point belonging to the ground in the real world coordinate system according to the original image and the disparity map, assuming the ground For the planar model, use RANSAC to fit the plane to obtain the ground model; convert the entire scene in the original grayscale image from camera coordinates to world coordinates, and generate a plan view, which is taken from the plan to occupy the map; The domain mark detection algorithm divides the position of each obstacle and converts it into the original image to mark it, and calculates the distance of the obstacle to the vehicle through the disparity map; when the current vehicle distance is less than a certain threshold, an alarm or an incoming decision module is performed. Participate in decision making.

According to the above-mentioned automobile anti-collision warning method, the obtaining the map by the plan view may include: first extracting points higher than the first threshold height of the ground according to the world coordinates of each point, and converting the points into the ground coordinate system; Points above the first threshold height and below the second threshold height are marked in the occupied map; the value occupying each pixel in the map is the sum of the heights of its respective points, thereby obtaining the occupied map.

According to the above-described automobile anti-collision warning method, the position of each obstacle obtained by the connected domain mark detection algorithm from the occupied map may include: converting the color to the color according to the size of each pixel value in the occupied map. The image is such that the larger the pixel value is, the more toward the red color, the smaller the pixel value is, the more the color is biased toward the blue color, and the different objects are segmented by the connected domain mark detection algorithm, thereby obtaining the position of each obstacle.

According to the car collision avoidance warning method described above, the binarizing the V disparity map may include: determining a maximum value of each row of pixel values, and setting a gray value of a pixel at which only the maximum value is in each row to 255, the remaining pixel gray value is set to 0.

According to the above automobile anti-collision warning method, using the RANSAC method to fit a piece of segmentation line may include repeatedly performing the following sequence of operations until a predetermined end criterion is reached: selecting a set of random subsets among the maximum points in the V disparity map To perform straight line fitting to obtain a straight line model; use the obtained straight line model to test all other data. If a point is applied to the estimated straight line model, it is considered to be an intra-point. If there are more than a predetermined number of points, it is classified as Intra-point, then the estimated model is considered reasonable, then re-estimate the model with all intra-site points, and estimate the error rate of the intra-point and model; if the error rate of the model is lower than the current best model, replace it with the model The best model at present; the best model obtained in the end is used as the segmentation line of the segment.

According to the above automobile anti-collision warning method, the using the RANSAC method to fit the multi-segment segment line may include: first extracting the first line according to the above method, and after the extraction is completed, belonging to The point of the first straight line is removed from the V disparity map, and then the second line is extracted for the remaining points according to the method of claim 5, and so on, until the number of remaining points is less than a predetermined threshold.

According to the above automobile anti-collision warning method, the smoothing of the straight line according to the multi-frame image may include: setting a time window, assuming that the line model is represented by ax+by+c=0, obtaining a line model parameter for each frame image, for each Each parameter is accumulated for each frame. When a new image is received for each frame, the line model parameters of the first frame image are subtracted from the accumulated parameter results, and the line model parameters of the current frame image are added, and then averaged as The linear model parameters for this frame.

According to the above automobile anti-collision warning method, the obtaining the travelable area in the original grayscale image by the extracted straight line includes: selecting, for each row in the V disparity map, a point at which the disparity value on the extracted straight line is d, In the row corresponding to the disparity map, the difference between the disparity value of each pixel and d is compared. When the difference is less than a certain threshold, the corresponding position of the original image is determined as a safe travelable area.

According to another aspect of the present invention, an automobile anti-collision warning system based on binocular stereo vision is provided, which may include: a binocular camera that continuously captures two left and right grayscale images in front of the car in the direction of travel of the vehicle; The device, including a memory, a processor, a communication interface, a bus, a memory, a communication interface, and a processor are all connected to the bus, wherein the computer stores executable instructions, and the computing device can obtain two left and right cameras obtained by the binocular camera via the communication interface. a grayscale image, when the processor executes the computer executable instruction, performing the following method: calculating a disparity map based on two left and right grayscale images; converting a disparity map from the disparity map; and performing a binary value on the V disparity map Using the RANSAC method to fit a segmented straight line from the binarized V-disparity map points; smoothing the straight line according to the multi-frame image; obtaining the travelable region in the original grayscale image by the extracted straight line; Calculate the three-dimensional coordinates of the points belonging to the ground in the real world coordinate system based on the original image and the disparity map. The ground is a plane model, and RANSAC is used to fit the plane to obtain a ground model; the entire scene in the original gray image is converted from camera coordinates to world coordinates, and a plan is generated at the same time, and the plan is taken to occupy the map; The connected domain mark detection algorithm divides the position of each obstacle and converts it into the original image to mark it, and calculates the distance of the obstacle to the vehicle through the disparity map. When the current vehicle distance is less than a certain threshold, an alarm or an incoming decision is made. Modules participate in decision making.

According to the above-mentioned automobile anti-collision warning system, obtaining a map by the plan view may include: first extracting points higher than the first threshold height of the ground according to the world coordinates of each point, and converting the points into the ground coordinate system; A point at a first threshold height and below a second threshold height is marked in the occupied map; the value occupying each pixel in the map is the sum of the heights of its corresponding points, thereby obtaining Occupy the map.

According to the above automobile anti-collision warning system, the position of each obstacle obtained by the connected domain mark detection algorithm from the occupied map may include: converting the color to the color according to the size of each pixel value in the occupied map. The image is such that the larger the pixel value is, the more toward the red color, the smaller the pixel value is, the more the color is biased toward the blue color, and the different objects are segmented by the connected domain mark detection algorithm, thereby obtaining the position of each obstacle.

According to the above automobile anti-collision warning system, wherein the binarizing the V disparity map comprises: obtaining a maximum value of each row of pixel values, and setting a gray value of a pixel at which only the maximum value is in each row to 255, The remaining pixel grayscale values are set to zero.

According to the above automobile anti-collision warning system, the using the RANSAC method to fit a piece of segmentation line may include repeatedly performing the following sequence of operations until a predetermined exit criterion is reached: selecting a random set of the maximum points in the V disparity map The subset is used to perform straight line fitting to obtain a straight line model; the obtained straight line model is used to test all other data. If a point is applied to the estimated straight line model, it is considered to be an intra-point, and if there is a point exceeding a predetermined number, The class is an intra-point, then the estimated model is considered reasonable, then the model is re-estimated with all intra-site points, and the error rate of the intra-point and model is estimated; if the error rate of the model is lower than the current best model, then The model replaces the current best model; the best model obtained last is used as the segmentation line.

According to the above automobile anti-collision warning system, the using the RANSAC method to fit the multi-segment segment line may include: extracting the first line, and after the extraction is completed, removing the points belonging to the first line from the V-disparity map, and then The remaining points are used to extract the second line, and so on, until the number of remaining points is less than a predetermined threshold.

According to the above automobile anti-collision warning system, the smoothing of the straight line according to the multi-frame image may include: setting a time window, assuming that the line model is represented by ax+by+c=0, obtaining a linear model parameter for each frame image, for each Each parameter is accumulated for each frame. When a new image is received for each frame, the line model parameters of the first frame image are subtracted from the accumulated parameter results, and the line model parameters of the current frame image are added, and then averaged as The linear model parameters for this frame.

According to the above automobile anti-collision warning system, the obtaining the travelable area in the original grayscale image by the extracted straight line may include: selecting, for each row in the V disparity map, a disparity value on the extracted straight line as d, In the corresponding row in the disparity map, the difference between the disparity value of each pixel and d is compared. When the difference is less than a certain threshold, the corresponding position of the original image is determined as a safe drivable region.

According to still another aspect of the present invention, an automobile anti-collision warning system may include: a binocular phase The machine is configured to capture two left and right grayscale images in front of the car in the direction of travel of the car; the parallax map calculation component calculates a parallax map from the left and right two grayscale images; the V parallax map conversion module converts the parallax map to obtain a V Parallax map; binarization module, binarization of V disparity map; RANSAC line fitting module, using RANSAC method to fit segmentation line from the binarized V disparity map points; multi-frame image filtering a module, according to the multi-frame image smoothing and filtering straight line; the original image travelable area determining module obtains the travelable area in the original grayscale image by the extracted straight line; the ground model fitting module calculates the belonging according to the original image and the disparity map The three-dimensional coordinates of the ground point in the real world coordinate system, assuming the ground is a plane model, using RANSAC to fit the plane to obtain the ground model; occupying the map seeking module, converting the entire scene in the original gray image from camera coordinates To the world coordinates, at the same time generate a plan, the plan is taken to occupy the map; the obstacle segmentation and distance calculation module, from the occupied map The position of each obstacle is obtained by the connected domain mark detection algorithm, and converted into the original image, and the distance of the obstacle to the vehicle is calculated by the disparity map; the alarm module alarms when the current vehicle distance is less than a certain threshold. Or pass the decision module to participate in the decision.

A stereo vision-based automobile anti-collision warning method and system according to an embodiment of the present invention obtains a ground model based on a binocular visual image, obtains a occupied map of the scene through the ground model, and obtains anti-collision information from the occupied map, and can adapt to various For various roads and road conditions, the algorithm has low accuracy requirements for disparity map, reduced front-end calculation, strong anti-interference ability, and no dependence on data and artificial design features.

DRAWINGS

These and/or other aspects and advantages of the present invention will become more apparent from the following detailed description of the embodiments of the invention.

1 shows a schematic diagram of an in-vehicle anti-collision warning system 100 in accordance with an embodiment of the present invention;

2 shows a general flow chart of a vehicle collision avoidance warning method based on binocular stereo vision according to an embodiment of the present invention;

FIG. 3 is a schematic diagram showing a situation in which a least square method incorrectly extracts a straight line in the presence of a large noise;

4 shows a flow diagram of a method 240 of fitting a straight line from points of a V disparity map, in accordance with an embodiment of the present invention;

FIG. 5 is a block diagram showing the structure of an automobile collision avoidance warning system 300 according to another embodiment of the present invention.

detailed description

The present invention will be further described in detail below in conjunction with the drawings and specific embodiments.

First, an explanation of the terms used in this article is given.

Disparity map: The disparity map is based on any image in the image pair, the size of which is the size of the reference image, and the element value is the image of the disparity value. The disparity map contains the distance information of the scene. The parallax map can be calculated from the left and right images taken by the binocular camera. The coordinates of a point in a common two-dimensional disparity map are represented by (u, v), where u is the abscissa and v is the ordinate; the pixel value of the pixel at the point (u, v) is represented by d(u, v), The pixel value represents the parallax at this point (u, v). Since the disparity map contains the distance information of the scene, the image matching of the disparity map from the stereo image pair has been the most active field in binocular vision research.

V disparity map: The V disparity map is converted from the disparity map, and the gray value of any point (d, v) in the V disparity map is the number of points in the line corresponding to the ordinate of the disparity map having the disparity value equal to d. To put it bluntly, the V-disparity map can be viewed as a side view of the disparity map. The plane in the original image is projected into a straight line by accumulating the number of identical points of the same line of disparity values.

RANSAC: Abbreviation for RANdom Sample Consensus, which is an algorithm for calculating valid mathematical data based on a set of sample data sets containing abnormal data.

Occupation map: In the early visual system, the environmental knowledge is represented by a grid, in which the 2D projection information of each object in the environment to the ground is preserved. Such a representation is called occupation. map.

Binocular stereo vision: Binocular Stereo Vision is an important form of machine vision. It is based on the parallax principle and uses the imaging device to acquire two images of the measured object from different positions. A method of positional deviation to obtain three-dimensional geometric information of an object. Combining the images obtained by the two eyes and observing the difference between them, we can obtain a clear sense of depth, establish the correspondence between the features, and map the same spatial physical points in different images. Corresponding to the point, this difference, we call the disparity image. The binocular stereo vision measurement method has the advantages of high efficiency, appropriate accuracy, simple system structure, and low cost. In the measurement of moving objects (including animals and human bodies), the stereoscopic method is a more effective measurement method because image acquisition is done in an instant. Binocular stereo vision system is one of the key technologies of computer vision. The distance information of spatial 3D scenes is also the most basic content in computer vision research.

1 shows a schematic diagram of a system 100 for detecting a travelable area of a vehicle, including a binocular camera 110 and a computing device 120, in accordance with an embodiment of the present invention.

The binocular camera 110 continuously captures two left and right grayscale images in front of the car in the direction of travel of the car.

The binocular camera 110 is mounted, for example, in front of the top of the vehicle such that its imaging range is concentrated on the road surface at the front of the vehicle.

Computing device 120 includes a memory 121, a processor 122, a communication interface 123, and a bus 124. The memory 121, the communication interface 123 and the processor 122 are both connected to the bus 124. The memory stores computer executable instructions, and the computing device can obtain the left and right grayscale images captured by the binocular camera via the communication interface, when the processor executes the A method of performing an automobile collision avoidance warning when the computer executable instruction is described.

An alarm 125 may also be included in the computing device 120 for providing an alert signal or sending a notification when a danger or emergency is detected.

The structure shown in Fig. 1 is merely an example, and can be added, reduced, replaced, etc. as needed.

In addition, it should be noted that some of the functions or functions may be implemented by different components as needed, for example, calculation from the left and right images to obtain the disparity map is described in the embodiment as being implemented by the computing device, but as needed It is within the scope of the present invention to add software, hardware or firmware for calculating a disparity map in a binocular camera, or to deploy a special component for calculating a disparity map based on left and right images in a vehicle.

A method of detecting a travelable area of a car in real time according to an embodiment of the present invention will be described in detail below with reference to FIG.

In the embodiment of the present invention, the technology for detecting the travelable area of the automobile in real time is obtained by the binocular camera sensor, the left and right images are obtained, the disparity map is obtained from the left and right images, and the V disparity map is constructed by using the disparity map (V-disparity). Map), then use RANSAC to score points on the V disparity map A straight line is formed, and the straight line is smoothed and filtered according to the multi-frame image, and finally the parallax corresponding to the straight lines obtains a safe area that can be driven in the original image.

2 shows a general flow diagram of a method 200 of detecting a travelable area of a vehicle in real time in accordance with an embodiment of the present invention.

In step S210, two left and right grayscale images in front of the car in the traveling direction of the vehicle are imaged by a binocular camera mounted on the vehicle body, and a parallax map is calculated.

Specifically, for example, according to the binocular stereo matching correlation algorithm, the correspondence between each pair of images is first found, and the disparity map of the current scene is obtained according to the triangulation principle.

Here, it is also possible to perform some denoising processing and the like on the parallax map.

In step S220, a V-disparity map is obtained from the parallax map conversion.

Specifically, for example, in the disparity map, the relative distance of the object with respect to the lens is represented by the change in the shade of gray, and the parallax of the ground is continuously changed according to the depth information included in the disparity map, and the segment line is approximated. It is assumed that M _d represents the pixel value of a point on the disparity map, and M _vd represents the pixel value of the corresponding point on the V disparity map. The conversion relationship between the disparity map and the V disparity map is represented by a function f(M _d )=M _{vd ,} which represents the number P _num of pixels having the same disparity on each line of the cumulative disparity map, such that the disparity is the horizontal axis The vertical axis is consistent with the disparity map, and P _num is the gray value of the corresponding pixel, so that a gray V disparity map is obtained.

In step S230, the V-disparity map is binarized.

In one example, the following method is used for binarization: the principle of binarization is to first determine the maximum value of each row, and the gray value of the pixel at which only the maximum value is located in each row is set to 255, and the gray values of the remaining pixels are set. Is 0.

In step S240, the RANSAC method is used to fit the segmentation line from the points of the binarized V-disparity map.

In the following, in the numerous straight line fitting algorithms, why the embodiment of the present invention selects the straight line fitting of the points of the V-disparity map after binarization using the RANSAC method.

The data in real life tends to have a certain deviation, or noise, which makes the mathematics fit difficult. For example, we know that there is a linear relationship between two variables X and Y, Y = aX + b, we want to determine the specific values of parameters a and b. Through experiments, a set of X and Y test values can be obtained. Although theoretically two equations of unknown numbers only need two sets of values to confirm, due to systematic errors, the values of a and b calculated by arbitrary two points are not the same. What we hope is that the final calculated theoretical model has the least error with the test value.

Usually the prior art uses a least squares method or a Hough transform to fit a straight line.

The disadvantage of the Hough transform is that the detection speed is too slow to control in real time; the accuracy is not high enough, the expected information is not detected, but the error is judged, and a large amount of redundant data is generated. This is mainly due to:

1. It takes a lot of memory space, which takes a long time and has poor real-time performance;

2. The images in reality are generally interfered by external noise, and the signal-to-noise ratio is relatively low. At this time, the performance of the conventional Hough transform will drop sharply. When the search for the parameter space maximum value is difficult to determine due to the appropriate threshold, it often appears. The problem of "virtual peaks" and "missing inspections".

The least squares method calculates the value of the minimum mean square error with respect to the partial derivatives of the parameters a, b being zero. In fact, in many cases, the least squares method is synonymous with linear regression. Unfortunately, the least squares method is only suitable for situations where the error is small. Imagine this situation. If you need to extract a model from a noisy dataset (for example, only 20% of the data fits the model), the least squares method is not enough. For example, in Figure 3, a straight line (pattern) can be easily seen by the naked eye, but the least squares method is wrong.

The present invention detects a road surface by extracting a straight line from a V-disparity map, and has a large noise in the parallax map. In this case, the least square method is used to extract the straight line, and it is likely that a wrong fit is obtained.

The RANSAC algorithm can estimate the parameters of the mathematical model in an iterative manner from a set of observation data sets containing “outside points”, which is very suitable for model parameter estimation of observation data with more noise. The data obtained in practical applications often contains noisy data, which will cause interference to the construction of the model. We call such noise data points outliers (outliers), which play an active role in model construction. We call them inliers (inside points), RANSAC do One thing is to randomly select some points, use these points to get a model (if the line is fitted, the so-called model is actually the slope), and then use this model to test the remaining points, if the test If the data point is within the error tolerance, the data point is judged as an intra-point, otherwise it is judged as an out-of-office point. If the number of intra-site points reaches a certain threshold, it indicates that the selected data point set has reached an acceptable level. Otherwise, all the steps after the previous random selection of the point set are continued, and the process is repeated until Finding the selected set of data points is acceptable, and the model obtained at this time can be considered as the optimal model construction for the data points.

4 shows a flow diagram of a method 240 of fitting a straight line from a point of a V disparity map, in accordance with an embodiment of the present invention. This method can be used for step S240 in Fig. 2.

In step S241, a set of random subsets of the points in the binarized V-disparity map is selected for straight line fitting to obtain a straight line model.

In step S242, the obtained line model is used to test all other data. If a point is applied to the estimated line model, it is considered to be the intra-point, and the number of points in the statistical office.

In step S243, it is judged whether or not the number of intra-office points is larger than the threshold, and if the result of the determination is YES, the process proceeds to step S245, otherwise, the process proceeds to step S244.

In step S244, it is determined that the estimated model is unreasonable, the model is discarded, and then proceeds to step S249.

In step S245, it is judged that the estimated model is reasonable, then the model is re-estimated with all the intra-site points, and the error rate of the intra-point and the model is estimated, and then proceeds to step S246.

In step S246, it is judged whether the error rate of the current estimated model is smaller than the error rate of the optimal model, and if the result is affirmative, it proceeds to step S247, otherwise proceeds to step S248.

In step S247, the current model is substituted for the best model, that is, because the current estimated model error rate is lower than the best model error rate according to the judgment of step S246, the performance is better than the best model, so the most replaced The good model becomes the new best model, and then proceeds to step S249.

In step S248, the estimated model is discarded, and then proceeds to step S249.

In step S249, it is determined whether or not the termination condition is reached, and if the termination condition is reached, the process is terminated, otherwise the process returns to step S241 to repeat the execution. The termination condition here may be, for example, the number of iterations reaches a threshold number of times, the error rate is lower than a predetermined threshold, and the like.

The method of extracting a straight line from the V disparity map by using the RANSAC method is described above with reference to FIG. 4, and the ground is not a plane, so it is reflected in the V disparity map as a plurality of consecutive segmented straight lines, and the method of extracting the multi-segment piecewise straight lines may be, for example, as follows : first according to the method described in connection with FIG. 4, for example. Extract the first line, after the extraction is completed, remove the point belonging to the first line from the V disparity map, and then extract the second line according to the same method for the remaining points, and then repeat until the remaining points The number is less than a predetermined threshold.

Returning to Fig. 2, after the completion of step S240, the process proceeds to step S250.

In step S250, the straight line is smoothed according to the multi-frame image.

As described above, in the patent document CN 103489175B, Kalman filtering is performed on the fitted straight line.

The inventor believes that the Kalman filtering method is based on the Gaussian distribution of the processing object, but the change of the road surface is not Gaussian. In addition, the Kalman filtering method runs slowly and cannot meet the detection of automatic driving. The real-time requirements of the car's driveable area.

The road surface detecting technology according to the embodiment of the present invention designs a method for smoothly smoothing a straight line according to a multi-frame image in accordance with real-time requirements. Since the slope of the road on which the automobile travels does not change greatly, and the change is uniformly slow, the variation of the straight line fitted according to the embodiment of the present invention also varies uniformly. On the other hand, since the parallax map obtained by the binocular camera has a lot of noise, the obtained straight line causes unnecessary jitter. In order to reduce such jitter and consider that the straight line obtained by the above fitting is uniform and slow, the embodiment of the present invention proposes smoothing filtering using a multi-frame image to obtain a smooth and uniform straight line model.

Specifically, linear smoothing filtering may be performed according to the multi-frame image as follows: a time window is set, and a straight line model is assumed to be ax+by+c=0, and a linear model parameter is obtained for each frame image, for each frame for each parameter. Accumulate, when each frame of a new image, subtract the line model parameters of the first frame image from the accumulated parameter results, plus the line model parameters of the current frame image, and then average the line as the frame Model parameters. For example, if the car is driving on the road, the current time is tc, and the new image is taken. At this time, for the fixed window, the first frame is removed from the window, and then the new image frame is added, and the line model of the image in the window is added. The parameters are averaged as the linear model parameters of the new image frame, that is, the estimated mathematical model parameters of the road surface in the V disparity map; then, as time progresses, the operation continues, which is equivalent to moving forward with time Slide the window.

In step S260, the travelable area in the original grayscale image is obtained by the extracted straight line.

In one example, the travelable area in the original grayscale image may be obtained by the line extracted in the V disparity map as follows: for each row in the V disparity map, the point of the disparity value on the extracted line is d, In the row corresponding to the disparity map, the difference between the disparity value of each pixel and d is compared. When the difference is less than a certain threshold, the corresponding position of the original image is determined as a safe travelable area.

Information on safe travelable areas in grayscale maps provides critical decision-making information for assisted driving, autonomous driving and driverless driving to prevent collisions and ensure safety.

Returning to FIG. 2, in step S270, according to the original image and the disparity map, the three-dimensional coordinates of the points belonging to the ground in the real world coordinate system are calculated, and the ground is assumed to be a plane model, and the plane is fitted using RANSAC to obtain the ground model. .

Specifically, in one example, the ground model is obtained by fitting the following operations: According to the original image and the disparity map, the three-dimensional coordinates of the points belonging to the ground in the real world coordinate system are calculated. In the real world coordinate system, the ground is assumed to be a planar model, denoted as ax+by+cz+d=0, and then RANSAC is used to fit the plane. RANSAC performs line fitting by repeatedly selecting a random subset of the maximum points in the ground candidate points. Since the disparity map contains a lot of noise, after performing a RANSAC, the points belonging to the ground are collected. A RANSAC is performed to fit the plane, and finally the ground model is obtained.

In step S280, the entire scene in the original grayscale image is converted from camera coordinates to world coordinates, while a plan view is generated, and the map is occupied by the plan view.

In one example, the process of occupying the map is: first extracting points above a certain height of the ground, and converting those points into the ground coordinate system; we will occupy points above a certain height and below a certain height. The map is marked; the value occupying each pixel in the map is the sum of the heights of its corresponding points, thereby obtaining the occupied map.

In step S290, the position of each obstacle is segmented from the occupied map by the connected domain mark detection algorithm, and converted into the original image, and the distance of the obstacle to the vehicle is calculated by the disparity map.

In one example, the image is converted to a color-marked image based on the size of each pixel value in the map. The larger the pixel value, the more red, and the smaller the pixel value, the more blue. Then, different objects () are segmented by the connected domain label detection algorithm. For the connected domain label detection algorithm, refer to the literature, Di Stefano, Luigi, and Andrea Bulgarelli. "A simple and efficient connected components labeling algorithm." Image Analysis and Processing , 1999. Proceedings.International Conference on. IEEE, 1999. The specific position of each obstacle is obtained and converted into the original image, and the parallax map is used to calculate the distance of the obstacle to the vehicle. The relevant distance can be found based on the relationship between the parallax and the depth.

In step S2901, when the current vehicle distance is less than a certain threshold, an alarm is issued or the decision module is involved in the decision.

FIG. 5 is a block diagram showing the structure of a real-time detection system 300 for a motor vehicle that can detect a travelable area of a vehicle in real time according to another embodiment of the present invention. The system 300 is placed in the vehicle for detecting the travelable area of the car in real time, providing critical support for the driver's assisted driving, automatic driving and driverless driving.

As shown in FIG. 5, the vehicle travelable area real-time detection system 300 may include: a binocular camera 310, a parallax map calculation unit 320, a V-disparity map conversion unit 330, a binarization unit 340, a RANSAC line fitting unit 350, and a multi-frame. The image filtering unit 360, the original image travelable area determining unit 370, the ground model fitting module 380, the occupation map obtaining module 390, the obstacle division and distance calculating module 391, and the alarm module 392.

The binocular camera 310 is configured to capture two left and right grayscale images in front of the car in the direction of travel of the car. The parallax map calculation unit 320 calculates a parallax map from the left and right two grayscale images. The V-disparity map conversion section 330 converts the parallax map to obtain a V-disparity map. The binarization unit 340 binarizes the V-disparity map. The RANSAC line fitting component 350 uses the RANSAC method to fit a segmentation line from the points of the binarized V-disparity map. The multi-frame image filtering section 360 smooth-filters the straight line according to the multi-frame image. The original image travelable area determining section 370 obtains the travelable area in the original grayscale image by the extracted straight line. The ground model fitting module 380 calculates the three-dimensional coordinates of the points belonging to the ground in the real world coordinate system according to the original image and the disparity map, and assumes that the ground is a plane model, and uses RANSAC to fit the plane to obtain a ground model. The Occupied Map Fetch Module 390 converts the entire scene in the original grayscale image from camera coordinates to world coordinates, while generating a floor plan, which is taken up by the plan view to occupy the map. The obstacle segmentation and distance calculation module 391 divides the position of each obstacle from the occupied map by the connected domain mark detection algorithm, converts it into the original image, and calculates the distance of the obstacle to the vehicle through the disparity map. . The alarm module 392 performs an alarm when the current vehicle distance is less than a certain threshold or enters a decision module to participate in the decision.

The parallax map calculation unit 320, the V-disparity map conversion unit 330, the binarization unit 340, the RANSAC line fitting unit 350, the multi-frame image filter unit 360, the original image travelable area determining unit 370, the ground model fitting module 380, Occupy map seeking module 390, obstacle segmentation and For the functions and specific implementations of the distance calculation module 391 and the alarm module 392, reference may be made to the description of the corresponding steps in FIG. 2, and details are not described herein again.

It should be noted that the binocular camera in this paper should be understood in a broad sense. Any camera that can obtain left and right images or a device with camera function can be regarded as a binocular camera in this paper.

The parallax map calculation unit 320, the V-disparity map conversion unit 330, the binarization unit 340, the RANSAC line fitting unit 350, the multi-frame image filter unit 360, the original image travelable area determining unit 370, the ground model fitting module 380, The occupation map entropy module 390, the obstacle segmentation and distance calculation module 391, and the alarm module 392 should also be understood in a broad sense. These components can be implemented in software, firmware or hardware or a combination of these, and the components can be combined with each other. Combinations or further splits and the like are intended to fall within the scope of the present disclosure.

The real-time detecting method and system for a travelable area of a vehicle according to an embodiment of the present invention can adapt to various road surfaces and road conditions, have low precision requirements for parallax maps, reduce front end calculation amount, have strong anti-interference ability, and improve real-time performance. Auto-safe driving of cars is critical.

The embodiments of the present invention have been described above, and the foregoing description is illustrative, not limiting, and not limited to the disclosed embodiments. Numerous modifications and changes will be apparent to those skilled in the art without departing from the scope of the invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims (17)

1. An automobile anti-collision warning method based on binocular stereo vision, comprising:

   The left and right two grayscale images in front of the car in the direction of travel of the car are captured by a binocular camera mounted on the vehicle body, and a parallax map is calculated;

   Converting a parallax map to a V-disparity map;

   Binarizing the V disparity map;

   Using the RANSAC method to fit a segmented straight line from the point of the binarized V disparity map;

   Smoothing the line according to the multi-frame image;

   Obtaining a travelable area in the original grayscale image by the extracted straight line;

   According to the original image and the disparity map, calculate the three-dimensional coordinates of the points belonging to the ground in the real world coordinate system, and assume that the ground is a plane model, and use RANSAC to fit the plane to obtain the ground model;

   Converting the entire scene in the original grayscale image from camera coordinates to world coordinates, and generating a plan view, which is obtained by the plan view;

   The position of each obstacle is segmented from the occupied map by the connected domain mark detection algorithm, and converted into the original image, and the distance of the obstacle to the vehicle is calculated by the disparity map;

   When the current vehicle distance is less than a certain threshold, an alarm is issued or the decision module is involved to participate in the decision.
2. The automobile anti-collision warning method according to claim 1, wherein the occupying the map by the plan view comprises:

   First, according to the world coordinates of each point, points higher than the first threshold height of the ground are extracted, and the points are converted into the ground coordinate system; points higher than the first threshold height and lower than the second threshold height occupy the map The mark is taken; the value occupying each pixel in the map is the sum of the heights of its corresponding points, thereby obtaining the occupied map.
3. The automobile anti-collision warning method according to claim 1, wherein the position of each obstacle obtained by the connected domain mark detection algorithm from the occupied map comprises:

   According to the size of each pixel in the map, it is converted into a color-marked image, so that the larger the pixel value, the more red, the smaller the pixel value is, the more blue, and the different objects are segmented by the connected domain mark detection algorithm. , thus obtaining the position of each obstacle.
4. The automobile collision avoidance warning method according to claim 1, wherein said binarizing the V-disparity map comprises:

   Find the maximum value of each row of pixel values, and set the gray value of the pixel where only the maximum value is in each row. Set to 255 and the remaining pixel grayscale values are set to zero.
5. The automobile anti-collision warning method according to claim 1, wherein the RANSAC method is used to fit a segmentation line including:

   Repeat the sequence of operations described below until the predetermined end criteria are reached:

   Selecting a set of random subsets in the maximum point in the V disparity map to perform a straight line fit to obtain a straight line model;

   Use the obtained line model to test all other data. If a point is applied to the estimated line model, it is considered to be an intra-point. If there are more than a predetermined number of points classified as intra-point, then the estimated model is considered to be Reasonable, then re-estimate the model with all intra-site points and estimate the error rate of the intra-point and model;

   If the error rate of the model is lower than the current best model, replace the current best model with the model;

   The best model obtained last is used as the segmentation line of the segment.
6. The automobile anti-collision warning method according to claim 5, wherein said using the RANSAC method to fit the multi-segment segmentation line comprises:

   Firstly, the first straight line is extracted according to the method of claim 5, and after the extraction is completed, the points belonging to the first straight line are removed from the V disparity map, and then the remaining points are according to the method according to claim 5. The second line is extracted, and so on, until the number of remaining points is less than a predetermined threshold.
7. The automobile anti-collision warning method according to claim 1, wherein said multi-frame image smoothing filtering straight line comprises:

   Set a time window, assuming that the line model is represented as ax+by+c=0, get the line model parameters for each frame of image, accumulate each frame for each parameter, and add a new image every time, from the accumulated The parameter of the line model of the first frame is subtracted from the parameter result, and the line model parameters of the current frame image are added, and then averaged as the line model parameter of this frame.
8. The automobile anti-collision warning method according to any one of claims 1 to 7, wherein the obtaining the travelable area in the original grayscale image by the extracted straight line comprises:

   For each row in the V disparity map, select a point on the extracted line whose disparity value is d, and compare the disparity value of each pixel with the difference of d in the row corresponding to the disparity map, when the difference is less than a certain value At the threshold value, the corresponding position of the original image is determined as a safe travelable area.
9. An automobile anti-collision warning system based on binocular stereo vision, comprising:

   The binocular camera continuously captures two left and right grayscale images in front of the car in the direction of travel of the car;

   The computing device, including a memory, a processor, a communication interface, a bus, a memory, a communication interface, and a processor are all connected to the bus, wherein the computer stores executable instructions, and the computing device can obtain the left and right images obtained by the binocular camera via the communication interface. A grayscale image, when the processor executes the computer executable instructions, performs the following method:

   Calculating a disparity map based on two left and right grayscale images;

   Converting a parallax map to a V-disparity map;

   Binarizing the V disparity map;

   Using the RANSAC method to fit a segmentation line from the points of the binarized V disparity map;

   Smoothing the line according to the multi-frame image;

   Obtaining a travelable area in the original grayscale image by the extracted straight line;

   According to the original image and the disparity map, calculate the three-dimensional coordinates of the points belonging to the ground in the real world coordinate system, and assume that the ground is a plane model, and use RANSAC to fit the plane to obtain the ground model;

   Converting the entire scene in the original grayscale image from camera coordinates to world coordinates, and generating a plan view, which is obtained by the plan view;

   The position of each obstacle is segmented from the occupied map by the connected domain mark detection algorithm, and converted into the original image, and the distance of the obstacle to the vehicle is calculated by the disparity map.

   When the current vehicle distance is less than a certain threshold, an alarm is issued or the decision module is involved to participate in the decision.
10. The automobile anti-collision warning system according to claim 9, wherein the occupying the map by the plan view comprises:

    First, according to the world coordinates of each point, points higher than the first threshold height of the ground are extracted, and the points are converted into the ground coordinate system; points higher than the first threshold height and lower than the second threshold height occupy the map The mark is taken; the value occupying each pixel in the map is the sum of the heights of its corresponding points, thereby obtaining the occupied map.
11. The automobile anti-collision warning system according to claim 9, wherein the position of each obstacle obtained by the connected domain mark detection algorithm from the occupied map comprises:

    Converts it to a color-marked image according to the size of each pixel in the map, so that the larger the pixel value, the more red, the smaller the pixel value, the more biased toward blue, detected by the connected domain mark The algorithm splits out different objects, thereby obtaining the position of each obstacle.
12. The automobile anti-collision warning system according to claim 9, wherein said binarizing the V-disparity map comprises:

    Find the maximum value of each row of pixel values, set the gray value of the pixel where only the maximum value is in each row to 255, and set the gray value of the remaining pixels to 0.
13. The automobile anti-collision warning system according to claim 9, wherein said using the RANSAC method to fit a segmented straight line comprises:

    Repeat the sequence of operations below until the scheduled exit criteria are reached:

    Selecting a set of random subsets in the maximum point in the V disparity map to perform a straight line fit to obtain a straight line model;

    Use the obtained line model to test all other data. If a point is applied to the estimated line model, it is considered to be an intra-point. If there are more than a predetermined number of points classified as intra-point, then the estimated model is considered to be Reasonable, then re-estimate the model with all intra-site points and estimate the error rate of the intra-point and model;

    If the error rate of the model is lower than the current best model, replace the current best model with the model;

    The best model obtained last is used as the segmentation line of the segment.
14. The automobile anti-collision warning system according to claim 13, wherein said using the RANSAC method to fit the multi-segment segmentation line comprises:

    Extract the first line, after the extraction is completed, remove the point belonging to the first line from the V disparity map, and then extract the second line for the remaining points, and so on until the number of remaining points is less than the predetermined number Threshold.
15. The automobile anti-collision warning system according to claim 9, wherein the smoothing filter line according to the multi-frame image comprises:

    Set a time window, assuming that the line model is represented as ax+by+c=0, get the line model parameters for each frame of image, accumulate each frame for each parameter, and add a new image every time, from the accumulated The parameter of the line model of the first frame is subtracted from the parameter result, and the line model parameters of the current frame image are added, and then averaged as the line model parameter of this frame.
16. The automobile anti-collision warning system according to claim 9, wherein the obtaining the travelable area in the original grayscale image by the extracted straight line comprises:

    For each row in the V disparity map, select the disparity value on the extracted line as d, in the disparity map In the corresponding row, the difference between the disparity value of each pixel and d is compared. When the difference is less than a certain threshold, the corresponding position of the original image is determined as a safe travelable area.
17. An automobile anti-collision warning system includes:

    a binocular camera configured to capture two left and right grayscale images in front of the car in the direction of travel of the car;

    A parallax map calculation component calculates a parallax map from two left and right grayscale images;

    V parallax map conversion module, which converts the disparity map to obtain a V disparity map;

    a binarization module that binarizes the V disparity map;

    The RANSAC straight line fitting module uses the RANSAC method to fit a segmented straight line from the points of the binarized V disparity map;

    a multi-frame image filtering module for smoothing and filtering a line according to a multi-frame image;

    The original image travelable area determining module obtains the travelable area in the original grayscale image by the extracted straight line;

    The ground model fitting module calculates the three-dimensional coordinates of the points belonging to the ground in the real world coordinate system according to the original image and the disparity map, and assumes that the ground is a plane model, and uses RANSAC to fit the plane to obtain a ground model;

    Occupying the map seeking module, converting the entire scene in the original grayscale image from camera coordinates to world coordinates, and generating a plan view, which is obtained by the plan view;

    The obstacle segmentation and distance calculation module divides the position of each obstacle from the occupied map by the connected domain mark detection algorithm, and converts it into the original image to mark it, and calculates the distance of the obstacle to the vehicle through the disparity map;

    The alarm module performs an alarm when the current vehicle distance is less than a certain threshold or enters a decision module to participate in the decision.

Images