CN109522837B

CN109522837B - Pavement detection method and device

Info

Publication number: CN109522837B
Application number: CN201811354494.8A
Authority: CN
Inventors: 赵英芹; 夏克江
Original assignee: Hisense Co Ltd
Current assignee: Hisense Co Ltd
Priority date: 2018-11-14
Filing date: 2018-11-14
Publication date: 2021-10-29
Anticipated expiration: 2038-11-14
Also published as: CN109522837A

Abstract

The invention provides a road surface detection method and a device, a vehicle-mounted binocular camera can calculate a target mask area of a current time period through a target road surface acquired according to the previous time period, then determines fitting candidate points in a V disparity map to be detected according to the target mask area of the current time period, and fits the target road surface of the current time period according to the fitting candidate points. The target mask area can be updated in real time according to the fitted road surface along with the time change, so that the mask area can be adaptive to the change of the road surface, and the detection precision of the road surface with large fluctuation is effectively improved.

Description

Pavement detection method and device

Technical Field

The invention relates to the technical field of auxiliary driving and image processing, in particular to a road surface detection method and device.

Background

In the barrier detection method based on binocular vision, it is very important to effectively detect a road surface. In the current binocular stereo vision, the road surface can be detected by fitting a straight line or a curve in the V disparity map. Besides being greatly influenced by the selection of the fitting algorithm, the road surface fitting effect also has a large influence on the fitting result due to the quality of the disparity map or the V disparity map.

The existing binocular stereo matching algorithm is influenced by factors such as illumination, shielding and dust, and high matching accuracy is difficult to guarantee, so that a large number of mismatching points inevitably exist in a generated disparity map, the mismatching points in the disparity map form noise points when being projected into a V disparity map, and the noise points have great influence on road surface fitting accuracy in the V disparity map. Meanwhile, the vehicle may experience a road surface with large fluctuation or generate large bump in the driving process, which brings difficulty to the road surface detection based on the V disparity map and affects the road surface detection precision.

Disclosure of Invention

In view of this, the present invention provides a road surface detection method and device to solve the problem of low road surface detection accuracy caused by uneven road surface in the prior art.

Specifically, the invention is realized by the following technical scheme:

the invention provides a road surface detection method, which is applied to a vehicle-mounted binocular camera and comprises the following steps:

in the current time period, calculating a target mask area of the current time period according to the target road surface acquired in the previous time period;

inputting a V disparity map to be detected, and determining fitting candidate points in the V disparity map to be detected according to a target mask region of a current time period;

and fitting the target road surface of the current time period according to the fitting candidate points.

Based on the same concept, the invention also provides a road surface detection device, which is applied to a vehicle-mounted binocular camera and comprises:

the calculating unit is used for calculating a target mask area of the current time period according to the target road surface acquired in the previous time period in the current time period;

the determining unit is used for inputting the V parallax image to be detected and determining fitting candidate points in the V parallax image to be detected according to the target mask area of the current time period;

and the fitting unit is used for fitting the target road surface of the current time period according to the fitting candidate points.

Based on the same conception, the invention also provides a vehicle-mounted binocular camera, which comprises a memory, a processor, a communication interface and a communication bus;

the memory, the processor and the communication interface are communicated with each other through the communication bus;

the memory is used for storing a computer program;

the processor is configured to execute the computer program stored in the memory, and the steps of any one of the road surface detection methods according to the present invention are implemented when the processor 72 executes the computer program.

The present invention also provides a computer-readable storage medium having stored therein a computer program which, when executed by a processor, implements the steps of any of the road surface detection methods of the present invention.

Therefore, the vehicle-mounted binocular camera can calculate the target mask area of the current time period through the target road surface acquired according to the last time period, then determines fitting candidate points in the V parallax image to be detected according to the target mask area of the current time period, and fits the target road surface of the current time period according to the fitting candidate points. The target mask area can be updated in real time according to the fitted pavement along with the change of time, and the self-adaptive mask area can accurately contain more pavement candidate points in real time along with the change of the pavement condition, so that the reliable guarantee is provided for the pavement fitting, and the detection precision of the pavement with large fluctuation is effectively improved.

Drawings

FIG. 1 is a schematic illustration of a road surface and noise in a V disparity map in an exemplary embodiment of the invention;

FIG. 2 is a process flow diagram of a method of road surface detection in an exemplary embodiment of the invention;

FIG. 3-1 is a schematic illustration of parallax in an exemplary embodiment of the invention;

3-2 is a V disparity map illustration of a partial disparity map projection in an exemplary embodiment of the invention;

3-3 are schematic illustrations of an initial target mask region in an exemplary embodiment of the invention;

FIG. 4-1 is a schematic illustration of a disparity map of a rough road surface V in an exemplary embodiment of the invention;

FIG. 4-2 is a schematic illustration of an adaptive mask region in an exemplary embodiment of the invention;

FIG. 5-1 is a schematic illustration of a slope in an exemplary embodiment of the invention;

FIG. 5-2 is a schematic illustration of a comparison of a fixed mask region and an adaptive mask region in an exemplary embodiment of the invention;

FIG. 6 is a logical block diagram of a road surface detecting device in an exemplary embodiment of the invention;

fig. 7 is a logical block diagram of an on-vehicle binocular camera according to an exemplary embodiment of the present invention.

Detailed Description

Reference will now be made in detail to the exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, like numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. As used in this application and the appended claims, the singular forms "a", "an", and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the term "and/or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.

It is to be understood that although the terms first, second, third, etc. may be used herein to describe various information, such information should not be limited to these terms. These terms are only used to distinguish one type of information from another. For example, first information may also be referred to as second information, and similarly, second information may also be referred to as first information, without departing from the scope of the present application. The word "if" as used herein may be interpreted as "at … …" or "when … …" or "in response to a determination", depending on the context.

In the barrier detection method based on binocular vision, it is very important to effectively detect a road surface. Currently, in the field of binocular vision, the detection of road surfaces is usually performed by the steps comprising: obtaining a disparity map through stereo matching; calculating a V disparity map according to the disparity map; fitting the road surface in the V disparity map; and determining the road surface in the disparity map according to the fitting result.

In all algorithms for detecting road surfaces and obstacles based on binocular stereo vision technology, obtaining a disparity map by applying stereo matching technology is a key step which must be passed through. The choice of matching features, constraint criteria, similarity measures, etc. will directly affect the quality of the stereo matching algorithm. Meanwhile, external factors such as illumination, shielding and dust can also have certain influence on the matching effect. Therefore, there are many mismatching points, i.e., noise (noise) considered in the present invention, which are unavoidable in the disparity map obtained by stereo matching.

Due to the influence of matching precision, noise in the disparity map will generate large interference on the road fitting effect in the V disparity map. As shown in fig. 1, the inclined white line is the inclined line of the road surface, and the incomplete area enclosed by the dotted line is the position of the noise point. The existence of the noise can generate certain interference to the currently common straight line fitting algorithm, such as least square fitting, Hough transform and the like. Taking Least squares fitting (Least squares fitting) as an example, it finds the best functional match of the data by minimizing the sum of the squares of the errors as a mathematical optimization method. The application of the least square algorithm in the road surface detection can be regarded as a unitary linear regression analysis problem, and n groups of observed values (x) in the total are obtained₁,y₁),(x₂,y₂),…,(x_n,y_n) There are numerous straight lines that can be fitted, but in order for the sample regression function to fit the set of values as well as possible, it is most reasonable that this line be at the data center location of the sample. Therefore, the noise enclosed by the dotted line in fig. 1 affects the effect of the road slope fitting.

In order to solve the problems in the prior art, the invention provides a road surface detection method and a road surface detection device. The target mask area can be updated in real time according to the fitted pavement along with the change of time, and the self-adaptive mask area can accurately contain more pavement candidate points in real time along with the change of the pavement condition, so that the reliable guarantee is provided for the pavement fitting, and the detection precision of the pavement with large fluctuation is effectively improved.

Referring to fig. 2, a processing flow chart of a road surface detection method according to an exemplary embodiment of the invention is shown, the method is applied to a vehicle-mounted binocular camera, and the method includes:

step 201, in a current time period, calculating a target mask area of the current time period according to a target road surface acquired in the previous time period;

as an example, when the vehicle-mounted binocular camera is initialized, an initial road surface may be fitted according to camera parameters; then calculating a mask fluctuation area according to the initialization parameters; and calculating an initial target mask area according to the mask fluctuation area by taking the initial road surface as a reference. Wherein, the process of calculating the initial road surface is as follows:

according to the road surface detection based on the V-disparity map shown in fig. 1, taking the coordinate system of the binocular camera as an example, the attitude of the road plane can be estimated in the V-disparity map:

assuming that the binocular camera is at a height h from the ground, the road plane can be represented in the V disparity map by the following equation:

wherein, b represents the base length of the binocular camera, delta represents the parallax value, f represents the focal length of the camera, and theta is the pitch angle of the binocular camera, and the size of the pitch angle is related to the installation posture of the camera. The further derivation for equation (one) is:

according to the formula (II), if parameters such as the installation height h, the base length b, the focal length f and the pitch angle theta of the binocular camera are obtained through measurement, an estimation can be obtained on the posture of the horizontal road surface in the V parallax map, namely the formula (II).

By measuring parameters of the binocular camera, initial estimation can be further performed on the attitude position of the road surface in the V disparity map according to a formula (II) to obtain an initial road surface.

In a general road surface calculation, the disparity map shown in fig. 3-1 is taken as an example, and a V disparity map obtained by calculating the disparity map of an area of interest (an area where a road surface is located) is shown in fig. 3-2; it can be seen from fig. 3-2 that a relatively inclined oblique line represents the position of the oblique line of the road surface, and the lower portion of the road surface has more scattered points, which are noise points and are interfered by the noise points, and it is difficult to select the fitting candidate points in fig. 3-2.

The present invention therefore proposes to calculate a mask fluctuation region from the initial camera parameters as a template for fitting candidate point selection, as shown in fig. 3-2. In the figure, oblique lines are road surface oblique line positions calculated according to camera parameters, and the mask region is generated by including regions of a certain range upward and downward from the road surface oblique line positions, wherein the range is a mask fluctuation region and is expressed by delta v.

As an example, Δ v may be calculated according to a set road surface fluctuation range Δ h, and the formula for calculating the variation range Δ v of the longitudinal coordinate of the road surface image in the binocular camera coordinate system according to the set road surface fluctuation range is as follows:

wherein, Δ h represents a road surface fluctuation area, b represents a base line length of the binocular camera, and Δ_disparityRepresenting the disparity value, theta is the binocular camera pitch angle. In consideration of the actual road surface fluctuation condition and the system calculation efficiency, Δ V may be set to a constant value of 10 in the present embodiment, i.e., it floats up to 10 pixel units and down to 10 pixel units from the position where the road surface is inclined in the V-disparity map.

After the mask fluctuation region is initially calculated, an initial target mask region can be calculated according to the mask fluctuation region with the initial road surface as a reference. As can be seen from fig. 3 to 3, after the mask fluctuation region is further added based on the road surface slope, an initial target mask region (gray region) is obtained, and thus an initial target road surface is generated by the initial target mask region. 3-3, it can be known that the target mask region can basically contain most of the road surface parallax points, and simultaneously some noise points are filtered out, so that the obtained initial road surface is relatively accurate. In this embodiment, the target road surface of the previous time period that can be obtained in the current time period is the initial target road surface if the current time period is the first time period after initialization. Further, the target mask area for the current time period may be calculated from the target road surface for the previous time period. Specifically, the target mask area for the current time period may be calculated from the mask fluctuation area with reference to the target road surface for the previous time period.

Step 202, inputting a to-be-detected V disparity map, and determining fitting candidate points in the to-be-detected V disparity map according to a target mask area of a current time period;

after the V disparity map to be detected is input in the current time period, the fitting candidate points may be selected according to the target mask region of the current time period.

As an embodiment, all pixel columns of the target mask region located in the current time period in the V-disparity map to be detected may be traversed, and at least one point from each pixel column may be selected as a fitting candidate point.

And step 203, fitting the target road surface in the current time period according to the fitting candidate points.

In this embodiment, the fitting candidate points may be connected to fit a road surface slope, so that the target road surface may be detected by returning to the disparity map according to the fitting result. The method for connecting the fitting candidate points can refer to the prior art, and the invention is not limited.

According to the invention, a new mask area can be calculated according to the target pavement obtained in the current time period, and then the new mask area is used as the target mask area in the next time period; therefore, in the next time period, the fitting candidate points of the next frame of parallax image to be detected can be calculated through the new target mask area, and the road surface oblique line of the next time period is further fitted. And the mask area is continuously updated along with the calculated latest pavement by the reciprocating circulation, so that the self-adaptive mask area is realized.

Because most of flat road surfaces can be detected by detecting the road surfaces through the fixed mask area, but if the road surfaces have large fluctuation, a large detection problem occurs, especially for road conditions with large fluctuation like mountain roads, and the like, if the fixed mask area is used for detecting each next frame, when a vehicle runs to the rugged mountain road areas, as shown in fig. 4-1, only a part of noise points (a dashed-line frame area) are protected in the gray mask area, and more road surface candidate points to be protected are distributed outside the mask area, so that the failure of correct road surface fitting is caused. The detection result of the road surface detection method based on the adaptive mask area provided by the invention is shown in fig. 4-2, and the road surface shown in fig. 4-1 generates fitting candidate points through the adaptive mask area M, so that most of the fitting candidate points can be included through the adaptive mask area M (the gray area in fig. 4-2), most of noise points are eliminated, and the road surface detection result of the detection method based on the adaptive mask area is more accurate compared with the detection method based on the fixed mask area.

In order to compare the difference in detection effect between the adaptive mask area and the fixed mask area, a specific embodiment is described below. Considering that the adaptive pavement mask area generation method proposed by the patent is similar to a method which continuously changes from time to space, the effectiveness can be verified by a group of time sequence samples.

Fig. 5-1 is a binocular camera reference diagram acquired at certain time intervals Δ t during driving of an automobile. The distance of the road section is an uphill road, and obviously, the road surface at the bottom of the slope has a bulge, and the road surface detection process is difficult due to the existence of the slope and the bulge.

Fig. 5-2 is a resultant V disparity map projected based on the above binocular camera disparity map, in which the left diagram shows the road surface detection effect of the fixed mask region, and the right diagram shows the road surface detection effect of the adaptive mask region. The method has the advantages that the pavement can be visually detected from time sequence samples, the fixed mask area can accurately detect the pavement at the beginning, but the detected pavement is inaccurate due to the fact that the fixed mask area cannot follow the pavement self-adaptive change after the pavement undulation changes due to time change; and the self-adaptive mask area can be changed in real time along with the change of the pavement condition.

Therefore, the pavement is detected through the self-adaptive mask area, the gradual change characteristic of the pavement is fully considered, and the target mask area of the current frame is calculated by referring to the pavement detection result of the previous frame, so that the pavement area can be effectively contained, more pavement candidate points can be accurately contained, and reliable guarantee is provided for pavement fitting. Therefore, the invention can solve the problem of flat road surface detection and has better detection effect on special road conditions with larger fluctuation and the like.

Based on the same conception, the invention also provides a road surface detection device, which can be realized by software, hardware or a combination of the software and the hardware. Taking software implementation as an example, the road surface detection device of the present invention is a logical device, and is implemented by reading a corresponding computer program instruction in a memory by a CPU of the device in which the road surface detection device is located and then operating the computer program instruction.

Referring to fig. 6, a road surface detecting device 600 according to an exemplary embodiment of the present invention is applied to a vehicle-mounted binocular camera, and the logical structure of the device 600 includes:

a calculating unit 601, configured to calculate, in a current time period, a target mask region of the current time period according to a target road surface acquired in a previous time period;

a determining unit 602, configured to input a to-be-detected V-disparity map, and determine fitting candidate points in the to-be-detected V-disparity map according to a target mask region of a current time period;

and a fitting unit 603, configured to fit the target road surface in the current time period according to the fitting candidate points.

As an embodiment, the apparatus further comprises:

an initialization unit 604, configured to fit an initial road surface according to the camera parameters during initialization; calculating a mask fluctuation area; and calculating an initial target mask area according to the mask fluctuation area by taking the initial road surface as a reference.

As an embodiment, the initialization unit 604 calculates a mask fluctuation region, specifically:

area of mask fluctuation

Wherein, Δ h represents a road surface fluctuation area, b represents a base line length of the binocular camera, and Δ_disparityRepresenting the disparity value, theta is the binocular camera pitch angle.

As an embodiment, the determining unit 602 is specifically configured to traverse a pixel column located in the target mask area in the to-be-detected V-disparity map; at least one point from each pixel column is selected as a fitting candidate point.

Based on the same concept, the present invention also provides an on-vehicle binocular camera, as shown in fig. 7, including a memory 71, a processor 72, a communication interface 73, and a communication bus 74;

the memory 71, the processor 72 and the communication interface 73 communicate with each other through the communication bus 74;

the memory 71 is used for storing computer programs;

the processor 72 is configured to execute the computer program stored in the memory 71, and when the processor 72 executes the computer program, any step of the road surface detection method provided in the embodiment of the present invention is implemented.

The invention further provides a computer-readable storage medium, in which a computer program is stored, and the computer program, when executed by a processor, implements any one of the steps of the road surface detection method provided by the embodiment of the invention.

All the embodiments in the present specification are described in a related manner, and the same and similar parts among the embodiments may be referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, for embodiments of the computer device and the computer-readable storage medium, since they are substantially similar to the method embodiments, the description is relatively simple, and in relation to what is described in the partial description of the method embodiments.

In summary, the vehicle-mounted binocular camera of the present invention may calculate the target mask area of the current time period through the target road surface acquired according to the previous time period, then determine the fitting candidate points in the to-be-detected V-disparity map according to the target mask area of the current time period, and fit the target road surface of the current time period according to the fitting candidate points. The target mask area can be updated in real time according to the fitted pavement along with the change of time, and the self-adaptive mask area can accurately contain more pavement candidate points in real time along with the change of the pavement condition, so that the reliable guarantee is provided for the pavement fitting, and the detection precision of the pavement with large fluctuation is effectively improved.

The implementation process of the functions and actions of each unit in the above device is specifically described in the implementation process of the corresponding step in the above method, and is not described herein again.

For the device embodiments, since they substantially correspond to the method embodiments, reference may be made to the partial description of the method embodiments for relevant points. The above-described embodiments of the apparatus are merely illustrative, and the units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the scheme of the application. One of ordinary skill in the art can understand and implement it without inventive effort.

The above description is only for the purpose of illustrating the preferred embodiments of the present invention and is not to be construed as limiting the invention, and any modifications, equivalents, improvements and the like made within the spirit and principle of the present invention should be included in the scope of the present invention.

Claims

1. A road surface detection method is characterized in that the method is applied to a vehicle-mounted binocular camera, and comprises the following steps:

in the current time period, calculating a target mask area of the current time period according to the target road surface acquired in the previous time period; wherein the initial target mask area implementation comprises: at the beginningDuring initialization, fitting an initial road surface according to camera parameters; calculating a mask fluctuation area; calculating an initial target mask area according to a mask fluctuation area by taking an initial road surface as a reference, wherein the calculating of the mask fluctuation area comprises the following steps: area of mask fluctuation

Wherein, Δ h represents a road surface fluctuation area, b represents a base line length of the binocular camera, and Δ_disparityRepresenting a parallax value, and theta is a binocular camera pitch angle;

inputting a V parallax image to be detected, and traversing pixel columns in the target mask area in the V parallax image to be detected; selecting at least one point from each pixel column as a fitting candidate point;

2. The utility model provides a road surface detection device which characterized in that, the device is applied to on-vehicle binocular camera, the device includes:

the calculating unit is used for calculating a target mask area of the current time period according to the target road surface acquired in the previous time period in the current time period; wherein the initial target mask area implementation comprises: during initialization, fitting an initial road surface according to camera parameters; calculating a mask fluctuation area; calculating an initial target mask area according to the mask fluctuation area by taking the initial road surface as a reference; the calculating the mask fluctuation area specifically comprises the following steps: area of mask fluctuation

the determining unit is used for inputting the V disparity map to be detected and traversing pixel columns in the target mask area in the V disparity map to be detected; selecting at least one point from each pixel column as a fitting candidate point;

3. The vehicle-mounted binocular camera is characterized by comprising a memory, a processor, a communication interface and a communication bus;

the memory is used for storing a computer program;

the processor is used for executing the computer program stored in the memory, and the steps of the method are realized when the processor executes the computer program.

4. A computer-readable storage medium, in which a computer program is stored which, when being executed by a processor, carries out the steps of the above-mentioned method.