CN104166834A

CN104166834A - Pavement detection method and pavement detection device

Info

Publication number: CN104166834A
Application number: CN201310187630.XA
Authority: CN
Inventors: 陈超; 游赣梅; 师忠超; 鲁耀杰; 王刚
Original assignee: Ricoh Co Ltd
Current assignee: Ricoh Co Ltd
Priority date: 2013-05-20
Filing date: 2013-05-20
Publication date: 2014-11-26
Anticipated expiration: 2033-05-20
Also published as: CN104166834B

Abstract

The invention provides a pavement detection method and a pavement detection device. The pavement detection method comprises the following steps: a disparity map and a gray-scale map including a pavement are obtained; a road marker capable of identifying the location of the pavement is detected from the gray-scale map; a full V disparity map based on a full map is constructed from the disparity map; a first pavement point is selected from the full V disparity map; a road indicative V disparity map based on the detected road marker is constructed from the disparity map; a second pavement point is selected from the road indicative V disparity map; and the pavement is extracted based on the first pavement point and the second pavement point. According to the pavement detection method and the pavement detection device provided by the embodiment of the invention, pavement extraction is carried out by adoption of a V disparity map based on a whole disparity map and a road marker based on things like left and right lane lines. The method and the device are very effective for detection of sloping pavements, and a pavement extracted by the method and the device is more in line with the actual situation. In addition, two-dimensional data is mainly operated and processed in the method, so that feature points are enhanced, and the amount of calculation is greatly reduced.

Description

Pavement detection method and apparatus

Technical field

The present invention relates to image and process, relate more specifically to pavement detection method and apparatus.

Background technology

The application of drive assist system is day by day universal.Road or track warning system (Lane/Road detection warning, LDW/RDW) are the subsystems of drive assist system, can avoid collision, determine more accurately and drive direction etc.Road Detection is very crucial for LDW/RDW system, only on the basis of having known road information, just may do further processing, for example warning.And it is also the very important part of 3D driving environment understanding technology that pavement of road detects, the roadside guardrail in 3D driving environment understanding technology is detected, vanishing Point Detection Method, vehicle identification etc. have a great impact.

Yet also there are some problems in existing road surface algorithm for estimating: they can carry out accurate pavement detection in simple environment, but the detection that may usually make a mistake in complex environment.

In the open US20090041337A1 of the United States Patent (USP) that is entitled as " Image Processing Apparatus and Method ", based on 3D information, detect roadmarking, and then utilize this roadmarking and 3D information to carry out road surface estimation.This invention directly utilizes 3D information to carry out lane detection in disparity map, then in disparity map, carry out road surface estimation, thereby calculated amount is large comparatively speaking, and complexity is high.

Author, be Zhencheng Hu, Francisco Lamosa, Keiichi Uchimura, be entitled as " AComplete U-V-Disparity Study for Stereovision Based3D Driving Environment Analysis. " Proceedings of the5th international Conference on3-D Digital Imaging and Modeling, Jun.13, in 2005. document, a kind of road scene analytical algorithm based on stereoscopic vision has been proposed.This technology is by construct V disparity map from original disparity map, to the road plane in 3D road scene, and non-road area, barriers etc. are classified.This technology is for the situation that has complex road conditions such as inclinations grade such as the road surface detection that may make a mistake.

Summary of the invention

Based on said circumstances, the present invention has been proposed.

According to an aspect of the present invention, propose a kind of pavement detection method, can comprise: obtained the disparity map and the gray-scale map that comprise road surface; From gray-scale map, detect the Sign for road that can identify position, road surface; From disparity map, build the full V disparity map based on full figure; From full V disparity map, select first via millet cake; From disparity map, build the indicative V disparity map of road of the Sign for road based on detecting; From the indicative V disparity map of road, select the second road surface point; And extract road surface based on first via millet cake and the second road surface point.

According to an aspect of the present invention, proposed a kind of road surface checking device, can comprise: image has obtained parts, for obtaining disparity map and the gray-scale map that comprises road surface; Sign for road detection part, for detecting the Sign for road that can identify position, road surface from gray-scale map; Full V disparity map builds parts, for building the full V disparity map based on full figure from disparity map; First via millet cake alternative pack, for selecting first via millet cake from full V disparity map; The indicative V disparity map of road builds parts, for build the indicative V disparity map of road of the Sign for road based on detecting from disparity map; The second road surface point selection parts, for selecting the second road surface point from the indicative V disparity map of road; And road surface extraction parts, for extracting road surface based on first via millet cake and the second road surface point.

Utilize pavement detection method according to the above embodiment of the present invention and road surface checking device, adopt the V disparity map of V disparity map based on whole disparity map and Sign for road based on such as left and right lane line simultaneously, carry out road surface extraction, detection for tilted road surface is very effective, and the road surface of extracting more tallies with the actual situation.In addition, utilize pavement detection method according to the above embodiment of the present invention and road surface checking device, in gray-scale map, detect the Sign for road such as lane line, and carry out road surface straight line estimation based on Sign for road structure V disparity map being detected, these two step process be all 2-D data, unique point has obtained enhancing thus, and calculated amount significantly reduces simultaneously.

According to another aspect of the present invention, propose a kind of pavement detection method, can comprise: obtained the disparity map that comprises road surface; From disparity map, build V disparity map; Select road surface point; And extract road surface based on road surface point.Wherein, select road surface point to comprise: the architectural feature of obtaining each pixel in V disparity map, the architectural feature of a pixel is based on the accumulation of all pixels in region centered by this pixel, that have given shape and pre-sizing and obtains, and the shape in this region is mated with the situation of road surface projection in V disparity map; Architectural feature based on the pixel of every row in V disparity map, determines the dynamic threshold of these row; Wherein to meet its Structural Eigenvalue highly minimum in the dynamic threshold of row and pixel that this road surface point is all dynamic thresholds that are greater than these row in these row under being greater than for selected road surface point.

According to another aspect of the present invention, propose a kind of road surface checking device, can comprise: image has obtained parts, obtained the disparity map that comprises road surface; V disparity map builds parts, for build V disparity map from disparity map; Road surface point selection parts, for selecting road surface point; And road surface extraction parts, for extracting road surface based on road surface point.Wherein, road surface point selection parts select road surface point to comprise: the architectural feature of obtaining each pixel in V disparity map, the architectural feature of a pixel is based on the accumulation of all pixels in region centered by this pixel, that have given shape and pre-sizing and obtains, and the shape in this region is mated with the situation of road surface projection in V disparity map; Architectural feature based on the pixel of every row in V disparity map, determines the dynamic threshold of these row; Wherein the architectural feature of selected road surface point meet be greater than under the dynamic threshold of row and this road surface point be highly minimum in the pixel of all dynamic thresholds that are greater than these row in these row.

Utilize pavement detection method according to the above embodiment of the present invention and road surface checking device, as previously mentioned, what consider due to the architectural feature of pixel is overall condition rather than an independent pixel of a pixel in region, so weak road surface has obtained corresponding enhancing.Meanwhile, owing to having calculated accumulated value, can effectively remove noise.

In addition, because according to road surface, the projection of shape in V disparity map decides for calculating the shape in region of the architectural feature of a pixel, therefore resulting architectural feature is more suitable for pavement detection application.

In choosing according to the road surface point of the present embodiment, because take to be suitable for for example dynamic threshold of every row of regional area, therefore can determine adaptively dynamic threshold, therefore can select better the road surface point of each regional area.

Accompanying drawing explanation

Fig. 1 be contribute to understand of the present invention, as the onboard system schematic diagram of the example of applied environment of the present invention.

Fig. 2 is the overall flow figure of pavement detection method 1000 according to an embodiment of the invention.

Fig. 3 schematically shows the result example of a lane detection.

(a) in Fig. 4 schematically shows the example that vehicle-mounted stereoscopic camera is taken the road area disparity map obtaining.(b) in Fig. 4 schematically shows the V disparity map that the conversion of disparity map as shown in Figure 3 obtains, also i.e. described full V disparity map herein.

Fig. 5 shows the process flow diagram of the illustrative methods 1500 of the indicative V disparity map of road that builds according to an embodiment of the invention the Sign for road based on detecting from disparity map.

Fig. 6 (a) is identical with Fig. 3, show the lane detection result in gray-scale map in an example, Fig. 6 (b) shows detection area-of-interest corresponding in definite in one example disparity map, wherein shows two regions as the region of interest (ROI) of detecting.

(a) in Fig. 7, (b), (c) show the V disparity map organigram based on left and right lane line of an example.(a) in Fig. 7 is identical with Fig. 6 (b), show detection area-of-interest corresponding in definite in one example disparity map, (b) in Fig. 7 shows the left side road V disparity map schematic diagram based on left-lane line ROI structure, (d) in Fig. 7 shows the right side road V disparity map schematic diagram based on right lane line ROI structure, and the V disparity map that (c) in Fig. 7 shows based on left and right lane line ROI corresponds respectively to corresponding part in the V disparity map based on full figure.

Fig. 8 shows the process flow diagram of the exemplary method 1700 based on first via millet cake and the second road surface point extraction road surface according to an embodiment of the invention.

It is that each road surface line segment extracting in the example of left and right lane line is unified in a schematic diagram in V disparity map that Fig. 9 shows at Sign for road.

Figure 10 has provided the schematic diagram of left side the second road surface line segment, right side the second road surface line segment and the road surface envelope that first via upper thread section draws entirely based on shown in Fig. 9.

Figure 11 shows the process flow diagram of road surface point extracting method 1400 according to an embodiment of the invention.

In Figure 12, schematically provided the testing result of road surface historical frames and definite surveyed area accordingly, and schematically shown based on the definite surveyed area of camera parameters.

Figure 13 shows according to an embodiment of the invention the schematic diagram for the region of the architectural feature of calculating pixel point Pi.

Figure 14 shows the schematic diagram of the road surface candidate point of tentatively choosing for the V disparity map based on full disparity map.

Figure 15 (a) schematically shows the road surface point example of tentatively choosing, and Figure 15 (b) has schematically shown the road surface point example of finally choosing after discrete point is removed.

What Figure 16 was schematically illustrated in each V disparity map Road millet cake chooses the set of result in a V disparity map.

Figure 17 (a1)-(c1) and (a2)-(c2) show for more traditional road surface point choosing method with according to the schematic diagram of the result of the road surface point choosing method of the embodiment of the present invention.

Figure 18 illustrates according to the overview flow chart of the pavement detection method 1000 ' of second embodiment of the invention.

Figure 19 is illustrated in the process flow diagram that the road surface of extracting in V disparity map is the exemplary road surface inverse mapping method 1800 in envelope situation.

Figure 20 (a) and Figure 10 are same, schematically show the road surface envelope extracting in V disparity map according to one embodiment of the invention, Figure 20 (b) schematically shows according to an embodiment of the invention the road surface point inverse mapping result in the disparity map of the road surface envelope based on extracting.

Figure 21 (a1)-(c1) and (a2)-(c2) has shown the road surface result that different detection methods obtain.

Figure 22 shows according to the process flow diagram of the pavement detection method 2000 of third embodiment of the invention.

Figure 23 shows the block diagram of road surface checking device 4000 according to an embodiment of the invention.

Figure 24 shows the block diagram of road surface checking device 5000 according to another embodiment of the present invention.

Figure 25 is the concept map illustrating according to the hardware configuration of the pavement detection system of the embodiment of the present invention 6000.

Embodiment

In order to make those skilled in the art understand better the present invention, below in conjunction with the drawings and specific embodiments, the present invention is described in further detail.In addition, for fear of obscuring main points of the present invention, for some technology well known in the art, will not elaborate.

To be described in the following order:

1, key concept introduction

2, applied environment signal

3, the first embodiment of pavement detection method

3.1, the overall flow of pavement detection method

3.2, the detailed example of road surface point extracting method

4, the second embodiment of pavement detection method

5, the 3rd embodiment of pavement detection method

6, the first embodiment of road surface checking device

7, the second embodiment of road surface checking device

8, system hardware configuration

9, sum up

1, key concept introduction

Introduce key concept below, so that understand.

Parallax, actually refers to respectively draw from a certain baseline two ends a straight line to same during compared with far object, the angle that become therebetween.Refer generally to from there being two points of certain distance to observe the direction difference that same target produces.From target, see two angles between point, be called the parallactic angle of these two points, the distance between 2 is called baseline.As long as know parallax angle and base length, just can calculate the distance between target and observer.

Disparity map (disparity map) is to take that to appoint a piece image be benchmark, and its size is the size of this benchmark image, the image that element value is parallax value.The range information that disparity map has comprised scene.The left image that disparity map can be taken from binocular camera and right image, calculate, or calculate by the depth map in three-dimensional view.

Certain point coordinate in ordinary two dimensional disparity map is with (x, y, d) expression, and wherein x is horizontal ordinate, and y is ordinate, and d represents the parallax value at this pixel place.

For ease of understanding, figuratively, V disparity map can be considered as the side view of disparity map, and U disparity map can be considered as the vertical view of disparity map.V disparity map can calculate from disparity map.In V-disparity map, the gray-scale value of any point (d, y) is the number that in the ordinate of the corresponding disparity map row that is y, parallax value equals the point of d.

Xian Huo road surface, road surface line segment: the broadside lines that represents pavement of road, in the real 3D world, road is roughly a 3D plane, but through after side projection, the road surface side facial contour forming in V disparity map is roughly the line segment of 2D, is called Xian Huo road surface, road surface line segment here.

2, applied environment schematic diagram

Fig. 1 be contribute to understand of the present invention, as the onboard system schematic diagram of the example of applied environment of the present invention.Road Detection parts or its part that software of the present invention or hardware can be used as wherein realize.

3, the first embodiment of pavement detection method

3.1, the overall flow of pavement detection method

Below with reference to Fig. 2, pavement detection method is according to an embodiment of the invention described.

As shown in Figure 2, in step S1100, obtain the disparity map and the gray-scale map that comprise road surface.

In an example, can take to obtain the gray-scale map (as left eye gray-scale map) that comprises region, road surface by vehicle-mounted stereoscopic camera, and from for example left eye gray-scale map and right eye gray-scale map by calculating corresponding disparity map.Stereoscopic camera is such as having binocular camera, many orders camera etc.

But, in another example, can take respectively and obtain gray-scale map and disparity map by different cameras, as long as can determine that the Conversion Relations between the coordinate system between image can be transformed in a unified coordinate system both.For example, can take and obtain gray-scale map by the general camera carrying on automobile, and take and calculate disparity map by the binocular camera carrying on automobile, in this case, the position relationship of given general camera and binocular camera and performance parameter separately, can determine the coordinate transformation relation between gray-scale map and disparity map.

In an example, the image capture device of photographic images and carry out image processing and can mutually be positioned at this locality to detect the signal processing apparatus on road surface, or even integrate.In another example, the camera of photographic images and carry out image processing with detect the signal conditioning package on road surface can be mutually away from, by for example wired or radio communication, gray-scale map and/or disparity map are delivered to signal processing apparatus.

Hereinafter, for convenience of description, take video camera and being all equipped on vehicle as example is described for the signal conditioning package of Road Detection.

Next, in step S1200, from gray-scale map, detect the Sign for road that can identify position, road surface.

Here, for example, Sign for road can be lane line, and the position of lane line has identified the position on road surface.But, Sign for road is not limited to this, but can be for identifying anything of position, road surface, such as the fence on road, and curb stone, road is on both sides of the road or middle grove etc.Particularly, for example, the bottom position of the fence on road has been indicated position, road surface.Similarly, the crowd that on road, a group is advanced also can be used as Sign for road, because crowd's bottom position has been indicated position, road surface.

But, because inspection vehicle diatom relatively easily and has more directly been indicated position, road surface in gray-scale map, therefore adopt lane line advantageous particularly as Sign for road possibility.Hereinafter, for convenience of description, the lane line of take is described as example.

About the lane detection based on gray-scale map, the method for many lane detections is disclosed in prior art, these existing method for detecting lane lines all can be for the present invention.

Provide one below according to the illustrative methods of carrying out lane detection of the embodiment of the present invention.Therefore first, determine the roughly surveyed area of lane line, for example, because video camera is fixed on vehicle, in Vehicle Driving Cycle process, lane line is arranged in the specific region of image, can this specific region be set to detect ROI(area-of-interest).Secondly, in detecting ROI, by the edge detection operator such as Sobel, process gray level image, thereby create outline map, wherein consider that lane line has specific angle, can be only by Grad, the marginal point within this angular range remains and carries out subsequent treatment; In addition, when carrying out lane detection, there are many features can be used for carrying out lane detection, such as lane width, length, color etc.Finally, can adopt Hough conversion in outline map, to carry out sectional straight line fitting.The lane line information of output is as shown in the formula shown in (1):

Left-lane line: ysin (θ _l)+xcos (θ _l)=ρ _l(1)

Right lane line: ysin (θ _r)-xcos (θ _r)=ρ _r

Wherein, x, y is the coordinate in gray-scale map, θ _l, ρ _l, θ _r, ρ _rbe respectively left and right lane line parameter.If there is no lane line on road surface, this module judges the accumulated value converting by Hough, track output parameter is set to 0 and identifies.Fig. 3 schematically shows the result example of a lane detection.

Next, in step S1300, from disparity map, build the full V disparity map based on full figure.

As previously mentioned, can, by being mapped to a little a vertical plane (V-vertical dimension, Δ-depth dimension) in original disparity map, generate V disparity map.Each point in former disparity map represents the point (U-horizontal dimension, V-vertical dimension, Δ-depth dimension) in three-dimensional world coordinate system.After mapping, in the middle of new image, horizontal dimension has disappeared, and this new image is exactly so-called V disparity map.In V disparity map, the intensity of each point is exactly the accumulation number of point in x coordinate (U-horizontal dimension) direction in former disparity map with same disparity value and identical y coordinate (V-vertical dimension).Therefore in this V disparity map, road surface has obtained enhancing, and the quantity of point reduced greatly, carries out efficiency and the accuracy that pavement detection can significantly improve pavement detection based on V disparity map.

Next, in step S1400, from full V disparity map, select first via millet cake.

If do not add processing, V disparity map is not carried out to line segment as road surface by line fitting method, because noise is a lot, testing result out of true; And calculated amount is very large.

According to one embodiment of the invention, filter V disparity map to carry out the selection of road surface point, to remove Null Spot, improve counting yield, prevent noise.

According to an example, when selecting road surface point, can consider that the intensity of road surface point should be greater than certain threshold value, and road surface point is generally positioned at the bottom of V disparity map.

According to one embodiment of present invention, from V disparity map, select road surface point to comprise: the architectural feature of obtaining each pixel V disparity map, the architectural feature of a pixel is based on the accumulation of all pixels in region centered by this pixel, that have given shape and pre-sizing and obtains, the shape in this region is mated with the situation of road surface projection in V disparity map, this is for when architectural feature is extracted, and the architectural feature that is positioned at the pixel on road surface can be included contiguous road surface point as much as possible.In one example, in binocular camera level, be equipped on vehicle and basic horizontal is taken forward in the situation that, this region be shaped as approximate parallelogram, as shown in figure 13, the approximate line style projection matchings at such region shape and the road surface approximate 135 degree angles that tilt in V disparity map.That is, generally, in camera level, carry on vehicle and the in the situation that level is taken to front lower place, in the situation (in reality, most of situation is like this) of non-fully-flattened, road surface, the projection approximation of road surface in V disparity map is parallelogram shape.In another example, the shape in region can be flattened rectangular, and when a pixel being carried out to structure-pixel extraction, this rectangle major axis is approximate to tilt along 135 directions, and this rectangle major axis is approximate parallel in the projection of V disparity map with road surface in other words.But, for example, when camera is taken road surface vertically downward, the projection line of road surface in V disparity map will be no longer vergence direction, but approximate 90 degree, perpendicular to transverse axis, now for extracting the shape in the region that architectural feature chooses, can correspondingly adjust, be for example that major axis is perpendicular to the rectangle of transverse axis.Above-mentioned shape (angle that comprises inclination, the size of shape etc.) of carrying out the region of pixel accumulation can be adjusted according to actual conditions.What consider due to architectural feature is overall condition rather than an independent pixel of a pixel in region, so weak road surface can access corresponding enhancing.Meanwhile, owing to having calculated accumulated value, can effectively remove noise.

In one example, can dynamically determine each row for filtering the threshold value of road surface point.For example, architectural feature that can be based on the pixel of every row in V disparity map, determines the dynamic threshold of these row.

In one example, should to meet its Structural Eigenvalue highly minimum in the dynamic threshold of row and pixel that this road surface point is all dynamic thresholds that are greater than these row in these row under being greater than for selected road surface point.

In one example, select road surface point also to comprise that only in particular detection region, carrying out road surface point chooses, and is excluded in outside road surface point candidate and directly will be positioned at the extra-regional pixel of this particular detection.For example, in one example, can determine the first surveyed area in V disparity map and based on determine the second surveyed area of V disparity map for obtaining parameter such as the setting angle etc. of the video camera of disparity map and gray-scale map based on historical trace information; And adopt the common factor of the first surveyed area and the second surveyed area as surveyed area, wherein only in surveyed area, carry out road surface point selection.Like this, can significantly reduce calculated amount and get rid of noise.

In one example, in the point selection of road surface, it is also conceivable that removal discrete point.For example, calculate road surface point apart from the distance of the road surface line previously having detected, and from the point of road surface, get rid of the point that is greater than predetermined threshold with the distance of the road surface line previously having detected.

Hereinafter, with reference to Figure 11, describe in detail and carry out according to an embodiment of the invention the example of road surface point selection.

In addition, in the application for a patent for invention that the old superfine application number of making is CN201210194074.4 by identical inventor application for a patent for invention and application number are CN201210513215.4, also the method example that has provided road surface point selection, these methods all can be applied to the present invention.Here by reference above-mentioned patent documentation is integrated with herein.

Next, in step S1500, from disparity map, build the indicative V disparity map of road of the Sign for road based on detecting.

The difference of this road sign V disparity map and previously described full V disparity map is, full V disparity map is based on view picture anaglyph, and road sign V disparity map is only the part of the disparity map based on corresponding with Sign for road domain of dependence.

The illustrative methods that builds according to an embodiment of the invention the indicative V disparity map of road of the Sign for road based on detecting from disparity map is described below with reference to Fig. 5.

As shown in Figure 5, the method is usingd the lane line information that detects and original disparity map as input 1510, in step S1520, according to the position of lane line, determines corresponding detection region of interest ROI in original disparity map.The number of region of interest is corresponding to the number of lane line.

Particularly, because disparity map and the coordinate of gray-scale map are corresponding or can determine Conversion Relations, therefore by the position of lane line in gray-scale map, can determine its position in disparity map.Meanwhile, according to camera parameters, can also in disparity map, determine real distance, the information such as width.Accordingly, centered by lane line, according to actual range, set ROI.In fact definite ROI is a rectangle in 3d space, but in anaglyph, is shown as a trapezoidal shape.

In step S1530, the region of interest ROI based on definite in original disparity map, constructs corresponding V disparity map.

Above, take that to have two lane lines be example, illustrated and two lane line region of interest detected, thereby correspondingly constructed two lane line V disparity maps.But the number of lane line is not limited to two, and can be one, three or more.And the number of lane line may in this case, can not construct corresponding lane line V disparity map for zero bar yet, and then follow-up pavement detection will only be carried out according to full V disparity map.

In step S1500, build after the indicative V disparity map of road completes, advance to step S1600.

In step S1600, from the indicative V disparity map of road, select the second road surface point.

In step S1600, from the indicative V disparity map of road, select the operation that selects on the second road surface to be similar to from full V disparity map, to select the operation of first via millet cake among step S1400.But it should be noted that, in step S1600 from the indicative V disparity map of road select operation that the second road surface selects can among step S1400, from full V disparity map, select the operation of first via millet cake different, can be that integrated operation step is different, also can be the details of operation difference wherein having, even neither same.

In the situation that there is the indicative V disparity map of a plurality of Roads, in the situation as shown in Fig. 6 (a), (b) and Fig. 7 (a)-(d), from the indicative V disparity map of each Road, select road surface point separately respectively.

In step S1600, from the indicative V disparity map of road, select to advance to step S1700 after the second road surface point.

In step S1700, based on first via millet cake and the second road surface point, extract road surface.

Below with reference to Fig. 8, the example of the method based on first via millet cake and the second road surface point extraction road surface is according to an embodiment of the invention described.

As shown in Figure 8, the first via millet cake that is input as aforementioned extraction of the method and the second road surface point 1710.

In step S1720, based on first via millet cake, extract the first via upper thread section that represents road surface; And the second road surface line segment that represents road surface based on the second road surface point extraction.Relevant to putting the method for matching or extraction line segment can for example utilize Hough transformation method or least square method, or utilize the combination of these two kinds of methods.The road surface line-fitting method providing in the application for a patent for invention that the aforesaid application for a patent for invention that is CN201210194074.4 by the old superfine application number of making of identical inventor and application number are CN201210513215.4 all can be applied to the present invention.

Here it should be noted that, the second road surface point may comprise many groups road surface point, for example, the in the situation that of the lane line of aforementioned detection left and right, as previously mentioned, comprise one group of left side second road surface point and one group of right side second road surface point.Certainly not only two of existing lane lines, but be in the situation of more, there are many groups of the second road surface points of corresponding number, or in the situation that lane line is one, there is one group of second road surface point, and therefore based on second road surface point extraction the second road surface line segment, also just can comprise that organize the second road surface point respectively extracts each corresponding the second road surface line segment based on each.

It is that each road surface line segment extracting in the example of left and right lane line is unified in a schematic diagram in V disparity map that Fig. 9 shows at Sign for road, wherein by the line segment of numeral 1 and associated arrows indication, indicated based on left side the second road surface point corresponding to left-hand lane line and left side the second road surface line segment that matching obtains, line segment by numeral 2 and associated arrows indication is indicated based on right side the second road surface point corresponding to right-hand lane line and right side the second road surface line segment that matching obtains, the line segment indication full first via upper thread section that corresponding first via millet cake matching obtains based on full V disparity map by numeral 3 and associated arrows indication.

After extraction obtains first via upper thread section and (one or more) second road surface line segment in step S1720, advance to step S1730.

In step S1730, extract the envelope of first via upper thread section and the second road surface line segment as road surface.Each road surface line segment obtaining based on matching in above-mentioned steps S1720, extracts their envelope as the road surface detecting.

For example, continuing take situation about describing in conjunction with Fig. 9 is above example, and on the left of supposing, the second road surface line segment, right side the second road surface line segment and full first via upper thread section can use respectively following formula (2), (3), (4) to represent:

h _L(d)＝k _L·d+b _L…(2)

h _R(d)＝k _R·d+b _R…(3)

h _W(d)＝k _W·d+b _W…(4)

Wherein, h _l(d), h _r(d), h _w(d) be the pavement-height in each V figure, i.e. vertical dimension coordinate in V disparity map, d is the parallax value that distance D (being actual physics distance) is located, i.e. horizontal dimension coordinate in V disparity map, k and b are the road surface line parameters estimating.Can be as shown in the formula the envelope that extracts each road surface line segment shown in (5), (6), (7) as pavement-height scope h _e(d):

h _E(d)＝[h _Emin(d),h _Emax(d)] (5)

h _Emax(d)＝max(h _L(d),h _R(d),h _W(d)) (6)

h _Emin(d)＝min(h _L(d),h _R(d),h _W(d)) (7)

Wherein, h wherein _emax(d) be illustrated in the maximal value of each road surface line segment height at parallax value d place, be left side the second road surface line segment, right side the second road surface line segment and full first via upper thread section at the pavement-height h at parallax value d place _l(d), h _r(d), h _w(d) maximal value in.H wherein _emin(d) be illustrated in the minimum value of each road surface line segment height at parallax value d place, be left side the second road surface line segment, right side the second road surface line segment and full first via upper thread section at the pavement-height h at parallax value d place _l(d), h _r(d), h _w(d) minimum value in.Thus, envelope represents the pavement-height minimum value h by each parallax value d place _eminand pavement-height maximal value h (d) _emax(d) the boundary line h forming _emax(d).And pavement-height scope h _e(d) in this pavement-height minimum value h _eminand pavement-height maximal value h (d) _emax(d) between.

The method 1700 is output as the road surface envelope 1740 of extraction.

Figure 10 has provided the schematic diagram of left side the second road surface line segment, right side the second road surface line segment and the road surface envelope that first via upper thread section draws entirely based on shown in Fig. 9.As shown in Figure 10, generally, real road surface broadside lines is not desirable line segment, but has certain thickness, because generally road surface is not strict level in x direction (horizontal direction), but has certain inclination.In the example of Figure 10, each section of matching line drawing has out been stated the different part in road surface: the tolerance that the road surface line simulating in the V figure based on full disparity map (being full first via upper thread section mentioned above) is rough side, whole road surface height; Based on a left side, the road surface line simulating in the V figure of right lane line (being left side the second road surface line segment and right side the second road surface line segment mentioned above) has been described respectively the road surface side height of left side with right side.So according to embodiments of the invention, extract the envelope that simulates line segment in each V figure describe actual medium dip, there is certain thickness road surface side facial contour.This envelope is the outline of all V figure Road upper thread, and this is realistic road conditions better.

In one example, lane line is two and substantially symmetrical about road surface, under this application example, the road surface that utilizes two groups of the left and right based on left and right V disparity map road surface point that the above embodiment of the present invention extracts and the road surface point based on full V disparity map extraction to extract can meet the actual conditions of road better.Certainly, lane line can be asymmetrical, and the number of lane line is not limited to two, and can be arbitrary number.

In above-mentioned example, based on road surface point on the same group not, distinguish matching road surface line segment respectively, and the envelope that extracts each road surface line segment is as road surface.But this is only preferred exemplary, but and nonrestrictive.In another example, property implementation, can merge first via millet cake and the second road surface point without distinction as the set of road surface point as an alternative, and is used as road surface based on the incompatible extraction of road surface point set road surface line segment; Or in an example again, also can in conjunction with directly extracting the outer envelope that such road surface point gathers, be used as road surface based on whole road surface points

In addition, in the road scene of above describing, there is the situation of lane line, the in the situation that of there is not lane line in road scene, according to one embodiment of the invention, can only adopt the line of matching in the V figure based on full disparity map as final road surface broadside lines, like this will be to some unstructured roads, as backroad, rural roads etc., have good adaptability.

Turn back to Fig. 2, after step S1700 finishes, pavement detection method finishes.

Described above according to the overall flow of the pavement detection method of first embodiment of the invention.

Utilization is according to the pavement detection method of first embodiment of the invention, adopt the V disparity map of V disparity map based on whole disparity map and Sign for road based on such as left and right lane line simultaneously, carry out road surface extraction, very effective for the detection of tilted road surface, and the road surface of extracting more tallies with the actual situation.

In addition, the embodiment of the present invention detects the Sign for road such as lane line in gray-scale map, and carries out road surface straight line estimation based on Sign for road structure V disparity map being detected, these two step process be all 2-D data, unique point has obtained enhancing thus, and calculated amount significantly reduces simultaneously.

3.2, the detailed example of road surface point extracting method

Below with reference to Figure 11, the example of road surface point extracting method is according to an embodiment of the invention described.

Figure 11 shows the process flow diagram of road surface point extracting method 1400 according to an embodiment of the invention.This road surface point extracting method 1400 can be applied to the road surface point extraction step S1400 shown in Fig. 2 and similar road surface point extraction step S1600, and both differences are that input is full V disparity map or the indicative V disparity map of the road of the Sign for road based on detecting.To be input as V disparity map, be described without distinction below.

As shown in figure 11, be input as V disparity map 1410.

In step S1420, limit surveyed area.

By limiting surveyed area, thereby the follow-up road surface point that only carries out in surveyed area rather than in view picture V disparity map detects, and can significantly reduce calculated amount, alleviates noise effect.

In one embodiment, can the historical trace information based on road surface frame determine the surveyed area in V disparity map.In other words, can be according to the surveyed area in the testing result restriction V disparity map of road surface historical frames.In Vehicle Driving Cycle process, can not undergo mutation in road surface, so there is very high probability the road surface line segment of present frame to be detected near historical surveyed area.The testing result of road surface historical frames and definite surveyed area accordingly in Figure 12, have schematically been provided, the road surface line that wherein the solid line indication former frame of label 1 indication detects, the region indication surveyed area definite according to this testing result that the solid line of label 2 indications is confined, the region indication that the dot-and-dash line of label 3 indications is confined is in addition based on the definite surveyed area of camera parameters.

In another embodiment, can be based on determine the surveyed area of V disparity map for obtaining the parameter of the video camera of disparity map and gray-scale map.Generally, for example the parameter of in-vehicle camera this video camera in taking road process is fixed, so the road surface of this shot by camera is also generally arranged in image fixed area.The parameter of video camera comprises intrinsic parameter and outer parameter.The outer parameter of video camera for example has video camera to the height on road surface, the distance of left and right image center point in the situation that of binocular camera, the angle on the plane of delineation and road surface etc.The intrinsic parameter of video camera is such as there being camera focus etc.Utilize these prioris, those skilled in the art can determine the surveyed area of road surface in V disparity map based on mathematical operation.The scope that the dot-and-dash line institute frame of label 3 indications in Figure 12 is got schematically shows based on the definite surveyed area of camera parameters.

In another embodiment, can determine the first surveyed area in V disparity map and based on determining the second surveyed area of V disparity map for obtaining the parameter of the video camera of disparity map and gray-scale map as setting angle based on historical trace information; And adopt the common factor of the first surveyed area and the second surveyed area as surveyed area, wherein only in surveyed area, carry out road surface point selection.Can make like this calculated amount greatly reduce, can get rid of the interference of a lot of noises simultaneously.It should be noted that, in the schematic diagram in Figure 12, it is inner that the first surveyed area of determining based on historical trace information is included in the second surveyed area of determining based on camera parameters, and therefore now the common factor of the two is the first surveyed area.But this is only example, and in fact the first surveyed area and the second surveyed area can intersect mutually, or the second surveyed area is inner at the first surveyed area, and now the common factor of the two is different from the first surveyed area.

Define surveyed area in step S1420 after, advance to step S1430.

In step S1430, for each pixel in surveyed area, extract architectural feature.

As previously mentioned, in embodiments of the present invention, the architectural feature of a pixel of definition is based on the accumulation of all pixels in region centered by this pixel, that have given shape and pre-sizing and obtains, the shape in this region is mated with the situation of road surface projection in V disparity map, this is for when architectural feature is extracted, and the architectural feature that is positioned at the pixel on road surface can will be close to road surface point as much as possible.For any pixel Pi, its architectural feature has been described the pixel distribution characteristics in its neighborhood.Figure 13 shows according to an embodiment of the invention the schematic diagram for the region of the architectural feature of calculating pixel point Pi, and this region is the parallelogram neighborhood of pixel Pi.Can be by calculating the intensity level sum of pixel in its parallelogram neighborhood as the value of the architectural feature of pixel Pi.

What consider due to the architectural feature of pixel is overall condition rather than an independent pixel of a pixel in region, so weak road surface has obtained corresponding enhancing.Meanwhile, owing to having calculated accumulated value, can effectively remove noise.

In above-mentioned example, in binocular camera level, be equipped on vehicle and basic horizontal is taken forward in the situation that, the shape in this region can be taken as approximate parallelogram.As shown in figure 13, the approximate line style projection matchings at such region shape and road surface approximate inclination 135 degree angles in V disparity map.More specifically, this considers following situation and sets: in camera level, carry on vehicle and the in the situation that level is taken to front lower place, in the situation (in reality, most of situation is like this) of non-fully-flattened, road surface, the projection approximation of road surface in V disparity map is parallelogram shape.In another example, the shape in region can be taken as flattened rectangular, and when a pixel being carried out to structure-pixel extraction, this rectangle major axis is taken as approximate along 135 directions inclinations, and this rectangle major axis is approximate parallel in the projection of V disparity map with road surface in other words.But, in another example, for example, when camera is taken road surface vertically downward, the projection line of road surface in V disparity map will be no longer vergence direction, but approximate 90 degree, perpendicular to transverse axis, now for extracting the shape in the region that architectural feature chooses, can correspondingly adjust, be for example taken as major axis perpendicular to the rectangle of transverse axis.。Above-mentioned shape (angle that comprises inclination, the size of shape etc.) of carrying out the region of pixel accumulation can be adjusted according to actual conditions.What consider due to architectural feature is overall condition rather than an independent pixel of a pixel in region, so weak road surface can access corresponding enhancing.Meanwhile, owing to having calculated accumulated value, can effectively remove noise.

Because according to road surface, the projection of shape in V disparity map decides for calculating the shape in region of the architectural feature of a pixel, therefore resulting architectural feature is more suitable for pavement detection application.

Calculate the architectural feature of each pixel in surveyed area in step S1430 after, advance to step S1440.

In step S1440, determine dynamic threshold.

In the prior art, conventionally fixing threshold value is used in whole region when selecting the candidate point of detected object etc.

But usually, determining of threshold size itself is exactly a more difficult problem.And such threshold value may not be suitable in view picture region everywhere.

For this reason, according to one embodiment of present invention, for every row of surveyed area, dynamically determine adaptively the threshold value of these row.Particularly, in surveyed area, can scan each row, calculate respectively the Structural Eigenvalue of each pixel, find maximum Structural Eigenvalue in each row.Then using a certain ratio 0.6 dynamic threshold as these row for example of this row max architecture eigenwert.When definite threshold, substitute to consider max architecture eigenwert, or except max architecture eigenwert additionally, can consider that the mean value, dispersion etc. of architectural feature carry out definite threshold.

According to another embodiment of the invention, be not for every row, but for some row for example 3 row or 4 row unify to set a threshold value, this threshold value is determined according to the Structural Eigenvalue of these some row.

Determine dynamic threshold in step S1440 after, advance to step S1450.

In step S1450, the dynamic threshold according to the architectural feature of each pixel and its altitude feature and affiliated row, carries out the primary election of road surface point.In other words, be referred to road surface point candidate or background dot.

For each row, only retain dynamic threshold and the pixel in bottommost that its Structural Eigenvalue is greater than these row, as road surface candidate point, other points all will be excluded.

Tentatively choose road surface point based on architectural feature and altitude feature in step S1450 after, advance to step S1460.

In step S1460, from the road surface point of tentatively choosing, remove discrete point.

In one embodiment, this can be by calculating road surface point apart from the distance of the road surface line that previously detect, and the point that the distance of eliminating and the road surface line that previously detected is greater than predetermined threshold from the point of road surface is realized.Relevant distance can be for example Euclidean distance.Predetermined threshold for example can be taken as all road surface points of tentatively choosing apart from the average of the Euclidean distance of the road surface line previously having detected.Thus, can exclude the point away from previous road surface line.Remaining names a person for a particular job as final road surface point set.

Figure 15 (a) schematically shows the road surface point example of tentatively choosing, and Figure 15 (b) has schematically shown the road surface point example of finally choosing after discrete point is removed.Point in rectangle frame in Figure 15 (a) is noise, from Figure 15 (b) and Figure 15 (a) relatively, after through discrete point Transformatin, has removed this noise.

For the V disparity map based on full disparity map and based on a left side, the V disparity map of right lane line, can adopt respectively above method to carry out choosing of road surface point.What Figure 16 was schematically illustrated in each V disparity map Road millet cake chooses the set of result in a V disparity map.

Figure 17 (a1)-(c1) and (a2)-(c2) show for more traditional road surface point choosing method with according to the schematic diagram of the result of the road surface point choosing method of the embodiment of the present invention, wherein, classic method is only carried out road surface point based on altitude feature and is chosen, and above-mentioned road surface point choosing method according to the embodiment of the present invention has adopted structure feature extraction and self-adaptation to determine dynamic threshold measure.Particularly, two groups of result comparisons of applying the road surface point choosing method of traditional road surface point choosing method and the embodiment of the present invention have been provided.Figure 17 (a1) shows a V disparity map example based on full disparity map, figure (b1) shows for the V disparity map shown in figure (a1) and applies the road surface point that traditional road surface point choosing method obtains, figure (c1) shows the road surface point that obtains of road surface point choosing method (wherein having merged the road surface point that the road surface point chosen based on full V disparity map and the left and right V disparity map based on left and right lane line are chosen) of the application embodiment of the present invention, wherein in figure (b1), the point in rectangle frame is the point that the guardrail on road surface causes, from figure (b1) and (c1) relatively, the road surface point choosing method of the embodiment of the present invention has been removed guardrail noise spot, and figure (c1) thus rectangle frame in point be by the present invention, to extract architectural feature road surface point is strengthened, the road surface point being retained thus, the road surface point choosing method of the visible embodiment of the present invention can strengthen road surface point.Similarly, Figure 17 (a2) shows a V disparity map example based on full disparity map, figure (b2) shows for the V disparity map shown in figure (a2) and applies the road surface point that traditional road surface point choosing method obtains, figure (c2) shows the road surface point that obtains of road surface point choosing method of the application embodiment of the present invention, wherein in figure (b2), the point in rectangle frame is noise, from figure (b2) and (c2) relatively, the road surface point choosing method of the embodiment of the present invention has been removed noise, and the point that presents two line segment forms in a rectangle frame of figure (c2) is the road surface point obtaining in road surface inclination situation, another rectangle frame shows the road surface point having obtained on road surface line segment more accurately, the road surface choosing method of the visible embodiment of the present invention can retain the road surface of inclination.

Visible, according in the road surface choosing method of the present embodiment, because extract the architectural feature of pixel, thereby road surface point has obtained enhancing, can retain better road surface.

According in the road surface choosing method of the present embodiment, because limit surveyed area, thereby the calculated amount of greatly reducing has improved efficiency.

According in the road surface choosing method of the present embodiment, by removing discrete point, can suppress the impact of noises such as fence or irrigation canals and ditches.

According in the road surface choosing method of the present embodiment, because take to be suitable for for example dynamic threshold of every row of regional area, therefore can determine adaptively dynamic threshold, therefore can select better the road surface point of each regional area.

It should be noted that, in the choosing method of road surface point according to an embodiment of the invention shown in Figure 11, adopted that surveyed area restriction, architectural feature are extracted, dynamic threshold is determined, discrete point removes all means and obtain best road surface point and choose result.But it should be noted that, this does not represent to adopt all these means to carry out road surface point to choose, on the contrary, can only take as required in these means certain or some carry out road surface point and choose, also can add as required in addition the performance that extra means or strategy improve road surface point choosing method.

4, the second embodiment of pavement detection method

Below with reference to Figure 18, describe according to the pavement detection method 1000 ' of second embodiment of the invention.

The pavement detection method of the pavement detection method of the second embodiment shown in Figure 18 and the first embodiment shown in Fig. 2 different have been road surface inverse mapping step S1800 many.Introduce in detail step S1800 below.Other step S1100-S1700 can, with reference to above in conjunction with the description of Fig. 2, repeat no more here.

According to one embodiment of present invention, V disparity map extract one or more line segment as road surface after, wish in disparity map, extracted road surface point to be showed i.e. so-called road surface point inverse mapping.

In other words, the road surface based on definite in V disparity map that the road surface point inverse mapping here refers to and relevant road surface point is mapped in disparity map, thus in disparity map, reflect road surface point.

Below in conjunction with Figure 19, be described in the example that the road surface of extracting in V disparity map is inverse mapping method in road surface in envelope situation.

As shown in figure 19, road surface inverse mapping method is input as road surface envelope and the original disparity map 1810 in V disparity map.

In step S1820, according to the road surface envelope extracting, calculate pavement-height.For road surface envelope altitude range h _e(d) be the h as shown in formula (5) above _e(d)=[h _emin(d), h _emax(d)] situation.For distance D place (having parallax value d), its pavement-height answers it in h _emin(d) and between hEmax (d).

In step S1830, in V disparity map, according to pavement-height, confirm real road surface point.This is that (the road surface point of namely selecting in the step S1400 of Fig. 2 or Figure 18 and S1600) is only road surface candidate point in fact because the road surface point of road surface line matching institute foundation, and possible its is not real road surface point.This step is after extracting road surface envelope, according to road surface envelope, confirms which real road surface point has, and particularly, a bit (d, the y) in V disparity map, according to its parallax value d, calculates its h based on formula (5) _e(d) value: if its y coordinate is in h _eminand h (d) _emax(d), between, it is pavement-height point.

Finally, in step S1840, road surface point (point with the pavement-height) inverse mapping of confirming is returned to former disparity map.It is prior art that some inverse mapping in V figure is returned to disparity map, repeats no more here.

Get back to Figure 19, in Figure 19, road surface inverse mapping method is output as the road surface detecting showing in disparity map.

The road surface point inverse mapping method of the road surface envelope based on extracting shown in Figure 19 is only example, and the present invention is not limited to this.For example, in another embodiment, can save road surface point in the V disparity map in Figure 19 and confirm that step S1830 and road surface point inverse mapping return the step S1840 of disparity map, and direct a bit (x based in formula (5) checking disparity map, y, d) whether be road surface point, particularly, for a pixel (x in disparity map, y, d), according to its parallax d, utilize formula (5) to calculate its h _e(d) value: if its y coordinate is in h _eminand h (d) _emax(d) between, it is the road surface point on disparity map, and vice versa.

Figure 21 (a1)-(c1) and (a2)-(c2) has shown the road surface result that different detection methods obtain.Figure 21 (a1) shows the original gray scale road image of an example; Figure 21 (b1) shows the pavement detection result (classic method is only to choose based on altitude feature the method that road surface line was put and extracted on road surface based on disparity map) obtaining about this original gray scale road image classic method, from Figure 21 (b1), its road surface point retains seldom; Figure 21 (c1) shows according to method shown in Figure 18 of the embodiment of the present invention (wherein, in the point selection step of road surface, adopted architectural feature and dynamic threshold) testing result that obtains, wherein in the rectangle frame in Figure 21 (c1), comprise lane line, it has retained the part of lane line as can be seen here.Figure 21 (a2) shows the original gray scale road image of an example, Figure 21 (b2) shows the pavement detection result obtaining about this original gray scale road image classic method, Figure 21 (c2) shows the testing result obtaining according to method shown in Figure 18 of the embodiment of the present invention, rectangle frame part in Figure 21 (b2) and Figure 21 (c2), this part road surface of classic method is very faint, and the pavement detection method of the embodiment of the present invention has obtained the road surface strengthening.As can be seen from Figure 21, because the pavement detection method of the embodiment of the present invention is the choosing of the enterprising walking along the street millet cake of V disparity map based on Sign for road such as left and right lane line, and extract road surface envelope, the road surface therefore tilting has obtained reservation.Owing to adopting architectural feature and dynamic threshold, road surface point has obtained enhancing in addition.The results show the validity of this method.

5, the 3rd embodiment of pavement detection method

Below with reference to Figure 22, describe according to the pavement detection method of third embodiment of the invention.

Figure 22 shows according to the process flow diagram of the pavement detection method 2000 of third embodiment of the invention.The pavement detection method characteristic of third embodiment of the invention is road surface point selection part, wherein based on architectural feature and dynamic threshold, selects road surface point.

As shown in figure 22, in step S2100, obtain the disparity map that comprises road surface.The realization of relevant step S2100 can, with reference to obtaining the description of the part of disparity map in step S1100 in relevant Fig. 2, repeat no more here.

In step S2200, from disparity map, build V disparity map.The realization of relevant step S2200 can, with reference to the description of the part in step S1300 in relevant Fig. 2, repeat no more here.

In step S2300, select road surface point.

Wherein, select road surface point to comprise:

Obtain the architectural feature of each pixel in V disparity map, the architectural feature of a pixel is based on the accumulation of all pixels in region centered by this pixel, that have given shape and pre-sizing and obtains, and the shape in this region is mated with the situation of road surface projection in V disparity map;

Architectural feature based on the pixel of every row in V disparity map, determines the dynamic threshold of these row;

Wherein to meet its Structural Eigenvalue highly minimum in the dynamic threshold of row and pixel that this road surface point is all dynamic thresholds that are greater than these row in these row under being greater than for selected road surface point.

It should be noted that, in step S2300, select the operation of road surface point not get rid of the utilization of other means.For example, in step S2300, can adopt the surveyed area of describing in conjunction with Figure 11 to limit the operation of step S1420 and discrete point removal step S1460, and can adopt other to be suitable for the means of road surface point selection.

About architectural feature in step S2300 in the present embodiment is extracted and the definite realization of dynamic threshold, can be with reference to above in conjunction with the description of the extraction step S1430 of architectural feature shown in Figure 11 and dynamic threshold determining step S1440.Here repeat no more.

In step S2400, based on road surface point, extract road surface.About extract the operation on road surface based on road surface point, can for example utilize Hough transformation method or least square method, or utilize the combination of these two kinds of methods.The road surface line-fitting method providing in the application for a patent for invention that the aforesaid application for a patent for invention that is CN201210194074.4 by the old superfine application number of making of identical inventor and application number are N201210513215.4 all can be applied to the present invention.

As previously mentioned, what consider due to the architectural feature of pixel is overall condition rather than an independent pixel of a pixel in region, so weak road surface has obtained corresponding enhancing.Meanwhile, owing to having calculated accumulated value, can effectively remove noise.

6, the first embodiment of road surface checking device

As shown in figure 23, road surface checking device 4000 can comprise: image obtains parts 4100, for obtaining disparity map and the gray-scale map that comprises road surface; Sign for road detection part 4200, for detecting the Sign for road that can identify position, road surface from gray-scale map; Full V disparity map builds parts 4300, for building the full V disparity map based on full figure from disparity map; First via millet cake alternative pack 4400, for selecting first via millet cake from full V disparity map; The indicative V disparity map of road builds parts 4500, for build the indicative V disparity map of road of the Sign for road based on detecting from disparity map; The second road surface point selection parts 4600, for selecting the second road surface point from the indicative V disparity map of road; And road surface extraction parts 4700, for extracting road surface based on first via millet cake and the second road surface point.

It should be noted that, the arrow between the parts shown in Figure 23 only represents a kind of logical relation.Even if do not draw arrow between two parts, do not represent not subsistence logic relation of two parts yet.

Operation and realization about road surface checking device 4000 each parts can, with reference to the description of carrying out in conjunction with the process flow diagram shown in Fig. 2, repeat no more here.

7, the second embodiment of road surface checking device

As shown in figure 24, this road surface checking device can comprise: image obtains parts 5100, obtains the disparity map that comprises road surface; V disparity map builds parts 5200, for build V disparity map from disparity map; Road surface point selection parts 5300, for selecting road surface point; And road surface extraction parts 5400, for extracting road surface based on road surface point.Wherein, road surface point selection parts 5300 select road surface point to comprise: the architectural feature of obtaining each pixel in V disparity map, the architectural feature of a pixel is based on the accumulation of all pixels in region centered by this pixel, that have given shape and pre-sizing and obtains, and the shape in this region is mated with the situation of road surface projection in V disparity map; Architectural feature based on the pixel of every row in V disparity map, determines the dynamic threshold of these row; Wherein the architectural feature of selected road surface point meet be greater than under the dynamic threshold of row and this road surface point be highly minimum in the pixel of all dynamic thresholds that are greater than these row in these row.

It should be noted that, the arrow between the parts shown in Figure 24 only represents a kind of logical relation.Even if do not draw arrow between two parts, do not represent not subsistence logic relation of two parts yet.

Operation and realization about road surface checking device 5000 each parts can, with reference to the description of carrying out in conjunction with the process flow diagram shown in Figure 22, repeat no more here.

8, system hardware configuration

The present invention can also implement by a kind of system that detects road surface.Figure 25 is the concept map illustrating according to the hardware configuration of the pavement detection system of the embodiment of the present invention 6000.As shown in figure 25, pavement detection system 6000 can comprise: input equipment 6100, for inputting image to be processed from outside, the stereo-picture that the left and right image of taking such as binocular camera, stereoscopic camera the are taken gray level image that even general camera is taken etc., the remote input equipment that this input equipment for example can comprise keyboard, Genius mouse and communication network and connect; Treatment facility 6200, above-mentioned according to the pavement detection method of the embodiment of the present invention for implementing, or be embodied as above-mentioned according to the pavement detection equipment of the embodiment of the present invention, what for example can comprise the central processing unit of computing machine or other has chip of processing power etc., can be connected to the network (not shown) such as the Internet, according to the needs of processing procedure and from Network Capture data for example left and right image, gray level image, anaglyph etc.; Output device 6300, for implement the result of above-mentioned pavement detection process gained to outside output, for example, can comprise display, printer and communication network and the long-range output device that connects etc.; And memory device 6400, for storing the related gray level image of pavement detection process, anaglyph, V disparity map, road surface point, road surface line segment parameter etc. in volatile or non-volatile mode, for example, can comprise the various volatile or nonvolatile memory of random-access memory (ram), ROM (read-only memory) (ROM), hard disk or semiconductor memory etc.

9, sum up

Pavement detection method and road surface checking device according to the embodiment of the present invention have above been described.

According to an aspect of the present invention, pavement detection method can comprise: obtain the disparity map and the gray-scale map that comprise road surface; From gray-scale map, detect the Sign for road that can identify position, road surface; From disparity map, build the full V disparity map based on full figure; From full V disparity map, select first via millet cake; From disparity map, build the indicative V disparity map of road of the Sign for road based on detecting; From the indicative V disparity map of road, select the second road surface point; And extract road surface based on first via millet cake and the second road surface point.

According to an aspect of the present invention, road surface checking device can comprise: image obtains parts, for obtaining disparity map and the gray-scale map that comprises road surface; Sign for road detection part, for detecting the Sign for road that can identify position, road surface from gray-scale map; Full V disparity map builds parts, for building the full V disparity map based on full figure from disparity map; First via millet cake alternative pack, for selecting first via millet cake from full V disparity map; The indicative V disparity map of road builds parts, for build the indicative V disparity map of road of the Sign for road based on detecting from disparity map; The second road surface point selection parts, for selecting the second road surface point from the indicative V disparity map of road; And road surface extraction parts, for extracting road surface based on first via millet cake and the second road surface point.

According to another aspect of the present invention, pavement detection method can comprise: obtain the disparity map that comprises road surface; From disparity map, build V disparity map; Select road surface point; And extract road surface based on road surface point.Wherein, select road surface point to comprise: the architectural feature of obtaining each pixel in V disparity map, the architectural feature of a pixel is based on the accumulation of all pixels in region centered by this pixel, that have given shape and pre-sizing and obtains, and the shape in this region is mated with the situation of road surface projection in V disparity map; Architectural feature based on the pixel of every row in V disparity map, determines the dynamic threshold of these row; Wherein to meet its Structural Eigenvalue highly minimum in the dynamic threshold of row and pixel that this road surface point is all dynamic thresholds that are greater than these row in these row under being greater than for selected road surface point.

According to another aspect of the present invention, road surface checking device can comprise: image obtains parts, obtains the disparity map that comprises road surface; V disparity map builds parts, for build V disparity map from disparity map; Road surface point selection parts, for selecting road surface point; And road surface extraction parts, for extracting road surface based on road surface point.Wherein, road surface point selection parts select road surface point to comprise: the architectural feature of obtaining each pixel in V disparity map, the architectural feature of a pixel is based on the accumulation of all pixels in region centered by this pixel, that have given shape and pre-sizing and obtains, and the shape in this region is mated with the situation of road surface projection in V disparity map; Architectural feature based on the pixel of every row in V disparity map, determines the dynamic threshold of these row; Wherein the architectural feature of selected road surface point meet be greater than under the dynamic threshold of row and this road surface point be highly minimum in the pixel of all dynamic thresholds that are greater than these row in these row.

The description of the embodiment of the present invention is only example, and those skilled in the art can change as required, substitute or combination.

In description above, the Sign for road of take is that the situation of lane line is described as example, but this is only example, Sign for road is not limited to this, but can be for identifying anything of position, road surface, such as the fence on road, curb stone, road on both sides of the road or middle grove etc.Particularly, for example, the bottom position of the fence on road has been indicated position, road surface.Similarly, the crowd that on road, a group is advanced also can be used as Sign for road, because crowd's bottom position has been indicated position, road surface.

In description above, the lane line of take is described as example as two, but this is only example, and the number of lane line can be less than or more than two.

In description above, in pavement detection, be applied to describe the present invention under the situation of drive assist system, but the pavement detection method and apparatus of the embodiment of the present invention can be applied to the situation that other need to carry out pavement detection.

Ultimate principle of the present invention has below been described in conjunction with specific embodiments, but, it is to be noted, for those of ordinary skill in the art, can understand whole or any steps or the parts of method and apparatus of the present invention, can be in the network of any calculation element (comprising processor, storage medium etc.) or calculation element, with hardware, firmware, software or their combination, realized, this is that those of ordinary skills use their basic programming skill just can realize in the situation that having read explanation of the present invention.

Therefore, object of the present invention can also realize by move a program or batch processing on any calculation element.Described calculation element can be known fexible unit.Therefore, object of the present invention also can be only by providing the program product that comprises the program code of realizing described method or device to realize.That is to say, such program product also forms the present invention, and the storage medium that stores such program product also forms the present invention.Obviously, described storage medium can be any storage medium developing in any known storage medium or future.

Also it is pointed out that in apparatus and method of the present invention, obviously, each parts or each step can decompose and/or reconfigure.These decomposition and/or reconfigure and should be considered as equivalents of the present invention.And, carry out the step of above-mentioned series of processes and can order naturally following the instructions carry out in chronological order, but do not need necessarily according to time sequencing, to carry out.Some step can walk abreast or carry out independently of one another.

Above-mentioned embodiment, does not form limiting the scope of the invention.Those skilled in the art should be understood that, depend on designing requirement and other factors, various modifications, combination, sub-portfolio can occur and substitute.Any modification of doing within the spirit and principles in the present invention, be equal to and replace and improvement etc., within all should being included in protection domain of the present invention.

Claims

1. a pavement detection method, comprising:

Acquisition comprises disparity map and the gray-scale map on road surface;

From gray-scale map, detect the Sign for road that can identify position, road surface;

From disparity map, build the full V disparity map based on full figure;

From full V disparity map, select first via millet cake;

From disparity map, build the indicative V disparity map of road of the Sign for road based on detecting;

From the indicative V disparity map of road, select the second road surface point; And

Based on first via millet cake and the second road surface point, extract road surface.

2. according to the pavement detection method of claim 1, describedly based on first via millet cake and the second road surface point, extract road surface and comprise:

Based on first via millet cake, extract the first via upper thread section that represents road surface;

Based on the second road surface point, extract the second road surface line segment that represents road surface; And

Obtain the envelope of first via upper thread section and the second road surface line segment as road surface.

3. according to the pavement detection method of claim 2, wherein said Sign for road is about the substantially symmetrical Sign for road of road,

Wherein from gray-scale map, detecting the Sign for road can identify position, road surface comprises and detects the Sign for road in left side and the Sign for road on right side;

The indicative V disparity map of road that wherein builds the Sign for road based on detecting from disparity map comprises the indicative V disparity map of left side road of Sign for road in left side and the indicative V disparity map of right side road of the Sign for road on the right side based on detecting building based on detecting;

Wherein from the indicative V disparity map of road, selecting the second road surface point to comprise from the indicative V disparity map of left side road selects the left side indicative V disparity map of the second Dian Hecong right side, road surface road to select right side the second road surface point;

Wherein based on the second road surface point extract the second road surface line segment represent road surface comprise based on left side the second road surface point extract left side the second road surface line segment and based on right side second road surface point extraction right side the second road surface line segment with; And

The envelope that wherein obtains first via upper thread section and the second road surface line segment comprises that as road surface the envelope that obtains first via upper thread section, left side the second road surface line segment and right side the second road surface line segment is as road surface.

4. according to the pavement detection method of claim 1, describedly from full V disparity map, select first via millet cake and/or select the second road surface point to comprise from the indicative V disparity map of road:

5. according to the pavement detection method of claim 1 or 4, wherein select road surface point also to comprise:

Based on historical trace information, determine the first surveyed area in V disparity map and based on determine the second surveyed area of V disparity map for obtaining the parameter of the video camera of disparity map and gray-scale map; And

Adopt the common factor of the first surveyed area and the second surveyed area as surveyed area, wherein only in surveyed area, carry out road surface point selection.

6. according to the pavement detection method of claim 1 or 4, wherein select road surface point also to comprise:

Calculate road surface point apart from the distance of the road surface line previously having detected, and from the point of road surface, get rid of the point that is greater than predetermined threshold with the distance of the road surface line previously having detected.

7. according to the pavement detection method of claim 2, describedly based on first via millet cake and the second road surface point, extract road surface and comprise:

According to the road surface envelope detecting, calculate pavement-height scope;

Based on pavement-height scope, in V view, select road surface point; And

Road surface point inverse mapping in V disparity map is returned to disparity map.

8. a pavement detection method, comprising:

Acquisition comprises the disparity map on road surface;

From disparity map, build V disparity map;

Select road surface point; And

Based on road surface point, extract road surface,

Wherein, select road surface point to comprise:

9. a road surface checking device, comprising:

Image obtains parts, for obtaining disparity map and the gray-scale map that comprises road surface;

Sign for road detection part, for detecting the Sign for road that can identify position, road surface from gray-scale map;

Full V disparity map builds parts, for building the full V disparity map based on full figure from disparity map;

First via millet cake alternative pack, for selecting first via millet cake from full V disparity map;

The indicative V disparity map of road builds parts, for build the indicative V disparity map of road of the Sign for road based on detecting from disparity map;

The second road surface point selection parts, for selecting the second road surface point from the indicative V disparity map of road; And

Parts are extracted on road surface, for extracting road surface based on first via millet cake and the second road surface point.

10. a road surface checking device, comprising:

Image obtains parts, obtains the disparity map that comprises road surface;

V disparity map builds parts, for build V disparity map from disparity map;

Road surface point selection parts, for selecting road surface point; And

Parts are extracted on road surface, for extracting road surface based on road surface point,

Wherein, point selection parts in road surface select road surface point to comprise:

Wherein the architectural feature of selected road surface point meet be greater than under the dynamic threshold of row and this road surface point be highly minimum in the pixel of all dynamic thresholds that are greater than these row in these row.