CN111768373A - Hierarchical pavement marking damage detection method based on deep learning - Google Patents
Hierarchical pavement marking damage detection method based on deep learning Download PDFInfo
- Publication number
- CN111768373A CN111768373A CN202010559262.7A CN202010559262A CN111768373A CN 111768373 A CN111768373 A CN 111768373A CN 202010559262 A CN202010559262 A CN 202010559262A CN 111768373 A CN111768373 A CN 111768373A
- Authority
- CN
- China
- Prior art keywords
- image
- mark
- detectable
- line
- original
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000001514 detection method Methods 0.000 title claims abstract description 27
- 238000013135 deep learning Methods 0.000 title claims abstract description 13
- 239000003550 marker Substances 0.000 claims abstract description 48
- 238000000034 method Methods 0.000 claims abstract description 36
- 239000011159 matrix material Substances 0.000 claims description 25
- 238000012545 processing Methods 0.000 claims description 14
- 230000011218 segmentation Effects 0.000 claims description 13
- 238000005516 engineering process Methods 0.000 claims description 8
- 239000007787 solid Substances 0.000 claims description 6
- 238000013461 design Methods 0.000 claims description 3
- 238000012423 maintenance Methods 0.000 abstract description 8
- 238000010586 diagram Methods 0.000 description 16
- 238000004364 calculation method Methods 0.000 description 2
- 238000006073 displacement reaction Methods 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 238000007781 pre-processing Methods 0.000 description 2
- 230000007547 defect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 238000011156 evaluation Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 238000009440 infrastructure construction Methods 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 230000000717 retained effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/0002—Inspection of images, e.g. flaw detection
- G06T7/0004—Industrial image inspection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/11—Region-based segmentation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/13—Edge detection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/60—Analysis of geometric attributes
- G06T7/62—Analysis of geometric attributes of area, perimeter, diameter or volume
Abstract
The invention provides a hierarchical pavement marker damage detection method and device based on deep learning. Acquiring an image of a road surface through a camera, and cutting the image to obtain an original image cut image; performing primary identification on the original image to identify detectable road surface marks in the original image; recognizing a contour area of the pavement marker according to a subgraph corresponding to the detectable pavement marker, and extracting damaged position information of the pavement marker; and restoring the outline region of the pavement marker to a world coordinate system from a pixel coordinate system by using external parameters of a camera, and calculating the damage rate of the pavement marker according to the damaged area and the outline area of the pavement marker in the world coordinate system. The method can effectively identify the damage position of the pavement marker, effectively evaluate the damage degree of the pavement marker and provide convenience for further pavement maintenance.
Description
Technical Field
The invention relates to the technical field of road traffic sign management, in particular to a hierarchical pavement sign damage detection method based on deep learning.
Background
With the increasing perfection of the infrastructure construction of Chinese roads and the rapid increase of road traffic volume, the large-scale maintenance requirement of road pavement traffic marks is increasing day by day. Meanwhile, the road surface marking maintenance for the road network with huge scale also brings huge challenges to the operation and maintenance unit of the traffic infrastructure.
In order to improve the efficiency of road pavement traffic marking maintenance, it is necessary to use an automated detection system to obtain important information such as the location of the damaged pavement marking and the degree of damage. The pavement marking damage detection is a branch of the pavement damage detection. From the sensor class of adoption, current road surface damage detection techniques can be mainly divided into two categories: the first type is damage detection based on a laser technology, and the type, size and severity of damage such as sink, pit and the like can be obtained by adopting a laser ranging mode and post-calculation processing; the second type is image technology-based processing, which takes a video of the road surface by a vehicle-mounted or hand-held camera, and identifies the type, size, and severity of the road surface damage by using image processing technology. The road surface damage detection based on the image technology has the advantages of simplicity, practicability, strong robustness, moderate cost and strong adaptability.
At present, the road surface detection method based on the image technology in the prior art mainly adopts a method of combining manual evaluation and traditional image processing. The disadvantages of this method are: there are disadvantages in that accuracy is to be improved and time cost is large. Secondly, although the road pavement marking plays a key role in guaranteeing safe driving of the vehicle, no method for detecting damage of the pavement marking exists in the prior art.
Disclosure of Invention
The embodiment of the invention provides a hierarchical pavement marker damage detection method based on deep learning, which is used for overcoming the defects of the prior art.
In order to achieve the purpose, the invention adopts the following technical scheme.
A hierarchical pavement marking damage detection method based on deep learning comprises the following steps:
acquiring an image of a road surface through a camera, and cutting the image to obtain an original image cut image;
preliminarily identifying the original image to be cut, selecting a detectable road mark in the original image to be cut, and judging whether a positioning image exists or not;
recognizing a contour area of the road detectable surface mark according to the subgraph corresponding to the road surface mark, and recognizing the damaged area information of the detectable road surface mark;
and restoring the outline area and the damaged area of the detectable road surface mark to a world coordinate system from a pixel coordinate system by utilizing the external parameters of the camera, and calculating the damage rate of the detectable road surface mark according to the outline area and the damaged area of the road surface mark under the world coordinate system.
Preferably, the acquiring, by the camera, the image of the road surface, and cutting the image to obtain an original image cut image includes:
the method comprises the steps of collecting an image of a road surface through a camera, cutting the upper half part of an original image according to the length of a dotted line lane line in the image of the road surface, removing a sky mark and a road surface mark of which the distance between the sky mark and a camera lens is larger than a set distance value, keeping related contents within 10 meters away from the camera lens, and obtaining a cut image of the original image, wherein the length of a current solid line lane line in the cut image is about 10 meters.
Preferably, the preliminary identification of the original image cut image to identify all the road surface marks in the original image cut image includes:
and preliminarily identifying the original image to be cut through a target detection algorithm, identifying the positions and the categories of all the road surface marks in the original image to be cut, and respectively surrounding each road surface mark by adopting a rectangular frame.
Preferably, the step of selecting a detectable road mark from the sub-identified road marks and determining whether the detected road mark is a positioning image comprises:
selecting detectable lane marking marks from the identified lane marking marks, wherein the detectable lane marking marks comprise full-line lane marking marks and dotted-line lane marking marks, and the following conditions are met:
1. the distance between the bottom end of the rectangular frame encircled by the mark and the bottom end of the original image to be cut is less than 20% of the height of the original image to be cut;
2. the height of the marked rectangular frame is more than 39% of the height of the original image to be cut;
3. the center of the marked rectangular frame is positioned at the left/right side of the middle line of the original image, and the distance between the center of the rectangular frame and the middle line of the original image is the shortest of all the rectangular frames of the lane line marks, the centers of which are positioned at the left/right side of the middle line and meet the two conditions;
if the detectable dotted type lane line mark exists in the original image, further judging whether the distance between the bottom edge of the rectangular frame of the dotted type lane line mark and the bottom edge of the original image is larger than 10% of the height of the original image; if the dotted lane line mark meets the condition and a detectable solid lane line mark exists in the original image, recording the original image as a positioning image and recording the detectable dotted lane line mark as a positioning mark; if a plurality of detectable dotted-line type lane line marks in the same original cutting image meet the condition that the distance between the bottom edge of the rectangular frame and the bottom edge of the original cutting image is greater than 10% of the height of the original cutting image, one detectable dotted-line type lane line mark with the largest distance between the bottom edge of the rectangular frame and the bottom edge of the original cutting image is selected as the positioning mark.
Preferably, said sorting out detectable indicia indicative of pavement marking from said identified pavement markings comprises:
selecting an indicating mark meeting the following conditions from the identified indicating pavement marks as a detectable indicating mark:
1. the distance between the bottom end of the rectangular frame encircled by the mark and the bottom end of the original image to be cut is smaller than 20% of the height of the original image to be cut;
2. the height of the marked rectangular frame is more than 39% of the height of the input original image cutting image;
3. the center line of the original clipped image may be partially covered with the rectangular frame portion, and the distance between the center point of the rectangular frame and the center line of the original clipped image is shortest compared with other indication mark rectangular frames satisfying the above condition.
Preferably, the recognizing a contour region of the pavement marker according to the sub-image corresponding to the pavement marker includes:
based on the subgraph of the detectable road surface mark, an example segmentation algorithm is adopted to identify the outline region of the detectable road surface mark in the subgraph, a contour region mask matrix coded by 0-1 dipolar values represents the outline region of the road surface mark, the size of the contour region mask matrix is consistent with the size of the subgraph of the detectable road surface mark, the corresponding value of the pixel points occupied by the marked outline region is 1, and the corresponding values of the pixel points at other positions are 0.
Preferably, said identifying a damaged area of said detectable pavement marking comprises:
and (2) digging out sub-graphs based on the rectangular frames of the detectable road surface marks, identifying the damaged areas of the detectable road surface marks in the sub-graphs by adopting a semantic segmentation algorithm, and representing the specific positions of the damaged areas of the road surface marks in the images by using a damaged segmentation matrix coded by 0-1 dipolar values, wherein the size of the damaged segmentation matrix is consistent with that of a mask matrix of the outline areas of the detectable road surface marks, the corresponding value of the pixel points occupied by the undamaged areas of the road surface marks is 1, and the corresponding value of the pixel points at other positions is 0.
Preferably, the method for calculating the damage rate of the road surface mark according to the outline area and the damaged area of the detectable road surface mark in the world coordinate system includes the steps of:
calibrating external parameters of the camera based on a contour region mask matrix of the positioning mark, restoring a contour region and a damaged region of the detectable road mark under a pixel coordinate system to a world coordinate system by using the external parameters of the camera, and calculating the damage rate of the road mark according to the contour area and the damaged area of the detectable road mark under the world coordinate system;
the method comprises the following steps of processing and calculating the breakage rate aiming at the detectable indication mark and the detectable broken line type lane line mark in the original image cutting image by the following method:
judging whether the mark appears in the image sequence of the video as a detectable road mark for the first time, if not; the damage rate is not calculated; if the mark appears in the image sequence of the video as the detectable road surface mark for the first time, calculating the actual physical distance m between the bottom edge of the rectangular frame and the top edge of the rectangular frame of the mark through the external parameters of the camera; finding one frame of image in the original image cutting images after the frame of original image cutting image by using an object tracking method so that the actual distance r between the top edge and the bottom edge of the rectangular frame marked in the image is closest to m/2; calculating the area a1 of the mark and the area d1 of the damaged part in the image under the world coordinate system;
finding a horizontal line in the current original cutting image to enable the actual distance from the line to the bottom edge of the marked rectangular frame to be r, and calculating the area a2 and the area d2 of a damaged part marked by the horizontal line and the bottom edge of the rectangular frame;
the detectable pavement marking has a breakage rate equal to (d1+ d2)/(a1+ a 2);
and (3) processing and calculating the breakage rate by adopting the following method aiming at the solid lane mark in the original cutting image:
and finding a horizontal straight line in the image based on the external parameters of the camera, so that the actual distance between the horizontal straight line and the bottom edge of the rectangular frame of the mark is equal to 2 meters, and calculating the area of the mark part clamped between the bottom edge of the rectangular frame and the horizontal straight line and the damaged area through the external parameters of the camera, thereby further calculating the damage rate of the mark.
Preferably, the calibrating the external parameters of the camera by the contour region mask matrix of the positioning mark based on the camera calibration technology includes:
searching a positioning image which is the smallest in frame number away from a current original cutting image, calculating the frame number away from the current original cutting image and any positioning image, wherein a positioning mark in the positioning image is a dotted line type lane line, and obtaining pixel coordinate system coordinates of four vertexes of the dotted line type lane line mark in the original image according to a lane line mark contour region mask matrix;
and judging the position relation between the detectable solid line type lane line and the positioning mark in the positioning image, and taking the upper vertex and the lower vertex of one side of the positioning mark facing the detectable solid line type lane line as positioning points. Horizontal lines are respectively made towards one side of the detectable solid line type lane lines through the two positioning points, pixels occupied by the first detectable solid line type lane line outline intersected with the horizontal lines are also set as the positioning points, and 4 positioning points can be obtained in total.
Setting the lower left corner of the dotted line lane line rectangle as the origin of a world coordinate system, obtaining the actual lengths of the dotted line lane line mark and the lane width according to the road and the design data of the mark so as to sequentially obtain the world coordinates of 4 positioning points, estimating the camera external parameters corresponding to the positioning image according to the world coordinates and the pixel coordinates of the 4 positioning points, and using the camera external parameters of the positioning image as the camera external parameters of the current image.
According to the technical scheme provided by the embodiment of the invention, the hierarchical pavement marker damage detection method based on deep learning can effectively identify the damage position of the pavement marker, effectively evaluate the damage degree of the pavement marker and provide convenience for further pavement maintenance and repair.
Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings needed to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.
Fig. 1 is a schematic processing flow diagram of a hierarchical pavement marking damage detection method based on deep learning according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of image preprocessing according to an embodiment of the present invention;
FIG. 3 is a schematic diagram of an initial pavement marking identification according to an embodiment of the present disclosure;
FIG. 4 is a schematic diagram of a left/right detectable lane marking selection according to an embodiment of the present invention;
FIG. 5 is a schematic diagram of a positioning mark according to an embodiment of the present invention;
FIG. 6 is a schematic diagram of a detectable marker selection according to an embodiment of the present invention;
FIG. 7 is a schematic view of a pavement marking contour region identification according to an embodiment of the present disclosure;
FIG. 8 is a schematic view of a damaged area identification of a pavement marker according to an embodiment of the present disclosure;
fig. 9-1, 9-2, 9-3, 9-4, and 9-5 are schematic diagrams illustrating a process of extracting the breakage rate information of the pavement marker according to the embodiment of the present invention.
Detailed Description
Reference will now be made in detail to embodiments of the present invention, examples of which are illustrated in the accompanying drawings, wherein like reference numerals refer to the same or similar elements or elements having the same or similar function throughout. The embodiments described below with reference to the accompanying drawings are illustrative only for the purpose of explaining the present invention, and are not to be construed as limiting the present invention.
As used herein, the singular forms "a", "an", "the" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and/or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof. It will be understood that when an element is referred to as being "connected" or "coupled" to another element, it can be directly connected or coupled to the other element or intervening elements may also be present. Further, "connected" or "coupled" as used herein may include wirelessly connected or coupled. As used herein, the term "and/or" includes any and all combinations of one or more of the associated listed items.
It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the prior art and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.
For the convenience of understanding the embodiments of the present invention, the following description will be further explained by taking several specific embodiments as examples in conjunction with the drawings, and the embodiments are not to be construed as limiting the embodiments of the present invention.
In recent years, the rapid development of deep learning promotes the breakthrough of key problems in the fields of image processing and the like, so that the precision, stability, reliability and the like of image processing are greatly improved. The embodiment of the invention provides a hierarchical pavement marker damage detection method based on deep learning, which can realize automatic high-precision detection of pavement marker damage degree and position and improve the intelligent level of operation and maintenance of traffic infrastructure.
The embodiment of the invention provides a hierarchical pavement marker damage detection method based on deep learning, which comprises the following processing flows:
step 1, collecting an image of a road surface through a camera, and cutting the image to obtain an original image.
Fig. 2 is a schematic diagram of image preprocessing according to an embodiment of the present invention, as shown in fig. 2, the upper half of an original image is cut off based on the length of a dashed lane line in the image, a sky and a road mark having a distance from a camera lens greater than a set distance value (10 meters) are excluded, contents related to the distance from the camera lens within 10 meters are retained, the length of a current solid lane line in the cut image is about 10 meters, and the cut size is a fixed value, and each image for detection is cut according to the fixed size once the value is determined.
And 2, performing primary identification on the cut image of the original image after cutting, and identifying detectable road surface marks in the cut image of the original image.
Fig. 3 is a schematic diagram of preliminary identification of a road surface mark according to an embodiment of the present invention, as shown in fig. 3, the original image cropped output in step 2 is input into an object detection algorithm, such as fast-RCNN or YOLOv3, which may frame the mark in the input image with a 2D frame, and output the coordinates of the center point of the frame of the mark in the pixel coordinate system and the size of the frame in the pixel coordinate system. And preliminarily identifying the original image to be cut through a target detection algorithm, and identifying the positions and the categories of all the road surface marks in the original image to be cut. The types of the road surface markings include solid-line lane line markings, broken-line lane line markings, and indication marks such as arrows, numbers, and characters.
Selecting a detectable lane marking from the road surface markings recognized from the image and judging whether the image is a positioning image, wherein the detectable lane marking comprises a solid line type lane marking and a dotted line type lane marking, and the following conditions are satisfied:
1. the distance between the bottom end of the circled rectangular frame and the bottom end of the original image to be cut is less than 20% of the height of the original image to be cut;
2. the height of the marked rectangular frame is more than 39% of the height of the original image to be cut;
3. the centers of the marked rectangular frames are positioned at the left/right sides of the middle line of the original image, and the distance between the centers of the rectangular frames and the middle line of the original image is the shortest of all the rectangular frames of the lane line marks, the centers of which are positioned at the left/right sides of the middle line, and the lane line marks meet the above two conditions. And the central point of the rectangular frame of the lane line mark is positioned on the left side of the central line of the original image to be named as a left lane line mark, and the central point is positioned on the right side of the central line to be named as a right lane line mark. Fig. 4 is a schematic diagram of a lane marking selection with left/right detectable according to an embodiment of the present invention.
If the detectable dotted-line type lane line mark exists in the original image, further judging whether the distance between the bottom edge of the rectangular frame of the dotted-line type lane line mark and the bottom edge of the original image is larger than 10% of the height of the original image; if the dashed-line lane line marks satisfy the condition and a detectable solid-line lane line mark is present in the original cut image, the original cut image is recorded as a positioning image, and the detectable dashed-line lane line mark is recorded as a positioning mark. If a plurality of detectable dotted-line lane line marks in the same original cutting image satisfy the condition that the distance between the bottom edge of the rectangular frame and the bottom edge of the original cutting image is greater than 10% of the height of the original cutting image, one detectable dotted-line lane line mark with the largest distance between the bottom edge of the rectangular frame and the bottom edge of the original cutting image is selected as the positioning mark. Fig. 5 is a schematic diagram of a positioning mark according to an embodiment of the present invention.
Selecting an indicating mark meeting the following conditions from the recognized pavement marks as a detectable indicating mark:
1. the distance between the bottom end of the circled rectangular frame and the bottom end of the input original image clipping image is less than 20% of the height of the original image clipping image;
2. the height of the marked rectangular frame is 39% greater than the height of the input original image cut image;
3. the center line of the original clipped image may be partially covered with the rectangular frame portion, and the distance between the center point of the rectangular frame and the center line of the original clipped image is shortest compared with other indication mark rectangular frames satisfying the above condition. A schematic diagram of a detectable marker selection provided by embodiments of the invention is shown in fig. 6.
If none of the marks in the original cut image meets the conditions listed above for the detectable lane marking and the indicator marking, the detectable road marking of the image is recorded as none.
And 3, accurately identifying the pavement marker, and identifying the outline area of the pavement marker.
Fig. 7 is a schematic diagram illustrating the outline region identification of a pavement marker according to an embodiment of the present invention, and as shown in fig. 7, a subgraph is extracted according to a rectangular frame of a detectable marker, an outline region of the detectable pavement marker is further identified in the subgraph corresponding to the extracted detectable pavement marker based on Mask-RCNN and other example segmentation algorithms, and a region covered by the detectable pavement marker is semantically segmented, and at this time, marker target region information is stored in a 0-1 dipolar value coding manner.
And (3) predicting the contour region of the detectable road surface mark in the subgraph by adopting an example segmentation algorithm such as Mask-RCNN (matrix-recursive least squares) based on the subgraph of the detectable road surface mark output in the step 2, and outputting a contour region Mask matrix coded by 0-1 dipolar values to represent the contour region.
The size of the mask matrix of the outline area is consistent with the size of a detectable subgraph of the pavement marker, the corresponding value of the pixel points occupied by the marked outline area is 1, and the corresponding values of the pixel points at other positions are 0.
And 4, identifying the damaged area of the pavement marker.
Fig. 8 is a schematic diagram illustrating the damaged portion identification of the road surface marker according to the embodiment of the present invention, as shown in fig. 8, the sub-picture output in step 2 is input into a semantic algorithm such as U-Net, the damaged portion of the road surface marker is identified by U-Net, and the information of the damaged portion of the road surface marker is stored by using a 0-1 dipolar value coding method.
And (3) predicting the damaged area of the road mark by adopting semantic segmentation algorithms such as U-Net and the like based on the detectable road mark subgraph output in the step 2, outputting a damaged segmentation matrix coded by 0-1 dipolar values, and finely depicting the specific position of the damaged area of the road mark in the image. The size of the damaged segmentation matrix is consistent with the size of a detectable outline area mask matrix of the pavement marker, the corresponding value of a pixel point occupied by an undamaged area of the pavement marker is 1, and the corresponding values of pixel points at other positions are 0.
And 5: and (3) calibrating the external parameters of the camera based on the camera calibration technology and the contour region mask matrix of the positioning mark obtained in the step (3), wherein the external parameters of the camera comprise a rotation matrix and a displacement vector, the rotation matrix reflects an included angle between an optical axis of the camera and a coordinate axis of a world coordinate system, and the displacement vector reflects a three-dimensional position of the camera under the world coordinate system.
The positioning image with the minimum frame number away from the current original cutting image is searched, the original cutting image is from an original, the positioning image is also from an original, the original is one frame in the video, and therefore the frame number away from the current original cutting image and any positioning image can be calculated (see step 2 for the description of the positioning image).
The positioning marks in the positioning image are broken-line type lane lines, and the coordinates of the pixel coordinate system of the four vertices of the broken-line type lane line marks in the original image are further obtained from the mask matrix of the lane line mark outline area obtained in step 3 (see step 2 for the description of the positioning marks).
And judging the position relation between the detectable solid line type lane line and the positioning mark in the positioning image, and taking the upper vertex and the lower vertex of one side of the positioning mark facing the detectable solid line type lane line as positioning points. Horizontal lines are respectively made towards one side of the detectable solid line type lane lines through the two positioning points, pixels occupied by the first detectable solid line type lane line outline intersected with the horizontal lines are also set as the positioning points, and 4 positioning points can be obtained in total.
Setting the lower left corner of the dotted line lane line rectangle as the origin of a world coordinate system, obtaining the actual lengths of the dotted line lane line mark and the lane width according to the road and the design data of the mark so as to sequentially obtain the world coordinates of 4 positioning points, estimating the camera external parameters corresponding to the positioning image according to the world coordinates and the pixel coordinates of the 4 positioning points, and using the camera external parameters of the positioning image as the camera external parameters of the current image.
Note that the positioning image of the current image is likely to be the current image itself.
And 6, extracting the damage rate information of the pavement marker.
Fig. 9-1, 9-2, 9-3, 9-4, and 9-5 are schematic diagrams illustrating a process of extracting the breakage rate information of a road marking according to an embodiment of the present invention, where i denotes a frame number, in fig. 9-1, i ═ 5 denotes an exemplary video stream start frame, in fig. 9-2, i ═ 10 denotes a detectable mark breakage start detection frame, and a solid line frame denotes a frame when the mark appears as a detectable mark for the first time in an image sequence; calculating the broken area and the mark area of the part of the mark in the area sandwiched by the 2 nd and 3 rd dotted lines from top to bottom in the graph to obtain a1 and b 1; in fig. 9-3, i is 15, which indicates a detectable mark breakage end detection frame, and the solid line frame represents the frame when the mark appears as a detectable mark in the image sequence; the region between the 1 st and 2 nd dotted lines from the top represents that in order to calculate the breakage rate of the mark, only the breakage area of the part of the mark in the region and the mark area are calculated to obtain a2 and b2, and finally the breakage rate of the mark is (b1+ b2)/(a1+ a 2). In fig. 9-4, i ═ 18. In fig. 9-5, i ═ 21 indicates a video stream end frame. Based on the outline area of the road surface mark and the ground mark damaged area identified in fig. 9-2 and 9-3, the outline area and the damaged area of the road surface mark are restored to the world coordinate system from the pixel coordinate system according to the camera parameters corresponding to the respective images, and the damage rate of the road surface mark, that is, the percentage of the damaged portion in the original image is further calculated.
The method comprises the following steps of processing and calculating the breakage rate aiming at the detectable indication mark and the detectable broken line type lane line mark in the original image cutting image by the following method: judging whether the mark appears in the image sequence of the video as a detectable road mark for the first time, if not; the breakage rate calculation is not made,
if the mark appears in the image sequence of the video as the detectable road surface mark for the first time, calculating the actual physical distance m between the bottom edge of the rectangular frame and the top edge of the rectangular frame of the mark through camera external parameters; and finding one frame of image in the original image cutting image after the frame of original image cutting image by adopting an object tracking method so that the actual distance r between the top edge and the bottom edge of the rectangular frame marked in the image is closest to m/2. The area a1 of the marker and the area d1 of the broken portion in that image were calculated under the world coordinate system. Since the target tracking algorithm can be used to track the trace of a given mark in a sequence of images, it can be determined in which images in the sequence of images the mark appears as a detectable pavement marker.
A horizontal line is found in the present original image so that the actual distance from the line to the bottom side of the marked rectangular frame is r, and the area a2 marked by the part sandwiched by the horizontal line and the bottom side of the rectangular frame and the damaged part area d2 are calculated.
The detectable pavement marking has a breakage rate equal to (d1+ d2)/(a1+ a 2).
And (3) processing and calculating the breakage rate by adopting the following method aiming at the solid lane mark in the original cutting image: and finding a horizontal straight line in the image based on the corresponding camera external parameter, enabling the actual distance between the horizontal straight line and the bottom edge of the marked rectangular frame to be equal to 2 meters, and calculating the area of the marked part between the bottom edge of the rectangular frame and the horizontal straight line and the damaged area through the camera external parameter, thereby further calculating the damaged rate.
All types of detectable pavement markings can determine the position of the marking in the world coordinate system based on the time stamp of the image and the GPS location information time stamp.
In summary, the hierarchical pavement marker damage detection method based on deep learning according to the embodiment of the present invention can effectively identify the damage position of the pavement marker, effectively evaluate the damage degree of the pavement marker, and provide convenience for further pavement maintenance.
Compared with the traditional method, the method has the advantages of simplicity, practicability, low cost, high identification precision, wide applicable scene and the like.
Those of ordinary skill in the art will understand that: the figures are merely schematic representations of one embodiment, and the blocks or flow diagrams in the figures are not necessarily required to practice the present invention.
From the above description of the embodiments, it is clear to those skilled in the art that the present invention can be implemented by software plus necessary general hardware platform. Based on such understanding, the technical solutions of the present invention may be embodied in the form of a software product, which may be stored in a storage medium, such as ROM/RAM, magnetic disk, optical disk, etc., and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the method according to the embodiments or some parts of the embodiments.
The embodiments in the present specification are described in a progressive manner, and the same and similar parts among the embodiments are referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, for apparatus or system embodiments, since they are substantially similar to method embodiments, they are described in relative terms, as long as they are described in partial descriptions of method embodiments. The above-described embodiments of the apparatus and system are merely illustrative, and the units described as separate parts may or may not be physically separate, and the parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment. One of ordinary skill in the art can understand and implement it without inventive effort.
The above description is only for the preferred embodiment of the present invention, but the scope of the present invention is not limited thereto, and any changes or substitutions that can be easily conceived by those skilled in the art within the technical scope of the present invention are included in the scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.
Claims (9)
1. A hierarchical pavement marking damage detection method based on deep learning is characterized by comprising the following steps:
acquiring an image of a road surface through a camera, and cutting the image to obtain an original image cut image;
preliminarily identifying the original image to be cut, selecting a detectable road mark in the original image to be cut, and judging whether a positioning image exists or not;
recognizing a contour area of the road detectable surface mark according to the subgraph corresponding to the road surface mark, and recognizing the damaged area information of the detectable road surface mark;
and restoring the outline region of the pavement marker to a world coordinate system from a pixel coordinate system by using the external parameters of the camera, and calculating the damage rate of the pavement marker according to the damaged area and the outline area of the pavement marker in the world coordinate system.
2. The method of claim 1, wherein the capturing the image of the road surface by the camera and cropping the image to obtain the cropped image of the original image comprises:
the method comprises the steps of collecting an image of a road surface through a camera, cutting the upper half part of an original image by taking the length of a lane line in the image of the road surface as a reference basis, removing the sky and a road surface mark with the distance between the sky and a camera lens larger than a set distance value, keeping related contents within about 10 meters away from the camera lens, and obtaining a cut image of the original image, wherein the length of a current solid lane line in the cut image is about 10 meters.
3. The method of claim 1, wherein said preliminary identifying said artwork cropped image includes:
and preliminarily identifying the original image to be cut through a target detection algorithm, identifying the positions and the categories of all the road surface marks in the original image to be cut, and respectively surrounding each road surface mark by adopting a rectangular frame.
4. The method of claim 3, wherein said selecting a detectable lane marking from said identified pavement markings and determining whether the detected lane marking is a scout image comprises:
selecting detectable lane marking marks from the recognized pavement markings, wherein the detectable lane marking marks comprise full-line lane marking marks and dotted-line lane marking marks, and the following conditions are met:
(1) the distance between the bottom end of the rectangular frame encircled by the mark and the bottom end of the original image to be cut is less than 20% of the height of the original image to be cut;
(2) the height of the marked rectangular frame is more than 39% of the height of the original image to be cut;
(3) the centers of the marked rectangular frames are positioned at the left/right sides of the middle line of the original image cutting image, and the distance between the centers of the rectangular frames and the middle line of the original image cutting image is the shortest of all lane line marks of which the centers are positioned at the left/right sides of the middle line and meet the above two conditions;
if the detectable dotted type lane line mark exists in the original image, further judging whether the distance between the bottom edge of the rectangular frame of the dotted type lane line mark and the bottom edge of the original image is larger than 10% of the height of the original image; recording the original cutting image as a positioning image and the detectable broken-line lane line mark as a positioning mark if the broken-line lane line mark satisfies the condition and a detectable solid-line lane line mark also exists in the original cutting image; if a plurality of detectable dotted-line type lane line marks in the same original cutting image meet the condition that the distance between the bottom edge of the rectangular frame and the bottom edge of the original cutting image is greater than 10% of the height of the original cutting image, one detectable dotted-line type lane line mark with the largest distance between the bottom edge of the rectangular frame and the bottom edge of the original cutting image is selected as the positioning mark.
5. The method of claim 3 wherein said selecting a detectable indicia pavement marking from said identified pavement markings comprises:
selecting an indicating mark meeting the following conditions from the recognized pavement marks as a detectable indicating mark:
(1) the distance between the bottom end of the rectangular frame encircled by the mark and the bottom end of the original image to be cut is smaller than 20% of the height of the original image to be cut;
(2) the marked rectangular frame height is greater than 39% of the input original image clipping image height;
(3) the center line of the original clipped image may be partially covered with the marked rectangular frame portion, and the distance between the center point of the rectangular frame and the center line of the original clipped image is shortest compared with other indication mark rectangular frames satisfying the above condition.
6. The method of claim 3, 4 or 5, wherein said identifying a contour region of said pavement marking from a corresponding sub-image of said detectable pavement marking comprises:
and (2) digging out the subgraph based on the rectangular frame of the detectable road mark, identifying the outline region of the detectable road mark in the subgraph by adopting an example segmentation algorithm, representing the outline region of the road mark by an outline region mask matrix coded by 0-1 dipolar values, wherein the size of the outline region mask matrix is consistent with that of the subgraph of the detectable road mark, the corresponding value of a pixel point occupied by the marked outline region is 1, and the corresponding value of a pixel point at other positions including the road surface, the damaged region and the like is 0.
7. The method of claim 6, wherein said identifying damaged areas of said detectable pavement marking comprises:
and identifying a damaged area of the detectable road mark in the subgraph by adopting a semantic segmentation algorithm based on the subgraph of the detectable road mark, and representing the specific position of the damaged area of the road mark in the image by using a damaged segmentation matrix coded by 0-1 dipolar values, wherein the size of the damaged segmentation matrix is consistent with the size of a mask matrix of a contour area of the detectable road mark, the corresponding value of a pixel point occupied by an undamaged area of the road mark is 1, and the corresponding values of pixel points at other positions including the road surface, the damaged area and the like are 0.
8. The method of claim 6, wherein the step of calculating the damage rate of the pavement marker in the world coordinate system by restoring the outline area and the damaged area of the detectable pavement marker to the world coordinate system from the pixel coordinate system by using the external parameters of the camera comprises the steps of:
calibrating external parameters of the camera based on a contour region mask matrix of the positioning mark by using a camera calibration technology, converting a contour region and a damaged region of the pavement mark under a pixel coordinate system into a world coordinate system by using the external parameters of the camera, and calculating the damage rate of the pavement mark under the world coordinate system according to the area of the damaged part of the pavement mark in the world coordinate system and the area of the pavement mark contour in the world coordinate system;
the method comprises the following steps of processing and calculating the breakage rate aiming at the detectable indication mark and the detectable broken line type lane line mark in the original image cutting image by the following method:
judging whether the mark appears in the image sequence of the video as a detectable road mark for the first time, if not; the damage rate is not calculated; if the mark appears in the image sequence of the video as the detectable road surface mark for the first time, calculating the actual physical distance m between the bottom edge of the rectangular frame and the top edge of the rectangular frame of the mark through the external parameters of the camera; finding one frame of image in the original image cutting images after the frame of original image cutting image by using an object tracking method so that the actual distance r between the top edge and the bottom edge of the rectangular frame marked in the image is closest to m/2; calculating the area a1 of the mark and the area d1 of the damaged part in the image under the world coordinate system;
finding a horizontal line in the current original cutting image to enable the actual distance from the line to the bottom edge of the marked rectangular frame to be r, and calculating the area a2 and the area d2 of a damaged part marked by the horizontal line and the bottom edge of the rectangular frame;
the detectable pavement marking has a breakage rate equal to (d1+ d2)/(a1+ a 2);
and (3) processing and calculating the breakage rate by adopting the following method aiming at the solid lane mark in the original cutting image:
and finding a horizontal straight line in the image based on the external parameters of the camera so that the actual distance between the horizontal straight line and the bottom edge of the rectangular frame of the mark is equal to 2 meters, and calculating the area of the mark part clamped between the bottom edge of the rectangular frame and the horizontal straight line under the world coordinate system and the damaged area through the external parameters of the camera, thereby further calculating the damage rate of the mark.
9. The method of claim 8, wherein calibrating the camera's external parameters based on the camera calibration technique using the outline area mask matrix of the positioning markers comprises:
searching a positioning image which is the smallest in frame number away from a current original cutting image, calculating the frame number away from the current original cutting image and any positioning image, wherein a positioning mark in the positioning image is a dotted line type lane line, and obtaining pixel coordinate system coordinates of four vertexes of the dotted line type lane line mark in the original image according to a lane line mark contour region mask matrix; and judging the position relation between the detectable solid line type lane line and the positioning mark in the positioning image, and taking the upper vertex and the lower vertex of one side of the positioning mark facing the detectable solid line type lane line as positioning points. Horizontal lines are respectively made towards one side of the detectable solid line type lane lines through the two positioning points, pixels occupied by the first detectable solid line type lane line outline intersected with the horizontal lines are also set as the positioning points, and 4 positioning points can be obtained in total.
Setting the lower left corner of the dotted line lane line rectangle as the origin of a world coordinate system, obtaining the actual lengths of the dotted line lane line mark and the lane width according to the road and the design data of the mark so as to sequentially obtain the world coordinates of 4 positioning points, estimating the camera external parameters corresponding to the positioning image according to the world coordinates and the pixel coordinates of the 4 positioning points, and using the camera external parameters of the positioning image as the camera external parameters of the current image.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010559262.7A CN111768373A (en) | 2020-06-18 | 2020-06-18 | Hierarchical pavement marking damage detection method based on deep learning |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010559262.7A CN111768373A (en) | 2020-06-18 | 2020-06-18 | Hierarchical pavement marking damage detection method based on deep learning |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111768373A true CN111768373A (en) | 2020-10-13 |
Family
ID=72721029
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010559262.7A Pending CN111768373A (en) | 2020-06-18 | 2020-06-18 | Hierarchical pavement marking damage detection method based on deep learning |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111768373A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114972509A (en) * | 2022-05-26 | 2022-08-30 | 北京利君成数字科技有限公司 | Method for quickly identifying tableware position |
| CN115965927A (en) * | 2023-03-16 | 2023-04-14 | 杭州枕石智能科技有限公司 | Pavement information extraction method and device, electronic equipment and readable storage medium |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR20040061223A (en) * | 2002-12-30 | 2004-07-07 | 엘지산전 주식회사 | Traffic information acquisition method and apparatus using camera calibration |
| CN108416320A (en) * | 2018-03-23 | 2018-08-17 | 京东方科技集团股份有限公司 | Inspection device, the control method of inspection device and control device |
| CN109801282A (en) * | 2019-01-24 | 2019-05-24 | 湖北大学 | Pavement behavior detection method, processing method, apparatus and system |
| CN110458161A (en) * | 2019-07-15 | 2019-11-15 | 天津大学 | A kind of mobile robot doorplate location method of combination deep learning |
| CN110766744A (en) * | 2019-11-05 | 2020-02-07 | 北京华捷艾米科技有限公司 | MR volume measurement method and device based on 3D depth camera |
-
2020
- 2020-06-18 CN CN202010559262.7A patent/CN111768373A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR20040061223A (en) * | 2002-12-30 | 2004-07-07 | 엘지산전 주식회사 | Traffic information acquisition method and apparatus using camera calibration |
| CN108416320A (en) * | 2018-03-23 | 2018-08-17 | 京东方科技集团股份有限公司 | Inspection device, the control method of inspection device and control device |
| CN109801282A (en) * | 2019-01-24 | 2019-05-24 | 湖北大学 | Pavement behavior detection method, processing method, apparatus and system |
| CN110458161A (en) * | 2019-07-15 | 2019-11-15 | 天津大学 | A kind of mobile robot doorplate location method of combination deep learning |
| CN110766744A (en) * | 2019-11-05 | 2020-02-07 | 北京华捷艾米科技有限公司 | MR volume measurement method and device based on 3D depth camera |
Non-Patent Citations (5)
| Title |
|---|
| 吴一全;刘莉;: "基于视觉的车道线检测方法研究进展", 仪器仪表学报, no. 12 * |
| 张文影;李礼夫;: "针对车辆与行人检测的感兴趣区域自适应分割算法", 科学技术与工程, no. 05 * |
| 王维等: "基于区域生长和最外边缘追踪的路面损坏检测", 中国民航大学学报, vol. 34, no. 6, pages 51 - 54 * |
| 贺思远: "道路场景中的多目标跟踪研究", 《中国优秀硕士学位论文全文数据库信息科技辑》 * |
| 陈龙;李清泉;毛庆洲;: "基于成像模型的车道线检测与跟踪方法", 中国公路学报, no. 06 * |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114972509A (en) * | 2022-05-26 | 2022-08-30 | 北京利君成数字科技有限公司 | Method for quickly identifying tableware position |
| CN114972509B (en) * | 2022-05-26 | 2023-09-29 | 北京利君成数字科技有限公司 | Method for quickly identifying tableware position |
| CN115965927A (en) * | 2023-03-16 | 2023-04-14 | 杭州枕石智能科技有限公司 | Pavement information extraction method and device, electronic equipment and readable storage medium |
| CN115965927B (en) * | 2023-03-16 | 2023-06-13 | 杭州枕石智能科技有限公司 | Pavement information extraction method and device, electronic equipment and readable storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109949578B (en) | Vehicle line pressing violation automatic auditing method based on deep learning | |
| EP3171292B1 (en) | Driving lane data processing method, device, storage medium and apparatus | |
| CN110619279B (en) | Road traffic sign instance segmentation method based on tracking | |
| CN111179152B (en) | Road identification recognition method and device, medium and terminal | |
| CN115717894B (en) | Vehicle high-precision positioning method based on GPS and common navigation map | |
| CN110689724B (en) | Automatic motor vehicle zebra crossing present pedestrian auditing method based on deep learning | |
| CN109271861B (en) | Multi-scale fusion point cloud traffic signboard automatic extraction method | |
| Borkar et al. | An efficient method to generate ground truth for evaluating lane detection systems | |
| CN112070756B (en) | Three-dimensional road surface disease measuring method based on unmanned aerial vehicle oblique photography | |
| CN106980855A (en) | Traffic sign quickly recognizes alignment system and method | |
| CN111768373A (en) | Hierarchical pavement marking damage detection method based on deep learning | |
| CN112819895A (en) | Camera calibration method and device | |
| CN114359181A (en) | Intelligent traffic target fusion detection method and system based on image and point cloud | |
| CN103632376A (en) | Method for suppressing partial occlusion of vehicles by aid of double-level frames | |
| CN107918775B (en) | Zebra crossing detection method and system for assisting safe driving of vehicle | |
| CN111383286B (en) | Positioning method, positioning device, electronic equipment and readable storage medium | |
| Wei et al. | Damage inspection for road markings based on images with hierarchical semantic segmentation strategy and dynamic homography estimation | |
| Hu | Intelligent road sign inventory (IRSI) with image recognition and attribute computation from video log | |
| CN113255578B (en) | Traffic identification recognition method and device, electronic equipment and storage medium | |
| Guerrieri et al. | Flexible and stone pavements distress detection and measurement by deep learning and low-cost detection devices | |
| CN115240086A (en) | Unmanned aerial vehicle-based river channel ship detection method, device, equipment and storage medium | |
| CN110210324B (en) | Road target rapid detection early warning method and system | |
| CN115497036A (en) | Waterlogging depth calculation method and system and readable storage medium | |
| CN112507887B (en) | Intersection sign extracting and associating method and device | |
| Ho et al. | Automatic image recognition of pavement distress for improving pavement inspection |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |