CN113962960A - Pavement disease detection method based on deep learning - Google Patents

Pavement disease detection method based on deep learning Download PDF

Info

Publication number
CN113962960A
CN113962960A CN202111232233.0A CN202111232233A CN113962960A CN 113962960 A CN113962960 A CN 113962960A CN 202111232233 A CN202111232233 A CN 202111232233A CN 113962960 A CN113962960 A CN 113962960A
Authority
CN
China
Prior art keywords
disease
detection
frame
pavement
coordinates
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202111232233.0A
Other languages
Chinese (zh)
Inventor
杜建超
李梦
于成龙
王彬凤
李卫斌
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Xidian University
Original Assignee
Xidian University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Xidian University filed Critical Xidian University
Priority to CN202111232233.0A priority Critical patent/CN113962960A/en
Publication of CN113962960A publication Critical patent/CN113962960A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/0002—Inspection of images, e.g. flaw detection
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
    • G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • General Engineering & Computer Science (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Evolutionary Computation (AREA)
  • Artificial Intelligence (AREA)
  • Biomedical Technology (AREA)
  • General Health & Medical Sciences (AREA)
  • Evolutionary Biology (AREA)
  • Health & Medical Sciences (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Molecular Biology (AREA)
  • Computing Systems (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Quality & Reliability (AREA)
  • Image Analysis (AREA)

Abstract

The invention discloses a pavement disease detection method based on deep learning, which mainly solves the problems that the specific categories of pavement diseases cannot be obtained and the counting of the diseases of different categories is not accurate in the prior art. The implementation scheme is as follows: shooting road video frame images through a vehicle-mounted camera to obtain a training sample set L; selecting a yolov5 network as a pavement disease detection model B; inputting the training sample set L into a pavement disease detection model B, and performing iterative training on the pavement disease detection model B by using a gradient descent method to obtain a trained pavement disease detection model B'; the types of road defects in the video are inspected; determining the type of the disease at the same position in the single-frame image of the video frame image set C; and carrying out duplicate removal and counting treatment on the same part of the disease in the continuous multi-frame images to obtain an accurate pavement disease detection result. The invention avoids manual intervention, improves the accuracy of disease detection, and can be used for maintenance and repair of road surfaces.

Description

Pavement disease detection method based on deep learning
Technical Field
The invention belongs to the technical field of image processing, and particularly relates to a pavement disease detection method which can be used for overhauling and maintaining a pavement.
Technical Field
The material and the structure of the highway pavement are affected irreversibly under the natural environment exposed for a long time, and the service life of the highway pavement is greatly shortened, so that the detection of pavement diseases is particularly important for the maintenance and overhaul of the highway pavement, the condition of pavement diseases needs to be evaluated and known in time, and traffic accidents are avoided.
The most important link for evaluating the road pavement performance is the distribution condition of pavement diseases, and the detection of the pavement diseases provides data support for the maintenance and overhaul of the pavement. The conventional method for detecting the pavement diseases mainly comprises a manual field evaluation and visual analysis method. The method for manually evaluating the site has the advantages that the traffic must be closed firstly, the traffic efficiency of the highway traffic is seriously influenced, the method is greatly influenced by subjective factors of monitoring personnel, the efficiency is low, and the method is not suitable for detecting the pavement diseases in a large range. The visual analysis method is mainly used for collecting the types and the damage degrees of the road surface diseases in a photographing and video recording mode, and the collected data are mainly processed in two modes: one is to complete the analysis of the diseases in the video by manpower, to carry out rough analysis on the road surface damage condition, and the data processing and calculation amount is large; and the other method is to complete the analysis and treatment of diseases in the video through the support of a specific image processing algorithm.
The two methods for detecting the pavement diseases have the conditions of low accuracy, complicated operation and low efficiency in execution, and are not suitable for actual development.
Patent document CN106529593A discloses a patent application named as "road surface disease detection method and system", which is implemented as follows: collecting a pavement image, and dividing the pavement image into sub-areas with preset sizes; determining a gray level co-occurrence matrix of each sub-region, and respectively obtaining the structural similarity between each sub-region and each adjacent sub-region according to each gray level co-occurrence matrix; counting the number of sub-regions corresponding to the structural similarity which is greater than a preset structural similarity threshold in the obtained structural similarities; and obtaining the ratio of the counted number of the sub-regions to the total number of the sub-regions segmented by the pavement image, and judging that the pavement image has pavement diseases when the ratio is smaller than a preset ratio threshold. The method does not influence the traffic efficiency of highway traffic by using the collected pavement images, but only can obtain whether the pavement has diseases or not, cannot know the specific categories of the pavement diseases and the accurate counting of the diseases of different categories, and has great problems in specific evaluation of the pavement diseases of the pavement.
Patent document CN101565925 discloses a patent application named as "road surface disease investigation and treatment method", which is implemented by the following steps: firstly, running an electronic satellite map program on a networked computer, finding out coordinates of a starting point, a route turning point and a terminal point of an investigated road section or obtaining information of reference objects on two sides of a road, such as villages, gas stations, intersections, bridges and the like on a detailed road traffic map, calculating length information of the road section and segmenting the investigated road; and then the road section survey information is imported into a palm computer, the palm computer is used for recording road surface disease position information on site according to disease profiles by different marks, disease survey statistics is carried out by the palm computer, disease data counted by the palm computer are opened on the computer, statistical calculation is carried out on the road surface diseases within a set distance of each fixed length, and a data table and a chart obtained by analysis are stored in the computer. Although the method solves the problem that the road traffic efficiency is influenced in the traditional road surface disease detection process, the road section is obtained on an electronic satellite map, the real-time property of road surface updating is poor, the method cannot be suitable for the road surface without a reference object, the specific category of the road surface disease cannot be obtained, the road surface disease information cannot be evaluated, and the effective overhaul and maintenance of the road surface are influenced.
Disclosure of Invention
The invention aims to overcome the defects in the prior art and provides a pavement disease detection method based on deep learning, so that pavement information is updated in real time, specific categories of pavement diseases are detected, accurate counting is carried out, and the pavement disease information is evaluated.
In order to achieve the purpose, the technical scheme adopted by the invention comprises the following steps:
(1) the method comprises the following steps of shooting road video frame images through a vehicle-mounted camera, and obtaining a training sample set L:
(2) a yolov5 network is selected as a pavement disease detection model B:
(3) inputting the training sample set L into a pavement disease detection model B, and performing iterative training on the pavement disease detection model B by using a gradient descent method to obtain a trained pavement disease detection model B';
(4) the types of the road surface diseases in the inspection video are as follows:
(4a) shooting a road surface video S to be detected by using a vehicle-mounted camera as the input of a road surface disease detection model B ', and comparing frame by frame and pixel by pixel to obtain a video S' marked with the road surface disease category and confidence information;
(4b) framing the video S' to obtain a video frame image set C ═ C1,C2,...,Cq,...,CnIn which C isqRepresenting the q-th video frame image containing the road surface disease information, wherein n represents the number of the video frame images, and n is more than or equal to 100;
(5) determining the types of diseases at the same position on the single-frame images in the video frame image set C:
(5a) determining the type of the disease according to the marking condition of the detected frame of the disease at the same position:
if the same position of a single-frame image in the video frame image set C is marked by a detection frame, the type of the detection frame is the type of the disease;
if the same position of the single frame image in the video frame image set C is marked by a plurality of detection frames, judging whether the types of the detection frames are the same or not; if the detection frame types are the same, executing (5 b);
(5b) combining the detection frames by adopting a disease processing method for spatial position combination to determine the type of the disease, and completing the de-duplication of the road disease detection result:
(5b1) sorting the confidence degrees of the output detection frames according to a descending order, reserving the first two detection frames, storing the coordinates of the upper left corner and the lower right corner of the first two detection frames in a set J, and removing all the rest detection frames; let the left upper-left coordinates of the two detection frames be (Lx1, Ly1), (Lx2, Ly2), and the right lower-right coordinates be (Rx1, Ry1), (Rx2, Ry 2);
(5b2) and combining the space positions of the two detection frames in the image to determine the type of the disease at the position:
when the four conditions of Lx1 < Lx2 < Rx1, Lx2 < Rx1 < Rx2, Ry2 < Ry1 < Ly2 and Ry1 < Ly2 < Ly1 are simultaneously met, combining two detection frames into a detection frame with the coordinates of the upper left corner (Lx-Lx 1, Ly-Ly 1) and the coordinates of the lower right corner (Rx-Rx 2 and Ry-Ry 2), wherein the type of the detection frame is the type of the disease;
when the four conditions of Lx2 < Lx1 < Rx2, Lx1 < Rx2 < Rx1, Ry1 < Ry2 < Ly1 and Ry2 < Ly1 < Ly2 are simultaneously met, combining two detection frames into a detection frame with the coordinates of the upper left corner (Lx-Lx 2, Ly-Ly 2) and the coordinates of the lower right corner (Rx-Rx 1 and Ry-Ry 1), wherein the type of the detection frame is the type of the disease;
when the four conditions of Lx1 < Lx2 < Rx1, Ry1 < Ry2 < Ly1, Lx2 < Rx1 < Rx2 and Ry2 < Ly1 < Ly2 are simultaneously met, two detection frames are combined into one detection frame with the coordinates of the upper left corner (Lx-Lx 1, Ly-Ly 2) and the coordinates of the lower right corner (Rx-Rx 2 and Ry-Ry 1), and the type of the detection frame is the type of the disease;
when the four conditions of Lx2 < Lx1 < Rx2, Ry2 < Ry1 < Ly2, Lx1 < Rx2 < Rx1 and Ry1 < Ly2 < Ly1 are simultaneously met, two detection frames are combined into one detection frame with the coordinates of the upper left corner (Lx-Lx 2, Ly-Ly 1) and the coordinates of the lower right corner (Rx-Rx 1 and Ry-Ry 2), and the type of the detection frame is the type of the disease;
when the four conditions of Lx1 < Lx2 < Rx1, Ry1 < Ly2 < Ly1, Lx1 < Rx2 < Rx1 and Ry1 < Ry2 < Ly1 are simultaneously met, two detection frames are combined into one detection frame with the coordinates of the upper left corner (Lx-Lx 1, Ly-Ly 1) and the coordinates of the lower right corner (Rx-Rx 1 and Ry-Ry 1), and the type of the detection frame is the type of the disease;
when the four conditions of Lx2 < Lx1 < Rx2, Ry2 < Ly1 < Ly2, Lx2 < Rx1 < Rx2 and Ry2 < Ry1 < Ly2 are simultaneously met, two detection frames are combined into one detection frame with the coordinates of the upper left corner (Lx-Lx 2, Ly-Ly 2) and the coordinates of the lower right corner (Rx-Rx 2 and Ry-Ry 2), and the type of the detection frame is the type of the disease;
(6) carrying out duplicate removal and counting treatment on the same disease in continuous multi-frame images:
(6a) respectively allocating a tracker for predicting the motion position of a target disease appearing in a video in the next frame by using a Kalman filtering algorithm, wherein each tracker corresponds to a disease tracking number m;
(6b) correcting the predicted position according to the detection result of the road surface disease position in the next frame to obtain the optimal position estimation value of the disease tracked in the frame;
(6c) comparing the disease at the optimal position in the current frame with the disease tracked in the previous frame:
if the two diseases are the same disease, the disease is subjected to duplicate removal counting in two frames before and after;
if the two are not the same disease, counting the tracked diseases to obtain the real-time tracking counting and detection result of the same disease in the continuous frames.
Compared with the prior art, the invention has the following advantages:
1) the invention uses the trained pavement disease detection model B' to detect the pavement to be evaluated to obtain the detection frame marked with the pavement disease category, thereby not only not influencing the traffic efficiency of highway traffic, but also realizing the detection of the pavement disease of specific category.
2) According to the method, the disease detection results of the pavement diseases in the single-frame images are combined by adopting a disease treatment method of spatial position combination to realize the de-duplication treatment of the diseases at the same position, the de-duplication treatment of the same disease in the continuous multi-frame images is realized by adopting a Kalman filtering algorithm, and the real-time accurate tracking and counting of the pavement diseases can be obtained.
3) According to the invention, the yolov5 network is selected as the pavement disease detection model B' which is finished by the iterative training of the pavement disease detection model B to detect the pavement diseases, so that the response speed is high, the efficiency is high, and the accuracy of the pavement disease detection is improved.
Drawings
FIG. 1 is a flow chart of an implementation of the present invention.
Detailed Description
The invention is described in further detail below with reference to the figures and specific examples.
Referring to fig. 1, the implementation steps of this example are as follows:
step 1, shooting road video frame images through a vehicle-mounted camera to obtain a training sample set L:
1.1) shooting road video frame images by using a vehicle-mounted camera, and extracting 8000 video frame images containing road diseases from the road video frame images, wherein A is { A ═ A }1,A2,...,Ai,...,ANAnd using a rectangular frame in the labellmg software to carry out image A on each video frameiMarking the road surface diseases of medium categories including transverse cracks, longitudinal cracks, cracks and ruts to obtain a label set P ═ P containing disease information1,P2,...,Pi,...,PNWhere N is 8000, AiRepresents the ith road surface image containing the fault, PiIs represented by AiA corresponding label containing road surface disease information;
1.2) set P ═ P of labels of different disease information1,P2,...,Pi,...,PNThe original video frame image A corresponding to the original video frame image A is equal to { A }1,A2,...,Ai,...,ANCombining to form a training sample set: l ═ L1,L2,...,Lj,...,LNIn which L isjIs PjAnd AjThe j-th training sample is formed by combining, wherein j is 1, 2.
And 2, selecting a yolov5 network as a pavement disease detection model B.
The yolov5 network comprises a Backbone network Backbone, a target characteristic network Neck and a detection network Head,
wherein: the Backbone network backhaul comprises a CSPdacrynet module and a spatial pyramid pooling SPP module, wherein the CSPdacrynet module comprises a Focus submodule, a convolution layer with 32 convolution kernels and a BottleneckCSP submodule, and the BottleneckCSP submodule comprises a convolution layer Conv, a batch normalization layer BN, an activation function Leaky Relu and a residual error layer which are sequentially connected;
the target feature network Neck is formed by splicing a feature pyramid FPN and a pyramid PAN from bottom to top;
the detection network Head consists of three convolution layers connected in parallel.
And sequentially cascading the Backbone network backhaul, the target characteristic network Neck and the detection network Head to form a pavement disease detection model B.
And 3, performing iterative training on the road surface disease detection model B.
Inputting the training sample set L into a pavement disease detection model B, and performing iterative training on the pavement disease detection model B by using a gradient descent method to obtain a trained pavement disease detection model B', wherein the following steps are implemented:
3.1) setting the maximum iteration number as Y, wherein Y is 800, and the pavement disease detection model of the Y iteration is ByThe number of initialization iterations y is 0 and B is By；
3.2) taking the training sample set L as a pavement disease detection model ByThe input of the network is pushed forward, and L is transmitted through a Focus sub-module in the backbone networkjSlicing to obtain a feature map of 304 × 12, performing a convolution operation with 32 convolution kernels to obtain a feature map of 304 × 32, and changing the feature map channel from 12 to 32 so as to retain position information and improve the receptive field;
3.3) inputting the result obtained in the step 3.2) into a convolutional layer Conv, a batch normalization layer BN, an activation function Leaky Relu and a residual layer which are sequentially connected in the BottleneckCSP submodule to finish the extraction of the features to obtain a feature map of 38 x 32;
3.4) inputting the result obtained in 3.3) into a spatial pyramid pooling module SPP, extracting features with different scales through pooling layers with different kernel sizes of 11,55,99 and 1313, and fusing the extracted features to obtain a feature map of 19 × 32;
3.5) sending the feature map obtained in the step 3.4) into a target feature network Neck, fusing the high-level features with the low-level features through a top-down FPN structure to obtain a predicted feature map, and transmitting the strong positioning features of the low level to the top level through a pyramid PAN structure from bottom to top to obtain the feature map containing the semantic and positioning information
Figure BDA0003316472230000061
Wherein
Figure BDA0003316472230000062
The jth feature map containing semantic and positioning information;
3.6) Using the frame Loss function CIOU _ Loss, by
Figure BDA0003316472230000063
And LjDetection model B for calculating road surface diseasesyCross entropy Loss value Loss ofs：
Figure BDA0003316472230000064
Wherein IOU represents evaluation
Figure BDA0003316472230000065
Middle prediction box and LjComparing the intersection of the real frames, wherein Distance _ C represents the Distance of a diagonal line in the minimum circumscribed rectangle of the target position, Distance _2 represents the Euclidean Distance between the two central points of the prediction frame and the minimum circumscribed rectangle, and v is a parameter for measuring the consistency of the length-width ratio;
3.7) passage through LosssModel B for detecting road surface diseasesyThe weight value of the road surface defect detection model B is updated to obtain a road surface defect detection model B after the y iterationy；
3.8) judging whether the current iteration number reaches the set maximum iteration number Y, if so, obtaining a trained pavement disease detection model B', otherwise, making Y equal to Y +1, and returning to 3.3).
And 4, polling the types of the road surface diseases in the video to obtain a video frame image set.
4.1) shooting a road video S to be detected by using a vehicle-mounted camera as the input of a road disease detection model B ', and comparing pixels in each frame of the video with the learned characteristics in the B ' to obtain a video S ' marked with road disease category and confidence information;
4.2) framing the video S' to obtain a video frame image set C ═ { C ═ C1,C2,...,Cq,...,CnIn which C isqThe q-th video frame image containing the road surface defect information is shown, n represents the number of the video frame images, and n is 200.
And 5, determining the type of the disease at the same position in the single-frame image of the video frame image set C.
5.1) determining the type of the disease according to the marking condition of the detected frame of the disease at the same position:
if the same position disease in the single frame image of the video frame image set C is marked by one detection frame, the type of the detection frame is the type of the disease;
if the same position disease in the single frame image of the video frame image set C is marked by a plurality of detection frames, judging whether the types of the detection frames are the same: if the detection frames are different in type, no processing is performed; if the detection frame types are the same, executing 5.2);
5.2) combining the detection frames by adopting a disease processing method for spatial position combination to determine the category of the disease, and completing the de-duplication of the road disease detection result:
5.2.1) sorting the confidence degrees of the output detection frames according to a descending order, reserving the first two detection frames, storing the coordinates of the upper left corner and the lower right corner of the detection frames in a set J, and removing all the rest detection frames; let the left upper-left coordinates of the two detection frames be (Lx1, Ly1), (Lx2, Ly2), and the right lower-right coordinates be (Rx1, Ry1), (Rx2, Ry 2);
5.2.2) merging according to the space positions of the two detection frames in the image to determine the type of the disease at the position:
when the four conditions of Lx1 < Lx2 < Rx1, Lx2 < Rx1 < Rx2, Ry2 < Ry1 < Ly2 and Ry1 < Ly2 < Ly1 are simultaneously met, combining two detection frames into a detection frame with the coordinates of the upper left corner (Lx-Lx 1, Ly-Ly 1) and the coordinates of the lower right corner (Rx-Rx 2 and Ry-Ry 2), wherein the type of the detection frame is the type of the disease;
when the four conditions of Lx2 < Lx1 < Rx2, Lx1 < Rx2 < Rx1, Ry1 < Ry2 < Ly1 and Ry2 < Ly1 < Ly2 are simultaneously met, combining two detection frames into a detection frame with the coordinates of the upper left corner (Lx-Lx 2, Ly-Ly 2) and the coordinates of the lower right corner (Rx-Rx 1 and Ry-Ry 1), wherein the type of the detection frame is the type of the disease;
when the four conditions of Lx1 < Lx2 < Rx1, Ry1 < Ry2 < Ly1, Lx2 < Rx1 < Rx2 and Ry2 < Ly1 < Ly2 are simultaneously met, two detection frames are combined into one detection frame with the coordinates of the upper left corner (Lx-Lx 1, Ly-Ly 2) and the coordinates of the lower right corner (Rx-Rx 2 and Ry-Ry 1), and the type of the detection frame is the type of the disease;
when the four conditions of Lx2 < Lx1 < Rx2, Ry2 < Ry1 < Ly2, Lx1 < Rx2 < Rx1 and Ry1 < Ly2 < Ly1 are simultaneously met, two detection frames are combined into one detection frame with the coordinates of the upper left corner (Lx-Lx 2, Ly-Ly 1) and the coordinates of the lower right corner (Rx-Rx 1 and Ry-Ry 2), and the type of the detection frame is the type of the disease;
when the four conditions of Lx1 < Lx2 < Rx1, Ry1 < Ly2 < Ly1, Lx1 < Rx2 < Rx1 and Ry1 < Ry2 < Ly1 are simultaneously met, two detection frames are combined into one detection frame with the coordinates of the upper left corner (Lx-Lx 1, Ly-Ly 1) and the coordinates of the lower right corner (Rx-Rx 1 and Ry-Ry 1), and the type of the detection frame is the type of the disease;
when the four conditions of Lx2 < Lx1 < Rx2, Ry2 < Ly1 < Ly2, Lx2 < Rx1 < Rx2 and Ry2 < Ry1 < Ly2 are simultaneously satisfied, two detection frames are combined into one detection frame with the coordinates of the upper left corner (Lx-Lx 2, Ly-Ly 2) and the coordinates of the lower right corner (Rx-Rx 2 and Ry-Ry 2), and the type of the detection frame is the type of the disease.
And 6, removing the weight of the same disease in the continuous multi-frame images.
6.1) respectively allocating a tracker for predicting the motion position of a target disease appearing in a video in the next frame by using a Kalman filtering algorithm, and respectively creating unique identifiers for different types of targets entering video frame images through a multi-target tracking sort in the Kalman filtering algorithm, wherein each tracker corresponds to a disease tracking number m;
6.2) correcting the predicted position according to the detection result of the road surface disease position in the next frame to obtain the optimal position estimation value tracked by the disease in the frame;
6.2.1 creating a tracker list TL for storing the tracker with the tracking number m corresponding to the successfully tracked diseases in the previous frame of the video;
6.2.2 creating a detection list DL for storing a detection result of the pavement disease obtained after the current frame is subjected to feature extraction by using a pavement disease detection model B';
6.2.3 traversing each tracker in the tracker list TL, predicting the position of the road surface disease in the current frame according to the position of the road surface disease stored in the TL in the previous frame, and storing all the predicted positions of the road surface disease in the current frame into the prediction list PL;
6.2.4 updating TL from DL and PL:
calculating the intersection and parallel ratio of each detection result in the DL and the PL and a rectangular frame represented by the prediction result to obtain an intersection and parallel ratio matrix IOU;
and matching each detection result with each prediction result by using the Hungarian algorithm by taking the IOU as a weight parameter to finish updating the tracker list TL, wherein the updated tracker list TL' stores the optimal position estimation value of the disease in the current video frame.
And 7, counting the diseases of different categories.
Comparing the disease at the optimal position in the current frame with the disease tracked in the previous frame:
7.1) if the two diseases are the same disease, carrying out duplicate removal counting on the disease in two frames before and after, and realizing the following steps:
7.1.1) respectively allocating a global counter for each different type of road surface diseases;
7.1.2) initializing a tracking number list for storing the number of the counted road surface diseases;
7.1.3) on the premise of ensuring the same disease, judging whether the tracking number of the road surface disease exists in a tracking number list:
if so, canceling repeated counting of the tracked diseases during counting;
and if the global counter does not exist, adding one to the global counter of the category to obtain updated pavement disease statistical information.
7.2) if the two are not the same disease, counting the tracked diseases to obtain the real-time tracking counting and detection result of the same disease in the continuous frames.
The foregoing description is only an example of the present invention and is not intended to limit the invention, so that it will be apparent to those skilled in the art that various changes and modifications in form and detail may be made therein without departing from the spirit and scope of the invention.

Claims (7)

1. A pavement disease detection method based on deep learning is characterized by comprising the following steps:
(1) the method comprises the following steps of shooting road video frame images through a vehicle-mounted camera, and obtaining a training sample set L:
(2) a yolov5 network is selected as a pavement disease detection model B:
(3) inputting the training sample set L into a pavement disease detection model B, and performing iterative training on the pavement disease detection model B by using a gradient descent method to obtain a trained pavement disease detection model B';
(4) the types of the road surface diseases in the inspection video are as follows:
(4a) shooting a road surface video S to be detected by using a vehicle-mounted camera as the input of a road surface disease detection model B ', and comparing frame by frame and pixel by pixel to obtain a video S' marked with the road surface disease category and confidence information;
(4b) framing the video S' to obtain a video frame image set C ═ C1,C2,...,Cq,...,CnIn which C isqRepresenting the q-th video frame image containing the road surface disease information, wherein n represents the number of the video frame images, and n is more than or equal to 100;
(5) determining the types of the diseases at the same position in the single-frame images of the video frame image set C:
(5a) determining the type of the disease according to the marking condition of the detected frame of the disease at the same position:
if the same position of a single-frame image in the video frame image set C is marked by a detection frame, the type of the detection frame is the type of the disease;
if the same position of the single frame image in the video frame image set C is marked by a plurality of detection frames, judging whether the types of the detection frames are the same or not; if the detection frame types are the same, executing (5 b);
(5b) combining the detection frames by adopting a disease processing method for spatial position combination to determine the type of the disease, and completing the de-duplication of the road disease detection result:
(5b1) sorting the confidence degrees of the output detection frames according to a descending order, reserving the first two detection frames, storing the coordinates of the upper left corner and the lower right corner of the first two detection frames in a set J, and removing all the rest detection frames; let the left upper-left coordinates of the two detection frames be (Lx1, Ly1), (Lx2, Ly2), and the right lower-right coordinates be (Rx1, Ry1), (Rx2, Ry 2);
(5b2) and combining the space positions of the two detection frames in the image to determine the type of the disease at the position:
when the four conditions of Lx1 < Lx2 < Rx1, Lx2 < Rx1 < Rx2, Ry2 < Ry1 < Ly2 and Ry1 < Ly2 < Ly1 are simultaneously met, combining two detection frames into a detection frame with the coordinates of the upper left corner (Lx-Lx 1, Ly-Ly 1) and the coordinates of the lower right corner (Rx-Rx 2 and Ry-Ry 2), wherein the type of the detection frame is the type of the disease;
when the four conditions of Lx2 < Lx1 < Rx2, Lx1 < Rx2 < Rx1, Ry1 < Ry2 < Ly1 and Ry2 < Ly1 < Ly2 are simultaneously met, combining two detection frames into a detection frame with the coordinates of the upper left corner (Lx-Lx 2, Ly-Ly 2) and the coordinates of the lower right corner (Rx-Rx 1 and Ry-Ry 1), wherein the type of the detection frame is the type of the disease;
when the four conditions of Lx1 < Lx2 < Rx1, Ry1 < Ry2 < Ly1, Lx2 < Rx1 < Rx2 and Ry2 < Ly1 < Ly2 are simultaneously met, two detection frames are combined into one detection frame with the coordinates of the upper left corner (Lx-Lx 1, Ly-Ly 2) and the coordinates of the lower right corner (Rx-Rx 2 and Ry-Ry 1), and the type of the detection frame is the type of the disease;
when the four conditions of Lx2 < Lx1 < Rx2, Ry2 < Ry1 < Ly2, Lx1 < Rx2 < Rx1 and Ry1 < Ly2 < Ly1 are simultaneously met, two detection frames are combined into one detection frame with the coordinates of the upper left corner (Lx-Lx 2, Ly-Ly 1) and the coordinates of the lower right corner (Rx-Rx 1 and Ry-Ry 2), and the type of the detection frame is the type of the disease;
when the four conditions of Lx1 < Lx2 < Rx1, Ry1 < Ly2 < Ly1, Lx1 < Rx2 < Rx1 and Ry1 < Ry2 < Ly1 are simultaneously met, two detection frames are combined into one detection frame with the coordinates of the upper left corner (Lx-Lx 1, Ly-Ly 1) and the coordinates of the lower right corner (Rx-Rx 1 and Ry-Ry 1), and the type of the detection frame is the type of the disease;
when the four conditions of Lx2 < Lx1 < Rx2, Ry2 < Ly1 < Ly2, Lx2 < Rx1 < Rx2 and Ry2 < Ry1 < Ly2 are simultaneously met, two detection frames are combined into one detection frame with the coordinates of the upper left corner (Lx-Lx 2, Ly-Ly 2) and the coordinates of the lower right corner (Rx-Rx 2 and Ry-Ry 2), and the type of the detection frame is the type of the disease;
(6) carrying out duplicate removal and counting treatment on the same disease in continuous multi-frame images:
(6a) respectively allocating a tracker for predicting the motion position of a target disease appearing in a video in the next frame by using a Kalman filtering algorithm, wherein each tracker corresponds to a disease tracking number m;
(6b) correcting the predicted position according to the detection result of the road surface disease position in the next frame to obtain the optimal position estimation value of the disease tracked in the frame;
(6c) comparing the disease at the optimal position in the current frame with the disease tracked in the previous frame:
if the two diseases are the same disease, the disease is subjected to duplicate removal counting in two frames before and after;
if the two are not the same disease, counting the tracked diseases to obtain the real-time tracking counting and detection result of the same disease in the continuous frames.
2. The method of claim 1: wherein, the training sample set L is obtained in (1) and is realized as follows:
(1a) shooting road video frame images by using a vehicle-mounted camera, and extracting N video frame images containing road diseases from the road video frame images, wherein A is { A ═ A }1,A2,...,Ai,...,ANAnd using a rectangular frame in the labellmg software to carry out image A on each video frameiMarking the road surface diseases of medium categories including transverse cracks, longitudinal cracks, cracks and ruts to obtain a label set P ═ P containing disease information1,P2,...,Pi,...,PNWherein N is more than or equal to 3000, AiRepresents the ith road surface image containing the fault, PiIs represented by AiA corresponding label containing road surface disease information;
(1b) label set P ═ P of different disease information1,P2,...,Pi,...,PNThe original video frame image A corresponding to the original video frame image A is equal to { A }1,A2,...,Ai,...,ANCombining to form a training sample set: l ═ L1,L2,...,Lj,...,LNIn which L isjIs PjAnd AjThe j-th training sample is formed by combining, wherein j is 1, 2.
3. The method of claim 1, wherein the yolov5 network in (2) comprises a Backbone network Backbone, a target feature network tack and a detection network Head;
the Backbone network backhaul comprises a CSPdacrynet module and a spatial pyramid pooling SPP module, wherein the CSPdacrynet module comprises a Focus submodule, a convolutional layer with 32 convolutional kernels and a BottleneckCSP submodule, and the BottleneckCSP submodule comprises a convolutional layer Conv, a batch normalization layer BN, an activation function Leaky Relu and a residual error layer which are sequentially connected;
the target feature network Neck is formed by splicing a feature pyramid FPN and a pyramid PAN from bottom to top;
the detection network Head consists of three convolution layers connected in parallel.
4. The method according to claim 1, wherein the pavement disease detection model B is iteratively trained by using a gradient descent method in (3) and the following is implemented:
3a) setting the maximum iteration number as Y, wherein Y is more than or equal to 500, and setting the pavement disease detection model of the Y iteration as ByThe number of initialization iterations y is 0 and B is By；
3b) Taking a training sample set L as a pavement disease detection model ByThe input of the network is pushed forward, and L is transmitted through a Focus sub-module in the backbone networkjSlicing to obtain a feature map of 304 × 12, and performing a convolution operation with 32 convolution kernels to obtain a feature map of 304 × 32;
3c) inputting the result obtained in the step 3b) into a convolutional layer Conv, a batch normalization layer BN, an activation function Leaky Relu and a residual layer which are sequentially connected in a BottleneckCSP submodule to perform feature extraction, and then obtaining a feature map of 38 x 32;
3d) inputting the result obtained in the step 3c) into a spatial pyramid pooling module SPP, extracting features with different scales through pooling layers of kernel sizes, and fusing the extracted features to obtain a feature map of 19 × 32;
3e) sending the feature map obtained in the step 3d) into a target feature network Neck, fusing the upper-layer features and the lower-layer features through a top-down FPN structure to obtain a predicted feature map, and transmitting the strong positioning features of the lower layer to the top layer through a pyramid PAN structure from bottom to top through the predicted feature map to obtain a feature map containing semantics and positioning information
Figure FDA0003316472220000041
Wherein
Figure FDA0003316472220000042
The jth feature map containing semantic and positioning information;
3f) adopting a frame Loss function CIOU _ Loss and passing
Figure FDA0003316472220000043
And LjDetection model B for calculating road surface diseasesyCross entropy Loss value Loss ofsThen through LosssModel B for detecting road surface diseasesyThe weight value of the road surface defect detection model B is updated to obtain a road surface defect detection model B after the y iterationy；
3g) And judging whether the current iteration number reaches the set maximum iteration number Y, if so, obtaining a trained pavement disease detection model B', otherwise, enabling Y to be Y +1, and returning to 3 c).
5. The method according to claim 1, wherein in (6a), a tracker for predicting the motion position of the disease in the next frame is respectively allocated to the target disease appearing in the video by using a kalman filtering algorithm, and a multi-target tracking sort in the kalman filtering algorithm is used to respectively create unique identifiers for different types of targets entering the video frame image, and the identifiers are the allocated trackers.
6. The method according to claim 1, wherein the predicted position is corrected according to the detection result of the road defect position in the next frame in (6b), and the following is implemented:
6b1) creating a tracker list TL for storing a tracker with a tracking number m corresponding to a successfully tracked disease in a previous frame of a video;
6b2) creating a detection list DL for storing a detection result of the pavement diseases obtained after feature extraction is carried out on the current frame by using a pavement disease detection model B';
6b3) traversing each tracker in the tracker list TL, predicting the position of the road disease in the current frame according to the position of the road disease stored in the TL in the previous frame, and storing all the predicted positions of the road disease in the current frame in the prediction list PL;
6b4) TL is updated according to DL and PL:
calculating the intersection and parallel ratio of each detection result in the DL and the PL and a rectangular frame represented by the prediction result to obtain an intersection and parallel ratio matrix IOU;
and matching each detection result with each prediction result by using the Hungarian algorithm by taking the IOU as a weight parameter to finish updating the tracker list TL, wherein the updated tracker list TL' stores the optimal position estimation value of the disease in the current video frame.
7. The method according to claim 1, wherein in (6c), the disease at the optimal position in the current frame is the same disease as the disease tracked in the previous frame, and the duplicate removal counting is performed in the previous frame and the next frame, which is implemented as follows:
6c1) respectively distributing a global counter for each different type of pavement diseases;
6c2) initializing a tracking number list for storing the number of the counted road surface diseases;
6c3) on the premise of ensuring the same disease, judging whether the tracking number of the road surface disease exists in a tracking number list:
if so, canceling repeated counting of the tracked diseases during counting;
and if the global counter does not exist, adding one to the global counter of the category to obtain updated pavement disease statistical information.
CN202111232233.0A 2021-10-22 2021-10-22 Pavement disease detection method based on deep learning Pending CN113962960A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202111232233.0A CN113962960A (en) 2021-10-22 2021-10-22 Pavement disease detection method based on deep learning

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202111232233.0A CN113962960A (en) 2021-10-22 2021-10-22 Pavement disease detection method based on deep learning

Publications (1)

Publication Number Publication Date
CN113962960A true CN113962960A (en) 2022-01-21

Family

ID=79466134

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202111232233.0A Pending CN113962960A (en) 2021-10-22 2021-10-22 Pavement disease detection method based on deep learning

Country Status (1)

Country Link
CN (1) CN113962960A (en)

Cited By (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114495068A (en) * 2022-04-18 2022-05-13 河北工业大学 Road surface health detection 'element' method based on man-machine interaction and deep learning
CN115272930A (en) * 2022-07-28 2022-11-01 广西北投交通养护科技集团有限公司 Ground penetrating radar-based road surface state evaluation method
CN116363530A (en) * 2023-03-14 2023-06-30 北京天鼎殊同科技有限公司 Method and device for positioning expressway pavement diseases
CN116485843A (en) * 2023-05-24 2023-07-25 江苏全栈智能科技有限公司 Road disease identification tracking method
CN116797436A (en) * 2023-08-29 2023-09-22 北京道仪数慧科技有限公司 Processing system for carrying out road disease inspection by utilizing bus
CN117078235A (en) * 2023-10-17 2023-11-17 深圳市城市交通规划设计研究中心股份有限公司 Road network maintenance method, electronic equipment and storage medium for comprehensive evaluation

Cited By (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114495068A (en) * 2022-04-18 2022-05-13 河北工业大学 Road surface health detection 'element' method based on man-machine interaction and deep learning
CN115272930A (en) * 2022-07-28 2022-11-01 广西北投交通养护科技集团有限公司 Ground penetrating radar-based road surface state evaluation method
CN116363530A (en) * 2023-03-14 2023-06-30 北京天鼎殊同科技有限公司 Method and device for positioning expressway pavement diseases
CN116363530B (en) * 2023-03-14 2023-11-03 北京天鼎殊同科技有限公司 Method and device for positioning expressway pavement diseases
CN116485843A (en) * 2023-05-24 2023-07-25 江苏全栈智能科技有限公司 Road disease identification tracking method
CN116797436A (en) * 2023-08-29 2023-09-22 北京道仪数慧科技有限公司 Processing system for carrying out road disease inspection by utilizing bus
CN116797436B (en) * 2023-08-29 2023-10-31 北京道仪数慧科技有限公司 Processing system for carrying out road disease inspection by utilizing bus
CN117078235A (en) * 2023-10-17 2023-11-17 深圳市城市交通规划设计研究中心股份有限公司 Road network maintenance method, electronic equipment and storage medium for comprehensive evaluation
CN117078235B (en) * 2023-10-17 2024-03-26 深圳市城市交通规划设计研究中心股份有限公司 Road network maintenance method, electronic equipment and storage medium for comprehensive evaluation

Similar Documents

Publication Publication Date Title
Spencer Jr et al. Advances in computer vision-based civil infrastructure inspection and monitoring
CN113962960A (en) Pavement disease detection method based on deep learning
CN113160192B (en) Visual sense-based snow pressing vehicle appearance defect detection method and device under complex background
CN108171112B (en) Vehicle identification and tracking method based on convolutional neural network
CN110135522B (en) Intelligent method for detecting and marking small target integration of remote sensing image
CN110264444B (en) Damage detection method and device based on weak segmentation
CN111985325B (en) Aerial small target rapid identification method in extra-high voltage environment evaluation
CN111274926B (en) Image data screening method, device, computer equipment and storage medium
CN108711148A (en) A kind of wheel tyre defect intelligent detecting method based on deep learning
CN114596500A (en) Remote sensing image semantic segmentation method based on channel-space attention and DeeplabV3plus
CN116229292A (en) Inspection system and method based on unmanned aerial vehicle road surface inspection disease
Li et al. Pixel‐Level Recognition of Pavement Distresses Based on U‐Net
CN116824399A (en) Pavement crack identification method based on improved YOLOv5 neural network
CN115995056A (en) Automatic bridge disease identification method based on deep learning
CN114973116A (en) Method and system for detecting foreign matters embedded into airport runway at night by self-attention feature
CN116109616A (en) Pavement crack detection and small-surface element fitting detection method based on YOLOv5
Guerrieri et al. Flexible and stone pavements distress detection and measurement by deep learning and low-cost detection devices
KC Enhanced pothole detection system using YOLOX algorithm
CN115719475A (en) Three-stage trackside equipment fault automatic detection method based on deep learning
CN113205511B (en) Electronic component batch information detection method and system based on deep neural network
CN113158954B (en) Automatic detection method for zebra crossing region based on AI technology in traffic offsite
Ranyal et al. Enhancing pavement health assessment: An attention-based approach for accurate crack detection, measurement, and mapping
CN113902793A (en) End-to-end building height prediction method and system based on single vision remote sensing image and electronic equipment
Wang et al. Instance segmentation of soft‐story buildings from street‐view images with semiautomatic annotation
CN110765900B (en) Automatic detection illegal building method and system based on DSSD

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination