CN111242017A - Multi-marking-line pavement crack identification method, device, equipment and storage medium - Google Patents
Multi-marking-line pavement crack identification method, device, equipment and storage medium Download PDFInfo
- Publication number
- CN111242017A CN111242017A CN202010027072.0A CN202010027072A CN111242017A CN 111242017 A CN111242017 A CN 111242017A CN 202010027072 A CN202010027072 A CN 202010027072A CN 111242017 A CN111242017 A CN 111242017A
- Authority
- CN
- China
- Prior art keywords
- crack
- training
- marking
- road surface
- model
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 60
- 238000012549 training Methods 0.000 claims abstract description 105
- 239000011159 matrix material Substances 0.000 claims description 37
- 230000006870 function Effects 0.000 claims description 33
- 238000000605 extraction Methods 0.000 claims description 19
- 238000005457 optimization Methods 0.000 claims description 12
- 238000007781 pre-processing Methods 0.000 claims description 8
- 238000009499 grossing Methods 0.000 claims description 6
- 238000011176 pooling Methods 0.000 claims description 6
- 230000004913 activation Effects 0.000 claims description 4
- 238000004590 computer program Methods 0.000 claims description 3
- 238000005070 sampling Methods 0.000 claims description 2
- 230000000694 effects Effects 0.000 abstract description 11
- 238000013135 deep learning Methods 0.000 abstract description 7
- 238000003062 neural network model Methods 0.000 abstract description 2
- 238000012360 testing method Methods 0.000 description 18
- 238000004891 communication Methods 0.000 description 10
- 238000004364 calculation method Methods 0.000 description 7
- 238000010586 diagram Methods 0.000 description 7
- 230000008569 process Effects 0.000 description 7
- 238000012545 processing Methods 0.000 description 7
- 238000005516 engineering process Methods 0.000 description 5
- 239000000523 sample Substances 0.000 description 5
- ORILYTVJVMAKLC-UHFFFAOYSA-N Adamantane Natural products C1C(C2)CC3CC1CC2C3 ORILYTVJVMAKLC-UHFFFAOYSA-N 0.000 description 4
- 238000013527 convolutional neural network Methods 0.000 description 4
- 230000008901 benefit Effects 0.000 description 3
- 238000011156 evaluation Methods 0.000 description 3
- 238000012986 modification Methods 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 238000013528 artificial neural network Methods 0.000 description 2
- 230000007613 environmental effect Effects 0.000 description 2
- 238000005286 illumination Methods 0.000 description 2
- 230000005012 migration Effects 0.000 description 2
- 238000013508 migration Methods 0.000 description 2
- 238000012567 pattern recognition method Methods 0.000 description 2
- 230000003068 static effect Effects 0.000 description 2
- 238000005299 abrasion Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 238000004422 calculation algorithm Methods 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000005336 cracking Methods 0.000 description 1
- 230000001419 dependent effect Effects 0.000 description 1
- 230000008034 disappearance Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000037361 pathway Effects 0.000 description 1
- 230000001737 promoting effect Effects 0.000 description 1
- 230000000644 propagated effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000012795 verification Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/10—Terrestrial scenes
- G06V20/13—Satellite images
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
- Y02T10/00—Road transport of goods or passengers
- Y02T10/10—Internal combustion engine [ICE] based vehicles
- Y02T10/40—Engine management systems
Abstract
One or more embodiments of the present specification provide a method, an apparatus, a device, and a storage medium for identifying a pavement crack with multiple marked lines, including: acquiring an actual road surface image; obtaining a marking model and a crack model; the marking model is obtained through training based on a first training pavement image set, and the first training pavement image set is associated with a first label representing a marking; the crack model is obtained through training based on a second training road surface image set, and the second training road surface image set is associated with a second label representing a crack; inputting the actual road surface image into the marking line model to obtain a marking line identification result; inputting the actual pavement image into a crack model to obtain a crack identification result; and subtracting the marking line identification result from the crack identification result to obtain the final identification result of the actual pavement crack. The method utilizes the neural network model and the deep learning mode to identify the actual road surface image, improves the identification accuracy, has extremely high identification speed, ensures that the identification result is slightly influenced by the environment through model superposition, and improves the identification effect.
Description
Technical Field
One or more embodiments of the present disclosure relate to the field of image recognition, and in particular, to a method, an apparatus, a device, and a storage medium for recognizing a multi-marked road crack.
Background
The current identification of the road marking is basically based on a digital image processing technology, and the road marking is easy to be identified as a crack by mistake due to cracking and abrasion on the marking, so that the identification effect of the crack is reduced. The road surface image collected in the actual environment is often influenced by illumination and noise, the crack is identified by the traditional image identification method through a manual selection algorithm, but effective identification can be carried out only under the conditions of good illumination condition and clear performance, and the identification accuracy rate is low in actual operation. In the prior art, the pattern recognition method based on the gray-scale image has low recognition speed and cannot realize real-time recognition. The single crack model is used for crack recognition, and the crack recognition method is influenced by factors such as pavement background diversity, crack diversity and construction places which do not need recognition in an actual scene, is greatly influenced by the environment, and is poor in recognition effect.
Disclosure of Invention
In view of the above, one or more embodiments of the present disclosure are directed to a method, an apparatus, a device, and a storage medium for identifying a pavement crack with multiple marked lines, so as to solve the problems of poor identification effect, low identification accuracy, slow identification speed, and large environmental impact in the prior art.
In view of the above objects, one or more embodiments of the present specification provide a multi-marking pavement crack recognition method, including:
acquiring an actual road surface image;
obtaining a marking model and a crack model; the marking model is obtained through training based on a first training pavement image set, and the first training pavement image set is associated with a first label representing a marking; the crack model is obtained through training based on a second training road surface image set, and a second label representing a crack is associated with the second training road surface image set;
inputting the actual road surface image into the marking line model to obtain a marking line identification result;
inputting the actual pavement image into the crack model to obtain a crack identification result;
and subtracting the marking line identification result from the crack identification result to obtain the final identification result of the actual pavement crack.
Optionally, the method further includes: preprocessing the actual road surface image;
the preprocessing operation comprises: and carrying out image filling and image adjusting operation on the actual road surface image.
Optionally, the marking model is obtained by training based on a first training road surface image set, and the first training road surface image set is associated with a first label representing a marking; the crack model is obtained by training based on a second training road surface image set, the second training road surface image set is associated with a second label representing a crack, the training is carried out for a plurality of rounds, and each round of the training comprises:
setting network parameters, and inputting the first training road surface image set into the marking model in the forward direction and inputting the second training road surface image set into the crack model in the forward direction;
the marking model performs characteristic extraction on the first training pavement image set to obtain a marking prediction result, and the crack model performs characteristic extraction on the second training pavement image set to obtain a crack prediction result;
calculating a reticle error between the reticle prediction result and a real value and a fracture error between the fracture prediction result and the real value by using a loss function;
and adjusting the network parameters by combining the marking line errors, the crack errors, the gradients of the network parameters and an optimization method, and taking the adjusted network parameters as the network parameters of the next round of training.
Optionally, the feature extraction includes:
down-sampling the first training road surface image set and the second training road surface image set through 2 x 2 maximum pooling;
and performing the feature extraction on the first training road surface image set and the second training road surface image set by using a 5 × 5 convolution kernel and a 3 × 3 convolution kernel, and a ReLU function and a sigmoid function as activation functions.
Optionally, the loss function is defined by
Wherein dice represents a sample similarity, X represents a probability matrix formed by the actual road surface image through network feedforward, and Y represents a mark matrix formed by the first or second training road surface image sets, | X |1L representing the probability matrix X1Norm, | Y |1L representing the mark matrix Y1Norm, X X Y represents the Hadamard product of the probability matrix X and the label matrix Y, and e is the first smoothing factor.
Optionally, the definition formula of the optimization method is
mt＝β1·mt-1+(1-β1)·gt
vt＝β2·vt-1+(1-β2)·gt 2
Wherein L is the loss function, gtFor the gradient of the loss function L in respect of the parameter theta at the t-th round, thetatThe value of the parameter theta at the t-th round, mtFor recording first moments, vtFor recording the second moment of the wave or waves,
is mtIs determined by the estimated value of (c),
is v istEstimate of (a), β1Is the first over-parameter β2Is the second hyperparameter, lr is the learning rate, e0Is the second smoothing factor.
Optionally, subtracting the marking line recognition result from the crack recognition result to obtain a final recognition result of the actual pavement crack, where the method includes:
the marking model identifies the actual pavement image to obtain a marking identification probability matrix, and the marking identification probability matrix is discretized to obtain a marking identification result;
the crack model identifies the actual pavement image to obtain a crack identification probability matrix, and discretizing the crack identification probability matrix to obtain a crack identification result;
and subtracting the marking line identification result from the crack identification result to obtain the final identification result of the actual pavement crack.
In view of the above objects, one or more embodiments of the present specification also provide a multi-marking pavement crack recognition apparatus including:
a first acquisition module configured to acquire an actual road surface image;
a second obtaining module configured to obtain a reticle model and a fracture model; the marking model is obtained through training based on a first training pavement image set, and the first training pavement image set is associated with a first label representing a marking; the crack model is obtained through training based on a second training road surface image set, and a second label representing a crack is associated with the second training road surface image set;
the first recognition module is configured to input the actual road surface image into the marking model to obtain a marking recognition result;
the second identification module is configured to input the actual road surface image into the crack model to obtain a crack identification result;
and the result output module is configured to subtract the marking line identification result from the crack identification result to obtain a final identification result of the actual pavement crack.
In view of the above, one or more embodiments of the present specification further provide an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements any one of the above-mentioned methods when executing the program.
In view of the above, one or more embodiments of the present specification also provide a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions for causing the computer to perform any of the methods.
As can be seen from the above description, in the multi-reticle pavement crack recognition method, the multi-reticle pavement crack recognition device, the multi-reticle pavement crack recognition equipment and the storage medium provided in one or more embodiments of the present disclosure, because the used convolutional neural network contains a large number of parameters, the recognition method has a strong abstract fitting capability, and can effectively improve the recognition accuracy, the average similarity of the reticle recognition performed on the pavement image by the data image processing technology is only about 60%, and the average similarity of the method provided in one or more embodiments of the present disclosure reaches more than 90%. The invention is suitable for the deep learning mode to carry out the image recognition, and actually leads a large number of calculation processes, so that a large number of calculation quantities can be reduced when processing unknown images, specifically, the unknown images can obtain automatically marked marking line areas only by carrying out previous items in a network, the calculation quantity is half of that of the training process, and the recognition speed is extremely high. Compared with the single model for identifying the cracks in the prior art, the method has the advantages that the crack model and the marking model are superposed, and the deep learning method is used for effectively identifying the marks and the cracks, so that the influence of the marks is eliminated, the influence of the environment is small, and the crack identification effect is improved.
Drawings
In order to more clearly illustrate one or more embodiments or prior art solutions of the present specification, the drawings that are needed in the description of the embodiments or prior art will be briefly described below, and it is obvious that the drawings in the following description are only one or more embodiments of the present specification, and that other drawings may be obtained by those skilled in the art without inventive effort from these drawings.
FIG. 1 is a flow diagram of a method for identifying pavement cracks in one or more embodiments of the present disclosure;
FIG. 2 is a diagram of a deep learning network architecture in one or more embodiments of the present disclosure;
FIG. 3 is a diagram of an overlay model identification scheme in one or more embodiments of the present description;
FIG. 4 is a graph illustrating the effectiveness of crack identification in one or more embodiments of the present disclosure;
FIG. 5 is a graph of reticle identification effectiveness in one or more embodiments of the present disclosure;
FIG. 6 is a schematic view of a pavement crack identification apparatus according to one or more embodiments of the present disclosure;
FIG. 7 is a block diagram of an electronic device in one or more embodiments of the disclosure.
Detailed Description
For the purpose of promoting a better understanding of the objects, aspects and advantages of the present disclosure, reference is made to the following detailed description taken in conjunction with the accompanying drawings.
It is to be noted that unless otherwise defined, technical or scientific terms used in one or more embodiments of the present specification should have the ordinary meaning as understood by those of ordinary skill in the art to which this disclosure belongs. The use of "first," "second," and similar terms in one or more embodiments of the specification is not intended to indicate any order, quantity, or importance, but rather is used to distinguish one element from another. The word "comprising" or "comprises", and the like, means that the element or item listed before the word covers the element or item listed after the word and its equivalents, but does not exclude other elements or items. The terms "connected" or "coupled" and the like are not restricted to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "upper", "lower", "left", "right", and the like are used merely to indicate relative positional relationships, and when the absolute position of the object being described is changed, the relative positional relationships may also be changed accordingly.
One or more embodiments of the present specification provide a multi-marked line pavement crack identification method, apparatus, device and storage medium.
Referring to fig. 1, a method of an embodiment of the invention includes the steps of:
s101 acquires an actual road surface image.
In this step, an actual road surface image needs to be acquired from an actual road surface through image acquisition. In practical application, the method is used for detecting and identifying the cracks on the road surface containing the multiple marked lines in the actual road surface maintenance work, as an optional embodiment, an image acquisition device, such as a camera or a road surface probe, is used for acquiring images, acquiring actual road surface images, locally identifying the cracks on the road surface by using the method provided by the invention, rapidly identifying the cracks on the road surface, or identifying the cracks on the road surface remotely by using a server, and more accurately identifying the cracks on the road surface.
In this embodiment, after obtaining the actual road surface image information, a preprocessing operation is performed on the actual road surface image, where the preprocessing operation includes: and carrying out image filling and image adjustment operation on the actual road surface image, and obtaining a gray level image after processing, so that the images of the input marking model and the crack model have consistency.
S102, obtaining a marking model and a crack model; the marking model is obtained through training based on a first training pavement image set, and the first training pavement image set is associated with a first label representing a marking; the crack model is obtained through training based on a second training road surface image set, and a second label representing a crack is associated with the second training actual road surface image set.
In the step, a convolution neural network is built, and a convolution layer and a pooling layer are built, so that a marking model and a crack model are built. The marking model needs to extract the characteristics of a first training pavement image set, the first training pavement image set is associated with a first label expressing a marking, and the input first training pavement image set is converted into a corresponding training marking prediction probability matrix, so that a marking prediction result is output; the crack model needs to perform feature extraction on a second training road surface image set, the second training actual road surface image set is associated with a second label representing a crack, the input second training road surface image set is converted into a corresponding training crack prediction probability matrix, and therefore a crack prediction result is output. In this embodiment, the first training road surface image set is used to train the marking model for 100 rounds, the second training road surface image set is used to train the crack model for 100 rounds, and each round of training specifically includes the following steps:
setting network parameters, and inputting the first training road surface image set into the marking model in the forward direction and inputting the second training road surface image set into the crack model in the forward direction;
the marking model performs characteristic extraction on the first training pavement image set to obtain a marking prediction result, and the crack model performs characteristic extraction on the second training pavement image set to obtain a crack prediction result;
calculating a reticle error between the reticle prediction result and a real value and a fracture error between the fracture prediction result and the real value by using a loss function;
and adjusting the network parameters by combining the marking line errors, the crack errors, the gradients of the network parameters and an optimization method, and taking the adjusted network parameters as the network parameters of the next round of training.
In this embodiment, the network parameters are model parameters such as a batch size and a learning rate, the value of the batch size is 32, and the value of the learning rate is 0.001, the convolutional neural network adopted in the present invention includes a convolutional layer and a pooling layer, referring to fig. 2, taking a marking model to perform feature extraction on a first training road image set as an example, initially performing feature extraction through a 5 × 5 convolutional kernel, and then adopting a 3 × 3 convolutional kernel, so as to obtain a larger receptive field and perform better feature extraction; the pooling layer performs downsampling through maximum pooling, so that main characteristics can be reserved, and parameters and calculated amount can be reduced; and (4) taking a ReLU function and a sigmoid function as activation functions, extracting features, and finally outputting an identification probability matrix. The ReLU function is high in calculation speed during forward propagation, the problem of gradient disappearance does not exist during reverse propagation, convergence of a neural network model is maintained in a stable state, the value range of the sigmoid function is (0,1) and meets the requirements in actual operation, and the sigmoid function has good symmetry, so that the selected activation functions in the embodiment are the ReLU function and the sigmoid function. In the embodiment, a ReLU function and a convolution kernel combination of 5 × 5, a ReLU function and a convolution kernel combination of 3 × 3 are adopted, Maxpool is adopted for feature extraction by a convolution kernel combination of Maxpool 2 × 2, sigmoid function and 1 × 1, an obtained actual road surface image is 3400 × 2200, a gray map of 1088 × 704 is adopted when a convolution neural network is input, and a final output prediction result is a probability matrix of 34 × 22 of a value range (0, 1).
In this embodiment, the loss function is defined by
Wherein dice represents a sample similarity, X represents a probability matrix formed by the actual road surface image through network feedforward, and Y represents a mark matrix formed by the first or second training road surface image sets, | X |1L representing the probability matrix X1Norm, | Y |1L representing the mark matrix Y1Norm, where X × Y represents the hadamard product of the probability matrix X and the label matrix Y, e is the first smoothing factor, and the value in this embodiment is 10-3. The optimization method used in training is an Adam optimization method, the first moment and the second moment of the loss function are recorded at the same time, the Adam optimization method is a self-adaptive network optimization mode, and the Adam optimization method is defined as
mt＝β1·mt-1+(1-β1)·gt
vt＝β2·vt-1+(1-β2)·gt 2
Wherein, gtThe gradient of a loss function L with respect to a parameter theta at the t-th round, L being the loss function, thetatIs the value of theta at the t-th round, mtFor recording first moments, vtFor recording the second moment of the wave or waves,
is mtIs determined by the estimated value of (c),
is v istEstimate of (a), β1Is the first over-parameter β2β in this embodiment as the second hyperparameter1Value of 0.9, β2The value is 0.999, lr is the learning rate, and the value is 10 in this embodiment-3，∈0The second smoothing factor is 10 in this embodiment-8。
In this embodiment, the first training road surface image set and the second training road surface image set are both pre-marked with real values, and in the training process, the reticle error between the reticle prediction result of the current round and the real value of the current round and the fracture error between the fracture prediction result of the current round and the real value of the current round are calculated by using a loss function, so as to observe the optimization of the training result. And calculating to obtain the adjustment size required by the network parameters set in the current round by combining the marking line error, the crack error, the gradient of each layer of network parameters and an Adam optimization method through back propagation, and taking the adjusted network parameters as the network parameters set in next round of training.
As an optional embodiment, after obtaining the marking model and the crack model, the marking model and the crack model need to be tested by using a test actual pavement image, the test actual pavement image is an actual pavement image without marking labels and crack labels, and the test of the marking model and the test of the crack model are separately performed, in this embodiment, the step of testing the marking model specifically includes:
the gray scale image of the preprocessed test actual pavement image is positively input into a marking model, and the marking model carries out feature extraction on the gray scale image of the preprocessed test actual pavement image;
the marking model carries out forward propagation to obtain a test marking prediction result of each test actual pavement image, namely a test marking prediction probability matrix;
and evaluating the recognition effect of the reticle model under the current parameters by using an image recognition evaluation method, and optimizing the parameters to obtain the reticle model with the best recognition effect.
In this embodiment, the step of testing the fracture model specifically includes:
the gray-scale image of the preprocessed test actual pavement image is positively input into a crack model, and the crack model performs feature extraction on the gray-scale image of the preprocessed test actual pavement image;
the crack model is propagated forwards to obtain a test crack prediction result of each test actual pavement image, namely a test crack prediction probability matrix;
and evaluating the recognition effect of the crack model under the current parameters by using an image recognition evaluation method, and optimizing the parameters to obtain the crack model with the best recognition effect.
The indexes of the evaluation method of image recognition used in the present embodiment include: obtaining a marking model with three optimal sample similarity, accuracy and recall rate; and obtaining the optimal crack model of the sample similarity, the accuracy and the recall rate.
S103, inputting the actual road surface image into the marking model to obtain a marking identification result.
In this step, the specific steps of marking identification include:
the gray scale image of the actual road surface image after pretreatment is positively input into a marking model, and the marking model performs characteristic extraction on the gray scale image of the actual road surface image after pretreatment;
and carrying out forward propagation on the marking model to obtain a marking identification probability matrix of the actual pavement image, and discretizing the marking identification probability matrix to obtain a marking identification result.
And S104, inputting the actual road surface image into the crack model to obtain a crack identification result.
In this step, the crack identification specifically includes:
the gray-scale image of the actual road surface image after pretreatment is positively input into a crack model, and the crack model performs characteristic extraction on the gray-scale image of the actual road surface image after pretreatment;
and carrying out forward propagation on the crack model to obtain a crack identification probability matrix of the actual pavement image, and discretizing the crack identification probability matrix to obtain a crack identification result.
And S105, subtracting the marking line recognition result from the crack recognition result to obtain a final recognition result of the actual pavement crack.
In the step, in the practical engineering application, only the forward propagation process of the marking model and the crack model is used, a threshold value is set to be 0.5 by referring to fig. 3, the marking recognition result and the crack recognition result obtained in the step are used for subtracting the marking recognition result from the crack recognition result to obtain a final recognition result of the actual pavement crack, whether the crack exists in the actual pavement image is judged by comparing the size relationship between the final recognition result and the set threshold value, and if the value of the final recognition result is greater than the set threshold value, the crack exists in the actual pavement image; and if the value of the final identification result is smaller than the set threshold value, no crack exists in the actual road surface image.
Referring to fig. 4 and 5, the graphs are actual recognition results in actual engineering application, the crack model recognizes cracks in an actual road surface image, the marking line model recognizes marking lines in the actual road surface image, and since the obtained final recognition result is a probability matrix of a value range (0,1) obtained after discretization, if the value of the final recognition result is greater than a set threshold value of 0.5, it is proved that the road surface corresponding to the actual road surface image contains cracks; and if the value of the final recognition result is less than the set threshold value of 0.5, the fact that the road surface corresponding to the actual road surface image has no cracks is proved.
In this embodiment, in the process of performing migration learning between terminals, the method for identifying a pavement crack provided by the present invention needs to build an environmental support of the preprocessing device and the deep learning device on the new terminal, and when performing migration learning in other data sets, only the model parts of the preprocessing device and the convolutional neural network need to be migrated, and the original image reading part of the training and testing link is written by itself.
As can be seen from the above description, in the multi-reticle pavement crack recognition method, the multi-reticle pavement crack recognition device, the multi-reticle pavement crack recognition equipment and the storage medium provided in one or more embodiments of the present disclosure, because the used convolutional neural network contains a large number of parameters, the recognition method has a strong abstract fitting capability, and can effectively improve the recognition accuracy, the average similarity of the reticle recognition performed on the pavement image by the data image processing technology is only about 60%, and the average similarity of the method provided in one or more embodiments of the present disclosure reaches more than 90%. The invention is suitable for the deep learning mode to carry out the image recognition, and actually leads a large number of calculation processes, so a large number of calculation amounts can be reduced when processing unknown images, specifically, the unknown images can obtain automatically marked line areas only by carrying out one previous item in a network, the calculation amount is half of the training process, after test verification, the original road surface images are subjected to line recognition by using the pattern recognition method based on the gray level image in the prior art, the recognition speed is only 1-6 per second, but the method provided by the invention has the recognition speed of 50-200 per second and extremely high recognition speed. Compared with the single model for identifying the cracks in the prior art, the method has the advantages that the crack model and the marking model are superposed, and the deep learning method is used for effectively identifying the marks and the cracks, so that the influence of the marks is eliminated, the influence of the environment is small, and the crack identification effect is improved. As the actual cracks have diversity, as an optional embodiment, training superposition of the models can be performed by training a plurality of different crack models, so that the recognition effect of the cracks is integrally improved.
Based on the same inventive concept, one or more embodiments of the invention also provide a multi-marking pavement crack recognition device, which comprises a first acquisition module, a second acquisition module, a first recognition module, a second recognition module and a result output module.
Referring to fig. 6, the present apparatus includes:
a first acquisition module configured to acquire an actual road surface image;
a second obtaining module configured to obtain a reticle model and a fracture model; the marking model is obtained through training based on a first training pavement image set, and the first training pavement image set is associated with a first label representing a marking; the crack model is obtained through training based on a second training road surface image set, and a second label representing a crack is associated with the second training actual road surface image set;
the first recognition module is configured to input the actual road surface image into the marking model to obtain a marking recognition result;
the second identification module is configured to input the actual road surface image into the crack model to obtain a crack identification result;
and the result output module is configured to subtract the marking line identification result from the crack identification result to obtain a final identification result of the actual pavement crack.
For convenience of description, the above devices are described as being divided into various modules by functions, and are described separately. Of course, the functionality of the modules may be implemented in the same one or more software and/or hardware implementations in implementing one or more embodiments of the present description.
The apparatus of the foregoing embodiment is used to implement the corresponding method in the foregoing embodiment, and has the beneficial effects of the corresponding method embodiment, which are not described herein again.
Based on the same inventive concept, one or more embodiments of the present invention further provide an electronic device, which includes a memory, a processor, and a computer program stored on the memory and executable on the processor, and the processor implements the above method when executing the program.
Fig. 7 is a schematic diagram illustrating a more specific hardware structure of an electronic device according to this embodiment, where the electronic device may include: a processor 701, a memory 702, an input/output interface 703, a communication interface 704, and a bus 705. Wherein the processor 701, the memory 702, the input/output interface 703 and the communication interface 704 are communicatively connected to each other within the device via a bus 705.
The processor 701 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an Application Specific Integrated Circuit (ASIC), or one or more Integrated circuits, and is configured to execute related programs to implement the technical solutions provided in the embodiments of the present specification.
The Memory 702 may be implemented in the form of a ROM (Read Only Memory), a RAM (Random access Memory), a static storage device, a dynamic storage device, or the like. The memory 702 may store an operating system and other application programs, and when the technical solution provided by the embodiments of the present specification is implemented by software or firmware, the relevant program codes are stored in the memory 702 and called to be executed by the processor 701.
The input/output interface 703 is used for connecting an input/output module to realize information input and output. The input/output/modules may be configured in the device as components or may be external to the device to provide corresponding functionality. The input devices may include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output devices may include a display, a speaker, a vibrator, an indicator light, etc.
The communication interface 704 is used for connecting a communication module to realize communication interaction of the device and other devices. The communication module can realize communication in a wired mode (such as USB, network cable and the like) and also can realize communication in a wireless mode (such as mobile network, WIFI, Bluetooth and the like).
It should be noted that although the above-mentioned device only shows the processor 701, the memory 702, the input/output interface 703, the communication interface 704 and the bus 705, in a specific implementation, the device may also include other components necessary for normal operation. In addition, those skilled in the art will appreciate that the above-described apparatus may also include only those components necessary to implement the embodiments of the present description, and not necessarily all of the components shown in the figures.
Computer-readable media of the present embodiments, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), Read Only Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), Digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information that can be accessed by a computing device.
Those of ordinary skill in the art will understand that: the discussion of any embodiment above is meant to be exemplary only, and is not intended to intimate that the scope of the disclosure, including the claims, is limited to these examples; within the spirit of the present disclosure, features from the above embodiments or from different embodiments may also be combined, steps may be implemented in any order, and there are many other variations of different aspects of one or more embodiments of the present description as described above, which are not provided in detail for the sake of brevity.
In addition, well-known power/ground connections to Integrated Circuit (IC) chips and other components may or may not be shown in the provided figures, for simplicity of illustration and discussion, and so as not to obscure one or more embodiments of the disclosure. Furthermore, devices may be shown in block diagram form in order to avoid obscuring the understanding of one or more embodiments of the present description, and this also takes into account the fact that specifics with respect to implementation of such block diagram devices are highly dependent upon the platform within which the one or more embodiments of the present description are to be implemented (i.e., specifics should be well within purview of one skilled in the art). Where specific details (e.g., circuits) are set forth in order to describe example embodiments of the disclosure, it should be apparent to one skilled in the art that one or more embodiments of the disclosure can be practiced without, or with variation of, these specific details. Accordingly, the description is to be regarded as illustrative instead of restrictive.
While the present disclosure has been described in conjunction with specific embodiments thereof, many alternatives, modifications, and variations of these embodiments will be apparent to those of ordinary skill in the art in light of the foregoing description. For example, other memory architectures (e.g., dynamic ram (dram)) may use the discussed embodiments.
It is intended that the one or more embodiments of the present specification embrace all such alternatives, modifications and variations as fall within the broad scope of the appended claims. Therefore, any omissions, modifications, substitutions, improvements, and the like that may be made without departing from the spirit and principles of one or more embodiments of the present disclosure are intended to be included within the scope of the present disclosure.
Claims (10)
1. A multi-marking pavement crack identification method is characterized by comprising the following steps:
acquiring an actual road surface image;
obtaining a marking model and a crack model; the marking model is obtained through training based on a first training pavement image set, and the first training pavement image set is associated with a first label representing a marking; the crack model is obtained through training based on a second training road surface image set, and the second training road surface image set is associated with a second label representing a crack;
inputting the actual road surface image into the marking line model to obtain a marking line identification result;
inputting the actual pavement image into the crack model to obtain a crack identification result;
and subtracting the marking line identification result from the crack identification result to obtain the final identification result of the actual pavement crack.
2. The method of claim 1, further comprising: preprocessing the actual road surface image;
the preprocessing operation comprises: and carrying out image filling and image adjusting operation on the actual road surface image.
3. The method of claim 1, wherein the reticle model is trained based on a first training road surface image set associated with a first label representing a reticle; the fracture model is obtained by training based on a second training road surface image set, the second training road surface image set is associated with a second label representing a fracture, the training is carried out for a plurality of rounds, and each round of the training comprises:
setting network parameters, and inputting the first training road surface image set into the marking model in the forward direction and inputting the second training road surface image set into the crack model in the forward direction;
the marking model performs characteristic extraction on the first training pavement image set to obtain a marking prediction result, and the crack model performs characteristic extraction on the second training pavement image set to obtain a crack prediction result;
calculating a reticle error between the reticle prediction result and a real value and a fracture error between the fracture prediction result and the real value by using a loss function;
and adjusting the network parameters by combining the marking line errors, the crack errors, the gradients of the network parameters and an optimization method, and taking the adjusted network parameters as the network parameters of the next round of training.
4. The method of claim 3, wherein the feature extraction comprises:
down-sampling the first training road surface image set and the second training road surface image set through 2 x 2 maximum pooling;
and performing the feature extraction on the first training road surface image set and the second training road surface image set by using a 5 × 5 convolution kernel and a 3 × 3 convolution kernel, and a ReLU function and a sigmoid function as activation functions.
5. The method of claim 3, wherein the loss function is defined by the formula
Wherein dice represents a sample similarity, X represents a probability matrix formed by the actual road surface image through network feedforward, and Y represents a mark matrix formed by the first or second training road surface image sets, | X |1L representing the probability matrix X1Norm, | Y |1L representing the mark matrix Y1Norm, X X Y represents the Hadamard product of the probability matrix X and the label matrix Y, and e is the first smoothing factor.
6. The method of claim 3, wherein the optimization method is defined by the formula
mt＝β1·mt-1+(1-β1)·gt
vt＝β2·vt-1+(1-β2)·gt 2
Wherein L is the loss function, gtFor the gradient of the loss function L in respect of the parameter theta at the t-th round, thetatThe value of the parameter theta at the t-th round, mtFor recording first moments, vtFor recording the second moment of the wave or waves,
is mtIs determined by the estimated value of (c),
is v istEstimate of (a), β1Is the first over-parameter β2Is the second hyperparameter, lr is the learning rate, e0Is the second smoothing factor.
7. The method of claim 1, wherein subtracting the marking recognition result from the crack recognition result to obtain a final actual pavement crack recognition result comprises:
the marking model identifies the actual pavement image to obtain a marking identification probability matrix, and the marking identification probability matrix is discretized to obtain a marking identification result;
the crack model identifies the actual pavement image to obtain a crack identification probability matrix, and discretizing the crack identification probability matrix to obtain a crack identification result;
and subtracting the marking line identification result from the crack identification result to obtain the final identification result of the actual pavement crack.
8. A multi-marking pavement crack recognition device, comprising:
a first acquisition module configured to acquire an actual road surface image;
a second obtaining module configured to obtain a reticle model and a fracture model; the marking model is obtained through training based on a first training pavement image set, and the first training pavement image set is associated with a first label representing a marking; the crack model is obtained through training based on a second training road surface image set, and a second label representing a crack is associated with the second training road surface image set;
the first recognition module is configured to input the actual road surface image into the marking model to obtain a marking recognition result;
the second identification module is configured to input the actual road surface image into the crack model to obtain a crack identification result;
and the result output module is configured to subtract the marking line identification result from the crack identification result to obtain a final identification result of the actual pavement crack.
9. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the method according to any of claims 1 to 7 when executing the program.
10. A non-transitory computer readable storage medium storing computer instructions for causing a computer to perform the method of any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010027072.0A CN111242017B (en) | 2020-01-10 | 2020-01-10 | Multi-marking pavement crack identification method, device, equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010027072.0A CN111242017B (en) | 2020-01-10 | 2020-01-10 | Multi-marking pavement crack identification method, device, equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111242017A true CN111242017A (en) | 2020-06-05 |
| CN111242017B CN111242017B (en) | 2024-01-23 |
Family
ID=70865747
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010027072.0A Active CN111242017B (en) | 2020-01-10 | 2020-01-10 | Multi-marking pavement crack identification method, device, equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111242017B (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114418937A (en) * | 2021-12-06 | 2022-04-29 | 北京邮电大学 | Pavement crack detection method and related equipment |
| CN116664582A (en) * | 2023-08-02 | 2023-08-29 | 四川公路桥梁建设集团有限公司 | Road surface detection method and device based on neural vision network |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR101926561B1 (en) * | 2018-03-13 | 2018-12-07 | 연세대학교 산학협력단 | Road crack detection apparatus of patch unit and method thereof, and computer program for executing the same |
| CN109816626A (en) * | 2018-12-13 | 2019-05-28 | 深圳高速工程检测有限公司 | Road surface crack detection method, device, computer equipment and storage medium |
| CN109949290A (en) * | 2019-03-18 | 2019-06-28 | 北京邮电大学 | Pavement crack detection method, device, equipment and storage medium |
| CN110120041A (en) * | 2019-05-14 | 2019-08-13 | 郑州大学 | Pavement crack image detecting method |
| CN110349119A (en) * | 2019-05-27 | 2019-10-18 | 北京邮电大学 | Pavement disease detection method and device based on edge detection neural network |
| CN110502972A (en) * | 2019-07-05 | 2019-11-26 | 广东工业大学 | A kind of pavement crack segmentation and recognition methods based on deep learning |
-
2020
- 2020-01-10 CN CN202010027072.0A patent/CN111242017B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR101926561B1 (en) * | 2018-03-13 | 2018-12-07 | 연세대학교 산학협력단 | Road crack detection apparatus of patch unit and method thereof, and computer program for executing the same |
| CN109816626A (en) * | 2018-12-13 | 2019-05-28 | 深圳高速工程检测有限公司 | Road surface crack detection method, device, computer equipment and storage medium |
| CN109949290A (en) * | 2019-03-18 | 2019-06-28 | 北京邮电大学 | Pavement crack detection method, device, equipment and storage medium |
| CN110120041A (en) * | 2019-05-14 | 2019-08-13 | 郑州大学 | Pavement crack image detecting method |
| CN110349119A (en) * | 2019-05-27 | 2019-10-18 | 北京邮电大学 | Pavement disease detection method and device based on edge detection neural network |
| CN110502972A (en) * | 2019-07-05 | 2019-11-26 | 广东工业大学 | A kind of pavement crack segmentation and recognition methods based on deep learning |
Non-Patent Citations (1)
| Title |
|---|
| 徐志刚: "基于多特征融合的路面破损图像自动识别技术研究" * |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114418937A (en) * | 2021-12-06 | 2022-04-29 | 北京邮电大学 | Pavement crack detection method and related equipment |
| CN114418937B (en) * | 2021-12-06 | 2022-10-14 | 北京邮电大学 | Pavement crack detection method and related equipment |
| CN116664582A (en) * | 2023-08-02 | 2023-08-29 | 四川公路桥梁建设集团有限公司 | Road surface detection method and device based on neural vision network |
| CN116664582B (en) * | 2023-08-02 | 2023-10-27 | 四川公路桥梁建设集团有限公司 | Road surface detection method and device based on neural vision network |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111242017B (en) | 2024-01-23 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108230390B (en) | Training method, key point detection method, device, storage medium and electronic equipment | |
| KR102166458B1 (en) | Defect inspection method and apparatus using image segmentation based on artificial neural network | |
| KR20210006971A (en) | System and method for geolocation prediction | |
| CN105938559A (en) | Digital image processing using convolutional neural networks | |
| CN109977191B (en) | Problem map detection method, device, electronic equipment and medium | |
| CN106920245B (en) | Boundary detection method and device | |
| CN114359283B (en) | Defect detection method based on Transformer and electronic equipment | |
| CN111127468A (en) | Road crack detection method and device | |
| CN114299034A (en) | Defect detection model training method, defect detection method and device | |
| CN108230292A (en) | The training method of object detecting method and neural network, device and electronic equipment | |
| KR20190086141A (en) | Simulation Data Preprocessing Technique for Development of Damage Detecting Method for Bridges Based on Convolutional Neural Network | |
| CN111429482A (en) | Target tracking method and device, computer equipment and storage medium | |
| CN111242017B (en) | Multi-marking pavement crack identification method, device, equipment and storage medium | |
| CN115222061A (en) | Federal learning method based on continuous learning and related equipment | |
| CN107992873A (en) | Object detection method and device, storage medium, electronic equipment | |
| CN116433661B (en) | Method, device, equipment and medium for detecting semiconductor wafer by multitasking | |
| CN112862706A (en) | Pavement crack image preprocessing method and device, electronic equipment and storage medium | |
| CN115861736B (en) | High-speed corrugated case printing defect detection method, system and storage medium based on knowledge distillation | |
| CN114418937B (en) | Pavement crack detection method and related equipment | |
| CN114882115B (en) | Vehicle pose prediction method and device, electronic equipment and storage medium | |
| JP2019086473A (en) | Learning program, detection program, learning method, detection method, learning device, and detection device | |
| CN114529815A (en) | Deep learning-based traffic detection method, device, medium and terminal | |
| CN113989632A (en) | Bridge detection method and device for remote sensing image, electronic equipment and storage medium | |
| CN112084889A (en) | Image behavior recognition method and device, computing equipment and storage medium | |
| JP2021108031A (en) | Model generation device, vehicle simulation system, model generation method, vehicle simulation method and computer program |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |