CN115619777A - Method and system for detecting ice and snow state of road surface and computer readable storage medium - Google Patents

Method and system for detecting ice and snow state of road surface and computer readable storage medium Download PDF

Info

Publication number
CN115619777A
CN115619777A CN202211545183.6A CN202211545183A CN115619777A CN 115619777 A CN115619777 A CN 115619777A CN 202211545183 A CN202211545183 A CN 202211545183A CN 115619777 A CN115619777 A CN 115619777A
Authority
CN
China
Prior art keywords
ice
snow
state
snow state
road surface
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202211545183.6A
Other languages
Chinese (zh)
Other versions
CN115619777B (en
Inventor
邢志伟
朱书杰
李彪
阚犇
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Civil Aviation University of China
Original Assignee
Civil Aviation University of China
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Civil Aviation University of China filed Critical Civil Aviation University of China
Priority to CN202211545183.6A priority Critical patent/CN115619777B/en
Publication of CN115619777A publication Critical patent/CN115619777A/en
Application granted granted Critical
Publication of CN115619777B publication Critical patent/CN115619777B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/0002—Inspection of images, e.g. flaw detection
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/764—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
    • G06V10/774—Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
    • G06V10/80—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level
    • G06V10/806—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level of extracted features
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/60—Type of objects
    • G06V20/69—Microscopic objects, e.g. biological cells or cellular parts
    • G06V20/698—Matching; Classification
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02A—TECHNOLOGIES FOR ADAPTATION TO CLIMATE CHANGE
    • Y02A90/00—Technologies having an indirect contribution to adaptation to climate change
    • Y02A90/10—Information and communication technologies [ICT] supporting adaptation to climate change, e.g. for weather forecasting or climate simulation

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Evolutionary Computation (AREA)
  • Multimedia (AREA)
  • Health & Medical Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Computing Systems (AREA)
  • Databases & Information Systems (AREA)
  • Artificial Intelligence (AREA)
  • Medical Informatics (AREA)
  • Software Systems (AREA)
  • Quality & Reliability (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Biomedical Technology (AREA)
  • Molecular Biology (AREA)
  • Image Analysis (AREA)

Abstract

The invention provides a method, a system and a computer readable storage medium for detecting the ice and snow state on a road surface, wherein the detection method comprises the following steps: acquiring microscopic image information of the ice and snow state of the road surface; generating an ice and snow state data set according to the microscopic image information; training iteration according to the ice and snow state data set based on a convolutional neural network to generate an ice and snow state perception model; and sensing a real-time microscopic image of the ice and snow state of the road surface through the ice and snow state sensing model, and outputting an ice and snow state sensing system. The ice accumulation image characteristics can be independently learned through the convolutional neural network for image recognition, the complex algorithm of machine learning is omitted, and the problem that small differences among ice types are difficult to distinguish can be better solved, so that the ice and snow state perception model has strong recognition capability of ice and snow state categories.

Description

Method and system for detecting ice and snow state of road surface and computer readable storage medium
Technical Field
The invention relates to the technical field of structure detection, in particular to a method and a system for detecting the ice and snow state of a road surface and a computer readable storage medium.
Background
In the related art, with the increasing annual throughput of airports, the air transportation industry is driven to develop continuously. However, in winter, the problem of icing on airport roads is often caused by ice and snow weather, and the duration is relatively long, which undoubtedly brings challenges to the flight operation of the airport, and the ice accumulation on the airport roads causes a series of chain reaction problems, such as the problems of leaving the airport for landing, delaying flights, staying in the airport for passengers, rushing out of runways, closing the airport and the like. After the airport runway is frozen, a thin ice layer is formed, then, accumulated ice is formed on the runway surface along with the expansion of the area of the ice layer, and the friction coefficient of the runway is sharply reduced. When ice is formed on the road surface and the friction coefficient of the road surface is lower than 0.3, the road surface is very smooth, and according to the stipulation of civil aviation bureaus, an airport must be closed and related deicing operation is carried out. The ice accumulated on the airport roads not only can influence the operation of airports and airlines, but also can bring loss of lives and properties to passengers when going out. Therefore, the method is particularly important for the perception research of the ice and snow condition on the airport pavement, can provide valuable reference for field staff, makes a proper deicing scheme according to the ice condition on the pavement, selects the best guarantee measure and improves the service guarantee capability of the airport.
At present, a common ice and snow condition detection method comprises a sensor detection method, and the ice accumulation detection sensor is widely applied mainly in an optical fiber type, an ultrasonic type, a microwave type, an infrared reflection type and the like: (1) optical fiber formula: determining the ice type according to the difference of the reflected and scattered light intensity of the light in different ice and snow states; (2) ultrasonic formula: when an ice layer is formed on the surface, the piezoelectric device receives ultrasonic waves reflected by an ice and air interface, and the ice accumulation condition is judged according to the time delay of the transmission and the reception of the ultrasonic waves in the ice; (3) microwave formula: the waveguide tube is arranged on the surface, and the phase constant of ice can be changed when the ice is gathered on the surface of the waveguide tube insulating layer, so that the resonance frequency is reduced, and the ice accumulation condition is obtained according to the offset of the resonance frequency; (4) infrared reflection formula: based on the fact that all solids can reflect infrared energy when the temperature is higher than absolute zero, the infrared energy reflects the surface temperature of the detection piece, the temperature difference value is very close to that of the detection piece when the detection piece is frozen, and ice accumulation information is obtained through detection of the temperature difference value.
Another common ice accumulation state sensing method is a computer vision method, which uses an image processing technology to perform preliminary processing on an ice and snow image of a road surface, and extracts characteristic information including color, texture, contour and the like from the image by combining a machine learning algorithm so as to detect and identify various ice and snow states of the road surface. The method uses the image acquisition device to acquire the ice and snow images of the pavement without contacting the pavement. However, the common computer vision method has high requirements on image quality, the ice and snow states of the road surface are variable, the image characteristics are complex, and the problem of large deviation between the sensing result and the actual result exists.
Through the above analysis, the problems and defects of the prior art are as follows:
(1) The ice accumulation detection sensor has a limited range for detecting ice accumulation, is difficult to comprehensively detect the ice accumulation area on the road, has low accuracy due to the influence of environmental factors, is high in cost for damage, maintenance and replacement and is difficult to widely use in the scene of sensing the ice and snow state of the area of the airport road;
(2) The states of ice and snow on the road surface are various, the existing computer vision method has insufficient capability of extracting the features of the ice and snow image on the airport road surface, and the accuracy of the ice and snow state perception model is limited.
The difficulty in solving the above problems and defects is: the information collected by the ice accumulation sensor is used for detecting that the icing condition of the airport pavement is greatly influenced by environmental factors, and the sensor is easy to damage, so that the detection result is wrong; the conventional image identification method is used for detecting the icing condition of the airport pavement, and has poor real-time performance and low accuracy.
Disclosure of Invention
The present invention is directed to solving or improving at least one of the above technical problems.
Therefore, a first object of the present invention is to provide a method for detecting a state of ice and snow on a road surface.
The second purpose of the invention is to provide a detection system for the ice and snow state on the road surface.
A third object of the present invention is to provide a computer-readable storage medium.
To achieve the first object of the present invention, an embodiment of the present invention provides a method for detecting a snow and ice state on a road surface, including: acquiring microscopic image information of the ice and snow state of the road surface; generating an ice and snow state data set according to the microscopic image information; training iteration according to the ice and snow state data set to generate an ice and snow state perception model based on a convolutional neural network; and sensing a real-time microscopic image of the ice and snow state of the road surface through the ice and snow state sensing model, and outputting an ice and snow state sensing system.
Optionally, the method for detecting the snow and ice state on the road surface further includes: establishing a YOLOv4 network model; dividing the ice and snow state data set into a training set, a testing set and a verification set according to a set proportion; and inputting the training set into the YOLOv4 network model for training to obtain the ice and snow state perception model.
Optionally, the establishing a YOLOv4 network model specifically includes: adding a global attention module in a main feature extraction network based on an original YOLOv4 target detection algorithm; introducing a self-adaptive spatial feature fusion structure into an enhanced feature extraction network; the original CIoU loss function is replaced with the SIoU loss function.
Optionally, the set ratio is: 8:1:1.
Optionally, outputting an ice and snow state sensing system through the ice and snow state sensing model specifically includes: and inputting the test set into the ice and snow state perception model so as to output the ice and snow state perception system.
Optionally, the generating an ice and snow state data set according to the microscopic image information specifically includes: classifying the microscopic image information according to an ice and snow state classification table by combining a crystal state and an image gray value; and marking the microscopic image by adopting an image marking tool to generate a label file with a standard text format type to form the ice and snow state data set.
Optionally, the classifying the microscopic image information according to the ice and snow state classification table by combining the crystal state and the image gray level value specifically includes: the status microscopic images were classified into 6 types: snow, snow melt, slush, frozen ice, wet ice, and water.
Optionally, the acquiring microscopic image information of the ice and snow state of the road surface specifically includes: and collecting the ice and snow state from the right above the road surface by adopting an electron microscope.
To achieve the second object of the present invention, an embodiment of the present invention provides a system for detecting a snow-ice state on a road surface, including: the ice and snow state acquisition module is suitable for acquiring microscopic image information of the ice and snow state of the road surface; the data processing module is in communication connection with the ice and snow state acquisition module and is suitable for generating an ice and snow state data set according to the microscopic image information; and the ice and snow state sensing module is in communication connection with the data processing module and is suitable for training and iterating the ice and snow state data set to generate an ice and snow state sensing model and output an ice and snow state sensing system.
To achieve the third object of the present invention, an embodiment of the present invention provides a computer-readable storage medium including: the storage medium stores a computer program that, when executed, implements any one of the methods for detecting an ice and snow state on a road surface.
Compared with the prior art, the advantages and the inventive technical effects of the invention are specifically described as follows:
1. the ice accumulation image characteristics can be automatically learned through the convolutional neural network for image recognition, the complex algorithm of machine learning is omitted, and the problem that small differences among ice types are difficult to distinguish can be better solved as long as the ice type image data samples are rich, so that the ice and snow state perception model has strong recognition capability of ice and snow state categories. The collected microcosmic images of the real-time ice and snow state of the road surface are input into the ice and snow state perception model, the real-time ice and snow state of the road surface can be perceived, and the ice and snow state perception system can be output, so that the ice accumulation condition of the road surface can be comprehensively and accurately known, and reliable data basis is provided for the subsequent ice and snow removal operation of the road surface.
2. According to the method, a track ice and snow state perception model is trained by improving YOLOv4, and aiming at the problems that the ice and snow crystal state presents different sizes and shapes, is interfered by impurities such as dust and coherent speckle noise and is difficult to perceive the state, a global attention module GC Block is added in a main feature extraction network structure to increase global context modeling, so that richer shallow and deep features are obtained.
3. An adaptive spatial feature fusion structure ASFF is added at the tail of a PANet of a reinforced feature extraction network of a YOLOv4 model, so that the defect of inconsistency of different scales in the PANet network is suppressed, the multi-scale features are fused in a self-adaptive manner, and the optimal fusion effect is achieved;
4. and in consideration of the vector angle between required regressions, redefining the penalty index, and replacing the original CIoU loss function with the SIoU loss function to better reflect the position relation between the prediction frame and the real frame and further improve the convergence speed of the perception model and the accuracy of reasoning.
5. By improving an image processing algorithm, the ice and snow state of the pavement is accurately sensed, a pavement ice and snow state sensing model with high accuracy is obtained along with accumulation of historical data, the ice accumulation condition of the airport pavement is accurately identified, an accurate and valuable reference is provided for ice and snow removal operation of the pavement, and the operation guarantee capability of the pavement is improved.
Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention.
Drawings
FIG. 1 is a flow chart of a method for detecting the snow and ice on a road surface according to an embodiment of the present invention;
FIG. 2 is a flow chart of a method for detecting the snow and ice on the road surface according to an embodiment of the present invention;
FIG. 3 is a flowchart of a method for detecting the snow and ice on the road surface according to the embodiment of FIG. 2, wherein a Yolov4 network model is established;
fig. 4 is a flowchart of the method of fig. 1 for detecting the snow and ice on the road surface according to the embodiment of the present invention for generating a data set of the snow and ice state from the microscopic image information;
FIG. 5 is a plan view of six snow and ice states of a method for detecting snow and ice on a road surface according to an embodiment of the present invention;
FIG. 6 is a block diagram of a global attention module of a method for detecting ice and snow on a road surface according to an embodiment of the present invention;
FIG. 7 is a block diagram of an adaptive spatial feature fusion module of a method for detecting the ice and snow status of a road surface according to an embodiment of the present invention;
FIG. 8a is a schematic view of an angle loss of a method for detecting an ice and snow state on a road surface according to an embodiment of the present invention;
FIG. 8b is a schematic diagram illustrating distance loss in a method for detecting an ice/snow status on a road surface according to an embodiment of the present invention;
fig. 8c is a IoU loss schematic diagram of a method for detecting an ice and snow state on a road surface according to an embodiment of the present invention.
Fig. 9 is a diagram of a sensing result based on an improved yollov 4 ice and snow state of the method for detecting the ice and snow state of the road surface according to the embodiment of the present invention;
fig. 10 is a block diagram of a system for detecting snow and ice on a road surface according to an embodiment of the present invention.
Fig. 11 is a flowchart of a method for detecting the ice and snow state of the aircraft road surface according to the method for detecting the ice and snow state of the road surface of the embodiment of the present invention.
Reference numerals
1-a system for detecting the ice and snow state of the road surface; 11-an ice and snow state acquisition module; 12-a data processing module; 13-ice and snow state sensing module.
Detailed Description
In order that the above objects, features and advantages of the present invention can be more clearly understood, a more particular description of the invention will be rendered by reference to the appended drawings. It should be noted that the embodiments and features of the embodiments of the present application may be combined with each other without conflict.
In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present invention, however, the present invention may be practiced in other ways than those specifically described herein, and therefore the scope of the present invention is not limited by the specific embodiments disclosed below.
The technical solutions of some embodiments of the present invention are described below with reference to fig. 1 to 10.
As shown in fig. 1, there is provided a method for detecting a snow-ice state on a road surface, comprising:
step S101: acquiring microscopic image information of the ice and snow state of the road surface;
step S103: generating an ice and snow state data set according to the microscopic image information;
step S105: training iteration according to the ice and snow state data set based on a convolutional neural network to generate an ice and snow state perception model;
step S107: and sensing a real-time microscopic image of the ice and snow state of the road surface through the ice and snow state sensing model, and outputting an ice and snow state sensing system.
The road surface can comprise a large-scale road surface such as an airport and the like, the characteristics of the ice-accumulated image can be automatically learned through a convolutional neural network for image recognition, a complex algorithm of machine learning is omitted, and the problem that small differences among ice types are difficult to distinguish can be better solved as long as the data samples of the ice type images are rich, so that the ice and snow state perception model has strong recognition capability of ice and snow state categories. The collected microcosmic images of the real-time ice and snow state of the road surface are input into the ice and snow state perception model, the real-time ice and snow state of the road surface can be perceived, and the ice and snow state perception system can be output, so that the ice accumulation condition of the road surface can be comprehensively and accurately known, and reliable data basis is provided for the subsequent ice and snow removal operation of the road surface.
Further, acquiring microscopic image information of the ice and snow state of the road surface specifically comprises: and collecting the ice and snow state from the right above the road surface by adopting an electron microscope. The ice and snow state images are collected from the position right above the road surface, so that the integrity and intuition of the ice and snow state information acquisition can be ensured, the ice and snow state image characteristics can be extracted by the sensing model, and the classification effect of the sensing model on the ice and snow state of the road surface is improved.
Further, the method for detecting the ice and snow state of the road surface further comprises the following steps:
step S201: establishing a YOLOv4 network model;
step S203: dividing the ice and snow state data set into a training set, a testing set and a verification set according to a set proportion;
step S205: and inputting the training set into a YOLOv4 network model for training to obtain an ice and snow state perception model.
The ice and snow state perception model of the road surface is trained through the improved YOLOv4 algorithm, the training set and the verification set are put into the perception model based on the improved YOLOv4 to be trained for 200 epochs to obtain the optimal perception model, then the real-time ice and snow state microscopic image of the road surface is collected by an electron microscope and is input into the ice and snow state perception model, and the real-time ice and snow state of the road surface can be perceived.
Further, as shown in fig. 5, classifying the microscopic image information according to the ice and snow state classification table and combining the crystal state and the image gray scale value specifically includes: the status microscopic images were classified into 6 types: snow (a), snow melt (b), slush (c), frozen ice (d), wet ice (e) and water (f).
The type name of model training is modified into snow, snow melt, slush, frozen ice, wet ice and water, the number of categories is modified into 6, the training iteration time epoch is set to be 200, and the one-time training sample number batch-size is set to be 16.
Further, acquiring microscopic image information of the ice and snow state of the road surface specifically comprises: and collecting the ice and snow state from the right above the road surface by adopting an electron microscope.
Further, establishing a YOLOv4 network model specifically includes:
step S301: adding a global attention module in a main feature extraction network based on an original YOLOv4 target detection algorithm;
step S303: introducing a self-adaptive spatial feature fusion structure into an enhanced feature extraction network;
step S305: the original CIoU loss function is replaced with the SIoU loss function.
Since the ice and snow crystal states have the characteristics of different sizes and shapes, and are interfered by impurities such as dust and coherent speckle noise, the state sensing is not easy to perform, for the original YOLOv4 target detection algorithm, a global attention module GC Block is added into a trunk feature extraction network CSPDarknet53, and the structure is shown in fig. 6. By introducing a GC Block module into a CSP structure of a CSP search net53 main feature extraction network, global context information can be extracted to obtain richer shallow and deep features. Therefore, for the original YOLOv4 target detection algorithm, a global attention module GC Block is added to the main feature extraction network CSPDarknet53, and global modeling is added.
The implementation steps of the GC Block global attention module comprise:
step 1: using a 1 × 1 convolution
Figure 352490DEST_PATH_IMAGE001
And the softmax function acquires attention weight and then performs attention pooling operation, thereby realizing the context modeling
Figure 738472DEST_PATH_IMAGE002
，
Figure 721471DEST_PATH_IMAGE003
For global attention pooling weights, the correlation of the positions is obtained by embedding a Gaussian (Embedded Gaussian),
Figure 585522DEST_PATH_IMAGE004
the calculation formula is as follows:
Figure 868736DEST_PATH_IMAGE005
（1）
and 2, step: using 1 × 1 convolution in sequence
Figure 160040DEST_PATH_IMAGE006
LayerNorm, reLU and 1X 1 convolution
Figure 895915DEST_PATH_IMAGE007
And the characteristic conversion operation is carried out, so that the aims of reducing the optimization difficulty and facilitating generalization are fulfilled. The partThe formula of the calculation is as follows:
Figure 829236DEST_PATH_IMAGE008
（2）
and step 3: feature fusion using addition operations, obtained by aggregating global context features onto the features of each location
Figure 966956DEST_PATH_IMAGE009
And substituting the results (1) and (2) to obtain a final output result:
Figure 429161DEST_PATH_IMAGE010
（3）
wherein
Figure 652332DEST_PATH_IMAGE011
For the purpose of inputting an example feature map,
Figure 858186DEST_PATH_IMAGE012
in order to output the characteristic diagram,
Figure 853342DEST_PATH_IMAGE013
in order to query the index location(s),
Figure 752028DEST_PATH_IMAGE014
enumerating locations for all possible; an example of an input is a picture,
Figure 462495DEST_PATH_IMAGE015
。
as shown in fig. 7, an adaptive spatial feature fusion structure ASFF is added at the tail of the PANet enhanced feature extraction network, and the disadvantage of inconsistency of different scales in the PANet network is suppressed, so that the multi-scale features are adaptively fused, and an optimal fusion effect can be achieved. Therefore, the self-adaptive spatial feature fusion structure ASFF is introduced into the reinforced feature extraction network PANet, and the multi-scale feature fusion effect is enhanced.
ASFIn the F structure, three enhanced feature layers, namely Level1, level2 and Level3, are obtained through a PANET enhanced feature extraction network, and the feature map of each layer is converted into the size of the other two layers of the three layers through an ASFF structure and then fusion prediction is carried out. Taking ASFF-3 as an example, feature layers of Level1 and Level2 are obtained through PANet, 1 × 1 convolution operation is carried out on the feature layers to compress the number of channels to be the same as that of Level3, then 4 times and 2 times of upsampling are respectively carried out on the feature layers to form feature maps with the same dimension as that of Level3, the feature maps are marked as Resize _ Level1 and Resize _ Level2, and then 1 × 1 convolution operation is carried out on Resize _ Level1, resize _ Level2 and Level3 to obtain weight parameters
Figure 206460DEST_PATH_IMAGE016
Finally, multiplying the weight parameters with Resize _ Level1, resize _ Level2 and Level3, and summing to obtain ASFF-3 with fused features, wherein the above process can be described by the following formula:
Figure 318772DEST_PATH_IMAGE017
（4）
wherein
Figure 122780DEST_PATH_IMAGE018
Representing a new feature map obtained by ASFF feature fusion;
Figure 320544DEST_PATH_IMAGE019
respectively represent layers 1, 2 and 3 to
Figure 133779DEST_PATH_IMAGE020
A feature vector of a layer feature layer;
Figure 100598DEST_PATH_IMAGE021
weight parameters of three different characteristic layers are respectively expressed by a softmax function
Figure 341086DEST_PATH_IMAGE022
。
And replacing the original CIoU loss function with the SIoU loss function, redefining penalty indexes by considering vector angles between required regressions, and further improving convergence speed and reasoning accuracy. Therefore, the SIoU loss function is used for replacing the original CIoU loss function, and the position relation of the predicted frame and the real frame can be better reflected. The sio loss function consists of four cost functions:
the Angle cost (Angle cost), as shown in FIG. 8a, is defined as follows:
Figure 760566DEST_PATH_IMAGE023
（5）
wherein
Figure 111913DEST_PATH_IMAGE024
The height difference between the center points of the real frame and the predicted frame,
Figure 933239DEST_PATH_IMAGE025
the distance between the center points of the real frame and the predicted frame,
Figure 344628DEST_PATH_IMAGE026
i.e. an angle
Figure 251405DEST_PATH_IMAGE027
。
Figure 140863DEST_PATH_IMAGE028
（6）
Figure 79345DEST_PATH_IMAGE029
（7）
Figure 661636DEST_PATH_IMAGE030
（8）
Figure 790129DEST_PATH_IMAGE031
Is the coordinates of the center of the real frame,
Figure 748857DEST_PATH_IMAGE032
to predict the frame center coordinates when
Figure 279196DEST_PATH_IMAGE027
At 0 or π/2, the angle loss is 0, if it is during training
Figure 32388DEST_PATH_IMAGE033
Then minimize
Figure 648177DEST_PATH_IMAGE027
Otherwise, minimize
Figure 145018DEST_PATH_IMAGE034
。
Distance cost, as shown in FIG. 8b, is defined as follows:
Figure 795442DEST_PATH_IMAGE035
（9）
Figure 453956DEST_PATH_IMAGE036
（10）
Figure 557042DEST_PATH_IMAGE037
（11）
wherein
Figure 591994DEST_PATH_IMAGE038
The width and height of the minimum bounding rectangle for the real box and the prediction box.
Shape loss (Sharp cost), defined as follows:
Figure 362504DEST_PATH_IMAGE039
（12）
Figure 191919DEST_PATH_IMAGE040
（13）
wherein
Figure 47880DEST_PATH_IMAGE041
And
Figure 877734DEST_PATH_IMAGE042
controlling the attention degree to the shape loss for the width and height of the prediction frame and the real frame, respectively, and setting to reduce the movement of the prediction frame for avoiding the excessive attention of the shape loss
Figure 971592DEST_PATH_IMAGE043
。
IoU loss (IoU cost), as shown in fig. 8c, is defined as follows:
Figure 503067DEST_PATH_IMAGE044
（14）
finally, the sio loss function is:
Figure 580745DEST_PATH_IMAGE045
（15）
and finishing the improvement of the YOLOv4 model, and starting to train the airport pavement ice and snow state perception model.
Further, the set ratio is: 8:1:1.
The ice and snow state data set is divided into a training set, a test set and a verification set according to the proportion of 8. The training set is high in proportion, data diversity is guaranteed, the training effect of the model is guaranteed, the verification set is used for checking the training degree of the model, the phenomenon that the model is over-fit and under-fit is prevented in the training process, the training set is assisted to construct the model, and the test set is used for evaluating the final effect of the model after the training is completed, the accuracy rate of the model, the error and the like.
Furthermore, the system for sensing the ice and snow state output by the ice and snow state sensing model specifically comprises: and inputting the test set into an ice and snow state sensing model so as to output an ice and snow state sensing system.
Further, generating the ice and snow state data set according to the microscopic image information specifically includes:
step S401: classifying the microscopic image information according to the ice and snow state classification table by combining the crystal state and the image gray value;
step S403: and (4) marking the microscopic image by adopting an image marking tool to generate a label file with a standard text format type to form an ice and snow state data set.
The historical ice and snow state microscopic image information collected by the electron microscope can be manually classified according to the ice and snow state classification table by combining the crystal state and the image gray value, and then the image is labeled by using an image labeling tool Labelimg to generate a label file with a standard text format type to form an ice and snow state data set, such as an xml type label file.
Example 2
As shown in fig. 10, a detection system 1 for a snow and ice state on a road surface includes: ice and snow state collection module 11, data processing module 12 and ice and snow state perception module 13, ice and snow state collection module 11 is suitable for the microcosmic image information who obtains the ice and snow state of road surface, data processing module 12 is connected with ice and snow state collection module 11 communication, data processing module 12 is suitable for according to microcosmic image information generation ice and snow state data set, ice and snow state perception module 13 is connected with data processing module 12 communication, ice and snow state perception module 13 is suitable for training the iteration to ice and snow state data set, in order to generate ice and snow state perception model, and output ice and snow state perception system.
The ice and snow state acquisition module 11 can be an electron microscope, and can autonomously learn the characteristics of ice accumulation images through a convolutional neural network based on image recognition, so that a complex algorithm for machine learning is omitted, and as long as the data samples of the ice type images are abundant, the problem that small differences between ice types are difficult to distinguish can be better solved.
Example 3
A computer-readable storage medium, comprising: the storage medium stores a computer program that, when executed, implements any one of the methods for detecting the state of ice and snow on a road surface.
The computer-readable storage medium provided by the embodiment of the present invention implements the steps of the method for detecting the snow and ice state of the road surface according to any one of the embodiments of the present invention, and thus has all the advantageous effects of the method for detecting the snow and ice state of the road surface according to any one of the embodiments of the present invention.
Example 4
Taking an airport pavement as an example, a method for detecting a snow and ice state on a pavement, as shown in fig. 11, specifically includes:
(1) Acquiring microscopic image information of the ice and snow state of the airport pavement by using an electron microscope;
(2) Manually classifying the ice and snow state microscopic images according to the ice and snow state classification table, labeling the images by using Labelimg, and generating an xml type label file to form a data set;
(3) Dividing a data set into a training set, a test set and a verification set according to a certain proportion;
(4) Modifying the model configuration file, and setting related training parameters;
(5) Introducing a GC Block global attention module into a CSP structure of a CSP search 53 main feature extraction network of YOLOv4, and adding global context modeling;
(6) An ASFF adaptive spatial feature fusion structure is added at the tail part of a PANet reinforced feature extraction network of YOLOv4, so that the multi-scale feature fusion effect is further enhanced;
(7) The SIoU loss function is used to replace the original CIoU function.
(8) And finishing the improvement of the YOLOv4 model, and starting to train the airport pavement ice and snow state perception model.
The ice and snow state at each stage was collected from directly above the airport runway using an electron microscope, as shown in fig. 5.
Dividing a data set into a training set, a test set and a verification set according to the proportion of 8.
After improvement of the YOLOv4 model is completed, an airport pavement ice and snow state perception model starts to be trained, in order to check the effect of the improved model in ice and snow state perception recognition, a perception model trained by the original YOLOv4 model and perception models trained by sequentially adding one, two or three improvement modules are compared with test set experiment results, and the test results of the four models are shown in the following table 1:
Figure 223079DEST_PATH_IMAGE046
as can be seen from the above table, the airport pavement ice and snow state sensing model based on the improved YOLOv4 model provided by the invention has a strong capability of identifying ice and snow state categories, and the improved YOLOv4 sensing result is shown in fig. 9, so that an airport operation center can comprehensively and accurately know the ice and snow state of the pavement, and a reliable decision basis is provided for the subsequent airport pavement ice and snow removal operation.
It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by a computer program that can be stored in a computer-readable storage medium and that when executed, can include the processes of the embodiments of the methods described above. The storage medium may be a magnetic Disk, an optical Disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a Flash Memory (Flash Memory), a Hard Disk (Hard Disk Drive, abbreviated as HDD), a Solid State Drive (SSD), or the like; the storage medium may also comprise a combination of memories of the kind described above.
The embodiment of the invention has the beneficial effects that:
according to the method for detecting the ice and snow state of the pavement, the electron microscope is used for collecting the ice and snow state microscopic image information of the pavement, the ice and snow state microscopic image of the pavement is manually classified according to an ice and snow state classification table and is labeled to form an ice and snow state microscopic image data set of the airport pavement; improving the YOLOv4 model, and improving the perception capability of the YOLOv4 model on the ice and snow image characteristics of the airport pavement by adding a global attention module, a self-adaptive spatial feature fusion structure and a modification loss function; and finally, inputting the microcosmic image data set of the ice and snow state of the airport pavement into an improved YOLOv4 model to train an ice and snow state perception model, and realizing real-time monitoring of the ice and snow state of the pavement. The method for sensing the ice and snow state of the road surface has the advantages of quick response, high accuracy, environment change, easiness in implementation and the like, and has a good application prospect.
In the present invention, the terms "first", "second", and "third" are used for descriptive purposes only and are not to be construed as indicating or implying relative importance; the term "plurality" means two or more unless expressly limited otherwise. The terms "mounted," "connected," "fixed," and the like are to be construed broadly, and for example, "connected" may be a fixed connection, a removable connection, or an integral connection; "connected" may be directly connected or indirectly connected through an intermediate. The specific meanings of the above terms in the present invention can be understood by those skilled in the art according to specific situations.
In the description of the present invention, it is to be understood that the terms "upper", "lower", "left", "right", "front", "rear", and the like indicate orientations or positional relationships based on those shown in the drawings, and are only for convenience of description and simplification of description, but do not indicate or imply that the referred device or unit must have a specific direction, be constructed in a specific orientation, and be operated, and thus, should not be construed as limiting the present invention.
In the description herein, the description of the terms "one embodiment," "some embodiments," "specific embodiments," etc., means that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the invention. In this specification, the schematic representations of the terms used above do not necessarily refer to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.
The present invention has been described in terms of the preferred embodiment, and it is not intended to be limited to the embodiment. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims (10)

1. A method for detecting the ice and snow state on a road surface is characterized by comprising the following steps:
acquiring microscopic image information of the ice and snow state of the road surface;
generating an ice and snow state data set according to the microscopic image information;
training iteration according to the ice and snow state data set to generate an ice and snow state perception model based on a convolutional neural network;
and sensing the real-time microscopic image of the ice and snow state of the road surface through the ice and snow state sensing model, and outputting an ice and snow state sensing system.
2. The method for detecting an icy or snowy state of a roadway surface according to claim 1, further comprising:
establishing a YOLOv4 network model;
dividing the ice and snow state data set into a training set, a test set and a verification set according to a set proportion;
and inputting the training set into the YOLOv4 network model for training to obtain the ice and snow state perception model.
3. The method for detecting the snow and ice state on the road surface according to claim 2, wherein the establishing of the YOLOv4 network model specifically comprises:
adding a global attention module in a main feature extraction network based on an original YOLOv4 target detection algorithm;
introducing a self-adaptive spatial feature fusion structure into an enhanced feature extraction network;
the original CIoU loss function is replaced with the SIoU loss function.
4. The method for detecting an ice and snow state on a road surface according to claim 2, wherein the set ratio is: 8:1:1.
5. The method for detecting an icy or snowy state of a road surface according to claim 2, wherein outputting an icy or snowy state sensing system by the icy or snowy state sensing model specifically comprises:
and inputting the test set into the ice and snow state perception model so as to output the ice and snow state perception system.
6. The method for detecting an icy or snowy state of a roadway surface according to any one of claims 1 to 5, wherein the generating an icy or snowy state data set from the microscopic image information specifically includes:
classifying the microscopic image information according to an ice and snow state classification table by combining a crystal state and an image gray value;
and marking the microscopic image by adopting an image marking tool to generate a label file with a standard text format type to form the ice and snow state data set.
7. The method for detecting the snow and ice state of the road surface according to claim 6, wherein the classifying the microscopic image information according to the snow and ice state classification table by combining the crystal state and the image gray level value specifically comprises: the status microscopic images were classified into 6 types: snow, snow melt, slush, frozen ice, wet ice, and water.
8. The method for detecting an ice and snow state on a road surface according to any one of claims 1 to 5, wherein the acquiring microscopic image information of the ice and snow state on the road surface specifically includes: and collecting the ice and snow state from the right above the road surface by adopting an electron microscope.
9. A system for detecting a snow and ice condition on a roadway, comprising:
the ice and snow state acquisition module is suitable for acquiring microscopic image information of the ice and snow state of the road surface;
the data processing module is in communication connection with the ice and snow state acquisition module and is suitable for generating an ice and snow state data set according to the microscopic image information;
and the ice and snow state sensing module is in communication connection with the data processing module and is suitable for training and iterating the ice and snow state data set to generate an ice and snow state sensing model and output an ice and snow state sensing system.
10. A computer-readable storage medium, comprising:
the storage medium stores a computer program that, when executed, implements a method of detecting an ice and snow state on a road surface according to any one of claims 1 to 8.
CN202211545183.6A 2022-12-05 2022-12-05 Method and system for detecting ice and snow state of road surface and computer readable storage medium Active CN115619777B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202211545183.6A CN115619777B (en) 2022-12-05 2022-12-05 Method and system for detecting ice and snow state of road surface and computer readable storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202211545183.6A CN115619777B (en) 2022-12-05 2022-12-05 Method and system for detecting ice and snow state of road surface and computer readable storage medium

Publications (2)

Publication Number Publication Date
CN115619777A true CN115619777A (en) 2023-01-17
CN115619777B CN115619777B (en) 2023-03-17

Family

ID=84879567

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202211545183.6A Active CN115619777B (en) 2022-12-05 2022-12-05 Method and system for detecting ice and snow state of road surface and computer readable storage medium

Country Status (1)

Country Link
CN (1) CN115619777B (en)

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113989230A (en) * 2021-10-28 2022-01-28 华东交通大学 Improved YOLOv 4-based road pavement disease detection method
CN114091185A (en) * 2022-01-24 2022-02-25 中国民航大学 Method and system for identifying evolution state of ice accumulated on surface of airplane
CN114332722A (en) * 2021-12-31 2022-04-12 吉林大学 Real-time estimation method for adhesion coefficient of mixed ice and snow road surface based on video data
CN114372960A (en) * 2021-12-17 2022-04-19 中国民航大学 Airplane ground ice accretion detection method, system, terminal and medium based on image understanding
CN114549914A (en) * 2022-04-24 2022-05-27 中国民航大学 Airplane large-area icing detection method and detection system
CN114584924A (en) * 2022-02-28 2022-06-03 长沙融创智胜电子科技有限公司 Intelligent unattended sensor system and target identification method
CN114596315A (en) * 2022-05-10 2022-06-07 中国民航大学 Aircraft ground detection icing method, device and system and computer equipment

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113989230A (en) * 2021-10-28 2022-01-28 华东交通大学 Improved YOLOv 4-based road pavement disease detection method
CN114372960A (en) * 2021-12-17 2022-04-19 中国民航大学 Airplane ground ice accretion detection method, system, terminal and medium based on image understanding
CN114332722A (en) * 2021-12-31 2022-04-12 吉林大学 Real-time estimation method for adhesion coefficient of mixed ice and snow road surface based on video data
CN114091185A (en) * 2022-01-24 2022-02-25 中国民航大学 Method and system for identifying evolution state of ice accumulated on surface of airplane
CN114584924A (en) * 2022-02-28 2022-06-03 长沙融创智胜电子科技有限公司 Intelligent unattended sensor system and target identification method
CN114549914A (en) * 2022-04-24 2022-05-27 中国民航大学 Airplane large-area icing detection method and detection system
CN114596315A (en) * 2022-05-10 2022-06-07 中国民航大学 Aircraft ground detection icing method, device and system and computer equipment

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
BAILIN LIU 等: ""An Improved Yolov5 for Multi-Rotor UAV Detection"" *
YANG YANG 等: ""GC-YOLOv3: You Only Look Once with Global Context Block"" *

Also Published As

Publication number Publication date
CN115619777B (en) 2023-03-17

Similar Documents

Publication Publication Date Title
CN108985194B (en) Intelligent vehicle travelable area identification method based on image semantic segmentation
Nie et al. Pavement Crack Detection based on yolo v3
Li et al. Automatic crack recognition for concrete bridges using a fully convolutional neural network and naive Bayes data fusion based on a visual detection system
CN101800890B (en) Multiple vehicle video tracking method in expressway monitoring scene
Cano-Ortiz et al. Machine learning algorithms for monitoring pavement performance
CN109325960A (en) A kind of infrared cloud image cyclone analysis method and analysis system
CN111415533B (en) Bend safety early warning monitoring method, device and system
CN102073852B (en) Multiple vehicle segmentation method based on optimum threshold values and random labeling method for multiple vehicles
CN106023257A (en) Target tracking method based on rotor UAV platform
CN103778435A (en) Pedestrian fast detection method based on videos
CN111860106B (en) Unsupervised bridge crack identification method
CN103745238A (en) Pantograph identification method based on AdaBoost and active shape model
CN113609895A (en) Road traffic information acquisition method based on improved Yolov3
CN112580575A (en) Electric power inspection insulator image identification method
CN114596315A (en) Aircraft ground detection icing method, device and system and computer equipment
CN115311229A (en) Laser radar-based pavement disease detection and classification method and system and storage medium
CN106529391B (en) A kind of speed limit road traffic sign detection of robust and recognition methods
CN113177929B (en) Sea surface oil spill detection method based on full convolution network, system and application thereof
Anwar et al. YOLOv4 based deep learning algorithm for defects detection and classification of rail surfaces
CN115619777B (en) Method and system for detecting ice and snow state of road surface and computer readable storage medium
Liu et al. Pavement surface defect recognition method based on vehicle system vibration data and feedforward neural network
Wang et al. Road surface state recognition based on semantic segmentation
CN115100173B (en) Road pavement image crack geometric property detection and crack identification method based on deep convolutional neural network
Shahbazi et al. Autonomous road pavement inspection and defect analysis for smart city maintenance
CN111429435A (en) Rapid and accurate cloud content detection method for remote sensing digital image

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant