CN115619777B - Method and system for detecting ice and snow state of road surface and computer readable storage medium - Google Patents

Method and system for detecting ice and snow state of road surface and computer readable storage medium Download PDF

Info

Publication number
CN115619777B
CN115619777B CN202211545183.6A CN202211545183A CN115619777B CN 115619777 B CN115619777 B CN 115619777B CN 202211545183 A CN202211545183 A CN 202211545183A CN 115619777 B CN115619777 B CN 115619777B
Authority
CN
China
Prior art keywords
ice
snow
snow state
state
road surface
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202211545183.6A
Other languages
Chinese (zh)
Other versions
CN115619777A (en
Inventor
邢志伟
朱书杰
李彪
阚犇
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Civil Aviation University of China
Original Assignee
Civil Aviation University of China
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Civil Aviation University of China filed Critical Civil Aviation University of China
Priority to CN202211545183.6A priority Critical patent/CN115619777B/en
Publication of CN115619777A publication Critical patent/CN115619777A/en
Application granted granted Critical
Publication of CN115619777B publication Critical patent/CN115619777B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/0002—Inspection of images, e.g. flaw detection
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/764—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
    • G06V10/774—Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
    • G06V10/80—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level
    • G06V10/806—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level of extracted features
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/60—Type of objects
    • G06V20/69—Microscopic objects, e.g. biological cells or cellular parts
    • G06V20/698—Matching; Classification
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02A—TECHNOLOGIES FOR ADAPTATION TO CLIMATE CHANGE
    • Y02A90/00—Technologies having an indirect contribution to adaptation to climate change
    • Y02A90/10—Information and communication technologies [ICT] supporting adaptation to climate change, e.g. for weather forecasting or climate simulation

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Evolutionary Computation (AREA)
  • Multimedia (AREA)
  • Health & Medical Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Computing Systems (AREA)
  • Databases & Information Systems (AREA)
  • Artificial Intelligence (AREA)
  • Medical Informatics (AREA)
  • Software Systems (AREA)
  • Quality & Reliability (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Biomedical Technology (AREA)
  • Molecular Biology (AREA)
  • Image Analysis (AREA)

Abstract

The invention provides a method, a system and a computer readable storage medium for detecting the ice and snow state on a road surface, wherein the detection method comprises the following steps: acquiring microscopic image information of the ice and snow state of the road surface; generating an ice and snow state data set according to the microscopic image information; training iteration according to the ice and snow state data set based on a convolutional neural network to generate an ice and snow state perception model; and sensing a real-time microscopic image of the ice and snow state of the road surface through the ice and snow state sensing model, and outputting an ice and snow state sensing system. The ice accumulation image characteristics can be autonomously learned through the convolutional neural network for image recognition, the complex algorithm of machine learning is omitted, and the problem that small differences among ice types are difficult to distinguish can be better solved, so that the ice and snow state perception model has strong recognition capability of ice and snow state categories.

Description

Method and system for detecting ice and snow state of road surface and computer readable storage medium
Technical Field
The invention relates to the technical field of structure detection, in particular to a method and a system for detecting the ice and snow state of a road surface and a computer readable storage medium.
Background
In the related art, with the increasing annual throughput of airports, the air transportation industry is also driven to develop continuously. However, in winter, the problem of icing on the airport pavement often occurs due to the ice and snow weather, and the duration is relatively long, which undoubtedly brings challenges to the flight operation of the airport, and the ice accumulation on the airport pavement causes a series of chain reaction problems, such as the problems of spare landing of a flight, delayed flight, passengers staying at the airport, airplane rushing out of a runway, airport closing and the like. After the airport runway is frozen, a thin ice layer is formed, then, accumulated ice is formed on the runway surface along with the expansion of the area of the ice layer, and the friction coefficient of the runway is sharply reduced. When ice is accumulated on the road surface and the friction coefficient of the road surface is lower than 0.3, the road surface is very smooth, and an airport must be closed and related deicing operation is carried out according to the stipulation of civil aviation bureaus. The ice accumulated on the airport roads not only can influence the operation of airports and airlines, but also can bring loss of lives and properties to passengers when going out. Therefore, the method is particularly important for the perception research of the ice and snow condition on the airport pavement, can provide valuable reference for field staff, makes a proper deicing scheme according to the ice condition on the pavement, selects the best guarantee measure and improves the service guarantee capability of the airport.
At present, the common ice and snow condition detection method includes a sensor detection method, and the ice detection sensor is widely applied mainly in an optical fiber type, an ultrasonic type, a microwave type, an infrared reflection type and the like: (1) optical fiber formula: determining the ice type according to the difference of the reflected and scattered light intensity of the light in different ice and snow states; (2) ultrasonic formula: when an ice layer is formed on the surface, the piezoelectric device receives ultrasonic waves reflected by an ice and air interface, and the ice accumulation condition is judged according to the time delay of the reflection and the reception of the ultrasonic waves in the ice; (3) microwave formula: the waveguide tube is installed on the surface, and the phase constant of ice gathered on the surface of the waveguide tube insulating layer can be changed, so that the resonant frequency is reduced, and the ice accumulation condition is obtained according to the offset of the resonant frequency; (4) infrared reflection formula: based on the fact that all solids can reflect infrared energy when the temperature is higher than absolute zero, the infrared energy reflects the surface temperature of the detection piece, the temperature difference value is very close to that of the detection piece when the detection piece is frozen, and ice accumulation information is obtained through detection of the temperature difference value.
Another common ice accumulation state sensing method is a computer vision method, which uses an image processing technology to perform preliminary processing on an ice and snow image of a road surface, and extracts characteristic information including color, texture, contour and the like from the image by combining a machine learning algorithm so as to detect and identify various ice and snow states of the road surface. The method uses the image acquisition device to acquire the ice and snow images of the pavement without contacting the pavement. However, the common computer vision method has high requirements on image quality, the ice and snow states of the road surface are variable, the image characteristics are complex, and the problem of large deviation between the sensing result and the actual result exists.
Through the above analysis, the problems and defects of the prior art are as follows:
(1) The ice accumulation detection sensor has a limited range for detecting ice accumulation, is difficult to comprehensively detect the ice accumulation area on the road, has low accuracy due to the influence of environmental factors, is high in cost for damage, maintenance and replacement and is difficult to widely use in the scene of sensing the ice and snow state of the area of the airport road;
(2) The states of ice and snow on the road surface are various, the existing computer vision method has insufficient capability of extracting the features of the ice and snow image on the airport road surface, and the accuracy of the ice and snow state perception model is limited.
The difficulty in solving the above problems and defects is: the information collected by the ice accumulation sensor is used for detecting that the icing condition of the airport pavement is greatly influenced by environmental factors, and the sensor is easy to damage, so that the detection result is wrong; the conventional image identification method is used for detecting the icing condition of the airport pavement, and has poor real-time performance and low accuracy.
Disclosure of Invention
The present invention is directed to solving or improving at least one of the above technical problems.
Therefore, a first object of the present invention is to provide a method for detecting a state of ice and snow on a road surface.
The second purpose of the invention is to provide a detection system for the ice and snow state on the road surface.
A third object of the present invention is to provide a computer-readable storage medium.
To achieve the first object of the present invention, an embodiment of the present invention provides a method for detecting a snow and ice state on a road surface, including: collecting the ice and snow state from the right above the road surface by adopting an electron microscope, and acquiring microscopic image information of the ice and snow state of the road surface; generating an ice and snow state data set according to the microscopic image information; dividing the ice and snow state data set into a training set, a test set and a verification set according to a set proportion; establishing an improved YOLOv4 network model based on a convolutional neural network; inputting the improved YOLOv4 network model according to the training set in the ice and snow state to carry out data set training iteration so as to generate an ice and snow state perception model; inputting the collected real-time ice and snow state microscopic image of the road surface into an ice and snow state perception model; and sensing a real-time microscopic image of the ice and snow state of the road surface through the ice and snow state sensing model, inputting the test set into the ice and snow state sensing model, and outputting an ice and snow state sensing system.
Optionally, the method for detecting the snow and ice state on the road surface further includes: establishing a YOLOv4 network model; dividing the ice and snow state data set into a training set, a testing set and a verification set according to a set proportion; and inputting the training set into the YOLOv4 network model for training to obtain the ice and snow state perception model.
Optionally, the establishing a YOLOv4 network model specifically includes: adding a global attention module in a main feature extraction network based on an original YOLOv4 target detection algorithm; introducing a self-adaptive spatial feature fusion structure into an enhanced feature extraction network; the original CIoU loss function is replaced with the SIoU loss function.
Optionally, the set ratio is: 8:1:1.
Optionally, outputting an ice and snow state sensing system through the ice and snow state sensing model specifically includes: and inputting the test set into the ice and snow state perception model so as to output the ice and snow state perception system.
Optionally, the generating an ice and snow state data set according to the microscopic image information specifically includes: classifying the microscopic image information according to an ice and snow state classification table by combining a crystal state and an image gray value; and marking the microscopic image by adopting an image marking tool to generate a label file with a standard text format type to form the ice and snow state data set.
Optionally, the classifying the microscopic image information according to the ice and snow state classification table by combining the crystal state and the image gray level value specifically includes: the status microscopic images were classified into 6 types: snow, snow melt, slush, frozen ice, wet ice, and water.
Optionally, the acquiring microscopic image information of the ice and snow state of the road surface specifically includes: and collecting the ice and snow state from the right above the road surface by adopting an electron microscope.
To achieve the second object of the present invention, an embodiment of the present invention provides a system for detecting a snow-ice state on a road surface, including: the ice and snow state acquisition module comprises an electron microscope, wherein the electron microscope acquires an ice and snow state from the right above a road surface and is suitable for acquiring microscopic image information of the ice and snow state of the road surface; the data processing module is in communication connection with the ice and snow state acquisition module and is suitable for generating an ice and snow state data set according to the microscopic image information and dividing the ice and snow state data set into a training set, a testing set and a verification set according to a set proportion; establishing a YOLOv4 network model based on a convolutional neural network; and the ice and snow state sensing module is in communication connection with the data processing module and is suitable for training and iterating the ice and snow state data set so as to generate an ice and snow state sensing model and output an ice and snow state sensing system.
To achieve the third object of the present invention, an embodiment of the present invention provides a computer-readable storage medium including: the storage medium stores a computer program that, when executed, implements any one of the methods for detecting an ice and snow state on a road surface.
Compared with the prior art, the advantages and the inventive technical effects of the invention are specifically described as follows:
1. the ice accumulation image characteristics can be independently learned through the convolutional neural network for image recognition, the complex algorithm of machine learning is omitted, and the problem that small differences between ice types are difficult to distinguish can be better solved as long as the data samples of the ice type images are abundant, so that the ice and snow state perception model has strong recognition capability of ice and snow state categories. The collected microcosmic images of the real-time ice and snow state of the road surface are input into the ice and snow state perception model, the real-time ice and snow state of the road surface can be perceived, and the ice and snow state perception system can be output, so that the ice accumulation condition of the road surface can be comprehensively and accurately known, and reliable data basis is provided for the subsequent ice and snow removal operation of the road surface.
2. According to the method, a track ice and snow state perception model is trained by improving YOLOv4, and aiming at the problems that the ice and snow crystal state presents different sizes and shapes, is interfered by impurities such as dust and coherent speckle noise and is difficult to perceive the state, a global attention module GC Block is added in a main feature extraction network structure to increase global context modeling, so that richer shallow and deep features are obtained.
3. An adaptive spatial feature fusion structure ASFF is added at the tail of a PANet of a reinforced feature extraction network of a YOLOv4 model, so that the defect of inconsistency of different scales in the PANet network is suppressed, the multi-scale features are fused in a self-adaptive manner, and the optimal fusion effect is achieved;
4. and in consideration of the vector angle between required regressions, redefining the penalty index, and replacing the original CIoU loss function with the SIoU loss function to better reflect the position relation between the prediction frame and the real frame and further improve the convergence speed of the perception model and the accuracy of reasoning.
5. By improving an image processing algorithm, the ice and snow state of the pavement is accurately sensed, a pavement ice and snow state sensing model with high accuracy is obtained along with accumulation of historical data, the ice accumulation condition of the airport pavement is accurately identified, an accurate and valuable reference is provided for ice and snow removal operation of the pavement, and the operation guarantee capability of the pavement is improved.
Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention.
Drawings
FIG. 1 is a flow chart of a method for detecting the snow and ice on a road surface according to an embodiment of the present invention;
FIG. 2 is a flow chart of a method for detecting the snow and ice on the road surface according to an embodiment of the present invention;
FIG. 3 is a flowchart of a method for detecting the snow and ice on the road surface according to the embodiment of FIG. 2, wherein a Yolov4 network model is established;
fig. 4 is a flowchart of the method of fig. 1 for detecting the snow and ice on the road surface according to the embodiment of the present invention for generating a data set of the snow and ice state from the microscopic image information;
FIG. 5 is a top view of six snow and ice states of a method for detecting snow and ice on a roadway surface according to an embodiment of the present invention;
FIG. 6 is a block diagram of a global attention module of a method for detecting an ice and snow status of a road surface according to an embodiment of the present invention;
FIG. 7 is a block diagram of an adaptive spatial feature fusion module of a method for detecting the ice and snow status of a road surface according to an embodiment of the present invention;
FIG. 8a is a schematic view of an angle loss of a method for detecting an ice and snow state on a road surface according to an embodiment of the present invention;
FIG. 8b is a schematic diagram illustrating distance loss in a method for detecting an ice/snow status on a road surface according to an embodiment of the present invention;
fig. 8c is a IoU loss schematic diagram of a method for detecting an ice and snow state on a road surface according to an embodiment of the present invention.
Fig. 9 is a diagram of a sensing result based on an improved yoolov 4 snow and ice state of a method for detecting the snow and ice state on a road surface according to an embodiment of the present invention;
fig. 10 is a block diagram of a system for detecting the snow and ice on the road surface according to an embodiment of the present invention.
Fig. 11 is a flowchart of a method for detecting the ice and snow state of the aircraft road surface according to the method for detecting the ice and snow state of the road surface of the embodiment of the present invention.
Reference numerals
1-a system for detecting the ice and snow state of the road surface; 11-an ice and snow state acquisition module; 12-a data processing module; 13-ice and snow state sensing module.
Detailed Description
In order that the above objects, features and advantages of the present invention can be more clearly understood, a more particular description of the invention, taken in conjunction with the accompanying drawings and detailed description, is set forth below. It should be noted that the embodiments and features of the embodiments of the present application may be combined with each other without conflict.
In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present invention, however, the present invention may be practiced in other ways than those specifically described herein, and therefore the scope of the present invention is not limited by the specific embodiments disclosed below.
The technical solutions of some embodiments of the present invention are described below with reference to fig. 1 to 10.
As shown in fig. 1, there is provided a method for detecting a snow-ice state on a road surface, comprising:
step S101: collecting an ice and snow state from the right above a pavement by adopting an electron microscope, and acquiring microscopic image information of the ice and snow state of the pavement;
step S103: generating an ice and snow state data set according to the microscopic image information;
step S105: dividing the ice and snow state data set into a training set, a testing set and a verification set according to a set proportion;
step S107: establishing an improved YOLOv4 network model based on a convolutional neural network;
step S109: training iteration according to the ice and snow state data set to generate an ice and snow state perception model;
step S111: inputting the collected real-time ice and snow state microscopic image of the road surface into an ice and snow state perception model;
step S113: and sensing a real-time microscopic image of the ice and snow state of the road surface through an ice and snow state sensing model, inputting a test set into the ice and snow state sensing model, and outputting an ice and snow state sensing system.
The road surface can comprise a large-scale road surface such as an airport and the like, the characteristics of the ice-accumulating image can be automatically learned through a convolutional neural network for image recognition, the complex algorithm of machine learning is omitted, and the problem that small differences among ice types are difficult to distinguish can be better solved as long as the data samples of the ice type images are abundant, so that the ice and snow state perception model has strong recognition capability of ice and snow state categories. The collected microcosmic images of the real-time ice and snow state of the road surface are input into the ice and snow state perception model, the real-time ice and snow state of the road surface can be perceived, and the ice and snow state perception system can be output, so that the ice accumulation condition of the road surface can be comprehensively and accurately known, and reliable data basis is provided for the subsequent ice and snow removal operation of the road surface.
Further, acquiring microscopic image information of the ice and snow state of the road surface specifically comprises: and collecting the ice and snow state from the right above the road surface by adopting an electron microscope. The ice and snow state images are collected from the position right above the road surface, so that the integrity and intuition of the ice and snow state information acquisition can be ensured, the ice and snow state image characteristics can be extracted by the sensing model, and the classification effect of the sensing model on the ice and snow state of the road surface is improved.
Further, the method for detecting the ice and snow state of the road surface further comprises the following steps:
step S201: establishing a YOLOv4 network model;
step S203: dividing the ice and snow state data set into a training set, a testing set and a verification set according to a set proportion;
step S205: and inputting the training set into a YOLOv4 network model for training to obtain an ice and snow state perception model.
The ice and snow state perception model of the road surface is trained through the improved YOLOv4 algorithm, the training set and the verification set are put into the perception model based on the improved YOLOv4 to be trained for 200 epochs to obtain the optimal perception model, then the real-time ice and snow state microscopic image of the road surface is collected by an electron microscope and is input into the ice and snow state perception model, and the real-time ice and snow state of the road surface can be perceived.
Further, as shown in fig. 5, classifying the microscopic image information according to the ice and snow state classification table and combining the crystal state and the image gray scale value specifically includes: the status microscopic images were classified into 6 types: snow (a), snow melt (b), slush (c), frozen ice (d), wet ice (e) and water (f).
The type name of model training is modified into snow, snow melt, slush, frozen ice, wet ice and water, the number of categories is modified into 6, the training iteration time epoch is set to be 200, and the one-time training sample number batch-size is set to be 16.
Further, acquiring microscopic image information of the ice and snow state of the road surface specifically comprises: and collecting the ice and snow state from the right above the road surface by adopting an electron microscope.
Further, establishing a YOLOv4 network model specifically includes:
step S301: adding a global attention module in a main feature extraction network based on an original YOLOv4 target detection algorithm; step S303: introducing a self-adaptive spatial feature fusion structure into an enhanced feature extraction network;
step S305: the original CIoU loss function is replaced with the SIoU loss function.
Since the states of the ice and snow crystals are characterized by different sizes and shapes, and interference of impurities such as dust and speckle noise is generated, the states are not easy to perceive, for the original YOLOv4 target detection algorithm, a global attention module GCBlock is added in a trunk feature extraction network CSPDarknet53, and the structure is shown in fig. 6. By introducing a GCBlock module into the CSP structure of the CSP search network 53, global context information can be extracted to obtain richer shallow and deep features. Therefore, for the original YOLOv4 target detection algorithm, a global attention module GC Block is added to the main feature extraction network CSPDarknet53, and global modeling is added.
The implementation steps of the GCBlock global attention module comprise:
step 1: using a 1 × 1 convolution of W k And the softmax function acquires attention weight and then carries out attention pooling operation, thereby realizing the context modeling sigma j α j x j ，α j For global attention pooling weights, the correlation of positions is obtained by embedding a Gaussian (Embedded Gaussian), α j The calculation formula is as follows:
Figure GDA0004066509840000061
step 2: using 1 × 1 convolution W in sequence v1 LayerNorm, reLU and 1 × 1 convolution W v2 And the characteristic conversion operation is carried out, so that the aims of reducing the optimization difficulty and facilitating generalization are fulfilled. The partial calculation formula is as follows:
δ(·)＝W v2 ReLU(LN(W v1 (·))) (2)
and step 3: feature fusion using an addition operation, by aggregating global context features onto the features of each location, F (x) i ，δ(∑ j α j x j ) Substituting the results (1) and (2) to obtain the final output result:
Figure GDA0004066509840000071
wherein x is an input example feature map, z is an output feature map, i is a query index position, and j is all possible enumeration positions; an example of an input is a picture, N p ＝H*W。
As shown in fig. 7, an adaptive spatial feature fusion structure ASFF is added at the tail of the PANet enhanced feature extraction network, and the disadvantage of inconsistency of different scales in the PANet network is suppressed, so that the multi-scale features are adaptively fused, and an optimal fusion effect can be achieved. Therefore, the self-adaptive spatial feature fusion structure ASFF is introduced into the reinforced feature extraction network PANet, and the multi-scale feature fusion effect is enhanced.
In the ASFF structure, three enhanced feature layers, namely Level1, level2 and Level3, are obtained through a PANet enhanced feature extraction network, and the feature map of each layer is converted into the size of the other two layers of the three layers through the ASFF structure and then fusion prediction is carried out. Taking ASFF-3 as an example, obtaining feature layers Level1 and Level2 through PANet, performing 1 × 1 convolution operation on the ASFF-3 to compress the number of channels to be the same as Level3, then performing 4-fold and 2-fold upsampling on the ASFF-3 to form a feature map with the same dimension as Level3, recording the feature map as Resize _ Level1 and Resize _ Level2, then performing 1 × 1 convolution operation on Resize _ Level1, resize _ Level2 and Level3 to obtain weight parameters α, β and γ, and finally multiplying the weight parameters by Resize _ Level1, resize _ Level2 and Level3 and summing up to obtain ASFF-3 with fused features, where the above process can be described by the following formula:
Figure GDA0004066509840000072
wherein
Figure GDA00040665098400000810
Representing a new feature map obtained by ASFF feature fusion;
Figure GDA00040665098400000812
respectively representing the feature vectors from the 1 st, 2 nd and 3 rd layers to the l-th layer feature layer;
Figure GDA00040665098400000811
weight parameters of three different characteristic layers are respectively expressed by a softmax function
Figure GDA0004066509840000081
And replacing the original CIoU loss function with the SIoU loss function, redefining penalty indexes by considering vector angles between required regressions, and further improving convergence speed and reasoning accuracy. Therefore, the SIoU loss function is used for replacing the original CIoU loss function, and the position relation of the predicted frame and the real frame can be better reflected. The sio loss function consists of four cost functions:
the Angle cost (Angle cost), as shown in FIG. 8a, is defined as follows:
Figure GDA0004066509840000082
wherein c is h Is the height difference between the center points of the real frame and the predicted frame, sigma is the distance between the center points of the real frame and the predicted frame,
Figure GDA0004066509840000083
i.e. the angle alpha.
Figure GDA0004066509840000084
Figure GDA0004066509840000085
Figure GDA0004066509840000086
Figure GDA0004066509840000087
Is the coordinates of the center of the real frame,
Figure GDA0004066509840000088
for predicting the frame center coordinates, when α is 0 or π/2, the angle loss is 0, if it is during training
Figure GDA0004066509840000089
Then alpha is minimized and beta is minimized otherwise.
Distance cost, as shown in FIG. 8b, is defined as follows:
Δ＝∑ t＝x，y (1-e -γρt ) (9)
Figure GDA0004066509840000091
γ＝2-Λ (11)
wherein (c) w ，c h ) The width and height of the minimum bounding rectangle for the real box and the prediction box.
Shape loss (Sharp cost), defined as follows:
Figure GDA0004066509840000095
Figure GDA0004066509840000092
wherein (w, h) and (w) gt ，h gt ) The attention degree of the shape loss is controlled for the width and height of the prediction frame and the real frame, respectively, and theta epsilon [2,6 ] is set to avoid reducing the movement of the prediction frame due to excessive attention on the shape loss]。
IoU loss (IoU cost), as shown in fig. 8c, is defined as follows:
Figure GDA0004066509840000093
finally, the sio loss function is:
Figure GDA0004066509840000094
and finishing the improvement of the YOLOv4 model, and starting to train the airport pavement ice and snow state perception model.
Further, the set ratio is: 8: 1.
The ice and snow state data set is divided into a training set, a testing set and a verification set according to the ratio of 8:1, a model configuration file is modified, and related training parameters are set. The training set is high in proportion, data diversity is guaranteed, the training effect of the model is guaranteed, the verification set is used for checking the training degree of the model, the phenomenon that the model is over-fit and under-fit is prevented in the training process, the training set is assisted to construct the model, and the test set is used for evaluating the final effect of the model after the training is completed, the accuracy rate of the model, the error and the like.
Furthermore, the system for sensing the ice and snow state output by the ice and snow state sensing model specifically comprises: and inputting the test set into an ice and snow state sensing model so as to output an ice and snow state sensing system.
Further, generating the ice and snow state data set according to the microscopic image information specifically includes:
step S401: classifying the microscopic image information according to the ice and snow state classification table by combining the crystal state and the image gray level value; step S403: and (4) marking the microscopic image by adopting an image marking tool to generate a label file with a standard text format type to form an ice and snow state data set.
The historical ice and snow state microscopic image information collected by the electron microscope can be manually classified according to the ice and snow state classification table by combining the crystal state and the image gray value, and then the image is labeled by using an image labeling tool Labelimg to generate a label file with a standard text format type to form an ice and snow state data set, such as an xml type label file.
Example 2
As shown in fig. 10, a detection system 1 for a snow and ice state on a road surface includes: ice and snow state collection module 11, data processing module 12 and ice and snow state perception module 13, ice and snow state collection module 11 is suitable for the microcosmic image information who obtains the ice and snow state of road surface, data processing module 12 is connected with ice and snow state collection module 11 communication, data processing module 12 is suitable for according to microcosmic image information generation ice and snow state data set, ice and snow state perception module 13 is connected with data processing module 12 communication, ice and snow state perception module 13 is suitable for training the iteration to ice and snow state data set, in order to generate ice and snow state perception model, and output ice and snow state perception system.
The ice and snow state acquisition module 11 can be an electron microscope, and can autonomously learn the characteristics of ice accumulation images through a convolutional neural network based on image recognition, so that a complex algorithm for machine learning is omitted, and as long as the data samples of the ice type images are abundant, the problem that small differences between ice types are difficult to distinguish can be better solved.
Example 3
A computer-readable storage medium, comprising: the storage medium stores a computer program that, when executed, implements any one of the methods for detecting the state of ice and snow on a road surface.
The computer-readable storage medium provided by the embodiment of the present invention implements the steps of the method for detecting the snow and ice state of the road surface according to any one of the embodiments of the present invention, and thus has all the advantageous effects of the method for detecting the snow and ice state of the road surface according to any one of the embodiments of the present invention.
Example 4
Taking an airport pavement as an example, a method for detecting a snow and ice state on a pavement, as shown in fig. 11, specifically includes:
(1) Acquiring microcosmic image information of the ice and snow state of the airport pavement by using an electron microscope;
(2) Manually classifying the ice and snow state microscopic images according to the ice and snow state classification table, labeling the images by using Labelimg, and generating an xml type label file to form a data set;
(3) Dividing a data set into a training set, a testing set and a verification set according to a certain proportion;
(4) Modifying the model configuration file and setting related training parameters;
(5) Introducing a GC Block global attention module into a CSP structure of a CSP search 53 main feature extraction network of YOLOv4, and adding global context modeling;
(6) An ASFF adaptive spatial feature fusion structure is added at the tail part of a PANet reinforced feature extraction network of YOLOv4, so that the multi-scale feature fusion effect is further enhanced;
(7) The SIoU loss function is used instead of the original CIoU function.
(8) And finishing the improvement of the YOLOv4 model, and starting to train the airport pavement ice and snow state perception model.
The ice and snow state at each stage was collected from just above the airport runway using an electron microscope, as shown in fig. 5.
Dividing a data set into a training set, a testing set and a verification set according to the proportion of 8.
After improvement of the YOLOv4 model is completed, an airport pavement ice and snow state perception model starts to be trained, in order to check the effect of the improved model in ice and snow state perception recognition, a perception model trained by the original YOLOv4 model and perception models trained by sequentially adding one, two or three improvement modules are compared with test set experiment results, and the test results of the four models are shown in the following table 1:
TABLE 1 comparison of test results of four models
Figure GDA0004066509840000111
As can be seen from the above table, the airport pavement ice and snow state sensing model based on the improved YOLOv4 model provided by the invention has a strong capability of identifying ice and snow state categories, and the improved YOLOv4 sensing result is shown in fig. 9, so that an airport operation center can comprehensively and accurately know the ice and snow state of the pavement, and a reliable decision basis is provided for the subsequent airport pavement ice and snow removal operation.
It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by a computer program that can be stored in a computer-readable storage medium and that when executed, can include the processes of the embodiments of the methods described above. The storage medium may be a magnetic Disk, an optical Disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a Flash Memory (Flash Memory), a Hard Disk (Hard Disk Drive, abbreviated as HDD), a Solid State Drive (SSD), or the like; the storage medium may also comprise a combination of memories of the kind described above.
The embodiment of the invention has the beneficial effects that:
according to the method for detecting the ice and snow state of the pavement, an electron microscope is used for collecting the ice and snow state microscopic image information of the pavement, the ice and snow state microscopic image of the pavement is manually classified according to an ice and snow state classification table and is marked to form an airport pavement ice and snow state microscopic image data set; improving the YOLOv4 model, and improving the perception capability of the YOLOv4 model on the ice and snow image characteristics of the airport pavement by adding a global attention module, a self-adaptive spatial feature fusion structure and a modification loss function; and finally, inputting the microcosmic image data set of the ice and snow state of the airport pavement into an improved YOLOv4 model to train an ice and snow state sensing model, so as to realize real-time monitoring of the ice and snow state of the pavement. The method for sensing the ice and snow state of the road surface has the advantages of quick response, high accuracy, environment change, easiness in implementation and the like, and has a good application prospect.
In the present invention, the terms "first", "second", and "third" are used for descriptive purposes only and are not to be construed as indicating or implying relative importance; the term "plurality" means two or more unless expressly limited otherwise. The terms "mounted," "connected," "fixed," and the like are used broadly and should be construed to include, for example, "connected" may be a fixed connection, a detachable connection, or an integral connection; "coupled" may be direct or indirect through an intermediary. The specific meanings of the above terms in the present invention can be understood by those skilled in the art according to specific situations.
In the description of the present invention, it is to be understood that the terms "upper", "lower", "left", "right", "front", "rear", and the like indicate orientations or positional relationships based on those shown in the drawings, and are only for convenience of description and simplification of description, but do not indicate or imply that the referred device or unit must have a specific direction, be constructed in a specific orientation, and be operated, and thus, should not be construed as limiting the present invention.
In the description herein, the description of the terms "one embodiment," "some embodiments," "specific embodiments," etc., means that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the invention. In this specification, the schematic representations of the terms used above do not necessarily refer to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.
The above is only a preferred embodiment of the present invention, and is not intended to limit the present invention, and various modifications and changes will occur to those skilled in the art. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims (6)

1. A method for detecting the ice and snow state on a road surface is characterized by comprising the following steps:
collecting an ice and snow state from the right above a pavement by adopting an electron microscope, and acquiring microscopic image information of the ice and snow state of the pavement;
generating an ice and snow state data set according to the microscopic image information;
dividing the ice and snow state data set into a training set, a testing set and a verification set according to a set proportion;
based on a convolutional neural network, establishing an improved YOLOv4 network model, which specifically comprises the following steps:
adding a global attention module in a main feature extraction network based on an original YOLOv4 target detection algorithm;
introducing a self-adaptive spatial feature fusion structure into an enhanced feature extraction network;
replacing the original CIoU loss function with the SIoU loss function;
inputting the improved YOLOv4 network model according to the training set in the ice and snow state data set for training iteration to generate an ice and snow state perception model;
inputting the collected real-time ice and snow state microscopic image of the road surface into an ice and snow state perception model;
and sensing a real-time microscopic image of the ice and snow state of the road surface through the ice and snow state sensing model, inputting the test set into the ice and snow state sensing model, and outputting an ice and snow state sensing system.
2. The method for detecting an ice and snow state on a road surface according to claim 1, wherein the set ratio is: 8:1:1.
3. The method for detecting an ice/snow state on a road surface according to claim 1 or 2, wherein generating an ice/snow state data set from the microscopic image information specifically includes:
classifying the microscopic image information according to an ice and snow state classification table by combining a crystal state and an image gray value;
and marking the microscopic image by adopting an image marking tool to generate a label file with a standard text format type to form the ice and snow state data set.
4. The method for detecting the snow and ice state of the road surface according to claim 3, wherein the classifying the microscopic image information according to the snow and ice state classification table by combining the crystal state and the image gray level value specifically comprises: the status microscopic images were classified into 6 types: snow, snow melt, slush, frozen ice, wet ice, and water.
5. A system for detecting a snow and ice condition on a roadway, comprising:
the ice and snow state acquisition module comprises an electron microscope, the electron microscope acquires an ice and snow state from the right above the road surface, and is suitable for acquiring and sending microscopic image information of the ice and snow state of the road surface;
the data processing module is in communication connection with the ice and snow state acquisition module and is suitable for generating an ice and snow state data set according to the microscopic image information and dividing the ice and snow state data set into a training set, a testing set and a verification set according to a set proportion; based on a convolutional neural network, establishing a Yolov4 network model, which specifically comprises the following steps: adding a global attention module in a main feature extraction network based on an original YOLOv4 target detection algorithm; introducing a self-adaptive spatial feature fusion structure into an enhanced feature extraction network; replacing the original CIoU loss function with the SIoU loss function;
and the ice and snow state sensing module is generated by inputting an improved YOLOv4 network model into the training set in the ice and snow state data set for training iteration, is in communication connection with the data processing module and the electronic microscope respectively, and is suitable for acquiring the test set of the data processing module and real-time microscopic images of the ice and snow state of the road surface sent by the electronic microscope to output an ice and snow state sensing system.
6. A computer-readable storage medium, comprising:
the storage medium stores a computer program that, when executed, implements a method of detecting an ice and snow state on a road surface according to any one of claims 1 to 4.
CN202211545183.6A 2022-12-05 2022-12-05 Method and system for detecting ice and snow state of road surface and computer readable storage medium Active CN115619777B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202211545183.6A CN115619777B (en) 2022-12-05 2022-12-05 Method and system for detecting ice and snow state of road surface and computer readable storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202211545183.6A CN115619777B (en) 2022-12-05 2022-12-05 Method and system for detecting ice and snow state of road surface and computer readable storage medium

Publications (2)

Publication Number Publication Date
CN115619777A CN115619777A (en) 2023-01-17
CN115619777B true CN115619777B (en) 2023-03-17

Family

ID=84879567

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202211545183.6A Active CN115619777B (en) 2022-12-05 2022-12-05 Method and system for detecting ice and snow state of road surface and computer readable storage medium

Country Status (1)

Country Link
CN (1) CN115619777B (en)

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113989230A (en) * 2021-10-28 2022-01-28 华东交通大学 Improved YOLOv 4-based road pavement disease detection method
CN114584924A (en) * 2022-02-28 2022-06-03 长沙融创智胜电子科技有限公司 Intelligent unattended sensor system and target identification method

Family Cites Families (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114372960B (en) * 2021-12-17 2024-08-02 中国民航大学 Airplane ground ice accumulation detection method, system, terminal and medium based on image understanding
CN114332722A (en) * 2021-12-31 2022-04-12 吉林大学 Real-time estimation method for adhesion coefficient of mixed ice and snow road surface based on video data
CN114091185B (en) * 2022-01-24 2022-04-12 中国民航大学 Method and system for identifying evolution state of ice accumulated on surface of airplane
CN114549914B (en) * 2022-04-24 2022-07-12 中国民航大学 Airplane large-area icing detection method and detection system
CN114596315A (en) * 2022-05-10 2022-06-07 中国民航大学 Aircraft ground detection icing method, device and system and computer equipment

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113989230A (en) * 2021-10-28 2022-01-28 华东交通大学 Improved YOLOv 4-based road pavement disease detection method
CN114584924A (en) * 2022-02-28 2022-06-03 长沙融创智胜电子科技有限公司 Intelligent unattended sensor system and target identification method

Also Published As

Publication number Publication date
CN115619777A (en) 2023-01-17

Similar Documents

Publication Publication Date Title
Hou et al. The state-of-the-art review on applications of intrusive sensing, image processing techniques, and machine learning methods in pavement monitoring and analysis
CN109325960B (en) Infrared cloud chart cyclone analysis method and analysis system
Cano-Ortiz et al. Machine learning algorithms for monitoring pavement performance
CN113486938B (en) Multi-branch time convolution network-based re-landing analysis method and device
CN111626290A (en) Infrared ship target detection and identification method under complex sea surface environment
CN111507233A (en) Multi-mode information fusion intelligent vehicle pavement type identification method
CN111415533B (en) Bend safety early warning monitoring method, device and system
CN110910531B (en) Rapid pavement friction coefficient detection method based on vehicle-mounted OBD information
CN111860106B (en) Unsupervised bridge crack identification method
CN113139594B (en) Self-adaptive detection method for airborne image unmanned aerial vehicle target
CN111582380A (en) Ship track density clustering method and device based on space-time characteristics
CN113177929B (en) Sea surface oil spill detection method based on full convolution network, system and application thereof
CN114372960A (en) Airplane ground ice accretion detection method, system, terminal and medium based on image understanding
CN103745238A (en) Pantograph identification method based on AdaBoost and active shape model
CN114596315A (en) Aircraft ground detection icing method, device and system and computer equipment
CN113609895A (en) Road traffic information acquisition method based on improved Yolov3
CN116403124A (en) Structural plane intelligent recognition method of three-dimensional point cloud surrounding rock based on DResNet-PointNet
CN115311229A (en) Laser radar-based pavement disease detection and classification method and system and storage medium
CN115619777B (en) Method and system for detecting ice and snow state of road surface and computer readable storage medium
Carrillo et al. Design of efficient deep learning models for determining road surface condition from roadside camera images and weather data
CN110210326A (en) A kind of identification of online train and speed estimation method based on fiber-optic vibration signal
CN117152706A (en) Aircraft runway accumulated water identification method, device and system
CN117218375A (en) Priori knowledge and data driven based environment visibility prediction method and device
CN112395956A (en) Method and system for detecting passable area facing complex environment
CN116630268A (en) Road disease detection method, system, equipment and medium

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant