CN114926456A - Rail foreign matter detection method based on semi-automatic labeling and improved deep learning - Google Patents

Rail foreign matter detection method based on semi-automatic labeling and improved deep learning Download PDF

Info

Publication number
CN114926456A
CN114926456A CN202210669222.7A CN202210669222A CN114926456A CN 114926456 A CN114926456 A CN 114926456A CN 202210669222 A CN202210669222 A CN 202210669222A CN 114926456 A CN114926456 A CN 114926456A
Authority
CN
China
Prior art keywords
rail
algorithm
foreign matter
improved
model
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202210669222.7A
Other languages
Chinese (zh)
Inventor
王国华
谢函
古卓恒
连佩仪
凌海涛
陈嘉豪
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
South China Agricultural University
Original Assignee
South China Agricultural University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by South China Agricultural University filed Critical South China Agricultural University
Priority to CN202210669222.7A priority Critical patent/CN114926456A/en
Publication of CN114926456A publication Critical patent/CN114926456A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/0002—Inspection of images, e.g. flaw detection
    • G06T7/0004—Industrial image inspection
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/20—Image preprocessing
    • G06V10/25—Determination of region of interest [ROI] or a volume of interest [VOI]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/20—Image preprocessing
    • G06V10/26—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/20—Image preprocessing
    • G06V10/28—Quantising the image, e.g. histogram thresholding for discrimination between background and foreground patterns
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/764—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
    • G06V10/774—Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10048—Infrared image
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20048—Transform domain processing
    • G06T2207/20061—Hough transform
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V2201/00—Indexing scheme relating to image or video recognition or understanding
    • G06V2201/07—Target detection

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Multimedia (AREA)
  • Evolutionary Computation (AREA)
  • Computing Systems (AREA)
  • Health & Medical Sciences (AREA)
  • Artificial Intelligence (AREA)
  • Software Systems (AREA)
  • General Health & Medical Sciences (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Databases & Information Systems (AREA)
  • Medical Informatics (AREA)
  • Molecular Biology (AREA)
  • General Engineering & Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Mathematical Physics (AREA)
  • Computational Linguistics (AREA)
  • Biophysics (AREA)
  • Biomedical Technology (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Quality & Reliability (AREA)
  • Image Processing (AREA)

Abstract

The invention discloses a rail foreign matter detection method with semi-automatic labeling and improved deep learning, which comprises the following steps: semi-automatically labeling the data set based on the thermal infrared human body segmentation result; aiming at the outdoor all-weather environment of the rail, the ResNet50 neural network is improved to finish the identification of different environment scenes; according to the scene recognition result, improving the Yolov5 neural network to recognize the rail foreign matter; and extracting the rail outline of the picture, designing an alarm strategy aiming at rail foreign matter invasion by combining the foreign matter position, and performing graded alarm judgment. By introducing scene classification of deep learning and combining with an improved target identification network, the rail foreign matter identification method improves the rail foreign matter identification effect under different environmental scenes, improves the accuracy of warning of foreign matter invading the rail, and reduces the possibility of train accidents caused by the invasion of foreign matter into the rail by combining with a scientific warning strategy.

Description

Rail foreign matter detection method based on semi-automatic labeling and improved deep learning
The technical field is as follows:
the invention relates to the field of computer vision and pattern recognition and Intelligent Transportation Systems (ITS), in particular to a rail foreign matter detection method with semi-automatic labeling and improved deep learning.
Background art:
the rail foreign matter intrusion detection alarm system in the railway driving auxiliary system is an important component of a railway intelligent traffic system, and the research theme directly comes from social and market demands, so that the rail foreign matter intrusion detection alarm system has a good application prospect and a good market value. According to the '2019 railway safety condition announcement', the number of specific fatalities of railway traffic accidents from 2016 to 2019 is 932, 898, 857 and 788 people in sequence (the '2019 railway safety condition announcement' of the national railway administration).
Although the railway adopts the modes of arranging an isolation wall, an isolation steel wire mesh, manual patrol and the like to isolate the interference of the outside world on the normal operation of the railway at present, railway operation accidents caused by the invasion of foreign matters on the track still often occur due to the fact that a plurality of local persons are not enough and are unattended and railway workers need to patrol frequently at night. The traditional detection method needs to install a plurality of cameras on the rail along the line, and is high in cost and poor in real-time performance.
In the traditional detection method, the end appreciation of the China railway science research institute (end appreciation, high-speed train operation environment monitoring wireless sensor network research [ D ]. China railway science research institute, 2017.) proposes to improve the existing railway monitoring equipment, add a wireless sending module, realize wireless transmission of information, simultaneously can receive monitoring information on the ground, and perform processing, storage and display, thereby effectively solving the problems that the wired network has high wiring cost and cannot be flexibly detected. The application research of a three-dimensional laser radar image in railway foreign object intrusion detection [ D ] of northern industry university, 2018.) of northern industry university provides that a three-dimensional laser radar is used for railway foreign object intrusion detection, a three-dimensional laser radar system is formed by combining a GL-1025 laser radar and a holder system for information acquisition, then format conversion, three-dimensional modeling and other operations are carried out on three-dimensional point cloud data, finally, an OTSU algorithm, a K mean value clustering and a DBSCAN clustering are adopted for target detection, and finally, an experimental result shows that the algorithm can automatically classify and detect targets, and meanwhile, the better detection effect is achieved in different scenes. However, due to the huge point cloud data and the requirement on hardware, it is difficult to complete real-time detection. An active track state monitoring system (Chinese patent grant publication No. CN113484420A, grant publication date: 2021.10.08) is additionally provided with the active track state monitoring system for a track, and a railway power grid inspection robot (Chinese patent grant publication No. CN114194231A, grant publication date: 2022.03.18) adopts a robot inspection mode, so that a new innovation direction is provided for rail foreign matter detection.
In the technical method based on deep learning, Wang Lei (Wang Lei, railway traffic safety-oriented high-precision pedestrian detection algorithm research and system design [ D ]. Beijing university of transportation, 2019.) provides a small-scale pedestrian detection algorithm, and the algorithm optimizes a network structure from weak semantic segmentation loss functions, channel radial transformation and feature fusion based on an Faster R-CNN network. Liuhui et al (Liuhui, zhangshishuai, shengjue, etc.. orchard pedestrian real-time detection method based on improved SSD [ J ]. agricultural machinery report, 2019,50(4):8.) propose a pedestrian real-time monitoring algorithm, and introduce void convolution into SSD to enlarge the receptive field of a convolutional neural network. Rainbow et al (Rainbow, Fieldy, easy, etc.. pedestrian detection based on deep convolutional neural network [ J ] computer engineering and applications, 2016,52(13):5.) propose an algorithm for detecting pedestrians by constructing convolutional neural network with a multilayer network structure, which has higher efficiency for pedestrian detection. Wanning (wanning, a railway intrusion pedestrian classification algorithm [ D ] based on feature fusion. beijing university of transportation) proposes a railway pedestrian classification algorithm with feature fusion, i.e., fusion of convolutional neural network features and manual pedestrian features. The method for detecting foreign matter intrusion in real time by combining a traditional machine learning algorithm to perform basic processing on an image, demarcating an intrusion area, performing multiple times of training by using a deep learning Yolo-V3 network model, detecting the demarcated intrusion area through a training model and detecting whether foreign matter exists in the detection area is possible, but the method needs to install a plurality of camera devices along the line, is high in cost, insufficient in maneuverability and does not fully exert the advantage of the real-time property of the Yolo 3 algorithm.
In summary, although the foreign object detection based on the deep learning technology method has been successful in other fields, there is still room for further improvement in the detection rate, detection accuracy, detection efficiency and robustness of the subdivided field of the railway intelligent transportation system.
The invention content is as follows:
the embodiment of the invention aims to provide a rail foreign matter detection method with semi-automatic labeling and improved deep learning, and aims to solve the problems that the existing rail foreign matter detection method is low in instantaneity and robustness and is difficult to cope with rail foreign matter invasion conditions in various complex environments.
A rail foreign matter detection method capable of semi-automatically labeling and improving deep learning comprises the following steps:
(1) a human body segmentation algorithm based on thermal infrared is provided, the segmentation result is mapped into a visible light image, a human body labeling frame of visible light is rapidly obtained, and a Yolov5 training data set is established;
(2) providing a scene classification algorithm based on an improved ResNet50 model, and guiding to execute human body identification methods of human body identification models in different scenes;
(3) training a classification model offline by using a target detection algorithm of an improved YOLOv5 model, and carrying out target detection on rail foreign matters to realize foreign matter detection;
(4) a rail contour extraction algorithm based on a traditional algorithm and improved deep learning is provided, rail segmentation is carried out on a single picture, and a contour is extracted;
(5) and providing a foreign matter alarm strategy algorithm according to the rail segmentation result and the foreign matter position, guiding and studying and judging the alarm grade of the whole picture, and realizing alarm grade judgment.
Further, the visible light human body labeling frame obtained quickly in the step (1) is obtained through the following steps: by selecting a thermal infrared camera and a visible light camera with consistent focal lengths, recording thermal infrared and visible light videos simultaneously in a scene with a scene temperature lower than 25 ℃; on the basis, by means of the characteristic that the thermal infrared human body is easily segmented by an otsu algorithm at the temperature of below 25 ℃, the segmentation of the thermal infrared human body image is automatically completed by the otsu algorithm, and then the segmentation result is mapped into the visible light image, so that a large number of visible light human body labeling frame results are directly obtained; and finally, carrying out a small amount of manual correction on results of a small number of possible segmentation errors, semi-automatically acquiring a visible human body labeling frame, and establishing a YOLOv5 training data set.
Further, the scene classification algorithm of the improved ResNet50 model in step (2) is as follows: aiming at the ResNet50 scene classification model, a transfer learning mechanism training model is introduced, and the weight of a model pre-trained on ImageNet is transferred to the model so as to accelerate the convergence of the network model; and (2) specifically, the Relu function is changed into the LRelu function in the selection of the activation function, so that the gradient is prevented from disappearing.
Further, the target detection algorithm of the improved YOLOv5 model in step (3) is as follows: aiming at a YOLOv5 target detection model, after data are obtained, Mosaic data enhancement and real-time data loading are adopted, a boundary box prediction loss function is improved, and the position precision and the classification precision are improved by utilizing a model of a GT target boundary box UID distributor, a GT target boundary box UID matcher, a boundary box position and classification loss weight algorithm; step (3) in the aspect of collecting combined target features, a convolution attention mechanism is added, for a prediction part, on the basis of a traditional sigmoid function sigma (x), an h-swish function is introduced into a MobileNet v3 network, and the CIOU _ Loss of a Loss function originally used in a Bounding box is changed into GIOU _ Loss; and finally, a depsort tracking module is added into a detection module of the YOLOv5 algorithm to prevent missing detection, so that the robustness of detecting foreign matters by the system is improved.
Further, the track line detection method based on the conventional algorithm in step (4) comprises: extracting the edge characteristics of the railway track by using Hough change for the straight line type rail; for a curve type rail, firstly segmenting the rail to obtain a segmented linear rail, then extracting the edge characteristics of the rail by utilizing Hough, then combining the characteristics to obtain the edge extraction of the curve rail, and providing a rail line contour segmentation algorithm suitable for four complex environments of different types, namely a straight line, a curve, a multi-rail straight line and a multi-rail curve by combining the two; and the rail contour extraction algorithm based on the improved deep learning in the step (4) is as follows: extracting the rail outline by adopting a deep learning algorithm based on a DeepLabv3+ semantic segmentation network; and finally, carrying out contour taking and set combining processing on the data results obtained by the two models, and leaving the common part of the two models as the finally extracted track contour.
Further, the foreign matter alarm strategy algorithm in the step (5) is as follows: and (4) obtaining corresponding coordinates of the track profile of the single picture by using the step (4), obtaining coordinates of foreign matters of the single picture by using the step (3), performing weighted judgment on the invasion condition of the foreign matters in the whole picture according to a proposed self-defined function and algorithm, and finally normalizing the judgment into the alarm grade of the whole picture.
Compared with the prior art for detecting the foreign matters on the track, the invention has the following advantages and effects: according to the method, the visible light pedestrian marking frame data set can be semi-automatically obtained in the data preprocessing stage, and the marking precision is further manually improved in the later review process; according to the method in the step (2), aiming at rail outdoor scenes with large differences of common illumination, weather and the like, the method introduces a scene classification algorithm based on improved deep learning, guides a subsequent human body detection model to finish high-precision foreign matter identification according to the scene classification result, and solves the problem that the model is difficult to adapt to complex environments in different scenes; after the target detection algorithm based on the improved YOLOv5 model in the step (3) obtains data, a loss function is improved, the position precision and the classification precision are improved, a convolution attention mechanism is added in the aspect of collecting combined target features, besides, a tracking module is added to prevent system omission, and the problem that the robustness of a track foreign matter detection system is not strong, and the traditional YOLOv5 algorithm framework is not accurate and insufficient in detection aiming at certain specific environmental conditions is finally solved; the rail outline extraction algorithm based on the traditional algorithm and the improved deep learning in the step (4) combines the advantages of the traditional rail outline segmentation method and the rail outline segmentation method based on the deep learning, specifically adopts the improved Hough transformation algorithm to detect the rail line and segment the rail outline, and uses the deep learning algorithm based on the deep Labv3+ semantic segmentation network to perform the semantic segmentation of the rail line and extract the rail outline, and then performs outline extraction and set processing on data results obtained by the two, so as to finally improve the accuracy of extracting the rail outline; the foreign body alarm strategy algorithm designed in the step (5) can guide and study the alarm grade of the whole picture, obtain the linear alarm grade according to the self-defined function, realize accurate judgment and processing of rail foreign body invasion, greatly reduce the occurrence of false touch alarm and excessive alarm conditions, and reduce the possibility of accidents.
The invention has the other advantage that the hardware condition of the system is based on the vehicle-mounted camera, and compared with the traditional track foreign matter detection method that static cameras are installed along the track in a segmented manner for continuously monitoring the track in real time for 24 hours all day, the method has stronger economic applicability and higher fault-tolerant rate, greatly improves the track foreign matter intrusion detection efficiency and reduces the possibility of train accidents caused by foreign matter intruding into the track.
Description of the drawings:
fig. 1 is a general flowchart of a rail foreign object detection method with semi-automatic labeling and improved deep learning according to an embodiment of the present invention.
Fig. 2 is a method for establishing a YOLOv5 training data set by mapping a segmentation result to a visible light image, quickly obtaining a visible light human body labeling box, according to the human body segmentation algorithm based on thermal infrared provided by the embodiment of the present invention.
Fig. 3 is a scene classification algorithm based on an improved ResNet50 model according to an embodiment of the present invention, which is used for guiding execution of a human body recognition method of a human body recognition model in different scenes.
Fig. 4 is a diagram of a network model architecture of YOLOv 5.
Fig. 5 is a diagram of an embodiment of a rail profile extraction algorithm based on a conventional algorithm and improved deep learning provided by an embodiment of the present invention.
The specific implementation mode is as follows:
in order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is further described in detail with reference to the following embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
The application of the principles of the present invention will be further described with reference to the accompanying drawings and specific embodiments.
The overall process of the invention is shown in fig. 1, and the rail foreign matter detection method with semi-automatic labeling and improved deep learning of the embodiment of the invention comprises the following steps:
s1: by selecting a thermal infrared camera and a visible light camera with consistent focal lengths, recording thermal infrared and visible light videos simultaneously in a scene with a scene temperature lower than 25 ℃; on the basis, by means of the characteristic that the thermal infrared human body is easily segmented by an otsu algorithm at the temperature of below 25 ℃, the segmentation of the thermal infrared human body image is automatically completed by the otsu algorithm, and then the segmentation result is mapped into the visible light image, so that a large number of visible light human body labeling frame results are directly obtained; and finally, carrying out a small amount of manual correction on results of a small number of possible segmentation errors, semi-automatically acquiring a visible human body labeling frame, and establishing a YOLOv5 training data set.
S2: dividing the training data set into various environments according to the environment scene of the picture and labeling the environments, wherein the environments comprise six categories of sunny days, cloudy days, rainy days, foggy days, snowy days and dark night, and then performing off-line training on the labeled data set by utilizing the proposed improved depth residual error network ResNet50 classification algorithm to obtain a ResNet50 model capable of identifying the picture environment; classifying the YOLOv5 training data set established in S1 by using the ResNet50 model to obtain various training data sets based on different environmental scenes; the improved ResNet50 classification algorithm comprises the following steps: firstly, introducing a transfer learning mechanism training model, and transferring the weight of a pre-trained model on ImageNet into the model to accelerate the convergence of the network model; secondly, the Relu function is changed into the LRelu function in the selection of the activation function, so that the gradient of the deep network is prevented from disappearing.
S3: then, by using an improved YOLOv 5-based target detection algorithm, performing offline training on the multiple training data sets obtained in S2 based on different environmental scenes separately to obtain an improved YOLOv5 model based on different environmental scenes; the target detection algorithm based on the improved YOLOv5 comprises the following steps: after the data are obtained, the method adopts the Mosaic data enhancement and the real-time data loading data to improve the prediction loss function of the boundary box, and improves the position precision and the classification precision by utilizing a model of a GT target boundary box UID distributor, a GT target boundary box UID matcher and a boundary box position and classification loss weight algorithm; in the aspect of collecting the combined target features, a convolution attention mechanism is added; finally, in a prediction part, on the basis of a traditional sigmoid function sigma (x), an h-swish function is introduced into MobileNet 3, and the CIOU _ Loss of the Loss function originally used in a Bounding box is changed into GIOU _ Loss; and finally, a depsort tracking module is added into a detection module of the YOLOv5 algorithm to prevent missing detection, so that the robustness of detecting foreign matters by the system is improved.
S4: the model obtained through the training detects whether foreign matter invades the limit in the process of the train running scene, and meanwhile, a new rail contour extraction algorithm based on improved deep learning is provided for rail segmentation and contour extraction. In the traditional track line detection method, the preprocessed track line image is detected by using Hough transformation, track lines possibly existing in the image can be roughly positioned, then the track line position is obtained by screening according to the characteristics of the track lines, and finally the track outline is extracted by segmentation; the deep learning-based rail contour extraction method adopts a DeepLabv3+ semantic segmentation network to perform rail line semantic segmentation so as to extract the rail contour. The method is characterized in that a traditional rail outline extraction method and a deep learning-based rail outline extraction method are improved, and the method specifically comprises the steps of carrying out outline extraction and set processing on data results obtained by two models, leaving a common part of the two models as a rail outline finally extracted by the user, so that the common part is used for alarm study and judgment more accurately, and transmitting results obtained by running the two models into an alarm strategy model after the rail outline is extracted.
S5: carrying out alarm study and judgment on the incoming result by utilizing the proposed alarm strategy model; the proposed alarm strategy model algorithm is as follows: performing weighted fusion on the upper-layer input features according to foreign matter detection results and rail outline extraction results transmitted from S3 and S4, and finally normalizing the upper-layer input features into the alarm level of a single picture; the principle of the alarm strategy model is as follows: and performing feature extraction on the input result of the foreign matter detection to obtain picture absolute coordinate vectors of the positions of the n foreign matters.
The absolute coordinate vector of the picture of the foreign body position is shown as (1), wherein x i Is the abscissa, y, of the foreign body in the picture i Is the ordinate of the foreign matter in the picture.
A i ＝(x i ，y i )，i∈{1，2，......，n} (1)
According to the result, the feature extraction is carried out on the input result of the semantic segmentation, and the rail pixel width absolute value of the ith row of pixels is obtained, and the expression is shown as (2).
Figure BDA0003692384050000091
Wherein W i1 ，W i2 For the semantic rail boundary at the ordinate y i The abscissa value of (a).
And (4) obtaining the distance of the foreign matters relative to the semantic boundary of the rail, wherein the expression is shown as (3).
Figure BDA0003692384050000092
The alarm determination value expression is shown in (4).
Figure BDA0003692384050000093
And when P is more than or equal to 0.5, alarming.
S6: and judging whether the motor car needs to give an alarm or not according to the output result of the alarm strategy model of S5.
As shown in fig. 2, the specific process of S1 includes:
s11: and selecting thermal infrared cameras and visible light cameras with consistent focal lengths to be deployed on the train.
S12: and simultaneously recording thermal infrared videos and visible light videos in the scene with the scene temperature lower than 25 ℃.
S13: and processing the video according to a frame extraction method by using the video stream recorded and collected in the step S12, wherein the time interval is 1S, namely, one picture is extracted from the video every 1S, and particularly, for the image with the human body foreign matter, the images collected by the two cameras are subjected to frame extraction processing while time sequence matching is performed by using a double thread.
S14: on the basis, by means of the characteristic that the thermal infrared human body is easily segmented by an otsu algorithm at the temperature of below 25 ℃, the segmentation of the thermal infrared human body image is automatically completed by the otsu algorithm.
S15: and mapping the segmentation result of the thermal infrared human body image into the visible light image so as to directly obtain a large amount of visible light human body labeling frame results.
S16: and carrying out a small amount of manual correction on results of a small amount of possible segmentation errors, and semi-automatically acquiring a visible human body marking frame.
S17: according to the result of the visible light human body labeling frame semi-automatically acquired in the step S15, according to the following steps: 1: 1 is divided into a training set, a testing set and a verification set, and a YOLOv5 training data set is finally established.
As shown in fig. 3, the specific process of S2 includes:
s21: firstly, transferring a loaded pre-training model to a scene classification model, then labeling a training data set established in S17, inputting the labeled training data set into an improved ResNet50 network for training, training and evaluating the performance of the model by using a cross validation method, and finally obtaining an optimal classifier capable of identifying the environmental scene where the training data set picture is located.
S22: in the aspect of transfer learning, the network weight pre-trained in ImageNet data set by the ResNet50 network needs to be acquired, and the command wget is usedhttps://download.pytorch.org/models/The rescet 50-19c8e357.pth downloads the pre-training model, and the model pre-training weights are loaded using the torch _ load _ state _ dit (torch _ load ("/rescet 50-pre.pth"), strict ═ False).
S23: in terms of activating the function, in order to avoid the problem of gradient disappearance, the Relu function is changed into an LRelu function, which is shown in (5), so as to ensure that the network can be converged well at a deeper layer.
Figure BDA0003692384050000101
S24: in the aspect of model training, the model is trained by taking batch _ size as 64 and epoch as 20, so as to obtain a classifier capable of identifying the environment scene where the training data set picture is located.
S25: according to the classifier, environment-scene-based classification is carried out on the established YOLOv5 training data set in S17, and finally a plurality of YOLOv5 training data sets based on different environment scenes are obtained.
As shown in fig. 4, the specific process of S3 includes:
s31: according to the multiple training data sets obtained in S25 based on different environmental scenes, for different environmental scenes, training an improved YOLOv5 model based on the environmental conditions by using the different environmental scene training data sets, and obtaining the rail foreign matter detection algorithm based on the improved YOLOv5 network under different environmental scene conditions.
S32: in the data loader, the input end of YOLOv5 adopts Mosaic data enhancement, a parameter of True is set in create _ dataloader, whether Mosaic in hyp.scratch.yaml is set to 1.0 is checked, original information is kept, and after a new picture is detected to be input each time, the purpose of real-time data enhancement is realized by adjusting the brightness of the picture and using a Mosaic method to enhance the Mosaic data. And splicing the 4 pictures in a random zooming, random cutting and random arrangement mode.
S33: in the aspect of collecting combined target features, a convolution attention mechanism is added, an attention mechanism Modified SAM is applied in YOLOv4, but a relevant attention mechanism is not applied in YOLOv5, so a convolution attention mechanism module CBAM is added to a feature extraction network, an accumulation function corresponding to a standard convolution layer and a CBAM module is added to commom.
S34: and (3) improving a boundary frame prediction Loss function, and utilizing a GT target boundary frame UID distributor, a GT target boundary frame UID matcher, a boundary frame position and classification Loss weight algorithm model to further improve the position precision and the classification precision, wherein the CIOU _ Loss of the Loss function originally used in the Bounding box is changed into the GIOU _ Loss, and the specific transformation is shown as (6) and (7).
Figure BDA0003692384050000111
Figure BDA0003692384050000112
S35: in terms of prediction, the traditional sigmoid function sigma (x) is very consuming in computing resources at a mobile terminal, while the Relu6 function has easy quantitative deployment, so that the speed can be increased while the precision is kept by introducing an h-swish function on the basis of the traditional sigmoid function sigma (x), wherein the h-swish function is shown as (8).
Figure BDA0003692384050000121
S36: and finally, adding a depsort tracking module into a detection module code of the YOLOv5 algorithm to prevent missing detection, and adding the appearance information of the target into the calculation of interframe matching, thereby improving the robustness of detecting foreign matters by the system.
As shown in fig. 5, the specific process of S4 includes:
s41: the traditional track line segmentation method is improved: extracting the edge characteristics of the railway track by using Hough transformation for the straight-line rail; for a curve type rail, segmenting the rail to obtain a segmented linear rail, extracting the edge characteristics of the rail by utilizing Hough transformation, and combining the characteristics to obtain the edge extraction of the curve rail; finally, a track line contour segmentation algorithm suitable for four different types of complex environments, namely straight lines, curves, multi-track straight lines and multi-track curves, is provided by combining the two algorithms.
S42: since a natural scene contains a large number of irrelevant objects, most of which have straight line components and can greatly affect the identification of a track region, an ROI (region of interest) region needs to be divided in an image, and the identification of a rail in the ROI region can greatly improve the identification accuracy and eliminate the interference of irrelevant factors. Using a cv2.setMouseCallback function, and obtaining coordinates of pixel points by clicking the pixel points in an image; the ROI area is rectangular, and when the ROI area is selected, only the coordinates of pixel points of two end points of a diagonal line of the rectangle need to be selected, corresponding coordinate values can be returned, and the ROI area is drawn through the coordinate values.
S43: before Hough transformation, a part of preprocessing flow is often included to enhance the image edge, and the preprocessing of the algorithm comprises three stages of gray value conversion, histogram equalization and Gaussian filtering. The GRAY value conversion phase we use the cv2.cvtcolor function with the parameters cv2.color _ RGB2GRAY to convert the original three-channel RGB color image into a GRAY scale image.
S44: after the image is converted into a single-channel gray scale image, histogram equalization is performed on the image to enhance contrast. Histogram equalization enhances image contrast by stretching the pixel intensity distribution range, the equalized image is only approximately uniformly distributed, and the dynamic range expansion of the image essentially expands the quantization interval and reduces the quantization level, with distinct boundaries between regions. Histogram equalization is carried out on the gray level image by using a cv2.equalizehist function, and for a natural scene under a normal illumination condition, the image contrast is always improved after the histogram equalization, so that the target edge detection is facilitated.
S45: and carrying out Gaussian filtering processing on the image, carrying out weighted average on the whole image by Gaussian filtering, and eliminating Gaussian noise in the image, wherein the value of each pixel point is obtained by carrying out weighted average on the pixel point and other pixel values in the field. The specific operation of gaussian filtering is: each pixel in the image is scanned with a gaussian kernel (also known as convolution, template) and the weighted average gray value of the pixel in the domain determined by the template is used to replace the value of the central pixel point of the template. We apply gaussian filtering processing to the image using the cv2.gaussian kernel function and define the gaussian kernel size by itself.
S46: and carrying out Hough transformation processing on the preprocessed pictures and extracting the rail outline. The Hough transformation principle is to transform a curve (including a straight line) in an image space into a parameter space, and determine a description parameter of the curve by detecting an extreme point in the parameter space, thereby extracting a regular curve in the image. A polar coordinate system is adopted for representation in Hough linear detection, and a linear polar coordinate expression is shown as (9).
ρ＝xcosθ+ysinθ (9)
In expression (9): ρ is the distance from the origin to the straight line, θ is the angle between the straight line and the y-axis, and (x, y) is the coordinate of a point on the straight line.
And (4) respectively taking each point coordinate in the coordinate system into the formula, respectively calculating the size of rho for each given theta, and counting the rho with the same size, thereby determining a straight line.
S47: the deep learning-based rail contour extraction method adopts a deep learning algorithm based on improved semantic segmentation to extract the rail contour. The specific process is as follows: and acquiring a rail image acquired in the running process of the train by using a vehicle-mounted camera, and sending the rail image into a DeepLabv3+ semantic segmentation network trained in advance by the system for semantic segmentation of the rail line.
S48: and carrying out perspective transformation on the binary image only containing the track line obtained by segmentation to obtain an aerial view, screening effective track line pixel points, carrying out polynomial fitting on the effective track line points by using a least square method, outputting polynomial fitting coefficients of left and right track lines, and determining and segmenting the rail outline according to the coefficients.
S49: and finally, carrying out contour extraction and collective processing on the data results obtained by the two models, and leaving the common parts of the two models as the rail contours extracted by segmentation for alarm study and subsequent data processing stages.
The specific processes of S5 and S6 are as follows:
transmitting the foreign matter detection result and the rail semantic segmentation result into an alarm strategy model, and outputting whether an alarm is needed or not according to the alarm strategy model; if the model judges that alarming is needed, alarming is carried out on the background, foreign matters are framed by a red frame in a foreground video frame, and operations such as motor car alarming, emergency braking and the like are triggered; if the model judges that the alarm is not needed, the foreign matters needing to be noticed are framed by other color frames in the foreground monitoring video stream.

Claims (6)

1. A rail foreign matter detection method capable of realizing semi-automatic labeling and improving deep learning is characterized by comprising the following steps:
(1) a human body segmentation algorithm based on thermal infrared is provided, the segmentation result is mapped into a visible light image, a human body labeling frame of visible light is rapidly obtained, and a Yolov5 training data set is established;
(2) providing a scene classification algorithm based on an improved ResNet50 model, and guiding to execute human body identification methods of human body identification models in different scenes;
(3) training a classification model offline by using a target detection algorithm of an improved YOLOv5 model, and carrying out target detection on rail foreign matters to realize foreign matter detection;
(4) a rail contour extraction algorithm based on a traditional algorithm and improved deep learning is provided, rail segmentation is carried out on a single picture, and a contour is extracted;
(5) and providing a foreign matter alarm strategy algorithm according to the rail segmentation result and the foreign matter position, guiding and studying and judging the alarm grade of the whole picture, and realizing alarm grade judgment.
2. The method for detecting foreign matters on a railway rail with semi-automatic labeling and improved deep learning according to claim 1, wherein the visible light human body labeling frame obtained in the step (1) is obtained by: by selecting a thermal infrared camera and a visible light camera with consistent focal lengths, thermal infrared and visible light videos are recorded simultaneously in a scene with the scene temperature lower than 25 ℃; on the basis, by means of the characteristic that the thermal infrared human body is easily segmented by an otsu algorithm at the temperature of below 25 ℃, the segmentation of the thermal infrared image is automatically completed by the otsu algorithm, and then the segmentation result is mapped into the visible light image, so that a large amount of visible light human body labeling frame results are directly obtained; and finally, carrying out a small amount of manual correction on results of a small number of possible segmentation errors, semi-automatically acquiring a visible human body labeling frame, and establishing a YOLOv5 training data set.
3. The method for semi-automatically labeling and improving deep learning of rail alien substances according to claim 1, wherein the scene classification algorithm of the improved ResNet50 model in the step (2) is as follows: aiming at the ResNet50 scene classification model, introducing a transfer learning mechanism training model, and transferring the weight of a model pre-trained on ImageNet into the model to accelerate the convergence of the network model; secondly, the Relu function is changed to the LRelu function in the selection of the activation function, so as to prevent the gradient from disappearing.
4. The method for detecting a rail alien substance according to claim 1, wherein the target detection algorithm of the improved YOLOv5 model in step (3) is as follows: aiming at a YOLOv5 target detection model, after data are obtained, Mosaic data enhancement and real-time data loading are adopted to improve a boundary box prediction loss function, and the position precision and the classification precision are improved by utilizing a model composition of a GT target boundary box UID distributor, a GT target boundary box UID matcher and a boundary box position and classification loss weight algorithm; in the aspect of collecting the combined target features, a convolution attention mechanism is added; finally, in a prediction part, on the basis of a traditional sigmoid function sigma (x), an h-swish function is introduced into the MobileNet v3, and the CIOU _ Loss of the Loss function originally used in the Bounding box is changed into the GIOU _ Loss; and finally, a depsort tracking module is added in the detection of the YOLOv5 algorithm to prevent missing detection, so that the robustness of detecting foreign matters by the system is improved.
5. The method for semi-automatically labeling and improving deep learning of rail alien substances according to claim 1, wherein the step (4) of the conventional algorithm and the improved deep learning-based rail profile extraction algorithm is as follows: the traditional rail contour segmentation method and the deep learning-based rail contour segmentation method are improved, and the data results obtained by the two models are subjected to contour extraction and set processing, and the common part of the two models is left as the finally extracted rail contour.
6. A method for semi-automatically labeling and improving deep learning rail alien material detection according to claim 1, wherein the alien material alarm strategy algorithm of step (5) is: and (4) obtaining corresponding coordinates of the track outline of the single picture, obtaining foreign body coordinates of the single picture by using the step (3), performing weighted judgment on the whole picture for the invasion situation of the foreign bodies according to the proposed self-defined function and algorithm, and finally normalizing the judgment into the alarm level of the whole picture.
CN202210669222.7A 2022-06-14 2022-06-14 Rail foreign matter detection method based on semi-automatic labeling and improved deep learning Pending CN114926456A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202210669222.7A CN114926456A (en) 2022-06-14 2022-06-14 Rail foreign matter detection method based on semi-automatic labeling and improved deep learning

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202210669222.7A CN114926456A (en) 2022-06-14 2022-06-14 Rail foreign matter detection method based on semi-automatic labeling and improved deep learning

Publications (1)

Publication Number Publication Date
CN114926456A true CN114926456A (en) 2022-08-19

Family

ID=82814395

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202210669222.7A Pending CN114926456A (en) 2022-06-14 2022-06-14 Rail foreign matter detection method based on semi-automatic labeling and improved deep learning

Country Status (1)

Country Link
CN (1) CN114926456A (en)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116229388A (en) * 2023-03-27 2023-06-06 哈尔滨市科佳通用机电股份有限公司 Method, system and equipment for detecting motor car foreign matters based on target detection network
CN117079245A (en) * 2023-07-05 2023-11-17 浙江工业大学 Traffic road target identification method based on wireless signals
WO2024179388A1 (en) * 2023-02-28 2024-09-06 深圳市绿洲光生物技术有限公司 Plankton object detection and classification method based on multi-layer neural network architecture

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2024179388A1 (en) * 2023-02-28 2024-09-06 深圳市绿洲光生物技术有限公司 Plankton object detection and classification method based on multi-layer neural network architecture
CN116229388A (en) * 2023-03-27 2023-06-06 哈尔滨市科佳通用机电股份有限公司 Method, system and equipment for detecting motor car foreign matters based on target detection network
CN116229388B (en) * 2023-03-27 2023-09-12 哈尔滨市科佳通用机电股份有限公司 Method, system and equipment for detecting motor car foreign matters based on target detection network
CN117079245A (en) * 2023-07-05 2023-11-17 浙江工业大学 Traffic road target identification method based on wireless signals

Similar Documents

Publication Publication Date Title
CN108171112B (en) Vehicle identification and tracking method based on convolutional neural network
CN109657581B (en) Urban rail transit gate traffic control method based on binocular camera behavior detection
CN108492319B (en) Moving target detection method based on deep full convolution neural network
CN112101175A (en) Expressway vehicle detection and multi-attribute feature extraction method based on local images
CN114926456A (en) Rail foreign matter detection method based on semi-automatic labeling and improved deep learning
CN103093249B (en) A kind of taxi identification method based on HD video and system
CN111611874B (en) Face mask wearing detection method based on ResNet and Canny
TW202013252A (en) License plate recognition system and license plate recognition method
CN107253485A (en) Foreign matter invades detection method and foreign matter intrusion detection means
CN111008632B (en) License plate character segmentation method based on deep learning
CN104134068B (en) Monitoring vehicle feature representation and classification method based on sparse coding
CN113888754B (en) Vehicle multi-attribute identification method based on radar vision fusion
CN110033431A (en) Non-contact detection device and detection method for detecting corrosion area on surface of steel bridge
Naufal et al. Preprocessed mask RCNN for parking space detection in smart parking systems
CN111915583A (en) Vehicle and pedestrian detection method based on vehicle-mounted thermal infrared imager in complex scene
Tao et al. Smoky vehicle detection based on range filtering on three orthogonal planes and motion orientation histogram
Lashkov et al. Edge-computing-facilitated nighttime vehicle detection investigations with CLAHE-enhanced images
Kheder et al. Transfer learning based traffic light detection and recognition using CNN inception-V3 model
CN112069997B (en) Unmanned aerial vehicle autonomous landing target extraction method and device based on DenseHR-Net
CN117745709A (en) Railway foreign matter intrusion detection method, system, equipment and medium
CN115147450B (en) Moving target detection method and detection device based on motion frame difference image
CN112115767B (en) Tunnel foreign matter detection method based on Retinex and YOLOv3 models
Mo et al. Research on expressway traffic event detection at night based on Mask-SpyNet
CN113052118A (en) Method, system, device, processor and storage medium for realizing scene change video analysis and detection based on high-speed dome camera
CN113191255A (en) Traffic sign identification method based on mobile robot

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination