CN116626177A - Rail damage identification method and device - Google Patents
Rail damage identification method and device Download PDFInfo
- Publication number
- CN116626177A CN116626177A CN202310498362.7A CN202310498362A CN116626177A CN 116626177 A CN116626177 A CN 116626177A CN 202310498362 A CN202310498362 A CN 202310498362A CN 116626177 A CN116626177 A CN 116626177A
- Authority
- CN
- China
- Prior art keywords
- rail
- model
- damage
- rail damage
- yolov7
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 230000006378 damage Effects 0.000 title claims abstract description 212
- 238000000034 method Methods 0.000 title claims abstract description 42
- 238000001514 detection method Methods 0.000 claims abstract description 118
- 229910000831 Steel Inorganic materials 0.000 claims abstract description 77
- 239000010959 steel Substances 0.000 claims abstract description 77
- 238000012549 training Methods 0.000 claims abstract description 56
- 238000012360 testing method Methods 0.000 claims abstract description 52
- 238000010586 diagram Methods 0.000 claims abstract description 39
- 238000007703 enzymatic ligation assisted by nucleases technique Methods 0.000 claims abstract description 39
- 230000007246 mechanism Effects 0.000 claims description 41
- 238000004590 computer program Methods 0.000 claims description 27
- 238000004422 calculation algorithm Methods 0.000 claims description 13
- 238000007781 pre-processing Methods 0.000 claims description 11
- 238000003860 storage Methods 0.000 claims description 10
- 238000012545 processing Methods 0.000 claims description 9
- 230000002776 aggregation Effects 0.000 claims description 7
- 238000004220 aggregation Methods 0.000 claims description 7
- 238000002372 labelling Methods 0.000 claims description 6
- 238000003064 k means clustering Methods 0.000 claims description 5
- 210000001503 joint Anatomy 0.000 claims description 4
- 238000010276 construction Methods 0.000 claims description 3
- 238000013480 data collection Methods 0.000 claims description 3
- 238000002604 ultrasonography Methods 0.000 claims description 2
- 238000009659 non-destructive testing Methods 0.000 abstract description 2
- 230000000694 effects Effects 0.000 description 10
- 230000004927 fusion Effects 0.000 description 7
- 230000006872 improvement Effects 0.000 description 6
- 239000000523 sample Substances 0.000 description 6
- 230000006870 function Effects 0.000 description 5
- 238000003062 neural network model Methods 0.000 description 5
- 238000004458 analytical method Methods 0.000 description 4
- 238000013135 deep learning Methods 0.000 description 4
- 230000008569 process Effects 0.000 description 4
- 238000011176 pooling Methods 0.000 description 3
- 230000008901 benefit Effects 0.000 description 2
- 238000004364 calculation method Methods 0.000 description 2
- 238000013527 convolutional neural network Methods 0.000 description 2
- 238000009826 distribution Methods 0.000 description 2
- 239000004973 liquid crystal related substance Substances 0.000 description 2
- 238000013468 resource allocation Methods 0.000 description 2
- 238000000528 statistical test Methods 0.000 description 2
- 230000002123 temporal effect Effects 0.000 description 2
- 238000012795 verification Methods 0.000 description 2
- 241000282414 Homo sapiens Species 0.000 description 1
- 208000027418 Wounds and injury Diseases 0.000 description 1
- 230000003044 adaptive effect Effects 0.000 description 1
- 230000009901 attention process Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 230000019771 cognition Effects 0.000 description 1
- 230000000295 complement effect Effects 0.000 description 1
- 238000012937 correction Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 238000007499 fusion processing Methods 0.000 description 1
- 208000014674 injury Diseases 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000010606 normalization Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
- G01N29/00—Investigating or analysing materials by the use of ultrasonic, sonic or infrasonic waves; Visualisation of the interior of objects by transmitting ultrasonic or sonic waves through the object
- G01N29/44—Processing the detected response signal, e.g. electronic circuits specially adapted therefor
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B61—RAILWAYS
- B61K—AUXILIARY EQUIPMENT SPECIALLY ADAPTED FOR RAILWAYS, NOT OTHERWISE PROVIDED FOR
- B61K9/00—Railway vehicle profile gauges; Detecting or indicating overheating of components; Apparatus on locomotives or cars to indicate bad track sections; General design of track recording vehicles
- B61K9/08—Measuring installations for surveying permanent way
- B61K9/10—Measuring installations for surveying permanent way for detecting cracks in rails or welds thereof
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
- G01N29/00—Investigating or analysing materials by the use of ultrasonic, sonic or infrasonic waves; Visualisation of the interior of objects by transmitting ultrasonic or sonic waves through the object
- G01N29/04—Analysing solids
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
- G01N29/00—Investigating or analysing materials by the use of ultrasonic, sonic or infrasonic waves; Visualisation of the interior of objects by transmitting ultrasonic or sonic waves through the object
- G01N29/44—Processing the detected response signal, e.g. electronic circuits specially adapted therefor
- G01N29/4409—Processing the detected response signal, e.g. electronic circuits specially adapted therefor by comparison
- G01N29/4418—Processing the detected response signal, e.g. electronic circuits specially adapted therefor by comparison with a model, e.g. best-fit, regression analysis
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
- G01N29/00—Investigating or analysing materials by the use of ultrasonic, sonic or infrasonic waves; Visualisation of the interior of objects by transmitting ultrasonic or sonic waves through the object
- G01N29/44—Processing the detected response signal, e.g. electronic circuits specially adapted therefor
- G01N29/4454—Signal recognition, e.g. specific values or portions, signal events, signatures
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
- G01N2291/00—Indexing codes associated with group G01N29/00
- G01N2291/02—Indexing codes associated with the analysed material
- G01N2291/023—Solids
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
- G01N2291/00—Indexing codes associated with group G01N29/00
- G01N2291/02—Indexing codes associated with the analysed material
- G01N2291/028—Material parameters
- G01N2291/0289—Internal structure, e.g. defects, grain size, texture
Landscapes
- Physics & Mathematics (AREA)
- Analytical Chemistry (AREA)
- General Health & Medical Sciences (AREA)
- Health & Medical Sciences (AREA)
- Life Sciences & Earth Sciences (AREA)
- Chemical & Material Sciences (AREA)
- Engineering & Computer Science (AREA)
- Biochemistry (AREA)
- Pathology (AREA)
- General Physics & Mathematics (AREA)
- Immunology (AREA)
- Signal Processing (AREA)
- Acoustics & Sound (AREA)
- Mechanical Engineering (AREA)
- Investigating Or Analyzing Materials By The Use Of Ultrasonic Waves (AREA)
Abstract
The invention discloses a method and a device for identifying steel rail damage, and relates to the technical field of nondestructive testing, wherein the method comprises the following steps: collecting historical steel rail detection data; marking the ultrasonic B display data according to the type of the steel rail damage to obtain a marked file; establishing a training set and a testing set; constructing a YOLOV7 model; the YOLOV7 model comprises a backbone network and a detection head; the backbone network comprises a plurality of ELANs, and the ELANs output a plurality of characteristic diagrams with different sizes to the detection head; training and testing the YOLOV7 model by using a training set and a testing set respectively to obtain a rail damage identification model; inputting the steel rail detection data to be identified into a steel rail damage identification model, and outputting a steel rail damage identification result; the rail damage identification result comprises a rail damage position, a rail damage type and a rail damage value. The invention can improve the accuracy, speed and efficiency of identifying the rail damage.
Description
Technical Field
The invention relates to the technical field of nondestructive testing, in particular to a rail damage identification method and device.
Background
This section is intended to provide a background or context to the embodiments of the invention that are recited in the claims. The description herein is not admitted to be prior art by inclusion in this section.
Along with the rapid development of rail transit, the rail transit is a main travel and transportation mode of people, and the safety and reliability of the rail transit are required to be higher. The rail flaw detection vehicle regularly detects the rail, and has very important significance for running safety and reducing or avoiding potential safety hazards of rail transit.
The wheel type ultrasonic probe is arranged on the left and right wheels of the rail flaw detection vehicle, when the flaw detection vehicle passes through the rail at a certain speed, the probe sends out ultrasonic signals to the rail and receives echo signals, so that the internal damage information of the rail can be acquired, and A, B type scanning images, namely an A display and a B display of ultrasonic are formed. Wherein the B display is converted by using a plurality of A display scanning results through coding. The B display is displayed in a two-dimensional image, and can display the damage condition of the longitudinal section of the steel rail, and the principle is that the internal damage is displayed based on the difference of the ultrasonic wave reflection echo intensity at the damage position. Compared with the display A, the display B has more visual image information quantity, and contains the position information of the injury on the track, and has small data quantity and convenient storage. The ultrasonic B display is utilized to obtain the flaw information and the corresponding position information in the steel rail, and a data basis is provided for carrying out the steel rail flaw detection based on target detection.
At present, analysis of ultrasonic steel rail flaw detection vehicle detection data in China mainly depends on manual playback analysis, however, the playback task amount is huge, and the playback effect also depends on the professional level of playback personnel. In the prior art, an intelligent rail damage identification method based on deep learning is used for converting the problem of 'object detection' of rail damage into the problem of 'classification' on the basis of a convolutional neural network architecture, for example, based on the generation principle of B display data of a rail flaw detection vehicle and rail damage classification. However, in practical application, the rail damage characteristics are smaller, the existing deep learning method is not high in recognition accuracy on the smaller rail damage characteristics, and in addition, the rail damage recognition speed and the image processing efficiency are still to be improved.
Disclosure of Invention
The embodiment of the invention provides a rail damage identification method, which is used for improving the rail damage identification precision, speed and efficiency, and comprises the following steps:
collecting historical steel rail detection data; the historical steel rail detection data comprise ultrasonic B display data;
marking the ultrasonic B display data according to the type of the rail damage corresponding to the historical rail detection data to obtain a marking file;
establishing a training set and a testing set according to the annotation file;
constructing a target detection algorithm YOLOV7 model; the YOLOV7 model comprises a backbone network and a detection head; the backbone network comprises a plurality of high-efficiency aggregation networks ELANs, and the ELANs output a plurality of characteristic diagrams with different sizes to the detection head;
training and testing the YOLOV7 model by using a training set and a testing set respectively to obtain a rail damage identification model;
inputting the steel rail detection data to be identified into a steel rail damage identification model, and outputting a steel rail damage identification result; the rail damage identification result comprises a rail damage position, a rail damage type and a rail damage value.
The embodiment of the invention also provides a device for identifying the rail damage, which is used for improving the accuracy, the speed and the efficiency of identifying the rail damage, and comprises the following steps:
the data collection module is used for collecting historical steel rail detection data; the historical steel rail detection data comprise ultrasonic B display data;
the data preprocessing module is used for marking the ultrasonic B display data according to the type of the rail damage corresponding to the historical rail detection data to obtain a marked file; establishing a training set and a testing set according to the annotation file;
the model construction module is used for constructing a target detection algorithm YOLOV7 model; the YOLOV7 model comprises a backbone network and a detection head; the backbone network comprises a plurality of ELANs, and the ELANs output a plurality of characteristic diagrams with different sizes to the detection head;
the model training and testing module is used for training and testing the YOLOV7 model by utilizing the training set and the testing set respectively to obtain a rail damage identification model;
the steel rail damage identification module is used for inputting steel rail detection data to be identified into the steel rail damage identification model and outputting a steel rail damage identification result; the rail damage identification result comprises a rail damage position, a rail damage type and a rail damage value.
The embodiment of the invention also provides computer equipment, which comprises a memory, a processor and a computer program stored on the memory and capable of running on the processor, wherein the method for identifying the rail damage is realized when the processor executes the computer program.
The embodiment of the invention also provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program realizes the rail damage identification method when being executed by a processor.
The embodiment of the invention also provides a computer program product, which comprises a computer program, wherein the computer program realizes the rail damage identification method when being executed by a processor.
In the embodiment of the invention, a YOLOV7 model is constructed, wherein the YOLOV7 model comprises a backbone network and a detection head, the backbone network comprises a plurality of ELANs, and the ELANs output a plurality of feature images with different sizes to the detection head; training and testing the YOLOV7 model by utilizing ultrasonic B display data in the historical steel rail detection data to obtain a steel rail damage identification model; the rail damage identification model can extract a plurality of characteristic diagrams with different sizes, such as shallow characteristic diagrams and deep characteristic diagrams, and the characteristic diagrams with different sizes have different receptive fields, so that multi-scale characteristic fusion and detection can be carried out on a detection head, the rail damage identification precision is improved, and particularly the identification precision and detection effect of small and medium targets in rail damage are improved; meanwhile, in the embodiment of the invention, ultrasonic B display data are marked in advance according to the type of the rail damage corresponding to the historical rail detection data in the preprocessing of the historical rail detection data, so that a training set and a testing set with sufficient data quantity are formed, and the improvement of the rail damage identification speed and efficiency is facilitated.
Drawings
In order to more clearly illustrate the embodiments of the invention or the technical solutions in the prior art, the drawings that are required in the embodiments or the description of the prior art will be briefly described, it being obvious that the drawings in the following description are only some embodiments of the invention, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art. In the drawings:
FIG. 1 is a schematic flow chart of a rail flaw identification method in an embodiment of the invention;
FIG. 2 is a schematic diagram of a YOLOV7 model in accordance with an embodiment of the present invention;
FIG. 3 is a second schematic diagram of a YOLOV7 model in accordance with an embodiment of the present invention;
FIG. 4 is a schematic diagram of an attention mechanism module according to an embodiment of the present invention;
FIG. 5 is a third schematic diagram of a YOLOV7 model in accordance with an embodiment of the present invention;
FIG. 6 is a schematic diagram IV of a YOLOV7 model in accordance with an embodiment of the present invention;
fig. 7 is a schematic view of a rail damage recognition device according to an embodiment of the present invention.
Detailed Description
For the purpose of making the objects, technical solutions and advantages of the embodiments of the present invention more apparent, the embodiments of the present invention will be described in further detail with reference to the accompanying drawings. The exemplary embodiments of the present invention and their descriptions herein are for the purpose of explaining the present invention, but are not to be construed as limiting the invention.
The applicant finds that the analysis of the ultrasonic steel rail flaw detection vehicle detection data in China at present mainly depends on manual playback analysis, however, the playback task amount is huge, and the playback effect also depends on the professional level of playback personnel. In the prior art, an intelligent rail damage identification method based on deep learning is used for converting the problem of 'object detection' of rail damage into the problem of 'classification' on the basis of a convolutional neural network architecture, for example, based on the generation principle of B display data of a rail flaw detection vehicle and rail damage classification. However, in practical application, the rail damage characteristics are smaller, the existing deep learning method is not high in recognition accuracy on the smaller rail damage characteristics, and in addition, the rail damage recognition speed and the image processing efficiency are still to be improved. Based on this, the applicant proposes a rail flaw identification method.
Fig. 1 is a schematic flow chart of a rail damage identification method in an embodiment of the present invention, as shown in fig. 1, the method includes:
step 101, collecting historical steel rail detection data; the historical steel rail detection data comprise ultrasonic B display data;
102, marking the ultrasonic B display data according to the type of the rail damage corresponding to the historical rail detection data to obtain a marking file;
step 103, building a training set and a testing set according to the labeling file;
104, constructing a YOLOV7 model; the YOLOV7 model comprises a backbone network and a detection head; the backbone network comprises a plurality of ELANs, and the ELANs output a plurality of characteristic diagrams with different sizes to the detection head;
step 105, training and testing the YOLOV7 model by using a training set and a testing set respectively to obtain a rail damage identification model;
step 106, inputting the steel rail detection data to be identified into a steel rail damage identification model, and outputting a steel rail damage identification result; the rail damage identification result comprises a rail damage position, a rail damage type and a rail damage value.
As can be seen from the flow chart shown in fig. 1, in the embodiment of the present invention, a YOLOV7 model is constructed, where the YOLOV7 model includes a backbone network and a detection head, and the backbone network includes a plurality of ELANs, and the ELANs output a plurality of feature maps with different sizes to the detection head; training and testing the YOLOV7 model by utilizing ultrasonic B display data in the historical steel rail detection data to obtain a steel rail damage identification model; the rail damage identification model can extract a plurality of characteristic diagrams with different sizes, such as shallow characteristic diagrams and deep characteristic diagrams, and the characteristic diagrams with different sizes have different receptive fields, so that multi-scale characteristic fusion and detection can be carried out on a detection head, the rail damage identification precision is improved, and particularly the identification precision and detection effect of small and medium targets in rail damage are improved; meanwhile, in the embodiment of the invention, ultrasonic B display data are marked in advance according to the type of the rail damage corresponding to the historical rail detection data in the preprocessing of the historical rail detection data, so that a training set and a testing set with sufficient data quantity are formed, and the improvement of the rail damage identification speed and efficiency is facilitated.
The method for identifying rail damage in the embodiment of the invention is explained in detail below.
First, historical rail test data, such as ultrasound B-display data, is collected.
Then, the collected historical rail detection data is preprocessed. In the embodiment of the invention, the ultrasonic B display data is marked according to the type of the rail damage corresponding to the historical rail detection data, and finally the marked file is obtained. In the implementation, because the collected historical steel rail detection data are more, automatic batch preprocessing can be realized by programming, ultrasonic B display data are sequentially acquired from a designated directory, and labeling is performed, so that labeling files in the format of XML and the like are formed. The steel rail damage types corresponding to the historical steel rail detection data comprise one or any combination of rail head nuclear damage, rail web hole crack, rail bottom crack, weld joint damage, rail surface damage, horizontal crack and the like, and other steel rail damage types possibly exist in some special structure.
In one embodiment, marking the ultrasonic B-display data according to the type of rail damage corresponding to the historical rail detection data may include: and marking the anchor frame and the corresponding steel rail damage type at each steel rail damage position of the ultrasonic B display data to form a sample set with sufficient data volume, and providing data support for training, testing and verifying the follow-up neural network model.
After the labeling is completed, the labeling file is divided according to a certain proportion, a training set, a testing set and a verification set are established, wherein the training set is used for training the neural network model and determining parameters, the verification set is used for determining the structure of the neural network model and adjusting the super parameters of the neural network model, and the testing set is used for checking the generalization capability of the neural network model.
Thereafter, a YOLOV7 model was constructed. YOLOV7 is a single-stage target detection algorithm with the most advantage in current speed, and can rapidly complete the positioning and classification of targets in images, and applicant considers that YOLOV7 is very suitable for rail damage identification based on ultrasonic B-display data. In implementation, based on the characteristics of YOLOV7, the basic modules such as a feature extraction module CBS, a high-efficiency network aggregation module ELAN, a downsampling module MP, a pyramid pooling integration module SPPCSPC, a repartitionizing convolution module RepConv, high-efficiency aggregation network modules REP and ELAN-H (similar to the ELAN module, the slightly different output numbers selected by the second branch) and a convolution module Conv are implemented to complete the construction of the YOLOV7 model. Fig. 2 is a schematic diagram of a YOLOV7 model in an embodiment of the present invention, as shown in fig. 2, where the YOLOV7 model includes a Backbone network (i.e. a backhaul part in fig. 2) and a detection Head (i.e. a Head part in fig. 2), where the Backbone network is used to extract features, and includes a plurality of CBS, a plurality of MPs, a plurality of ELANs, and the like, and the detection Head is used to fuse the features and locate detection, and includes SPPCSPC, a plurality of RepConv, a plurality of ELAN-H, a plurality of Conv, and the like.
The basic idea of YOLOV7 model operation is to input test data and to simultaneously regress the type of rail damage and the corresponding location at the output layer. The specific flow is as follows:
1) Preprocessing input picture data, and aligning the input picture data into RGB pictures with fixed resolution, such as 640 multiplied by 640;
2) Inputting pictures with consistent resolution into a backbone network, wherein the backbone network has 50 layers, firstly, changing the whole input image into a feature map with the size of 160 multiplied by 128 through 4 CBS modules, wherein the CBS modules mainly comprise Conv and Batch Normalization standardized and activated functions SiLU;
3) Then, three MP+ELANs are processed, the length and width of the image are reduced, the channel is doubled, three ELANs respectively output three-scale feature images C3, C4 and C5, and the resolution sizes are respectively 80 multiplied by 512, 40 multiplied by 1024 and 20 multiplied by 1024; the ELAN is a high-efficiency aggregation network structure, and the network can learn more characteristics and has stronger robustness by controlling the shortest gradient path and the longest gradient path; the MP layer is provided with two branches, the effect is to perform downsampling, the first branch firstly passes through a maximum pooling layer, the maximum effect is to perform downsampling, then the channel number is changed through a 1x1 convolution, the second branch firstly passes through a 1x1 convolution to perform the channel number change, then the second branch passes through a 3x3 convolution kernel and a convolution block with the step length of 2, the convolution block is also used for downsampling, and finally the results of the first branch and the second branch are added together to obtain a super downsampling result;
4) The detection head receives the feature images C3, C4 and C5, performs feature image fusion, wherein the feature image C5 is a downsampled feature image which is finally output through calculation of all modules of a backbone network, firstly, the feature image C5 passes through an SPPCSPC module, the SPPCSPC module can improve the receptive field of a model, meanwhile, the model can adapt to or input images with different resolutions, and the channel number is changed from 1024 to 512; then, fusing C5 with the features C4 and C3 in a top-down mode to obtain P3, P4 and P5, and fusing P3, P4 and P5 in a bottom-up mode, wherein a typical PAFPN (Path Aggregation Feature pyramid network) structure is adopted, and the feature images with different scales are fused from top to bottom and from bottom to top to enhance the features and improve the detection performance of the network on targets with various scales; the three scale feature images after fusion are processed by the RepConv module and the Conv module, three tasks (classification, front and back background classification and frame) of image detection are predicted, and regression parameters and category information of each feature point on the three different scale feature images are output. Taking an ultrasonic B-display data set as an example, the ultrasonic B-display data set comprises a plurality of steel rail damage types, and the finally output prediction results are (20,20,33), (40,40,33) and (80,80,33) respectively, wherein 20, 40 and 80 respectively represent three different feature map scales, 33 comprises data corresponding to 11 kinds of information of steel rail damage identification results of 3 anchor points (anchors), for example, the method comprises the following steps: the method comprises the steps of determining regression parameters of a target frame of a position, judging whether a characteristic point comprises confidence of a steel rail damage type or not, determining probability information of the steel rail damage type, and obtaining a steel rail damage identification result after decoding the prediction result.
It should be noted that, as the image data is deeply calculated in the network backbone, the size of the feature map is continuously reduced, the receptive field is continuously increased, and in fig. 2, the network backbone includes three mp+elan, and the feature maps output by ELAN are C3, C4, and C5 from shallow to deep, respectively. In order to further improve the recognition accuracy of the small rail damage feature, in the embodiment of the present invention, the number of MPs and ELANs in the network backbone and the output feature map are not limited, for example, the network backbone in fig. 2 may further increase the number of MPs 1 and ELANs and the output shallow feature map.
Fig. 3 is a schematic diagram two of a YOLOV7 model in the embodiment of the present invention, as shown in fig. 3, the number of MP1 and ELAN is not increased based on the structure of the YOLOV7 model in fig. 2, but the receptive field of the shallow feature map C2 and C2 output by ELAN is smaller than that of C3, C4 and C5, and the shallow feature map C2 with smaller receptive field is introduced into the detection head to perform deeper multi-scale feature fusion, so that the detection accuracy of rail damage identification can be further improved. The shallow feature map is added into the feature fusion process, so that the identification of smaller rail damage features in the picture can be facilitated, a single picture can correspond to longer rail process mileage, and detection of a small target is utilized.
And training and testing the constructed YOLOV7 model by using a training set and a testing set to obtain a rail damage identification model, wherein training parameters are shown in table 1, and table 1 is a part of training parameters of the YOLOV7 model.
TABLE 1Yolov7 model part training parameters
In one embodiment, when the YOLOV7 model is trained by using the training set, a residual network structure can be introduced for assisting training, and a complex residual network structure can be equivalent to a common 3×3 convolution during reasoning, so that the network complexity is reduced, but the reasoning performance of the network is not reduced.
After training, a steel rail damage identification Model is obtained, the steel rail damage identification Model is tested by utilizing a test set, and statistical indexes of the test comprise average precision mAP, recall and Model Size, as shown in a table 2, and the table 2 is the test statistical index of the steel rail damage identification Model in the embodiment of the invention.
Table 2 test statistics index of rail damage identification model in the embodiment of the present invention
Finally, inputting the steel rail detection data to be identified into a steel rail damage identification model, and outputting a steel rail damage identification result; the rail damage identification result comprises a rail damage position, a rail damage type and a rail damage value.
The rail damage recognition result can be output in the form of a picture, the rail damage position, the rail damage type and the rail damage value are displayed, for example, a target frame corresponding to the rail damage position, the rail damage type and the confidence value of the rail damage are displayed in the picture, and the rail damage recognition result can also be output in the form of a thermodynamic diagram and a pseudo-color diagram.
In order to obtain the most suitable size of the target frame, the length and width shape of the anchor frame in the marking file is required to be closest to the real target frame, in one embodiment, before a training set and a test set are established according to the marking file, the length and width values of the anchor frame in the marking file are clustered by adopting a K-Means clustering algorithm to obtain a length and width result value, wherein the length and width result value is used for representing the length and width value of the target frame which corresponds to the historical rail detection data and is suitable for the length and width value of the target frame which represents the rail damage position in the final rail damage identification result.
The K-Means algorithm flow is:
(1) K points are randomly selected from the length and width of an anchor frame in the annotation file to serve as the centers of initial clustering,the selected center point is C= { C 1 ,c 2 ,......,c k -wherein K is an integer;
(2) For each sample in the data set, calculating the distance from each sample to each clustering center point, and dividing the distance from each sample to which clustering center point is the smallest into the class of the corresponding clustering center;
(3) For each rail damage type, recalculating the clustering center of the categoryWherein i represents the total number of the steel rail damage type data;
(4) Repeating the step (2) and the step (3) until the position of the clustering center is not changed or the designated iteration times are reached, and obtaining the final clustering center.
And clustering the anchor frames on the data set by using a K-Means clustering algorithm to obtain the length and width of the optimal prediction target frame suitable for the data set.
The black background in the ultrasonic B-mode ultrasonic data is relatively more, the occupied area of the steel rail damage and the normal steel rail structure in a B-display image is small, and the difficulty of detecting the damage under the condition is higher than that of the normal data. To improve the accuracy and speed of rail damage identification and improve the target detection capability of the YOLOV7 model, in one embodiment, constructing the YOLOV7 model may include: and constructing a backbone network in the YOLOV7 model by using an attention mechanism. The attention mechanism is derived from a biological system of human beings, when a large amount of information is processed, such as pictures, a certain part of more interesting parts in the pictures are often focused, the attention mechanism simulates the biological cognition habit, so that the vision of a machine focuses on important areas of the images rather than ignores irrelevant contents, and the capability of a computer for analyzing complex scenes is improved. In general, the attention mechanisms can be divided into four basic categories: channel Attention (Channel Attention), spatial Attention (Spatial Attention), temporal Attention (Temporal Attention), branching Attention (Branch Attention). As a means of resource allocation, the attention mechanism may solve the problem of information overload, and use limited computing resources to process more important information, and any of the above types of attention mechanisms may be used in embodiments of the present invention.
In one embodiment, the attention mechanism CBAM (Convolutional Block Attention Module) module is encoded based on the combined attention mechanism of channel attention and spatial attention, which is introduced into the YOLOV7 model. The spatial attention can make the model pay more attention to the pixel area which plays a role in classifying in the image and neglect the insignificant area, the channel attention is used for processing the distribution relation of the characteristic map channels, and meanwhile, attention distribution is carried out on two dimensions, so that the attention mechanism enhances the improvement effect on the performance of the model. And the CBAM module has no large number of convolution structures and a small number of pooling layers and feature fusion operations, and the structure avoids a large number of computations caused by convolution multiplication, so that the module has low complexity and small computation amount.
On a given one of the intermediate feature maps, the CBAM module sequentially generates an attention map along two independent dimensions of the channel and space, and then multiplies the attention map by the input feature map to effect adaptive feature correction. The CBAM module has the characteristics of light weight, universality and the like, can be seamlessly integrated into the YOLOV7 model of the embodiment of the invention, can perform end-to-end training together with the YOLOV7 model, and does not introduce excessive calculation amount. Fig. 4 is a schematic diagram of an attention mechanism module in an embodiment of the present invention, where the CBAM module is embedded in the YOLOV7 model, and a feature map of the input CBAM module may be expressed as:
F∈R C×H×W
wherein F is a feature map input into the CBAM module, R represents a real space, E represents a belonging number, C represents a channel number, H represents a high of the feature map, and W represents a width of the feature map. The CBAM module sequentially generates a 1D channel attention profile Mc and a 2D spatial attention profile Ms:
Mc∈R C×1×1
Ms∈R 1×H×W
the entire attention process can be described as:
wherein, the liquid crystal display device comprises a liquid crystal display device,representing element-by-element multiplication, and adopting a broadcasting mechanism in the middle for dimension transformation and matching. For a given feature map, the CBAM module uses the channel attention and spatial attention modules to place in sequence, calculate complementary attention, and let the model learn "what to pay attention to" and "where to pay attention to", respectively.
Fig. 5 is a schematic diagram III of a YOLOV7 model in an embodiment of the present invention, as shown in fig. 5, in this example, a CBAM module is embedded into a backbone network in the YOLOV7 model, and by using the CBAM module, a resource allocation manner can be changed according to the importance degree of a target of interest, so that resources are more inclined towards a steel rail damage to be focused, the focus of the YOLOV7 model on the damage and the steel rail structure is improved, and meanwhile, the focus of an algorithm model on an irrelevant black background area is inhibited.
Similarly, training and testing are performed after the attention mechanism is introduced, and statistical indexes of testing comprise mAP, recall and Model Size, as shown in Table 3, and Table 3 is a statistical index of testing of a rail damage recognition Model of the attention mechanism in the embodiment of the invention.
TABLE 3 statistical test index of Rail damage identification model for introducing attention mechanism in the embodiment of the invention
Considering the simultaneous introduction of the attention mechanism and the increase of the shallow feature map, the recognition accuracy and efficiency of the rail damage recognition model may be further improved, and in an embodiment, inputting the rail detection data to be recognized into the rail damage recognition model in step 106, outputting the rail damage recognition result may include: the rail detection data to be identified is input into a rail damage identification model, the rail damage identification model is sequentially processed by a plurality of convolution layers, a plurality of ELANs and attention mechanisms in a backbone network, characteristic diagrams with different sizes are respectively output from each ELAN and attention mechanism to a detection head, and finally a rail damage identification result is output through a regression process.
Fig. 6 is a schematic diagram four of a YOLOV7 Model in the embodiment of the present invention, as shown in fig. 6, to simultaneously introduce an attention mechanism and increase shallow feature patterns, a YOLOV7 Model which is more focused on a steel rail damage area and has a better small target detection effect is constructed and trained, a test set is used to test a steel rail damage recognition Model which simultaneously introduces an attention mechanism and increases shallow feature patterns, and statistical indexes of the test include mAP, recall, and Model Size, as shown in table 4, and table 4 is a test statistical index of a steel rail damage recognition Model which simultaneously introduces an attention mechanism and increases shallow feature patterns in the embodiment of the present invention.
TABLE 4 statistical test index of Rail damage identification model for simultaneously introducing attention mechanism and increasing shallow feature map in the embodiment of the invention
Compared with the original rail damage identification model, for example, in the first schematic diagram of the YOLOV7 model shown in fig. 2, the rail damage identification model which introduces a attention mechanism and increases shallow feature images simultaneously realizes larger improvement in detection precision, is more suitable for rail damage detection tasks on rail B display image data, can realize rapid and accurate intelligent rail damage identification, and overcomes the defects of low conventional rail flaw detection efficiency, time consumption and labor consumption.
The embodiment of the invention also provides a rail damage identification device, as described in the following embodiment. Because the principle of the device for solving the problems is similar to that of the rail damage identification method, the implementation of the device can be referred to the implementation of the rail damage identification method, and the repetition is not repeated.
Fig. 7 is a schematic view of a rail damage identifying device according to an embodiment of the present invention, as shown in fig. 7, the device includes:
the data collection module 701 is used for collecting historical steel rail detection data; the historical steel rail detection data comprise ultrasonic B display data;
the data preprocessing module 702 is configured to label the ultrasonic B-display data according to the type of the rail damage corresponding to the historical rail detection data, so as to obtain a label file; establishing a training set and a testing set according to the annotation file;
a model building module 703, configured to build a target detection algorithm YOLOV7 model; the YOLOV7 model comprises a backbone network and a detection head; the backbone network comprises a plurality of ELANs, and the ELANs output a plurality of characteristic diagrams with different sizes to the detection head;
the model training and testing module 704 is configured to train and test the YOLOV7 model by using the training set and the testing set, respectively, so as to obtain a rail damage identification model;
the rail damage identification module 705 is configured to input rail detection data to be identified into a rail damage identification model, and output a rail damage identification result; the rail damage identification result comprises a rail damage position, a rail damage type and a rail damage value.
In one embodiment, the rail damage type comprises any one or any combination of the following:
rail head core damage, rail web hole crack, rail bottom crack, weld joint damage, rail surface damage and horizontal crack.
In one embodiment, the data preprocessing module 701 is specifically configured to:
and marking the anchor frame and the corresponding steel rail damage type at each steel rail damage position of the ultrasonic B display data.
In one embodiment, the apparatus further comprises:
the clustering processing module is used for clustering the length and width values of the anchor frame in the annotation file by adopting a K-Means clustering algorithm before the data preprocessing module 701 establishes the training set and the testing set according to the annotation file to obtain a length and width result value; the length and width result value is used for representing the shape and the size of the rail damage characteristic corresponding to the historical rail detection data.
In one embodiment, the model building module 703 is specifically configured to:
and constructing a backbone network in the YOLOV7 model by using an attention mechanism.
In one embodiment, the attention mechanism comprises: spatial attention mechanisms and/or channel attention mechanisms.
In one embodiment, rail damage identification module 705 is specifically configured to:
the rail detection data to be identified is input into a rail damage identification model, and is processed by a plurality of convolution layers, a plurality of ELANs and attention mechanisms in a backbone network in sequence, and feature images with different sizes are respectively output from each ELAN and attention mechanism to a detection head, and a rail damage identification result is output.
In one embodiment, when training the YOLOV7 model with a training set, a residual network structure is introduced to assist in training.
The embodiment of the invention also provides computer equipment, which comprises a memory, a processor and a computer program stored on the memory and capable of running on the processor, wherein the method for identifying the rail damage is realized when the processor executes the computer program.
The embodiment of the invention also provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program realizes the rail damage identification method when being executed by a processor.
The embodiment of the invention also provides a computer program product, which comprises a computer program, wherein the computer program realizes the rail damage identification method when being executed by a processor.
In the embodiment of the invention, a YOLOV7 model is constructed, wherein the YOLOV7 model comprises a backbone network and a detection head, the backbone network comprises a plurality of ELANs, and the ELANs output a plurality of feature images with different sizes to the detection head; training and testing the YOLOV7 model by utilizing ultrasonic B display data in the historical steel rail detection data to obtain a steel rail damage identification model; the rail damage identification model can extract a plurality of characteristic diagrams with different sizes, such as shallow characteristic diagrams and deep characteristic diagrams, and the characteristic diagrams with different sizes have different receptive fields, so that multi-scale characteristic fusion and detection can be carried out on a detection head, the rail damage identification precision is improved, and particularly the identification precision and detection effect of small and medium targets in rail damage are improved; meanwhile, in the embodiment of the invention, ultrasonic B display data are marked in advance according to the type of the rail damage corresponding to the historical rail detection data in the preprocessing of the historical rail detection data, so that a training set and a testing set with sufficient data quantity are formed, and the improvement of the rail damage identification speed and efficiency is facilitated.
It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
The foregoing description of the embodiments has been provided for the purpose of illustrating the general principles of the invention, and is not meant to limit the scope of the invention, but to limit the invention to the particular embodiments, and any modifications, equivalents, improvements, etc. that fall within the spirit and principles of the invention are intended to be included within the scope of the invention.
Claims (19)
1. A method for identifying rail flaws, comprising:
collecting historical steel rail detection data; the historical steel rail detection data comprise ultrasonic B display data;
marking the ultrasonic B display data according to the type of the rail damage corresponding to the historical rail detection data to obtain a marking file;
establishing a training set and a testing set according to the annotation file;
constructing a target detection algorithm YOLOV7 model; the YOLOV7 model comprises a backbone network and a detection head; the backbone network comprises a plurality of high-efficiency aggregation networks ELANs, and the ELANs output a plurality of characteristic diagrams with different sizes to the detection head;
training and testing the YOLOV7 model by using a training set and a testing set respectively to obtain a rail damage identification model;
inputting the steel rail detection data to be identified into a steel rail damage identification model, and outputting a steel rail damage identification result; the rail damage identification result comprises a rail damage position, a rail damage type and a rail damage value.
2. The method of claim 1, wherein the type of rail damage comprises any one or any combination of the following:
rail head core damage, rail web hole crack, rail bottom crack, weld joint damage, rail surface damage and horizontal crack.
3. The method of claim 1, wherein labeling the ultrasound B-display data according to the type of rail damage corresponding to the historical rail detection data comprises:
and marking the anchor frame and the corresponding steel rail damage type at each steel rail damage position of the ultrasonic B display data.
4. The method of claim 3, further comprising, prior to building the training set and the test set from the annotation file:
clustering the length and width values of the anchor frames in the annotation file by adopting a K-Means clustering algorithm to obtain a length and width result value; the length and width result value is used for representing the shape and the size of the rail damage characteristic corresponding to the historical rail detection data.
5. The method of claim 1, wherein constructing a YOLOV7 model comprises:
and constructing a backbone network in the YOLOV7 model by using an attention mechanism.
6. The method of claim 5, wherein the attention mechanism comprises: spatial attention mechanisms and/or channel attention mechanisms.
7. The method of claim 5, wherein inputting rail detection data to be identified into the rail damage identification model and outputting rail damage identification results comprises:
the rail detection data to be identified is input into a rail damage identification model, and is processed by a plurality of convolution layers, a plurality of ELANs and attention mechanisms in a backbone network in sequence, and feature images with different sizes are respectively output from each ELAN and attention mechanism to a detection head, and a rail damage identification result is output.
8. The method of claim 1, wherein a residual network structure is introduced to assist training when training the YOLOV7 model with a training set.
9. A rail damage identification device, comprising:
the data collection module is used for collecting historical steel rail detection data; the historical steel rail detection data comprise ultrasonic B display data;
the data preprocessing module is used for marking the ultrasonic B display data according to the type of the rail damage corresponding to the historical rail detection data to obtain a marked file; establishing a training set and a testing set according to the annotation file;
the model construction module is used for constructing a target detection algorithm YOLOV7 model; the YOLOV7 model comprises a backbone network and a detection head; the backbone network comprises a plurality of high-efficiency aggregation networks ELANs, and the ELANs output a plurality of characteristic diagrams with different sizes to the detection head;
the model training and testing module is used for training and testing the YOLOV7 model by utilizing the training set and the testing set respectively to obtain a rail damage identification model;
the steel rail damage identification module is used for inputting steel rail detection data to be identified into the steel rail damage identification model and outputting a steel rail damage identification result; the rail damage identification result comprises a rail damage position, a rail damage type and a rail damage value.
10. The apparatus of claim 9, wherein the type of rail damage comprises any one or any combination of the following:
rail head core damage, rail web hole crack, rail bottom crack, weld joint damage, rail surface damage and horizontal crack.
11. The apparatus of claim 9, wherein the data preprocessing module is specifically configured to:
and marking the anchor frame and the corresponding steel rail damage type at each steel rail damage position of the ultrasonic B display data.
12. The apparatus as recited in claim 11, further comprising:
the clustering processing module is used for clustering the length and width values of the anchor frame in the annotation file by adopting a K-Means clustering algorithm before the training set and the testing set are established according to the annotation file, so as to obtain a length and width result value; the length and width result value is used for representing the shape and the size of the rail damage characteristic corresponding to the historical rail detection data.
13. The apparatus of claim 9, wherein the model building module is specifically configured to:
and constructing a backbone network in the YOLOV7 model by using an attention mechanism.
14. The apparatus of claim 13, wherein the attention mechanism comprises: spatial attention mechanisms and/or channel attention mechanisms.
15. The apparatus of claim 13, wherein the rail damage identification module is specifically configured to:
the rail detection data to be identified is input into a rail damage identification model, and is processed by a plurality of convolution layers, a plurality of ELANs and attention mechanisms in a backbone network in sequence, and feature images with different sizes are respectively output from each ELAN and attention mechanism to a detection head, and a rail damage identification result is output.
16. The apparatus of claim 9, wherein a residual network structure is introduced to assist training when training the YOLOV7 model with a training set.
17. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the method of any of claims 1 to 8 when executing the computer program.
18. A computer readable storage medium, characterized in that the computer readable storage medium stores a computer program which, when executed by a processor, implements the method of any of claims 1 to 8.
19. A computer program product, characterized in that the computer program product comprises a computer program which, when executed by a processor, implements the method of any of claims 1 to 8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310498362.7A CN116626177A (en) | 2023-05-05 | 2023-05-05 | Rail damage identification method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310498362.7A CN116626177A (en) | 2023-05-05 | 2023-05-05 | Rail damage identification method and device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116626177A true CN116626177A (en) | 2023-08-22 |
Family
ID=87637343
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310498362.7A Pending CN116626177A (en) | 2023-05-05 | 2023-05-05 | Rail damage identification method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116626177A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116862915A (en) * | 2023-09-05 | 2023-10-10 | 西南石油大学 | Method for identifying defects of video stream in fan |
| CN117253066A (en) * | 2023-11-20 | 2023-12-19 | 西南交通大学 | Rail surface state identification method, device, equipment and readable storage medium |
-
2023
- 2023-05-05 CN CN202310498362.7A patent/CN116626177A/en active Pending
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116862915A (en) * | 2023-09-05 | 2023-10-10 | 西南石油大学 | Method for identifying defects of video stream in fan |
| CN117253066A (en) * | 2023-11-20 | 2023-12-19 | 西南交通大学 | Rail surface state identification method, device, equipment and readable storage medium |
| CN117253066B (en) * | 2023-11-20 | 2024-02-27 | 西南交通大学 | Rail surface state identification method, device, equipment and readable storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112200161B (en) | Face recognition detection method based on mixed attention mechanism | |
| Zhang et al. | Real-time detection of cracks on concrete bridge decks using deep learning in the frequency domain | |
| Han et al. | Using channel-wise attention for deep CNN based real-time semantic segmentation with class-aware edge information | |
| CN116626177A (en) | Rail damage identification method and device | |
| CN111080629A (en) | Method for detecting image splicing tampering | |
| CN111563418A (en) | Asymmetric multi-mode fusion significance detection method based on attention mechanism | |
| CN114399672A (en) | Railway wagon brake shoe fault detection method based on deep learning | |
| CN111932511B (en) | Electronic component quality detection method and system based on deep learning | |
| CN115035361A (en) | Target detection method and system based on attention mechanism and feature cross fusion | |
| CN114049356B (en) | Method, device and system for detecting structure apparent crack | |
| CN117593304B (en) | Semi-supervised industrial product surface defect detection method based on cross local global features | |
| CN115984537A (en) | Image processing method and device and related equipment | |
| Uzar et al. | Performance analysis of YOLO versions for automatic vehicle detection from UAV images | |
| CN114677349A (en) | Image segmentation method and system for edge information enhancement and attention guidance of encoding and decoding | |
| Ni et al. | Toward high-precision crack detection in concrete bridges using deep learning | |
| Yin et al. | Road Damage Detection and Classification based on Multi-level Feature Pyramids. | |
| CN117152414A (en) | Target detection method and system based on scale attention auxiliary learning method | |
| CN116052110A (en) | Intelligent positioning method and system for pavement marking defects | |
| CN115937736A (en) | Small target detection method based on attention and context awareness | |
| CN115359346B (en) | Small micro-space identification method and device based on street view picture and electronic equipment | |
| Dong et al. | Intelligent pixel-level pavement marking detection using 2D laser pavement images | |
| Liu et al. | Efficient dense attention fusion network with channel correlation loss for road damage detection | |
| Yazid et al. | Automated system form concrete damage classification identification using pretrained deep learning model | |
| Pan et al. | Quantitative Performance Evaluation of Improved U-Net Model in Crack Detection | |
| Nguyen | Crack Detection and Classification using Digital Image Processing |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |