CN113689392A - Railway fastener defect detection method and device - Google Patents
Railway fastener defect detection method and device Download PDFInfo
- Publication number
- CN113689392A CN113689392A CN202110950466.8A CN202110950466A CN113689392A CN 113689392 A CN113689392 A CN 113689392A CN 202110950466 A CN202110950466 A CN 202110950466A CN 113689392 A CN113689392 A CN 113689392A
- Authority
- CN
- China
- Prior art keywords
- railway
- sample
- determining
- railway fastener
- target
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/0002—Inspection of images, e.g. flaw detection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30108—Industrial image inspection
- G06T2207/30164—Workpiece; Machine component
Abstract
The invention provides a method and a device for detecting defects of railway fasteners, and relates to the technical field of railway detection. The railway fastener defect detection method comprises the following steps: determining a sample to be detected with position information and a first target sample with the position information according to the acquired image information of the railway fastener; expanding the first target sample according to a target algorithm, and determining an expanded second target sample; generating a detection model for detecting the railway fastener according to the second target sample and the first target sample; and inputting the sample to be detected into the detection model, and determining a detection result. According to the scheme, the detection model for detecting the railway fastener is generated according to the expanded second target sample and the first target sample with the position information, the problem that the defect samples in the prior art are unbalanced is solved, and the detection accuracy is improved.
Description
Technical Field
The invention relates to the technical field of railway detection, in particular to a method and a device for detecting defects of railway fasteners.
Background
With the push of railway network layout and the rapid development of high-speed rail technology, railway operation, maintenance and detection are responsible for more and more important responsibilities. The safe operation of train needs the safety guarantee of railway system, and the railway fastener is the important subassembly of connecting rail and sleeper, guarantees the track distance, plays important effect in high-speed railway detecting system. However, due to the large distribution span of railway infrastructure, large environmental change, and the influence of train vibration and the like, the railway fastener has the defects of loosening, breaking, even missing and the like, which seriously influences the railway operation safety and even causes railway derailment accidents. Therefore, the railway fasteners need to be periodically tested to ensure safe operation of the railway.
At present, the types of the fasteners of railway lines in China are different, the traditional detection method needs manual operation, the method consumes a large amount of time, has high labor intensity and low efficiency, and brings unknown potential safety hazards to inspectors. With the rapid development of high-speed railways, the detection of railway fasteners faces great challenges, and therefore, there is a great need for researching rapid and accurate railway fastener identification algorithms.
In the prior art, a railway fastener defect detection method based on two-dimensional images is developed rapidly, the detection speed and the detection precision of fasteners are accelerated by combining a deep learning algorithm, and a corresponding detection system is developed. But because the two-dimensional image is greatly influenced by the illumination direction and the light condition, the defect detection of the railway fastener with lower image quality is more difficult, and the recall rate is lower. In addition, because the quantity of samples of the railway fastener is huge and the quantity of the defect samples is far less than that of the normal samples, the detection model cannot effectively record and extract the characteristics of the defect samples, and particularly when the deep learning algorithm model is used for detection, the model learns more characteristic attributes of the normal samples, so that the false detection rate is increased, and the final detection result is influenced.
In the prior art, the research of detecting the fasteners by utilizing three-dimensional information is less and immature, and most of the fasteners are segmented and positioned by utilizing template matching, so that the speed is low, and no effective solution is provided for the problem of unbalanced defect samples.
In summary, the two-dimensional image railway fastener defect detection method based on machine vision has the following disadvantages: the defect samples of the railway fasteners are few, the samples are difficult to label, and the defect detection of the two-dimensional railway fasteners is greatly influenced by illumination conditions; the method for detecting the fasteners based on the three-dimensional information is immature, and the problem of unbalance of defect samples is not effectively solved.
Disclosure of Invention
The embodiment of the invention provides a method and a device for detecting defects of railway fasteners, and aims to solve the problems of unbalanced samples and low sample labeling efficiency in the detection method in the prior art.
In order to solve the technical problems, the invention adopts the following technical scheme:
the embodiment of the invention provides a railway fastener defect detection method, which comprises the following steps:
determining a sample to be detected with position information and a first target sample with the position information according to the acquired image information of the railway fastener;
expanding the first target sample according to a target algorithm, and determining an expanded second target sample;
generating a detection model for detecting the railway fastener according to the second target sample and the first target sample;
and inputting the sample to be detected into the detection model, and determining a detection result.
Optionally, according to the acquired image information of the railway fastener, determining a sample to be detected with position information and a first target sample with position information, including:
acquiring railway fastener image information, wherein the railway fastener image information comprises a depth image and an intensity image; the pixels of the depth image and the intensity image correspond to each other one by one;
determining the position information of the railway fastener in the railway fastener image information according to the pixels of the depth image;
and determining a sample to be detected with the position information and a first target sample with the position information according to the position information of the railway fastener and the pixels of the intensity image.
Optionally, determining the position information of the railway fastener in the image information of the railway fastener according to the pixels of the depth image, including:
determining a railway fastener area and a rail area according to pixels of the depth image;
and determining the position information of the railway fastener in the railway fastener image information according to the railway fastener area and the rail area.
Optionally, the determining the location information of the railway clip in the image information of the railway clip according to the area of the railway clip and the area of the rail includes:
determining lateral coordinates of the railway clip from the railway clip area and the rail area;
determining a lateral position of the railway fastener according to the lateral coordinate and the depth image;
wherein the lateral coordinate satisfies the following formula:
xk∈[xa-dx,xa]∪[xa+h,xa+h+dx](ii) a Wherein x iskTransverse coordinate, x, representing the area of the railway fasteneraIs a lateral coordinate of a rail-side of the rail area, h is a rail width of the rail area, dxIs a first preset threshold.
Optionally, the determining the location information of the railway clip in the image information of the railway clip according to the area of the railway clip and the area of the rail includes:
according to the depth image, filtering the determined transverse position of the railway fastener along the transverse direction of the railway fastener, and determining a target height threshold value and a target height curve of the railway fastener;
determining the longitudinal coordinate of the railway fastener according to the target height threshold and the target height curve;
wherein the longitudinal coordinate satisfies the following formula:
yk∈[y1-Δy,y2+Δy](ii) a Wherein, ykRepresenting the longitudinal coordinate, y, of the railway fastener area1And y2And a height coordinate of the rail area is represented, and delta y is a second preset threshold value.
Optionally, determining the sample to be detected with the position information and the first target sample with the position information according to the position information of the railway fastener and the pixels of the intensity image, including:
performing mean filtering on the region of the railway fastener according to the pixels of the intensity image;
and determining a sample to be detected with position information and a first target sample with position information according to the filtered position information of the railway fastener and the railway fastener.
Optionally, the expanding the first target sample according to the target algorithm, and determining an expanded second target sample, includes:
expanding the defective fastener sample of the first target sample according to a generative antagonistic neural network algorithm, and determining an expanded second target sample;
wherein the generative confrontation neural network algorithm comprises a generator and an arbiter; the generator is used for expanding according to the defective fastener sample; the discriminator is used to determine the categories of the expanded sample and the original sample.
Optionally, the generating a detection model for detecting a railway fastener according to the second target sample and the first target sample includes:
merging the second target sample and the first target sample into a target training set;
generating a detection model for detecting the railway fastener according to the target training set;
and the middle layers of the target training set adopt a convolution kernel of 3x 3.
Optionally, the first target sample is expanded according to a target algorithm according to the pixels of the depth image, and after determining an expanded second target sample, the method further includes:
obtaining the type of the railway fastener after the position information is determined; the railway fastener types at least include: normal railway fasteners, missing nut railway fasteners, spring strip broken railway fasteners, missing spring strip railway fasteners;
according to the type of the railway fastener, corresponding labels are set for the railway fastener after the position information is determined, and each label comprises: the type of railway fastener.
The embodiment of the invention also provides a railway fastener defect detection device, which comprises:
the first determining module is used for determining a sample to be detected with position information and a first target sample with the position information according to the acquired image information of the railway fastener;
the second determination module is used for expanding the first target sample according to a target algorithm and determining an expanded second target sample;
the generating module is used for generating a detection model for detecting the railway fastener according to the second target sample and the first target sample;
and the third determining module is used for inputting the sample to be detected into the detection model and determining a detection result.
Embodiments of the present invention further provide a readable storage medium, on which a program is stored, where the program, when executed by a processor, implements the steps of the method for detecting defects of railway fasteners as described in any one of the above.
The invention has the beneficial effects that:
in the technical scheme, the method comprises the steps of determining a sample to be detected with position information and a first target sample with the position information according to acquired image information of the railway fastener; expanding the first target sample according to a target algorithm, and determining an expanded second target sample; generating a detection model for detecting the railway fastener according to the second target sample and the first target sample; and inputting the sample to be detected into the detection model, and determining a detection result. According to the technical scheme, the first target sample is expanded according to the target algorithm, the expanded second target sample is determined, the problem of sample imbalance in the prior art is solved, and the detection accuracy is improved through the detection model generated by the method.
Drawings
FIG. 1 is a schematic flow chart of a method for detecting defects of a railway fastener according to an embodiment of the present invention;
FIG. 2 shows a depth view of a railway fastener provided by an embodiment of the present invention;
FIG. 3 illustrates a longitudinal height variation curve of a railway fastener provided in accordance with an embodiment of the present invention;
FIG. 4a shows a diagram of a railway fastener before noise reduction;
FIG. 4b shows a view of the railway fastener after noise reduction;
FIG. 5 illustrates a fastener defect detection network architecture of the present invention;
FIG. 6 shows a loss curve during training of the test model of the present invention;
FIG. 7 shows the average accuracy curve of the present invention;
fig. 8 is a schematic block diagram of a railway fastener defect detecting apparatus according to an embodiment of the present invention.
Detailed Description
In order to make the technical problems, technical solutions and advantages of the present invention more apparent, the following detailed description is given with reference to the accompanying drawings and specific embodiments. In the following description, specific details such as specific configurations and components are provided only to help the full understanding of the embodiments of the present invention. Thus, it will be apparent to those skilled in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the invention. In addition, descriptions of well-known functions and constructions are omitted for clarity and conciseness.
It should be appreciated that reference throughout this specification to "one embodiment" or "an embodiment" means that a particular feature, structure or characteristic described in connection with the embodiment is included in at least one embodiment of the present invention. Thus, the appearances of the phrases "in one embodiment" or "in an embodiment" in various places throughout this specification are not necessarily all referring to the same embodiment. Furthermore, the particular features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.
In various embodiments of the present invention, it should be understood that the sequence numbers of the following processes do not mean the execution sequence, and the execution sequence of each process should be determined by its function and inherent logic, and should not constitute any limitation to the implementation process of the embodiments of the present invention.
The invention provides a method and a device for detecting defects of railway fasteners, aiming at the problems of unbalanced samples and low efficiency of sample marking in the detection method in the prior art.
It should be understood that there are many types of fasteners in high-speed railways, each of which comprises a metal elastic strip and a central bolt located at the center of the metal elastic strip, and plays an important role in connecting rails and sleepers and ensuring the stability and reliability of the rails. The general defects of fasteners fall into three categories: missing center bolts, missing metal spring strips, metal spring strip breakage and the like.
As shown in fig. 1, an embodiment of the present invention provides a method for detecting a defect of a railway fastener, including:
in this embodiment, the first target sample is expanded according to a target algorithm, which can be understood as expanding the defective samples in the first target sample. Here, the target algorithm is a sample expansion algorithm.
and 400, inputting the sample to be detected into the detection model, and determining a detection result.
In the defect detection process of the railway fastener, the structural optical equipment is used for collecting railway scene samples including a depth map and an intensity map, the fastener is positioned and segmented from a scene by using the depth map through the segmentation method provided by the invention, and the intensity map can be positioned and segmented from the scene according to the corresponding relation between the intensity map and the depth map; in order to solve the problem of sample imbalance, the invention provides a method for expanding the number of samples by using a generative antagonistic neural network, combining original samples to form a new training set, training a detection model (deep neural network), inputting the samples to be detected into the detection model, and finally outputting the detection result of the railway fastener. The invention solves the problem of sample imbalance, and expands the number of defective samples, namely, the first target sample is expanded according to a target algorithm; and finally, completing the defect detection and identification of the railway fastener by using the generated detection model.
It should be understood that accurate positioning of the railway fastener is the key to automatic detection of the fastener, and usually, the distance between the fastener and the rail on the high-speed railway is a fixed value dxAccordingly, the position range of the fastener in the scene can be reduced. Although the distance between the fasteners is relatively fixed, the distance between the fasteners varies to different degrees due to the large amount and variation of the railway network, and the positions of the fasteners cannot be directly positioned by using the distance. Therefore, the present invention provides a position location method with adaptive altitude according to step 100, which is as follows.
Optionally, the step 100 includes:
step 110, acquiring railway fastener image information, wherein the railway fastener image information comprises a depth image and an intensity image; the pixels of the depth image and the intensity image correspond to each other one by one;
step 130, determining a sample to be detected with position information and a first target sample with position information according to the position information of the railway fastener and the pixels of the intensity image.
In this embodiment, the acquisition of the image information of the railway fastener may be performed by using an acquisition sample device, which is preferably a 3D camera, and may directly obtain a depth map and an intensity map of the entire scene, including three parts, namely a steel rail, a sleeper, and a railway fastener, where the depth map and the intensity map are in one-to-one correspondence at pixel positions, and the depth map includes more position coordinate information, and may be used to complete positioning and segmentation of the railway fastener; the intensity map contains more semantic information and can be used for the expansion of defect samples and the defect detection. And generating a three-dimensional point cloud of the railway scene by using the depth map and the acquisition resolution.
Specifically, according to the depth map of the railway fastener shown in fig. 2, the step 120 includes:
step 121, determining a railway fastener area and a railway rail area according to pixels of the depth image;
step 122, determining the location information of the railway fastener in the railway fastener image information according to the railway fastener area and the rail area.
In this embodiment, the railway fastener area and the rail area are determined according to the pixels of the depth image, the map shown in fig. 2 is constructed according to the railway fastener area and the rail area, and the position information of the rail fastener in the railway fastener image information is determined according to the railway fastener area and the rail area.
Specifically, the step 122 includes:
determining lateral coordinates of the railway clip from the railway clip area and the rail area;
determining a lateral position of the railway fastener according to the lateral coordinate and the depth image;
wherein the lateral coordinate satisfies the following formula:
xk∈[xa-dx,xa]∪[xa+h,xa+h+dx](ii) a Wherein x iskTransverse coordinate, x, representing the area of the railway fasteneraIs a lateral coordinate of a rail-side of the rail area, h is a rail width of the rail area, dxIs a first preset threshold.
In this embodiment, the lateral coordinates of the railway fastener are located according to step 122. In a depth map (the lateral coordinates of the railway clip described in FIG. 2 may be denoted as D (i, j)), the lateral zone position of the clip should satisfy, xk∈[xa-dx,xa]∪[xa+h,xa+h+dx](ii) a Wherein x iskTransverse coordinate, x, representing the area of the railway fasteneraIs a lateral coordinate of a rail-side of the rail area, h is a rail width of the rail area, dxFor the first predetermined threshold, the shape of the railway fastener is generally within a certain range. And (5) intercepting the original three-dimensional image by using the formula to obtain D' (i, j).
Specifically, the step 122 includes:
according to the depth image, filtering the determined transverse position of the railway fastener along the transverse direction of the railway fastener, and determining a target height threshold value and a target height curve of the railway fastener;
determining the longitudinal coordinate of the railway fastener according to the target height threshold and the target height curve;
here, the purpose of the filtering is to reduce noise of a three-dimensional point cloud in the depth map, and after determining a target height threshold, obtaining a position of a fastener by using an intersection point of the target height threshold and a target height curve;
wherein the longitudinal coordinate satisfies the following formula:
yk∈[y1-Δy,y2+Δy](ii) a Wherein, ykRepresenting the longitudinal coordinate, y, of the railway fastener area1And y2And a height coordinate of the rail area is represented, and delta y is a second preset threshold value.
In this embodiment, the railway fastener is longitudinally self-adaptive in height position. On the processed depth map D '(i, j), the height of the fastener is higher than the height of the surrounding object and within a certain range, and therefore, summing D' (i, j) along the x direction can obtain the variation curve of the height of the fastener in the y direction. Because the depth map has large noise and the fluctuation range of the height variation curve is large and not smooth, in order to reduce the noise of the three-dimensional point cloud in the depth map, the three-dimensional point cloud is filtered, and a moving average filter is defined as
Wherein HiIs a filtered sequence of heights, hiFor the height sequence before filtering, win is the window width moving along the data. After filtering, the original data are smoother, and a target height threshold value T is sethThe intersection point of the target height threshold and the filtered target height curve is used to obtain the area range of the fastener in the longitudinal direction, and the final position is determined, that is, the intersection point of the target height threshold and the filtered target height curve is used to obtain the position of the fastener, that is, as shown in fig. 3. The intersections y1 and y2 determine the longitudinal position of the railway fastener while providing a small threshold Δ y to ensure the integrity of the fastener when the railway fastener is zone-split. Longitudinal position range y of railway fastenerkComprises the following steps:
yk∈[y1-Δy,y2+Δy](ii) a Wherein, ykRepresenting the longitudinal coordinate, y, of the railway fastener area1And y2And a height coordinate of the rail area is represented, and delta y is a second preset threshold value.
It should be noted that, considering the existence of the missing or missing elastic strip, etcSituation, setting a height threshold ThIn the crosstie position, as shown in FIG. 3, the spring bar area and the nut area, both of which are well above ThTherefore, the railway fastener has defects and the like, and the positioning and the division of the area cannot be influenced, so the stability of the provided positioning and dividing method is strong.
Optionally, the step 130 includes:
step 131, performing mean filtering on the area of the railway fastener according to the pixels of the intensity image;
it should be noted that, when acquiring track sample data, the acquisition sample device is preferably a 3D camera, i.e., a structured light camera, and due to external interference such as sight shielding, obstacles, and the like, there are many discrete noise points in the intensity map of the railway fastener, which affect the accuracy of subsequent feature extraction, and seriously interfere with the image detection result, so the invention provides to denoise the fastener intensity map by using a mean value filtering algorithm, improve the image quality, and disperse the noise to surrounding pixels by changing the image pixel value into the pixel average value of itself and surrounding 8 pixels, thereby weakening the influence of the noise on the image.
And step 132, determining the sample to be detected with the position information and the first target sample with the position information according to the filtered position information of the railway fastener and the railway fastener.
In this embodiment, a template is set for the target area in the intensity map according to the pixels of the intensity image, the template includes the target pixel and 8 pixels around the target pixel, and all pixels in the template are summed and averaged to replace the target pixel. Assuming the target pixel in the intensity map as I (I, j), a 3 × 3 template is selected, the template is composed of the target pixel and its surrounding 8 pixels, the pixel mean value of all pixels in the template is determined by the following formula as the target pixel value g (I, j) of I (I, j),
where n is the total number of pixels in the template and s is the selected 3x3 domain range. The noise reduction can be obviously seen after the railway fastener sample image is processed by utilizing the mean value filtering, the image is smoother, particularly, the railway fastener image before noise reduction in figure 4a and the railway fastener image after noise reduction in figure 4b can be seen, a plurality of noise points obviously disappear by utilizing the railway fastener image after noise reduction in the mean value, and pixels of the image are clearer.
Optionally, the step 200 includes:
expanding the defective fastener sample of the first target sample according to a generative antagonistic neural network algorithm, and determining an expanded second target sample;
wherein the generative confrontation neural network algorithm comprises a generator and an arbiter; the generator is used for expanding according to the defective fastener sample; the discriminator is used to determine the categories of the expanded sample and the original sample.
It should be noted that, in order to solve the problem of sample imbalance, the present invention proposes to expand the number of defect samples by using a generative confrontation neural network algorithm, learn features from the overall distribution of the defect samples, train a new neural network model on the original data set and the generated data set, and complete the fastener defect detection, where the generative confrontation neural network algorithm is a generative confrontation neural network DCGAN.
In this embodiment, the generative confrontation neural network GAN is a generative model, and learning training is completed through a confrontation process composed of two generative confrontation neural networks, which are respectively the discriminator D and the generator G. The discriminator is used for distinguishing the real samples from the generated samples, and the generator learns the distribution characteristics of the real samples to generate the defect samples. To enable control of the distribution p of the defect samples generatedgDefining a priori noise distribution pz(z) and then mapping the data space with G, where G (z; θ)g) Is represented by a convolutional neural network with a parameter thetagA differentiable function of; meanwhile, the discriminator is denoted by D, and outputs the possibility of judging a real sample and generating a sample. Training D to correctly distinguish between real and generated samples, and G to learn distribution pgAs close as possible to p with real samplesdEquation of the objective functionCan be expressed as:
the DCGAN adopted by the invention is an improvement on an original GAN network, so that the training of the GAN is more stable, and a generator can learn deeper sample characteristics. The specific improvements are as follows: replacing the pooling layer with a full convolution layer, adding a Batch Normalization (Batch Normalization) operation in the generator, where a Linear rectification function (ReLU function) is used as the activation function, and in the arbiter, a LeakyRelu function (ReLU function is used to set all negative values to zero, and conversely, a Leaky Relu function is used to give a non-zero slope to all negative values) is used as the activation function, etc.
The network structure is shown in fig. 5, the generator and the discriminator are respectively composed of four convolution layers, wherein the generator inputs 100-dimensional vectors obeying gaussian distribution, and generates images with the same dimensionality as the original samples after upsampling convolution. The discriminator is equivalent to a classifier, the input includes real samples and generated samples, and the output is the category of the real samples and the generated samples.
Optionally, the step 300 includes:
merging the second target sample and the first target sample into a target training set;
generating a detection model for detecting the railway fastener according to the target training set;
and the middle layers of the target training set adopt a convolution kernel of 3x 3.
In this embodiment, when a DCGAN is used to generate a railway fastener defect sample, a 100-dimensional vector is randomly generated, the convolution layer 1 is used to expand the scale to 4 × 1024, 4 convolution kernels with the size of 5 × 5 and deconvolution with the step size of 2 are used for the expansion tensor, and after each deconvolution, the ReLu function is used as an activation function and the Batch Normalization operation is performed. And finally, generating 64 x 1 samples of the defective railway fasteners. The input of the discriminator network comprises a real sample and a generated sample, the image is classified by using four convolution layers, similarly, the convolution kernel size is 5x5, the step length is 2, and finally, a full connection layer is added to complete true and false discrimination classification.
After the model is trained, a 100-dimensional vector is randomly generated and input into a generator, the output is the generated railway fastener defect sample, samples of different railway fastener defect types are respectively trained in the training process, the network is ensured not to combine different defect types, and unreasonable defect samples are generated.
In the railway fastener identification part, the invention proposes that ResNet is used as a backbone network to build a neural network with 18 layers of 8 residual blocks, a preset function, namely a focal loss function is used as a loss function, original data and generated data are combined together to be used as a new training set, and the whole detected network structure is shown in figure 5.
In the network structure, the input layer is a railway fastener image, the middle layer comprises 5 convolution layers and 1 full-connection layer, wherein 8 short-connection residual blocks use convolution kernels of 3x3, and the output layer has dimension of 5 and corresponds to 4 defective railway fasteners and normal railway fasteners.
Optionally, after step 200, the method further includes:
obtaining the type of the railway fastener after the position information is determined; the railway fastener types at least include: normal railway fasteners, missing nut railway fasteners, spring strip broken railway fasteners, missing spring strip railway fasteners;
according to the type of the railway fastener, corresponding labels are set for the railway fastener after the position information is determined, and each label comprises: the type of railway fastener.
It should be noted that the tag only includes the type of the railway fastener and does not include the location information, and after the location information is obtained, the railway fastener is separated from the scene picture, and here, the tag setting operation is performed only on the sample of the railway fastener.
In this embodiment, the acquisition sample device is preferably a 3D camera, i.e., a structured light camera, for example, the acquisition sample device acquires 500 railway scene images in total, and acquires 754 railway fastener samples in total after the fastener positioning and splitting are completed, i.e., after the step 100 and 200 are completed, the sample lacks marking information, so that the method can acquire the type of the railway fastener after the position information is determined, set a corresponding label for the railway fastener after the position information is determined, and complete the marking of each image, i.e., 659 normal samples and 95 defect samples are acquired in total after the positioning, splitting and marking are performed, wherein 32 broken samples, 27 nut missing samples and 36 elastic bar missing samples are acquired, and each defect sample is expanded by 150 generated antagonistic neural networks. And combining the generated samples with the original samples to generate a final data set, wherein the final data set comprises 1204 pieces of data in total, and the data set is divided into a training set and a testing set according to the proportion of 7: 3.
In a specific embodiment, the invention utilizes a SICK 3D camera and an Osela 5mW line laser with 660nm wavelength in the prior art to build a structured light measurement device. During actual test, the distance range of rail surface to fastener is between 210mm to 310mm, and it is within 0.1mm to measure this degree of depth scope error to survey sample collection equipment through the experiment. The structured light measuring equipment is installed on an inspection train, the inspection train runs at a fixed speed, on-site dynamic testing is completed, precision verification is carried out on the on-site dynamic testing, and the measurement precision is measured to be within 0.8mm through experiments.
In the method, a railway scene sample is acquired by structured light equipment, and the method comprises the following steps: a depth map and an intensity map. The fasteners in the depth map can be further segmented by the corresponding relation between the depth map and the strength map through segmenting the fasteners in the depth map by the method. And generating an expanding sample number of the confrontation network, solving the problem of sample imbalance, combining the original samples to form a new training set, and training the railway fastener detection network. In addition, when online detection is carried out, railway sample images can be collected in real time and segmented, then prediction is carried out by using a detection model trained offline, and a detection result can be obtained quickly and accurately.
Specifically, in the step 300, a detection model for detecting the railway fastener is generated according to the second target sample and the first target sample; it should be noted that, in order to adapt to the network input shown in fig. 5, the image of the data set needs to be scaled to 64 × 64, the fastener defect detection adopts an 18-layer network (ResNet18) with weight as a backbone network, a deep learning algorithm (Adam algorithm) is selected as an optimizer, the batch size (batch _ size) is 64, the iteration number (epochs) is set to 50, the experiment is run on a preset graphics card, and the running time is 5 min.
As shown in fig. 6, fig. 6 is a curve of the loss during the training process of the model, and it can be seen that the loss is continuously decreased with the increase of the epoch, and reaches the minimum and is kept stable when the epoch is about 45, at which time the loss is 0.085, which indicates that the training of the detection model is completed.
Specifically, in the step 400, as shown in fig. 7, the average accuracy curves of the four fastener types in the test set are shown in fig. 7, and it can be seen that when the epoch is about 42, the average accuracy already reaches about 97.6%, and the detection performance of the model is good.
Through the steps 100 to 400, 362 samples of the test set are taken, wherein 198 samples of the normal samples, 55 samples of the broken samples, 53 samples of the nut missing samples and 56 samples of the elastic strip missing samples are taken, and the confusion matrix on the test set is shown in table 1. As can be seen from the figure, for three defect types of elastic strip loss, nut loss and elastic strip breakage, the model prediction is only 1, 2 and 1 prediction errors respectively, the average accuracy of the detection of the defective fastener reaches 97.6 percent, the test time is about 30s, about 12 images are detected per second on average, and the actual detection requirement is met.
Table 1: confusion matrix for test set
In summary, (1) the invention provides a railway fastener defect detection method combining a depth map and an intensity map based on deep learning, position information and texture information are fused, and the detection accuracy is as high as 97.6%; (2) the invention provides a depth map-based railway fastener positioning and segmenting method, which utilizes a fastener image longitudinal height self-adaptive method to complete automatic positioning and segmentation of a fastener; (3) aiming at the problem of unbalance of defective samples, the invention firstly provides a method for expanding the number of samples by using a generating type antagonistic neural network, the generated samples and original samples are combined to form a new data set, the detection accuracy of defective fasteners is improved, and the false detection alarm rate is reduced; (4) the invention designs a deep neural network by using ResNet18 as a backbone network to finish the detection of the defective fastener on a synthetic data set, and designs an experiment to finish the verification. The future work can be expanded around a network structure and a loss function, the network is reasonably designed according to the characteristics of the railway fastener, and the characteristics with higher distinguishability are extracted; for defects which are difficult to identify, regular terms are added in the loss function, accuracy of identifying difficult samples is improved, and performance of the model is further improved.
As shown in fig. 8, an embodiment of the present invention further provides a railway fastener defect detecting apparatus, including:
the first determining module 10 is configured to determine a to-be-detected sample with position information and a first target sample with position information according to the acquired image information of the railway fastener;
a second determining module 20, configured to expand the first target sample according to a target algorithm, and determine an expanded second target sample;
a generating module 30, configured to generate a detection model for detecting a railway fastener according to the second target sample and the first target sample;
and a third determining module 40, configured to input the sample to be detected into the detection model, and determine a detection result.
Optionally, the first determining module 10 includes:
the first acquisition submodule is used for acquiring railway fastener image information, and the railway fastener image information comprises a depth image and an intensity image; the pixels of the depth image and the intensity image correspond to each other one by one;
the first determining submodule is used for determining the position information of the railway fastener in the image information of the railway fastener according to the pixels of the depth image;
and the second determining submodule is used for determining the sample to be detected with the position information and the first target sample with the position information according to the position information of the railway fastener and the pixels of the intensity image.
Optionally, the first determining sub-module includes:
a first determining unit for determining a railway fastener area and a rail area according to pixels of the depth image;
a second determining unit configured to determine location information of the rail clip in the rail clip image information according to the rail clip area and the rail area.
Optionally, the second determining unit includes:
a first determining subunit for determining lateral coordinates of the railway clip from the railway clip area and the rail area;
the second determining subunit is used for determining the transverse position of the railway fastener according to the transverse coordinate and the depth image;
wherein the lateral coordinate satisfies the following formula:
xk∈[xa-dx,xa]∪[xa+h,xa+h+dx](ii) a Wherein x iskTransverse coordinate, x, representing the area of the railway fasteneraIs a lateral coordinate of a rail-side of the rail area, h is a rail width of the rail area, dxIs a first preset threshold.
Optionally, the second determining unit includes:
a third determining subunit, configured to filter the determined lateral position of the railway fastener along the lateral direction of the railway fastener according to the depth image, and determine a target height threshold and a target height curve of the railway fastener;
determining the longitudinal coordinate of the railway fastener according to the target height threshold and the target height curve;
wherein the longitudinal coordinate satisfies the following formula:
yk∈[y1-Δy,y2+Δy](ii) a Wherein, ykIndicating the railway buckleLongitudinal coordinate of the part area, y1And y2And a height coordinate of the rail area is represented, and delta y is a second preset threshold value.
Optionally, the second determining sub-module includes:
the filtering unit is used for carrying out mean value filtering on the area of the railway fastener according to the pixels of the intensity image;
and the third determining unit is used for determining the sample to be detected with the position information and the first target sample with the position information according to the filtered position information of the railway fastener and the railway fastener.
Optionally, the second determining module 20 includes:
the fourth determining unit is used for expanding the defective fastener sample of the first target sample according to a generative antagonistic neural network algorithm and determining an expanded second target sample;
wherein the generative confrontation neural network algorithm comprises a generator and an arbiter; the generator is used for expanding according to the defective fastener sample; the discriminator is used to determine the categories of the expanded sample and the original sample.
Optionally, the generating module 30 includes:
a merging unit, configured to merge the second target sample and the first target sample into a target training set;
the generating unit is used for generating a detection model for detecting the railway fastener according to the target training set;
and the middle layers of the target training set adopt a convolution kernel of 3x 3.
Optionally, the apparatus further comprises:
the acquisition module is used for acquiring the type of the railway fastener after the position information is determined; the railway fastener types at least include: normal railway fasteners, missing nut railway fasteners, spring strip broken railway fasteners, missing spring strip railway fasteners;
the processing module is used for setting corresponding labels for the railway fasteners after the position information is determined according to the types of the railway fasteners, and each label comprises: the type of railway fastener.
The embodiment of the present invention further provides a readable storage medium, where a program is stored, and when the program is executed by a processor, the program implements each process of the embodiment of the method for detecting a defect of a railway fastener, and can achieve the same technical effect, and in order to avoid repetition, the detailed description is omitted here. The readable storage medium may be a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk.
Finally, it is also noted that, in the present invention, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or terminal apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.
While the preferred embodiments of the present invention have been described, it will be understood by those skilled in the art that various changes and modifications may be made without departing from the spirit and scope of the invention as defined in the following claims.
Claims (11)
1. A railway fastener defect detection method is characterized by comprising the following steps:
determining a sample to be detected with position information and a first target sample with the position information according to the acquired image information of the railway fastener;
expanding the first target sample according to a target algorithm, and determining an expanded second target sample;
generating a detection model for detecting the railway fastener according to the second target sample and the first target sample;
and inputting the sample to be detected into the detection model, and determining a detection result.
2. The method of claim 1, wherein determining the sample to be tested having position information and the first target sample having position information based on the acquired railway fastener image information comprises:
acquiring railway fastener image information, wherein the railway fastener image information comprises a depth image and an intensity image; the pixels of the depth image and the intensity image correspond to each other one by one;
determining the position information of the railway fastener in the railway fastener image information according to the pixels of the depth image;
and determining a sample to be detected with the position information and a first target sample with the position information according to the position information of the railway fastener and the pixels of the intensity image.
3. The method of claim 2, wherein determining location information for a railroad clip in the railroad clip image information from pixels of the depth image comprises:
determining a railway fastener area and a rail area according to pixels of the depth image;
and determining the position information of the railway fastener in the railway fastener image information according to the railway fastener area and the rail area.
4. The method of claim 3, wherein said determining location information for a rail clip in said rail clip image information based on said rail clip area and said rail area comprises:
determining lateral coordinates of the railway clip from the railway clip area and the rail area;
determining a lateral position of the railway fastener according to the lateral coordinate and the depth image;
wherein the lateral coordinate satisfies the following formula:
xk∈[xa-dx,xa]∪[xa+h,xa+h+dx](ii) a Wherein x iskTransverse coordinate, x, representing the area of the railway fasteneraIs a lateral coordinate of a rail-side of the rail area, h is a rail width of the rail area, dxIs a first preset threshold.
5. The method of claim 4, wherein said determining location information for a rail clip in said rail clip image information based on said rail clip area and said rail area comprises:
according to the depth image, filtering the determined transverse position of the railway fastener along the transverse direction of the railway fastener, and determining a target height threshold value and a target height curve of the railway fastener;
determining the longitudinal coordinate of the railway fastener according to the target height threshold and the target height curve;
wherein the longitudinal coordinate satisfies the following formula:
yk∈[y1-Δy,y2+Δy](ii) a Wherein, ykRepresenting the longitudinal coordinate, y, of the railway fastener area1And y2And a height coordinate of the rail area is represented, and delta y is a second preset threshold value.
6. The method of claim 2, wherein determining the sample to be tested having position information and the first target sample having position information based on the position information of the railway fastener and the pixels of the intensity image comprises:
performing mean filtering on the region of the railway fastener according to the pixels of the intensity image;
and determining a sample to be detected with position information and a first target sample with position information according to the filtered position information of the railway fastener and the railway fastener.
7. The method of claim 1, wherein the expanding the first target sample according to a target algorithm and determining an expanded second target sample comprises:
expanding the defective fastener sample of the first target sample according to a generative antagonistic neural network algorithm, and determining an expanded second target sample;
wherein the generative confrontation neural network algorithm comprises a generator and an arbiter; the generator is used for expanding according to the defective fastener sample; the discriminator is used to determine the categories of the expanded sample and the original sample.
8. The method of claim 1, wherein generating a testing model for testing railroad fasteners from the second target specimen and the first target specimen comprises:
merging the second target sample and the first target sample into a target training set;
generating a detection model for detecting the railway fastener according to the target training set;
and the middle layers of the target training set adopt a convolution kernel of 3x 3.
9. The method of claim 1, wherein the first target sample is augmented according to a target algorithm, and wherein after determining the augmented second target sample, the method further comprises:
obtaining the type of the railway fastener after the position information is determined; the railway fastener types at least include: normal railway fasteners, missing nut railway fasteners, spring strip broken railway fasteners, missing spring strip railway fasteners;
according to the type of the railway fastener, corresponding labels are set for the railway fastener after the position information is determined, and each label comprises: the type of railway fastener.
10. A railway fastener defect detecting device, characterized by includes:
the first determining module is used for determining a sample to be detected with position information and a first target sample with the position information according to the acquired image information of the railway fastener;
the second determination module is used for expanding the first target sample according to a target algorithm and determining an expanded second target sample;
the generating module is used for generating a detection model for detecting the railway fastener according to the second target sample and the first target sample;
and the third determining module is used for inputting the sample to be detected into the detection model and determining a detection result.
11. A readable storage medium, having stored thereon a program which, when executed by a processor, carries out the steps of the method of detecting defects in railway fasteners of any one of claims 1 to 9.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110950466.8A CN113689392A (en) | 2021-08-18 | 2021-08-18 | Railway fastener defect detection method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110950466.8A CN113689392A (en) | 2021-08-18 | 2021-08-18 | Railway fastener defect detection method and device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113689392A true CN113689392A (en) | 2021-11-23 |
Family
ID=78580460
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110950466.8A Pending CN113689392A (en) | 2021-08-18 | 2021-08-18 | Railway fastener defect detection method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113689392A (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114309652A (en) * | 2022-01-06 | 2022-04-12 | 北京铁科首钢轨道技术股份有限公司 | Manufacturing method of 3D printed railway fastener elastic strip |
| CN114549438A (en) * | 2022-02-10 | 2022-05-27 | 浙江大华技术股份有限公司 | Reaction kettle buckle detection method and related device |
| CN114663882A (en) * | 2022-05-24 | 2022-06-24 | 昆山斯沃普智能装备有限公司 | Electric automobile chassis scratch three-dimensional detection method based on deep learning |
| CN114820599A (en) * | 2022-06-27 | 2022-07-29 | 南通奥尔嘉橡塑有限公司 | Injection molding buckle defect detection method and device based on computer vision |
-
2021
- 2021-08-18 CN CN202110950466.8A patent/CN113689392A/en active Pending
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114309652A (en) * | 2022-01-06 | 2022-04-12 | 北京铁科首钢轨道技术股份有限公司 | Manufacturing method of 3D printed railway fastener elastic strip |
| CN114549438A (en) * | 2022-02-10 | 2022-05-27 | 浙江大华技术股份有限公司 | Reaction kettle buckle detection method and related device |
| CN114663882A (en) * | 2022-05-24 | 2022-06-24 | 昆山斯沃普智能装备有限公司 | Electric automobile chassis scratch three-dimensional detection method based on deep learning |
| CN114663882B (en) * | 2022-05-24 | 2022-09-06 | 昆山斯沃普智能装备有限公司 | Electric automobile chassis scratch three-dimensional detection method based on deep learning |
| CN114820599A (en) * | 2022-06-27 | 2022-07-29 | 南通奥尔嘉橡塑有限公司 | Injection molding buckle defect detection method and device based on computer vision |
| CN114820599B (en) * | 2022-06-27 | 2022-09-02 | 南通奥尔嘉橡塑有限公司 | Injection molding buckle defect detection method and device based on computer vision |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113689392A (en) | Railway fastener defect detection method and device | |
| CN115272334B (en) | Method for detecting tiny defects on surface of steel rail under complex background | |
| CN111080622B (en) | Neural network training method, workpiece surface defect classification and detection method and device | |
| CN112330591B (en) | Steel rail surface defect detection method and device capable of achieving sample-less learning | |
| CN105809668B (en) | The body surface deformation behaviour extracting method of three-dimensional point cloud is scanned based on line | |
| CN110211101A (en) | A kind of rail surface defect rapid detection system and method | |
| Wu et al. | Train rail defect classification detection and its parameters learning method | |
| CN108711148A (en) | A kind of wheel tyre defect intelligent detecting method based on deep learning | |
| CN111311567A (en) | Method for identifying fastener and steel rail diseases of track line image | |
| Chen et al. | CUFuse: Camera and ultrasound data fusion for rail defect detection | |
| CN106441107A (en) | Method for automatic detection of steel rail abrasion | |
| CN111754460A (en) | Method, system and storage medium for automatically detecting gap of point switch | |
| CN106274979A (en) | Rail wear automatic detection device | |
| CN109360188A (en) | A kind of detection method of high iron catenary equipotential line defective mode | |
| Zhao et al. | Image-based comprehensive maintenance and inspection method for bridges using deep learning | |
| CN115131330A (en) | Method and system for detecting and measuring surface defects of steel rail by fusing multi-source sensors | |
| CN115526864A (en) | Steel rail surface defect detection method based on improved characteristic pyramid network and metric learning | |
| CN108508023A (en) | The defect detecting system of end puller bolt is contacted in a kind of railway contact line | |
| CN103258218B (en) | Masking-out detects skeleton generating method, device, defect inspection method and device | |
| Aydin et al. | Defect classification of railway fasteners using image preprocessing and alightweight convolutional neural network | |
| CN114067296A (en) | Method and device for identifying surface defects of steel rail | |
| Hassan et al. | Road crack detection using the particle filter | |
| CN115393346A (en) | Detection method for flaw detection of surface of steel rail | |
| KR101782366B1 (en) | Vision inspection method based on learning data using normalization of sample scale | |
| CN114693617A (en) | Method, system, equipment and storage medium for detecting defects of railway steel rails |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |