CN112733742A - Deep learning-based fault detection method for round pin of lower pull rod of railway wagon - Google Patents
Deep learning-based fault detection method for round pin of lower pull rod of railway wagon Download PDFInfo
- Publication number
- CN112733742A CN112733742A CN202110048089.9A CN202110048089A CN112733742A CN 112733742 A CN112733742 A CN 112733742A CN 202110048089 A CN202110048089 A CN 202110048089A CN 112733742 A CN112733742 A CN 112733742A
- Authority
- CN
- China
- Prior art keywords
- round pin
- image
- pull rod
- round
- network
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/20—Scenes; Scene-specific elements in augmented reality scenes
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/22—Matching criteria, e.g. proximity measures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/0002—Inspection of images, e.g. flaw detection
- G06T7/0004—Industrial image inspection
- G06T7/0008—Industrial image inspection checking presence/absence
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10004—Still image; Photographic image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30248—Vehicle exterior or interior
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- General Physics & Mathematics (AREA)
- Physics & Mathematics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Evolutionary Biology (AREA)
- Evolutionary Computation (AREA)
- Bioinformatics & Computational Biology (AREA)
- General Engineering & Computer Science (AREA)
- Artificial Intelligence (AREA)
- Life Sciences & Earth Sciences (AREA)
- Multimedia (AREA)
- Quality & Reliability (AREA)
- Image Analysis (AREA)
Abstract
A method for detecting faults of a round pin of a lower pull rod of a railway wagon based on deep learning belongs to the technical field of image detection. The invention aims to solve the problems of low detection efficiency and high detection accuracy rate of the existing detection and identification method. Firstly, inputting a lower pull rod image into a segmentation network of a lower pull rod round pin to predict round hole areas of the lower pull rod round pin and circular pins which are not installed, judging whether the round pin has a loss fault, and if the round pin is lost, determining the loss position of the round pin; the round pin sub-image is obtained by intercepting the round pin positioned in the round pin area from the lower pull rod image according to the position of the round pin in the segmentation result, and the round pin sub-image is input into a classification network of the lower pull rod round pin for carrying out round pin sub-image classification; if the round hole is identified, determining the position of the round hole without the round pin; if the cotter pin is identified to be lost, determining the cotter pin lost position; if no fault is detected, the next pull-down bar image is processed. The method is mainly used for fault detection of the round pin of the lower pull rod of the railway wagon.
Description
Technical Field
The invention relates to a method for detecting a fault of a round pin of a lower pull rod. Belonging to the technical field of image detection.
Background
The opening pin of the lower pull rod of the truck is lost, and the round pin can cause the falling of the lower pull rod after being lost, thereby causing major accidents and causing serious loss to personal property.
The drop of the lower pull rod is one of the most frequent factors of the railway freight car in application accidents, fault detection is basically carried out in a mode of manually checking images in fault detection of the opening pin of the lower pull rod, and due to the fact that the conditions of fatigue, omission and the like are easily caused in the working process of car inspection personnel, the appearance of missed inspection and wrong inspection is caused, and the driving safety is influenced. And the mode of adopting image automatic identification not only can save a large amount of manpower and materials, can effectively reduce the influence of human factor moreover, reduces and misses and the false retrieval, improves detection efficiency and stability. In recent years, image processing, deep learning, and artificial intelligence have been developed, and technologies have become mature. Therefore, the fault identification of the round pin of the lower pull rod is carried out by adopting image processing and deep learning, and the detection accuracy can be effectively improved.
Disclosure of Invention
The invention aims to solve the problems of low detection efficiency and high detection accuracy rate of the existing detection and identification method.
A rail wagon lower pull rod round pin fault detection method based on deep learning comprises the following steps:
acquiring a to-be-detected pull-down rod image, inputting the pull-down rod image into a segmentation network of a pull-down rod round pin, predicting a pull-down rod round pin area and a round hole area without the round pin, and acquiring a predicted multi-value image, wherein a value 1 in the multi-value image is the round pin area, and a value 2 in the multi-value image is the round hole area without the round pin; judging whether the round pin has a loss fault or not according to the number of the round pin areas and the number of the round hole areas without the round pins, and if the round pins are lost, determining the lost positions of the round pins;
if the round pin area is located, split pin loss detection is performed: according to the position of the round pin in the segmentation result of the segmentation network of the round pin of the lower pull rod, a round pin subimage is obtained by intercepting from the lower pull rod image, and the round pin subimage is input into a classification network of the round pin of the lower pull rod for classification; the classification result comprises a normal lower pull rod round pin, a cotter pin lost round pin and a round hole without the round pin; if the round hole is identified, determining the position of the round hole without the round pin; if the cotter pin is identified to be lost, determining the cotter pin lost position; if no fault is detected, the next pull-down bar image is processed.
Further, the process of obtaining the image of the lower link to be detected includes the steps of:
acquiring a real vehicle passing image as an original image to be detected; and roughly positioning the lower pull rod in the original image according to the wheel base information and the prior information of the lower pull rod part to obtain a lower pull rod image of the lower pull rod image.
Further, before inputting the down-link image into the segmentation network of the down-link round pin for prediction, local histogram equalization enhancement needs to be performed on the down-link image.
Further, the process of judging whether the round pin has the loss fault or not according to the number of the round pin areas and the round hole areas without the round pins comprises the following steps: and if the number of the round pin areas is less than 2 and the number of the round hole areas is more than 4, judging that the round pins are lost.
Further, the segmentation network of the round pin of the lower pull rod is a similar dynamic graph convolution network, and the processing process of the similar dynamic graph convolution network comprises the following steps:
firstly, a basic convolution neural network is utilized to generate a segmentation result and a corresponding characteristic map, which is called a coarse characteristic map;
taking the rough characteristic diagram as input, obtaining a characteristic diagram, namely a backbone network characteristic diagram, by utilizing a backbone network, taking each point on the backbone network characteristic diagram as a node of the diagram, and taking the similarity between the nodes as the weight of the edge of the adjacent matrix to obtain an adjacent matrix A;
then, graph convolution is carried out: z ═ σ (AXW);
x is an input feature diagram, namely a backbone network feature diagram; w is the network parameter to be learned, σ is a nonlinear activation function, and Z is a feature map after the map convolution;
obtaining a fine characteristic diagram after graph convolution; and connecting the fine feature map and the coarse feature map, and then using 1-by-1 convolution to obtain a final prediction result.
Further, the process of obtaining the adjacency matrix a by using each point on the backbone network characteristic diagram as a node of the diagram and using the similarity between the nodes as a weight of an edge of the adjacency matrix includes the following steps:
feature x for two nodesiAnd xjThe similarity of these two features is measured using two linear transformations φ and φ':
F(xi,xj)＝φ(xi)Tφ′(xj)
and then performing softmax normalization to obtain an adjacency matrix A, wherein the elements in the adjacency matrix A are as follows:
and N is the number of all edges connected with the ith node.
Further, the split network of drop link round pins employs a ResNet50 network.
Further, the training process of the class dynamic graph convolution network comprises the following steps:
firstly, a basic convolution neural network is utilized to generate a segmentation result and a corresponding characteristic map, which is called a coarse characteristic map;
taking the rough characteristic diagram as input, obtaining a characteristic diagram, namely a backbone network characteristic diagram, by utilizing a backbone network, taking each point on the backbone network characteristic diagram as a node of the diagram, and taking the similarity between the nodes as the weight of the edge of the adjacent matrix to obtain an adjacent matrix A;
and (3) performing dynamic sampling point selection and composition based on a segmentation result predicted by a basic convolutional neural network: selecting points by utilizing a segmentation result P predicted by a basic convolutional neural network and a result G marked in a training set, and selecting a point set of target Samples of each category based on a Samples formula:
Samples＝P-P∩G+G-P∩G+r·P∩G
wherein the first term P-P.andgate G is a difficult negative sample; the second term G-P.andgate G is a difficult positive sample; the last term P.andgate.G is a simple positive sample; when selecting points, all difficult samples are selected, and then simple samples are selected according to the ratio r;
after the dynamic sampling point selection composition is finished, graph convolution is carried out, and the feature graph X input in the graph convolution process in the training process is a feature graph corresponding to the dynamic sampling point selection composition;
obtaining a fine characteristic diagram after graph convolution; connecting the fine feature map and the coarse feature map, and then using 1-by-1 convolution to obtain a final prediction result;
training was performed using an AdaBelief optimizer.
Further, the loss function of the segmented network of drop link round pins is:
L＝α·lc+β·lf+γ·la
wherein lcFor coarse segmentation loss,/fFor fine segmentation loss,/aLoss is monitored for assistance; alpha, beta, gamma balance parameters.
Further, the classification network of the lower link round pin adopts a cross entropy loss function as a loss function of classification.
Advantageous effects
1. And the automatic image identification mode is used for replacing manual detection, so that the detection efficiency and accuracy are improved.
2. The deep learning algorithm is applied to automatic fault recognition of the round pin of the lower pull rod, and the stability and the precision of the whole algorithm are improved.
3. The deep learning network is used for cutting the positioning round pin component and then the classification network is used for cotter pin detection, so that the system identification accuracy is high.
4. Round pin loss is detected again in the classification network, and missing detection caused by split pin loss based on segmentation for the first time due to noise interference is effectively avoided.
5. The system uses the similar dynamic graph convolution as a network for segmenting the round pin of the lower pull rod, so that the context information can be better learned and fused, the segmentation of the components is accurate, and the final fault identification effect of the system can be improved.
6. And a partial marking and initial training mode is adopted to manufacture the segmentation data set, so that the data set manufacturing efficiency is improved.
7. An AdaBelief optimizer is adopted in the deep learning model training process in the system, convergence is rapid, and accuracy is high.
8. The system performs subsequent identification after performing adaptive histogram equalization enhancement on the coarse positioning image, avoids the influence of darker noise and image on the identification effect, and has high system fault detection accuracy.
Drawings
FIG. 1 is a schematic flow diagram of an embodiment;
FIG. 2 is a flow chart of fault identification;
FIG. 3 is a flow chart of weight coefficient calculation;
FIG. 4 is a diagram illustrating a structure of a residual block;
FIG. 5 is a schematic diagram of a dynamics-like convolutional network.
Detailed Description
It should be noted that, in the present invention, the embodiments disclosed in the present application may be combined with each other without conflict.
The first embodiment is as follows: the present embodiment is specifically described with reference to figure 1,
the embodiment is a deep learning-based fault detection method for a round pin of a lower pull rod of a railway wagon, and the method comprises the following steps of:
acquiring a to-be-detected pull rod image, inputting the pull rod image into a segmentation network of a pull rod round pin to predict round hole areas of the pull rod round pin and the round pin which is not installed, and acquiring a predicted multi-value image, wherein 1 value in the multi-value image is the round pin area, and 2 value is the round hole area which is not installed with the round pin; judging whether the round pin has a loss fault or not according to the number of the round pin areas and the number of the round hole areas without the round pins, and if the round pins are lost, determining the lost positions of the round pins;
as soon as the round pin area is located, a cotter pin loss detection is performed: according to the position of the round pin in the segmentation result of the segmentation network of the round pin of the lower pull rod, a round pin subimage is obtained by intercepting from the lower pull rod image, and the round pin subimage is input into a classification network of the round pin of the lower pull rod for classification; the classification result comprises a normal lower pull rod round pin, a cotter pin lost round pin and a round hole without the round pin; if the round hole is identified, determining the position of the round hole without the round pin; if the cotter pin is identified to be lost, determining the cotter pin lost position; if no fault is detected, the next pull-down bar image is processed.
The second embodiment is as follows:
the embodiment is a method for detecting faults of a round pin of a lower pull rod of a railway wagon based on deep learning, and the process of obtaining the image of the lower pull rod to be detected comprises the following steps:
acquiring a real vehicle passing image as an original image to be detected; and roughly positioning the lower pull rod in the original image according to the wheel base information and the prior information of the lower pull rod part to obtain a lower pull rod image of the lower pull rod image.
Other steps and parameters are the same as in the first embodiment.
The third concrete implementation mode:
the embodiment is a method for detecting faults of a round pin of a lower pull rod of a railway wagon based on deep learning, and before a lower pull rod image is input into a segmentation network of the round pin of the lower pull rod for prediction, local histogram equalization enhancement needs to be carried out on the lower pull rod image.
Other steps and parameters are the same as in the first or second embodiment.
The fourth concrete implementation mode:
the embodiment is a deep learning-based fault detection method for a round pin of a lower pull rod of a railway wagon, and the process of judging whether the round pin has a loss fault or not according to the number of round pin areas and round hole areas without round pins comprises the following steps: and if the number of the round pin areas is less than 2 and the number of the round hole areas is more than 4, judging that the round pins are lost.
Other steps and parameters are the same as in one of the first to third embodiments.
The fifth concrete implementation mode:
the embodiment is a deep learning-based fault detection method for a round pin of a lower pull rod of a railway wagon, a segmentation network of the round pin of the lower pull rod is a similar dynamic graph convolution network, and the similar dynamic graph convolution network processing process comprises the following steps:
firstly, a basic convolution neural network is utilized to generate a segmentation result and a corresponding characteristic map, which is called a coarse characteristic map;
taking the rough characteristic diagram as input, obtaining a characteristic diagram, namely a backbone network characteristic diagram, by utilizing a backbone network, taking each point on the backbone network characteristic diagram as a node of the diagram, and taking the similarity between the nodes as the weight of the edge of the adjacent matrix to obtain an adjacent matrix A;
then, graph convolution is carried out: z ═ σ (AXW);
x is an input feature diagram, namely a backbone network feature diagram; w is the network parameter to be learned, σ is a nonlinear activation function, and Z is a feature map after the map convolution;
obtaining a fine characteristic diagram after graph convolution; and connecting the fine feature map and the coarse feature map, and then using 1-by-1 convolution to obtain a final prediction result.
Other steps and parameters are the same as in one of the first to fourth embodiments.
The sixth specific implementation mode:
the embodiment is a deep learning-based rail wagon pull rod round pin fault detection method, each point on a main network characteristic diagram is used as a node of the diagram, and the similarity between the nodes is used as the weight of the edge of an adjacent matrix, so that the process of obtaining the adjacent matrix A comprises the following steps:
feature x for two nodesiAnd xjThe similarity of these two features is measured using two linear transformations φ and φ':
F(xi,xj)＝φ(xi)Tφ′(xj)
and then performing softmax normalization to obtain an adjacency matrix A, wherein the elements in the adjacency matrix A are as follows:
and N is the number of all edges connected with the ith node.
Other steps and parameters are the same as those in the fifth embodiment.
The seventh embodiment:
the embodiment is a deep learning-based fault detection method for a round pin of a lower pull rod of a railway wagon, and a ResNet50 network is adopted as a segmentation network of the round pin of the lower pull rod.
Other steps and parameters are the same as in one of the first to sixth embodiments.
The specific implementation mode is eight:
the embodiment is a deep learning-based method for detecting faults of round pins of a lower pull rod of a railway wagon, and the training process of a similar dynamic graph convolution network comprises the following steps:
firstly, a basic convolution neural network is utilized to generate a segmentation result and a corresponding characteristic map, which is called a coarse characteristic map;
taking the rough characteristic diagram as input, obtaining a characteristic diagram, namely a backbone network characteristic diagram, by utilizing a backbone network, taking each point on the backbone network characteristic diagram as a node of the diagram, and taking the similarity between the nodes as the weight of the edge of the adjacent matrix to obtain an adjacent matrix A;
and (3) performing dynamic sampling point selection and composition based on a segmentation result predicted by a basic convolutional neural network: selecting points by utilizing a segmentation result P predicted by a basic convolutional neural network and a result G marked in a training set, and selecting a point set of target Samples of each category based on a Samples formula:
Samples＝P-P∩G+G-P∩G+r·P∩G
wherein the first term P-P.andgate G is a difficult negative sample; the second term G-P.andgate G is a difficult positive sample; the last term P.andgate.G is a simple positive sample; when selecting points, all difficult samples are selected, and then simple samples are selected according to the ratio r;
after the dynamic sampling point selection composition is finished, graph convolution is carried out, and the feature graph X input in the graph convolution process in the training process is a feature graph corresponding to the dynamic sampling point selection composition;
obtaining a fine characteristic diagram after graph convolution; connecting the fine feature map and the coarse feature map, and then using 1-by-1 convolution to obtain a final prediction result;
training was performed using an AdaBelief optimizer.
Other steps and parameters are the same as in one of the first to seventh embodiments.
The specific implementation method nine:
the embodiment is a fault detection method for a round pin of a lower pull rod of a railway wagon based on deep learning, and a loss function of a segmentation network of the round pin of the lower pull rod is as follows:
L＝α·lc+β·lf+γ·la
wherein lcFor coarse segmentation loss,/fFor fine segmentation loss,/aLoss is monitored for assistance; alpha, beta, gamma balance parameters.
The other steps and parameters are the same as in embodiment eight.
The detailed implementation mode is ten:
the embodiment is a deep learning-based fault detection method for a round pin of a lower pull rod of a railway wagon, and a classification network of the round pin of the lower pull rod adopts a cross entropy loss function as a classification loss function.
Other steps and parameters are the same as in one of the first to ninth embodiments.
Examples
The embodiment is a deep learning-based method for detecting a fault of a round pin of a lower pull rod of a railway wagon, and as shown in fig. 2, the method specifically comprises the following steps:
1. establishing a sample data set
High-definition equipment is built around the track of the truck, and the truck acquires a high-definition image after passing through the equipment. The image is a sharp grayscale image.
The truck parts can be influenced by natural conditions such as rainwater, mud, oil, black paint and the like or artificial conditions. Also, there may be differences in the images taken at different sites. Thus, the drop-down bar images vary widely. Therefore, in the process of collecting the pull-down image data, the pull-down images under various conditions are collected as completely as possible to ensure the diversity.
The drop down link members will vary in configuration among different types of trucks. But some of the less common truck types have draw down bar sections that are more difficult to collect due to the greater frequency of occurrence of the differences between the different types. Therefore, all types of pull-down rod parts are collectively referred to as a class, and sample data sets are established according to the class, and are divided into a pull-down rod segmentation sample data set and a pull-down rod round pin sample data set.
a. The lower pull rod segmentation sample data set comprises the following steps: a grayscale image set and a marker image set. There is a one-to-one correspondence between the grayscale image data set and the marker image data set, i.e. one marker image per grayscale image.
According to the wheel base information and the prior information of the pull-down rod part, roughly positioning a high-definition gray image shot by equipment to determine a pull-down rod image, wherein the pull-down rod image is a gray image, and the pull-down rod image forms a gray image set.
In the process of determining the pull-down rod image by roughly positioning the high-definition gray image shot by the equipment according to the axle distance information and the prior information of the pull-down rod part, because the ratio r _ xs between the left starting position and the right starting position of the pull-down rod and the distance between the two axles is not changed, and the ratio r _ ys between the starting position and the ending position of the pull-down rod in the vertical direction and the height of the original image is not changed, the rough positioning of the pull-down rod can be completed according to the axle distance position and the ratio information.
Acquiring a segmented image of the pull-down rod part based on the pull-down rod image, wherein the segmented image of the marked pull-down rod part is a marked image, and the marked image forms a marked image set; the divided image of the pull-down rod member is a multi-valued image in which the round pin (including the missing of the cotter pin) is 1, the round hole to which the round pin is not attached is 2, and the remaining gray value is a background area of 0.
Aiming at the characteristics that a round hole without a round pin in a lower pull rod image is easy to segment from a background, a cotter pin is lost or the normal round pin is difficult to segment, the invention adopts a partial marking and initial training mode to manufacture a lower pull rod segmentation sample data set: firstly, manually segmenting and marking part of sample data A, obtaining a lower pull rod image sample segmentation network after training, predicting the rest sample data B to be segmented by using the weight, and manually correcting lost split pins with wrong segmentation or normal round pins to obtain a marker image set. The method can reduce the sample data set making time and improve the working efficiency of the whole system.
The pull-down rod image sample segmentation network and the pull-down rod round pin segmentation network are a framework, and the segmentation effects are different due to different weight coefficients in the networks. The pull-down rod image sample segmentation network has less training data, so that the segmentation effect of the pull-down rod image sample segmentation network is not good, but circular holes can be segmented well, the efficiency of making a training data set can be improved, and other categories need to be manually corrected in the segmentation effect to be used as training samples.
b. The lower pull rod round pin sample data set comprises: and the round pin of the normal lower pull rod and the round pin of the cotter pin are lost, and the round hole of the round pin is not installed. The lower pull rod round pin sample set can be manually intercepted, and the position of the round pin in the rough positioning lower pull rod image segmentation prediction result can be automatically intercepted by utilizing a segmentation network according to the lower pull rod round pin. The lower pull rod round pin sample data set is used for storing three types of images (a normal lower pull rod round pin, a cotter pin lost round pin and a round hole without the round pin) in different folders.
Although the establishment of the pull-down rod segmentation sample data set and the pull-down rod round pin sample data set comprises images under various conditions, the data amplification of the sample data set is still required to improve the stability of the algorithm. The amplification form comprises operations of image translation, image zooming, image mirroring and the like, wherein each operation is performed under random conditions, so that the diversity and applicability of the sample can be guaranteed to the greatest extent. And the sample data is continuously amplified according to the system operation effect.
2. Calculating the sample data set weights, as shown in fig. 3, includes the following processes:
2.1 image segmentation network
The similar dynamic graph convolution network is used as a division network of the lower pull rod round pin, and the division network of the lower pull rod round pin divides an image into three types of round pins (including the condition that cotter pins are lost), round holes without round pins and background areas. The class dynamic graph convolution network adopts a frame from coarse to fine, and as shown in fig. 5, the processing procedure of the class dynamic graph convolution network includes the following steps:
firstly, generating a rough segmentation result and a corresponding feature map by using a Basic convolutional neural Network (Basic Network), namely a rough feature map, then putting the rough feature map into a similar dynamic map convolution processing process for fine adjustment, then connecting the fine feature map and the rough feature map, and then obtaining a final prediction result by using 1 x 1 convolution;
the detailed structure of the dynamic-like graph convolution processing process mainly comprises two steps, wherein one step is graph construction, and the other step is graph convolution. When the graph is constructed, each point on the feature graph output by the backbone network is taken as a node of the graph through the backbone network (resnet 50). The graph convolution process in the network is carried out class by class, namely a class-by-class learning strategy is mainly used for carrying out picture composition on points belonging to the same class and preventing the points belonging to other classes from interfering with the learning process of the class, so that a segmentation result predicted by a basic convolutional neural network is required to be used as a filter to sample corresponding points of different classes to carry out picture composition. Then, a difficult sample (such as an edge portion of a pin of a draw bar, a noise disturbance which suddenly appears in an image, and the like) is sampled during the patterning process, and dynamic sampling is used to optimize the difficult sample. Then, the fine features are obtained through the graph convolution process, and then, a 1-by-1 convolution is carried out to obtain the same shape as the previous coarse features, so that the later combination and prediction are convenient.
The graph is constructed in a manner that the similarity between nodes is used as the weight of the edges of the adjacent matrix. Since the patterning in the graph convolution is actually building the adjacency matrix. In particular, if for the characteristics of two nodes, xiAnd xjDefining two linear transformations, φ and φ', which can be learned to measure the similarity of the two features, we derive the formula:
F(xi,xj)＝φ(xi)Tφ′(xj)
the two linear transformations are learnable, which is equivalent to a dynamically learnt composition mode, and then the adjacency matrix a is obtained by performing softmax normalization, and the formula of the elements in the adjacency matrix a is as follows:
wherein N is the number of all edges connecting the ith node;
the composition mode is a very naive composition mode, and in the implementation, the composition mode is a composition mode for carrying out full connection on points of the same category.
Dynamic sampling point selection composition: in the training process, points are selected by using a segmentation result P predicted by a basic convolutional neural network and a result G marked in a training set, and the following formula of Samples is a point set sampled by each class of targets (round pins or round holes without round pins):
Samples＝P-P∩G+G-P∩G+r·P∩G
wherein, the first item P-P n G is a difficult negative sample, namely a point which does not belong to the current class but is sampled; the second term G-P.andgate G is a difficult positive sample, i.e., one that is originally in this class that has not been sampled; the last term P.andgate G is a simple positive sample, a point that belongs to this class and is sampled in. As can be seen from the Samples formula, all the difficult Samples are selected during the point selection, and then the simple Samples are selected according to a certain ratio r to assist the training process.
After the composition is finished, Graph convolution (Class-wise Graph reading) can be carried out to obtain a fine feature Graph; the following form is adopted when the graph is convoluted:
Z＝σ(AXW)
a is the adjacency matrix, X is an input feature map, W is the network parameter to be learned, σ is the nonlinear activation function Relu, and Z is the feature after the map convolution;
2.2 image classification network
According to the position of the round pin in the segmentation network segmentation result of the round pin of the lower pull rod, a round pin sub-image is obtained by intercepting from the gray image;
ResNet50 is used as a sorting net for the lower link pins, the input to which is the round pin sub-image. The classical model in image classification is CNN, but CNN shows a degradation problem with the increase of the number of layers, i.e. a deep level network is rather inferior to a slightly shallow level network performance; this is not a result of over-fitting, since degradation gaps are shown on the training set. And ResNet can solve this problem well.
ResNet is a residual neural network. The main idea of ResNet is that a skip connection structure is designed in a residual block, so that the network has a stronger identity mapping capability, the depth of the network is expanded, and the performance of the network is improved.
The structure of the residual block is shown in fig. 4; f (x) is h (x) -x, x is the shallow output, h (x) is the deep output, f (x) is the transformation of two layers sandwiched between the shallow x, when the feature of the shallow x is mature enough, if any change to the feature x will make loss large, f (x) will automatically tend to learn to be 0, and x will continue to be passed from the path of the identity map. This achieves the initial objective without increasing the computational cost: in the forward process, when the output of the shallow layer is mature enough (optimal), the layer behind the deep network can realize the role of identity mapping.
The segmentation result of the segmentation network based on the round pin of the lower pull rod can be used for judging whether the round pin is lost (the whole round pin is lost with the split pin) or not according to the number of the round holes, and meanwhile, a round pin sub-image can be obtained in a gray image according to the position of the round pin (normal or the split pin is lost) in a multi-valued image of the segmentation result, and the round pin sub-image can identify whether the split pin is lost (a small iron sheet on the round pin) or not after passing through the classification network of the round pin of the lower pull rod.
2.3 loss function
The loss function of the dynamic graph convolution-like network in the split network of the lower pull rod round pin is as follows:
L＝α·lc+β·lf+γ·la
wherein lcFor coarse segmentation loss,/fFor fine segmentation loss,/aTo assist supervisionLoss, i.e. loss of the final segmentation of the model (difference of the final segmentation result and the labeling data); α, β, γ are used to balance the three loss parameters.
The classification network resnet50 of the drop link round pin is a feature extraction network, and the cross entropy loss function is used as the loss function of classification.
2.4 optimizer
The weights of the segmentation task of the segmentation network of the lower pull rod round pin and the classification task of the classification network of the lower pull rod round pin are trained by an AdaBelief optimizer. The AdaBelief optimizer combines the fast convergence properties of Adam with good generalization properties of SGDM. AdaBelief adjusts the step size according to the belief in the current gradient direction, and takes the Exponential Moving Average (EMA) of the noisy gradient as the gradient prediction of the next step. If the observed gradient deviates significantly from the prediction, then the current observation is not trusted and a smaller step size is taken; if the observed gradient is close to the predicted value, the current observation is trusted and a larger step size is taken;
AdaBelief takes into account the curvature of the loss function; AdaBelief considers the sign of the gradient in the denominator. The AdaBelief optimizer takes training efficiency and stability into account.
Firstly, initializing weight, then obtaining a prediction result after data transformation through each neural network, and calculating a new weight coefficient after a loss function and an optimizer. And if the loss function calculated by the new weight coefficient is reduced, updating the weight, and finishing one training iteration. The program will repeat this process, completing a fixed number of iterations for all images until the optimal weight coefficients are found.
3. Lower pull rod fault discrimination
Acquiring a real vehicle passing image as an original image to be detected;
according to the wheel base information and the prior information of the lower pull rod part, the lower pull rod is roughly positioned in an original image to obtain a lower pull rod image, and the lower pull rod image is a gray image; and (3) carrying out local histogram equalization enhancement on the roughly positioned pull-down rod image (the process is carried out according to specific conditions), and reducing the influence of noise, too dark image and the like on subsequent identification. And (3) predicting circular hole areas of the lower pull rod round pin and the non-installed round pin by using a trained similar dynamic graph convolution network according to the enhanced lower pull rod image to obtain a predicted multi-value image, wherein a value 1 in the multi-value image is the circular pin area, and a value 2 in the multi-value image is the circular hole area without the installed round pin. The normal lower pull rod is provided with 2 round pin areas and 4 round hole areas; there will be 4 2 value regions in the normal lower tie rod; in the normal lower pull rod, 3 round holes are formed in the left side of the lower pull rod, and 2 round holes are left after one round pin is installed to fix the lower pull rod; the right side of the lower pull rod is provided with 3 round holes, and 2 round holes are remained after one of the 3 round holes is provided with an upper round pin to fix the lower pull rod. And after the round pin is lost, the number of the 2-value areas is changed, and the round pin loss fault is judged according to the number of the 1-value areas and the 2-value areas.
If the round pin loss fault occurs, the flat iron can deviate, in a multi-value lower pull rod image, the number of the round pins and the number of the round holes are calculated in an image processing mode, and if the number of the round pins is smaller than 2 and the number of the round holes is larger than 4, the position information of the round pin loss fault is calculated and an alarm is given. As soon as the round pin area is located, a cotter pin loss detection is performed: according to the position of the round pin in the segmentation network segmentation result of the round pin of the lower pull rod, a round pin sub-image is obtained in the lower pull rod image in a capturing mode; and (4) carrying out circular pin subimage classification by using a trained classification network of the lower pull rod circular pin. The classification result comprises a normal lower pull rod round pin, a cotter pin lost round pin and a round hole without the round pin; if the round hole is identified by the image, calculating the position of fault information and reporting that the round pin is lost, and if the round hole is identified by the image, calculating the position of the lost split pin and reporting that the split pin is lost; if no fault is detected, the next pull-down bar image is processed.
Procedure for calculating the missing position of the cotter: and recording the original image as an image 1, then intercepting a lower pull rod image 2 according to prior knowledge, and obtaining a round pin sub-image 3 according to a segmentation result. And identifying the missing position of the cotter pin in the round pin sub-image 3, then converting the position of the cotter pin in the lower pull rod image 2 according to the round pin sub-image 3, and converting the position into the original image 1 to obtain the specific position of the fault in the original image.
The present invention is capable of other embodiments and its several details are capable of modifications in various obvious respects, all without departing from the spirit and scope of the present invention.
Claims (10)
1. A rail wagon lower pull rod round pin fault detection method based on deep learning is characterized by comprising the following steps:
acquiring a to-be-detected pull-down rod image, inputting the pull-down rod image into a segmentation network of a pull-down rod round pin, predicting a pull-down rod round pin area and a round hole area without the round pin, and acquiring a predicted multi-value image, wherein a value 1 in the multi-value image is the round pin area, and a value 2 in the multi-value image is the round hole area without the round pin; judging whether the round pin has a loss fault or not according to the number of the round pin areas and the number of the round hole areas without the round pins, and if the round pins are lost, determining the lost positions of the round pins;
if the round pin area is located, split pin loss detection is performed: according to the position of the round pin in the segmentation result of the segmentation network of the round pin of the lower pull rod, a round pin subimage is obtained by intercepting from the lower pull rod image, and the round pin subimage is input into a classification network of the round pin of the lower pull rod for classification; the classification result comprises a normal lower pull rod round pin, a cotter pin lost round pin and a round hole without the round pin; if the round hole is identified, determining the position of the round hole without the round pin; if the cotter pin is identified to be lost, determining the cotter pin lost position; if no fault is detected, the next pull-down bar image is processed.
2. The method for detecting the round pin fault of the lower pull rod of the railway wagon based on the deep learning as claimed in claim 1, wherein the process of obtaining the image of the lower pull rod to be detected comprises the following steps:
acquiring a real vehicle passing image as an original image to be detected; and roughly positioning the lower pull rod in the original image according to the wheel base information and the prior information of the lower pull rod part to obtain a lower pull rod image of the lower pull rod image.
3. The method for detecting the round pin fault of the railway wagon pull-down rod based on the deep learning as claimed in claim 1, wherein before the pull-down rod image is input into a segmentation network of the round pin of the pull-down rod for prediction, local histogram equalization enhancement needs to be performed on the pull-down rod image.
4. The method for detecting the round pin fault of the lower pull rod of the railway wagon based on the deep learning as claimed in claim 1, wherein the process of judging whether the round pin has the loss fault or not according to the number of the round pin areas and the round hole areas without the round pins comprises the following steps: and if the number of the round pin areas is less than 2 and the number of the round hole areas is more than 4, judging that the round pins are lost.
5. The method for detecting the round pin fault of the pull-down rod of the railway wagon based on the deep learning as claimed in one of claims 1 to 4, wherein the segmented network of the round pin of the pull-down rod is a dynamic graph convolution-like network, and the dynamic graph convolution-like network processing process comprises the following steps:
firstly, a basic convolution neural network is utilized to generate a segmentation result and a corresponding characteristic map, which is called a coarse characteristic map;
taking the rough characteristic diagram as input, obtaining a characteristic diagram, namely a backbone network characteristic diagram, by utilizing a backbone network, taking each point on the backbone network characteristic diagram as a node of the diagram, and taking the similarity between the nodes as the weight of the edge of the adjacent matrix to obtain an adjacent matrix A;
then, graph convolution is carried out: z ═ σ (AXW);
x is an input feature diagram, namely a backbone network feature diagram; w is the network parameter to be learned, σ is a nonlinear activation function, and Z is a feature map after the map convolution;
obtaining a fine characteristic diagram after graph convolution; and connecting the fine feature map and the coarse feature map, and then using 1-by-1 convolution to obtain a final prediction result.
6. The method for detecting the round pin failure of the pull rod of the rail wagon based on the deep learning of claim 5, wherein the process of obtaining the adjacency matrix A by taking each point on the main network characteristic diagram as a node of the diagram and taking the similarity between the nodes as the weight of the edge of the adjacency matrix comprises the following steps:
feature x for two nodesiAnd xjThe similarity of these two features is measured using two linear transformations φ and φ':
F(xi,xj)＝φ(xi)Tφ′(xj)
and then performing softmax normalization to obtain an adjacency matrix A, wherein the elements in the adjacency matrix A are as follows:
and N is the number of all edges connected with the ith node.
7. The deep learning-based rail wagon pull-down rod round pin fault detection method as claimed in claim 6, wherein the split network of the pull-down rod round pin adopts a ResNet50 network.
8. The deep learning-based rail wagon pull-down rod round pin fault detection method as claimed in claim 6, wherein the training process of the dynamics-like graph convolution network comprises the following steps:
firstly, a basic convolution neural network is utilized to generate a segmentation result and a corresponding characteristic map, which is called a coarse characteristic map;
taking the rough characteristic diagram as input, obtaining a characteristic diagram, namely a backbone network characteristic diagram, by utilizing a backbone network, taking each point on the backbone network characteristic diagram as a node of the diagram, and taking the similarity between the nodes as the weight of the edge of the adjacent matrix to obtain an adjacent matrix A;
and (3) performing dynamic sampling point selection and composition based on a segmentation result predicted by a basic convolutional neural network: selecting points by utilizing a segmentation result P predicted by a basic convolutional neural network and a result G marked in a training set, and selecting a point set of target Samples of each category based on a Samples formula:
Samples＝P-P∩G+G-P∩G+r·P∩G
wherein the first term P-P.andgate G is a difficult negative sample; the second term G-P.andgate G is a difficult positive sample; the last term P.andgate.G is a simple positive sample; when selecting points, all difficult samples are selected, and then simple samples are selected according to the ratio r;
after the dynamic sampling point selection composition is finished, graph convolution is carried out, and the feature graph X input in the graph convolution process in the training process is a feature graph corresponding to the dynamic sampling point selection composition;
obtaining a fine characteristic diagram after graph convolution; connecting the fine feature map and the coarse feature map, and then using 1-by-1 convolution to obtain a final prediction result;
training was performed using an AdaBelief optimizer.
9. The method for detecting the fault of the round pin of the lower pull rod of the railway wagon based on the deep learning of claim 8, wherein the loss function of the segmented network of the round pin of the lower pull rod is as follows:
L＝α·lc+β·lf+γ·la
wherein lcFor coarse segmentation loss,/fFor fine segmentation loss,/aLoss is monitored for assistance; alpha, beta, gamma balance parameters.
10. The deep learning-based round pin fault detection method for the wagon lower pull rod as claimed in claim 9, wherein the classification network of the round pin lower pull rod adopts a cross entropy loss function as a classification loss function.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110048089.9A CN112733742B (en) | 2021-01-14 | 2021-01-14 | Deep learning-based fault detection method for round pin of lower pull rod of railway wagon |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110048089.9A CN112733742B (en) | 2021-01-14 | 2021-01-14 | Deep learning-based fault detection method for round pin of lower pull rod of railway wagon |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112733742A true CN112733742A (en) | 2021-04-30 |
| CN112733742B CN112733742B (en) | 2022-02-01 |
Family
ID=75593008
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110048089.9A Active CN112733742B (en) | 2021-01-14 | 2021-01-14 | Deep learning-based fault detection method for round pin of lower pull rod of railway wagon |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112733742B (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115273044A (en) * | 2022-07-15 | 2022-11-01 | 哈尔滨市科佳通用机电股份有限公司 | Vehicle door damage fault identification and detection method based on improved graph convolution network |
| CN115346172A (en) * | 2022-08-16 | 2022-11-15 | 哈尔滨市科佳通用机电股份有限公司 | Method and system for detecting loss and breakage of hook lifting rod return spring |
| CN115526874A (en) * | 2022-10-08 | 2022-12-27 | 哈尔滨市科佳通用机电股份有限公司 | Round pin of brake adjuster control rod and round pin split pin loss detection method |
| CN116452858A (en) * | 2023-03-24 | 2023-07-18 | 哈尔滨市科佳通用机电股份有限公司 | Rail wagon connecting pull rod round pin breaking fault identification method and system |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2019136946A1 (en) * | 2018-01-15 | 2019-07-18 | 中山大学 | Deep learning-based weakly supervised salient object detection method and system |
| CN110232687A (en) * | 2019-06-21 | 2019-09-13 | 华北电力大学（保定） | The detection method of cotter bolt defect in a kind of electric inspection process image |
| CN111080650A (en) * | 2019-12-12 | 2020-04-28 | 哈尔滨市科佳通用机电股份有限公司 | Method for detecting looseness and loss faults of small part bearing blocking key nut of railway wagon |
| CN111079746A (en) * | 2019-12-12 | 2020-04-28 | 哈尔滨市科佳通用机电股份有限公司 | Railway wagon axle box spring fault image identification method |
| CN111091541A (en) * | 2019-12-12 | 2020-05-01 | 哈尔滨市科佳通用机电股份有限公司 | Method for identifying fault of missing nut in cross beam assembly of railway wagon |
| CN111091542A (en) * | 2019-12-12 | 2020-05-01 | 哈尔滨市科佳通用机电股份有限公司 | Image identification method for breakage fault of spring supporting plate of railway wagon bogie |
| WO2020181685A1 (en) * | 2019-03-12 | 2020-09-17 | 南京邮电大学 | Vehicle-mounted video target detection method based on deep learning |
| CN112132821A (en) * | 2020-09-30 | 2020-12-25 | 哈尔滨市科佳通用机电股份有限公司 | Cotter pin loss detection method based on image processing |
-
2021
- 2021-01-14 CN CN202110048089.9A patent/CN112733742B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2019136946A1 (en) * | 2018-01-15 | 2019-07-18 | 中山大学 | Deep learning-based weakly supervised salient object detection method and system |
| WO2020181685A1 (en) * | 2019-03-12 | 2020-09-17 | 南京邮电大学 | Vehicle-mounted video target detection method based on deep learning |
| CN110232687A (en) * | 2019-06-21 | 2019-09-13 | 华北电力大学（保定） | The detection method of cotter bolt defect in a kind of electric inspection process image |
| CN111080650A (en) * | 2019-12-12 | 2020-04-28 | 哈尔滨市科佳通用机电股份有限公司 | Method for detecting looseness and loss faults of small part bearing blocking key nut of railway wagon |
| CN111079746A (en) * | 2019-12-12 | 2020-04-28 | 哈尔滨市科佳通用机电股份有限公司 | Railway wagon axle box spring fault image identification method |
| CN111091541A (en) * | 2019-12-12 | 2020-05-01 | 哈尔滨市科佳通用机电股份有限公司 | Method for identifying fault of missing nut in cross beam assembly of railway wagon |
| CN111091542A (en) * | 2019-12-12 | 2020-05-01 | 哈尔滨市科佳通用机电股份有限公司 | Image identification method for breakage fault of spring supporting plate of railway wagon bogie |
| CN112132821A (en) * | 2020-09-30 | 2020-12-25 | 哈尔滨市科佳通用机电股份有限公司 | Cotter pin loss detection method based on image processing |
Non-Patent Citations (4)
| Title |
|---|
| HANZHE HU ET AL: "Class-wise Dynamic Graph Convolution for Semantic Segmentation", 《ARXIV.ORG》 * |
| 周云燕: "基于图像分析理论的机械故障诊断研究", 《中国优秀博硕士学位论文全文数据库(博士)信息科技辑》 * |
| 张晓光等: "《射线检测焊接缺陷的提取和自动识别》", 31 October 2004 * |
| 王一丁等: "《数字图像处理》", 31 August 2015 * |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115273044A (en) * | 2022-07-15 | 2022-11-01 | 哈尔滨市科佳通用机电股份有限公司 | Vehicle door damage fault identification and detection method based on improved graph convolution network |
| CN115346172A (en) * | 2022-08-16 | 2022-11-15 | 哈尔滨市科佳通用机电股份有限公司 | Method and system for detecting loss and breakage of hook lifting rod return spring |
| CN115526874A (en) * | 2022-10-08 | 2022-12-27 | 哈尔滨市科佳通用机电股份有限公司 | Round pin of brake adjuster control rod and round pin split pin loss detection method |
| CN116452858A (en) * | 2023-03-24 | 2023-07-18 | 哈尔滨市科佳通用机电股份有限公司 | Rail wagon connecting pull rod round pin breaking fault identification method and system |
| CN116452858B (en) * | 2023-03-24 | 2023-12-15 | 哈尔滨市科佳通用机电股份有限公司 | Rail wagon connecting pull rod round pin breaking fault identification method and system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112733742B (en) | 2022-02-01 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112733742B (en) | Deep learning-based fault detection method for round pin of lower pull rod of railway wagon | |
| CN111091541B (en) | Method for identifying fault of missing nut in cross beam assembly of railway wagon | |
| CN111079747B (en) | Railway wagon bogie side frame fracture fault image identification method | |
| CN107316064B (en) | Asphalt pavement crack classification and identification method based on convolutional neural network | |
| CN111079821A (en) | Derailment automatic braking pull ring falling fault image identification method | |
| CN111652295B (en) | Railway wagon coupler yoke key joist falling fault identification method | |
| CN111091548B (en) | Railway wagon adapter dislocation fault image identification method and system based on deep learning | |
| CN111091543B (en) | Railway wagon swing bolster spring loss fault target detection method | |
| CN111080600A (en) | Fault identification method for split pin on spring supporting plate of railway wagon | |
| CN110276357A (en) | A kind of method for recognizing verification code based on convolutional neural networks | |
| CN111080607B (en) | Rolling bearing oil slinging fault detection method based on image recognition | |
| CN111080614A (en) | Method for identifying damage to rim and tread of railway wagon wheel | |
| CN111079630B (en) | Fault identification method for railway wagon brake beam with incorrect installation position | |
| CN113989257A (en) | Electric power comprehensive pipe gallery settlement crack identification method based on artificial intelligence technology | |
| CN111079822A (en) | Method for identifying dislocation fault image of middle rubber and upper and lower plates of axle box rubber pad | |
| CN111079628A (en) | Floor damage fault image identification method | |
| CN111091551A (en) | Method for detecting loss fault of brake beam strut opening pin of railway wagon | |
| CN114627106A (en) | Weld defect detection method based on Cascade Mask R-CNN model | |
| CN112329858B (en) | Image recognition method for breakage fault of anti-loosening iron wire of railway motor car | |
| CN111091554B (en) | Railway wagon swing bolster fracture fault image identification method | |
| CN117670855A (en) | RoadU-Net-based intelligent recognition and classification method for asphalt pavement diseases | |
| CN110349119B (en) | Pavement disease detection method and device based on edge detection neural network | |
| CN111652228A (en) | Railway wagon sleeper beam hole foreign matter detection method | |
| CN113239865B (en) | Deep learning-based lane line detection method | |
| CN112766174B (en) | Railway train carriage group bottom plate loss fault detection method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |