CN114170136A - Method, system, device and medium for detecting defects of fasteners of contact net bracket device - Google Patents

Method, system, device and medium for detecting defects of fasteners of contact net bracket device Download PDF

Info

Publication number
CN114170136A
CN114170136A CN202111297730.9A CN202111297730A CN114170136A CN 114170136 A CN114170136 A CN 114170136A CN 202111297730 A CN202111297730 A CN 202111297730A CN 114170136 A CN114170136 A CN 114170136A
Authority
CN
China
Prior art keywords
model
fastener
cyolo
image
rrnet
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202111297730.9A
Other languages
Chinese (zh)
Inventor
蔡长青
刘爽
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Guangzhou University
Original Assignee
Guangzhou University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Guangzhou University filed Critical Guangzhou University
Priority to CN202111297730.9A priority Critical patent/CN114170136A/en
Publication of CN114170136A publication Critical patent/CN114170136A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/0002—Inspection of images, e.g. flaw detection
    • G06T7/0004—Industrial image inspection
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/23—Clustering techniques
    • G06F18/232—Non-hierarchical techniques
    • G06F18/2321—Non-hierarchical techniques using statistics or function optimisation, e.g. modelling of probability density functions
    • G06F18/23213—Non-hierarchical techniques using statistics or function optimisation, e.g. modelling of probability density functions with fixed number of clusters, e.g. K-means clustering
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30108—Industrial image inspection
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30168—Image quality inspection

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • General Physics & Mathematics (AREA)
  • Evolutionary Computation (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Artificial Intelligence (AREA)
  • General Engineering & Computer Science (AREA)
  • General Health & Medical Sciences (AREA)
  • Molecular Biology (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Software Systems (AREA)
  • Mathematical Physics (AREA)
  • Health & Medical Sciences (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • Computing Systems (AREA)
  • Evolutionary Biology (AREA)
  • Probability & Statistics with Applications (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Quality & Reliability (AREA)
  • Image Analysis (AREA)

Abstract

The invention discloses a method and a system for detecting defects of fasteners of a contact net bracket device, a computer device and a storage medium. The CYOLO model has the performance of extracting limited effective information from highly redundant information so as to position a fastener region in an image to be detected, and the RRNet model detects the defects of fasteners in different directions through a detector based on a rotation anchor point. The method provided by the invention is used for detecting the characteristics of the images of the contact net bracket device by combining the advantages of the CYOLO model and the RRNet model, and can keep higher processing speed and achieve reliable detection precision. The invention is widely applied to the technical field of image processing.

Description

Method, system, device and medium for detecting defects of fasteners of contact net bracket device
Technical Field
The invention relates to the technical field of image processing, in particular to a method and a system for detecting defects of fasteners of a contact net bracket device, a computer device and a storage medium.
Background
The railway vehicle is in contact with the contact net system (OCS) through a pantograph mounted on a contact net bracket device (CSD), so that the contact net supplies power to the railway vehicle. The contact force between the pantograph and the contact net system is large, and the railway vehicle moves at a high speed, so that strong vibration excitation is caused to the contact net bracket device, and the fasteners such as bolts and the like on the contact net bracket device are easy to loosen and even damage, so that the normal operation of the railway vehicle is influenced.
In the prior art for detecting the defects of fasteners on a contact net support device, a camera is arranged near the installation position of the contact net support device on a vehicle, an image of the contact net support device is shot in the driving process, and the defects of the contact net support device are identified by carrying out image analysis on the shot image. The image shot by the contact net support device is characterized in that the image itself may have tens of millions of pixels, but each fastener part in the image may have only hundreds of pixels, so that semantic information of fastener defects contained in the shot image is actually very limited, and different fasteners have different orientations, which become obstacles for positioning and defect identification of the fasteners in the fastener defect detection process.
Disclosure of Invention
The invention aims to provide a method and a system for detecting defects of fasteners of a contact net support device, a computer device and a storage medium, aiming at least one technical problem that images shot by the contact net support device contain less semantic information, the orientations of the fasteners are lack of regularity, and obstacles are caused to positioning and defect identification of the fasteners.
On one hand, the embodiment of the invention provides a method for detecting defects of fasteners of a contact net bracket device, which comprises the following steps:
acquiring an image to be detected; the image to be detected comprises a contact net bracket device;
establishing a CYOLO model;
establishing an RRNet model;
inputting the image to be detected into the CYOLO model, and extracting a fastener area of a contact net bracket device in the image to be detected by the CYOLO model and outputting the fastener area;
inputting the fastener region into the RRNet model, wherein the RRNet model outputs a defect identification result of the fastener region.
Further, the method for detecting the defect of the fastener of the contact net bracket device further comprises a training step of the CYOLO model, and the training step of the CYOLO model comprises the following steps:
acquiring a first sample image and a first label; the first sample image comprises an overhead line system bracket device and fasteners on the overhead line system bracket device, and the first label is used for marking the types of the fasteners in the first sample image;
marking fasteners in the first sample image with a frame;
clustering the frames in the first sample image by a size-based k-means clustering algorithm;
the CYOLO model takes the fastener in the first sample image as a detection target, sets a prediction grid with a corresponding size according to the size of the detection target, detects the detection target in the prediction grid by using a box, and outputs a fastener type identification result;
setting a first loss function;
determining a value of the first loss function from the fastener type identification and the first tag;
and adjusting the network parameters of the CYOLO model according to the value of the first loss function or finishing the training.
Further, the training of the CYOLO model further includes:
and carrying out grid mask processing on the first sample image before the CYOLO model detects the first sample image.
Further, the first loss function is:
Figure RE-GDA0003469355040000021
wherein S represents the number of meshes in the prediction mesh, B represents the prediction mesh,
Figure RE-GDA0003469355040000022
and
Figure RE-GDA0003469355040000023
is a binary value and is used as a binary value,
Figure RE-GDA0003469355040000024
representing a prediction offset vector, vijkA target vector representing a ground truth value,
Figure RE-GDA0003469355040000025
which is indicative of a prediction reference,
Figure RE-GDA0003469355040000026
denotes the target reference, λcoordWeight factor, λ, representing coordinate loss termobjAnd λnoobjA weight factor, λ, representing a reference loss termclsWeight factor, coordinate loss L, representing the reference loss term for negative sample anchoringregIs L1Loss, reference loss LconfFor focus loss, class loss LclsIn order to have a sigmoidal cross entropy, i, j and k represent the serial numbers, x and y represent the coordinates of the center of the box, w represents the width of the box, h represents the height of the box,
Figure RE-GDA0003469355040000027
a predicted probability distribution, P, representing the result of the identification of the type of fasteneri j1 is the true value of the first label.
Further, the method for detecting the defect of the fastener of the contact net bracket device further comprises a training step of the RRNet model, and the training step of the RRNet model comprises the following steps:
acquiring a second sample image and a second label; the second sample image comprises fasteners, and the second label is used for marking the types of the fastener defects in the second sample image;
setting parameters of a rotating frame;
the RRNet model predicts the offset vector of the fastener in the second sample image through a prediction frame, adjusts the direction of the fastener in the second sample image through the rotation of the rotation frame, sets an anchoring frame to sample and identify the defect type of the fastener in the second sample image, and outputs a defect type identification result;
setting a second loss function;
determining the value of the second loss function according to the defect type identification result and the second label;
and adjusting the network parameters of the RRNet model according to the value of the second loss function or finishing the training.
Further, the parameters of the rotating frame are as follows:
tx＝(x-xa)/wa，ty＝(y-ya)/ha
tw＝log(w/wa)，th＝log(h/ha)，tθ＝θ-θa
Figure RE-GDA0003469355040000031
Figure RE-GDA0003469355040000032
wherein x and y represent center coordinates of the rotating frame, w represents a width of the rotating frame, h represents a height of the rotating frame, θ represents a direction angle of the rotating frame, xaAnd yaRepresenting the center coordinates of the anchor box, waRepresents the width of the anchor frame, haRepresenting the height, θ, of the anchoring frameaRepresents the angle of orientation of the anchor frame,
Figure RE-GDA0003469355040000033
and
Figure RE-GDA0003469355040000034
represents the coordinates of the center of the prediction box,
Figure RE-GDA0003469355040000035
represents the width of the prediction box and the prediction frame,
Figure RE-GDA0003469355040000036
represents the height of the prediction box and the height of the prediction box,
Figure RE-GDA0003469355040000037
representing the directional angle of the prediction box.
Further, the second loss function is:
Figure RE-GDA0003469355040000038
wherein the content of the first and second substances,
Figure RE-GDA0003469355040000039
representing a prediction offset vector, vijkA target vector representing a ground truth value,
Figure RE-GDA00034693550400000310
is the binary value, N is the number of anchors, pnA predicted probability distribution, t, representing the result of the defect type recognitionnRepresenting a probability distribution of the second label,
Figure RE-GDA0003469355040000041
representing the IoU value, λ, re-represented between the minimum bounding rectangle of the nth anchor and the ground truth targetcoord、λclsAnd λiouRepresents a weight factor, LsregRepresents the smoothing L1Loss, LregRepresents L1Loss, LclsIndicating a loss of focus.
On the other hand, the embodiment of the invention also comprises a system for detecting the defects of the fasteners of the contact net bracket device, wherein the system for detecting the defects of the fasteners of the contact net bracket device comprises:
the first module is used for acquiring an image to be detected; the image to be detected comprises a contact net bracket device;
a second module for establishing a CYOLO model;
the third module is used for establishing an RRNet model;
the fourth module is used for inputting the image to be detected into the CYOLO model, and the CYOLO model extracts a fastener area of the contact net bracket device in the image to be detected and outputs the fastener area;
a fifth module for inputting the fastener region into the RRNet model, the RRNet model outputting a defect identification result for the fastener region.
In another aspect, the embodiment of the present invention further includes a computer device, which includes a memory and a processor, where the memory is used to store at least one program, and the processor is used to load the at least one program to execute the method for detecting a defect in a fastener of an overhead contact system support device in the embodiment.
In another aspect, the present invention further provides a storage medium, in which a processor-executable program is stored, and the processor-executable program is used for executing the catenary bracket device fastener defect detection method in the embodiment when the processor executes the program.
The invention has the beneficial effects that: according to the method for detecting the defects of the fasteners of the contact net bracket device, the CYOLO model has good performance of identifying a small part of fastener regions in an image to be detected, namely the CYOLO model has the performance of extracting limited effective information from highly redundant information, so that the fastener regions in the image to be detected are positioned; the used RRNet model can detect the defects of the fasteners in different directions through the detector based on the rotation anchor points, therefore, an image to be detected is input into the CYOLO model, the CYOLO model can extract the fastener region, the fastener region is input into the RRNet model, and the RRNet model can output the defect identification result of the fastener region, so that the detection of the defects of the fasteners of the contact net support device is completed. According to the method for detecting the defects of the fasteners of the contact network bracket device, the images of the contact network bracket device are detected by combining the advantages of a CYOLO model and an RRNet model according to the characteristics of the images of the contact network bracket device, so that higher processing speed can be kept and reliable detection precision can be achieved.
Drawings
FIG. 1 is a flow chart of a method for detecting defects in fasteners of a bracket device of an overhead line system in an embodiment;
FIG. 2 is a schematic structural diagram of a CYOLO model in an embodiment;
FIG. 3 is a schematic diagram of the CYOLO model training in the embodiment;
FIG. 4 is a schematic diagram of an embodiment of grid masking;
FIG. 5 is a schematic structural diagram of an RRNet model in the embodiment;
fig. 6 is a working principle diagram of the RRNet model in the embodiment.
Detailed Description
In this embodiment, referring to fig. 1, the method for detecting the defects of the fasteners of the bracket device of the overhead line system includes the following steps:
s1, obtaining an image to be detected; the image to be detected comprises a contact net bracket device;
s2, establishing a CYOLO model;
s3, establishing an RRNet model;
s4, inputting the image to be detected into a CYOLO model, and extracting a fastener area of the contact net bracket device in the image to be detected by the CYOLO model and outputting the fastener area;
and S5, inputting the fastener area into an RRNet model, and outputting a defect identification result of the fastener area by the RRNet model.
In step S1, the contact net support device may be photographed by a camera mounted on the railway vehicle to obtain an image to be detected, or the photographed image to be detected may be stored in a database and the image to be detected may be read from the database.
In step S2, a CYOLO model may be established based on the YOLOv3 model. If the original YOLOv3 model is used, then the coefficients in the YOLOv3 model are typically set to 32, and the image is adjusted to 320 × 320, 416 × 416, or 608 × 608 resolution by five downsampling layers to reduce computational overhead and memory usage. However, joints containing hundreds of pixels or even thousands of pixels in the captured image of the catenary stent device are usually compressed by five downsampling layers, so that the number of connected effective pixels is very limited. Therefore, the structure of the CYOLO model established based on the YOLOv3 model in the present embodiment is shown in fig. 2.
Referring to fig. 2(a), only four downsampling layers are used in the CYOLO model. Similar to YOLOv3, the basic unit of the CYOLO model used in the present embodiment is also a CBL block composed of Conv, BN and LRelu layers. As shown in fig. 2(B), adding BN and Relu layers after the Conv layer in each CBL block can ensure fast convergence and non-linearity of the model. Specifically, RDB _ N (i.e., remaining block N) is composed of RDBs (1 to N) including two cbl. For example, RDB _2 (i.e., N — 2) includes a residual block _1 and a residual block _2, as shown in fig. 2 (C). The input image is resized to 416 x 512 to match the size ratio of the original image. Thus, a 416 × 512 image will generate a 26 × 32 prediction grid. Meanwhile, to address the problem of small objects and lack of semantic information, feature layer reconstruction links the available semantic information of 52 × 64, 104 × 128, and 208 × 256 layers to the last yolo prediction layer, allowing the network to obtain finer feature information from the last extended yolo layers (26 × 32 and 52 × 64). For example, in fig. 2(D), CYOLO uses stride ═ 4 to connect the w/4 × h/4 × 2c feature output by RDB _1 (output feature map size: w × h × c) to the last yolo prediction layer (52 × 64). Notably, the shallow layer contains rich object edge information, so two sets of features are reconstructed from RDB _1 (n × n × c is reconstructed to n/4 × n/4 × 2c) at two different starting points with a down-sampling factor of 4 to preserve the boundary features. Wherein w, h and c represent the width, height and number of channels of the output feature map, respectively.
After the CYOLO model is established, before the image to be detected is detected by using the CYOLO model, a training step of the CYOLO model is also executed. In this embodiment, the training of the CYOLO model includes:
p1a. acquiring a first sample image and a first tag; the first sample image comprises an overhead line system bracket device and fasteners on the overhead line system bracket device, and the first label is used for marking the types of the fasteners in the first sample image;
p2a. mark fasteners in the first sample image with a frame;
P3A, clustering the frames in the first sample image through a k-means clustering algorithm based on the size;
the P4A.CYOLO model takes the fastener in the first sample image as a detection target, sets a prediction grid with corresponding size according to the size of the detection target, detects the detection target by using a box in the prediction grid, and outputs a fastener type identification result;
p5a. setting a first loss function;
p6a. determining a value of a first loss function from the fastener type identification and the first tag;
p7a. adjust the network parameters of the CYOLO model or end the training according to the values of the first loss function.
In step P1A, the first sample image may be a pre-captured image including a catenary bracket assembly including fasteners such as screws, bolts, pins, etc., and in step P2A, the type of the fastener in the first sample image is marked by a first label. Referring to fig. 3(a), fasteners in the first sample image are labeled by six types of boxes C0, C1, C2, C3, C4, and C5, and the boxes are clustered by a size-based k-means algorithm in step P3A.
In step P4A, the largest three boxes are used on the 26 × 32 prediction grid to detect large objects and the other three boxes are used on the 52 × 64 prediction grid to detect small objects. The boxes in fig. 3(B) represent ground truth values for the bird shields (indicated by the boxes). The boxes in fig. 3(C) represent the three boxes (the three largest boxes or the other three smallest boxes) generated by the k-means algorithm. For the default box on the last M × N feature map, 5 class objects, one confidence, and four coordinates (x, y, w, h) of the box are predicted, and thus, the output tensor of the CYOLO model has M × N × (5+5) × 3 dimensions.
In step P5A, the first loss function is set as:
Figure RE-GDA0003469355040000071
wherein S represents the number of meshes in the prediction mesh, B represents the prediction mesh,
Figure RE-GDA0003469355040000072
and
Figure RE-GDA0003469355040000073
is a binary value, in particular if
Figure RE-GDA0003469355040000074
Then the box is responsible for testing the object. Otherwise, it is not responsible for testing the object,
Figure RE-GDA0003469355040000075
representing a prediction offset vector, vijkA target vector representing a ground truth value,
Figure RE-GDA0003469355040000076
which is indicative of a prediction reference,
Figure RE-GDA0003469355040000077
denotes the target reference, λcoordWeight factor, λ, representing coordinate loss termobjAnd λnoobjA weighting factor representing a reference loss term, in particular λobjReference loss term weight, λ, representing the positive sample anchornoobjReference loss term weight, λ, representing the negative sample anchorclsWeight factor, coordinate loss L, representing the reference loss term for negative sample anchoringregIs L1Loss, reference loss LconfFor focus loss, class loss LclsFor S-shaped cross entropy, i, j and k tablesIndicating the serial number, x and y indicating the coordinates of the center of the cassette, w indicating the width of the cassette, h indicating the height of the cassette,
Figure RE-GDA0003469355040000078
a predicted probability distribution, P, representing the result of the identification of the type of fasteneri j1 is the true value of the first label.
In step P6A, the fastener type recognition result and the first label may be substituted into the formula of the first loss function, the value of the first loss function is calculated, if the obtained value of the first loss function does not reach the preset range, the process jumps to step P1A again after the network parameters of the CYOLO model are adjusted in step P7A to start executing steps P1A-P6A, and if the obtained value of the first loss function is within the preset range, the training of the CYOLO model may be ended.
In this embodiment, when training the CYOLO model, the first sample image is also subjected to mesh masking processing before performing steps P4A-P7A, that is, before the CYOLO model detects the first sample image.
In this embodiment, a process of performing mesh masking on the first sample image may be represented as Output (i, j) ═ Input (i, j) × Mask (i, j), where Input (i, j) ∈ RH × W × C represents the first sample image before the mesh masking, Output (i, j) represents the first sample image after the mesh masking, and Mask (i, j) represents a Mask used by the mesh masking. Each point in Mask (i, j) is a binary value, and if Mask (i, j) is equal to 1, the pixel value of point (i, j) in Input (i, j) will be kept in image Output (i, j) when the grid Mask processing is performed, otherwise it will be truncated.
Referring to fig. 4, the shape of the masks is similar to a grid, each mask being formed by tiling cells (i.e., small rectangles on the left side of fig. 4), as shown on the left side of fig. 4. The mask may be defined by four parameters (s, l, Δ x, Δ y). s and l are the ratio of the gray edge in the unit and the unit length, respectively. Δ x and Δ y are the horizontal and vertical distances from the first cell to the nearest border of the image, respectively.
To make full use of the mesh masking method, the ratio of masksThe rate k is calculated as
Figure RE-GDA0003469355040000081
This means that k is proportional to the reserved area over the entire input image. If an incomplete unit in the mask is ignored, k can be calculated as:
k＝1-(1-s)2＝2s-s2。
furthermore, to enrich the diversity of CSD data sets in view of randomness, l can be randomly selected from the following ranges:
l＝random(lmin，lmax)。
thus, Δ x and Δ y are selected from the following ranges:
Δx(Δy)＝random(0，l-1)。
a balance between over-fitting and under-fitting can be achieved by appropriate selection of the above parameters. In this embodiment,/minIs set to 96, lmaxSet to 224 and s to 0.6.
By carrying out grid masking processing on the first sample image, overfitting caused by similarity between samples and limited sample problems can be overcome, and reasonable balance is realized between retention and deletion of information of certain areas in the image, so that data enhancement is realized.
In step S3, the established RRNet model is a single-stage rotating detector based on the retina network, i.e., a rotating retina network (RRNet). Referring to fig. 5, RRNet is composed of two parts: (1) a back bone network for extracting features and (2) a sub-network for regression and classification. Wherein a Feature Pyramid Network (FPN) is used as a backbone network that enhances the detector by top-down paths and horizontally connected multi-scale pyramid structures. Each layer in the pyramid is used to detect objects of different proportions. The backbone of RRNet is composed of ResNet and FPN, as shown in fig. 5(a) and 5 (b). P denotes pyramid layers (P3, P4, P5, P6). Each pyramid level contains two subnets for classification and coordinate regression, as shown in fig. 5(c) and (d). The size of the feature map from the classification network is W × H × K × a, where W and H are the width and height of the feature map, respectively, K is the number of classes, and a is the number of generated boxes.
In anchor-based detectors, the S-value is the reduction factor (an exponential power of 2) of the feature map relative to the original image. RRNet can balance semantic information and location information using C3 and C4 in ResNet as a compressed cascade storage mechanism. For example, for the compressed cascaded storage mechanism, the network upsamples P5 to fuse C4 to obtain P4.
As shown on the left side in fig. 6, these anchor points "+" generated on P7(S ═ 128) are sparsely arranged on the input image. Therefore, in the case where the hyper parameter setting IoU is 0.4, the anchor boxes (dashed boxes) on these anchor points are difficult to match with the ground truth (solid boxes), resulting in the discarding of a large number of anchor boxes (e.g., max IoU is 0.31 and 0.16, as shown in fig. 6). Thus, a larger S may tilt the object directly at the P7 prediction layer. In contrast, as shown on the right side in fig. 6, these anchor points "+" generated on P6(S ═ 64) are densely arranged on the input image, so a smaller S more easily samples the region of interest. Furthermore, according to previous experience, most fasteners in CSD data sets (e.g., nuts and pins) are smaller than 128 pixels, so only four levels are used in the pyramid [ P3, P4, P5, P7 ].
After the RRNet model is established, before the RRNet model is used for detecting the image to be detected, a training step of the RRNet model is also executed. In this embodiment, the training of the RRNet model includes:
p1b. acquiring a second sample image and a second label; the second sample image comprises fasteners, and the second label is used for marking the types of the fastener defects in the second sample image;
P2B, setting parameters of a rotating frame;
predicting the offset vector of the fastener in the second sample image through a prediction frame by the P3B.RRNet model, adjusting the direction of the fastener in the second sample image through rotation of a rotation frame, setting an anchor frame to perform sampling and defect type identification on the fastener in the second sample image, and outputting a defect type identification result;
P4B, setting a second loss function;
P5B, determining the value of a second loss function according to the defect type identification result and a second label;
and P6B, adjusting network parameters of the RRNet model according to the value of the second loss function or finishing training.
In step P1B, the second sample image may be a pre-captured image including fasteners, and the defects of the fasteners in the different second sample images may be of different types, such as loosening, damage, etc. In step P2B, the parameters of the prediction box, the anchor box, and the rotation box are set:
tx＝(x-xa)/wa，ty＝(y-ya)/ha
tw＝log(w/wa)，th＝log(h/ha)，tθ＝θ-θa
Figure RE-GDA0003469355040000091
Figure RE-GDA0003469355040000092
wherein x and y represent the center coordinates of the rotating frame, w represents the width of the rotating frame, h represents the height of the rotating frame, θ represents the direction angle of the rotating frame, xaAnd yaCenter coordinates, w, representing the anchoring frameaWidth of anchor frame, haIndicating the height, theta, of the anchoring frameaIndicating the angle of orientation of the anchor frame,
Figure RE-GDA0003469355040000093
and
Figure RE-GDA0003469355040000094
representing the center coordinates of the prediction box,
Figure RE-GDA0003469355040000095
the width of the prediction box is represented,
Figure RE-GDA0003469355040000101
which represents the height of the prediction box,
Figure RE-GDA0003469355040000102
representing the directional angle of the prediction box. t is tx、tw、
Figure RE-GDA0003469355040000103
Represents regression of the rotation box.
Specifically, the prediction box, the anchor box, and the rotation box can be represented by five parameters (x, y, w, h, θ), respectively, which is consistent with OpenCV. Specifically, θ refers to the angle of the frame with respect to the x-axis, with the corresponding side defined as the width w and the other side defined as the height h.
In this embodiment, the loss of the RRNet model is defined as follows:
Figure RE-GDA0003469355040000104
wherein, the first and second guide rollers are arranged in a row,
Figure RE-GDA0003469355040000105
representing a prediction offset vector, vijkA target vector representing a ground truth value,
Figure RE-GDA0003469355040000106
is a binary value and is used as a binary value,
Figure RE-GDA0003469355040000107
can be 0 or 1 if
Figure RE-GDA0003469355040000108
Equal to 1, there is one object in the anchor, if
Figure RE-GDA0003469355040000109
Equal to 0, then there are no objects in the anchor, N is the number of anchors, pnA predicted probability distribution, t, representing the result of the defect type recognitionnRepresenting the probability distribution, λ, of the second labelcoord、λclsAnd λiouRepresenting a weighting factor.
A new IoU-based rotation loss is introduced into the loss function, which is defined as:
Figure RE-GDA00034693550400001010
wherein
Figure RE-GDA00034693550400001011
Representing the IoU value re-represented between the n 'th anchor's minimum bounding rectangle and the ground truth target.
The second loss function in step P4B is the sum of the loss and the rotation loss of the RRNet model, i.e. the second loss function is:
Figure RE-GDA00034693550400001012
this loss can be resolved by adding a constraint to the angle regression to solve the mispredicted angle problem, LsregRepresents the smoothing L1Loss, LregRepresents L1Loss, LclsIndicating a loss of focus.
In step P5B, the defect type recognition result and the second label may be substituted into the formula of the second loss function, the value of the second loss function is calculated, if the obtained value of the second loss function does not reach the preset range, the network parameters of the RRNet model are adjusted in step P6B, and then the process jumps to step P1B again to start executing steps P1B-P5B, and if the obtained value of the second loss function is within the preset range, the training of the RRNet model may be ended.
The CYOLO model trained in the steps P1A-P7A and the RRNet model trained in the steps P1B-P6B have the capability of positioning the fastener of the image to be detected and detecting the defects. Specifically, the trained CYOLO model has good performance of identifying a small fastener region in an image to be detected, namely the CYOLO model has the performance of extracting limited effective information from highly redundant information, so that the fastener region in the image to be detected is positioned; the trained RRNet model can detect defects of fasteners in different directions through a detector based on rotational anchor points. Step S4 may be executed to input the image to be detected into the CYOLO model, the CYOLO model extracts the fastener region of the catenary bracket device in the image to be detected for output, and step S5 is executed to input the fastener region into the RRNet model, and the RRNet model outputs the defect identification result of the fastener region, thereby completing the detection of the defect of the fastener of the catenary bracket device.
The method for detecting the defects of the fasteners of the contact network bracket device in the embodiment is used for detecting the images of the contact network bracket device by combining the advantages of the CYOLO model and the RRNet model according to the characteristics of the images of the contact network bracket device, and can keep higher processing speed and achieve reliable detection precision.
In this embodiment, contact net support device fastener defect detecting system includes:
the first module is used for acquiring an image to be detected; the image to be detected comprises a contact net bracket device;
a second module for establishing a CYOLO model;
the third module is used for establishing an RRNet model;
the fourth module is used for inputting the image to be detected into the CYOLO model, and the CYOLO model extracts the fastener area of the contact net bracket device in the image to be detected and outputs the fastener area;
and the fifth module is used for inputting the fastener region into the RRNet model, and the RRNet model outputs a defect identification result of the fastener region.
The first module, the second module, the third module, the fourth module and the fifth module are respectively a hardware module, a software module or a combination of a hardware module and a software module having corresponding functions. The first module may be configured to perform step S1 in the method for detecting a defect of a fastener of an overhead line system bracket device in this embodiment, the second module may be configured to perform step S2, the third module may be configured to perform step S3, the fourth module may be configured to perform step S4, and the fifth module may be configured to perform step S5, so that when the system for detecting a defect of a fastener of an overhead line system bracket device is operated, the same technical effect as that of the method for detecting a defect of a fastener of an overhead line system bracket device may be achieved.
The method for detecting the defect of the fastener of the bracket device of the overhead contact system in the embodiment can be implemented by writing a computer program for executing the method for detecting the defect of the fastener of the bracket device of the overhead contact system in the embodiment, writing the computer program into a computer device or a storage medium, and executing the method for detecting the defect of the fastener of the bracket device of the overhead contact system in the embodiment when the computer program is read out and operated, thereby achieving the same technical effects as the method for detecting the defect of the fastener of the bracket device of the overhead contact system in the embodiment.
It should be noted that, unless otherwise specified, when a feature is referred to as being "fixed" or "connected" to another feature, it may be directly fixed or connected to the other feature or indirectly fixed or connected to the other feature. Furthermore, the descriptions of upper, lower, left, right, etc. used in the present disclosure are only relative to the mutual positional relationship of the constituent parts of the present disclosure in the drawings. As used in this disclosure, the singular forms "a", "an", and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. In addition, unless defined otherwise, all technical and scientific terms used in this example have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used in the description of the embodiments herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used in this embodiment, the term "and/or" includes any combination of one or more of the associated listed items.
It will be understood that, although the terms first, second, third, etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element of the same type from another. For example, a first element could be termed a second element, and, similarly, a second element could be termed a first element, without departing from the scope of the present disclosure. The use of any and all examples, or exemplary language ("e.g.," such as "or the like") provided with this embodiment is intended merely to better illuminate embodiments of the invention and does not pose a limitation on the scope of the invention unless otherwise claimed.
It should be recognized that embodiments of the present invention can be realized and implemented by computer hardware, a combination of hardware and software, or by computer instructions stored in a non-transitory computer readable memory. The methods may be implemented in a computer program using standard programming techniques, including a non-transitory computer-readable storage medium configured with the computer program, where the storage medium so configured causes a computer to operate in a specific and predefined manner, according to the methods and figures described in the detailed description. Each program may be implemented in a high level procedural or object oriented programming language to communicate with a computer system. However, the program(s) can be implemented in assembly or machine language, if desired. In any case, the language may be a compiled or interpreted language. Furthermore, the program can be run on a programmed application specific integrated circuit for this purpose.
Further, operations of processes described in this embodiment can be performed in any suitable order unless otherwise indicated herein or otherwise clearly contradicted by context. The processes described in this embodiment (or variations and/or combinations thereof) may be performed under the control of one or more computer systems configured with executable instructions, and may be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) collectively executed on one or more processors, by hardware, or combinations thereof. The computer program includes a plurality of instructions executable by one or more processors.
Further, the method may be implemented in any type of computing platform operatively connected to a suitable interface, including but not limited to a personal computer, mini computer, mainframe, workstation, networked or distributed computing environment, separate or integrated computer platform, or in communication with a charged particle tool or other imaging device, and the like. Aspects of the invention may be embodied in machine-readable code stored on a non-transitory storage medium or device, whether removable or integrated into a computing platform, such as a hard disk, optically read and/or write storage medium, RAM, ROM, or the like, such that it may be read by a programmable computer, which when read by the storage medium or device, is operative to configure and operate the computer to perform the procedures described herein. Further, the machine-readable code, or portions thereof, may be transmitted over a wired or wireless network. The invention described in this embodiment includes these and other different types of non-transitory computer-readable storage media when such media include instructions or programs that implement the steps described above in conjunction with a microprocessor or other data processor. The invention also includes the computer itself when programmed according to the methods and techniques described herein.
A computer program can be applied to input data to perform the functions described in the present embodiment to convert the input data to generate output data that is stored to a non-volatile memory. The output information may also be applied to one or more output devices, such as a display. In a preferred embodiment of the invention, the transformed data represents physical and tangible objects, including particular visual depictions of physical and tangible objects produced on a display.
The above description is only a preferred embodiment of the present invention, and the present invention is not limited to the above embodiment, and any modifications, equivalent substitutions, improvements, etc. within the spirit and principle of the present invention should be included in the protection scope of the present invention as long as the technical effects of the present invention are achieved by the same means. The invention is capable of other modifications and variations in its technical solution and/or its implementation, within the scope of protection of the invention.

Claims (10)

1. The method for detecting the defects of the fasteners of the contact net bracket device is characterized by comprising the following steps of:
acquiring an image to be detected; the image to be detected comprises a contact net bracket device;
establishing a CYOLO model;
establishing an RRNet model;
inputting the image to be detected into the CYOLO model, and extracting a fastener area of a contact net bracket device in the image to be detected by the CYOLO model and outputting the fastener area;
inputting the fastener region into the RRNet model, wherein the RRNet model outputs a defect identification result of the fastener region.
2. The catenary bracket device fastener defect detection method of claim 1, further comprising a training step of the CYOLO model, wherein the training step of the CYOLO model comprises:
acquiring a first sample image and a first label; the first sample image comprises an overhead line system bracket device and fasteners on the overhead line system bracket device, and the first label is used for marking the types of the fasteners in the first sample image;
marking fasteners in the first sample image with a frame;
clustering the frames in the first sample image by a size-based k-means clustering algorithm;
the CYOLO model takes the fastener in the first sample image as a detection target, sets a prediction grid with a corresponding size according to the size of the detection target, detects the detection target in the prediction grid by using a box, and outputs a fastener type identification result;
setting a first loss function;
determining a value of the first loss function from the fastener type identification and the first tag;
and adjusting the network parameters of the CYOLO model according to the value of the first loss function or finishing the training.
3. The method of claim 2, wherein the training step of the CYOLO model further comprises:
and carrying out grid mask processing on the first sample image before the CYOLO model detects the first sample image.
4. The method of claim 2 or 3, wherein the first loss function is:
Figure FDA0003337221760000021
wherein S represents the number of meshes in the prediction mesh, B represents the prediction mesh,
Figure FDA0003337221760000022
and
Figure FDA0003337221760000023
is a binary value and is used as a binary value,
Figure FDA0003337221760000024
representing a prediction offset vector, vijkA target vector representing a ground truth value,
Figure FDA0003337221760000025
which is indicative of a prediction reference,
Figure FDA0003337221760000026
denotes the target reference, λcoordWeight factor, λ, representing coordinate loss termobjAnd λnoobjA weight factor, λ, representing a reference loss termclsWeight factor, coordinate loss L, representing the reference loss term for negative sample anchoringregIs L1Loss, reference loss LconfFor focus loss, class loss LclsIn order to have a sigmoidal cross entropy, i, j and k represent the serial numbers, x and y represent the coordinates of the center of the box, w represents the width of the box, h represents the height of the box,
Figure FDA0003337221760000027
indicating fastenersThe predicted probability distribution of the type recognition result,
Figure FDA0003337221760000028
is the true value of the first label.
5. The method of claim 1, further comprising a training step of the RRNet model, wherein the training step of the RRNet model comprises:
acquiring a second sample image and a second label; the second sample image comprises fasteners, and the second label is used for marking the types of the fastener defects in the second sample image;
setting parameters of a rotating frame;
the RRNet model predicts the offset vector of the fastener in the second sample image through a prediction frame, adjusts the direction of the fastener in the second sample image through the rotation of the rotation frame, sets an anchoring frame to sample and identify the defect type of the fastener in the second sample image, and outputs a defect type identification result;
setting a second loss function;
determining the value of the second loss function according to the defect type identification result and the second label;
and adjusting the network parameters of the RRNet model according to the value of the second loss function or finishing the training.
6. The method for detecting the defects of the fasteners of the overhead line system bracket device according to claim 5, wherein the parameters of the rotating frame are as follows:
tx＝(x-xa)/wa，ty＝(y-ya)/ha
tw＝log(w/wa)，th＝log(h/ha)，tθ＝θ-θa
Figure FDA0003337221760000031
Figure FDA0003337221760000032
wherein x and y represent center coordinates of the rotating frame, w represents a width of the rotating frame, h represents a height of the rotating frame, θ represents a direction angle of the rotating frame, xaAnd yaRepresenting the center coordinates of the anchor box, waRepresents the width of the anchor frame, haRepresenting the height, θ, of the anchoring frameaRepresents the angle of orientation of the anchor frame,
Figure FDA0003337221760000033
and
Figure FDA0003337221760000034
represents the coordinates of the center of the prediction box,
Figure FDA0003337221760000035
represents the width of the prediction box and the prediction frame,
Figure FDA0003337221760000036
represents the height of the prediction box and the height of the prediction box,
Figure FDA0003337221760000037
representing the directional angle of the prediction box.
7. The method of claim 5 or 6, wherein the second loss function is:
Figure FDA0003337221760000038
wherein the content of the first and second substances,
Figure FDA0003337221760000039
representing a prediction offset vector, vijkA target vector representing a ground truth value,
Figure FDA00033372217600000310
is the binary value, N is the number of anchors, pnA predicted probability distribution, t, representing the result of the defect type recognitionnRepresenting a probability distribution of the second label,
Figure FDA00033372217600000311
representing the IoU value, λ, re-represented between the minimum bounding rectangle of the nth anchor and the ground truth targetcoord、λclsAnd λiouRepresents a weight factor, LsregRepresents the smoothing L1Loss, LregRepresents L1Loss, LclsIndicating a loss of focus.
8. The utility model provides a contact net support device fastener defect detecting system which characterized in that, contact net support device fastener defect detecting system includes:
the first module is used for acquiring an image to be detected; the image to be detected comprises a contact net bracket device;
a second module for establishing a CYOLO model;
the third module is used for establishing an RRNet model;
the fourth module is used for inputting the image to be detected into the CYOLO model, and the CYOLO model extracts a fastener area of the contact net bracket device in the image to be detected and outputs the fastener area;
a fifth module for inputting the fastener region into the RRNet model, the RRNet model outputting a defect identification result for the fastener region.
9. A computer apparatus comprising a memory for storing at least one program and a processor for loading the at least one program to perform the method of detecting a defect in a fastener of an overhead contact system bracket apparatus of any of claims 1 to 7.
10. A storage medium having stored therein a processor-executable program, wherein the processor-executable program, when executed by a processor, is configured to perform the catenary bracket device fastener defect detection method of any of claims 1-7.
CN202111297730.9A 2021-11-04 2021-11-04 Method, system, device and medium for detecting defects of fasteners of contact net bracket device Pending CN114170136A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202111297730.9A CN114170136A (en) 2021-11-04 2021-11-04 Method, system, device and medium for detecting defects of fasteners of contact net bracket device

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202111297730.9A CN114170136A (en) 2021-11-04 2021-11-04 Method, system, device and medium for detecting defects of fasteners of contact net bracket device

Publications (1)

Publication Number Publication Date
CN114170136A true CN114170136A (en) 2022-03-11

Family

ID=80478017

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202111297730.9A Pending CN114170136A (en) 2021-11-04 2021-11-04 Method, system, device and medium for detecting defects of fasteners of contact net bracket device

Country Status (1)

Country Link
CN (1) CN114170136A (en)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114913232A (en) * 2022-06-10 2022-08-16 嘉洋智慧安全生产科技发展（北京）有限公司 Image processing method, apparatus, device, medium, and product
CN115753066A (en) * 2022-12-23 2023-03-07 湖北中程科技产业技术研究院有限公司 New energy automobile fastener intellectual detection system
CN116939532A (en) * 2023-09-18 2023-10-24 黑龙江伯安科技有限公司 5G-based communication tower remote monitoring system

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114913232A (en) * 2022-06-10 2022-08-16 嘉洋智慧安全生产科技发展（北京）有限公司 Image processing method, apparatus, device, medium, and product
CN114913232B (en) * 2022-06-10 2023-08-08 嘉洋智慧安全科技(北京)股份有限公司 Image processing method, device, equipment, medium and product
CN115753066A (en) * 2022-12-23 2023-03-07 湖北中程科技产业技术研究院有限公司 New energy automobile fastener intellectual detection system
CN116939532A (en) * 2023-09-18 2023-10-24 黑龙江伯安科技有限公司 5G-based communication tower remote monitoring system
CN116939532B (en) * 2023-09-18 2023-12-19 黑龙江伯安科技有限公司 5G-based communication tower remote monitoring system

Similar Documents

Publication Publication Date Title
CN114170136A (en) Method, system, device and medium for detecting defects of fasteners of contact net bracket device
US10192107B2 (en) Object detection method and object detection apparatus
WO2019033572A1 (en) Method for detecting whether face is blocked, device and storage medium
CN109800682B (en) Driver attribute identification method and related product
WO2021217924A1 (en) Method and apparatus for identifying vehicle type at traffic checkpoint, and device and storage medium
WO2018154709A1 (en) Movement learning device, skill discrimination device, and skill discrimination system
CN111626295B (en) Training method and device for license plate detection model
CN109508636A (en) Vehicle attribute recognition methods, device, storage medium and electronic equipment
CN110245554B (en) Pedestrian movement trend early warning method, system platform and storage medium
CN112419202B (en) Automatic wild animal image recognition system based on big data and deep learning
CN111932596B (en) Method, device and equipment for detecting camera occlusion area and storage medium
CN110796701B (en) Identification method, device and equipment of mark points and storage medium
CN108229281B (en) Neural network generation method, face detection device and electronic equipment
Mutneja et al. GPU accelerated face detection from low resolution surveillance videos using motion and skin color segmentation
JP2006323779A (en) Image processing method and device
CN109657577B (en) Animal detection method based on entropy and motion offset
CN112991397B (en) Traffic sign tracking method, apparatus, device and storage medium
CN112101139B (en) Human shape detection method, device, equipment and storage medium
CN114241354A (en) Warehouse personnel behavior identification method and device, computer equipment and storage medium
JP7215569B2 (en) Object feature quantity extraction device, object feature quantity extraction method and program
CN113762017A (en) Action recognition method, device, equipment and storage medium
CN113256600B (en) Camera dust detection method and device based on artificial intelligence and electronic equipment
CN117079344A (en) Pedestrian behavior detection method, detection device, equipment and storage medium
CN117523282A (en) Anchor-free example-level human body part detection method and device and storage medium
CN117689714A (en) Image-based target position identification method and device and electronic equipment

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination