CN117470859B

CN117470859B - Insulator internal defect detection method and device

Info

Publication number: CN117470859B
Application number: CN202311804168.3A
Authority: CN
Inventors: 范亮; 汤坚; 王秋媚; 张磊; 郑路铭
Original assignee: Guangzhou Zhongke Zhi Tour Technology Co ltd
Current assignee: Guangzhou Zhongke Zhi Tour Technology Co ltd
Priority date: 2023-12-25
Filing date: 2023-12-25
Publication date: 2024-03-22
Anticipated expiration: 2043-12-25
Also published as: CN117470859A

Abstract

The application provides a method and a device for detecting internal defects of an insulator. The method comprises the steps of obtaining a patrol video of a power transmission line collected by an unmanned aerial vehicle; the inspection video is an infrared image acquired by infrared acquisition equipment carried by the unmanned aerial vehicle; inputting each frame of infrared image in the inspection video into a trained defect detection model, and carrying out frame-by-frame defect detection on the inspection video by using the trained defect detection model so as to output a defect detection result; the trained defect detection model comprises a backbone network; the main network trains the main network by comparing and learning the features of the same semantics of the visible light image sample set and the infrared image sample set in different domains, so as to obtain a trained main network; the trained backbone network is used for determining parameters of the backbone network; if the defect detection result is that the infrared image contains the defective insulator, the position information and the category information of the defective insulator of the infrared image are marked. Thus, the defect detection of the insulator is realized.

Description

Insulator internal defect detection method and device

Technical Field

The invention relates to the technical field of power transmission line overhaul, in particular to a method and a device for detecting internal defects of an insulator.

Background

The insulator is an indispensable component in the transmission line, and plays a dual role of preventing current from being grounded and supporting wires in the transmission line. The insulator is exposed in the field for a long time and is subjected to strong electric fields, changed temperature and humidity and mechanical stress of line pulling, so that faults are easy to occur. The insulator is subjected to self-explosion, so that flashover can occur during operation of the power transmission line, and safety is endangered; the pollution and dust on the insulator may not be obviously harmful in a dry environment, but when the pollution meets a wet environment, the impedance of the insulator string is greatly reduced, and the risk is generated; the resistance value of the zero-value insulator formed after the insulator is broken down is obviously reduced, so that the insulating capability of the insulator is reduced, and the operation risk is generated. Regular maintenance of the insulators is necessary. The maintenance method of the related art mostly depends on manual climbing of the tower, and is low in efficiency and high in danger.

Disclosure of Invention

The application provides an improved method and device for detecting internal defects of an insulator.

The application provides a method for detecting internal defects of an insulator, which comprises the following steps:

acquiring a patrol video of a power transmission line acquired by an unmanned aerial vehicle; the inspection video is an infrared image acquired by an infrared acquisition device carried by the unmanned aerial vehicle;

Inputting each frame of infrared image of the inspection video into a trained defect detection model, and performing frame-by-frame defect detection on the inspection video by using the trained defect detection model so as to output a defect detection result; the trained defect detection model comprises a backbone network; the main network trains the main network by comparing and learning the features of the same semantics of the visible light image sample set and the infrared image sample set in different domains to obtain a trained main network; the trained backbone network is used for determining parameters of the backbone network;

and if the defect detection result is that the infrared image contains the defective insulator, marking the position information and the category information of the defective insulator of the infrared image.

Further, the method further comprises: the defect detection model is obtained by training in the following way:

obtaining a visible light image sample in a visible light image sample set and an infrared image sample in an infrared image sample set;

performing image registration on a scene-matched visible light image sample and an infrared image sample to obtain a visible light and infrared image pair;

Inputting the visible light and infrared image pairs into a backbone network of a defect detection model to be trained, learning the features of the same semantics of a visible light image sample set and an infrared image sample set in different domains, and training the backbone network of the defect detection model to be trained to obtain a trained backbone network; the trained backbone network is embedded in the defect detection model to be trained, and the defect detection model to be trained is updated, so that parameters of the backbone network of the defect detection model to be trained are known and determined;

acquiring an infrared image sample set with a label;

inputting the infrared image sample set into the defect detection model to be trained, and training the defect detection model to be trained to obtain the trained defect detection model; wherein the training of the defect detection model to be trained includes training initialization parameters of a network layer of the defect detection model to be trained.

Further, the image registration of the scene-matched visible light image sample and the infrared image sample to obtain a visible light and infrared image pair includes:

respectively extracting characteristic points from a visible light image and an infrared image acquired under the same scene to obtain the characteristic points of the visible light image and the characteristic points of the infrared image;

Performing image registration on the characteristic points of the visible light image and the characteristic points of the infrared image to obtain matching point pairs;

transforming parameters based on the matching point pairs to obtain a homography matrix;

and performing perspective transformation on the visible light image by utilizing the homography matrix to obtain the visible light and infrared image pair.

Further, the training the backbone network of the defect detection model to be trained to obtain a trained backbone network includes:

extracting a feature space of the visible light and infrared image pair;

clustering visible light features and infrared light features of the feature space with the same semantic information by contrast learning;

and training the backbone network in a self-supervision manner by comparing the loss function, and increasing the inter-class distance of the features when the intra-class distance of the features is shortened in the iteration process to obtain the trained backbone network.

Further, the defect detection model to be trained comprises a first feature mapping branch and a second feature mapping branch; the first feature mapping branch is used for mapping the visible light image into unit feature vectors, the second feature mapping branch is used for mapping the infrared image into unit feature vectors, and each branch comprises a main network of a defect detection model to be trained and a multi-layer perceptron MLP;

Training the backbone network of the defect detection model to be trained to obtain a trained backbone network, wherein the training comprises the following steps:

initializing weight parameters of the backbone network and the MLP by using a random value;

b, inputting the visible light image sample and the infrared image sample into the backbone network and the MLP, and outputting a first feature vector mapped by the visible light image sample and a second feature vector mapped by the visible light image sample;

c, calculating an error between the first feature vector and the second feature vector by using a contrast loss function;

d, calculating a gradient according to the derivative of the contrast loss function, counter-propagating the error along the minimum gradient direction, and correcting each weight parameter of the main network and the MLP;

and E, repeatedly executing the A to the D until an iteration stop condition is reached, and obtaining the trained backbone network.

Further, the acquiring the infrared image sample set with the label includes:

randomly selecting a predetermined number of images from all the acquired infrared images;

marking an annotation frame on an insulator of the selected image and marking the annotation frame to generate an infrared image sample set with the annotation frame; the information of the annotation frame comprises category names and position information, wherein the category names comprise zero value insulators, low value insulators and normal insulators.

Further, the inputting the infrared image sample set into the defect detection model to be trained, training the defect detection model to be trained, and obtaining the trained defect detection model includes:

dividing the training rounds into a front N rounds and a rear N rounds; the N is the number of turns;

inputting the infrared image sample set into a defect detection model to be trained, freezing weight parameters of a backbone network in the previous N rounds, and performing partial training on the network layer;

and in the last N rounds, carrying out overall training on the trunk network and the network layer to obtain the trained defect detection model.

Further, in the following N rounds, performing overall training on the backbone network and the network layer to obtain the trained defect detection model, including:

a, initializing parameters of the network layer of the defect detection model to be trained by using random values; migrating parameters of the trained backbone network to the defect detection model to be trained;

b, inputting the infrared image sample set into the defect detection model to be trained, and outputting insulator identification, defect type and positioning results;

c, obtaining an error sum of insulator identification, defect category and label frame position regression by using the loss function;

d, obtaining a current gradient according to the derivative of the loss function; according to the historical gradient of each parameter of the defect detection model to be trained, the learning rate of each parameter is adjusted, the error sum is reversely propagated along the minimum gradient direction, and each weight parameter of the defect detection model to be trained is corrected;

and e, repeatedly executing the a to the d until the iteration stopping condition is reached, and obtaining the trained defect detection model.

Further, after the marking of the positional information and the category information of the insulator of the defect of the infrared image, the method further includes:

after the unmanned aerial vehicle inspection is finished, generating a visual report by using the position information and the category information of the marked insulator of the defect of the infrared image; the visual report is used for realizing analysis and processing of the power transmission line by a user.

The application provides an inside defect detection device of insulator, include:

the information acquisition module is used for acquiring the inspection video of the power transmission line acquired by the unmanned aerial vehicle; the inspection video is an infrared image acquired by an infrared acquisition device carried by the unmanned aerial vehicle;

The detection module is used for inputting each frame of infrared image of the inspection video into a trained defect detection model, and carrying out frame-by-frame defect detection on the inspection video by using the trained defect detection model so as to output a defect detection result; the trained defect detection model comprises a backbone network; the main network trains the main network by comparing and learning the features of the same semantics of the visible light image sample set and the infrared image sample set in different domains to obtain a trained main network; the trained backbone network is used for determining parameters of the backbone network;

and the marking module is used for marking the position information and the category information of the defective insulator of the infrared image if the defect detection result is that the infrared image contains the defective insulator.

The present application provides an insulator internal defect detection system comprising one or more processors configured to implement the method of any one of the above.

There is provided a computer readable storage medium having stored thereon a program which, when executed by a processor, implements a method as claimed in any one of the preceding claims.

In some embodiments, the method for detecting the internal defects of the insulator comprises the steps of acquiring a patrol video of a power transmission line acquired by an unmanned aerial vehicle; the inspection video is an infrared image acquired by infrared acquisition equipment carried by the unmanned aerial vehicle; inputting each frame of infrared image in the inspection video into a trained defect detection model, and carrying out frame-by-frame defect detection on the inspection video by using the trained defect detection model so as to output a defect detection result; the trained defect detection model comprises a backbone network; the main network trains the main network by comparing and learning the features of the same semantics of the visible light image sample set and the infrared image sample set in different domains, so as to obtain a trained main network; the trained backbone network is used for determining parameters of the backbone network; if the defect detection result is that the infrared image contains the defective insulator, the position information and the category information of the defective insulator of the infrared image are marked. In the embodiment of the application, the inspection video of the power transmission line is detected through the trained defect detection model, the defective insulator is determined, and then the position information and the category information of the defective insulator of the infrared image are marked to determine the specific category of the defective insulator and the position of the defective insulator. Therefore, the internal defect detection of the insulator is realized, the detection is not dependent on manpower, the efficiency is high, and the danger is low.

Drawings

Fig. 1 is a schematic flow chart of an internal defect detection method of an insulator according to an embodiment of the present application;

FIG. 2 is a schematic diagram of a training process for obtaining a trained defect detection model in the method for detecting internal defects of the insulator shown in FIG. 1;

FIG. 3 is a schematic diagram of a domain-adaptive self-monitoring training framework corresponding to the trained defect detection model shown in FIG. 2;

FIG. 4 is a schematic diagram of the YOLOv5 model of FIG. 2 used in the training process to obtain a trained defect detection model;

FIG. 5 is a schematic diagram showing a specific flow of the training process for obtaining the trained defect detection model shown in FIG. 2;

fig. 6 is a schematic structural diagram of an internal defect detecting device for an insulator according to an embodiment of the present disclosure;

fig. 7 is a block diagram of an insulator internal defect detection system according to an embodiment of the present application.

Detailed Description

Reference will now be made in detail to exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, the same numbers in different drawings refer to the same or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with one or more embodiments of the present specification. Rather, they are merely examples of apparatus and methods consistent with aspects of one or more embodiments of the present description as detailed in the accompanying claims.

It should be noted that: in other embodiments, the steps of the corresponding method are not necessarily performed in the order shown and described in this specification. In some other embodiments, the method may include more or fewer steps than described in this specification. Furthermore, individual steps described in this specification, in other embodiments, may be described as being split into multiple steps; while various steps described in this specification may be combined into a single step in other embodiments.

In order to solve the technical problems that the overhauling method in the related art mostly depends on manual climbing of a tower, the efficiency is low and the danger is high, the embodiment of the application provides an insulator internal defect detection method.

Acquiring a patrol video of a power transmission line acquired by an unmanned aerial vehicle; the inspection video is an infrared image acquired by infrared acquisition equipment carried by the unmanned aerial vehicle; inputting each frame of infrared image in the inspection video into a trained defect detection model, and carrying out frame-by-frame defect detection on the inspection video by using the trained defect detection model so as to output a defect detection result; the trained defect detection model comprises a backbone network; the main network trains the main network by comparing and learning the features of the same semantics of the visible light image sample set and the infrared image sample set in different domains, so as to obtain a trained main network; the trained backbone network is used for determining parameters of the backbone network; if the defect detection result is that the infrared image contains the defective insulator, the position information and the category information of the defective insulator of the infrared image are marked.

In the embodiment of the application, the inspection of the power transmission line is detected through the trained defect detection model, the defective insulator is determined, and then the position information and the category information of the defective insulator of the infrared image are marked to determine the specific category of the defective insulator and the position of the defective insulator, so that the detection of the internal defect of the insulator is realized, the manual work is not relied on, the efficiency is high, and the danger is low.

In the related technology, scene detail information in a visible light image and heat radiation information in an infrared image are combined through a fusion technology, an image with comprehensive characteristics is output, and the error detection rate of defects is effectively reduced by detecting on the fusion image, so that the accurate judgment of internal defects of an insulator is realized. Further, in the related art, a defect detection model based on deep learning needs massive labeling data for generalization training, however, if only a small amount of data is used for model training, the model is very easy to be over-fitted. At present, part of related technologies directly migrate a model obtained by training a visible light image to a defect detection task of a fusion image, and then fine adjustment training is performed by using a small amount of labeling data so as to realize knowledge migration of the visible light image, but due to the fact that distribution differences exist between two types of data, the model can be forcedly fitted with uncorrelated characteristics between the two types of data in the fine adjustment training process, so that negative effects are caused, and model fitting performance is reduced.

Compared with the related art, in the embodiment of the application, the main network is trained by solely comparing and learning the features of the same semantics of the visible light image sample set and the infrared image sample set in different domains through the main network, so that the trained main network is obtained, semantic knowledge of the visible light domain can be migrated to the infrared light domain, the main network learns the feature extraction capability with domain invariance, the negative influence is reduced, and the model fitting performance is improved. And a large number of samples such as a visible light image sample set and an infrared image sample set are used for training a backbone network, and the model is prevented from being over-fitted through big data driving, so that the fitting performance of the model is improved.

Fig. 1 is a flow chart illustrating a method for detecting an internal defect of an insulator according to an embodiment of the present application.

As shown in fig. 1, the method for detecting internal defects of an insulator may include, but is not limited to, the following steps 110 to 130:

step 110, acquiring a patrol video of a power transmission line acquired by an unmanned aerial vehicle; the inspection video is an infrared image acquired by an infrared acquisition device carried by the unmanned aerial vehicle. Only the acquisition with the infrared acquisition device is used at this time.

The infrared acquisition device is used for acquiring infrared images. The infrared acquisition device may also include, but is not limited to, an infrared acquisition camera, a video camera, and other devices with infrared acquisition functions. Other devices with infrared acquisition function such as infrared acquisition handheld terminals and the like.

Step 120, inputting each frame of infrared image of the inspection video into a trained defect detection model, and performing frame-by-frame defect detection on the inspection video by using the trained defect detection model to output a defect detection result; the trained defect detection model comprises a backbone network; the main network trains the main network by comparing and learning the features of the same semantics of the visible light image sample set and the infrared image sample set in different domains, so as to obtain a trained main network; the trained backbone network is used to determine parameters of the backbone network.

Since the visible light image sample set is in the visible light domain and the infrared image sample set is in the infrared light domain, the visible light image sample set and the infrared image sample set are in the different domains.

The trained defect detection model of the embodiment of the application is deployed in a computing platform. Of course, the training process of the defect detection model to be trained can be deployed on the computing platform, or can be deployed on other computing platforms independently. Such a computing platform may include, but is not limited to, a computer, which may also be referred to as a computer.

The trained defect detection model herein may also be referred to as a domain-adaptive self-supervision training framework, or simply as a self-supervision framework, for details, see below.

In step 130, if the defect detection result is that the infrared image includes the defective insulator, the position information and the type information of the defective insulator of the infrared image are marked. The specific type of the defective insulator and the position of the defective insulator are determined by the position information and the type information of the defective insulator marked with the infrared image. If the defect detection result is that the infrared image does not include the defective insulator, the process returns to step 110 to continue execution. So realize unmanned aerial vehicle's inspection. The insulators of the above defects may be, but are not limited to, zero value insulators and low value insulators, and the above defects include internal defects. Since external defects such as self-explosion, breakage and pollution of the insulator sheet can cause temperature change in the insulator, the embodiment of the application determines the defective insulator by detecting the insulator with abnormal temperature.

In some embodiments, after the step 130, the method for detecting an internal defect of an insulator further includes generating a visual report by using position information and category information of the marked insulator with the infrared image defect after the inspection of the unmanned aerial vehicle is completed; the visual report is used for realizing analysis and processing of the transmission line by a user. Thus, the user can analyze and process the transmission line conveniently.

The trained defect detection module may be, but not limited to, YOLO model and other deep learning-based defect detection models. For other fault detection models based on deep learning, if the model structure is composed of a backbone network and other downstream task networks, the backbone network can be trained through the domain self-adaptive self-supervision training framework in the embodiment of the application. Further, one or more of a trained YOLOv5 model, a trained YOLOv6 model, and a trained YOLOv9 model may be used.

The following description is made by taking the trained YOLOv5 model as an example: and deploying the trained YOLOv5 model in a computing platform. When the internal defect detection of the insulator of the power transmission line is carried out, the inspection video of the power transmission line acquired by the unmanned aerial vehicle is transmitted into a computing platform through a network, the platform starts a trained YOLOv5 model to detect the video frame by frame, each frame of infrared image in the video is input into the trained YOLOv5 model to detect the internal defect of the insulator of the image, if the insulator containing the defect in the image is detected, the defect position information and the category information in the image are marked, and after the video detection is finished, a visual report is generated so that power staff can analyze and process the line.

In the embodiment of the application, on the basis of adopting the YOLOv5 model as a defect detection model, firstly, a domain self-adaptive self-supervision training framework is used for independently training a backbone network of the YOLOv5 model, and the migration of semantic knowledge from a visible light domain to an infrared light domain is realized by learning the characteristic of domain invariance between a visible light image and an infrared image. Thus, the self-supervision training framework is used for reasonably approaching the characteristics of the visible light domain and the infrared light domain; and because the self-supervision training does not need to label the images and does not need to consume extra labor and time cost, the main network can be trained by using a large amount of data easily, the model is prevented from being fitted through big data driving, and then the downstream network of the model is subjected to fine tuning training by adopting a small amount of labeling data. Secondly, embedding the trained backbone network into a YOLOv5 model, and training the whole model by using a small amount of marked infrared image data; finally, the YOLOv5 model after double-stage training can realize high-precision insulator internal defect detection by using only infrared images.

Fig. 2 is a schematic diagram of a training flow of a trained defect detection model in the method for detecting an internal defect of the insulator shown in fig. 1.

As shown in fig. 2, the defect detection model is trained by the following steps 210 to 250:

step 210, obtaining a visible light image sample in a visible light image sample set and an infrared image sample in an infrared image sample set; the visible light image sample set and the infrared image sample set are used as self-supervision training data sets.

The visible light image sample set may be a visible light image collected by a visible light collection device mounted on an unmanned aerial vehicle. The visible light acquisition device is used for obtaining visible light images. The visible light collection device can also include, but is not limited to, a visible light collection camera, a video camera and other devices with visible light collection functions. Other devices with visible light collection function such as a visible light collection handheld terminal and the like.

And 220, performing image registration on the scene-matched visible light image sample and the infrared image sample to acquire a visible light and infrared image pair.

The synchronous shooting is scene matching, but because the shot images are stored in the memories of the sensors, the scene matching is needed after the two images are taken out of the memories.

As shown in connection with fig. 2, the above step 220 may further include the steps of (1) to (4) as follows:

(1) And respectively extracting characteristic points from the visible light image and the infrared image acquired under the same scene to obtain the characteristic points of the visible light image and the characteristic points of the infrared image.

FIG. 3 is a schematic diagram of a domain-adaptive self-monitoring training framework corresponding to the trained defect detection model shown in FIG. 2.

As shown in fig. 3, the visible light image and the infrared image are captured by a visible light camera and a thermal infrared imager at the same time. After the registration process is performed,the combination of the two images of the obtained visible light image and the infrared image is a visible light and infrared image pair. The visible light and infrared image pair is used in self-monitoring training framework, and is shown in FIG. 3, and the input is visible light imageAnd infrared image->。

(2) Carrying out image registration on the characteristic points of the visible light image and the characteristic points of the infrared image to obtain matching point pairs;

(3) Transforming parameters based on the matching point pairs to obtain a homography matrix;

(4) And performing perspective transformation on the visible light image by utilizing the homography matrix to obtain a visible light and infrared image pair.

In the early stage, the unmanned aerial vehicle with the visible light camera and the thermal infrared imager is used for carrying out the inspection and flying of the power transmission line to acquire the data of the sub-images of the insulators, and for the same scene, the visible light image and the infrared image in the scene are acquired. In order to avoid the influence of factors such as even heating of the insulator, target infrared reflection radiation generated in the daytime, increase of night humidity and the like caused by sunlight irradiation in the daytime on the real temperature of the insulator, the image acquisition is carried out under the conditions that sunlight interference does not exist at night or in the early morning and evening, and the air humidity is not more than or equal to 70% RH (Relative Humidity ).

Because the problems of different focal lengths and different shooting angles of the visible light camera and the thermal infrared imager exist in the actual image acquisition process, in order to align the spatial positions of the same target in different images, the acquired visible light image and infrared image need to be registered, and a visible light image pair and an infrared image pair are acquired. The registration may include, but is not limited to, extracting feature points first, matching the feature points, and calculating a homography matrix last. In this embodiment, for the visible light image and the infrared image acquired in the same scene, it is further defined as follows: firstly, respectively extracting characteristic points by using a SuperPoint algorithm; secondly, matching the characteristic points of the two images through a LightGlue algorithm to obtain matching point pairs; and finally, calculating a findHomograph function input by the matching point pair into an opencv tool kit to obtain a homography matrix, and performing perspective transformation on the visible light image by using the homography matrix to realize image registration. As such, the SuperPoint algorithm and the LightGlue algorithm are used in order to achieve registration faster.

Step 230, inputting the visible light and infrared image pairs into a backbone network of a defect detection model to be trained, learning features of the same semantics of a visible light image sample set and an infrared image sample set in different domains, and training the backbone network of the defect detection model to be trained to obtain a trained backbone network; and the trained backbone network is embedded in the defect detection model to be trained, and the defect detection model to be trained is updated, so that parameters of the backbone network of the defect detection model to be trained are known and determined.

Step 240, a sample set of infrared images with labels is acquired.

The above-mentioned label can be reflected by the label box. This tag may include, but is not limited to, category names and location information. The infrared image sample set with the label may be the same as or different from the infrared image sample set of the training backbone network, which is not limited herein.

The step 240 may further include the following two steps:

step 1, randomly selecting a predetermined number of images from all the acquired infrared images.

The random selection process is completed by the computing platform randomly. The model training involves two stages of training, firstly training using a self-supervision framework to train the feature extraction capability of the backbone network, and the stages do not need any labeling data; and secondly, training of the whole model, wherein a small amount of infrared images and labeling data are required to train the feature mapping capability of the model, so that the model can identify the category and position the insulator.

Step 2, marking a marking frame on an insulator of the selected image and marking the marking frame to generate an infrared image sample set with the marking frame; the information of the annotation frame comprises category names and position information, wherein the category names comprise zero value insulators, low value insulators and normal insulators.

Randomly selecting 1/10 number of images from all the infrared images acquired in the step 1 to carry out sample labeling, and framing out insulators in the images by using a labeling tool, wherein the information of the labeling frame comprises category names and position information, and the labeled categories comprise three categories of zero-value insulators, low-value insulators, normal insulators and the like. It should be noted that, in the embodiment of the present application, the characteristics of heat radiation of the insulator may be accurately reflected by using the infrared image, and the three types of insulators are distinguished and correctly labeled according to brightness, compared with a normal insulator, the temperature of the zero-value insulator is lower, so that the brightness of the position corresponding to the zero-value insulator is darker; the heating power of the low-value insulator is increased, and the infrared energy radiated outwards is also enhanced, so that the brightness of the corresponding position is brighter.

When labeling, the category names use the pinyin initials of the insulator categories; the position information contains information of the upper left corner and the lower right corner of the label frame, respectively Xmin (upper left corner X coordinate of the label frame), ymin (upper left corner Y coordinate of the label frame), xmax (lower right corner X coordinate of the label frame), and Ymax (lower right corner Y coordinate of the label frame). After an image is marked, all marked information in the image is stored in an xml tag file in a VOC data format. In the embodiment of the present application, not only the category name but also the position information may be displayed on the insulator of the image.

Step 250, inputting an infrared image sample set into a defect detection model to be trained, and training the defect detection model to be trained to obtain a trained defect detection model; wherein training the defect detection model to be trained includes training initialization parameters of a network layer of the defect detection model to be trained.

The initialization parameters of the network layer may include, but are not limited to, the initialization parameters of a feature fusion network (neg) and the initialization parameters of a Head network (Head) for training a defect detection model to be trained. Further, FIG. 4 is a schematic diagram of the YOLOv5 model used in the training process of FIG. 2 to obtain a trained defect detection model. As shown in fig. 4, the YOLOv5 model is described as an example. The FOC is a Focus module in the YOLOv5 model, and the module performs slicing operation on an input picture. CBL is a Convolution module in YOLOv5 model, consisting of Convolution operation (C), batch normalization operation (Batch Normalization, B) and activation function (leak ReLU, L). CSP is a cross-phase local (Cross Stage Partial, CSP) network module in the YOLOv5 model. Also in the YOLOv5 model are the feature map stitching operations Concat, convolutional layer Conv, and the spatial pyramid pooling operations (Spatial Pyramid Pooling, SPP). The YOLOv5 model may consist of a Backbone network (Backbone), a feature fusion network (Neck), and a Head of detection network (Head), as follows: the main network is used as a main body of the model and is used for extracting the characteristics of the image, and a CSP-Darknet53 characteristic extraction network is adopted. The feature fusion network is used for carrying out feature fusion on a plurality of layers, improving the detection capability of the model on targets with different scales, and adopting a combined network of FPN (Feature Pyramid Network ) and PAN (Path Aggregation Network, path aggregation network). The Head network (Head) is used to perform feature mapping through a small number of convolution layers to generate final output, namely, the type and position of the target. Thus, the YOLOv5 model is a data-driven model, and reasonable application of the data usage mode and the model training mode can improve the performance of the YOLOv5 model more effectively.

The backbone network is based on the structural design of a CSPDarknet53 network, adopts a cross-stage local network CSP concept, generates feature layers with different scales from an input image, and extracts rich information features. The feature fusion network combines a feature pyramid network FPN and a path aggregation network PAN. For a multi-scale feature layer output by a backbone network, strong semantic features are conveyed through a self-deep-to-shallow path of the FPN, and then strong positioning features are conveyed through a self-shallow-to-deep network of the PAN, so that fusion of shallow positioning information and deep semantic information on different scales is realized. And the detection Head network (Head) performs feature mapping on the feature graphs of the three scales through three detection branches, and outputs the position information and the category information of insulators of defects in the images.

In the embodiment of the application, the self-supervision framework is used for training, so that the main network has strong feature extraction capability, a large amount of marking data is not needed for training the whole model, and only 1/10 of infrared images are used for marking at the stage.

The step 230 may further include the following steps of < 1 > and < 3 >: and (3) extracting the feature space of the visible light and infrared image pair. And 2, clustering visible light features and infrared light features with the same semantic information in the feature space by contrast learning. And 3, self-monitoring and training the main network by comparing the loss function, and increasing the inter-class distance of the features when the intra-class distance of the features is shortened in the iterative process to obtain the trained main network.

For the pair of visible and infrared images acquired in step 220, the two images in the pair of visible and infrared images have been registered, so that the visible and infrared features at the same location have the same semantic information (the same object or the same scene) although there is a difference in data distribution. Based on the premise, the domain self-adaptive self-supervision training framework of the embodiment of the application trains the backbone network of the YOLOv5 model, the features extracted from the backbone network are clustered by contrast learning, the visible light features and the infrared light features with the same semantic information in the feature space, the inter-class distance of the features is increased while the intra-class distance of the features is shortened, so that semantic knowledge of the visible light domain is migrated to the infrared light domain, and the backbone network learns the feature extraction capability with domain invariance.

In the embodiment of the application, the domain self-adaptive self-supervision training framework and the contrast loss function are combined, the main network can be trained through contrast loss self-supervision on the basis of no need of any manual label, the visible light features and the infrared features with the same semantics are pulled up in iteration, the knowledge migration of the visible light domain is realized, and the feature extraction capability of the main network is improved.

FIG. 5 is a schematic diagram showing a specific flow of the training process for obtaining the trained defect detection model shown in FIG. 2.

Referring to fig. 3, as shown in fig. 5, the defect detection model to be trained includes a first feature mapping branch and a second feature mapping branch; the first feature mapping branch is used for mapping the visible light image into unit feature vectors, the second feature mapping branch is used for mapping the infrared image into unit feature vectors, and each branch comprises a main network of a defect detection model to be trained and a multi-layer perceptron MLP. As such, each branch may be composed of a backbone network of defect detection models to be trained and a multi-layer perceptron MLP, mapping the input visible and infrared images into feature vectors. The domain-adaptive self-supervision training framework is shown in fig. 3 and 5 below, and has two feature mapping branches for mapping visible light images respectivelyAnd infrared image->Mapped into unit feature vectors. Each branch consists of a Backbone network (Backbone) of the YOLOv5 model and a multi-Layer perceptron (MLP), wherein the network structure of the multi-Layer perceptron comprises an average pooling Layer (AvgPooling) at a spatial level, a hidden Layer (Layer 1) with a neuron number of 512 and an output Layer (Layer 2) with a neuron number of 128. The backbone networks of the two feature map branches are weighted in common.

Based on the above, the training of the backbone network of the defect detection model to be trained, to obtain a trained backbone network, includes the following steps a to E: and A, initializing weight parameters of the backbone network and the MLP by using the random value. And B, inputting the visible light image sample and the infrared image sample into a backbone network and an MLP, and outputting a first feature vector mapped by the visible light image sample and a second feature vector mapped by the visible light image sample. And C, calculating the error between the first characteristic vector and the second characteristic vector by using the contrast loss function. The contrast loss function is used for calculating the contrast loss of the visible light feature vector and the infrared feature vector, and the calculated contrast loss is minimum only when the infrared feature vector is close to the visible light feature vector corresponding to the same serial number in the training batch and is far away from other feature vectors. And D, calculating a gradient according to the derivative of the contrast loss function, back-propagating the error along the minimum direction of the gradient, and correcting each weight parameter of the backbone network and the MLP. And E, repeatedly executing the step A to the step D until the iteration stopping condition is reached, and obtaining the trained backbone network. Thus, during the self-supervision training framework training, only the feature extraction capability of the Backbone network (backhaul) of the model is trained, and in the overall model training, the parameters of the feature fusion network (neg) and the Head network (Head), namely the feature fusion capability and the feature mapping capability, need to be trained and adjusted.

And E, determining that the iteration stopping condition is met until the preset iteration times are reached, and stopping iteration. The preset iteration number may be greater than or equal to 100 rounds and less than or equal to 201 rounds. Alternatively, the preset number of iterations may be 200 rounds.

The specific process of the feature mapping is as follows: visible light imageAnd infrared image->Inputting into a backbone network for feature extraction, and respectively obtaining the size of +.>Feature map of->And->. Firstly, inputting the feature map into an average pooling layer to pool the space level to generate +.>Is mapped by the hidden layer and the output layer to obtain a vector with length of 128 +.>And->Finally, in order to enable the vector to perform contrast learning in the unit feature space, the vector needs to be unitized, and the vector can be expressed as:

in the above-mentioned formula, the first step,the values of (2) are 1 and 2, respectively, representing a visible light image and an infrared image,/-for example>Vector expressed as visible light, +.>Vectors denoted as infrared images +.>Denoted as->Intra-vector +.>The values of the individual elements.

The domain adaptive self-monitoring training framework requires a batch of visible light and infrared image pairs as input during training, each batch containingFor visible and infrared image pairs, each image can thus be represented as +. >，/>，/>When equal to 1, this image is visible light, and +.>When equal to 2 it means that the image is an infrared image, and (2)>Representing the serial number of the image in a batch, the vector output by the image after feature mapping can be represented as +.>. The vector unitization is to enable the vector to perform contrast learning in a unit feature space, specifically: the contrast loss calculated by the image pairs of the same batch is the overall error (loss) of the same batch calculated subsequently under the same space>The accumulation operation is reasonable. And, formula for calculating overall loss +.>And their association. Thus, contrast loss of one training lot +.>The contrast loss function can be expressed as: />

Wherein,a hyper-parameter expressed as an adjustment vector inner product size; />Expressed as an exponential function with a base of a natural constant e; />Expressed asExcept->Other sequence number sets than->Represented asElements in the collection, ++>Denoted as->Indicate->And vector obtained by feature mapping of the visible light image. It can be seen that only the vector +.>And->Near and in addition to other vectors->Contrast loss at far distance>The value of (2) will be the smallest. Therefore, the framework can train the backbone network in a self-supervision manner through contrast loss on the basis of no manual label, and the visible light features and infrared features with the same semantics are pulled in the iteration, so that the knowledge migration of the visible light domain is realized, and the feature extraction capability of the backbone network is improved.

The frame training uses random gradient descent as a training optimization strategy for a total of 200 rounds of training. The learning rate is dynamically adjusted from 0.03, and after each round of updating is completed, the learning rate is multiplied by 0.9. In order to further improve the diversity of training images, the visible light images and the infrared images of the input training frame need to be enhanced by random data, including three modes of random cutting, size adjustment and horizontal overturning. Finally, the backbone network of the last training round is extracted.

In the embodiment of the application, the self-supervision training stage is used for training the feature extraction capability, wherein the extracted features are texture features for shallow features and structural semantic features for deep features, and the features are not basically the whole insulator target. In the detection head network of the YOLOv5 model, the extracted features are subjected to feature mapping, and the category, the position and the size of the target object are mapped. And the model main body is mainly used for feature extraction, and has the most huge network structure and preferential training. Is relatively easy to train compared to the fusion and mapping capabilities of the following features.

As shown in connection with fig. 2, the above step 250 may further include the following steps 1 > to 3 >: 1 >, dividing the training rounds into a front N rounds and a rear N rounds; n is the number of turns; wherein N is a positive integer multiple of 2. And the above N may be, but not limited to, 38 or less and 5 or less. Alternatively, N may be, but is not limited to, 40, 50, 60, 80, etc. And are not exemplified here. 2 >, inputting the infrared image sample set into a defect detection model to be trained, freezing the weight parameters of the main network in the previous N rounds, and performing partial training on the network layer; and 3 >, in the latter N rounds, carrying out overall training on the main network and the network layer to obtain a trained defect detection model. Therefore, the frozen backbone network means that the weight parameters of the backbone network are not dynamically adjusted when the defect detection model to be trained is trained, on one hand, the training speed of the model is improved, on the other hand, because the feature extraction network and the detection head network are not trained yet, larger losses can be generated in classification losses and regression losses, and larger parameter adjustment is most likely to be caused to the trained backbone network when gradient is propagated.

The step of 3 > above may further comprise the steps of a to e: a, initializing parameters of a network layer of a defect detection model to be trained by using a random value; and migrating the parameters of the trained backbone network to a defect detection model to be trained, wherein the defect detection model to be trained contains the parameters of the trained backbone network. b, inputting the infrared image sample set into a defect detection model to be trained, and outputting insulator identification, defect category and positioning results. And c, obtaining the error sum of insulator identification, defect type and label frame position regression by using the loss function. d, obtaining a current gradient according to the derivative of the loss function; and according to the historical gradient of each parameter of the defect detection model to be trained, the learning rate of each parameter is adjusted, the error sum is reversely propagated along the minimum direction of the gradient, and each weight parameter of the defect detection model to be trained is corrected. And e, repeatedly executing the steps a to d until the iteration stopping condition is reached, and obtaining a trained defect detection model. The loss function uses a loss function of an original model, such as a YOLOv5 model, and is not described herein.

And e, until training reaches the preset iteration times, determining that an iteration stopping condition is met, stopping iteration, and stopping training when the loss value is not reduced for M continuous rounds. M is 5 or more.

Illustratively, the backbone network trained in step 230 is embedded in the YOLOv5 model, and the training set of infrared images is used to train the whole model. Model training uses an adaptive estimation optimizer as a training optimization strategy for the network for a total of 50 rounds of training. The learning rate was dynamically adjusted starting from 0.0003, and after each round of updating was completed, the learning rate was multiplied by 0.9. In order to improve the iteration efficiency of the model, the weight parameters of the backbone network are frozen in the first 25 rounds, and only the unfrozen network layer is subjected to fine adjustment to participate in gradient updating in back propagation; the last 25 rounds train the model as a whole. Model loss is calculated after each round of training, and training is stopped when the loss value does not drop for 5 consecutive rounds. And obtaining weight parameter files of a plurality of models through multiple times of training, comparing model precision of different weight parameter files under a test set, and selecting a model with highest precision as a final model.

According to the principle that registered visible light images and infrared images have the same semantic information at the same position, the domain self-adaptive self-supervision training framework provided by the application trains a backbone network in a self-supervision manner through contrast loss on the basis of no need of any manual label, and the visible light features and infrared features with the same semantic are pulled in iteration to realize knowledge migration of a visible light domain, so that the backbone network learns feature extraction capability with domain invariance. And then the main network is embedded in the YOLOv5 model, the robustness of the image features provided by the main network for the subsequent insulator internal defect detection task is higher, and the influence of the complex infrared image background and low resolution on the detection precision is effectively solved. Compared with the existing detection method based on fusion of the visible light image and the infrared image, the method and the device have the advantages that knowledge migration in the visible light domain is more reasonably realized through the self-supervision training framework, a large amount of marking data is not needed when the whole model is trained, and consumption of manpower and material resources is reduced.

An infrared image background existing in an insulator internal detection technology of an infrared image in the related technology is complex, and a region with concentrated temperature possibly exists, so that the contrast of an insulator to be detected is low; poor image resolution, blurred visual effect, etc. Compared with the related art, the characteristics of the fully acquired insulator can be obtained in the embodiment of the application.

Fig. 6 is a schematic structural diagram of an apparatus for detecting an internal defect of an insulator according to an embodiment of the present application.

As shown in fig. 6, the insulator internal defect detection device includes the following modules:

the information acquisition module 41 is used for acquiring the inspection video of the power transmission line acquired by the unmanned aerial vehicle; the inspection video is an infrared image acquired by infrared acquisition equipment carried by the unmanned aerial vehicle;

the detection module 42 is configured to input each frame of infrared image of the inspection video into the trained defect detection model, and perform frame-by-frame defect detection on the inspection video using the trained defect detection model to output a defect detection result; the trained defect detection model comprises a backbone network; the main network trains the main network by comparing and learning the features of the same semantics of the visible light image sample set and the infrared image sample set in different domains, so as to obtain a trained main network; the trained backbone network is used for determining parameters of the backbone network;

And a marking module 43, configured to mark the position information and the type information of the defective insulator of the infrared image if the defect detection result is that the infrared image includes the defective insulator.

In some embodiments, the internal defect detection of the insulator further includes a training module, and the defect detection model is obtained through training by the training module:

the self-supervision training data set acquisition unit is used for acquiring visible light image samples in the visible light image sample set and infrared image samples in the infrared image sample set;

the visible light and infrared image pair acquisition unit is used for carrying out image registration on a visible light image sample and an infrared image sample matched with a scene to acquire a visible light and infrared image pair;

the local training unit is used for inputting the visible light and infrared image pairs into a backbone network of the defect detection model to be trained, learning the characteristics of the same semantics of the visible light image sample set and the infrared image sample set in different domains, and training the backbone network of the defect detection model to be trained to obtain a trained backbone network; the trained backbone network is embedded in the defect detection model to be trained, and the defect detection model to be trained is updated, so that parameters of the backbone network of the defect detection model to be trained are known and determined;

The infrared image sample set acquisition unit is used for acquiring an infrared image sample set with a label;

the integral training unit inputs the infrared image sample set to a defect detection model to be trained, trains the defect detection model to be trained, and obtains a trained defect detection model; wherein training the defect detection model to be trained includes training initialization parameters of a network layer of the defect detection model to be trained.

In some embodiments, the detecting of the internal defect of the insulator further includes a generating module, configured to generate a visual report after the inspection of the unmanned aerial vehicle is finished after marking the position information and the category information of the insulator of the defect of the infrared image; the visual report is used for realizing analysis and processing of the transmission line by a user.

The implementation process of the functions and actions of each module/unit in the above device is specifically detailed in the implementation process of the corresponding steps in the above method, so that the same technical effects can be achieved, and will not be described herein again.

The insulator internal defect detection system provided by the embodiment of the application comprises the insulator internal defect detection device.

Fig. 7 is a block diagram of an insulator internal defect detection system 50 according to an embodiment of the present application.

As shown in fig. 7, the insulator internal defect detection system 50 includes one or more processors 51 for implementing the insulator internal defect detection method as described above.

In some embodiments, the insulator internal defect detection system 50 may include a computer readable storage medium 59, the computer readable storage medium 59 may store a program that may be invoked by the processor 51, and may include a non-volatile storage medium. In some embodiments, the insulator internal defect detection system 50 may include a memory 58 and an interface 57. In some embodiments, the insulator internal defect detection system 50 may also include other hardware depending on the application.

The computer-readable storage medium 59 of the embodiment of the present application has stored thereon a program for implementing the insulator internal defect detection method described above when executed by the processor 51.

The present application may take the form of a computer program product embodied on one or more computer-readable storage media 59 (including but not limited to disk storage, CD-ROM, optical storage, etc.) having program code embodied therein. Computer-readable storage media 59 include both non-transitory and non-transitory, removable and non-removable media, and may be implemented in any method or technology for information storage. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of computer readable storage media 59 include, but are not limited to: phase change memory (PRAM), static Random Access Memory (SRAM), dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), read Only Memory (ROM), electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital Versatile Disks (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium, may be used to store information that may be accessed by the computing device.

The foregoing description of the preferred embodiments is provided for the purpose of illustration only, and is not intended to limit the scope of the disclosure, since any modifications, equivalents, improvements, etc. that fall within the spirit and principles of the disclosure are intended to be included within the scope of the disclosure.

It should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, the phrase "comprising one … …" does not exclude the presence of additional identical elements in a process, method, article, or apparatus that comprises the depicted element.

Claims

1. A method for detecting an internal defect of an insulator, comprising:

If the defect detection result is that the infrared image contains a defective insulator, marking position information and category information of the defective insulator of the infrared image;

the method further comprises the steps of: the defect detection model is obtained by training in the following way: obtaining a visible light image sample in a visible light image sample set and an infrared image sample in an infrared image sample set; performing image registration on a scene-matched visible light image sample and an infrared image sample to obtain a visible light and infrared image pair; inputting the visible light and infrared image pairs into a backbone network of a defect detection model to be trained, learning the features of the same semantics of a visible light image sample set and an infrared image sample set in different domains, and training the backbone network of the defect detection model to be trained to obtain a trained backbone network; the trained backbone network is embedded in the defect detection model to be trained, and the defect detection model to be trained is updated, so that parameters of the backbone network of the defect detection model to be trained are known and determined; acquiring an infrared image sample set with a label; inputting the infrared image sample set into the defect detection model to be trained, and training the defect detection model to be trained to obtain the trained defect detection model; wherein the training of the defect detection model to be trained comprises training initialization parameters of a network layer of the defect detection model to be trained;

Training the backbone network of the defect detection model to be trained to obtain a trained backbone network, wherein the training comprises the following steps: extracting a feature space of the visible light and infrared image pair; clustering visible light features and infrared light features with the same semantic information in the feature space through contrast learning; the trunk network is trained in a self-supervision mode through a contrast loss function, the inter-class distance of the features is increased when the intra-class distance of the features is shortened in the iteration process, and the trained trunk network is obtained;

correspondingly, the defect detection model to be trained comprises a first characteristic mapping branch and a second characteristic mapping branch; the first feature mapping branch is used for mapping the visible light image into unit feature vectors, the second feature mapping branch is used for mapping the infrared image into unit feature vectors, and each branch comprises a main network of a defect detection model to be trained and a multi-layer perceptron MLP; training the backbone network of the defect detection model to be trained to obtain a trained backbone network, wherein the training comprises the following steps: initializing weight parameters of the backbone network and the MLP by using a random value; b, inputting the visible light image sample and the infrared image sample into the backbone network and the MLP, and outputting a first feature vector mapped by the visible light image sample and a second feature vector mapped by the visible light image sample; c, calculating an error between the first feature vector and the second feature vector by using a contrast loss function; d, calculating a gradient according to the derivative of the contrast loss function, counter-propagating the error along the minimum gradient direction, and correcting each weight parameter of the main network and the MLP; and E, repeatedly executing the A to the D until an iteration stop condition is reached, and obtaining the trained backbone network.

2. The method for detecting an internal defect of an insulator according to claim 1, wherein the image registration of the scene-matched visible light image sample and the infrared image sample to obtain a visible light and infrared image pair comprises:

3. The method for detecting internal defects of an insulator according to claim 1, wherein the acquiring a sample set of infrared images with labels comprises:

4. The method for detecting internal defects of an insulator according to claim 1, wherein the inputting the infrared image sample set into the defect detection model to be trained, training the defect detection model to be trained, and obtaining the trained defect detection model comprises:

5. The method for detecting internal defects of an insulator according to claim 4, wherein the step of performing overall training on the backbone network and the network layer in the subsequent N rounds to obtain the trained defect detection model comprises:

6. The insulator internal defect detection method according to claim 1, wherein after the marking of the positional information and the category information of the insulator of the defect of the infrared image, the method further comprises:

7. An insulator internal defect detection device, characterized by comprising:

the marking module is used for marking the position information and the category information of the insulators of the defects of the infrared image if the defect detection result is that the infrared image contains the insulators of the defects;

the apparatus further comprises: the defect detection model is obtained by training in the following way: obtaining a visible light image sample in a visible light image sample set and an infrared image sample in an infrared image sample set; performing image registration on a scene-matched visible light image sample and an infrared image sample to obtain a visible light and infrared image pair; inputting the visible light and infrared image pairs into a backbone network of a defect detection model to be trained, learning the features of the same semantics of a visible light image sample set and an infrared image sample set in different domains, and training the backbone network of the defect detection model to be trained to obtain a trained backbone network; the trained backbone network is embedded in the defect detection model to be trained, and the defect detection model to be trained is updated, so that parameters of the backbone network of the defect detection model to be trained are known and determined; acquiring an infrared image sample set with a label; inputting the infrared image sample set into the defect detection model to be trained, and training the defect detection model to be trained to obtain the trained defect detection model; wherein the training of the defect detection model to be trained comprises training initialization parameters of a network layer of the defect detection model to be trained;