CN111382709A

CN111382709A - Insulator image detection method based on unmanned aerial vehicle inspection

Info

Publication number: CN111382709A
Application number: CN202010167880.7A
Authority: CN
Inventors: 刘黎; 韩睿; 齐冬莲; 闫云凤
Original assignee: Zhejiang University ZJU; State Grid Zhejiang Electric Power Co Ltd; Electric Power Research Institute of State Grid Zhejiang Electric Power Co Ltd
Current assignee: Zhejiang University ZJU; State Grid Zhejiang Electric Power Co Ltd; Electric Power Research Institute of State Grid Zhejiang Electric Power Co Ltd
Priority date: 2020-03-11
Filing date: 2020-03-11
Publication date: 2020-07-07

Abstract

The invention relates to the technical field of image recognition and detection applied to insulators, and provides an insulator image detection method based on unmanned aerial vehicle routing inspection, which comprises the following steps: acquiring an aerial image set through shooting equipment carried by an unmanned aerial vehicle during cruising; step two: classifying the aerial image set in a manual labeling mode and forming a plurality of related task packages; step three: amplifying a plurality of related task packages in a data amplification mode, and performing parallel learning training on the insulator detection model by using the amplified data packages; step four: in the multi-task learning training, a training evaluation formula is introduced; if the result value generated by the calculation of the training evaluation formula is larger than the set threshold value, immediately stopping continuously learning the auxiliary task; step five: verifying the effectiveness in insulator detection through quantitative analysis and comparison of experimental results; the detection performance and accuracy are improved.

Description

Insulator image detection method based on unmanned aerial vehicle inspection

Technical Field

The invention relates to the technical field of image recognition and detection applied to insulators, in particular to an insulator image detection method based on unmanned aerial vehicle inspection.

Background

On transmission line patrols and examines, carry out fault detection investigation through discernment unmanned aerial vehicle aerial photograph image to the insulator, play important role in maintaining transmission system safety and stability operation. With the help of the unmanned aerial vehicle, the insulator detection can be realized based on an image recognition algorithm. Traditional image recognition algorithms, such as an insulator detection method based on skeleton feature extraction, an insulator detection method based on insulator string feature extraction, insulator detection based on image threshold segmentation and the like, are easily affected by an environmental background, have poor robustness, are easily subjected to false detection or missing detection in the detection of insulator images with large number, complex background and different angles shot by an unmanned aerial vehicle, and cannot perform accurate detection.

In recent years, the continuous progress of deep learning theory has made possible its application in image segmentation. Compared with the traditional method, the deep learning-based method can automatically learn more abstract and expressive characteristics from the image data, such as the algorithm based on the convolutional neural network and the algorithm based on the full convolutional neural network.

The characteristics of the detection algorithm can be automatically extracted in a layered manner while the whole information and the local information are considered by utilizing a Convolutional Neural Network (CNN), the calculation efficiency is improved by adopting the algorithm of the network structure, and the accuracy of insulator detection is greatly improved. However, in the process of using CNN to classify the pixels of the image shot by the drone, the size of the pixel block generated by classification is much smaller than that of the whole shot image, so that only some local features can be extracted from the generated pixel block, which greatly limits the classification performance (detection performance).

Compared with a convolutional neural network, the image segmentation method based on the full convolutional neural network (FCN) is more advanced, the dense prediction under the condition of a non-full connection layer is realized by the algorithm, and the algorithm efficiency is improved by generating a segmentation mask map with any size. However, FCN reduces the resolution of the feature map during multiple sampling, and further loses detailed information such as the target location, so that FCN is difficult to accurately locate an insulator image with the characteristics of a complex background, resulting in a final detection result with a low accuracy.

In addition, the existing method only considers insulator detection as a segmentation task or a bounding box detection task, ignores the mutual connection between the two tasks, and limits the detection performance of the model.

Therefore, an insulator image detection method with high detection performance and high accuracy is urgently needed.

Disclosure of Invention

The technical problem to be solved by the invention is to overcome the defects in the prior art, and provide an insulator image detection method based on unmanned aerial vehicle routing inspection, so that the detection performance and the accuracy are improved.

In order to achieve the purpose, the invention is realized by the following technical scheme: an insulator image detection method based on unmanned aerial vehicle routing inspection comprises the following steps: acquiring an aerial image set through shooting equipment carried by an unmanned aerial vehicle during cruising; step (ii) ofII, secondly: classifying the aerial image set in a manual labeling mode and forming a plurality of related task packages; step three: amplifying a plurality of related task packages in a data amplification mode, and performing parallel learning training on the insulator detection model by using the amplified data packages; step four: in the multitask learning training, let

The loss function value chosen for the kth task on the validation set,

selecting a loss function value for the kth task on the training set; if the result value generated by the formula calculation is larger than the set threshold value epsilon, immediately stopping continuously learning the auxiliary task; the training evaluation formula is as follows,

h represents the iteration step size of the multi-task learning; med represents the median of multitask learning; wherein

δ_k(. cndot.) represents the relative rate of fall of the loss generated during the k-th task training; t represents the total number of iterations performed in the current training process; t represents an equalization factor for controlling the weighting magnitude of the task weighting processing; step five: and verifying the effectiveness in insulator detection through quantitative analysis and comparison of experimental results.

The further preferable scheme of the invention is as follows: in the second step, the manual marking mode comprises 3 types, namely insulator segmentation marking, insulator surrounding frame marking and picture background attribute marking are sequentially carried out; the insulator segmentation marks are used as main tasks, and insulator surrounding frame marks and picture background attributes are used as auxiliary tasks; the training of the auxiliary task is stopped before the auxiliary task starts to over-fit the training set and suppresses the main task from converging.

The further preferable scheme of the invention is as follows: the insulator detection model is obtained by pre-training on a COCO data set.

The further preferable scheme of the invention is as follows: and in the second step, constructing a characteristic pyramid on the network structure by using the obtained data, and fusing the high-dimensional characteristic and the low-dimensional characteristic.

The further preferable scheme of the invention is as follows: and optimizing the output result by adopting a corrosion expansion algorithm.

The further preferable scheme of the invention is as follows: and secondly, before manual marking, randomly cutting all the insulator images shot by the unmanned aerial vehicle into three different sets, namely a training set, a verification set and a test set, in a data amplification mode.

The further preferable scheme of the invention is as follows: the process of constructing the feature pyramid is as follows: 1) firstly, realizing feedforward calculation of a trunk convolution neural network by adopting a bottom-up characteristic channel, and generating a characteristic diagram which has a scaling step length of 2 and is gradually reduced; 2) in the feedforward calculation process of the characteristic channel from bottom to top, the convolution layers which generate the characteristic mapping layers with the same size are classified into the same-stage layer of a characteristic pyramid; further, the output of the last layer of each stage layer is taken as the output of the corresponding stage layer; defining the output of each phase layer as { C1, C2, …, Cn }, which is scaled by a factor of 2 with respect to the input; 3) by performing 2 times of up-sampling on the features of each stage layer from top to bottom, the feature channel from top to bottom can generate the features with rough spatial information and strong image semantic information; 4) and adding feature maps with lower resolution from top to bottom and the same size in the output of the upper stage layer from bottom according to elements, thereby realizing the fusion of high-dimensional and low-dimensional features.

The further preferable scheme of the invention is as follows: the result optimization by adopting a corrosion expansion algorithm comprises the following steps:

1) firstly, corroding short false detection line segments by using a corrosion algorithm for n times to eliminate the false detection line segments, and meanwhile considering that in the process of corroding and correcting the false segmentation regions by using the corrosion algorithm, the regions which are originally correctly segmented are corroded, so that the region occupied by the insulator in each insulator image block is much larger than the region occupied by the false segmentation region segments;

2) after the n-time corrosion algorithm is adopted, the mistakenly divided region sections in the insulator image block disappear, and the correct divided regions in the insulator image block are reserved;

3) and then, adopting an expansion algorithm for n times for each insulator image block, wherein the algorithm is used for restoring all areas corroded by the corrosion algorithm, and finally obtaining an optimization result of insulator detection for removing the mistakenly-segmented area sections after the corrosion algorithm and the expansion algorithm.

The further preferable scheme of the invention is as follows: the parameters used for the results of the quantitative analysis and comparative experiments were as follows:

1) DSC (dice Similarity coefficient), wherein the Similarity coefficient is an index for evaluating an image segmentation repetition rate in the case of performing manual segmentation and automatic segmentation on an image photographed by an unmanned aerial vehicle, and the larger the DSC value is, the higher the detection accuracy is, and the formula thereof is:

in the above formula: TP represents the number of correctly predicted insulator pixel points; FP represents the number of insulator pixel points which are wrongly predicted; FN represents the number of background pixels that were mispredicted.

2) The positive Predictive value refers to a ratio of the number of the insulator points, which is accurately divided, in the image shot by the unmanned aerial vehicle to the number of the insulator points, which is a result generated by dividing the image, wherein the larger the PPV value is, the lower the false detection rate is, and the formula is as follows:

3) sensitivity refers to the ratio of the number of correctly divided insulator points in an image shot by an unmanned aerial vehicle to the number of true insulator points in the image, the larger the Sensitivity value is, the higher the recall rate is, and the formula is as follows:

in conclusion, the invention has the following beneficial effects: an insulator picture is obtained by aerial photography of an unmanned aerial vehicle, a characteristic pyramid is constructed by fusing multi-dimensional characteristic information, loss of detail information such as a target position is avoided, and efficient detection of an insulator in a complex background is achieved; and a multi-task learning algorithm is introduced into the convolutional neural network, so that the generalization capability of the model is further improved, and the insulator detection precision is improved. The insulator actual image obtained by unmanned aerial vehicle aerial photography is used for carrying out experiments, and the result shows that the insulator detection precision can be improved to 95.3% by the method, and the method has high engineering application value.

Drawings

FIG. 1 is a flow chart of the present invention.

FIG. 2 is a graph showing the effect of comparative experiments.

FIG. 3 is a diagram of model iterations with different strategies.

Detailed Description

The present invention will be described in further detail with reference to the accompanying drawings.

The present embodiment is only for explaining the present invention, and it is not limited to the present invention, and those skilled in the art can make modifications without inventive contribution to the present embodiment as needed after reading the present specification, but all of them are protected by patent law within the scope of the claims of the present invention.

As shown in fig. 1, the invention provides an insulator image detection method based on unmanned aerial vehicle routing inspection, which comprises the following steps.

The method comprises the following steps: through the shooting equipment that unmanned aerial vehicle carried on acquireing aerial photography image set when cruising.

Step two: and classifying the aerial image set in a manual labeling mode and forming a plurality of related task packages. Specifically, before manual labeling, all the insulator images shot by the unmanned aerial vehicle are randomly cut and divided into three different sets, namely a training set, a verification set and a test set, in a data amplification mode. The manual marking mode comprises 3 types, namely insulator segmentation marking, insulator surrounding frame marking and picture background attribute marking in sequence. And constructing a feature pyramid on the network structure of the obtained data, and fusing the high-dimensional features and the low-dimensional features. The process of constructing the feature pyramid is as follows:

1) firstly, realizing feedforward calculation of a trunk convolution neural network by adopting a bottom-up characteristic channel, and generating a characteristic diagram which has a scaling step length of 2 and is gradually reduced;

2) in the feedforward calculation process of the characteristic channel from bottom to top, the convolution layers which generate the characteristic mapping layers with the same size are classified into the same-stage layer of a characteristic pyramid; further, the output of the last layer of each stage layer is taken as the output of the corresponding stage layer; defining the output of each phase layer as { C1, C2, …, Cn }, which is scaled by a factor of 2 with respect to the input;

3) by performing 2 times of up-sampling on the features of each stage layer from top to bottom, the feature channel from top to bottom can generate the features with rough spatial information and strong image semantic information;

4) and adding feature maps with lower resolution from top to bottom and the same size in the output of the upper stage layer from bottom according to elements, thereby realizing the fusion of high-dimensional and low-dimensional features.

And (4) dividing and marking the insulator as a main task, and marking the insulator surrounding frame and the picture background attribute as auxiliary tasks. The training of the auxiliary task is stopped before the auxiliary task starts to over-fit the training set and suppresses the main task from converging.

Step three: and amplifying a plurality of related task packages in a data amplification mode, and performing parallel learning training on the insulator detection model by using the amplified data packages. The insulator detection model is obtained by pre-training on a COCO data set.

Step four: in the multitask learning training, let

The loss function value chosen for the kth task on the validation set,

selecting a loss function value for the kth task on the training set; if the result value generated by the formula calculation is larger than the set threshold value epsilon, the continuous learning of the item is immediately stoppedAnd (5) assisting the task.

The training evaluation formula is as follows,

h represents the iteration step size of the multi-task learning; med represents the median of multitask learning. The first item of the above formula, which is larger than the product on the left side of the sign, represents the variation trend of the error generated in the training process, and if the error generated in the training process is rapidly reduced in the multi-task learning iteration interval with h as the step length, the numerical value of the first item of the above formula is very small, which represents that the task is still valuable, and the training can be continued; otherwise, it indicates that the task tends to be stopped. The second term in the above equation, which is larger than the left side of the number, is a generalization error compared to the training set error, and is an importance coefficient of the kth auxiliary task error, which can be learned by gradient descent, and the magnitude of the value indicates the time length of the task influence.

Wherein

δ_k(. cndot.) represents the relative rate of fall of the loss generated during the k-th task training; t represents the total number of iterations performed in the current training process; t denotes an equalization factor for controlling the weight magnitude of the weighting process for the tasks. A larger T indicates a more balanced weight distribution for different tasks. When T is very large, it means that the weight assigned to each task in the weighting process is equal.

Step five: and optimizing the output result by adopting a corrosion expansion algorithm. And verifying the effectiveness in insulator detection through quantitative analysis and comparison of experimental results.

The result optimization by adopting a corrosion expansion algorithm comprises the following steps:

The parameters used for the results of the quantitative analysis and comparative experiments were as follows:

the specific working principle is that the used experimental data are derived from insulator images aerial photographed by an unmanned aerial vehicle, 3000 images are obtained in total, the size of each image is 3860 × 2160, and a 512 × 512 insulator image training set, a verification set and a test set are obtained by means of data amplification operation.

Before training on the insulator data set, pre-training is carried out on the COCO data set, and then the pre-trained model is placed on the insulator data set to be trained. And training the insulator detection network by adopting a random gradient descent mode, wherein the initial learning rate is 0.01, the total iteration times are 50000 times, the learning rate is reduced to 0.001 at the 20000 th time, the learning rate is reduced to 0.0001 at the 40000 times, and the whole model is trained on the NVIDIA Titan X GPU.

In order to verify the application effect of the detection network in insulator detection, the detection network is compared with a convolutional neural network-based insulator detection method and an FCN-based insulator detection method, and the results are shown in table 1.

TABLE 1 comparison of the results of the experiments with different detection methods

	DSC	PPV	Sensitivity
				Methods of the invention	0.953	0.923	0.901
Convolutional neural network	0.723	0.745	0.753
				FCN	0.814	0.801	0.805

As can be seen from the experimental results in table 1, the proposed improved convolutional neural network insulator detection model is much greater than the other two classical methods in the insulator segmentation performance.

Fig. 2 is a comparison of experimental results on the test set. Fig. 2(a) - (e) show the original image, the segmentation label, the text method detection result, the convolutional neural network method detection result, and the FCN method detection result, respectively. As can be seen from fig. 2, the detection results of the convolutional neural network and the FCN method have more false detections, because in the process of down-sampling, the feeling is also increased, the position information of the insulator is lost due to the use of the aggregation context, part of the background close to the insulator in the image shot by the unmanned aerial vehicle is erroneously detected as the insulator in the detection process, and the environmental adaptability of the convolutional neural network and the FCN method is poor. Compared with a convolutional neural network and an FCN detection method, the provided insulator detection method based on the feature pyramid and the multitask learning network can effectively eliminate the interference of the similar background and accurately segment the insulator.

The proposed add-multitask and auxiliary task early stop strategy was experimentally verified. Firstly, on the premise of keeping the network structure unchanged, the addition of the auxiliary task proves that the segmentation performance of the insulator can be improved by the multitask model compared with the single-task model, and the experimental results are shown in table 2.

TABLE 2 multitask comparative test results

	DSC	PPV	Sensitivity
				Segmentation	0.884	0.871	0.874
Partition + bounding box	0.932	0.903	0.889
				Segmentation + background classification	0.892	0.882	0.878
Segmentation + bounding box + background classification	0.953	0.923	0.901

As can be seen from table 2, by adding a plurality of tasks, various performance indexes can be improved, thereby improving the detection accuracy of the insulator.

The training error of the deep learning algorithm gradually decreases with the passage of time, but the error of the verification set increases again, which means that the model is over-fitted, and at this time, an early stopping strategy can be adopted to obtain the parameters which perform best on the verification set. Fig. 3(a) is a multitask training loss curve when the auxiliary task early stopping strategy is not introduced, and when the bounding box regression task and the background classification task converge, the segmentation task also enters a convergence state. FIG. 3(b) is a multitasking curve when an auxiliary task early stop strategy is introduced, after early stopping an already converged auxiliary task, the loss of the split task is reduced compared to the loss of not stopping the auxiliary task early, which illustrates that stopping the auxiliary task early is beneficial for the convergence of the main task.

Claims

1. An insulator image detection method based on unmanned aerial vehicle inspection is characterized by comprising the following steps,

the method comprises the following steps: acquiring an aerial image set through shooting equipment carried by an unmanned aerial vehicle during cruising;

step two: classifying the aerial image set in a manual labeling mode and forming a plurality of related task packages;

step three: amplifying a plurality of related task packages in a data amplification mode, and performing parallel learning training on the insulator detection model by using the amplified data packages;

step four: in the multitask learning training, let

The loss function value chosen for the kth task on the validation set,

selecting a loss function value for the kth task on the training set; if the result value generated by the calculation of the training evaluation formula is larger than the set threshold value epsilon, immediately stopping continuously learning the auxiliary task; the training evaluation formula is as follows,

h represents the iteration step size of the multi-task learning; med represents the median of multitask learning;

wherein

δ_k(. cndot.) represents the relative rate of fall of the loss generated during the k-th task training; t represents the total number of iterations performed in the current training process; t represents an equalization factor for controlling the weighting magnitude of the task weighting processing;

step five: and verifying the effectiveness in insulator detection through quantitative analysis and comparison of experimental results.

2. The insulator image detection method according to claim 1, characterized in that: in the second step, the manual marking mode comprises 3 types, namely insulator segmentation marking, insulator surrounding frame marking and picture background attribute marking are sequentially carried out; the insulator segmentation marks are used as main tasks, and insulator surrounding frame marks and picture background attributes are used as auxiliary tasks; the training of the auxiliary task is stopped before the auxiliary task starts to over-fit the training set and suppresses the main task from converging.

3. The insulator image detection method according to claim 2, characterized in that: the insulator detection model is obtained by pre-training on a COCO data set.

4. The insulator image detection method according to claim 2, characterized in that: and in the second step, constructing a characteristic pyramid on the network structure by using the obtained data, and fusing the high-dimensional characteristic and the low-dimensional characteristic.

5. The insulator image detection method according to claim 2, characterized in that: and optimizing the output result by adopting a corrosion expansion algorithm.

6. The insulator image detection method according to claim 1, characterized in that: and secondly, before manual marking, randomly cutting all the insulator images shot by the unmanned aerial vehicle into three different sets, namely a training set, a verification set and a test set, in a data amplification mode.

7. The insulator image detection method according to claim 4, characterized in that: the process of constructing the feature pyramid is as follows:

8. The insulator image detection method according to claim 5, characterized in that: the result optimization by adopting a corrosion expansion algorithm comprises the following steps:

9. The insulator image detection method according to claim 1, characterized in that: the parameters used for the results of the quantitative analysis and comparative experiments were as follows: