CN117541534A

CN117541534A - Power transmission line inspection method based on unmanned plane and CNN-BiLSTM model

Info

Publication number: CN117541534A
Application number: CN202311319249.4A
Authority: CN
Inventors: 玄志恒; 杨东东; 杨绍辉; 田野; 邵震; 韩明奇; 刘瑞华; 赵含笑; 陈鹏; 张悦
Original assignee: Puyang Power Supply Co of State Grid Henan Electric Power Co Ltd
Current assignee: Puyang Power Supply Co of State Grid Henan Electric Power Co Ltd
Priority date: 2023-10-12
Filing date: 2023-10-12
Publication date: 2024-02-09

Abstract

The invention relates to a power transmission line inspection method based on an unmanned aerial vehicle and a CNN-BiLSTM model. Firstly, collecting a large number of different power transmission line images under different weather conditions through unmanned aerial vehicle aerial image data, wherein the power transmission line images comprise normal states and various defect conditions. The image is then preprocessed, including denoising, uniform size, etc. Next, CNN is used to extract image features and serialize them for the time series modeling of BiLSTM. Then, a CNN-BiLSTM mixed model is constructed to effectively capture time sequence information among components, and identification of complex defects is achieved. Next, the model is trained and optimized using the prepared training data, and the super parameters are adjusted to achieve optimal performance. And finally, applying the trained CNN-BiLSTM model to a new unmanned aerial vehicle aerial image to detect defects, automatically identifying various defects and returning position and name information. By the method, efficient and accurate inspection of the power transmission line is realized, and the efficiency and accuracy of defect detection are improved.

Description

Power transmission line inspection method based on unmanned plane and CNN-BiLSTM model

Technical Field

The invention belongs to the technical field of power transmission of power systems, and particularly relates to a power transmission line inspection method based on an unmanned plane and a CNN-BiLSTM model.

Background

At present, unmanned aerial vehicle transmission line inspection is used as an efficient and safe inspection mode, and is widely applied in the power industry. However, the existing unmanned aerial vehicle transmission line inspection technology still has some bottlenecks, limitations and disadvantages. Firstly, the traditional unmanned aerial vehicle inspection mainly relies on manual visual inspection, and for a large-scale transmission line network, the problems of fatigue, omission and the like are easily caused due to heavy and time-consuming tasks. Meanwhile, manual inspection is affected by experience level of operators and environmental factors, and subjectivity and consistency differences may exist in inspection results. Second, traditional unmanned aerial vehicle inspection lacks intelligent image analysis and recognition capabilities. Although the existing target detection algorithm can detect the defects of the power transmission equipment, the problems of missed detection or false detection often exist when the aerial image of a multi-scale and complex scene is processed, and especially the defects with small size or blurring are difficult to accurately identify.

Along with the continuous development of computer vision, deep learning and other technologies, unmanned aerial vehicle transmission line inspection technology is developing towards the intelligent and automatic direction. In recent years, convolutional Neural Networks (CNNs) in deep learning have made a significant breakthrough in the fields of image processing and object detection, which have advantages in feature extraction and image recognition. In addition, a variant of the Recurrent Neural Network (RNN), a two-way long short-term memory network (BiLSTM), is excellent in terms of sequence modeling, gradually applied to image sequence processing tasks. Aiming at challenges in unmanned aerial vehicle transmission line inspection, a hybrid model combining CNN and BiLSTM is considered as a future development trend. The model can effectively integrate the advantages of image feature extraction and sequence modeling, enhance the detection capability of complex scenes and small-size defects, simultaneously lighten the burden of manual visual inspection and improve the automation degree and the accuracy of inspection.

The invention aims to provide a power transmission line inspection image detection method based on a CNN-BiLSTM hybrid model, which combines a multi-scale target detection and attention mechanism to solve a series of problems in the traditional unmanned aerial vehicle inspection. By combining a Convolutional Neural Network (CNN) and a two-way long-short-term memory network (BiLSTM), the method can overcome the bottleneck and the limit existing in the conventional unmanned aerial vehicle inspection, and intelligent image analysis and recognition are realized. The traditional manual visual inspection has the problems of visual fatigue, omission and false detection, and the mixed model can effectively capture complex scenes and small-size defects, and improves detection accuracy and robustness. The method introduces a multi-scale target detection and attention mechanism, further enhances the adaptability of the model to defects of different scales, and improves the inspection efficiency and accuracy. By the application of the technology, labor cost and time investment can be obviously reduced, and partial automatic line inspection is realized, so that great economic benefits are brought to the power industry. Meanwhile, the defects of the power transmission equipment are timely and accurately found and processed, the reliability and the safety of a power system are improved, the occurrence rate of fault accidents is reduced, and the stability of power supply is improved. Therefore, the invention has important innovation and application prospect in the technology, and is expected to bring revolutionary transformation and advantages to the unmanned aerial vehicle transmission line inspection field.

Disclosure of Invention

The invention relates to a power transmission line inspection method based on a CNN-BiLSTM model, in particular to the technical field of power transmission equipment defect detection by using unmanned aerial vehicle aerial images.

The existing transmission line inspection method generally depends on manual inspection and image analysis, is time-consuming and has subjective factors to influence the judgment accuracy. In order to solve the problems, the invention provides an automatic and efficient power transmission line inspection method, which is used for processing aerial images of an unmanned aerial vehicle by adopting a CNN-BiLSTM model to realize automatic detection and identification of defects of power transmission equipment so as to improve inspection efficiency and accuracy.

The invention aims to provide a power transmission line inspection method based on a CNN-BiLSTM model, which comprises the following steps:

step S1: and (5) data acquisition and preprocessing. And collecting a large amount of unmanned aerial vehicle aerial image data under different power transmission lines and different weather conditions, wherein the unmanned aerial vehicle aerial image data comprise a normal state and various defect conditions. The image data is preprocessed, including image denoising, uniform size, etc., to prepare the data for subsequent model training.

Step S2: feature extraction and serialization. And adopting a Convolutional Neural Network (CNN) to extract the characteristics of the preprocessed image. CNNs are able to capture local and global features of images, providing powerful support for subsequent sequence modeling. The extracted image features are serialized for sequential modeling by the BiLSTM model.

Step S3: and establishing a CNN-BiLSTM model. And (3) building a CNN-BiLSTM hybrid model, and inputting the image features extracted by CNN into BiLSTM for sequence modeling. BiLSTM is capable of effectively capturing timing information between components, and is advantageous for identifying complex defect conditions.

Step S4: model training and optimization. And (3) training the CNN-BiLSTM model by using the training data prepared in the step (1), and performing super-parameter tuning to obtain the optimal performance. The goal of model training is to enable it to accurately identify different types of power transmission equipment defects.

Step S5: and (5) detecting image defects. And performing defect detection on the new unmanned aerial vehicle aerial image by using the trained CNN-BiLSTM model. The model can automatically identify common defects such as insulator breakage, bolt loss, bolt rust, bird nest, foreign matter winding and the like, and return defect position and name information.

Specifically, the step S1 of data acquisition and preprocessing includes the following sub-steps:

step S1.1: and (5) data collection. Firstly, collecting a large number of unmanned aerial vehicle aerial image data under different power transmission lines and different weather conditions. These data include transmission line images in a normal state and transmission line images in various defect cases, such as broken insulators, lost pins, rusted bolts, bird nests, foreign object entanglement, and the like. The data acquisition can be carried out on site through unmanned aerial vehicles or other applicable aerial photographing equipment, so that the authenticity and diversity of the data are ensured.

Step S1.2: and (5) marking data. After the image data are obtained, accurate labeling of the data is required to be used as a basis for model training. In the labeling process, the defect type and position information existing in each image should be clarified. For each defect, it is necessary to specify its coordinates or area in the image, as well as the name of the defect. The data labeling is a key step of training the CNN-BiLSTM model, and the accuracy of the data labeling directly influences the detection effect of the model.

Step S1.3: and (5) preprocessing data. Preprocessing of the image data is necessary before model training. The data preprocessing aims at enabling the image data to reach uniform format and quality, and ensuring that the model can be trained under uniform conditions. The pretreatment comprises the following aspects:

denoising an image: noise and unclear portions may exist because aerial images may be affected by wind, vibration, and the like. Therefore, denoising techniques, such as filtering algorithms, should be used to remove noise from the image.

The size unifies: the CNN-BiLSTM model requires a fixed-size input, and therefore, all images need to be adjusted to the same size. The size unification can be realized by an interpolation method or a clipping mode, so that the image input is ensured to have the same dimension.

Normalization: the image pixel values are normalized, typically scaling the pixel values to a range of 0 to 1, to better process the image data during model training.

Step S1.4: data set partitioning. After the data preprocessing is completed, the data set should be divided into a training set, a validation set and a test set for training, tuning and evaluation of the CNN-BiLSTM model. Typically, most of the data is used for the training set, some for the validation set and a small amount for the test set. The aim of dividing the data set is to verify the generalization capability of the model on unseen data and to adjust the hyper-parameters of the model to improve the performance of the model.

Specifically, the step S2 feature extraction and serialization is to capture effective visual features from unmanned aerial vehicle aerial image data, and serialize the features into an input format suitable for processing by the BiLSTM model. The method comprises the following substeps:

step S2.1: convolutional Neural Network (CNN) feature extraction. The image data is feature extracted by a pre-trained Convolutional Neural Network (CNN). Specifically, the CNN model adopts a classical convolution layer, a pooling layer and an activation function, and extracts local features and global features of an image through a series of convolution operations. The weights and parameters of the CNN model are optimized through large-scale image data in a pre-training stage, so that the CNN model has effective feature extraction capability on images.

Step S2.2: the image features are serialized. The image features extracted from the CNN are serialized into one-dimensional vectors. Specifically, the image characteristics are obtained by flattening the output of the last layer of CNN. The flattening operation converts the image features from a multi-dimensional matrix to one-dimensional vectors to facilitate subsequent BiLSTM sequence modeling processing.

Step S2.3: time step division. The image features of a one-dimensional vector are divided into equal length time steps. In particular, the length of the time step is determined by the requirements of the input BiLSTM model, typically set according to the specific application scenario and computing resources. The purpose of equally dividing the image features into time steps is to enable the BiLSTM to model in sequential order, capturing timing information in the image data.

Step S2.4: and serializing the image feature matrix. The time-stepped image features are reorganized into a serialized feature matrix. Specifically, the number of rows of the feature matrix corresponds to the number of time steps, and the number of columns is equal to the dimension of the image feature in each time step. Through such serialization operations, the image features are effectively organized into an input format acceptable to the BiLSTM model.

Specifically, the step S3 of establishing the CNN-BiLSTM model comprises the following detailed steps:

Step S3.1: and building a convolutional neural network. In order to achieve image feature extraction, the convolutional neural network part adopts a structure with a plurality of convolutional layers and pooled layers. The convolution layer is used for extracting local features of the image, and the pooling layer is used for reducing the number of parameters and the calculation complexity of the model. Let the input aerial image be X, the output of the first layer convolution layer be C1, and the calculation formula of C1 is as follows:

C1＝f(W ₁ *X+b ₁ )

wherein W is ₁ Weight parameters representing convolution kernels, b ₁ For bias terms, denote convolution operations, f is an activation function (e.g., a ReLU function), and the output of C1 is a feature map.

Step S3.2: and (5) pooling layer treatment. In the CNN, the pooling layer is configured to downsample the feature map, and reduce the data dimension. The usual pooling operation is maximum pooling or average pooling. Assuming that the output after the pooling layer processing is P1, the calculation formula of P1 is as follows:

P1＝pooling(C1)

wherein, pooling represents pooling operation, and P1 is a feature map after downsampling.

Step S3.3: multi-layer rolling and pooling. To enhance the feature extraction capability of CNNs, a multi-layer convolution and pooling structure may be employed. Parameters such as the convolution kernel size, step size, activation function and the like of each layer can be set according to specific tasks. After multi-layer rolling and pooling, richer image features can be obtained.

Step S3.4: the image features are serialized. In order to facilitate the time sequence modeling of the BiLSTM model, the feature map output by the CNN part needs to be subjected to serialization processing. And flattening the feature map obtained by the multi-layer convolution and pooling into a one-dimensional vector to obtain a serialized image feature sequence. Thus, each serialized feature vector represents both local and global information in the image.

Step S3.5: two-way long and short term memory network (BiLSTM). To time-sequential model the serialized image feature sequence, the BiLSTM model was introduced. BiLSTM is a recurrent neural network capable of capturing sequence information. The model has both forward (forward) and backward (reverse) directions in time sequence, and processes the sequence data respectively. Let the serialized image feature sequence be S, output of BiLSTM be H, and the calculation process of forward and backward LSTM is as follows:

forward LSTM calculation:

backward LSTM calculation:

wherein,input sequence representing forward direction,/->Representing the reverse input sequence,/->For the output of the forward LSTM +.>Is the output of the reverse LSTM.

Step S3.6: the forward and backward outputs are fused. In order to fuse the outputs of the forward and backward LSTM, the BiLSTM model connects the outputs of the two directions to obtain the final output H, as follows:

Wherein [ (C) represents the ligation operation and H is the final output signature sequence of BiLSTM).

Specifically, in order to establish a high-efficiency accurate CNN-BiLSTM model for power transmission line inspection, the invention adopts the following specific steps to carry out model training and optimization:

step S4.1: data set partitioning

The model training firstly divides the data set into three parts of a training set, a verification set and a test set. The training set is used for learning model parameters, the verification set is used for adjusting super parameters and avoiding overfitting, and the test set is used for evaluating the final model performance.

Step S4.2: and (5) preprocessing the image input. The image input preprocessing is to convert the image data into a format acceptable to the model. And carrying out normalization processing on the images in the training set, the verification set and the test set to ensure that the input images have the same size and pixel range so as to eliminate the difference between the images.

Step S4.3: CNN network parameters are initialized. The CNN network parameter initialization stage adopts a pretrained convolutional neural network for initialization, such as VGG, resNet and other networks pretrained on large-scale image data. Such initialization is advantageous for improving convergence speed and generalization capability of the model.

Step S4.4: and (5) model training. The model training process is carried out in an end-to-end mode, namely the preprocessed image is input into a CNN network, then the characteristic sequence extracted by the CNN is input into a BiLSTM network, and finally the output of the BiLSTM is connected into a full-connection layer for defect classification. And measuring the error between the model predicted value and the real label by adopting a cross entropy loss function, updating the model parameters by using a back propagation algorithm, and optimizing the performance of the model.

Step S4.5: and (5) super-parameter tuning. The super-parameter tuning is used for finding the optimal configuration of the CNN-BiLSTM model. The super parameters of CNN and BiLSTM, such as learning rate, convolution kernel size, hidden unit number of BiLSTM, etc. are adjusted by the techniques of grid search, random search or Bayesian optimization, etc. to improve the generalization capability and performance of the model.

Step S4.6: and (5) model verification. And in the model verification stage, a verification set is used for evaluating the CNN-BiLSTM model obtained through training. And calculating the loss value and the accuracy of the model on the verification set, and further adjusting the structure and the super parameters of the model according to the verification result until the better verification performance is achieved.

Step S4.7: model testing and evaluation. And in the model test stage, the test set is used for carrying out final evaluation on the optimized CNN-BiLSTM model. And calculating performance indexes such as accuracy and recall rate of the model on the test set, and evaluating the actual effect of the model in actual transmission line inspection.

Step S4.8: model preservation and deployment. The model preservation and deployment stage preserves the optimized CNN-BiLSTM model as a deployable format, such as HDF5 or ONNX. The model can be loaded and operated in real time in an unmanned aerial vehicle inspection system, and defect detection and identification can be carried out on aerial images of the power transmission line.

The image defect detection step involves applying a trained CNN-BiLSTM model to new unmanned aerial vehicle aerial image data to automatically identify defects of the power transmission equipment and return location and name information. The specific image defect detection steps are as follows:

step S5.1: image input and preprocessing. Before image defect detection, firstly inputting an image acquired by aerial photography of an unmanned aerial vehicle into a trained CNN-BiLSTM model. Preprocessing operations, including image denoising, size unification, etc., are performed on the input image to ensure the quality and uniformity of the image data, making it suitable for processing of the model.

Step S5.2: and extracting CNN characteristics. And performing feature extraction on the preprocessed image by using CNN. The CNN has been trained in previous steps with a large amount of image data, which automatically learns the local and global features of the image. Through operations such as rolling and pooling, the CNN can capture important features such as texture, shape, and structure in the image.

Step S5.3: and (5) serializing the images. After obtaining the image features extracted by the CNN, these features are subjected to a serialization process. By image serialization, it is meant that the image features are organized into sequences in an order that allows the BiLSTM to time-sequence model these sequences. For example, for a plurality of towers and insulators in a transmission line, their features are serialized into a whole and input into the BiLSTM.

Step S5.4: biLSTM timing modeling and defect detection. After image serialization, the serialized image features are input into a BiLSTM model for time sequence modeling. The BiLSTM model can effectively capture the time sequence relationship between components, thereby detecting defects in the image sequence. Through learning the dependency relationship among a plurality of input images, biLSTM can discern common defect conditions such as insulator damage, bolt lose, bolt rust, bird nest, foreign matter winding.

Step S5.5: and outputting defect information. After the processing of the BiLSTM model and the image defect detection stage, the system outputs a defect detection result. The output result includes location information of the defect and defect name information. The location information may be represented by coordinates or pixel locations in the image to determine the location of the defect on a particular power transmission device. The defect name information indicates the specific type of defect detected, such as broken insulator, lost pin, etc. The output information will help the power sector determine which devices are defective and thus are targeted for repair and treatment.

The invention has the beneficial effects that: the power transmission line inspection method based on the CNN-BiLSTM model has obvious beneficial effects. By introducing a Convolutional Neural Network (CNN) and a two-way long-short-term memory network (BiLSTM), the method can automatically identify defects and abnormal conditions of power transmission equipment when processing unmanned aerial vehicle aerial image data. Compared with the traditional manual inspection and image analysis method, the method provided by the invention realizes the automation and high efficiency of line inspection, effectively reduces the workload of manual processing, and improves the inspection efficiency and accuracy. Meanwhile, by comprehensively utilizing the image feature extraction of CNN and the sequence modeling of BiLSTM, the method can accurately identify defects of different types, such as insulator damage, bolt loss, bolt corrosion and the like, provides a reliable defect detection means for a power transmission department, and powerfully improves the safety and stability of a power transmission line. The implementation of the invention can also save labor cost for the power transmission department, reduce the influence of subjective factors on the inspection result, and bring great value and help to the power transmission industry.

Drawings

FIG. 1 is a flow chart of the main steps of the present invention;

FIG. 2 is a diagram of the steps of data acquisition and preprocessing according to the present invention;

FIG. 3 is a flow chart of the feature extraction and serialization steps of the present invention;

FIG. 4 is a flowchart showing the steps for establishing a CNN-BiLSTM model according to the present invention;

FIG. 5 is a flow chart of the model training and optimization steps of the present invention;

FIG. 6 is a flowchart illustrating the steps of detecting an image defect according to the present invention;

FIG. 7 is a diagram of a CNN-BiLSTM network architecture of the present invention;

fig. 8 is a schematic diagram of a detection effect in the embodiment of the invention, wherein the left detection frame is internally provided with a damper for displacement, and the right detection frame is internally provided with a nut for tooth-out.

Detailed Description

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. All other embodiments, which can be made by those skilled in the art without making any inventive effort, based on the embodiments of the present invention are within the scope of the present invention, and are specifically described below in connection with the embodiments.

The specific embodiment of the invention is described through the following specific steps and methods to realize the transmission line inspection method based on the unmanned aerial vehicle and the CNN-BiLSTM model. The steps of this embodiment are as follows:

step S1: and (5) data acquisition and preprocessing.

S1.1: and (5) data collection. Firstly, collecting a large number of unmanned aerial vehicle aerial image data under different power transmission lines and different weather conditions. These data include transmission line images in a normal state and transmission line images in various defect cases, such as broken insulators, lost pins, rusted bolts, bird nests, foreign object entanglement, and the like. The data acquisition can be carried out on site through unmanned aerial vehicles or other applicable aerial photographing equipment, so that the authenticity and diversity of the data are ensured.

S1.2: and (5) marking data. After the image data are obtained, accurate labeling of the data is required to be used as a basis for model training. In the labeling process, the defect type and position information existing in each image should be clarified. For each defect, it is necessary to specify its coordinates or area in the image, as well as the name of the defect. The data labeling is a key step of training the CNN-BiLSTM model, and the accuracy of the data labeling directly influences the detection effect of the model.

S1.3: and (5) preprocessing data. Preprocessing of the image data is necessary before model training. The data preprocessing aims at enabling the image data to reach uniform format and quality, and ensuring that the model can be trained under uniform conditions. The pretreatment comprises the following aspects:

S1.4: data set partitioning. After the data preprocessing is completed, the data set should be divided into a training set, a validation set and a test set for training, tuning and evaluation of the CNN-BiLSTM model. Typically, most of the data is used for the training set, some for the validation set and a small amount for the test set. The aim of dividing the data set is to verify the generalization capability of the model on unseen data and to adjust the hyper-parameters of the model to improve the performance of the model.

Step S2: feature extraction and serialization.

S2.1: convolutional Neural Network (CNN) feature extraction. The image data is feature extracted by a pre-trained Convolutional Neural Network (CNN). Specifically, the CNN model adopts a classical convolution layer, a pooling layer and an activation function, and extracts local features and global features of an image through a series of convolution operations. The weights and parameters of the CNN model are optimized through large-scale image data in a pre-training stage, so that the CNN model has effective feature extraction capability on images.

Step S3: establishing a CNN-BiLSTM model

Step S3.1: convolutional neural network construction

Before establishing the CNN-BiLSTM model, a Convolutional Neural Network (CNN) part is first established. The adopted CNN model comprises a plurality of convolution layers and pooling layers so as to realize the feature extraction of the image. Specifically, the output C1 of the first convolution layer is obtained from the input aerial image X by the following calculation:

C1＝f(W ₁ *X+b ₁ )

wherein W is ₁ Weight parameters representing convolution kernels, b ₁ For bias terms, the convolution operation is represented, f is an activation function (e.g., a ReLU function), and the output of C1 is a feature map, which contains the local features of the image.

Step S3.2: pooling layer processing

To reduce the number of model parameters and computational complexity, a pooling layer is introduced to downsample the feature map. The usual pooling operation is maximum pooling or average pooling. Let the output after the pooling layer processing be P1, its calculation formula is as follows:

P1＝pooling(C1)

wherein, pooling represents pooling operation, and P1 is a feature map after downsampling, which contains main feature information in the feature map.

Step S3.3: multi-layer rolling and pooling

To enhance the feature extraction capability of CNNs, a multi-layer convolution and pooling structure may be employed. Parameters such as the convolution kernel size, step size, activation function and the like of each layer can be set according to specific tasks. By multi-layer convolution and pooling, more rich image features can be obtained.

Step S3.4: serializing image features

In order to facilitate time sequence modeling of the BiLSTM model, serialization processing is carried out on the feature map output by the CNN part. And flattening the feature map obtained by the multi-layer convolution and pooling into a one-dimensional vector to obtain a serialized image feature sequence. Thus, each serialized feature vector represents both local and global information in the image.

Step S3.5: bi-directional long-short term memory network (BiLSTM) construction

To time-sequence model the serialized image feature sequence, a two-way long-short-term memory network (BiLSTM) model is introduced. The BiLSTM is a cyclic neural network capable of capturing sequence information and is characterized by having a forward direction and a backward direction (reverse direction) in time sequence, and processing sequence data respectively. Let the serialized image feature sequence be S, output of BiLSTM be H, and the calculation process of forward and backward LSTM is as follows:

forward LSTM calculation:

backward LSTM calculation:

wherein,input sequence representing forward direction,/->Representing the reverse directionInput sequence of->For the output of the forward LSTM +.>Is the output of the reverse LSTM.

Step S3.6: fusing forward and backward outputs

To fuse the outputs of the forward and backward LSTM, the outputs of the two directions are connected to obtain the final output H, as follows:

Step S4: model training and optimization

Step S4.1 data set partitioning

In the model training process, the data set is first divided into three parts, namely a training set, a verification set and a test set. The training set is used for learning model parameters, the verification set is used for adjusting super parameters and avoiding overfitting, and the test set is used for evaluating the final model performance.

Step S4.2 image input pretreatment

Preprocessing operations are performed on the images in the training set, validation set, and test set to convert the image data into a format acceptable to the model. This includes image denoising, size unification, etc. to ensure that the input images have the same size and pixel range, thereby eliminating the variability between images.

Step S4.3 CNN network parameter initialization

Initialization is performed using a pretrained convolutional neural network, such as a VGG, resNet, or the like, pretrained on large-scale image data. Such initialization is advantageous for improving convergence speed and generalization capability of the model.

Step S4.4 model training

The model training process is performed in an end-to-end manner. Firstly, inputting the preprocessed image into a CNN network, inputting the feature sequence extracted by the CNN into a BiLSTM network, and finally, accessing the output of the BiLSTM into a full-connection layer for defect classification. And measuring the error between the model predicted value and the real label by adopting a cross entropy loss function, updating the model parameters by using a back propagation algorithm, and optimizing the performance of the model.

Step S4.5 super parameter tuning

In order to find the optimal configuration of the CNN-BiLSTM model, super-parameter tuning is performed. The super parameters of CNN and BiLSTM, such as learning rate, convolution kernel size, hidden unit number of BiLSTM, etc. are adjusted by the techniques of grid search, random search or Bayesian optimization, etc. to improve the generalization capability and performance of the model.

Step S4.6 model verification

The trained CNN-BiLSTM model is evaluated using a validation set. And calculating the loss value and the accuracy of the model on the verification set, and further adjusting the structure and the super parameters of the model according to the verification result until the better verification performance is achieved.

Step S4.7 model test and evaluation

And finally evaluating the optimized CNN-BiLSTM model by using a test set. And calculating performance indexes such as accuracy and recall rate of the model on the test set, and evaluating the actual effect of the model in actual transmission line inspection.

Step S4.8 model preservation and deployment

And storing the optimized CNN-BiLSTM model into a deployable format, such as HDF5 or ONNX. The model can be loaded and operated in real time in an unmanned aerial vehicle inspection system, and defect detection and identification can be carried out on aerial images of the power transmission line.

Step S5: image defect detection

Step S5.1: image input and preprocessing

Inputting the new unmanned aerial vehicle aerial image into a CNN-BiLSTM model which is trained and optimized for defect detection. Preprocessing the input image, including image denoising, size unification and other operations, so as to ensure that the quality and format of the image data are consistent with those of training data, and facilitate subsequent model processing.

Step S5.2: CNN feature extraction

And extracting the characteristics of the preprocessed image by utilizing the pretrained CNN model. The CNN model adopts a structure with a plurality of convolution layers and pooling layers, captures local features of an image through convolution operation, reduces data dimension through pooling operation, and obtains a feature map. The parameters of the CNN model have been optimized by a large amount of image data in the previous model training phase.

Step S5.3: image serialization

And carrying out serialization treatment on the feature map extracted by the CNN, and organizing the feature map into a sequence form according to a certain sequence so as to facilitate the time sequence modeling of BiLSTM. For example, for a plurality of towers and insulators in a transmission line, their features are serialized into a whole and input into the BiLSTM. The serialized image features are one-dimensional vectors representing local and global information in the image.

Step S5.4: biLSTM timing modeling and defect detection

After image serialization, the serialized image features are input into a BiLSTM model for time sequence modeling. BiLSTM is a recurrent neural network with processing power in both the forward and backward directions. The BiLSTM can effectively capture the time sequence relation in the image data and detect the defects in the image sequence. Through learning the dependency relationship among a plurality of input images, biLSTM can discern common defect conditions such as insulator damage, bolt loss, bolt corrosion.

Step S5.5: defect information output

After the processing of the BiLSTM model and the image defect detection stage, the system outputs a defect detection result. The output result includes location information of the defect and defect name information. The location information may be represented by coordinates or pixel locations in the image to determine the location of the defect on a particular power transmission device. The defect name information indicates the specific type of defect detected, such as broken insulator, lost pin, etc.

Step S5.6: defect result processing and feedback

And according to the output defect detection result, the defect information is corresponding to the image and marked on the original image, so that transmission inspection personnel can intuitively know the detection result. And the defect detection result is fed back to the power transmission inspection system, so that data recording and further analysis are facilitated, and support is provided for the power transmission department to make maintenance plans and decisions.

Step S5.7: defect detection Performance assessment

And performing performance evaluation on the output defect detection result, and calculating indexes such as accuracy, recall rate and the like of the model on the test set so as to verify the performance of the model. The CNN-BiLSTM model is continuously optimized and adjusted to improve the accuracy and stability of defect detection.

The validity of the present invention is verified as follows:

in order to verify the effectiveness of the CNN-BiLSTM model in transmission line inspection, various common defect types are selected for marking and training. These defect types include, but are not limited to, insulator breakage, pin loss, bolt rust, bird nest, foreign object entanglement, and the like. And carrying out accurate defect labeling on the collected aerial image data by professional transmission line inspection personnel. In the labeling process, labeling personnel label a defect area by using a rectangular frame according to the actual image content aiming at each image, and designate the defect type. Meanwhile, for the same defect type, samples with different positions and sizes are adopted, so that the model can be well generalized and detect defects under different conditions.

In order to verify the performance of the CNN-BiLSTM model in transmission line inspection, the following method is adopted for performance detection: the collected plurality of aerial image data is divided into a training set, a validation set, and a test set. The training set is used for model parameter learning, the verification set is used for adjusting super parameters and avoiding overfitting, and the test set is used for final performance evaluation. In the model training process, a cross entropy loss function is adopted to measure the error between the model predicted value and the real label. This may help to optimize the model to better fit the actual transmission line inspection data. And the performance of the model is evaluated by adopting common indexes such as accuracy, recall rate, F1-score and the like. The accuracy rate measures the accuracy rate of model prediction, the recall rate measures the detection capability of the model to real defects, the F1-score comprehensively considers the accuracy rate and the recall rate, and the comprehensive performance of the model is evaluated.

In the performance detection stage, super parameters of the CNN-BiLSTM model, such as learning rate, convolution kernel size, hidden unit number of BiLSTM and the like, are repeatedly adjusted to achieve better performance. Through the technology of grid search, random search or Bayesian optimization, the optimal hyper-parameter configuration is found, so that the model has higher accuracy and recall rate, and can show good performance on a test set.

In order to verify the feasibility and the practicability of the CNN-BiLSTM model in an actual power transmission line inspection scene, unmanned aerial vehicles are used for aerial photography on site, and the acquired images are input into the optimized model for defect detection. In the actual test, the detection result of the model is compared and verified with the actual inspection result of the professional inspection personnel, so that the accuracy and stability of the model in an actual scene are ensured.

It will be evident to those skilled in the art that the invention is not limited to the details of the foregoing illustrative embodiments, and that the present invention may be embodied in other specific forms without departing from the spirit or essential characteristics thereof. The present embodiments are, therefore, to be considered in all respects as illustrative and not restrictive, the scope of the invention being indicated by the appended claims rather than by the foregoing description, and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein. Any reference sign in a claim should not be construed as limiting the claim concerned.

Claims

1. The power transmission line inspection method based on the unmanned plane and the CNN-BiLSTM model is characterized by comprising the following steps of:

step S1: data acquisition and pretreatment; collecting a large number of unmanned aerial vehicle aerial image data under different power transmission lines and different weather conditions, wherein the unmanned aerial vehicle aerial image data comprise a normal state and various defect conditions; preprocessing image data, including image denoising, uniform size, etc., to prepare the data for subsequent model training;

step S2: feature extraction and serialization; performing feature extraction on the preprocessed image by adopting a Convolutional Neural Network (CNN); the CNN can capture local and global characteristics of the image and provide powerful support for subsequent sequence modeling; serializing the extracted image features so as to perform time sequence modeling on the BiLSTM model;

step S3: establishing a CNN-BiLSTM model; building a CNN-BiLSTM hybrid model, and inputting the image features extracted by CNN into BiLSTM for sequence modeling; the BiLSTM can effectively capture time sequence information among components, and has advantages for identifying complex defect conditions;

step S4: model training and optimizing; training the CNN-BiLSTM model by using the training data prepared in the step 1, and performing super-parameter tuning to obtain the optimal performance; the model training aims to enable the model training to accurately identify defects of different types of power transmission equipment;

Step S5: detecting image defects; performing defect detection on a new unmanned aerial vehicle aerial image by using a trained CNN-BiLSTM model; the model can automatically identify common defects such as insulator breakage, bolt loss, bolt rust, bird nest, foreign matter winding and the like, and return defect position and name information.

2. The method for inspecting the transmission line based on the unmanned aerial vehicle and the CNN-BiLSTM model as claimed in claim 1, wherein the data acquisition and preprocessing sub-steps S1.1 to S1.4 of the step S1 comprise the following steps:

step S1.1: collecting data; collecting unmanned aerial vehicle aerial image data under different power transmission lines and different weather conditions, including power transmission line images in a normal state and power transmission line images under various defect conditions, such as insulator breakage, bolt loss, bolt rust, bird nest, foreign object winding and the like; the data acquisition is carried out on site through an unmanned plane or other applicable aerial photographing equipment, so that the authenticity and diversity of the data are ensured;

step S1.2: marking data; after the image data are obtained, the data are accurately marked, and the defect type and position information existing in each image are clear in the marking process; for each defect, it is necessary to specify its coordinates or area in the image, and the name of the defect; the data annotation ensures the basis of model training, and the accuracy of the data annotation directly influences the detection effect of the model;

Step S1.3: preprocessing data; preprocessing operations such as image denoising and uniform size are carried out, so that image data reach uniform format and quality, and training of a model under uniform conditions is ensured;

step S1.4: dividing a data set; the data set is divided into a training set, a verification set and a test set and is used for training, optimizing and evaluating the CNN-BiLSTM model so as to verify the generalization capability of the model on unseen data.

3. The method for inspecting transmission lines based on the unmanned aerial vehicle and the CNN-BiLSTM model according to claim 1, wherein the feature extraction and serialization substeps S2.1 to S2.4 of step S2 include the following:

step S2.1: convolutional Neural Network (CNN) feature extraction; the image data is subjected to feature extraction through a pre-trained CNN, and local features and global features of the image are extracted through operations such as a convolution layer, a pooling layer and an activation function;

step S2.2: serializing the image features; the image features extracted from the CNN are serialized into one-dimensional vectors, and the image features are converted into one-dimensional vectors from a multi-dimensional matrix through flattening operation, so that the subsequent BiLSTM sequence modeling processing is facilitated;

step S2.3: dividing time steps; dividing the image features of the one-dimensional vector into equal-length time steps, wherein the length of the time steps is determined by the requirement of inputting a BiLSTM model so that the BiLSTM can model according to a sequence order, and capturing time sequence information in image data;

Step S2.4: serializing an image feature matrix; reorganizing the image features divided by time steps into a serialized feature matrix, wherein the number of rows of the feature matrix corresponds to the number of time steps, and the number of columns is equal to the dimension of the image features in each time step so as to provide the BiLSTM model for time sequence modeling.

4. The transmission line inspection method based on the unmanned aerial vehicle and the CNN-BiLSTM model as claimed in claim 1, comprising the following steps: the steps for establishing the CNN-BiLSTM model are as follows:

step S3.1: building a Convolutional Neural Network (CNN) to realize image feature extraction; the CNN network adopts a structure with a plurality of convolution layers and pooling layers; specifically, the convolution layer is used for extracting local features of the image, and the pooling layer is used for reducing the number of parameters and reducing the computational complexity of the model; let the input aerial image be X, the output of the first layer convolution layer be C1, and the calculation formula of C1 is as follows:

C1＝f(W ₁ *X+b ₁ )

wherein W is ₁ Weight parameters representing convolution kernels, b ₁ As bias term, represent convolution operation, f is activation function (such as ReLU function), and the output of C1 is characteristic diagram;

step S3.2: carrying out pooling layer treatment; in the CNN, the pooling layer is used for downsampling the feature map so as to reduce the data dimension; the common pooling operation has the maximum pooling or average pooling; assuming that the output after the pooling layer processing is P1, the calculation formula of P1 is as follows:

P1＝pooling(C1)

Wherein, pooling represents pooling operation, P1 is a feature map after downsampling, and the feature map contains main feature information in the feature map;

step S3.3: multi-layer rolling and pooling; to enhance the feature extraction capability of CNN, a multi-layer convolution and pooling structure may be employed; parameters such as the convolution kernel size, the step length, the activation function and the like of each layer can be set according to specific tasks; after multi-layer rolling and pooling, richer image features can be obtained;

step S3.4: serializing the image features; in order to facilitate time sequence modeling of the BiLSTM model, the feature map output by the CNN part needs to be subjected to serialization processing; flattening the feature map obtained by multi-layer convolution and pooling into a one-dimensional vector to obtain a serialized image feature sequence; thus, each serialized feature vector represents both local and global information in the image;

step S3.5: building a two-way long-short-term memory network (BiLSTM); in order to perform time sequence modeling on the serialized image feature sequence, the BiLSTM model is introduced; biLSTM is a recurrent neural network capable of capturing sequence information; the model has two directions of forward (forward) and backward (reverse) in time sequence, and processes sequence data respectively; let the serialized image feature sequence be S, output of BiLSTM be H, and the calculation process of forward and backward LSTM is as follows:

Forward LSTM calculation:

backward LSTM calculation:

wherein,input sequence representing forward direction,/->Representing the reverse input sequence,/->For the output of the forward LSTM +.>Is the output of the reverse LSTM;

s3.6: fusing the forward and backward outputs; in order to fuse the outputs of the forward and backward LSTM, the BiLSTM model connects the outputs of the two directions to obtain the final output H, as follows:

5. The transmission line inspection method based on the unmanned aerial vehicle and the CNN-BiLSTM model as claimed in claim 1, which is characterized by comprising the following steps:

step S4.1: a data set dividing step of dividing the image data set into a training set, a verification set and a test set;

step S4.2: an image input preprocessing step, namely carrying out normalization processing on images in a training set, a verification set and a test set to ensure that the input images have the same size and pixel range;

step S4.3: initializing CNN network parameters by using a pretrained convolutional neural network;

step S4.4: model training, namely inputting the preprocessed image into a CNN network, inputting a feature sequence extracted by the CNN into a BiLSTM network, and finally accessing the output of the BiLSTM into a full-connection layer for defect classification;

Step S4.5: super-parameter tuning, namely, adjusting the super-parameters of CNN and BiLSTM through grid searching, random searching or Bayesian optimization and other technologies to improve the generalization capability and performance of the model;

step S4.6: a model verification step, namely evaluating the CNN-BiLSTM model obtained through training by using a verification set, and further adjusting the structure and super parameters of the model;

step S4.7: the model testing and evaluating step, namely, finally evaluating the optimized CNN-BiLSTM model by using a testing set, and calculating performance indexes such as accuracy, recall rate and the like of the model on the testing set;

step S4.8: and a model storage and deployment step, wherein the optimized CNN-BiLSTM model is stored into a deployable format and is used for loading and running in real time in an unmanned aerial vehicle inspection system, and defect detection and identification are carried out on aerial images of the power transmission line.

6. The method for inspecting transmission lines based on the unmanned aerial vehicle and the CNN-BiLSTM model according to claim 1, wherein in the image processing method, the image defect detecting step comprises the following sub-steps:

step S5.1: inputting images acquired by aerial photography of the unmanned aerial vehicle into a trained CNN-BiLSTM model, and performing preprocessing operations such as image denoising and uniform size;

Step S5.2: a CNN feature extraction step, namely performing feature extraction on the preprocessed image by using CNN, and capturing important features such as textures, shapes, structures and the like in the image;

step S5.3: an image serialization step, namely carrying out serialization processing on the image features extracted by the CNN, and inputting an image feature sequence into the BiLSTM for time sequence modeling;

step S5.4: the method comprises the steps of BiLSTM time sequence modeling and defect detection, wherein after image serialization, the serialized image features are input into a BiLSTM model for time sequence modeling, and common defect conditions such as insulator breakage, bolt loss, bolt rust, bird nest, foreign object winding and the like in a power transmission line are identified;

step S5.5: and outputting defect information, namely outputting a defect detection result after the image defect detection stage is completed, wherein the defect detection result comprises position information and defect name information of the defect, and helping a power transmission department to determine the defect of equipment and maintain and process the defect.

7. The power transmission line inspection method based on the unmanned aerial vehicle and the CNN-BiLSTM model according to claim 1, wherein in the image processing method, the position information in the image defect detection step S5.5 represents the position of the defect on specific power transmission equipment, and the position information is represented by coordinates or pixel positions in an image; the defect name information indicates the specific type of defect detected, such as insulator breakage, pin loss.

8. The power transmission line inspection method based on the unmanned aerial vehicle and the CNN-BiLSTM model as claimed in claim 1, wherein in the image processing method, error between a model predicted value and a real label is measured by adopting a cross entropy loss function, model parameters are updated by using a back propagation algorithm, and the performance of the model is optimized.

9. The method for inspecting transmission lines based on unmanned aerial vehicle and CNN-BiLSTM model as claimed in claim 1, wherein in the image processing method, CNN network is initialized by pretrained convolutional neural network, such as VGG, resNet and other networks, so as to improve convergence rate and generalization capability of the model.

10. The method for inspecting transmission lines based on unmanned aerial vehicle and CNN-BiLSTM model as claimed in claim 1, wherein in the image processing method, super-parameter tuning adopts grid search, random search or Bayesian optimization technology, and super-parameters of CNN and BiLSTM, such as learning rate, convolution kernel size, hidden units of BiLSTM, etc., are adjusted to improve generalization capability and performance of the model.