CN110826588A - Drainage pipeline defect detection method based on attention mechanism - Google Patents

Abstract

The invention relates to a drainage pipeline defect detection method based on an attention mechanism, comprising: extracting defect pictures from a pipeline defect detection report, and classifying them according to their corresponding defect types, which are divided into: deformation, corrosion, scaling , wrong mouth, deposition, leakage and rupture; build the attention mechanism module, the attention mechanism module used is the CBAM module; build the neural network model based on the attention mechanism, the convolutional network removes the last 3 of the VGG16 network. A convolutional layer is added, and several CBAM modules are added on the basis; the neural network is trained using the backpropagation algorithm, and the verification is performed during the training process. When the verification accuracy is the highest, the optimal model for network training is saved; The optimal model is tested on the test set, and the classification results of pipeline defect pictures are obtained.

Description

An attention mechanism-based detection method for drainage pipe defects

technical field

The aspects involved in the present invention include computer fields such as computer vision, computer image processing and deep learning, and the field of abnormal detection of drainage pipes. The present invention is more focused on the application of deep learning technology to the detection of drainage pipe defects.

Background technique

The sewer system is one of the city's largest infrastructures, designed to collect and transport wastewater and stormwater. The normal use of this system is very important for the drainage safety of the city. With the rapid development of cities in recent years, my country has highlighted the problems of insufficient underground pipeline construction scale and low management level. Incidents such as torrential rain and road collapse occurred one after another in some cities, which seriously affected people's lives and the order of urban operations. Therefore, regular inspection and repair of drainage pipes is an indispensable measure in urban construction.

Pipeline CCTV (Closed Circuit Television Inspection, CCTV) is by far the most widely used technology in pipeline inspection. The system consists of a pipeline inspection robot and a CCTV camera installed on the robot. The device obtains video data containing the internal information of the pipeline under the remote operation of the staff or the automatic control of the computer [1]. Inspectors perform manual defect identification based on the captured video and then write a pipeline defect detection report. This method often relies too much on the experience of the inspector, has a lot of subjectivity, and consumes a lot of time and energy.

In recent years, deep learning, especially Convolutional Neural Network (CNN), has made great progress in the field of computer vision. Subsequently, pipeline defect detection methods based on convolutional neural networks have emerged. For example, Kumar et al. proposed in 2018 to use a simple classification convolutional neural network to classify three defects of root invasion, deposition and cracks [2]; Li et al. used a deep convolutional neural network with hierarchical classification to detect and classify sewer defect types from unbalanced CCTV inspection data detection [3]. However, none of these methods take advantage of the unique characteristics of pipeline data, and the effect is not ideal.

references

[1] Costello, S., Chapman, D., Rogers, C., Metje, N.: Underground assetlocation and condition assessment technologies. Tunnelling and UndergroundSpace Technology 22(5-6), 524-542 (2007).

[2] Kumar, Srinath S., et al. "Automated defect classification in sewerclosed circuit television inspections using deep convolutional neural networks." Automation in Construction 91 (2018): 273-283.

[3] Li, Duanshun, Anran Cong, and Shuai Guo. "Sewer damage detection fromimbalanced CCTV inspection data using deep convolutional neural networks withhierarchica1 classification."Automation in Construction 101(2019): 199-208.

SUMMARY OF THE INVENTION

The present invention uses the VGG-16 convolutional neural network-based structure as the main model, proposes an automatic detection method for pipeline defects based on attention mechanism to solve the above problems, and uses a large number of existing marked pipeline abnormal samples for training and testing, The invention can quickly and accurately determine the abnormal type of the drainage pipeline. The technical scheme of the present invention is as follows:

An attention-based defect detection method for drainage pipes, the general steps are described as follows:

Step 1: Extract the defect pictures from the pipeline defect inspection report, and classify them according to their corresponding defect types, which are divided into seven categories: deformation, corrosion, scaling, misalignment, deposition, leakage and rupture.

Step 2: Divide the dataset into training set, test set and validation set.

Step 3: Build an attention mechanism module. The attention mechanism module used is the CBAM module (Convolutional Block Attention Module). This module can be directly added to the convolutional neural network model and used as a network layer. The input of this module is the output of the convolutional layer O∈R ^C×H×W , where C is the number of channels of the Feature map, H and W are the height and width of the Feature map, respectively. O passes through the channel attention module A _C ∈ R ^C×1×1 and the spatial attention module A _S ∈ R ^1×H×W finally get the output O ^a ∈ R ^C×H×W .

表示逐元素乘法，O¹表示中间结果。经过CBAM模块获得的O^a能够直接再送入卷积神经网络的其它部分。in

means element-wise multiplication, O ¹ means intermediate result. The O ^a obtained by the CBAM module can be directly fed into other parts of the convolutional neural network.

Step 4: Build a neural network model based on the attention mechanism. The convolutional network removes the last three convolutional layers of the VGG16 network, and adds several CBAM modules on the basis. The overall structure of the convolutional neural network is:

The first convolution layer, the convolution kernel size is 3*3, a total of 64 filters, stride=1.

The first attention module, the CBAM module, has a scaling factor r=8.

The second convolution layer, the convolution kernel size is 3*3, a total of 64 filters, stride=1.

The first maximum pooling layer, the pooling size is 2*2, and the stride=2.

The third convolutional layer, the convolution kernel size is 3*3, a total of 128 filters, stride=1.

The second attention module, the CBAM module, has a scaling factor r=8.

The fourth convolutional layer, the convolution kernel size is 3*3, a total of 128 filters, stride=1.

The second maximum pooling layer, the pooling size is 2*2, and the stride=2.

The fifth convolutional layer, the convolution kernel size is 3*3, a total of 256 filters, stride=1.

The third attention module, the CBAM module, has a scaling factor r=8.

The sixth convolutional layer, the convolution kernel size is 3*3, a total of 256 filters, stride=1.

The fourth attention module, the CBAM module, has a scaling factor r=8.

The seventh convolutional layer, the convolution kernel size is 3*3, a total of 256 filters, stride=1.

The third maximum pooling layer, the pooling size is 2*2, and the stride=2.

The eighth convolutional layer, the convolution kernel size is 3*3, a total of 512 filters, stride=1.

The fifth attention module, the CBAM module, has a scaling factor r=8.

The ninth convolutional layer, the convolution kernel size is 3*3, a total of 512 filters, stride=1.

The sixth attention module, the CBAM module, has a scaling factor r=8.

The tenth convolution layer, the convolution kernel size is 3*3, a total of 512 filters, stride=1.

The fourth maximum pooling layer, the pooling size is 2*2, and the stride=2.

The first fully connected layer has 4096 nodes.

The second fully connected layer has 4096 nodes.

The Softmax layer finally outputs the probability that the image corresponds to 7 abnormal categories.

Step 5: Use the back-propagation algorithm to train the neural network, verify during the training process, and save the optimal model for network training when the verification accuracy is the highest.

Step 6: Use the saved optimal model to test the test set to obtain the classification result of the pipeline defect picture.

Description of drawings

Fig. 1 is the flow chart of the present invention

Fig. 2 is the network structure diagram of the present invention

Figure 3 is the structure diagram of CBAM

Detailed ways

In order to make the objectives, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below with reference to the accompanying drawings and examples. Obviously, the described implementations are only some of the embodiments of the present invention and are not exhaustive of all embodiments. Also, the features of the implementations and examples in this description may be combined with each other without conflict.

The processing steps of the present invention include: data preparation and processing, establishment of training set validation set test set, training convolutional neural network, using the trained network model for automatic defect classification and other major steps.

Step 1: Data preparation and processing. The python program is used to extract the pictures of the internal defects of the pipeline from the obtained drainage pipe defect detection report, and the pictures are classified according to the defect type according to the report content. Select seven types of pictures with better picture quality, such as deformation, corrosion, scaling, misalignment, deposition, leakage and cracking.

Step 2: Divide the selected seven types of pictures of defect types such as deformation, corrosion, scaling, misalignment, deposition, and leakage into data sets, and divide them into training sets, validation sets and test sets, with a division ratio of 3:1 : 1.

Step 3: Build the attention mechanism module, and the attention mechanism structure adopted is the CBAM module. The input of this module is the output of the convolutional layer O∈R ^C×H×W , where C is the number of channels of the feature map, and H and W are the height and width of the feature map, respectively. The attention mechanism module can be divided into two parts, the first part is the channel attention module, and the second part is the spatial attention module.

The first part passes the input O through the channel attention module A _C (O)∈R ^C×1×1 , and obtains the intermediate output O ¹ ∈ R ^C×H×W .

This part can be divided into the following 3 steps:

(1) First, perform global average pooling GloAvgPool and global maximum pooling GloMaxPool respectively, and then pass the results through two fully connected layers Dense ₁ and Dense ₂ to obtain two outputs of the same dimension d ₁ ∈ R ^{C×1 ×1} and d ₂ ∈ R ^C×1×1 , namely:

d ₁ = Dense ₂ (Dense ₁ (GloAvgPool(O)))

d ₂ = Dense ₂ (Dense ₁ (GloMaxPool(O)))

r表示缩放系数，在本实验中r＝8。Dense₂的结点个数为C。The number of nodes in Dense ₁ is

r represents the scaling factor, in this experiment r=8. The number of nodes in Dense ₂ is C.

(2) Add d ₁ and d ₂ , and send the result of the addition to the sigmoid function to obtain the channel attention weight A _C (O)∈R ^C×1×1 . Calculated as follows:

A _C (O)=Sigmoid(d ₁ +d ₂ )

表示逐元素乘法，则计算公式如下：(3) Perform element-wise multiplication of the input O and the channel attention weight A _C (O)∈R ^C×1×1 , and finally obtain the output O ¹ ∈ R ^C×H×W of the first part.

Represents element-wise multiplication, and the calculation formula is as follows:

The second part passes the obtained intermediate result O ¹ through the spatial attention module A _S (O ¹ )∈R ^1×H×W and finally obtains the output O ^a ∈R ^C×H×W . This part can be divided into the following 4 steps:

(1) Perform average pooling ChanAvgPool and max pooling ChanMaxPool on the input channel dimension, and then concatenate the obtained results. Calculated as follows:

d _c = [ChanAvgPool(O ¹ ); ChanMaxPool(O ¹ )]

(2) The obtained result _{dc passes} through a convolutional layer, which has a filter and the filter size is 7×7, and then sends the output of the convolutional layer to the sigmoid function to obtain the spatial attention weight A _S (O ¹ )∈R ^1×H×W .

A _S (O ¹ )=Sigmoid(Con2D _7×7 (d _c ))

(3) Perform element-wise multiplication of the intermediate result O ¹ and the spatial attention weight A _S (O ¹ )∈R ^1×H×W , and finally obtain the output O ^a ∈R ^C×H×W . Calculated as follows:

Step 4: Build a convolutional neural network model based on the attention mechanism. The convolutional network removes the last three convolutional layers of the VGG16 network, and adds several CBAM modules on the basis. The overall network structure of the model is as follows:

The first convolution layer, the convolution kernel size is 3*3, a total of 64 filters, stride=1.

The first attention module, the CBAM module, has a scaling factor r=8.

The second convolution layer, the convolution kernel size is 3*3, a total of 64 filters, stride=1.

The first maximum pooling layer, the pooling size is 2*2, and the stride=2.

The third convolutional layer, the convolution kernel size is 3*3, a total of 128 filters, stride=1.

The second attention module, the CBAM module, has a scaling factor r=8.

The fourth convolutional layer, the convolution kernel size is 3*3, a total of 128 filters, stride=1.

The second maximum pooling layer, the pooling size is 2*2, and the stride=2.

The fifth convolutional layer, the convolution kernel size is 3*3, a total of 256 filters, stride=1.

The third attention module, the CBAM module, has a scaling factor r=8.

The sixth convolutional layer, the convolution kernel size is 3*3, a total of 256 filters, stride=1.

The fourth attention module, the CBAM module, has a scaling factor r=8.

The seventh convolutional layer, the convolution kernel size is 3*3, a total of 256 filters, stride=1.

The third maximum pooling layer, the pooling size is 2*2, and the stride=2.

The eighth convolutional layer, the convolution kernel size is 3*3, a total of 512 filters, stride=1.

The fifth attention module, the CBAM module, has a scaling factor r=8.

The ninth convolutional layer, the convolution kernel size is 3*3, a total of 512 filters, stride=1.

The sixth attention module, the CBAM module, has a scaling factor r=8.

The tenth convolution layer, the convolution kernel size is 3*3, a total of 512 filters, stride=1.

The fourth maximum pooling layer, the pooling size is 2*2, and the stride=2.

The first fully connected layer has 4096 nodes.

The second fully connected layer has 4096 nodes.

The Softmax layer finally outputs the probability that the image corresponds to 7 abnormal categories.

Step 5: Train the convolutional neural network model and save the optimal model weights.

Step 6: Use the saved optimal model weights for automatic classification of pipeline defect pictures.

The above network structure uses TensorFlow as the background and is built through the Keras deep learning library. The language used is Python. In this embodiment, the function model in the Keras deep learning library is used as the overall structure, and the convolutional layer, the maximum pooling layer, the fully connected layer, the Softmax layer, the Add function, the Multiply layer are constructed according to the writing method of the convolutional network layer in the library. Convolutional neural network structure and internal operations such as functions.

The pipeline abnormality type detection method provided by the embodiment of the present invention does not require the user to manually define features for classification after acquiring the image to be recognized, and directly uses the pre-trained deep learning network to determine the type of the image to be recognized. And because the attention mechanism module is added to the network, the network can gradually extract the unique features of different defect pictures during the training process, which greatly improves the network's ability to identify pipeline defect pictures, and can more accurately realize the automatic detection of pipeline defects. Detection and classification, with good application value.

Claims (3)

1. A method for detecting defects in drainage pipes based on an attention mechanism, comprising the following steps: Step 1: Extract the defect pictures from the pipeline defect inspection report, and classify them according to their corresponding defect types, which are divided into seven categories: deformation, corrosion, scaling, misalignment, deposition, leakage and rupture. Step 2: Divide the dataset into training set, test set and validation set; Step 3: Build the attention mechanism module. The attention mechanism module used is the CBAM module (Convolutional Block Attention Module). This module can be directly added to the convolutional neural network model and used as a network layer; the input of this module is the convolutional block. The output of the layer O∈R ^C×H×W , where C is the number of channels of the Feature map, H and W are the height and width of the Feature map, respectively. O passes through the channel attention module A _C ∈ R ^C×1×1 and The spatial attention module A _S ∈ R ^1×H×W finally gets the output O ^a ∈ R ^C×H×W .

in

Represents element-wise multiplication, O ¹ represents the intermediate result; O ^a obtained by the CBAM module can be directly sent to other parts of the convolutional neural network; Step 4: Build a neural network model based on the attention mechanism. The convolutional network removes the last three convolutional layers of the VGG16 network, and adds several CBAM modules on the basis. The overall structure of the convolutional neural network is: The first convolution layer, the convolution kernel size is 3*3, a total of 64 filters, stride=1; The first attention module, CBAM module, scaling factor r=8; The second convolution layer, the convolution kernel size is 3*3, a total of 64 filters, stride=1; The first maximum pooling layer, the pooling size is 2*2, and the stride=2; The third convolution layer, the convolution kernel size is 3*3, a total of 128 filters, stride=1; The second attention module, CBAM module, scaling factor r=8; The fourth convolution layer, the convolution kernel size is 3*3, a total of 128 filters, stride=1; The second maximum pooling layer, the pooling size is 2*2, and the stride=2; The fifth convolution layer, the convolution kernel size is 3*3, a total of 256 filters, stride=1; The third attention module, CBAM module, scaling factor r=8; The sixth convolution layer, the convolution kernel size is 3*3, a total of 256 filters, stride=1; The fourth attention module, CBAM module, scaling factor r=8; The seventh convolution layer, the convolution kernel size is 3*3, a total of 256 filters, stride=1; The third maximum pooling layer, the pooling size is 2*2, and the stride=2; The eighth convolution layer, the convolution kernel size is 3*3, a total of 512 filters, stride=1; The fifth attention module, CBAM module, scaling factor r=8; The ninth convolution layer, the convolution kernel size is 3*3, a total of 512 filters, stride=1; The sixth attention module, CBAM module, scaling factor r=8; The tenth convolution layer, the convolution kernel size is 3*3, a total of 512 filters, stride=1; The fourth maximum pooling layer, the pooling size is 2*2, and the stride=2; The first fully connected layer has 4096 nodes; The second fully connected layer has 4096 nodes; The Softmax layer finally outputs the probability that the image corresponds to 7 abnormal categories; Step 5: Use the back-propagation algorithm to train the neural network, verify during the training process, and save the optimal model for network training when the verification accuracy is the highest; Step 6: Use the saved optimal model to test the test set to obtain the classification result of the pipeline defect picture. 2. The method according to claim 1, characterized in that, in step 2), the data enhancement method adopted is Gaussian noise, image flipping and color dithering. 3. The method according to claim 1, wherein in step 5), network training is performed using Nvidia GPU.

Images