CN110084804A - A kind of underwater works defect inspection method based on Weakly supervised deep learning - Google Patents

Abstract

The invention discloses an underwater structure defect detection method based on weakly supervised deep learning, comprising the following steps: S1, weakly labeling an input image with semantic information as a label, training a convolutional neural network model under weak supervision, and detecting normal Classify images and defective images; S2, use the third-layer convolution layer information of the convolutional neural network model to implement a depth saliency detection algorithm; S3, according to the detection results of the depth saliency algorithm, perform an iterative clustering unified detection algorithm , train a reliable outliers classifier for underwater structure images; S4, use the iterative clustering unified detection classifier to evaluate and test the underwater structure image dataset. The invention provides an underwater structure defect detection method based on weakly supervised deep learning, which solves the problem that the underwater structure defect detection model is difficult to construct, and can better assist the inspectors to complete the defect detection task of the underwater structure target.

Description

A weakly supervised deep learning-based defect detection method for underwater structures

technical field

The invention specifically relates to an underwater structure defect detection method based on weakly supervised deep learning, and belongs to the technical field of image processing analysis and understanding.

Background technique

With the development of water conservancy and hydropower construction, more and more water conservancy facilities are used in production and life. Underwater structures such as dams and doorways are easily affected by structural stress and underwater environment, resulting in defects such as cracks and rust. At present, underwater imaging technology based on optical image is widely used in underwater detection because of its high resolution and detection accuracy. The image detection method based on deep learning can realize the automatic detection of defects. It trains according to the given sample data set, and obtains a model that can complete the expected detection effect, which is applied to image detection. However, most of the traditional deep learning methods require massive pixel annotation datasets for training. The underwater structure defect detection process is difficult to obtain massive labeled data due to the complex underwater environment and the random variety of structure defects.

SUMMARY OF THE INVENTION

The technical problem to be solved by the present invention is to overcome the defects of the prior art, and to provide a weakly supervised deep learning-based underwater underwater structure that only needs the semantic information of the image as an annotation for training, and can solve the problem of difficulty in constructing a defect detection model for underwater structures. Defect detection method for lower structures.

The invention proposes a weakly supervised deep learning-based underwater structure defect detection method, which solves the problem of constructing an auxiliary detection model in the weakly supervised training data set with only the semantic information of the image as a label, so as to realize the detection of the acquired underwater structures. Semantic detection of images.

In order to solve the above-mentioned technical problems, the technical scheme adopted in the present invention is:

A method for defect detection of underwater structures based on weakly supervised deep learning, which is characterized by comprising the following steps: S1, weakly labeling an input image with semantic information as a label, training a convolutional neural network model under weak supervision, Classification of images and defective images;

S2, using the third-layer convolutional layer information of the convolutional neural network model to implement a deep saliency detection algorithm;

S3, according to the detection result of the depth saliency algorithm, perform an iterative clustering unified detection algorithm to train a reliable classifier of abnormal points of underwater structure images;

S4, the iterative clustering unified detection classifier is used for the evaluation test on the underwater structure image dataset.

The steps of the depth saliency detection algorithm in S2 are as follows:

S21, using the feature map of the third layer t in the convolutional layer in the convolutional neural network model as the intermediate image ^Im for generating the salient point;

S22, use the maximum filter to find the local maximum value in ^Im , and save it in the list L;

S23, sort L in descending order of strength, and the strength is expressed as strength(x), where x is each element in L;

S24, perform the following operations on each element x in L: first, add x to the list γ to achieve initialization, then traverse the elements y that belong to the same ^Im in the 8 adjacent fields of each element x, if the formula strength( y)∈(strength(x)-z,strength(x)) holds, then let and will marked as candidate points, if will is added to the list γ as a valid maximum value; second, traverse the elements For each element y in the 8-adjacent domain of , if the strength of the adjacent point y is in the interval (strength(x)-z, strength(x)), it is also marked as a candidate point; continue to visit the 8-adjacency of the candidate point domain, repeat the above marking operation until it is found that the strength of all adjacent points y of a candidate point satisfies strength(y)<strength(x)-z, and the process ends, where z represents the tolerance of the gray value in this algorithm degree, determined by the range seeded around the maximum, and if the visited neighbor y already exists in the list L, it is removed from L as the weaker maximum; finally, by computing all elements in γ to generate a salient point, the geometric center is characterized by the maximum value in the wider image area.

The steps of the iterative clustering unified algorithm in S3 are as follows:

S31, let I _n and I _a represent the normal image and the defective image training set, respectively;

S32, let P _n and P _{a be} the salient point sets extracted from In and I _a by using the DSD algorithm;

S33, iteratively execute the following operations S times, where S>1: for each normal image I _n , extract the point-like cross feature PCFM of P _n , and use K-means to cluster the point-like cross feature into a normal set Cluster Q, the elements in the cluster are represented by N _q , q=1, 2,...Q; for each defective image I _a , the point-like cross feature representation of P _a is extracted, and its point-like cross feature is adopted K-means clustering is an abnormal cluster R, and the elements in the cluster are represented by A _r , r=1, 2,...R;

S34, let Q=Q·S, R=R·S;

_S35 , perform the following operations on each element A _r , _{r =} 1, 2, _. , N _q ), q=1,2,...Q, then sort the distances d _rq in ascending order; then, calculate the normalized distance d _rq12 =d _rq1 /d _rq2 , where d _rq1 and d _rq2 represent A, respectively The distance between _r and its nearest neighbor cluster midpoints N _q1 and N _q2 , that is, the distance between clusters is estimated using the Euclidean distance between cluster centroids;

S36, the average normalized distance d _q12 is estimated from all the distances d _rq12 obtained in S35;

S37, perform the following operations on each element Ar in the abnormal cluster: if the formula d _rq12 <d _q12 is established, remove the _{element Ar} from the abnormal cluster and classify it into the normal cluster.

In S4, the evaluation test detects and identifies the salient points related to the defects of underwater structures through the K-means clustering method.

The specific steps are as follows: images of underwater structures that are classified as defective set to the input image, right Do the following operations for each salient point m in: extract the point-like cross feature PCFM feature of point m; then calculate the distance from m to all clusters, that is, each point in the A _r ∪ N _q cluster; according to the K nearest neighbors of m The classes to which the points belong are voted by majority to classify detected salient points as normal or abnormal.

Defective areas of underwater structures are indicated by rectangular boxes.

Beneficial effects of the present invention: The invention provides a weakly supervised deep learning-based underwater structure defect detection method, which only needs the semantic information of the image as an annotation for training, and an ideal image detection model can be obtained. The architecture includes a unified algorithm of deep saliency detection and iterative clustering. In the weakly supervised training dataset with only the semantic information of images as labels, it solves the problem of difficulty in building a defect detection model for underwater structures, and can better assist the inspectors. Complete the task of defect detection for underwater structure targets.

Description of drawings

Fig. 1 is a training flow chart of a weakly supervised deep learning-based underwater structure defect detection method;

Figure 2. The test flow chart of the weakly supervised deep learning-based defect detection method for underwater structures.

Detailed ways

The present invention will be further described below with reference to the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solutions of the present invention, and cannot be used to limit the protection scope of the present invention.

As shown in Figure 1 and Figure 2, the present invention provides a weakly supervised deep learning-based underwater structure defect detection method, comprising the following steps:

Step 1: Weakly label the input images with semantic information as labels, and train a weakly supervised convolutional neural network model (WCNN) to classify normal and abnormal image training sets.

The second step is to use the information of the third convolution layer of the convolutional neural network model to realize the depth saliency detection algorithm, which is used to detect the salient points of the defective underwater structure image and the non-defective image. The depth saliency detection algorithm (DSD) Proceed as follows:

a, the feature map of the third layer t in the convolutional layer in the convolutional neural network model is used as an intermediate image ^Im for generating salient points;

b, use the max filter to find the local maximum in ^Im and save it to the list L;

c, sort L in descending order of strength, and the strength is expressed as strength(x), where x is each element in L;

d. Do the following for each element x in L: first, add x to the list γ to achieve initialization, then traverse the 8 adjacent fields of each element x that belong to the same element y of ^Im , if the formula strength( y)∈(strength(x)-z,strength(x)) holds, then let and will marked as candidate points, if will is added to the list γ as a valid maximum value; second, traverse the elements For each element y in the 8-adjacent domain of , if the strength of the adjacent point y is in the interval (strength(x)-z, strength(x)), it is also marked as a candidate point; continue to visit the 8-adjacency of the candidate point domain, repeat the above marking operation until it is found that the strength of all adjacent points y of a candidate point satisfies strength(y)<strength(x)-z, and the process ends, where z represents the tolerance of the gray value in this algorithm degree, determined by the range seeded around the maximum, and if the visited neighbor y already exists in the list L, it is removed from L as the weaker maximum; finally, by computing all elements in γ to generate a salient point, the geometric center is characterized by the maximum value in the wider image area.

Step 3: According to the detection results of the depth saliency algorithm, perform the iterative clustering unified algorithm (ICU) detection, and the iterative clustering unified algorithm extracts the point-like cross feature (PCFM) composed of salient points, and marks it as abnormal or normal. A Reliable Outlier Classifier for Images of Underwater Structures. An ideal detection model is obtained by training the WCNN network model and ICU classifier with large-scale underwater structure image data in a weakly supervised environment. The steps of the iterative clustering unified algorithm are as follows:

a, let In and _{Ia denote} the normal image and the defective image training set, respectively;

b, let P _n and P _{a be} the salient point sets extracted from In and I _a by the DSD algorithm;

c, iteratively perform the following operations S times, where S>1: for each normal image I _n , extract the point-like cross feature PCFM of P _n , and use K-means to cluster the point-like cross feature into a normal set Cluster Q, the elements in the cluster are represented by N _q , q=1, 2,...Q; for each defective image I _a , the point-like cross feature representation of P _a is extracted, and its point-like cross feature is adopted K-means clustering is an abnormal cluster R, and the elements in the cluster are represented by A _r , r=1, 2,...R;

d, let Q=Q·S, R=R·S;

e. Do the following for each element Ar, _r =1, 2,...R in the anomaly cluster _R : Between Ar and _Nq , calculate the distance d _rq (A _r , N _q ), q=1,2,...Q, then sort the distances d _rq in ascending order; then, calculate the normalized distance d _rq12 =d _rq1 /d _rq2 , where d _rq1 and d _rq2 represent A, respectively The distance between _r and its nearest neighbor cluster midpoints N _q1 and N _q2 , that is, the distance between clusters is estimated using the Euclidean distance between cluster centroids;

f, the average normalized distance d _q12 is estimated from all the distances d _rq12 obtained in step e;

_g . Perform the following operations on each element Ar in the abnormal cluster: if the formula d _rq12 <d _q12 is established, remove the element _Ar from the abnormal cluster and classify it into the normal cluster.

In step 4, the iterative clustering unified detection classifier is used for the evaluation test on the underwater structure image data set. The evaluation test detects and identifies the salient points related to the defects of underwater structures through K-means clustering method. The specific steps are as follows: images of underwater structures that are classified as defective set to the input image, right Do the following operations for each salient point m in: extract the point-like cross feature PCFM feature of point m; then calculate the distance from m to all clusters, that is, each point in the A _r ∪ N _q cluster; according to the K nearest neighbors of m The classes to which the points belong are voted by majority to classify detected salient points as normal or abnormal. Defective areas of underwater structures are indicated by rectangular boxes.

In a weakly supervised deep learning-based underwater structure defect detection method of the present invention, in the model training stage, DSD is applied to defect-free and defective images, and the information extracted from the feature map of the WCNN convolutional layer is used to represent the underwater structure salient features in an image. In the testing phase, DSD is applied to the defect images classified by WCNN to obtain defects represented by salient points. A novel iterative clustering unification algorithm of the present invention classifies the salient points detected by the DSD algorithm under weak supervision based on the clustering method, involving training and testing stages. During training, images of defective and normal structures are received and their salient points are clustered on their vector representations by K-means clustering. In the testing stage, it is mainly responsible for processing the abnormal images with defects, obtaining some abnormal points of the defects of underwater structures, and indicating the defect areas of the structures with a rectangular frame.

The above is only the preferred embodiment of the present invention, it should be pointed out that: for those skilled in the art, without departing from the principle of the present invention, several improvements and modifications can also be made, and these improvements and modifications are also It should be regarded as the protection scope of the present invention.

Claims (6)

1. a method for detecting defects of underwater structures based on weakly supervised deep learning, is characterized in that: comprise the following steps: S1, weakly label the input image with semantic information as the label, train a convolutional neural network model under weak supervision, and classify normal images and defective images; S2, using the third-layer convolutional layer information of the convolutional neural network model to implement a deep saliency detection algorithm; S3, according to the detection result of the depth saliency algorithm, perform an iterative clustering unified detection algorithm to train a reliable classifier of abnormal points of underwater structure images; S4, the iterative clustering unified detection classifier is used for the evaluation test on the underwater structure image dataset. 2. a kind of underwater structure defect detection method based on weak supervision deep learning according to claim 1, is characterized in that: in S2, the step of depth saliency detection algorithm is as follows: S21, using the feature map of the third layer t in the convolutional layer in the convolutional neural network model as the intermediate image ^Im for generating the salient point; S22, use the maximum filter to find the local maximum value in ^Im , and save it in the list L; S23, sort L in descending order of strength, and the strength is expressed as strength(x), where x is each element in L; S24, perform the following operations on each element x in L: first, add x to the list γ to achieve initialization, then traverse the elements y that belong to the same ^Im in the 8 adjacent fields of each element x, if the formula strength( y)∈(strength(x)-z,strength(x)) holds, then let and will marked as candidate points, if will is added to the list γ as a valid maximum value; second, traverse the elements For each element y in the 8-adjacent domain of , if the strength of the adjacent point y is in the interval (strength(x)-z, strength(x)), it is also marked as a candidate point; continue to visit the 8-adjacency of the candidate point domain, repeat the above marking operation until it is found that the strength of all adjacent points y of a candidate point satisfies strength(y)<strength(x)-z, and the process ends, where z represents the tolerance of the gray value in this algorithm degree, determined by the range seeded around the maximum, and if the visited neighbor y already exists in the list L, it is removed from L as the weaker maximum; finally, by computing all elements in γ to generate a salient point, the geometric center is characterized by the maximum value in the wider image area. 3. a kind of underwater structure defect detection method based on weak supervision deep learning according to claim 1, is characterized in that: in S3, iterative clustering unified algorithm steps are as follows: S31, let I _n and I _a represent the normal image and the defective image training set, respectively; S32, let P _n and P _{a be} the salient point sets extracted from In and I _a by using the DSD algorithm; S33, iteratively execute the following operations S times, where S>1: for each normal image I _n , extract the point-like cross feature PCFM of P _n , and use K-means to cluster the point-like cross feature into a normal set Cluster Q, the elements in the cluster are represented by N _q , q=1, 2,...Q; for each defective image I _a , the point-like cross feature representation of P _a is extracted, and its point-like cross feature is adopted K-means clustering is an abnormal cluster R, and the elements in the cluster are represented by A _r , r=1, 2,...R; S34, let Q=Q·S, R=R·S; _S35 , perform the following operations on each element A _r , _{r =} 1, 2, _. , N _q ), q=1,2,...Q, then sort the distances d _rq in ascending order; then, calculate the normalized distance d _rq12 =d _rq1 /d _rq2 , where d _rq1 and d _rq2 represent A, respectively The distance between _r and its nearest neighbor cluster midpoints N _q1 and N _q2 , that is, the distance between clusters is estimated using the Euclidean distance between cluster centroids; S36, the average normalized distance d _q12 is estimated from all the distances d _rq12 obtained in S35; S37, perform the following operations on each element Ar in the abnormal cluster: if the formula d _rq12 <d _q12 is established, remove the _{element Ar} from the abnormal cluster and classify it into the normal cluster. 4. a kind of underwater structure defect detection method based on weakly supervised deep learning according to claim 1, is characterized in that: in S4, the evaluation test passes the K-means clustering method to the salient points relevant to the underwater structure defect detected and identified. 5. A weakly supervised deep learning-based underwater structure defect detection method according to claim 4, characterized in that: the concrete steps are: to be classified as a defective underwater structure image set to the input image, right Do the following operations for each salient point m in: extract the point-like cross feature PCFM feature of point m; then calculate the distance from m to all clusters, that is, each point in the A _r ∪ N _q cluster; according to the K nearest neighbors of m The classes to which the points belong are voted by majority to classify detected salient points as normal or abnormal. 6 . A weakly supervised deep learning-based defect detection method for underwater structures according to claim 5 , wherein a rectangular frame is used to indicate the defect area of the underwater structures. 7 .