CN114332008A - Unsupervised defect detection and positioning method based on multi-level feature reconstruction - Google Patents

Unsupervised defect detection and positioning method based on multi-level feature reconstruction Download PDF

Info

Publication number
CN114332008A
CN114332008A CN202111625694.4A CN202111625694A CN114332008A CN 114332008 A CN114332008 A CN 114332008A CN 202111625694 A CN202111625694 A CN 202111625694A CN 114332008 A CN114332008 A CN 114332008A
Authority
CN
China
Prior art keywords
feature
reconstruction
abnormal
method based
network
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202111625694.4A
Other languages
Chinese (zh)
Other versions
CN114332008B (en
Inventor
陈平平
毛焕
陈锋
林志坚
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Fuzhou University
Original Assignee
Fuzhou University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Fuzhou University filed Critical Fuzhou University
Priority to CN202111625694.4A priority Critical patent/CN114332008B/en
Priority claimed from CN202111625694.4A external-priority patent/CN114332008B/en
Publication of CN114332008A publication Critical patent/CN114332008A/en
Application granted granted Critical
Publication of CN114332008B publication Critical patent/CN114332008B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Landscapes

  • Image Analysis (AREA)

Abstract

The invention relates to an unsupervised defect detection and positioning method based on multi-level feature reconstruction. The method comprises the following steps: acquiring abnormal images without abnormal images and different types of defects of products; extracting a multi-scale feature group by using an abnormal-free image input feature extraction network; inputting the features of the highest dimensionality into a reconstruction network to reconstruct a new feature group corresponding to different scales layer by layer; constructing a loss function training reconstruction network for the two groups of feature groups; and in the testing stage, an anomaly map and an anomaly score are calculated according to the difference condition of the two characteristic groups and are used for judging the anomaly and positioning the defect area. The method effectively utilizes the characteristic information difference of the abnormal image and the abnormal image in different dimensions, and can realize the detection of the defective area of the product, thereby avoiding manual marking and improving the detection efficiency of the product quality.

Description

Unsupervised defect detection and positioning method based on multi-level feature reconstruction
Technical Field
The invention relates to the field of computer vision, in particular to an unsupervised defect detection and positioning method based on multi-level feature reconstruction.
Background
The defect detection has wide and important application in the field of industrial production all the time, and in the industrial production, whether a product is a defective product or not is detected, the defect area of the product is positioned, and the yield of the product can be effectively ensured. In an actual scene, because defect samples have the characteristic of difficult acquisition, the number of the defect samples is very rare relative to the number of normal samples, the types of the defects have uncertainty, and the sizes and the positions of the defects are random, the characteristics of the defects cannot be predicted in advance. In addition, under the actual industrial production scene, certain requirements are also placed on the real-time performance of the defect detection algorithm.
With the continuous development of deep learning in the field of target detection, defect detection has also made a great progress as a branch of the field of target detection. In defect detection, only normal samples or an additional small part of defect samples are usually used as a training set, and the method of supervised learning is not practical in a defect detection task, so the method of unsupervised or weakly supervised learning is usually adopted. How to avoid the need for data labeling and realize the positioning of different types of defect areas is an important issue at present. In order to avoid the need of data labeling, the method is generally established on the characteristic extraction of a normal sample, and the characteristic distribution of the normal sample can be learned by reconstructing or learning the characteristic, so that a certain deviation can occur when a defective sample is faced as the basis of abnormal judgment. In order to realize the positioning of the defect area, an abnormal graph with the size of an input image is constructed, so that the prediction result of the network model on the image defect condition can be reflected intuitively, but the direct reconstruction of the normal image possibly faces the problem of over-strong generalization capability, so that the defect image is reconstructed, and in addition, the positioning of the image block level also does not meet the requirement of positioning the defect areas with different scales.
Disclosure of Invention
The invention aims to provide an unsupervised defect detection and positioning method based on multi-level feature reconstruction, namely, a defect detection framework which is high in efficiency and combines multi-level feature information with different scales is provided, the requirement for marking training data is avoided, and the defect positioning of different types with different sizes is realized. The scheme reduces the cost of manual labeling, improves the detection efficiency, has strong adaptability, and can be used in the defect detection links of different products.
In order to achieve the purpose, the technical scheme of the invention is as follows: an unsupervised defect detection and positioning method based on multi-level feature reconstruction comprises the following steps:
step S1, acquiring abnormal images and abnormal image making data sets of the target product;
s2, constructing a multi-level feature-based feature extraction and reconstruction network;
step S3, inputting a training image data set into the network constructed in the step S2 for training;
step S4, inputting the test image data set into the parameter optimal model for reasoning;
and step S5, obtaining a detection result by adopting an abnormal graph based on multi-level feature difference.
In an embodiment of the present invention, the data set used in step S1 is an industrial inspection abnormal data set MVTecAD, where the MVTecAD data set includes 15 categories of high resolution images with a total of more than 5 thousand, the categories include 10 categories of objects and 5 categories of textures, the training image data set in each category includes only normal images, and the test image data set includes normal images and multiple types of defect images and provides an annotation of a defect area.
In an embodiment of the present invention, the defective region label provided by the data set is converted into a mask image by a binarization method, that is, pixels of the normal region and the background region are set to 0, and pixels of the defective region are set to 255, and the input image and the mask image are adjusted by using a scaling and clipping method for network training and reasoning.
In an embodiment of the present invention, in step S2, a feature extraction and reconstruction network based on multi-level features is constructed, and the feature extraction and reconstruction network is composed of a feature extraction module and a feature reconstruction module; and extracting 3 feature graphs with different scales from the feature extraction module, taking the minimum size feature graph as the input of the feature reconstruction module, and extracting 3 feature graphs with corresponding scales from the feature reconstruction module, wherein the two groups of feature graphs are used for subsequent training and reasoning.
In one embodiment of the invention, the feature extraction module is based on a wideResNet50-2 network architecture.
In an embodiment of the present invention, in step S3, a loss function is used to train parameters of an optimized network model, parameters of a feature extraction module in a reconstructed network and a feature extraction module based on multi-level features remain unchanged, parameters of a feature reconstruction module participate in optimization, and training is performed under a Pytorch deep learning framework; loss function using multi-level feature reconstruction function Lf
Multilevel feature reconstruction function LfThe expression of (a) is as follows:
Figure BDA0003439438910000021
wherein f isnAnd f'nAnd respectively representing the feature map extracted by the nth layer and the corresponding reconstructed feature map, wherein the two feature maps are normalized by L2, and MSE represents a mean square error loss function.
In an embodiment of the present invention, in step S5, a group of residual feature maps is obtained by comparing differences between two groups of feature maps output by the feature extraction module and the feature reconstruction module, an abnormal map having the same size as an input image is obtained by performing upsampling, adding, and gaussian filtering operations on the residual feature maps of different layers, an abnormal score is obtained by taking a maximum value in the abnormal map, and a detection and location result of a defect is obtained by selecting an appropriate threshold to perform threshold segmentation on the abnormal map and the abnormal score.
In one embodiment of the present invention, residual error feature map for the nth layer
Figure BDA0003439438910000022
The expression of (a) is as follows:
Figure BDA0003439438910000023
wherein f isnAnd f'nRespectively representing the characteristic diagram extracted at the n-th layer and the corresponding reconstruction characteristic diagram, wherein the two characteristic diagrams are standardized by L2, | | · | | survival rate2Representing the L2 norm.
In one embodiment of the present invention, the expression for the anomaly map m is as follows:
Figure BDA0003439438910000031
wherein G is4Denoted as a gaussian filter with variance of 4, U is denoted as an upsampling operation using a bilinear interpolation, wn,hn,cnRespectively representing residual characteristic diagrams
Figure BDA0003439438910000032
Width, height and number of channels.
Compared with the prior art, the invention has the following beneficial effects:
1. the invention provides an efficient multi-level feature extraction module and a feature reconstruction module, which enhance the positioning capability of an abnormal defect area.
2. The invention integrates the difference of the extracted features and the reconstructed features under different scales as output, has strong adaptability to the defect detection of different classes, and improves the defect detection performance.
3. The invention carries out feature reconstruction by the features of the input image, realizes unsupervised learning and reduces the cost of manual marking;
4. the invention can also be applied to product defect detection in other scenes.
Drawings
FIG. 1 is a flow chart of the structure of the embodiment of the present invention.
Fig. 2 is an exemplary diagram of the MVTecAD data set in step S1 according to an embodiment of the present invention.
Fig. 3 is a diagram of a network structure for extracting and reconstructing features based on multi-level features in step S2 according to an embodiment of the present invention.
Fig. 4 is a block diagram of the reconstruction of the constructed features in step S2 according to the embodiment of the present invention.
Fig. 5 shows the detection result of the step S4 of inputting the test image data set into the parameter optimization model for inference in the embodiment of the present invention.
Detailed Description
The technical scheme of the invention is specifically explained below with reference to the accompanying drawings.
The unsupervised defect detection and positioning method based on multi-level feature reconstruction integrates the difference between multi-level features and reconstruction features, realizes the detection and positioning of defect samples under the condition of unsupervised learning, and simultaneously has small calculation overhead of a used network model, thereby meeting the performance requirement and the real-time requirement of a defect detection scene.
The invention relates to an unsupervised defect detection and positioning method based on multi-level feature reconstruction, which comprises the following steps:
step S1, acquiring abnormal images and abnormal image making data sets of the target product;
s2, constructing a multi-level feature-based feature extraction and reconstruction network;
step S3, inputting a training image data set into the network constructed in the step S2 for training;
step S4, inputting the test image data set into the parameter optimal model for reasoning;
and step S5, obtaining a detection result by adopting an abnormal graph based on multi-level feature difference.
The following are specific embodiments of the present invention.
As shown in fig. 1, the present embodiment provides an unsupervised defect detection and location method based on multi-level feature reconstruction, which includes the following steps:
step S1, acquiring abnormal images and abnormal image making data sets of the target product;
in this example, the step S1 is implemented as follows:
taking an MVTecAD dataset as an example, the label of the defect sample is labeled by the defect area provided by the dataset, a binary mask image is generated according to the labeling information, and then operations of scaling, clipping and standardization are applied to the sample image and the mask image.
S2, constructing a multi-level feature-based feature extraction and reconstruction network;
s21, constructing a feature extraction module based on the wideResNet50-2 network, extracting 3 layers of feature graphs with different scales, and taking the layer of feature graph with the minimum size and the maximum number of channels as the input of a next feature reconstruction module;
in this example, the step S21 is implemented as follows:
firstly, the first 3 convolutional layers (Conv2, Conv3 and Conv4) of the wideResNet50-2 network are cut as a feature extraction module, and after an image is input, a feature map { f) with different scales of 3 layers is output1,f2,f3H, will f3As input to the feature reconstruction module.
And step S22, constructing a feature reconstruction module, reconstructing from high dimension to low dimension by using three convolution layers with the high-dimension feature map as input, adjusting the size of the feature map by using an up-sampling operation from layer to layer, and outputting the feature maps of 3 layers corresponding to the size in the feature extraction module.
In this example, the step S22 includes the steps of:
step S221, constructing a feature reconstruction module, which is composed of three convolution layers (Conv4 ', Conv3 ', Conv2 '), each convolution layer includes 3 convolution operations with convolution kernel sizes of 1, 3, and 1, each convolution operation is accompanied by batch normalization operation and a LeakyReLu activation function, each convolution layer samples the output feature map by 2 times by using a bilinear interpolation method, and an output branch of each layer of feature map integrates information between channels by using a convolution operation with a convolution kernel size of 1.
Step S222, the feature f with the largest number of channels and the smallest size output by the feature extraction module3As an input to the feature reconstruction module,outputting a reconstructed feature set { f 'of corresponding scale size from each convolution layer of the feature reconstruction module'1,f′2,f′3}。
Step S3, inputting a training image data set into the network constructed in the step S2 for training;
in this example, the step S3 includes the steps of:
and S31, training parameters of an optimized network model by adopting a loss function and an SGD optimizer, keeping parameters of a feature extraction module in the network unchanged, participating in optimization by parameters of a feature reconstruction module, and training under a Pythrch deep learning framework. Loss function using multi-level feature reconstruction function Lf
Multilevel feature reconstruction function LfThe expression of (a) is as follows:
Figure BDA0003439438910000051
wherein f isnAnd f'nAnd respectively representing the feature map extracted by the nth layer and the corresponding reconstructed feature map, wherein the two feature maps are normalized by L2, and MSE represents a mean square error loss function.
Step S32, in the training stage, the training data set is divided into 20% as a verification set, the training condition of the current model parameters is evaluated on the verification set, and the loss L of the model on the verification set is calculatedfAnd taking the training round with the minimum average value as the model with the optimal parameters and storing the model.
Step S4, inputting the test image data set into the parameter optimal model for reasoning;
step S5, obtaining a detection result by adopting an abnormal graph based on multi-level feature difference;
step S51, calculating difference values between the feature maps of each layer by using the feature map and the reconstructed feature map obtained by reasoning in the step S4 to obtain a residual feature map
Figure BDA0003439438910000052
Residual profile for nth layer
Figure BDA0003439438910000053
The expression of (a) is as follows:
Figure BDA0003439438910000054
wherein f isnAnd f'nRespectively representing the characteristic diagram extracted at the n-th layer and the corresponding reconstruction characteristic diagram, wherein the two characteristic diagrams are standardized by L2, | | · | | survival rate2Representing the L2 norm.
Step S52, for the residual feature map of each layer, after upsampling by a bilinear interpolation method, adding the residual feature maps of each layer and passing through a gaussian filter to obtain a final abnormal map m:
residual profile for nth layer
Figure BDA0003439438910000055
The expression of (a) is as follows:
Figure BDA0003439438910000056
wherein G is4Denoted as a gaussian filter with variance of 4, U is denoted as an upsampling operation using a bilinear interpolation, wn,hn,cnRespectively representing residual characteristic diagrams
Figure BDA0003439438910000057
Width, height and number of channels.
In this example, the step S52 is implemented as follows:
and selecting the maximum value in the abnormal graph as an abnormal score, and performing threshold segmentation on the abnormal graph and the abnormal score by selecting a proper threshold to obtain a defect detection and positioning result.
The above are preferred embodiments of the present invention, and all changes made according to the technical scheme of the present invention that produce functional effects do not exceed the scope of the technical scheme of the present invention belong to the protection scope of the present invention.

Claims (9)

1. An unsupervised defect detection and positioning method based on multi-level feature reconstruction is characterized by comprising the following steps:
step S1, acquiring abnormal images and abnormal image making data sets of the target product;
s2, constructing a multi-level feature-based feature extraction and reconstruction network;
step S3, inputting a training image data set into the network constructed in the step S2 for training;
step S4, inputting the test image data set into the parameter optimal model for reasoning;
and step S5, obtaining a detection result by adopting an abnormal graph based on multi-level feature difference.
2. The unsupervised defect detecting and locating method based on multi-level feature reconstruction as claimed in claim 1, wherein the data set used in step S1 is an industrial inspection abnormal data set MVTecAD, the MVTecAD data set includes 15 categories of high resolution images with a total of more than 5 k, the categories include 10 categories of objects and 5 categories of textures, the training image data set in each category includes only normal images, and the testing image data set includes normal images and multiple types of defect images and provides an annotation of defect regions.
3. The unsupervised defect detection and localization method based on multi-level feature reconstruction as claimed in claim 2, wherein the defect region label provided by the data set is converted into the mask image by the binarization method, i.e. the pixels of the normal region and the background region are set to 0, and the pixels of the defect region are set to 255, and the input image and the mask image are adjusted by scaling and cropping method for network training and reasoning.
4. The unsupervised defect detection and localization method based on multilevel feature reconstruction as claimed in claim 1, wherein the step S2 is to construct a multilevel feature-based feature extraction and reconstruction network, which is composed of a feature extraction module and a feature reconstruction module; and extracting 3 feature graphs with different scales from the feature extraction module, taking the minimum size feature graph as the input of the feature reconstruction module, and extracting 3 feature graphs with corresponding scales from the feature reconstruction module, wherein the two groups of feature graphs are used for subsequent training and reasoning.
5. The method as claimed in claim 4, wherein the feature extraction module is based on a wideResNet50-2 network structure.
6. The unsupervised defect detection and localization method based on multilevel feature reconstruction as claimed in claim 4, wherein in step S3, a loss function is used to train parameters of an optimized network model, parameters of a feature extraction module in the feature extraction and reconstruction network based on multilevel features remain unchanged, parameters of a feature reconstruction module participate in optimization, and training is performed under a Pytorch deep learning framework; loss function using multi-level feature reconstruction function Lf
Multilevel feature reconstruction function LfThe expression of (a) is as follows:
Figure FDA0003439438900000011
wherein f isnAnd f'nAnd respectively representing the feature map extracted by the nth layer and the corresponding reconstructed feature map, wherein the two feature maps are normalized by L2, and MSE represents a mean square error loss function.
7. The unsupervised defect detection and localization method based on multi-level feature reconstruction as claimed in claim 4, wherein in step S5, a group of residual feature maps is obtained by comparing the difference between two groups of feature maps output by the feature extraction module and the feature reconstruction module, an abnormal map with the same size as the input image is obtained by performing up-sampling, addition and gaussian filtering operations on the residual feature maps of different layers, the maximum value in the abnormal map is taken to obtain the abnormal score, and the abnormal map and the abnormal score are subjected to threshold segmentation by selecting a proper threshold to obtain the defect detection and localization result.
8. The unsupervised defect detection and localization method based on multi-level feature reconstruction as claimed in claim 7, wherein the residual feature map of the nth layer
Figure FDA0003439438900000021
The expression of (a) is as follows:
Figure FDA0003439438900000022
wherein f isnAnd f'nRespectively representing the characteristic diagram extracted at the n-th layer and the corresponding reconstruction characteristic diagram, wherein the two characteristic diagrams are standardized by L2, | | · | | survival rate2Representing the L2 norm.
9. The unsupervised defect detecting and locating method based on multi-level feature reconstruction as claimed in claim 8, wherein the expression for the abnormal graph m is as follows:
Figure FDA0003439438900000023
wherein G is4Denoted as a gaussian filter with variance of 4, U is denoted as an upsampling operation using a bilinear interpolation, wn,hn,cnRespectively representing residual characteristic diagrams
Figure FDA0003439438900000024
Width, height and number of channels.
CN202111625694.4A 2021-12-28 Unsupervised defect detection and positioning method based on multi-level feature reconstruction Active CN114332008B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202111625694.4A CN114332008B (en) 2021-12-28 Unsupervised defect detection and positioning method based on multi-level feature reconstruction

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202111625694.4A CN114332008B (en) 2021-12-28 Unsupervised defect detection and positioning method based on multi-level feature reconstruction

Publications (2)

Publication Number Publication Date
CN114332008A true CN114332008A (en) 2022-04-12
CN114332008B CN114332008B (en) 2024-06-28

Family

ID=

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114821187A (en) * 2022-05-19 2022-07-29 济南大学 Image anomaly detection and positioning method and system based on discriminant learning
CN115205650A (en) * 2022-09-15 2022-10-18 成都考拉悠然科技有限公司 Unsupervised abnormal positioning and detecting method and unsupervised abnormal positioning and detecting device based on multi-scale standardized flow
TWI812363B (en) * 2022-07-06 2023-08-11 緯創資通股份有限公司 Image defect detecting system, generating method thereof and computer readable medium of the method

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2019144575A1 (en) * 2018-01-24 2019-08-01 中山大学 Fast pedestrian detection method and device
CN111401384A (en) * 2020-03-12 2020-07-10 安徽南瑞继远电网技术有限公司 Transformer equipment defect image matching method
CN113034478A (en) * 2021-03-31 2021-06-25 太原科技大学 Weld defect identification and positioning method and system based on deep learning network
US20210319265A1 (en) * 2020-11-02 2021-10-14 Zhengzhou University Method for segmentation of underground drainage pipeline defects based on full convolutional neural network

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2019144575A1 (en) * 2018-01-24 2019-08-01 中山大学 Fast pedestrian detection method and device
CN111401384A (en) * 2020-03-12 2020-07-10 安徽南瑞继远电网技术有限公司 Transformer equipment defect image matching method
US20210319265A1 (en) * 2020-11-02 2021-10-14 Zhengzhou University Method for segmentation of underground drainage pipeline defects based on full convolutional neural network
CN113034478A (en) * 2021-03-31 2021-06-25 太原科技大学 Weld defect identification and positioning method and system based on deep learning network

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
郑思凡: ""双粒度光流流形学习的刮刷总成摆杆摆幅检测"", 《华南理工大学学报(自然科学版)》, 31 January 2020 (2020-01-31) *

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114821187A (en) * 2022-05-19 2022-07-29 济南大学 Image anomaly detection and positioning method and system based on discriminant learning
TWI812363B (en) * 2022-07-06 2023-08-11 緯創資通股份有限公司 Image defect detecting system, generating method thereof and computer readable medium of the method
CN115205650A (en) * 2022-09-15 2022-10-18 成都考拉悠然科技有限公司 Unsupervised abnormal positioning and detecting method and unsupervised abnormal positioning and detecting device based on multi-scale standardized flow
CN115205650B (en) * 2022-09-15 2022-11-29 成都考拉悠然科技有限公司 Unsupervised abnormal positioning and detecting method and unsupervised abnormal positioning and detecting device based on multi-scale standardized flow

Similar Documents

Publication Publication Date Title
CN108961235B (en) Defective insulator identification method based on YOLOv3 network and particle filter algorithm
CN111815601B (en) Texture image surface defect detection method based on depth convolution self-encoder
CN111680614B (en) Abnormal behavior detection method based on video monitoring
EP3885989A1 (en) Anomaly detection based on an autoencoder and clustering
CN112036513B (en) Image anomaly detection method based on memory-enhanced potential spatial autoregression
CN112967243A (en) Deep learning chip packaging crack defect detection method based on YOLO
CN110648310B (en) Weak supervision casting defect identification method based on attention mechanism
CN113610822B (en) Surface defect detection method based on multi-scale information fusion
CN111079539B (en) Video abnormal behavior detection method based on abnormal tracking
CN113362320A (en) Wafer surface defect mode detection method based on deep attention network
CN115115921B (en) Wafer defect mode identification method based on residual network
CN113762265B (en) Classified segmentation method and system for pneumonia
WO2024021461A1 (en) Defect detection method and apparatus, device, and storage medium
CN116934725A (en) Method for detecting sealing performance of aluminum foil seal based on unsupervised learning
CN113947597B (en) Industrial defect detection method, device and medium based on shielding reconstruction
Li et al. Fabric defect segmentation system based on a lightweight GAN for industrial Internet of Things
CN114565605A (en) Pathological image segmentation method and device
CN112733861B (en) Text erasing and character matting method based on U-shaped residual error network
CN116563250A (en) Recovery type self-supervision defect detection method, device and storage medium
CN114332008B (en) Unsupervised defect detection and positioning method based on multi-level feature reconstruction
CN114332008A (en) Unsupervised defect detection and positioning method based on multi-level feature reconstruction
CN115690001A (en) Method for detecting defects in steel pipe welding digital radiographic image
CN114862803A (en) Industrial image anomaly detection method based on fast Fourier convolution
CN115909077A (en) Hyperspectral image change detection method based on unsupervised spectrum unmixing neural network
CN115631197A (en) Image processing method, device, medium, equipment and system

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant