CN114066811B

CN114066811B - Industrial product abnormality detection method, system, device and storage medium

Info

Publication number: CN114066811B
Application number: CN202111191074.4A
Authority: CN
Inventors: 胡广华; 唐辉雄; 何文亮; 涂千禧; 焦安强; 徐志佳; 王清辉
Original assignee: South China University of Technology SCUT
Current assignee: South China University of Technology SCUT
Priority date: 2021-10-13
Filing date: 2021-10-13
Publication date: 2024-03-19
Anticipated expiration: 2041-10-13
Also published as: CN114066811A

Abstract

The invention discloses an industrial product abnormality detection method, a system, a device and a storage medium, wherein the method comprises the following steps: obtaining an image to be detected, inputting the image to be detected into an encoder to obtain a first coding feature, and performing cosine similarity calculation according to the first coding feature and a preset global template feature to obtain a similarity score graph; comparing the similarity score graph with a preset threshold score graph to obtain abnormal features, and replacing the abnormal features with template features at corresponding positions; inputting the coding features with the corrected features into a decoder to obtain a reconstructed image; and calculating similarity scores of the image to be detected and the reconstructed image, and performing threshold segmentation according to the similarity scores to obtain an image region with an abnormality in the image to be detected. The invention establishes template characteristics according to the characteristics of the actual detection image, overcomes the defect that the reconstruction model cannot reconstruct serious abnormality, has self-adaptability and anti-interference capability, and can be widely applied to the technical field of industrial product abnormality detection.

Description

Industrial product abnormality detection method, system, device and storage medium

Technical Field

The present invention relates to the field of industrial product anomaly detection technologies, and in particular, to a method, a system, a device, and a storage medium for industrial product anomaly detection.

Background

Before the final packaging of industrial products such as a new energy automobile battery pack, electrical equipment, a PCB (printed circuit board) and the like is completed, the products need to be subjected to anomaly detection, and the purposes of finding out surface defects of parts and identifying whether foreign matters scattered in the products such as bolts, nuts and the like exist are achieved. Abnormal conditions of industrial products not only affect the quality and appearance of the products, but also can affect the service performance of the products to a great extent, and even cause serious potential safety hazards, such as short circuit accidents caused by metal foreign matters in battery packs. However, existing industrial product anomaly detection mainly relies on experienced detection staff to perform through manual visual observation, is low in efficiency and precision, is tedious in work, and is extremely susceptible to detection omission or false detection caused by subjective factors of detectors. Therefore, the development of the automatic and intelligent industrial product abnormality visual inspection system has remarkable application value.

The machine vision abnormality detection method proposed by the current academic world and industry is mainly divided into a background differential detection method, a supervised detection method based on deep learning, an unsupervised detection method based on deep learning and the like; however, these methods have obvious disadvantages in terms of stability, accuracy, versatility, and intelligence, and the like, and are briefly described as follows:

(1) Background difference-based detection methods. The method belongs to a traditional detection method, and comprises the following steps: firstly, collecting a plurality of standard product sample images without defects, which are called template images, for reference; during detection, the image to be detected is aligned with the template image, differential operation is carried out on the image to be detected and the template image, so that an error image between the image to be detected and the template image is obtained, and a region with a larger error value on the image is regarded as an abnormal (foreign matter) region. The method has small calculated amount and simple realization, but has the following inherent problems: (1) first, image alignment and positioning typically requires the use of special features on the test object, such as geometric feature holes, artificial mark patterns, etc., the feasibility of which is limited by the specific product. The more general method is to use image gradient features such as SIFT, SURF descriptors and the like to match the geometric transformation relationship between the image to be detected and the template image based on the similarity between the feature vectors, so as to correct the image to be detected and align the image to be detected with the template image. However, this method is limited to objects whose feature points, such as edge corners, are easily extracted, and which will fail or be unreliable for weak textures, such as metal object surfaces. (2) Secondly, because of factors such as manufacturing and assembling tolerance, elastic deformation of materials, offset of shooting visual angles and the like, the image to be detected cannot be accurately aligned with the template image, and errors generated by dislocation cannot be distinguished from real anomalies by an algorithm. (3) Again, the acquisition environment of the template image and the acquisition environment of the image to be detected cannot be strictly consistent, and additional noise is also introduced due to environmental illumination change and the like, so that unacceptable false detection is caused. Due to the above factors, the method is limited to detection of simple objects such as glass, film surface defects and the like, and is not suitable for complex scenes such as detection of foreign matters in battery packs.

(2) A supervised anomaly detection method based on deep learning. Such methods generally assume that the anomaly (foreign object) type is known, and that the anomaly, foreign object morphology of each class has a good consistency. Abnormal targets are identified and detected by utilizing a multi-target detection framework such as FasterRCNN, SSD or YOLO series. For example, the YOLO series deep learning target detection framework solves object detection as a regression problem, and achieves end-to-end target detection by fitting the width and height of a marked real frame, the image position and the category of the object in the frame, and the speed and accuracy are high. However: (1) because the target detection method needs to collect a large amount of images containing abnormal types and data of a real frame provided by manual marking, the acquisition of a large amount of abnormal samples in the actual production process is difficult, the manual marking workload is extremely large, and the manual marking workload is extremely easy to be influenced by subjective factors of marking personnel to generate false detection or omission detection. (2) More seriously, most types of anomalies (foreign objects) are unknown in an industrial environment, and the types of anomalies (foreign objects) vary in shape, appearance, and posture. However, the existing YOLO-like object detection frames have no capability of detecting untrained ("unseen") anomalies (foreign objects), and cannot meet the actual needs.

(3) On the other hand, the unsupervised anomaly detection method does not need to collect anomaly samples in advance, and only needs to input normal images into network models such as AE (self-encoder), VAE (variable self-encoder), f-AnoGAN (increasing the generation of an encoder template against a network) and variants thereof, compress high-dimensional normal features by using a bottleneck layer of the network, obtain representative normal features or fit the distribution of the normal features, filter redundant information, and reconstruct the compressed information into an original image. In the test process, the network executes the same forward reasoning calculation on the input image to be tested, and due to the action of the mechanism, abnormal characteristics are filtered by a bottleneck layer, so that the image to be tested is reconstructed into a region which is obviously different from an original image in an abnormal region, and meanwhile, normal reconstruction of a normal region is ensured, and the abnormal region is highlighted so as to facilitate segmentation. In theory, the method is independent of any defect and foreign matter sample in network learning, and detection of various unknown foreign matters can be realized by collecting a plurality of normal sample images. However, due to the excessively strong generalization capability of the network models (1) ae, VAE, etc., serious abnormal features such as large-area breakage, etc. are easily over-reconstructed (over-fitting), that is, the reconstructed image still contains abnormal features, abnormal regions cannot be highlighted, and difficulty is brought to later segmentation. (2) Meanwhile, AE and VAE essentially use the information of the input data to carry out the deep convolution operation, the input is mapped to the high-dimensional normal feature space, and the decoder decodes again through the mapping information to complete the reconstruction task. During detection, once a large number of defects occur to a detection object, such as a part with a larger battery pack defect size or part breakage, an input image to be detected is seriously inconsistent with a normal image, at this time, a reconstruction network cannot map 'abnormal' input to a high-dimensional normal feature space, namely, input information deviates from a mapping space fitted by an encoder, so that a decoding module fails to reconstruct and abnormal areas cannot be separated. The essence of the method is that the existing image reconstruction or restoration framework model lacks a global constraint mechanism for the characteristics, only local information (receiving domains) can be utilized to reconstruct an image block which is highly similar to a normal sample, the relative position relation between the receiving domains is ignored, and severely damaged images cannot be restored and reconstructed, so that the accuracy and stability of detection are affected.

Disclosure of Invention

In order to solve at least one of the technical problems existing in the prior art to a certain extent, the invention aims to provide an industrial product anomaly detection method, system, device and storage medium for fusing global template feature constraint and convolutional neural network image reconstruction.

The technical scheme adopted by the invention is as follows:

an industrial product anomaly detection method comprises the following steps:

obtaining an image to be detected, inputting the image to be detected into an encoder to obtain a first coding feature, and performing cosine similarity calculation according to the first coding feature and a preset global template feature to obtain a similarity score graph;

comparing the similarity score graph with a preset threshold score graph, judging the position with similarity score smaller than the threshold score graph as an abnormal feature, and replacing the abnormal feature with a template feature at a corresponding position;

inputting the coding features with the corrected features into a decoder to obtain a reconstructed image;

and calculating similarity scores of the image to be detected and the reconstructed image, and performing threshold segmentation according to the similarity scores to obtain an image region with an abnormality in the image to be detected.

Further, the global template features and the threshold score map are obtained by:

Acquiring an industrial product image without abnormality, and dividing the acquired industrial product image into a training set and a verification set;

training an image reconstruction model using the training set, the image reconstruction model comprising an encoder and a decoder;

extracting the characteristics of each image in the training set by adopting the trained encoder to obtain second coding characteristics, and flattening all the second coding characteristics in the training set into a two-dimensional tensor;

k-means clustering is carried out on the two-dimensional tensors, K template feature vectors are obtained, and the K template feature vectors are remodeled into K global template features;

inputting the verification set into an encoder, acquiring third coding features of the verification set, performing cosine similarity calculation on the third coding features of each image and all global template features to obtain k average scores, and selecting a template corresponding to the maximum value from the k average scores as a nearest template;

cosine similarity comparison is carried out on the nearest similar template and the third coding feature to obtain a similarity score matrix, the similarity score matrix of all images is accumulated and averaged to obtain a threshold score graph, and the threshold score graph is used as a judgment basis for feature replacement in the detection process;

The size of the global template feature is consistent with the size of the coding feature, and k is a positive integer.

Further, the training the image reconstruction model using the training set includes:

constructing a combined loss function, wherein the combined loss function comprises an L2 norm loss function, a structural similarity loss function and a multi-scale gradient amplitude similarity loss function;

inputting the original image in the training set into the image reconstruction model to obtain a reconstructed image;

and comparing the differences of the original image and the reconstructed image by adopting the combined loss function, and training the image reconstruction model according to the differences obtained by comparison.

Further, the expression of the combined loss function is as follows:

wherein lambda is ₁ 、λ ₂ And lambda (lambda) ₃ For the weight of each loss function,X _i representing an input image +.>The reconstructed image is defined, D represents the decoder,e represents an encoder; l2_loss (·) represents the L2 norm Loss function, ssim_loss (·) represents the structural similarity Loss function, and msgms_loss (·) represents the multi-scale gradient magnitude similarity Loss function;

the expression of the structural similarity loss function is as follows:

wherein mu is brightness and sigma is contrast; constant C ₁ Indicating negative avoidanceNear 0 causes instability of the system, constant C ₂ Indicating avoidance->Near 0, causing instability of the system, H, W represents the height and width of the image, respectively;

the expression of the multi-scale gradient amplitude similarity loss function is as follows:

wherein g (·) represents gradient information of the image obtained by convolution operation of the image by the Prewitt filter, N _l Representing the number of pixels at scale l, the constant c avoids system instability when the denominator approaches 0.

Further, a third coding feature of each image is calculated using the following formulaWith k global template features Z _T Cosine similarity of (c):

template with maximum average valueThird coding feature->And (2) each of>And (3) carrying out cosine similarity comparison on the corresponding positions to obtain similarity scores, wherein the formula is as follows:

where i represents the ith drawing of the validation set, j, k represent the jth row and kth column features of the feature drawing, c is the number of characteristic channels, N is the number of samples of the verification set, and finally a threshold score map S, S epsilon R is obtained ^h×w 。

Further, the calculating the similarity score of the image to be detected and the reconstructed image, and the threshold segmentation according to the similarity score, to obtain the image area with the abnormality in the image to be detected, includes:

performing MSGMS similarity score calculation on the image to be detected and the reconstructed image to obtain a score heatmap with the same resolution as the image to be detected;

And carrying out threshold segmentation on the pair score heat map to obtain a binarized image, and obtaining an area with a pixel value of 1 in the binarized image as an abnormal area.

Further, the acquiring the image to be detected includes:

conveying the industrial product to be detected to a preset position;

shooting the industrial product by adopting an image acquisition device to obtain an image to be detected;

the image acquisition device is fixed in position relation with the preset position.

The invention adopts another technical scheme that:

an industrial product anomaly detection system, comprising:

the feature extraction module is used for acquiring an image to be detected, inputting the image to be detected into the encoder to obtain a first coding feature, and performing cosine similarity calculation according to the first coding feature and a preset global template feature to obtain a similarity score graph;

the feature replacement module is used for comparing the similarity score graph with a preset threshold score graph, judging the position with the similarity score smaller than the threshold score graph as an abnormal feature, and replacing the abnormal feature with a template feature at a corresponding position;

the image reconstruction module is used for inputting the coding features subjected to the feature correction into a decoder to obtain a reconstructed image;

The anomaly identification module is used for calculating the similarity score of the image to be detected and the reconstructed image, and carrying out threshold segmentation according to the similarity score to obtain an image area with anomalies in the image to be detected.

The invention adopts another technical scheme that:

an industrial product anomaly detection device, comprising:

at least one processor;

at least one memory for storing at least one program;

the at least one program, when executed by the at least one processor, causes the at least one processor to implement the method described above.

The invention adopts another technical scheme that:

a computer readable storage medium, in which a processor executable program is stored, which when executed by a processor is adapted to carry out the method as described above.

The beneficial effects of the invention are as follows: the invention establishes template characteristics according to the characteristics of the actual detection image, overcomes the defect that the reconstruction model cannot reconstruct serious abnormality, has good self-adaptability and anti-interference capability, and can be effectively used for automatic online detection of the abnormality of the industrial fixed scene product.

Drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following description is made with reference to the accompanying drawings of the embodiments of the present invention or the related technical solutions in the prior art, and it should be understood that the drawings in the following description are only for convenience and clarity of describing some embodiments in the technical solutions of the present invention, and other drawings may be obtained according to these drawings without the need of inventive labor for those skilled in the art.

FIG. 1 is a training flow diagram of a deep learning unsupervised self-encoder model in an embodiment of the present invention;

FIG. 2 is a flow chart of global template feature generation in an embodiment of the invention;

FIG. 3 is a flowchart of threshold score map generation in an embodiment of the present invention;

FIG. 4 is a flow chart of an image reconstruction test for fusing global template features in an embodiment of the present invention;

FIG. 5 is a schematic flow chart of an industrial product anomaly detection method in an embodiment of the invention;

FIG. 6 is a schematic diagram of a detection result and a segmentation result according to an embodiment of the present invention;

FIG. 7 is a graph showing the reconstruction results for a larger area of an artificial defect in an embodiment of the present invention;

fig. 8 is a schematic representation of the reconstruction results for product loss (plain background) in an embodiment of the present invention.

Detailed Description

Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein like or similar reference numerals refer to like or similar elements or elements having like or similar functions throughout. The embodiments described below by referring to the drawings are illustrative only and are not to be construed as limiting the invention. The step numbers in the following embodiments are set for convenience of illustration only, and the order between the steps is not limited in any way, and the execution order of the steps in the embodiments may be adaptively adjusted according to the understanding of those skilled in the art.

In the description of the present invention, it should be understood that references to orientation descriptions such as upper, lower, front, rear, left, right, etc. are based on the orientation or positional relationship shown in the drawings, are merely for convenience of description of the present invention and to simplify the description, and do not indicate or imply that the apparatus or elements referred to must have a particular orientation, be constructed and operated in a particular orientation, and thus should not be construed as limiting the present invention.

In the description of the present invention, a number means one or more, a number means two or more, and greater than, less than, exceeding, etc. are understood to not include the present number, and above, below, within, etc. are understood to include the present number. The description of the first and second is for the purpose of distinguishing between technical features only and should not be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated or implicitly indicating the precedence of the technical features indicated.

In the description of the present invention, unless explicitly defined otherwise, terms such as arrangement, installation, connection, etc. should be construed broadly and the specific meaning of the terms in the present invention can be reasonably determined by a person skilled in the art in combination with the specific contents of the technical scheme.

When the image reconstruction model encounters serious damage of a product (such as incomplete product caused by product truncation, missing of large-area parts in an assembly body, and the like), as the receiving domain of the network model is only in a certain local range, once large-area abnormal pixels appear, the network cannot recover to a normal image through a small amount of normal features, so that the abnormal region is excessively reconstructed (output is similar to input, but not similar to the normal image), and defect segmentation cannot be successfully performed. Even in the case of a complete missing product (product drop, etc.) the network cannot obtain any valid information of the product from the image, necessarily resulting in a reconstruction failure.

Therefore, in order to overcome the problems described above, the embodiment provides an industrial product anomaly intelligent detection method for reconstructing an image by fusing global template features and a deep convolutional neural network. The internal output characteristics of the encoder are constrained forcedly, so that the problem of excessive reconstruction of abnormal areas in detection is solved effectively, and the stability and accuracy of detection are improved greatly. The trained encoder outputs the coding features of the training set, clusters the coding features into k template features with global information according to the types or characteristics of the products, and meanwhile, effectively completes the online automatic detection of the industrial products by verifying the average score of similarity comparison between the coding features of the training set and the corresponding positions of nearest template features as a threshold value.

It is worth noting that the method of the embodiment is different from other anomaly detection methods based on feature clustering image reconstruction, and the method does not consider the characteristic of conforming to most products that the image content of different pixel areas is different, and is only suitable for the conditions of regular textures and single image features. The invention clusters the features of the same position of all pictures, thereby obtaining k templates with the most representation, namely, the features of each position of the templates are only related to the position, and the position information is effectively reserved, so that even if the position is changed from the product features to the background features due to the lack and the like, the method of the embodiment can not ignore the abnormality due to the high consistency with the background features. In addition, the method of the embodiment does not use some papers to replace the abnormal features by surrounding features, and the main reason is that when large-area abnormality occurs, since the surrounding of the features in the abnormal region are all abnormal features, all the abnormal features cannot be completely removed by only one replacement, so that the abnormality can be rebuilt. The method of the embodiment replaces the abnormal features at the same position with the template features based on the template position information, so that the replacement process is not influenced by surrounding features, all possible abnormal features can be effectively replaced, and the image is ensured to be reconstructed into a normal image.

The industrial product abnormality detection method provided by the embodiment comprises the following steps:

s1, acquiring an image without abnormality of an industrial product, and dividing the acquired image into a training set and a verification set.

The imaging condition of the image is controllable, namely the position relation between the industrial camera and the product is fixed, the imaging condition has the characteristics of fixed scene and basically aligned targets on the image, and the imaging condition can be realized in the following way:

in the actual abnormal detection process of industrial products, products are often conveyed to a designated position through a conveyor belt or a UGV trolley, an industrial mechanical arm is used for clamping an industrial camera to perform fixed-point shooting, the shooting point is fixed in position, and the acquired image information only slightly deviates due to the position accuracy of conveying equipment or the mechanical arm, namely, the positions of targets in images are consistent in height each time, namely, the targets are basically aligned. And in a practical in-line process, the kind of product is fixed, i.e. the kind is known, because of mass production. The acquired image has the characteristics of fixed scene and basically aligned target in the image.

S2, inputting a training set TData training image reconstruction model.

And comparing differences of the original image and the reconstructed image by using an L2 loss function, an SSIM (structural similarity) loss function and an MSGMS (multi-scale gradient amplitude similarity) loss function, and training a coding and decoding model.

The training process of the image reconstruction model is described below in detail with AE as an example, but it should be noted that the image reconstruction model is not limited to this model:

AE self-encoders are mainly composed of an encoder E,decoder D is configured to input image X _i Input to encoder E for processing to obtain encoded featuresThe resolution of the feature map is 16×16, i e (1, …, N) represents the ith image of the training set. Because of the "necked layer" of the encoder, the image information is compressed, and features extracted by the encoder have the characteristics of high-level abstraction and principal components in order to be able to reconstruct back into the original image. The coding features are then input to a decoder D for processing to obtain a reconstructed image +.>

The self-encoder consists of an encoder E and a decoder D, and the aim of the self-encoder is to enable the self-encoder to learn to extract the characteristics of normal images, and the self-encoder is trained by establishing the following loss functions:

wherein lambda is ₁ 、λ ₂ And lambda (lambda) ₃ For the weight of each loss function,l2_loss (·) represents the L2 norm Loss, i.e., the square root of the sum of squares of the elements in the matrix, ssim_loss (·) represents the structural similarity Loss, and the specific calculation is as follows:

where μ is brightness and σ is contrast. MSGMS_Loss is a multi-scale gradient magnitude Loss function, wherein the similarity of GMS gradient magnitudes under different resolutions of original image and reconstructed image is used, and the specific calculation is as follows:

Where g (·) represents the square root of the sum of squares of each element, N, obtained by convolving the image with a Prewitt filter of 3×3 convolution kernel size to obtain gradient information of the image _l Representing the number of pixels at scale l.

S3, manufacturing global template features.

The encoder pair training set TData completed through training, wherein TData epsilon R ^N×H×W×3 Performing forward propagation once to obtain compressed coding characteristic Z _f ，Z _f ∈R ^N×h×w×C . The coding feature retains the image position information, i.e. h, w are not equal to 1. To flattened coding features Z _f ′，Z _f ′∈R ^N×(h×w×C) K-means clustering is carried out to obtain K template feature vectors Z _T ', wherein Z _T ′∈R ^k×(h×w×C) . And Reshape these vectors to k global template features Z _T ∈R ^k×h×w×C The global template feature size is consistent with the code feature size, facilitating independent similarity comparisons for each location.

Specifically, inputting a training set to a trained encoder to obtain normal features Z of all images _f ＝E(X _i ) Will Z _f Tensor Z of shape Nx65536 that can be processed by flattening into K-means _f ' where N represents the number of images, 65536=16×16×256. Second to Z _f ' K-means clustering is carried out, the number K of clustering centers is set according to the characteristics of the product, and if the appearance of the toothbrush bristles is consistent except that the toothbrush bristles have three colors, k=3. Obtaining k×65536 feature tensors by K-means, and finally recovering the feature tensors to the shape of the original coding feature, namely K global modes of 16×16×256 Board feature Z _T ∈R ^{k×16×16×256} 。

S4, manufacturing a threshold score map S.

Features obtained by encoding each image using verification set VDataTemplate features most similar to it->The similarity comparison of cosine of the corresponding position is carried out to obtain a similarity score graph S _i ∈R ^h×w The approximate evaluation standard is that the similarity score obtained by the previous similarity comparison is averaged, and the similarity with the highest average similarity is taken as the nearest similar template. Finally for all S _i The similarity score of the corresponding positions is accumulated and averaged to obtain a threshold score chart S epsilon R ^h×w The judgment basis is used for characteristic replacement in the later test process.

Specifically, a forward propagation is performed by using the verification set to obtain a coded feature Z _f Wherein the encoding characteristics of each image of the set are verifiedAnd k template features Z _T And (3) performing corresponding position cosine similarity comparison, wherein the specific formula is as follows:

template with maximum average valueZ _f And (2) each of>And (3) carrying out cosine similarity comparison on the corresponding positions to obtain similarity scores, wherein the formula is as follows:

where i represents the ith drawing of the validation set, j, k represents the jth row and kth column features of the feature drawing, c is the number of characteristic channels, N is the number of samples of the verification set, and finally a threshold score map S, S epsilon R is obtained ^h×w Wherein S is _jk ＝Scores_threshold _jk 。

S5, testing. Inputting the image to be detected into an encoder to obtain a reconstructed image.

Each image to be detected outputs characteristics through an encoderThe specific steps are similar to the step S4 described above, and are similar to the most similar global template feature +.>Performing corresponding position cosine similarity comparison, and enabling each position similarity score to be smaller than a characteristic vector ++of the threshold score map S>Template feature vector T replaced with corresponding position _jk ∈R ^1×1×C The corrected feature->Input to the decoder to complete image reconstruction.

S6, image difference.

Reconstruction of images and artwork (i.e. images to be detected) And performing MSGMS similarity score calculation, performing threshold segmentation according to the similarity score, and binarizing to finally obtain an image region with the abnormality of the test image. Specifically, the MSGMS gradient amplitude similarity score is used for the reconstructed image and the original image, the calculation method refers to the description of the step S2, a scoring heatmap with the same resolution as the original image is obtained, and the scoring heatmap is subjected to threshold segmentation so as to obtain a binarized image I _mask The region with a value of 1 is the abnormal region.

In summary, the method of the present embodiment specifically includes two stages: an off-line training phase and an on-line real-time detection phase.

Offline training stage: when offline training is performed, only a small amount of industrial product images without abnormality are needed, and the images are divided into a training set and a verification set. And training the reconstruction model offline, and generating global template features and a threshold score map according to the steps.

On-line real-time detection stage: and in the online real-time detection stage, inputting the image to be detected into an encoder, comparing the cosine similarity of the corresponding positions of the output coding features and all template features, selecting the template feature with the largest average cosine similarity score as the nearest neighbor, taking a similarity score graph of the nearest neighbor template feature and the coding feature for carrying out replacement feature judgment, replacing the feature smaller than the threshold score graph with the template feature at the same position, and finally inputting the corrected feature into a decoder to finish the image reconstruction work. And (3) calculating two images of abnormal score heatmaps of the reconstructed image and the original image through MSGMS, and setting 1 and the rest 0 of the pixel area with higher abnormal score through threshold binarization so as to complete abnormal segmentation.

The above method is explained in detail with reference to specific examples.

An industrial product abnormal intelligent detection method integrating global template features and deep convolutional neural network image reconstruction comprises the following steps:

And step 1, acquiring an industrial product image without abnormality, and dividing the image into a training set and a verification set. As shown in fig. 1, the network structure diagram of the present invention is mainly divided into two modules: encoder E, decoder D.

And step 2, training the self-encoder by using the training set.

As shown in fig. 1, the self-encoder is trained to have a normal image reconstruction function, the training iteration number is 800, the earl vstop maximum tolerance iteration number is 50, the batch_size is 4, and Adam gradient descent is mainly used in the stage; the loss function is classified into three types of loss of L2, loss of SSIM and loss of MSGMS according to the above, wherein the weight ratio of each is 1:1:1. The encoder of the self-encoder comprises a five-time downsampling module, wherein each downsampling comprises two convolution layers, two BN layers and two activation functions (ReLU). The decoder is similar to the encoder, each up-sampling comprises an up-sampling layer, a convolution layer, a BN layer and an activation line number ReLU, and finally the pre-output activation function layer is changed into Tanh to ensure the same numerical range as the input data.

Step 3, generating coding features Z for each image in the training set by using the trained self-encoder _f And then flattening all the features in the training set into a two-dimensional Tensor with the shape of N multiplied by d, wherein N is the number of training set pictures, d=h multiplied by w multiplied by C, C is the number of feature map channels, and h and w are the height and width of the feature map respectively. K-means clustering is carried out on the two-dimensional Tensor, the clustering center is K types, the specific value is determined according to the characteristics of a data set, the flattening operation is equivalent to clustering each feature vector at a corresponding position, K-means clustering is carried out to obtain K multiplied dimension Tensor, the reshape is returned to the K multiplied by h multiplied by w multiplied by C shape according to the size of an original feature map to serve as a global template feature, and the flow is shown in figure 2.

Step 4, inputting the verification set into the encoder to obtain the coding characteristic Z of the verification set _f And carrying out cosine similarity calculation on the coding features of each image and all global template features to obtain k average scores, and selecting a template with the maximum value from the k average scores as a nearest neighbor template. Then uses the template and coding feature Z _f Cosine similarity comparison is carried out pixel by pixel, so as to obtain a similarity score matrix S with the same size as the coding feature size _i ∈R ^h×w . The similarity score matrix of all images is accumulated and averaged to finally obtain a threshold score table S epsilon R ^h×w The above flow is shown in fig. 3 as a threshold query graph of test phase replacement features.

And 5, the testing stage is similar to the training stage, the only difference is that the coded feature needs to be subjected to cosine similarity calculation with the global template feature nearest to the coded feature, the obtained similarity score graph needs to be compared with the threshold score graph, the position with the similarity score smaller than the threshold score table is judged to be the feature with the possibility of abnormality, and the position is replaced with the template feature at the corresponding position, so that the corrected coded feature is obtained. Finally, the data is input to a decoder to finish the final reconstruction work, and the overall working flow of the test is shown in fig. 4.

And 6, obtaining an abnormal score heatmap by the original image and the reconstructed image through MSGMS gradient amplitude similarity, separating the region with higher abnormal score by a set threshold value, thereby obtaining the position of an abnormal pixel, and completing the final abnormal segmentation work.

The above-described flow may be better described by way of fig. 5. The embodiment selects the industrial products MVTec data set bottle, capsule, pill, toothbrush and transistor as the experimental results to show that the images of the products have single background, little change of the appearance of the product types and invariable relative layout of the details, but the embodiment is only a preferred embodiment of the invention and is not limited by the invention, and any modification, equivalent replacement, improvement and the like which are within the spirit and principle of the invention are included in the protection scope of the invention. As a result, as shown in FIG. 6, it can be seen that the present invention has a good effect of reconstructing an abnormal region, which may occur in practice, into a normal region, and can accurately divide defects in industrial products. Further, we demonstrate the better detectability and stability of the present invention for images with large area anomalies by artificial defects such as drawing closed polygons, graffiti, and pure background (missing product), the results of which are shown in fig. 7 and 8. In fig. 6, 7 and 8, (a) corresponds to a button, (b) corresponds to a capsule, (c) corresponds to a pill, (d) corresponds to a tothbrush, and (e) corresponds to a transducer.

In summary, compared with the prior art, the method of the embodiment has the following beneficial effects:

(1) Compared with the existing unsupervised anomaly detection method, the method is not easily affected by the strong expression capacity of the reconstructed model. The strong expressive power of the various image reconstruction models themselves enables reconstruction of abnormal regions, and in addition, serious abnormalities and defects cannot be reconstructed into normal regions because the information of normal features cannot be utilized inside the abnormal regions. The image reconstruction forced constraint decoding feature fused with the global template feature provided by the embodiment replaces the abnormal feature with the corresponding position template feature before decoding, and the feature information of the surrounding neighborhood is not required to be filled, so that the feature before decoding is effectively ensured not to contain any abnormal information, and the expression capability of the image reconstruction network is fully utilized and the influence of the abnormal feature is restrained. And because of the method for replacing the bit alignment template, the situation that the abnormal characteristics are similar to the normal characteristics but the positions are wrong is better solved, so that the method provided by the embodiment can detect the abnormality even if the image is a pure background (the product falls off and the like).

(2) Compared with a supervised abnormality detection method, the method of the embodiment does not need a large amount of manual labeling and comprises a large amount of abnormality samples to detect the abnormality of the industrial product. The method of the embodiment utilizes the compression characteristic and the reconstruction characteristic of the image reconstruction network, and simultaneously utilizes the global template characteristic to replace the abnormal characteristic, so that the abnormal image area is effectively reconstructed into the normal area, the reconstructed image is a version of the input image without abnormality, and the abnormal area can be obtained by carrying out differential analysis on the input image and the reconstructed image. Meanwhile, the method of the embodiment can work well under the condition of no knowledge of the types of the anomalies and the large-area missing of the product, which is impossible for the supervised anomaly detection.

(3) Compared with the image difference technology, the method of the embodiment has stronger applicability and robustness and is not easy to be influenced by feature point extraction. According to the method, k global template features can be formed by setting the number of Kmeans clustering centers according to a specific scene, one set of features does not need to be manually selected according to each type of product, and universality is higher.

The embodiment also provides an industrial product abnormality detection system, which comprises:

The industrial product abnormality detection system of the embodiment can execute any combination implementation steps of the industrial product abnormality detection method provided by the method embodiment of the invention, and has the corresponding functions and beneficial effects of the method.

The embodiment also provides an industrial product abnormality detection device, including:

at least one processor;

at least one memory for storing at least one program;

The industrial product abnormality detection device provided by the embodiment of the invention can be used for executing any combination implementation steps of the embodiment of the method, and has the corresponding functions and beneficial effects.

The present application also discloses a computer program product or a computer program comprising computer instructions stored in a computer readable storage medium. The computer instructions may be read from a computer-readable storage medium by a processor of a computer device, and executed by the processor, to cause the computer device to perform the method shown in fig. 5.

The embodiment also provides a storage medium which stores instructions or programs for executing the industrial product abnormality detection method provided by the embodiment of the method, and when the instructions or programs are run, any combination of the embodiments of the method can be executed to implement steps, so that the method has corresponding functions and beneficial effects.

In some alternative embodiments, the functions/acts noted in the block diagrams may occur out of the order noted in the operational illustrations. For example, two blocks shown in succession may in fact be executed substantially concurrently or the blocks may sometimes be executed in the reverse order, depending upon the functionality/acts involved. Furthermore, the embodiments presented and described in the flowcharts of the present invention are provided by way of example in order to provide a more thorough understanding of the technology. The disclosed methods are not limited to the operations and logic flows presented herein. Alternative embodiments are contemplated in which the order of various operations is changed, and in which sub-operations described as part of a larger operation are performed independently.

Furthermore, while the invention is described in the context of functional modules, it should be appreciated that, unless otherwise indicated, one or more of the described functions and/or features may be integrated in a single physical device and/or software module or one or more functions and/or features may be implemented in separate physical devices or software modules. It will also be appreciated that a detailed discussion of the actual implementation of each module is not necessary to an understanding of the present invention. Rather, the actual implementation of the various functional modules in the apparatus disclosed herein will be apparent to those skilled in the art from consideration of their attributes, functions and internal relationships. Accordingly, one of ordinary skill in the art can implement the invention as set forth in the claims without undue experimentation. It is also to be understood that the specific concepts disclosed are merely illustrative and are not intended to be limiting upon the scope of the invention, which is to be defined in the appended claims and their full scope of equivalents.

The functions, if implemented in the form of software functional units and sold or used as a stand-alone product, may be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention may be embodied essentially or in a part contributing to the prior art or in a part of the technical solution, in the form of a software product stored in a storage medium, comprising several instructions for causing a computer device (which may be a personal computer, a server, a network device, etc.) to perform all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), a magnetic disk, or an optical disk, or other various media capable of storing program codes.

Logic and/or steps represented in the flowcharts or otherwise described herein, e.g., a ordered listing of executable instructions for implementing logical functions, can be embodied in any computer-readable medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, processor-containing system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions. For the purposes of this description, a "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device.

More specific examples (a non-exhaustive list) of the computer-readable medium would include the following: an electrical connection (electronic device) having one or more wires, a portable computer diskette (magnetic device), a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). In addition, the computer readable medium may even be paper or other suitable medium on which the program is printed, as the program may be electronically captured, via, for instance, optical scanning of the paper or other medium, then compiled, interpreted or otherwise processed in a suitable manner, if necessary, and then stored in a computer memory.

It is to be understood that portions of the present invention may be implemented in hardware, software, firmware, or a combination thereof. In the above-described embodiments, the various steps or methods may be implemented in software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, may be implemented using any one or combination of the following techniques, as is well known in the art: discrete logic circuits having logic gates for implementing logic functions on data signals, application specific integrated circuits having suitable combinational logic gates, programmable Gate Arrays (PGAs), field Programmable Gate Arrays (FPGAs), and the like.

In the foregoing description of the present specification, reference has been made to the terms "one embodiment/example", "another embodiment/example", "certain embodiments/examples", and the like, means that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the invention. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiments or examples. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

While embodiments of the present invention have been shown and described, it will be understood by those of ordinary skill in the art that: many changes, modifications, substitutions and variations may be made to the embodiments without departing from the spirit and principles of the invention, the scope of which is defined by the claims and their equivalents.

While the preferred embodiment of the present invention has been described in detail, the present invention is not limited to the above embodiments, and various equivalent modifications and substitutions can be made by those skilled in the art without departing from the spirit of the present invention, and these equivalent modifications and substitutions are intended to be included in the scope of the present invention as defined in the appended claims.

Claims

1. An industrial product anomaly detection method is characterized by comprising the following steps:

performing similarity score calculation on the image to be detected and the reconstructed image, and performing threshold segmentation according to the similarity score to obtain an image area with abnormality in the image to be detected;

the global template features and the threshold score map are obtained by:

wherein the size of the global template feature is consistent with the size of the coding feature, and k is a positive integer;

calculating a third encoding characteristic for each image using the following formulaWith k global template features Z _T Cosine similarity of (c):

2. The method of claim 1, wherein training an image reconstruction model using the training set comprises:

3. The industrial product anomaly detection method of claim 2, wherein the expression of the combined loss function is as follows:

wherein lambda is ₁ 、λ ₂ And lambda (lambda) ₃ For the weight of each loss function,X _i representing an input image +.>Representing the reconstructed image, D representing the decoder, E representing the encoder; l2_loss (·) represents the L2 norm Loss function, ssim_loss (·) represents the structural similarity Loss function, and msgms_loss (·) represents the multi-scale gradient magnitude similarity Loss function;

The expression of the structural similarity loss function is as follows:

wherein mu is brightness and sigma is contrast; constant C ₁ Representation avoidanceNear 0 causes instability of the system, constant C ₂ Indicating avoidance->Near 0, causing instability of the system, H, W represents the height and width of the image, respectively;

4. The method for detecting an abnormality of an industrial product according to claim 1, wherein the calculating the similarity score of the image to be detected and the reconstructed image, and the threshold segmentation according to the similarity score, obtaining the image area in which the abnormality exists in the image to be detected, includes:

and carrying out threshold segmentation on the scoring heat map to obtain a binarized image, and obtaining an area with a pixel value of 1 in the binarized image as an abnormal area.

5. The method for detecting an abnormality of an industrial product according to claim 1, wherein said acquiring an image to be detected includes:

conveying the industrial product to be detected to a preset position;

6. An industrial product anomaly detection system, comprising:

the anomaly identification module is used for calculating similarity scores of the image to be detected and the reconstructed image, and carrying out threshold segmentation according to the similarity scores to obtain an image area with anomalies in the image to be detected;

The global template features and the threshold score map are obtained by:

where i represents the ith drawing of the validation set, j, k represent the jth row and kth column features of the feature drawing,c is the number of characteristic channels, N is the number of samples of the verification set, and finally a threshold score map S, S epsilon R is obtained ^h×w 。

7. An industrial product anomaly detection device, comprising:

at least one processor;

at least one memory for storing at least one program;

the at least one program, when executed by the at least one processor, causes the at least one processor to implement the method of any one of claims 1-5.

8. A computer readable storage medium, in which a processor executable program is stored, characterized in that the processor executable program is for performing the method according to any of claims 1-5 when being executed by a processor.