CN117121053A

CN117121053A - Apparatus and method for manufacturing anomaly detection

Info

Publication number: CN117121053A
Application number: CN202280026413.XA
Authority: CN
Inventors: 金大焕; 赵汉相
Original assignee: Samsung Electro Mechanics Co Ltd
Current assignee: Samsung Electro Mechanics Co Ltd
Priority date: 2021-05-25
Filing date: 2022-03-02
Publication date: 2023-11-24

Abstract

A method and apparatus for anomaly detection are provided. The method comprises the following steps: generating second image data by applying the first abnormality detection model to the first image data; generating third image data corresponding to an image difference between the first image data and the second image data by performing a first logical operation on the first image data and the second image data; generating image mask data having characteristic information between the first image data and the second image data by applying a second abnormality detection model to the first image data and the second image data; and generating fourth image data having abnormality indication information by performing a second logical operation on the third image data and the generated image mask data. The method may include learning the first anomaly detection model and the second anomaly detection model using a plurality of pieces of image data of a predetermined good artifact.

Description

Apparatus and method for manufacturing anomaly detection

Technical Field

The present disclosure relates to an apparatus and method for manufacturing anomaly detection.

Background

An apparatus for abnormality inspection or detection may inspect whether an image of an article of manufacture in the manufacturing industry (hereinafter, referred to as "manufacturing image") includes an abnormality, for example, for quality control or abnormality correction. A typical apparatus for anomaly detection may employ typical image processing, for example, in accordance with an image processing algorithm.

Alternatively, if anomaly checking/detection is performed using deep learning techniques, such as through Convolutional Neural Networks (CNNs), a large amount of training data and labels may be needed or desired for training the CNNs, but example usability of the appropriate data and labels may be limited.

(prior art literature)

(patent document 1) Japanese patent laid-open No. 2020-139905 (2019.04.03)

(patent document 2) Korean patent publication No. 10-2020-0135530 (2019.05.22)

(patent document 3) Korean patent publication No. 10-2021-0050186 (2019.10.28)

Disclosure of Invention

Technical problem

An aspect of the present disclosure is provided to introduce a selection of concepts in a simplified form that are further described below in the detailed description. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.

Solution to the technical problem

In one general aspect, an apparatus includes: an image generator configured to learn a first abnormality detection model using a plurality of pieces of image data of a predetermined good product, and apply the learned first abnormality detection model to the first image data to generate second image data; a first logic operator configured to perform a first logic operation on the first image data and the second image data, and output third image data corresponding to an image difference between the first image data and the second image data; a feature extractor configured to learn a second abnormality detection model using the plurality of pieces of image data of the predetermined good product, and apply the learned second abnormality detection model to the first image data and the second image data to generate image mask data having feature information between the first image data and the second image data; and a second logic operator configured to perform a second logic operation on the third image data and the image mask data to generate fourth image data having abnormality indication information.

The learning of the first abnormality detection model may include performing compression-recovery learning based on the predetermined good.

The first anomaly detection model learned may be configured to: when the first image data corresponds to data of a defective article, compression-recovery is performed on the data of the defective article to generate the second image data corresponding to one of a plurality of predetermined good articles.

For the first logical operation, the first logical operator may include a subtraction logical operation unit configured to subtract the first image data and the second image data.

The first logic operator may be configured to perform a process of learning a Structural Similarity (SSIM) self-encoder algorithm.

The feature extractor may be configured to: feature vector information between the first image data and the second image data is extracted, and the image mask data having the feature information is generated based on the feature vector information.

The learning of the second abnormality detection model may include learning a reverse embedding algorithm that compares image information input to the first abnormality detection model with image information output by the first abnormality detection model for each of a plurality of corresponding patch units, and classifies non-defective articles and defective articles based on a result of the comparison for each of the plurality of corresponding patch units.

The feature extractor may be configured to: extracting feature vector information from the first image data using the learned second abnormality detection model for respective tile units of the first image data, and generating the image mask data having one selected between non-defective product information and defective product information for each of the respective tile units based on the extracted feature vector information.

The second logic operator may include a multiplication logic operation unit configured to multiply the third image data with the image mask data.

In one general aspect, an apparatus includes a processor configured to: reconstructing defect-free article image data of the input image data using a learned first anomaly detection model, wherein the learned first anomaly detection model comprises a Structural Similarity (SSIM) -self-encoder; generating image mask data having characteristic information between the input image data and the reconstructed defect-free article image data using a second anomaly detection model based on learning of a reverse embedding algorithm; and generating defect indication information based on a result of the SSIM self-encoder and the generated image mask data.

In one general aspect, a processor-implemented method includes: generating second image data by applying the learned first abnormality detection model to the first image data; generating third image data corresponding to an image difference between the first image data and the second image data by performing a first logical operation on the first image data and the second image data; generating image mask data having characteristic information between the first image data and the second image data by applying a learned second abnormality detection model to the first image data and the second image data; and generating fourth image data having abnormality indication information by performing a second logical operation on the third image data and the generated image mask data.

The method may further comprise: learning the first anomaly detection model using a plurality of pieces of image data of a predetermined good artifact; and learning the second abnormality detection model using the plurality of pieces of image data of the predetermined good product.

The learning of the first anomaly detection model may include learning a Structural Similarity (SSIM) self-encoder algorithm, and the generating of the third image data may include implementing the learned SSIM self-encoder algorithm.

The generating of the image mask data may include: extracting the feature vector information from the first image data for respective tile units of the first image data, and generating the image mask data having one selected between non-defective product information and defective product information for each of the respective tile units based on the extracted feature vector information.

When the first image data corresponds to data of a defective article, the learned first anomaly detection model may perform compression-recovery on the data of the defective article to generate the second image data corresponding to one of a plurality of predetermined good articles.

Performing the first logical operation on the first image data and the second image data may include subtracting the first image data and the second image data.

The generating of the image mask data may include extracting the feature vector information between the first image data and the second image data, and generating the image mask data based on the extracted feature vector information.

The performing of the second logical operation may include multiplying the third image data with the image mask data.

Other features and aspects will be apparent from the following detailed description, the drawings, and the claims.

Advantageous effects of the invention

According to one or more embodiments, regarding an original image and an image restored by an image generator such as a self-encoder, etc., for a manufactured image in a manufacturing process, contrast learning of feature vectors may be performed by performing inverse embedding algorithm learning on each tile unit to reduce a difference between feature vectors at the same location and increase a difference between feature vectors at different locations, thereby rapidly detecting a defect of the manufactured image.

In addition, in one or more embodiments, only good articles may be learned, and thus production of articles may be performed in a timely manner. In some cases, it may be desirable to measure the size of the defective article. In an example, to solve the above-described problem with deep learning, a segmentation method may be used. For example, when an automatically generated segmentation mask is used as a segmentation learning label, there is an effect of significantly reducing the period of time constituting data for segmentation.

Drawings

FIG. 1 is a diagram illustrating an apparatus for anomaly detection in accordance with one or more embodiments.

FIG. 2 is a diagram illustrating example image data of FIG. 1 in accordance with one or more embodiments.

FIG. 3 is a diagram illustrating example operations of an anomaly detection model in accordance with one or more embodiments.

Fig. 4 is a diagram illustrating an example learning operation of an anomaly detection model of a device for anomaly detection using a reverse embedding method in accordance with one or more embodiments.

FIG. 5 is a diagram illustrating example anomaly detection results in accordance with one or more embodiments.

FIG. 6 is a diagram illustrating example results of anomaly detection for example capacitor components in accordance with one or more embodiments.

FIG. 7 is a diagram illustrating example results of anomaly detection for example camera module components in accordance with one or more embodiments.

FIG. 8 is a diagram illustrating an example method for anomaly detection in accordance with one or more embodiments.

Throughout the drawings and detailed description, the same reference numerals will be understood to refer to the same or similar elements, features and structures unless otherwise described or provided. The figures may not be to scale and the relative sizes, proportions, and depictions of elements in the figures may be exaggerated for clarity, illustration, and convenience.

Detailed Description

The following detailed description is provided to assist the reader in obtaining a thorough understanding of the methods, apparatus, and/or systems described herein. However, various changes, modifications, and equivalents of the methods, apparatus, and/or systems described herein will be readily apparent after an understanding of the present disclosure. For example, the order of operations described herein is merely an example and is not limited to the order set forth herein, but rather variations that will be readily understood after an understanding of the present disclosure may be made in addition to operations that must occur in a specific order. In addition, descriptions of known features after understanding the present disclosure may be omitted for the sake of clarity and conciseness.

The features described herein may be embodied in different forms and are not to be construed as limited to the examples described herein. Rather, the examples described herein have been provided solely to illustrate some of the many possible ways in which the methods, devices, and/or systems described herein may be implemented that will be readily appreciated after a review of the present disclosure.

The terminology used herein is for the purpose of describing various examples only and is not intended to be limiting of the disclosure. Singular forms also are intended to include plural forms unless the context clearly indicates otherwise. The terms "comprises," "comprising," and "having" are intended to specify the presence of stated features, integers, operations, elements, and/or groups thereof, but do not preclude the presence or addition of one or more other features, integers, operations, elements, and/or groups thereof.

Throughout the specification, when an element is described as being "connected to" or "coupled to" another element, the element may be directly "connected to" or directly "coupled to" the other element, or there may be one or more other elements interposed therebetween. In contrast, when an element is referred to as being "directly connected to" or "directly coupled to" another element, there can be no other element intervening therebetween. As used herein, the term "and/or" includes any one of the items listed in relation to and any combination of any two or more.

Although terms such as "first," "second," and "third" may be used herein to describe various elements, components, regions, layers or sections, these elements, components, regions, layers or sections should not be limited by these terms. Rather, these terms are only used to distinguish one element, component, region, layer or section from another element, component, region, layer or section. Thus, a first member, first component, first region, first layer, or first portion referred to in the examples described herein may also be referred to as a second member, second component, second region, second layer, or second portion without departing from the teachings of the examples.

Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs based on an understanding of the present disclosure. Terms (such as those defined in commonly used dictionaries) should be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and the present disclosure and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein. The use of the term "may" herein with respect to an example or embodiment (e.g., with respect to what the example or embodiment may include or implement) means that there is at least one example or embodiment that includes or implements such feature, and not all examples include or implement such feature.

As described above, if anomaly checking/detection (hereinafter referred to as "anomaly detection") is performed using deep learning techniques, such as by using a typical Convolutional Neural Network (CNN), the availability of appropriate data and tags found herein may be limited. For example, appropriate defective data and labels may be very limited in manufacturing sites that may require anomaly detection (such as manufacturing sites that may require or desire rapid changes).

For example, typically, a relatively large amount of data may be collected for training such CNNs. For example, if anomaly detection is performed using such CNNs, such training data may be collected from a state in which the data of good articles and the data of defective articles are balanced. However, in view of such an anomaly detection method based on the example CNN, there may be a disadvantage in that it takes a long time (for example, two (2) weeks to one (1) month or more) to collect data in this way, especially in the case where defects have an extremely low occurrence frequency.

Since typical deep learning methods applied to anomaly detection may be highly data-dependent, even if algorithms based on Social Networking Service (SNS) data or general images to be studied are optionally applied and unsupervised learning is considered, it is found that their performance may be degraded here, particularly in view of difficulty in collecting and learning based on defective data. Some methods may be implemented in a typical unsupervised learning algorithm for collecting incorrect data. However, in the case where such unsupervised learning is applied to anomaly detection, if unsupervised learning is applied to data collection, learning may still be performed mainly based on good artifacts (e.g., based on good enough artifacts for learning to take place).

Thus, although unsupervised learning can generally have the following advantages: the data collection period may be greatly shortened and operations for data tags may not be required, but there may be problems: such typical unsupervised learning methods may not be suitable for data with a large number of types and amounts of good artifacts, such as in the case of anomaly detection, because anomaly detection learning may also be expected to be based on defective artifacts.

In general, in the case of general unsupervised learning, the self-encoder structure may have a deep learning structure that can efficiently compress data, and may be a structure widely used in unsupervised learning. For example, in the case where such unsupervised learning is applied to anomaly detection, the data distribution of an image can be learned using a self-encoder having a CNN structure. For example, when the data of a good article can be learned, the data of the defective article can be transferred through the self-encoder structure and can be restored to the data of the good article. Therefore, the image difference from the original image can be used to detect an abnormality.

However, in this example in which such unsupervised learning is applied to anomaly detection, the self-encoder may have a bottleneck structure, and thus, when data is recovered, there may be a problem of blurring in a high-frequency region, which may make it difficult to distinguish between a fine defect region and a high-frequency normal region, and thus, the performance of desired anomaly detection may deteriorate.

As another example, an anomaly detection method using a Generated Antagonism Network (GAN) or using a variational self-encoder (VAE) instead of a self-encoder structure may be employed. However, in GAN, the following disadvantages may exist: the random vector may be expected to be fine-tuned when expected, and in the VAE, there may be the following problems: the quality of the restored image may deteriorate, such as in the operation of the self-encoder example.

Thus, when anomaly detection is applied in a manufacturing site, it may be desirable to address potential data loss problems due to bottlenecks of the example self-encoder. This may also be advantageous for anomaly detection to be applicable to a wide variety of good products.

Fig. 1 is a diagram illustrating an apparatus for anomaly detection in accordance with one or more embodiments, and fig. 2 is a diagram illustrating example image data of fig. 1 in accordance with one or more embodiments.

Referring to fig. 1 and 2, an apparatus 10 for anomaly detection may include, for example, an image generator 100, a first logic operator 200, a feature extractor 300, and a second logic operator 400. The device 10 and the devices described in fig. 2-4 each represent one or more processors and one or more non-transitory memories, wherein the learning and/or inference operations are implemented by hardware or by a combination of hardware and software, respectively, such as by instructions stored in the one or more memories that, when executed by at least one of the one or more processors, configure at least one or any combination of the one or more processors to implement any, any combination, or all of the operations or methods described herein. As another non-limiting example, any of the image generator 100, the first logic operator 200, the feature extractor 300, and the second logic operator 400, respectively, represent such one or more processors, and may represent such one or more non-transitory memories that may further store such respective instructions for such respective learning and/or reasoning operations.

The image generator 100 may learn (i.e., train) the first anomaly detection model using pieces of image data of good articles as learning targets. The image generator 100 may apply the learned first abnormality detection model to the first image data VD1 as the inspection object to generate the second image data VD2.

The first logic operator 200 may be configured to perform a first logic operation on the first image data VD1 corresponding to the original image and the second image data VD2 as the image restored by the image generator 100, and may output third image data corresponding to an image difference between the first image data VD1 and the second image data VD2. As a non-limiting example, the first logic operator 200 may be a first logic operation unit.

Feature extractor 300 may learn (i.e., train) the second anomaly detection model using the pieces of image data of the good article as learning targets. These pieces of image data of the good product may be pieces of image data of the same good product as those of the good product used as a learning target in the learning of the first abnormality detection model, or may be pieces of image data of different good products. The feature extractor 300 may apply the learned second abnormality detection model to the first image data VD1 and the second image data VD2 as the inspection objects to output the image mask data VMD having feature information between the first image data VD1 and the second image data VD2.

For example, a reverse embedding method may be used in the second abnormality detection model, in which comparison of the first image data and the second image data of the good product as a learning target for each tile unit may be learned, and therefore, the good product and the defective product with respect to the first image data (learning target) for each tile unit may be classified.

For example, the second anomaly detection model may implement a reverse embedded learning method, which will be described in more detail below with reference to, for example, FIG. 4 and equation 2.

In an example, the combined model may be derived by combining the learned second anomaly detection model of the feature extractor 300 and the learned first anomaly detection model of the image generator 100. In another example, the image generator 100 having the first abnormality detection model may be predetermined, and the feature extractor 300 having the learned second abnormality detection model may be combined with the image generator 100, the result of the image generator 100 may be compared with the result of the feature extractor 300, and a final abnormality detection result may be generated based on the result of the comparison.

Thus, as a non-limiting example, one or more embodiments may perform anomaly detection using two such models/networks (or an example combined model/network), where the final result of anomaly detection is based on the respective results of the two such models/networks.

The image generator 100 may be configured to compress and then recover the first image data VD1 of good articles. In this case, when a defective article passes through the image generator 100, when recovery is performed, second image data VD2 similar to data of a good article may be generated. When the difference between the two image data VD1 and VD2 is obtained by the implemented Structural Similarity (SSIM) algorithm, the third image data VD3 having the first abnormality detection map may be generated.

The feature extractor 300 may perform a reverse embedding learning method on the image data of the good product as the learning target, and then may compare the patch unit feature vector between the first image data VD1 of the original image as the inspection object and the second image data VD2 restored by the image generator 100 to generate the image mask data VMD having the second abnormality detection map.

As a non-limiting example, the feature extractor 300 may compare the first image data VD1 of each block nx32×32 with the second image data VD2 of each block nx32×32. In this case, as a non-limiting example, N represents the number of blocks, and "32×32" represents the size of an image having thirty-two (32) pixels by thirty-two (32) pixels.

A second logical operation (e.g., multiplication) may be performed on the third image data VD3 and the image mask data VMD to generate fourth image data VD4 having a final abnormality detection map. For example, the fourth image data VD4 can be used to detect abnormal data, and can also be used as a division mask. Accordingly, the fourth image data VD4 may be or include information indicating an abnormality.

The second logic operator 400 may be configured to perform a second logic operation (e.g., multiplication) on the third image data VD3 and the image mask data VMD to generate fourth image data VD4 having abnormality information. As a non-limiting example, the second logic operator 400 may be a second logic operation unit.

Accordingly, in the case where a good product as a learning target has been learned by the first abnormality detection model in the image generator 100, first image data corresponding to data of a defective product in the inspection process may be input, a recovery operation may be performed, and a second abnormality detection model corresponding to data of the good product may be generated using the result of the first abnormality detection model in the image generator 100.

The first logical operation of the first logical operator 200 may include a subtraction logical operation of subtracting the first image data VD1 and the second image data VD 2.

As a non-limiting example, the first logical operation of the first logical operator 200 may correspond to a process of learning a Structural Similarity (SSIM) -self-encoder algorithm.

For example, as a non-limiting example, the learning of the image generator 100 may use a process of learning SSIM-self-encoder algorithms and SSIM using the loss function presented below in equation 1.

[ 1]

As a non-limiting example, in equation 1, p and q are obtained by clipping the first image data VD1, the original image, and the second image data VD2 restored from the encoder to<k×k>Image data, μ obtained _p Sum mu _q Is the average intensity, sigma _p Sum sigma _q Is the variance, sigma _pq Is covariance, and c ₁ And c ₂ Is constant and is typically 0.01 and 0.03, respectively.

As described above, when the image data of a good product is learned by the self-encoder, the data distribution of the good product can be learned. Thus, when the image data of the defective product passes through the self-encoder, the image data of the defective product can be restored to the image data corresponding to the data of the good product.

Referring to fig. 3, the operation processing of the first abnormality detection model will be described. For example, the first abnormality detection model may have a learned data distribution of the first image data VD1, and may include a process of restoring the input image to a good product.

For example, when an image of an n×n size is input, an input image of a w×h size may be compressed into a vector of a one-dimensional M size, and then the compressed vector may be restored to an image of a w×h size (i.e., an original size of the input image before compression).

In this process, the important elements in the data of VD1 can be compressed into vectors of M size. When the non-learned data (e.g., a defective image) is input as VD1 to the first abnormality detection model, one of the previously learned images may be restored according to the input VD 1.

Referring to fig. 4, the feature extractor 300 may extract feature vector information FVI between the first image data VD1 and the second image data VD2, and may generate image mask data VMD having feature information based on the feature vector information FVI.

For example, the feature extractor 300 may learn a reverse embedding algorithm that compares the first image data VD1 and the second image data VD2 of good products (learning targets) for each tile unit, and may classify the good products and defective products on the first image data VD1 for each tile unit.

Accordingly, the feature extractor 300 may apply the learned reverse embedding algorithm to extract feature vector information from the first image data as the inspection object for each tile unit, and may output the image mask data VMD having good product information and defective product information for each tile unit based on the feature vector information FVI.

For example, as a non-limiting example, the loss function used in learning by the reverse embedding algorithm is presented below in equation 2.

[ 2]

In formula 2, z _i Is a feature vector in which a block of first image data (original image) is obtained by the feature extractor 300, and z _j Is a feature vector in which a block of the second image data restored by the image generator 100 is obtained by the feature extractor 300. sim is cosine similarity and τ is an hyper-parameter.

Referring to fig. 4, the feature extractor 300 may include, for example, a cropping operator 310, a contrast learner 330, and an anomaly score calculator 350.

First, the cropping operator 310 may crop the original image and the image restored by the image generator 100 for each tile unit.

The contrast learner 330 may learn cosine similarity for the same location in the augmented image and may learn cosine similarity for different locations in the reduced image. In this case, the contrast learner 330 may further include a projector. For example, the projector may function to compress only the important elements of such corresponding indications in the vector extracted by the feature extractor.

The image generator 100 may render feature vectors in the same position as each other and may render feature vectors in different positions from each other to clearly generate a difference depending on whether or not there is an abnormality. Thus, as a non-limiting example, the anomaly score calculator 350 may calculate the anomaly score using the above.

For example, in the abnormality detection map M, the first abnormality detection map M may be obtained by adding the first abnormality detection map M to the third image data obtained by the image generator 100 through the SSIM algorithm _ssim And is included in the inverse embedding algorithm by the feature extractor 300Second abnormality detection map M in the obtained image mask data VMD _contra Multiplication to obtain the final abnormality detection map M. As a non-limiting example, as presented below in equation 3, the anomaly score (AnomalyScore) may be selected to be, for example, a maximum value from the anomaly detection map M.

[ 3]

M＝M _ssim ×M _contra

AnomalyScore＝max _i，j M

For example, the second logical operation of the second logical operator 400 may be performed by a multiplication logical operation unit configured to multiply the third image data VD3 and the image mask data VMD.

In addition, as described above with reference to fig. 4 and equation 2, for example, the second abnormality detection model may correspond to a reverse embedding learning algorithm. For example, for each tile unit, the second abnormality detection model may be trained by performing reverse embedding learning on the second image data VD2 corresponding to the image restored by the image generator 100 and the first image data VD1 corresponding to the original image to reduce the difference between feature vectors at the same position and to increase the difference between feature vectors at different positions.

Referring to fig. 5, the table shown in fig. 5 represents example learning results in accordance with one or more embodiments as checking accuracy.

First, example capacitor (e.g., MLCC) data is applied in an example implementation of the apparatus 10 of fig. 1, e.g., such that the capacitor assembly 1 is learned using 6880 good product images, with 587 defective product images and 830 good product images being tested. In this case, as shown in fig. 5, the capacitor assembly 1 achieves an inspection accuracy of 97.1%.

Next, another example capacitor (e.g., MLCC) data is applied in the example implementation of the apparatus 10 of fig. 1, e.g., such that the capacitor assembly 2 is learned using 2201 good images, where the 639 good images and 1013 defective images are used for testing. In this case, as shown in fig. 5, the capacitor assembly 2 achieves an inspection accuracy of 94.5%.

Fig. 6 is a diagram illustrating example results of anomaly detection for a capacitor assembly in accordance with one or more embodiments, and fig. 7 is a diagram illustrating example results of anomaly detection for a camera module assembly in accordance with one or more embodiments.

Referring to fig. 6, for the capacitor assembly, original images (corresponding to the first image data VD 1) of cases C1 to C6 having six (6) different defects are inspected. As a result, as can be seen in fig. 6, the defect in the original image is accurately displayed on the inspection image (corresponding to the fourth image data VD 4).

Referring to fig. 7, for the camera module assembly, the original images (VD 1) of the cases C1 to C4 having four (4) different defects are checked. As a result, it can be seen in fig. 7 that the defect in the original image is accurately displayed on the inspection image (VD 4).

In short, the above example descriptions made with reference to fig. 1 to 7 also set forth corresponding operations of the example methods for anomaly detection. Likewise, the above and following descriptions of example methods for anomaly detection may be implemented by any of the devices and components described above with respect to fig. 1-7, and any combination thereof. Accordingly, in the following description of the example method with respect to fig. 8, duplicate descriptions may be omitted.

In operation S100, the first abnormality detection model may be learned using pieces of image data of good products as learning targets, and the learned first abnormality detection model may be applied to the first image data VD1 as an inspection object to generate the second image data VD2. As a non-limiting example, this may be performed in the image generator 100 discussed above.

In operation S200, a first logical operation may be performed on the first image data VD1 and the second image data VD2, and third image data VD3 corresponding to an image difference between the first image data VD1 and the second image data VD2 may be output. As a non-limiting example, operation S200 may be performed by the first logic operator 200 discussed above.

In operation S300, the second abnormality detection model may be learned using pieces of image data of good products as learning targets, and the learned second abnormality detection model may be applied to the first image data VD1 and the second image data VD2 as inspection objects to output the image mask data VMD having characteristic information between the first image data VD1 and the second image data VD 2. As a non-limiting example, operation S300 may be performed by the feature extractor 300 discussed above.

In operation S400, a second logical operation may be performed on the third image data VD3 and the image mask data VMD to generate fourth image data VD4 having abnormality information. As a non-limiting example, operation S400 may be performed by the second logic operator 400 discussed above.

The learned first abnormality detection model may perform compression-recovery learning on the good product, and when first image data corresponding to the defective product is input and a recovery operation is performed, second image data corresponding to the good product may be generated.

For example, the first logical operation of operation S200 may include a subtraction logical operation of subtracting the first image data VD1 and the second image data VD 2.

For example, the first logical operation of operation S200 may correspond to a process of learning a Structural Similarity (SSIM) -self-encoder algorithm.

In operation S300, feature vector information FVI between the first image data VD1 and the second image data VD2 may be extracted, and image mask data VMD having feature information based on the feature vector information FVI may be generated.

In operation S300, a reverse embedding algorithm that compares the first image data VD1 and the second image data VD2 of good products as learning targets may be learned for each patch unit based on the first abnormality detection model or when the first abnormality detection model is learned, and the good products and defective products on the first image data VD1 may be classified for each patch unit.

In operation S300, the learned reverse embedding algorithm may be applied to extract the feature vector information FVI from the first image data VD1 as the inspection object for each tile unit, and output the image mask data VMD having good product information and defective product information for each tile unit based on the feature vector information FVI.

The second logical operation of operation S400 may include a multiplication logical operation of multiplying the third image data VD3 and the image mask data VMD.

The devices for anomaly detection (including electronic devices, image generators, logic operators, feature extractors, cropping operators, contrast learners, anomaly score calculators, projectors, and other devices, apparatuses, units, modules, and components described herein with respect to fig. 1-8) are implemented by hardware components. Examples of hardware components that may be used to perform the operations described in this application include, where appropriate: a controller, a sensor, a generator, a driver, a memory, a comparator, an arithmetic logic unit, an adder, a subtractor, a multiplier, a divider, an integrator, and any other electronic component configured to perform the operations described in this disclosure. In other examples, one or more of the hardware components that perform the operations described in this application are implemented by computing hardware (e.g., by one or more processors or computers). The processor or computer may be implemented by one or more processing elements (such as logic gate arrays, controllers and arithmetic logic units, digital signal processors, microcomputers, programmable logic controllers, field programmable gate arrays, programmable logic arrays, microprocessors or any other devices or combination of devices configured to respond to and execute instructions in a defined manner to achieve desired results). In one example, a processor or computer includes or is connected to one or more memories storing instructions or software that are executed by the processor or computer. The hardware components implemented by the processor or computer may execute instructions or software, such as an Operating System (OS) and one or more software applications running on the OS, to perform the operations described in the present application. The hardware components may also access, manipulate, process, create, and store data in response to execution of instructions or software. For simplicity, the singular term "processor" or "computer" may be used to describe the examples described in the present disclosure, but in other examples, multiple processors or computers may be used, or a processor or computer may include multiple processing elements or multiple types of processing elements, or both. For example, a single hardware component or two or more hardware components may be implemented by a single processor or two or more processors or processors and controllers. One or more hardware components may be implemented by one or more processors or processors and controllers, and one or more other hardware components may be implemented by one or more other processors or another processor and another controller. One or more processors or processors and controllers may implement a single hardware component or two or more hardware components. The hardware components may have any one or more of a variety of processing configurations, examples of which include single processor, stand alone processor, parallel processor, single Instruction Single Data (SISD) multiprocessing, single Instruction Multiple Data (SIMD) multiprocessing, multiple Instruction Single Data (MISD) multiprocessing, and Multiple Instruction Multiple Data (MIMD) multiprocessing.

The methods shown in fig. 1-8, which perform the operations described in this application, are performed by computing hardware (e.g., by one or more processors or computers) implemented as described above, which execute instructions or software to perform the operations described in this application as performed by the methods. For example, a single operation or two or more operations may be performed by a single processor, or two or more processors, or a processor and controller. One or more operations may be performed by one or more processors or processors and controllers, and one or more other operations may be performed by one or more other processors or another processor and another controller. One or more processors or processors and controllers may perform a single operation or two or more operations.

Instructions or software for controlling computing hardware (e.g., one or more processors or computers) to implement the hardware components and perform the methods described above may be written as computer programs, code segments, instructions, or any combination thereof to individually or collectively instruct or configure the one or more processors or computers to operate as a machine computer or special purpose computer to perform the operations performed by the hardware components and methods described above. In one example, the instructions or software include machine code (such as machine code produced by a compiler) that is executed directly by one or more processors or computers. In another example, the instructions or software include high-level code that is executed by one or more processors or computers using an interpreter. The instructions or software may be written in any programming language based on the block diagrams and flowcharts shown in the figures and the corresponding descriptions herein (which disclose algorithms for performing the operations performed by the hardware components and methods described above).

Instructions or software for controlling computing hardware (e.g., one or more processors or computers) to implement the hardware components and perform the methods described above, as well as any associated data, data files, and data structures, may be recorded, stored, or fixed in or on one or more non-transitory computer-readable storage media. Examples of non-transitory computer readable storage media include read-only memory (ROM), random-access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random-access memory (RAM), dynamic random-access memory (DRAM), static random-access memory (SRAM), flash memory, nonvolatile memory, CD-ROM, CD-R, CD + R, CD-RW, CD + RW, DVD-ROM, DVD-R, DVD + R, DVD-RW, DVD + RW, DVD-RAM, BD-ROM, BD-R, BD-rlth, BD-RE, blu-ray or optical disk storage, hard Disk Drive (HDD), solid State Drive (SSD), flash memory, card-type memory such as a multimedia card mini-or card (e.g., secure Digital (SD) or extreme digital (XD)), magnetic tape, floppy disk, magneto-optical data storage, hard disk, solid state disk, and any associated data, data files and data structures configured to store the instructions or software in a non-transitory manner and to provide the instructions or software and any associated data, data files and data structures to a computer or processor or one or more processors to execute the instructions. In one example, the instructions or software and any associated data, data files, and data structures are distributed in a networked computer system such that the instructions and software and any associated data, data files, and data structures are stored, accessed, and executed by one or more processors or computers in a distributed fashion.

While this disclosure includes particular examples, it will be readily understood after an understanding of the present disclosure that various changes in form and details may be made therein without departing from the spirit and scope of the claims and their equivalents. The examples described herein are to be construed in an illustrative, and not a limitative sense. The description of features or aspects in each example will be considered applicable to similar features or aspects in other examples. Suitable results may be obtained if the described techniques are performed in a different order and/or if components in the described systems, architectures, devices or circuits are combined in a different manner and/or replaced or supplemented by other components or their equivalent components. Therefore, the scope of the present disclosure may be defined by the claims and their equivalents in addition to the above disclosure, and all modifications within the scope of the claims and their equivalents are to be construed as being included in the present disclosure.

[ description of reference numerals ]

100: image generator

200: first arithmetic unit

300: feature extractor

400: second arithmetic unit

VD1: first image data

VD2: second image data

VD3: third image data

VMD: image mask data

VD4: fourth image data

Claims

1. An apparatus, comprising:

an image generator configured to learn a first abnormality detection model using a plurality of pieces of image data of a predetermined good product, and apply the learned first abnormality detection model to the first image data to generate second image data;

a first logic operator configured to perform a first logic operation on the first image data and the second image data, and output third image data corresponding to an image difference between the first image data and the second image data;

a feature extractor configured to learn a second abnormality detection model using the plurality of pieces of image data of the predetermined good product, and apply the learned second abnormality detection model to the first image data and the second image data to generate image mask data having feature information between the first image data and the second image data; and

and a second logic operator configured to perform a second logic operation on the third image data and the image mask data to generate fourth image data having abnormality indication information.

2. The apparatus of claim 1, wherein the learning of the first anomaly detection model includes performing compression-recovery learning based on the predetermined good.

3. The device of claim 1, wherein the learned first anomaly detection model is configured to: when the first image data corresponds to data of a defective article, compression-recovery is performed on the data of the defective article to generate the second image data corresponding to one of a plurality of predetermined good articles.

4. The apparatus of claim 1, wherein for the first logical operation, the first logical operator comprises a subtraction logical operation unit configured to subtract the first image data and the second image data.

5. The device of claim 1, wherein the first logic operator is further configured to perform a process of learning a Structural Similarity (SSIM) self-encoder algorithm.

6. The device of claim 1, wherein the feature extractor is configured to: feature vector information between the first image data and the second image data is extracted, and the image mask data having the feature information is generated based on the feature vector information.

7. The apparatus of claim 1, wherein the learning of the second anomaly detection model includes learning a reverse embedding algorithm that compares image information input to the first anomaly detection model with image information output by the first anomaly detection model for each of a plurality of respective tile units, and classifies non-defective articles and defective articles based on a result of the comparison for each of the plurality of respective tile units.

8. The device of claim 7, wherein the feature extractor is configured to: extracting feature vector information from the first image data using the learned second abnormality detection model for respective tile units of the first image data, and generating the image mask data having one selected between non-defective product information and defective product information for each of the respective tile units based on the extracted feature vector information.

9. The apparatus of claim 1, wherein the second logic operator comprises a multiplication logic operation unit configured to multiply the third image data with the image mask data.

10. An apparatus, comprising:

a processor configured to:

reconstructing defect-free article image data of the input image data using a learned first anomaly detection model, wherein the learned first anomaly detection model comprises a Structural Similarity (SSIM) self-encoder;

generating image mask data having characteristic information between the input image data and the reconstructed defect-free article image data using a second anomaly detection model based on learning of a reverse embedding algorithm; and

defect indication information is generated based on the result of the structural similarity self-encoder and the generated image mask data.

11. A processor-implemented method, comprising:

generating second image data by applying the learned first abnormality detection model to the first image data;

generating third image data corresponding to an image difference between the first image data and the second image data by performing a first logical operation on the first image data and the second image data;

generating image mask data having characteristic information between the first image data and the second image data by applying a learned second abnormality detection model to the first image data and the second image data; and

Fourth image data having abnormality indication information is generated by performing a second logical operation on the third image data and the generated image mask data.

12. The method of claim 11, the method further comprising:

learning the first anomaly detection model using a plurality of pieces of image data of a predetermined good artifact; and

the second abnormality detection model is learned using the pieces of image data of the predetermined good product.

13. The method of claim 12, wherein the learning of the first anomaly detection model includes performing compression-recovery learning based on the predetermined good.

14. The method according to claim 12,

wherein the learning of the first anomaly detection model includes learning a Structural Similarity (SSIM) self-encoder algorithm, an

Wherein the generation of the third image data includes implementing the learned structural similarity self-encoder algorithm.

15. The method according to claim 12,

wherein the learning of the second abnormality detection model includes learning a reverse embedding algorithm that compares image information input to the first abnormality detection model with image information output by the first abnormality detection model for each of a plurality of corresponding patch units, and classifies non-defective articles and defective articles based on a result of the comparison for each of the plurality of corresponding patch units.

16. The method of claim 15, wherein the generating of the image mask data comprises: extracting the feature vector information from the first image data for respective tile units of the first image data, and generating the image mask data having one selected between non-defective product information and defective product information for each of the respective tile units based on the extracted feature vector information.

17. The method of claim 11, wherein when the first image data corresponds to data of a defective article, the first anomaly detection model that is learned performs compression-recovery on the data of the defective article to generate the second image data corresponding to one of a plurality of predetermined good articles.

18. The method of claim 11, wherein performing the first logical operation on the first and second image data comprises subtracting the first and second image data.

19. The method of claim 11, wherein the generating of the image mask data includes extracting the feature vector information between the first image data and the second image data, and generating the image mask data based on the extracted feature vector information.

20. The method of claim 11, wherein the performing of the second logical operation comprises multiplying the third image data with the image mask data.