KR20210141060A - Machine learning based image anomaly detection system - Google Patents

Abstract

The present invention relates to a machine learning-based image anomaly detection system. According to the present invention, a normal part is learned through machine learning from an image acquired from an object and an abnormal part is detected for visualization and quantification for the purpose of object error detection. The present invention includes: an auto encoder algorithm-based image learning module outputting a rebuilt image after an original image acquired from an object is input; a differential image generating unit generating a differential image by receiving the input of the original image and the rebuilt image and calculating the structural similarity between the original image and the rebuilt image; an average map generating unit receiving the input of the original image and generating an average map reflecting common spatial information; an attention map generating unit receiving the input of the differential image and generating an attention map reflecting position information requiring concentration during anomaly detection based on physical space coordinate information; an image arithmetic unit generating an arithmetic image by receiving the input of the differential image, the average map, and the attention map; an anomaly image generating unit receiving the input of the arithmetic image and generating an abnormal part-highlighting anomaly image; and an anomaly numerical value calculation unit receiving the input of the anomaly image and generating an anomaly numerical value indicating the degree of anomaly of the anomaly image.

Description

Machine learning-based image anomaly detection system {MACHINE LEARNING BASED IMAGE ANOMALY DETECTION SYSTEM}

The present invention relates to an image processing system, and more particularly, to a machine learning-based image anomaly detection system for detecting whether an image acquired from an object is abnormal in order to detect a defect of the object.

Anomaly detection system is a technology for detecting anomaly that is significantly different from normal and is being used in many fields. If there is a clear physical method that can distinguish between normal and abnormal well, anomalies can be easily detected by creating a rule based model. However, in reality, there are many cases in which the criteria for distinguishing normal from abnormal are ambiguous, or it is difficult to define characteristics for distinguishing abnormalities. In addition, in order to define these characteristics, related expertise or a lot of time and effort may be required. Above all, in the case of an abnormality, a new case that is not previously defined may occur. If the frequency of occurrence of anomalies is low in a stable situation, it is very difficult to secure bad data, so it is realistically impossible to secure all bad data in all possible cases in advance.

In order to overcome these difficulties, many studies have been conducted to learn data using machine learning to detect abnormalities. As a learning method, a method of remembering past cases through instance-based learning or learning an inference model is used. And, when new data is input, it is possible to determine whether it is normal or abnormal by measuring the similarity with existing data or by substituting it into an inference model.

In particular, in the case of images, through deep learning, it has established itself as a trend in image processing by exceeding the limits of traditional computer vision and demonstrating cognitive abilities beyond humans in a specific area. Deep learning can easily extract key features from images even if humans do not define them. Among the various known deep learning algorithms, an autoencoder is representative as an algorithm that can be used to detect anomalies in an image. Autoencoder is one of the unsupervised learning methods, and many autoencoder derivative algorithms are being studied because it is easy to learn and has excellent performance.

Embodiments of the present invention provide a machine learning-based image anomaly detection system that can learn a normal part through machine learning from an image obtained from an object to detect a defect of the object, and visualize and quantify only the abnormal part. it is to do

An image abnormality detection system according to an embodiment of the present invention includes an image learning module based on an autoencoder algorithm that receives an original image obtained from an object and outputs a reconstructed image; a differential image generating unit that receives the original image and the reconstructed image and generates a differential image by calculating the structural similarity between the original image and the reconstructed image; an average map generator that receives the original image and generates an average map reflecting common spatial information; an attention map generation unit receiving the differential image and generating an attention map reflecting location information to be focused on when detecting an anomaly according to coordinate information in physical space; an image operation unit configured to receive the differential image, the average map, and the attention map and generate a calculated image; an abnormal image generating unit that receives the calculated image and generates an abnormal image in which only an abnormal portion is emphasized; and an abnormality calculation unit that receives the abnormal image and generates an abnormality value obtained by digitizing the abnormality of the abnormal image. The differential image generator may further include a clustering unit configured to perform clustering on a plurality of the differential images.

The original image may include a gray scale image. The original image may include a normal image obtained from the non-defective object, and the image learning module may be learned using a plurality of the normal images. The image learning module includes: an encoder network that receives the original image and generates a latent vector; and a decoder network that has a structure symmetrical to that of the encoder network and generates the reconstructed image having the same size as the original image by receiving the latent vector. In addition, the image learning module includes: a pre-distribution unit for generating a real sample from a preset probability distribution; and a discrimination network that randomly receives the real sample and the fake sample generated by the encoder network and trains itself and the encoder network based on the discrimination loss to distinguish whether the input sample is real or fake. In addition, the image learning module may include: a noise adding unit configured to generate a noise-added image by receiving the original image and adding randomly generated noise to the original image; an encoder network including a plurality of serially connected layers and generating a latent vector by receiving the noise-added image; and a decoder network including a plurality of layers connected in series, having a structure symmetrical to the encoder network, and receiving the latent vector and generating the reconstructed image having the same size as the original image.

In the differential image generator, the structural similarity between the original image and the reconstructed image may be calculated using a Structural Similarity Index Map (SSIM). The original image may include a normal image obtained from the non-defective object, and the average map generator may generate the average map by averaging a plurality of the normal images in units of pixels. The original image may include a normal image obtained from the non-defective object, and the attention map generating unit generates the attention map by averaging the plurality of differential images corresponding to each of the plurality of normal images in units of pixels. can The original image may include a test image obtained from the object to detect a defect of the object and a normal image obtained from a non-defective object, and the calculated image may include a weighted summation image, and the weighted summation image may include a weighted summation image. The image may be generated by adding a weight to each of the differential image corresponding to the test image, the average map calculated from the plurality of normal images, and the attention map calculated from the plurality of normal images, and then summing them. The abnormal image generator may generate the abnormal image by comparing the operation image with a first reference value preset in units of pixels and changing the pixel value of the pixel that is less than or equal to the first reference value to a maximum value or a minimum value. The abnormal value calculation unit compares the abnormal image with a second reference value preset in units of pixels, counts only pixels that are equal to or greater than the second reference value, and divides the number of counted pixels by all pixels of the abnormal image to calculate the abnormal value can

An image abnormality detection system according to an embodiment of the present invention includes an image learning module based on an autoencoder algorithm that receives an original image obtained from an object and outputs a reconstructed image; a differential image generating unit that receives the original image and the reconstructed image and generates a differential image by calculating the structural similarity between the original image and the reconstructed image; a pooling image generator which receives the differential image and generates an average pooled image, a maximum pooled image, and a standard deviation pooled image, respectively; a fusion image generation unit generating a fusion image by summing the average pooled image, the maximum pooled image, and the standard deviation pooled image; an attention image generating unit generating an attention map reflecting the location information to be focused upon when detecting anomalies according to coordinate information in physical space, and generating an attention image by reflecting the generated attention map to the fusion image; an image calculating unit generating a calculated image in which the magnification of the attention image is adjusted; an abnormal image generating unit that receives the calculated image and generates an abnormal image in which only an abnormal portion is emphasized; and an abnormality calculation unit that receives the abnormal image and generates an abnormality value obtained by digitizing the abnormality of the abnormal image. The differential image generator may further include a clustering unit configured to perform clustering on a plurality of the differential images.

The original image may include a color image. The original image may include a normal image obtained from the non-defective object, and the image learning module may be learned using a plurality of the normal images. The image learning module includes: an encoder network that receives the original image and generates a latent vector; and a decoder network that has a structure symmetrical to that of the encoder network and generates the reconstructed image having the same size as the original image by receiving the latent vector. In addition, the image learning module includes: a pre-distribution unit for generating a real sample from a preset probability distribution; and a discrimination network that randomly receives the real sample and the fake sample generated by the encoder network and trains itself and the encoder network based on the discrimination loss to distinguish whether the input sample is real or fake. In addition, the image learning module may include: a noise adding unit configured to generate a noise-added image by receiving the original image and adding randomly generated noise to the original image; an encoder network including a plurality of serially connected layers and generating a latent vector by receiving the noise-added image; and a decoder network including a plurality of layers connected in series, having a structure symmetrical to the encoder network, and receiving the latent vector and generating the reconstructed image having the same size as the original image.

In the differential image generator, the structural similarity between the original image and the reconstructed image may be calculated using a Structural Similarity Index Map (SSIM). The original image may include a test image obtained from the object to detect a defect of the object, and each of the average pooled image, the maximum pooled image, the standard deviation pooled image and the fusion image is in the test image. may be corresponding. The fusion image may be generated by adding weights to each of the average pooled image, the maximum pooled image, and the standard deviation pooled image, and then summed them. The original image may include a normal image obtained from the non-defective object, and the attention image generating unit generates the attention map by averaging the plurality of differential images corresponding to each of the plurality of normal images in units of pixels. can The attention image generator may generate the attention image by multiplying the fusion image with the attention map to which a weight is reflected. The image operation unit may include a multiplier, and the operation image may include a multiplied image. The abnormal image generator may generate the abnormal image by comparing the operation image with a first reference value preset in units of pixels and changing the pixel value of the pixel that is less than or equal to the first reference value to a maximum value or a minimum value. The abnormal value calculation unit compares the abnormal image with a second reference value preset in units of pixels, counts only pixels that are equal to or greater than the second reference value, and divides the number of counted pixels by all pixels of the abnormal image to calculate the abnormal value can

This technology, based on the means of solving the above-mentioned problems, learns a normal part from an original image obtained from an object, detects only an abnormal part, visualizes and digitizes it, and provides the user with an effect of effectively detecting the defect of the object. have.

In addition, it is possible to track the chronic defect of the object, and there is an effect that the root cause of the defect can be identified and the cause can be provided.

In addition, it is possible to improve the efficiency of the manufacturing process by providing an index for judging a good product and a bad product in the object manufacturing process, and there is an effect that can improve the quality and yield of the object.

1 is a block diagram showing the configuration of an image anomaly detection system according to a first embodiment of the present invention.
2 is a block diagram showing the configuration of an image anomaly detection system according to a second embodiment of the present invention.
3 is a block diagram illustrating the configuration of an image anomaly detection system according to a third embodiment of the present invention.
4 is a block diagram illustrating the configuration of an image anomaly detection system according to a fourth embodiment of the present invention.

Advantages and features of the present invention and methods of achieving them will become apparent with reference to the embodiments described below in detail in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but will be implemented in a variety of different forms, and only these embodiments allow the disclosure of the present invention to be complete, and common knowledge in the technical field to which the present invention belongs It is provided to fully inform the possessor of the scope of the invention, and the present invention is only defined by the scope of the claims. Sizes and relative sizes of layers and regions in the drawings may be exaggerated for clarity of description. Like reference numerals refer to like elements throughout.

The present invention can be embodied as computer-readable codes on a computer-readable recording medium, and the computer-readable recording medium includes all types of recording devices in which data readable by a computer system is stored. . Examples of the computer-readable recording medium include ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc. also includes In addition, the computer-readable recording medium may be distributed in a network-connected computer system, and the computer-readable code may be stored and executed in a distributed manner.

Combinations of each block in the accompanying block diagram and each step in the flowchart may be performed by computer program instructions. These computer program instructions may be embodied in a processor of a general-purpose computer, special purpose computer, or other programmable data processing equipment, such that the instructions executed by the processor of the computer or other programmable data processing equipment may be configured in the respective blocks in the block diagram or in the flowchart. Each step creates a means for performing the described functions. These computer program instructions may also be stored in a computer-usable or computer-readable memory that may direct a computer or other programmable data processing equipment to implement a function in a particular manner, and thus the computer-usable or computer-readable memory. The instructions stored in the block diagram may also produce an item of manufacture containing instruction means for performing a function described in each block of the block diagram or each step of the flowchart. The computer program instructions may also be mounted on a computer or other programmable data processing equipment, such that a series of operational steps are performed on the computer or other programmable data processing equipment to create a computer-executed process to create a computer or other programmable data processing equipment. It is also possible that instructions for performing the processing equipment provide steps for carrying out the functions described in each block of the block diagram and each step of the flowchart.

Further, each block or each step may represent a module, segment, or portion of code that includes one or more executable instructions for executing the specified logical function(s). It should also be noted that, in some alternative embodiments, it is possible for the functions recited in blocks or steps to occur out of order. For example, it is possible that two blocks or steps shown one after another may in fact be performed substantially simultaneously, or that the blocks or steps may sometimes be performed in the reverse order according to the corresponding function.

An embodiment of the present invention, which will be described later, is intended to provide a machine learning-based image anomaly detection system. More specifically, an embodiment of the present invention learns a normal part through machine learning from an image obtained from an object in order to detect a defect of an object, and detects only an abnormal part to visualize and quantify an image abnormality based on machine learning. To provide a detection system. To this end, the image anomaly detection system according to an embodiment of the present invention may include an image learning module and an anomaly detection module based on an AutoEncoder algorithm.

In the machine learning-based image anomaly detection system according to an embodiment of the present invention, various well-known products may be applied as objects. For example, the object may be a predetermined structure, for example, a wafer on which a pattern is formed. In addition, various known imaging means may be used to acquire an image from an object. For example, the image acquired from the object may be an image captured based on various light sources, such as laser, visible light, and infrared light. As another example, the image acquired from the object may be an image taken by measuring equipment such as a scanning electron microscope (SEM), a transmission electron microscope (TEM), or an atomic force microscope (AFM).

Hereinafter, a machine learning-based image anomaly detection system according to an embodiment of the present invention will be described in detail with reference to the drawings. In an embodiment to be described later, the original image may refer to an image obtained from an object and input to the image anomaly detection system, and may include a normal image and a test image. In this case, the normal image may refer to an original image obtained from a first object that is a good product, and the test image may refer to an original image obtained from a second object having the same configuration as the first object for detecting object defects. have.

1 is a block diagram showing the configuration of an image anomaly detection system according to a first embodiment of the present invention.

As shown in FIG. 1 , the image anomaly detection system 10 according to the first embodiment of the present invention may include an image learning module 100 and an anomaly detection module 200 . Here, the image anomaly detection system 10 according to the first embodiment may detect an abnormality in a grayscale image obtained from an object, that is, a black-and-white image.

The image learning module 100 may be configured based on a generative model. Specifically, the image learning module 100 may be configured based on an autoencoder algorithm. More specifically, the image learning module 100 according to the first embodiment may be configured based on an Adversarial AutoEncoder (AAE) algorithm. The AAE algorithm may refer to an algorithm in which a Variational AutoEncoder (VAE) and a Generative Adversarial Network (GAN) are combined. The image learning module 100 based on the AAE algorithm may include an autoencoder block 102 and a discriminator block 104 . Meanwhile, as a modification of the first embodiment, the image learning module 100 ′ may be configured based on a Variational AutoEncoder (VAE) algorithm. That is, the image learning module 100 ′ according to a modification of the first embodiment may be configured only with the autoencoder block 102 .

The autoencoder block 102 in the image learning module 100 may include an encoder network 110 and a decoder network 120 having a symmetric structure. That is, the autoencoder block 102 may refer to a neural network in which the encoder network 110 and the decoder network 120 have a symmetrical structure. The autoencoder block 102 receives the original image and reduces (or compresses) the original image into a low-dimensional latent space through the encoder network 110, and converts the reduced original image to the decoder network 120. It is possible to output a reconstruction image by restoring the original dimension (or size) again through the Here, the dimension of the image may refer to the total size of the image or the total number of pixels. For example, when the original image is a black-and-white image composed of 10 pixels horizontally and vertically, the dimension of the original image may be 100 dimensions. Also, when the original image is a color image composed of 10 pixels horizontally and vertically and expressed in RGB, the dimension of the original image may be 300 dimensions.

The autoencoder block 102 may learn to reduce the difference between the original image and the reconstructed image in the dimensionality reduction and restoration process. In other words, the autoencoder block 102 trains the encoder network 110 and the decoder network 120 to reduce the difference between the original image and the reconstructed image by obtaining a reconstruction error in the dimensionality reduction and restoration process. can This is a method to solve by replacing unsupervised learning with supervised learning. The original image input to the autoencoder block 102 and the reconstructed image output through the autoencoder block 102 have the same size. can have

The encoder network 110 may be composed of two or more layers. The encoder network 110 may receive an original image, learn to extract key features from the original image, and reduce the high-dimensional original image to a low-dimensional latent space to output a latent vector. The latent vector is an output value output through the encoder network 110 and may refer to a value in the form of a dimensionally reduced (or compressed) vector including main features of the original image. Meanwhile, the latent vector may be referred to as a fake sample generated by the encoder network 110 .

Like the encoder network 110 , the decoder network 120 may consist of two or more layers. The decoder network 120 may serve to generate training data, that is, a reconstructed image, and may be trained to generate a reconstructed image very similar to the original image. In other words, the decoder network 120 may play a role of receiving a latent vector and generating and outputting a reconstructed image restored to the same dimension as the original image.

The image learning module 100 according to the first embodiment may be learned using only a plurality of easily obtainable normal images to facilitate image abnormality detection. That is, the autoencoder block 102 and the determination block 104 can be trained using only a plurality of normal images. If the autoencoder block 102 is trained using only a plurality of normal images, the autoencoder block 102 may have the ability to extract key features from the normal images. Through this, it is possible to determine normal or abnormal through the difference between the learned test image input to the autoencoder block 102 and the reconstructed image generated through the autoencoder block 102 . At this time, since the autoencoder block 102 according to the first embodiment was learned only with a normal image, if the test image input to detect the defect of the object is normal, that is, if the object to be inspected is of good quality, the image can be reconstructed well. Therefore, it is possible to generate a reconstruction image with little difference from the test image. On the other hand, when the test image is abnormal, that is, when the object to be inspected is defective, a large difference between the test image and the reconstructed image is inevitable because the abnormal image is not learned. That is, since the image learning module 100 according to the first embodiment is learned using only normal images based on the above-described principle, it is possible to effectively detect defects that are not defined in advance.

The autoencoder block 102 is easy to learn when estimating assuming a Gaussian distribution, but performance may be degraded if it does not follow an actual normal distribution. The discrimination block 104 is to prevent the performance degradation of the autoencoder block 102 described above, and may include a Prior Distribution unit 130 and a Discriminator network 140 .

The prior distribution unit 130 may serve to generate a real sample from a preset probability distribution. Here, the preset probability distribution may be a probability distribution that is easy to process, such as a normal distribution. For reference, the prior distribution unit 130 may extract a real sample from a preset probability distribution without receiving an original image. The real sample corresponds to the original image, but instead of receiving the original image, it may refer to a distribution desired by the user, that is, an arbitrary value extracted from a preset probability distribution. In this way, after extracting a real sample from a preset probability distribution in the prior distribution unit 130 , by learning through the discrimination network 140 , the preset probability distribution is followed and a specific distribution can be forced.

The discriminant network 140 may perform a role of examining the latent vector output from the encoder network 110 and forcing it into a desired type of probability distribution. At this time, the discriminant network 140 does not assume a specific probability distribution, but learns so that the distribution of the latent vector output from the encoder network 110 , that is, the fake sample, follows the distribution of the real sample output from the prior distribution unit 130 . Thus, the performance of the image learning module 100 can be improved. To this end, the determination network 140 randomly receives real samples and fake samples output from the prior distribution unit 130 and the encoder network 110, respectively, and determines whether the samples input to the determination network 140 are real or fake. It can learn based on discriminator loss to distinguish well. At this time, the performance of the image learning module 100 can be further improved by learning the discrimination network 140 itself based on the discrimination loss and at the same time additionally learning the encoder network 110 that provides fake samples to the discrimination network 140 . can Here, the discriminant loss, which is the output value of the discriminant network 140, is a measure defined for learning in machine learning. If the predicted value and the actual value are large, the loss is large, and if the difference is small, the loss is small. It may refer to a metric that designates a loss function and assigns a weight to learning. In the determination network 140 , a user may define a direct measure, but in a situation in which it is difficult to define a direct measure, it may be defined using a loss function or an error function.

As described above, the image learning module 100 according to the first embodiment is an autoencoder block 102 including an encoder network 110 composed of two or more layers and a decoder network 120 composed of two or more layers. By having , there is an advantage that robust learning is possible. In addition, by having the determination block 104 together with the autoencoder block 102, there is an advantage that it is possible to learn precisely enough to make it difficult to distinguish a real sample (or an original image) from a fake sample (or a reconstructed image). As such, the image anomaly detection system 10 according to the first embodiment includes the image learning module 100 that has been precisely learned using only the normal image, thereby generating a reconstructed image corresponding to the original image and at the same time between the two images. Based on the difference in , effective image anomaly detection is possible.

The anomaly detection module 200 according to the first embodiment may receive an original image and a reconstruction image corresponding to the original image from the image learning module 100, detect only an abnormal portion in the original image, and visualize and quantify it. Specifically, the anomaly detection module 200 may receive an original image and a reconstructed image in which the original image is output through the image learning module 100, detect only an abnormal portion in the original image, and generate a visualized abnormal image. In addition, the abnormality detection module 200 may generate an abnormality value obtained by quantifying the abnormality image so that the user can easily determine whether the abnormality is normal or abnormal based on the abnormality image. That is, the abnormality detection module 200 may serve to objectively, quickly and accurately detect a defect of an object from the original image and the reconstructed image. At this time, as the original image uses the gray scale image obtained from the object, the anomaly detection module 200 generates a differential image generator 210, an average map generator 220, and an attention map. It may include an attention map generator 230 , an image calculation unit 240 , an anomaly image generator 250 , and an anomaly score calculation unit 260 .

The differential image generator 210 may serve to generate a differential image by receiving the original image and the reconstructed image, calculating the structural similarity between the original image and the reconstructed image. At this time, the structural similarity between the original image and the reconstructed image is calculated using SSIM (Structural Similarity Index Map), which can calculate the difference between the luminance (l), contrast (c), and structure (s) between the two images. can be calculated using The original image input to the differential image generator 210 may include a test image and a plurality of normal images.

SSIM can be defined as in Equation 1 below. Meanwhile, since Equation 1 is a well-known equation, detailed descriptions of equations and variables will be omitted herein.

On the other hand, the differential image generator 210 may further include a clustering unit 212 for clustering similar differential images in a plurality of differential images corresponding to each of the plurality of original images and assigning a weight to each other. . The distribution of each of a plurality of original images input to the differential image generator 210 and a plurality of reconstructed images corresponding thereto is varied, or each of the plurality of original images and a plurality of reconstructed images corresponding thereto is the fineness of the object. In the case of corresponding to the (micro) region, the performance of the anomaly detection module 200 may be improved by performing clustering. The optimal number of clusters for multiple differential images is dimensionality reduction using Kernel PCA (Kernel principal component analysis) of Radial Basis Function (RBF), clustering of differential images using Gaussian Mixture Model (GMM), and BIC ( Based on the Bayesian Information Criterion), it can be calculated through a series of processes to identify the optimal cluster. On the other hand, the distribution of each of the plurality of original images input to the differential image generator 210 and the plurality of reconstructed images corresponding thereto is not varied, and in a certain case, a plurality of differential images corresponding to each of the plurality of original images It is also possible to skip clustering for the .

The average map generator 220 may serve to receive a plurality of original images and generate an average map in which common spatial information is reflected. In this case, the average map may be generated by averaging a plurality of original images pixelwise. Specifically, the average map generator 220 calculates the weight of the common area by averaging the pixel values at the same position from the plurality of original images, and through this, a pattern that appears the same in each of the plurality of original images, that is, the common space. It is possible to generate an average map reflecting the above information. Here, the plurality of original images input to the average map generator 220 may be a plurality of normal images obtained from each of the non-defective objects. That is, the average map may be calculated from a plurality of normal images.

The average map may be generated based on Equation 2 below.

Here, 'M' may indicate an average map, 'n' may indicate the number of original images input to the average map generator 220 , and 'x _i ' may indicate an original image.

The attention map generator 230 is to reflect a variation appearing between a plurality of normal images that cannot be obtained from the average map, and may serve to detect only an abnormal portion, excluding a normal portion. In other words, the attention map generator 230 may serve to generate an attention map in which location information to be focused on when detecting an anomaly is reflected according to coordinate information in a physical space. Specifically, the attention map generator 230 may receive a plurality of differential images from the differential image generator 210 , and may generate an attention map by averaging the plurality of differential images in units of pixels. Here, a plurality of differential images input to the attention map generator 230 is to be calculated through SSIM from a plurality of normal images used to learn the image learning module 100 and a plurality of corresponding reconstruction images. can That is, the attention map may be calculated from a plurality of normal images.

The attention map may be generated based on Equation 3 below.

'는 정상 이미지, '

'는 정상 이미지에 상응하는 재건 이미지를 지칭할 수 있다. Here, 'A' is the attention map, 'n' is the number of differential images input to the attention map generator 230, '

' is the normal image, '

' may refer to a reconstructed image corresponding to a normal image.

). 이는, 기존 스케일을 유지하기 위한 것으로, 가중치의 총 합은 1 이하이거나, 또는 1 이상일 수도 있다. 즉, 가중 합산 이미지를 생성하는 과정에서 가중치는 사용자가 원하는 이상 이미지를 생성하기 위해 조건에 따라 최적화가 필요한 변수일 수 있다. The image calculating unit 240 may serve to generate a calculated image by receiving a differential image, an average map, and an attention map. In this case, the calculation image may include a weighted sum image. The weighted summation image is an average map that reflects spatial information common to the differential image that reflects the structural similarity between the original image and the reconstructed image, and the attention map that reflects the location information to focus on when detecting anomalies according to the coordinate information in physical space. After that, it can be generated by summing. Here, the weight of each of the differential image, the average map, and the attention map is a user's arbitrary setting value for combining when generating an abnormal image, which is the basis of anomaly detection performance, and the sum of the weights given to each image may be 1 (

). This is to maintain the existing scale, and the total sum of weights may be 1 or less, or 1 or more. That is, in the process of generating the weighted summation image, the weight may be a variable that needs to be optimized according to conditions in order to generate an abnormal image desired by the user.

The weighted summation image may be generated according to Equation 4 below.

'은 가중 합산 이미지, '

'는 가중치가 부여된 차동 이미지, '

'는 가중치가 부여된 주의 맵, '

'은 가중치가 부여된 평균 맵를 지칭할 수 있다.here, '

'is the weighted sum image,'

' is the weighted differential image, '

' is a weighted map of states, '

' may refer to a weighted average map.

The abnormal image generating unit 250 receives a calculated image, for example, a weighted sum image, and compares the first reference value preset in pixel units with the weighted sum image to generate an abnormal image emphasizing only the abnormal part. In this case, the abnormal image generating unit 250 may generate an abnormal image in which only the abnormal portion is emphasized by changing the pixel value of the pixel that is less than or equal to the first reference value in the weighted sum image to the maximum value or the minimum value. That is, only an abnormal portion of the abnormal image output from the abnormal image generating unit 250 may be shaded.

The abnormality calculation unit 260 is for improving the convenience of management, and may receive an abnormal image and compare it with a second reference value preset in units of pixels to express the degree of image abnormality numerically. In this case, the abnormal value calculating unit 260 may calculate an abnormal value by counting only pixels equal to or greater than the second reference value in the abnormal image, and dividing the counted number of pixels by the total number of pixels. Here, when the average value of the abnormal image is simply used when calculating the abnormal value, the discrimination power may be reduced when there is a singularity and when there is a slight difference as a whole. To prevent this, the abnormal value calculation unit 260 according to the first embodiment may introduce a concept of density and assign weights to the abnormal value when calculating the abnormal value. In this case, the density may refer to calculating the number of pixels counted on the abnormal image, that is, the number of pixels equal to or greater than the second reference value, as the total number of pixels. Through this, it is possible to improve the discriminating power of the abnormal value.

Meanwhile, the first reference value and the second reference value preset in the abnormal image generating unit 250 and the abnormal numerical calculation unit 260 may be set to any value desired by the user within the range of 0 to 1, respectively. Since normality or abnormality changes depending on the reference value, if the reference value is set low, even a small abnormality can be detected and operated sensitively. can be established

As described above, the image abnormality detection system 10 according to the first embodiment of the present invention learns a normal part from an original image obtained from an object, detects only an abnormal part, visualizes and digitizes it, and provides it to the user. Defects can be effectively detected. In addition, it is possible to track a chronic defect of an object, and identify the root cause of the defect and provide the cause. In addition, by providing an index for judging a good product and a defective product in the object manufacturing process, the efficiency of the manufacturing process can be improved, and the quality and yield of the object can be improved.

2 is a block diagram showing the configuration of an image anomaly detection system according to a second embodiment of the present invention. Hereinafter, for convenience of description, the same reference numerals are used for the same components as those of the first embodiment, and detailed descriptions thereof will be omitted.

As shown in FIG. 2 , the image anomaly detection system 20 according to the second embodiment of the present invention may include an image learning module 300 and an anomaly detection module 200 . Here, the image anomaly detection system 20 according to the second embodiment may detect an abnormality in a gray-scale image obtained from an object, that is, a black-and-white image.

In the image anomaly detection system 20 according to the second embodiment, the anomaly detection module 200 may be the same as the anomaly detection module 200 described in the first embodiment. That is, the image anomaly detection system 20 according to the second embodiment may be different from the image anomaly detection system 20 according to the first embodiment only in the image learning module 300 . Therefore, in the second embodiment, a detailed description of the anomaly detection module 200 will be omitted.

The image learning module 300 according to the second embodiment may be configured based on a generative model. Specifically, the image learning module 300 may be configured based on an autoencoder algorithm. More specifically, the image learning module 300 according to the second embodiment may be configured based on a Stacked Denoising AutoEncoder (SDAE) algorithm.

The image learning module 300 configured based on the SDAE algorithm includes a noise adding unit 310, a plurality of layers 321, 322, 323, an encoder network 320 and an encoder network 320 connected in series. It has a symmetric structure and may include a decoder network 330 in which a plurality of layers 331 , 332 , 333 are connected in series. In the second embodiment, the encoder network 320 exemplifies a case in which the first layer 321 to the n-th layer 323, n is a natural number) are sequentially connected in series, and the first layer 321 to the n-th layer 323 ) direction can be gradually reduced in dimension. In addition, the decoder network 330 exemplifies a case in which the first layer 331 to the n-th layer 333 are sequentially connected in series, and gradually changes to the original image in the direction from the first layer 321 to the n-th layer 333 . The dimension can be restored to be close. Here, the number of layers in the encoder network 320 and the number of layers in the decoder network 330 may be the same or different from each other.

The noise adding unit 310 may serve to generate a noise addition image by receiving an original image and adding randomly generated noise to the original image. In this case, the noise added to the original image may be generated in various ways. The encoder network 320, in which a plurality of layers 321, 322, and 323 are connected in series, receives the noise-added image, learns to extract the main features from the noise-added image, and converts the high-dimensional noise-added image into a low-dimensional latent image. It is possible to output a latent vector by reducing the dimensions to space. The decoder network 330 in which a plurality of layers 331, 332, 333 are connected in series may serve to generate training data, that is, a reconstructed image, and may be trained to generate a reconstructed image similar to the original image. .

The image learning module 300 according to the second embodiment includes an encoder network 320 in which a plurality of layers 321, 322, 323 are serially connected and a plurality of layers 331, 332, 333 for a noise-added image. ) by using the serially connected decoder network 330 to reduce and restore the dimension, the performance of the image learning module 300 can be improved.

3 is a block diagram illustrating the configuration of an image anomaly detection system according to a third embodiment of the present invention. Hereinafter, for convenience of description, the same reference numerals are used for the same components as those of the first embodiment, and detailed descriptions thereof will be omitted.

As shown in FIG. 3 , the image anomaly detection system 30 according to the third embodiment of the present invention may include an image learning module 100 and an anomaly detection module 400 . Here, the image anomaly detection system according to the third embodiment may detect an abnormality in a color image obtained from an object. The color image may be composed of a red channel, a green channel, and a blue channel.

In the image anomaly detection system 30 according to the third embodiment, the image learning module 100 may be the same as the image learning module 100 according to the first embodiment. That is, the image anomaly detection system 30 according to the third embodiment may be different from the image anomaly detection system 10 according to the first embodiment only in the anomaly detection module 400 . Therefore, in the third embodiment, a detailed description of the image learning module 100 will be omitted.

The abnormality detection module 200 according to the third embodiment may receive an original image and a reconstruction image corresponding to the original image from the image learning module 100, detect only an abnormal portion in the original image, and visualize and quantify it. Specifically, the anomaly detection module 400 may receive an original image and a reconstruction image in which the original image is output through the image learning module 100, detect only an abnormal portion in the original image, and generate a visualized abnormal image. In addition, the abnormality detection module 400 may generate an abnormality value obtained by digitizing the abnormality image so that the user can easily determine whether the abnormality is normal or abnormal based on the abnormality image. That is, the anomaly detection module 400 may serve to objectively, quickly and accurately detect a defect of an object from the original image and the reconstructed image. At this time, as the color image obtained from the object is used as the original image, the anomaly detection module 400 includes a differential image generator 410 , a pooling image generator 420 , and a fusion image generator (Blended image generator 430), attention image generator 440, image calculation unit 450, anomaly image generator 460 and anomaly score calculation unit 470).

The differential image generator 410 may serve to generate a differential image by receiving the original image and the reconstructed image and calculating the structural similarity between the original image and the reconstructed image. At this time, the structural similarity between the original image and the reconstructed image is calculated using SSIM (Structural Similarity Index Map), which can calculate the difference between the luminance (l), contrast (c), and structure (s) between the two images. can be calculated using The original image input to the differential image generator 410 may include a test image and a plurality of normal images.

Meanwhile, the differential image generator 410 may further include a clustering unit 412 for clustering similar differential images in a plurality of differential images corresponding to each of the plurality of original images and assigning a weight to each other. . The distribution of each of the plurality of original images input to the differential image generating unit 410 and the plurality of reconstructed images corresponding thereto is varied, or each of the plurality of original images and the plurality of reconstructed images corresponding thereto is the fineness of the object. When it corresponds to the (micro) region, the performance of the anomaly detection module 400 may be improved by performing clustering. The optimal number of clusters for multiple differential images is dimensionality reduction using Kernel PCA (Kernel principal component analysis) of Radial Basis Function (RBF), clustering of differential images using Gaussian Mixture Model (GMM), and BIC ( Based on the Bayesian Information Criterion), it can be calculated through a series of processes to identify the optimal cluster. On the other hand, the distribution of each of the plurality of original images input to the differential image generator 410 and the plurality of reconstructed images corresponding thereto is not varied, and in a certain case, a plurality of differential images corresponding to each of the plurality of original images It is also possible to omit the clustering of the fields.

The pooling image generator 420 may play a role of receiving a differential image and generating a pooling image. In this case, the differential image input to the pooling image generator 420 to generate the pooling image may be a differential image corresponding to the test image. Specifically, the pooling image generator 420 includes an average pooling image generator 422 , a max pooling image generator 424 , and a standard deviation pooling image generator. generator, 426).

The average pooling image generation unit 422 receives the differential image and averages the difference for each color, that is, the difference for each channel in pixel unit to generate an average pooling image in which common spatial information is reflected in all channels. have. The maximum pooling image generator 424 may generate a maximum pooling image by receiving a differential image and reflecting a value having the greatest difference in each channel in pixel units as a representative value. And, the standard deviation pooling image generating unit 426 calculates the standard deviation value between each channel so as to detect a large change in color and reflects the difference between the channels. A standard deviation pooling image. can create Meanwhile, in each of the average pooling image generation unit 422, the maximum pooling image generation unit 424, and the standard deviation pooling image generation unit 426, a channel refers to a unit color and includes a red channel, a green channel, and a blue channel. can

The fusion image generator 430 receives the pooled images generated by the pooling image generator 420, that is, the average pooled image, the maximum pooled image, and the standard deviation pooled image, and fuses them to generate a blended image. can In this case, the fusion image generator 430 may generate a fusion image by adding weights to each of the average pooled image, the maximum pooled image, and the standard deviation pooled image, and then summed them. Here, the weight may be an arbitrary set value of the user. That is, according to the user's arbitrary setting, the fusion image may be the sum of all of the average pooled image, the maximum pooled image, and the standard deviation pooled image at a predetermined ratio, or two selected images among them are summed at a predetermined ratio. it might be In addition, the fusion image may be any one of an average pooled image, a maximum pooled image, and a standard deviation pooled image.

The attention image generator 440 is to reflect a variation appearing among a plurality of normal images, and may serve to detect only an abnormal portion, excluding a normal portion. Specifically, the attention image generating unit 440 generates an attention map reflecting the location information to be focused on when detecting anomalies according to the coordinate information in physical space, and reflects the weight of the attention map generated in the input fusion image to generate the attention image. It can play a role in creating That is, the attention image may be generated by multiplying the fusion image corresponding to the input test image by the weight of the attention map. The attention map generated by the attention image generator 440 may be generated by receiving a plurality of differential images from the differential image generator 410 and averaging the plurality of differential images in units of pixels. Here, the plurality of differential images for generating the attention map may be those calculated through SSIM from a plurality of normal images used for learning the image learning module 100 and a plurality of corresponding reconstruction images.

The image operation unit 450 may serve to generate an operation image whose magnification is adjusted for the attention image. In order to adjust the magnification of the attention image, the image operation unit 450 may include a multiplier, and the operation image may include a multiplied image in which the magnification of the attention image is adjusted.

The abnormal image generating unit 460 may serve to generate an abnormal image in which only the abnormal portion is emphasized by receiving the multiplied image and comparing the multiplied image with a first reference value preset in units of pixels. In this case, the abnormal image generator 460 may generate an abnormal image in which only the abnormal portion is emphasized by changing the pixel value of the pixel that is less than or equal to the first reference value in the multiplied image to a maximum value or a minimum value. That is, in the abnormal image output from the abnormal image generating unit 460 , only an abnormal portion may be shaded.

The abnormality calculation unit 470 is for improving the convenience of management, and may receive an abnormal image and compare it with a second reference value preset in units of pixels to express the degree of image abnormality numerically. In this case, the abnormal value calculating unit 470 may calculate an abnormal value by counting only pixels equal to or greater than the second reference value in the abnormal image, and dividing the counted number of pixels by the total number of pixels. Here, when the average value of the abnormal image is simply used when calculating the abnormal value, the discrimination power may be reduced when there is a singularity and when there is a slight difference as a whole. In order to prevent this, the abnormal value calculation unit 470 according to the third embodiment may introduce a concept of density and assign weights to the abnormal value when calculating the abnormal value. In this case, the density may refer to calculating the number of pixels counted on the abnormal image, that is, the number of pixels equal to or greater than the second reference value, as the total number of pixels. Through this, it is possible to improve the discriminating power of the abnormal value.

Meanwhile, the first reference value and the second reference value preset in the abnormality image generating unit 460 and the abnormality numerical calculation unit 470 may be designated as arbitrary values desired by the user within the range of 0 to 1, respectively. Since normality or abnormality changes depending on the reference value, if the reference value is set low, even a small abnormality can be detected and operated sensitively. can be established

As described above, the image abnormality detection system 30 according to the third embodiment of the present invention learns a normal part from an original image obtained from an object, detects only an abnormal part, visualizes and digitizes it, and provides it to the user. Defects can be effectively detected. In addition, it is possible to track a chronic defect of an object, and identify the root cause of the defect and provide the cause. In addition, by providing an index for judging a good product and a defective product in the object manufacturing process, the efficiency of the manufacturing process can be improved, and the quality and yield of the object can be improved.

4 is a block diagram illustrating the configuration of an image anomaly detection system according to a fourth embodiment of the present invention. Hereinafter, for convenience of description, the same reference numerals are used for the same components as those of the second and third embodiments, and detailed descriptions thereof will be omitted.

As shown in FIG. 4 , the image anomaly detection system 40 according to the fourth embodiment of the present invention may include an image learning module 300 and an anomaly detection module 400 . Here, the image abnormality detection system according to the fourth embodiment may detect abnormality in a color image obtained from an object. The color image may be composed of a red channel, a green channel, and a blue channel.

The image learning module 300 in the image anomaly detection system 40 according to the fourth embodiment may be the same as the image learning module 300 of the image anomaly detection system 20 according to the second embodiment. In addition, the abnormality detection module 400 in the image abnormality detection system 40 according to the fourth embodiment may be the same as the abnormality detection module 400 of the image abnormality detection system 30 according to the third embodiment. That is, an image anomaly detection system having various structures can be implemented through a combination of the above-described embodiments.

Although the present invention has been described in detail with reference to preferred embodiments, the present invention is not limited to the above embodiments, and various modifications are possible by those skilled in the art within the scope of the technical spirit of the present invention. do.

10, 20, 30, 40: Image anomaly detection system
100, 300: image learning module 200, 400: anomaly detection module
110, 320: encoder network 120, 330: decoder network
130: prior distribution unit 140: discriminant network
210, 410: differential image generator 212, 412: clustering unit
220: average map generation unit 230: attention map generation unit
240, 450: image operation unit 250, 460: abnormal image generation unit
260, 470: abnormal value calculation unit 310: noise adding unit
420: pooling image generating unit 430: fusion image generating unit
440: attention image generator

Claims (28)

an image learning module based on an autoencoder algorithm that receives an original image obtained from an object and outputs a reconstructed image;
a differential image generating unit that receives the original image and the reconstructed image and generates a differential image by calculating the structural similarity between the original image and the reconstructed image;
an average map generator that receives the original image and generates an average map reflecting common spatial information;
an attention map generation unit receiving the differential image and generating an attention map reflecting location information to be focused on when detecting an anomaly according to coordinate information in physical space;
an image operation unit configured to receive the differential image, the average map, and the attention map and generate a calculated image;
an abnormal image generating unit that receives the calculated image and generates an abnormal image in which only an abnormal portion is emphasized; and
An abnormality numerical calculation unit that receives the abnormal image and generates an abnormality value obtained by quantifying the abnormality of the abnormal image
An image anomaly detection system comprising a. According to claim 1,
The differential image generator further includes a clustering unit configured to perform clustering on a plurality of the differential images. According to claim 1,
The original image is an image anomaly detection system including a grayscale image. According to claim 1,
The original image includes a normal image obtained from the object of good quality, and the image learning module is an image abnormality detection system that is learned by using a plurality of the normal images. According to claim 1,
The image learning module is
an encoder network that receives the original image and generates a latent vector; and
A decoder network that has a symmetric structure with the encoder network and generates the reconstructed image having the same size as the original image by receiving the latent vector
An image anomaly detection system comprising a. 6. The method of claim 5,
The image learning module is
a pre-distribution unit that generates a real sample from a preset probability distribution; and
A discrimination network that receives the real sample and the fake sample generated by the encoder network at random and trains itself and the encoder network based on the discrimination loss to distinguish whether the input sample is real or fake
An image anomaly detection system comprising a. According to claim 1,
The image learning module is
a noise adding unit that receives the original image and adds randomly generated noise to the original image to generate a noise-added image;
an encoder network including a plurality of serially connected layers and generating a latent vector by receiving the noise-added image; and
A decoder network that includes a plurality of layers connected in series, has a symmetric structure with the encoder network, and generates the reconstructed image having the same size as the original image by receiving the latent vector
An image anomaly detection system comprising a. According to claim 1,
Structural similarity between the original image and the reconstructed image in the differential image generator is an image anomaly detection system for calculating using a Structural Similarity Index Map (SSIM). According to claim 1,
The original image includes a normal image obtained from the non-defective object, and the average map generator generates the average map by averaging a plurality of the normal images in units of pixels. According to claim 1,
The original image includes a normal image obtained from the non-defective object, and the attention map generator generates the attention map by averaging the plurality of differential images corresponding to each of the plurality of normal images in units of pixels. detection system. According to claim 1,
The original image includes a test image obtained from the object in order to detect a defect of the object, and a normal image obtained from a good object,
The computed image includes a weighted summation image, wherein the weighted summation image includes the differential image corresponding to the test image, the average map computed from a plurality of the normal images, and the attention map computed from a plurality of the normal images, respectively. An image anomaly detection system that generates weights by weighting and summing them. According to claim 1,
The abnormal image generating unit compares the calculated image with a first reference value preset in units of pixels, and changes the pixel value of a pixel that is less than or equal to the first reference value to a maximum value or a minimum value to generate the abnormal image. According to claim 1,
The abnormal value calculating unit compares the abnormal image with a second reference value preset in units of pixels, counts only pixels that are equal to or greater than the second reference value, and divides the number of counted pixels by all pixels of the abnormal image to calculate the abnormal value Image Anomaly Detection System. an image learning module based on an autoencoder algorithm that receives an original image obtained from an object and outputs a reconstructed image;
a differential image generating unit that receives the original image and the reconstructed image and generates a differential image by calculating the structural similarity between the original image and the reconstructed image;
a pooling image generator which receives the differential image and generates an average pooled image, a maximum pooled image, and a standard deviation pooled image, respectively;
a fusion image generation unit generating a fusion image by summing the average pooled image, the maximum pooled image, and the standard deviation pooled image;
an attention image generating unit generating an attention map reflecting location information to be focused upon when detecting anomalies according to coordinate information in physical space, and generating an attention image by reflecting the generated attention map to the fusion image;
an image calculating unit generating a calculated image in which the magnification of the attention image is adjusted;
an abnormal image generating unit that receives the calculated image and generates an abnormal image in which only an abnormal portion is emphasized; and
An abnormality calculation unit for receiving the abnormal image and generating an abnormality value obtained by quantifying the abnormality of the abnormal image
An image anomaly detection system comprising a. 15. The method of claim 14,
The differential image generator further includes a clustering unit configured to perform clustering on a plurality of the differential images. 15. The method of claim 14,
The original image is an image anomaly detection system including a color image. 15. The method of claim 14,
The original image includes a normal image obtained from the object of good quality, and the image learning module is an image abnormality detection system that is learned by using a plurality of the normal images. 15. The method of claim 14,
The image learning module is
an encoder network that receives the original image and generates a latent vector; and
A decoder network that has a symmetric structure with the encoder network and generates the reconstructed image having the same size as the original image by receiving the latent vector
An image anomaly detection system comprising a. 19. The method of claim 18,
The image learning module is
a pre-distribution unit that generates a real sample from a preset probability distribution; and
A discrimination network that receives the real sample and the fake sample generated by the encoder network at random and trains itself and the encoder network based on the discrimination loss to distinguish whether the input sample is real or fake
An image anomaly detection system comprising a. 15. The method of claim 14,
The image learning module is
a noise adding unit that receives the original image and adds randomly generated noise to the original image to generate a noise-added image;
an encoder network including a plurality of serially connected layers and generating a latent vector by receiving the noise-added image; and
A decoder network that includes a plurality of layers connected in series, has a symmetric structure with the encoder network, and generates the reconstructed image having the same size as the original image by receiving the latent vector
An image anomaly detection system comprising a. 15. The method of claim 14,
Structural similarity between the original image and the reconstructed image in the differential image generator is an image anomaly detection system for calculating by using a Structural Similarity Index Map (SSIM). 15. The method of claim 14,
The original image includes a test image obtained from the object to detect a defect of the object, and each of the average pooled image, the maximum pooled image, the standard deviation pooled image and the fusion image corresponds to the test image. Image Anomaly Detection System. 15. The method of claim 14,
The fusion image is an image anomaly detection system generated by adding weights to each of the average pooled image, the maximum pooled image, and the standard deviation pooled image, and then summed them. 15. The method of claim 14,
The original image includes a normal image obtained from the non-defective object, and the attention image generating unit generates the attention map by averaging the plurality of differential images corresponding to each of the plurality of normal images in units of pixels. detection system. 25. The method of claim 24,
The image anomaly detection system for the attention image generating unit generates the image of the attention by multiplying the fusion image with the weighted attention map. 15. The method of claim 14,
The image operation unit includes a multiplier, and the operation image is an image anomaly detection system including a multiplied image. 15. The method of claim 14,
The abnormal image generating unit compares the calculated image with a first reference value preset in units of pixels, and changes the pixel value of a pixel that is less than or equal to the first reference value to a maximum value or a minimum value to generate the abnormal image. 15. The method of claim 14,
The abnormal value calculating unit compares the abnormal image with a second reference value preset in units of pixels, counts only pixels that are equal to or greater than the second reference value, and divides the number of counted pixels by all pixels of the abnormal image to calculate the abnormal value Image Anomaly Detection System.

Images