KR102475851B1 - System and method of defect inspection using generation and transformation of input and output data based on deep learning - Google Patents

Abstract

The present invention relates to a deep learning-based product defect inspection system and method using input/output data generation and transformation that can effectively solve the shortage of original data, and when an original image is input, a plurality of restored images identical or similar to the original image. an interpolation image generation module for generating an interpolation image and generating an interpolation image using the reconstructed image; and a defect inspection module configured to classify product defects by learning the reconstructed image and the interpolated image using a deep learning method.

Description

System and method of defect inspection using generation and transformation of input and output data based on deep learning}

The present invention relates to a deep learning-based product defect inspection system using input/output data generation and transformation, and more particularly, a deep learning-based product defect inspection system using input/output data generation and transformation that can effectively solve the shortage of original data. and methods.

Recently, various technologies for smart factory implementation have been intensively researched. For example, in order to implement a smart factory, various process data such as raw material amount, material quality, manufacturing process data (temperature, humidity, etc.), and facility operating conditions are required, and the resulting product quality data must be databased. That is, it is necessary to predict quality by analyzing the correlation between input and output of the factory, prevent mass defects in advance, and predict facility failures in advance to respond preemptively.

To implement such a smart factory, a defect inspection system that inspects product defects in real time and determines product quality is one of the core technologies.

In general, the configuration of a vision system for defect inspection is as follows.

It consists of three major components: an optical module composed of lighting and a camera that visualizes the state of the product, a recognition module that automatically determines the location and type of defects in the image, and a database that stores the product image and judgment results. can

In particular, a recognition module that determines the location and type of a defect in an image is a major factor influencing defect inspection performance. Conventionally, in order to determine normality and defect, the system was developed as an algorithm development method that extracts the main factor of the feature by relying on the rule-base method based on the developer's empirical judgment, but this method takes a long development period and , the disadvantage of having to develop a key factor extraction algorithm every time depending on the product and defect, and the consequent degradation in recognition performance. Therefore, in recent years, there is a tendency to develop recognition algorithms using deep learning technology that can independently extract features for defect detection and classification.

However, when deep learning technology, one of data-driven technologies, is applied to actual manufacturing, effective performance can be secured only when there is a sufficient amount of data. In practice, this happened frequently. This lack of data has often made it difficult to apply artificial intelligence to manufacturing. When learning is performed using such a small amount of data, an over-fitting problem occurs, which shows excellent performance for the learned data, but deteriorates performance when performing defect inspection using the actually learned model.

The present invention is intended to solve various problems, including the above problems, and is a groundbreaking technology that can secure excellent generalization performance with only a small amount of data. Defect features can be extracted by learning a large amount of restored and interpolated images, and interpolation images can be additionally generated from restored images to solve the data shortage problem, and a variety of and large amounts of restored and interpolated images can be quickly generated. It can be acquired within a short period of time, thereby preventing the shortage of original images, greatly improving the precision and accuracy of judgment for defect inspection and quality diagnosis of manufacturing products. As an essential technology for realizing a smart factory, input/output data generation that can be applied to technology that optimizes production conditions by determining the state of facilities according to production operating conditions and predicting product quality according to facility conditions And to provide a deep learning-based product defect inspection system and method using deformation. However, these tasks are illustrative, and the scope of the present invention is not limited thereby.

Deep learning-based product defect inspection system using input/output data generation and transformation according to the spirit of the present invention for solving the above problems generates a plurality of restored images identical or similar to the original image when an original image is input, and the restored image An interpolation image generation module for generating an interpolation image using an image; and a defect inspection module configured to classify product defects by learning the reconstructed image and the interpolated image using a deep learning method.

In addition, according to the present invention, the interpolation image generation module may include: a first encoder for mapping an input original image to a feature map; a first decoder for restoring the feature map into a first reconstruction image based on a first restoration jet value Z1 and restoring a second reconstruction image based on a second restoration jet value Z2; And the feature map is calculated by changing the variable (a) of the quantum interpolation equation N times by a quantum interpolation equation using both the first restoration jet value Z1 and the second restoration jet value Z2 A second decoder for restoring a plurality of N-th interpolated images based on a plurality of N-th interpolated jet values ZNnew.

Further, according to the present invention, the first restoration jet value and the second restoration jet value may be intermediate random hidden variables that are learned so that the first reconstruction image is generated identically or similarly to the original image.

Also, according to the present invention, the quantum interpolation equation may be ZNnew = a*Z1 + (1-a)*Z2, where 0 < a < 1).

In addition, according to the present invention, the new label Y according to the quantum interpolation equation is YNnew = a * Y1 + (1-a) * Y2, (where 0 < a < 1), [a, 1 Like -a, 0, 0, 0,,,]. It can be expressed as a soft label vector value in which weights for each class are reflected.

In addition, according to the present invention, the first encoder may include a convolution layer for extracting a feature map having a smaller size by convolving an input value with a filter; a batch normalization layer that generates normalized output values based on the mean or standard deviation of the batches; an activation function for additionally activating data after a convolution operation; and a pooling layer for subsampling data as much as the size of the pooling kernel in the feature map into data of a smaller size.

In addition, according to the present invention, the first decoder or the second decoder may include a deconvolution layer for extracting a feature map having a larger size by deconvoluting an input value with a filter; a batch normalization layer that generates normalized output values based on the mean or standard deviation of the batches; an activation function for additionally activating data after a convolution operation; and an unpooling layer upsampling data as large as the size of the pooling kernel in the feature map to data of a larger size.

In addition, according to the present invention, the defect inspection module extracts a defect feature map from a plurality of the reconstructed images and the interpolated image, including at least one or more of a convolution layer, a pooling layer, an activation function, and combinations thereof. a defect feature extraction unit; a defect restoration unit that restores the defect feature map into a defect restoration image, including at least one of a deconvolution layer, an unpooling layer, an activation function, and combinations thereof; and a classifier for classifying the defect restoration image according to defect types.

On the other hand, a deep learning-based product defect inspection method using input and output data generation and transformation according to the spirit of the present invention for solving the above problems is, (a) when an original image is input, a plurality of restored images identical or similar to the original image are input. generating, and generating an interpolation image using the reconstructed image; and (b) classifying defects of the product by learning the reconstructed image and the interpolated image using a deep learning method.

In addition, according to the present invention, the step (a) may include (a-1) mapping the input original image to a feature map; (a-2) restoring the feature map into a first reconstruction image based on a first restoration jet value Z1, and restoring a second reconstruction image based on a second restoration jet value Z2; and (a-3) a variable (a) of the quantum interpolation equation is set to N by a quantum interpolation equation using both the first restoration jet value Z1 and the second restoration jet value Z2 for the feature map. and restoring a plurality of N-th interpolated images based on the plurality of N-th interpolated jet values ZNnew calculated while changing the number of times.

In addition, according to the present invention, in the step (b), (b-1) a plurality of the reconstructed images and the interpolated images using at least one or more of a convolution layer, a pooling layer, an activation function, and combinations thereof extracting a defect feature map from; (b-2) restoring the defect feature map into a defect reconstructed image by including at least one or more of a deconvolution layer, an unpooling layer, an activation function, and combinations thereof; and (b-3) classifying the defect reconstruction image according to defect types.

According to various embodiments of the present invention made as described above, it is an innovative technology capable of securing excellent generalization performance with only a small amount of data, and a large amount of restoration by restoring a small amount of original image multiple times by itself using a deep learning method. Defect features can be extracted by learning images and interpolation images, and interpolation images can be additionally generated from restored images to solve the data shortage phenomenon, and various and large amounts of restored images and interpolated images can be acquired in a short time It can greatly improve the precision and accuracy of judgment for defect inspection and quality diagnosis of manufacturing products by preventing the shortage of original images, and can be used not only for product defect inspection but also for diagnostic fields such as non-destructive inspection. , As an essential technology for realizing a smart factory, it has an effect that can be applied to technology that optimizes production conditions by determining the state of facilities according to production operating conditions and predicting product quality according to facility conditions. Of course, the scope of the present invention is not limited by these effects.

1 is a block diagram conceptually illustrating a deep learning-based product defect inspection system using input/output data generation and transformation according to some embodiments of the present invention.
FIG. 2 is a conceptual diagram illustrating an interpolation image generating module of a deep learning-based product defect inspection system using input/output data generation and transformation of FIG. 1 .
FIG. 3 is a conceptual diagram showing an interpolation image generation module of the deep learning-based product defect inspection system using the generation and transformation of input/output data of FIG. 1 in more detail.
FIG. 4 is a conceptual diagram conceptually illustrating an example of a relationship between a first encoder, an Nth restoring jet value, and a first decoder of the deep learning-based product defect inspection system using the generation and transformation of input/output data of FIG. 1 .
5 is a conceptual diagram conceptually illustrating a defect inspection module of a deep learning-based product defect inspection system using the generation and transformation of input/output data of FIG. 1 .
6 is a flowchart illustrating a deep learning-based product defect inspection method using input/output data generation and transformation according to some embodiments of the present invention.
7 is a flowchart illustrating a deep learning-based product defect inspection method using input/output data generation and transformation according to some other embodiments of the present invention.

Hereinafter, several preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

The embodiments of the present invention are provided to more completely explain the present invention to those skilled in the art, and the following examples may be modified in many different forms, and the scope of the present invention is as follows It is not limited to the examples. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the spirit of the invention to those skilled in the art. In addition, the thickness or size of each layer in the drawings is exaggerated for convenience and clarity of explanation.

Terms used in this specification are used to describe specific embodiments and are not intended to limit the present invention. As used herein, the singular form may include the plural form unless the context clearly indicates otherwise. Also, when used herein, "comprise" and/or "comprising" specifies the presence of the recited shapes, numbers, steps, operations, elements, elements, and/or groups thereof. and does not exclude the presence or addition of one or more other shapes, numbers, operations, elements, elements and/or groups.

Hereinafter, embodiments of the present invention will be described with reference to drawings schematically showing ideal embodiments of the present invention. In the drawings, variations of the depicted shape may be expected, depending on, for example, manufacturing techniques and/or tolerances. Therefore, embodiments of the inventive concept should not be construed as being limited to the specific shape of the region shown in this specification, but should include, for example, a change in shape caused by manufacturing.

1 is a block diagram conceptually showing a deep learning-based product defect inspection system 100 using input/output data generation and transformation according to some embodiments of the present invention.

First, as shown in FIG. 1, the deep learning-based product defect inspection system 100 using input/output data generation and transformation according to some embodiments of the present invention includes an interpolation image generation module 10 and a defect inspection module. (20) may be included.

Here, for example, as shown in FIG. 1, the interpolation image generating module 10 generates a plurality of reconstructed images 2 identical or similar to the original image 1 when the original image 1 is input, and , A module for generating an interpolated image 3 using the reconstructed image 2. For example, the interpolated image generation module 10 can operate in two modes: a machine learning mode and an object detection mode. .

More specifically, for example, in the machine learning mode, the interpolation image generating module 10 may perform learning on object classification through learning data (GTD, Ground Truth Data).

At this time, the interpolation image generation module 10 includes a neural network capable of learning about an image, for example, a convolutional neural network (CNN) to perform machine learning and classification of a specific entity in the image. can do.

In addition, the learning data may machine-learn the entity classification module through labeled ground truth data (GTD). For example, the training data may include a defect image associated with a plurality of classification names of a specific entity.

In addition, for example, as shown in FIG. 1, the defect inspection module 20 is a module for classifying product defects by learning the reconstructed image 2 and the interpolated image 3 using a deep learning method, For example, the defect inspection module 20 may operate in two modes: a machine learning mode and an object detection mode.

For example, in the machine learning mode, the defect inspection module 20 may perform learning about object classification through learning data (GTD, Ground Truth Data).

At this time, the defect inspection module 20 includes a neural network capable of learning about an image, for example, a convolutional neural network (CNN) to perform machine learning and classification of a specific object in an image. can

In addition, the learning data may machine-learn the entity classification module through labeled ground truth data (GTD). For example, the training data may include a defect image associated with a plurality of classification names of a specific entity.

FIG. 2 is a conceptual diagram illustrating an interpolation image generating module 10 of the deep learning-based product defect inspection system 100 using the input/output data generation and transformation of FIG. 1 .

More specifically, for example, as shown in FIG. 2, the interpolation image generation module 10, as a deep learning model in the form of a variational auto-encoder, features an input original image. A first encoder (E1) that maps to a map, and the feature map is reconstructed into a first restored image based on the first restoration jet value (Z1), and based on the second restoration jet value (Z2), the second The first decoder (D1) for restoring the reconstructed image and the feature map by a quantum interpolation equation using both the first restoration jet value Z1 and the second restoration jet value Z2 It may include a second decoder D2 for restoring a plurality of N-th interpolated images based on a plurality of N-th interpolated jet values ZNnew calculated by changing the variable a N times.

Here, N is a positive integer, and the first restoration jet value Z1 and the second restoration jet value Z2 are learned so that the first reconstruction image is generated identically or similarly to the original image. Each random hidden variable, and the quantum interpolation equation is

ZNnew = a*Z1 + (1-a)*Z2, where 0 < a < 1)

can be

At this time, the new label Y according to the quantum interpolation equation,

YNnew = a*Y1 + (1-a)*Y2, (where 0 < a < 1), as in [a, 1-a, 0, 0, 0,,,]. It can be expressed as a soft label vector value in which weights for each class are reflected.

That is, the new label vector value is not an existing hard label, that is, one-hot encoding in which class is 1 and the rest is 0, but, for example, a less than 1 (class-specific It can be a soft label vector value, such as [0.3, 0.7, 0, 0, 0] or [0.9, 0.1, 0, 0, 0], which represents a classification variable that is a weight) as a class.

Therefore, it is possible to learn a deep learning model so that the input image and the output image are the same or similar using a variational auto-encoder structure composed of an encoder and a decoder, After removing the encoder part from the learned variational autoencoder structure, a large number of data is generated by inputting various jet values (ZN) for restoration to the decoder part, and through this, big data to be input to the defect inspection model is generated. can do.

Here, the jet value may generate various interpolated images by interpolating the jet value using a quantum interpolation formula without using any jet value, for example, a combination of Z1, Z2, ... ZN. When creating a new ZNnew using , each Z1, Z2. ... , a label Y corresponding to ZN can also be interpolated to create a new Ynew.

By doing this, it is possible to implement a deep learning model that has excellent generalization performance by giving weights to labels as well as data augmentation effect. Therefore, a new image Xnew can be generated by inputting the new ZNnew to a decoder, and a large amount of big data paired with a new Ynew, which is a label of Xnew, can be generated.

FIG. 3 is a conceptual diagram showing the interpolation image generation module 10 of the deep learning-based product defect inspection system 100 using the input/output data generation and transformation of FIG. 1 in more detail.

More specifically, for example, as shown in FIG. 3, the first encoder E1 is a convolution layer 11 (convolution layer 11) that extracts a feature map of a smaller size by convolving an input value with a filter. ), a batch normalization layer (12) that generates normalized output values based on the average or standard deviation of the batch, and an activation function (13) that additionally activates data after the convolution operation (activation function) and a pooling layer 14 for subsampling data as much as the size of the pooling kernel in the feature map into data of a smaller size.

In addition, the first decoder (D1) or the second decoder (D2) deconvolution layer (15) (deconvolution layer) for extracting a feature map of a larger size by deconvolution of the input value with a filter; A batch normalization layer (16) that generates normalized output values based on the average or standard deviation of the batch, an activation function (17) that additionally activates the data after the convolution operation, and a feature map may include an unpooling layer 18 (unpooling layer) for upsampling data equal to the size of the pooling kernel to data of a larger size.

For example, as shown in FIG. 3 , the convolution layer 11 of the first encoder E1 may include a first convolution layer and a second convolution layer. In this case, the second convolution layer is behind the first convolution layer. That is, the second convolution layer may use the output of the first convolution layer.

More specifically, the first convolution layer may extract a feature map based on a convolution filter kernel having a size smaller than that of the input original image. For example, the convolution filter kernel is a matrix having a smaller size than the original size. In this case, the first convolution layer may extract a feature map by multiplying the original image while moving the convolution filter kernel. At this time, after completing the convolution operation, the activation function 13, for example, ReLU, may be additionally passed to the batch normalization layer 12 that generates a normalized output value or to activate data.

After passing through the convolution layer 11, it may pass through the pooling layer 14 to reduce the amount of data and computation.

In this case, the pooling layer 14 may subsample data as large as the size of a pooling kernel (for example, n x m) in the feature map into data of a smaller size.

In this way, the CNN of the first encoder E1 may perform an operation by repeatedly passing the plurality of convolution layers 11 and the pooling layer 14 .

In addition, for example, as shown in FIG. 3, the first decoder D1 deconvolution layer 15 (deconvolution layer) for extracting a feature map of a larger size by deconvolution of the input value with a filter And, a batch normalization layer 16 that generates normalized output values based on the mean or standard deviation of the batch, and an activation function 17 that additionally activates data after the convolution operation (activation function) and an unpooling layer 18 (unpooling layer) for upsampling data as large as the size of the pooling kernel in the feature map to data of a larger size.

Also, for example, the deconvolution layer 15 of the first decoder D1 may include a first deconvolution layer and a second deconvolution layer. In this case, the second deconvolution layer is behind the first deconvolution layer. That is, the second deconvolution layer may use the output of the first deconvolution layer.

More specifically, the first deconvolution layer may extract a feature map having a larger size than the size of the input feature map. In this case, the first deconvolution layer may extract a feature map by multiplying the original image while moving the deconvolution filter kernel. At this time, after completing the deconvolution operation, the activation function 17, for example, ReLU, may be additionally passed to the batch normalization layer 16 that generates a normalized output value or to activate data.

After passing through the deconvolution layer, it may pass through the unpooling layer 18 to increase the amount of data and computation.

In this case, the unpooling layer 18 may upsample data corresponding to the size of the unpooling kernel in the feature map to data of a larger size.

As such, the CNN of the first decoder D1 may perform an operation by repeatedly passing the plurality of deconvolution layers 15 and the unpooling layer 18.

FIG. 4 shows the relationship between the first encoder E1, the Nth restoring jet value ZN, and the first decoder D1 of the deep learning-based product defect inspection system 100 using the input/output data generation and transformation of FIG. 1 It is a conceptual diagram conceptually showing an example of

As shown in FIGS. 2 to 4 , the first reconstruction jet value Z1 and the second restoration jet value Z2 are generated such that the first reconstruction image is identical to or similar to the original image. An intermediate random hidden variable to be learned, and the Nth interpolation jet value ZNnew may be an intermediate random hidden variable determined by the first restoring jet value Z1 and the second restoring jet value Z2. .

Therefore, as shown in FIG. 4, these intermediate random hidden variables (latent variables) are learning to be output by varying the weight in the process of converting the input data between the input layer and the output layer. As a variable, the present invention is a plurality of reconstructed images (which are identical or similar to the original image (1) reconstructed from the same original image (1) by varying the intermediate random hidden variable between the encoder and the decoder ( 2) and interpolated images 3 derived from these reconstructed images.

Therefore, it is possible to solve the data shortage phenomenon by generating a plurality of reconstructed images and interpolated images in large quantities from one original image, and to drastically reduce the time of labeling work, and to reduce label errors that may occur during labeling work. (label error) phenomenon can be drastically reduced, and various and large amounts of restored images and interpolated images can be acquired in a short time, preventing the lack of original images, thereby improving the accuracy of defect inspection and quality diagnosis of manufacturing products. and the accuracy of judgment can be greatly improved.

FIG. 5 is a conceptual diagram conceptually illustrating the defect inspection module 20 of the deep learning-based product defect inspection system 100 using the generation and transformation of input/output data of FIG. 1 .

On the other hand, more specifically, for example, as shown in FIGS. 1 and 5, the defect inspection module 20 includes at least a convolution layer 211, a pooling layer 212, an activation function 213, and these A defect feature extractor 21 extracting a defect feature map from a plurality of reconstructed images and the interpolated image including any one or more combinations of, at least a deconvolution layer 221 and an unpooling layer 222 , activation function 223, a defect restoration unit 22 that restores the defect feature map into a defect restoration image by including any one or more of these combinations, and a classifier 23 that classifies the defect restoration image by defect type. can include

Here, for example, the convolution layer 211 (convolution layer) of the defect feature extraction unit 21 is a layer capable of extracting a feature map of a smaller size by convolving an input value with a filter, and the pooling The layer 212 is a layer capable of subsampling data as much as the size of the pooling kernel in the feature map into data of a smaller size, and the activation function 213 may be a function that additionally activates data after the convolution operation. .

In addition, for example, the deconvolution layer 221 of the defect restoration unit 22 is a layer capable of extracting a feature map of a larger size by deconvolution of an input value with a filter, and the unpooling layer ( 222) is a layer capable of upsampling data as large as the size of the pooling kernel in the feature map to data of a larger size, and the activation function 223 may be a function that additionally activates data after the convolution operation.

For example, as shown in FIG. 5 , the convolution layer 211 of the defect feature extractor 21 may include a first convolution layer and a second convolution layer. In this case, the second convolution layer is behind the first convolution layer. That is, the second convolution layer may use the output of the first convolution layer.

More specifically, the first convolution layer may extract a feature map based on a convolution filter kernel having a size smaller than that of the input original image. For example, the convolution filter kernel is a matrix having a smaller size than the original size. In this case, the first convolution layer may extract a feature map by multiplying the original image while moving the convolution filter kernel. At this time, after completing the convolution operation, the activation function 213, for example, ReLU, may be additionally passed to activate data.

After passing through the convolution layer, it may pass through the pooling layer 212 to reduce the amount of data and computation.

In this case, the pooling layer 212 may subsample data as large as the size of a pooling kernel (for example, n x m) in the feature map into data of a smaller size.

In this way, the CNN of the defect feature extraction unit 21 may perform an operation by repeatedly passing a plurality of convolution layers 211 and a pooling layer 212 .

Also, for example, the deconvolution layer 221 of the defect restoration unit 22 may include a first deconvolution layer and a second deconvolution layer. In this case, the second deconvolution layer is behind the first deconvolution layer. That is, the second deconvolution layer may use the output of the first deconvolution layer.

More specifically, the first deconvolution layer may extract a feature map having a larger size than the size of the input feature map. In this case, the first deconvolution layer may extract a feature map by multiplying the original image while moving the deconvolution filter kernel. At this time, after completing the deconvolution operation, the activation function 223, for example, ReLU, may be additionally passed to activate data.

After passing through the deconvolution layer, it may pass through the unpooling layer 222 to increase the amount of data and computation.

In this case, the unpooling layer 222 may upsample data corresponding to the size of the unpooling kernel in the feature map to data of a larger size.

In this way, the CNN of the defect restoration unit 22 may perform an operation by repeatedly passing a plurality of deconvolution layers 221 and an unpooling layer 222 .

Therefore, by using the defect feature extraction unit 21, the defect restoration unit 22, and the classifier 23, the above-described reconstructed images and the interpolated images are repeatedly learned to finally classify product defects. It can be discriminated with high accuracy and precision.

6 is a flowchart illustrating a deep learning-based product defect inspection method using input/output data generation and transformation according to some embodiments of the present invention.

As shown in FIGS. 1 to 6, in the deep learning-based product defect inspection method using input/output data generation and transformation according to some embodiments of the present invention, (a) when an original image 1 is input, the original image Generating a plurality of reconstructed images (2) identical or similar to (1), and generating an interpolated image (3) using the reconstructed image (2), and (b) the reconstructed image (2) and the interpolated image It may include a step of classifying product defects by learning (3) in a deep learning method.

7 is a flowchart illustrating a deep learning-based product defect inspection method using input/output data generation and transformation according to some other embodiments of the present invention.

1 to 7, a deep learning-based product defect inspection method using input/output data generation and transformation according to some other embodiments of the present invention, (a) when an original image 1 is input, the original generating a plurality of reconstructed images (2) identical to or similar to image (1) and generating an interpolated image (3) using the reconstructed image (2); and (b) the reconstructed image (2) and the interpolation It may include the step of classifying defects of the product by learning the image (3) using a deep learning method, wherein the (a) step is to map the input original image (1) to a feature map (a-2) restoring the feature map into a first reconstruction image based on the first restoration jet value Z1, and restoring the feature map into a second reconstruction image based on the second restoration jet value Z2 and (a-3) a variable (a) of the quantum interpolation equation by a quantum interpolation equation using both the first restoration jet value Z1 and the second restoration jet value Z2 for the feature map. It may include a step of restoring a plurality of N th interpolated images based on a plurality of N th interpolation jet values ZNnew calculated by changing N times, wherein the step (b) comprises: (b-1) at least convolution Defect feature maps are extracted from the plurality of reconstructed images 2 and the interpolated images 3 using one or more of a solution layer 211, a pooling layer 212, an activation function 213, and combinations thereof. (b-2) converting the defect feature map into a defect reconstruction image, including at least one of a deconvolution layer 221, an unpooling layer 222, an activation function 223, and combinations thereof; and (b-3) classifying the defect restoration image 2 according to defect types.

Therefore, even under circumstances where it is difficult to obtain original images, various and large amounts of reconstructed images and binary images can be acquired in a short time, thereby preventing the lack of original images, thereby improving the accuracy and quality of defect inspection and quality diagnosis of manufacturing products. It can greatly improve the accuracy of judgment, and can be used not only for product defect inspection, but also for diagnostic fields such as non-destructive inspection. It can also be applied to technology that optimizes production conditions by predicting product quality according to facility conditions.

Although the present invention has been described with reference to the embodiments shown in the drawings, this is merely exemplary, and those skilled in the art will understand that various modifications and equivalent other embodiments are possible therefrom. Therefore, the true technical protection scope of the present invention should be determined by the technical spirit of the appended claims.

1: original video
2: Restoration image
3: interpolated image
10: interpolation image generation module
E1: first encoder
D1: first decoder
D2: second decoder
Z1: jet value for first restoration
Z2: jet value for second restoration
ZNnew: Nth interpolation jet value
a: variable
11: Convolution layer
12: Batch normalization layer
13: activation function
14: pooling layer
15: deconvolution layer
16: Batch normalization layer
17: Activation function
18: unpooling layer
20: defect inspection module
21: defect feature extraction unit
211: convolution layer
212: pooling layer
213: activation function
22: defect restoration unit
221: deconvolution layer
222: unpooling layer
223: activation function
23: classifier
100: Deep learning-based product defect inspection system

Claims (11)

an interpolation image generation module for generating a plurality of reconstructed images identical to or similar to the original image when an original image is input, and generating an interpolated image using the reconstructed image; and
A defect inspection module for classifying defects of a product by learning the reconstructed image and the interpolated image using a deep learning method;
The interpolation image generation module,
A first encoder for mapping an input original image to a feature map;
a first decoder for restoring the feature map into a first reconstruction image based on a first restoration jet value Z1 and restoring a second reconstruction image based on a second restoration jet value Z2; and
The feature map is calculated by changing the variable (a) of the quantum interpolation equation N times by a quantum interpolation equation using both the first restoration jet value Z1 and the second restoration jet value Z2. a second decoder for restoring a plurality of N-th interpolated images based on the N-th interpolated jet values ZNnew;
Including, deep learning-based product defect inspection system using input and output data generation and transformation. delete According to claim 1,
The first restoring jet value and the second restoring jet value are intermediate random hidden variables that are learned so that the first restored image is generated identically or similarly to the original image. Deep learning using input/output data generation and transformation Based product defect inspection system. According to claim 3,
The quantum interpolation equation,
ZNnew = a*Z1 + (1-a)*Z2, where 0 < a < 1)
Deep learning-based product defect inspection system using input/output data generation and transformation. According to claim 4,
The new label Y according to the quantum interpolation equation,
YNnew = a*Y1 + (1-a)*Y2, where 0 < a < 1),
Like [a, 1-a, 0, 0, 0,,,]. Deep learning-based product defect inspection system using input/output data generation and transformation, represented by soft label vector values with weights for each class reflected. According to claim 1,
The first encoder,
a convolution layer that extracts a feature map of a smaller size by convolving input values with a filter;
a batch normalization layer that generates normalized output values based on the mean or standard deviation of the batches;
an activation function for additionally activating data after a convolution operation; and
a pooling layer for subsampling data as much as the size of the pooling kernel in the feature map into data of a smaller size;
Including, deep learning-based product defect inspection system using input and output data generation and transformation. According to claim 1,
The first decoder or the second decoder,
a deconvolution layer for extracting a feature map of a larger size by deconvoluting input values with a filter;
a batch normalization layer that generates normalized output values based on the mean or standard deviation of the batches;
an activation function for additionally activating data after a convolution operation; and
an unpooling layer upsampling data as large as the size of the pooling kernel in the feature map to data of a larger size;
Including, deep learning-based product defect inspection system using input and output data generation and transformation. According to claim 1,
The defect inspection module,
a defect feature extractor extracting a defect feature map from the plurality of reconstructed images and the interpolated image including at least one of a convolution layer, a pooling layer, an activation function, and combinations thereof;
a defect restoration unit that restores the defect feature map into a defect restoration image, including at least one of a deconvolution layer, an unpooling layer, an activation function, and combinations thereof; and
a classifier for classifying the defect restoration image according to defect types;
Including, deep learning-based product defect inspection system using input and output data generation and transformation. (a) generating a plurality of restored images identical to or similar to the original image when an original image is input, and generating an interpolated image using the restored image; and
(b) classifying defects of the product by learning the reconstructed image and the interpolated image using a deep learning method;
In step (a),
(a-1) mapping the input original image to a feature map;
(a-2) restoring the feature map into a first reconstruction image based on a first restoration jet value Z1, and restoring a second reconstruction image based on a second restoration jet value Z2; and
(a-3) By a quantum interpolation equation using both the first restoration jet value Z1 and the second restoration jet value Z2 for the feature map, the variable (a) of the quantum interpolation equation is changed N times. restoring a plurality of N-th interpolated images based on a plurality of N-th interpolated jet values ZNnew calculated while changing;
Including, deep learning-based product defect inspection method using input and output data generation and transformation. delete According to claim 9,
In step (b),
(b-1) extracting defect feature maps from the plurality of reconstructed images and the interpolated image using at least one or more of a convolution layer, a pooling layer, an activation function, and combinations thereof;
(b-2) restoring the defect feature map into a defect reconstructed image by including at least one or more of a deconvolution layer, an unpooling layer, an activation function, and combinations thereof; and
(b-3) classifying the defect restoration image according to defect types;
Including, deep learning-based product defect inspection method using input and output data generation and transformation.

Images