KR20200087297A - Defect inspection method and apparatus using image segmentation based on artificial neural network - Google Patents

Abstract

According to the present invention, a defect detection method using an image segmentation based on an artificial neural network comprises the steps of: inputting, by a computer device, a source image including a product object to a learning network; and outputting, by the computer device, a segmented image generated by the learning network. The learning network is a convolutional encoder-decoder, and outputs the segmented image obtained by classifying a defective part in the product object.

Description

DEFECT INSPECTION METHOD AND APPARATUS USING IMAGE SEGMENTATION BASED ON ARTIFICIAL NEURAL NETWORK

The technique described below relates to a defect detection technique based on image processing.

Machine vision (machine vision) has been widely studied because it can perform high-precision inspection repeatedly by replacing a manual inspector in applications such as defect inspection, classification, and recognition.

The machine learning technology, which has recently attracted attention, is actively being studied in applications where it is difficult to apply the existing machine vision technology because it can learn the judgment form of a manual inspector.

Korean Registered Patent No. 10-1688458

The technique described below is intended to provide a technique for performing product defect inspection by segmenting an image based on an artificial neural network.

A method of detecting a defect using image segmentation based on an artificial neural network includes inputting a source image including a product object into a learning network by a computer device and outputting a split image generated by the learning network by the computer device. The learning network is a convolutional encoder-decoder, and outputs the segmented image in which defective parts are separated from the product object.

A defect detection device using artificial neural network-based image segmentation includes an input device that receives a source image containing a product object, a storage device that stores a plurality of convolutional encoder-decoders that generate output values that distinguish defective areas from the input image, and The source image is input to each of the plurality of convolutional encoder-decoders to generate output values for each pixel of the source image, and each of the plurality of convolutional encoder-decoders outputs the output values for each pixel. And a computing device that classifies a final class for pixels and generates an output image based on the final class for each pixel.

The technique described below analyzes an image using a convolutional encoder-decoder and outputs an image indicating a defective area. The technique described below expresses a defective portion so that the worker can more conveniently check whether the product is defective or not.

1 is an example of a defect detection system using image segmentation.
2 is an example of a convolutional encoder-decoder.
3 is an example of a neural network model for segmenting an image.
4 is another example of a neural network model for segmenting an image.
5 is an example of a configuration of a defect detection apparatus using image segmentation.

The technique described below may be applied to various changes and may have various embodiments, and specific embodiments will be illustrated in the drawings and described in detail. However, this is not intended to limit the technology described below to specific embodiments, and should be understood to include all modifications, equivalents, or substitutes included in the spirit and scope of the technology described below.

Terms such as first, second, A, B, etc. can be used to describe various components, but the components are not limited by the above terms, and only for distinguishing one component from other components Used only. For example, the first component may be referred to as a second component, and similarly, the second component may be referred to as a first component without departing from the scope of the technology described below. The term and/or includes a combination of a plurality of related described items or any one of a plurality of related described items.

In the terminology used herein, a singular expression should be understood to include a plurality of expressions unless clearly interpreted differently in the context, and terms such as “comprises” describe features, numbers, steps, operations, and components described. It is to be understood that it means that a part or a combination thereof is present, and does not exclude the presence or addition possibility of one or more other features or numbers, step operation components, parts or combinations thereof.

Prior to the detailed description of the drawings, it is intended to clarify that the division of components in this specification is only divided by the main functions of each component. That is, two or more components to be described below may be combined into one component, or one component may be divided into two or more for each subdivided function. In addition, each of the constituent parts to be described below may additionally perform some or all of the functions of other constituent parts in addition to the main functions of the constituent parts, and some of the main functions of the constituent parts are different. Needless to say, it may also be carried out in a dedicated manner.

In addition, in performing the method or the method of operation, each process constituting the method may occur differently from the specified order unless a specific order is explicitly stated in the context. That is, each process may occur in the same order as specified, may be performed substantially simultaneously, or may be performed in the reverse order.

The technique described below is a technique for detecting defects in products using a machine learning model. The technology described below is a machine vision technique that analyzes an image of a product to inspect product quality. There are various models of machine learning models as is well known. The technique described below is assumed to analyze an image using an artificial neural network.

Hereinafter, a subject analyzing an image using an artificial neural network is called an image processing apparatus. The image processing device corresponds to a device capable of processing and calculating data. For example, the image processing device may be implemented as a device such as a PC, a smart device, and a server. The image processing apparatus processes an input image using a previously trained artificial neural network model.

The image processing apparatus analyzes an image including a product (object). The image to be analyzed is hereinafter referred to as a source image. The image processing apparatus inputs a source image into an artificial neural network to generate an analysis result (classification).

On the other hand, although the output result of a single artificial neural network itself may be a result indicating whether it is defective, the output result of the artificial neural network may be constantly processed to determine whether or not the final failure occurs. Therefore, a device that performs a defect inspection on a product with a processed value of a result of analyzing the source image and further processing is called a computer device. The computer device is a concept including the above-described image processing device.

The computer device may output a constant image of the analysis result of the source image using the artificial neural network. For example, the computer device may generate a result of the analysis as a constant image by segmenting the source image. Hereinafter, an image indicating whether or not the defect is defective is referred to as a segmented image. Various artificial neural network models for image segmentation have been studied, and the technique described below may be performed by adopting any one of them.

The user means the subject who checks for product defects.

1 is an example of a defect detection system 100 using image segmentation. The defect detection system 100 of FIG. 1 may be a system installed in a product processing line. The cameras 111, 112, and 113 photograph the appearance of the product to be inspected. 1 shows three cameras 111, 112, and 113 as examples. Defect inspection can be performed using individual cameras in multiple process lines. Furthermore, defect inspection may be performed using a plurality of cameras photographing one product from different viewpoints. The cameras 111, 112 to 113 capture the source image.

In FIG. 1, the detection server 150 and the detection PC 180 are illustrated as computer devices for detecting defects. The defect detection system 100 may include either the detection server 150 or the detection PC 180. Furthermore, the defect detection system 100 may include both the detection server 150 and the detection PC 180.

The detection server 150 receives source images from the cameras 111, 112 to 113. The detection server 150 may receive the source image through a wired or wireless network. The detection server 150 generates an analysis result indicating whether the product is defective or a divided image and transmits it to the user 10. The user 10 may check the analysis result or the divided image of the detection server 150 through the user terminal. The user terminal means a device such as a PC, a smart device, and a display device.

The detection PC 180 receives the source image from the cameras 111, 112 to 113. The detection PC 180 may receive the source image through a wired or wireless network. The detection PC 180 generates an analysis result indicating whether the product is defective or a divided image and transmits it to the user 10. The user 20 may check whether the product is defective through the detection PC 180.

The computer device may detect whether the defect is or not by performing division on the source image. The computer device may use various image segmentation techniques. For convenience of description, it is assumed that the computer device uses a convolutional encoder/decoder.

2 is an example of a convolutional encoder-decoder. The convolutional encoder-decoder consists of a convolutional encoder and a convolutional decoder. The convolutional encoder-decoder has a mirror image structure with each other. The convolutional encoder-decoder reduces the size of the feature map extracted from the source image and then makes it as large as the source image size, and classifies the class as a classification result for each pixel of the source image.

Convolutional encoders have multiple layers. 2 shows an example of a convolutional encoder having 5 layers. One layer has a convolutional layer and a pooling layer. Although FIG. 2 shows only the convolutional layer, the layer may further include a batch normalization layer and a non-linear activation layer in addition to the convolutional layer. In this case, one layer may have a convolutional layer, a batch normalization layer, and a nonlinearization layer order. This layer can be repeated multiple times. 2 shows a layer repeated twice. That is, one layer has (i) (convolutional layer, batch normalization layer, and nonlinearization layer) × n times + (ii) a pooling layer structure.

The convolutional layer outputs a feature map through convolutional operation on the input image. At this time, a filter that performs convolutional operations is also called a kernel. The size of the filter is called the filter size or kernel size. The operation parameters constituting the kernel are referred to as kernel parameters, filter parameters, or weights. In the convolutional layer, different types of filters can be used for one input.

The convolutional layer performs a convolution operation on a specific area of the input image. The computational domain is called a window. The window can be moved one space from the top left to the bottom right of the image, and the size of the move can be adjusted at one time. The size of the movement is called stride. The convolutional layer performs a convolution operation on all areas of the input image while moving a window in the input image. Meanwhile, the convolutional layer pads at the edge of the image to maintain the dimension of the input image after the convolution operation.

The pooling layer sub-samples the feature map obtained as a result of the computation of the convolutional layer. The pooling operation includes maximum pooling and average pooling. Maximum pooling selects the largest sample value within the window. The average pooling is sampled with the average value of the values included in the window. In general, pooling is such that the stride and the window have a size.

The nonlinear operation layer is a layer that determines output values from neurons (nodes). The nonlinear operation layer uses a transfer function. Transfer functions include Relu and sigmoid functions.

The convolutional encoder generates a feature map for the source image.

The convolutional decoder generates a constant image using the feature map generated by the convolutional encoder. The convolutional decoder is composed of a plurality of layers. 2 shows an example of a convolutional decoder having 5 layers.

One layer has an upsampling layer and a deconvolutional layer. The inverse convolutional layer performs the inverse operation of the convolutional layer of the convolutional encoder. The structure of the convolutional layer, the convolutional layer, the batch standardization layer, and the non-linearization layer may be repeated. However, since the convolutional decoder has a symmetrical structure with the convolutional encoder, it has the same number of convolutional layers and the same convolutional filter size and number as the convolutional encoder.

The upsampling layer performs the reverse operation of the pooling layer. The upsampling layer performs upsampling. The upsampling layer, unlike the pooling layer, serves to expand the dimension.

The inverse convolutional layer performs an inverse operation of the convolutional layer. The inverse convolutional layer performs a convolution operation in the opposite direction to the convolutional layer. The inverse convolutional layer receives a feature map as an input and generates an output image by convolution operation using the kernel. If stride is 1, the inverse convolutional layer outputs an image in which the horizontal and vertical sizes of the feature map are the same as the horizontal and vertical dimensions of the output. If stride is 2, the inverse convolutional layer outputs half the size of the feature map horizontally and vertically.

The convolutional decoder may finally have a softmax classfier.

3 is an example of a neural network model for segmenting an image. 3 is an example of a neural network model including one convolutional encoder-decoder. The learning network N of FIG. 3 may correspond to the convolutional encoder-decoder of FIG. 2.

The convolutional encoder extracts features from the source image and outputs a feature map. The convolutional decoder receives a feature map and outputs a constant divided image. The video segmentation using the convolutional encoder-decoder will be briefly described.

The convolutional encoder-decoder can classify which class belongs to each pixel-wise. The last layer of the convolutional encoder-decoder has the size of the horizontal size of the input image × the vertical size of the input image × the number of classes. The last layer means classification information for the k class of the (i,j)th pixel.

The softmax layer performs softmax for each pixel. An example of the soft max equation is shown in Equation 1 below.

input is input information for the Softmax layer. input(i,j,k) means classification information for the k class of the (i,j) th pixel. output is the output value of the Softmax layer. output(i,j,k) means classification information for the k class of the (i,j)th pixel.

와 같이, 각 픽셀 별로 클래스에 대한 차원(dimension)을 모두 더하면 1의 값이 나온다.

As shown in the figure, if all dimensions of a class are added for each pixel, a value of 1 is obtained.

The output value of the softmax layer represents a classification probability for a class for each pixel. In this case, classification may be performed with a class having the highest probability for the corresponding pixel.

Convolutional encoder-decoder has a loss function similar to other neural network models. In the learning process or update process, the weight of the neural network can be optimized based on the loss function. For example, the weight optimization may use a gradient descent method. The computer device calculates the gradient d _parameter of each parameter to be learned for the defined loss function. The computer device can optimize the weight by finding a value that minimizes the slope. The computer device may update the value of each parameter using the slope of each parameter. The parameter _new to be updated can be determined as follows. parameter _new = parameter _prev -learnRate* d _parameter . Here, the learning rate (learnRate) is a setting value that is set directly during model training.

The loss function may consist of two parts. The loss function may include a cross entropy and a regularization term. The loss function can be the sum of the cross entropy and the normalization term. The loss function may include a normalization term to normalize such that the total cross entropy loss value determined by the cross entropy of each pixel and the result value of the softmax function are similar in the reference value range between neighboring pixels for the same class.

The cross entropy part may be expressed as Equation 2 below.

In Equation 2, w _k is a weight for the k-th class, t(i,j,k) is affiliation information for the k-th class of (i,j) pixels, and p(i,j,k) is (i, j) The softmax output score for the k-th class of pixels.

The value of t(i,j,k)*log(p(i,j,k)) multiplied by the weight w _k for each class is the loss of cross entropy for class k of the pixel at the (i,j)th position. Mean value. When w _k is not detected, a higher weight is given to the higher cost, or the non-uniformity of the number of pixels corresponding to each class of the training data set is compensated.

Equation (2) sums the loss values for each class, calculates the average cross-entropy loss value by averaging for each pixel position.

The normalization term can be expressed as Equation 3 below.

In Equation 3, w _k is a weight for the k-th class, t(i,j,k) is affiliation information for the k-th class of (i,j) pixels, and p(i,j,k) is (i, j) The softmax output score for the k-th class of pixels, the regularizer is a hyper parameter that determines the weight of the loss value for the loss _{regularizationTerm} .

The normalization term calculates the loss value by squaring the difference so that similar values can be obtained for the same class between surrounding pixels with respect to the result value of the softmax function. That is, the normalization term serves to normalize such that similar values appear for the same class in adjacent pixels. The range of adjacent pixels can be adjusted. In Equation 3, a value similar to the pixel (i-1, j) and the pixel (i, j-1) directly adjacent to the pixel is adjusted based on the (i,j) pixel. The degree of similarity may be determined by a different formula from Equation 3. After all, it is sufficient to adjust the normalization term so that the result of the softmax function is not significantly different for a specific pixel of the same class of adjacent pixels.

In FIG. 3, the learning network N may output a specific segmented image based on classification information (output value) for each pixel. For example, the learning network N may determine a pixel value with the same color for the same output value. Through this, the segmented image becomes an image in which the region for the classification value is divided.

4 is another example of a neural network model for segmenting an image. 4 is an example of a neural network model including a plurality of learning networks (N1, N2, ..., Nn). In FIG. 4, the individual learning networks N1, N2, ..., or Nn perform the same or similar operation as the learning network of FIG. However, the individual learning networks N1, N2, ... or Nn output classification information (output value) for each pixel of the source image. The segmented image may be generated based on a value output by a plurality of learning networks (N1, N2, ... or Nn).

Each of the individual learning networks N1, N2, ... or Nn has a configuration as described in FIG. Accordingly, each of the individual learning networks N1, N2, ... or Nn generates an output value indicating classification information for each pixel for the input source image.

At this time, the plurality of learning networks (N1, N2, ..., Nn) may be different from each other by setting the pooling size and stride size of the pooling layer differently, and the size of the feature map output by the encoder. This is to detect product defects of various sizes using a plurality of learning networks.

For output values output from a plurality of learning networks (N1, N2, ..., Nn), a final output value (score) for each pixel may be determined using a kind of bagging technique. For example, the computer device reports the softmax output value of each learning network as a score for each class for each pixel of each learning network, and any of the criteria such as maximum, median, mode, arithmetic mean, harmonic mean, geometric mean, and truncated mean You can calculate the final score for each class using one.

The computer device may generate a segmented image based on the final score for each pixel. The computer device may output a divided image by setting a pixel value having a specific color according to the value of the final score of each pixel.

5 is an example of a configuration of a defect detection apparatus 200 using image segmentation. The defect detection device 200 corresponds to the above-described computer device. The defect detection device 200 shows the configuration of the detection server 150 to the detection PC 180 of FIG. 1.

The defect detection device 200 is a device that generates the above-described source image analysis result or split image. The defect detection device 200 may be physically implemented in various forms. For example, the defect detection device 200 may take the form of a computer device such as a PC, a server of a network, a chipset dedicated to image processing, and the like. The computer device may include a mobile device such as a smart device.

The defect detection device 200 includes a storage device 210, a memory 220, a computing device 230, an interface device 240, and a communication device 250.

The storage device 210 stores a neural network model for image processing. For example, the storage device 210 may store a learning network as shown in FIG. 3 or/and a learning network as shown in FIG. 4. Furthermore, the storage device 210 may store programs or source codes required for image processing. The storage device 210 may store the input source image and the generated split image.

The memory 220 may store data and information generated in the process of generating the source image and the divided image received by the defect detection device 200.

The interface device 240 is a device that receives certain commands and data from the outside. The interface device 240 may receive a source image from a physically connected input device or an external storage device. The interface device 240 may receive various neural network models for image processing. The interface device 240 may receive learning data, information, and parameter values for generating a neural network model.

The communication device 250 refers to a configuration that receives and transmits certain information through a wired or wireless network. The communication device 250 may receive a source image from an external object. The communication device 250 may also receive various neural network models and data for model training. The communication device 250 may transmit the generated segmented image as an external object.

The communication device 250 to the interface device 240 is a device that receives certain data or commands from the outside. The communication device 250 to the interface device 240 may be referred to as an input device.

The computing device 230 uses the neural network model or program stored in the storage device 210 to generate information or segmented images for detecting the quantity. The computing device 230 may train the neural network model used in the image processing process using the given learning data. The computing device 230 may generate a segmented image for the source image using the neural network constructed through the above-described process. The computing device 230 may be a device such as a processor, an AP, or a chip embedded with a program that processes data and processes a certain operation.

In addition, the above-described method for detecting a quantity or generating a segmented image may be implemented as a program (or application) including an executable algorithm that can be executed on a computer. The program may be stored and provided in a non-transitory computer readable medium.

The non-transitory readable medium means a medium that stores data semi-permanently and that can be read by a device, rather than a medium that stores data for a short time, such as registers, caches, and memory. Specifically, the various applications or programs described above may be stored and provided in a non-transitory readable medium such as a CD, DVD, hard disk, Blu-ray disk, USB, memory card, ROM, and the like.

The drawings attached to the present embodiment and the present specification merely show a part of the technical spirit included in the above-described technology, and are easily understood by those skilled in the art within the scope of the technical spirit included in the above-described technical specification and drawings. It will be apparent that all of the examples and specific examples that can be inferred are included in the scope of the above-described technology.

100: defect detection system
111, 112, 113: Camera
150: detection server
180: detecting PC
10, 20: user
200: image processing device
210: storage device
220: memory
230: computing device
240: interface device
250: communication device

Claims (13)

A computer device inputting a source image including a product object into a learning network; And
The computer device comprises the step of outputting the divided image generated by the learning network,
The learning network is a convolutional encoder-decoder, and a defect detection method using image segmentation based on an artificial neural network that outputs the segmented image in which a defective part is classified from the product object. According to claim 1,
The encoder extracts a feature in units of pixels to generate a feature map, and the decoder detects a defect using artificial neural network based image segmentation that outputs the segmented image as an input of the feature map. According to claim 1,
The loss function of the learning network is a similar value within a reference value range between neighboring pixels for a class in which the result of the total cross entropy loss value and the softmax function determined by the cross entropy of each pixel is the same class. A defect detection method using artificial neural network based image segmentation including a normalization term that normalizes to appear. According to claim 1,
The loss function of the learning network is a function of _summing the first loss function (loss _crossEntropy ) and the second loss function (loss _{regularizationTerm} ), and detecting a defect using artificial neural network based image segmentation determined by the following formula.

,

(Where w _k is the weight for the k-th class, t(i,j,k) is the affiliation information for the k-th class of (i,j) pixels, and p(i,j,k) is (i,j ) Softmax output score for the k-th class of pixels, regularizer is a _{hyperparameter} that determines the weight of the loss value for loss _{regularizationTerm} ) A computer device inputting a source image including product objects into a plurality of learning networks;
Outputting an output value for each pixel by the plurality of learning networks; And
And the computer device classifying a final class for each pixel based on an output value for each pixel output by the plurality of learning networks.
The learning network is a convolutional encoder-decoder, and a defect detection method using image segmentation based on an artificial neural network that outputs the output value for classifying a defective part in the product object. The method of claim 5,
The loss function of the learning network is a similar value within a reference value range between neighboring pixels for a class in which the result of the total cross entropy loss value and the softmax function determined by the cross entropy of each pixel is the same class. A defect detection method using artificial neural network based image segmentation including a normalization term that normalizes to appear. The loss function of the learning network is a function of _summing the first loss function (loss _crossEntropy ) and the second loss function (loss _{regularizationTerm} ), and detecting a defect using artificial neural network based image segmentation determined by the following formula.

,

(Where w _k is the weight for the k-th class, t(i,j,k) is the affiliation information for the k-th class of (i,j) pixels, and p(i,j,k) is (i,j ) Softmax output score for the k-th class of pixels, regularizer is a _{hyperparameter} that determines the weight of the loss value for loss _{regularizationTerm} ) The method of claim 5,
The computer device determines a final score for each pixel as at least one of a maximum value, a median value, a mode value, an arithmetic average value, a harmonic average value, a geometric average value, and a truncated average value of output values for each pixel output by the plurality of learning networks, and A defect detection method using image segmentation based on an artificial neural network that classifies the final class by a final score. The method of claim 5,
The computer device detects a defect using artificial neural network based image segmentation that generates a segmented image based on a final class for each pixel. An input device that receives a source image including a product object;
A storage device for storing a plurality of convolutional encoder-decoder generating an output value for distinguishing a defective part from the input image; And
The source image is input to each of the plurality of convolutional encoder-decoders to generate output values for each pixel of the source image, and each of the plurality of convolutional encoder-decoders is output based on output values for each pixel. A defect detection apparatus using artificial neural network based image segmentation, comprising an operation device for classifying a final class for a pixel and generating an output image based on the final class for each pixel. The method of claim 10,
The plurality of convolutional encoder-decoder is a defect detection apparatus using artificial neural network based image segmentation with different sizes of feature maps output from the encoder. The method of claim 10,
The loss function of the convolutional encoder-decoder is a function of _summing a first loss function (loss _crossEntropy ) and a second loss function (loss _{regularizationTerm} ), and a defect detection apparatus using artificial neural network based image segmentation determined by the following formula .

,

(Where w _k is the weight for the k-th class, t(i,j,k) is the affiliation information for the k-th class of (i,j) pixels, and p(i,j,k) is (i,j ) Softmax output score for the k-th class of pixels, regularizer is a _{hyperparameter} that determines the weight of the loss value for loss _{regularizationTerm} ) The method of claim 10,
The computing device determines a final score for each pixel as at least one of a maximum value, a median value, a mode value, an arithmetic mean value, a harmonic mean value, a geometric mean value, and a truncation mean value for each pixel output from the plurality of convolutional encoder-decoders. And, the defect detection device using the artificial neural network based image segmentation to classify the final class by the final score.

Images