KR102141302B1 - Object detection method based 0n deep learning regression model and image processing apparatus - Google Patents

Abstract

An image object detection method based on a deep learning regression model includes the steps of: inputting a source image including a target object and a background to a deep learning regression model by an image processing device; and generating an output image in which the target object is classified from the source image using the deep learning regression model by the image processing device, wherein the deep learning regression model is a convolutional encoder-decoder structure and the deep learning regression model distinguishes between the target object and the background using a pixel-based regression method.

Description

OBJECT DETECTION METHOD BASED 0N DEEP LEARNING REGRESSION MODEL AND IMAGE PROCESSING APPARATUS}

The technique described below relates to a technique for detecting a specific object in an image. In particular, the technique described below relates to an object detection technique using a pixel unit regression model.

Machine vision (machine vision) has been widely studied because it can perform a highly accurate inspection repeatedly by replacing a manual inspector in applications such as defect inspection, classification, and recognition.

Machine learning technology, which has recently attracted attention, is actively being studied in applications where it is difficult to apply the existing machine vision technology because it can learn the judgment form of a manual inspector.

Matthias Haselmann, Dieter P. Gruber, Paul Tabatabai, Anomaly Detection using Deep Learning based Image Completion, 2018 17th IEEE International Conference on Machine Learning and Applications

It is not easy to detect a specific object (such as a defective part) in an image having a similar pattern background. In the related art, a method of determining a difference value by comparing a source image with a specific object excluded from a source image is used. However, since the process of detecting a specific object in the background of a similar pattern or a noisy background is not easy, the accuracy of product defect judgment is poor.

The technique described below is intended to provide a technique for detecting a specific object included in a source image using a pixel-wise regression-type deep learning model. The technique described below is intended to provide a technique for detecting a specific object by comparing an object with a background from which the distortion component is removed.

A method for detecting an image object based on a regression deep learning model includes: a processing device inputting a source image including a target object and a background into the regression deep learning model, and the image processing device uses the regression deep learning model to detect the image object in the source image. And generating an output image in which the target object is classified.

An object detection apparatus based on a regression deep learning model includes an input device that receives a source image including an object and a background, a storage device that stores a regression deep learning model that generates an output image in which the target object is separated from the source image, and the And a computing device that inputs a source image to the regression deep learning model to generate the output image.

The regression deep learning model is a convolutional encoder-decoder structure, and the regression deep learning model distinguishes the target object and the background by a pixel-wise regression method.

The technique described below accurately detects a specific object even in a source image having many distortion components using a pixel-wise regression method. The technology described below facilitates product defect detection by generating an image in which a specific object is identified (divided).

1 is an example of a defect detection system using a regression deep learning model.
2 is an example of a convolutional encoder-decoder.
3 is an example of a learning network structure of a regression deep learning model.
4 is an example of a source image and an output image.
5 is an example of the configuration of an object detection device.

The technique described below may be applied to various changes and may have various embodiments, and specific embodiments will be illustrated in the drawings and described in detail. However, this is not intended to limit the techniques described below to specific embodiments, and should be understood to include all changes, equivalents, or substitutes included in the spirit and scope of the techniques described below.

Terms such as first, second, A, B, etc. can be used to describe various components, but the components are not limited by the above terms, and only for distinguishing one component from other components Used only. For example, the first component may be referred to as a second component, and similarly, the second component may be referred to as a first component without departing from the scope of the technology described below. The term and/or includes a combination of a plurality of related described items or any one of a plurality of related described items.

In the terminology used herein, a singular expression should be understood to include a plurality of expressions unless clearly interpreted differently in the context, and terms such as “comprises” describe features, numbers, steps, operations, and components described. It is to be understood that it means that a part or a combination thereof is present, and does not exclude the presence or addition possibility of one or more other features or numbers, step operation components, parts or combinations thereof.

Prior to the detailed description of the drawings, it is intended to clarify that the division of components in this specification is only divided by the main functions of each component. That is, two or more components to be described below may be combined into one component, or one component may be divided into two or more for each subdivided function. In addition, each of the components to be described below may additionally perform some or all of the functions in charge of other components in addition to the main functions in charge thereof, and some of the main functions in charge of each of the components are different. Needless to say, it may also be carried out in a dedicated manner.

In addition, in performing the method or the method of operation, each process constituting the method may occur differently from the specified order unless a specific order is explicitly stated in the context. That is, each process may occur in the same order as specified, may be performed substantially simultaneously, or may be performed in the reverse order.

The technique described below is a technique for detecting defects in products using a machine learning model. The technique described below may be applied to a machine vision technique that analyzes an image of a product and inspects product quality. There are various models of machine learning models as is well known. The technique described below is assumed to analyze an image using an artificial neural network.

Hereinafter, a subject analyzing an image using an artificial neural network is called an image processing apparatus. The image processing device corresponds to a device capable of processing and calculating data. The image processing device is a computing device having an arithmetic function. For example, the image processing device may be implemented as a device such as a PC, a smart device, and a server. The image processing apparatus processes the input image using a previously trained artificial neural network model.

The image processing apparatus may analyze an image including a product (object). The input image to be analyzed is hereinafter referred to as a source image. The image processing device generates an analysis result by inputting a source image into the artificial neural network. The image processing apparatus may generate an analysis result as a constant image. The specific object detected by the image processing device in the source image is called a target object. For example, the target object may be a defective part, a specific part, or a specific kind of object. The source image includes the target object and background.

1 is an example of a defect detection system 100 using a regression deep learning model. 1 is a system for detecting a defective site object using a regression deep learning model. The regression deep learning model will be described later. The regression deep learning model is a neural network model that detects a specific object in a source image. Therefore, the defect detection system 100 corresponds to one application to which a regression deep learning model is applied.

The defect detection system 100 of FIG. 1 may be a system installed in a product processing line. The cameras 111, 112, and 113 photograph the appearance of the product to be inspected. 1 shows three cameras 111, 112, and 113 as examples. Defect inspection can be performed using individual cameras in multiple process lines. Furthermore, defect inspection may be performed using a plurality of cameras photographing one product from different viewpoints. The cameras 111, 112 to 113 capture the source image.

In FIG. 1, the detection server 150 and the detection PC 180 are illustrated as image processing devices for detecting defects. The defect detection system 100 may include either the detection server 150 or the detection PC 180. Furthermore, the defect detection system 100 may include both the detection server 150 and the detection PC 180.

The detection server 150 receives the source image from the cameras (at least one of 111, 112 and 113). The detection server 150 may receive the source image through a wired or wireless network. The detection server 150 generates an analysis result indicating whether the product is defective or a divided image and transmits it to the user 10. The user 10 may check the analysis result or the divided image of the detection server 150 through the user terminal. The user terminal means a device such as a PC, a smart device, and a display device.

The detection PC 180 receives the source image from the cameras (at least one of 111, 112, and 113). The detection PC 180 may receive the source image through a wired or wireless network. The detection PC 180 generates an analysis result indicating whether the product is defective or a divided image and transmits it to the user 10. The user 20 may check whether the product is defective through the detection PC 180.

The image processing apparatus may detect a target object from the source image using a regression deep learning model. The regression deep learning model may be a model of a convolutional encoder/decoder structure. First, the convolutional encoder-decoder will be described.

2 is an example of a convolutional encoder-decoder. 2 is an example of a general convolutional encoder-decoder. The convolutional encoder-decoder consists of a convolutional encoder and a convolutional decoder. The convolutional encoder-decoder has a mirror image structure with each other. The convolutional encoder-decoder reduces the size of the feature map extracted from the source image and then makes it as large as the source image size, and classifies the class as a classification result for each pixel of the source image.

Convolutional encoders have multiple layers. 2 shows an example of a convolutional encoder having 5 layers. One layer has a convolutional layer and a pooling layer. Although FIG. 2 shows only the convolutional layer, the layer may further include a batch normalization layer and a non-linear activation layer in addition to the convolutional layer. In this case, one layer may have a convolutional layer, a batch normalization layer, and a nonlinearization layer order. This layer can be repeated multiple times. 2 shows a layer repeated twice. That is, one layer has (i) (convolutional layer, batch normalization layer, and nonlinearization layer) x n times + (ii) a pooling layer structure.

The convolutional layer outputs a feature map through convolutional operation on the input image. At this time, a filter that performs convolutional operations is also called a kernel. The size of the filter is called the filter size or kernel size. The operation parameters constituting the kernel are referred to as kernel parameters, filter parameters, or weights. In the convolutional layer, different types of filters can be used for one input.

The convolutional layer performs a convolution operation on a specific area of the input image. The computational domain is called a window. The window can be moved one space from the top left to the bottom right of the image, and the size of the move can be adjusted at one time. The size of the movement is called stride. The convolutional layer performs a convolution operation on all areas of the input image while moving a window in the input image. Meanwhile, the convolutional layer pads at the edge of the image to maintain the dimension of the input image after the convolution operation.

The pooling layer sub-samples the feature map obtained as a result of the computation of the convolutional layer. The pooling operation includes maximum pooling and average pooling. Maximum pooling selects the largest sample value within the window. The average pooling is sampled with the average value of the values included in the window. In general, pooling is such that the stride and the window have a size.

The nonlinear operation layer is a layer that determines output values from neurons (nodes). The nonlinear operation layer uses a transfer function. Transfer functions include Relu and sigmoid functions.

The convolutional encoder generates a feature map for the source image.

The convolutional decoder generates a constant image using the feature map generated by the convolutional encoder. The convolutional decoder is composed of a plurality of layers. 2 shows an example of a convolutional decoder having 5 layers.

One layer has an upsampling layer and a deconvolutional layer. The inverse convolutional layer performs an inverse operation of the convolutional layer of the convolutional encoder. The structure of the convolutional layer, the convolutional layer, the batch standardization layer, and the non-linearization layer may be repeated. However, since the convolutional decoder has a symmetric structure with the convolutional encoder, it has the same number of convolutional layers and the same convolutional filter size and number as the convolutional encoder.

The upsampling layer performs the reverse operation of the pooling layer. The upsampling layer performs upsampling. The upsampling layer, unlike the pooling layer, serves to expand the dimension.

The inverse convolutional layer performs an inverse operation of the convolutional layer. The inverse convolutional layer performs a convolution operation in the opposite direction to the convolutional layer. The inverse convolutional layer receives a feature map as an input and generates an output image by convolution operation using the kernel. If stride is 1, the inverse convolutional layer outputs an image in which the horizontal and vertical sizes of the feature map are the same as the horizontal and vertical dimensions of the output. If stride is 2, the inverse convolutional layer outputs half the size of the feature map horizontally and vertically.

3 is an example of a learning network structure of a regression deep learning model. The regression deep learning model has a convolutional encoder-decoder structure. 3, the learning network N refers to a learning network that is a regression deep learning model. The learning network N includes an encoder N1 and a decoder N2.

The encoder N1 receives a source image and outputs a feature map. The decoder N2 outputs a specific image based on the received feature map. The image output by the decoder N2 is called an output image.

The encoder N1 generates a feature map of the source image and extracts only some valid feature values while reducing the size of the feature map. The decoder N2 increases the size of the feature map based on valid feature values output from the encoder N1. The decoder N2 finally outputs an output image having the same size as the source image.

The encoder N1 includes a plurality of encoder stages. 3 shows three encoder stages (encoder 1, encoder 2 and encoder 3) as examples.

The encoder stage may be composed of a plurality of convolutional blocks and one pooling layer. One convolutional block includes a convolutional layer, a nonlinearization layer, and a standardization layer. A relu layer can be used for the nonlinear layer. The standardization layer may use a batch standardization layer. The operation of each layer is as described above.

The pooling layer can perform max pooling. The pooling layer stores the pixel position having the maximum value and delivers it to the upsampling layer of the corresponding decoder stage (indicated by the dotted arrow in FIG. 3).

The decoder N2 includes a plurality of decoder stages. 3 shows three decoder stages (decoder 1, decoder 2 and decoder 3) as an example. As described above, the decoder N1 has a structure (mirror image) that is symmetrical to the encoder N1. Therefore, the decoder stage may be composed of one upsampling layer and a plurality of (equal to the number of convolutional blocks) inverse convolutional blocks. The decoder stage reverses the operation of the encoder stage.

The upsampling layer outputs the value of the input feature map at the maximum pixel position received from the maximum pooling layer of the corresponding encoder stage, and outputs a value of '0' other than the maximum pixel position. The decoder stage has the same number of filters and filter sizes as the convolutional block and convolutional layer of the corresponding encoder stage.

In the deep learning model, parameters are trained in the direction of reducing the loss function. The loss function can optimize the weight of the neural network through the learning process. For example, the weight optimization may use a gradient descent method. The image processing apparatus calculates a gradient d _parameter of each parameter to be learned for the defined loss function. The image processing apparatus may optimize the weight by finding a value that minimizes the slope. The image processing apparatus may update the value of each parameter by using the slope of each parameter. The parameter _new to be updated can be determined as follows. parameter _new = parameter _prev -learnRate* d _parameter . Here, the learning rate (learnRate) is a setting value that is set directly during model training. The learning network of FIG. 3 may use the loss function L as shown in Equation 1 below.

In Equation 1, n is the number of data used for learning, d is the number of channels, c is the number of columns, and r is the number of rows. Columns and rows indicate pixel locations in the image. Y is output data and T is correct answer data. R is a normalization term. The loss function has the form of combining the mean squared error (MSE) and the normalization term between the output data and the correct answer data. The MSE eventually ensures that the output data is the same as the correct answer data. λ is a value that is set in advance as a hyper parameter. Determine the weight for the normalized term of the loss function.

Equation 2 below shows a normalization term.

In Equation 2, ε means an arbitrary small positive number to prevent a situation in which the value inside the square root becomes 0.

The normalization term R causes the values of the output data not to be distributed sporadically, but to output similar values to the surrounding pixels. The normalization term R normalizes total variation (TV) between peripheral pixels in the row direction and column direction of the output data Y. TV normalization allows similar values between surrounding pixels, while preserving the border relatively in the border portion where the value changes rapidly in the image. The normalization term allows certain pixels to have similar values within a certain range from adjacent surrounding pixels.

The background of the source image may have noise or non-uniform lighting components. The normalization term serves to remove or smooth distortion components in this background.

The trained regression deep learning model illustrated in FIG. 3 calculates a difference value from a background for each pixel by a pixel unit regression method. In the process of the encoder N1 extracting features in units of pixels, noise of a background is removed by a regression method, and a difference value between the noise-removed background and the current pixel (eg, a specific object part) is calculated. Through this, even if the background is complicated or the background has a specific pattern, the background and the target object are distinguished. The decoder N2 generates an output image based on the feature map having the distinguished features.

4 is an example of a source image and an output image. In FIG. 4, A represents an example of a source image, and B represents an example of an output image. The output image is an example of displaying the difference value in red color to intuitively display the result. Here, the difference value means the difference between the pixel and the background. The red pixel (area) represents the region with the highest difference value, the yellow pixel represents the region where the difference value is lower than the red color, and blue represents the background without the difference value.

5 is an example of the configuration of the object detection device 200. The object detection device 200 corresponds to the image processing device described above. The object detection device 200 may also correspond to the detection server 150 to detection PC 180 of FIG. 1.

The object detection apparatus 200 generates an output image in which the object is identified in the source image using the regression deep learning model. The object detection device 200 may be physically implemented in various forms. For example, the object detection device 200 may take the form of a computer device such as a PC, a server of a network, a chipset dedicated to image processing, and the like. The computer device may include a mobile device such as a smart device.

The object detection device 200 includes a storage device 210, a memory 220, a computing device 230, an interface device 240, and a communication device 250.

The storage device 210 stores a neural network model for image processing. For example, the storage device 210 may store a regression deep learning model as shown in FIG. 3. The regression deep learning model is assumed to be pre-trained. Furthermore, the storage device 210 may store programs or source codes required for image processing. The storage device 210 may store the input source image and the generated split image.

The memory 220 may store data and information generated in the process of generating the source image and the output image received by the object detection apparatus 200.

The interface device 240 is a device that receives certain commands and data from the outside. The interface device 240 may receive a source image from a physically connected input device or an external storage device. The interface device 240 may receive various neural network models for image processing. The interface device 240 may receive learning data, information, and parameter values for generating a neural network model.

The communication device 250 refers to a configuration that receives and transmits certain information through a wired or wireless network. The communication device 250 may receive a source image from an external object. The communication device 250 may also receive various neural network models and data for model training. The communication device 250 may transmit the generated output image as an external object.

The communication device 250 to the interface device 240 is a device that receives certain data or commands from the outside. The communication device 250 to the interface device 240 may be referred to as an input device.

The computing device 230 generates an output image in which the target object is detected using a neural network model or program stored in the storage device 210. The computing device 230 may learn a regression deep learning model used in the image processing process using the given learning data. The computing device 230 may generate an output image for the source image using the regression deep learning model constructed through the above-described process. The computing device 230 may be a device such as a processor, an AP, or a chip embedded with a program that processes data and processes a certain operation.

In addition, the above-described image processing method, a method of generating an image in which a target object is classified from a background, or a method of detecting a quantity may be implemented as a program (or application) including an executable algorithm that can be executed on a computer. The program may be stored and provided in a non-transitory computer readable medium.

The non-transitory readable medium means a medium that stores data semi-permanently and that can be read by a device, rather than a medium that stores data for a short time, such as registers, caches, and memory. Specifically, the various applications or programs described above may be stored and provided in a non-transitory readable medium such as a CD, DVD, hard disk, Blu-ray disk, USB, memory card, ROM, and the like.

The drawings attached to the present embodiment and the present specification merely show a part of the technical spirit included in the above-described technology, and are easily understood by those skilled in the art within the scope of the technical spirit included in the above-described technical specification and drawings. It will be apparent that all of the examples and specific examples that can be inferred are included in the scope of the above-described technology.

100: defect detection system
111, 112, 113: Camera
150: detection server
180: detecting PC
10, 20: user
200: object detection device
210: storage device
220: memory
230: computing device
240: interface device
250: communication device

Claims (12)

Inputting a source image including a target object and a background into the regression deep learning model by the image processing apparatus; And
The image processing apparatus includes the step of generating an output image in which the target object is separated from the source image using the regression deep learning model.
The regression deep learning model is a convolutional encoder-decoder structure, and the regression deep learning model distinguishes the target object from the background by a pixel unit regression method, and the loss function of the encoder is A method for detecting an image object based on a regression deep learning model defined by the following equation.

,

(Where L is the loss function, n is the number of data used for learning, d is the number of channels, c is the number of columns, r is the number of rows, Y is the output data, T is the correct answer data, R is the normalization term, λ is the weight, ε is any small positive number) According to claim 1,
The encoder extracts a feature in units of pixels to generate a feature map, and the decoder outputs the output image as an input of the feature map, based on a regression deep learning model-based image object detection method. According to claim 1,
The encoder uses a pixel-wise regression to smooth a distortion component in the background, and extracts a feature for the target object and a background from which the distortion component is removed. According to claim 1,
The encoder is a regression deep learning model-based image object detection method including a plurality of encoder blocks and one pooling layer. According to claim 1,
The encoder loss function includes a normalization term in which a specific pixel is adjusted to a similar value within a predetermined range from surrounding pixels. delete delete delete delete delete delete delete

Images