KR20230030259A - Deep learning-based data augmentation method for product defect detection learning - Google Patents

Abstract

The present invention relates to a deep learning-based data augmentation method for product defect detection learning, which can generate large amounts of defect data. According to the present invention, the deep learning-based data augmentation method for product defect detection learning comprises: a first step of labeling a small amount of given training data while generating a segmentation map of a defective object to remove the background along with a bounding box used for object detection; a second step of using a cropped image of the bounding box area of each object and a manually labeled defect object segmentation map to train a semantic segmentation model, and applying the semantic segmentation network to a bounding box area object image without a segmentation label when training of the semantic segmentation network is completed, thereby acquiring a segmentation map capable of segmenting the defective object; a third step of removing the background from the bounding box area image of each defect object with the segmentation map acquired in the first and second steps to generate a background-free defect image and then inputting the background-free defect image into a generative adversarial network (GAN)-based data generation model to learn the distribution of defect data; and a fourth step of inputting the generated defective object into a trained background color or density classification model to calculate the background color or density suitable for the generated object and pasting the generated defect object into the defect-free image data to generate a defect image.

Description

Deep learning-based data augmentation method for product defect detection learning}

The present invention relates to a method for augmenting data, and more particularly, to a method for augmenting object detection learning data based on semantic segmentation and generative adversarial networks (GAN).

Recently, deep learning technology has made remarkable progress, surpassing human capabilities in several fields. However, deep learning technology still has the disadvantage of requiring a lot of training data. To overcome these problems, various studies on few-shot learning, which is a technique for performing learning in a small data environment, are being conducted. The easiest method to use when data is scarce is data augmentation. In the case of learning on image data, spatial level augmentation such as image rotation or cropping, or pixel level augmentation that adjusts brightness or contrast, etc. is commonly used, and is usually built into deep learning platforms and automatically performed during the learning process. .

In addition, since this augmentation technique cannot generate new data with different shapes, a method of generating new data within the distribution by learning the distribution of training data through a generative model such as Generative Adversarial Networks (GAN) has also been proposed. However, even at this time, when the learning data has a background, it is necessary to additionally solve the problem of processing the background of the generated data or the problem of attaching a label (correct answer) to the generated data for supervised learning.

Systematically collecting and labeling (labeling) product defect data at manufacturing sites is expensive, so it is often difficult to obtain enough data for learning. Accordingly, in order to secure learning data, defect data may be manually created through an image editing program or a program for creating virtual defects may be created and used.

In other words, it is essential to secure a sufficient amount of defect data in order to obtain excellent performance in a vision inspection system using deep learning. A method to effectively secure the necessary data is required.

Shorten, C., Khoshgoftaar, T.M. A survey on Image Data Augmentation for Deep Learning.J Big Data 6, 60 (2019).

The present invention has been proposed to solve the above technical problems, and provides a deep learning-based data augmentation method capable of generating a large amount of defect data based on minimal manual work when a small amount of defect data exists using deep learning technology. to provide.

According to an embodiment of the present invention for solving the above problem, labeling of a given small amount of training data is performed while creating a segmentation map of a defective object for background removal along with a bounding box used for object detection. In the first step, the semantic segmentation model is learned using the image of the bounding box area of each object cut out and the manually labeled defective object segmentation map. The second step is to obtain a segmentation map that can be applied to the image to segment the defect object, and the background is removed by using the segmentation map obtained in the first and second stages in the bounding box area image of each defect object to obtain a defect without a background. A third step of generating an image and inputting it to a Generative Adversarial Networks (GAN)-based data generation model to learn the distribution of defect data, and classifying the defect object created using the GAN-based data generation model as the learned background color or density. The fourth step of creating a defect image by inputting it into the model to calculate the background color (in case of color image) or density (in case of black and white image) suitable for the created object, and then pasting the created defect object to image data without defects. There is provided a deep learning-based data augmentation method for product defect detection learning that includes.

In addition, according to another embodiment of the present invention, a first step of separating a defect image from a normal image in the entire data, a second step of displaying the type and location (bounding box) of each defect in the defect image, , The third step of cutting out the part where the defect is located in the defect image and saving it as a defect part image, the fourth step of selecting some of the defect part images and displaying them as a defect segmentation image, and the defect part image and the defect segmentation image. The fifth step of learning the segmentation model, the sixth step of extracting the defect segmentation image by inserting the defect region unmarked defect region image into the learned defect segmentation model, and the defect region images obtained through the third to sixth steps, and The seventh step of learning a defect generation model using the defect region, the eighth step of generating a desired number of random defects through the learned defect generation model, and defect part images obtained through the third to sixth steps, A ninth step of deriving the average background color or density for each defect using the defect area, a tenth step of learning a background color or density classification model using the defect area and the average background color or density, and a defect generated in the eighth step. An 11th step of deriving an appropriate color or density range by putting one of them into the learned background color or density classification model, and a 12th step of importing a normal image for the defect in the 11th step and pasting it into a position having an appropriate color or density range. There is provided a deep learning-based data augmentation method for product defect detection learning that includes.

In addition, the present invention is characterized in that the 11th to 12th steps are repeatedly performed for each defect generated in the 8th step.

Further, in the present invention, the normal image is an image without a defect, the defect image is an image with a defect, the defect area image is an image obtained by cutting out a defect area in a rectangle from the defect image, and the defect segmentation image Is an image in which the defective area in the defect area image is displayed in black and the other areas in white, the defect background is an image displaying the rest of the defect area image except for the defect area, and the defect area image is the defect area It is characterized in that it means an image in which the defect background is displayed in black in the image.

The present invention uses GAN (Generative Adversarial Networks)-based learning data augmentation techniques together with semantic segmentation model learning-based defect background removal and classification model learning-based background color or density selection for smooth augmentation of learning data for detecting defective objects. We propose an object detection learning data augmentation technique. The added semantic segmentation learning and color and density classification learning are used to remove the background of the generated defective object image and select a background of appropriate color or density, respectively. As a result of applying the proposed technique to a small amount of 4 types of steel plate defect image datasets, natural augmented images were obtained. As a result of applying the augmented images to object detection learning, a program to create virtual defects using a geometric method was written and generated a better data set. Excellent detection results were obtained.

1 is a diagram illustrating the meaning of terms in the present invention
Figure 2 is a diagram showing the entire process of the data augmentation technique of the present invention
Figure 2a is a diagram showing the f-DAGAN network structure
Figure 3 is a flow chart showing the entire process of the data augmentation technique of Figure 2 in more detail
4 is a diagram showing a defective object sample image;
5 is a view showing a sample image of a defect object from which a background has been removed;
6 is a view showing a generated defective object sample image
7 is a view showing a sample image of a defect object in which a background is combined;

Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings in order to describe in detail enough for those skilled in the art to easily implement the technical idea of the present invention.

- Proposed data augmentation technique

1 is a diagram illustrating the meaning of terms in the present invention.

Referring to FIG. 1, (1) a normal image is an image without a defect, (2) a defect image is an image with a defect, (3) a defect image is a rectangular image of a defect region from a defect image, ( 4) The defect division image is an image in which the defect area is displayed in black and the other parts are white in the defect area image, (5) the defect background is the rest of the defect area image excluding the defect area, (6) the defect area image is the defect area In the image, it means an image in which the defect background is displayed in black.

The proposed data augmentation technique is based on GAN (Generative Adversarial Networks)-based learning data augmentation technique that learns the object distribution of a given small amount of training data and creates objects within the distribution. In order to automatically generate label information of a defective object to be created, the region image of the defective object should be input to the generation model instead of the entire image used for learning. However, since the generative model learns the distribution of the entire input image, it learns not only the defect object but also the background distribution to generate an image, and the generated image has a background mismatch problem, making it difficult to use it as an augmented image. In the present invention, to solve this problem, the background of the defective part image is removed before applying the generative model through partial labeling and semantic segmentation network.

2 is a diagram showing the entire process of the data augmentation technique of the present invention.

Referring to FIG. 2, the entire process is (1) a process of creating a defect area image and a defect segmentation image through labeling of a defect image and learning a generation model with this data to generate an arbitrary defect area image, (2) manual work In order to reduce the labeling cost, manual labeling is performed on only a portion of the labeling process, and based on this, a semantic segmentation model is learned to segment the defect area. (3) The average color or density of the defect area and defect background It consists of a process of generating a defect image by learning with a classification model to derive an appropriate background color or density for the defect generated in step (1), importing a normal image and pasting the defect to a location corresponding to the appropriate color or density. . Here, process (2) is selectively executable according to the data situation.

Again, looking at the entire process of the data augmentation technique of the present invention,

First, labeling of a given small amount of training data is performed. At this time, along with a bounding box generally used for object detection, a segmentation map of the defective object is also created for background removal. However, since object segmentation labeling in pixel units is much more expensive than bounding box labeling, only part of the labeling is performed.

Second, a semantic segmentation model is trained using the image of each object's bounding box area cut out and the manually labeled defective object segmentation map. Google's DeepLabV3+ can be used as a semantic segmentation network. When the learning of the semantic segmentation network is completed, it is applied to the bounding box area object image without segmentation label to obtain a segmentation map capable of segmenting the defective object.

Third, the background is removed from the bounding box area image of each defect object using the segmentation maps obtained in the first and second steps to create a defect image without a background, which is input to a GAN (Generative Adversarial Networks)-based data generation model. to learn the distribution of defect data. In the present invention, f-DAGAN (feature-based DAGAN), which was developed based on DAGAN (Data Augmentation GAN), was used as a data generation network.

Figure 2a is a diagram showing the f-DAGAN network structure. f-DAGAN has the structure of FIG. 2a and has the feature of learning the distribution through two images and performing discriminator learning at both the feature vector level and the generated image level.

Fourth, the created defect object is pasted into the defect-free image data. However, since the defective object created by f-DAGAN is created according to the distribution of learning data, the color or density of the created object may appear in various ways. Therefore, if a defective object is pasted at an arbitrary location, it may be difficult to identify or an unnatural result may be obtained if it does not match the color or density of the background of the pasted location. To solve this problem, a step of learning a classification model capable of predicting the color or density of the background for the object area of the training data is added. To this end, a classification model is trained with the background-removed object area image as input and the average color or density of the removed background as output. As a background color or concentration classification model, SE-Net, etc., which shows excellent performance by readjusting learning features according to importance, can be used. If the defect object created through the third step is entered into the learned background color or density classification model, the background color or density suitable for the created object can be obtained.

Finally, the final image is completed by pasting the defective object created in the third step to the normal product image without defects. At this time, the position to be pasted is randomly selected. If the background color or density of the selected position is not within the appropriate background color or density range obtained in the fourth step, the position to be pasted is searched until the background color or density condition is satisfied.

3 is a flowchart showing the entire process of the data augmentation technique of FIG. 2 in more detail.

Referring to FIG. 3, the detailed steps of the entire process are shown in a flowchart, which includes (1) separating a defect image and a normal image from the entire data, (2) the type of defect for each defect in the defect image. (3) cutting out the defect location from the defect image and saving it as a defect area image, (4) selecting some of the defect area images and displaying them as a defect segmentation image; (5) learning a defect segmentation model using the defect segmentation image and the defect segmentation image; (6) extracting the defect segmentation image by inserting the defect area unmarked defect segment image into the learned defect segmentation model; Steps of learning a defect generation model using the defect region images and defect regions obtained through (3) to (6), (8) generating a desired number of random defects through the learned defect generation model, (9) Deriving the average background color or density for each defect using the defect area image and defect area obtained through steps (3) to (6), (10) Background color or density classification model using the defect area and average background color or density , (11) putting one defect generated in step 8 into the learned background color or density classification model to derive an appropriate color or density range, (12) taking a normal image for the defect in step 11 and titrating It consists of pasting in a position with a range of colors or densities. Here, steps (11) to (12) are repeated for each defect created in step (8).

- Implementation results and evaluation

4 is a diagram showing a defective object sample image, and FIG. 5 is a diagram showing a defect object sample image from which a background has been removed.

In the proposed technique, the semantic segmentation and background concentration classification models were implemented with the Pytorch platform, and the f-DAGAN model was implemented with the TensorFlow platform. Training was conducted in an AMD RYZEN Threadripper 1920x CPU and two Nvidia RTX 2080TI GPU environments.

A total of 100 images were given to evaluate the proposed technique, consisting of 60 images with defects and 40 images without defects. Each image is 1280×1280 in size and has 256 densities of grayscale. The defect images are classified into four types: black spots, white spots, scratches, and stains. Fig. 4 shows sample images cut from the left based on the bounding box for defect objects of black, white, contamination, and scratch types. The result of removing the background using the semantic segmentation map obtained through the first and second steps of each sample image is shown in FIG. 5 .

6 is a diagram illustrating a generated defective object sample image.

6 shows samples of each type of defective object created through f-DAGAN. The order of defect types is the same as in FIG. 4 . Since the image with the background removed was learned, it can be confirmed that the generated image also has no background.

7 is a diagram illustrating a defective object sample image in which a background is combined.

Finally, Fig. 7 shows the result of pasting the created defect object image to the normal background image and cutting out the bounding box area. At this time, it can be seen that a background having an appropriate concentration has been selected through a process of finding a position to paste using the learned background concentration classification model.

<Table 1>

Table 1 is a table showing the object detection accuracy of the model trained with the generated data.

In order to check the quality of the generated augmented image, performance evaluation was performed by using it for object detection network training. The YOLOv5L6 model was used as the object detection network, and the results of learning with the dataset generated by the data generation program that was developed and used in-house were compared. Table 1 shows the object detection learning accuracy results of the generated augmented data. Both the proposed data augmentation technique and the program-based augmentation technique generated 10K size data for each defect type and trained for 60 epochs. For accuracy results, Box F1-Score with an IoU threshold of 50% was used. As a result of the evaluation, it can be seen that the dataset created by the proposed method has better learning efficiency than the dataset created by a program that creates virtual defects using a geometric method.

- conclusion

The present invention is a learning method using GAN-based training data augmentation technique, semantic segmentation-based background processing, color or density classification-based background selection, etc. together to automatically create a high-quality dataset that can solve the problem of insufficient object detection training data. A data augmentation technique is proposed. In addition to the high learning efficiency of the generated dataset, the proposed technique has the advantage of high usability because it automatically learns the distribution and performs augmentation for various types of objects. Although the proposed technique is currently geared toward augmenting objects with relatively homogeneous backgrounds, it could be applied to augmentation of objects with more complex backgrounds.

As such, those skilled in the art to which the present invention pertains will be able to understand that the present invention may be embodied in other specific forms without changing its technical spirit or essential features. Therefore, the embodiments described above should be understood as illustrative in all respects and not limiting. The scope of the present invention is indicated by the following claims rather than the detailed description above, and all changes or modifications derived from the meaning and scope of the claims and their equivalent concepts should be interpreted as being included in the scope of the present invention. do.

Claims (4)

A first step of labeling a given small amount of training data while generating a segmentation map of a defective object for background removal along with a bounding box used for object detection;
A semantic segmentation model is trained using the image of the bounding box region of each object cut out and the manually labeled defective object segmentation map. a second step of obtaining a segmentation map capable of segmenting ;
In the bounding box area image of each defect object, the background is removed using the segmentation maps obtained in the first and second steps to create a defect image without a background, and the defect image is input to a GAN (Generative Adversarial Networks)-based data generation model to generate defects. a third step of learning the distribution of data; and
A fourth step of generating a defect image by inputting the generated defect object into the learned background color or density classification model to calculate the background color or density suitable for the generated object, and then pasting the generated defect object into image data without defects. ;
Deep learning-based data augmentation method for learning product defect detection, including a.
A first step of separating a defective image and a normal image from all data;
A second step of displaying the type and location (bounding box) of each defect in the defect image;
A third step of cutting out a region where a defect is located in a defect image and saving it as a defect region image;
A fourth step of selecting some of the defect part images and displaying them as a defect segmentation image;
A fifth step of learning a defect segmentation model using the defect region image and the defect segmentation image;
A sixth step of extracting a defect segmentation image by inserting the defect area image of the defect area unmarked into the learned defect segmentation model;
A seventh step of learning a defect generation model using the defect region image and the defect region obtained through the third to sixth steps;
An eighth step of generating a desired number of random defects through the learned defect generation model;
a ninth step of deriving an average background color or density for each defect using the defect area image and the defect area obtained through the third to sixth steps;
A tenth step of learning a background color or density classification model using the defect area and the average color or density of the background;
An eleventh step of deriving an appropriate color or density range by inserting the defect generated in the eighth step into the learned background color or density classification model; and
a twelfth step of taking a normal image of the defect of the eleventh step and pasting it into a position having an appropriate color or density range;
Deep learning-based data augmentation method for learning product defect detection, including a.
According to claim 2,
A deep learning-based data augmentation method for product defect detection learning, characterized in that steps 11 to 12 are repeatedly performed for each defect generated in step 8.
According to claim 2 or 3,
The normal image is an image without a defect, and the defect image is an image with a defect,
The defect portion image is an image obtained by rectangularly cutting out a defect portion from the defect image,
The defect segmentation image is an image in which the defect area is displayed in black and the other portion is displayed in white in the defect portion image,
The defect background is an image displaying the rest of the defect area image except for the defect area,
The deep learning-based data augmentation method for product defect detection learning, characterized in that the defect area image means an image in which a defect background is displayed in black in the defect portion image.

Images