KR20230116985A - System and method for detecting pit defects on film - Google Patents

Abstract

The present invention relates to a film pit defect detection system, wherein an inspection film image including image information about the inspection film is image-processed according to a predetermined rule-based machine vision technique, so that of the entire area of the inspection film image a pre-processing module extracting defect candidate regions in which pit defect candidates predicted as pit defects of the inspection film exist, and generating defect candidate region images including image information on the defect candidate regions; and a CNN binary classification module for classifying the defect candidate region images using a CNN algorithm to which a deep learning-based machine vision technique is applied, and determining a defect candidate region image in which the pit defect actually exists among the defect candidate region images. .
According to the present invention, in order to detect pit defects in a film manufacturing environment with weak features and insufficient learning data, localization is performed using a preprocessing module to which rule-based machine vision is applied, CNN binary classification to which deep learning-based machine vision is applied By implementing a 2-Stage detection model that performs classification using modules, stable pit defect detection performance can be secured compared to conventional deep learning-based machine vision composed of a 1-Stage detection model that simultaneously performs localization and classification.

Description

System and method for detecting pit defects on film}

The present invention relates to a film pit defect detection system and method.

A pit defect is a defect that occurs due to a fine stamping phenomenon among various defects that occur in a manufacturing process of a film used for a display panel, and has a fine size that is difficult to identify with the naked eye. Automatic detection of these pit defects is a very important technology for automating the film manufacturing process.

Conventionally, in the manufacturing field, rule-based machine vision, which performs inspection work according to predetermined rules, is mainly used to automatically detect defects in a product during a manufacturing process. However, conventional rule-based machine vision shows high performance when feature information is limited and distinct, but is unsuitable for use in systems in which various types of defects such as pit defects in the film manufacturing process occur irregularly. there was.

Recently, in the manufacturing field, unlike rule-based machine vision, which allows inspection work to proceed only within predetermined rules, it is possible to inspect atypical items through data learning, and various types of defects can be detected through excellent data mining performance. The dissemination of deep learning based machine vision, which can accurately and quickly detect , is increasing.

Machine vision based on deep learning continues to evolve and is used in many fields. However, unlike the refined training data set that is open to the public for deep learning-based machine vision learning, in the real world, data with irregular sizes, colors, and patterns occur even for the same class. Among them, a low detection rate for small objects, in particular, remains a problem to be solved in the field of deep learning-based machine vision.

The Faster R-CNN model, which is popularly used in the object detection field, and other conventional machine vision based on deep learning generate feature maps by performing iterative convolution and downsampling operations. Accordingly, in the conventional deep learning-based machine vision, information on small objects is lost in repetitive convolution and downsampling processes, making normal learning difficult. In addition, in the conventional deep learning-based machine vision, since the anchor box is designed to be suitable for medium and large objects, the anchor box does not fit within the size range of small objects, resulting in poor performance.

As described above, the conventional deep learning-based machine vision has a problem in that it is not suitable for detecting small objects. In order to solve this problem, methods for improving detection performance by modifying the neural network architecture of deep learning-based machine vision or applying properties suitable for small object detection have been studied.

In conventional studies on machine vision based on deep learning for small object detection, a training data set containing at least 300 training images is used, and the images included in these trained data sets each have distinct features in color, texture, pattern, etc. and is well refined to include objects of various sizes.

However, pit defects exist at a very small ratio of about 1/480,000 in the film transmission image. Accordingly, conventional deep learning-based machine vision for detecting small objects has a high probability of feature loss of pit defects, and detection that requires predicting both the location and class of pit defects due to the characteristics of pit defects without distinct features. There are problems in that it is difficult to learn the model, and it is difficult to secure a minimum amount of training data required for detecting pit defects due to the low occurrence frequency of pit defects.

An object of the present invention is to provide an improved film pit defect detection system and method to precisely detect pit defects occurring in a film manufacturing process.

A film pit defect detection system according to a preferred embodiment of the present invention for solving the above problems is image processing of an inspection film image including image information on a inspection film according to a predetermined rule-based machine vision technique, Defect candidate regions in which pit defect candidates predicted as pit defects of the test film exist are extracted from the entire region of the inspection film image, and defect candidate region images including image information on the defect candidate regions are generated. a pre-processing module; and a CNN binary classification module for classifying the defect candidate region images using a CNN algorithm to which a deep learning-based machine vision technique is applied, and determining a defect candidate region image in which the pit defect actually exists among the defect candidate region images. .

A film pit defect detection method according to another preferred embodiment of the present invention for solving the above problems is image processing of an inspection film image including image information on a inspection film according to a predetermined rule-based machine vision technique, Extracting defect candidate regions in which pit defect candidates predicted as pit defects of the inspection film exist among the entire region of the inspection film image, and obtaining defect candidate region images including image information on the defect candidate regions localization step to create; and a classification step of classifying the defect candidate area images using a CNN algorithm to which a deep learning-based machine vision technique is applied, and discriminating a defect candidate area image in which the pit defect actually exists among the defect candidate area images.

The present invention relates to a film pit defect detection system and method, and has the following effects.

First, in the present invention, in order to detect pit defects in a film manufacturing environment with weak features and insufficient learning data, localization is performed using a preprocessing module to which rule-based machine vision is applied, CNN binary classification to which deep learning-based machine vision is applied By implementing a 2-Stage detection model that performs classification using modules, stable pit defect detection performance can be secured compared to conventional deep learning-based machine vision composed of a 1-Stage detection model that simultaneously performs localization and classification.

Second, the present invention can reduce training errors of the CNN binary classification module due to insufficient training data by increasing or decreasing the training data using a data augmentation module that implements a data augmentation technique suitable for the characteristics of pit defects.

1 is a block diagram showing a schematic configuration of a film pit defect detection system according to a preferred embodiment of the present invention.
2 is a block diagram showing a schematic configuration of a preprocessing module and a CNN binary classification module shown in FIG. 1;
Figure 3 is a film image for inspection.
Figure 4 is a film image for explaining the characteristics of pit defects.
5 is a block diagram showing a schematic configuration of the data augmentation module shown in FIG. 1;
6 is a block diagram for explaining a film pit defect detection method using the film pit defect detection system shown in FIG. 1;
Fig. 7 is a block diagram for explaining a method of implementing the localization step shown in Fig. 6;
8 is a block diagram for explaining a method of implementing the data augmentation step shown in FIG. 6;

Hereinafter, some embodiments of the present invention will be described in detail through exemplary drawings. In adding reference numerals to components of each drawing, it should be noted that the same components have the same numerals as much as possible even if they are displayed on different drawings. In addition, in describing an embodiment of the present invention, if it is determined that a detailed description of a related known configuration or function hinders understanding of the embodiment of the present invention, the detailed description will be omitted.

In describing the components of the embodiment of the present invention, terms such as first, second, A, B, (a), and (b) may be used. These terms are only used to distinguish the component from other components, and the nature, order, or order of the corresponding component is not limited by the term. In addition, unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by a person of ordinary skill in the art to which the present invention belongs. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning in the context of the related art, and unless explicitly defined in the present application, they should not be interpreted in an ideal or excessively formal meaning. don't

1 is a block diagram showing a schematic configuration of a film pit defect detection system according to a preferred embodiment of the present invention, and FIG. 2 is a block diagram showing a schematic configuration of a preprocessing module and a CNN binary classification module shown in FIG. 3 is a film image for inspection.

1 and 2, the film pit defect detection system 1 according to a preferred embodiment of the present invention includes an inspection film image including image information on the inspection film for which pit defects F need to be detected. (F0) is image-processed according to a predetermined rule-based machine vision technique to extract defect candidate regions in which pit defect candidates predicted as pit defects (F) of the film for inspection exist, and to at least one of the defect candidate regions. The pre-processing module 10 for generating defect candidate area images F2 each including image information for the test from the film image F0 for inspection, and the defect candidate area images using a CNN algorithm to which a deep learning-based machine vision technique is applied. A CNN binary classification module 20 for classifying (F2) and determining a defect candidate region image (F2) in which pit defect (F) actually exists among defect candidate region images (F2), and a pre-refined training film image It may include a data augmentation module 30 that generates a plurality of training image patches for training of the CNN binary classification module 20 from.

In the present specification, the film is preferably a film applied to a display panel. However, it is not limited thereto, and in the present specification, the film may be a variety of films applicable to various products other than display panels.

In addition, in the present specification, a film image refers to a transmission image generated by photographing the film with a camera in a state in which light for inspection is transmitted through the film.

First, the preprocessing module 10 uses rule-based machine vision to perform localization to extract defect candidate region images F2 in which pit defect candidates are located, respectively, from a film image F0 for inspection. It is a device.

The pre-processing module 10 image-processes the inspection film image F0 according to a predetermined rule-based machine vision technique to generate defect candidate region images F2. Classification by the CNN binary classification module 20 (Classification) is prepared so that it can be performed in advance. Through this, the preprocessing module 10 prevents loss of pit defect information that occurs when the CNN binary classification module 20 simultaneously performs localization and classification without the preprocessing module 10, and CNN binary classification It can perform a function of reducing the amount of calculation of the module 20. Here, the inspection film image F0 refers to an image generated by photographing an inspection film for which the pit defect F needs to be detected.

The structure of this preprocessing module 10 is not particularly limited. For example, as shown in FIG. 2 , the pre-processing module 10 may include a region-of-interest extraction unit 12, a candidate region extraction unit 14, and the like.

The ROI extracting unit 12 processes the inspection film image F0 according to a rule-based machine vision technique to obtain a region of interest including information about the inspection film among the entire area of the inspection film image F0. A1) is prepared to be selectively extracted. Through this, the ROI extracting unit 12 may perform a function of generating the ROI image F1 including the ROI A1 from the film image F0 for inspection.

As shown in FIG. 3 , when the inspection film image F0 is generated by photographing the inspection film, the inspection film image F0 includes image information about the region of interest A1 where the inspection film is present. and image information about the background area A2 where the inspection film is not located may be included together. When the pit defect detection operation is performed on the film image including the image information on the background area A2, errors may occur in the pit defect detection result due to the image information on the background area A2. Accordingly, the ROI extracting unit 12 extracts the ROI image F1 selectively including only the image information on the ROI A1 from the inspection film image F0.

The configuration of this region-of-interest extraction unit 12 is not particularly limited. For example, as shown in FIG. 2 , the ROI extraction unit 12 includes a resizing unit 12a, a black and white conversion unit 12b, a binarization unit 12c, a labeling unit 12d, It may have an image cropping unit 12e or the like.

The resizing unit 12a generates a second inspection film image by reducing the resolution of the inspection film image F0 by a predetermined ratio. For example, the resizing unit 12a reduces the resolution of the film image F0 for inspection to 2048*1500, which is a quarter size, when the film image F0 for inspection has a high resolution of 8192*6000. , it is possible to generate a film image for the second inspection.

The black-and-white conversion unit 12b converts the second inspection film image into black and white to generate a third inspection film image composed of the black-and-white image.

The binarization unit 12c generates a fourth inspection film image by performing binarization on the third inspection film image based on a predetermined threshold. According to this binary hatching, the region of interest (A1) where the test film is present among the entire regions of the fourth test film image is displayed brighter than the background region (A2).

The labeling unit 12d performs a labeling operation on the binarized fourth inspection film image, and converts the fourth inspection film image into a bright area corresponding to the inspection film and a dark area corresponding to the background area A2. divide That is, the labeling unit 12d divides the binarized fourth inspection film image according to brightness.

The image cropping unit 12e cuts out a bright area from the fourth inspection film image subjected to the labeling operation as an area of interest A1 including image information on the inspection film, and includes the image information on the area of interest A1. A region-of-interest image F1 is created.

The candidate region extraction unit 14 processes the region-of-interest image F1 according to a predetermined rule-based machine vision technique to remove noise included in the region-of-interest image F1 so as to be part of the pit defect F. It is provided to improve the visibility of pit defect candidates while selectively extracting predicted pit defect candidates. Through this, the candidate region extraction unit 14 may perform a function of generating defect candidate region images F2 each including image information on at least one pit defect candidate from the region of interest image F1. .

The configuration of this candidate region extraction unit 14 is not particularly limited. For example, as shown in FIG. 2 , the candidate region extraction unit 14 includes a median filtering unit 14a, a morphology calculation unit 14b, a binarization unit 14c, a labeling unit 14d, , an image cropping unit 14e, and the like.

The median filtering unit 14a performs median filtering or image blurring to mitigate noise in units of pixels on the ROI image F1 input from the ROI extraction unit 12, 2 Create an image of the region of interest.

The morphology calculation unit 14b generates a third ROI image by performing a morphology operation to remove light or dark noise in units of small regions included in the second ROI image.

The morphology calculator 14b performs an opening operation to remove bright noise and a closing operation to remove dark noise, and then calculates the difference between the opening operation result image and the closed association result image, thereby performing a fit from the second region-of-interest image. Pit defect candidates predicted to be part of the defect F are extracted in units of pixels. Through this, the morphology calculation unit 14b generates a third region-of-interest image from which pit defect candidates are clearly image-processed and noise included in the normal region A3 in which pit defect candidates do not exist is removed, thereby generating a defect candidate region. The operation of the CNN binary classification module 20 may be reduced by minimizing the number of generated images F2.

The binarization unit 14c generates a fourth ROI image by performing binarization on the third ROI image based on a predetermined threshold. According to the binarization unit 14c, pixels belonging to the pit defect candidate among all pixels included in the fourth region-of-interest image are displayed brighter than pixels belonging to the normal region A3.

The labeling unit 14d performs a labeling operation on the binarized fourth ROI image, and divides the fourth ROI image into bright pixels belonging to pit defect candidates and dark pixels belonging to the normal region A3.

That is, the labeling unit 14d divides the binarized fourth region-of-interest image according to brightness so that pit defect candidates having pixel units can be grouped into one region-by-region.

It is preferable that the labeling unit 14d performs 4-way connectivity labeling. In this case, the labeling unit 14d examines the top, bottom, left, and right pixels of each pixel with respect to all pixels included in the fourth region-of-interest image, and if it is determined that they are pixels of the same type as the center pixel, the center pixel and the top, bottom, left, and right pixels of each pixel are examined. The pixels are grouped into one area. For example, if the center pixel and the top, bottom, left, and right pixels all correspond to bright pixels, the labeling unit 14d ties the center pixel and the top, bottom, left, and right pixels into one area, and the center pixel and the top, bottom, left, and right pixels all correspond to dark pixels. Then, the center pixel and the top, bottom, left and right pixels are grouped into one area.

The image cropping unit 14e groups and crops at least one pit defect candidate for each region from the labeled fourth region-of-interest image, and cuts out at least one pit defect candidate among all pit defect candidates included in the fourth region-of-interest image. Defect candidate area images F2 each including image information are generated.

As described above, pit defects correspond to small objects that are remarkably small in size. Accordingly, the image cropping unit 14e uses only regions of a predetermined standard size or less among the entire regions generated by grouping bright pixels into one region by the labeling unit 14d as the defect candidate region image F2, and uses the fourth region of interest as the defect candidate region image F2. It is desirable to selectively extract from an image and remove regions exceeding a standard size as noise. Here, the standard size is not particularly limited. For example, the image cropping unit 14e may, on a per-pixel basis, determine if the area exceeds 10*10, the horizontal width exceeds 10, or the vertical width among regions generated by grouping bright pixels into one region. Areas exceeding 10 may be removed as noise.

Next, the CNN binary classification module 20 uses a CNN-based model to which deep learning-based machine vision is applied, and pit defect candidates included in each of the defect candidate area images F2 input from the preprocessing module 10 described above. It is a device for performing classification to determine whether or not is actually a pit defect (F).

A CNN-based model usable as the CNN binary classification module 20 is not particularly limited. For example, the CNN binary classification module 20 may be composed of a ResNet Model, a MobileNet Model, a SqueezeNet Model, and the like.

The Resnet model is a representative CNN-based model with significantly improved performance. Assuming that the output value of a specific layer L is F(x), the output value x of L-1 is added to the next layer in the form of F(x)+x. By applying the forwarded Skip-Connection method, problems related to learning of deep-layer models can be solved.

The mobilenet model is a representative lightweight CNN-based model, which divides and calculates the output map for each channel and then sums each output map again. It can record the same image classification performance as the existing point-wise summation method.

The squeezenet model is another representative lightweight CNN model, which compresses channels using 1*1 convolution and then uses 1*1 and 3*3 convolution kernels again to make the output map the same as the number of channels in the original input. By introducing the Fire Module that generates , it is possible to record the same performance as the existing CNN-based model while greatly reducing the model size.

FIG. 4 is a film image for explaining characteristics of pit defects, and FIG. 5 is a block diagram showing a schematic configuration of the data augmentation module shown in FIG. 1 .

Next, the data augmentation module 30 processes the training film image according to a predetermined data augmentation technique, and generates a plurality of training image patches for learning of the CNN binary classification module 20 from the training film image. am.

In the case where the film image has a high resolution of 8192*6000 size, pit defects F exist with a size of 10*10 or less. As shown in FIG. 4 , the pit defects F mainly have a round dot shape. The pit defect (F) is characterized by having a bright value and having no other color information other than having a dark shade around it.

This pit defect (F) occurs very rarely in the film manufacturing process, and there is a limiting condition that the CNN binary classification module 20 must learn using a small number of film images for learning. Here, the training film image refers to an image generated by taking a training film refined according to a predetermined refinement method for learning the deep learning-based CNN algorithm provided in the CNN binary classification module 20.

As described above, the feature information of the pit defect (F) with low discrimination and the amount of insufficient training data become factors that degrade the learning performance of the CNN binary classification module 20. To solve this problem, the film pit defect detection system 1 includes a data augmentation module 30 arranged to extract a plurality of training image patches from a small number of training film images using a data augmentation technique.

Regarding data augmentation, a contrast difference due to a difference in brightness between a pit defect (F) and the surrounding area of the pit defect (F) may be an important consideration, and the location of shadows existing in the surrounding area of the pit defect (F) Depending on the pit defect (F), the fact that each pit defect (F) has slightly different characteristics may also be considered. Accordingly, the data augmentation module 30 extracts original image patches from the training film image, and then processes these original image patches using a predetermined data augmentation technique to generate a plurality of training film image patches from a small number of original image patches. can create

As shown in FIG. 5, the data augmentation module 30 includes an original extractor 32 for extracting original image patches from a film image for learning according to a predetermined extraction method, and an original image according to a predetermined data augmentation technique. An enhancement unit 34 that generates a plurality of film image patches for learning from enhancement image patches including at least patches, original image patches generated by the original extraction unit 32, and the enhancement unit 34 It may be provided with a storage unit 36, etc. in which the film image patches for learning, etc. are stored.

In addition, the data augmentation technique usable in the augmentation unit 34 is not particularly limited. For example, the augmentation unit 34 may generate a plurality of film image patches for learning from a small number of image patches for augmentation using at least one data augmentation technique among a brightness change technique, a rotation technique, and an inversion technique. In this case, as shown in FIG. 5, the enhancement unit 34 includes a brightness change unit 34a that generates a plurality of film image patches for learning from the image patches for enhancement according to a predetermined brightness change technique; A rotation unit 34b that generates a plurality of film image patches for learning from augmentation image patches according to a predetermined rotation technique, and an inversion unit that generates a plurality of film image patches for learning from the augmentation image patches according to a predetermined inversion technique ( 34c), etc.

Here, the type of image patch usable as an image patch for augmentation is not particularly limited. For example, an image patch for augmentation may include an original image patch and an image patch for training previously generated by a specific data augmentation technique among data augmentation techniques. That is, the brightness changing unit 34a, the rotating unit 34b, and the inverting unit 34c convert an image patch for learning previously generated by a specific data augmentation technique among the brightness variation technique, the rotation technique, and the inversion technique to the specific data. It can be used as an image patch for augmentation to implement other data augmentation techniques other than augmentation techniques.

The brightness change unit 34a may generate a plurality of training image patches from the augmentation image patch while gradually adjusting the brightness of the augmentation image patch within a predetermined brightness control range. A brightness control range of the augmentation image patch is not particularly limited. For example, the brightness change unit 34a may generate a plurality of training image patches from the augmentation image patch by adjusting the brightness of the augmentation image patch within a brightness control range of -30% to +30%. .

The rotation unit 34b may generate a plurality of training image patches having different arrangement angles from the augmentation image patch while rotating the augmentation image patch step by step within a predetermined rotation angle range. A rotation angle range of the augmentation image patch is not particularly limited. For example, the rotation unit 34b may generate a plurality of training image patches from the augmentation image patch by rotating the augmentation image patch within a rotation angle range of -180° to +180°.

The reversing unit 34c may generate a plurality of training image patches inverted in different directions from the augmentation image patch while inverting the augmentation image patch in a predetermined inversion direction. The inversion direction of the augmentation image patch is not particularly inverted. For example, the reversing unit 34c may generate a plurality of training image patches from the augmentation image patch by horizontally inverting or vertically inverting the augmentation image patch.

Table 1 below is a table showing an example in which the data augmentation module 30 generates a plurality of training image patches from training film images.

Type Pit No-Pit Train Test Train Test Original 45 5 45 5 B 780 85 780 85 R 830 90 830 90 F 140 15 140 15 B+R+F 25,070 2,725 25,070 2,725

- Pit: An image patch containing image information about a pit defect- No-Pit: Image patch containing image information about the area around the pit defect.

- Train: image patch used only for training of CNN binary classification model (20)

- Test: image patch used only for performance evaluation of CNN binary classification module (20)

- O: Original image patch

- B: Image patches created according to the Brightness technique

- R: image patch created according to the Rotation technique

- F: Image patch created according to Flip technique

- B+R+F: total image patch generated according to the data augmentation technique

Referring to Table 1, the original extractor 32 may be confused with 50 first original image patches including image information on pit defects (F) from a total of 20 training film images and pit defects (F). 50 second original image patches for the surrounding area of the pit defect F may be extracted respectively. Also, the augmentation unit 34 may apply the aforementioned data augmentation technique to each of the first original image patch and the second original image patch extracted by the original extractor 32 . Through this, the augmentation unit 34 can generate about 28,000 first training image patches for pit defects F from 50 first original image patches, and pit defects from 50 second original image patches ( It is possible to generate about 28,000 second training image patches for peripheral areas that may be confused with F).

In general, a conventional deep learning-based machine vision for detecting defects from a product is composed of a 1-Stage detection model that simultaneously performs localization and classification. If pit defects are detected from high-resolution inspection film images using such conventional deep learning-based machine vision, pit defects that are remarkably small in size compared to the overall size of the inspection film images in the process of repetitive convolution and down-sampling are detected. Problems such as loss of information and a problem that the detection speed of pit defects is significantly reduced due to the enormous amount of calculation due to the processing of high-resolution inspection film images based on deep learning.

By the way, the film pit defect detection system 1 localizes the film image F0 for inspection using rule-based machine vision to obtain a low-resolution defect candidate where the pit defect candidate is located from the film image F0 for inspection. A pit defect ( F) includes a CNN binary classification module 20 that determines the defect candidate region image F2 that actually exists. As such, the film pit defect detection system 1 is composed of a 2-Stage detection model that individually performs localization and classification, and thus may be referred to as IPCNN (Image Processing & Convolution Neural Networks). Through this, the film pit defect detection system 1 performs the localization and CNN binary classification module 20 according to the preprocessing module 10 for the 1-Stage detection operation of the conventional deep learning-based machine vision, which simultaneously performs localization and classification. It can be replaced with a 2-Stage detection operation that sequentially performs classification according to

Hereinafter, experimental results for evaluation of the pit defect detection performance of the film pit defect detection system 1 will be described.

In order to evaluate the pit defect detection performance of the film pit defect detection system 1, a pit defect detection experiment was performed on a total of 20 training film images including pit defects F. Here, the film image for learning refers to a film image refined to include the actual pit defect F.

In the case of the film pit defect system, the CNN binary classification module 20 performed a pit defect detection test on three embodiments composed of any one of the Resnet model, the Mobilenet model, and the Squeezenet model. In addition, in the case of a conventional deep learning-based machine vision composed of a 1-Stage detection model, pit defect detection tests for two comparative examples in which the deep learning-based machine vision is composed of either a Faster R-CNN model or a YOLO-V3 model was carried out.

The evaluation of the pit defect detection performance calculates the degree of overlap between the predicted bounding box and the labeled correct answer GT (Ground Truth), and if the threshold value is 50% or more and the class prediction value matches the correct answer value, the correct prediction is made. It was judged and carried out. The evaluation indexes used in the evaluation of such pit defect detection performance are TP (True Positive), when correct values are accurately predicted, TN (True Negative), when incorrect answers are predicted as incorrect, and FP, when incorrect answers are predicted as correct. (False Positive), and FN (False Negative) that did not predict the correct answer. In addition, using the above four indicators, precision and recall were calculated as shown in Equations 1 and 2 below.

Table 2 below shows pit defect detection performance of the preprocessing module 10 .

TP TN FP FN Precision Recall 97 0 193 5 33.4% 95.1%

In Table 2, the results for each indicator represent the average value of the results for a total of 20 learning film images. The preprocessing module 10 had a high detection rate of about 95% for finding pit defects (F) in the learning film image, but showed very low performance in terms of accuracy because the rate of falsely detected FPs was very high. However, in terms of finding 95% of the actual pit defects F, it can be seen that the performance of the preprocessing module 10 is excellent.

Table 3 below shows the CNN binary classification module 20 according to three embodiments composed of any one of the Resnet model, the Mobilenet model, and the Squeezenet model, and any one of the Faster R-CNN model and the YOLO-V3 model. Pit defect detection performance results for each training film image of conventional deep learning-based machine vision according to two comparative examples composed of 1-Stage detection models are shown.

Model TP TN FP FN Precision[%] Recall[%] Time[s] IPCNN_ResNet 121 179 82 29 59.6 80.7 1.51 IPCNN_MobileNet 123 200 76 28 61.6 81.3 0.42 IPCNN_SqueezeNet 127 182 83 26 59.9 82.7 1.33 Faster R-CNN 0 0 150 150 0 0 1.25 YOLO-V3 0 0 0 150 0 0 0.3

In the case of the CNN binary classification module 20 according to the three embodiments, the pit defect detection performance was tested based on using the training film image patches extracted and augmented from the training film image by the data augmentation module 30.

In addition, in Table 3, among the pit defect detection performance results of the CNN binary classification module 20 according to the three embodiments and the conventional deep learning-based machine vision according to the two comparative examples, the highest performance for each index is indicated in bold font .

First, the conventional deep learning-based machine vision according to the two comparative examples did not detect any pit defects (F), and in particular, the YOLO-V3 model could not predict even one pit defect (F).

In the case of the YOLO-V3 model, the loss of object information caused by convolution in the forward propagation process and the large anchor box size compared to the fit defect (F) are analyzed as the main causes, and from the 8192*6000 high-resolution learning film image In the case of detecting a small pit defect (F) of less than 10*10, the lack of learning data with only 20 learning film images is analyzed as an additional cause.

In the case of Faster R-CNN model, it predicts bounding boxes containing fit defects (F), but the Intersection over Union (IoU) is calculated with very low values due to large anchor boxes, and all bounding boxes due to poor localization performance. is classified as false detection.

Next, the CNN binary classification module 20 according to the three embodiments detects the pit defect F with an accuracy of about 60% or more and a detection rate of about 80% or more.

Looking at the characteristics of each embodiment, the MobileNet model and the SqueezeNet model showed better performance in terms of accuracy and detection rate than the ResNet model with the most learning parameters. In this regard, the film pit defect detection system 1, although it is possible to augment insufficient learning data using the data augmentation module 30, is not substantially for various patterns of pit defects F, so that possible occurrence It has a characteristic that it is difficult to generalize by specifying patterns of all types of pit defects F. However, according to Table 3, it is analyzed that the poor generalization performance of the data of the film pit defect detection system 1 is improved in the process of reducing the weight of the MobileNet model and the SqueezeNet model.

In addition, considering both speed and detection performance, it can be seen that the mobilenet model is most suitable for the CNN binary classification module 20 as it has about 3 times faster speed and the highest accuracy compared to the other two models. can

In addition, it can be seen that the mobilenet model is excellent in terms of calculation speed as well as pit defect detection performance because the difference in detection speed from the YOLO-V3 model, which is specialized for fast detection speed (time), is only 0.1 s.

In addition, the mobilenet model classifies about 70% of the defect candidate area images F2 extracted by the preprocessing module 10 as peripheral areas rather than pit defects F, which indicates that the pit defect classification performance is excellent. can know that

As described above, the film pit defect detection system 1 implements a 2-Stage detection model for detecting pit defects F in a film manufacturing environment having weak features and insufficient learning data. This film pit defect detection system 1 is based on localization and classification in order to reduce the detection performance degradation of small objects of the conventional deep learning-based machine vision composed of a 1-Stage detection model and learning errors due to insufficient training data. It is individually performed using the preprocessing module 10 and the CNN binary classification module 20, and by increasing or decreasing the learning data using the data augmentation module 30 that implements a data augmentation technique suitable for the characteristics of the pit defect (F), stable Pit defect detection performance can be secured.

6 is a block diagram for explaining a film pit defect detection method using the film pit defect detection system 1 shown in FIG. 1, and FIG. 7 is a block diagram for explaining a method for performing the localization step shown in FIG. , and FIG. 8 is a block diagram for explaining a method of implementing the data augmentation step shown in FIG. 6 .

Referring to FIG. 6 , the film pit defect detection method using the film pit defect detection system 1 may include a localization step (S 10), a classification step (S 20), and a data augmentation step (S 30). can

First, in the localization step (S 10), the inspection film image F0 including image information on the inspection film is image-processed according to a predetermined rule-based machine vision technique, so that the entire inspection film image F0 Among the regions, defect candidate regions in which pit defect candidates predicted as pit defects F of the inspection film exist are extracted, and defect candidate region images F2 including image information on the defect candidate regions are generated. To this end, as shown in FIG. 7 , the localization step ( S10 ) may include a region of interest ( A1 ) extraction step ( S12 ), a candidate region extraction step ( S14 ), and the like.

In the step of extracting the region of interest (A1) (S 12), the region of interest (A1) in which the film for inspection exists is selectively extracted from among the entire regions of the film image (F0) for inspection, and image information on the region of interest (A1) is extracted. A region of interest image F1 including a is generated. To this end, the step of extracting the region of interest A1 (S 12) includes a resizing step (S 121), a black and white conversion step (S 123), a binarization step (S 125), a labeling step (S 127), and an image A cropping step (S 129) and the like may be included.

In the resizing step (S 121), a second inspection film image is generated from the inspection film image F0 by reducing the resolution of the inspection film image F0 by a predetermined ratio.

In the black and white step ( S 123 ), the second inspection film image is converted to black and white to generate a third inspection film image composed of black and white images from the second inspection film image.

In the binarization step (S125), binarization is performed on the third inspection film image based on a predetermined threshold, so that the area of interest A1 from the third inspection film image is the background area where the inspection film does not exist ( A fourth inspection film image marked brighter than that of A2) is generated.

In the labeling step ( S127 ), a labeling operation is performed on the fourth inspection film image, and the fourth inspection film image is divided into a region of interest A1 and a background region A2.

In the image cropping step (S 129), the region of interest A1 is cut out from the fourth inspection film image subjected to the labeling operation in the labeling step (S 127), and the region of interest image including image information on the region of interest A1. (F1).

In the candidate region extraction step (S 14), defect candidate regions in which pit defect candidates exist are extracted from the region of interest image F1, and defect candidate region images including image information on at least one of the defect candidate regions ( F2) is created. To this end, as shown in FIG. 7 , the candidate region extraction step (S 14) includes a median filtering step (S 141), a morphology calculation step (S 143), a binarization step (S 145), and a labeling step (S 147). ), an image cropping step (S 149), and the like.

In the median filtering step ( S141 ), a second ROI image is generated from the ROI image F1 by performing median filtering to mitigate noise in units of pixels on the ROI image F1 .

In the morphology calculation step S143, a third ROI image is generated from the second ROI image by performing morphology calculation on the second ROI image to remove noise included in the second ROI image.

In the binarization step (S 145), binarization is performed on the third region-of-interest image, and pit defect candidates included in the third region-of-interest image are brightly displayed compared to the normal region A3 where the pit defect candidates do not exist. , A fourth ROI image is generated from the third ROI image.

In the labeling step ( S147 ), a labeling operation is performed on the fourth ROI image to divide the fourth ROI image into pit defect candidates and the normal region A3 .

In the image cropping step (S 149), at least one pit defect candidate is grouped and cut out from the fourth region-of-interest image labeled in the labeling step (S 147) by region, and the defect candidate region images F2 are formed into at least one pit defect candidate region. It is created to include image information for each of the defect candidates.

Next, in the classification step (S20), the defect candidate region images F2 are classified using the CNN algorithm to which the deep learning-based machine vision technique is applied, and the pit defect F among the defect candidate region images F2 is determined. A defect candidate region image F2 that actually exists is determined.

This classification step may be performed using a CNN-based model such as a ResNet Model, a MobileNet Model, or a SqueezeNet Model.

Next, in the data augmentation step (S30), a learning film image including image information on a predetermined learning film is processed according to a predetermined data augmentation technique, and a number of data for learning the CNN algorithm used in the classification step are processed. Create image patches for training. To this end, as shown in FIG. 8 , the data augmentation step (S 30) may include an original extraction step (S 32) and a learning patch generation step (S 34).

In the original extraction step (S32), original image patches are extracted from the training film image according to a predetermined extraction method.

In the training patch generation step (S34), a plurality of training film image patches are generated from the augmentation image patches including at least the original image patches according to a predetermined data augmentation technique. Here, the augmentation image patches may include original image patches generated in the original extraction step and training image patches previously generated in the learning patch generation step previously performed.

In the learning patch generation step (S 34), a brightness change step (S 342), a rotation step (S 344), an inversion step (S 346), and the like may be generated.

In the brightness change step (S342), a plurality of training image patches are generated from the augmentation image patches by stepwise adjusting the brightness of the augmentation image patches within a predetermined brightness control range.

In the rotation step (S344), a plurality of training image patches having different arrangement angles are generated from the augmentation image patches by stepwise rotation of the augmentation image patches within a predetermined rotation angle range.

In the inversion step (S346), a plurality of training image patches inverted in different directions are generated from the augmentation image patches by inverting the augmentation image patches in a predetermined inversion direction.

The above description is merely an example of the technical idea of the present invention, and various modifications and variations can be made to those skilled in the art without departing from the essential characteristics of the present invention.

Therefore, the embodiments disclosed in the present invention are not intended to limit the technical idea of the present invention, but to explain, and the scope of the technical idea of the present invention is not limited by these embodiments. The protection scope of the present invention should be construed according to the claims below, and all technical ideas within the equivalent range should be construed as being included in the scope of the present invention.

1: Film Pit Defect Detection System
10: preprocessing module
12: region of interest extraction unit 12a: resizing unit
12b: black and white conversion unit 12c: binarization unit
12d: labeling unit 12e: image cropping unit
14: candidate area extraction unit 14a: median filtering unit
14b: morphology calculation unit 14c: binarization unit
14d: labeling unit 14e: image cropping unit
20: CNN binary classification module
30: data augmentation module
32: original extraction unit
34: augmenting unit 34a: brightness changing unit
34b: rotating part 34c: inverting part
36: storage unit
F0: film image for inspection
A1: region of interest A2: background region
F1: region of interest image
F2: Defect Candidate Area Image
F: pit defect

Claims (26)

Pit defect candidates predicted as pit defects of the inspection film among the entire area of the inspection film image by image processing the inspection film image including image information on the inspection film according to a predetermined rule-based machine vision technique a pre-processing module that extracts the existing defect candidate regions and generates defect candidate region images including image information on the defect candidate regions; and
A CNN binary classification module for classifying the defect candidate area images using a CNN algorithm to which a deep learning-based machine vision technique is applied, and determining a defect candidate area image in which the pit defect actually exists among the defect candidate area images, Film pit defect detection system. According to claim 1,
The preprocessing module,
a region-of-interest extracting unit configured to selectively extract a region of interest in which the film for examination is present from among entire regions of the film image for examination, and generate a region-of-interest image including image information on the region of interest; and
and a candidate region extraction unit that extracts the defect candidate regions in which the pit defect candidates exist from the region-of-interest image, and generates the defect candidate region images. According to claim 2,
The region of interest extraction unit,
a binarization unit that performs binarization on the inspection film image based on a predetermined threshold value, and brightens the ROI among the entire area of the inspection film image compared to a background area in which the inspection film does not exist;
a labeling unit dividing the inspection film image into the ROI and the background area by performing a labeling operation on the inspection film image binarized by the binarization unit; and
and an image cropping unit configured to generate the ROI image by cutting out the ROI from the inspection film image subjected to the labeling operation by the labeling unit. According to claim 3,
The ROI extraction unit further has a resizing unit that reduces the resolution of the inspection film image by a predetermined ratio,
The film pit defect detection system of claim 1 , wherein the binarization unit performs binarization on the inspection film image resized by the resizing unit. According to claim 4,
The ROI extracting unit further has a black and white conversion unit for converting the inspection film image resized by the resizing unit to black and white;
The binarization unit performs binarization on the inspection film image converted to black and white by the black and white conversion unit, the film pit defect detection system. According to claim 2,
The candidate region extraction unit,
a morphology calculation unit to remove noise included in the ROI image by performing a morphology operation on the ROI image;
a binarization unit that performs binarization on the morphology-operated region-of-interest image by the morphology calculation unit, and brightens pit defect candidates included in the region-of-interest image compared to a normal region in which the pit defect candidates do not exist;
a labeling unit performing a labeling operation on the ROI image binarized by the binarization unit, and dividing the ROI image into the pit defect candidates and the normal region; and
At least one pit defect candidate region is grouped and cut out from the region of interest image labeled by the labeling unit, and the defect candidate region images are generated to include image information for at least one pit defect candidate among the pit defect candidates. A film pit defect detection system having an image cropping unit. According to claim 6,
The candidate region extraction unit,
Further comprising a median filtering unit performing median filtering for mitigating noise in units of pixels with respect to the ROI image;
The film pit defect detection system of claim 1 , wherein the morphology calculation unit performs morphology calculation on the region-of-interest image median-filtered by the median-value filtering unit. According to claim 1,
A data augmentation module for generating a plurality of training image patches for learning of the CNN binary classification module by processing a training film image including image information on a pre-refined training film according to a predetermined data augmentation technique. , film pit defect detection system. According to claim 8,
The data augmentation module,
an original extractor extracting original image patches from the learning film image according to a predetermined extraction method; and
A film pit defect detection system having an enhancement unit generating a plurality of film image patches for learning from image patches for enhancement including at least the original image patches according to a predetermined data augmentation technique. According to claim 9,
The enhancement unit has a brightness change unit for generating the learning image patches from the enhancement image patches by stepwise adjusting the brightness of the enhancement image patches within a predetermined brightness control range. Film pit defect detection system. According to claim 9,
The enhancement unit has a rotation unit configured to generate the training image patches having different arrangement angles from the enhancement image patches by stepwise rotation of the enhancement image patch within a predetermined rotation angle range, and film pit defect detection. system. According to claim 9,
The enhancement unit has an inversion unit for inverting the enhancement image patch in a predetermined inversion direction and generating the learning image patches inverted in different directions from the enhancement image patches. Film pit defect detection system. According to claim 9,
The image patches for enhancement include the original image patches and image patches for learning previously generated by the enhancement unit, the film pit defect detection system. Pit defect candidates predicted as pit defects of the inspection film among the entire area of the inspection film image by image processing the inspection film image including image information on the inspection film according to a predetermined rule-based machine vision technique a localization step of extracting the existing defect candidate regions and generating defect candidate region images including image information on the defect candidate regions; and
and a classification step of classifying the defect candidate area images using a CNN algorithm to which a deep learning-based machine vision technique is applied, and determining a defect candidate area image in which the pit defect actually exists among the defect candidate area images. Fault detection method. According to claim 14,
The localization step is
a region of interest extracting step of selectively extracting a region of interest in which the film for examination is present from among entire regions of the film image for examination, and generating a region of interest image including image information on the region of interest; and
and a candidate region extraction step of extracting the defect candidate regions in which the pit defect candidates exist from the ROI image to generate the defect candidate region images. According to claim 15,
In the step of extracting the region of interest,
A binarization step of performing binarization on the inspection film image based on a predetermined threshold value and brightly displaying the ROI of the entire area of the inspection film image compared to a background area in which the inspection film does not exist;
a labeling step of dividing the film image for inspection into the region of interest and the background region by performing a labeling operation on the film image for inspection binarized in the binarization step; and
and an image cropping step of generating an image of the region of interest by cutting out the region of interest from the inspection film image subjected to the labeling operation in the labeling step. According to claim 16,
The step of extracting the region of interest further includes a resizing step of reducing the resolution of the film image for inspection by a predetermined ratio,
In the binarization step, the film pit defect detection method of performing binarization on the inspection film image resized in the resizing step. According to claim 17,
The step of extracting the region of interest further includes a black and white conversion step of converting the film image for inspection resized in the resizing step to black and white,
In the binarization step, the film pit defect detection method of performing binarization on the inspection film image converted to black and white in the black and white conversion step. According to claim 15,
In the step of extracting the candidate region,
a morphology calculation step of removing noise included in the ROI image by performing a morphology calculation on the ROI image;
a binarization step of performing binarization on the morphologically calculated region-of-interest image, and brightly displaying pit defect candidates included in the region-of-interest image compared to a normal region in which the pit defect candidates do not exist;
a labeling step of dividing the ROI image into the pit defect candidates and the normal region by performing a labeling operation on the binarized ROI image;
An image cropping step of grouping and cropping at least one pit defect candidate by region from the labeled region-of-interest image, and generating the defect candidate region images to include image information about at least one pit defect candidate among the pit defect candidates, respectively. Including, film pit defect detection method. According to claim 19,
The step of extracting the candidate region further includes a median filtering step of performing median filtering for mitigating noise in units of pixels with respect to the ROI image;
Wherein the morphology calculation unit performs a morphology calculation on the median-filtered region-of-interest image. According to claim 14,
Further comprising a data augmentation step of generating a plurality of training image patches for learning of the CNN algorithm by processing a training film image including image information on a predetermined training film according to a predetermined data augmentation technique, film pit Fault detection method. According to claim 21,
The data augmentation step,
an original extraction step of extracting original image patches from the learning film image according to a predetermined extraction method; and
A film pit defect detection method comprising a learning patch generating step of generating a plurality of training film image patches from augmentation image patches including at least the original image patches according to a predetermined data augmentation technique. The method of claim 22,
The generating of the training patches includes a brightness changing step of generating the training image patches from the augmentation image patches by stepwise adjusting the brightness of the augmentation image patches within a predetermined brightness control range. detection method. The method of claim 22,
The generating patch for learning step includes a rotation step of generating the learning image patches having different arrangement angles from the augmentation image patches by stepwise rotating the augmentation image patch within a predetermined rotation angle range. , Film pit defect detection method. The method of claim 22,
The generating of the training patches includes a reversing step of inverting the augmentation image patches in a predetermined inversion direction to generate the training image patches inverted in different directions from the augmentation image patches, the film pit defect detection method. The method of claim 22,
Wherein the augmentation image patches include the original image patches and training image patches generated in advance in the learning patch generation step.