KR20170127269A - Method and apparatus for detecting and classifying surface defect of image - Google Patents

Abstract

The present invention relates to a method and an apparatus for accurately detecting and easily classifying defects on an image surface. The method comprises: detecting defects on an image surface through filtering using an adjacent difference filter which emphasizes differences between pixels in a specific pixel area and removes similarity; and classifying the defects in the filtered image by feature by using an algorithm.

Description

METHOD AND APPARATUS FOR DETECTING AND CLASSIFYING SURFACE DEFECT OF IMAGE FIELD OF THE INVENTION [0001]

The present invention relates to a method and apparatus for detecting and classifying defects on an image surface and more particularly to detecting defects by filtering with an adjacent difference filter emphasizing differences between specific pixel regions and eliminating similarities, And a method and apparatus for extracting feature values from data and classifying the defects by type.

Recently, with the rapid growth of the mobile display industry, a display panel module and a production method thereof have been developed in a new form, so that a high resolution camera and various types of illumination are being applied to judge the quality of a display surface. However, as can be seen from FIG. 1, the image obtained by the camera according to various illumination positions and angles has a problem that a deviation occurs between the images.

As a result, every time the illumination or the camera angle changes, the technician continuously intervenes in the parameter setting, making it difficult to accurately set the parameters and the productivity is lowered.

In addition, even if the parameters are correctly set, in the case of highly integrated display parts, there is a texture pattern, and in order to detect defects present on the surface, it is necessary to be able to distinguish the defect from the texture pattern. However, There is a problem that the good product is recognized as a defective product due to the similarity between the pattern and the defect.

SUMMARY OF THE INVENTION The present invention has been made in view of the above problems,

We propose a neighboring difference filter which can extract the texture pattern and defects separately and extract only the defective area without distinguishing or adding the application areas even in local illumination difference, different illumination and camera angle.

Also, a method of easily classifying types of defects by extracting characteristic values of image surface defects filtered by the adjacent difference filter is proposed.

According to an aspect of the present invention, there is provided a method of detecting and classifying an image surface defect,

Receiving image data obtained by an image sensor; And a filtering step of performing at least a part of the pixels of the image data with a Neighboring Difference Filter (NDF), wherein the neighboring difference filter is a filter for filtering a specific area of pixels of the image data And may be a filter that compares a difference in the pixel value with an area spaced from the specific area by a predetermined distance.

The filtering may further include dividing the image data into a plurality of sub image data so that each pixel of the image data includes a plurality of sub pixels, Performing at least a portion of pixels of each sub-image data with the adjacent difference filter; And converting each sub-image data obtained by performing the composite multiplication into one image data.

The filtering may further include converting a plurality of subpixels included in each pixel into one pixel when each pixel of the image data includes a plurality of subpixels; And performing a synthesizing multiplication of at least a part of the converted pixels of the image data with the adjacent difference filter.

The image surface defect detection and classification method may further include the step of setting a predetermined threshold value to display a region having a lower brightness level or a higher brightness level than the background of the image among the defects of the image surface .

Preferably, the image surface defect detection and classification method includes: a feature extraction step of extracting feature values of filtered image data; And comparing the extracted feature value with a defect type DB to classify the defect of the image data.

In addition, the feature extraction step may extract at least a part of the geometric feature, the statistical feature, the intensity feature, and the texture feature of the filtered image data.

The feature extraction step may further include: calculating a co-occurrence matrix from each pixel of the filtered image data; And extracting a quantitative feature value of the filtered image data using the calculated concurrent matrices.

The image surface defect detection and classification method may further include updating the defect type DB using the extracted feature value and the classified result.

The adjacent difference filter is a square matrix including a first region, a second region surrounding the first region, and a third region surrounding the second region, and the value of the element in the first region is a positive value , The value of the element in the second region is 0, and the value of the element in the third region may be a negative value.

Preferably, the adjacent difference filter is a (2M + L) by (2M + L) size of the first region and a size of the first region and the second region is L by L, (2N + 2M + L) by (2N + 2M + L), the value of the element in the first region is 1 / (L ^ 2), and the size of the first region, the second region, , And the value of the element in the third region may be -1 / (4N * (L + 2M + N)).

According to an aspect of the present invention, there is provided an image surface defect detection and classification apparatus,

An image data receiving unit for receiving image data obtained by an image sensor; A filtering unit for performing at least a part of pixels of the image data with a Neighboring Difference Filter (NDF); A processor for extracting feature values of the filtered image data and comparing the extracted feature values with a defect type DB to classify defects of the image data; And a memory unit in which the defect information DB is stored, wherein the proximity difference filter compares a specific area of the image data pixels with a predetermined distance from the specific area to emphasize the difference of the pixel value Filter.

The defect of the image surface excluding the texture pattern can be more accurately detected using the adjacent difference filter according to the present invention,

It is possible to easily classify types of defects by extracting feature values of image surface defects.

BRIEF DESCRIPTION OF THE DRAWINGS The accompanying drawings, which are included to provide a further understanding of the invention and are incorporated in and constitute a part of the specification, illustrate embodiments of the invention and, together with the description, serve to explain the principles of the invention.
FIG. 1 shows a problem that a deviation occurs according to the position and angle of an illumination in the conventional display surface inspection.
FIG. 2 shows an example of a problem of recognizing a good product as a defective product because the defect can not be distinguished from the texture pattern of the conventional display.
FIG. 3 is a flowchart illustrating a method of detecting and classifying defects on an image surface according to the present invention.
4 is a view for explaining the manufacturing principle of an adjacent difference filter according to the present invention.
Figure 5 shows a proximity difference filter (NDF) according to the present invention.
6 shows a specific example of the production of an adjacent difference filter (NDF) according to the present invention.
FIG. 7 shows an example of image surface defects selectively extracted according to contrast.
8 is a sample image of an image surface defect.
FIG. 9 shows a result of filtering the sample image of FIG. 8 with an adjacent difference filter.
Figure 10 shows a graph of fitness values of the 3-CV data set according to the number of feature values and an accuracy value of an arbitrarily selected training set and test set.
11 compares the performance of the conventional filtering method and the filtering method according to the present invention.
FIG. 12 shows a result of applying PCA (Principal Components Analysis) to image data.
FIG. 13 shows a performance graph according to the number of trees of a random forest.
FIG. 14 shows a configuration of an image surface defect detection and classification apparatus according to the present invention.

BRIEF DESCRIPTION OF THE DRAWINGS The present invention is capable of various modifications and various embodiments, and specific embodiments will be described in detail below with reference to the accompanying drawings.

The following examples are provided to aid in a comprehensive understanding of the methods, apparatus, and / or systems described herein. However, this is merely an example and the present invention is not limited thereto.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the following description, well-known functions or constructions are not described in detail since they would obscure the invention in unnecessary detail. The following terms are defined in consideration of the functions of the present invention, and may be changed according to the intention or custom of the user, the operator, and the like. Therefore, the definition should be based on the contents throughout this specification. The terms used in the detailed description are intended only to describe embodiments of the invention and should in no way be limiting. Unless specifically stated otherwise, the singular form of a term includes plural forms of meaning. In this description, the expressions "comprising" or "comprising" are intended to indicate certain features, numbers, steps, operations, elements, parts or combinations thereof, Should not be construed to preclude the presence or possibility of other features, numbers, steps, operations, elements, portions or combinations thereof.

It is also to be understood that the terms first, second, etc. may be used to describe various components, but the components are not limited by the terms, and the terms may be used to distinguish one component from another .

Hereinafter, exemplary embodiments of a magneto-electric force microscope according to the present invention will be described in detail with reference to the accompanying drawings.

FIG. 3 is a flowchart illustrating a method of detecting and classifying defects on an image surface according to the present invention.

As shown in FIG. 3, the defect detection and classification method of the image surface according to the present invention includes a step S310 of receiving image data, a step S320 of filtering the received image, (Step S330).

First, in step S310, the image surface defect detection and classification apparatus 100 receives image data from an image sensor such as a camera, a scanner, or a microscope.

In step S320, the image data is filtered to facilitate defect detection. The filtering may be performed by convoluting pixel values of the image data using a neighboring difference filter (NDF) according to the present invention as shown in Equation 1 below.

[Equation 1]

Here, H x W is the size of the image data, and V _{(x, y)} is the image patch based on the (x, y) point.

In addition, the data for the color image among the image data may include a plurality of subpixels in one pixel. For example, an RGB image includes red, green, and blue subpixels. In this case, the filtering step according to the present invention can be performed in two ways. For convenience of explanation, it is assumed that an image including a sub-pixel is an RGB image, and is described below

First, the image data is divided into three sub-image data having only Red sub-pixels, Green sub-pixels, and Blue sub-pixels, and each sub-image data is filtered with an adjacent difference filter. Then, Data can be converted back to gray level.

Secondly, the Red subpixel, the Green subpixel, and the Blue subpixel of each pixel of the image data may be converted into one gray-level pixel, and then filtered by the adjacent difference filter. A method of converting each RGB subpixel to a gray level can be performed using, for example, an equation such as gray level pixel value = 0.299R + 0.587G + 0.114B.

Hereinafter, the neighborhood difference filter according to the present invention will be described in more detail with reference to FIGS. 4 to 6. FIG.

4 is a view for explaining the manufacturing principle of an adjacent difference filter according to the present invention.

The key idea of the neighborhood difference filter according to the present invention is a method of strategically comparing pixels of a specific region with pixels of a region spaced by a predetermined distance in order to emphasize the difference between each pixel of the image data and remove similarities.

As can be seen in FIG. 4 (a), adjacent image patches exhibit a similar pattern to each other. Also, as can be seen from FIG. 4 (b), an image patch having a pixel value that is not similar to the reference image patch is farther away from the reference image patch. That is, when each pixel is compared with an image including a texture pattern, it is difficult to grasp the characteristics of each pixel, but when compared with an image patch unit, it can be seen that a similar image patch is repeated in the adjacent region.

Therefore, it may be aimed to design an adjacent difference filter according to the present invention in which a specific area of the image data pixels is compared with an area spaced by a predetermined pixel distance from the specific area so that the difference of the pixel value is emphasized have.

Here, the image patch means a pixel region belonging to a predetermined range among the entire pixel regions of the image data.

5 shows an example of an adjacent difference filter (NDF) according to the present invention.

인 제1 영역, 상기 제1 영역을 둘러싸고 원소 값이 0인 제2 영역, 상기 제2 영역을 둘러싸고 원소 값이

인 제3 영역으로 구분될 수 있다. 여기서,

,

,

로 정의된다.As can be seen in Figure 5, the neighborhood difference filter according to the present invention is a square matrix,

A second region surrounding the first region and having an element value of 0, an element value surrounding the second region,

And a third region, which is the second region. here,

,

,

.

6 shows a specific example of the production of an adjacent difference filter (NDF) according to the present invention.

,

,

인 인접차이 필터이다.6 (b), l = 3, m = 2 and n = 2 are determined

,

,

Lt; / RTI >

In addition, the filtered image data may be classified into a relatively dark region or a bright region by comparing the background of the image of the defects on the image surface using Equation (2) below.

&Quot; (2) "

T is the threshold value and is always greater than zero. p ∈ {1, 0, 1} is the image normalization parameter, and if the brightness of the defect is lighter than the surrounding area (p = 1), the dark case (p = -1) Can be extracted.

FIG. 7 illustrates an example of a defect of image data selectively extracted according to brightness and darkness using Equation (2).

7 (a) is an actual image, and FIG. 7 (b) is a view showing an example in which a shadow part of the image surface is darker than the background in the condition that the adjacent difference filter is l = 10, m = 10, n = 10 and T = (p = 1). Fig. 7 (c) shows the defect of the image surface which is brighter than the background, (P = 0).

FIG. 8 is a sample image of an image surface defect, and FIG. 9 is a result of filtering the sample image of FIG. 8 with an adjacent difference filter.

(A) is scratch, (b) is long dust, (c) is Pit, and (d) is a condition where the adjacent difference filter is l = 10, m = 10, n = 10, Circle Dust, (e) represents Stain. Through the filtering process, it can be seen that the defects of the image surface are clearly distinguished from the background.

Adjacent difference filters according to the present invention have several advantages. First, it shows robust performance in defect detection as compared to other conventional methods, as can be seen in FIG. Secondly, by replacing the difference information with neighboring pixels at each position of the image, it is possible to remove the repetitive pattern and separate the defects. Such substitution can be effectively used to define feature values in the step of classifying defects. Third, it is possible to selectively distinguish a bright region or a dark region from a background in an image surface defect.

In step S330, a feature value of a defect of the filtered image data is extracted.

Specifically, the present invention proposes a set of feature values comprising a geometrical feature, an intensity feature, a statistical feature, and a textural feature. Among them, the texture feature is one of the most important features to define specifically what appears on the image surface.

In the present invention, a texture feature based on a co-occurrence matrix generated from an image filtered by an adjacent difference filter is used to extract feature values for an image. The coincidence matrix P _kl is an estimate of the joint probability of two pixels having a particular value i and j at a point d apart in different directions along the vector (k, l) or (-k, -l). The co-occurrence matrix is a matrix indicating the intensity distribution between pixels at a specific distance. Various secondary statistics that can represent the texture of the image and the like can be calculated from the co-occurrence matrix.

_x,

_y,

_x and

_y 는 각각 동시발생 행렬의 행(row)의 합 및 열(column)의 합의 평균 및 표준 편차를 의미한다. N_g는 이산 강도 레벨(discrete intensity level)의 수를 의미하고, p(i, j) = P_kl(i, j)이다.The measurement of the texture is made by the coincidence matrix, and the coincidence matrix is calculated using the following Table 1. < tb >< TABLE > here,

_x ,

_y ,

_x and

_y means the mean and standard deviation of the sum of the rows and the sum of the columns of the co-occurrence matrix, respectively. N _g is the number of discrete intensity levels, and p (i, j) = P _kl (i, j).

[Table 1]

In order to construct optimal feature values, first, all 44 features consisting of geometric, statistical, intensity and texture features are calculated, and then important feature values are selected from among them. The 33 texture feature sets extracted from the co-occurrence matrices of the image filtered by the neighborhood difference filter are divided into four directions (0, 180, 45, 225, 90, 270, (See Table 1) of three distances (d = 1, 2, 3) along the horizontal axis (135 °, -315 °). Table 2 below shows eleven feature sets composed of geometric features F1 to F6, statistical features F7 and F8, and intensity features F9 to F11 extracted from the image filtered by the adjacent difference filter .

[Table 2]

It is possible to extract the quantified characteristic values through the methods of Table 1 and Table 2, and it becomes possible to use the extracted values as input of the machine learning technique.

A three-fold cross validation (3-CV) method with elimination backward that starts with all variable sets and removes nonessential ones continuously can be used to determine important information and avoid duplication of information . The data set is divided into three, two-thirds of the data are arbitrarily selected and trained. The remaining data is used for verification.

To evaluate a set of variables along their predictive ability, a fitness value is used as a score for evaluating the variables of the set during each sequence. The fitness value is an average of mean accuracy values in the 3-CV data set.

Figure 10 shows a graph of fitness values of the 3-CV data set according to the number of feature values and an accuracy value of an arbitrarily selected training set and test set.

According to the number of features shown in FIG. 10, the fitness value begins to fall sharply in the last eight characteristic values. Therefore, it can be confirmed that more than 9 characteristic values should be selected in order to keep the result value constant. The selected features include three geometric features F1, F2, F3, one statistical feature F7, two intensity features F10, F11 and three texture features Tex2 for distance = 1, Tex4 for distance = 3, Tex11 for distance = 2).

Here, training data is data used to create an algorithm for classifying data, and test data means data specifically recognized for use in testing an algorithm.

In addition, the defect information DB used to classify defects may be updated through a learning process in addition to classifying defects by extracting feature values of the image surface.

In the present invention, feature values are trained using random forest, which is one of ensemble learning methods. A random forest is a set of multiple decision trees. It trains many different trees with bootstrap aggregating (bagging) and feature value bagging processes. The bagging allows each tree to be trained from a whole set of training data to a portion of arbitrarily selected data. The bagging process avoids excessive training for the system, and the trained trees are uncorrelated with each other. The feature value bagging process is used to arbitrarily select some of the input features in all tree nodes. This is to avoid repetition effects and correlation by features that strongly affect response variables during training. The random forest shows various effects depending on how to quantify the features.

However, the method of updating the classification method using the classification result of the image surface defect is not limited to the method using the random forest.

Experimental results on image surface defect detection and classification method according to the present invention will be described below.

A. Adjacent Difference Filter (NDF) Processing Result

Referring to Table 3 and FIG. 11, the performance of the adjacent difference filter according to the present invention can be confirmed. The quality of the filtered image can be determined by a ratio of correctly classified pixels (RCCP) per unit image. As shown in Table 3, it can be confirmed that the best performance is obtained in the adjacent difference filter according to the present invention.

[Table 3]

B. Classification Result

All types of defects in the data set were identified by a skilled engineer. The training data is set to 50% of the total data, and the remaining data is set to the test data. Table 4 below shows the number of defects.

[Table 4]

Three types of inputs are used in this experiment. Adjacent difference filter was applied as a preprocessing process in all feature value extraction methods.

The first method is to apply a multi-resolution pyramid image. The center of the boundary rectangle of the defects of each image filtered by the adjacent difference filter is aligned and four pyramid images of 8x8, 16x16, 32x32, and 64x64 pixels are used as feature value vectors.

The second method is to apply principal components analysis (PCA) to each image filtered by the neighborhood difference filter. By applying the PCA, it can be expected to reduce the redundancy of the input. The PCA is applied after adjusting the image to 28x28 size, and is applied to the PCA space of 10-dimensional and 20-dimensional, respectively, for training as a feature value vector, as can be seen in Fig.

The third method utilizes a feature combination consisting of geometric features, intensity features, statistical features and texture features disclosed in the present invention. Using the random forest, perform the six test methods in Table 5 below and compare them.

[Table 5]

In order to check the performance of the classification method according to the present invention, Precision = TP / (TP + FP), Recall = TP / (TP + FN), and Accuracy = FP + FN + TN). Here, true positives (TP) and true negatives (TN) denote the correct classification, and false positives (FP) and false negatives (FT) denote the wrong classification.

Additionally, the Precision refers to the proportion of samples that are actually positive in the sample that is expected to be positive and is also referred to as the fit rate. Further, the recall is also referred to as the detection rate in the ratio of the samples determined to be positive in the actually positive sample. Further, the Accuracy is also referred to as a determination rate by dividing TP and TN by the sum of all of them.

The evaluation results of the six experiments are shown in [Table 6], [Table 7] and [Table 8] below, and Table 9 below shows the training time and test time of each experiment. During the entire test on the test data the parameters of the adjacent difference filter were set to l = 10, m = 10, n = 10, T = 70 and p = 1. The random forest was set to 300 trees and a depth of 20.

[Table 6]

[Table 7]

[Table 8]

[Table 9]

Referring to FIG. 13, performance varies according to the number of random forest trees. The performance increases sharply until the number of trees is about 40, but it is maintained at a similar level thereafter.

As a result, it shows the best performance in terms of training time, classification time, and all aspects in the feature value input condition defined as nine at T6. Tables 10 and 11 below show the prediction results of each defect through the confusion matrix of T5 and T6.

FIG. 14 shows a configuration of an image surface defect detection and classification apparatus 100 according to the present invention.

14, the image defect detection and classification apparatus 100 according to the present invention may include an image data receiving unit 110, a filtering unit 120, and a processing unit 130. As shown in FIG. In addition, it may further include a memory unit 140 in which a defect information DB for defect classification is stored.

The image data receiving unit 110 may receive image data obtained by an image sensor such as a camera, a scanner, and a microscope from the image sensor.

The filtering unit 120 may filter the image data using a Neighboring Difference Filter (NDF) according to the present invention. The adjacent difference filter may compare a specific area of pixels of the image data with a predetermined distance from the specific area to emphasize a difference between the pixel values. That is, through the filtering process, the image surface defects and the background of the image are visually distinct (see FIGS. 8 and 9).

The processor 130 extracts the feature value of the filtered image data and compares the extracted feature value with the defect type DB to classify the defect of the image data. Also, the processing unit 130 may update the defect type DB through a learning process.

Meanwhile, the embodiments of the present invention described above can be written in a program that can be executed in a computer, and the created program can be stored in a medium.

The medium may be a storage such as a magnetic storage medium (e.g., ROM, floppy disk, hard disk, etc.), an optical reading medium (e.g. CD ROM, DVD, etc.) and a carrier wave Media, but is not limited thereto.

The foregoing description is merely illustrative of the technical idea of the present invention, and various changes and modifications may be made by those skilled in the art without departing from the essential characteristics of the present invention. Therefore, the embodiments described in the present invention are not intended to limit the technical spirit of the present invention, but are intended to be illustrative and not restrictive. The scope of protection of the present invention should be construed by the following claims, and all technical ideas within the scope of equivalents thereof should be construed as being included in the scope of the present invention.

100: image surface defect detection and classification device
110:
120: Filtering unit
130:
140:

Claims (12)

Receiving image data obtained by an image sensor; And
A filtering step of performing at least a part of pixels of the image data with a Neighboring Difference Filter (NDF) to perform a convolution,
Wherein the adjacent difference filter comprises:
Wherein the image data is a filter that compares a specific area of pixels of the image data with an area spaced by a predetermined distance from the specific area and emphasizes a difference between the pixel values.
The method according to claim 1,
Wherein the filtering step comprises:
When each pixel of the image data includes a plurality of subpixels,
Dividing the image data into a plurality of sub-image data so as to have only sub-pixels of the same type;
Performing at least a portion of pixels of each sub-image data with the adjacent difference filter; And
And converting each of the sub-image data obtained by performing the composite product into one piece of image data.
The method according to claim 1,
Wherein the filtering step comprises:
When each pixel of the image data includes a plurality of subpixels,
Converting a plurality of subpixels contained in each pixel into one pixel; And
Further comprising performing a synthetic product with at least some of the transformed pixels of the image data with the adjacent difference filter.
The method according to claim 1,
The image surface defect detection and classification method includes:
Further comprising the steps of setting a predetermined threshold value and displaying the defective image surface in a region having a lower brightness or a region having a higher brightness than the background of the image.
The method according to claim 1,
The image surface defect detection and classification method includes:
A feature extraction step of extracting feature values of the filtered image data; And
And comparing the extracted feature value with a defect type DB to classify defects in the image data.
6. The method of claim 5,
The feature extraction step may include:
Wherein at least some of the geometric, statistical, intensity, and texture features of the filtered image data are extracted.
6. The method of claim 5,
The feature extraction step may include:
Calculating a co-occurrence matrix from each pixel of the filtered image data; And
And extracting a quantitative feature value of the filtered image data using the calculated concurrent matrices.
6. The method of claim 5,
The image surface defect detection and classification method includes:
Further comprising the step of updating the defect type DB using the extracted feature value and the classified result.
The method according to claim 1,
Wherein the adjacent difference filter comprises:
A square matrix including a first area, a second area surrounding the first area, and a third area surrounding the second area,
Wherein the value of the element in the first region is a positive value, the value of the element in the second region is 0, and the value of the element in the third region is a negative value.
10. The method of claim 9,
Wherein the adjacent difference filter comprises:
Wherein the size of the first area is L by L and the size of the area including the first area and the second area is (2M + L) by (2M + L), and the size of the first area, (2N + 2M + L) by (2N + 2M + L).
Wherein the value of the element in the first region is 1 / (L ^ 2) and the value of the element in the third region is -1 / (4N * (L + 2M + N) And classification methods.
An image data receiving unit for receiving image data obtained by an image sensor;
A filtering unit for performing at least a part of pixels of the image data with a Neighboring Difference Filter (NDF);
A processor for extracting feature values of the filtered image data and comparing the extracted feature values with a defect type DB to classify defects of the image data; And
And a memory unit in which the defect information DB is stored,
Wherein the adjacent difference filter comprises:
Wherein the image data is a filter that compares a specific area of the image data pixels with an area spaced a predetermined distance from the specific area so that a difference between the pixel values is emphasized.
11. A computer program stored on a medium in combination with hardware to perform the image surface defect detection and classification method of any one of claims 1 to 10.

Images