JP2022055953A - Defect classification device, defect classification method and program - Google Patents

Abstract

To provide a defect classification device, a defect classification method and a program capable of improving accuracy in categorizing defects present in a checkup object.SOLUTION: A defect classification device 10 comprises: a defect detection unit 120 which detects defects from a captured image of a checkup object; a feature amount calculation unit 130 which calculates a feature amount of the defect detected by the defect detection unit 120; a first classification unit 140 which classifies the defect detected by the defect detection unit 120 by using a learned model 40 learned by deep learning; and a second classification unit 160 which classifies the defect detected by the defect detection unit 120, based on the feature amount calculated by the feature amount calculation unit 130 and a classification result by the first classification unit 140.SELECTED DRAWING: Figure 3

Description

The present invention relates to a defect classification device, a defect classification method and a program.

There is known a technique for classifying defects existing in an inspection target based on an captured image of the inspection target. For example, Patent Document 1 discloses a visual inspection apparatus having an automatic defect classification function. The visual inspection apparatus disclosed in Patent Document 1 calculates the feature amount of the defect detected from the inspection image, and classifies the defect using the calculated feature amount. On the other hand, with the recent development of artificial intelligence (AI) technology, the development of a method for classifying defects existing in the inspection target using deep learning (deep learning) is also underway.

Japanese Unexamined Patent Publication No. 2006-266872

The method of classifying defects based on the feature amount as disclosed in Patent Document 1 has a problem that the classification performance does not improve for complicated defects. The main reason for this is that it is difficult to select the optimum feature amount for classification and to set the classification condition using the optimum feature amount for classification of complicated defects. On the other hand, the method of classifying defects using deep learning can classify complex defects with high accuracy, which is difficult with the method based on feature quantity, but the method based on feature quantity classification can classify them with high accuracy. Classification, which was relatively easy, can be difficult. The main reason for this is that deep learning needs to learn defects sorted by classification type in advance, but it is necessary to set a huge number of classification types in order to improve classification accuracy. In view of such a situation, it is required to improve the classification accuracy of defects existing in the inspection target.

The present invention has been made to solve the above-mentioned problems, and an object of the present invention is to provide a defect classification device or the like capable of improving the classification accuracy of defects existing in an inspection target.

In order to achieve the above object, the defect classification device according to the first aspect of the present invention is
A defect detection unit that detects defects from the captured image of the inspection target, and
A feature amount calculation unit that calculates the feature amount of the defect detected by the defect detection unit, and a feature amount calculation unit.
Using the trained model trained by deep learning, a first classification unit that classifies the defects detected by the defect detection unit, and a first classification unit.
A second classification unit for classifying the defects detected by the defect detection unit is provided based on the feature amount calculated by the feature amount calculation unit and the classification result by the first classification unit.

In order to achieve the above object, the defect classification method according to the second aspect of the present invention is:
Defect detection step to detect defects from the captured image of the inspection target,
A feature amount calculation step for calculating the feature amount of the defect detected in the defect detection step, and a feature amount calculation step.
Using the trained model trained by deep learning, a first classification step for classifying the defects detected in the defect detection step, and a first classification step.
Includes a second classification step for classifying the defects detected in the defect detection step based on the feature amount calculated in the feature amount calculation step and the classification result in the first classification step. ..

In order to achieve the above object, the program according to the third aspect of the present invention is
Computer,
Defect detection unit that detects defects from the captured image of the inspection target,
A feature amount calculation unit that calculates the feature amount of the defect detected by the defect detection unit,
A first classification unit that classifies the defects detected by the defect detection unit using a trained model trained by deep learning.
It functions as a second classification unit that classifies the defects detected by the defect detection unit based on the feature amount calculated by the feature amount calculation unit and the classification result by the first classification unit.

According to the present invention, it is possible to improve the classification accuracy of defects existing in the inspection target.

It is a block diagram which shows the hardware composition of the defect classification apparatus which concerns on Embodiment 1. FIG. (A) to (D) are diagrams showing examples of defects existing in the inspection target in the first embodiment, respectively. It is a block diagram which shows the functional structure of the defect classification apparatus which concerns on Embodiment 1. FIG. It is a figure which shows the example of the captured image which the inspection target was imaged in Embodiment 1. FIG. FIG. 3 is a conceptual diagram of a neural network used for classifying defects by the first classification unit in the first embodiment. (A) to (C) are diagrams showing examples of defects that are difficult to classify by features, respectively. (A) to (C) are diagrams showing examples of defects that are difficult to classify by deep learning, respectively. It is a figure which shows the example of the classification condition of the defect by the 2nd classification part in Embodiment 1. FIG. It is a flowchart which shows the flow of the defect classification process executed by the defect classification apparatus which concerns on Embodiment 1. FIG. It is a block diagram which shows the functional structure of the defect classification apparatus which concerns on Embodiment 2.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. The same or corresponding parts in the figure are designated by the same reference numerals.

(Embodiment 1)
FIG. 1 shows the configuration of the defect classification device 10 according to the first embodiment of the present invention. The defect classification device 10 is a device that inspects the inspection target by classifying the defects existing in the inspection target. Here, the inspection target is a product, a product, or the like to be inspected for defects. The inspection target is, for example, a film-shaped (sheet-shaped) member used in a lithium ion secondary battery (LIB: Lithium-Ion Battery). However, the inspection target is not limited to this and may be anything.

Defects existing in the inspection target are parts that may deteriorate the function, performance, etc. of the inspection target. Specifically, the defects existing in the inspection target include scratches, wrinkles, stains and the like existing in the inspection target, and foreign substances such as dust and dirt adhering to the surface of the inspection target.

2 (A) to 2 (D) show examples of defects existing in the inspection target. The dot-like defects shown in FIGS. 2 (A) and 2 (B) represent, for example, foreign substances such as dust and dirt adhering to the surface of the inspection target. Further, the linear defects shown in FIGS. 2C and 2D represent, for example, scratches existing on the surface of the inspection target.

As shown in FIG. 1, the defect classification device 10 includes a control unit 11, a storage unit 12, an input reception unit 13, a display unit 14, a communication unit 15, and an image pickup unit 16.

The control unit 11 includes a CPU (Central Processing Unit), a ROM (Read Only Memory), and a RAM (Random Access Memory). The CPU includes a microprocessor and the like, and is a central processing unit that executes various processes and operations. In the control unit 11, the CPU reads out the control program stored in the ROM and controls the operation of the entire defect classification device 10 while using the RAM as the work memory.

Further, the control unit 11 includes a processor for image processing such as a DSP (Digital Signal Processor) and a GPU (Graphics Processing Unit), and a buffer memory for temporarily storing the processed image. The control unit 11 processes the captured image in which the appearance of the inspection target is captured by using a well-known image processing method.

The storage unit 12 includes a non-volatile memory such as a flash memory and a hard disk, and serves as a so-called secondary storage device or auxiliary storage device. The storage unit 12 stores programs and data used by the control unit 11 to perform various processes. Further, the storage unit 12 stores data generated or acquired by the control unit 11 performing various processes. For example, the storage unit 12 stores the trained model 40, the training data 50, and the classification table 60. Details of the trained model 40 and the training data 50 will be described later.

The input receiving unit 13 includes an input device such as a keyboard, a mouse, a button, a touch pad, and a touch panel, and receives an operation input from the user. For example, the user can input a process start instruction and various settings in the defect classification device 10 by operating the input receiving unit 13.

The display unit 14 includes a display device such as a liquid crystal display and an organic EL (Electro Luminescence) display, and displays various images under the control of the control unit 11. For example, the display unit 14 displays an image showing the result of classification by the defect classification device 10.

The communication unit 15 includes a communication interface for communicating with an external device of the defect classification device 10. For example, the communication unit 15 communicates with an external device in accordance with well-known communication standards such as LAN (Local Area Network) and USB (Universal Serial Bus).

The image pickup unit 16 includes a lens that collects visible light, an image pickup element such as a CCD (Charge Coupled Device) or CMOS (Complementary Metal Oxide Semiconductor) arranged at the focus position of the lens, and an image obtained by the image pickup element. It is provided with an A / D (Analog / Digital) converter that converts an analog signal representing the above into digital data. The image pickup unit 16 takes an image of the inspection target under the control of the control unit 11 and acquires an image of the appearance of the inspection target.

FIG. 3 shows a functional configuration of the defect classification device 10. As shown in FIG. 3, the defect classification device 10 functionally includes a captured image acquisition unit 110, a defect detection unit 120, a feature amount calculation unit 130, a first classification unit 140, and a learning unit 150. A second classification unit 160, a classification condition setting unit 170, and a pass / fail determination unit 180 are provided. In the control unit 11, the CPU reads the program stored in the ROM into the RAM, executes the program, and controls the program, thereby functioning as each of these units.

The captured image acquisition unit 110 acquires an captured image in which the inspection target is captured. Specifically, the captured image acquisition unit 110 causes the imaging unit 16 to image the inspection target in a state where the inspection target is within the field of view of the imaging unit 16. As a result, the captured image acquisition unit 110 acquires an captured image in which the appearance of the inspection target is captured. The captured image acquisition unit 110 is realized by the control unit 11 cooperating with the image pickup unit 16 and the like.

The defect detection unit 120 detects defects existing in the inspection target from the captured image of the inspection target. Specifically, the defect detection unit 120 analyzes the captured image acquired by the captured image acquisition unit 110 by using a well-known image processing technique. Then, the defect detection unit 120 detects a region in the captured image that satisfies a specific condition as a defect region in which the defect may be captured.

Specifically, FIG. 4 shows an example of the captured image 30 in which the inspection target is captured by the imaging unit 16. The defect detection unit 120 analyzes the pixel value (luminance value) of each pixel included in the captured image 30, and determines whether or not there is a region in which the difference in pixel value from the surrounding pixels is larger than the reference value. Then, when the defect detection unit 120 has a region in the captured image 30 in which the difference in pixel value from the surrounding pixels is larger than the reference value, the defect detection unit 120 detects the region as the defect region.

For example, when three defects X1, X2, and X3 are shown in the captured image 30, the defect detection unit 120 detects each region of the defects X1, X2, and X3 in the captured image 30 as a defect region. The defect detection unit 120 is realized by the control unit 11 cooperating with the storage unit 12 and the like.

Returning to FIG. 3, the feature amount calculation unit 130 calculates the feature amount of the defect detected by the defect detection unit 120. Here, the feature amount of the defect is a value indicating the feature of the defect. Specifically, the feature amount calculation unit 130 calculates the size, area, roundness, and brightness of the defect as the feature amount of the defect.

The size and area of the defect are indicators of the size of the defect. Specifically, the size of the defect corresponds to the length of the defect region detected by the defect detection unit 120 in any one or more directions (for example, the vertical direction, the horizontal direction, etc.). Further, the area of the defect corresponds to the number of pixels included in the defect region detected by the defect detection unit 120.

The roundness of a defect is a value indicating how much the shape of the defect deviates from the circle. For example, since the shapes of the defects X1 and X2 shown in FIG. 4 are close to a circle, their roundness is close to 0. On the other hand, since the shape of the defect X3 is significantly different from that of the circle, the roundness deviates from 0.

The brightness of a defect is an index represented by the color information of the defect, and indicates whether the defect is close to white or black. The brightness of the defect is represented by, for example, a representative value such as an average value or a mode value of the pixel values of a plurality of pixels included in the defect region detected by the defect detection unit 120.

The feature amount calculation unit 130 analyzes the defect region detected from the captured image 30, and calculates the size, area, roundness, and brightness of the defect region as the feature amount of the defect existing in the inspection target. The feature amount calculation unit 130 is realized by cooperating with the control unit 11, the storage unit 12, and the like.

The first classification unit 140 classifies the defects detected by the defect detection unit 120 by using the trained model 40 learned by deep learning. Specifically, the first classification unit 140 classifies the defects detected by the defect detection unit 120 into one of a plurality of predetermined types. Here, the type of defect is to distinguish by the type of defect, and is information indicating which of the defects existing in the inspection target corresponds to, for example, scratches, wrinkles, pinholes, foreign substances, etc. be.

The first classification unit 140 refers to the trained model 40 stored in the storage unit 12 in order to classify the defects detected by the defect detection unit 120. The trained model 40 is a model that outputs a classification result of classifying the defect into one of a plurality of types in response to the input of the image in which the defect existing in the inspection target is captured.

The learning unit 150 generates a trained model 40 by learning the relationship between a defect image for learning and the type of defect shown in the defect image by deep learning. The learning unit 150 refers to the learning data 50 stored in the storage unit 12 in order to execute learning by deep learning.

The learning data 50 is data (training data) used for learning by the learning unit 150, and includes a plurality of data sets used as teacher data. Each of the plurality of data sets defines the correspondence between the defect image for learning and the type of defect shown in the defect image. The defect image for learning is an image showing a defect as a sample whose type is known in advance.

The learning unit 150 uses a plurality of data sets included in the learning data 50 as supervised data, and performs supervised learning by a deep learning method. In order to carry out deep learning, the learning unit 150 generates a neural network, more specifically, a convolutional neural network (CNN). Here, CNN may be various methods improved from CNN, such as R-CNN (Regional-CNN) and the like.

As shown in FIG. 5, the CNN outputs an input layer into which input data is input, a convolution layer for convolving the image input to the input layer, a pooling layer for pooling, and an output from the pooling layer. It is composed of a fully connected layer that binds and an output layer that obtains an output value using an activation function. The fully connected layer contains a plurality of intermediate layers (hidden layers). In the CNN, a plurality of sets of the convolution layer and the pooling layer are provided, and the convolution process and the pooling process are repeated a plurality of times for the image input to the input layer. The fully connected layer combines the data after being processed by the convolutional layer and the pooling layer in the intermediate layer. The output layer outputs the values combined in the fully connected layer as output data using an activation function such as softmax. The configuration of the CNN shown in FIG. 5 is an example, and the configuration of the neural network generated by the learning unit 150 is not limited to this.

The CNN accepts the data of the image in which the defect is captured as the input data, and outputs the information indicating the type of the defect as the output data. The number of nodes in the output layer is predetermined by the number of types of defects that may exist in the inspection target. In the example of FIG. 5, each node of the output layer is associated with a type A, a type B, ..., A type N (for example, scratches, wrinkles, pinholes, foreign matter, etc.).

Each node of the output layer outputs a score which is a value of 0 or more and 1 or less. The score is a value indicating the certainty that the defect in the image input to the input layer corresponds to the type associated with each node. The closer the score output from a node is to 1, the higher the probability that the defect in the image input to the input layer corresponds to the type associated with that node. In the example of FIG. 5, the score output from the node of type B in the output layer is 0.78, which is higher than the score output from the node of other types. Therefore, the probability that the defect in the image input to the input layer corresponds to the type B is relatively high.

The learning unit 150 uses a plurality of data sets included in the training data 50 as teacher data to adjust parameters in the neural network, specifically, filter values in the convolution layer and coupling weights in the intermediate layer. Specifically, the learning unit 150 inputs the defect image for learning included in each data set as input data. Then, in the learning unit 150, among the scores output from each node of the output layer, the score output from the node corresponding to the correct answer type approaches 1, and the score output from the other nodes becomes 0. Adjust the parameters using the backpropagation method or the like so that they are closer to each other. Here, the correct answer type is the type of defect associated with the defect image for learning input to the input layer.

The learning unit 150 optimizes the weight of the connection of each layer in the neural network by executing such a process for each of the plurality of data sets included in the training data 50. As a result, the learning unit 150 constructs a neural network that receives the input of the image in which the defect is captured and outputs a score indicating the certainty that the defect corresponds to each of a plurality of predetermined types. When the learning unit 150 constructs the neural network, the learned neural network is stored in the storage unit 12 as a trained model 40. The learning unit 150 is realized by the control unit 11 cooperating with the storage unit 12 and the like.

The first classification unit 140 estimates the type of the defect detected by the defect detection unit 120 using the trained model 40 generated by the learning unit 150 in this way, and is detected by the defect detection unit 120. Classify the defects into the estimated types. Specifically, the first classification unit 140 inputs the image of the defect detected from the captured image 30 by the defect detection unit 120 into the trained model 40 as input data. Then, in the first classification unit 140, it is certain that the defects detected by the defect detection unit 120 output from the trained model 40 with respect to such an input correspond to each of the plurality of predetermined types. Refer to the score that shows the uniqueness. Then, the first classification unit 140 classifies the defects detected by the defect detection unit 120 into the type having the highest score output from the trained model 40 among the plurality of types.

When, for example, as shown in FIG. 4, when a plurality of defects X1, X2, and X3 are detected in the captured image 30, the first classification unit 140 performs such a classification process on the detected defects X1, 1. Execute for each of X2 and X3. As a result, the first classification unit 140 classifies each of the plurality of defects X1, X2, and X3 into one of a plurality of predetermined types. The first classification unit 140 is realized by cooperating with the control unit 11, the storage unit 12, and the like.

Returning to FIG. 3, the second classification unit 160 detects the defect detected by the defect detection unit 120 based on the feature amount calculated by the feature amount calculation unit 130 and the classification result by the first classification unit 140. Classify. In other words, the second classification unit 160 classifies defects by combining both the feature amount calculated by the feature amount calculation unit 130 and the classification result by the first classification unit 140.

Specifically, classification based only on features such as defect size, area, roundness, brightness, etc. is a classification performance for complex defects that humans intuitively judge. There is a problem that it does not rise. For example, as shown in FIGS. 6 (A) to 6 (C), defects having an irregular shape (FIG. 6 (A)) and defects in which wrinkles, scratches, etc. are widely distributed (FIG. 6 (B)). )) And defects in which they are combined (FIG. 6 (C)) tend to be difficult to classify based on the feature amount alone. This main factor is that it is difficult to select the optimum feature amount for classification and to set the classification condition using the optimum feature amount for classification.

On the other hand, by using deep learning, it is possible to improve the classification accuracy for complicated defects that were difficult to classify only by the feature amount. However, in the classification using deep learning, the classification based on the feature amount may be erroneously determined by a relatively simple classification. For example, as shown in FIGS. 7A to 7C, there is a tendency that deep learning is not good at classifying a plurality of defects having similar shapes but different sizes. The main reason for this is that deep learning needs to learn the defects sorted by classification type in advance, but if the classification type is set for each defect with a slightly different size, it is necessary to set a huge number of classification types. Because it becomes. In particular, when such classification type setting is performed manually, the work time related to sorting becomes enormous, which hinders improvement of classification accuracy by deep learning.

As described above, simply replacing the classification based only on the feature amount with the classification based only on deep learning is not suitable for improving the classification accuracy of defects. In consideration of such a situation, the second classification unit 160 is based on both the feature amount calculated by the feature amount calculation unit 130 and the classification result using deep learning by the first classification unit 140. And classify the defects.

Specifically, the second classification unit 160 refers to the classification table 60 stored in the storage unit 12. The classification table 60 is a table in which the defect classification conditions set by the second classification unit 160 are set. The second classification unit 160 classifies the defects detected by the defect detection unit 120 according to the classification conditions set in the classification table 60.

FIG. 8 shows an example of the classification conditions defined in the classification table 60. The classification table 60 includes a plurality of types of features calculated by the feature calculation unit 130 and the types and scores of defects obtained by classification using the trained model 40 by the first classification unit 140 as classification conditions. Correspondingly, 10 groups G1 to G10 are defined.

Specifically, the classification table 60 is classified differently with respect to the feature amount depending on whether the type of the defect classified by the first classification unit 140 is "scratch", "foreign matter", or "wrinkle". The conditions are set. For example, when the type of defect is "scratch", the second classification unit 160 classifies the defect into one of the groups G1 and G2 depending on whether the size of the defect is larger than L1. .. Further, when the type of the defect is "foreign matter", depending on whether the brightness of the defect is brighter than B1 and whether the area of the defect is larger than A1 or A2. The second classification unit 160 classifies the defect into one of the groups G3 to G6. Furthermore, when the type of defect is "wrinkle", the second classification depends on whether the score is larger than S1 and whether the size of the defect is larger than L2 or L3. Part 160 classifies the defect into any of the groups G7 to G10.

In this way, the second classification unit 160 is a combination of the feature amount calculated by the feature amount calculation unit 130 and the type and score obtained as the classification result by the first classification unit 140 added as a new feature amount. Determines which of the plurality of groups set in the classification table 60 corresponds to. Then, the second classification unit 160 classifies the defects detected by the defect detection unit 120 into the corresponding group. The second classification unit 160 is realized by cooperating with the control unit 11, the storage unit 12, and the like.

The classification table 60 shown in FIG. 8 is an example, and the second classification unit 160 can classify defects according to various classification conditions, not limited to the classification conditions set in the classification table 60.

Returning to FIG. 3, the classification condition setting unit 170 sets the defect classification condition by the second classification unit 160 according to the operation received from the user. Specifically, the classification condition setting unit 170 classifies from among a plurality of types of feature quantities calculated by the feature quantity calculation unit 130 and the types and scores obtained as the classification result by the first classification unit 140. Set the combination to be used. Further, the classification condition setting unit 170 sets the boundary value of the group in each of the feature amount, the type, and the score, that is, the reference value for the pass / fail determination unit 180, which will be described later, to determine the pass / fail of the defect.

By operating the input receiving unit 13 and inputting an instruction, the user can set an arbitrary classification condition including the classification condition set in the classification table 60 shown in FIG. 8, for example. For example, when the inspection target is a product manufactured on a production line, different classification conditions can be set depending on the product. Specifically, one product allows wrinkles regardless of size but cannot tolerate pinholes larger than the standard value, and another product does not allow wrinkles regardless of size but is brighter than the standard value. It is acceptable, etc.

The classification condition setting unit 170 sets the classification condition according to the input received from the user via the input reception unit 13 in this way. The second classification unit 160 classifies the defects detected by the defect detection unit 120 according to the classification conditions set by the classification condition setting unit 170. The classification condition setting unit 170 is realized by the control unit 11 cooperating with the input reception unit 13.

Further, whether the classification condition setting unit 170 classifies the defect based on both the feature amount calculated by the feature amount calculation unit 130 and the classification result using deep learning by the first classification unit 140 as the classification condition. , Or classify defects based on only one of them.

For example, when the classification condition setting unit 170 is set to classify defects based on both the feature amount and the classification result using deep learning, the second classification unit 160 is set to the feature amount such as size and deep. Defects are classified based on the type and the score, which are the classification results using learning. On the other hand, when the classification condition setting unit 170 is set to classify defects based on only one of the feature amount and the classification result using deep learning, the second classification unit 160 is set. Defects are classified based on features only or deep learning only.

In this way, the user can freely combine the classification using the feature amount and the classification using the deep learning. As a result, even after the introduction of classification using deep learning, it is possible to perform classification using only conventional features. In other words, high compatibility can be ensured between the classification using deep learning and the classification using only the conventional features. Therefore, for example, in a production line, it is possible to gradually introduce classification using deep learning while assessing its performance.

Returning to FIG. 3, the pass / fail determination unit 180 determines the pass / fail of the defect detected by the defect detection unit 120 based on the classification result by the second classification unit 160. Here, the pass / fail of a defect means whether or not the defect is allowed to be included in the inspection target.

As shown in FIG. 8, the classification table 60 sets the pass / fail of the defect when the defect is classified into each group by the second classification unit 160. When the defect is classified into a group of "OK" (specifically, groups G2, G4, G6, G8, G10) by the second classification unit 160, the pass / fail determination unit 180 is a defect to which the defect is allowed. Judge that there is. On the other hand, when the defect is classified into the "NG" group (specifically, the groups G1, G3, G5, G7, G9) by the second classification unit 160, the pass / fail determination unit 180 does not allow the defect. Judged as a defect.

When the pass / fail determination is made, the pass / fail determination unit 180 outputs the determination result. Specifically, the pass / fail determination unit 180 displays an image showing the determination result by the pass / fail determination unit 180 on the display unit 14, and causes the user to recognize whether or not the inspection target contains a defect. The pass / fail determination unit 180 is realized by the control unit 11 cooperating with the display unit 14 and the like.

The pass / fail determination unit 180 may output the pass / fail determination result by voice instead of or in addition to displaying it on the display unit 14. Alternatively, the pass / fail determination unit 180 may output information indicating the determination result by the pass / fail determination unit 180 to an external device via the communication unit 15.

The flow of the defect classification process executed by the defect classification device 10 configured as described above will be described with reference to the flowchart shown in FIG.

Before the defect classification process shown in FIG. 9 is executed, the learning unit 150 generates the trained model 40 by performing deep learning using the learning data 50. Then, in a state where the generated learned model 40 is stored in the storage unit 12, when an instruction to start processing is received from the user via the input reception unit 13, the defect classification process shown in FIG. 9 starts.

When the defect classification process is started, in the defect classification device 10, the control unit 11 functions as the classification condition setting unit 170 and sets the classification conditions (step S1). Specifically, the control unit 11 sets the classification conditions defined in, for example, the classification table 60 shown in FIG. 8 according to the input received from the user via the input reception unit 13.

When the classification condition is set, the control unit 11 functions as the captured image acquisition unit 110 and acquires the captured image captured by the inspection target (step S2). Specifically, the control unit 11 acquires the captured image 30 in which the surface of the inspection target is captured, as shown in FIG. 4, for example, by capturing the appearance of the inspection target by the imaging unit 16.

When the captured image is acquired, the control unit 11 functions as the defect detection unit 120 and detects the defect from the captured image (step S3). Specifically, the control unit 11 analyzes the pixel value of each pixel in the captured image and determines the presence or absence of a region in which the pixel value is significantly different from the surroundings. Then, when there is a region whose pixel value is significantly different from that of the surrounding area, the control unit 11 detects the region as a defect region in which a defect may be reflected.

When the defect is detected, the control unit 11 functions as the feature amount calculation unit 130 and calculates the feature amount of the defect (step S4). Specifically, the control unit 11 calculates a plurality of types of feature quantities such as the size, area, roundness, and brightness of the defect detected in step S3.

When the feature amount of the defect is calculated, the control unit 11 functions as the first classification unit 140 and classifies the defect using the trained model 40 (step S5). Specifically, the control unit 11 inputs an image of the defect detected in step S3 into the trained model 40 stored in the storage unit 12. Then, the control unit 11 refers to the scores of the plurality of types output from the trained model 40, and classifies the detected defects into the type having the highest score among the plurality of types.

When the defect is classified using the trained model 40, the control unit 11 functions as the second classification unit 160, and further classifies the defect based on the feature amount and the classification result using the trained model 40. (Step S6). Specifically, the control unit 11 determines the feature amount of the defect calculated in step S4 and the type and score of the defect classified in step S5 from the plurality of groups according to the classification conditions set in step S1. Identify the group to which the combination applies. Then, the control unit 11 classifies the defects detected in step S3 into the specified group.

When the defects are classified, the control unit 11 functions as a pass / fail determination unit 180, determines and outputs the pass / fail of the defect (step S7). Specifically, the control unit 11 determines in step S6 whether the defect is classified into a group in which the pass / fail is set to "OK" or a group in which the pass / fail is set to "NG". .. As a result of the determination, when the defect is classified into the group whose pass / fail is set to "NG", the control unit 11 determines that the defect is unacceptable (defective) and displays that fact on the display unit 14. do. On the other hand, when the defect is classified into the group whose pass / fail is set to "OK", the control unit 11 determines that the defect is acceptable (not defective) and displays that fact on the display unit 14. ..

When a plurality of defects are detected in the captured image 30 in step S3, the control unit 11 executes the processes of steps S4 to S7 for each of the detected defects. As a result, the control unit 11 inspects whether or not the inspection target contains an unacceptable (corresponding to a defect) defect. As a result, the defect classification process shown in FIG. 9 is completed.

As described above, the defect classification device 10 according to the first embodiment calculates the feature amount of the defect detected from the captured image captured by the inspection target, and uses the trained model 40 learned by deep learning. And classify the detected defects. Then, the defect classification device 10 according to the first embodiment further classifies the detected defects based on the feature amount and the classification result by the trained model 40. As described above, the defect classification device 10 according to the first embodiment classifies the defects existing in the inspection target by combining the classification using the feature amount and the classification using the deep learning. Therefore, defects that are difficult to classify using only one of the case where only the feature amount is used and the case where only the deep learning is used can be classified with high accuracy. As a result, the accuracy of classifying defects existing in the inspection target can be improved.

Further, the defect classification device 10 according to the first embodiment handles the classification result by deep learning as a new feature amount in combination with the feature amount such as the conventional size. Therefore, without changing the method of setting the classification condition using the feature amount such as the conventional size, either the classification using the feature amount or the classification by deep learning should be selected and used, or both. It can be easily used in combination. Thereby, the optimum classification conditions can be easily set according to the defect to be inspected and the purpose of classification.

In particular, for example, when a product manufactured on a manufacturing line is an inspection target, defects generated in the manufacturing process of the product are high for various defects from simple to complex depending on the manufacturing method and manufacturing process. Classification performance is required. On top of that, it is necessary to facilitate the setting of classification conditions and reduce the time required to set the classification conditions. The defect classification device 10 according to the first embodiment improves the classification accuracy of various defects existing in the inspection target by handling the classification result by deep learning as a new feature amount in combination with the feature amount such as the conventional size. At the same time, the classification conditions can be easily set.

Further, since the defect classification device 10 according to the first embodiment handles the classification result by deep learning as a new feature amount in combination with the feature amount such as the conventional size, a classification sequence using the feature amount such as the conventional size is used. It can be introduced without change. As a result, the defect classification device 10 according to the first embodiment can be introduced without significantly changing the defect determination criteria, for example, when a product manufactured on a production line is an inspection target.

Furthermore, in the classification results by AI such as deep learning, since the classification process cannot be seen from the outside, it is often impossible for humans to explain the classification criteria. Therefore, for example, in controlling the quality of products on a production line, it is difficult to replace all defect classifications with AI. Since the defect classification device 10 according to the first embodiment can be introduced without changing the conventional classification sequence using features such as size, defects due to deep learning are maintained while maintaining classification criteria that are easy for humans to explain. Classification can be introduced.

(Embodiment 2)
Next, the second embodiment of the present invention will be described. The same configurations and functions as those of the first embodiment will be omitted as appropriate.

FIG. 10 shows a functional configuration of the defect classification device 20 according to the second embodiment. As shown in FIG. 10, the defect classification device 20 functionally includes a captured image acquisition unit 110, a defect detection unit 120, a feature amount calculation unit 130, a first classification unit 140, and a second classification unit. It includes 160, a classification condition setting unit 170, a pass / fail determination unit 180, and a model setting unit 210. The CPU reads the program stored in the ROM into the RAM, executes the program, and controls the program to function as each of these parts.

More specifically, the defect classification device 10 according to the first embodiment includes the learning unit 150, whereas the defect classification device 20 according to the second embodiment does not include the learning unit 150. Instead, the defect classification device 20 according to the second embodiment includes a model setting unit 210.

The model setting unit 210 sets the trained model 40 according to the operation received from the user. Specifically, the defect classification device 20 stores a plurality of trained models 40 generated by deep learning using different learning data 50 in the storage unit 12. The model setting unit 210 selects one of the plurality of learned models 40 stored in the storage unit 12 according to the input received from the user via the input reception unit 13. Then, the model setting unit 210 sets the selected trained model 40 as the trained model 40 used by the first classification unit 140 for classification. The model setting unit 210 is realized by the control unit 11 cooperating with the input reception unit 13.

Since the defect classification device 20 does not include the learning unit 150, each of the plurality of trained models 40 is not generated by the defect classification device 20 itself, but is generated by an external device different from the defect classification device 20. .. The defect classification device 10 acquires a plurality of learned models 40 generated by an external device and stores them in the storage unit 12, for example, by communicating with the device via the communication unit 15.

The first classification unit 140 classifies the defects detected by the defect detection unit 120 by using the trained model 40 set by the model setting unit 210. Since the functions of each unit other than the model setting unit 210, including the first classification unit 140, are the same as those in the first embodiment, the description thereof will be omitted.

As described above, the defect classification device 20 according to the second embodiment does not have the function of the learning unit 150, and sets the learned model 40 used for classifying the defects according to the operation received from the user. This allows the user to flexibly adjust the setting of defect classification using deep learning.

(Modification example)
Although the embodiments of the present invention have been described above, it is possible to combine the embodiments and appropriately modify or omit the embodiments.

For example, in the above embodiment, the feature amount calculation unit 130 calculates four values of the defect size, area, roundness, and brightness as the feature amount of the defect detected by the defect detection unit 120. However, the feature amount calculation unit 130 is not limited to calculating all of these four values as the feature amount of the defect, and may calculate only a part of these four values. Further, the feature amount calculation unit 130 may calculate a value other than these four values as the feature amount of the defect, and which of at least one of these four values and a value other than these four values is used. May also be calculated.

In the above embodiment, the second classification unit 160 classifies the defects detected by the defect detection unit 120 with reference to the classification table 60 stored in the storage unit 12. At this time, the second classification unit 160 may classify the defects by using the random forest method. The random forest method is a method of generating a large number of decision trees by learning based on randomly sampled training data (training data). Specifically, the second classification unit 160 uses the feature amount calculated by the feature amount calculation unit 130 and the type and score obtained as the classification result by the first classification unit 140 as explanatory variables. , Generates a classifier composed of a large number of decision trees. Then, the second classification unit 160 classifies the defects by using the classifier generated by the random forest method.

In the above embodiment, the second classification unit 160 includes a feature amount calculated by the feature amount calculation unit 130, a type classified by the first classification unit 140, and a score indicating the certainty corresponding to the type. The defects detected by the defect detection unit 120 were classified based on the above. However, the second classification unit 160 is detected by the defect detection unit 120 based on the feature amount calculated by the feature amount calculation unit 130 and the type classified by the first classification unit 140 without using the score. Defects may be classified.

Alternatively, the second classification unit 160 may use a plurality of types and a plurality of scores. For example, the second classification unit 160 is not limited to the type having the highest score and its score among the plurality of types output from the plurality of nodes in the output layer of the neural network, and has two or more in descending order of the score. The types and the scores of those types may be used.

In the above embodiment, the defect classification devices 10 and 20 include an imaging unit 16. However, the defect classification devices 10 and 20 themselves do not include the image pickup unit 16, and an image pickup device independent of the defect classification devices 10 and 20 may image the inspection target. When the image pickup device is provided separately from the defect classification devices 10 and 20, the image pickup image acquisition unit 110 communicates with the image pickup device via the communication unit 15, and the image pickup unit 16 captures the appearance of the inspection target. Acquire the captured image.

In the above embodiment, in the control unit 11 of the defect classification devices 10 and 20, the CPU functions as each unit shown in FIG. 3 or FIG. 10 by executing the program stored in the ROM or the storage unit 12. However, the control unit 11 is provided with dedicated hardware such as an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), and various control circuits instead of the CPU, and the dedicated hardware is shown in FIG. Alternatively, it may function as each part shown in FIG. In this case, the functions of each part may be realized by individual hardware, or the functions of each part may be collectively realized by a single hardware. Further, some of the functions of each part may be realized by dedicated hardware, and other parts may be realized by software or firmware.

By applying a program that defines the operation of the defect classification devices 10 and 20 to an existing computer such as a personal computer and an information terminal device, the computer can be made to function as the defect classification devices 10 and 20. Further, the distribution method of such a program is arbitrary, and for example, a computer-readable recording such as a CD-ROM (Compact Disk ROM), a DVD (Digital Versatile Disk), an MO (Magneto Optical Disk), or a memory card. It may be stored in a medium and distributed, or may be distributed via a communication network such as the Internet.

The present invention allows for various embodiments and modifications without departing from the broad spirit and scope of the invention. Further, the above-described embodiment is for explaining the present invention, and does not limit the scope of the present invention. That is, the scope of the present invention is shown not by the embodiment but by the claims. And various modifications made within the scope of the claims and within the scope of the equivalent invention are considered to be within the scope of the invention.

10, 20 Defect classification device, 11 Control unit, 12 Storage unit, 13 Input reception unit, 14 Display unit, 15 Communication unit, 16 Imaging unit, 30 Captured image, 40 Trained model, 50 Learning data, 60 Classification table, 110 Image acquisition unit, 120 Defect detection unit, 130 Feature amount calculation unit, 140 First classification unit, 150 Learning unit, 160 Second classification unit, 170 Classification condition setting unit, 180 Pass / fail judgment unit, 210 Model setting unit , X1, X2, X3 Defects

Claims (10)

A defect detection unit that detects defects from the captured image of the inspection target, and
A feature amount calculation unit that calculates the feature amount of the defect detected by the defect detection unit, and a feature amount calculation unit.
Using the trained model trained by deep learning, a first classification unit that classifies the defects detected by the defect detection unit, and a first classification unit.
A second classification unit for classifying the defects detected by the defect detection unit based on the feature amount calculated by the feature amount calculation unit and the classification result by the first classification unit is provided.
Defect classifier. Further, a classification condition setting unit for setting the classification condition of the defect by the second classification unit according to the operation received from the user is provided.
The second classification unit classifies the defects detected by the defect detection unit according to the classification conditions set by the classification condition setting unit.
The defect classification device according to claim 1. As the classification condition, the classification condition setting unit classifies the defect based on both the feature amount calculated by the feature amount calculation unit and the classification result by the first classification unit, or the feature. It is set whether to classify the defect based on only one of the feature amount calculated by the quantity calculation unit and the classification result by the first classification unit.
The defect classification device according to claim 2. Further provided with a model setting unit for setting the trained model according to the operation received from the user.
The first classification unit classifies the defects detected by the defect detection unit using the trained model set by the model setting unit.
The defect classification device according to any one of claims 1 to 3. The trained model outputs a score indicating the certainty that the defect detected by the defect detection unit corresponds to each of a plurality of predetermined types.
The first classification unit classifies the defect detected by the defect detection unit into the type having the highest score output from the trained model among the plurality of types.
The second classification unit is based on the feature amount calculated by the feature amount calculation unit, the type classified by the first classification unit, and the score indicating the certainty corresponding to the type. Then, the defect detected by the defect detection unit is classified.
The defect classification device according to any one of claims 1 to 4. The feature amount calculation unit calculates at least one of the size, area, roundness, and brightness of the defect detected by the defect detection unit as the feature amount.
The defect classification device according to any one of claims 1 to 5. A learning unit that generates the trained model by learning the relationship between the defect image for learning and the type of defect shown in the defect image by the deep learning is further provided.
The first classification unit classifies the defects detected by the defect detection unit using the trained model generated by the learning unit.
The defect classification device according to any one of claims 1 to 6. Further, a pass / fail determination unit for determining the pass / fail of the defect detected by the defect detection unit is further provided based on the classification result by the second classification unit.
The defect classification device according to any one of claims 1 to 7. Defect detection step to detect defects from the captured image of the inspection target,
A feature amount calculation step for calculating the feature amount of the defect detected in the defect detection step, and a feature amount calculation step.
Using the trained model trained by deep learning, a first classification step for classifying the defects detected in the defect detection step, and a first classification step.
Includes a second classification step for classifying the defects detected in the defect detection step based on the feature amount calculated in the feature amount calculation step and the classification result in the first classification step. ,
Defect classification method. Computer,
Defect detection unit that detects defects from the captured image of the inspection target,
A feature amount calculation unit that calculates the feature amount of the defect detected by the defect detection unit,
A first classification unit that classifies the defects detected by the defect detection unit using a trained model trained by deep learning.
It functions as a second classification unit for classifying the defects detected by the defect detection unit based on the feature amount calculated by the feature amount calculation unit and the classification result by the first classification unit.
program.

Images