JP2020085583A - Inspection device and inspection method - Google Patents

Abstract

To provide technology that reduces the effects of learning data having an inaccurate label.SOLUTION: An inspection device of the present invention comprises: a first learning execution unit for separating learning data that is a set of labeled data into training data and validation data and executing the first learning of a neural network using the training data, as well as executing the verification of the validation data using the neural network after the first learning, thereby extracting label-inaccurate data, out of the validation data, the label of which is inaccurate; a second learning execution unit for executing the second learning of the neural network using learning data; and an inspection execution unit for inspecting an inspection object using the neural network after the second learning. The second learning execution unit randomly selects one label from among a plurality of predetermined labels as to the label-inaccurate data out of the learning data and uses the selected label in the second learning in the state of it being added.SELECTED DRAWING: Figure 1

Description

The present disclosure relates to a technique for inspecting an inspection target using an image.

Patent Document 1 discloses a technique for executing an inspection of an inspection target using an image using a neural network. Before performing such an inspection, the learning of the neural network is executed in advance using the learning data to which the inspection result visually inspected by the inspector is given as the correct label. The correct answer label is, for example, a label indicating whether the inspection object is a good product or a defective product. In order to perform the inspection by the image with sufficient accuracy, it is important to give a large amount of training data with correct labeling to the neural network.

JP, 2004-191112, A

However, the training data labeled by the inspector often includes data with incorrect labels. The cause of the inaccurate label may be simply due to a human error, or may be due to the variation of the quality criterion by the inspector. Therefore, there has conventionally been a demand for reducing the influence of learning data having an incorrect label when learning a neural network.

According to an aspect of the present disclosure, there is provided an inspection device that inspects an inspection target using a neural network. This inspection apparatus divides learning data, which is a set of labeled data, into training data and validation data, executes the first learning of the neural network using the training data, and performs the first learning after the first learning. A first learning execution unit for extracting label inaccurate data in which the label is inaccurate from the validation data by executing verification of the validation data using the neural network; A second learning execution unit that executes a second learning of the neural network; and an inspection execution unit that inspects the inspection target using the neural network after the second learning, the second learning Of the learning data, the execution unit uses the label inaccurate data for the second learning in a state where one label is randomly selected from a plurality of predetermined labels and given.

The block diagram of an inspection system. Explanatory drawing which shows the structural example of a neural network. Explanatory drawing which shows the example of a determination score. The flowchart which shows the whole process sequence of embodiment. Explanatory drawing which shows the structural example of learning data. The flowchart which shows the detailed procedure of step S100 of FIG. Explanatory drawing which shows the content of cross validation processing. Explanatory drawing which shows the example of the verification result and label correction by cross validation processing. The flowchart which shows the detailed procedure of step S200 of FIG. The graph which shows the ROC curve of an Example and a comparative example.

A. First Embodiment FIG. 1 is a block diagram of an inspection system in the embodiment. This inspection system includes an imaging device 100, a visual inspection device 200, a database 300, and an inspection device 400.

The imaging device 100 captures an image MD of the inspection object. The visual inspection apparatus 200 is used by an inspector to visually observe the image MD to determine the quality of the inspection object, and to attach a quality label to the image MD. The image with the quality label is registered in the database 300 as learning data LD.

The inspection device 400 includes an information acquisition unit 410, a learning execution unit 420, a neural network 430, an inspection execution unit 440, and an inspection result output unit 450. Each of these units may be realized by a processor executing a computer program, or may be realized by a hardware circuit.

The information acquisition unit 410 acquires the image MD captured by the imaging device 100 or the learning data LD stored in the database 300.

The learning execution unit 420 executes learning of the neural network 430. At this time, the learning execution unit 420 acquires the learning data LD from the database 300 via the information acquisition unit 410. The learning execution unit 420 includes a first learning execution unit 422 and a second learning execution unit 424. The processing contents of the first learning execution unit 422 and the second learning execution unit 424 will be described later.

The inspection execution unit 440 uses the learned neural network 430 to execute the determination on the image MD. At this time, the inspection execution unit 440 acquires the image MD captured by the imaging device 100 via the information acquisition unit 410.

The inspection result output unit 450 has a function of outputting the inspection result obtained by the inspection execution unit 440. The inspection result output unit 450 can be realized by, for example, a display device.

FIG. 2 is an explanatory diagram showing a configuration example of the neural network 430. The neural network 430 has an input layer 432, an intermediate layer 434, a fully connected layer 436, and an output layer 438. The neural network 430 is a convolutional neural network in which the intermediate layer 434 includes a convolution filter and a pooling layer. However, a neural network other than the convolutional neural network may be used.

The image MD captured by the image capturing apparatus 100 is input to the input layer 432. The middle layer 434 includes a convolutional filter layer and a pooling layer. The intermediate layer 434 may include a plurality of these filter layers and pooling layers. In the intermediate layer 434, a plurality of feature quantities corresponding to the image MD are output and input to the fully-combined layer 436. Full bond layer 436 may include a plurality of full bond layers. The output layer 438 includes three output nodes, and the three determination scores S1, S2, and S3 are output from these output nodes. The three determination scores S1 to S3 are normalized so that the sum of them is 1.0.

FIG. 3 is an explanatory diagram showing an example of the determination scores S1 to S3. The determination scores S1, S2, S3 are, for example, the following values.
(1) Judgment score S1: Probability that the inspection object is non-defective.
(2) Judgment score S2: Probability that the inspection object is the first-type defect A.
(3) Judgment score S3: Probability that the inspection object is the second type defect B.

Based on the determination score S1 of non-defective product, the quality is determined according to the determination threshold Th. Specifically, when the determination score S1 of the non-defective product exceeds the determination threshold Th, the non-defective product label is adopted as the determination result. On the other hand, if the non-defective judgment score S1 is equal to or lower than the judgment threshold Th, the defective label having the highest score among the defective judgment scores S2 and S3 is adopted. For example, in the example of FIG. 3, the value of the determination score S1 for the non-defective product is 0.8, which exceeds the determination threshold Th. Therefore, the result of “non-defective product” is used as the determination label. These determination scores S1 to S3 are examples, and the neural network 430 may be configured to output other determination scores. However, the neural network 430 is configured to output one determination score indicating the probability of being “non-defective” and one or more determination scores each indicating the probability of being one or more “defective”. Preferably.

FIG. 4 is a flowchart showing the overall processing procedure of the embodiment. In step S100, a first learning is performed on the neural network 430 to extract label inaccurate data whose label is inaccurate. In step S200, the second learning is executed on the neural network 430 in a state where labels are randomly given to the label inaccurate data. In step S300, the inspection of the inspection object is executed using the neural network 430 after the second learning is executed. Detailed procedures of steps S100 and S200 will be described later.

FIG. 5 is an explanatory diagram showing a configuration example of the learning data LD. Each data of the learning data LD is the image MD with the quality label LB. Since the learning data LD is a set of a plurality of data, it is also called “learning data set”. In ordinary machine learning of neural networks, learning by iteration and epoch is often executed. The iteration means that a plurality of data for one batch is extracted from the learning data LD and the learning is executed by the data for one batch. The epoch means learning using the entire learning data LD by repeating iterations. In the present embodiment, such learning is executed in the second learning in step S200.

FIG. 6 is a flowchart showing the detailed procedure of step S100 of FIG. The process of step S100 is executed by the first learning execution unit 422. In step S110, the learning data LD is acquired. In steps S120 to S170, the cross validation processing described below is executed. In the cross validation processing, the learning data LD is divided into training data and validation data, and the division is changed to perform learning and verification multiple times.

FIG. 7 is an explanatory diagram showing the contents of the cross validation process. In this example, the entire learning data LD is divided into three sub data sets DS1 to DS3. In the first processing, learning of the neural network 430 is executed using the first and second sub data sets DS1 and DS2 as training data. Learning using the training data is called "first learning". When the first learning is completed, the learned neural network 430 is verified using the third sub data set DS3 as validation data. By this verification, the validation data DS3 is classified into data DS3a in which the determination result using the neural network 430 and the original determination label match, and data DS3b in which they do not match. The data DS3b whose determination results do not match is extracted as label inaccurate data whose label is inaccurate.

In the second cross-validation processing, the first learning of the neural network 430 is executed using the first and third sub-data sets DS1 and DS3 as training data. This first learning is executed for the initialized neural network 430, not for the one learned in the first processing. Then, the learned neural network 430 is verified using the second sub-data set DS2 as validation data. As a result of this verification, the validation data DS2 is classified into data DS2a in which the determination result using the neural network 430 and the original determination label match, and data DS2b in which they do not match. The data DS2b whose determination results do not match is extracted as label inaccurate data whose label is inaccurate.

In the third processing, the neural network 430 is trained using the second and third sub-data sets DS2 and DS3 as training data, and the first sub-data set DS1 is used as validation data. The verification of the neural network 430 is executed. As a result of this verification, the validation data DS1 is classified into data DS1a in which the determination result using the neural network 430 and the original determination label match, and data DS1b in which they do not match. The data DS1b whose determination results do not match is extracted as label inaccurate data whose label is inaccurate.

When this cross-validation process is completed, as shown in the lower part of FIG. 7, the original learning data LD is classified into the label correct data LDa with the correct label and the label inaccurate data LDb with the incorrect label. .. The number of classifications of the learning data LD in the cross validation process is not limited to 3, and can be set to any integer of 2 or more.

Instead of performing the above-described cross-validation processing, a part of the learning data LD may be used as training data and the other part may be used as validation data to extract the label inaccurate data. However, if the cross-validation processing is performed, it is possible to determine whether or not each data forming the learning data LD corresponds to the label inaccurate data.

Steps S120 to S170 of FIG. 6 are detailed procedures of the cross validation process. First, in step S120, the learning data LD is divided into training data and validation data. In step S130, the neural network 430 is initialized. In step S140, the first learning is performed using the training data. In step S150, the label inaccurate data is extracted by performing verification using the validation data. In step S160, it is determined whether or not the cross-validation process is completed. If not completed, the process proceeds to step S170, the classification of the training data and the validation data is changed, and steps S130 to S150 are executed again. When the cross validation process is completed, the process proceeds from step S160 to step S180.

In step S180, a part of the label inaccurate data LDb extracted by the cross validation process is selected to narrow down the label inaccurate data, and in step S190, the label of the narrowed down label inaccurate data is corrected.

FIG. 8 is an explanatory diagram showing an example of the verification result and the label correction by the cross validation process. In the upper part of FIG. 8, four images D1 to D4 and their pass/fail labels are drawn for the original learning data LD. The quality label of the first image D1 is "good", the quality label of the second image D2 is "defective A", the quality label of the third image D3 is "bad B", and the quality of the fourth image D4 is good. The label is "good product". In FIG. 8, a pass/fail label score value Sref corresponding to the pass/fail label is also drawn as a part of the original learning data. The quality label score value Sref is a value that is set to 1.0 when the quality label is “good” and is set to 0 when the quality label is not “good”. As will be described later, the quality label score value Sref is set by the first learning execution unit 422 and is used in the process of narrowing down the label inaccurate data in step S180.

In the middle part of FIG. 8, the determination labels and determination scores S1 to S3 obtained by the verification using the four images D1 to D4 as validation data, the label inaccuracy F described later, and the determination result of match/mismatch are shown. Is drawn. In the first image D1, the first determination score S1 indicating the probability that the inspection object is a non-defective item is 0.9, which is equal to or greater than the determination threshold Th described in FIG. 3, and thus the determination label is “non-defective item”. The determination label of the second image D2 is also “non-defective”, the determination label of the third image D3 is “defective B”, and the determination label of the fourth image D4 is “defective A”. At the bottom of the verification result, the match/mismatch between the determination label obtained by the verification and the original pass/fail label is further shown. The data of the images D2 and D4 in which the determination label obtained by the verification and the original label do not match are extracted as label inaccurate data in which the label is inaccurate in step S150 of FIG.

The label inaccuracy F shown in FIG. 8 can be used in the narrowing-down process in step S180 of FIG. As the narrowing down process, for example, one of the narrowing down process using the label inaccuracy F and the narrowing down process by visual inspection can be used.

<Refining process using label inaccuracy F>
The label inaccuracy F is a value indicating the degree to which the original pass/fail label is inaccurate, and is calculated, for example, by the following equation.
F=|Sref-S1|... (1)
Here, Sref is a pass/fail label score indicating the pass/fail label of the original learning data, and S1 is a determination score indicating the probability that the inspection object is a pass item. As shown in FIG. 8, the label inaccuracy F of the image D2 is 0.8 and the label inaccuracy F of the image D4 is 0 out of the two images D2 and D4 whose coincidence determination is “mismatch”. .6. In this example, the inaccuracy of the original pass/fail label is higher in the image D2 than in the image D4.

In the process of narrowing down the label inaccurate data using the label inaccuracies F, the label inaccurate data is preferentially selected in the descending order of the label inaccuracies F and narrowed down. Specifically, for example, the label inaccuracy data can be narrowed down by selecting the label inaccurate data whose label inaccuracy F is higher than a predetermined threshold value. Alternatively, the label inaccurate data may be narrowed down by selecting a preset number of label inaccurate data in descending order of the label inaccuracies F. In this way, data with high label inaccuracy can be extracted as label inaccurate data, so that the influence of learning data having an inaccurate label can be reduced more efficiently.

In the label correction example shown at the bottom of FIG. 8, the pass/fail label of the image D2 having a higher label inaccuracy F is corrected from “bad A” to “inaccurate”. Further, the pass/fail label of the image D4 having a lower label inaccuracy F is corrected from “good item” to “defective A”. The latter corrected label “Bad A” is the determination label obtained by the second learning execution unit 424 in the verification. The data whose pass/fail label is “inaccurate” indicates that the data is selected as the label inaccurate data in step S180 of FIG. Instead of using the quality label “inaccurate”, other data such as a flag may be used to distinguish the label inaccurate data from the label accurate data.

<Visual narrowing down process>
In the visual narrowing-down process, the label inaccuracy data is narrowed down by visually reconfirming an image in which the determination label obtained by the verification and the original pass/fail label do not match. At this time, if the data that clearly has an error in the original pass/fail label is corrected to the determination label obtained from the verification result, and if it is difficult to determine whether the pass/fail label has an error, the label is “inaccurate”. It may be corrected to. In this process, data with high label inaccuracy can be extracted as label inaccurate data while visually confirming.

Note that step S180 in FIG. 6 may be omitted, and the label inaccurate data extracted in step S150 may be used as it is for the label correction in step S190. In this case, the original pass/fail labels are corrected to “inaccurate” for all the label inaccurate data extracted in step S150. When the process of step S100 is completed in this way, the process of step S200 of FIG. 4 is executed.

FIG. 9 is a flowchart showing the detailed procedure of step S200 of FIG. The process of step S200 is executed by the second learning execution unit 424. In step S210, the neural network 430 is initialized. However, this initialization may be omitted, and the second learning may be performed using the neural network 430 that has been learned in the first learning.

In step S220, a plurality of data corresponding to one batch is acquired from the learning data LD whose label has been corrected in the first learning. When acquiring the data for one batch, it is preferable to select the label inaccurate data whose quality label is “inaccurate” at a preset selection ratio. In step S230, one label is randomly selected from a plurality of predetermined labels and given to the label inaccurate data in one batch of data. In the present embodiment, one of a plurality of non-defective labels, “defective A”, “defective A”, and “defective B”, which should be originally used, is randomly selected as the label to be randomly given. In step S240, the second learning is executed using the learning data for one batch. In step S250, it is determined whether or not the learning is completed, and if it is not completed, the processes in step S210 and thereafter are repeated. For example, when the learning of one epoch using the entire learning data LD is executed by a predetermined number of epochs, it is determined that the learning is completed.

As described above, in the second learning, by randomly assigning labels to the inaccurate label data, noise can be intentionally added to the learning data at the time of learning, and the neural network 430 is regularized to make the determination. It is possible to improve accuracy. In the second learning, it is preferable to improve the determination accuracy of the neural network 430 by executing the learning of one epoch using the entire learning data LD for several tens to several hundreds of epochs or more.

When the processing of step S200 is completed in this way, in step S300 of FIG. 4, the inspection execution unit 440 executes the inspection of the inspection target using the learned neural network 430.

FIG. 10 is a graph showing ROC curves of the example and the comparative example. The horizontal axis of FIG. 10 is the missed rate, and the vertical axis is the false detection rate. The missed rate is a rate of erroneously determining that the data whose original quality label is not “good” is “good”. The erroneous detection rate is a rate at which the data whose original quality label is “non-defective” is erroneously determined as “defective A” or “defective B”. A plurality of points in FIG. 10 indicate values when the determination threshold Th in FIG. 3 is different. When the determination threshold Th becomes smaller, the rate of being determined to be non-defective increases and the rate of determining to be defective decreases. Therefore, the smaller the determination threshold Th is, the lower the false detection rate is, but the higher the oversight rate is.

The ROC curve of the example is the result of accuracy evaluation using the actual inspection data using the neural network 430 that has been trained according to the above-described embodiment. The ROC curve of the comparative example is the result when the accuracy evaluation is performed using the same inspection data as in the example using the neural network 430 that has been trained using the original learning data LD as it is. From this result, it was confirmed that the determination accuracy of the neural network 430 is improved by randomly applying the label to the label inaccurate data.

As described above, in the above-described embodiment, for the label inaccurate data of the learning data LD, the learning of the neural network 430 is executed in the state where the label is randomly assigned, and thus the inaccurate label is included in the learning data LD. Even if there is data that has, it is possible to reduce the effect and perform more correct inspection.

B. Other Embodiments The present disclosure is not limited to the above-described embodiments, and can be realized in various forms without departing from the spirit of the present disclosure. For example, the present disclosure can also be realized by the following aspects. The technical features in the above embodiments corresponding to the technical features in the embodiments described below are to solve some or all of the problems of the present disclosure, or some or all of the effects of the present disclosure. In order to achieve the above, it is possible to appropriately replace or combine. If the technical features are not described as essential in this specification, they can be deleted as appropriate.

(1) According to the first aspect of the present disclosure, there is provided an inspection device that inspects an inspection target using a neural network. This inspection apparatus divides learning data, which is a set of labeled data, into training data and validation data, executes the first learning of the neural network using the training data, and performs the first learning after the first learning. A first learning execution unit for extracting label inaccurate data in which the label is inaccurate from the validation data by executing verification of the validation data using the neural network; A second learning execution unit that executes a second learning of the neural network; and an inspection execution unit that inspects the inspection target using the neural network after the second learning, the second learning Of the learning data, the execution unit uses the label inaccurate data for the second learning in a state where one label is randomly selected from a plurality of predetermined labels and given.
According to this inspection device, for the label inaccurate data of the learning data, since the learning of the neural network is executed in the state where the label is randomly assigned, even if the learning data having the inaccurate label exists, It is possible to reduce the influence and perform a more correct inspection.

(2) In the inspection device, the first learning execution unit changes the classification of the training data and the validation data of the learning data, and performs cross validation processing for performing the first learning and the verification. It may be possible to determine whether or not each data forming the learning data corresponds to the label inaccurate data.
According to this inspection device, since it is determined whether or not each data forming the learning data is the label inaccurate data, it is possible to reduce the influence of the learning data having the inaccurate label and perform the correct inspection. Becomes

(3) In the inspection device, the first learning execution unit estimates the inaccuracy of the label of the validation data from the result of verification of the validation data using the neural network after the first learning, The validation data may be preferentially extracted in the descending order of the inaccuracy as the label inaccurate data.
According to this inspection device, since data with high label inaccuracy can be extracted as label inaccurate data, the influence of learning data having an inaccurate label can be reduced more efficiently.

(4) According to the second aspect of the present disclosure, there is provided an inspection method for inspecting an inspection object using a neural network. This inspection method extracts training data and validation data from training data that is a set of labeled data, executes the first learning of the neural network using the training data, and executes the first learning after the first learning. A first learning execution step of extracting label inaccurate data in which the label is inaccurate from the validation data by executing verification of the validation data using the neural network of A second learning execution step of executing a second learning of the neural network; and an inspection execution step of inspecting the inspection object by using the neural network after the second learning, the second learning In the execution step, of the learning data, the label inaccurate data is used for the second learning in a state where one label is randomly selected from a plurality of predetermined labels and given.
According to this inspection method, the learning of the neural network is performed in a state where labels are randomly assigned to the inaccurate label data of the learning data, so that there is learning data having an inaccurate label in the learning data. Even if it does, the influence can be reduced and a more correct inspection can be performed.

100... Imaging device, 200... Visual inspection device, 300... Database, 400... Inspection device, 410... Information acquisition part, 420... Learning execution part, 422... First learning execution part, 424... Second learning execution part, 430... Neural network, 432... Input layer, 434... Intermediate layer, 436... Full connection layer, 438... Output layer, 440... Inspection execution unit, 450... Inspection result output unit

Claims (4)

An inspection device for inspecting an inspection object using a neural network,
Learning data, which is a set of labeled data, is divided into training data and validation data, the first learning of the neural network is performed using the training data, and the neural network after the first learning is executed. A first learning execution unit for extracting label inaccurate data in which the label is inaccurate from the validation data by executing verification of the validation data using
A second learning execution unit that executes second learning of the neural network using the learning data;
An inspection execution unit that inspects the inspection object using the neural network after the second learning;
Equipped with
Of the learning data, the second learning execution unit randomly selects one label from a plurality of predetermined labels for the label inaccurate data, and applies the label to the second learning in a state where the label is randomly selected. Inspection device to be used. The inspection apparatus according to claim 1, wherein
The first learning execution unit changes the classification of the training data and the validation data of the learning data, and performs cross validation to perform the first learning and the verification. The inspection device for determining whether or not the label inaccurate data corresponds to. The inspection apparatus according to claim 1 or 2, wherein
The first learning execution unit estimates the inaccuracy of the label of the validation data from the result of the verification of the validation data using the neural network after the first learning, and the inaccuracy in descending order of the inaccuracy. An inspection device that preferentially extracts validation data as the label inaccurate data. An inspection method for inspecting an inspection object using a neural network,
Training data and validation data are extracted from the learning data that is a set of labeled data, the first learning of the neural network is performed using the training data, and the neural network after the first learning is executed. A first learning execution step of extracting label inaccurate data in which the label is inaccurate among the validation data by executing verification of the validation data using
A second learning execution step of performing a second learning of the neural network using the learning data;
An inspection execution step of inspecting the inspection object using the neural network after the second learning;
Equipped with
In the second learning execution step, for the label inaccurate data among the learning data, one label is randomly selected from a plurality of predetermined labels and is given to the second learning. The inspection method to use.

Images