WO2022202456A1 - Appearance inspection method and appearance inspection system - Google Patents

Abstract

The following occur during an appearance inspection according to the present invention: training data, which is a set of training samples in which training images capturing an item to be inspected for training purposes are paired with correct evaluation values for the training images is stored in a storage resource; correct evaluation values for training samples included in the training data are altered in accordance with a modifiable prescribed variation distribution; expanded training samples, which are training samples in which the altered values are set as correct evaluation values, are generated; expanded training data, which is a set of expanded training samples, is generated; the relationship between the training images and the evaluation values is learned, on the basis of the expanded training data, to determine internal parameters of an evaluation engine; an inspection image capturing the item to be inspected is acquired; the inspection image is inputted into the evaluation engine; and an estimated evaluation value, which is an estimate of the inspection image evaluation value, is acquired from output from the evaluation engine. (Selected Drawing: FIG. 2)

Description

APPEARANCE INSPECTION METHOD AND APPEARANCE INSPECTION SYSTEM

The present disclosure relates to an appearance inspection device and an appearance inspection method based on machine learning. More specifically, an evaluation value is obtained from an inspection image of an actual inspection object using an evaluation engine that has learned the relationship between a learning image obtained by imaging an inspection object for learning in advance and its correct evaluation value. An apparatus and method for highly accurate estimation and automatic evaluation of the performance of an inspected object are disclosed.

In many industrial products including machinery, metals, chemicals, food, textiles, etc., based on inspection images, defects such as shape defects, assembly defects, adhesion of foreign substances, internal defects and criticality, surface scratches, spots, stains, etc. Appearance inspection is widely performed to evaluate various workmanship. Conventionally, many of these appearance inspections have been performed by visual judgment by inspectors. On the other hand, with the increasing demand for mass production and quality improvement, inspection costs and the burden on inspectors are increasing. Also, sensory tests based on human senses require particularly high experience and skill. Reproducibility and reproducibility are also issues, as the evaluation values differ depending on the inspector, and the results differ each time the inspection is performed. There is a strong demand for automation of inspections in order to address issues such as inspection costs, skills, and individuality.

In recent years, the performance of machine learning has dramatically improved due to the proposal of deep network models represented by Convolutional Neural Network (CNN) (for example, Non-Patent Document 1). Many appearance inspection methods utilizing evaluation engines based on machine learning have been proposed. For example, WO2020/129617 (Patent Document 1) discloses a method of determining shape defects of welded portions using machine learning. ing.

In the learning of the evaluation engine, the image of the inspection object for learning (learning image) is input, and the difference between the estimated evaluation value output from the evaluation engine and the correct evaluation value taught by the inspector is reduced. Update the evaluation engine's internal parameters (such as network weights and biases). As for the timing of updating the internal parameters, it is common to divide the training data into sets called mini-batches and update the internal parameters for each mini-batch instead of learning all the training samples at once. be. This is called mini-batch learning, and when all mini-batches have been learned, all training samples have been used for learning. Learning all these mini-batches once is called one epoch. By repeating epochs many times, we optimize the internal parameters. We may also shuffle the training samples contained in the mini-batch for each epoch.

In machine learning, the so-called "overfitting" is a problem, in which the evaluation engine is over-optimized for specific learning samples and generalization performance declines. One of the factors is learning with a small number of learning samples. If a huge number of learning samples can be prepared, optimization of internal parameters with high generalization performance can be expected without optimizing for specific samples. On the other hand, collection of learning samples is often manually performed by inspectors, and it is often difficult to prepare a large number of learning samples.

Data Augmentation is known as a general method for suppressing overfitting. This is a technique of artificially padding a learning image prepared by an inspector by adding processing such as translation and rotation to the learning image. For example, if an inspection target with a certain correct evaluation value happens to appear in the upper left of all learning images, even if the correct evaluation value and the position of the inspection target on the image are irrelevant, the evaluation engine may erroneously learn that location is the key criterion. Such erroneous learning can be suppressed by using data augmentation to increase position variation. WO2020/129617 (Patent Literature 1) also discloses data extension such as changing the position of a defective shape portion.

International Publication WO2020/129617

As mentioned above, it is generally desirable to have many variations in inspection images. of effort. In addition, the correct evaluation value taught by the inspector includes fluctuations in judgment due to individual differences and erroneous teaching. In some cases, one learning sample may have multiple correct evaluation values. It is considered that over-learning is particularly likely to occur under such circumstances. On the other hand, as a practical matter, it is difficult to improve the quality of learning samples only by the efforts of inspectors. In addition, the above-described padding of the inspection image alone cannot sufficiently solve the problem of variations in correct evaluation values. Therefore, even if the quality of learning samples deteriorates, a mechanism is required to maintain the performance of estimating evaluation values by machine learning.

A visual inspection method and a visual inspection system according to one aspect of the present disclosure include:
(a) storing learning data, which is a set of learning samples that are pairs of learning images obtained by imaging inspection objects for learning and correct evaluation values for the learning images, in a storage resource;
(b) changing the correct evaluation value of the learning sample included in the learning data according to a changeable predetermined variation distribution, and generating an extended learning sample that is a learning sample having the changed value as the correct evaluation value;
(c) generating augmented learning data that is a set of the augmented learning samples;
(d) determining the internal parameters of the evaluation engine by learning the relationship between the learning image and the evaluation value based on the augmented learning data;
(e) Acquiring an inspection image of the inspection object,
(f) inputting the inspection image to the evaluation engine, and obtaining an estimated evaluation value, which is an estimated evaluation value of the inspection image, from the output of the evaluation engine;

According to this aspect, in the automation of appearance inspection using machine learning, the evaluation engine does not over-learn even against variations in correct evaluation values due to fluctuations in judgment, incorrect teaching, existence of multiple correct evaluation values, etc. It can be suppressed, and an improvement in the accuracy of the evaluation engine can be expected.

1 is a diagram showing an automatic visual inspection system and an overall processing sequence; FIG. FIG. 10 is a diagram showing expansion of learning data in the learning phase; It is a figure which shows an example of a to-be-tested object. FIG. 10 is a diagram showing an example of a distribution of correct evaluation values; It is a figure which shows an example of a to-be-tested object. FIG. 10 is a diagram showing an example of a distribution of correct evaluation values; FIG. 5 is a diagram showing an example of a known learning sequence; It is a figure which shows an example of a learning sequence. It is a figure which shows an example of a learning sequence. FIG. 10 is a diagram showing updating of the fluctuating distribution in the learning phase; FIG. 10 is a diagram showing a GUI for inputting and displaying a variation distribution of correct evaluation values; It is a figure which shows the hardware constitutions of an automatic visual inspection system.

The present embodiment will be described with reference to the drawings. This embodiment has the following features. However, the features included in this embodiment are not limited to those shown below.

(1) A learning image acquisition step of imaging an inspection object for learning and acquiring learning images {f_i} (i=1,...,Nf, Nf: number of images); A learning data input step of inputting a set {(f_i, g_i)} (i=1,...,Nf) of pairs (f_i, g_i) of correct evaluation values g_i taught by the user as learning samples as learning data. , a variation distribution input step of inputting the variation distribution d(g'_i;g_i) of the correct evaluation value, and based on the variation distribution d(g'_i;g_i), the correct evaluation value g_i an augmented learning data generation step that generates augmented learning data consisting of multiple augmented learning samples (f_i,g'_ij) (j=1,...,NS_i, NS_i: number of extensions) generated by changing the value of ; A learning step of learning the relationship between the learning image and the evaluation value based on the learning data to determine the internal parameters of the evaluation engine; Nf'', Nf'': the number of images), and an evaluation step of inputting the inspection image f''_i to the trained evaluation engine and outputting the estimated evaluation value g''^_i. The variation distribution d(g'_i;g_i) given in the variation distribution input step is characterized in that the distribution changes with the correct evaluation value g_i as a parameter.

Supplementary information about this feature. Variations in correct evaluation values are inherently present, and it is difficult to completely remove them. Therefore, in this embodiment, data expansion is performed based on the statistical variation distribution d of correct evaluation values. That is, from the correct evaluation value g_i given by the inspector, a plurality of extended correct evaluation values {g′_ij} (j=1, . These are learned as an extended learning sample group {(f_i,g'_ij)}. By varying the correct evaluation value, even if the taught correct value is somewhat inaccurate, excessive optimization for each learning sample can be suppressed, and an improvement in generalization performance can be expected.

The problem with this process is how to appropriately give the variation distribution d. In the unlikely event that an erroneous variation distribution is given, a large amount of false learning data will be generated, and the performance of estimating the evaluation value may deteriorate. This embodiment is characterized in that the variation distribution d is switched according to the value of the correct evaluation value g_i. That is, the variation distribution was given for each correct evaluation value as a probability distribution d(g'_i;g_i) with the extended correct evaluation value g'_i as a variable and the correct evaluation value g_i as a parameter. g'_ij is the jth extended correct evaluation value g'_i generated based on the variation d(g'_i;g_i).

(2) In the extended learning data generation step and learning step, one extended learning sample (f_i, g'_i) is generated and replaced with the learning sample (f_i, g_i) to generate an extended mini-batch m'_s.

Supplementary information about this feature. In general data augmentation methods, a plurality of augmented learning samples are generated from one learning sample, which increases the learning time. That is, due to the increase in the number of learning samples, either the number of learning samples included in one mini-batch before expansion, the number of mini-batches, or both increases compared to before data expansion. In this embodiment, since each learning sample (f_i, g_i) before extension is simply replaced with one extended learning sample (f_i, g'_i), the number of learning samples and the number of mini-batches do not change. Considering the variation distribution d as a probability distribution, a random number following this probability distribution is used to generate the extended evaluation correct value g'_i. When the number of generations is small, there may be bias in the augmented learning samples, but as the learning progresses with repeated epochs, the distribution of the correct augmented evaluation values in the trained augmented learning samples can approach the dispersion distribution d. Be expected.

(3) In the extended learning data generation step and the learning step, when the evaluation engine is iteratively trained while generating the extended learning data, the variation distribution d is updated during the iterations, and the extended learning data is generated based on the updated variation distribution. is characterized by generating As a specific example, the reliability R(g'^_i) of the estimated evaluation value g'^_i output by inputting the extended learning sample to the evaluation engine during learning is calculated, and the reliability R(g'^_i) is a parameter, and the variation distribution is updated during iteration.

Supplementary information about this feature. There is a limit to giving a highly accurate variation distribution from the beginning. Therefore, it is considered to change the variation distribution during learning to obtain a more appropriate distribution. As a method for changing the variation distribution, consider adaptively changing it according to the learning state of the evaluation engine. For example, the learning image f_i is input to the evaluation engine during learning, the estimated evaluation value g'^_i is output, and the reliability R(g'^_i) of g'^_i is calculated. As an example of how to calculate the reliability, the difference between the correct evaluation value g_i and the estimated evaluation value g'^_i, or the variation distribution d(g'_i;g_i) is regarded as a probability distribution, and the estimated evaluation value g'^_i is calculated as The imputed value d(g'^_i;g_i) (a measure of how likely it is that the estimated value is g'^_i), and for classification problems, the method based on the variation in the degree of membership to each classification class is mentioned. Regarding the degree of belonging to each classification class, if the degree of belonging to the correct classification class is outstanding, the reliability is high, but if there is also the degree of belonging to other classification classes, the reliability decreases according to the degree.

The variation distribution is changed during learning according to such reliability. By repeatedly updating the variation distribution and the evaluation engine, it is possible to estimate an appropriate variation distribution that could not be assumed before learning, and to obtain a higher performance evaluation engine by expanding the data based on this variation distribution. .
It should be noted that the embodiments described below do not limit the present invention, and not all of the elements described in the embodiments and their combinations are essential to the solution of the invention.

1. Automatic Visual Inspection System and Overall Processing Sequence FIG. 1 shows an automatic visual inspection system and overall processing sequence in the present invention. The processing sequence is roughly divided into a learning phase 100 and an inspection phase 101 .

In the learning phase, an inspection object for learning is imaged to acquire learning images {f_i} (i=1,...,Nf, Nf: number of images) (102). The learning image is acquired by imaging the surface or inside of the object to be inspected as a digital image with an imaging device such as a CCD camera, an optical microscope, a charged particle microscope, an ultrasonic inspection device, an X-ray inspection device, or the like. As another example of "acquisition", it is also possible to simply receive an image captured by another system and store it in the storage resource of the automatic visual inspection system.

Next, a correct evaluation value g_i is assigned to each learning image f_i (103). The evaluation value is an index for evaluating various workmanship such as shape defects, assembly defects, adhesion of foreign matter, defects and criticality inside the object to be inspected, surface scratches, spots, dirt, etc. can be defined. For example, evaluation values may be obtained by quantifying the criticality of defects and surface conditions (g_i=[g_min,g_max]), or by classifying the types of defects that have occurred as labels (g_i={ class1,…,classN}). A correct evaluation value g_i is assigned to these evaluation criteria based on the inspector's visual judgment and numerical values analyzed by other inspection devices and methods. Of course, it is desirable that this correct evaluation value g_i is accurate, but there is a possibility that it may include variations due to fluctuations in the judgment of the inspector, misinstruction, existence of multiple correct evaluation values, and the like.

A pair (f_i, g_i) of each learning image f_i and a correct evaluation value g_i is called a learning sample, and a set of learning samples {(f_i, g_i)} (i=1,..., Nf) is called learning data. The training data is used to train the rating engine (104). An evaluation engine is an estimator based on machine learning that takes an inspection image f_i as an input and outputs an estimated evaluation value g^_i. Various existing machine learning engines can be used as the evaluation engine. -nearest neighbor (k-NN) and the like. These evaluation engines can handle classification and regression problems.

Generally, learning samples are used to optimize the internal parameters of the evaluation engine so that an estimated evaluation value g^_i close to the taught correct evaluation value g_i is output. In this embodiment, the variation distribution d of the correct evaluation value g_i is input. Based on this variation distribution d, multiple augmented learning samples (f_i,g'_ij) (j=1,...,NS_i, NS_i : expansion number), and trains the evaluation engine based on the augmented learning data to optimize the internal parameters (106).

In the inspection phase, the actual inspection object is imaged to acquire inspection images f''_i (i=1,..., Nf'', Nf'': number of images) (102). The inspection image f''_i is input to the trained evaluation engine, and the estimated evaluation value g''^_i is output (107). The estimated evaluation value is checked by an inspector as necessary (108), and if there is a defect, countermeasures are fed back to the manufacturing process.

In the above-described embodiment, each of the learning image f_i and the inspection image f''_i is one image. A rating value may be estimated as an input to the rating engine. In this case, the learning image group and the inspection image group are f_i and f''_i, respectively. Similarly, there may be multiple types of evaluation values. Combining both, an evaluation engine with multiple inputs and multiple outputs may be used.

2. Data Extension 2.1 Data Extension Method A data extension method in this embodiment is shown in FIG. As described above, the pair (f_i, g_i) of the learning image f_i and the correct evaluation value g_i taught by the inspector is called a learning sample (in FIG. 2, three learning samples 201, 202, and 203 are shown as examples), A set of learning samples {(f_i, g_i)} (i=1, . . . , Nf) is called learning data (200). The correct evaluation value g_i in the learning data varies due to fluctuations in the judgment of the inspector, misinstruction, existence of a plurality of correct evaluation values, and the like. Variations in correct evaluation values are inherently present, and it is difficult to completely remove them. Therefore, in this embodiment, the variation of the correct evaluation value is given as the variation distribution d (211). A plurality of extended correct evaluation values {g′_ij} (j=1, . The expansion number Ns_i may be changed for each learning sample. However, since training samples with a large number of expansions may affect the learning of the evaluation engine more than training samples with a small number of expansions, weights w_i (i=1,..., Nf)(212) can be multiplied (e.g. weighted by the reciprocal of the expansion number). A pair (f_i, g'_ij) of a learning image f_i and an extended correct evaluation value g'_ij is called an extended learning sample (in FIG. 2, as an example, three extended learning samples 208 generated from a learning sample (f_1, g_1) , 209, and 210), and the set of extended learning samples generated from the i-th learning sample (f_i, g_i) is called an extended learning sample group S_i={(f_i, g'_ij)} (in FIG. (shows three extended learning sample groups 205, 206, 207). g'_ij is the j-th extended correct evaluation value g'_i generated from the learning sample (f_i, g_i). Furthermore, all the extended learning sample groups are collectively called extended learning data {(F_k, G_k)} (k=1,..., NF, NF=ΣNS_i) (204). That is, {(F_k, G_k)}={S_1,...,S_Nf}. The variation distribution d can be regarded as the tendency of the correct evaluation value to be mistaken or the probability distribution of the correct evaluation value. Even if there is, it can be expected to suppress excessive optimization for each learning sample and improve generalization performance.

The problem with this process is how to appropriately give the variation distribution d. In the unlikely event that an erroneous variation distribution is given, a large amount of false learning data will be generated, and the performance of estimating the evaluation value may deteriorate. On the other hand, a plurality of inspectors may assign a correct evaluation value g_i to each learning sample f_i, and the variation distribution d may be obtained from the actual degree of variation depending on the inspector. However, evaluation by a plurality of inspectors for each learning sample f_i significantly increases the inspection cost and the load on the inspectors. This embodiment is characterized in that the variation distribution d is switched according to the value of the correct evaluation value g_i. That is, the variation distribution was given for each correct evaluation value as a probability distribution d(g'_i;g_i) with the extended correct evaluation value g'_i as a variable and the correct evaluation value g_i as a parameter. g'_ij is the jth extended correct evaluation value g'_i generated based on the variation d(g'_i;g_i). The variation distribution d(g'_i;g_i) may be given by a histogram as shown in FIG. 4, which will be described later, or by a polygonal line as shown in FIG. 6, which will be described later. Also, it may be given by a combination of parametric functions (Gaussian distribution, etc.), or given by a free curve as shown in FIG. 11, which will be described later.

2.2 Concrete example 1 of data expansion
Taking the defect classification shown in FIG. 3 as an example of visual inspection, a specific example of variation distribution for this example will be described with reference to FIG. This inspection is an example of classifying foreign matter adhering to an inspection object into four defect classes D1 to D4. FIGS. 3A to 3D show two schematic diagrams of defects belonging to defect classes D1 to D4, respectively. 4A to 4D show defects 300 and 301 belonging to defect class D1, defects 302 and 303 belonging to defect class D2, defects 304 and 305 belonging to defect class D3, and defect class D4. Defects 306 and 307 belonging to , respectively, are displayed. Variation distributions for the correct evaluation values D1 to D4 are shown in FIGS. 4(a) to 4(d). In this example, the variation distribution is given by a histogram.

The variation distribution d(g'_i;g_i) (g_i="D1") shown in FIG. 4(a) will be described in detail. In d(g'_i;"D1"), the extended correct evaluation value D1 on the horizontal axis has the highest frequency of 3 on the vertical axis (400). This indicates that the training sample taught as D1 is most likely to have an actual correctness evaluation value of D1 as well. Therefore, when generating the extended learning sample group S_i={(f_i, g'_ij)} from the learning sample (f_i, g_i), the extended learning sample whose extended correct evaluation value g'_i is D1 is the most generate a lot. Based on the histogram frequency ratio accurately, the extended correct evaluation value of 3/6=50% of the extended learning samples in the extended learning sample group S_i is D1. D2 also has a relatively high frequency of 2 (401). This is because, as can be seen by comparing (a) and (b) of FIG. 3, the defect classes D1 and D2 are similar in that they are both jagged-shaped defects, and are easy to confuse in teaching. Therefore, the extended correct evaluation value of 2/6≈33% of the extended learning samples in the extended learning sample group S_i is D2. On the other hand, the power of D3 is 0 (402). This is because, as can be seen by comparing (a) and (c) of FIG. 3, it is impossible to confuse a jagged defect with a round defect. Therefore, the extended learning sample group S_i does not include an extended learning sample with an extended correct evaluation value of D3. In addition, although the degree of D4 is small at 1, it has a value (403). As can be seen by comparing (a) and (c) in Fig. 3, the defect classes D1 and D4 are not very similar, but there is some possibility of confusion in that they are both bumpy defects. It's for. Therefore, the extended correct evaluation value of 1/6≈17% of the extended learning samples in the extended learning sample group S_i is D4.

The variation distributions in FIGS. 4(b) to (d) are also explained in the same way. Incidentally, since the defect class D3 is unlikely to be confused with other defect classes, the variation distribution d(g'_i;"D3") shown in FIG. 404). In other words, it is not necessary to expand the learning sample whose correct evaluation value is D3.

2.3 Concrete example 2 of data expansion
Taking the surface roughness evaluation shown in FIG. 5 as an example of visual inspection, a specific example of the variation distribution for this example will be described with reference to FIG. This inspection is an example of quantifying and evaluating the roughness level of the surface of the inspection object from 1.0 to 3.0. The evaluation value is the roughness level, and the smaller the value, the better the condition. Schematic diagrams (500-504) of surface images with roughness levels of 1.0, 1.5, 2.0, 2.5 and 3.0 are shown in FIGS. 5(a)-(e), respectively. Although the correct evaluation value g_i of the roughness level was given in increments of 0.5 in the learning samples, the roughness level is a continuous value, and there are inspection objects with intermediate roughness levels. Therefore, the estimation engine deals with regression problems. Figures 6(a) to 6(e) show the variation distribution for correct evaluation values of 1.0 to 3.0. In this example, the variation distribution is given by a polygonal line.

The variation distribution d(g'_i;g_i) (g_i=1.0) shown in FIG. 6(a) will be described in detail. A polygonal line of d(g'_i;1.0) indicates the probability distribution that the actual evaluation value of the learning sample whose correct evaluation value g_i is 1.0 is the expanded correct evaluation value g'_i on the horizontal axis. Therefore, the extended correct evaluation value g'_i is generated using the value of the polygonal line as the generation probability. The value of the polygonal line at the extended correct evaluation value of 1.0 is the highest (601), but as the extended correct evaluation value increases, the value of the polygonal line gradually decreases, reaching 0 at the extended correct evaluation value of 2.0 (604). Since the roughness level changes continuously, it is highly likely that the correct evaluation values will also vary continuously. Therefore, when generating the extended learning sample group S_i={(f_i, g'_ij)} from the training samples (f_i, g_i), the largest number of extended learning samples with a coarseness level value of 1.0 are generated ( 601), some extended learning samples with a coarseness level value of 1.5 are also generated (603), but no extended learning samples with a coarseness level value of 2.0 or greater are generated (604-606). In addition, extended learning samples with intermediate roughness levels as extended correct evaluation values are not limited to roughness levels 1.0 and 1.5 (indicated by black circles in Fig. 6(a)) included in the learning data. (indicated by white circles in FIG. 6(a)). For example, augmented learning samples with coarseness level 1.25 can be generated 602 with a frequency intermediate between the generation frequencies of coarseness levels 1.0 and 1.5.

The variation distributions in FIGS. 6(b) to (e) are similarly explained. As a major tendency, in any variation distribution, there is a probability distribution in the evaluation values around the correct evaluation value g_i, and the value of the probability distribution decreases as the distance from the correct evaluation value g_i increases. However, in this example, there is a discontinuous probability distribution change between roughness levels 1.5 and 2.0. As shown in FIG. 5, a roughness level of 1.5 or less is within the normal range and can be shipped as a product. On the other hand, roughness levels of 2.0 and above are of poor quality and cannot be shipped. The inspection images of roughness levels 1.5 and 2.0 shown in FIGS. 5(b) and 5(c), respectively, also have discontinuous changes in appearance, and the decision line (505) for determining whether or not to ship is set between them. For this reason, there is a tendency that a decision error that crosses this decision line is less likely to occur. Therefore, this tendency was reflected in the variation distribution. 6(b) and 6(c) show this tendency in an easy-to-understand manner. In FIG. 6(b), as the value of the extended correct value g'_i on the horizontal axis increases from 1.5 to 2.0, the value of the broken line sharply decreases (607→608). Similarly, in FIG. 6(c), as the value of the extended correct value g'_i on the horizontal axis increases from 2.0 to 1.5, the value of the broken line sharply decreases (610→609). In this way, it is possible to easily reflect the tendency of the correct evaluation value to be mistaken in the variation distribution.

2.4 Effects of data expansion Figures 4 and 6 are just examples of variation distributions, and various other variation distributions can be considered depending on the inspection object and evaluation items. In addition, it is considered that the correct evaluation value g_i and the variation distribution are highly dependent in many cases. Therefore, it was considered effective to switch the distribution according to the value of the correct evaluation value g_i. Variation in learning data is a trend of error susceptibility obtained from field experience, and can be thought of as so-called "domain knowledge." According to this embodiment, it is possible to effectively and efficiently incorporate such domain knowledge in training of the evaluation engine. In other words, for example, defining a variation distribution for each learning sample would require a huge amount of work, and conversely, defining a uniform variation distribution for all training samples would impair accuracy and generate a large number of erroneous extended learning samples. There is fear. On the other hand, according to the present embodiment, it is possible to relatively easily reflect the properties of the object to be inspected in the variation distribution, and this was shown using the two specific examples described above. It was found that even if incorrect teaching is included, it is possible to achieve both accuracy and work cost by defining the variation distribution d using the correct evaluation value g_i given by the inspector as a parameter.

3. Learning sequence (generation timing and distribution of augmented learning data)
In the learning phase, the learning sequence of the evaluation engine using the augmented learning data according to the present embodiment includes several examples. In each embodiment, the number of expansions of the expansion learning data, the timing of expansion processing, and the like are different. A typical example will be specifically described below.

3.1 Learning sequence 1
First, a general learning sequence of an evaluation engine using learning data {(f_i, g_i)} will be described with reference to FIG. In the learning of the evaluation engine, the image of the inspection object for learning (learning image f_i) is input, and the difference between the estimated evaluation value g^_i output from the evaluation engine and the correct evaluation value g_i taught by the inspector Update the evaluation engine's internal parameters (network weights, biases, etc.) so that As for the timing of updating the internal parameters, instead of learning all the training samples at once, the training data (704) is divided into several mini-batches {m_s} (s=1,...,Nm, Nm: number of mini-batches, m_s It is common to divide 705 into a set called ⊂{(f_i,g_i)} and update the internal parameters for each mini-batch.This is called mini-batch learning, and when all mini-batches have been learned All the training samples have been used for training.This single learning of all mini-batches is called one epoch, and by repeating the epoch many times, the internal parameters are optimized. is denoted as {e_t} (t = 1, ..., Ne, Ne: the number of epochs).In Fig. 7, the state of mini-batch division is illustrated in the first epoch e_1 (700).The illustration is omitted. However, the same mini-batch division is performed in the second and subsequent epochs e_2 to e_Ne (701 to 703), and the learning samples included in the mini-batch may be shuffled for each epoch.

An example of the learning sequence of the evaluation engine using the extended learning data {(f_i, g'_ij)} in this embodiment will be described with reference to FIG. 800-803 are epochs and 805 is mini-batch. In this embodiment, the learning data {(f_i, g_i)} (704) in FIG. 7 are replaced with extended learning data {(f_i, g'_ij)} (804). At this time, the mini-batches generated by dividing the augmented learning data are divided into augmented mini-batches {m'_s} (s=1,...,Nm', Nm': the number of augmented mini-batches, m'_s⊂{(f_i,g'_ij)} Generally, the number of extended learning samples NF in the extended learning data is larger than the number of learning samples Nf in the learning data (NF>Nf), so in the embodiment of FIG. Alternatively, the number of samples included in one extended mini-batch increases, so if the number of epochs remains the same, the learning time increases.

3.2 Learning sequence 2
An example of the learning sequence of the evaluation engine using the extended learning data {(f_i, g'_ij)} in this embodiment will be described with reference to FIG. In this embodiment, for one i-th learning sample (f_i, g_i) included in the s-th mini-batch m_s obtained by dividing the learning data (904) given by the inspector, based on the variation distribution d It is characterized by generating an extended mini-batch m'_s by generating one extended learning sample (f_i, g'_i) and replacing it with the learning sample (f_i, g_i). For example, one extended learning sample (f_1, g'_1) (916) is generated from one learning sample (f_1, g_1) (909) included in the first mini-batch m_1 (905) (911). This is done for all learning samples (909, 910, etc.) included in all mini-batches (905-908, etc.) to generate extended mini-batches (912-915, etc.) and learning samples (916, 917, etc.).

Supplementary information about this feature. In general data augmentation, since a plurality of augmented learning samples are generated from one learning sample, the learning time increases. That is, due to the increase in the number of learning samples, either the number of learning samples included in one mini-batch before expansion, the number of mini-batches, or both increases compared to before data expansion. In this embodiment, since one learning sample (f_i, g_i) before extension is simply replaced with one extended learning sample (f_i, g'_i), the number of learning samples and the number of mini-batches do not change. Considering the variation distribution d as a probability distribution, a random number following this probability distribution is used to generate the extended evaluation correct value g'_i. Generation of the extended evaluation correct value g'_i using this random number is performed at each epoch. Therefore, even for the same learning sample, the value of the extended evaluation correct value g'_i may change depending on the epoch. When the number of generations is small, there may be bias in the augmented learning samples, but as the learning progresses with repeated epochs, the distribution of the correct augmented evaluation values in the trained augmented learning samples can approach the dispersion distribution d. Be expected. As a result, if the number of epochs is the same, it is possible to learn the augmented learning data reflecting the information of the variation distribution in the same time as the learning of the learning data {(f_i, g_i)}. By distributing the augmented learning data, which has an increased number of samples due to data augmentation, to be learned over the entire epoch, it is possible to take into account the variation distribution of the correct evaluation values without increasing the learning cost as a result. can also

4. Variation distribution change during learning 4.1 Basic processing It is characterized by generating augmented learning data based on the distribution. As a specific example, the reliability R(g'^_i) of the estimated evaluation value g'^_i output by inputting the extended learning sample to the evaluation engine during learning is calculated, and the reliability R(g'^_i) is a parameter, and the variation distribution is updated during iteration.

Supplementary information about this feature. There is a limit to giving a highly accurate variation distribution from the beginning. Therefore, it is considered to change the variation distribution during learning to obtain a more appropriate distribution. As a method for changing the variation distribution, consider adaptively changing it according to the learning state of the evaluation engine.

A specific example of changing the variation distribution during learning will be described with reference to FIG. FIG. 10 incorporates a mechanism for changing the variation distribution during learning to the learning sequence described with reference to FIG. However, it is not limited to the learning sequence described with reference to FIG. 9 that the mechanism for changing the variation distribution during learning can be incorporated. For example, the change of variation distribution during learning can be applied to the learning sequence described with reference to FIG. 8 and other learning sequences. The epoch corresponding to the t-th epoch e_t in FIG. 9 is indicated at 1000, but the same applies to other epochs. First, generate mini-batches {m_s} (1002, 1003) obtained by dividing the learning data (1001), and for each learning sample (f_i, g_i) (1004-1006) included in the mini-batch, the variation distribution d Generate (1007) augmented learning samples (f_i, g'_i) (1010-1012) based on (1017) and replace them with training samples (f_i, g_i) to obtain augmented mini-batch {m'_s} (1008, 1009). Then learn the extended mini-batch {m'_s} to update the internal parameters of the evaluation engine (1013). The processing up to this point is the same as in FIG. Next, each learning image f_i is input to the evaluation engine (1013) being trained, the estimated evaluation value g'^_i is output, and the reliability of g'^_i R(g'^_i) (1014 to 1016) Calculate As an example of how to calculate the reliability, the difference between the correct evaluation value g_i and the estimated evaluation value g'^_i, or the variation distribution d(g'_i;g_i) is regarded as a probability distribution, and the estimated evaluation value g'^_i is calculated as The imputed value d(g'^_i;g_i) (a measure of how likely it is that the estimated value is g'^_i), and for classification problems, the method based on the variation in the degree of membership to each classification class is mentioned. Regarding the degree of belonging to each classification class, if the degree of belonging to the correct classification class is outstanding, the reliability is high, but if there is also the degree of belonging to other classification classes, the reliability decreases according to the degree. Variation distribution (1017) is changed during learning according to such reliability. The timing of the change may be for each mini-batch learning or for each epoch. Based on the modified variation distribution (1017) in the next mini-batch or epoch, generate (1007) the augmented learning samples (f_i,g'_i) (1010-1012) and use them to run the evaluation engine (1013). learn.

By repeatedly updating the variation distribution and the evaluation engine, it is possible to estimate an appropriate variation distribution that could not be assumed before learning, and to obtain a higher performance evaluation engine by expanding the data based on this variation distribution. .

4.2 Processing Variations Processing variations for the basic processing of changing the variation distribution described in 4.1 will be described. In 4.1, the variation distribution was changed based on the reliability R(g'^_i) of the estimated evaluation value g'^_i of the augmented learning sample. If the reliability is high, there is a high possibility that the variation distribution that generated the extended learning sample was appropriate. Conversely, if the reliability is low, it is necessary to change the variation distribution. The variation distribution may be changed (A) for each correct evaluation value g_i, or (B) for each learning sample (f_i, g_i). In addition to the reliability R(g'^_i), (C) the evaluation result of the verification data or (D) a heuristic method may be used as clues for changing the variation distribution.

First, we will describe in detail the variations of the parameters that change the variation distribution. In (A) above, the variation distribution is switched according to the value of the correct evaluation value g_i, and the variation distribution is given by d(g'_i;g_i) using g_i as a parameter. That is, the variation distribution is changed based on the reliability of the estimated evaluation value g'^_i for the augmented learning sample whose correct evaluation value is g_i.

In (B) above, the variation distribution is switched for each learning sample (f_i,g_i), and the variation distribution is given by d(g'_i;(f_i,g_i)) with (f_i,g_i) as a parameter. be done. Since the reliability can be calculated for each extended learning sample (f_i,g'_ij), it is possible to evaluate the validity of the variation distribution for each learning sample (f_i,g_i) that is the basis of the extended learning sample. is. Therefore, it is possible to change the variation distribution for each learning sample (f_i, g_i). Manually giving a variation distribution for each learning sample requires a huge amount of work cost, and it is also difficult to give an appropriate variation distribution. By updating the variability distribution based on the reliability of the training sample to improve the performance of the evaluation engine, the variability distribution is optimized for each training sample in parallel with optimizing the internal parameters of the evaluation engine without human intervention. become possible.

Next, we will describe in detail the variations of the clues that change the variation distribution. In (C) above, the variation distribution is changed based on the evaluation result of the verification data. In the learning of the evaluation engine, data called verification data is prepared separately from the training data used for learning (called learning data in this disclosure) in order to obtain internal parameters with high generalization performance for unlearned data. It is common to In learning, the internal parameters are successively updated to improve the estimation results for the training data, but which internal parameters are finally adopted depends on the estimation results of the verification data (untrained data) that are not used in training. An internal parameter that becomes In this embodiment, the variation distribution may be selected based on the reliability of the estimated evaluation value of the verification data instead of the learning data.

In (D) above, a heuristic method may be used to optimize the variability distribution. In other words, the variability distribution given before learning is used as the initial value, and the variability distribution is slightly changed during learning, resulting in an update to a variability distribution that improves the performance of the evaluation engine (accuracy rate and reliability of estimated evaluation values). continue. Such a method allows obtaining a heuristically relevant variability distribution without using an analytical approach.

5. GUI
In this embodiment, FIG. 11 shows an example of a graphical user interface (GUI) for a user such as an inspector to specify and confirm the variation distribution (1100). This GUI can display the variation distribution for each correct evaluation value (1102 to 1104 in 1101). The variation distribution to be displayed can be displayed by switching between the distribution initially specified by the user and the distribution updated during learning using radio buttons or the like (1105). In the latter, by specifying the ID of the epoch (1106), it is possible to display the variation distribution during the update at any epoch. Variation distribution can be given by a histogram (e.g. Fig. 4), a line (e.g. Fig. 6), a combination of parametric functions, a free curve (e.g. 1102 or 1103 in Fig. 11), etc., and the method of giving can be selected with radio buttons. (1107).

The information of the learning sample (f_i, g_i) can be displayed (1108) as a material for determining the validity of the processing result or as a material for determining the variation distribution. Several training samples can be displayed side by side and compared (1109, 1121, 1133). The information "display 1" (1109) of one learning sample will be taken up and the details of the display contents will be explained. The learning sample to be displayed can be specified by the ID of the learning image (1110), and can be filtered by the correct evaluation value g_i (1111). The inspection image f_i, the estimated evaluation value g^_i for the learning sample (f_i, g_i), and the reliability R(g^_i) can be displayed (1112, 1113, 1114). An inspection image (1112) in this example is a defect image inside a semiconductor device captured by an ultrasonic inspection apparatus. Information on the extended learning sample {(f_i,g'_ij)} generated from the learning sample (f_i,g_i) can also be displayed (1115). Information of multiple extended samples can also be displayed side by side (1116, 1117). Display contents include extended correct evaluation value g'_ij, estimated evaluation value g'^_ij for extended learning sample (f_i,g'_ij), confidence R(g'^_ij) (1118, 1119, 1120) . The information "display 2" (1121) of other learning samples is the same as "display 1". That is, 1122-1132 correspond to 1110-1120. In addition, in this example, information on learning samples is displayed in displays 1109 and 1121, but information on inspection samples (inspection image f''_i, estimated evaluation value g''^_i, etc.) can also be displayed in the same way. be.

This embodiment suppresses the over-learning of the evaluation engine against variations in correct evaluation values due to fluctuations in judgment, incorrect teaching, and the existence of multiple correct evaluation values in the automation of appearance inspections using machine learning. can do. As a practical matter, it is difficult to improve the quality of learning samples only by the efforts of inspectors. In addition, the variation in correct evaluation values cannot be sufficiently resolved by padding the inspection image alone. For this reason, the present embodiment provides a mechanism that maintains the performance of estimating an evaluation value by machine learning even if the quality of learning samples is degraded. This makes it possible to estimate the evaluation value from the inspection image of the inspection object with high accuracy and automatically evaluate the performance of the inspection object.

In this embodiment, two-dimensional image data is used as input information, but one-dimensional signals such as received waves of ultrasonic waves, or three-dimensional volume data acquired by a laser range finder, etc., can also be used as input information. It is possible to apply the technique of this embodiment. The method of the present embodiment can also be applied when there are multiple input images and multiple types of estimated evaluation values (evaluation engine has multiple inputs and multiple outputs).

6. Hardware Configuration of Automatic Visual Inspection System FIG. 12 shows an automatic visual inspection system that implements the visual inspection method described in the above embodiments. The automatic visual inspection system is composed of the imaging device described above and a computer. Examples of imaging devices have already been described.

The computer is a component for processing the visual inspection method described in this embodiment, and has the following.
*Processor: Examples of processors include CPU, GPU, and FPGA, but other components may be used as long as they can process the visual inspection method.
* Storage resources: Examples of storage resources include RAM, ROM, HDD, and non-volatile memory (flash memory, etc.). Note that the storage resource may include a volatile memory (the aforementioned RAM is one example). The storage resource may store a program (referred to as a visual inspection program) that causes a processor to execute the visual inspection method described in the above embodiments. Also, the storage resource may store data referred to or generated by the visual inspection program. An example of data stored in that storage resource is:
**Learning image, correct evaluation value, learning data **Extended learning sample, extended learning data,
** Internal parameters of the rating engine,
**Inspection image,
** Estimated rating.
*GUI device: Examples of a GUI device include a display and a projector, but other devices may be used as long as they can display a GUI.
* Input device: Examples of input devices include keyboards, mice, and touch panels, but other devices may be used as long as they are configured to accept operations from the user. Also, the input device and the GUI device may be an integrated device.
*Communication interface device: Examples of communication interfaces include USB, Ethernet, and Wi-Fi. Any interface that can receive images directly from an imaging device or that allows the user to send the images to a computer Other interface devices may be used. Alternatively, a portable nonvolatile storage medium (for example, flash memory, DVD, CD-ROM, Blu-ray disc, etc.) storing the image may be connected to the communication interface, and the image may be stored in the computer.
The above is the hardware configuration of the computer. It should be noted that the automatic visual inspection system may include a plurality of computers and a plurality of imaging devices.

Note that the aforementioned visual inspection program may be stored in the computer through the following path:
* Store the visual inspection program in a portable non-volatile storage medium, and distribute the program to computers by connecting the medium to a communication interface.
* The program distribution server distributes the appearance inspection program to the computer. The program distribution server has a storage resource that stores the appearance inspection program, a processor that performs distribution processing for distributing the appearance inspection program, and a communication interface device that can communicate with the communication interface device of the computer.
This completes the description of the embodiment. As described above, the embodiments described so far do not limit the claimed invention, and all of the elements described in the embodiments and their combinations are essential to the solution of the invention. not necessarily.
This application claims the benefit of priority based on Japanese Patent Application No. 2021-47889 filed on March 22, 2021, the entire disclosure of which is incorporated herein by reference.

100... Learning phase 101... Inspection phase 201, 202, 203... Learning samples 205, 206, 207... Extended learning sample groups 208, 209, 210... Extended learning samples

Claims (10)

1. (a) storing learning data, which is a set of learning samples that are pairs of learning images obtained by imaging inspection objects for learning and correct evaluation values for the learning images, in a storage resource;
   (b) changing the correct evaluation value of the learning sample included in the learning data according to a changeable predetermined variation distribution, and generating an extended learning sample that is a learning sample having the changed value as the correct evaluation value;
   (c) generating augmented learning data that is a set of the augmented learning samples;
   (d) determining the internal parameters of the evaluation engine by learning the relationship between the learning image and the evaluation value based on the augmented learning data;
   (e) Acquiring an inspection image of the inspection object,
   (f) inputting the inspection image to the evaluation engine and obtaining an estimated evaluation value, which is an estimated evaluation value of the inspection image, from the output of the evaluation engine;
   Appearance inspection method.
2. In the above (b), from each of the learning samples included in the learning data, generating an extended learning sample by changing the correct evaluation value using a random number according to the variation distribution,
   In (c) above, generating the extended learning data by replacing the learning sample of the learning data with an extended learning sample generated from the learning sample;
   In (d) above, updating internal parameters of the evaluation engine based on the augmented learning data;
   Repeating (b) to (d) as one epoch, repeating a plurality of epochs while regenerating the correct evaluation value in (b) for each epoch;
   The appearance inspection method according to claim 1.
3. In (a) above, further dividing the learning data into a plurality of mini-batches that are subsets;
   In the above (b), from each of the learning samples included in each of the mini-batches, generating an extended learning sample by changing the correct evaluation value using a random number according to the variation distribution,
   In the above (c), generating a plurality of extended mini-batches that are a set of the extended learning samples in units of the mini-batch, and generating the extended learning data with the plurality of extended mini-batches as a subset;
   In (d), learning the extended learning data for each extended mini-batch, and updating the internal parameter each time the extended mini-batch is learned;
   The appearance inspection method according to claim 2.
4. Furthermore,
   (g) displaying the variation distribution in a GUI so that the user can check it;
   The appearance inspection method according to claim 1.
5. In (d) in repeating the epoch, the learning image is input to the evaluation engine during learning, and the variation distribution used in (b) is changed based on the estimated evaluation value output from the evaluation engine. do,
   The appearance inspection method according to claim 2.
6. calculating reliability based on the estimated evaluation value output from the evaluation engine, and changing the variation distribution used in (b) based on the reliability;
   The appearance inspection method according to claim 5.
7. In the above (d) in repeating the epoch, each time the extended mini-batch is learned, the learning image is input to the evaluation engine during learning, and based on the estimated evaluation value output from the evaluation engine, the ( changing the variation distribution used in b);
   The appearance inspection method according to claim 3.
8. calculating reliability based on the estimated evaluation value output from the evaluation engine, and changing the variation distribution used in (b) based on the reliability;
   The appearance inspection method according to claim 7.
9. A visual inspection system having a processor and storage resources,
   The processor
   (a) storing learning data, which is a set of learning samples that are pairs of learning images obtained by imaging inspection objects for learning and correct evaluation values for the learning images, in the storage resource;
   (b) changing the correct evaluation value of the learning sample included in the learning data according to a changeable predetermined variation distribution, and generating an extended learning sample that is a learning sample having the changed value as the correct evaluation value;
   (c) generating augmented learning data that is a set of the augmented learning samples;
   (d) determining the internal parameters of the evaluation engine by learning the relationship between the learning image and the evaluation value based on the augmented learning data;
   (e) Acquiring an inspection image of the inspection object,
   (f) inputting the inspection image to the evaluation engine and obtaining an estimated evaluation value, which is an estimated evaluation value of the inspection image, from the output of the evaluation engine;
   Appearance inspection system.
10. (a) storing learning data, which is a set of learning samples that are pairs of learning images obtained by imaging inspection objects for learning and correct evaluation values for the learning images, in a storage resource;
    (b) changing the correct evaluation value of the learning sample included in the learning data according to a changeable predetermined variation distribution, and generating an extended learning sample that is a learning sample having the changed value as the correct evaluation value;
    (c) generating augmented learning data that is a set of the augmented learning samples;
    (d) determining the internal parameters of the evaluation engine by learning the relationship between the learning image and the evaluation value based on the augmented learning data;
    (e) Acquiring an inspection image of the inspection object,
    (f) inputting the inspection image to the evaluation engine and obtaining an estimated evaluation value, which is an estimated evaluation value of the inspection image, from the output of the evaluation engine;
    A visual inspection program that allows a computer to do things.
    
    

Images