JP2024008670A - Machine learning apparatus - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a machine learning apparatus capable of saving labor for constructing a learned model.

SOLUTION: A machine learning apparatus 1 comprises an arithmetic processing apparatus 2 and a storage apparatus 3. The storage apparatus 3 comprises: an image data storage unit 20 for storing training image data P1, P2 and test image data P5, P6; a model storage unit 21 for storing a learned model M; and a defective factor training image data storage unit 22 for storing defective factor training image data P3 in which the defective factor training image data P2 labeled as defective is linked with defective factor information including a position of a defective factor 30 as an annotation. The arithmetic processing apparatus 2 comprises: a model generation unit 11; a quality determination unit 12 for determining whether each of the test image data P5, P6 is a non-defective product or a defective product; and an analysis and teaching unit 13 for analyzing an erroneous determination factor based on the defective factor information and teaches a solution according to the erroneous determination factor if a true defective part of the defective product is erroneously determined.

SELECTED DRAWING: Figure 1

COPYRIGHT: (C)2024,JPO&INPIT

Description

The present invention relates to a machine learning device.

Conventionally, as described in Patent Document 1 (Japanese Unexamined Patent Application Publication No. 2021-174417), a trained model generated by machine learning training image data is used to determine whether image data to be inspected is a good product or a non-defective product. Inspection systems that estimate and evaluate whether a product is defective are known.

JP 2021-174417 Publication

In order to evaluate the accuracy of the trained model, test image data for evaluation is used to estimate whether the test image data is a good product or a defective product. At this time, if the determination accuracy of the trained model is not sufficient, a defective product may be erroneously determined to be a non-defective product. In order to improve the judgment accuracy of trained models, it is necessary to analyze the causes of erroneous judgments and take appropriate measures such as adding training image data and re-collecting data.

However, according to the prior art, only whether the test image data is a non-defective product or a defective product is output. For this reason, it is sometimes difficult to analyze the cause of a defective product being erroneously determined as a non-defective product. Furthermore, even if the cause is identified, it is still necessary to consider what specific measures can be taken to improve the judgment accuracy, which increases the amount of effort required to build a trained model. there were.

The present invention has been made in view of such problems, and aims to provide a machine learning device that can save labor for constructing a trained model.

One aspect of the present invention is
A machine learning device comprising an arithmetic processing unit and a storage device,
The storage device is
an image data storage unit that stores training image data for learning and test image data for evaluation;
a model storage unit that stores a learned model generated by performing machine learning using the training image data and the test image data;
A defective factor training image data storage unit that stores defective factor training image data in which defective product training image data labeled as defective among the training image data is linked with defective factor information including the location of the defective factor as an annotation. and,
Equipped with
The arithmetic processing device is
a model generation unit that generates the learned model by performing the machine learning using the training image data and the test image data;
a quality determination unit that uses the trained model to determine whether the test image data labeled as a non-defective product or a defective product is a non-defective product or a defective product;
When a true defective part of a defective product is erroneously determined based on the determination result of the quality determining section, the erroneous determination factor is analyzed based on the defect factor information, and a solution is taught in accordance with the erroneous determination factor. analysis teaching department;
There is a machine learning device equipped with.

According to one aspect of the present invention, it is possible to know what countermeasures should be applied to training image data so that test image data that has been incorrectly determined will be correctly determined. This makes it possible to reduce the effort required to estimate the cause of the erroneous determination and to devise countermeasures based on the estimated cause for the erroneously determined test image data. As a result, the overall effort required to construct a trained model can be reduced.

As described above, according to the above aspect, it is possible to save labor for constructing a learned model in a machine learning device.

1 is a diagram showing the configuration of a machine learning device according to a first embodiment; FIG. 3 is a flowchart showing the overall operation of the machine learning device according to the first embodiment. 3 is a diagram for explaining the operations of S1 to S3 in FIG. 2. FIG. FIG. 3 is a diagram for explaining the operations of S4 to S5 in FIG. 2. FIG. FIG. 6 is a diagram for explaining a case where a truly defective part is erroneously determined. FIG. 3 is a diagram for explaining what is taught regarding solution 1; FIG. 6 is a diagram for explaining padding processing that is executed when solution 1 is selected. FIG. 7 is a diagram for explaining what is taught regarding solution 2. FIG. 7 is a diagram for explaining screen splitting processing that is executed when solution 2 is selected. 10 is a diagram for explaining screen division processing using model α in FIG. 9 as an example. FIG. FIG. 7 is a diagram for explaining the content taught regarding Solution 3. FIG. 7 is a diagram for explaining masking processing that is executed when solution 3 is selected. FIG. 7 is a diagram for explaining what is taught regarding Solution 4 in the machine learning device of Embodiment 2; 12 is a flowchart showing the overall operation of the machine learning device according to the third embodiment.

(Embodiment 1)
1. Outline of Machine Learning Device The machine learning device according to the present invention is applied to an inspection device that inspects the quality of a workpiece. As an inspection device, it can be applied to, for example, an inspection device for inspecting die-cast parts used in automobiles. The inspection device determines the quality of inspection image data obtained by capturing an image of the inspection object. For example, it is possible to determine whether there is an abnormality in the appearance of an object to be inspected. Furthermore, for example, it is possible to determine the presence or absence of scratches generated on the surface of the workpiece, or to determine the degree of shape error of the workpiece.

An outline of a machine learning device 1 according to Embodiment 1 of the present invention will be described with reference to FIGS. 1, 3, and 4. As shown in FIG. 1, a machine learning device 1 according to the present embodiment includes an arithmetic processing device 2 and a storage device 3. The machine learning device 1 is connected to an input device 4 and a display device 5.

The input device 4 is an input interface for transmitting various image data, which will be described later, to an image data acquisition unit 10, which will be described later. For example, a camera, a scanner, etc. can be used as the input interface. Further, as the input interface, a reading device for reading images stored in an external storage device such as a CD or a DVD is also exemplified.

The display device 5 displays content taught to the user by the analysis teaching section 13, which will be described later. As the display device 5, for example, a monitor, a printing device, etc. can be adopted.

The arithmetic processing device 2 includes an image data acquisition unit 10 , a model generation unit 11 , a quality determination unit 12 , an analysis teaching unit 13 , a failure factor integrated image data generation unit 14 , and a relearning training image data generation unit 15 and.

The image data acquisition unit 10 is connected to the input device 4 and receives training image data P1 and P2 for learning, test image data P5 and P6 for evaluation, and defect factor training image data P3 from the input device 4. (See Figures 3 and 4).

As shown in FIG. 3, the training image data P1 and P2 include good product training image data P1 labeled as a non-defective product and defective product training image data P2 labeled as a defective product. As shown in FIG. 4, the test image data P5 and P6 include test image data P5 of a non-defective product labeled as a non-defective product and test image data P6 of a defective product labeled as a defective product.

As shown in FIG. 3, the defective factor training image data P3 is obtained by linking defective product training image data P2 and defective product test image data P6 with defective factor information including the position of the defective factor 30 as an annotation. . The defective factor training image data P3 has a true defective part that is the cause of the defective product. Examples of the defective factors 30 include scratches, irregularities, dimensional defects, and distortions. The position, size, shape, etc. of the defective factor 30 are linked to the defective factor training image data P3 at the pixel level.

As shown in FIG. 1, the image data acquisition unit 10 transmits training image data P1, P2, test image data P5, P6, and defect factor training image data P3 to the storage device 3. Further, the image data acquisition unit 10 transmits the training image data P1, P2 and the test image data P5, P6 to the model generation unit 11, the analysis teaching unit 13, and the relearning training image data generation unit 15. Further, the image data acquisition section 10 transmits the defect factor training image data P3 to the defect factor integrated image data generation section 14.

The model generation unit 11 generates a learned model M by a known method by performing machine learning using the training image data P1, P2 and the test image data P5, P6 (see FIG. 3). As shown in FIG. 1, the model generation unit 11 transmits the generated trained model M to the storage device 3 and the quality determination unit 12.

The quality determining unit 12 uses the learned model M to determine whether the non-defective product test image data P5 and the defective product test image data P6 are non-defective products or defective products. The quality determining unit 12 transmits the determination results regarding the non-defective product test image data P5 and the defective product test image data P6 to the analysis teaching unit 13.

When a true defective part of a defective product is erroneously determined based on the determination result of the quality determining section 12, the analysis teaching unit 13 analyzes the erroneous determination factor based on the defective factor information and solves the problem according to the erroneous determination factor. Teach them strategies. The analysis teaching unit 13 teaches at least one solution. The analysis teaching unit 13 teaches the solution to the user by displaying the solution on the display device 5. The analysis teaching unit 13 transmits the solution to the relearning training image data generation unit 15.

The defective factor integrated image data generation unit 14 generates defective factor integrated image data P4 by integrating a plurality of defective factor training image data P3 (see FIG. 3). The defect factor integrated image data P4 is one piece of image data in which a plurality of defect factor training image data P3 are accumulated. In the defect factor integrated image data P4, the probability of occurrence of the defect factor 30 in the defect factor training image data P3 with respect to the number of defect factor training image data P3 is recorded for each pixel included in one sheet of image data. The probability of occurrence of the defective factor 30 is the value obtained by dividing the number of pixels corresponding to the position of the defective factor 30 by the total number of pixels.

Furthermore, the failure factor integrated image data P4 can count the probability of occurrence of the failure factor 30 at the pixel level. That is, in the defective factor integrated image data P4, the number of pieces of defective factor training image data P3 in which the position of the defective factor 30 was present is recorded for each pixel as one piece of image data. In short, the defect factor integrated image data P4 records information as to whether or not the defect factor 30 was present in the defect factor training image data P3 for each pixel forming the image.

As shown in FIG. 1, the failure factor integrated image data generation unit 14 transmits the generated failure factor integrated image data P4 to the storage device 3, the analysis teaching unit 13, and the relearning training image data generation unit 15.

The relearning training image data generation section 15 includes an padding processing section 16, an image division processing section 17, and a masking processing section 18.

The inflating processing unit 16 performs inflating processing on the defective product training image data P2 including the erroneously determined true defective part based on the analysis result of the analysis teaching unit 13, and sends the inflated image data to the model generation unit 11. Let this be the input image data. The true defective part means the defect factor 30 included in the defective product test image data P6. The erroneously determined true defective portion refers to a true defective portion when the quality determining unit 12 determines that a defective product including the true defective portion is a non-defective product.

The image division processing section 17 divides the training image data P1 and P2 based on the analysis result of the analysis and teaching section 13, and uses the divided image data as input image data to the model generation section 11.

The masking processing unit 18 performs a masking process on a predetermined area of the training image data P1 and P2, and inputs the masked image data to the model generation unit 11.

The storage device 3 includes an image data storage section 20, a model storage section 21, a defect factor training image data storage section 22, a correct judgment trimming image data storage section 23, and a defect factor integrated image data storage section 24. .

The image data storage section 20 stores training image data P1, P2 and test image data P5, P6 received from the image data acquisition section 10. The image data storage section 20 transmits training image data P1 and P2 to the model generation section 11. Further, the image data storage unit 20 transmits the test image data P5 and P6 to the quality determining unit 12 and the analysis teaching unit 13. The image data storage section 20 also transmits the training image data P1, P2 and the test image data P5, P6 to the relearning training image data generation section 15.

The model storage unit 21 stores the trained model M sent from the model generation unit 11. The model storage unit 21 transmits the learned model M to the quality determining unit 12.

The defect factor training image data storage section 22 receives the defective product training image data P2 from the image data acquisition section 10 and stores it. The defect factor training image data storage section 22 transmits the defective product training image data P2 to the defect factor integrated image data generation section 14.

The correct determination trimming image data storage unit 23 stores correct determination trimming image data (not shown) including a true defective part when the true defective part is correctly determined. The correct determination trimming image data is image data when a truly defective part was correctly determined when image data was determined in the past as a non-defective product or a defective product in an inspection apparatus.

The position, size, and shape of a truly defective part in past inspections are recorded in association with the correct determination trimming image data. In addition, since the correct judgment trimming image data stored in the correct judgment trimming image data storage unit 23 is the one in which the true defective part has been correctly determined, the image size of the correct judgment trimming image data is such that the true defective part is correctly determined. This can be said to be an appropriate image size that can be determined correctly.

The defective factor integrated image data storage section 24 stores the defective factor integrated image data P4 transmitted from the defective factor integrated image data generating section 14. The failure factor integrated image data storage unit 24 transmits the failure factor integrated image data P4 to the analysis teaching unit 13 and the relearning training image data generation unit 15.

2. Overall Operation of Machine Learning Device 1 Next, the overall operation of the machine learning device 1 according to this embodiment will be described with reference to FIG. 2. When the process of the machine learning device 1 of this embodiment is started, in S1, the input device 4 acquires training image data P1, P2, test image data P5, P6, and defect factor training image data P3. The input device 4 transmits training image data P1, P2, test image data P5, P6, and defect factor training image data P3 to the image data acquisition unit 10. The image data acquisition section 10 transmits training image data P1, P2 and test image data P5, P6 to the image data storage section 20. The image data storage unit 20 stores training image data P1, P2 and test image data P5, P6. The image data acquisition unit 10 transmits training image data P1 and P2 to the model generation unit 11. Further, the image data acquisition unit 10 transmits the defective factor training image data P3 to the defective factor training image data storage unit 22. The defective factor training image data storage unit 22 stores defective factor training image data P3.

In S2, the model generation unit 11 generates a learned model M by performing machine learning using the training image data P1 and P2. The model generation unit 11 transmits the learned model M to the model storage unit 21. The model storage unit 21 stores the trained model M.

In S3, the defective factor integrated image data generation unit 14 acquires the defective factor training image data P3 from the defective factor training image data storage unit 22, and generates the defective factor integrated image data P4 from the defective factor training image data P3. The defect factor integrated image data generation section 14 transmits the defect factor integrated image data P4 to the defect factor integrated image data storage section 24. The defect factor integrated image data storage section 24 stores defect factor integrated image data P4.

In S3, the quality determination unit 12 acquires the trained model M from the model storage unit 21.

In S4, the quality determining section 12 obtains test image data P5 and P6 from the image data storage section 20.

In S5, the quality determining unit 12 uses the learned model M to determine whether the test image data P5 and P6 labeled as non-defective or defective are non-defective or defective. The quality determining section 12 transmits the determination result to the analysis teaching section 13.

In S6, the analysis teaching unit 13 acquires the determination result from the quality determining unit 12. The analysis teaching unit 13 acquires defective factor training image data P3 from the defective factor training image data P3. The analysis teaching unit 13 also acquires correct determination trimming image data from the correct determination trimming image data storage unit 23 . Furthermore, the analysis teaching section 13 acquires defective factor integrated image data P4 from the defective factor integrated image data storage section 24.

When a true defective part of a defective product is erroneously determined based on the determination result of the quality determining section 12, the analysis teaching unit 13 analyzes the erroneous determination factor based on the defective factor information and solves the problem according to the erroneous determination factor. derive a strategy.

The analysis teaching unit 13 causes the display device 5 to display the derived solution.

After the user views the solution displayed on the display device 5, the user determines whether or not the solution is adopted. Accordingly, one of the solutions is selected by the user.

In S7, the relearning training image data generation unit 15 acquires training image data P1 and P2 from the image data storage unit 20. Next, the relearning training image data generating section 15 acquires the defective factor training image data P3 from the defective factor training image data storage section 22 or acquires the defective factor training image data P3 from the correct judgment trimming image data storage section depending on the selected solution. The correct determination trimming image data is obtained from 23, or the defective factor integrated image data P4 is obtained from the defective factor integrated image data storage section 24.

The relearning training image data generation unit 15 generates relearning training image data P1 and P2 based on the selected solution.

With the above, the overall operation of the machine learning device 1 is completed.

3. Specific Operations of Machine Learning Device 1 Next, specific operations of the machine learning device 1 according to this embodiment will be described. First, the operations in S1 to S3 above will be explained with reference to FIG. As shown in FIG. 3, one trained model M is generated from a plurality of good product training image data P1 labeled as non-defective products and a plurality of defective product training image data P2 labeled as defective products. Moreover, one defective factor integrated image data P4 is generated from the plurality of defective factor training image data P3.

Next, the operations in S4 to S5 described above will be explained with reference to FIGS. 4 to 6. As shown in FIG. 4, the quality determination unit 12 acquires non-defective product test image data P5 labeled as a non-defective product and defective product test image data P6 labeled as a defective product. The quality determining unit 12 determines whether the non-defective product test image data P5 and the defective product test image data P6 are non-defective products or defective products based on the generated learned model M.

The determination that the quality determining unit 12 makes on the non-defective product test image data P5 and the defective product test image data P6 is as follows.
-Determination A (true negative): It is determined that the non-defective product test image data P5 is a non-defective product. Since this judgment is correct, it is a correct judgment. There is no problem because the non-defective products are correctly determined as non-defective products.
- Judgment B (false positive): The non-defective product test image data P5 is determined to be a defective product based on the false defective part. This judgment is incorrect and is therefore an erroneous judgment. However, from a manufacturing standpoint, this simply reduces yield and does not result in genuine defective products being distributed on the market. Therefore, this is not considered a problem in this embodiment.
- Judgment C (first true positive): Regarding the defective product test image data P6, the true defective part is correctly recognized and it is determined that the product is defective. Since this judgment is correct, it is a correct judgment. There is no problem because the defective product is correctly determined to be defective.
- Determination D (second true positive): As shown in FIG. 5, the defective product test image data P6 is determined to be a defective product based on the false defective region rather than the true defective region. The defective product is determined to be a defective product, and the determination itself is correct, so the determination is correct. However, if a trained model M that has made such a determination is used, there is a possibility that defective products whose true defective parts have been overlooked may be distributed on the market. Therefore, in this embodiment, this determination is treated as a case where a truly defective part is erroneously determined and a countermeasure is required.
- Judgment E (false positive): As shown in FIG. 5, the defective product test image data P6 is determined to be a non-defective product without recognizing a true defective part. This judgment is incorrect and is therefore an erroneous judgment. Further, similar to the above-described determination D, there is a possibility that a defective product with a truly defective part overlooked will be distributed in the market. Therefore, in this embodiment, this determination is treated as a case where a truly defective part is erroneously determined and a countermeasure is required.

However, the example of defective product test image data P6 shown in FIG. 5 is used for convenience of explanation of determination D and determination E, and is not intended to limit the content of the subsequent explanation.

As mentioned above, the biggest concern at the manufacturing site is that the true defective portion of the workpiece W is not recognized. Therefore, as shown in FIG. 4, in this embodiment, the defective product test image data P6 for which the above determinations D and E have been made are extracted as a case in which a true defective part has been erroneously determined. However, like determination B, a configuration may be adopted in which a case where the non-defective product test image data P5 is determined to be a defective product based on a false defective part is extracted.

The quality determination unit 12 creates a class activation map P8 (or heat map, localization map) indicating the frequency at which the parts that serve as the basis for determination are focused on the extracted defective product test image data P6 (FIG. 6). reference). In the class activation map P8, the frequency of attention focused by the quality determining unit 12 is stored for each pixel. In the class activation map P8, the frequency of interest is expressed as tone shading. True defective parts are labeled with respect to the created class activation map P8.

When a true defective part of a defective product is erroneously determined, the analysis teaching unit 13 analyzes the erroneous determination factor that led to the erroneous determination of the true defective part based on the defect factor information, and teaches a solution according to the erroneous determination factor. do. There may be one solution, or there may be two or more solutions. In this embodiment, an example will be described in which three solutions 1 to 3 are taught.

4. Solution 1
Solution 1 will be explained with reference to FIGS. 6 and 7. The analysis teaching unit 13 acquires the class activation map P8 from the quality determining unit 12. In addition, the analysis teaching unit 13 calculates the maximum frequency of interest Nmax in the false judgment test image data P7, which is the test image data P5 and P6 incorrectly judged by the pass/fail judgment unit 12, and the true The frequency of attention N in the defective region is obtained (see FIG. 6).

The degree of interest N will be explained in detail. The analysis teaching unit 13 acquires a true defective part in the false determination test image data P7 based on the false determination test image data P7 and the failure factor information linked to the failure cause training image data P4. Next, the analysis teaching unit 13 acquires the frequency of interest N in the true defective part that is erroneously determined in the erroneously determined test image data P7.

FIG. 6(a) shows a class activation map P8 created from defective product test image data P6 in which a true defective part has been erroneously determined.

The class activation map P8 includes a rectangle labeled as a true defective part (N=5) (hereinafter referred to as the N5 area) and a rectangle labeled as a true defective part (N=10) (hereinafter referred to as the N10 area). and a polygon (hereinafter referred to as N50 region) labeled as a false defective region (Nmax=50).

The above-mentioned N=5 and N=10 indicate that the above-mentioned degree of attention N is 5 and 10, respectively. Further, Nmax=50 indicates that the above-mentioned maximum degree of attention Nmax is 50. However, the numbers N and Nmax are not limited to the above numbers.

In the N5 region, there is a scratch as a defective factor 30. This scratch is a genuine defective part. However, in this embodiment, the quality determining unit 12 did not recognize the scratches in the N5 area. That is, the flaw in the N5 area is incorrectly determined to be a true defective part.

In the N10 region, there is a flaw which is a defective factor 30. This scratch is a genuine defective part. In this embodiment, the quality determining unit 12 recognized the scratches within the N10 area. That is, for the flaw in the N10 area, it is correctly determined that the true defective part is the true defective part.

There are no scratches within the N50 region. In this embodiment, the quality determining unit 12 recognized that there was a flaw in the N50 area and determined that the product was defective. That is, since the flaw in the N50 area is determined to be a defective product based on the false defective part, the true defective part is erroneously determined.

As shown in FIG. 6(b), the analysis teaching unit 13 calculates an insufficient focused frequency ΔN, which is the difference between the maximum focused frequency Nmax=50 and the focused frequencies N=5, 10 in the true defective part. For the N5 region, ΔN is 45, and for the N10 region, ΔN is 40. As a precaution, in the N50 region, ΔN is 0 because it is the difference from Nmax=50.

Based on the calculated ΔN, the analysis and teaching unit 13 analyzes that the insufficient quantity of the defective product training image data P2 among the training image data P1 and P2 is the cause of the erroneous determination of a truly defective part. In detail, based on the calculated ΔN, the analysis and teaching unit 13 determines that, among the training image data P1 and P2, the lack of quantity of the defective product training image data P2 that includes the erroneously determined true defective part is due to the true defective part. Analyze this as the cause of the misjudgment.

As shown in FIG. 6(c), the analysis teaching unit 13 calculates the optimum focus frequency Nbest for the true defective region obtained based on the insufficient focus frequency ΔN. In this embodiment, Nbest in the N5 region is 50, and Nbest in the N10 region is 50. As a precaution, Nbest in the N50 region is 50.

The analysis teaching unit 13 displays at least one of the insufficient attention frequency ΔN and Nbest as information for a solution on the display device 5 and instructs the user.

When solution 1 is selected by the user, the padding processing unit 16 pads the training image data P1 and P2 that include the erroneously determined true defective part. The number of training image data P1 and P2 to be padded may be an arbitrary number determined by the user based on the insufficient focus frequency ΔN, or may be a number taught as Nbest.

As shown in FIG. 7A, the padding processing unit 16 acquires defective product training image data P2 and non-defective product training image data P1 from the image data storage unit 20.

As shown in FIG. 7(b), based on the analysis result of the analysis and teaching unit 13, the padding processing unit 16 calculates the true defective part formed at the position corresponding to the true defective part from the defective product training image data P2. Extract the area containing. The cut out range includes the true defective part and is wider than the true defective part.

As shown in FIG. 7(b), the padding processing unit 16 pastes the cut out area to the non-defective training image data P1. Of the non-defective training image data P1, the position where the cut out area is pasted is not particularly limited. For example, the cut out area may be pasted at or near the position where the true defective part has occurred in the defective product training image data P2 or the defective product test image data P6, or in the defective factor integrated image data P4. It may be pasted at a position where the probability of occurrence of the failure factor 30 is relatively high, or it may be pasted at a position where the frequency of interest N is relatively small in the class activation map P8.

As shown in FIG. 7C, the padding processing unit 16 pads the good quality training image data P1 to which the cut out region has been pasted. Specifically, the truly defective part in the cut out area is rotated at an arbitrary angle, shifted to an arbitrary position, and enlarged or reduced at an arbitrary magnification. As a result, a plurality of new defective product training image data P2 are created. However, in the padding process, the truly defective portion may be deformed.

5. Solution 2
Solution 2 will be explained with reference to FIGS. 8 to 10. As shown in FIG. 8, the analysis and teaching unit 13 acquires the false determination test image data P7 in which a truly defective part has been falsely determined from the image data storage unit 20. Further, the analysis teaching unit 13 acquires defect factor integrated image data P4 from the defect factor integrated image data storage unit 24.

The analysis teaching unit 13 acquires the position of the erroneously determined true defective part from the erroneously determined test image data P7. The analysis and teaching unit 13 specifies a position corresponding to a true defective part that has been erroneously determined in the erroneously determined test image data P7, in the defective factor integrated image data P4.

The analysis and teaching unit 13 cuts out, from the defective factor integrated image data P4, an area including the defective factor group 31, which is generated at a position corresponding to the true defective part incorrectly determined in the false determination test image data P7. In this embodiment, the analysis teaching unit 13 cuts out three types of regions with different sizes. However, the number of regions to be cut out may be one, two, or four or more.

The defective factor group 31 mentioned above refers to a group of a plurality of defective factors 30 arranged consecutively or closely in the defective factor integrated image data P4. In other words, the defective factor group 31 is an accumulation of defective factors 30 included in the plurality of defective factor training image data P3 used to generate the defective product training image data P2.

The analysis teaching unit 13 calculates the ratio of the local image size including the defect factor 30 in the defect factor integrated image data P4 to the total image size of the defect factor integrated image data P4. Note that the above-mentioned local image size is the image size of one defective factor group 31 that includes a position corresponding to a true defective part that is erroneously determined in the defective product test image data P6.

The analysis teaching unit 13 acquires the correct judgment trimming image data in which the defect factor 30 is recorded at the position corresponding to the true defective part incorrectly judged in the incorrect judgment test image data P7 from the correct judgment trimming image data storage unit 23. . The analysis teaching unit 13 calculates, from the correct judgment trimmed image data, the ratio of the local image size of the true defective part in the correct judgment trimmed image data to the total image size of the correct judgment trimmed image data.

The analysis teaching unit 13 calculates the proportion of the local image size of the true defective part in the correct judgment trimmed image data to the total image size of the correct judgment trimmed image data, and the defect factor integrated image with respect to the total image size of the defective factor integrated image data P4. Based on the ratio of the local image size including the failure factor 30 in the data P4, it is determined that the ratio of the local image size of the incorrectly determined true defective part to the total image size of the defective product test image data P6 is inappropriate. , it is analyzed that the true defective part is the cause of the erroneous determination.

In other words, the analysis and teaching unit 13 determines that the ratio of the local image size to the total image size in the defective product test image data P6 is inappropriate based on the correct determination trimmed image data. Analyze it as a determining factor.

The analysis teaching unit 13 determines the size of the total area that allows the pass/fail judgment unit 12 to appropriately recognize the true defective part based on the ratio of the local image size of the true defective part in the correct judgment trimmed image data to the total image size of the correct judgment trimmed image data. Determine image size. Based on the determined total image size, the analysis teaching unit 13 teaches the user the appropriate total image size of the training image data P1 and P2 used for machine learning by displaying it on the display device 5.

When solution 2 is selected by the user, as shown in FIG. do. FIG. 9 shows an example in which the defective product training image data P2 is subjected to image division processing. Note that the non-defective training image data P1 may be subjected to image division processing. The image division processing unit 17 enlarges or reduces the divided defective product training image data P2 to an appropriate total image size. In this embodiment, model α, model β, and model γ are illustrated as examples of the divided defective product training image data P2. However, the regions of the image to be divided are not limited to the above models α to γ.

The image division processing unit 17 pads the image data that has been enlarged or reduced to an appropriate total image size. In FIG. 10, the padding process will be explained using model α as an example.

As shown in FIG. 10, image data enlarged or reduced to an appropriate total image size is cut out from the defective product training image data P2. The image division processing unit 17 rotates the true defective part of the image data at an arbitrary angle, shifts it to an arbitrary position, transforms it into an arbitrary shape, and enlarges or reduces it at an arbitrary magnification. . As a result, a plurality of new defective product training image data P2 are created.

The image division processing unit 17 transmits a set of image data padded based on the model α to the model generation unit 11. The model generation unit 11 generates one learned model M by performing machine learning based on the model α. Similarly, one trained model M is generated based on model β, and one trained model M is generated based on model γ. That is, the model generation unit 11 generates a trained model M corresponding to each model divided by the image division processing unit 17.

6. Solution 3
Solution 3 will be explained with reference to FIGS. 11 and 12. As shown in FIG. 11, the analysis and teaching section 13 acquires defect factor integrated image data P4 from the defect factor integrated image data storage section 24. The analysis teaching unit 13 acquires a failure factor area 32 including a plurality of failure factors 30 in the failure factor integrated image data P4. In FIG. 11, the defect cause area 32 is shown as three squares in the defect cause integrated image data P4.

The defect cause area 32 is designated by the user. The position, size, and shape of the defective cause area 32 are arbitrarily set based on the position, shape, and size of the defective cause 30. In this embodiment, the failure cause area 32 is specified as a quadrilateral, but it may be a polygon such as a triangle or a pentagon, or may be a circle, an ellipse, a track, or any other shape. The number of failure factor areas 32 may be one, two, or four or more.

The analysis and teaching unit 13 acquires the false determination test image data P7 in which the quality determination unit 12 has erroneously determined a truly defective part. The analysis and teaching unit 13 determines, based on the defect factor integrated image data P4 labeled with the defect factor region 32 and the false judgment test image data P7, that the false judgment test image data P7 includes a part that should not be focused on. However, it is analyzed that the true defective part is the cause of the erroneous determination.

The analysis and teaching unit 13 designates an area excluding the defect cause area 32 as a masking area 40 . The analysis teaching unit 13 generates masked image data P9 in which the masking region 40 is subjected to masking processing on the false determination test image data P7. The analysis teaching unit 13 teaches the user the masked image data P9 by displaying it on the display device 5. Thereby, the analysis teaching unit 13 teaches the user the masking area 40 for the training image data P1 and P2 as solution information.

When solution 3 is selected by the user, as shown in FIG. 12, the masking processing section 18 acquires the defect factor integrated image data P4 from the defect factor integrated image data storage section 24. The masking processing unit 18 acquires a defective cause area 32 including a plurality of defective causes 30 in the defective factor integrated image data P4.

The masking processing unit 18 acquires the good product training image data P1 and the defective product training image data P2 from the image data storage unit 20. The masking processing unit 18 performs masking processing on the non-defective product training image data P1 and the defective product training image data P2, excluding the defective cause region 32. Thereby, the masking processing unit 18 generates masked image data P9 in which the masking process is performed on the good product training image data P1 and the defective product training image data P2. The masked image data P9 generated from the non-defective training image data P1 and the masked image data P9 generated from the defective training image data P2 are sent to the model generation unit 11 as new training image data P1 and P2, respectively. input image data.

7. Effects of this embodiment According to this embodiment, the analysis teaching unit 13 teaches a solution according to the cause of the erroneous determination, so the user can learn about the solution for correctly determining the erroneous determination test image data P7. You can know. Thereby, it is possible to reduce the effort required to estimate the cause of the misjudgment regarding the misjudgment test image data P7 and devise countermeasures based on the estimated cause. As a result, the effort required to construct the trained model M can be reduced as a whole.

According to this embodiment, since a plurality of solutions are taught, the user only has to select one of the plurality of solutions taught. Thereby, the effort required to construct the trained model M can be further reduced.

In the defect factor integrated image data P4, the distribution state of the defect factors 30 is visualized in one piece of image data. Thereby, when considering the distribution state of the failure factors 30 when devising a solution for improving the learned model M, the solution can be devised efficiently.

(Embodiment 2)
Next, the machine learning device 1 according to the second embodiment will be described with reference to FIG. 13. This embodiment is substantially the same as Embodiment 1 except that Solution 4 is taught instead of Solution 3, so the same components are given the same reference numerals and redundant explanations will be omitted.

Solution 4 will be explained with reference to FIG. 13. As shown in FIG. 13, the analysis and teaching section 13 acquires defect factor integrated image data P4 from the defect factor integrated image data storage section 24. The analysis teaching unit 13 acquires a failure factor area 32 including a plurality of failure factors 30 in the failure factor integrated image data P4. In FIG. 13, the defect cause area 32 is shown as three squares in the defect cause integrated image data P4.

The analysis teaching unit 13 acquires the false determination test image data P7 in which a true defective part is falsely determined, and the class activation map P8 generated based on the false determination test image data P7. The analysis teaching unit 13 acquires a low frequency of interest region 41 in which the frequency of interest N described above is equal to or less than a predetermined frequency in the class activation map P8 based on the false determination test image data P7. In this embodiment, the predetermined frequency is 10. In the class activation map P8 shown in FIG. 13, the boundary of N=10 is shown by a black curve. A portion whose tone is darker than the boundary of N=10 is a low focused frequency region 41 where the focused frequency N is 10 or less. The predetermined frequency that defines the boundary of the low focused frequency region 41 is not limited to 10, but can be set to any value such as 20 or 30.

The analysis and teaching unit 13 designates an area excluding the defect cause area 32 and the low frequency area 41 as a masking area 40 . The analysis teaching unit 13 generates masked image data P9 in which the masking region 40 is subjected to masking processing on the false determination test image data P7. The analysis teaching unit 13 teaches the user the masked image data P9 by displaying it on the display device 5. Thereby, the analysis teaching unit 13 teaches the user the masking area 40 for the training image data P1 and P2 as solution information.

According to the present embodiment, an area excluding the low frequency area of interest 41 whose frequency of interest is less than or equal to a predetermined frequency is designated as the masking area 40 . Thereby, parts that should not be of interest can be effectively removed from the training data. As a result, the accuracy of the learned model M can be improved.

(Embodiment 3)
Next, the machine learning device 1 of Embodiment 3 will be described with reference to FIG. 14. The overall operation of the machine learning device 1 according to this embodiment will be described with reference to FIG. 14. Since S1 to S5 are the same as in the first embodiment, redundant explanation will be omitted.

In S10, the analysis teaching unit 13 acquires the determination result from the quality determining unit 12. The analysis teaching unit 13 acquires defective factor training image data P3 from the defective factor training image data P3. The analysis teaching unit 13 also acquires correct determination trimming image data from the correct determination trimming image data storage unit 23 . Furthermore, the analysis teaching section 13 acquires defective factor integrated image data P4 from the defective factor integrated image data storage section 24.

When a true defective part of a defective product is erroneously determined based on the determination result of the quality determining section 12, the analysis teaching unit 13 analyzes the erroneous determination factor based on the defective factor information and solves the problem according to the erroneous determination factor. derive a strategy.

The analysis teaching unit 13 transmits the derived solution to the relearning training image data generation unit 15. The relearning training image data generation unit 15 acquires training image data P1 and P2 from the image data storage unit 20. Next, the relearning training image data generating section 15 acquires the defective factor training image data P3 from the defective factor training image data storage section 22 according to the solution sent from the analysis teaching section 13, and performs correct judgment trimming. The correct determination trimming image data is obtained from the image data storage section 23, and the defective factor integrated image data P4 is obtained from the defective factor integrated image data storage section 24.

The relearning training image data generation unit 15 generates relearning training image data P1 and P2 based on the solution sent from the analysis teaching unit 13. The relearning training image data generation unit 15 transmits the generated relearning training image data P1 and P2 to the model generation unit 11.

In S11, the model generation unit 11 performs machine learning using the acquired training image data P1 and P2 for relearning, thereby creating a new trained model that is different from the trained model M generated in S2. generate. The model generation unit 11 transmits the generated new trained model to the quality determination unit 12.

In S12, the quality determining unit 12 acquires a new learned model from the model generating unit 11. Furthermore, the quality determining section 12 acquires test image data P5 and P6 from the image data storage section 20.

The quality determining unit 12 uses the newly learned model to determine whether the test image data P5, P6 acquired in S4 is a non-defective product or a defective product. The quality determining unit 12 transmits the determination result of the new learned model to the analysis teaching unit 13.

In S13, the analysis teaching unit 13 teaches the solution to the user by displaying the solution derived in S10 on the display device 5.

In S14, the analysis teaching unit 13 instructs the user about the effectiveness of the solution derived in S10 by displaying it on the display device 5.

The effectiveness of the solution is not particularly limited. As the effectiveness of the solution, for example, the number of incorrectly determined test image data P7 in which a true defective part was incorrectly determined in the learned model evaluation in S5, and the number of incorrectly determined test image data P7 in which a true defective part was incorrectly determined in the learned model evaluation in S12. By comparing the number of determination test image data P7, a solution with a smaller number of erroneous determination test image data P7 may be more effective.

For example, among the false judgment test image data P7 in which the true defective part was incorrectly judged in the trained model evaluation in S5, the test image data P5 and P6 in which the true defective part was judged to be correct in the trained model evaluation in S12. A solution with a larger number of solutions may be more effective.

With the above, the overall operation of the machine learning device 1 is completed.

The configuration other than the above is substantially the same as in the first embodiment, so the same members are denoted by the same reference numerals and redundant explanations will be omitted.

According to this embodiment, it is possible to know which solution is effective among a plurality of proposed solutions before considering the plurality of solutions. Thereby, the effort required to construct the learned model M can be further reduced.

The present invention is not limited to the above-mentioned embodiments, but can be applied to various embodiments without departing from the gist thereof.

DESCRIPTION OF SYMBOLS 1...Machine learning device, 2...Arithmetic processing unit, 3...Storage device, 11 Model generation section, 12 Acceptability determination section, 13 Analysis teaching section, 14 Failure factor integrated image data generation section, 16... Padded processing section, 17 Image division Processing unit, 20... Image data storage unit, 21... Model storage unit, 22... Failure factor training image data storage unit, 23... Correct judgment trimming image data storage unit, 30... Failure factor, 31... Failure factor group, 32... Failure Factor area, 40... Masking area, 41... Low focused frequency area, N... Focused frequency, Nbest... Optimal focused frequency, Nmax... Maximum focused frequency, M... Learned model, ΔN... Insufficient focused frequency, P1... Good quality training image data (training image data), P2...defective product training image data (training image data), P3...defective factor training image data, P4...defective factor integrated image data, P5...defective product test image data (test image data), P6...defective product training image data Good product test image data (test image data), P7... Erroneous judgment test image data, P8... Class activation map, P9... Masked image data

Claims (18)

A machine learning device comprising an arithmetic processing unit and a storage device,
The storage device is
an image data storage unit that stores training image data for learning and test image data for evaluation;
a model storage unit that stores a learned model generated by performing machine learning using the training image data and the test image data;
A defective factor training image data storage unit that stores defective factor training image data in which defective product training image data labeled as defective among the training image data is linked with defective factor information including the location of the defective factor as an annotation. and,
Equipped with
The arithmetic processing device is
a model generation unit that generates the learned model by performing the machine learning using the training image data and the test image data;
a quality determination unit that uses the trained model to determine whether the test image data labeled as a non-defective product or a defective product is a non-defective product or a defective product;
When a true defective part of a defective product is erroneously determined based on the determination result of the quality determining section, the erroneous determination factor is analyzed based on the defect factor information, and a solution is taught in accordance with the erroneous determination factor. analysis teaching department;
A machine learning device equipped with. What is the case where the true defective part of the defective product is incorrectly determined?
If the quality determination unit erroneously determines a defective product containing a truly defective part as a non-defective product,
If the quality determining unit correctly determines that the defective product is a defective product, but the defective product is determined to be a defective product based on a false defective portion rather than a true defective portion,
The machine learning device according to claim 1, comprising any one of the following. The machine learning device according to claim 1, wherein the analysis teaching unit teaches a plurality of solutions and the effectiveness of the plurality of solutions. The machine learning device according to any one of claims 1 to 3, wherein the analysis and teaching unit analyzes that the cause of the erroneous determination is an insufficient quantity of defective product training image data among the training image data. The machine according to claim 4, wherein the analysis and teaching unit analyzes that the cause of the erroneous determination is an insufficient quantity of the defective product training image data that includes the erroneously determined true defective part among the training image data. learning device. The quality determining unit determines whether the product is a good product or a defective product, and generates a frequency of interest for each pixel,
The analysis teaching section is
Obtaining the maximum frequency of interest Nmax in the erroneously determined test image data that is the test image data that has been erroneously determined by the quality determining unit;
Obtaining the frequency of interest N in the true defective part that has been erroneously determined in the erroneously determined test image data;
Analyzing that the lack of quantity of the defective product training image data is the cause of the misjudgment, based on the insufficient focus frequency ΔN, which is the difference between the maximum focus frequency Nmax and the focus frequency N in the true defective part;
The machine according to claim 4, wherein at least one of the insufficient focus frequency ΔN and the optimal focus frequency Nbest for the true defective part obtained based on the insufficient focus frequency ΔN is taught as information for the solution. learning device. The analysis teaching section is
obtaining the true defective part in the false determination test image data based on the false determination test image data and the defect factor information;
The machine learning device according to claim 6 , wherein the machine learning device acquires the frequency of interest N in the true defect portion that has been incorrectly determined in the incorrectly determined test image data. 5. The arithmetic processing device further includes an inflating processing unit that performs inflating processing on the defective product training image data including the erroneously determined true defective part based on the analysis result of the analysis teaching unit. machine learning device. The analysis teaching unit analyzes that the cause of the erroneous determination is that the ratio of the local image size of the erroneously determined true defective region to the total image size of the test image data is inappropriate. 3. The machine learning device according to any one of 3. The storage device further includes a correct judgment trimming image data storage unit that stores correct judgment trimming image data including the true defective part when the true defective part is correctly judged,
The analysis teaching unit analyzes that the cause of the false determination is that the ratio of the local image size to the total image size in the test image data is inappropriate, based on the correct determination trimmed image data. Machine learning device described. The arithmetic processing device further includes a defective factor integrated image data generation unit that generates defective factor integrated image data by integrating the plurality of defective factor training image data,
The analysis teaching section is
a ratio of the local image size of the true defective part in the correct judgment trimmed image data to the total image size of the correct judgment trimmed image data;
Based on the ratio of the local image size including the defective factor in the defective factor integrated image data to the total image size of the defective factor integrated image data,
The machine learning device according to claim 10, wherein the cause of the misjudgment is analyzed to be that the ratio of the local image size to the total image size in the test image data is inappropriate. In the defective factor integrated image data, a group of a plurality of defective factors arranged consecutively or closely is defined as a defective factor group;
The defect factor integrated image data has a plurality of the defect factor groups,
The local image size including the defective factor in the defective factor integrated image data is the image size of one of the defective factor group including a position corresponding to the true defective part that is erroneously determined in the test image data. The machine learning device according to 11. The machine learning device according to claim 9, wherein the analysis teaching unit teaches an appropriate total image size of the training image data used for the machine learning as information for the solution. The arithmetic processing device further comprises an image division processing unit that divides the training image data based on the analysis result of the analysis teaching unit and uses the divided image data as input image data to the model generation unit. The machine learning device according to item 9. The analysis teaching unit analyzes that the cause of the erroneous determination is that the erroneously determined test image data that is the test image data that has been erroneously determined by the quality determining unit includes a region that should not be focused on. The machine learning device according to any one of 1 to 3. The machine learning device according to claim 15, wherein the analysis teaching unit teaches a masking area for the training image data as the solution information. The arithmetic processing device further includes a defective factor integrated image data generation unit that generates defective factor integrated image data by integrating the plurality of defective factor training image data,
The analysis teaching section is
obtaining a defect cause area including a plurality of defect factors in the defect factor integrated image data;
Designating an area excluding the defect cause area as the masking area,
The machine learning device according to claim 16, wherein the masking area for the training image data is taught as information on the solution. The arithmetic processing device further includes a defective factor integrated image data generation unit that generates defective factor integrated image data by integrating the plurality of defective factor training image data,
The analysis teaching section is
Obtaining a defect cause area including a plurality of defect factors in the defect factor integrated image data;
Obtaining a low frequency of interest region that is less than or equal to a predetermined frequency among the frequencies of interest in the erroneously determined test image data that is the test image data that has been erroneously determined by the quality determining unit;
Designating an area excluding the defect cause area and the low attention frequency area as the masking area,
The machine learning device according to claim 16, wherein the masking region for the training image data is taught as information on the solution.

Images