JP2021174194A - Learning data processing device, learning device, learning data processing method, and program - Google Patents

Abstract

To provide a learning data processing device, a data processing method, a program, and a learning device that efficiently remove a defective image mixed with a plurality of conforming images in a data set of machine learning.SOLUTION: In an information processing device, a learning data processing part 30 has a first image re-constitution part which reconstitutes an image, an abnormality level calculation part, and a data cleansing part. The abnormality level calculation part includes: a first difference image generation part which generates a difference image D321 between a learning image and a first re-constituted image D31 reconstituted by the first image re-constitution part; and a score calculation part which calculates scores based upon the difference image D321. The data cleansing part includes: a threshold calculation part which calculates a threshold D33 indicative of a range of conforming images based upon scores of all learning images; and a defective data removal part which removes learning images not included in the range of the conforming images from a data set D30 based upon the threshold D33.EFFECT: A defective image is efficiently removed which is mixed with a plurality of conforming images in the data set of machine learning.SELECTED DRAWING: Figure 4

Description

The present invention relates to a learning data processing device, a learning device, a learning data processing method, and a program.

In the manufacturing process of foods, pharmaceuticals, industrial products, etc., an inspection process may be provided on the manufacturing line to perform inspections such as detection of defective products. Product inspection is performed visually by humans (visual inspection), and there is a problem that human cost is high. Therefore, in order to automate a part or all of the inspection process, a system for automatically inspecting products by a machine is being developed.

For example, Patent Document 1 discloses a technique for performing defect inspection using machine learning. Specifically, the first learning unit learns a first model for discriminating normal data by using a set of normal data. This first learning unit performs machine learning only with non-defective images. As described above, an inspection device that performs machine learning using only non-defective product images and sorts out non-defective products and defective products is widely known.

JP-A-2018-120300

However, a large number of good images of thousands to tens of thousands may be required for a data set for machine learning. When a large number of images are prepared in this way, defective product data may be unintentionally mixed in the data set. In such a case, there arises a problem that the accuracy of the generated trained model becomes low.

An object of the present invention is to provide a technique for efficiently removing defective images mixed in a plurality of non-defective images in a machine learning data set.

In order to solve the above problems, the first invention of the present application is a learning data processing device that removes an inappropriate image from a plurality of learning images included in a learning data set, and each of the learning images. It has an abnormality degree calculation unit for calculating a score indicating an abnormality degree, and a data cleansing unit for removing an inappropriate learning image from the data set based on the score.

The second invention of the present application is the learning data processing apparatus of the first invention, further including a first image reconstructing unit for outputting a reconstructed image of the input image, and the abnormality degree calculation unit is described above. A first difference image generation unit that generates a difference image between the learning image input to the first image reconstruction unit and the reconstruction image output from the first image reconstruction unit, and the first difference image. The data cleansing unit includes a score calculation unit that calculates the score based on the difference image generated by the generation unit, and the data cleansing unit uses the score calculated by the abnormality degree calculation unit for all the learning images. Based on the threshold calculation unit that calculates the threshold indicating the range of the non-defective product, and the learning image that is not included in the range of the non-defective product is removed from the data set based on the threshold calculated by the threshold calculation unit. Includes a non-defective data removal unit.

The third invention of the present application is the learning data processing apparatus of the second invention, in which a first learning model that inputs and outputs images and a plurality of sample images are input to the first learning model and input. The first image reconstructing unit further has a first trained data trained to output a reconstructed image of the sample image, and the first image reconstructing unit applies the input image to the first trained data. Input to the first training model to generate a reconstructed image of the input image.

The fourth invention of the present application is the learning data processing apparatus of the third invention, and the first trained data is trained by inputting a plurality of learning images included in the data set as the sample images. It is trained data.

The fifth invention of the present application is the learning data processing apparatus of the third invention or the fourth invention, and the first learning unit which generates the first trained data by performing unsupervised learning on the first learning model. Have.

The sixth invention of the present application is the learning data processing apparatus of the first invention, which is a first learning model in which an image is input and a score which is a feature amount is output, and a plurality of sample images are used as the first learning model. The first trained data is further provided with the first trained data that is input to and trained to output the result of detecting the outlier of the input sample image as the score, and the abnormality degree calculation unit is the input image. Is input to the first training model to which the first trained data is applied, and the score is acquired.

The seventh invention of the present application is a learning device for constructing an inspection device for inspecting an object, and is a first invention for removing an inappropriate image from a plurality of learning images included in a learning data set. It also has a learning data processing device according to any one of the sixth inventions, and a second learning unit that performs learning using a processed data set from which inappropriate images have been removed by the learning data processing device. The second learning unit inputs a plurality of the learning images included in the processed data set into a second learning model that inputs and outputs images, and the second learning model is input to the learning model. Image reconstruction Generates second trained data trained to output an image.

The eighth invention of the present application is a training data processing method for removing an inappropriate image from a plurality of training images included in a training data set, and a) indicates an abnormality degree for each of the training images. It has an abnormality degree calculation step of calculating a score, and b) a data cleansing step of removing an inappropriate learning image from the data set based on the score.

The ninth invention of the present application is the learning data processing method of the eighth invention, and the step a) is a1) an image reconstruction step of reconstructing the learning image, and a2) the step a1). Based on the first difference image generation step of generating the difference image between the input learning image and the reconstructed image output in the step a1) and the difference image generated in a3) step a2). Including the score calculation step of calculating the score, the step b) is a threshold indicating the range of non-defective products based on the score calculated in the step a) for b1) all the learning images. A threshold calculation step for calculating the above, and a defective product data removal step for removing the learning image not included in the range of good products from the data set based on the threshold calculated in b2) step b1). including.

The tenth invention of the present application is a program for causing a computer to perform training data processing for removing an inappropriate image from a plurality of training images included in a training data set, and the computer is used to perform a. ) An abnormality degree calculation step of calculating a score indicating an abnormality degree for each of the training images, and b) a data cleansing step of removing an inappropriate learning image from the data set based on the score. Let it run.

The eleventh invention of the present application is the program of the tenth invention, and the step a) is an image reconstruction step of a1) reconstructing the learning image and a2) the input in the step a1). The score is based on the first difference image generation step of generating a difference image between the learning image and the reconstructed image output in the step a1) and the difference image generated in a3) step a2). Including the score calculation step of calculating the above, the step b) is a threshold for calculating a threshold indicating the range of non-defective products based on the score calculated in the step a) for b1) all the learning images. It includes a calculation step and a defective product data removal step of removing the learning image not included in the range of non-defective products from the data set based on the threshold value calculated in b2) step b1).

According to the first to eleventh inventions of the present application, it is possible to efficiently remove defective images mixed in a plurality of non-defective images in a machine learning data set.

It is a figure which shows the inspection apparatus of 1st Embodiment. It is a figure which shows the hardware configuration of the information processing apparatus of 1st Embodiment. It is a figure which shows the functional structure which the information processing apparatus of 1st Embodiment has. It is a figure which shows a part of the functional structure which the information processing apparatus of 1st Embodiment has. It is a flowchart which shows the flow of the 1st learning process of 1st Embodiment. It is a figure which conceptually shows the state that the good product image was input to the trained 1st learning model of 1st Embodiment. It is a figure which conceptually shows the state that the defective product image was input to the trained 1st learning model of 1st Embodiment. It is a flowchart which shows the flow of the data cleansing process in the defect detection part of 1st Embodiment. It is a figure which showed the example of the image obtained at each stage of the data cleansing process of 1st Embodiment. It is a figure which showed the example of the image obtained at each stage of the data cleansing process of 1st Embodiment. It is an example of the figure which showed the difference image about the object of a good product and a defective product in the data cleansing process of 1st Embodiment. It is a figure which shows the statistical data of the score in the data cleansing process of 1st Embodiment. It is a flowchart which shows the flow of the 2nd learning process of 1st Embodiment. It is a flowchart which shows the flow of the defect detection process of 1st Embodiment. It is a figure which shows the functional structure which the information processing apparatus of 2nd Embodiment has.

Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. It should be noted that the components described in this embodiment are merely examples, and the scope of the present invention is not limited to them. In the drawings, the dimensions and numbers of each part may be exaggerated or simplified as necessary for easy understanding.

<1. First Embodiment>
<1-1. Inspection device configuration>
FIG. 1 is a diagram showing an inspection device 10 of the first embodiment. The inspection device 10 detects defects in the object 90 by analyzing the image of the object 90. The object 90 is specifically a tablet, but is not limited to a tablet. The inspection device 10 includes a camera 110 and an information processing device 120. The camera 110 is electrically connected to the information processing device 120. The camera 110 includes an image sensor. The camera 110 outputs an image signal obtained by imaging the object 90 using the image sensor to the information processing device 120. The object 90 imaged by the camera 110 may be stopped at a predetermined position, or may be moved in a predetermined direction by a transport mechanism such as a belt conveyor.

FIG. 2 is a diagram showing a hardware configuration of the information processing device 120 of the first embodiment. The information processing device 120 has a configuration as a computer. Specifically, the information processing device 120 includes a processor 121, a RAM 123, a storage unit 125, an input unit 127, a display unit 129, an apparatus I / F 131, and a communication I / F 133. The processor 121, the RAM 123, the storage unit 125, the input unit 127, the display unit 129, the device I / F 131, and the communication I / F 133 are electrically connected to each other via the bus 135.

Specifically, the processor 121 includes a CPU or a GPU. The RAM 123 is a storage medium capable of reading and writing information, and specifically, an SDRAM. The storage unit 125 is a recording medium capable of reading and writing information, and specifically includes an HDD (hard disk drive) or an SSD (solid state drive). The storage unit 125 may include a portable optical disk, magnetic disk, semiconductor memory, or the like. The storage unit 125 stores the program P. The processor 121 realizes various functions by executing the program P with the RAM 123 as a work area. The program P may be provided or distributed to the information processing apparatus 120 via the network.

The input unit 127 is an input device that accepts user's operation input, and specifically, a mouse, a keyboard, or the like. The display unit 129 is a display device that displays images representing various types of information, and is specifically a liquid crystal display.

The device I / F 131 is an interface for electrically connecting the camera 110 to the information processing device 120. The communication I / F 133 is an interface for connecting the information processing device 120 to a network such as the Internet. The camera 110 may be connected to the information processing device 120 via the communication I / F 133. That is, the inspection device 10 is not essential to include the camera 110, and may include only the information processing device 120.

FIG. 3 is a diagram showing a functional configuration included in the information processing device 120 of the first embodiment. FIG. 4 is a diagram showing a part of the functional configuration included in the information processing apparatus 120 in more detail.

As shown in FIG. 3, the information processing apparatus 120 includes a first learning unit 20, a learning data processing unit 30, a second learning unit 40, and a defect detection unit 50.

The first learning unit 20, the learning data processing unit 30, and the second learning unit 40 are configured to construct a defect detection unit 50 that constitutes an inspection device. In order to accurately learn the second learning model M2, which will be described later, of the defect detection unit 50, a large number of non-defective images of objects (hereinafter referred to as “non-defective images”) are required. However, when preparing a data set for learning including images of a large number of objects, a defective image (hereinafter referred to as "defective image") may be mixed. Therefore, the information processing device 120 uses a learning data set in which defective image images may be mixed, and uses an inappropriate defective product image from a plurality of learning images included in the learning data set. It has a learning data processing unit 30 for removing the above.

In the present embodiment, the defect detection unit 50, which is a part of the inspection device, the first learning unit 20, the learning data processing unit 30, and the second learning unit 40 for constructing the defect detection unit 50 are the same information processing device. Although included in 120, the present invention is not limited to this. The first learning unit 20, the learning data processing unit 30, and the second learning unit 40 are not necessarily provided in the inspection device 10, and may be provided in another computer.

The first learning unit 20 learns the first learning model M1 described later in the learning data processing unit 30. Specifically, the first learning unit 20 performs unsupervised learning on the first learning model M1.

The learning data processing unit 30 is an example of a learning data processing device that removes inappropriate images from a plurality of learning images included in a learning data set. The learning data processing unit 30 processes the learning data set D30 used when the second learning unit 40 learns the second learning model M2 described later in the defect detection unit 50.

The learning data processing unit 30 includes a first learning model M1, a first learned data D1, a first image reconstruction unit 31, an abnormality degree calculation unit 32, and a data cleansing unit 33. The first learning model M1 and the first learned data D1 are stored in the storage unit 125. The first image reconstruction unit 31, the abnormality degree calculation unit 32, and the data cleansing unit 33 are functions realized by operating the processor 121 according to the program P.

The first learning model M1 is a learning model that inputs an image and outputs an image. The first learning model M1 is, for example, an autoencoder using a neural network or a variational autoencoder (Variational Autoencoder).

The first learned data D1 is learned data obtained by the first learning unit 20 learning the first learning model M1. The first learning unit 20 generates the first learned data D1 by machine learning and stores it in the storage unit 125.

Specifically, the first trained data D1 inputs a plurality of sample images D20, which are images of objects for training, into the first learning model M1 and outputs a reconstructed image of the input sample image D20. It is the trained data trained in this way.

The plurality of sample images D20 are images obtained by capturing a plurality of objects 90 with the camera 110. At this time, it is preferable that the plurality of sample images D20 are all non-defective images. However, from the viewpoint of realistic preparation, a plurality of sample images D20 used by the first learning unit 20 for learning the first learning model M1 and a data set D30 to be subjected to data cleansing processing by the learning data processing unit 30. The same image is used as the plurality of learning images included in. That is, the first trained data D1 is trained data obtained by inputting a plurality of training images included in the data set D30 as sample images D20 and training them.

The first learning model M1 (hereinafter referred to as "learned first learning model L1") to which the first trained data D1 is applied receives the sample image D20 as an input and outputs a reconstructed image of the input sample image. Works on. Since most of the plurality of sample images D20 used for learning are non-defective images, when a non-defective image is input to the trained first learning model L1, the reproducibility of the output reconstructed image is high. On the other hand, when an image of a defective product is input to the trained first learning model L1, the reproducibility of the output reconstructed image becomes low.

The first image reconstruction unit 31 outputs a reconstruction image of the input image. Specifically, the first image reconstruction unit 31 inputs a data set D30 including a plurality of learning images into the trained first learning model L1, and a plurality of learning images included in the data set D30. The process of generating the first reconstructed image D31 including each of the reconstructed images of the above is executed. The first image reconstruction unit 31 delivers the first reconstruction image D31 output from the trained first learning model L1 to the abnormality degree calculation unit 32.

The abnormality degree calculation unit 32 calculates a score indicating the abnormality degree for each of the learning images included in the data set D30. As shown in FIG. 4, the abnormality degree calculation unit 32 includes a first difference image generation unit 321 and a score calculation unit 322.

The first difference image generation unit 321 includes each of the plurality of learning images included in the data set D30 input to the first image reconstruction unit 31 and the first reconstruction output from the first image reconstruction unit 31. Differences are taken from each of the images D31 to generate a difference image D321.

The score calculation unit 322 calculates a score indicating the degree of abnormality for each of the learning images included in the data set D30 based on the difference image D321. The score is, for example, the total brightness of all the pixels of the difference image D321. Then, the score calculation unit 322 delivers the score data D32 including all the scores for each learning image of the data set D30 to the data cleansing unit 33.

The data cleansing unit 33 removes an inappropriate learning image from the data set D30 based on the score calculated by the abnormality degree calculation unit 32. The data cleansing unit 33 has a threshold value calculation unit 331 and a defective product data removal unit 332.

The threshold value calculation unit 331 calculates the threshold value D33 indicating the range of non-defective products based on all the scores included in the score data D32. In the present embodiment, the threshold value calculation unit 331 statistically processes the score included in the score data D32, and calculates the threshold value D33 based on the result. The defective product data removing unit 332 generates a processed data set D40 from which the learning image not included in the range of the non-defective product is removed from the data set D30 based on the threshold value D33.

As shown in FIG. 3, the second learning unit 40 learns the second learning model M2, which will be described later, of the defect detection unit 50. Specifically, the second learning model M2 of the defect detection unit 50 is trained using the processed data set D40 from which the inappropriate learning image has been removed by the learning data processing unit 30.

The defect detection unit 50 detects defects in the object 90 based on the object image obtained by photographing the object 90 with the camera 110. The defect detection unit 50 includes a second learning model M2, a second learned data D2, a second image reconstruction unit 51, a second difference image generation unit 52, and a defect detection processing unit 53. The second learning model M2 and the second learned data D2 are stored in the storage unit 125. The second image reconstruction unit 51, the second difference image generation unit 52, and the defect detection processing unit 53 are functions realized by operating the processor 121 according to the program P.

The second learning model M2 is a learning model that inputs an image and outputs an image. The second learning model M2 is, for example, an autoencoder using a neural network or a variational autoencoder (Variational Autoencoder). Exactly the same learning model may be used for the first learning model M1 and the second learning model M2, or different learning models may be used. For example, the same autoencoder may be used for the first learning model M1 and the second learning model M2, or autoencoders having different numbers of layers and input / output parameters may be used.

The second learned data D2 is learned data obtained by the second learning unit 40 learning the second learning model M2. The second learning unit 40 generates the second learned data D2 by machine learning and stores it in the storage unit 125.

Specifically, the second trained data D2 inputs the processed data set D40 into the second learning model M2, and outputs a reconstructed image of the training image included in the input processed data set D40. It is the trained data trained by.

The second learning model M2 (hereinafter referred to as “learned second learning model L2”) to which the second trained data D2 is applied inputs a learning image included in the processed data set D40, and is used for input learning. Image reconstruction Operates to output an image. The learning image included in the processed data set D40 used for learning has a very high probability of being a good image due to the data cleansing process of the learning data processing unit 30. Therefore, the trained second learning model L2 outputs an image close to a non-defective product as a reconstructed image regardless of whether the input image is a non-defective product image or the input image is a defective product image.

The second image reconstruction unit 51 outputs a reconstruction image of the input image. Specifically, the object image D50 obtained by capturing the object 90 to be inspected for defects with the camera 110 is input to the second image reconstruction unit 51. Then, the second image reconstruction unit 51 inputs the object image D50 into the trained second learning model L2, and the second difference between the second reconstructed image D51 reconstructed by the trained second learning model L2. It is handed over to the image generation unit 52.

The second difference image generation unit 52 takes a difference between the object image D50 input to the second image reconstruction unit 51 and the second reconstruction image D51 output from the second image reconstruction unit 51, and obtains a second difference. 2 The difference image D52 is generated.

The defect detection processing unit 53 detects the presence or absence of defects based on the second difference image D52, and outputs the presence or absence of defects as the inspection result D60.

The processor 121 may display the inspection result D60 output by the defect detection processing unit 53 on the display unit 129.

<1-2. 1st learning process>
The first learning process S10 performed by the first learning unit 20 will be described with reference to FIG. FIG. 5 is a flowchart showing the flow of the first learning process S10 of the present embodiment. In the first learning process S10, the first learning model M1 is used to generate the first learned data D1 which is a parameter or the like for outputting the reconstructed image of the input image.

As shown in FIG. 5, the information processing apparatus 120 first performs a preparatory step of preparing a plurality of sample images D20 (step S11). Specifically, the processor 121 acquires a plurality of sample images D20 by capturing a plurality of objects 90 with the camera 110. Then, the processor 121 stores the acquired plurality of sample images D20 in the storage unit 125.

The plurality of sample images D20 are also used as the data set D30 before processing. At this time, most of the plurality of objects 90 to be prepared are non-defective products, but in rare cases, defective products may be mixed. Therefore, there is a possibility that defective products are mixed in a very small part of the plurality of sample images D20.

When the preparation step S11 is completed, the first learning unit 20 inputs a part (for example, 100 images) of the plurality of sample images D20 acquired in the preparation step S11 into the first learning model M1 as a small data set. The first learning step of generating the first learned data D1 is executed (step S12). The term "generate trained data" includes updating the trained data that has already been generated.

Specifically, in the first learning process S12, the first learning unit 20 inputs a small data set into the trained first learning model L1 (the first learning model M1 in the first first learning process S12). , Generate a reconstructed image. Then, an evaluation index of how close the input image and the reconstructed image are is calculated. For the evaluation index, for example, the sum of squares of the difference (error) between the input sample image and the output reconstructed image is used. Then, based on the calculated evaluation index, the first trained data D1 including the weighting parameters of the neural network constituting the first learning model M1 is updated by backpropagation.

Subsequently, the first learning unit 20 determines whether or not the learning of the first learning model M1 is completed (step S13). Specifically, it is determined whether or not the evaluation index calculated in the first learning step S12 immediately before has converged within a predetermined range. That is, it is determined whether or not the first learning model M1 to which the first learned data D1 generated in the immediately preceding first learning step S12 is applied generates a reconstructed image approximated to the sample image with desired accuracy. do. Specifically, for example, when the evaluation index calculated in the first learning step S12 is smaller than the predetermined threshold value, it is determined that the learning of the first learning model M1 is completed. In step S13, even when the learning of all the sample images D20 is completed a predetermined number of times, it may be determined that the learning is completed.

If it is determined in step S13 that the learning of the first learning model M1 has not been completed (step S13: No), the processor 121 returns to the first learning step S12 and learns about the next small data set.

When it is determined in step S13 that the learning of the first learning model M1 is completed (step S13: Yes), the processor 121 stores the first learned data D1 generated by the first learning unit 20 in the storage unit 125. Is executed (step S14).

FIG. 6 is a diagram conceptually showing a state in which a non-defective image G11 is input to the first learning model M1 to which the first learned data D1 of the present embodiment is applied, that is, the trained first learning model L1. be. FIG. 7 is a diagram conceptually showing a state in which the defective product image G12 is input to the trained first learning model L1.

As shown in FIGS. 6 and 7, the first learning model M1 is specifically a neural network having an encoder and a decoder. The encoder finds the latent variable by dimensionally compressing the input image. The decoder reproduces the original input image from the latent variables. The number of elements of the input layer and the output layer of the first learning model M1 shown in FIG. 6 and the number of hidden layers are merely examples, and are not limited thereto.

As shown in FIG. 6, when a non-defective image G11 in which a defect-free object 90 is imaged is input to the trained first learning model L1, the reconstructed image G21 output from the trained first learning model L1 is compared. High reproducibility. That is, the reconstructed image G21 has a small difference from the input non-defective image G11. On the other hand, as shown in FIG. 7, when the defective image G12 obtained by imaging the object 90 having the defective portion 91 is input to the trained first learning model L1, it is output from the trained first learning model L1. The reconstructed image G22 does not sufficiently reproduce the defective portion 91 of the defective product. That is, the reconstructed image G22 has a large difference from the input defective image G12. The defective portion 91 is, for example, dirt, foreign matter, chips, etc. attached to the object 90.

<1-3. Data cleansing process>
Next, the data cleansing process S20 performed by the learning data processing unit 30 will be described with reference to FIG. FIG. 8 is a flowchart showing the flow of the data cleansing process S20 of the present embodiment. The data cleansing process S20 is a data process for removing an inappropriate image from a plurality of learning images included in the learning data set D30 by using the trained first learning model L1.

As shown in FIG. 8, the information processing apparatus 120 first performs a preparatory step of preparing the data set D30 before processing (step S21). Specifically, in the present embodiment, since the plurality of learning images included in the data set D30 before processing are the same as the plurality of sample images D20 used in the first learning processing S10, the processor 121 A plurality of sample images D20 are read from the storage unit 125 as the data set D30 before processing.

Next, the first image reconstruction unit 31 generates the first reconstruction image D31 using the trained first learning model L1 from each learning image of the data set D30 obtained in the preparation step S21. An image reconstruction step is performed (step S22). Specifically, the first image reconstruction unit 31 inputs the learning image included in the data set D30 into the trained first learning model L1 and acquires the output of the first reconstruction image D31. Then, the processor 121 stores the acquired first reconstructed image D31 in the storage unit 125.

Subsequently, the first difference image generation unit 321 includes the learning image of the data set D30 stored in the storage unit 125 and the first reconstruction image D31 generated from the learning image in the first image reconstruction step S22. The first difference image generation step of generating the first difference image D321 is performed from the above (step S23). Specifically, the first difference image D321 is generated by taking the difference between the brightness of each pixel of the learning image and the brightness of each pixel of the first reconstructed image D31. Then, the processor 121 stores the acquired first difference image D321 in the storage unit 125.

Here, FIGS. 9 and 10 are diagrams showing examples of images obtained at each stage of the data cleansing process according to the present embodiment. The learning image G31 in FIG. 9 is an image when the object 90 is a non-defective product, whereas the learning image G32 in FIG. 10 is a defective product when the object 90 has a defective portion 91. It is an image of. FIG. 11 is an enlarged view of the first difference images G51 and G52 shown in FIGS. 9 and 10.

When the object 90 is a non-defective product, as shown in FIG. 9, the learning image G31 input in the first image reconstruction step S22 and the first reconstruction image G41 generated in the first image reconstruction step S22. Is an approximate image. That is, in this case, the brightness of each pixel of the first reconstructed image G41 is close to the brightness of each pixel of the learning image G31. Therefore, as shown in FIG. 9, the first difference image G51 is an image in which the brightness of the approximated image is canceled out and the brightness of each pixel is small.

On the other hand, when the object 90 is a defective product, as shown in FIG. 10, the defective portion 91 appears in the learning image G32 input in the first image reconstruction step S22. In such a case, as described above, the first reconstructed image G42 generated from the learning image G32 including the defective portion 91 becomes an image in which the defective portion 91 is removed from the learning image G32.

Therefore, in the normal portion of the object 90 in the first difference image G52, the brightness of the image approximated to the learning image G32 is canceled out, and the brightness of each pixel is small. On the other hand, in the defect portion 91 of the first difference image G42, since the difference in brightness between the learning image G32 and the first reconstructed image G42 is large, the brightness of the portion is larger than that of the normal portion. Therefore, as shown in FIG. 10, the defective portion 91 is displayed whitish in the first difference image G52. By obtaining the first difference image G52 in this way, the defective portion 91 can be specifically extracted.

After that, for each of the learning images included in the data set D30, the score calculation unit 322 performs a score calculation step of calculating a score indicating the degree of abnormality (step S24). In the present embodiment, the score is the total brightness of all the pixels of the difference image D321. Then, the score calculation unit 322 delivers the score data D32 including all the scores for each learning image of the data set D30 to the threshold value calculation unit 331 of the data cleansing unit 33.

The threshold value calculation unit 331 performs a threshold value calculation step of calculating a threshold value D33 indicating a range of non-defective products based on all the scores included in the score data D32 (step S25). The threshold value calculation unit 331 of the present embodiment statistically processes all the scores included in the score data D32, and calculates the threshold value D33 based on the processing result.

FIG. 12 is a diagram showing a histogram of scores included in an example of score data D32. In FIG. 12, the horizontal axis is the score value, and the vertical axis is the number of data in each score.

In the present embodiment, as described above, in the difference image D321, the brightness of the defective portion 91 of the object 90 is increased. Therefore, the score in the difference image D321 of the non-defective product has a substantially normal distribution, and the score in the difference image D321 of the defective product takes a larger value than the score of the non-defective product. Therefore, in the frequency distribution of all the scores included in the score data D32, as shown in FIG. 12, a frequency distribution having a longer right tail (the one with the larger score) of the normal distribution can be obtained.

For such score data D32, the threshold value calculation unit 331 performs statistical processing for obtaining values such as the average value S of the scores and the standard deviation σ in the threshold value calculation step S25. Then, the threshold value calculation unit 331 calculates the threshold value D33 indicating the range of non-defective products. The threshold value D33 includes a first threshold value Th1 indicating a lower limit value of a non-defective product and a second threshold value Th2 indicating an upper limit value of a score of a non-defective product.

In the present embodiment, the first threshold value Th1 and the second threshold value Th2 are calculated as follows with reference to the average value S of the scores and the standard deviation σ.
Th1 = S-3 * σ
Th2 = S + 3 * σ

The threshold value calculation unit 331 of the present embodiment calculates the threshold value D33 by performing statistical processing, but the present invention is not limited to this. The threshold value calculation unit 331 may select the difference images D321 for the number of predetermined numbers in descending order of the score, and set the threshold value D33 so that these difference images D321 are excluded. Further, the threshold value calculation unit 331 may not provide the first threshold value Th1 which is the lower limit value of the score of the non-defective product, and may calculate the second threshold value Th2 which is the upper limit value of the score of the non-defective product as a constant multiple of the average value S of the score. The calculation method of the threshold value D33 of the threshold value calculation unit 331 is not limited to these, and may be appropriately selected according to conditions such as the type and purpose of the object 90.

Finally, the defective product data removing unit 332 performs a defective product data removing step of removing the learning image not included in the range of the non-defective product from the data set D30 based on the threshold value D33 (step S26). Specifically, the defective product data removing unit 332 corresponds to the learning image corresponding to the difference image D321 having a score smaller than the first threshold Th1 and the difference image D321 having a score larger than the second threshold Th2. The training image is removed from the data set D30 to generate the processed data set D40. Then, the processor 121 stores the generated processed data set D40 in the storage unit 125.

The data cleansing process S20 generates a processed data set D40 in which defective images are removed from the data set D30 before processing. By learning the second learning model M2 used for defect detection using this processed data set D40, the accuracy of machine learning of the second learning model M2 can be improved.

<1-4. Second learning process>
The second learning process S30 performed by the second learning unit 40 will be described with reference to FIG. FIG. 13 is a flowchart showing the flow of the second learning process S30 of the present embodiment. In the second learning process S30, the second learning model M2 is used to generate the second learned data D2 which is a parameter or the like for outputting the reconstructed image of the input image.

As shown in FIG. 13, the information processing apparatus 120 first performs a preparatory step of preparing the processed data set D40 (step S31). Specifically, the processor 121 reads out the processed data set D40 stored in the storage unit 125.

When the preparation step S31 is completed, the second learning unit 40 uses a part of a plurality of learning images (for example, 100 images) included in the processed data set D40 acquired in the preparation step S31 as a small data set. The second learning step of inputting to the second learning model M2 to generate the second trained data D2 is executed (step S32). The second learning step S32 of the present embodiment is performed in the same manner as the first learning step S12 of the first learning process S10.

Subsequently, the second learning unit 40 determines whether or not the learning of the second learning model M2 is completed (step S33). Step S33 of this embodiment is performed in the same manner as step S13 of the first learning process S10.

If it is determined in step S33 that the learning of the second learning model M2 has not been completed (step S33: No), the processor 121 returns to the second learning step S32 and learns about the next small data set.

When it is determined in step S33 that the learning of the second learning model M2 is completed (step S33: Yes), the processor 121 stores the second learned data D2 generated by the second learning unit 40 in the storage unit 125. Is executed (step S34).

In this way, by performing the second learning process S30 in the same procedure as the first learning process S10, the trained second learning model L2 is based on the input image as in the trained first learning model L1. It can be trained as a learning model that outputs a reconstructed image that does not sufficiently reproduce the defective part. At this time, in the second learning process S30, machine learning can be performed using the processed data set D40 having a lower mixing rate of defective products than the pre-processed data set S30 used for learning in the first learning process S10. can. As a result, the trained second learning model L2 can perform learning more accurately.

<1-5. Defect detection process>
Finally, the defect detection process S40 using the trained second learning model L2 in the defect detection unit 50 will be described with reference to FIG. In this defect detection process S40, defects in the object 90 are detected based on the object image D50 obtained by imaging the object 90. FIG. 14 is a flowchart showing the flow of the defect detection process in the defect detection unit 50 of the present embodiment.

As shown in FIG. 14, the information processing apparatus 120 first takes an image of the object 90 to be inspected and performs an imaging step of acquiring the object image D50 (step S41). Specifically, the processor 121 acquires the object image D50 by photographing the object to be inspected by using the camera 110. Then, the processor 121 stores the acquired object image D50 in the storage unit 125.

Next, the second image reconstruction unit 51 generates a second reconstruction image D51 from the object image D50 obtained in the imaging step S41 using the trained second learning model L2, a second image reconstruction step. Is executed (step S42). Specifically, the second image reconstruction unit 51 inputs the object image D50 into the second learning model M2 to which the second trained data D2 is applied, and acquires the output of the second reconstruction image D51. do. Then, the processor 121 stores the acquired second reconstructed image D51 in the storage unit 125.

Subsequently, the second difference image generation unit 52 executes a second difference image generation step of generating the second difference image D52 from the object image D50 and the second reconstructed image D51 stored in the storage unit 125 (. Step S43). Specifically, the second difference image generation unit 52 obtains the second difference image D52 by taking the difference between the brightness of each pixel of the object image D50 and the brightness of each pixel of the second reconstructed image D51. Generate. Then, the processor 121 stores the generated second difference image D52 in the storage unit 125.

Finally, the defect detection processing unit 53 executes a defect detection step of determining whether or not the object 90 has a defect based on the second difference image D52 (step S44). As described above, in the second difference image D52, the defective portion 91 of the object 90 is a region having high brightness. Using this feature, the defect detection processing unit 53 determines whether or not the object 90 has a defect. The specific judgment method can be appropriately selected. Then, the defect detection processing unit 53 outputs the presence or absence of defect detection as the inspection result D60.

<2. Second Embodiment>
FIG. 15 is a diagram showing a functional configuration included in the information processing apparatus 120A of the second embodiment. The information processing device 120A shown in FIG. 15 has a first learning unit 20A and a learning data processing unit 30A having a configuration different from that of the first embodiment, and a second learning unit 40 and a defect having the same configuration as the first embodiment. It has a detection unit 50. In FIG. 15, the same reference numerals are given to the same configurations as those in the first embodiment.

The first learning unit 20A learns the first learning model M1A, which will be described later, of the learning data processing unit 30A. Specifically, the first learning unit 20A performs unsupervised learning on the first learning model M1A.

The learning data processing unit 30A is an example of a learning data processing device that removes an inappropriate image from a plurality of learning images included in a learning data set. The learning data processing unit 30A includes a first learning model M1A, a first learned data D1A, an abnormality degree calculation unit 32A, and a data cleansing unit 33A.

The first learning model M1A is a One Class SVM (One Class Support Vector Machine). One Class SVM can detect outliers by performing classification. The first learning model M1A may be another machine learning model that performs unsupervised learning, such as a One Class neural network.

The first learned data D1A is learned data obtained by the first learning unit 20A learning the first learning model M1A. The first learning unit 20A generates the first learned data D1A by machine learning.

Specifically, the first trained data D1A appropriately detects outliers by inputting a plurality of sample images D20, which are images of objects for training, into the first learning model M1A and calculating the feature amount. It is the trained data trained to perform.

The first learning model M1A (hereinafter referred to as "trained first learning model L1A") to which the first trained data D1A is applied receives the sample image D20 as an input, and whether or not the input sample image is an outlier. Is output as a score.

The abnormality degree calculation unit 32A inputs each of the learning images included in the data set D30 into the trained first learning model L1A, and acquires a score for each learning image. Then, the abnormality degree calculation unit 32A delivers the score data D32A including all the scores for each learning image of the data set D30 to the data cleansing unit 33A.

The data cleansing unit 33A removes an inappropriate learning image from the data set D30 based on the score calculated by the abnormality degree calculation unit 32A. That is, the data cleansing unit 33A outputs the processed data set D40A obtained by removing the learning image determined in advance by the trained first learning model L1A as an outlier from the data set D30.

<3. Modification example>
Although the embodiments have been described above, the present invention is not limited to the above, and various modifications and combinations are possible.

Although the present invention has been described in detail, the above description is exemplary in all aspects and the invention is not limited thereto. It is understood that innumerable variations not illustrated can be assumed without departing from the scope of the present invention. The configurations described in the above embodiments and the modifications can be appropriately combined or omitted as long as they do not conflict with each other.

10 Inspection device 20, 20A 1st learning unit 31 1st image reconstruction unit 32, 32A Abnormality calculation unit 33, 33A Data cleansing unit 40 2nd learning unit 50 Defect detection unit 90 Object 321 1st difference image generation unit 322 Score calculation unit 331 Threshold calculation unit 332 Defective product data removal unit D1, D1A 1st trained data D2 2nd trained data D20 Sample image D30 data set D31 1st reconstruction image D32, D32A Score data D321 1st difference image D33 Threshold D40, D40A Processed data set L1, L1A Trained first learning model L2 Trained second learning model M1, M1A First learning model M2 Second learning model P program

Claims (11)

A training data processing device that removes inappropriate images from a plurality of training images included in a training data set.
An abnormality degree calculation unit that calculates a score indicating an abnormality degree for each of the learning images, and an abnormality degree calculation unit.
A data cleansing unit that removes inappropriate training images from the data set based on the score.
A data processing device for learning. The learning data processing device according to claim 1.
Reconstruction of the input image Further having a first image reconstruction unit for outputting an image,
The abnormality degree calculation unit
A first difference image generation unit that generates a difference image between the learning image input to the first image reconstruction unit and the reconstruction image output from the first image reconstruction unit.
A score calculation unit that calculates the score based on the difference image generated by the first difference image generation unit, and a score calculation unit.
Including
The data cleansing unit
A threshold value calculation unit that calculates a threshold value indicating the range of non-defective products based on the score calculated by the abnormality degree calculation unit for all the learning images.
Based on the threshold value calculated by the threshold value calculation unit, a defective product data removal unit that removes the learning image not included in the non-defective product range from the data set, and a defective product data removal unit.
Data processing equipment for learning, including. The learning data processing device according to claim 2.
The first learning model that inputs and outputs images,
The first trained data in which a plurality of sample images are input to the first training model and trained to output a reconstructed image of the input sample images, and
Have more
The first image reconstruction unit is
A training data processing device that inputs an input image to the first learning model to which the first trained data is applied to generate a reconstructed image of the input image. The learning data processing device according to claim 3.
The first trained data is a training data processing device which is trained data obtained by inputting a plurality of training images included in the data set as the sample images and training them. The learning data processing device according to claim 3 or 4.
A learning data processing device having a first learning unit that generates first learned data by performing unsupervised learning on the first learning model. The learning data processing device according to claim 1.
The first learning model that inputs an image and outputs a score that is a feature,
The first trained data obtained by inputting a plurality of sample images into the first training model and training the input results of outlier detection of the sample images to be output as the score.
Have more
The abnormality degree calculation unit
A training data processing device that inputs an input image into the first learning model to which the first trained data is applied and acquires the score. It is a learning device for constructing an inspection device that inspects an object.
The learning data processing apparatus according to any one of claims 1 to 6, which removes inappropriate images from a plurality of learning images included in a learning data set.
A second learning unit that performs learning using a processed data set from which inappropriate images have been removed by the learning data processing device, and
Have,
The second learning unit inputs a plurality of the learning images included in the processed data set into a second learning model that inputs and outputs images, and the second learning model is input to the learning model. Image reconstruction A learning device that generates second trained data trained to output an image. A training data processing method that removes inappropriate images from multiple training images contained in a training dataset.
a) An abnormality degree calculation step of calculating a score indicating an abnormality degree for each of the learning images, and
b) A data cleansing step that removes inappropriate learning images from the dataset based on the score.
A data processing method for learning. The learning data processing method according to claim 8.
In step a),
a1) An image reconstruction step of reconstructing the learning image and
a2) A first difference image generation step of generating a difference image between the learning image input in the step a1) and the reconstructed image output in the step a1).
a3) A score calculation step of calculating the score based on the difference image generated in the step a2), and
Including
In step b),
b1) A threshold calculation step of calculating a threshold indicating the range of non-defective products based on the score calculated in the step a) for all the learning images.
b2) A defective product data removal step of removing the learning image not included in the range of non-defective products from the data set based on the threshold value calculated in the step b1).
Data processing methods for learning, including. A program for causing a computer to perform training data processing for removing inappropriate images from a plurality of training images included in a training data set.
On the computer
a) An abnormality degree calculation step of calculating a score indicating an abnormality degree for each of the learning images, and
b) A data cleansing step that removes inappropriate learning images from the dataset based on the score.
A program that runs. The program according to claim 10.
In step a),
a1) An image reconstruction step of reconstructing the learning image and
a2) A first difference image generation step of generating a difference image between the learning image input in the step a1) and the reconstructed image output in the step a1).
a3) A score calculation step of calculating the score based on the difference image generated in the step a2), and
Including
In step b),
b1) A threshold calculation step of calculating a threshold indicating the range of non-defective products based on the score calculated in the step a) for all the learning images.
b2) A defective product data removal step of removing the learning image not included in the range of non-defective products from the data set based on the threshold value calculated in the step b1).
Including the program.

Images