JP7505256B2 - Image inspection device, image inspection method, and trained model generation device - Google Patents

Description

The present invention relates to an image inspection device, an image inspection method, and a trained model generation device.

Conventionally, image inspection devices are known that inspect an object based on an image of the object.

For example, Patent Document 1 describes an abnormality determination device that performs abnormality determination based on inputted image data to be determined, which has a process execution means that generates reconstructed image data from the features of the image data to be determined using reconstruction parameters for reconstructing normal image data from features extracted from a group of normal image data, and executes an abnormality determination process to determine an abnormality based on difference information between the generated reconstructed image data and the image data to be determined.

When the image data to be judged includes image data of multiple channels, the anomaly judgment device of Patent Document 1 uses reconstruction parameters to generate reconstructed image data for each channel from the features of the image data of each channel, and judges anomalies based on difference information between each reconstructed image data generated and the image data of each channel of the image data to be judged.

JP 2018-5773 A

In Patent Document 1, a trained autoencoder, which is a trained model, is used to generate a reconstructed image from an image to be judged. Here, for example, when a special pattern exists locally in an image of a good inspection object, if the trained model has low expressive power, it may not be possible to reconstruct the special pattern in the image generated by the trained model. In this case, there is a risk that the image of the good inspection object may be erroneously judged to be defective.

In addition, if an image of a good inspection object shows a pattern that is good in one location or part but defective in another location or part, the image generated by the trained model may show a defective pattern, causing the defective inspection object to be overlooked.

Therefore, one of the objectives of the present invention is to provide an image inspection device, an image inspection method, and a trained model generation device that can restore special patterns and suppress the generation of defective patterns.

An image inspection device according to one aspect of the present invention includes a segmented image generation unit that inputs inspection segmented images, which are images obtained by segmenting an image of a non-defective inspection object, and label information for the inspection segmented images, to a trained model that has been trained to output restored segmented images using as input non-defective segmented images, which are images obtained by segmenting an image of a non-defective inspection object, and label information for the non-defective segmented images, to generate restored segmented images, and an inspection unit that inspects the inspection object based on the restored segmented images generated by the segmented image generation unit.

According to this aspect, it is possible to generate an inspection split image and a restored split image according to the label information of the inspection split image. Therefore, if the inspection split image includes a special pattern at a specific position, it is possible to restore the special pattern. Furthermore, even if the inspection split image partially includes a defective product pattern, it is possible to generate a restored split image including a non-defective product pattern, thereby preventing the generation of a defective product pattern.

In the above aspect, the segmented image generation unit may input a plurality of input data sets, each of which is composed of an inspection segmented image and label information of the inspection segmented image, to the trained model to generate a plurality of restored segmented images, and the inspection unit may inspect the inspection object based on the plurality of restored segmented images.

According to this aspect, inspection can be performed based on multiple restored split images, making it possible to inspect the object more accurately.

In the above aspect, the apparatus may further include a restored image generating unit that generates a restored image by combining a plurality of restored divided images, and the inspection unit may inspect the object to be inspected based on the difference between the image of the object to be inspected and the restored image.

According to this aspect, the difference between the image of the object to be inspected and the restored image becomes clear, making it possible to inspect the object to be inspected more accurately.

In the above aspect, the trained model is a neural network having multiple layers including an input layer, and the segmented image generation unit may input the inspection segmented image to the input layer and input label information to a layer having a dimension lower than that of the input layer.

According to this aspect, when label information is input to a layer with a dimension lower than that of the input layer in the neural network, the label information contributes more to the output of the trained model than when the label information is input to the input layer. As a result, it becomes possible to generate a more appropriately restored segmented image.

In the above aspect, if the first inspection split image is similar to the second inspection split image, the split image generation unit may input label information of the first inspection split image and the second inspection split image into the trained model.

According to this aspect, it becomes possible to use the same label information for similar inspection division images.

In the above embodiment, the inspection unit may determine whether the object being inspected is good or bad.

This aspect allows for more detailed inspection of the object being inspected.

In the above aspect, the inspection unit may detect defects in the object being inspected.

This aspect allows for more detailed inspection of the object being inspected.

In the above embodiment, an imaging unit may be further provided for capturing an image of the object to be inspected.

This aspect makes it easy to obtain images of the object being inspected.

In the above aspect, the device may further include a division unit that divides the image of the object to be inspected into a plurality of inspection divided images.

According to this aspect, it is possible to inspect the object to be inspected even if the image of the object to be inspected has not been divided in advance.

An image inspection method according to another aspect of the present invention is an image inspection method by a computer having a processor, in which the processor inputs inspection split images, which are images obtained by dividing an image of an inspection object that is a good product, and label information for the inspection split images, into a trained model that has been trained to output restored split images using as input good product split images, which are images obtained by dividing an image of an inspection object that is a good product, and label information for the good product split images, to generate restored split images, and inspect the inspection object based on the generated restored split images.

According to this aspect, it is possible to generate an inspection split image and a restored split image according to the label information of the inspection split image. Therefore, if the inspection split image includes a special pattern at a specific position, it is possible to restore the special pattern. Furthermore, even if the inspection split image partially includes a defective product pattern, it is possible to generate a restored split image including a non-defective product pattern, thereby preventing the generation of a defective product pattern.

A trained model generating device according to another aspect of the present invention includes a model generating unit that performs a training process using a plurality of training data sets each of which is composed of a combination of good-item divided images obtained by dividing an image of a good-item inspection object and label information for the good-item divided images, and generates a trained model that receives the good-item divided images and the label information as inputs and outputs restored divided images.

According to this aspect, it is possible to generate an inspection split image and a restored split image according to the label information of the inspection split image. Therefore, if the inspection split image includes a special pattern at a specific position, it is possible to restore the special pattern. Furthermore, even if the inspection split image partially includes a defective product pattern, it is possible to generate a restored split image including a non-defective product pattern, thereby preventing the generation of a defective product pattern.

The present invention provides an image inspection device, an image inspection method, and a trained model generation device that can restore special patterns and suppress the generation of defective patterns.

1 is a schematic configuration diagram of an image inspection system according to an embodiment of the present invention. FIG. 2 is a functional block diagram showing the configuration of a trained model generation device according to the embodiment. FIG. 2 is a diagram illustrating a learning data set generated by a learning data generating unit. FIG. 13 is a diagram showing an example of a non-defective product image. 5 is a diagram showing an example of label information assigned to each of a plurality of non-defective divided images included in the non-defective image of FIG. 4 . FIG. 5 is a diagram showing an example of label information assigned to each of a plurality of non-defective divided images included in the non-defective image of FIG. 4 . FIG. 5 is a diagram showing an example of label information assigned to each of a plurality of non-defective divided images included in the non-defective image of FIG. 4 . FIG. FIG. 2 is a diagram for explaining a model trained by a model generation unit according to an embodiment of the present invention. FIG. 2 is a functional block diagram showing the configuration of the image inspection device according to the embodiment. 2 is a functional block diagram showing a configuration of a processing unit according to the embodiment. FIG. 11 is a diagram for explaining a process up to when a processing unit generates a restored image based on an inspection image. FIG. FIG. 13 is a diagram showing an example of an inspection image. FIG. 13 is a diagram showing an example of a restored image generated based on an inspection image. FIG. 13 is a diagram showing a difference image which is the difference between an inspection image and a restored image. FIG. 1 is a diagram showing the physical configuration of an image inspection device and a trained model generation device according to an embodiment of the present invention. 1 is a flowchart illustrating an example of a flow in which a trained model generation device generates a trained model. 13 is a flowchart showing an example of a flow in which an image inspection device performs inspection of an object to be inspected using a trained model based on an image of the object to be inspected. FIG. 13 is a diagram showing an example of an inspection image.

A preferred embodiment of the present invention will be described with reference to the attached drawings.

Figure 1 is a schematic diagram of an image inspection system 1 according to one embodiment of the present invention. The image inspection system 1 includes an image inspection device 20 and a light 25. The light 25 irradiates light L onto an object to be inspected 30. The image inspection device 20 captures reflected light R and inspects the object to be inspected 30 based on an image of the object to be inspected 30 (hereinafter also referred to as an "inspection image"). The image inspection device 20 is connected to a trained model generation device 10 via a communication network 15. The trained model generation device 10 generates a trained model that is used by the image inspection device 20 to inspect the object to be inspected 30.

FIG. 2 is a functional block diagram showing the configuration of the trained model generation device 10 according to this embodiment. The trained model generation device 10 includes a storage unit 100, a training data generation unit 110, a model generation unit 120, and a communication unit 130.

The storage unit 100 stores various types of information. In this embodiment, the storage unit 100 includes a good-quality image DB 102, a training data DB 104, and a trained model DB 106. A plurality of good-quality images are stored in the good-quality image DB 102. The good-quality images are images of good-quality inspection objects. In addition, the training data DB 104 stores a plurality of training data sets, each of which is composed of a combination of divided good-quality images obtained by dividing a good-quality image and label information of the divided good-quality images. In addition, the trained model DB 106 stores trained models generated by the trained model generation device 10 described below.

The training data generation unit 110 can generate a training data set that is used by the model generation unit 120 to perform the training process. The training data set generated by the training data generation unit 110 will be described with reference to FIG. 3.

The learning data generation unit 110 acquires a good-quality image from the good-quality image DB 102, and generates a plurality of good-quality divided images by dividing the good-quality image 40. In this embodiment, the learning data generation unit 110 divides the good-quality image 40 vertically and horizontally into four, thereby generating a total of 16 good-quality divided images. Note that the good-quality image 40 may be divided into 2 to 15 good-quality divided images, or into 17 or more good-quality divided images. Furthermore, the shape of the good-quality divided images is not limited to a rectangle, and may be any shape.

The learning data generation unit 110 also assigns label information to each of the multiple non-defective divided images to generate multiple learning data sets, each of which is composed of the non-defective divided images and the assigned label information. The learning data generation unit 110 may assign label information to the non-defective divided images based on a pre-specified algorithm, or may assign label information to the non-defective divided images based on a user operation.

For example, suppose that a good-quality divided image is generated for each of a number of mutually different good-quality images. In this case, the same label information is assigned to all of the generated good-quality divided images that are located at the same position in the good-quality image.

The label information assigned to two similar good-quality divided images that are different in position in the good-quality image may be the same label information or different label information. Here, the determination of whether the two good-quality divided images are similar may be made based on, for example, whether the degree of agreement between the two good-quality divided images is equal to or greater than a predetermined threshold value.

When the learning data generation unit 110 assigns label information to the non-defective divided images, the learning data generation unit 110 may store information indicating which position of the non-defective divided images has been assigned which label information in the trained model DB 106.

For example, the first good-quality divided image 400 is assigned label information (A), and the first good-quality divided image 400 and the label information (A) constitute one learning data set. Similarly, the learning data generation unit 110 generates learning data sets based on each of the 16 good-quality divided images, such as a learning data set of the second good-quality divided image 402 and label information (B) and a learning data set of the third good-quality divided image 404 and label information (C).

Now, with reference to Figs. 4 to 7, specific examples of label information that the learning data generation unit 110 assigns to the good-quality divided images will be described. Fig. 4 is a diagram showing an example of a good-quality image 300. The good-quality image 300 shown in Fig. 4 includes four patterns (a first pattern 302, a second pattern 304, a third pattern 306, and a fourth pattern 308). Similarly to the good-quality image 40 shown in Fig. 3, the good-quality image 300 shown in Fig. 4 is divided into four vertically and horizontally, resulting in a total of 16 good-quality divided images.

Each of Figures 5 to 7 shows an example of label information assigned to each of the 16 non-defective divided images. Note that in Figures 5 to 7, the four patterns shown in Figure 4 are omitted. Also, the numbers shown on the non-defective divided images in Figures 5 to 7 represent the label information.

In the good-quality image 300 shown in FIG. 5, label information is assigned to each of the 16 good-quality divided images in the order of raster scan. That is, in the good-quality image 300 shown in FIG. 5, label information of 0 to 15 is assigned to each of the 16 good-quality divided images in the order shown by the arrows. For example, label information of 0 to 3 is assigned to each of the four good-quality divided images in the top row, starting from the good-quality divided image 320 on the left side. Furthermore, label information of 4 to 7 is assigned to each of the four good-quality divided images in the second row from the top, starting from the good-quality divided image 322 on the left side. Furthermore, label information of 8 to 15 is assigned to each of the four good-quality divided images in the third and fourth rows from the top, in the order shown by the arrows.

Note that, here, the top left non-defective divided image 320 has been described as the starting non-defective divided image with label information of 0, but any non-defective divided image may be the starting non-defective divided image. Also, the numerical values of the label information may be assigned to the non-defective divided images in any order, not limited to the order of raster scan.

In the good product image 300 shown in FIG. 6, each of the 16 good product divided images is given label information in the form of an orthogonal coordinate system of (X, Y) indicating a pair of integers X and Y. Here, X indicates the column number, and Y indicates the row number. X is 0 for the leftmost column, and increases by 1 as it moves to the right. Y is 0 for the topmost row, and increases by 1 as it moves downward. Therefore, (X, Y) indicates the good product divided image located in the (Y-1)th position from the top of the (X-1)th column from the left. For example, the good product divided image 346 located in the fourth position from the top of the second column from the left is given label information of (1, 3).

The values of X and Y are not limited to the above values, and may be any values that can identify the position (i.e., row and column) of the non-defective divided image. In the above example, the label information is described as being expressed in the form of an (X, Y) Cartesian coordinate system, but the form of the label information is not limited to this, and may be, for example, in the form of a (ρ, θ) polar coordinate system.

In FIG. 7, each of the 16 good product divided images is assigned label information according to the category to which the good product divided image belongs. Specifically, each of the 16 good product divided images is classified into a category according to the similarity between the good product divided images, and each good product divided image is assigned label information according to the category. More specifically, each of multiple similar good product divided images is classified into the same category. At this time, the same label information is assigned to good product divided images that belong to the same category.

For example, as shown in FIG. 4, of the four good-quality divided images on the left side, three good-quality divided images 320, 322, and 326 are similar. Therefore, each of these three good-quality divided images 320, 322, and 326 is classified into the same category. In addition, each of the three good-quality divided images 320, 322, and 326 is given label information of 0 as the same label information. Similarly, of the four good-quality divided images in the second column from the left side, three good-quality divided images 340, 342, and 346 are also similar. Therefore, each of these three good-quality divided images 340, 342, and 346 is given label information of 1 as the same label information. In addition, the third good-quality divided image 324 from the top of the leftmost column is not similar to any of the other good-quality divided images, so it is given label information of 3 as label information different from the label information of the other good-quality divided images. Furthermore, non-defective segmented images that do not contain patterns are given the same label information, label information 2.

Here, the label information is generated based on the similarity between the good-quality divided images, but the method of generating the label information is not limited to this. For example, if the good-quality image includes multiple parts, label information may be assigned to each part. As an example, a case where a smartphone is visually inspected as an inspection target will be described. In this case, each of the multiple good-quality divided images included in the good-quality image is classified into an area of the smartphone display (hereinafter referred to as a "first area") and an area other than the display (hereinafter referred to as a "second area"). For example, label information of 1 may be assigned to the good-quality divided images classified into the category of the first area, and label information of 2 may be assigned to the good-quality divided images classified into the category of the second area.

The training data generation unit 110 stores the generated training data set in the training data DB 104. In this embodiment, the training data generation unit 110 stores the training data set generated for multiple good product images in the training data DB 104.

The model generation unit 120 performs a learning process using multiple learning datasets, and generates a trained model that receives a good-quality segmented image and label information as input and outputs a restored segmented image. Here, the restored segmented image is an image that restores the good-quality segmented image.

8 is a diagram for explaining a model trained by the model generation unit 120 according to this embodiment. In this embodiment, the model generation unit 120 trains the model by simultaneously inputting a non-defective divided image and label information to the model. A technique for training a neural network by simultaneously inputting an image and its additional information to the neural network is described in, for example, Reference 1.
(Reference 1) Murase, Hirakawa, Yamashita, Fujiyoshi, "Improving the accuracy of autonomous driving control using CNN with self-state information", PRMU2017-82, vol. 117, no. 238, pp. 85-90

Figure 8 shows a model 50 that is the target of the learning process by the model generation unit 120. In this embodiment, the model 50 includes a neural network with multiple layers including an input layer. More specifically, the model 50 is an autoencoder, and is composed of an input layer 500, an output layer 508, and multiple layers arranged between the input layer 500 and the output layer 508. When input data is input to the input layer 500, the input data is compressed into a feature vector in the intermediate layer 504, and output data is output from the output layer 508. Note that the model used to construct the trained model is not limited to an autoencoder. In addition, the number of layers constituting the neural network is not limited to five layers.

The model generation unit 120 inputs the good-quality segmented image and label information to the model 50. Inputting the good-quality segmented image means inputting each of the multiple pixel values contained in the good-quality segmented image. As a result, an image is output as output data from the output layer 508 of the model 50. The model generation unit 120 trains the model 50 by updating the weighting parameters between each layer contained in the model 50 so that the output data becomes data that restores the good-quality segmented image. In summary, the model generation unit 120 generates a trained model by inputting multiple training data sets to the model 50 and updating the weighting parameters. The model generation unit 120 stores the generated trained model in the trained model DB 106.

In this embodiment, the model generation unit 120 inputs the good-product segmented image to the input layer 500 of the model 50, and inputs the label information to the intermediate layer 504, which has a dimension lower than the dimension of the input layer. This allows the label information to contribute more to learning than the pixels of the good-product segmented image, making it possible to generate a more appropriate trained model.

The label information may be input to a layer other than the intermediate layer 504. The label information may be input to the input layer 500, for example. In addition, at least one of the non-defective segmented image and the label information input to the model 50 may be weighted as necessary before being input to the model 50. This allows the model 50 to learn more appropriately.

Returning to FIG. 2, the communication unit 130 provided in the trained model generating device 10 will be described. The communication unit 130 can transmit and receive various information. For example, the communication unit 130 can transmit the trained model to the image inspection device 20 via the communication network 15. At this time, information indicating which label information has been assigned to which position of the non-defective divided image in the non-defective image is also transmitted to the image inspection device 20.

Figure 9 is a functional block diagram showing the configuration of an image inspection device 20 according to this embodiment. The image inspection device 20 includes a communication unit 200, a storage unit 210, an imaging unit 220, and a processing unit 230.

The communication unit 200 can transmit and receive various types of information. For example, the communication unit 200 can receive a trained model from the trained model generation device 10 via the communication network 15. The communication unit 200 can also store the trained model and the like in the memory unit 210.

The memory unit 210 stores various types of information. In this embodiment, the memory unit 210 includes a trained model DB 106. Trained models are stored in the trained model DB 106. The trained model DB 106 also stores information indicating which label information is assigned to which position of a good-quality divided image in a good-quality image. The various types of information stored in the memory unit 210 are referenced by the processing unit 230 as necessary.

The imaging unit 220 includes various known imaging devices and captures an image of the inspection object 30. In this embodiment, the imaging unit 220 receives reflected light R from the inspection object 30 and captures an image of the inspection object 30. The imaging unit 220 transmits the captured image to the processing unit 230.

The processing unit 230 can perform various processes on the image of the inspection object to inspect the inspection object. FIG. 10 is a functional block diagram showing the configuration of the processing unit 230 according to this embodiment. The processing unit 230 includes a pre-processing unit 231, a division unit 232, a label information assignment unit 233, a divided image generation unit 234, a restored image generation unit 235, a post-processing unit 236, and an inspection unit 237.

The preprocessing unit 231 performs various preprocessing operations on the image of the object to be inspected. For example, the preprocessing unit 231 can perform processing to correct positional deviation on the image of the object to be inspected. The preprocessing unit 231 transmits the image that has been subjected to the preprocessing operations to the division unit 232.

The division unit 232 can divide the image of the inspection object to generate multiple inspection split images. In this embodiment, the division unit 232 divides the image of the inspection object using a method similar to that used to divide a non-defective product image in the trained model generation device 10. Specifically, the division unit 232 divides the image of the inspection object vertically and horizontally into four parts, thereby generating 16 inspection split images. The division unit 232 transmits the generated inspection split images to the label information assignment unit 233.

The label information assignment unit 233 assigns label information to the inspection split image. The label information assignment unit 233 refers to the trained model DB 212 to refer to information indicating which label information was assigned to which position of the good-quality split image in the good-quality image when generating the trained model, and assigns the label information of the good-quality split image at the corresponding position to the inspection split image. The combination of the inspection split image and the assigned label information becomes an input dataset. The label information assignment unit 233 assigns label information to each of the multiple inspection split images to generate multiple input datasets, and transmits the multiple input datasets to the split image generation unit 234.

The segmented image generation unit 234 can input an input data set (a combination of an inspection segmented image and label information) to the trained model to generate a restored segmented image. The trained model is a trained model generated by the trained model generation device 10 that has been trained to output a restored segmented image using a non-defective segmented image and label information as input.

In this embodiment, the split image generation unit 234 inputs a plurality of input data sets, each of which is composed of an inspection split image and label information, into the trained model to generate a plurality of restored split images. Each of the generated restored split images corresponds to each of the plurality of input data sets. The restored split images are images restored from non-defective split images. Therefore, if the inspection split image contains a defect or the like, an image from which the defect has been removed is output from the trained model as the restored split image. In this embodiment, the split image generation unit 234 generates a restored split image for each of the 16 inspection split images generated based on the inspection images, and transmits the generated 16 restored split images to the restored image generation unit 235.

The restored image generating unit 235 generates a restored image by combining multiple restored divided images. In this embodiment, the restored image generating unit 235 generates a restored image by combining 16 restored divided images generated by the divided image generating unit 234. Specifically, the restored image generating unit 235 generates a restored image by placing each of the generated 16 restored divided images at the position of the corresponding inspection divided image and combining them. The restored image is an image obtained by restoring a non-defective image. Therefore, if the inspection image contains a defect, an image with the defect removed is generated as the restored image.

With reference to FIG. 11, an example of the process in which the processing unit 230 generates a restored image 44 based on the inspection image 42 will be described.

The division unit 232 divides the inspection image 42 vertically and horizontally into four parts, generating 16 inspection split images. The label information assignment unit 233 assigns label information to each of the 16 generated inspection split images. For example, the first inspection split image 420 is assigned label information (A), the second inspection split image 422 is assigned label information (B), and the third inspection split image 424 is assigned label information (C). The combination of the inspection split image and the label information becomes the input data set.

As described above, similar inspection split images may be assigned the same label information. For this reason, for example, if the first inspection split image 420 and the second inspection split image 422 are similar to each other, the first inspection split image 420 may be assigned the label information (B) of the second inspection split image 422. Alternatively, the second inspection split image 422 may be assigned the label information (A) of the first inspection split image.

The split image generating unit 234 inputs each of the 16 generated input data sets into the trained model, and generates 16 restored split images. For example, the first restored split image 440 is generated based on the first inspection split image 420, the second restored split image 442 is generated based on the second inspection split image 422, and the third restored split image 444 is generated based on the third inspection split image 424. The restored image generating unit 235 generates the restored image 44 by synthesizing the generated 16 restored split images.

Returning to FIG. 10, the post-processing unit 236 will be described. The post-processing unit 236 can perform post-processing on the restored image. For example, the post-processing unit 236 can calculate the difference between the restored image and the inspection image to generate a difference image. Specifically, the post-processing unit 236 can generate a difference image by calculating the difference between the corresponding pixel values of the inspection image from each of the multiple pixel values that make up the restored image.

The difference image generated by the post-processing unit 236 will be described with reference to Figs. 12 to 14. Fig. 12 is a diagram showing an example of an image 60 of the inspection object 30 according to this embodiment. Fig. 13 is a diagram showing an example of a restored image 62 generated based on the image 60. Fig. 14 is a diagram showing a difference image 64 which is the difference between the image 60 of the inspection object and the restored image 62. As shown in Fig. 12, the image 60 includes a linear defect image 600. The defect image is an image of a defect in the inspection object. On the other hand, as shown in Fig. 13, the defective image has been removed from the restored image 62. Therefore, the difference image 64 which shows the difference between the image 60 of the inspection object and the restored image 62 mainly includes a defect image 640. In this embodiment, the inspection object is inspected based on the difference image 64 which includes the defect image 640.

Returning to FIG. 10, the inspection unit 237 will be described. The inspection unit 237 can inspect the inspection object 30 based on the restored split images generated by the split image generation unit 234. In this embodiment, the inspection unit 237 inspects the inspection object 30 based on a plurality of restored split images.

In this embodiment, the inspection unit 237 inspects the inspection object based on the difference between the inspection image and the restored image. Specifically, the inspection unit 237 inspects the inspection object based on the difference image generated by the post-processing unit 236.

The inspection unit 237 can also detect defects in the inspection object 30. For example, the inspection unit 237 can detect defects in the inspection object 30 by detecting the defect image 640 included in the difference image 64 shown in FIG. 14. Alternatively, the inspection unit 237 can determine whether the inspection object 30 is good or bad. Specifically, the inspection unit 237 can determine whether the inspection object 30 is good or bad based on the size of the defect image included in the difference image 64. More specifically, the inspection unit 237 can determine that the inspection object is defective when the size of the defect image included in the difference image 64 exceeds a predetermined threshold value.

Figure 15 is a diagram showing the physical configuration of the trained model generating device 10 and image inspection device 20 according to this embodiment. The trained model generating device 10 and image inspection device 20 each have a CPU (Central Processing Unit) 10a corresponding to a calculation unit, a RAM (Random Access Memory) 10b corresponding to a storage unit, a ROM (Read only Memory) 10c corresponding to a storage unit, a communication unit 10d, an input unit 10e, and a display unit 10f. Each of these components is connected via a bus so that they can send and receive data to each other.

In this example, the trained model generation device 10 and the image inspection device 20 are each described as being configured with a single computer, but the trained model generation device 10 and the image inspection device 20 may each be realized by combining multiple computers. Also, the image inspection device 20 and the trained model generation device 10 may be configured with a single computer. Also, the configuration shown in FIG. 15 is an example, and the trained model generation device 10 and the image inspection device 20 may have other configurations or may not have some of these configurations.

The CPU 10a is a calculation unit that controls the execution of programs stored in the RAM 10b or ROM 10c and calculates and processes data. The CPU 10a of the trained model generation device 10 is a calculation unit that executes a program (learning program) that performs learning processing using training data and generates a trained model. The CPU 10a of the image inspection device 20 is a calculation unit that executes a program (image inspection program) that inspects an object to be inspected using an image of the object to be inspected. The CPU 10a receives various data from the input unit 10e and the communication unit 10d, and displays the calculation results of the data on the display unit 10f or stores them in the RAM 10b.

RAM 10b is a storage unit in which data can be rewritten, and may be composed of, for example, a semiconductor memory element. RAM 10b may store data such as programs executed by CPU 10a, learning data, and learned models. Note that these are merely examples, and RAM 10b may store data other than these, or some of these data may not be stored.

ROM 10c is a memory unit from which data can be read, and may be composed of, for example, a semiconductor memory element. ROM 10c may store, for example, an image inspection program, a learning program, and data that is not rewritten.

The communication unit 10d is an interface that connects the image inspection device 20 to other devices. The communication unit 10d may be connected to a communication network such as the Internet.

The input unit 10e accepts data input from a user and may include, for example, a keyboard and a touch panel. The input unit 10e may accept input of, for example, label information of a non-defective divided image or an inspection divided image.

The display unit 10f visually displays the results of calculations performed by the CPU 10a and may be configured, for example, with an LCD (Liquid Crystal Display). The display unit 10f may display, for example, the results of an inspection of an object to be inspected.

The image inspection program may be provided by being stored in a computer-readable storage medium such as RAM 10b or ROM 10c, or may be provided via a communication network connected by the communication unit 10d. In the trained model generating device 10, the CPU 10a executes the training program to realize various operations described using FIG. 2 and the like. In the image inspection device 20, the CPU 10a executes the image inspection program to realize various operations described using FIG. 9 and FIG. 10 and the like. Note that these physical configurations are merely examples and do not necessarily have to be independent configurations. For example, each of the trained model generating device 10 and the image inspection device 20 may include an LSI (Large-Scale Integration) in which the CPU 10a is integrated with the RAM 10b and ROM 10c.

Figure 16 is a flowchart showing an example of the flow by which the trained model generation device 10 generates a trained model.

First, the learning data generating unit 110 divides a good-quality image stored in the good-quality image DB 102 into a plurality of good-quality divided images (step S101). At this time, if a plurality of good-quality images are stored in the good-quality image DB 102, the learning data generating unit 110 may divide each of the plurality of good-quality images to generate good-quality divided images corresponding to each of the good-quality images.

Next, the training data generation unit 110 assigns label information to each of the non-defective segmented images generated in step S103, and generates multiple training data sets (step S103). The training data generation unit 110 stores the generated training data sets in the training data DB 104.

Next, the model generation unit 120 executes a learning process using multiple learning data sets stored in the learning data DB 104, and generates a trained model that receives the good-quality segmented image and the label information as input and outputs a restored segmented image (step S105). The model generation unit 120 stores the generated trained model in the trained model DB 106.

Next, the communication unit 130 transmits the trained model generated in step S105 to the image inspection device 20 (step S107). This enables the image inspection device 20 to use the trained model generated by the trained model generation device 10.

Figure 17 is a flowchart showing an example of the process in which the image inspection device 20 inspects an object to be inspected using a trained model based on an image of the object to be inspected.

First, the imaging unit 220 included in the image inspection device 20 captures an image of the object to be inspected (step S201). The imaging unit 220 transmits the captured image to the processing unit 230.

Next, the preprocessing unit 231 included in the processing unit 230 performs preprocessing on the image captured in step S201 (step S203). Next, the division unit 232 divides the inspection image that has been preprocessed in step S203 to generate multiple inspection split images (step S205). Next, the label information assignment unit 233 assigns label information to each of the multiple inspection split images generated in step S205, and generates multiple input data sets (step S207).

Next, the segmented image generating unit 234 inputs the multiple input data sets generated in step S207 into the trained model, respectively, to generate multiple restored segmented images (step S209). Next, the restored image generating unit 235 generates a restored image by synthesizing the multiple restored segmented images generated in step S209 (step S211). Next, the post-processing unit 236 calculates the difference between the inspection image captured in step S201 and the restored image generated in step S211 to generate a difference image (step S213).

Next, the inspection unit 237 inspects the object to be inspected based on the difference image generated in step S213 (step S215).

According to this embodiment, a restored split image is generated according to the label information of the inspection split image. Therefore, even if a special pattern is partially included, the special pattern can be restored. Furthermore, even if a pattern that is good in one position or part is defective in another position or part, the good pattern can be properly restored. As a result, the generation of defective patterns can be suppressed.

The effect of this embodiment will be described in more detail with reference to FIG. 18. FIG. 18 is a diagram showing an example of an inspection image 70. The inspection image 70 is divided into six inspection split images 700, 702, 704, 706, 708, and 710. Of these six inspection split images, inspection split images 702, 706, and 708 are similar to each other. Furthermore, inspection split image 704 includes a special pattern that is different from the other inspection split images.

Let us assume that a trained model is generated using these six inspection segmentation images as training data without using label information. If inspection segmentation image 704 is input to this trained model, if the trained model has low expressive power, inspection segmentation images 702, 704, or 708 will be output, and the special pattern may not be restored.

On the other hand, the image inspection device 20 according to this embodiment uses label information in addition to the inspection split image. Therefore, it is possible to generate an inspection split image and a restored split image according to the position of the inspection split image. As a result, even if the inspection split image includes a special pattern at a specific position, it becomes possible to restore the special pattern. For example, even an inspection split image that shows a special pattern, such as inspection split image 704, can be properly restored.

In addition, in an image of a good inspection object, a pattern that is good in one position or part may be defective in another position or part. Even in such a case, the image inspection device 20 according to this embodiment can generate a restored divided image of a good product from an inspection split image that includes a defective pattern, thereby preventing the generation of a defective pattern. As a result, it is possible to prevent defective products from being overlooked.

The above-described embodiments are intended to facilitate understanding of the present invention, and are not intended to limit the present invention. The elements of the embodiments and their arrangements, materials, conditions, shapes, sizes, etc. are not limited to those exemplified, and may be modified as appropriate. In addition, configurations shown in different embodiments may be partially substituted or combined.

[Additional Note]
a divided image generating unit (234) that generates a restored divided image by inputting an inspection divided image, which is an image obtained by dividing an image of an inspection object (30), and label information for the inspection divided image, into a trained model that has been trained to output a restored divided image using as input a non-defective divided image, which is an image obtained by dividing an image of an inspection object of a non-defective product, and label information for the non-defective divided image;
an inspection unit (237) that inspects an object to be inspected based on the restored divided image generated by the divided image generation unit (234);
An image inspection device (20) comprising:

1...Image inspection system, 10...Trained model generation device, 110...Training data generation unit, 120...Model generation unit, 20...Image inspection device, 210...Storage unit, 220...Imaging unit, 230...Processing unit, 231...Preprocessing unit, 232...Segmentation unit, 233...Label information assignment unit, 234...Segmented image generation unit, 235...Restored image generation unit, 236...Post-processing unit, 237...Inspection unit, 25...Illumination, 30...Inspection object, 40...Good product image image, 42... inspection image, 62... restored image, 64... difference image, 400... first good product divided image, 402... second good product divided image, 404... third good product divided image, 420... first inspection divided image, 422... second inspection divided image, 424... third inspection divided image, 440... first restored divided image, 442... second restored divided image, 444... third restored divided image, 500... input layer, 504... intermediate layer, 508... output layer, 600... defect image

Claims (10)

a segmented image generating unit that inputs an inspection segmented image, which is an image obtained by segmenting an image of an inspection object, and label information about the inspection segmented image to a trained model that has been trained to output a restored segmented image using a non-defective segmented image, which is an image obtained by segmenting an image of an inspection object that is a non-defective item, and label information about the non-defective segmented image, and generates the restored segmented image;
an inspection unit that inspects the inspection object based on the restored divided image generated by the divided image generation unit;
Equipped with
The image inspection device further includes a label information adding unit that adds label information of the inspection divided image to the inspection divided image. The segmented image generation unit inputs a plurality of input data sets each composed of the inspection segmented image and label information of the inspection segmented image to the trained model, and generates a plurality of the restored segmented images;
The inspection unit inspects the inspection object based on the plurality of restored divided images.
The image inspection device according to claim 1 . a restored image generating unit that generates a restored image by synthesizing the plurality of restored divided images,
the inspection unit inspects the object to be inspected based on a difference between the image of the object to be inspected and the restored image.
3. The image inspection device according to claim 2. The trained model is a neural network having a plurality of layers including an input layer,
the segmented image generation unit inputs the inspection segmented image to the input layer, and inputs the label information to a layer having a dimension lower than a dimension of the input layer;
4. An image inspection device according to claim 1. When the first inspection split image is similar to the second inspection split image, the split image generation unit inputs label information of the first inspection split image and the second inspection split image into the trained model.
5. An image inspection device according to claim 1. The inspection unit judges whether the object to be inspected is good or bad.
6. An image inspection device according to claim 1. The inspection unit detects defects in the inspection object.
7. An image inspection device according to claim 1. Further comprising an imaging unit for capturing an image of the inspection object.
8. An image inspection device according to claim 1. A division unit that divides the image of the inspection object into a plurality of the inspection divided images.
9. An image inspection device according to claim 1. 1. A computer-implemented image inspection method comprising:
A trained model is trained to output a restored divided image using a non-defective divided image, which is an image obtained by dividing an image of an inspection object, and label information about the non-defective divided image as input, and to generate the restored divided image.
inspecting the inspection object based on the generated restored divided image;
Including,
The image inspection method further includes adding label information of the inspection divided image to the inspection divided image.

Images