JP7289427B2 - Program, information processing method and information processing apparatus - Google Patents

Description

Application of Article 30, Paragraph 2 of the Patent Act Released on the website of Pros Cons Co., Ltd. on September 19, 2019 (https://proscons.co.jp/2019/09/19/gemini-eye- lp/) (https://proscons.co.jp/ad/) (https://proscons.co.jp/2019/11/15/et-iot-technology-2019/) (https://proscons.co.jp/2019/11/15/et-iot-technology-2019/) co.jp/2019/11/26/et-iot-technology-2019-report/)

Application of Patent Law Article 30, Paragraph 2 Released at Embedded Technology 2019 and IoT Technology 2019 on November 20, 2019

The present invention relates to a program, a method for generating a trained model, an information processing method, and an information processing apparatus.

2. Description of the Related Art In recent years, in a process of visual inspection of an object to be inspected, it has been widely practiced to take an image of the object to be inspected with a camera and process the image (see Patent Document 1).

JP 2019-095217 A

However, the technique according to Patent Literature 1 has the problem that it takes time and effort to learn and operate.

One aspect is to provide a program or the like that facilitates learning and operation.

In one aspect, the program receives selection of either the learning mode or the operation mode, and when the learning mode is selected, acquires an image for learning including the inspection target, and generates by unsupervised learning. The first learning model receives setting of hyperparameters of a learning model, and outputs a reconstructed image corresponding to the learning image when the acquired learning image is input based on the received hyperparameter. obtaining an inspection image including an inspection object; dividing the obtained inspection image into a plurality of mutually overlapping small piece images; input to the learning model, output reconstructed images of the respective small piece images from the first learning model, and reconstruct corresponding to the plurality of divided small piece images and the respective small piece images output by the first learning model Calculate the degree of anomaly of each pixel with the image for each small piece image, count the number of times that the degree of anomaly of each pixel calculated for each small piece image is added, and calculate the average degree of anomaly for each pixel based on the counted number of times. A computer performs processing for calculating and receiving a setting of a threshold value of the degree of abnormality for verifying the inspection image , and displaying the detected abnormal image based on the calculated average degree of abnormality and the received threshold value of the degree of abnormality. to execute.

In one aspect, it is easier to learn and operate.

BRIEF DESCRIPTION OF THE DRAWINGS It is explanatory drawing which shows the outline|summary of a visual inspection system. It is a block diagram which shows the structural example of a computer. FIG. 4 is an explanatory diagram showing an example of a record layout of a learning model management DB; It is a functional block diagram showing the processing operation of the appearance inspection system. It is an explanatory view explaining processing which generates a non-defective product learning model. FIG. 4 is an explanatory diagram showing a menu of the visual inspection system; FIG. 10 is an explanatory diagram showing an example of a material creation screen in learning mode; FIG. 9 is an explanatory diagram showing an example of a learning screen in learning mode; FIG. 11 is an explanatory diagram showing an example of a verification screen in learning mode; FIG. 10 is a flow chart showing a processing procedure for generating a non-defective product learning model; FIG. 4 is a flowchart showing a processing procedure when verification processing is executed based on an image for inspection; FIG. 5 is an explanatory diagram showing an example of an operation mode screen; 4 is a flow chart showing a processing procedure when an abnormality is detected in an operation mode; 4 is a flow chart showing a processing procedure for detecting an abnormality based on a small piece image; FIG. 10 is a flow chart showing a processing procedure for generating a non-defective product learning model for each small piece image size; FIG. FIG. 11 is a block diagram showing a configuration example of a computer according to Embodiment 3; It is explanatory drawing which outputs the detection result which detected the presence or absence of abnormality using an abnormality detection model. FIG. 11 is an explanatory diagram showing an example of a screen for collecting incorrectly determined images; FIG. 11 is a flowchart showing an example of a processing procedure for generating an anomaly detection model; FIG. FIG. 12 is a block diagram showing a configuration example of a computer according to Embodiment 4; FIG. 10 is an explanatory diagram showing an example of a record layout of an image number DB; FIG. 10 is a flowchart showing a processing procedure for obtaining the maximum number of small piece images; FIG. FIG. 4 is an explanatory diagram for explaining an outline of processing for detecting an abnormality in an inspection object using an abnormality detection model; FIG. 10 is an explanatory diagram showing an example of a screen for detecting an abnormality in an inspection object in verification mode; FIG. 10 is an explanatory diagram showing an example of a screen for detecting an abnormality in an inspection object in an operation mode; 4 is a flow chart showing a processing procedure for detecting an abnormality in an inspection object using an abnormality detection model; FIG. 10 is an explanatory diagram showing an example of a verification mode screen for detecting an abnormality in an inspection object using both a non-defective product learning model and an abnormality detection model; FIG. 10 is an explanatory diagram showing an example of an operation mode screen for detecting an abnormality in an inspection target using both a non-defective product learning model and an abnormality detection model; FIG. 10 is a flow chart showing a processing procedure for detecting an abnormality in an inspection object using both a non-defective product learning model and an abnormality detection model; FIG. FIG. 10 is a flow chart showing a processing procedure for detecting an abnormality in an inspection object using both a non-defective product learning model and an abnormality detection model; FIG. 4 is a flow chart showing a processing procedure when verifying an image including an inspection object; FIG. 10 is a flowchart showing a processing procedure for learning an anomaly detection model using an overlooked image; FIG. FIG. 11 is an explanatory diagram showing an example of a verification mode screen for detecting an abnormality of an inspection object according to Modification 1;

Hereinafter, the present invention will be described in detail based on the drawings showing its embodiments.

(Embodiment 1)
Embodiment 1 relates to a mode in which an appearance inspection is performed by artificial intelligence (AI) based on an image including an inspection object to detect abnormalities in the inspection object. Objects to be inspected are products, products, or parts in industries such as metals, foods, daily necessities, medical care, and electronic devices. Objects to be inspected are, for example, bearings, screws, product containers, glass bottles, semiconductor packages, printed circuit boards, and the like. Appearance inspection is to detect defects in appearance such as scratches, stains, foreign matter, etc., on an object to be inspected. For example, appearance defects such as chipped bearings, foreign matter, wrong labels on product containers, wrong shapes and fine bends in electronic device parts can be detected by visual inspection.

FIG. 1 is an explanatory diagram showing an overview of an appearance inspection system. The system of this embodiment includes an information processing device 1 and an imaging device 2, and each device transmits and receives information via a network N such as the Internet, Ethernet (registered trademark), or USB (Universal Serial Bus) connection.

The information processing device 1 is an information processing device that processes, stores, and transmits/receives various information. The information processing device 1 is, for example, a server device, a personal computer, a general-purpose tablet PC (personal computer), or the like. In the present embodiment, the information processing apparatus 1 is assumed to be a personal computer, which will be replaced with the computer 1 for the sake of simplicity.

The imaging device 2 captures an image of an inspection object to generate an image. The imaging device 2 of this embodiment includes a wireless communication unit. The wireless communication unit is a wireless communication module for performing processing related to communication, and transmits/receives captured images to/from the computer 1 or the like via the network N. Note that, instead of the imaging device 2, a personal computer, a smart phone, or a mobile monitoring robot capable of imaging an object to be inspected may be used.

The computer 1 according to the present embodiment acquires an image for learning including the inspection object from the imaging device 2 when the selection of the learning mode is received. The computer 1 receives settings of hyperparameters of a non-defective learning model (first learning model) generated by unsupervised learning. Based on the received hyperparameters, the computer 1 generates a non-defective product learning model that outputs a reconstructed image corresponding to the acquired image for learning when the image is input. Note that the non-defective product learning model will be described later.

The computer 1 acquires an inspection image including an inspection target from the imaging device 2, and receives setting of an abnormality threshold for verifying the inspection image. The computer 1 calculates the degree of abnormality between the acquired inspection image and the reconstructed image output by the learned non-defective product learning model. The computer 1 detects an abnormality in the object to be inspected based on the calculated abnormality degree and the threshold value of the received abnormality degree. The computer 1 displays an abnormality image indicating the detected abnormality.

When the computer 1 receives the selection of the operation mode, the computer 1 acquires an operation image including the inspection object from the imaging device 2 . The computer 1 accepts selection of a learned non-defective learning model. The computer 1 calculates the degree of abnormality between the acquired operation image and the reconstructed image output by the non-defective product learning model. The computer 1 detects whether or not there is an abnormality in the inspection object based on the calculated degree of abnormality.

FIG. 2 is a block diagram showing a configuration example of the computer 1. As shown in FIG. The computer 1 includes a control section 11 , a storage section 12 , a communication section 13 , an input section 14 , a display section 15 , a reading section 16 and a mass storage section 17 . Each configuration is connected by a bus B.

The control unit 11 includes arithmetic processing units such as a CPU (Central Processing Unit), an MPU (Micro-Processing Unit), and a GPU (Graphics Processing Unit). , various information processing, control processing, etc. related to the computer 1 are performed. Note that although FIG. 2 illustrates the controller 11 as a single processor, it may be a multiprocessor.

The storage unit 12 includes memory elements such as RAM (Random Access Memory) and ROM (Read Only Memory), and stores the control program 1P or data necessary for the control unit 11 to execute processing. The storage unit 12 also temporarily stores data and the like necessary for the control unit 11 to perform arithmetic processing. The communication unit 13 is a communication module for performing processing related to communication, and transmits/receives information to/from the imaging device 2 or the like via the network N.

The input unit 14 is an input device such as a mouse, keyboard, touch panel, buttons, etc., and outputs received operation information to the control unit 11 . The display unit 15 is a liquid crystal display, an organic EL (electroluminescence) display, or the like, and displays various information according to instructions from the control unit 11 .

The reader 16 reads a portable storage medium 1a including CD (Compact Disc)-ROM or DVD (Digital Versatile Disc)-ROM. The control unit 11 may read the control program 1P from the portable storage medium 1a via the reading unit 16 and store it in the large-capacity storage unit 17 . Alternatively, the control unit 11 may download the control program 1P from another computer via the network N or the like and store it in the large-capacity storage unit 17 . Furthermore, the control unit 11 may read the control program 1P from the semiconductor memory 1b.

The large-capacity storage unit 17 includes a recording medium such as an HDD (Hard disk drive) or an SSD (Solid State Drive). The large-capacity storage unit 17 includes a non-defective learning model 171 and a learning model management DB (database) 172 . The non-defective product learning model 171 is a kind of deep generative model, and is a trained model that models data distribution and generates new data therefrom. The learning model management DB 172 stores files of learned models and the like.

In addition, in this embodiment, the storage unit 12 and the large-capacity storage unit 17 may be configured as an integrated storage device. Also, the large-capacity storage unit 17 may be composed of a plurality of storage devices. Furthermore, the large-capacity storage unit 17 may be an external storage device connected to the computer 1 .

In this embodiment, the computer 1 is described as a single information processing device, but it may be distributed to a plurality of devices for processing, or may be composed of virtual machines.

FIG. 3 is an explanatory diagram showing an example of the record layout of the learning model management DB 172. As shown in FIG. The learning model management DB 172 includes a model ID column, a learning model file column, a generation date/time column, a template image column, and a remarks column. The model ID column stores the ID of a uniquely identified learned model file in order to identify each learned model file.

The learning model file column stores the file of the learned model or the storage location of the file of the learned model. The generation date/time column stores the date/time information when the file of the trained model was generated. The template image sequence stores reference images for indicating regions from which images containing inspection objects are cut out. The remarks column stores descriptive information for files of trained models.

In this embodiment, an example in which learning models are managed by the learning model management DB 172 has been described, but the present invention is not limited to this. For example, the learning model file itself may be managed in the storage unit 12 or the large-capacity storage unit 17 of the computer 1 for each learning model. In this case, the learning model is identified based on the uniquely identified file name. Also, the template image corresponding to the learning model may be stored in the same storage location as the learning model file.

Note that the storage form of each DB described above is an example, and other storage forms may be used as long as the relationship between data is maintained.

FIG. 4 is a functional block diagram showing processing operations of the visual inspection system. The visual inspection system includes a learning mode and an operational mode. The learning mode is a mode in which a non-defective product learning model 171 is generated by unsupervised learning based on non-abnormal non-defective product images including inspection objects. The operation mode is a mode for detecting an abnormality based on the degree of abnormality between the operation image including the inspection object and the reconstructed image output by the learned non-defective product learning model 171 .

First, the process of generating the non-defective product learning model 171 in the learning mode will be described. When the computer 1 accepts the selection of the learning mode, the computer 1 acquires a learning image (normal image) including the inspection object from the imaging device 2 . Note that images for learning may be stored in the large-capacity storage unit 17 in advance, or may be transferred from an external device.

The computer 1 creates an image to be learned by the learning model based on the template image. If the template image has not been registered (created), the computer 1 registers (creates) the template image. Specifically, the computer 1 accepts designation of the area of the inspection object. For example, the computer 1 accepts the selection of the area of the inspection object by dragging the mouse cursor onto the image in which the rectangular selection range is displayed. The computer 1 registers a template image based on the selected area of the inspection object.

The computer 1 accepts selection of a registered template image. Based on the selected template image, the computer 1 recognizes the inspection object in the learning image acquired from the imaging device 2 using, for example, a template matching method. The template matching method is a process of searching an input image for a specific pattern that is most similar to a template image (partial image). Correlation).

The computer 1 cuts out an image obtained by recognizing the inspection object from the learning image and stores it in a predetermined learning image folder. The learning image folder may be, for example, a predetermined image folder or an image folder designated by the user (operator).

Also, based on the template image, it is possible to create an image for the learning model to learn from the moving image. Specifically, the computer 1 accepts selection of a moving image file containing the inspection object. Based on the template image, the computer 1 uses an object detection algorithm using, for example, a CNN (Convolution Neural Network) to recognize the inspection object included in the selected moving image. The computer 1 cuts out a plurality of images in which the inspection object is recognized from the moving image and stores them in the predetermined learning image folder described above.

As for the object detection algorithm, RCNN (Regions with Convolutional Neural Network), template matching (for example, SSD, SAD), or other object detection algorithms may be used instead of CNN.

The computer 1 accepts setting of hyperparameters of the non-defective product learning model 171 generated by unsupervised learning. Hyperparameters are parameters that control the behavior of machine learning algorithms and include RGB (Red Green Blue)/black and white, Depth, Channel and Epoch. RGB/BW is a parameter that sets the color mode for the image. Color modes include RGB color and grayscale (monochrome).

Depth is a parameter that sets the number of layers of the encoder or decoder of the learning model. Channel is a parameter that sets the number of neurons (number of filters) per layer. Increasing the number of layers or the number of channels increases the learning rate, but it takes time to learn. Epoch is a parameter that sets the number of repetitions of one piece of training data for learning.

The computer 1 generates the non-defective product learning model 171 based on the set hyperparameters of the non-defective product learning model 171 using the images stored in the predetermined learning image folder.

FIG. 5 is an explanatory diagram for explaining the process of generating the non-defective product learning model 171. As shown in FIG. The good product learning model 171 is constructed by deep learning and used as a program module that is part of artificial intelligence software. The good product learning model 171 is, for example, Auto Encoder, VAE (Variational Auto Encoder), U-Net (Convolutional Networks for Biomedical Image Segmentation), GAN (Generative Adversarial Networks), etc. is. When there are no abnormal images (defective product images) or few abnormal images of the inspection object, only normal images (good product images) can be learned by unsupervised learning.

In the following, VAE is used as an example of the non-defective product learning model 171 for explanation. The non-defective product learning model 171 is a probability density estimator that has learned normal learning data (normal images including inspection objects), and is a model that has learned the probability density features of normal initial learning data. Anomaly detection by probability density estimation learns the occurrence probability of normal communication patterns based on normal traffic, and detects communications with a low occurrence probability as anomalies. Thus, VAE allows anomaly detection without knowledge of all malignant conditions.

Based on the reconstruction error, the VAE adjusts the weights of the encoder (encoder) and the decoder (decoder) to find the latent variable z (condensed feature). ing. Specifically, VAE uses probability distributions to determine the latent variable z. Neural networks are used for the encoder and decoder for estimating and generating probability distributions, respectively, and parameters are updated by optimization methods such as stochastic gradient descent.

The VAE finds parameters that maximize the Evidence Lower Bound (ELBO) of the logarithmic likelihood function, which is the objective function, and performs manifold learning on the latent space. A good product learning model 171 is generated by training such that the input image of the encoder and the output image (reconstructed image) of the decoder are the same.

A VAE learns using a loss function usually represented by the following equation (1).

However, D _VAE (x), A _VAE (x) and M _VAE (x) are represented by the following equations (2), (3) and (4) respectively.

VAE is a model that performs learning by maximizing the lower bound of model evidence (ELBO) for a generative model p _θ (x) represented by a parameter θ. A VAE consists of an encoder that transforms an input x into a latent variable z and a decoder that reconstructs the input from the latent variable z. Both the variational posterior distribution q _φ (z|x) and the conditional probability p _θ (x|z) are modeled as multivariate normal distributions with diagonal variance-covariance matrices. After receiving the input x, the encoder outputs two elements, the mean vector μ _z and the standard deviation vector σ _z in the variational posterior distribution q _φ (z|x). Instead of performing Monte Carlo sampling of the variational posterior distribution q _φ (z|x), the maximum posterior probability estimate μ _z of the latent variable z is used in anomaly detection.

i and j indicate the element numbers of data x and latent variable z, respectively. VAE is composed of loss function L _VAE (x) = D _VAE (x) + A _VAE (x) + M _VAE (x) and three terms. The first D _VAE (x) is a term that has the property of learning high-frequency features from given data and ignoring low-frequency features. The second A _VAE (x) is a term that has the property of adjusting according to the complexity of the features in the given data. The third M _VAE (x) is a term that directly expresses the reproduction error.

The VAE in this embodiment is constructed based on an anomaly detection algorithm using non-normalized anomaly degrees. The non-normalized anomaly score is the anomaly score in the deep generative model, that is, the negative logarithmic degree minus the normalization term. The unnormalized anomaly score is robust to the complexity that the data potentially contains and can be evaluated without relying on the frequency of appearance of each part in the image.

The normalization term D _VAE (x) and the logarithm A _VAE (x) of the normalization constant are terms that represent the frequency and complexity of the appearance pattern (cluster) of the target site itself, so they are not used to calculate the degree of abnormality. . Thus, D _VAE (x) and A _VAE (x) are removed and only M _VAE (x) is used to calculate the degree of anomaly. A non-normalized loss function is represented by the following equation (5).

Therefore, an abnormality degree formula for each pixel in abnormality detection by VAE is represented by the following formula (6).

x is a term representing an input image. _xi is a term representing a pixel value. μ is a term representing the reconstructed image corresponding to x. μ _xi is a term representing the mean vector in the conditional probability p _θ (x|z). μ _z is a term representing the mean vector in the variational posterior distribution q _φ (z|x). σ is a term representing the uncertainty and complexity of the group to which x belongs. σ _xi is a term representing the standard deviation vector in the conditional probability p _θ (x|z). By dividing by σ rather than by the difference (absolute distance) between x and μ, as in Equation (6), an anomaly score robust to the inherent uncertainty and complexity of the image is calculated.

The computer 1 generates a non-defective product learning model 171 based on the anomaly detection algorithm using the non-normalized anomaly degree. The computer 1 stores the generated non-defective product learning model 171 in the learning model management DB 172 . Specifically, the computer 1 assigns a model ID, and stores the learned model file, generation date and time, template image, and remarks as one record in the learning model management DB 172 .

Next, referring back to FIG. 4, the verification process will be described based on the inspection image in the learning mode. The computer 1 acquires an inspection image including an inspection target from the imaging device 2 . The image for inspection may be stored in advance in the large-capacity storage unit 17, or may be transferred from an external device. Based on the template image, the computer 1 cuts out an inspection image and stores it in a predetermined verification image folder.

The computer 1 accepts the setting of the threshold of the degree of abnormality for verifying the inspection image. The degree of anomaly is the degree of deviation between the actual image (input image) and the reconstructed image (output image). The threshold value of the degree of abnormality is set to a value such as "1 to 100" or "1 to 300", for example.

The computer 1 inputs the clipped image to the learned non-defective product learning model 171 and outputs a reconstructed image corresponding to the image. When an image is input to the non-defective product learning model 171, the non-defective product learning model 171 can calculate “frequency” and “complexity” by eliminating D _VAE (x) and A _VAE (x) from L _VAE (x). A reconstructed image erased from the image is output. As a result, anomalies can be detected for each pixel with the same threshold for the reconstructed image.

The computer 1 detects an abnormality based on the degree of abnormality between the clipped image and the reconstructed image corresponding to the image. First, the computer 1 calculates the degree of abnormality of each pixel in the clipped image and the reconstructed image according to the degree of abnormality formula represented by Equation (6). Specifically, the computer 1 calculates the difference between the pixel value (μ _xi ) of the reconstructed image and the pixel value (x _i ) of the clipped image. The computer 1 calculates the degree of abnormality by dividing the calculated difference by the standard deviation vector (σ _xi ). Next, the computer 1 determines whether the calculated degree of abnormality of each pixel exceeds a threshold value (eg, 50). When the computer 1 determines that at least one of the degrees of abnormality of each pixel exceeds the threshold, it detects it as an abnormality.

In addition, it does not restrict to the abnormality determination process by the abnormality degree for every pixel mentioned above. For example, if the computer 1 determines that the area of the image exceeds a predetermined area threshold (second abnormality degree threshold) among the areas of the image that exceed the above threshold, it may be detected as abnormal. For example, the computer 1 aggregates the number of pixels whose anomaly exceeds a first anomaly threshold (e.g., 50), and determines whether the aggregated number of pixels exceeds a second anomaly threshold (e.g., 30 pixels). do. When the computer 1 determines that the number of pixels whose degree of abnormality exceeds the first threshold of degree of abnormality exceeds the second threshold of degree of abnormality, the computer 1 detects it as an abnormality.

In addition, it is possible to present information as an anomaly degree map (heat map) as to which site (portion) of the inspection object determined to be abnormal has an anomalous point. The degree of anomaly map is a heat map that visualizes the location of anomalies. For example, the degree of anomaly is normalized from 1 to 100 and colored logarithmically, and dark colored portions in the heat map are recognized as abnormal. The computer 1 generates an image anomaly degree map based on the calculated anomaly degree of each pixel.

The computer 1 displays the name of the non-defective product learning model 171, the threshold of the degree of abnormality, and the degree of abnormality map of the generated image. It should be noted that the process of generating the image anomaly degree map is not essential. For example, when an image anomaly map is not generated, the computer 1 may display the name of the non-defective product learning model 171, an anomaly degree threshold, and an anomaly image. Alternatively, the computer 1 may use an alert icon or the like indicating the abnormality to clearly indicate the abnormal location on the abnormal image.

In addition, it is not limited to the abnormality detection process described above. For example, the computer 1 uses an SSD to calculate the degree of similarity between an input image and a reconstructed image corresponding to the input image. When the computer 1 determines that the calculated similarity is equal to or less than a predetermined similarity threshold (for example, 80%), the computer 1 may detect it as an anomaly. It should be noted that the abnormality may be detected based on the difference in brightness or brightness of the image. For example, the computer 1 calculates the luminance difference between the input image and the reconstructed image corresponding to the input image for each pixel. When the computer 1 determines that the calculated luminance difference is equal to or greater than a predetermined difference threshold value, the computer 1 detects an abnormality.

FIG. 6 is an explanatory diagram showing the menu of the visual inspection system. The learning mode button 10a is a button for transitioning to the learning mode screen. The operation mode button 10b is a button for transitioning to the operation mode screen. When the computer 1 accepts the touch (click) operation of the learning mode button 10a, the computer 1 transitions to the learning mode screen (FIG. 7). When the computer 1 receives a touch operation on the operation mode button 10b, the computer 1 transitions to the operation mode screen (FIG. 12).

FIG. 7 is an explanatory diagram showing an example of the material creation screen in the learning mode. The material creation screen includes a material creation tab 11a, a learning tab 11b, a verification tab 11c, a clipped image selection button 11d, an image folder selection button 11e, a video file selection button 11f, an accuracy threshold input field 11g, and a learning image folder selection button. 11h, a create button 11i and an image list field 11j.

The material creation tab 11a is a tab for displaying a material creation screen (main screen) in the learning mode. The learning tab 11b is a tab for displaying a learning screen (FIG. 8) in the learning mode. The verification tab 11c is a tab for displaying a verification screen (FIG. 9) in the learning mode. The clipped image selection button 11d is a button for selecting a template image. The image folder selection button 11e is a button for selecting an image folder in which images including inspection objects are stored. The moving image file selection button 11f is a button for selecting a moving image including an inspection object.

The precision threshold input field 11g is a text field for inputting a recognition precision threshold for recognizing an inspection object from a moving image. The learning image folder selection button 11h is a button for selecting a folder for storing learning images cut out based on the template image. The create button 11i is a button for creating the non-defective product learning model 171 . The image list column 11j is a display column for displaying images output to the learning image folder.

The computer 1 displays a material creation screen when receiving a touch operation on the material creation tab 11a. The computer 1 displays a learning screen when receiving a touch operation on the learning tab 11b. The computer 1 displays the verification screen when the touch operation of the verification tab 11c is received.

When the computer 1 accepts the touch operation of the clipped image selection button 11d, the computer 1 accepts selection of a template image including the inspection object. When the computer 1 accepts the touch operation of the image folder selection button 11e, the computer 1 accepts the selection of the folder storing the images including the inspection object. When the computer 1 accepts the touch operation of the moving image file selection button 11f, it accepts the selection of the moving image file including the object. When the computer 1 accepts a touch operation on the learning image folder selection button 11h, the computer 1 accepts selection of a folder for storing learning images cut out based on the template image.

When the computer 1 receives the touch operation of the create button 11i, the computer 1 cuts out images for learning stored in the image folder selected by the image folder selection button 11e based on the template image selected by the cutout image selection button 11d. . The computer 1 stores the clipped image in the learning image folder selected by the learning image folder selection button 11h. The computer 1 displays the images stored in the learning image folder in the image list field 11j.

When a moving image file is selected by the moving image file selection button 11f, the computer 1 recognizes the inspection object from the selected moving image file based on the template image and the threshold entered in the accuracy threshold input field 11g. Crop multiple images. The computer 1 stores the clipped image in the learning image folder selected by the learning image folder selection button 11h. The computer 1 displays the images stored in the learning image folder in the image list column 11j.

FIG. 8 is an explanatory diagram showing an example of a learning screen in the learning mode. Note that the material creation tab 11a, the learning tab 11b, and the verification tab 11c are the same as those in FIG. 7, so description thereof will be omitted. The learning screen includes a learning image folder selection button 12a, a color mode setting radio button 12b, a depth input field 12c, a channel selection button 12d, an Epoch input field 12e, an AI model storage folder selection button 12f, a learning button 12g, and a learning information display field 12h. including.

The learning image folder selection button 12a is a button for selecting an image folder in which learning images are stored. The image folder stores images of normal inspection objects with no abnormalities for learning. The color mode setting radio button 12b is a button for receiving the setting of RGB color mode or grayscale mode for the learning image. The depth input field 12c is a text field for inputting the layer number of the encoder or decoder of the learning model. The number of layers of the learning model is set, for example, to a numerical value of "1 to 4".

The Channel selection button 12d is a button for selecting the number of channels (the number of neurons per layer). The Epoch input field 12e is a text field for inputting the number of learning times. The number of learning times is set, for example, to a numerical value of "1 to 100". In this embodiment, Depth, Channel, and Epoch are given as examples of hyperparameters, but the hyperparameters are not limited to these. For example, the learning coefficient, the number of filters, or the number of units may be set.

The AI model storage folder selection button 12f is a button for selecting a folder for storing files of learned models generated by executing learning. The learning button 12g is a button for generating a learning model based on the set parameters. The learning information display column 12h is a display column for displaying learning information such as explanation information of screen operation and progress of learning (generation) of the learning model.

When the computer 1 accepts a touch operation of the learning image folder selection button 12a, the computer 1 accepts selection of an image folder in which images for learning are stored. When the computer 1 accepts the touch operation of the AI model storage folder selection button 12f, the computer 1 accepts the selection of the AI model storage folder.

When the computer 1 accepts the touch operation of the learning button 12g, the color mode is selected by the color mode setting radio button 12b, the number of layers is input by the depth input field 12c, the number of channels is selected by the channel selection button 12d, and Epoch is input. The field 12e accepts the input of the number of learning times. The computer 1 generates the non-defective product learning model 171 using the image selected by the learning image folder selection button 12a based on the received color mode, number of layers, number of channels, and number of times of learning.

The computer 1 stores the file of the generated non-defective product learning model 171 in the folder selected by the AI model storage folder selection button 12f. The computer 1 stores the file of the generated non-defective product learning model 171 in the learning model management DB 172 . In addition, the learning progress and the like are displayed in the learning information display column 12h.

FIG. 9 is an explanatory diagram showing an example of a verification screen in the learning mode. Note that the material creation tab 11a, the learning tab 11b, and the verification tab 11c are the same as those in FIG. 7, so description thereof will be omitted. The verification screen includes an AI model file selection button 13a, a verification image folder selection button 13b, anomaly degree threshold input field 13c, a verification result storage folder selection button 13d, a verification button 13e, and a verification information display field 13f.

The AI model file selection button 13a is a button for accepting selection of a learned good product learning model 171 file. The verification image folder selection button 13b is a button for selecting an image folder in which inspection images including inspection objects are stored. The anomaly threshold entry field 13c is a text field for entering an anomaly threshold.

The verification result saving folder selection button 13d is a button for selecting a folder for saving the verification result. The verification button 13e is a button for verifying an inspection image. The verification information display field 13f is a display field for displaying verification information such as screen operation explanation information and verification results.

The computer 1 acquires the file of the learned non-defective product learning model 171 from the learning model management DB 172 of the large-capacity storage unit 17 when receiving the touch operation of the AI model file selection button 13 a. For example, the computer 1 displays the learned non-defective product learning model 171 (a plurality of acquired models) in a list box (not shown). The computer 1 accepts selection of the non-defective product learning model 171 by the user.

When the computer 1 accepts the touch operation of the verification image folder selection button 13b, the computer 1 accepts the selection of the image folder in which the images for inspection are stored. When the computer 1 accepts the touch operation of the verification result storage folder selection button 13d, the computer 1 accepts selection of a folder for storing the verification result.

When the computer 1 receives the touch operation of the verification button 13e, the computer 1 inputs the image selected by the verification image folder selection button 13b to the non-defective product learning model 171 selected by the AI model file selection button 13a, and corresponds to the image. output the reconstructed image. The computer 1 calculates the degree of abnormality between the selected image and the reconstructed image corresponding to the image. The computer 1 detects an anomaly based on the calculated anomaly degree and the anomaly degree threshold entered in the anomaly degree threshold entry field 13c. The computer 1 stores the detected abnormal image in the folder selected by the verification result storage folder selection button 13d, and displays it in the verification information display field 13f.

FIG. 10 is a flow chart showing a processing procedure for generating the non-defective product learning model 171. As shown in FIG. The control unit 11 of the computer 1 receives the selection of the learning mode through the input unit 14 (step S111). The control unit 11 acquires a learning image (normal image) including the inspection target from the imaging device 2 via the communication unit 13 (step S112). The control unit 11 receives selection of a template image including the inspection object via the input unit 14 (step S113).

Based on the acquired template image, the control unit 11 cuts out the learning image acquired from the imaging device 2 and stores it in a predetermined learning image folder (step S114). Specifically, based on the selected template image, the control unit 11 recognizes the inspection object from the learning image using, for example, an SSD. The control unit 11 cuts out an image in which the inspection object is recognized from the learning images and stores the cutouts in a predetermined learning image folder. The control unit 11 receives settings of hyperparameters of the non-defective product learning model 171 including RGB/monochrome, Depth, Channel and Epoch via the input unit 14 (step S115).

Based on the set hyperparameters, the control unit 11 generates a non-defective product learning model 171 that outputs a reconstructed image corresponding to an input image (step S116). The control unit 11 assigns a model ID, stores the file of the generated non-defective learning model 171, generation date and time, template image and remarks as one record in the learning model management DB 172 (step S117), and ends the process.

FIG. 11 is a flowchart showing a processing procedure for executing verification processing based on an inspection image. The control unit 11 of the computer 1 acquires an inspection image including the inspection object from the imaging device 2 via the communication unit 13 (step S121). The control unit 11 receives selection of the learned non-defective product learning model 171 via the input unit 14 (step S122). Specifically, the control unit 11 acquires the file of the learned non-defective product learning model 171 from the learning model management DB 172 of the large-capacity storage unit 17 when the touch operation of the AI model file selection button is received. The control unit 11 displays the learned non-defective product learning model 171 that has been acquired. The control unit 11 accepts selection of the non-defective product learning model 171 by the user.

The control unit 11 receives, via the input unit 14, the setting of the threshold value of the degree of abnormality for verifying the inspection image (step S123). Based on the template image corresponding to the selected non-defective product learning model 171, the control unit 11 cuts out an inspection image and stores it in a predetermined verification result storage folder (step S124). The control unit 11 inputs the clipped image to the encoder of the learned non-defective product learning model 171 (step S125). The control unit 11 causes the decoder of the non-defective product learning model 171 to output the reconstructed image of the image (step S126).

The control unit 11 calculates the degree of abnormality of each pixel of the clipped image and the reconstructed image corresponding to the image according to the degree of abnormality formula represented by Equation (6) (step S127). The control unit 11 determines whether at least one of the calculated degrees of abnormality of each pixel exceeds a threshold value (eg, 50) (step S128). When determining that at least one of the degrees of abnormality of each pixel exceeds the threshold value (YES in step S128), the control unit 11 creates an abnormality degree map (heat map) is generated (step S129). For example, the control unit 11 uses a heat map library to generate a heat map that indicates, in terms of color density, information about which site (portion) of the inspection object determined to be abnormal has an abnormal point. For example, the darker the color, the higher the degree of abnormality.

The control unit 11 causes the display unit 15 to display the verification result including the name of the non-defective product learning model 171, the threshold of the degree of abnormality, and the degree of abnormality map of the generated image (step S130), and ends the process. The control unit 11 may simultaneously display an anomaly degree map (heat map) and an anomaly degree numerical value corresponding to a color density indicating an anomaly point on the anomaly degree map. When the control unit 11 determines that the degree of abnormality of all pixels does not exceed the threshold value (NO in step S128), the control unit 11 displays the verification result of no abnormality on the display unit 15 (step S131), and executes the process. finish.

Next, referring back to FIG. 4, the process of detecting an abnormality in the operational mode will be described. When the computer 1 accepts the selection of the operation mode, the computer 1 acquires an operation image including the inspection object from the imaging device 2 . The computer 1 accepts the selection of the learned non-defective product learning model 171, and based on the template image corresponding to the selected non-defective product learning model 171, cuts out the operation image acquired from the imaging device 2 and stores it in a predetermined operation image folder. Remember.

The computer 1 inputs the clipped image to the non-defective product learning model 171 and outputs a reconstructed image corresponding to the image. The computer 1 calculates the degree of abnormality of each pixel in the clipped image and the reconstructed image corresponding to the image. In addition, since the calculation process of the degree of abnormality is the same as the calculation process described above, a description thereof will be omitted.

The computer 1 determines whether at least one of the calculated degrees of abnormality of each pixel exceeds a threshold. When determining that at least one of the calculated degrees of abnormality of each pixel exceeds the threshold value, the computer 1 outputs a detection result indicating that an abnormality has been detected. In addition, the computer 1 generates an anomaly degree map based on the calculated anomaly degree of each pixel, and simultaneously displays the generated anomaly degree map and the anomaly degree associated with the color representing the anomaly on the anomaly degree map. You can output. When the computer 1 determines that the degree of abnormality of all pixels does not exceed the threshold, it outputs a detection result of no abnormality.

FIG. 12 is an explanatory diagram showing an example of the operation mode screen. The operation mode screen includes an AI model file selection button 14a, a monitored folder selection button 14b, a monitoring camera selection button 14c, a log file selection button 14d, an operation button 14e, and an operation information display field 14f.

The AI model file selection button 14a is a button for selecting a file of the good product learning model 171 that has been learned. The monitoring target folder selection button 14b is a button for selecting a folder in which images for operation are stored. The monitoring camera selection button 14c is a button for selecting the imaging device 2 for monitoring. The log file selection button 14d is a button for selecting a log file for storing log data in which an abnormality of an inspection object is detected. The operation button 14e is a button for detecting the presence or absence of abnormality in the inspection object. The operational information display field 14f is a display field for displaying operational information such as the presence or absence of an abnormality and an abnormal image.

When the computer 1 accepts the touch operation of the AI model file selection button 14a, the computer 1 stores the file of the non-defective product learning model 171 that has been learned in the learning model management D of the large-capacity storage unit 17.
Obtained from B172. For example, the computer 1 displays the learned non-defective product learning model 171 (a plurality of acquired models) in a list box (not shown). The computer 1 accepts selection of the non-defective product learning model 171 by the user.

When the computer 1 accepts the touch operation of the monitoring target folder selection button 14b, the computer 1 accepts the selection of the folder in which the operation image is stored. When the computer 1 receives the touch operation of the monitoring camera selection button 14c, the computer 1 receives selection of the imaging device 2 for monitoring. When the computer 1 receives the touch operation of the log file selection button 14d, the computer 1 receives the selection of the log file.

When the operation button 14e is touched, the computer 1 converts the operation image stored in the monitoring target folder selected by the monitoring target folder selection button 14b into the non-defective product learning model selected by the AI model file selection button 14a. 171 to output a reconstructed image corresponding to the operation image. The computer 1 calculates the degree of abnormality between the operational image and the reconstructed image corresponding to the operational image.

The computer 1 detects whether or not there is an abnormality in the inspection object based on the calculated degree of abnormality. The computer 1 displays the presence or absence of the detected abnormality in the operation information display column 14f. The computer 1 outputs the log data for detecting the abnormality of the inspection object to the log file selected by the log file selection button 14d.

Although the abnormality detection process described above detects an abnormality based on an operation image stored in advance in a monitoring target folder, the invention is not limited to this. For example, an abnormality may be detected based on a moving image captured in real time by the imaging device 2 .

Specifically, the computer 1 obtains in real time a moving image including the inspection object from the imaging device 2 selected by the surveillance camera selection button 14c. The computer 1 uses, for example, an object detection algorithm using CNN based on the template image corresponding to the non-defective product learning model 171 selected by the AI model file selection button 14a, and the inspection object included in the selected moving image to recognize

The computer 1 cuts out a plurality of images in which the inspection object is recognized from the moving image and stores them in the storage unit 12 or a predetermined image folder. The computer 1 inputs the clipped image to the non-defective product learning model 171 and outputs a reconstructed image corresponding to the image. The computer 1 detects an abnormality based on the degree of abnormality between the clipped image and the reconstructed image corresponding to the image. After that, the process of displaying the abnormality and outputting the log data is the same as the process described above, so the description thereof will be omitted.

FIG. 13 is a flow chart showing a processing procedure when detecting an abnormality in the operation mode. The control unit 11 of the computer 1 receives the selection of the operation mode through the input unit 14 (step S141). The control unit 11 receives selection of the learned non-defective product learning model 171 via the input unit 14 (step S142). The control unit 11 receives the selection of the imaging device 2 for monitoring through the input unit 14 (step S143). The control unit 11 acquires the operation image including the inspection object from the selected imaging device 2 via the communication unit 13 (step S144).

Based on the template image corresponding to the selected non-defective product learning model 171, the control unit 11 cuts out an operational image and stores it in a predetermined operational image storage folder (step S145). The control unit 11 inputs the clipped image to the encoder of the learned non-defective product learning model 171 (step S146). The control unit 11 causes the decoder of the non-defective product learning model 171 to output a reconstructed image corresponding to the image (step S147).

The control unit 11 calculates the degree of abnormality of each pixel in the clipped image and the reconstructed image corresponding to the image according to the degree of abnormality formula represented by Equation (6) (step S148). The control unit 11 determines whether at least one of the calculated degrees of abnormality of each pixel exceeds a threshold value (eg, 50) (step S149).

When the control unit 11 determines that at least one of the calculated degrees of abnormality of each pixel exceeds the threshold (YES in step S149), the control unit 11 obtains the name of the non-defective product learning model 171, the threshold of the degree of abnormality, and the abnormal image. is displayed on the display unit 15 (step S150). The control unit 11 stores (outputs) the log data indicating the detection of the abnormality of the inspection object in the log file (step S151), and terminates the process. When the control unit 11 determines that the degree of abnormality of all pixels does not exceed the threshold value (NO in step S149), the control unit 11 displays the detection result of no abnormality on the display unit 15 (step S152), and executes the process. finish.

According to this embodiment, it is possible to output a reconstructed image corresponding to an image including an inspection object from the non-defective product learning model 171 generated by unsupervised learning based on the image.

According to this embodiment, it is possible to detect the presence or absence of an abnormality based on the degree of abnormality between an image including an inspection target and a reconstructed image of the image output by the non-defective product learning model 171 .

According to this embodiment, since learning is possible using only a small number of non-defective product images, it is possible to detect the presence or absence of an abnormality when the definition of a defective product is unclear.

According to this embodiment, learning and operation are facilitated by providing a learning mode and an operation mode.

(Embodiment 2)
Embodiment 2 relates to an embodiment in which an image including an inspection object is divided into a plurality of small piece images, and an abnormality of the inspection object is detected based on the plurality of divided small piece images. In addition, description is abbreviate|omitted about the content which overlaps with Embodiment 1. FIG.

The computer 1 divides the image containing the inspection object into a plurality of mutually overlapping strip images (eg 10×10 pixels). Specifically, the computer 1 extracts a plurality of small piece images while sliding the small piece images so that they partially overlap each other in the image. For example, a predetermined coordinate (eg, 10, 10) in the image is set as the center point of the small piece image, and the computer 1 cuts out (trimming) the small piece image to a predetermined size (eg, 10×10). The computer 1 horizontally slides the center point by a predetermined pixel unit (for example, 20 pixels) to obtain a second center point. Based on the obtained second center point, the computer 1 cuts out a small piece image of the same size as the small piece image cut out in the above-described process so that it partially overlaps with the small piece image. The computer 1 returns to the initial center point after extracting a plurality of strip images along the lateral direction. The computer 1 slides the center point vertically by the same pixel unit to obtain the next center point. Based on the acquired center point, the computer 1 cuts out the small piece images in the same size along the horizontal direction so that the small piece images partially overlap each other in the same manner as in the processing described above. In this way, the image can be divided into multiple strip images that overlap each other.

The computer 1 inputs a plurality of divided small piece images to the non-defective product learning model 171 that has been trained in the first embodiment, to the encoder of the non-defective product learning model 171, and the decoder of the non-defective product learning model 171 corresponds to each small piece image Output the reconstructed image. The computer 1 calculates the degree of anomaly of each pixel of a plurality of divided small-piece images and the reconstructed image corresponding to each small-piece image as the degree of anomaly represented by the equation (6) in the first embodiment for each small-piece image. Calculate according to the formula.

The computer 1 adds the degree of abnormality of each pixel, and totals the number of times the degree of abnormality is added. The computer 1 calculates the average degree of abnormality for each pixel by dividing by the total number of times for the entire image. After that, the computer 1 compares the calculated average degree of abnormality with a predetermined threshold value of the degree of abnormality, and detects the presence or absence of an abnormality in the object to be inspected in the same manner as in the first embodiment.

FIG. 14 is a flow chart showing a processing procedure for detecting an abnormality based on a small piece image. The control unit 11 of the computer 1 acquires an image including the inspection object from the imaging device 2 through the communication unit 13 (step S161). The control unit 11 divides the obtained image into a plurality of mutually overlapping small piece images (for example, 10×10 pixels) (step S162).

The computer 1 inputs a plurality of divided small piece images to the encoder of the non-defective product learning model 171 (step S163), and outputs each small piece from the decoder of the non-defective product learning model 171. A reconstructed image corresponding to the image is output (step S164). The control unit 11 calculates the degree of abnormality of each pixel of the divided plural small piece images and the reconstructed image corresponding to each of the small piece images, for each small piece image, using the abnormality expressed by the equation (6) in the first embodiment. It is calculated according to the degree formula (step S165).

The control unit 11 calculates the average degree of abnormality for each pixel (step S166). Specifically, the control unit 11 adds the degree of abnormality of each pixel, and counts the number of times the degree of abnormality is added. The control unit 11 calculates the average degree of abnormality for each pixel by dividing by the total number of times for the entire image. The control unit 11 determines whether or not at least one of the calculated average degrees of abnormality of each pixel exceeds the threshold of the degree of abnormality (step S167).

When the control unit 11 determines that at least one of the average degrees of abnormality of each pixel exceeds the threshold value (YES in step S167), the control unit 11 causes the display unit 15 to display the detection result including the abnormal image (step S168), the process ends. When the control unit 11 determines that the average degree of abnormality of all pixels does not exceed the threshold value (NO in step S167), the control unit 11 displays the detection result of no abnormality on the display unit 15 (step S169), and performs processing. exit.

In addition, it is not limited to the abnormality detection process described above. The computer 1 inputs the divided plural small piece images to the encoder of the non-defective product learning model 171, and causes the decoder of the non-defective product learning model 171 to output a reconstructed image corresponding to each small piece image. The computer 1 combines the output reconstructed images to generate a combined image. Note that, when the computer 1 performs the process of generating the combined image, the brightness of the overlapping portion of the small piece images may be calculated using an average value. The computer 1 calculates the degree of abnormality of each pixel of the generated combined image and the original image of division according to the degree of abnormality formula represented by the formula (6) of the first embodiment. After that, as in the first embodiment, the computer 1 detects an abnormality in the inspection object based on the calculated degree of abnormality of each pixel.

According to this embodiment, by dividing an image including an inspection object into small piece images, calculation convergence is improved, so that it is possible to shorten the parallel calculation processing time for reconstructing a plurality of small piece images.

<Modification 1 of Embodiment 2>
Modification 1 of Embodiment 2 divides an image including an inspection object into a plurality of small piece images for each of different small piece image sizes, and detects an abnormality of the inspection object based on the plurality of divided small piece images. Regarding. It should be noted that duplicate descriptions of the items that have already been described will be omitted.

The size of the small piece image may be determined according to the size of the abnormality (defect) of the inspection object. Anomaly detection is facilitated by dividing the image into multiple slice images according to the size of the anomaly. For example, for the image of the object to be inspected, the sizes of the piece images, which are 10×10 pixels, 15×15 pixels, and 20×20 pixels, may be determined in advance.

FIG. 15 is a flow chart showing a processing procedure for generating the non-defective item learning model 171 for each small piece image size. In addition, the same code|symbol is attached|subjected about the content which overlaps with FIG. 10, and description is abbreviate|omitted. The control unit 11 of the computer 1 executes steps S21 to 25 (steps S111 to S115 in FIG. 10). The control unit 11 identifies a plurality of sizes of small piece images (step S26). For example, the control unit 11 identifies a plurality of sizes of small piece images by receiving input of sizes of small piece images by the user via the input unit 14 .

The control unit 11 acquires the size of one small piece image from among the plurality of sizes of the identified small piece images (step S27). The control unit 11 divides the image including the inspection object into a plurality of mutually overlapping small piece images according to the acquired small piece image size (step S28). The control unit 11 generates a non-defective product learning model 171 by unsupervised learning based on the divided small piece images (step S29). Note that the process of generating the non-defective product learning model 171 is the same as that of the first embodiment, so the description is omitted. The control unit 11 stores the generated non-defective product learning model 171 in the learning model management DB 172 of the large-capacity storage unit 17 (step S30).

The control unit 11 determines whether or not the size of the small piece image is the last size among the plurality of sizes of the identified small piece images (step S31). When the control unit 11 determines that the size of the small piece image is not the last size (NO in step S31), the control unit 11 returns to step S27. When the control unit 11 determines that the size of the small piece image is the last size (YES in step S31), the control unit 11 ends the process.

Next, an example using the learned non-defective product learning model 171 generated by the above processing in the operation mode will be described. The control unit 11 of the computer 1 acquires an operation image including the inspection object from the imaging device 2 . The control unit 11 receives an input of the size of the small piece image through the input unit 14 . The control unit 11 divides the acquired operational image into a plurality of mutually overlapping small piece images according to the size of the small piece image received. The control unit 11 inputs the plurality of divided small piece images to the encoder of the good product learning model 171 corresponding to the size of the small piece image, and causes the decoder of the good product learning model 171 to output the reconstructed image of each small piece image. . Since the subsequent abnormality detection processing is the same as that of the second embodiment, description thereof is omitted.

According to Modification 1, it is possible to improve the accuracy of abnormality detection by having the non-defective product learning model 171 learn for each different size of the small piece image.

(Embodiment 3)
The third embodiment relates to a form of generating a second learning model based on collected abnormal images and detecting an abnormality of an inspection object using the generated second learning model. Note that the description of the contents overlapping those of the first and second embodiments will be omitted.

FIG. 16 is a block diagram showing a configuration example of the computer 1 of the third embodiment. In addition, the same code|symbol is attached|subjected about the content which overlaps with FIG. 2, and description is abbreviate|omitted. An abnormality detection model 173 (second learning model) is stored in the large-capacity storage unit 17 . The anomaly detection model 173 is a detector that detects the presence or absence of an anomaly based on an image including an inspection object, and is a trained model generated by machine learning.

In the actual operation of unsupervised anomaly detection, some images are misjudged as normal or abnormal. For example, an image may be determined to be an abnormal image, but the image is actually a normal image. Also, for example, an image may be determined to be a normal image, but the image may actually be an abnormal image. The abnormality detection model 173 can be constructed by learning the feature amounts of these erroneously determined images. Using the anomaly detection model 173, the computer 1 outputs anomaly information corresponding to the image when an image including an inspection object is input.

FIG. 17 is an explanatory diagram for outputting the detection result of detecting the presence or absence of an abnormality using the abnormality detection model 173. In FIG. The anomaly detection model 173 is constructed by deep learning and used as a program module that is part of artificial intelligence software. The abnormality detection model 173 is a detector that has already constructed (generated) a neural network that receives an image including an inspection object as an input and outputs a detection result of detecting the presence or absence of an abnormality in the inspection object.

The computer 1 constructs (generates) the abnormality detection model 173 by performing deep learning for learning the feature amount of the inspection object in the image. For example, the anomaly detection model 173 is a CNN (Convolution Neural Network), and is learned by an input layer that receives an input of an image including an inspection object, an output layer that outputs the detection result of detecting the presence or absence of an anomaly, and back propagation. and a pre-filled intermediate layer.

The input layer has a plurality of neurons that receive pixel values of pixels included in the image, and passes the input pixel values to the intermediate layer. The intermediate layer has a plurality of neurons for extracting image features, and passes the extracted image features to the output layer. In the intermediate layer, a convolution layer that convolves the pixel value of each pixel input from the input layer and a pooling layer that maps the pixel value convolved in the convolution layer are alternately connected to obtain pixel information of the image. is compressed, and finally the feature quantity of the image is extracted. Then, the hidden layer predicts whether or not the image has anomalies with respect to the inspected object by means of a fully connected layer whose parameters have been learned by back propagation. The prediction results are output to an output layer having multiple neurons.

The image including the inspection object may be input to the input layer after passing through alternately connected convolution layers and pooling layers to extract feature amounts. Any object detection algorithm such as RCNN, Fast RCNN, Faster RCNN, SSD or YOLO (You Only Look Once) may be used instead of CNN.

The computer 1 performs learning using a combination of images including inspection objects (for example, normal images and abnormal images) and training data (teacher data) in which the presence or absence of abnormality in each image is associated. The training data is data in which the presence or absence of abnormality is labeled with respect to the image including the inspection object. Note that the process of generating training data will be described later.

The computer 1 inputs an image, which is training data, to the input layer, performs arithmetic processing in the intermediate layer, and acquires a detection result indicating the presence or absence of an abnormality from the output layer. The detection result output from the output layer may be a value discretely indicating the presence or absence of an abnormality (for example, a value of "0" or "1"), or a continuous probability value (for example, from "0" to " value in the range up to 1”).

The control unit 11 compares the detection result output from the output layer with the information labeled for the image in the training data, that is, the correct value, and controls the intermediate layer so that the output value from the output layer approaches the correct value. Optimize the parameters used in the calculation process. The parameters are, for example, weights (coupling coefficients) between neurons, coefficients of activation functions used in each neuron, and the like. Although the parameter optimization method is not particularly limited, for example, the computer 1 optimizes various parameters using the error backpropagation method.

The computer 1 performs the above processing on each image included in the training data to generate an anomaly detection model 173 . As a result, for example, the computer 1 learns the abnormality detection model 173 using the training data, thereby constructing a model capable of detecting the presence or absence of an abnormality in the inspection object.

When the computer 1 acquires an image including an inspection target, the computer 1 inputs the acquired image to the abnormality detection model 173 . The computer 1 performs arithmetic processing for extracting the feature amount of the image in the intermediate layer of the abnormality detection model 173, inputs the extracted feature amount to the output layer of the abnormality detection model 173, and determines the presence or absence of an abnormality in the inspection object. Get the detection result shown as an output.

The computer 1 determines whether or not there is an abnormality in the inspection object based on the detection results output from the output layer of the abnormality detection model 173 . For example, the computer 1 may determine that an abnormality is detected when the probability value of abnormality (eg, 0.95) is equal to or greater than a predetermined threshold value (eg, 0.85).

FIG. 18 is an explanatory diagram showing an example of a screen for collecting incorrectly determined images. FIG. 18A is an explanatory diagram showing an example of a screen for collecting abnormal images of erroneous determinations. The collection screen of erroneously determined abnormal images includes an image display field 15a, an OK button 15b, and an NG button 15c. The image display column 15a is a display column for displaying an abnormal image determined by AI in the first and second embodiments. The OK button 15b is a button for accepting from the user (operator) the determination result that the abnormal image determined by AI is correctly determined to be abnormal. The NG button 15c is a button for accepting from the user a determination result that an abnormal image determined by AI is erroneously determined to be abnormal.

FIG. 18B is an explanatory diagram illustrating an example of a collection screen of erroneously determined normal images. The image display column 15a is a display column for displaying normal images determined by AI in the first and second embodiments. The OK button 15b is a button for accepting from the user the determination result that the normal image determined by AI is correctly determined to be normal. The NG button 15c is a button for accepting from the user a determination result that an AI-determined normal image is erroneously determined to be normal.

Note that processing in response to touch operations on the OK button 15b and the NG button 15c will be described with reference to FIG.

FIG. 19 is a flow chart showing an example of the processing procedure for generating the anomaly detection model 173 . Based on FIG. 19, the processing contents of the processing for generating the abnormality detection model 173 by machine learning will be described.

The control unit 11 of the computer 1 acquires a plurality of abnormal images from the imaging device 2 or an external device through the communication unit 13 (step S181). Incidentally, the abnormal image may be stored in the large-capacity storage unit 17 in advance. The control unit 11 acquires one abnormal image from a plurality of abnormal images, and determines whether or not the user's touch operation of the NG button 15c (FIG. 18) has been accepted for the acquired abnormal image (step S182). .

When the control unit 11 determines that the touch operation of the NG button 15c has been accepted (YES in step S182), it is erroneously determined to be abnormal, and the first training is performed in which data indicating normality is added to the abnormal image. Data is generated (step S184). The control unit 11 transitions to step S186, which will be described later. When determining that the touch operation of the NG button 15c has not been accepted (NO in step S182), the control unit 11 determines whether or not the touch operation of the OK button 15b (FIG. 18) has been accepted (step S183).

When the control unit 11 determines that the touch operation of the OK button 15b has been accepted (YES in step S183), the control unit 11 correctly determines that there is an abnormality, and the third training data obtained by adding data indicating an abnormality to the abnormal image. is generated (step S185). When the control unit 11 determines that the touch operation of the OK button 15b has not been received (NO in step S183), the control unit 11 returns to the process of step S182.

The control unit 11 determines whether or not the abnormal image is the last abnormal image among the plurality of abnormal images (step S186). When the control unit 11 determines that the abnormal image is not the last abnormal image (NO in step S186), the control unit 11 returns to the process of step S182. When the control unit 11 determines that the abnormal image is the last abnormal image (YES in step S186), the communication unit 13 acquires a plurality of normal images from the imaging device 2 or an external device (step S187).

The control unit 11 obtains one normal image from a plurality of normal images, and determines whether or not the user has touched the NG button 15c on the obtained normal image (step S188). When the control unit 11 determines that the touch operation of the NG button 15c has been accepted (YES in step S188), the control unit 11 erroneously determines that the image is normal. Data is generated (step S190). The control unit 11 transitions to step S192, which will be described later.

When determining that the touch operation of the NG button 15c has not been accepted (NO in step S188), the control unit 11 determines whether or not the touch operation of the OK button 15b has been accepted (step S189). When the control unit 11 determines that the touch operation of the OK button 15b has been accepted (YES in step S189), the control unit 11 correctly determines that the image is normal, and the normal image is added with data indicating that it is normal. is generated (step S191). When the control unit 11 determines that the touch operation of the OK button 15b has not been received (NO in step S189), the control unit 11 returns to the process of step S188.

The control unit 11 determines whether or not the normal image is the last normal image among the plurality of normal images (step S192). When the control unit 11 determines that the normal image is not the last normal image (NO in step S192), the control unit 11 transitions to the process of step S188. When the control unit 11 determines that the normal image is the last normal image (YES in step S192), the control unit 11 generates a plurality of first training data, second training data, third training data, and third training data. A combination of the four training data is used to generate an anomaly detection model 173 that, when an image is input, detects (outputs) anomaly information for the image (step S193).

Specifically, the control unit 11 inputs an image, which is training data, to the input layer of the neural network, and acquires the detection result of detecting the presence or absence of an abnormality corresponding to the image from the output layer. The control unit 11 compares the obtained detection result with the correct value of the training data (information labeled with respect to the image), and performs calculations in the intermediate layer so that the detection result output from the output layer approaches the correct value. Optimize the parameters (weights, etc.) used for processing. The control unit 11 stores the generated abnormality detection model 173 in the learning model management DB 172 of the large-capacity storage unit 17 (step S194), and ends the series of processes.

The computer 1 accepts selection of at least one of the learned non-defective product learning model 171 (first learning model) built in Embodiment 1 and the anomaly detection model 173 (second learning model) built in this embodiment. . When the non-defective product learning model 171 is selected, the computer 1 detects an abnormality in the inspection object based on the abnormality detection processing in the first or second embodiment. When the anomaly detection model 173 is selected, the computer 1 detects anomalies in the inspected object based on the anomaly detection processing of this embodiment.

According to this embodiment, it is possible to generate the abnormality detection model 173 based on the collected normal images and abnormal images, and detect the abnormality of the inspection object using the generated abnormality detection model 173 .

According to this embodiment, it is possible to construct the abnormality detection model 173 by supervised learning based on the collected normal images and abnormal images.

According to this embodiment, by using both a trained model constructed by unsupervised learning and a trained model constructed by supervised learning, it is possible to increase the accuracy of abnormality detection.

(Embodiment 4)
Embodiment 4 determines the maximum number of small piece images to be simultaneously input to the good product learning model 171 according to at least one of the hardware configuration of the computer 1, the hyperparameters of the good product learning model 171, and the size of the small piece images. Regarding. Note that the description of the contents overlapping those of the first to third embodiments will be omitted.

In the case of abnormality detection processing based on a plurality of small piece images, the hardware configuration of the computer 1, the hyperparameter of the non-defective product learning model 171, or the size of the small piece image, the arithmetic processing capacity ( For example, a difference in processing speed, etc. occurs. As the number of images to be processed at the same time increases, the memory usage rate increases and the processing speed of the GPU decreases. In order to maximize the processing power of the computer 1, it is necessary to determine the maximum number of small piece images that can be processed for the non-defective product learning model 171. FIG.

The hardware configuration is specs of GPU (processor), memory, hard disk, and the like. In addition, in this embodiment, an example of a GPU has been described, but the same applies to a CPU, an MPU, and the like. The hyperparameters of the non-defective product learning model 171 include Depth (the number of layers of the model), the number of Channels (the number of channels), Epoch (the number of times of learning), and the like.

FIG. 20 is a block diagram showing a configuration example of the computer 1 of the fourth embodiment. 16 are assigned the same reference numerals, and description thereof will be omitted. A number-of-images DB 174 is stored in the large-capacity storage unit 17 . The number-of-images DB 174 stores the maximum number of images simultaneously input to the non-defective product learning model 171 based on the hardware configuration of the computer 1, the hyperparameters of the non-defective product learning model 171, and the size of the small piece image.

Note that the maximum number of small piece images stored in the image number DB 174 may be determined through a GPU load test, for example.

FIG. 21 is an explanatory diagram showing an example of the record layout of the number-of-images DB 174. As shown in FIG. The image count DB 174 includes a GPU column, a hyperparameter column, an image size column, and an image count column. The GPU column includes a CUDA (Compute Unified Device Architecture) core number column, a core block column, a memory amount column and a memory block column. The CUDA core number column stores the number of CUDA cores of the GPU. The core block column stores core blocks of the GPU. The memory capacity column stores the memory capacity of the GPU. The memory block column stores memory blocks of the GPU.

The hyperparameter sequences include the Depth sequence, the Channel sequence and the Epoch sequence. The depth column stores the number of layers of the learning model. The Channel sequence stores the number of channels of the learning model. The Epoch column stores the number of learning times. The image size column stores the size of the piece image (eg, 50×50 pixels). The number-of-images column stores the maximum number of small-piece images simultaneously input to the non-defective product learning model 171 .

FIG. 22 is a flowchart showing a processing procedure for obtaining the maximum number of small piece images. The control unit 11 of the computer 1 acquires the hardware configuration using a system information acquisition API (Application Programming Interface) (step S11). For example, the number of CUDA cores, core blocks, memory amount, memory blocks, etc. of the GPU are acquired. The control unit 11 acquires the hyperparameters of the learned non-defective product learning model 171 (step S12). For example, depth, number of channels, epoch, etc. are acquired.

The control unit 11 receives an input of the size of the small piece image through the input unit 14 (step S13). Based on the hardware configuration of the computer 1, the hyperparameters of the non-defective product learning model 171, and the size of the received small piece image, the control unit 11 collates the image number DB 174 of the large-capacity storage unit 17 to obtain the maximum number of small piece images. The number of sheets is obtained (step S14). The control unit 11 acquires an image including the inspection object from the imaging device 2 through the communication unit 13 (step S15). Similar to the small piece image division processing in the second embodiment, the control unit 11 divides the acquired image of the inspection object into a plurality of mutually overlapping small piece images based on the maximum number of acquired small piece images (step S16). ). After that, similarly to the second embodiment, the control unit 11 detects an abnormality of the inspection object based on the divided plural small piece images.

Note that FIG. 22 shows an example of processing for determining the maximum number of small piece images to be simultaneously input to the non-defective product learning model 171 according to the hardware configuration of the computer 1, the hyperparameters of the non-defective product learning model 171, and the size of the small piece images. Although explained, it is not limited to this. The maximum number of image strips may be determined according to at least one of the hardware configuration, hyperparameters, or size of the image strips.

Note that the processing for determining the maximum number of small piece images is not limited to the processing described above. For example, the maximum number of small piece images may be obtained using a formula (algorithm) created based on the hardware configuration, hyperparameters, or the size of the small piece images.

According to the present embodiment, the maximum number of small piece images simultaneously input to the good product learning model 171 is determined according to at least one of the hardware configuration of the computer 1, the hyperparameters of the good product learning model 171, and the size of the small piece images. becomes possible.

According to this embodiment, when learning or operation processing is performed based on the maximum number of small piece images, the computer 1 can maximize its processing ability.

(Embodiment 5)
Embodiment 5 relates to detecting the presence or absence of an abnormality in an inspection object using an abnormality detection model 173 (second learning model) generated by supervised learning. Note that descriptions of the contents overlapping those of the first to fourth embodiments will be omitted.

Note that the configuration of the computer of this embodiment is the same as that of the computer of the third embodiment, so the description is omitted.

The anomaly detection model 173 in this embodiment is used as a program module that is part of artificial intelligence software. When an image including an object to be inspected is input, the abnormality detection model 173 outputs information indicating an area estimated to contain an abnormality and the degree of certainty (probability) that the abnormality appears in that area. Learning model. The abnormality of the object to be inspected is an external defect such as a scratch, stain, or foreign matter on the object to be inspected.

FIG. 23 is an explanatory diagram for explaining an overview of processing for detecting anomalies in an inspection object using the anomaly detection model 173. As shown in FIG. The anomaly detection model 173 of this embodiment performs estimation using RCNN. The anomaly detection model 173 includes a region candidate extractor 73a and a classifier 73b. An image including an inspection target is input to the anomaly detection model 173 . The area candidate extraction unit 73a extracts area candidates of various sizes from the image.

The classification unit 73b includes a neural network (not shown). Neural networks include convolution layers, pooling layers and fully connected layers. The classification unit 73b has a configuration in which a convolution layer that convolves the pixel value of each pixel extracted from the region candidate extraction unit 73a and a pooling layer that maps the pixel value convoluted in the convolution layer are alternately connected. While compressing the pixel information of the area candidate, the feature amount of the area candidate is finally extracted. The classifying unit 73b classifies whether or not the subject appearing in the region candidate is abnormal based on the feature amount of the extracted region candidate using a fully connected layer whose parameters have been learned by back propagation.

The abnormality detection model 173 repeats the extraction and classification of region candidates to output information indicating the presence or absence of abnormality in the inspection object.

The computer 1 performs learning using a combination of training data in which an image including an inspection object and information indicating the presence or absence of an abnormality in the inspection object corresponding to the image are associated with each other. The training data is data in which an image is labeled with information indicating an abnormal location corresponding to the image. The label is, for example, "dent", "dirt" or "scratch".

During learning, a large amount of past anomaly detection data that exists in a database that stores anomaly information may be used as training data, or an overlooked image determined based on the anomaly detection result in the first embodiment may be used. It is also possible to use training data created by In the overlooked image, an abnormality of the inspection object is not detected from the image based on the non-defective product learning model 171 in the abnormality detection processing of the first embodiment, but an inspection (for example, visual inspection) by the inspector detects an abnormality of the inspection object. is detected. The computer 1 learns the abnormality detection model 173 using the obtained overlooked image and information indicating the existence of abnormality in the inspection object corresponding to the overlooked image input by the inspector as training data.

Specifically, the computer 1 acquires an overlooked image in which an abnormality of the inspection object is not detected based on the non-defective item learning model 171 . The computer 1 sets information (for example, an abnormality label such as “dent”, “dirt” or “flaw”) indicating the presence of an abnormality in the inspection object corresponding to the acquired overlooked image. Accept input. The computer 1 creates training data in which the acquired overlooked image and information indicating the presence of an abnormality in the inspection object corresponding to the received overlooked image are associated with each other. By using overlooked images as training data, the more the system is continued, the more the anomaly detection model 173 is updated, making it possible to perform more accurate anomaly detection.

The computer 1 inputs an image, which is training data, to the region candidate extraction unit 73a, and acquires information indicating the presence or absence of an abnormality in the inspection object corresponding to the image through arithmetic processing in the classification unit 73b. The computer 1 compares the information indicating the presence or absence of abnormality in the inspection object output from the classification unit 73b with the information labeled for the image in the training data, that is, the correct value, and the output value from the classification unit 73b is Parameters used for arithmetic processing in the classification unit 73b (convolution layer, pooling layer and fully connected layer) are optimized so as to approach the correct value. The parameters are, for example, weights (coupling coefficients) between neurons, coefficients of activation functions used in each neuron, and the like. Although the parameter optimization method is not particularly limited, for example, the computer 1 uses error backpropagation to optimize various parameters.

The computer 1 performs the above processing on each image included in the training data to generate an anomaly detection model 173 . The computer 1 stores the generated anomaly detection model 173 in the learning model management DB 172 . In the learning model management DB 172, for example, the computer 1 learns the abnormality detection model 173 using the training data, thereby constructing a model capable of outputting information indicating the presence or absence of an abnormality in the inspection object corresponding to the image. can do. When the computer 1 acquires an image, it uses the abnormality detection model 173 to output information indicating the presence or absence of an abnormality in the inspection object corresponding to the image.

The anomaly detection model 173 outputs the range of the area and the certainty that the anomaly is present for the area candidate classified as having an anomaly that is higher than a predetermined threshold. In the example shown in FIG. 23, an area in which a dent abnormality is shown with a certainty of 0.87, which is higher than a predetermined dent abnormality threshold (for example, 0.80), and a predetermined contamination abnormality threshold (for example, 0.85) with a certainty of 0.92, which is higher than 0.85).

Any object detection algorithm such as CNN, Fast RCNN, Faster RCNN, SSD, or YOLO may be used instead of RCNN. Alternatively, a segmentation network such as U-Net may be used.

FIG. 24 is an explanatory diagram showing an example of a screen for detecting an abnormality in an inspection object in verification mode. The screen includes an AI model selection button 16a, a verification image folder selection button 16b, a label name selection combo box 16c, a certainty threshold input field 16d, an update button 16e, a certainty display field 16f, a save folder selection button 16g, and a verification button 16h. , an abnormality type tab 16i and a detection result display field 16j.

The AI model selection button 16a is a button for accepting selection of a learned anomaly detection model 173 file. The verification image folder selection button 16b is a button for selecting an image folder in which images including inspection objects are stored. The label name selection combo box 16c is a combo box that accepts selection of the name of the label attached to the image. The certainty threshold input field 16d is a text field for receiving input of thresholds for certainty of various types of abnormality. The update button 16e is a button for updating thresholds of certainty of various types of abnormality.

The certainty display column 16f is a display column for displaying the set thresholds of the certainty of various types of abnormality. The storage folder selection button 16g is a button for selecting a folder for storing the verification result. The verification button 16h is a button for verifying an image including an inspection object. The abnormality type tab 16i is a tab for switching between the type of abnormality (for example, dent, stain, or scratch) or no abnormality detection. As illustrated, the abnormality type tab 16i includes a "dent" tab, a "dirt" tab, a "flaw" tab, and a "no detection" tab. The detection result display column 16j is a display column for displaying the detection result of detecting the presence or absence of an abnormality in the inspection object.

The computer 1 acquires the file of the learned abnormality detection model 173 from the learning model management DB 172 when receiving the touch operation of the AI model selection button 16 a. For example, the computer 1 displays the acquired learned anomaly detection model 173 (a plurality of models) in a list box (not shown). The computer 1 accepts selection of the anomaly detection model 173 by the user. When the computer 1 accepts the touch operation of the verification image folder selection button 16b, the computer 1 accepts the selection of the image folder in which the verification images are stored.

When the computer 1 accepts the selection operation of the label name selection combo box 16c, the computer 1 accepts selection of the label name of the corresponding abnormality. When accepting an input operation in the certainty threshold input field 16d, the computer 1 accepts an input of the certainty threshold corresponding to the label name of the abnormality selected by the label name selection combo box 16c. When the touch operation of the update button 16e is received, the computer 1 associates the abnormality certainty threshold input in the certainty threshold input field 16d with the label name of the abnormality selected in the label name selection combo box 16c. to update. The computer 1 stores the updated thresholds of certainty of various types of abnormality in the storage unit 12 or the large-capacity storage unit 17, and displays the stored thresholds of certainty of various types of abnormality in the degree of certainty display column 16f.

When the computer 1 accepts the touch operation of the storage folder selection button 16g, the computer 1 accepts selection of a folder for storing the verification result. When the computer 1 receives the touch operation of the verification button 16h, the computer 1 detects the presence or absence of an abnormality in the inspection object based on various thresholds of certainty of abnormality. Specifically, the computer 1 inputs the image selected by the verification image folder selection button 16b to the anomaly detection model 173 selected by the AI model selection button 16a, and the set thresholds for the certainty of various anomalies. For a region candidate classified as containing an abnormality with a higher certainty than , output the range of the region and the degree of certainty that the anomaly is shown.

The computer 1 stores the detected abnormal image in the folder selected by the save folder selection button 16g. The computer 1 classifies single or multiple images based on the classification information (eg, dent, dirt, scratch, and no detection) shown in the abnormality type tab 16i. The computer 1 displays the detection result of each classified image in the detection result display column 16j corresponding to the corresponding abnormality type tab 16i.

Specifically, the computer 1 calculates the range (coordinates) of the region in which the abnormality appears, the type of abnormality, the degree of certainty, etc., together with each image, from the image in which the abnormality of the inspection object is detected. It is displayed in the detection result display field 16j corresponding to the corresponding abnormality type tab 16i ("dent" tab, "dirt" tab, or "flaw" tab) according to the type. The computer 1 detects an abnormality in the detection result display column 16j corresponding to the abnormality type tab 16i, which is the "no detection" tab, for the images in which no abnormality is detected in the inspection object, together with each image. to display detection results that are not

FIG. 25 is an explanatory diagram showing an example of a screen for detecting anomalies in inspection objects in the operation mode. Note that the same reference numerals are given to the contents that overlap with those in FIG. 24, and the description thereof is omitted. The screen includes a monitoring method selection radio button 17a, a monitoring folder selection button 17b, a camera selection combo box 17c, a resolution selection combo box 17d, a log file selection button 17e and an operation button 17f.

The monitoring method selection radio button 17a is a radio button for selecting "folder monitoring" or "camera monitoring". The monitor folder selection button 17b is a button for selecting a folder in which images for operation are stored. When the computer 1 accepts the touch operation of the monitoring folder selection button 17b, the computer 1 accepts selection of the folder in which the operation image is stored.

The camera selection combo box 17c is a combo box for selecting the connected imaging device 2 for monitoring. The resolution selection combo box 17d is a combo box for receiving the selection of the resolution of the selected imaging device 2 for monitoring. When the computer 1 accepts the selection operation of the camera selection combo box 17c, the computer 1 accepts the selection of the imaging device 2 for monitoring. When the computer 1 accepts the selection operation of the resolution selection combo box 17d, the computer 1 accepts the selection of the resolution corresponding to the monitoring imaging device 2 selected by the camera selection combo box 17c.

The log file selection button 17e is a button for selecting a log file for storing log data in which an abnormality of an inspection object is detected. When the computer 1 receives the touch operation of the log file selection button 17e, the computer 1 receives the selection of the log file.

The operation button 17f is a button for detecting the presence or absence of abnormality in the inspection object. The computer 1 acquires an operation image when receiving a touch operation of the operation button 17f. Specifically, when the monitoring method selected by the monitoring method selection radio button 17a is "folder monitoring", the computer 1 displays the operation image stored in the monitoring target folder selected by the monitoring folder selection button 17b. get. When the monitoring method selected by the monitoring method selection radio button 17a is "camera monitoring", the computer 1 obtains a moving image captured in real time by the monitoring imaging device 2 selected by the camera selection combo box 17c.

The computer 1 inputs the acquired operation image to the learned abnormality detection model 173 selected by the AI model selection button 16a, and outputs information indicating the presence or absence of an abnormality in the inspection object corresponding to the operation image. Let The computer 1 displays in the detection result display column 16j the information indicating the presence or absence of an abnormality in the inspection object output from the abnormality detection model 173 together with the operation image.

FIG. 26 is a flow chart showing a processing procedure for detecting anomalies in inspection objects using the anomaly detection model 173. As shown in FIG. The control unit 11 of the computer 1 acquires an image including the inspection target through the communication unit 13 or the input unit 14 (step S21). The control unit 11 receives the setting of the threshold value of the certainty of various kinds of abnormality through the input unit 14 (step S22). Note that when preset thresholds for the certainty of abnormality are stored in the storage unit 12 or the large-capacity storage unit 17, the control unit 11 stores various thresholds for the certainty of anomalies in the storage unit 12 or the large-capacity storage. Acquired from the unit 17.

The control unit 11 inputs the acquired image to the abnormality detection model 173 based on the received various thresholds of certainty of abnormality, and outputs information indicating the presence or absence of an abnormality in the inspection object corresponding to the image as a detection result. (step S23). The control unit 11 displays the image and the detection result on the display unit 15 (step S24), and terminates the process.

According to this embodiment, it is possible to detect an abnormality in an inspection object using the abnormality detection model 173 generated by supervised learning.

According to this embodiment, it is possible to adjust the accuracy of detection of various types of abnormality with respect to the inspection object by setting the thresholds of the degrees of certainty of various types of abnormality.

(Embodiment 6)
Embodiment 6 relates to a configuration in which a non-defective product learning model 171 (first learning model) and an abnormality detection model 173 (second learning model) are used together to detect an abnormality in an inspection object. Note that the description of the contents overlapping those of the first to fifth embodiments will be omitted.

A computer 1 acquires an image including an inspection object. First, the computer 1 detects a first detection result indicating the presence or absence of an abnormality in the inspection object corresponding to the image based on the non-defective item learning model 171 . Specifically, the computer 1 accepts settings of an abnormality threshold and an area threshold for verifying an image. The computer 1 accepts the setting of the threshold value of the degree of matching between the template image created based on the region of the inspection object and the acquired image.

The computer 1 uses a template matching technique such as SSD, SAD, or NCC to calculate the degree of matching between the template image and the acquired image. The computer 1 determines whether or not the calculated matching degree exceeds the received matching degree threshold. When the computer 1 determines that the calculated degree of matching exceeds the threshold value of the degree of matching, the computer 1 executes abnormality detection processing, which will be described later. It should be noted that the matching degree determination process is not essential. The computer 1 may execute the abnormality detection process for all the acquired images without executing the matching degree determination process described above.

The computer 1 inputs the obtained image to the trained non-defective product learning model 171 and outputs a reconstructed image corresponding to the image. The computer 1 calculates the degree of abnormality of each pixel between the acquired image and the reconstructed image output by the non-defective product learning model 171 . Note that the processing for calculating the degree of anomaly is the same as in the first embodiment, so the description is omitted. The computer 1 determines whether or not the calculated degree of abnormality of each pixel exceeds the threshold value (for example, 50) of the received degree of abnormality.

The computer 1 tallies the number of pixels whose degree of abnormality exceeds the threshold of the degree of abnormality among the regions of the image in which the calculated degree of abnormality of each pixel exceeds the threshold. The computer 1 determines whether or not the counted number of pixels (the area of the region) exceeds the received area threshold (eg, 30 pixels). When the computer 1 determines that the area of the region exceeds the area threshold, it detects it as an abnormality.

The computer 1 detects the first detection result indicating the presence or absence of abnormality in the inspection object by executing the above-described processing. The first detection result includes, for example, the range of the area where the abnormality is shown.

Next, the computer 1 uses the learned abnormality detection model 173 to detect a second detection result indicating the presence or absence of an abnormality in the inspection object. Specifically, the computer 1 inputs the acquired image to the abnormality detection model 173 learned in the fifth embodiment, and detects a second detection result indicating the presence or absence of an abnormality in the inspection object corresponding to the image. . The second detection result includes the range of area candidates classified as containing an abnormality, the degree of certainty that the abnormality is contained, the type, and the like.

When determining that an abnormality is detected from the first detection result, the computer 1 generates an abnormality degree map of the image based on the degree of abnormality of each pixel of the image. The computer 1 displays the detected first detection result and second detection result together with the generated abnormality degree map of the image. When the computer 1 determines that no abnormality is detected from the first detection result, it displays the detected first detection result and second detection result together with the image.

FIG. 27 is an explanatory diagram showing an example of a verification mode screen for detecting an abnormality in an inspection object using both the non-defective product learning model 171 and the abnormality detection model 173. As shown in FIG. The verification mode screen includes a good product learning model setting input field 18a, an anomaly detection model setting input field 18b, a good product folder selection button 18c, a defective product folder selection button 18d, a storage folder selection button 18e, a verification button 18f, a classification tab 18g, and verification. A result display field 18h is included.

The non-defective product learning model setting input field 18a is a setting input field for receiving selection of a learned non-defective product learning model 171 file and input of an abnormality threshold and an area threshold. The anomaly detection model setting input field 18b is a setting input field for receiving the selection of the learned anomaly detection model 173 file and the input of the threshold value of the certainty of various anomalies. The non-defective product folder selection button 18c is a button for selecting a folder in which non-defective product images including inspection objects are stored. The defective product folder selection button 18d is a button for selecting a folder in which defective product images including inspection objects are stored. The storage folder selection button 18e is a button for selecting a folder for storing the verification result. The verification button 18f is a button for verifying the image.

The classification tab 18g is a tab for switching classification information consisting of a contrast relationship between the first detection result and the second detection result. As illustrated, the classification information includes "defective product-defective product determination", "defective product-defective product determination", "defective product-defective product determination", and "defective product-defective product determination". In the “non-defective product-non-defective product determination” classification, both the first detection result and the second detection result are non-defective products (no abnormality). In the "non-defective product-defective product determination" classification, the first detection result is a non-defective product, and the second detection result is a defective product (abnormal). In the "defective product-defective product determination" classification, the first detection result is a defective product, and the second detection result is a non-defective product. In the "defective product-defective product determination" classification, both the first detection result and the second detection result are defective products. The verification result display column 18h is a display column for displaying the first detection result and the second detection result.

When the computer 1 receives a setting operation in the non-defective product learning model setting input field 18a, the computer 1 receives selection of the learned non-defective product learning model 171 file and inputs of the threshold of the degree of abnormality and the threshold of area. When the computer 1 accepts a setting operation in the anomaly detection model setting input field 18b, it accepts selection of a learned anomaly detection model 173 file and input of various anomaly certainty thresholds.

When the computer 1 accepts the touch operation of the non-defective product folder selection button 18c, the computer 1 accepts selection of a folder storing non-defective product images determined to be non-defective products. When the computer 1 accepts the touch operation of the defective product folder selection button 18d, the computer 1 accepts selection of a folder storing defective product images determined as defective products. Although the non-defective product images and the defective product images are stored in separate folders, they may be stored in the same folder. When the computer 1 accepts the touch operation of the storage folder selection button 18e, the computer 1 accepts selection of a folder for storing the verification result.

When the computer 1 receives the touch operation of the verification button 18f, the image of the non-defective product stored in the folder selected by the non-defective product folder selection button 18c and the defective product image stored in the folder selected by the defective product folder selection button 18d are displayed. Get an image. The computer 1 inputs the inspection images including the acquired non-defective product images and defective product images to the learned non-defective product learning model 171 selected by the non-defective product learning model setting input field 18a, and corresponds to each inspection image. Output the reconstructed image.

The computer 1 calculates the degree of abnormality between each inspection image and the reconstructed image corresponding to each inspection image output by the non-defective product learning model 171 . Based on the calculated degree of abnormality, the computer 1 detects a first detection result indicating the presence or absence of an abnormality in the inspection object corresponding to each inspection image.

The computer 1 inputs each inspection image to the abnormality detection model 173 selected by the abnormality detection model setting input field 18b, and indicates whether or not there is an abnormality in the inspection object corresponding to each inspection image. 2 Detect the detection result. The computer 1 stores the detected first detection result and second detection result in the folder selected by the storage folder selection button 18e. The computer 1 performs the first detection result and the second detection based on the classification information including "defective product-defective product determination", "defective product-defective product determination", "defective product-defective product determination", and "defective product-defective product determination". A plurality of inspection images are classified according to the results.

The computer 1 displays each inspection image in the verification result display field 18h corresponding to the classification tab 18g of "non-defective product-non-defective product determination" for the inspection image classified as "non-defective product-non-defective product determination". For the inspection images classified as "defective product - good product judgment", the computer 1 generates an abnormality degree map for each inspection image based on the abnormality degree of each pixel of each inspection image. do. The computer 1 displays the verification result display column 18h corresponding to the classification tab 18g of "defective product - non-defective product judgment" together with the abnormality degree map of each inspection image, and displays the abnormality corresponding to each inspection image. The first detection result indicating the location is displayed with an anchor box.

For the inspection images classified as "non-defective-defective product determination", the computer 1 displays each inspection image in the verification result display field 18h corresponding to the classification tab 18g "defective-defective product determination". In addition, the second detection result indicating the abnormal portion corresponding to each inspection image is displayed by color coding corresponding to the type of abnormality. For example, when a "hit" abnormality is detected, the second detection result including the certainty of the "hit" may be displayed in a yellow anchor box superimposed on the image at the "hit". Alternatively, when the "dirt" abnormality is detected, the second detection result including the degree of certainty of "dirt" may be superimposed on the image with a blue anchor box at the "dirt" location.

For the inspection images classified as "defective product-defective product judgment", the computer 1 creates an abnormality degree map for each inspection image based on the abnormality degree of each pixel of each inspection image. Generate. The computer 1 displays the verification result display field 18h corresponding to the classification tab 18g "defective product-defective product judgment", together with the generated abnormality degree map of each inspection image, to each inspection image. The first detection result and the second detection result indicating the corresponding abnormal location are displayed in different colors corresponding to the type of abnormality. For example, the computer 1 may display a first detection result (for example, an unknown defect) indicating an abnormal location corresponding to the inspection image so as to be superimposed on the abnormality degree map with a red anchor box. Alternatively, the computer 1 may display the second detection result (for example, the degree of certainty of the dent abnormality) indicating the abnormal location corresponding to the inspection image, superimposed on the abnormality degree map in a yellow anchor box.

FIG. 28 is an explanatory diagram showing an example of an operation mode screen for detecting an abnormality in an inspection target using both the non-defective product learning model 171 and the abnormality detection model 173. FIG. 25 and 27 are denoted by the same reference numerals, and description thereof will be omitted. The operation mode screen includes a non-defective product learning model setting input field 19a and an operation button 19b.

The non-defective product learning model setting input field 19a is used to select a learned non-defective product learning model 171 file, a threshold for the degree of matching between the template image created based on the area of the inspection object and the operation image, the threshold for the degree of abnormality, and the area. This is a setting input field for receiving input of a threshold value. When the computer 1 accepts a setting operation in the non-defective product learning model setting input field 19a, the computer 1 accepts selection of a learned non-defective product learning model 171 file, input of a matching threshold, an abnormality threshold, and an area threshold.

The operation button 19b is a button for detecting the presence or absence of abnormality in the inspection object. When the computer 1 receives a touch operation on the operation button 19b, the computer 1 acquires an operation image including the inspection object. Specifically, when the monitoring method selected by the monitoring method selection radio button 17a is "folder monitoring", the computer 1 displays the operation image stored in the monitoring target folder selected by the monitoring folder selection button 17b. get. When the monitoring method selected by the monitoring method selection radio button 17a is "camera monitoring", the computer 1 obtains a moving image captured in real time by the monitoring imaging device 2 selected by the camera selection combo box 17c.

The computer 1 inputs the obtained operating image to the learned non-defective product learning model 171 selected by the non-defective product learning model setting input field 19a, and outputs a reconstructed image corresponding to the operational image. The computer 1 calculates the degree of abnormality between the obtained operation image and the reconstructed image corresponding to the operation image output by the non-defective product learning model 171 . The computer 1 detects a first detection result indicating whether or not there is an abnormality in the inspection object based on the calculated degree of abnormality. The computer 1 inputs the acquired operation image to the abnormality detection model 173 selected by the abnormality detection model setting input field 18b, and a second detection result indicating whether or not there is an abnormality in the inspection object corresponding to the operation image. to detect

When the computer 1 determines that an abnormality in the inspection object is detected from the first detection result, the computer 1 generates an abnormality degree map of the operational image based on the degree of abnormality of each pixel of the operational image. The computer 1 displays the first detection result and the second detection result in the verification result display column 18h together with the generated abnormality degree map of the operation image. When the computer 1 determines from the first detection result that no abnormality has been detected in the inspection object, it displays the first detection result and the second detection result in the verification result display column 18h together with the operation image.

FIGS. 29 and 30 are flow charts showing a processing procedure for detecting an abnormality in an inspection object using both the non-defective product learning model 171 and the abnormality detection model 173. FIG. The control unit 11 of the computer 1 acquires an image including the inspection object (step S31). For example, when an image including an inspection object is stored in the storage unit 12 or the large-capacity storage unit 17 , the control unit 11 acquires the image from the storage unit 12 or the large-capacity storage unit 17 .

The control unit 11 receives through the input unit 14 settings of an abnormality threshold and an area threshold for verifying the image (step S32). The control unit 11 receives the setting of the threshold value of the certainty of various types of abnormality through the input unit 14 (step S33). The control unit 11 receives through the input unit 14 the setting of the threshold for the degree of matching between the template image created based on the region of the inspection object and the acquired image (step S34).

The control unit 11 uses a template matching technique such as SSD to calculate the degree of matching between the template image and the acquired image (step S35). The control unit 11 determines whether or not the calculated matching degree exceeds the received matching degree threshold (step S36). When determining that the calculated degree of matching does not exceed the threshold value of the degree of matching (NO in step S36), the control unit 11 returns to the process of step S31. When the control unit 11 determines that the calculated degree of matching exceeds the threshold value of the degree of matching (YES in step S36), the control unit 11 inputs the acquired image to the learned non-defective product learning model 171 to generate a reconstructed image corresponding to the image. is output (step S37).

The control unit 11 calculates the degree of abnormality of each pixel in the acquired image and the reconstructed image output by the non-defective product learning model 171 (step S38). The control unit 11 tallies the number of pixels whose abnormality degree exceeds the threshold value (step S39), from among the regions of the image in which the calculated abnormality degree of each pixel exceeds the abnormality degree threshold value (for example, 50). The control unit 11 determines whether or not the counted number of pixels (the area of the region) exceeds the received area threshold value (for example, 30 pixels) (step S40).

When determining that the area of the region exceeds the area threshold (YES in step S40), the control unit 11 determines that there is an abnormality (step S41), and transitions to the processing of step S43 described later. If the control unit 11 determines that the area of the region does not exceed the area threshold (NO in step S40), it determines that there is no abnormality (step S42). The control unit 11 detects the first detection result based on the abnormality determination result (step S43). The first detection result includes, for example, the range of the area where the abnormality is shown.

The control unit 11 inputs the image acquired in the process of step S31 to the abnormality detection model 173 (step S44). The control unit 11 uses the learned abnormality detection model 173 to detect a second detection result indicating the presence or absence of an abnormality in the inspection target based on the received thresholds of certainty of various types of abnormality (step S45). The second detection result includes the area candidate range classified as containing an abnormality, the degree of certainty that the abnormality is contained, and the like.

The control unit 11 determines whether or not an abnormality of the inspection object is detected from the detected first detection result (step S46). When determining that an abnormality is detected from the first detection result (YES in step S46), the control unit 11 generates an abnormality degree map of the image based on the degree of abnormality of each pixel of the image (step S47). The control unit 11 displays the generated abnormality degree map of the image and the detected first detection result and second detection result on the display unit 15 (step S48), and ends the process. When the control unit 11 determines that no abnormality is detected from the first detection result (NO in step S46), the display unit 15 displays the image and the detected first detection result and second detection result. (Step S49), the process ends. Note that when no abnormality is detected from the first detection result, the control unit 11 may display an abnormality degree map of the image and the detected first detection result and second detection result.

FIG. 31 is a flow chart showing a processing procedure for verifying an image including an inspection object. The control unit 11 of the computer 1 acquires a plurality of images including the inspection object from the storage unit 12 or the large-capacity storage unit 17 (step S61). The control unit 11 receives selection of the learned non-defective product learning model 171 via the input unit 14 (step S62). Specifically, the control unit 11 acquires the file of the learned non-defective product learning model 171 from the learning model management DB 172 of the large-capacity storage unit 17 when the touch operation of the AI model file selection button is received. The control unit 11 displays the learned non-defective product learning model 171 that has been acquired. The control unit 11 accepts selection of the non-defective product learning model 171 by the user. The control unit 11 receives selection of the abnormality detection model 173 via the input unit 14 (step S63).

Based on the selected non-defective product learning model 171, the control unit 11 detects a first detection result indicating the presence or absence of an abnormality in the inspection object corresponding to each acquired image (step S64). Based on the selected abnormality detection model 173, the control unit 11 detects a second detection result indicating the presence or absence of an abnormality in the inspection object corresponding to each acquired image (step S65). It should be noted that the detection processing of the first detection result and the second detection result is the same as the processing described above, so the description is omitted.

The control unit 11 determines the first detection result and the second A plurality of images are classified according to the detection result (step S66). The control unit 11 selects a classification tab created based on the classification information (for example, a "non-defective product-defective product judgment" tab, a "non-defective product-defective product judgment" tab, a "defective product-non-defective product judgment" tab, or a "defective product-defective product Judgment” tab) is accepted by the input unit 14 (step S67). It should be noted that the selection is not limited to the selection of the classification tab by the user. For example, the "non-defective product-defective product judgment" tab may be set as the default tab.

The control unit 11 acquires a single or multiple images from the classified images according to the selected classification tab (step S68). The control unit 11 determines whether or not the selected classification tab is “defective product-defective product determination” or “defective product-defective product determination” (step S69). When the control unit 11 determines that the selected classification tab is “defective product-defective product determination” or “defective product-defective product determination” (YES in step S69), each image acquired according to the classification is displayed. An abnormality degree map for each image is generated based on the degree of abnormality for each pixel in the image (step S70).

The control unit 11 determines whether or not the selected classification tab is "defective product-defective product determination" (step S71). When the control unit 11 determines that the classification tab is “defective product-defective product determination” (YES in step S71), the abnormality degree map of each image corresponding to the “defective product-defective product determination” classification, The detected first detection result and second detection result are displayed on the display unit 15 (step S72), and the process ends. When the control unit 11 determines that the classification tab is not “defective product-defective product determination” (NO in step S71), the abnormality degree map of each image corresponding to the “defective product-good product determination” classification and the detected The first detection result is displayed on the display unit 15 (step S73), and the process ends.

If the control unit 11 determines that the selected classification tab is other than "defective product-defective product determination" or "defective product-defective product determination" (NO in step S69), the selected classification tab is "defective product- It is determined whether or not it is "defective product determination" (step S74). When the control unit 11 determines that the selected classification tab is “non-defective product determination” (YES in step S74), each image corresponding to the “non-defective product-defective product determination” classification and the detected number 2 detection results are displayed on the display unit 15 (step S75), and the process ends. When the control unit 11 determines that the selected classification tab is not “non-defective-defective product determination” (NO in step S74), the display unit 15 displays images corresponding to the “defective-defective product determination” classification. (Step S76), the process ends.

FIG. 32 is a flowchart showing a processing procedure for learning the abnormality detection model 173 using an overlooked image. An oversight image is an image in which an abnormality of an inspection object is not detected from the image based on the non-defective product learning model 171, but an abnormality of the inspection object is detected by an inspector's inspection (for example, visual inspection).

The control unit 11 of the computer 1 acquires a plurality of images to be verified from the storage unit 12 or the large-capacity storage unit 17 (step S81). Based on the non-defective product learning model 171, the control unit 11 detects a first detection result indicating the presence or absence of an abnormality in the inspection object corresponding to each acquired image (step S82). The control unit 11 acquires one image from a plurality of images to be verified (step S83).

The control unit 11 determines whether or not an abnormality is detected from the first detection result corresponding to the acquired image (step S84). When determining that an abnormality is detected (YES in step S84), the control unit 11 returns to the process of step S83 and acquires the next image. When the control unit 11 determines that no abnormality is detected (NO in step S84), the input unit 14 receives the input of the result of determination by the inspector as to whether or not the image is an overlooked image (step S85). . Note that the inspector may manually distribute the images to the storage unit 12 or the large-capacity storage unit 17 (folder).

The control unit 11 determines whether or not the image is an overlooked image based on the determination result input by the inspector (step S86). When the control unit 11 determines that the image is not an overlooked image (NO in step S86), the process returns to step S83. When the control unit 11 determines that the image is an overlooked image (YES in step S86), the input unit 14 receives information input by the inspector indicating that there is an abnormality in the inspection object corresponding to the overlooked image ( step S87). The information indicating the presence of abnormality includes the type of abnormality (for example, dent, stain, or scratch), the range (coordinates) of the area in which the abnormality is shown, and the like.

The control unit 11 determines whether or not the image is the last among the plurality of images to be verified (step S88). If the control unit 11 determines that the image is not the last image (NO in step S88), the process returns to step S83. When the control unit 11 determines that the image is the last one (YES in step S88), the training data in which the image and the information indicating the presence of an abnormality in the inspection object corresponding to the received image are associated with each other. (step S89).

The control unit 11 uses the created training data to learn (re-learn) the abnormality detection model 173 (step S90). Note that the learning process of the abnormality detection model 173 is the same as the learning process of the abnormality detection model 173 in the fifth embodiment, so the explanation is omitted. The control unit 11 updates the learning model management DB 172 of the large-capacity storage unit 17 using the learned abnormality detection model 173 (step S91), and terminates the process.

In addition to the process of learning the abnormality detection model 173 using the overlooked image, the abnormality detection model 173 can be learned using the overdetected image. An overdetected image is an image in which an abnormality of an inspection object is detected based on the non-defective product learning model 171 in the abnormality detection processing of the first embodiment, but an inspection by an inspector (for example, visual inspection) reveals an abnormality of the inspection object. is not detected. The computer 1 learns the abnormality detection model 173 using the acquired overdetected images and information indicating normality of the inspection object corresponding to the overdetected images input by the inspector as training data.

Specifically, the computer 1 acquires an overdetected image in which an abnormality of the inspection object is detected based on the non-defective product learning model 171 . The computer 1 receives a setting input for setting information (for example, a label such as "normal" or "no abnormality") indicating the normality of the inspection object corresponding to the acquired overdetected image. The computer 1 creates training data in which the acquired overdetected image and the information indicating the normality of the inspection object corresponding to the received overdetected image are associated with each other. By using overdetected images as training data, the more the system is continued, the more the anomaly detection model 173 is updated, and more accurate anomaly detection can be performed.

In addition to the processing described above, the non-defective product learning model 171 can be learned using an over-detected image obtained by detecting an abnormality in the inspection object based on the abnormality detection model 173 . Specifically, the computer 1 acquires an overdetected image in which an abnormality is detected based on the abnormality detection model 173 . The computer 1 uses an autoencoder, a VAE, or the like, for example, to learn the non-defective product learning model 171 using the acquired overdetected image as a normal image. Note that the learning process of the non-defective product learning model 171 is the same as that of the first embodiment, so the description is omitted. By executing such a learning process, it is possible to improve the detection accuracy by regarding an over-detected abnormal portion as being normal.

Furthermore, the computer 1 may execute the abnormality detection process based on the non-defective product learning model 171 after executing the abnormality detection process of the inspection object based on the abnormality detection model 173 . According to such a processing order, an abnormality that is not detected by the abnormality detection model 173 can be detected based on the non-defective product learning model 171 learned based on normal images.

According to this embodiment, by detecting the presence/absence of an abnormality in an inspection target using both the non-defective product learning model 171 and the abnormality detection model 173, it is possible to prevent overlooking of important findings in image inspection.

<Modification 1>
FIG. 33 is an explanatory diagram showing an example of a verification mode screen for detecting anomalies in an inspection object according to Modification 1. FIG. 27 are assigned the same reference numerals, and descriptions thereof are omitted. The verification mode screen includes a classification tab 18g, a graph display field 18i, and good/defective check boxes 18j.

The classification tab 18g is a tab for switching classification information consisting of a contrast relationship between the first detection result and the second detection result. As illustrated, the classification information includes non-defective-OK judgment, defective-NG judgment, overdetection and non-detection.

The non-defective-OK determination is a classification in which an inspection object corresponding to an image is actually determined to be non-defective and OK (positive). When it is determined that the first detection result is OK and the second detection result is OK, the image is classified as "non-defective product-OK determination". Defective product-NG determination is a classification in which an inspection object corresponding to an image is actually determined to be defective and NG (erroneous). If either or both of the first detection result and the second detection result are determined to be NG, the image is classified as "defective product-NG".

Overdetection is a classification in which an object to be inspected corresponding to an image is judged to be NG even though it is actually a non-defective product. If either one or both of the first detection result and the second detection result are judged to be NG, but the inspector judges them to be OK, the image is classified as "overdetected." Undetected is a classification in which an object to be inspected corresponding to an image is actually defective but has not been detected as defective. If an image is not detected as a defective product from both the first detection result and the second detection result, but is judged as a defective product by the inspector, the image is classified as "undetected".

The graph display column 18i is a display column for displaying a graph showing the distribution of non-defective product images or defective product images. The non-defective product/defective product check box 18j is a check box for accepting selection of a non-defective product image or a defective product image to be tabulated.

When the computer 1 receives the touch operation of the verification button 18f, the image of the non-defective product stored in the folder selected by the non-defective product folder selection button 18c and the defective product image stored in the folder selected by the defective product folder selection button 18d are displayed. Get an image. The computer 1 detects a first detection result and a second detection result indicating whether or not there is an abnormality in the inspection object corresponding to the acquired inspection image including the non-defective product image and the defective product image. It should be noted that the detection processing of the first detection result and the second detection result is the same as the processing described above, so the description is omitted.

The computer 1 classifies a plurality of inspection images according to the first detection result and the second detection result based on the classification information including non-defective product-OK judgment, defective product-NG judgment, over-detection and non-detection. The computer 1 generates an abnormality degree map of each inspection image based on the degree of abnormality of each pixel of each inspection image. The computer 1 displays the first detection result and the second detection result in the verification result display field 18h corresponding to the applicable classification tab 18g together with the abnormality degree map of each inspection image classified into the classification. In addition, the first detection result and the second detection result may be displayed together with the image for inspection without being limited to the degree of abnormality map.

Specifically, when an abnormality is detected from the first detection result, for example, the computer 1 displays the first detection result (for example, an unknown defect) indicating an abnormal location corresponding to the inspection image as an abnormality in a red anchor box. It may be superimposed on the degree map and displayed. Alternatively, the computer 1 may display an arbitrary closed curve (a continuous curve whose both ends are the same) for the region of the image in which the degree of abnormality of each pixel exceeds the threshold.

When an abnormality is detected from the second detection result, the computer 1 generates a second detection result indicating an abnormal location corresponding to each inspection image together with an abnormality degree map of each inspection image. It is displayed by color-coding corresponding to the type. For example, when a "hit" abnormality is detected, the second detection result including the certainty of the "hit" may be displayed in a yellow anchor box superimposed on the image at the "hit". Alternatively, when the "dirt" abnormality is detected, the second detection result including the degree of certainty of "dirt" may be superimposed on the image with a blue anchor box at the "dirt" location.

Also, information indicating a comparison between the first detection result and the second detection result can be displayed in text format. For example, an "OK" or "NG" label indicating whether the detection result is correct or not may be displayed on the screen together with the image. As illustrated, when both the first detection result and the second detection result are erroneous, the computer 1 compares "NG" indicating the first detection result with "NG" indicating the second detection result. indicate. When the first detection result is positive and the second detection result is incorrect, the computer 1 displays "NG" indicating the first detection result and "OK" indicating the second detection result in contrast. do.

The computer 1 generates a fluff showing the distribution of good product images or bad product images based on the first detection result and the second detection result. The graph includes an abnormality distribution graph and an area distribution graph exceeding the abnormality degree. The degree-of-abnormality distribution graph is a graph showing the distribution of non-defective product images or defective product images based on the degree of abnormality for each classification including non-defective product-OK determination, defective product-NG determination, overdetection, and non-detection. The area distribution graph is a graph showing the distribution of non-defective product images or defective product images based on the area exceeding the degree of abnormality for each classification including non-defective product-OK judgment, defective product-NG judgment, overdetection and non-detection. .

FIG. 33 illustrates a distribution graph of the degree of abnormality. The computer 1 accepts the selection of the non-defective product image or the defective product image for which the distribution of the degree of abnormality is to be tabulated by using the non-defective product/defective product check box 18j. Based on the detected first detection result and second detection result, the computer 1 generates a histogram showing the distribution of the degree of abnormality for the received non-defective product image or defective product image.

As shown, the horizontal axis of the histogram indicates the value of the degree of anomaly (Anomaly Value), and the vertical axis indicates the number of images (Count). The computer 1, based on the defective product image received by the non-defective product/defective product check box 18j, determines the non-defective product-OK judgment, the defective product-NG judgment, the over-detection and the non-existence The number of defective product images to be classified into each classification is counted according to classification information including detection. The computer 1 generates a histogram based on the number of defective product images counted for each classification. The computer 1 displays the generated histogram in the graph display field 18i. As shown in the drawing, for example, when the first detection result is determined to be OK (below the threshold of the degree of abnormality) and the second detection result is determined to be NG, the histogram is indicated by hatching upward to the right. For example, when the first detection result is determined to be OK or NG and the second detection result is determined to be OK, the histogram is indicated by lower right hatching. For example, when both the first detection result and the second detection result are determined to be OK, the histogram is shown without hatching.

Although FIG. 33 exemplifies the distribution graph of the degree of abnormality, it can also be applied to the distribution graph of the non-defective product image or the defective product image based on the area exceeding the degree of abnormality. In this case, the computer 1 may generate a histogram based on a set area threshold instead of the abnormality threshold.

Note that the computer 1 may calculate the overdetection rate or the non-detection rate based on the first detection result and the second detection result and display it on the screen (not shown). Specifically, the computer 1 counts the number of non-defective images and the number of overdetected images, and calculates the overdetection rate based on the counted number of nondefective images and the number of overdetected images. The computer 1 counts the number of defective product images and the number of undetected images, and calculates the non-detection rate based on the counted number of defective product images and the number of undetected images. The computer 1 displays the calculated over-detection rate and non-detection rate on the screen.

According to this modification, it is possible to display the first detection result and the second detection result according to different classification information (non-defective product-OK judgment, defective product-NG judgment, overdetection and non-detection).

The embodiments disclosed this time are illustrative in all respects and should be considered not restrictive. The scope of the present invention is indicated by the scope of the claims rather than the meaning described above, and is intended to include all modifications within the scope and meaning equivalent to the scope of the claims.

1 Information processing device (computer)
REFERENCE SIGNS LIST 11 control unit 12 storage unit 13 communication unit 14 input unit 15 display unit 16 reading unit 17 large-capacity storage unit 171 non-defective product learning model 172 learning model management DB
173 Abnormality detection model 174 Image number reference DB
1a portable storage medium 1b semiconductor memory 1P control program 2 imaging device

Claims (16)

accepts the choice of either learning mode or operational mode,
Acquire an image for learning including the inspection object when the learning mode is selected,
Accept settings of hyperparameters of the first learning model generated by unsupervised learning,
generating the first learning model that outputs a reconstructed image corresponding to the learning image when the acquired learning image is input based on the received hyperparameter;
Acquiring an image for inspection including the inspection object,
dividing the acquired inspection image into a plurality of overlapping small-piece images;
inputting the plurality of divided small piece images to the first learning model that has been trained, and outputting a reconstructed image of each small piece image from the first learning model;
calculating, for each small piece image, the degree of abnormality of each pixel in the plurality of divided small piece images and the reconstructed image corresponding to each of the small piece images output by the first learning model;
Aggregate the number of times the anomaly degree of each pixel calculated for each small piece image is added,
Based on the counted number of times, calculate the average anomaly degree for each pixel,
Receiving the setting of the threshold of the degree of abnormality for verifying the image for inspection,
A program for causing a computer to execute a process of displaying a detected abnormal image based on the calculated average degree of abnormality and the received threshold value of the degree of abnormality. identifying a plurality of sizes of the small piece image;
2. The program according to claim 1 , for generating a first learning model for outputting a reconstructed image corresponding to said image when an image is input for each size of said small piece image specified. Determining the maximum number of small piece images to be simultaneously input to the first learning model according to at least one of the hardware configuration of the computer, the hyperparameter of the first learning model, and the size of the small piece images. 3. The program according to claim 1 or 2 , which causes Calculating the degree of abnormality of each pixel between an image containing the inspection object and a reconstructed image corresponding to the image,
The program according to any one of claims 1 to 3 , which executes a process of detecting an abnormality based on the calculated degree of abnormality. 5. The program according to any one of claims 1 to 4 , wherein when it is determined that at least one of the degrees of abnormality of each pixel exceeds the threshold value of the degree of abnormality, it is detected as an abnormality. accepting a selection of a template image created based on a region of the inspection object;
based on the selected template image, extracting an image of the inspection target from the image and storing it in a predetermined image folder;
6. The program according to any one of claims 1 to 5 , causing a process of inputting the stored image to the first learning model. accepting a selection of a template image created based on a region of the inspection object;
Acquire a video containing the inspection object,
based on the selected template image, cutting out a plurality of images in which the inspection object is recognized from the moving image and storing them in a predetermined image folder;
7. The program according to any one of claims 1 to 6 , wherein the stored image is inputted to the first learning model. The program according to any one of claims 1 to 7 , which causes execution of a process of displaying an abnormal image detected based on the name of the first learning model, the threshold value of the degree of abnormality, and the first learning model. When the selection of the operation mode is accepted,
9. The program according to any one of claims 1 to 8 , for executing a process of accepting selection of a folder in which operation images including the learned first learning model and an inspection object are stored. Detecting whether or not there is an abnormality in the inspection object based on the degree of abnormality between the operation image stored in the folder and the reconstructed image output by the first learning model,
10. The program according to claim 9 , which causes execution of a process of outputting presence/absence of the detected abnormality. 11. The program according to claim 9 or 10 , which causes execution of a process of storing log data in which an abnormality of the inspection object is detected. Receiving input as to whether or not the abnormal image is correct,
If the input is incorrect, generating first training data with data indicating that the abnormal image is normal,
Receiving input as to whether or not the normal image is correct,
If the input is incorrect, generating second training data with data indicating that the normal image is abnormal;
generating a second learning model that outputs abnormality information corresponding to an image when an image is input based on the combination of the generated first training data and the second training data; 11. A program according to any one of 11 . Accepting selection of at least one of the learned first learning model or the second learning model;
13. The program according to claim 12 , which executes a process of detecting an abnormality in an image including an inspection target using the received learning model. Acquiring the inspection image, which is not detected as the abnormal image, from the inspection images;
Receiving input of information indicating that there is an abnormality in the acquired image for inspection,
acquiring a plurality of combinations of training data in which the inspection image and the received information indicating the existence of an abnormality are associated;
generating a second learning model that outputs information indicating whether or not there is an abnormality in the object to be inspected when an inspection image is input based on a plurality of combinations of the acquired training data; 11. A program according to any one of 11 . accepts the choice of either learning mode or operational mode,
Acquire an image for learning including the inspection object when the learning mode is selected,
Receiving settings of hyperparameters of a first learning model generated by unsupervised learning, and generating a reconstructed image corresponding to the learning image when the acquired learning image is input based on the received hyperparameter. generating the first learning model to be output;
Acquiring an image for inspection including the inspection object,
dividing the acquired inspection image into a plurality of overlapping small-piece images;
inputting the plurality of divided small piece images to the first learning model that has been trained, and outputting a reconstructed image of each small piece image from the first learning model;
calculating, for each small piece image, the degree of abnormality of each pixel in the plurality of divided small piece images and the reconstructed image corresponding to each of the small piece images output by the first learning model;
Aggregate the number of times the anomaly degree of each pixel calculated for each small piece image is added,
Based on the counted number of times, calculate the average anomaly degree for each pixel,
Receiving the setting of the threshold of the degree of abnormality for verifying the image for inspection,
An information processing method for executing a process of displaying a detected abnormal image based on the calculated average degree of abnormality and the received threshold value of the degree of abnormality. a first reception unit that receives a selection of either the learning mode or the operation mode;
a first acquiring unit that acquires a learning image including the inspection object when the learning mode is selected;
a second reception unit that receives settings of hyperparameters of the first learning model generated by unsupervised learning;
a generation unit that generates the first learning model that outputs a reconstructed image corresponding to the learning image when the acquired learning image is input based on the received hyperparameter;
a second acquisition unit that acquires an inspection image including the inspection target;
a division unit that divides the acquired inspection image into a plurality of mutually overlapping small piece images;
a control unit that inputs the plurality of divided small piece images to the first learning model that has been trained, and outputs a reconstructed image of each small piece image from the first learning model;
a first calculation unit for calculating, for each small piece image, the degree of abnormality of each pixel of the plurality of divided small piece images and the reconstructed image corresponding to each of the small piece images output by the first learning model;
a tallying unit that tallies the number of times that the degree of abnormality of each pixel calculated for each small piece image is added;
A second calculation unit that calculates the average degree of anomaly for each pixel based on the counted number of times;
a third reception unit that receives setting of an abnormality threshold for verifying the inspection image;
An information processing apparatus comprising: a display unit that displays an abnormal image detected based on the calculated average degree of abnormality and the received threshold value of the degree of abnormality.

Images