WO2023223790A1

WO2023223790A1 - Training device, training method, abnormality detection device, and abnormality detection method

Info

Publication number: WO2023223790A1
Application number: PCT/JP2023/016448
Authority: WO
Inventors: 貴一奥野
Original assignee: コニカミノルタ株式会社
Priority date: 2022-05-18
Filing date: 2023-04-26
Publication date: 2023-11-23

Abstract

Disclosed is a technology for effectively training a plurality of machine learning models cooperating with each other. A training device according to one aspect of the present disclosure comprises: a to-be-trained model storage unit that stores a second model for classifying a candidate abnormality location detected by a first model; a data augmentation unit that performs data augmentation on first training data for the first model to acquire second training data obtained through such augmentation as to correspond to the result of detection by the first model; and a training unit that trains the second model by using the second training data.

Description

Training device, training method, abnormality detection device and abnormality detection method

The present disclosure relates to a training device, a training method, an abnormality detection device, and an abnormality detection method.

Due to advances in machine learning technology such as deep learning, machine learning technology is being widely used in various technical fields. For example, machine learning technology has come to be used in visual inspections for quality control of products and the like. In a typical appearance inspection, an image of a product captured by a camera or the like is input to a machine learning model, and the machine learning model predicts whether there are any abnormalities in the product. For example, when an anomaly is detected in a product, the machine learning model identifies the abnormal location on the product (e.g., a rectangular area in an image, a bounding box, etc.) and the classification result indicating the type of anomaly (e.g., dirt, scratch, etc.). (transformation, etc.).

Japanese Patent Application Publication No. 2021-93004

In such detection of anomaly locations and classification results, two machine learning models can be used: an anomaly location detection model that detects anomaly locations, and an anomaly classification model that detects classification results for each detected anomaly location. . Generally, anomalies occur with low frequency, making it difficult and costly to collect the anomaly data required for training machine learning models. Accordingly, when multiple such cooperative machine learning models are trained individually, the same training dataset may be utilized. That is, a training data set including normal images showing normal products and abnormal images showing defective products can be used to train both the abnormal location detection model and the abnormality classification model. For example, a normal image for training is given a label indicating a "normal" classification result, and an abnormal image for training is given a rectangular area indicating an abnormal location and a label indicating the classification result of each abnormal location. Ru.

Typically, when a trained anomaly detection model receives an image of a product to be detected, it predicts a rectangular area indicating a candidate anomaly location of the product and an anomaly probability indicating the possibility of an anomaly at the location. , an image showing a rectangular area having an abnormality probability equal to or higher than a threshold value is output as an abnormal image. On the other hand, when the trained anomaly classification model receives an image containing a rectangular area output from the trained anomaly detection model, the trained anomaly classification model generates a classification result for each rectangular area indicating an abnormality candidate area (for example, normal, dirty, scratched, deformed, etc.). ).

In this case, the candidate anomaly locations predicted by the trained anomaly location detection model may increase or decrease depending on the threshold of the anomaly probability that is set. For example, when the threshold value is set relatively high, the number of abnormality candidate locations to be detected is reduced, and the probability that the detected abnormality candidate locations are abnormal increases, but the detection of abnormality candidate locations may occur. On the other hand, when the threshold value is set relatively low, the number of detected abnormality candidate locations increases and the number of missed detections decreases. However, the discrimination accuracy of an anomaly classification model that does not use such overdetection data as training data may decrease.

In view of the above problems, one objective of the present disclosure is to provide a technique for effectively training multiple machine learning models that cooperate.

One aspect of the present disclosure provides a training target model storage unit that stores a second model that classifies abnormality candidate locations detected by the first model, and a training target model storage unit that stores data for first training data of the first model. a data expansion unit that performs expansion and obtains second training data expanded to correspond to the detection results of the first model; and a training unit that trains the second model using the second training data. A training device comprising:

According to the present disclosure, multiple machine learning models that cooperate can be effectively trained.

FIG. 1 is a schematic diagram showing abnormality detection processing according to an embodiment of the present disclosure. 2A and 2B are schematic diagrams illustrating anomaly location detection and anomaly classification according to an embodiment of the present disclosure. FIG. 3 is a schematic diagram showing training processing and inference processing of an abnormal location detection model and an abnormal classification model according to an embodiment of the present disclosure. FIG. 4 is a schematic diagram illustrating data expansion according to one embodiment of the present disclosure. FIG. 5 is a schematic diagram illustrating a training device and an abnormality detection device according to an embodiment of the present disclosure. FIG. 6 is a block diagram showing the hardware configuration of a training device and an abnormality detection device according to an embodiment of the present disclosure. FIG. 7 is a block diagram showing the functional configuration of a training device according to an embodiment of the present disclosure. 8A-8D are diagrams illustrating augmented training data according to one embodiment of the present disclosure. FIG. 9 is a flowchart illustrating training processing according to an embodiment of the present disclosure. FIG. 10 is a block diagram showing the functional configuration of an abnormality detection device according to an embodiment of the present disclosure. FIG. 11 is a flowchart showing abnormality detection processing according to an embodiment of the present disclosure.

Hereinafter, embodiments of the present disclosure will be described with reference to the drawings.

In the following example, an anomaly is detected using a training device that trains an anomaly detection model and an anomaly classification model that operate in conjunction, and an anomaly detection model and an anomaly classification model trained by the training device. An abnormality detection device is disclosed.

[Summary]
In the anomaly detection process according to the following embodiment, as shown in FIG. An image is output from the abnormal location detection model 10. In the abnormal location detection model 10, for example, as shown in FIG. 2A, normal location candidates (anomaly undetected regions) and abnormal location candidates (abnormality detection regions) are separated. As shown in FIG. 1, the abnormal location detection image output from the abnormal location detection model 10 is generated by superimposing detected abnormal location candidates (for example, rectangular areas, bounding boxes, etc.) on the input image. Good too.

Next, the acquired abnormality detection image is input to the abnormality classification model 20, and the detection result indicating the classification result (for example, normal, dirt, flaw, deformation, etc.) of each abnormality candidate part of the abnormality detection image is generated. , is output from the anomaly classification model 20. The abnormality classification model 20 predicts the classification result of the abnormality location candidates detected by the abnormality detection model 10, for example, as shown in FIG. 2B. For example, label A may correspond to the abnormality type "deformation," label B may correspond to "normal," and label C may correspond to the abnormality type "stain."

The anomaly detection model 10 and the anomaly classification model 20 can typically be trained using the same training data set. Specifically, a training data set including normal images showing normal products and abnormal images showing defective products is used as training data for training both the abnormal location detection model 10 and the abnormality classification model 20. be prepared. For example, a normal image for training is given a label indicating a "normal" classification result. On the other hand, a rectangular region indicating an abnormal location and a label indicating a classification result (for example, dirt, scratch, deformation, etc.) of each abnormal location are added to the abnormal image for training.

When the trained abnormality detection model 10 receives an image of a product to be detected, it predicts a rectangular area indicating an abnormality candidate area of the product and an abnormality probability indicating the possibility of an abnormality in the area, An image showing a rectangular region with an abnormality probability of is output. On the other hand, when the trained abnormality classification model 20 receives an image including a rectangular region indicating an abnormality candidate location outputted from the trained abnormality detection model 10, the trained abnormality classification model 20 receives the classification result of each rectangular region (for example, normal, dirt, scratch, deformation, etc.).

The anomaly candidate locations predicted by the trained anomaly location detection model 10 can be increased or decreased depending on the threshold of the anomaly probability that is set. For example, when the threshold value is set relatively high, the number of abnormality candidate locations to be detected is reduced, and the probability that the detected abnormality candidate locations are abnormal increases, but the detection of abnormality candidate locations may occur. On the other hand, when the threshold value is set relatively low, the number of detected abnormality candidate locations increases and the number of missed detections decreases. However, since the anomaly classification model 20 is trained using the same training data set as the anomaly location detection model 10, it cannot deal with over-detected anomaly location detection images, and the discrimination accuracy of the anomaly classification model 20 may decrease.

In the following embodiments, data expansion is performed on the training data of the abnormality detection model 10, and training data expanded to correspond to the detection results of the abnormality detection model 10 is obtained. Then, an anomaly classification model 20 for classifying anomaly candidate locations over-detected by the anomaly location detection model 10 is trained using the expanded training data. Thereby, even if the abnormality candidate location is over-detected by the abnormality location detection model 10, the abnormality classification model 20 can appropriately classify the abnormality candidate location.

[Anomaly location detection model and anomaly classification model]
An abnormal location detection model 10 and an abnormality classification model 20 according to an embodiment of the present disclosure will be described with reference to FIGS. 3 to 6. For example, the anomaly detection model 10 is a machine learning model such as a neural network that detects anomaly candidate locations from an image showing a detection target, and the anomaly classification model 20 classifies anomaly candidate locations from an image showing the anomaly candidate locations. It may also be realized as a machine learning model such as a neural network.

FIG. 3 is a schematic diagram showing training processing and inference processing using the abnormal location detection model 10 and the abnormal classification model 20 according to an embodiment of the present disclosure. As shown in FIG. 3, the abnormal location detection model 10 is trained using a training data set that includes normal images of the detection target, and abnormal images that indicate the abnormal locations in the detection target and the classification results of the abnormal locations. For example, the illustrated training data is an abnormal image showing the appearance of a product on which a rectangular region labeled with the abnormality type "stain" is superimposed. When trained using such a training data set, the anomaly location detection model 10 outputs an anomaly location detection result indicating an anomaly candidate location for the received detection target data. The illustrated abnormality location detection result includes a plurality of abnormality candidate locations.

On the other hand, the anomaly classification model 20 is trained using expanded training data obtained by performing data expansion on the training data used to train the abnormal location detection model 10. Specifically, as shown in FIG. 4, expanded locations (for example, rectangular regions) are arranged corresponding to the positions of abnormal locations in the training data of the training data set. For example, rectangular areas may be randomly added to positions corresponding to abnormal locations in all training data in the training data set. Then, the classification result of the added rectangular area is determined based on the presence or absence of overlap between the rectangular area and the abnormal location in the training data. For example, if a rectangular region at least partially overlaps with an abnormal location in the training data, the classification result of the overlapping abnormal location is assigned to the rectangular region. For example, in the example of the expanded training data shown in FIG. 4, a rectangular region that at least partially overlaps with an abnormal location ("dirt") in the training data is labeled with the classification result "dirt". On the other hand, if the rectangular area does not overlap with an abnormal location in the training data, the rectangular area is classified as "normal."

The anomaly classification model 20 is trained using the training data expanded in this way. When the trained abnormality classification model 20 acquires the abnormality detection results inferred by the trained abnormality detection model 10, it infers abnormality detection results indicating the classification results of each abnormality candidate location.

The training target abnormality detection model 10 and the training target abnormality classification model 20 are trained by the training device 100 using the training data set. Specifically, as shown in FIG. 5, the training device 100 trains the anomaly detection model 10 and the anomaly classification model 20 using training data and expanded training data, respectively, and trains the trained anomaly detection model 10 and The trained anomaly classification model 20 is provided to the anomaly detection device 200.

On the other hand, the trained abnormality detection model 10 and the trained abnormality classification model 20 are used by the abnormality detection device 200 for abnormality detection processing. Specifically, upon receiving the detection target data, the abnormality detection device 200 inputs the detection target data into the trained abnormality location detection model 10 and obtains the abnormality candidate location detection data. Then, the anomaly detection device 200 inputs the acquired anomaly candidate location detection data to the trained anomaly classification model 20, and obtains an anomaly detection result for the detection target data.

Note that although the following embodiments will be described with a focus on the abnormality detection model 10 and the abnormality classification model 20, the present disclosure is not necessarily limited to the abnormality detection model 10 and the abnormality classification model 20; It may be applied to multiple machine learning models that work together. That is, the training device according to the present disclosure performs data augmentation on the training data of model A, obtains training data that has been extended to correspond to the output result of model A, and uses the extended training data to expand the training data of model A. Model B may be trained to perform specific processing on the output results output by. Further, when the inference device according to the present disclosure acquires data indicating an inference target, the inference device uses a trained model A that acquires an output result from the data and a trained model B that acquires a processing result for the inference target data. Then, the inference result for the data to be inferred may be obtained.

Here, the training device 100 and the abnormality detection device 200 may be realized by a computing device such as a server, a personal computer (PC), a smartphone, or a tablet, and have a hardware configuration as shown in FIG. 6, for example. It's okay. That is, each of the training device 100 and the abnormality detection device 200 includes a drive device 101, a storage device 102, a memory device 103, a processor 104, a user interface (UI) device 105, and a communication device 106 that are interconnected via bus B. .

Programs or instructions for realizing various functions and processes to be described later in the training device 100 and the abnormality detection device 200 may be stored in a removable storage medium such as a CD-ROM (Compact Disk-Read Only Memory) or a flash memory. good. When the storage medium is set in the drive device 101, a program or instruction is installed from the storage medium into the storage device 102 or the memory device 103 via the drive device 101. However, the program or instructions do not necessarily need to be installed from a storage medium, and may be downloaded from any external device via a network or the like.

The storage device 102 is realized by a hard disk drive or the like, and stores installed programs or instructions as well as files, data, etc. used to execute the programs or instructions.

The memory device 103 is realized by random access memory, static memory, etc., and when a program or instruction is started, reads the program or instruction, data, etc. from the storage device 102 and stores it. The storage device 102, the memory device 103, and the removable storage medium may be collectively referred to as a non-transitory storage medium.

The processor 104 may be realized by one or more CPUs (Central Processing Units), GPUs (Graphics Processing Units), processing circuits, etc. that may be configured from one or more processor cores, and may include memory. to device 103 Various functions and processes of the training device 100 and the abnormality detection device 200, which will be described later, are executed according to the stored programs, instructions, and data such as parameters necessary to execute the programs or instructions.

The user interface (UI) device 105 may include input devices such as a keyboard, mouse, camera, and microphone, output devices such as a display, speaker, headset, and printer, and input/output devices such as a touch panel. An interface between the device 100 and the abnormality detection device 200 is realized. For example, a user operates a GUI (Graphical User Interface) displayed on a display or a touch panel using a keyboard, a mouse, etc., and operates the training device 100 and the abnormality detection device 200.

The communication device 106 is realized by various communication circuits that perform wired and/or wireless communication processing with communication networks such as external devices, the Internet, LAN (Local Area Network), and cellular networks.

However, the hardware configuration described above is merely an example, and the training device 100 and the abnormality detection device 200 according to the present disclosure may be realized by any other suitable hardware configuration.

[Training device]
Next, a training device 100 according to an embodiment of the present disclosure will be described with reference to FIGS. 7 and 8. The training device 100 according to the present embodiment executes data expansion on the training data of the training data set used to train the abnormal location detection model 10, and trains the abnormality classification model 20 using the expanded training data. .

FIG. 7 is a block diagram showing the functional configuration of the training device 100 according to an embodiment of the present disclosure. As shown in FIG. 7, the training device 100 includes a training target model storage section 110, a data expansion section 120, and a training section 130. For example, one or more functional units of the training target model storage unit 110, the data extension unit 120, and the training unit 130 may be realized by one or more processors 104 executing one or more programs or instructions. .

The training target model storage unit 110 stores a second model that classifies the abnormality candidate locations detected by the first model. Specifically, the training target model storage unit 110 stores an abnormality classification model 20 that classifies the abnormality candidate locations detected by the abnormality location detection model 10 as a training target machine learning model. For example, the abnormality detection model 10 and/or the abnormality classification model 20 may be a neural network such as a convolutional neural network, but the abnormality detection model 10 and/or the abnormality classification model 20 according to the present disclosure are limited to this. However, it may be any other type of multiple machine learning models working together. Furthermore, when the training device 100 also trains the abnormal location detection model 10, the training target model storage unit 110 may also store the abnormal location detection model 10.

The data expansion unit 120 performs data expansion on the training data of the first model, and obtains training data expanded to correspond to the detection results of the first model. For example, the data extension unit 120 may extend the training data by arranging an extension location corresponding to the position of an abnormal location in the training data. Specifically, the data expansion unit 120 identifies abnormal locations from all the abnormal images of the training data set used to train the abnormal location detection model 10, and creates an abnormal location list from the rectangular area of the identified abnormal locations. may be configured. For example, the abnormality list may indicate the coordinates of each vertex of a plurality of rectangular areas on the XY plane.

Then, the data expansion unit 120 randomly selects one or more rectangular areas from the abnormal location list, and superimposes the selected one or more rectangular areas on the abnormal image to be expanded. Thereafter, the data expansion unit 120 determines that when the superimposed rectangular area at least partially overlaps with the abnormal location in the abnormal image (that is, when the IoU (Intersection of Union) between the rectangular area and the abnormal location is non-zero), The classification result of the superimposed rectangular area may be set as the classification result of the abnormal location. For example, if the classification result of the overlapping abnormal location is "dirt", the data expansion unit 120 may set the classification result of the overlapped rectangular area as "dirt". On the other hand, if the superimposed rectangular area does not overlap with the abnormal part of the abnormal image (that is, if the IoU between the rectangular area and the abnormal part is zero), the data expansion unit 120 classifies the classification result of the superimposed rectangular area as "normal". ” may also be set.

For example, when adding a rectangular area as shown in FIG. 8A to an abnormal image of the training data set based on the abnormal location list, the data expansion unit 120 adds a rectangular area such as that shown in FIG. Since there is no overlap with any location, the classification result may be set as "normal".

Alternatively, when the data expansion unit 120 adds a rectangular area as shown in FIG. 8B to the abnormal image of the training dataset based on the abnormal area list, the added rectangular area is a "stain" abnormal area. Since the classification result overlaps with that of "dirt", the classification result may be set as "dirt".

Alternatively, when adding a rectangular area as shown in FIG. 8C to the normal image of the training dataset based on the abnormal location list, the data expansion unit 120 classifies the added rectangular area as "normal". You can also set it as .

Note that the above-described addition of rectangular areas may be repeated. As shown in FIG. 8D, the data expansion unit 120 may repeatedly add a plurality of rectangular areas as shown in FIG. 8D to the abnormal image of the training dataset based on the abnormal location list, and add The classification result of the rectangular area may be set depending on whether or not there is an overlap between the rectangular area and the abnormal location of the abnormal image.

In this way, by adding rectangular areas using the anomaly area list created based on the training dataset, it is possible to add rectangular areas that may be detected by the anomaly area detection model 10, and The classification results of the rectangular areas can be automatically obtained depending on the presence or absence of overlap. This allows training data to be expanded efficiently. Furthermore, even if a relatively low threshold is applied to the abnormality probability in detecting an abnormality, that is, even if a normal place is likely to be detected as an abnormality candidate, the abnormality classification model 20 can be appropriately classified as "normal". Even if such an over-detected abnormality detection model 10 is applied, highly accurate abnormality detection can be achieved by using it in conjunction with the abnormality classification model 20.

Additionally, the data expansion unit 120 performs a minute position change and/or size change (for example, randomly set) to the rectangular area in the abnormality list, and superimposes the changed rectangular area on the training data. You may let them. This makes it possible for the data expansion unit 120 to increase variations in rectangular areas, and to cope with deterioration in accuracy due to positional shifts and the like.

The training unit 130 trains the second model using the expanded training data. Specifically, the training unit 130 trains the anomaly classification model 20 using the training data expanded by the data expansion unit 120. For example, when the anomaly classification model 20 is implemented as a convolutional neural network, the training unit 130 inputs the expanded training images to the convolutional neural network to be trained, and combines the output results from the convolutional neural network to be trained and the expanded training images. The parameters of the convolutional neural network to be trained may be updated according to the error backpropagation method according to the error with the training image. The training unit 130 repeats the parameter update process, such as executing the above-described parameter update process on all expanded training data, until a predetermined termination condition is satisfied. When a predetermined termination condition is satisfied, the training unit 130 may end the training of the anomaly classification model 20 to be trained, and may provide the obtained anomaly classification model 20 to the anomaly detection device 200 as a trained model.

The training unit 130 may further train the abnormal location detection model 10. Specifically, the training target model storage unit 110 further stores the anomaly location detection model 10 as the training target, and the training unit 130 trains the anomaly location detection model 10 using unexpanded training data of the training data set. It's okay.

For example, in order to avoid non-detection, the training unit 130 may train the abnormal location detection model 10 and/or the abnormality classification model 10 using a loss function that more severely punishes non-detection. Such a loss function may be defined as follows, for example.
loss=(1-flag)*loss0+flag*(0.1*loss1)
Here, the variable flag is "0" or "1", and if the rectangular area to be calculated is normal, the flag is set to "0", and if the rectangular area to be calculated is abnormal, the flag is set to " It may be set to 1”. While this can reduce the possibility that an abnormal location is not detected by the abnormal location detection model 10, the overall error classification accuracy rate may decrease. Classification learning that allows undetected results allows tuning without constraining the loss function, and can maximize the overall classification accuracy rate.

Although the present embodiment has been described with a focus on the anomaly detection model 10 and/or the anomaly classification model 20, the machine learning model to be trained according to the present disclosure is not limited to this, and is trained using the same training data set. It may be similarly applied to multiple machine learning models that are linked together.

[Training process]
Next, with reference to FIG. 9, a training process according to an embodiment of the present disclosure will be described. The training process is executed by the training device 100 described above, and more specifically, one or more processors 104 of the training device 100 execute one or more programs or instructions stored in one or more memory devices 103. It may be realized by doing. FIG. 9 is a flowchart illustrating training processing according to an embodiment of the present disclosure.

As shown in FIG. 9, in step S101, the training device 100 performs data expansion on the training data of the abnormal location detection model 10. Specifically, the training device 100 may extend the training data by arranging an extension location (for example, a rectangular area) corresponding to the position of the abnormal location in the training data. Then, the training device 100 may classify the expanded locations depending on whether or not there is an overlap between the abnormal location and the extended location in the training data. For example, when the abnormal part of the training data and the extended part overlap at least partially, the training device 100 classifies the extended part so as to match the classification result of the abnormal part, and the training device 100 classifies the extended part so as to match the classification result of the abnormal part. If there is no overlap, the expanded location may be classified as normal.

That is, the training device 100 adds a rectangular region to the abnormal image of the training dataset, and when the added rectangular region at least partially overlaps with an abnormal location in the abnormal image, the training device 100 uses the classification result of the rectangular region as the abnormal location. It may also be set in the classification results. After determining the classification result of the rectangular area, the training device 100 may extend the training data by superimposing the rectangular area to which the set classification result is given on the training data. On the other hand, the training device 100 may add a rectangular region to the normal image of the training dataset. In this case, the training device 100 may expand the training data by superimposing the rectangular region classified as normal on the training data.

In step S102, the training device 100 trains the anomaly classification model 20 using the expanded training data. For example, when the anomaly classification model 20 is implemented as a convolutional neural network, the training device 100 inputs training images that have been expanded to the convolutional neural network that is the training target, and combines the output results from the convolutional neural network that is the training target with the expanded training images. The parameters of the convolutional neural network to be trained may be updated according to the error backpropagation method according to the error with the training image. The training device 100 repeats the parameter update process until a predetermined termination condition is satisfied. When a predetermined termination condition is satisfied, the training device 100 may end the training of the anomaly classification model 20 to be trained, and may provide the obtained anomaly classification model 20 to the anomaly detection device 200 as a trained model.

Additionally, the training device 100 may train the abnormal location detection model 10 using unenhanced training data. For example, when the anomaly detection model 10 is realized as a convolutional neural network, the training device 100 inputs a training image to the convolutional neural network to be trained, and combines the output result from the convolutional neural network to be trained with the training image. Depending on the error, parameters of the convolutional neural network to be trained may be updated according to the error backpropagation method. The training device 100 repeats the parameter update process until a predetermined termination condition is satisfied. When a predetermined termination condition is satisfied, the training device 100 may end the training of the abnormal location detection model 10 to be trained, and may provide the acquired abnormal location detection model 10 to the abnormality detection device 200 as a trained model.

[Anomaly detection device]
Next, with reference to FIG. 10, an abnormality detection device 200 according to an embodiment of the present disclosure will be described. The anomaly detection device 200 uses the trained anomaly location detection model 10 and the trained anomaly classification model 20 acquired from the training device 100 to perform anomaly detection on an image showing a product or the like to be detected.

As shown in FIG. 10, the anomaly detection device 200 includes a trained model storage section 210 and an anomaly detection section 220.

The trained model storage unit 210 stores a trained first model that detects abnormality candidate locations from images showing detection targets, and a trained second model that classifies abnormality candidate locations from detection target images showing abnormality candidate locations. Store the model of. Here, the first model is an anomaly location detection model 10 that detects an anomaly candidate location from an image showing a detection target, and the second model is an anomaly classification model that classifies an anomaly candidate location from an image showing an anomaly candidate location. Model 20 may also be used.

Upon acquiring an image showing a detection target, the anomaly detection unit 220 uses the trained first model and the trained second model to classify the abnormality candidate location and the abnormality candidate location in the detection target image. Get the results. Specifically, upon acquiring an image to be inferred, the anomaly detection unit 220 inputs the image to be inferred to the trained abnormality detection model 10, and extracts abnormality candidate locations in the input image from the trained abnormality detection model 10. Obtain an abnormal point detection image showing the For example, if a relatively low threshold is set to prevent abnormalities from not being detected, and a rectangular area exhibiting an abnormality probability greater than or equal to the set threshold is detected, the abnormality detection unit 220 detects a relatively large number of abnormalities. Potential abnormalities can be detected.

Upon detecting an abnormality candidate location, the anomaly detection unit 220 inputs the detected abnormality location detection image to the trained abnormality classification model 20 and obtains the classification result of each abnormality candidate location in the input image from the trained abnormality classification model 20. . For example, if the abnormality candidate location is normal, the abnormality detection unit 220 classifies the abnormality candidate location as “normal”, and if the abnormality candidate location is “dirt”, “flaw”, “deformation”, etc. If a type of abnormality is indicated, a detection result is obtained that classifies the abnormality candidate location into the corresponding abnormality type.

[Anomaly detection processing]
Next, with reference to FIG. 11, abnormality detection processing according to an embodiment of the present disclosure will be described. The abnormality detection process is executed by the above-mentioned abnormality detection device 200, and more specifically, one or more processors 104 of the abnormality detection device 200 execute one or more programs or programs stored in one or more memory devices 103. This may be accomplished by executing instructions. FIG. 11 is a flowchart showing abnormality detection processing according to an embodiment of the present disclosure.

As shown in FIG. 11, in step S201, the abnormality detection device 200 acquires an image showing the detection target. Specifically, the abnormality detection device 200 acquires an image of the appearance of a product to be detected, captured by an imaging means such as a camera.

In step S202, the anomaly detection device 200 uses the trained abnormality detection model 10 and the trained abnormality classification model 20 to obtain abnormality candidate locations and their classification results in the acquired detection target image. Specifically, the anomaly detection device 200 first inputs the acquired image into the trained abnormality detection model 10, and obtains an abnormality detection image indicating abnormality candidate locations in the input image from the trained abnormality detection model 10. . Next, the anomaly detection device 200 inputs the acquired abnormality detection image to the trained abnormality classification model 20, and obtains a detection result indicating the classification result of each abnormality candidate location in the input image from the trained abnormality classification model 20. .

In addition, the following additional notes are further disclosed regarding the above description.
(Additional note 1)
a training target model storage unit that stores a second model that classifies the abnormality candidate locations detected by the first model;
a data expansion unit that executes data expansion on first training data of the first model and obtains second training data expanded to correspond to the detection result of the first model;
a training unit that trains the second model using the second training data;
A training device with.
(Additional note 2)
The training device according to supplementary note 1, wherein the data expansion unit generates the second training data by arranging an expansion location corresponding to a position of an abnormal location in the first training data.
(Additional note 3)
The training device according to supplementary note 2, wherein the data expansion unit classifies the expanded location depending on whether or not there is an overlap between an abnormal location in the first training data and the expanded location.
(Additional note 4)
When the abnormal location of the first training data and the extended location at least partially overlap, the data expansion unit classifies the extended location so as to match the classification result of the abnormal location; The training device according to supplementary note 3, wherein if an abnormal location in the training data and the expanded location do not overlap, the expanded location is classified as normal.
(Appendix 5)
The first model is an anomaly location detection model that detects an anomaly candidate location from an image showing a detection target,
The training device according to any one of Supplementary Notes 1 to 4, wherein the second model is an abnormality classification model that classifies the abnormality candidate location from an image showing the abnormality candidate location.
(Appendix 6)
performing data augmentation on first training data of a first model, and obtaining second training data augmented to correspond to the detection results of the first model;
training a second model for classifying abnormality candidate locations detected by the first model using the second training data;
A training method performed by a computer.
(Appendix 7)
A trained model that stores a trained first model that detects an abnormality candidate location from an image showing a detection target, and a trained second model that classifies the abnormality candidate location from an image showing the abnormality candidate location. a storage section;
When an image indicating the detection target is acquired, the trained first model and the trained second model are used to classify the abnormality candidate location and the abnormality candidate location in the detection target image. an anomaly detection unit that obtains the
has
The trained second model performs data augmentation on the first training data of the trained first model, and is expanded to correspond to the detection result of the trained first model. The anomaly detection device is trained using the second training data.
(Appendix 8)
When an image showing a detection target is acquired, a trained first model that detects an abnormality candidate location from the image showing the detection target, and a trained first model that classifies the abnormality candidate location from the image showing the abnormality candidate location. 2, to obtain an abnormality candidate location in the detection target image and a classification result of the abnormality candidate location,
The trained second model performs data augmentation on the first training data of the trained first model, and is expanded to correspond to the detection result of the trained first model. a computer-implemented anomaly detection method, the computer-implemented anomaly detection method being trained on second training data;

Although the embodiments of the present disclosure have been described in detail above, the present disclosure is not limited to the specific embodiments described above, and various modifications may be made within the scope of the gist of the present disclosure described in the claims. - Can be changed.

The disclosure contents of the specification, drawings, and abstract included in the Japanese application No. 2022-081576 filed on May 18, 2022 are all incorporated into the present application.

10 Anomaly location detection model 20 Anomaly classification model 100 Training device 110 Training target model storage section 120 Data extension section 130 Training section 200 Anomaly detection device 210 Trained model storage section 220 Anomaly detection section

Claims

a training target model storage unit that stores a second model that classifies the abnormality candidate locations detected by the first model;
a data expansion unit that executes data expansion on first training data of the first model and obtains second training data expanded to correspond to the detection result of the first model;
a training unit that trains the second model using the second training data;
A training device with.
The training device according to claim 1, wherein the data extension unit generates the second training data by arranging an extension location corresponding to a position of an abnormal location in the first training data.
The training device according to claim 2, wherein the data expansion unit classifies the expanded location depending on whether or not there is an overlap between an abnormal location in the first training data and the expanded location.
When the abnormal location of the first training data and the extended location at least partially overlap, the data expansion unit classifies the extended location so as to match the classification result of the abnormal location; 4. The training device according to claim 3, wherein if an abnormal location in the training data and the expanded location do not overlap, the expanded location is classified as normal.
The first model is an anomaly location detection model that detects an anomaly candidate location from an image showing a detection target,
The training device according to any one of claims 1 to 4, wherein the second model is an abnormality classification model that classifies the abnormality candidate location from an image showing the abnormality candidate location.
performing data augmentation on first training data of a first model, and obtaining second training data augmented to correspond to the detection results of the first model;
training a second model for classifying abnormality candidate locations detected by the first model using the second training data;
A training method performed by a computer.
A trained model that stores a trained first model that detects an abnormality candidate location from an image showing a detection target, and a trained second model that classifies the abnormality candidate location from an image showing the abnormality candidate location. a storage section;
When an image indicating the detection target is acquired, the trained first model and the trained second model are used to classify the abnormality candidate location and the abnormality candidate location in the detection target image. an anomaly detection unit that obtains the
has
The trained second model performs data augmentation on the first training data of the trained first model, and is expanded to correspond to the detection result of the trained first model. The anomaly detection device is trained using the second training data.
When an image showing a detection target is acquired, a trained first model that detects an abnormality candidate location from the image showing the detection target, and a trained first model that classifies the abnormality candidate location from the image showing the abnormality candidate location. 2, to obtain an abnormality candidate location in the detection target image and a classification result of the abnormality candidate location,
The trained second model performs data augmentation on the first training data of the trained first model, and is expanded to correspond to the detection result of the trained first model. a computer-implemented anomaly detection method, the computer-implemented anomaly detection method being trained on second training data;