WO2020255224A1

WO2020255224A1 - Abnormality detection device, learning device, abnormality detection method, learning method, abnormality detection program, and learning program

Info

Publication number: WO2020255224A1
Application number: PCT/JP2019/023973
Authority: WO
Inventors: 崇之梅田; 慎吾安藤; 淳嵯峨田
Original assignee: 日本電信電話株式会社
Priority date: 2019-06-17
Filing date: 2019-06-17
Publication date: 2020-12-24

Abstract

A mask processing unit (60) generates, for each cell obtained by dividing an input image, a mask image in which the cell is masked. A restoration unit (64) receives the mask image as an input, generates a restored image for each cell using a pre-learned restoration model for image restoration, and acquires a restored cell image that is a cell in the restored image. A combination unit (66) generates a combined image by combining the restored cell images acquired for the respective cells. An abnormality detection unit (68) compares the combined image with the input image and detects an abnormal part.

Description

Anomaly detection device, learning device, anomaly detection method, learning method, anomaly detection program, and learning program

The disclosed technology relates to an abnormality detection device, a learning device, an abnormality detection method, a learning method, an abnormality detection program, and a learning program.

Anomaly detection is an important technology in modern industry. Its applications are wide-ranging, such as visual inspection of products, deterioration detection of industrial machines themselves, and deterioration detection of various infrastructures.

Although abnormal conditions have been detected manually by old visitors, with the recent advancement of machine learning, abnormality detection by machine learning has been proposed from images taken of inspection targets.

In abnormality detection based on machine learning, a large amount of abnormal state data is generally required, but it is difficult to secure data due to the low frequency of abnormal state occurrence.

For this problem, a technique for reducing abnormal state data required for learning data has been proposed (Patent Documents 1 and 2).

JP 2016-110290 JP-A-2018-81442

A major issue in the prior art is that a certain amount of abnormal state data is still required. It is necessary to enumerate various abnormal states during model learning, but in the real world, abnormalities that were not initially expected may occur. It is difficult to deal with this with the prior art.

The disclosed technology was made in view of the above points, and is an abnormality detection device, a learning device, an abnormality detection method, a learning method, and an abnormality detection capable of detecting an abnormality location without requiring abnormality state data. The purpose is to provide programs and learning programs.

The first aspect of the present disclosure is an abnormality detection device, in which a mask processing unit that generates a mask image that masks the cell for each cell in which the input image is divided, and the mask image as input for each cell. , A restoration unit that generates a restoration image using a pre-learned restoration model for restoring an image and acquires a restoration cell image that is the cell in the restoration image, and the restoration cell acquired for each cell. It is configured to include a joining unit that generates a combined image in which images are combined, and an abnormality detecting unit that compares the combined image with the input image and detects an abnormal portion.

A second aspect of the present disclosure is a learning device, in which a mask processing unit that generates a mask image that masks the cell for each cell in which a learning image representing a normal state is divided, and a mask for each cell. A restored image is generated by using an image as an input and a restored model for restoring the image, a restored cell image which is the cell in the restored image is acquired, and the restored cell image and the learning are used for each cell. It is configured to include a learning unit that learns the restoration model so that the cell of the image matches.

A third aspect of the present disclosure is an abnormality detection method, in which the mask processing unit generates a mask image masking the cell for each cell in which the input image is divided, and the restoration unit generates the mask image for each cell. Using the mask image as input, a restored image is generated using a pre-learned restoration model for restoring the image, the restored cell image which is the cell in the restored image is acquired, and the joining portion is for each cell. A combined image is generated by combining the restored cell images acquired in the above, and the abnormality detection unit compares the combined image with the input image and detects an abnormal portion.

A fourth aspect of the present disclosure is a learning method, in which the mask processing unit generates a mask image masking the cell for each cell in which the learning image representing the normal state is divided, and the learning unit generates the cell. Each time, the mask image is input, a restored image is generated using a restored model for restoring the image, a restored cell image which is the cell in the restored image is acquired, and the restored cell is obtained for each cell. The restoration model is trained so that the image and the cell of the training image match.

A fifth aspect of the present disclosure is an abnormality detection program, in which a mask image masking the cell is generated for each cell in which the input image is divided, and the mask image is input to each cell to restore the image. A restored image is generated, a restored cell image which is the cell in the restored image is acquired, and a combined image obtained by combining the restored cell images acquired for each cell is obtained. It is a program for causing a computer to generate and compare the combined image with the input image to detect an abnormal portion.

A sixth aspect of the present disclosure is a learning program, in which a mask image masking the cell is generated for each cell in which a learning image representing a normal state is divided, and the mask image is input to each cell. , A restored image is generated using a restored model for restoring an image, a restored cell image which is the cell in the restored image is acquired, and the restored cell image and the cell of the learning image are obtained for each cell. Is a program for causing a computer to learn the restoration model so as to match.

According to the disclosed technology, it is possible to detect an abnormal part without requiring abnormal state data.

It is a figure for demonstrating the method of learning the restoration model. It is a figure for demonstrating the method of detecting an abnormal part. It is a schematic block diagram of an example of a computer functioning as a learning device and an abnormality detection device of this embodiment. It is a block diagram which shows the structure of the learning apparatus of this embodiment. It is a block diagram which shows the structure of the abnormality detection device of this embodiment. It is a flowchart which shows the learning processing routine of the learning apparatus of this embodiment. It is a flowchart which shows the abnormality detection processing routine of the abnormality detection apparatus of this embodiment.

Hereinafter, an example of the embodiment of the disclosed technology will be described with reference to the drawings. The same reference numerals are given to the same or equivalent components and parts in each drawing. In addition, the dimensional ratios in the drawings are exaggerated for convenience of explanation and may differ from the actual ratios.

<Outline of this embodiment>
In the present embodiment, a learning model representing a normal state is divided into a plurality of cells, and a restoration model for restoring a masked image of each cell to the original state is learned.

At the time of the test, the input image is similarly divided into a plurality of cells, and the image masking each cell is input to the learned restoration model. By combining the restored cell images, which are the masked parts of the restored image output from the restored model, and taking the difference from the input image, the part with a large difference is detected as an abnormal part.

The processing during learning and the processing during testing will be explained in detail below.

First, the teacher data required for learning the restoration model will be described with reference to FIG.

The learning image I is an image showing a normal state. This will be divided into an arbitrary number of cells. Here, a case where the number of grids = 4 and the cells are divided into 16 cells (= 4 * 4) will be described.

M ₀ to M _n are masks, and only the cells to be masked are 1 and the others are 0.

G ₀ to G _n are mask images in which each cell is masked. The mask image is, for example, an image in which the cell portion is filled. However, n = 16.

The mask images G ₀ to G _n are input to the restoration model R, and the obtained outputs are the restoration images C ₀ to C _n , respectively.

The restoration model is a deep neural network that reconstructs the input image, for example, U-Net shown in Non-Patent Document 1 and Dilated Conv. It is preferable to use a network structure using layers.

The partial images L ₀ to L _n are obtained by copying 8 cells in the vicinity of the restored cell from the learning image I and combining the restored cell image with the center cell.

The partial images L _0'to L _n'are images having the same position and size as the partial images L ₀ to L _n , but the center cell is not a restored cell image but a copy from the cell of the learning image I. is there.

The combined image A is a restored image having the same size as the learning image I, in which the restored cell images are combined.

The partial images L _{0 ~} L _n and combined image A, input to the first discriminator F _L and a second discriminator F _A which is constructed similarly to the GAN discriminator shown in Non-Patent Document 1. Here, the first classifier _FL takes the combined image A as an input and discriminates whether or not it is a true image. The second discriminator F _A receives as input each of the partial images L _{0 ~} L _n, identifying respectively whether the true image.
Liu Y. et.al., "Deep Blind Image Inpainting", Internet Search <URL: https://arxiv.org/pdf/1712.09078.pdf> Yu J. et.al., "Generative Image Inpainting with Contextual Attention", Internet Search <URL: http://openaccess.thecvf.com/content_cvpr_2018/papers/Yu_Generative_Image_Inpainting_CVPR_2018_paper.pdf> Ian J. Goodfellow et al., "Generative Adversarial Nets", Internet Search <URL: http://datascienceassn.org/sites/default/files/Generative%20Adversarial%20Nets.pdf>

Loss function L _R on restoring model R is as follows. * Denotes multiplication for each element, as an object only the portion masked, calculates the loss function L _R comprising matching degree for each pixel.

First discriminator _{F L,} the loss function _L L for the second discriminator _{F A,} the _{L A} is as follows.

The loss function _L _R, L L, learns recovery model R to optimize _{L A,} first discriminator _{F L,} and a second discriminator _{F A.} As a result, the masked portion is restored, and the restoration model R is learned so that the restored cell image becomes natural with respect to the image of the peripheral portion and the combined image becomes natural.

Next, using the restoration model R obtained by learning, the processing when actually detecting an abnormality will be described with reference to FIG.

For any input image I, mask images G ₀ to G _n for each cell are input to the restoration model R as in the case of learning, and the restored cell images extracted from the output image are combined to form a combined image. _{Get A.} At this time, even if the cell represents an abnormal state, it is restored as representing a normal state in the restored cell image.

Then, as shown in the following equation, the abnormality detection map H is obtained by taking the difference between the input image I and the combined image A.
H = IA

Here, the combined image A is all restored to the normal system including the abnormal part by the restoration model R. Therefore, by taking the difference from the input image I including the abnormal part, ideally, the value other than the abnormal part becomes 0 and the value of the abnormal part becomes 1, and the abnormal part can be detected.

<Structure of learning device according to this embodiment>
FIG. 3 is a block diagram showing a hardware configuration of the learning device 10 of the present embodiment.

As shown in FIG. 3, the learning device 10 includes a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, a storage 14, an input unit 15, a display unit 16, and a communication interface (Random Access Memory) 13. It has an I / F) 17. Each configuration is communicably connected to each other via a bus 19.

The CPU 11 is a central arithmetic processing unit that executes various programs and controls each part. That is, the CPU 11 reads the program from the ROM 12 or the storage 14, and executes the program using the RAM 13 as a work area. The CPU 11 controls each of the above configurations and performs various arithmetic processes according to the program stored in the ROM 12 or the storage 14. In the present embodiment, the ROM 12 or the storage 14 stores a learning program for learning the restoration model. The learning program may be one program, or may be a group of programs composed of a plurality of programs or modules.

ROM 12 stores various programs and various data. The RAM 13 temporarily stores a program or data as a work area. The storage 14 is composed of an HDD (Hard Disk Drive) or an SSD (Solid State Drive), and stores various programs including an operating system and various data.

The input unit 15 includes a pointing device such as a mouse and a keyboard, and is used for performing various inputs.

The display unit 16 is, for example, a liquid crystal display and displays various types of information. The display unit 16 may adopt a touch panel method and function as an input unit 15.

The communication interface 17 is an interface for communicating with other devices, and for example, standards such as Ethernet (registered trademark), FDDI, and Wi-Fi (registered trademark) are used.

Next, the functional configuration of the learning device 10 will be described. FIG. 4 is a block diagram showing an example of the functional configuration of the learning device 10.

The learning device 10 can be represented by a configuration including a learning data storage unit 20, a mask processing unit 22, and a learning unit 24.

The learning data storage unit 20 stores a plurality of learning images representing a normal state.

The mask processing unit 22 generates a mask image in which only the cell is masked for each cell in which the learning image is divided for each of the plurality of learning images.

The learning unit 24 generates a restored image for each of the plurality of learning images by inputting a mask image masking the cell for each cell and using the restoration model, and the restoration cell which is the cell in the restoration image. An image is acquired, and a combined image is generated by combining the restored cell images acquired for each cell.

Learning unit 24, for each of a plurality of learning images, the loss function L _R, L _L, by optimizing the L _A, and the cells restored the cell image and learning images per cell match and , The combined image is identified as a true image by the first classifier, and the partial image in which the cell of the learning image is replaced with the restored cell image for each cell is the true image by the second classifier. The reconstruction model, the first classifier, and the second classifier are trained so that they can be identified as.

<Configuration of abnormality detection device according to this embodiment>
FIG. 3 is a block diagram showing a hardware configuration of the abnormality detection device 50 of the present embodiment.

As shown in FIG. 3, the abnormality detection device 50 has a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, a storage 14, and an input unit, similarly to the learning device 10. It has a display unit 16 and a communication interface (I / F) 17. In the present embodiment, the ROM 12 or the storage 14 stores an abnormality detection program for detecting an abnormality portion of the input image.

Next, the functional configuration of the abnormality detection device 50 will be described. FIG. 5 is a block diagram showing an example of the functional configuration of the abnormality detection device 50.

Functionally, as shown in FIG. 5, the abnormality detection device 50 includes a mask processing unit 60, a model storage unit 62, a restoration unit 64, a coupling unit 66, and an abnormality detection unit 68.

The mask processing unit 60 generates a mask image in which only the cell is masked for each cell in which the input image is divided.

The model storage unit 62 stores the restored model learned by the learning device 10.

The restoration unit 64 inputs a mask image for each cell, generates a restoration image using the restoration model, and acquires the restoration cell image which is the cell in the restoration image.

The combining unit 66 generates a combined image in which the restored cell images acquired for each cell are combined.

The abnormality detection unit 68 compares the combined image with the input image and detects the abnormality portion. For example, a region consisting of pixels whose pixel value difference is equal to or greater than a threshold value is detected as an abnormal portion.

<Operation of the learning device according to this embodiment>
Next, the operation of the learning device 10 will be described. FIG. 6 is a flowchart showing the flow of the learning process by the learning device 10. The learning process is performed by the CPU 11 reading the learning program from the ROM 12 or the storage 14, expanding it into the RAM 13 and executing it. Further, a learning image in a normal state is stored in advance in the learning data storage unit 20 of the learning device 10.

In step S100, the CPU 11, as the mask processing unit 22, generates a mask image in which only the cell is masked for each cell in which the learning image is divided for each of the plurality of learning images.

In step S102, the CPU 11, as the learning unit 24, generates a restored image by using the restoration model, inputting a mask image masking the cell for each cell for each of the plurality of learning images.

In step S104, as the learning unit 24, the CPU 11 acquires the restored cell image, which is the cell in the restored image, for each cell of each of the plurality of learning images, and combines the acquired restored cell images for each cell. Generate a combined image.

In step S106, CPU 11 has a learning section 24, for each of a plurality of learning images, so as to optimize the loss function _L _R, L L, the _{L A,} recovery model, first discriminator, and a second Learn the classifier. As a result, the cells of the restored cell image and the learning image match each cell, the combined image is identified as a true image by the first classifier, and the learning image is identified for each cell. The restored model, the first classifier, and the second classifier are trained so that the partial image in which the cell is replaced with the restored cell image is identified by the second classifier as the true image.

In step S108, the CPU 11 determines whether or not to end the repetition, and if it is determined not to end the repetition, the process returns to step S100. On the other hand, when the CPU 11 determines that the repetition is finished, the CPU 11 ends the learning processing routine.

<Operation of the abnormality detection device according to this embodiment>
Next, the operation of the abnormality detection device 50 will be described.

FIG. 7 is a flowchart showing the flow of abnormality detection processing by the abnormality detection device 50. The abnormality detection process is performed by the CPU 11 reading the abnormality detection program from the ROM 12 or the storage 14, expanding it into the RAM 13 and executing it. Further, the model storage unit 62 of the abnormality detection device 50 stores the restored model learned by the learning device 10. Further, the input unit 15 inputs an input image to be detected to the abnormality detection device 50.

In step S110, the CPU 11, as the mask processing unit 60, generates a mask image in which only the cell is masked for each cell in which the input image is divided.

In step S112, the CPU 11 generates a restored image as the restoration unit 64 by inputting a mask image for each cell and using the restoration model, and acquires the restoration cell image which is the cell in the restoration image.

In step S114, the CPU 11 generates a combined image in which the restored cell images acquired for each cell are combined as the combining unit 66.

In step S116, the CPU 11, as the abnormality detection unit 68, compares the combined image with the input image, detects the abnormality portion, displays it on the display unit 16, and ends the abnormality detection processing routine.

As described above, the learning device according to the present embodiment generates a mask image in which the cells are masked for each cell in which the learning image representing the normal state is divided, and the mask image is input to each cell as an image. A restored image is generated using the restored model for restoring the image, and the restored cell image, which is a cell in the restored image, is acquired. The learning device generates a combined image by combining the restored cell images acquired for each cell. In the learning device, the restored cell image and the learning image match each cell, the combined image is identified as a true image by the first classifier, and the learning device is used for learning cell by cell. The restoration model, the first classifier, and the second classifier are trained so that the image in which the cell of the image is replaced with the restored cell image is identified as a true image by the second classifier. As a result, it is possible to learn a restoration model for detecting an abnormal portion without requiring abnormal state data.

The abnormality detection device according to the present embodiment generates a mask image that masks the cells for each cell in which the input image is divided. The anomaly detection device takes a mask image as an input for each cell, uses a pre-learned restoration model for restoring the image, generates a restoration image, and acquires a restoration cell image which is a cell in the restoration image. The abnormality detection device generates a combined image in which the restored cell images acquired for each cell are combined, compares the combined image with the input image, and detects an abnormal portion. As a result, the abnormal location can be detected without requiring the abnormal status data. Further, since the abnormal portion can be detected without recognizing the type of abnormality, it can be used as a pre-stage of various abnormality detecting techniques.

<Modification example>
The present invention is not limited to the above-described embodiment, and various modifications and applications are possible without departing from the gist of the present invention.

For example, in the present embodiment, the number of divided grids is fixed, but learning / restoration may be performed for each of the plurality of grids to obtain a plurality of abnormality detection maps H. By taking the average of the plurality of abnormality detection maps H obtained in this way, it is possible to deal with fluctuations in the scale of the abnormal portion.

Further, the learning device and the abnormality detection device may be configured as one device. Further, although described as an embodiment in which the program is pre-installed in the specification of the present application, it is also possible to provide the program by storing it in a computer-readable recording medium.

Further, various processors other than the CPU may execute various processes in which the CPU reads the software (program) and executes the software (program) in each of the above embodiments. In this case, the processors include PLD (Programmable Logic Device) whose circuit configuration can be changed after manufacturing FPGA (Field-Programmable Gate Array), and ASIC (Application Specific Integrated Circuit) for executing ASIC (Application Special Integrated Circuit). An example is a dedicated electric circuit or the like, which is a processor having a circuit configuration designed exclusively for the purpose. Further, the learning process and the abnormality detection process may be executed by one of these various processors, or a combination of two or more processors of the same type or different types (for example, a plurality of FPGAs, and a CPU and an FPGA). It may be executed in combination with). Further, the hardware structure of these various processors is, more specifically, an electric circuit in which circuit elements such as semiconductor elements are combined.

Further, in each of the above embodiments, the mode in which the learning program and the abnormality detection program are stored (installed) in the storage 14 in advance has been described, but the present invention is not limited to this. The program is a non-temporary storage medium such as a CD-ROM (Compact Disk Read Only Memory), a DVD-ROM (Digital Versailles Disk Online Memory), and a USB (Universal Serial Bus) memory. It may be provided in the form. Further, the program may be downloaded from an external device via a network.

Regarding the above embodiments, the following additional notes will be further disclosed.

(Appendix 1)
Anomaly detection device
With memory
With at least one processor connected to the memory
Including
The processor
A mask image that masks the cell is generated for each cell in which the input image is divided.
For each cell, the mask image is input, a restored image is generated using a pre-learned restoration model for restoring the image, and the restored cell image which is the cell in the restored image is acquired.
A combined image obtained by combining the restored cell images acquired for each cell is generated.
An abnormal portion is detected by comparing the combined image with the input image.
Anomaly detection device configured as.

(Appendix 2)
A non-temporary storage medium that stores a program that can be executed by a computer to execute anomaly detection processing.
The abnormality detection process is
A mask image that masks the cell is generated for each cell in which the input image is divided.
For each cell, the mask image is input, a restored image is generated using a pre-learned restoration model for restoring the image, and the restored cell image which is the cell in the restored image is acquired.
A combined image obtained by combining the restored cell images acquired for each cell is generated.
A non-temporary storage medium that detects an abnormal portion by comparing the combined image with the input image.

(Appendix 3)
It ’s a learning device,
With memory
With at least one processor connected to the memory
Including
The processor
A mask image that masks the cell is generated for each cell in which the learning image representing the normal state is divided.
For each cell, the mask image is input, a restored image is generated using the restored model for restoring the image, and the restored cell image which is the cell in the restored image is acquired.
The restoration model is trained so that the restoration cell image and the cell of the learning image match for each cell.
A learning device configured to be.

(Appendix 4)
A non-temporary storage medium that stores a program that can be executed by a computer to perform a learning process.
The learning process is
A mask image that masks the cell is generated for each cell in which the learning image representing the normal state is divided.
For each cell, the mask image is input, a restored image is generated using the restored model for restoring the image, and the restored cell image which is the cell in the restored image is acquired.
A non-temporary storage medium that learns the restoration model so that the restoration cell image and the cell of the learning image match for each cell.

10 Learning device 20 Learning

data storage unit

22, 60 Mask processing unit 24 Learning unit 50 Anomaly detection device 62 Model storage unit 64 Restoration unit 66 Coupling unit 68 Anomaly detection unit

Claims

A mask processing unit that generates a mask image that masks the cells for each cell in which the input image is divided,
A restoration unit that receives the mask image as input for each cell, generates a restoration image using a pre-learned restoration model for restoring the image, and acquires a restoration cell image that is the cell in the restoration image. When,
A combined portion that generates a combined image by combining the restored cell images acquired for each cell, and
An abnormality detection unit that detects an abnormality by comparing the combined image with the input image,
Anomaly detection device including.
A mask processing unit that generates a mask image that masks the cell for each cell in which the learning image representing the normal state is divided.
For each cell, the mask image is input, a restored image is generated using the restored model for restoring the image, and the restored cell image which is the cell in the restored image is acquired.
A learning unit that learns the restoration model so that the restoration cell image and the cell of the learning image match for each cell.
Learning device including.
The learning unit
For each cell, the mask image is input, a restored image is generated using the restored model for restoring the image, and the restored cell image which is the cell in the restored image is acquired.
A combined image obtained by combining the restored cell images acquired for each cell is generated.
For each cell, the restored cell image and the cell of the learning image match, and
The combined image is identified by the first classifier as a true image and
For each cell, the restored model, the first classifier, so that the image in which the cell of the learning image is replaced with the restored cell image is identified as a true image by the second classifier. The learning device according to claim 2, wherein the second classifier is learned.
The mask processing unit generates a mask image that masks the cells for each cell that divides the input image.
The restoration unit generates a restoration image for each cell by inputting the mask image and using a pre-learned restoration model for restoring the image, and generates a restoration cell image which is the cell in the restoration image. Acquired,
The merged portion generates a merged image in which the restored cell images acquired for each cell are combined.
An abnormality detection method in which an abnormality detection unit detects an abnormality by comparing the combined image with the input image.
The mask processing unit generates a mask image in which the cell is masked for each cell in which the learning image representing the normal state is divided.
The learning unit generates a restored image using the restored model for restoring the image by inputting the mask image for each cell, and acquires the restored cell image which is the cell in the restored image.
A learning method in which the restoration model is learned so that the restoration cell image and the cell of the learning image match for each cell.
A mask image that masks the cell is generated for each cell in which the input image is divided.
For each cell, the mask image is input, a restored image is generated using a pre-learned restoration model for restoring the image, and the restored cell image which is the cell in the restored image is acquired.
A combined image obtained by combining the restored cell images acquired for each cell is generated.
An abnormality detection program for causing a computer to detect an abnormal part by comparing the combined image with the input image.
A mask image that masks the cell is generated for each cell in which the learning image representing the normal state is divided.
For each cell, the mask image is input, a restored image is generated using the restored model for restoring the image, and the restored cell image which is the cell in the restored image is acquired.
A learning program for causing a computer to learn the restoration model so that the restoration cell image and the cell of the learning image match for each cell.