JP2023146059A - Method for detecting abnormal area, system, and program - Google Patents

Abstract

To provide a method for detecting an abnormal area in which overlook of abnormality is reduced.SOLUTION: A method includes: (a) a step of performing processing of extracting a non-defective feature quantity group from a plurality of patches on a plurality of non-defective images; (b) a step of extracting a target feature quantity being the feature quantity of a target image from the plurality of patches; (c) a step of clustering a plurality of non-defective feature quantities to acquire a plurality of representative points being representative points of a plurality of clusters; (d) a step of creating, for each of the plurality of representative points, an appearance likelihood map for each of the plurality of patches; and (e) a step of calculating, for each feature extraction layer of a plurality of feature extraction layers, the most neighboring distance between the target feature quantity and the representative point with high appearance likelihood as the degree of abnormality for each of the plurality of patches by using the appearance likelihood map, thereby creating an abnormality degree map.SELECTED DRAWING: Figure 1

Description

The present disclosure relates to a method, system, and program for detecting an abnormal area.

Non-Patent Document 1 describes how to detect anomalies by acquiring a group of features of non-defective products and features of input data for inspection from a model trained on ImageNet in patch units, and measuring the distance between the two in the feature space. A technique for performing this has been disclosed.

Niv Cohen, Yedid Hoshen “Sub-Image Anomaly Detection with Deep Pyramid Correspondences”, arXiv:2005.02357, Wed, 3 Feb 2021.

However, if there are features that are close to the features of the acquired inspection input data among the features of a non-defective product, the degree of abnormality may be calculated to be low, increasing the possibility that an abnormality will be overlooked. be.

According to one aspect of the present disclosure, a method of detecting an abnormal region is provided. This method is a method for detecting an abnormal region included in a target image using a feature extraction model configured as a convolutional neural network having multiple feature extraction layers, the method includes: The feature extraction model is input to the feature extraction model, and in each of the plurality of feature extraction layers, a group of non-defective features, which are the features of the non-defective image, are applied to each of a plurality of patches corresponding to a plurality of patch images obtained by partitioning the image. (b) inputting the target image into the feature extraction model, and in each of the plurality of feature extraction layers, extracting a feature amount of the target image. (c) extracting a target feature for each of the plurality of patches, and (c) clustering the plurality of non-defective features for each of the feature extraction layers of the plurality of feature extraction layers to form a plurality of clusters. (d) for each of the plurality of representative points, obtaining a plurality of representative points, each of which is a representative point, for each of the plurality of patches, calculating the probability that the representative point will appear; (e) for each of the plurality of feature extraction layers of the plurality of feature extraction layers, using the appearance possibility map for each of the plurality of patches; The method includes the step of creating an anomaly degree map by calculating the nearest neighbor distance between the target feature amount and the representative point that is likely to appear as an anomaly degree.

FIG. 1 is a block diagram showing the configuration of an abnormality detection system. FIG. 2 is an explanatory diagram showing a configuration example of a feature extraction model. The figure which shows the flow of creating a stratified abnormality degree map. Examples of non-defective and defective images. Flowchart of abnormality detection processing. Flowchart of distance calculation processing. Flowchart of nearest neighbor distance array processing. A diagram illustrating distance calculation results. The figure explaining the calculation result of nearest neighbor distance. A diagram explaining an appearance possibility map. Flowchart of abnormality degree calculation processing. FIG. 3 is a diagram conceptually explaining an abnormality degree calculation method. 7 is a flowchart of the first half of abnormality detection processing according to the second embodiment. 7 is a flowchart of the second half of the abnormality detection process according to the second embodiment. An example of a GUI displayed on the display.

A. First embodiment:
A1. System configuration:
FIG. 1 is a block diagram showing the configuration of an abnormality detection system that executes the abnormality detection process shown in FIG. 5. As shown in FIG. 1, the abnormality detection system includes an information processing device 100 and a camera 400. The camera 400 captures an image of the object to be detected as an abnormality. The information processing device 100 functions as an abnormality detection device that detects abnormalities included in images captured by the camera 400. The information processing device 100 can be realized by, for example, a personal computer.

The information processing device 100 includes a processor 110, a memory 120, an interface circuit 130, an input device 140 connected to the interface circuit 130, and a display section 150. For example, but not limited to, the processor 110 not only has the function of executing the processes detailed below, but also displays data obtained by the processes and data generated in the process of the processes on the display unit 150. It also has the function of A camera 400 is also connected to the interface circuit 130.

The processor 110 functions as an anomaly detection unit 112 that detects anomalies included in images. The functions of the abnormality detection unit 112 are realized by the processor 110 executing a computer program stored in the memory 120. However, the function of the abnormality detection section 112 may be realized by a dedicated electronic circuit. Processor as used herein is a term that also includes such dedicated electronic circuits. Further, the processor that executes the processing of the abnormality detection unit 112 may be a processor included in a remote computer connected to the information processing device 100 via a network. Further, the processing of the abnormality detection unit 112 may be executed by a plurality of processors.

The memory 120 stores a feature extraction model 200, a group of non-defective products images GM, and a group of non-defective product features GF. The feature extraction model 200 is a machine learning model that obtains feature amounts from images, and has been pre-trained on an image database. Specifically, ResNet, which is obtained by pre-learning ImageNet, can be used as the feature extraction model 200. The non-defective image group GM is an image group of non-defective products that does not contain any abnormality. The non-defective product feature group GF is composed of feature amounts obtained when each non-defective product image of the non-defective product image group GM is input to the feature extraction model 200.

FIG. 2 is an explanatory diagram showing a configuration example of the feature extraction model 200. As shown in FIG. 2, the feature extraction model 200 includes, after the input layer receiving the input image IM, a first feature extraction layer FE1, a second feature extraction layer FE2, a third feature extraction layer FE3, and an average pooling layer. 240 is a convolutional neural network arranged in this order. The first feature extraction layer FE1, the second feature extraction layer FE2, and the third feature extraction layer FE3 are also collectively referred to as the feature extraction layer FE. Further, the feature extraction layer FE may be simply referred to as a "layer". Among the four layers FE1 to FE3, 240, the first feature extraction layer FE1 is the lowest layer, and the average pooling layer 240 is the highest layer. Although three feature extraction layers FE are used in the example of FIG. 2, the number of feature extraction layers FE is arbitrary. However, it is preferable to use two or more feature extraction layers FE.

The three feature extraction layers FE1 are convolution layers that respectively extract features of the input image IM. In the example of FIG. 2, the plane size of the input image IM is 224×224 pixels. The first feature extraction layer FE1 has a plane size of 56×56 nodes, and the number of channels is Z1. Node means neuron. The second feature extraction layer FE2 has a plane size of 28×28 and a number of channels Z2. The third feature extraction layer FE3 has a plane size of 14×14 and a channel number of Z3. The number of channels Z1, Z2, and Z3 in each layer can be set to any number greater than or equal to 1. The 16×16 pixel image in the input image IM is called a patch image PT0 or patch PT0. The patch PT0 in the input image IM corresponds to a 4×4 patch PT1 in the first feature extraction layer FE1, corresponds to a 2×2 patch PT2 in the second feature extraction layer FE2, and corresponds to a 2×2 patch PT2 in the third feature extraction layer FE3. This corresponds to a 1×1 size patch PT3.

In the abnormality detection process, feature amounts are extracted for each patch PT1 from the outputs of a plurality of nodes belonging to the first feature extraction layer FE1. The feature amount for one patch PT1 in the first feature extraction layer FE1 is expressed by the output of [plane size of patch PT1]×[number of channels] nodes, that is, 4×4×Z1 nodes. Therefore, for example, the feature amount corresponding to one patch PT1 is a vector consisting of 4×4×Z1 elements. Similarly, in the second feature extraction layer FE2, a feature amount is extracted for each patch PT2, and in the third feature extraction layer FE3, a feature amount is extracted for each patch PT3. Therefore, each of the three feature extraction layers FE extracts respective feature amounts from 14×14 patches. Since the feature extraction layer FE corresponds to an intermediate layer, the feature amounts extracted from the feature extraction layer FE can also be called "intermediate feature amounts."

The average pooling layer 240 executes an operation to obtain an output of 1×1×Z3 by calculating the average value of the outputs from the 14×14 nodes of the third feature extraction layer FE3 for each channel. The output of the average pooling layer 240 is a feature vector representing features of the entire image. However, since the output of the average pooling layer 240 is not used in the abnormality detection process according to this embodiment, the average pooling layer 240 may be omitted.

A2. Overview of anomaly detection processing:
In the abnormality detection process, a first layered abnormality map AM1, a second layered abnormality map AM2, and a third layered abnormality map AM3 corresponding to each of the three feature extraction layers FE are created. The first stratified abnormality map AM1, the second stratified abnormality map AM2, and the third stratified abnormality map AM3 are also collectively referred to as the stratified abnormality map AM.

FIG. 3 is a diagram showing the flow of creating one stratified abnormality map AM. As described above, the non-defective image group GM shown in FIG. 3 is an image group of non-defective products. A group of feature quantities F obtained when each non-defective image of the non-defective image group GM is input to the feature extraction model 200 is a non-defective feature group GF. In this embodiment, N non-defective images are used.

Similar to the non-defective image, the target image TM to be inspected is input to the feature extraction model 200, and the feature amount F is extracted for each patch. The extracted feature amount F is called a target feature amount TF.

Roughly speaking, the stratified abnormality map AM is created by determining the degree of similarity between the target feature value TF and the feature value F of the non-defective feature value group GF for each patch, and arranging the determined degree of similarity at patch positions. Created. Here, the degree of similarity specifically refers to a distance such as Eugrid distance or Manhattan distance, or a degree of similarity such as cosine similarity or Pearson similarity. The higher the degree of similarity, the higher the degree of similarity. The shorter the distance, the higher the degree of similarity. In this specification, the term distance is used to mean the degree of similarity. That is, in this specification, distance is a concept that includes similarity.

In this embodiment, in order to find the degree of similarity between the target feature TF and the feature F of the non-defective feature group GF, the feature F of the non-defective feature group GF that is closest to the target feature TF and the target feature The nearest neighbor distance is used, which is the distance to the quantity TF. Here, since the number of data in the non-defective feature group GF is enormous, in order to find the nearest neighbor distance, the distance between all the features F that make up the non-defective feature group GF and the target feature TF is calculated. The amount of calculation becomes enormous. Therefore, in this embodiment, the distance between the representative point RP obtained by clustering and the target feature amount TF is determined. This makes it possible to reduce the computational load required for calculations of similar degrees. Note that instead of the nearest neighbor distance between the feature F of the non-defective feature group GF and the target feature TF, the nearest neighbor distance between the representative point RP of the non-defective feature group GF and the target feature TF is determined. The method is also called approximate nearest neighbor search.

More specifically, the nearest neighbor distance between the representative point RP of each group obtained by clustering the non-defective features GF and each of the plurality of target features TF is calculated for each patch. Ru. A stratified abnormality map AM, which is a heat map, is created by arranging the calculated averages of the plurality of nearest neighbor distances for each patch at patch positions. This makes it possible to visualize the position determined to be abnormal. The stratified abnormality map AM is color-coded according to the length of the nearest neighbor distance. The larger the nearest neighbor distance, the lower the degree of similarity and the higher the degree of abnormality.

The inventors have focused on the fact that when determining the degree of abnormality as described above, there are defective images in which it is difficult to detect an abnormality. FIG. 4 is an example of a non-defective image containing no abnormality and a defective image containing an abnormality that is difficult to detect. FIG. 4 schematically shows a non-defective image and a defective image of a cable, and a non-defective image and a defective image of a transistor.

The cable image is a cross-sectional image of a three-core cable. The three-core cable has a configuration in which a first cable WY1, a second cable WY2, and a third cable WY3 each having a bundle of conducting wires covered with a first coating layer IL1 are collectively covered with a second coating layer IL2. There is. The first cable WY1, the second cable WY2, and the third cable WY3 have different colors of the first coating layer IL1. The defect image differs from the non-defective image in that the first coating layer IL1 of the first cable WY1 and the first coating layer IL1 of the second cable WY2 have the same color.

The transistor image is an image in which a transistor TR having a terminal TW is mounted on a substrate SB. The defect image differs from the non-defective image in that the transistor TR is not arranged.

In the cable defect image, the feature amount F extracted from the first target patch PTT1 located in the first covering layer IL1 of the first cable WY1 is different from the feature amount F extracted from the first target patch PTT1 located in the first covering layer IL1 of the second cable WY2 in the non-defective image. It is similar to the feature amount F extracted from the first non-defective patch PG1. The feature amount F extracted from the first non-defective patch PG1 is included in the non-defective feature group GF shown in FIG. Therefore, in the above method, the degree of abnormality in the first target patch PTT1 is calculated to be low.

Similarly, in the defect image of the transistor, the feature amount F extracted from the second target patch PTT2 located at the planned placement location of the transistor TR is the same as the feature amount F extracted from the second target patch PG2 located on the substrate SB of the non-defective image. Similar to quantity F. The feature amount F extracted from the second non-defective patch PG2 is included in the non-defective feature group GF. Therefore, in the above method, the degree of abnormality in the second target patch PTT2 is calculated to be low.

A3. Anomaly detection processing:
The abnormality detection process according to this embodiment is designed to detect the illustrated abnormalities. FIG. 5 is a flowchart of the abnormality detection process. FIG. 6 is a flowchart of distance calculation processing. FIG. 7 is a flowchart of nearest neighbor distance arrangement processing. FIG. 8 is a diagram illustrating the results of distance calculation, which will be described later. FIG. 9 is a diagram illustrating the calculation result of the nearest neighbor distance, which will be described later. FIG. 10 is a diagram illustrating an appearance possibility map MP, which will be described later. FIG. 11 is a flowchart of the abnormality degree calculation process. In step S10 shown in FIG. 5, the abnormality detection unit 112 inputs the plurality of good product image groups GM into the feature extraction model 200, extracts the feature amount F, and prepares the good product feature amount group GF.

In the non-defective product feature amount extraction process, the feature amount F is extracted using the feature extraction layer FE for each patch for each feature extraction layer FE for one non-defective image of the non-defective image group GM. Specifically, for one non-defective image, the process of extracting the feature amount F for each patch is repeated until it is completed for all feature extraction layers FE. Then, extraction of the feature amount F for each feature extraction layer FE is repeatedly performed until extraction of the feature amount F is completed for all non-defective images included in the non-defective image group GM. The feature quantity F extracted for each patch is converted into data without spatial information, and then added to the non-defective feature quantity group GF.

In step S20 of FIG. 5, similarly to step S10, the abnormality detection unit 112 inputs the target image TM to the feature extraction model 200 and extracts the target feature amount TF.

In step S30, the abnormality detection unit 112 obtains k representative points RP of the non-defective feature group GF by clustering. In this embodiment, the k-means method is used for clustering.

In step S40, the abnormality detection unit 112 performs appearance possibility map creation processing. The appearance possibility map creation process includes a distance calculation process in step S410 and a nearest neighbor distance arrangement process in step S420.

As shown in FIG. 6, in the distance calculation process, the abnormality detection unit 112 calculates the distance between the representative point RP and the feature amount F for each patch, targeting one feature extraction layer FE for one of the non-defective images. Step S411 is performed. Then, the abnormality detection unit 112 repeatedly performs step S411 until distances are calculated for all representative points. Then, the distance calculation is repeated until the distance is calculated for all the feature extraction layers FE for one non-defective image, and the distance calculation is repeated until the distance calculation is completed for all the non-defective images.

Specifically, in step S411, the abnormality detection unit 112 calculates the average distance between the representative point RP and each of the plurality of feature amounts F as the distance between the representative point RP and the feature amount F for each patch. A map created by arranging the distances between the representative point RP and the feature amount F at patch positions for one non-defective image is called a distance map MD. As shown in FIG. 8, N distance maps MD, which are the same as the number of non-defective images, are created for each representative point RP for each feature extraction layer FE. By performing step S410, in the example of FIG. 2, a tensor of [N×k×14×14] is created for one feature extraction layer FE.

As shown in FIG. 7, in the nearest neighbor distance arrangement process, the abnormality detection unit 112 determines the distance between the representative point RP of the N non-defective images and the feature amount F as the nearest neighbor distance for each patch in step S421. Obtain the minimum distance value. Then, in step S422, the obtained nearest neighbor distance is binarized using a predetermined threshold th to create an appearance possibility map MP. In this embodiment, when the nearest neighbor distance is less than the threshold th, the appearance possibility is set to "0" to indicate that the appearance possibility is high. When the nearest neighbor distance is equal to or greater than the threshold th, the appearance possibility is set to "1" to indicate that the appearance possibility is low. The anomaly detection unit 112 repeatedly performs steps S421 and S422 until it creates appearance possibility maps MP for all representative points RP, and repeatedly performs steps S421 and S422 until it creates appearance possibility maps MP for all feature extraction layers FE. As shown in FIG. 9, one appearance possibility map MP is created for each representative point RP in each feature extraction layer FE.

FIG. 10 is a diagram illustrating the appearance possibility map MP. The non-defective image of the transistor includes images of each of the substrate SB, the transistor TR, and the terminal TW. The representative point RP of the feature quantity F group extracted from the image of the board SB is the first representative point RP1. The representative point RP of the feature quantity F group extracted from the image of the transistor TR is the 418th representative point RP418. The representative point RP of the feature quantity F group extracted from the image of the terminal TW is the 756th representative point RP756. According to the above method, in the appearance possibility map MP of the first representative point RP1, the range of the image of the substrate SB is a range where the appearance possibility is high. In the appearance possibility map MP shown in FIG. 10, patches with an appearance possibility of "0" are shown without hatching, and patches with an appearance possibility of "1" are shown with diagonal hatching. Similarly, in the appearance possibility map MP of the 418th representative point RP418, the range of the image of the transistor TR is a range with a high possibility of appearance. In the appearance possibility map MP of the 756th representative point RP756, the range of the image of the terminal TW is a range where the appearance possibility is high.

In step S50 of FIG. 5, the abnormality detection unit 112 performs abnormality degree calculation processing. As shown in FIG. 11, the anomaly detection unit 112 calculates the degree of anomaly for each patch in one feature extraction layer FE. Specifically, in step S501, the abnormality detection unit 112 extracts a representative point RP that has a high probability of appearing for the target patch, that is, the probability of appearing is "0". In step S502, the abnormality detection unit 112 calculates the nearest distance between the representative point RP and the target feature amount TF as the degree of abnormality for each extracted representative point RP. Specifically, the distance between each of the plurality of target feature quantities TF and the target representative point RP is calculated, and the minimum value among the plurality of calculated distances is obtained as the nearest neighbor distance. The nearest neighbor distance obtained in this way reflects the possibility of appearance, so the distance to the representative point RP with a low possibility of appearance is calculated as the nearest neighbor distance, and the degree of abnormality is calculated to be low. Can be suppressed.

The anomaly detection unit 112 repeatedly calculates the degree of abnormality until it calculates the degree of abnormality for all patches, and repeatedly calculates the degree of abnormality until it calculates the degree of abnormality for all feature extraction layers FE.

FIG. 12 is a diagram conceptually explaining the abnormality degree calculation method according to this embodiment. The group of representative points RP acquired using the non-defective image includes a plurality of representative points RP from which features of the image of the board SB are extracted, a plurality of representative points RP from which features of the image of the transistor TR are extracted, and an image of the terminal TW. A plurality of representative points RP from which features are extracted are included. When calculating the degree of abnormality for a patch in the target image TM at the position where the transistor TR is scheduled to be placed, the representative points RP group from which the characteristics of the image of the transistor TR are extracted are unlikely to appear, so nearest neighbor distance calculation is necessary. excluded from the scope of Therefore, the distance to a group of representative points RP excluding the representative point RP from which features of the image of the transistor TR are extracted is used for calculating the degree of abnormality. Therefore, the degree of abnormality is prevented from being calculated to be low when the representative point RP that is the object of distance calculation is not limited. Therefore, it is possible to detect abnormalities such as misplacement of parts and missing parts as explained using FIG. 4.

Note that in order to detect abnormalities such as misplacement of parts or missing parts, a method of comparing the target image TM and the non-defective image for each image position may be considered. When comparing each image position in this way, it is expected that even a slight shift in the position of a component with respect to the image will be detected as an abnormality. In this regard, according to the abnormality detection method according to the present embodiment, by using the index of appearance possibility, it becomes possible to make a gentle judgment regarding the positional deviation of the component with respect to the image. Then, as described in the second embodiment, by adjusting the threshold th, it is possible to adjust the permissible width of the positional shift of the component with respect to the image.

In step S60 of FIG. 5, the anomaly detection unit 112 arranges the nearest neighbor distances at patch positions to create a stratified anomaly degree map AM as an anomaly degree map. As a result, as shown in the lower part of FIG. 2, three stratified abnormality maps AM are created for each of the three feature extraction layers FE. The first layered abnormality degree map AM1 is a map showing the abnormality degree of each patch PT1. Similarly, the second layered abnormality degree map AM2 is a map that shows the degree of abnormality for each patch PT2, and the third layered abnormality degree map AM3 is a map that shows the degree of abnormality for each patch PT3.

In step S100, the abnormality detection unit 112 displays the stratified abnormality degree map AM on the display unit 150, and ends this processing routine. By displaying the stratified abnormality degree map AM on the display unit 150, the user can confirm abnormal regions with a high degree of abnormality.

According to the first embodiment described above, in step S50, the abnormality detection unit 112 calculates the degree of abnormality using the appearance possibility map MP. Thereby, the nearest neighbor distance between the representative point RP and the target feature amount TF, which has a low probability of appearing, is adopted as the degree of abnormality, and it is possible to prevent an abnormality from being overlooked due to a low degree of abnormality being calculated.

In addition, in step 420, the abnormality detection unit 112 calculates the nearest neighbor distance between the representative point RP and the plurality of non-defective features as the probability that the representative point RP appears, and for patches whose nearest neighbor distance is less than the threshold th. An appearance possibility map MP is created by setting the appearance possibility high and setting the appearance possibility low for patches whose nearest neighbor distance is equal to or greater than the threshold th. Thereby, an appearance possibility map can be created by comparing the nearest neighbor distance between the representative point RP and the non-defective product feature amount and the threshold value th.

Furthermore, in step S501, the abnormality detection unit 112 extracts a representative point RP that is likely to appear for the target patch. In step S502, the abnormality detection unit 112 calculates the nearest distance between the extracted representative point RP and the target feature amount TF as the degree of abnormality. This allows the probability of occurrence to be reflected in the degree of abnormality.

B. Second embodiment:
FIGS. 13 and 14 are flowcharts showing the procedure of abnormality detection processing according to this embodiment. The anomaly detection process according to the present embodiment differs from the anomaly detection process according to the first embodiment in that an integrated anomaly degree map AMT is created, and that a change in the threshold th used for creating the appearance possibility map MP is accepted. different. Steps that are the same as the processing steps according to the first embodiment are given the same reference numerals, and detailed explanations will be omitted as appropriate.

As shown in FIG. 13, similarly to the first embodiment, the abnormality detection unit 112 performs steps S10 to S60.

In step S70, the abnormality detection unit 112 performs weighted addition of the stratified abnormality maps AM1, AM2, and AM3 to create an integrated abnormality map AMT shown in FIG. The integrated abnormality map AMT is created by first performing resolution adjustment to equalize the resolution of the stratified abnormality maps AM1, AM2, and AM3, and then adding the stratified abnormality maps AM1, AM2, and AM3 after the resolution adjustment. Ru. When added, the weights of the stratified abnormality maps AM1, AM2, and AM3 may be added equally, or may be added so that the weights are unequal. Here, unequal means that the weights are different from each other. The resolution adjustment is preferably performed to match the size of the input image IM to the feature extraction model 200. In the example of FIG. 2, the size of the input image IM is 224×224 pixels, so the size of each of the stratified abnormality maps AM1, AM2, and AM3 after resolution adjustment is also 224×224 pixels. Further, when adjusting the resolution, pixel values may be interpolated. Further, before or after the addition of the stratified abnormality maps AM1, AM2, and AM3, blurring processing may be performed using a blurring filter such as a Gaussian filter.

In step S80 shown in FIG. 14, the abnormality detection unit 112 detects a patch whose degree of abnormality is equal to or higher than a predetermined abnormality degree threshold as an abnormal location in the integrated abnormality degree map AMT. In step S110, the abnormality detection unit 112 identifies the detected abnormality location and causes the display unit 150 to display the stratified abnormality map AM and the integrated abnormality map AMT. Specifically, the abnormality detection unit 112 displays the abnormal location in a different color from other locations, for example, red, in the stratified abnormality map AM and the integrated abnormality map AMT. This allows the user to check the abnormal area.

FIG. 15 is an example of a GUI (Graphical User Interface) displayed on the display unit 150. As shown in FIG. 15, a window 500 displays a target image TM, an integrated abnormality map AMT, a first layered abnormality map AM1, a second layered abnormality map AM2, and a third layered abnormality map. Map AM3 is displayed. Further, the window 500 displays a slider 501 and a recalculation button 502. The slider 501 accepts changes in the threshold th. In the initial state, the recalculation button 502 is grayed out and cannot be selected. Then, when the threshold value th is changed from the initial state, it is displayed in a selectable valid state. If the user wants to change the threshold value th, after moving the slider 501, the user selects the recalculation button 502 which is in the enabled state.

In step S120 of FIG. 14, the abnormality detection unit 112 determines whether or not a change in the threshold th has been received. Specifically, when the recalculation button 502 shown in FIG. 15 is selected, it is determined that the change of the threshold th has been accepted. On the other hand, if the recalculation button 502 is displayed in a non-selectable state and the recalculation button 502 is not selected, it is determined that the change of the threshold th has been accepted.

In step S120, if the abnormality detection unit 112 determines that the change in the threshold value th has not been accepted, the abnormality detection unit 112 ends this processing routine. On the other hand, if the abnormality detection unit 112 determines that the change in the threshold th has been accepted, it performs the processes from step S420 to step S110 using the threshold th accepted by the slider 501. Thereby, the user can adjust the slider 501 by looking at the displayed stratified abnormality map AM and integrated abnormality map AMT. For example, the user can adjust the threshold value th to a high value when a slight positional shift of a component is detected as an abnormality.

According to the second embodiment described above, the integrated abnormality degree map AMT is created in step S70, and abnormal locations are displayed on the display unit 150 in step S110. This allows the user to see and confirm the abnormal location.

Moreover, in step S120, when the threshold value th has been changed, the abnormality detection unit 112 performs steps S420 to S110 again. Thereby, the user can adjust the threshold th by looking at the stratified abnormality map AM and the integrated abnormality map AMT.

C. Other embodiments:
(C1) Other embodiment 1:
In the first embodiment, in step S421, the probability of appearance is determined using the nearest distance, which is the smallest distance, among the distances calculated for N non-defective images. The method of calculating the probability of appearance is not limited to this, but for example, the number of non-defective images that are less than or equal to a predetermined distance threshold is calculated as the frequency among the distances calculated for N non-defective images, and the frequency is determined as the frequency is determined in advance. The occurrence probability map MP may be created by determining that patches whose frequency is equal to or higher than a frequency threshold are considered to have a high probability of appearing.

(C2) Other Embodiment 2:
In the first embodiment, the stratified abnormality map AM is displayed on the display unit 150 and used. The method of using the stratified abnormality map AM is not limited to the case where it is displayed on the display unit 150 and used as an image. For example, the target image TM can be classified using the position of a patch having an abnormality level greater than or equal to an abnormality threshold, and used for defect analysis. Further, it may be used to determine that a target image TM in which a patch having a degree of abnormality equal to or higher than an abnormality threshold is located at a specific position is defective.

(C3) Other Embodiment 3:
In the first embodiment, the k-means method is used for clustering, but other methods such as a single link method may be used.

(C4) Other Embodiment 4:
In the first embodiment, in step S422, the nearest neighbor distance between the representative point RP and the feature amount F of a non-defective product is binarized using the threshold th to create an appearance possibility map MP. The method of creating the appearance possibility map MP is not limited to this.
(C4-1) For example, the nearest neighbor distance map (MD': distance map before being binarized) may be treated as MP without being binarized and reflected in the abnormality degree map. In this case, the nearest neighbor distance of the corresponding coordinates of the nearest representative point MD' is added to the nearest neighbor distance calculated for the representative point RP, and this is reflected in the degree of abnormality.
Specifically, a map created by arranging the nearest neighbor distances having position information acquired in step S422 before binarization at patch positions may be used as the appearance possibility map MP. The following method is available as a method for creating an abnormality map using the appearance possibility map MP according to the present embodiment (C4-1). Regarding the target patch, the nearest neighbor distance used as the degree of abnormality calculated for each representative point RP is the nearest neighbor distance used as the probability of appearance of the target patch for all representative points RP, that is, in this embodiment ( The degree of abnormality may be an index obtained by adding the shortest distance among the nearest neighbor distances of the appearance possibility map MP related to C4-1).
(C4-2) Alternatively, MP may be calculated by normalizing the nearest neighbor distance map. In this case, the distance from the representative point RP is weighted by MP and then the nearest neighbor distance is determined, which is reflected in the degree of abnormality.
Specifically, a map created by arranging the values obtained by normalizing the nearest neighbor distances of the appearance possibility map MP according to the present embodiment (C4-1) at patch positions is used as the appearance possibility map MP. It may also be used as The following method is available as a method for creating an abnormality map using the appearance possibility map MP according to the present embodiment (C4-2). The normalized nearest neighbor distance, which is used as the probability of appearance of the target patch, is used for the nearest neighbor distance, which is used as the degree of abnormality calculated for the target patch, so that the higher the probability of appearance, the higher the degree of abnormality. An index that is weighted may be used as the degree of abnormality.

D. Other forms:
The present disclosure is not limited to the embodiments described above, and can be implemented in various configurations without departing from the spirit thereof. For example, the technical features of the embodiments corresponding to the technical features in each form described below are used to solve some or all of the above-mentioned problems or achieve some or all of the above-mentioned effects. In order to do so, it is possible to replace or combine them as appropriate. Further, unless the technical feature is described as essential in this specification, it can be deleted as appropriate.

(1) According to the first aspect of the present disclosure, a method of detecting an abnormal region is provided. This method is a method for detecting an abnormal region included in a target image using a feature extraction model configured as a convolutional neural network having multiple feature extraction layers, the method includes: (a) converting a non-defective image into the feature extraction model and in each of the plurality of feature extraction layers, a process of extracting a group of non-defective feature quantities, which are feature quantities of the non-defective image, for each of a plurality of patches corresponding to a plurality of patch images obtained by partitioning the image. (b) inputting the target image into the feature extraction model, and in each of the plurality of feature extraction layers, calculating the target feature quantity that is the feature quantity of the target image; , extracting for each of the plurality of patches, and (c) clustering the plurality of non-defective product features for each of the feature extraction layers of the plurality of feature extraction layers to extract representative points of each of the plurality of clusters. (d) for each of the plurality of representative points, for each of the plurality of patches, calculating the probability that the representative point will appear; , creating an appearance possibility map; (e) for each of the plurality of feature extraction layers, for each of the plurality of patches, using the appearance possibility map, determining the target feature amount and The method includes the step of creating an abnormality degree map by calculating the nearest neighbor distance to the representative point with a high possibility of appearance as the abnormality degree. According to this form, the nearest neighbor distance between a representative point with a high probability of appearing and the target feature is calculated as the abnormality degree, so the nearest neighbor distance between the representative point with a low probability of appearing and the target feature is calculated as the abnormality degree. By adopting this method, it is possible to prevent an abnormality from being overlooked by calculating a low abnormality degree.

(2) In the above embodiment, in the step (d), at least one of the frequency and probability of the appearance of the representative point may be calculated as the appearance possibility. According to this embodiment, the probability of appearance can be calculated using at least one of the frequency and probability of the appearance of the representative point.

(3) In the above embodiment, in the step (d), the nearest neighbor distance between the representative point and the plurality of non-defective product features is calculated as the probability for each of the plurality of patches, and The appearance possibility map may be created by setting the appearance possibility high for the patch whose distance is less than the threshold value and setting the appearance possibility low for the patch whose nearest neighbor distance is the threshold value or more. good. According to this form, the appearance possibility map can be created by comparing the nearest neighbor distance between the representative point and the non-defective product feature amount and the threshold value.

(4) In the above embodiment, in the step (e), for the target patch, the representative point with a high probability of appearance is extracted, and the nearest neighbor between the extracted representative point and the target feature amount. The distance may be calculated as the degree of abnormality. According to this embodiment, the probability of appearance can be reflected in the degree of abnormality by calculating the nearest neighbor distance between the extracted representative point with a high possibility of appearance and the target feature amount as the degree of abnormality.

(5) In the above embodiment, after the step (e), further (f) a resolution adjustment is performed to equalize the resolution of the plurality of abnormality maps obtained for the plurality of feature extraction layers, and after the resolution adjustment (g) detecting, in the integrated abnormality map, pixels for which the abnormality is equal to or higher than an abnormality threshold as abnormal locations; and (h) displaying the abnormal location on a display unit. According to this embodiment, the user can confirm the abnormality by looking at the abnormality displayed on the display unit.

(6) In the above embodiment, the threshold value is predetermined, and after the step (h), further performs the step of (i) accepting a change in the threshold value, and using the accepted threshold value, The steps from step (d) to step (h) may be performed. According to this embodiment, the user can freely change the threshold value, look at the display section, and confirm the detected abnormality using the changed threshold value.

The present disclosure can also be implemented in various forms other than the method of detecting an abnormal region. For example, it can be realized in the form of a computer program that implements a system for detecting an abnormal area or a method for detecting an abnormal area, a non-temporary recording medium on which the computer program is recorded, or the like.

DESCRIPTION OF SYMBOLS 100... Information processing device, 110... Processor, 112... Abnormality detection part, 120... Memory, 130... Interface circuit, 140... Input device, 150... Display part, 200... Feature extraction model, 240... Average pooling layer, 400... Camera , 500...Window, 501...Slider, 502...Recalculation button, AM1...First layered abnormality degree map, AM2...Second layered abnormality degree map, AM3...Third layered abnormality degree map, AMT...Integrated abnormality degree Map, F...feature quantity, FE1...feature extraction layer, FE2...second feature extraction layer, FE3...third feature extraction layer, GF...good product feature quantity group, GM...good product image group, IL1...first covering layer, IL2 ...second covering layer, IM...input image, MD...distance map, MP...appearance possibility map, PG1...first good patch, PG2...second good patch, PT0, PT1, PT2, PT3...patch, PTT1...th 1 target patch, PTT2...2nd target patch, RP...representative point, RP1...1st representative point, RP418...418th representative point, RP756...756th representative point, SB...substrate, TF...target feature amount, TM...target Image, TR...transistor, TW...terminal, WY1...first cable, WY2...second cable, WY3...third cable, th...threshold

Claims (8)

A method for detecting an abnormal region included in a target image using a feature extraction model configured as a convolutional neural network having multiple feature extraction layers, the method comprising:
(a) A non-defective product image is input to the feature extraction model, and in each of the plurality of feature extraction layers, a group of non-defective feature quantities, which are the feature quantities of the non-defective product image, are extracted into a plurality of non-defective feature quantities corresponding to a plurality of patch images obtained by partitioning the image. performing a process of extracting each of the patches on the plurality of non-defective images;
(b) inputting the target image into the feature extraction model, and extracting a target feature quantity, which is a feature quantity of the target image, for each of the plurality of patches in each of the plurality of feature extraction layers; and,
(c) for each of the feature extraction layers of the plurality of feature extraction layers, clustering the plurality of non-defective product feature quantities to obtain a plurality of representative points that are representative points of each of the plurality of clusters;
(d) creating an appearance possibility map by calculating the appearance possibility of the representative point appearing for each of the plurality of patches for each of the plurality of representative points;
(e) For each of the plurality of feature extraction layers of the plurality of feature extraction layers, for each of the plurality of patches, use the appearance possibility map to determine the relationship between the target feature amount and the representative point with a high possibility of appearance. A method comprising: creating an anomaly degree map by calculating a nearest neighbor distance as an anomaly degree. The method according to claim 1, comprising:
In the step (d), at least one of a frequency and a probability that the representative point appears is calculated as the appearance possibility. 3. The method according to claim 2,
In the step (d), for each patch of the plurality of patches, the nearest neighbor distance between the representative point and the plurality of non-defective product features is calculated as the probability, and the nearest neighbor distance is less than the threshold value. A method of creating the appearance possibility map by setting the appearance possibility high for patches and setting the appearance possibility low for the patches whose nearest neighbor distance is equal to or greater than the threshold value. The method according to claim 1, comprising:
In the step (e), for the target patch, the representative point with a high probability of appearance is extracted, and the nearest distance between the extracted representative point and the target feature is calculated as the abnormality degree. how to. 4. The method of claim 3, further comprising, after step (e),
(f) Performing resolution adjustment to equalize the resolution of the plurality of anomaly degree maps obtained for the plurality of feature extraction layers, and adding the plurality of anomaly degree maps after the resolution adjustment to create an integrated anomaly degree map. The process of creating
(g) detecting, in the integrated abnormality map, pixels whose abnormality is equal to or higher than an abnormality threshold as abnormal locations;
(h) A method including the step of displaying the abnormal location on a display unit. 6. The method according to claim 5,
The threshold value is predetermined, and after the step (h), further:
(i) performing a step of accepting a change in the threshold;
A method of performing the steps from step (d) to step (h) again using the received threshold value. A system for detecting abnormal regions included in a target image using a feature extraction model configured as a convolutional neural network having multiple feature extraction layers, the system comprising:
a memory that stores the feature extraction model and a plurality of non-defective product features obtained from each of the plurality of feature extraction layers when a plurality of non-defective product images are input to the feature extraction model;
one or more processors that execute calculations using the feature extraction model;
Equipped with
The processor includes:
(a) A non-defective image is input to the feature extraction model, and in each of the plurality of feature extraction layers, the non-defective feature quantity, which is the feature quantity of the non-defective image, is input to a plurality of patch images corresponding to a plurality of patch images obtained by partitioning the image. a step of performing extraction processing on each of the patches on the plurality of non-defective images;
(b) inputting the target image into the feature extraction model, and extracting a target feature quantity, which is a feature quantity of the target image, for each of the plurality of patches in each of the plurality of feature extraction layers; and,
(c) for each of the feature extraction layers of the plurality of feature extraction layers, clustering the plurality of non-defective product feature quantities to obtain a plurality of representative points that are representative points of each of the plurality of clusters;
(d) creating an appearance possibility map by calculating the appearance possibility of the representative point appearing for each of the plurality of patches for each of the plurality of representative points;
(e) For each of the plurality of feature extraction layers of the plurality of feature extraction layers, for each of the plurality of patches, use the appearance possibility map to determine the relationship between the target feature amount and the representative point with a high possibility of appearance. A system that performs the steps of: creating an anomaly degree map by calculating a nearest neighbor distance as an anomaly degree. A program for detecting abnormal regions included in a target image using a feature extraction model configured as a convolutional neural network having multiple feature extraction layers, the program comprising:
(a) A non-defective image is input to the feature extraction model, and in each of the plurality of feature extraction layers, the non-defective feature quantity, which is the feature quantity of the non-defective image, is input to a plurality of patch images corresponding to a plurality of patch images obtained by partitioning the image. a process of performing an extraction process for each of the patches on the plurality of non-defective images;
(b) A process of inputting the target image into the feature extraction model and extracting a target feature quantity, which is a feature quantity of the target image, for each of the plurality of patches in each of the plurality of feature extraction layers. and,
(c) a process of clustering the plurality of non-defective product feature values for each of the feature extraction layers of the plurality of feature extraction layers to obtain a plurality of representative points that are representative points of each of the plurality of clusters;
(d) creating an appearance possibility map by calculating the appearance possibility of the representative point appearing for each of the plurality of patches for each of the plurality of representative points;
(e) For each of the plurality of feature extraction layers of the plurality of feature extraction layers, for each of the plurality of patches, use the appearance possibility map to determine the relationship between the target feature amount and the representative point with a high possibility of appearance. A program that causes a computer to execute a process of creating an anomaly degree map by calculating the nearest neighbor distance as an anomaly degree.

Images