JP2021111306A - Computer system and program - Google Patents

Abstract

To automatically analyze a machine learning model for image recognition.SOLUTION: A computer system according to an embodiment acquires, for each of multiple images, a correctness determination result indicating whether an image recognition result obtained by inputting the image into a machine learning model is correct, calculates, for each of the multiple images, a relationship degree indicating the relationship between an area of interest in the image in the machine learning model and each of one or more attribute labels associated with the image in advance, and estimates, based on the correctness determination result and the relationship degree of each of the multiple images, an attribute label related to incorrect image recognition from the one or more attribute labels.SELECTED DRAWING: Figure 4

Description

One aspect of the disclosure relates to computer systems, processing methods, programs, and / or data structures.

A technique for analyzing a machine learning model for image recognition is known (see, for example, Patent Documents 1 to 3).

Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. Why should I trust you ?: Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pages 1135-1144. ACM, 2016. RR Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, D. Batra, et al. Grad-cam: Visual explanations from deep networks via gradient-based localization. In ICCV, pages 618-626, 2017 .. Avanti Shrikumar, Peyton Greenside, Anshul Kundaje. Learning important features through propagating Activation differences. Proceedings of the 34th International Conference on Machine Learning, PMLR 70, 3145-3153 (2017).

One aspect of the disclosure is intended to automatically analyze machine learning models for image recognition.

The computer system according to one aspect of the present disclosure includes a processor. The processor acquires a correct / incorrect judgment result indicating whether or not the image recognition result obtained by inputting the image into the machine learning model is correct for each of the plurality of images, and machine learning is performed for each of the plurality of images. The degree of relationship indicating the relationship between the region of interest in the image in the model and each of the one or more attribute labels associated with the image in advance is calculated, and based on the correctness determination result and the degree of relationship of each of the plurality of images. , The attribute label related to erroneous image recognition is estimated from one or more attribute labels.

It is a figure which shows an example of the analysis method by the analysis system which concerns on embodiment as process flow S1. It is a figure which shows an example of the highlight image which shows a region of interest. It is a figure which shows an example of the hardware composition of the computer used for the analysis system which concerns on embodiment. It is a figure which shows an example of the functional structure of the analysis system which concerns on embodiment. It is a figure which shows some example of the attribute label. It is a flowchart which shows an example of the analysis processing by the analysis system which concerns on embodiment. It is a flowchart which shows an example of the process of determining the correctness of image recognition. It is a flowchart which shows an example of the process of calculating the degree of relationship between a region of interest and an attribute label. It is a flowchart which shows an example of the process of estimating the attribute label related to erroneous image recognition.

Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In the description of the drawings, the same or equivalent elements are designated by the same reference numerals, and duplicate description will be omitted.

[System overview]
The analysis system 1 according to the embodiment is a computer system that analyzes a machine learning model for image recognition. More specifically, the analysis system 1 estimates the cause of the machine learning model erroneous image recognition.

An image is information expressed on a predetermined medium so that a person can visually recognize an event. The image may be an image generated by an image pickup device such as a camera, or may be an image subjected to any other image processing after shooting. Electronic data indicating an image, that is, image data is processed by a computer to visualize the image, and as a result, a person can visually recognize the image. For example, the image may be a still image, that is, a photograph, or a frame image constituting a moving image. Image recognition refers to a technique for identifying the characteristics of a scene projected on an image.

Machine learning is a method of autonomously finding a law or rule by iteratively learning based on given information. The specific method of machine learning is not limited, and it may be designed by any policy. Machine learning uses a machine learning model. Generally, machine learning models are constructed using multi-layered neural networks. A neural network is a model of information processing that imitates the mechanism of the human cranial nerve system. In the present disclosure, the machine learning model is an algorithm that processes vector data indicating image data as an input vector and outputs vector data indicating a recognition result as an output vector. The machine learning model that is operated is generated by learning and is therefore also called a "trained model". The trained model is the best computational model estimated to have the highest prediction accuracy and can therefore be referred to as the "best trained model". However, keep in mind that this best trained model is not always the “best in reality”.

The computer produces the best trained model by processing training data that contains many combinations of images and correct recognition results. The computer calculates the output vector indicating the recognition result by inputting the input vector indicating the image into the machine learning model. Then, the computer finds the error between the output vector and the recognition result shown in the training data, that is, the difference between the estimation result and the correct answer label. The computer then updates a given parameter in the machine learning model based on that error. The computer repeats such learning to generate the best trained model. In one example, this trained model is stored in the memory of the analysis system 1. The process of generating the best trained model can be called the training phase. The computer that generates the trained model is not limited, and may be, for example, an analysis system 1 or a computer or computer system different from the analysis system 1.

Analysis of a machine learning model is a process for clarifying the mechanism of a machine learning model. The analysis system 1 analyzes the machine learning model in order to understand the mechanism by which the machine learning model recognizes an image. In one example, the analysis of a machine learning model is to estimate the cause of the machine learning model's misrecognition of images.

FIG. 1 is a diagram showing an example of an analysis method by the analysis system 1 as a processing flow S1. The analysis system 1 processes a data set containing n combinations of the image x, the correct label k, and the attribute label r. The image x can be said to be the original image. The correct label is the output value (that is, the correct answer) that should be obtained by processing the image with a machine learning model. In one example, the correct label indicates the correct answer to the classification problem, that is, the class (group) to which the image belongs. The attribute label is a value indicating an attribute related to the image. Image-related attributes are arbitrary information that characterizes an image. In one example, a dataset is prepared by manually or automatically setting correct and attribute labels for individual images.

In step S11, the analysis system 1 executes image recognition by inputting the image x into the machine learning model. This machine learning model is a trained model and therefore its image recognition corresponds to the operational phase.

In step S12, the analysis system 1 executes a highlighting process for extracting a region of interest (also referred to as a “region of interest” in the present disclosure) that the machine learning model has focused on in its image recognition. In one example, the analysis system 1 extracts a region of interest from the image x based on the method described in Non-Patent Document 1 above, and generates a highlight image g indicating the region of interest. The number of images of the highlight image g is equal to the number of images of the image x. FIG. 2 is a diagram showing an example of a highlight image. In this example, the highlight image 202 obtained from image 201 particularly represents a sign plate indicating that the speed limit is 30 km / h as a region of interest. This means that the machine learning model paid particular attention to the sign board when recognizing the image 201. The black-filled portion in the highlight image 202, that is, the portion other than the region of interest, indicates that the region of interest is the region that the machine learning model has paid little or no attention to in image recognition.

In step S13, the analysis system 1 compares the result of the image recognition with the correct answer label k to determine whether or not the recognition result is correct, and obtains the correct / incorrect determination result c.

In step S14, the analysis system 1 executes a collation process for calculating the degree of relationship between the highlight image g (that is, the region of interest) and the attribute label r. The degree of relationship is an index indicating the strength of the relationship between the area of interest and the attribute label. In the present disclosure, it is assumed that the higher the degree of relationship, the stronger the relationship between the area of interest and the attribute label.

The analysis system 1 executes the processes of steps S11 to S14 for each combination in the data set. In step S15, the analysis system 1 executes an analysis based on the correctness determination result and the collation result (that is, the degree of relation) for n combinations to estimate the attribute label related to the incorrect image recognition. The analysis system 1 outputs the attribute label as an analysis result.

[System configuration]
FIG. 3 is a diagram showing an example of the hardware configuration of the computer 110 used for the analysis system 1. For example, the analysis system 1 has a control circuit 100. In one example, the control circuit 100 has one or more processors 101, a memory 102, a storage 103, a communication port 104, and an input / output port 105. Processor 101 executes the operating system and application programs. The storage 103 is composed of a non-temporary storage medium such as a hard disk, a non-volatile semiconductor memory, and a retrievable medium (for example, a magnetic disk, an optical disk, etc.), and stores an operating system and an application program. The memory 102 temporarily stores the program loaded from the storage 103 or the calculation result by the processor 101. In one example, the processor 101 functions as each functional module by executing a program in cooperation with the memory 102. The communication port 104 performs data communication with another device via the communication network NW in accordance with a command from the processor 101. The input / output port 105 executes input / output of an electric signal to / from an input / output device (user interface) such as a keyboard, a mouse, and a monitor in accordance with a command from the processor 101.

The storage 103 stores a program 120 for making the computer 110 function as the analysis system 1. When the processor 101 executes this program 120, each functional module of the analysis system 1 is realized. The program 120 may be provided after being fixedly recorded on a non-temporary recording medium such as a CD-ROM, a DVD-ROM, or a semiconductor memory. Alternatively, program 120 may be provided via a communication network as a data signal superimposed on a carrier wave.

The analysis system 1 may consist of one or more computers 110. When a plurality of computers 110 are used, one analysis system 1 is logically configured by connecting these computers 110 to each other via a communication network NW.

The computer 110 that functions as the analysis system 1 is not limited. For example, the analysis system 1 may be composed of a large computer 110 such as a business server, or a small computer 110 such as a personal computer.

FIG. 4 is a diagram showing an example of the functional configuration of the analysis system 1. In one example, the analysis system 1 includes an information processing device 10, a storage device 20, an input device 30, and an output device 40. In one example, the information processing device 10 operates according to an instruction signal input from the input device 30, accesses the storage device 20, reads and writes data, and outputs data indicating the processing result to the output device 40.

The information processing device 10 includes a data acquisition unit 11, an image recognition unit 12, a correctness determination unit 13, a highlight unit 14, a collation unit 15, and an analysis unit 16 as functional modules. The data acquisition unit 11 is a functional module for acquiring a data set. The image recognition unit 12 is a functional module that executes image recognition using a machine learning model. This machine learning model is a trained model, and therefore the image recognition performed by the image recognition unit 12 corresponds to the operation phase. The correctness determination unit 13 is a function module that determines whether or not the result of the image recognition is correct. The highlight unit 14 is a functional module that extracts a region of interest in the image recognition. The collation unit 15 is a functional element for calculating the degree of relationship indicating the relationship between the region of interest and the attribute label. The analysis unit 16 is a functional module that estimates attribute labels related to erroneous image recognition based on the correctness determination result of image recognition and the degree of relationship. The attribute label related to erroneous image recognition is the attribute label that is presumed to have caused the image recognition by the machine learning model to be erroneous. In short, the attribute label that contributed to erroneous recognition (misclassification). Is. Note that the "attribute label associated with erroneous image recognition" is the result estimated by the analysis system 1 and is not necessarily the attribute label that "realistically" misleads the image recognition by the machine learning model. I want to be.

The storage device 20 temporarily or permanently stores various data processed by the analysis system 1. In one example, the storage device includes a data set prepared in advance, an image recognition result obtained by the image recognition unit 12, a correctness determination result obtained by the correctness determination unit 13, and a highlight result obtained by the highlight unit 14. , At least one of the collation result (relationship degree) obtained by the collation unit 15 and the analysis result obtained by the analysis unit 16 is stored.

[Attribute label]
At least one attribute label is set for each image in the dataset, that is, each original image. The attributes expressed as attribute labels are not limited. The target to which the attribute label is set is not limited, and for example, the attribute label may be set for the entire image or may be set for any partial area in the image. Examples of attribute labels set for the entire image include shooting conditions (eg, shooting date, weather), and attributes for the entire image (eg, subject type, resolution, contrast, brightness, noise). , Not limited to these. Examples of attribute labels set for a sub-region are geometric information (eg, position, area, shape in the image) and attributes for that sub-region (eg, type, color, orientation, brightness, obstruction). (Presence or absence), but is not limited to these. The partial area may be set arbitrarily. The partial area may be set for each object (that is, a subject), and may be set for each object such as a person, a car, or a sign. Alternatively, the subregion may be set for each individual component of the individual object (subject). For example, if the object (subject) is a sign, a partial area may be set for each individual component such as a sign board, a border, an illustration, characters, and a support. Alternatively, the subregion may be set for each superpixel or for each pixel. A super pixel is an area composed of a plurality of consecutively arranged pixels having similar pixel information such as brightness and color.

In one example, in each image, at least one attribute label is set for each pixel. At least one attribute label corresponding to a certain pixel can be represented by a data structure represented by a one-dimensional array, and the data structure is referred to as a "pixel label vector" in the present disclosure. Each element of the pixel label vector indicates a corresponding attribute label. Therefore, the number of elements in the pixel label vector is equal to the total number of attribute label types. For example, if the pixel label vector of the (i, j) th pixel of a certain image is r _ij , the individual elements of the pixel label vector are e, and the total number of attribute label types is T, r _ij = [e. _ij ¹ , e _ij ² , ..., e _ij ^T ]. The values set for the individual elements of the pixel label vector are not limited. In one example, at least one element may be represented by a binary value of 0 or 1. For example, when a certain attribute label is given to a certain pixel, 1 is set for the element corresponding to the attribute label, and when the attribute level is not given to the pixel, the element is set. May be set to 0. As a specific example, when the pixel label vector of a pixel is _rij = [1,1,0,0, ..., 0,1], this pixel corresponds to the first and second elements. Has two attribute labels, and does not have two attribute labels corresponding to the third and fourth elements. At least one element of the pixel label vector may be represented by a continuous value, and for example, the resolution may be set as it is in the element corresponding to the resolution of the image.

FIG. 5 is a diagram showing some examples of attribute labels. Example (a) of FIG. 5 shows an image 211 showing a sign of merging and an illustration 212 for explaining the attribute label of the image 211. In the example (a), "empty", "sign", "tree", and "shielded" are set as the attribute labels of the image 211. Specifically, the pixel corresponding to the sky is given the attribute label "empty", the pixel corresponding to the sign is given the attribute level "mark", and the pixel corresponding to the tree is given the attribute label "tree". Granted. The attribute label "shielded" is given to the pixel corresponding to the portion where the sign is blocked by the tree. In this example, the "shielded" attribute label indicates that the sign is blocked by another object.

Example (b) of FIG. 5 shows an image 221 showing a sign indicating that the vehicle is prohibited from traveling outside the designated direction, and an illustration 222 for explaining the attribute label of the image 221. In the example (b), "empty", "sign", "tree", "pillar", and "shielded" are set as the attribute labels of the image 221. Specifically, the pixel corresponding to the sky is given the attribute label "empty", the pixel corresponding to the marker is given the attribute level "mark", and the pixel corresponding to the tree is given the attribute label "tree". The attribute label "pillar" is given to the pixel corresponding to the pillar. The attribute label "with shielding" is given to the pixel corresponding to the portion where the sign is blocked by the pillar. Again, the "shielded" attribute label indicates that the sign is blocked by another object.

It is assumed that a pixel label vector indicating five types of attribute labels, that is, sky, sign, tree, pillar, and presence / absence of occlusion, in this order is commonly defined in images 211 and 221. Then, in each element of the pixel label vector, 0 is given when the attribute label is not given, and 1 is given when the attribute label is given. An example of setting the attribute label is shown under this premise. Regarding the image 211, the pixel label vector corresponding to the unobstructed marker portion is [0,1,0,0,0]. The pixel label vector corresponding to the portion of the sign blocked by the tree is [0,0,1,0,1]. The pixel label vector corresponding to the part of the tree that does not block the sign is [0,0,1,0,0]. For the image 211 in which the pillar is not shown, there is no pixel label vector in which the fourth vector element is 1. Similarly for image 221 the pixel label vector corresponding to the sky is [1,0,0,0,0]. The pixel label vector corresponding to the portion of the sign blocked by the pillar is [0,0,0,1,1]. The pixel label vector corresponding to the part of the pillar that does not block the sign is [0,0,0,1,0].

[Processing procedure in the system]
The operation of the analysis system 1 will be described with reference to FIGS. 6 to 9, and the analysis method according to the present embodiment will be described. FIG. 6 is a flowchart showing an example of the overall flow of the analysis process as the process flow S2. FIG. 6 also shows the correspondence with the processing flow S1. FIG. 7 is a flowchart showing in detail an example of the process of determining the correctness of image recognition. FIG. 8 is a flowchart showing in detail an example of the process of calculating the degree of relationship between the region of interest and the attribute label. FIG. 9 is a flowchart showing in detail an example of the process of estimating the attribute label related to erroneous image recognition.

As shown in FIG. 6, in step S21, the data acquisition unit 11 acquires the data set. As described above, the data set is prepared in advance and stored in a predetermined storage device 20 in advance. The data acquisition unit 11 accesses the storage device 20 and reads out the data set.

In step S22, the image recognition unit 12 executes image recognition by the machine learning model for each image of the data set. This corresponds to step S11. For each of the one or more images in the data set, the image recognition unit 12 inputs the image into the machine learning model, acquires the data output from the machine learning model as the image recognition result, and stores the image recognition result. It is stored in the device 20.

式（１）は、画像がＫ個のクラス（グループ）のいずれに属するかを求めることを示す。［０，１］は、或るクラス（グループ）に属するか否かの二択を示す。変数ｐは画像が或るクラスに属する確率であり、変数ｐ_ｋは画像がクラスｋに属する確率を示す。分類モデルｆは、画像ｘを確率に変換する関数ｐ＝ｆ（ｘ）であるともいえる。分類モデルｆを用いて得られる結果はｋ^＊＝ａｒｇｍａｘ_ｋｐ_ｋで表される。これは、最大のｐ_ｋが得られるクラスｋを認識結果ｋ^＊として得ることを意味する。 In one example, this image recognition corresponds to the classification model represented by the following equation (1).

Equation (1) indicates which of the K classes (groups) the image belongs to. [0,1] indicates two choices as to whether or not it belongs to a certain class (group). The variable p is the probability that the image belongs to a certain class, and the variable p _k is the probability that the image belongs to the class k. It can be said that the classification model f is a function p = f (x) that converts the image x into a probability. Results obtained using a classification model f is expressed by ^_k * = argmax ^k p ^k. This means that the class k from which the maximum p _k is obtained is obtained as the recognition result k ^* .

In step S23, the highlight unit 14 extracts an area of interest in image recognition by the image recognition unit 12 for each image in the data set. This corresponds to step S12. As described above, in one example, the highlight unit 14 extracts the region of interest based on the method described in Non-Patent Document 1 and generates a highlight image showing the region of interest. As a result, a highlight image as shown in FIG. 2 is generated for each of the one or more images in the dataset. In one example, each pixel of the highlighted image is associated with a degree of influence on a small change in the corresponding pixel in the original image (ie, image x). This degree of influence is a scalar value, which is an example of information indicating a region of interest. The degree of influence of all pixels of the highlight image corresponding to the image x classified in the class k _{can be expressed by the gradient vector ∇ x} f _k (x). The number of elements in the gradient vector is equal to the number of pixels in the highlight image and therefore equal to the number of pixels in the original image. A gradient vector indicating the degree of influence of each pixel is also an example of information indicating a region of interest. The highlight unit 14 stores the highlight image including the gradient vector in the storage device 20 as the highlight result.

In step S24, the correctness determination unit 13 determines the correctness of image recognition for each image in the data set. This corresponds to step S13. The detailed flow of this determination process will be described with reference to FIG. 7.

In step S241, the correctness determination unit 13 selects one image to be processed. In step S242, the correctness determination unit 13 acquires the image recognition result for the image. This image recognition result is data output from the machine learning model in the image recognition unit 12. In step S243, the correctness determination unit 13 refers to the data set and acquires the correct answer label corresponding to the image. In step S244, the correctness determination unit 13 determines the correctness of image recognition by comparing the image recognition result with the correct answer label. The correctness determination unit 13 determines that the image recognition is correct when the image recognition result matches the correct answer label, and determines that the image recognition is incorrect otherwise. The correctness determination unit 13 stores the correctness determination result in the storage device 20. As shown in step S245, the correctness determination unit 13 executes the processes of steps S241 to S244 for each image of the data set. Step S24 ends when all the images in the dataset have been processed (YES in step S245).

Returning to FIG. 6, in step S25, the collating unit 15 calculates the degree of relationship between the region of interest and the attribute label for each image in the data set. This corresponds to step S14. The detailed flow of this calculation process will be described with reference to FIG.

In step S251, the collating unit 15 selects one image to be processed. In step S252, the collating unit 15 acquires the highlight result for the image. This highlight result is a highlight image generated by the highlight unit 14. In step S253, the collation unit 15 refers to the data set and acquires one or more attribute labels corresponding to the image. In step S254, the collating unit 15 calculates the degree of relationship between the region of interest and the attribute label based on the highlight result and the attribute label. Specifically, the collating unit 15 calculates the inner product of the gradient vector and the attribute label as the degree of relationship for each of the one or more attribute labels. This inner product is an example of the degree of relationship. The larger the inner product, the higher the degree of relationship. Therefore, the larger the inner product, the stronger the relationship between the region of interest and the attribute label.

The calculation of the inner product for one image will be described. Matching unit 15 for each of the one or more attribute label t, it generates a vector r _t consisting of a set of element values of the attribute label t corresponding to all pixels in the image. In the present disclosure refers to the vector r _t be an "image label vector". The number of elements of the image label vector corresponding to one attribute label t is equal to the number of pixels of the image. Matching unit 15 calculates the inner product between the image label vector _{r t} and the gradient vector ∇ _x f _{k (x).} This inner product is expressed by the following equation (2). Equation (2) represents for the n-th image, the inner product of the image label vector _{r t} and the gradient vector ∇ _x f _{k (x).}

As shown in step S255, the collating unit 15 executes the processes of steps S251 to S254 for each image of the data set. When all the images in the dataset have been processed (YES in step S255), step S25 ends.

Returning to FIG. 6, in step S26, the analysis unit 16 estimates the attribute label related to the erroneous image recognition based on the collation result and the correctness determination result. This corresponds to step S15. The detailed flow of this estimation process will be described with reference to FIG.

In step S261, the analysis unit 16 acquires the collation results for the individual images in the dataset. In step S262, the analysis unit 16 acquires the correctness determination result for each of the images. In step S263, the analysis unit 16 estimates the attribute label related to the erroneous image recognition based on the collation result and the correctness determination result. In this embodiment, two estimation methods are illustrated below.

ここで、ｘ_ｉｊは画素値を示す。α_ｔは補正重みを示し、これは０以上１以下の範囲で設定される。補正重みα_ｔは画素を変更するためのパラメータであるともいえ、より具体的には、個々の画素値を変更するためのパラメータであるともいえる。ｒ_ｉｊｔは画素に設定された属性ラベルｔの値を示し、したがって、ｒ_ｉｊｔが０であれば画素値は補正されない。 The first method is based on the consideration that if the image can be correctly recognized by correcting the pixels that contributed to the erroneous image recognition (that is, misclassification), the attribute label that contributed to the correction is related to the erroneous recognition. In the first method, the analysis unit 16 corrects the pixels using the correction weights. The correction weight is an example of the degree of contribution, which is an index indicating how much the attribute label contributed to erroneous image recognition. In the present disclosure, it is assumed that the attribute label having a larger correction weight has a higher degree of contribution to erroneous image recognition. The pixel correction for one attribute label t is expressed by the following equation (3).

Here, x _ij indicates a pixel value. α _t indicates a correction weight, which is set in the range of 0 or more and 1 or less. It can be said that the correction weight α _t is a parameter for changing the pixel, and more specifically, it can be said that it is a parameter for changing the individual pixel value. r _ijt indicates the value of the attribute label t set for the pixel, and therefore, _{if r ijt} is 0, the pixel value is not corrected.

ここで、ｆ_ｋは補正された画像を確率に変換する関数（分類モデル）を示す。ａｒｇｍａｘ_ｋｆ_ｋは、その確率が最大になるクラスｋを認識結果として得ることを意味する。 The analysis unit 16 searches for _{a correction weight α t} that correctly classifies images with incorrect image recognition results by correction and correctly classifies images with correct image recognition results even after correction. This search is a process for finding attribute labels related to misclassification under the constraint of considering correctly classified images. The combination of the image x, the attribute label r, and the correct label k corresponding to the misclassified N images is expressed as {x ⁽ⁿ⁾ , r ⁽ⁿ⁾ , k ⁽ⁿ⁾ } _{n = 1} ^N. Further, corresponding to the correctly classified the M image, image x', attribute label r', and a combination of true label ^{k'{x'(m), r'} (n), k'(n)} It is expressed as _{m = 1} ^M. The analysis unit 16 identifies these combinations using the correctness determination result. In this case, it can be said that _{the search for the correction weight α t is to search for α t} that satisfies the following equation (4).

Here, f _k indicates a function (classification model) that converts the corrected image into a probability. argmax _k f _k means that the class k having the maximum probability is obtained as the recognition result.

The analysis unit 16 _{acquires the attribute label corresponding to the maximum correction weight α t} as the attribute label related to erroneous image recognition. The objective function indicating the search for the correction weight α _t is defined by the following equation (5).

The analysis unit 16 approximates this objective function by a first-order Taylor expansion. This approximation is defined by the following equation (6), and in this calculation, the inner product (relationship degree) which is the collation result is used.

_{The analysis unit 16 calculates the maximum correction weight α t} based on the equation (6), and acquires the attribute label corresponding to the maximum correction weight α _t as an attribute label related to erroneous image recognition.

Similar to the first method, in the second method, if the images can be correctly classified by correcting the pixels that contributed to the incorrect image recognition (that is, misclassification), the attribute label that contributed to the correction is incorrect. Based on the consideration that it is related to cognition. In the second method, the correction weight α _t is used as in the first method.

Also in the second method, the analysis unit 16 searches for _{a correction weight α t} that correctly classifies the image with the incorrect image recognition result by correction and correctly classifies the image with the correct image recognition result even after the correction. .. Similar to the first method, the analysis unit 16 correctly classified the combination of the image x, the attribute label r, and the correct answer label k corresponding to the N misclassified images using the correct / incorrect judgment result. The combination of the image x', the attribute label r', and the correct label k'corresponding to the M images is specified.

ここで、ｆ_ｋ（ｎ）は画像が正解のクラスに属する確率を示し、ｆ_ｈは画像が誤ったクラス（すなわち、正解クラス以外のクラス）に属する確率を示す。したがって、この目的関数は、正しく分類された画像を補正後も正しく分類しつつ、正解クラスの確率と誤ったクラスの確率との差が最も大きくなるような補正重みα_ｔを得ることを目的とする。 In the second method, the _{objective function indicating the search for the maximum correction weight α t} is defined by the following equation (7).

Here, f _{k (n)} indicates the probability that the image belongs to the correct class, and f _h indicates the probability that the image belongs to the wrong class (that is, a class other than the correct class). Therefore, the purpose of this objective function is to obtain _{a correction weight α t} that maximizes the difference between the probability of the correct class and the probability of the wrong class while correctly classifying the correctly classified images even after correction. do.

The analysis unit 16 approximates this objective function by a first-order Taylor expansion. In one example, this approximation is defined by the following equation (8), and in this calculation, the inner product (relationship degree) which is the collation result is used.

_{The analysis unit 16 calculates the maximum correction weight α t} based on the equation (8), and acquires the attribute label corresponding to the maximum correction weight α _t as an attribute label related to erroneous image recognition.

ここで、λΣ_ｍβ_ｍは或る一定の範囲で制約を緩和するための制約項である。λはその制約項を制御するためのパラメータである。γΣ_ｔ｜α_ｔ｜は、補正重みα_ｔを疎にして（すなわち、なるべく多くの補正重みα_ｔを０にして）、重要な属性ラベルを抽出することを目的とする項である。γは、複数の属性ラベルが誤認識に関連したものとして推定されないようにするためのパラメータである。言い換えると、γは、誤認識の原因と推定される属性ラベルを少なくするためのパラメータである。 As a modification of the second method, the analysis unit 16 may approximate the objective function by a first-order Taylor expansion represented by the following equation (9). _{The analysis unit 16 calculates the maximum correction weight α t} based on the equation (9), and acquires the attribute label corresponding to the maximum correction weight α _t as an attribute label related to erroneous image recognition. In this calculation as well, the inner product (degree of relationship), which is the collation result, is used.

Here, λΣ _m β _m is a constraint term for relaxing the constraint in a certain range. λ is a parameter for controlling the constraint term. γΣ _t | α _t | is a term for the purpose of extracting important attribute labels by making the correction weight α _t sparse (that is, _{setting as many correction weights α t as possible to 0).} γ is a parameter for preventing a plurality of attribute labels from being presumed to be related to misrecognition. In other words, γ is a parameter for reducing the attribute label presumed to be the cause of misrecognition.

In both the first and second methods, the new image recognition result obtained by inputting the first corrected image obtained by applying the correction weight to the image for which the image recognition result was correct is correctly input to the machine learning model. In addition, the correction weight contributes to the correction of the new image recognition result obtained by inputting the second correction image obtained by applying the correction weight to the image in which the image recognition result is incorrect into the machine learning model. This is an example of processing calculated as a degree. The analysis unit 16 calculates the correction weight for each of the one or more attribute labels. Then, the analysis unit 16 estimates the attribute label having the highest correction weight (contribution degree) as the attribute label related to erroneous image recognition.

Both the first and second methods use both an image with a correct image recognition result and an image with an incorrect image recognition result, but if there are few or no images with incorrect image recognition results, the image recognition result is correct. You may search for α using only. In this search, the correctness determination unit 13 is used to calculate the probability of belonging to the correct answer class in the input image. The combination of the image x, the attribute label r, and the correct answer label k corresponding to N images classified below the threshold value with the probability of the correct answer class as the threshold value is {x ⁽ⁿ⁾ , r ⁽ⁿ⁾ , k ^{(n). )} } _{N = 1} ^N. Further, the combinations of the image x', the attribute label r', and the correct answer label k'corresponding to the M images classified by the threshold value or more are {x' ^(m) , r ^{'(n )} , k'(n). ⁾ } ^M _{= 1 M.} In the above combination, by using either of the first and second methods, the attribute label affecting the erroneous recognition is estimated.

The analysis unit 16 outputs the estimated attribute label as the analysis result. The output method of the analysis result is not limited. For example, the analysis unit 16 may store the analysis result in the storage device 20, display it on a monitor, or transmit it to another computer.

[effect]
As described above, the computer system according to one aspect of the present disclosure includes a processor. The processor acquires a correct / incorrect judgment result indicating whether or not the image recognition result obtained by inputting the image into the machine learning model is correct for each of the plurality of images, and machine learning is performed for each of the plurality of images. The degree of relationship indicating the relationship between the region of interest in the image in the model and each of the one or more attribute labels associated with the image in advance is calculated, and based on the correctness determination result and the degree of relationship of each of the plurality of images. , The attribute label related to erroneous image recognition is estimated from one or more attribute labels.

The program according to one aspect of the present disclosure includes, for each of a plurality of images, a step of acquiring a correct / incorrect judgment result indicating whether or not the image recognition result obtained by inputting the image into a machine learning model is correct. For each of the plurality of images, a step of calculating the degree of relationship indicating the relationship between the region of interest in the image in the machine learning model and each of one or more attribute labels associated with the image in advance, and a step of calculating the degree of relationship of the plurality of images. A computer is made to perform a step of estimating an attribute label related to erroneous image recognition from one or more attribute labels based on each correctness determination result and the degree of relationship.

In such an aspect, by obtaining the above-mentioned degree of relationship, the attribute label related to the region of interest of machine learning in the image can be grasped. Then, by considering the degree of relationship and the correctness of image recognition in a plurality of images, it is estimated which attribute label affected the incorrect image recognition. By executing this series of processes, the machine learning model for image recognition can be automatically analyzed. This automatic analysis makes it possible to efficiently identify the cause of misrecognition in a machine learning model by analyzing a large number of images, for example, without relying on visual inspection by an expert. In another example, the cause of misrecognition, which cannot be covered by the knowledge of experts, can be identified by its automatic analysis.

In the computer system according to the other aspect, the processor calculates the degree of contribution to erroneous image recognition for each of one or more attribute labels based on the correctness judgment result and the degree of relation of each of the plurality of images. From the above attribute labels, at least the attribute label having the highest contribution may be estimated as the attribute label related to erroneous image recognition. By calculating the contribution of each attribute label and selecting the attribute label with the highest contribution, the attribute label estimated to be most related to erroneous image recognition can be extracted.

In the computer system according to the other aspect, a new image recognition result obtained by the processor inputting the first corrected image obtained by applying the correction weight to the image for which the image recognition result was correct is input to the machine learning model. The correction weight that corrects the new image recognition result obtained by inputting the second correction image obtained by applying the correction weight to the image that is correct and the image recognition result is not correct into the machine learning model. The degree of contribution may be calculated for each of one or more attribute labels. If the image can be correctly recognized by correcting the pixels that contributed to the erroneous image recognition, it is highly probable that the attribute label that contributed to the correction is related to the erroneous recognition. Therefore, by using the above correction weight as the contribution degree, it is possible to accurately estimate the attribute label related to the erroneous image recognition.

In a computer system according to another aspect, the processor may estimate the attribute label having the highest contribution by approximating the objective function for determining the contribution by a first-order Taylor expansion. Since the calculation of contribution can be simplified by introducing the first-order Taylor expansion, the attribute label related to erroneous image recognition can be estimated in a shorter time.

In the computer system according to the other aspect, the processor may calculate the inner product of the gradient vector indicating the region of interest and the image label vector indicating the attribute label as the degree of relationship. By using this inner product, the degree of relationship between each attribute label and the region of interest can be easily obtained, so that the attribute label related to erroneous image recognition can be estimated in a shorter time.

[Modification example]
The above description has been made in detail based on the embodiments of the present disclosure. However, the present disclosure is not limited to the above embodiment. The present disclosure can be modified in various ways without departing from its gist.

In the above embodiment, the analysis system 1 calculates the image recognition result, the correctness determination result, and the highlight result. However, the computer system according to the present disclosure may obtain at least one of these results from an external computer system. That is, at least one of image recognition, correctness determination, and highlighting processing is not essential processing in the computer system according to the present disclosure.

In the above embodiment, the inner product is shown as an example of the degree of relationship indicating the relationship between the region of interest and the attribute label, but parameters other than the inner product may be used as the degree of relationship. In the above embodiment, the correction weight is shown as an example of the contribution degree indicating how much the attribute label contributed to the erroneous image recognition, but a parameter other than the correction weight may be used as the contribution degree.

In the above embodiment, the analysis unit 16 estimates the attribute label having the highest contribution (correction weight) as the attribute label related to erroneous image recognition. However, the estimated attribute label is not limited to this. For example, the computer system according to the present disclosure may estimate one or more attribute labels whose contribution (eg, correction weight) is greater than or equal to a given threshold as attribute labels associated with erroneous image recognition. In any case, the computer system according to the present disclosure estimates at least the attribute label having the highest contribution (correction weight) as the attribute label related to erroneous image recognition.

In the present disclosure, the expression "the processor executes the first process, executes the second process, ... executes the nth process", or the expression corresponding thereto is the first process. The concept including the case where the execution subject (that is, the processor) of n processes from the nth process to the nth process changes in the middle is shown. That is, this expression shows a concept including both a case where all n processes are executed by the same processor and a case where the processor changes according to an arbitrary policy in n processes.

When comparing the magnitude relations of two numbers in a computer system, either of the two criteria "greater than or equal to" and "greater than" may be used, and the two criteria "less than or equal to" and "less than" Either of them may be used. The selection of such criteria does not change the technical significance of the process of comparing the magnitude relations of two numbers.

The processing procedure of the method executed by the processor is not limited to the example in the above embodiment. For example, some of the steps (processes) described above may be omitted, or each step may be executed in a different order. In addition, any two or more steps of the above-mentioned steps may be combined, or a part of the steps may be modified or deleted. Alternatively, other steps may be performed in addition to each of the above steps.

The embodiments described in all or part of the above embodiments include control of image processing, improvement of processing speed, improvement of processing accuracy, improvement of usability, improvement of functions using data, provision of appropriate functions, and the like. Providing appropriate data, programs, recording media, devices and / or systems, such as improving or providing appropriate functions, reducing the capacity of data and / or programs, miniaturizing devices and / or systems, and data, programs. Any one of data, programs, recording media, equipment and / or system production / manufacturing optimization such as reduction of production / manufacturing cost of equipment or system, facilitation of production / manufacturing, shortening of production / manufacturing time, etc. Solve one problem.

1 ... analysis system, 10 ... information processing device, 11 ... data acquisition unit, 12 ... image recognition unit, 13 ... correctness judgment unit, 14 ... highlight unit, 15 ... collation unit, 16 ... analysis unit, 20 ... storage device.

Claims (6)

Equipped with a processor
The processor
For each of the plurality of images, a correct / incorrect judgment result indicating whether or not the image recognition result obtained by inputting the image into the machine learning model is correct is acquired.
For each of the plurality of images, the degree of relationship indicating the relationship between the region of interest in the image in the machine learning model and each of one or more attribute labels associated with the image in advance was calculated.
Based on the correctness determination result and the degree of relationship of each of the plurality of images, the attribute label related to the erroneous image recognition is estimated from the one or more attribute labels.
Computer system. The processor
For each of the one or more attribute labels, the degree of contribution to the erroneous image recognition is calculated based on the correctness determination result and the degree of relationship of each of the plurality of images.
From the one or more attribute labels, at least the attribute label having the highest contribution is estimated as the attribute label related to the erroneous image recognition.
The computer system according to claim 1. The new image recognition result obtained by the processor inputting the first corrected image obtained by applying the correction weight to the image for which the image recognition result was correct is input to the machine learning model, and the new image recognition result is correct and said. The contribution of the correction weight that makes the new image recognition result correct by inputting the second correction image obtained by applying the correction weight to the image in which the image recognition result is incorrect into the machine learning model. Calculated for each of the above 1 or more attribute labels.
The computer system according to claim 2. The processor estimates the attribute label with the highest contribution by approximating the objective function for the contribution by a first-order Taylor expansion.
The computer system according to claim 2 or 3. The processor calculates the inner product of the gradient vector indicating the region of interest and the image label vector indicating the attribute label as the degree of relationship.
The computer system according to any one of claims 1 to 4. For each of the plurality of images, a step of acquiring a correct / incorrect judgment result indicating whether or not the image recognition result obtained by inputting the image into the machine learning model is correct, and
For each of the plurality of images, a step of calculating the degree of relationship indicating the relationship between the region of interest in the image in the machine learning model and each of one or more attribute labels associated with the image in advance, and
A program that causes a computer to perform a step of estimating the attribute label related to erroneous image recognition from the one or more attribute labels based on the correctness determination result and the degree of relationship of each of the plurality of images.

Images