JP2021060692A - Inference result evaluation system, inference result evaluation device, and method thereof - Google Patents

Abstract

To provide an inference result evaluation system which can visualize which element or which part of learning data has had an influence on an inputted inference result of input data with respect to a machine learning model.SOLUTION: In accordance with an embodiment, a server 12 of a monitoring system 1 executes inference using a machine learning model 36 on the basis of test data, extracts learning data having contributed to an inference result of input data, as extracted learning data from a plurality of pieces of learning data, calculates an influence degree on the inference result of each of elements or each of prescribed parts of the extracted learning data, and visualizes the influence degree of each of the elements or each of the prescribed parts of the extracted learning data.SELECTED DRAWING: Figure 5

Description

Embodiments of the present invention relate to an inference result evaluation system, an inference result evaluation device, and a method thereof.

Inference techniques using machine learning models obtained by learning by machine learning are widely used. The machine learning model is generated using the learning data which is the teacher data. By using a machine learning model, it is possible to obtain inference results for the input data, but it is not possible to know which element of the training data influenced the inference results of the input data.

Japanese Unexamined Patent Publication No. 2019-82883

Therefore, the embodiment is an inference result evaluation system, an inference result evaluation device, and an inference result evaluation device capable of visualizing which element or part of the training data affects the inference result of the input input data in the machine learning model. An object of the present invention is to provide an inference result evaluation method.

The inference result evaluation system of the embodiment includes an input data designation unit that specifies input data, an inference unit that executes inference using a machine learning model based on the input data, and the inference unit for the input data. The learning data that contributed to the inference result is used as the extraction learning data, and the learning data extraction unit that extracts the learning data from a plurality of learning data and the degree of influence on the inference result are calculated for each element or a predetermined part of the extracted learning data. It has an influence degree calculation unit and an influence degree visualization unit that visualizes the influence degree for each element or each predetermined part in the extracted learning data.

It is a block diagram of the monitoring system which concerns on embodiment. It is a block diagram of the server which concerns on embodiment. It is a figure which shows the flow of the process of the model learning phase in the processor which concerns on embodiment. It is a figure which shows the data structure of the learning database which concerns on embodiment. It is a figure which shows the example of the process flow which analyzes which element or the part of the learning data had a great influence on the inference result about the test data which concerns on embodiment, and displays the analysis result. It is a figure which shows the example of the process flow of the inference execution part which concerns on embodiment. It is a figure which shows the flow of the correction processing of the learning data which concerns on embodiment. It is a figure which shows the example of the graphical user interface generated based on GUI data which concerns on embodiment. It is a figure which shows the other example of the graphical user interface generated by the result output part which concerns on embodiment. It is a figure which shows the display example of the graphical user interface after re-learning which concerns on embodiment. It is a flowchart which shows the flow of the process from the supply of the test data to the server to the re-learning of learning data which concerns on embodiment.

Hereinafter, embodiments will be described with reference to the drawings.
(Constitution)
FIG. 1 is a configuration diagram of a monitoring system according to the present embodiment. The monitoring system 1 is an inference system including a monitoring device 11, a server 12 on the cloud 12a, and a network 13 such as the Internet that connects the monitoring device 11 and the server 12. The monitoring system 1 of the present embodiment is a system for a service that infers an object or the state of the object to be reflected in an image using a machine learning model that is an inference model and outputs the inference result.

The monitoring device 11 includes a personal computer (hereinafter referred to as a PC) 21, a camera 22 as an imaging device, and a monitor 23 as a display device. A camera 22 is connected to the PC 21, and an image pickup signal from the camera 14 is input. The PC 21 transmits the image data of the image pickup signal to the server 12 via the network 13. A mouse 21a and a keyboard 21b are connected to the PC 21. A mouse 21a, a keyboard 21b, and a monitor 23 constitute a graphical user interface (hereinafter referred to as GUI).

The server 12 can output the inference result for the received image data by using the machine learning model. In the present embodiment, the inference result for the image data transmitted from the PC 21 is transmitted from the server 12 to the PC 21 via the network 13. The PC 21 outputs the inference result to the monitor 23. The user of the monitoring system 1 can know the inference result displayed on the monitor 23.

In the following embodiment, the input data of the machine learning model is only the image data, but the detection data of the detection device 22a for detecting temperature, pressure, sound, etc. as shown by the dotted line is also combined with the image data. It may be used as input data.

Furthermore, in the present embodiment, the learning function, the inference function, and the visualization function described later of the monitoring system 1 are provided in the server 12, but the PC 21 of the monitoring device 11 provides the learning function, the inference function, and the visualization function. You may have.

FIG. 2 is a configuration diagram of the server 12. The server 12 has an inference device 31 and a communication interface (hereinafter, referred to as a communication I / F) 32 with the network 13. The inference device 31 can communicate with the monitoring device 11 via the network 13 by the communication I / F 32. The inference device 31 includes a processor 33 and a memory 34. The processor 33 includes a central processing unit (hereinafter referred to as a CPU), a ROM, and a RAM, and realizes a predetermined function by reading the memory 34 and the software program stored in the ROM, expanding the software program into the RAM, and executing the program. To do. Predetermined functions include a learning function, an inference function, and a visualization function, which will be described later.

The processor 33 may be configured by other processing devices such as GPU (Graphics Processing Unit), FPGA (Field Programmable Gate Array), and TPU (Tensor Processing Unit) instead of the CPU, or the CPU, GPU, It may be configured in combination with at least two processing devices such as FPGA and TPU.

Further, in the present embodiment, the server 12 is a server on the cloud 12a, but may be a so-called on-premises server.

The memory 34 is a rewritable non-volatile memory, for example, a hard disk drive. The memory 34 includes a program area 34a in which various software programs such as a learning program, an inference program, and a visualization processing program are stored. Further, the memory 34 stores a database for learning data (hereinafter referred to as a learning database) 35 and a machine learning model 36 including machine learning model parameters (hereinafter referred to as model parameters) of the machine learning model 36. There is.

A plurality of learning data (training data) are stored in the learning database 35. Each learning data includes one input data and correct label data for the input data. Here, one input data is image data, and one correct label is identification class information.

In the present embodiment, the input data is image data, but vector data and numerical string data related to IoT (Internet of Things) may be used, and the input data may be vector data and numerical values in addition to the image data. Other data such as column data may also be included. The correct label data may be objective variables, classification class information such as normal / abnormal, and the like.

The machine learning model 36 includes data of model parameters of the inference model by deep learning. The machine learning model 36 outputs an inference result based on the input data. For example, when the model parameter is θ and the number of data is N, the machine learning model 36 outputs an inference result y _i (x _t , θ) _{based on the input data x t.}

The machine learning model 36 is not limited to the deep learning method, but may be a method such as logistic regression or support vector machine (SVM).
(Learning)
The learning function of the server 12 will be described. FIG. 3 is a diagram showing a processing flow of the model learning phase in the processor 33. FIG. 4 is a diagram showing a data structure of the learning database 35. The learning unit 41 that realizes the learning function is a learning program stored in the program area 34a of the memory 34. The learning unit 41 includes a data input unit 41a, a data preprocessing unit 41b, an inference execution unit 41c, and a model parameter update unit 41d.

The data input unit 41a acquires the learning data from the learning database 35 and supplies it to the data preprocessing unit 41b. The data pre-processing unit 41b executes a predetermined process as pre-processing on the input data which is the learning data, and converts the input data. For example, the data preprocessing unit 41b performs normalization processing so that the values of the input data have an average of 0 and a variance of 1. The data preprocessing unit 41b may perform inflating processing. Inflating is to add noise to the input data or to deform the image such as translation, rotation, and scaling.

As shown in FIG. 4, the training data includes a plurality of input data, and each training data is composed of table data TBL including an identification number (No.), link information, label information, and meta information. There is. The identification number is an identifier for identifying each learning data. The link information is information indicating a memory area in which input data is stored. The label information is the information of the correct answer label.

The meta information is information indicating a learning target portion in the input data (hereinafter referred to as learning target portion information). In the present embodiment, since the input data is image data, the learning target partial information is the learning target area information in the image data. That is, the meta information is information indicating which part or region of the input data is used as the input data to be learned.

The inference execution unit 41c executes inference on the input data preprocessed by the data preprocessing unit 41b. Here, the inference is first performed by extracting a predetermined feature amount from the image data of the region specified by the meta information in the image data, and using the machine learning model 36 for the extracted feature amount. For example, by default, the meta information sets the entire area of the image data as the learning target area, that is, the gaze area, but as described later, when the learning area, that is, the gaze area of the learning data is set by the user, the meta information is gazed. The area is changed to the set gaze area (hereinafter referred to as a designated gaze area). That is, one or more gaze areas (including a designated gaze area) can be set in the meta information.

When a plurality of gaze areas are set in one image data in the meta information, weighting is performed for each gaze area, and each probability of the inference result for each gaze area is changed according to the weighting. You may.

The inference execution unit 41c extracts a predetermined feature amount from the input data of the learning target area specified by the meta information, for example, based on the model parameters of the machine learning model 36. Based on the extracted features, the inference execution unit 41c outputs the inference result to the model parameter update unit 41d.

The model parameter update unit 41d updates the model parameters based on the execution result of the inference execution unit 41c. Here, the loss function L (y _t (x _t , θ), z _t ) is defined. The loss function is a function that becomes smaller as the correct label z and the inference result y match. For example, for regression problems, squared error is used, and for classification problems, cross entropy is used.

ここでは、式（１）は、Λ（θ）の正則化項を含み、その正則化項を含めて最小化される。最小値は、勾配降下法などを用いて求められる。正則化項に特定の関数、例えばＬ２ｎｏｒｍ（全てのθの二乗和）、を選ぶと、θ^＊を、入力データｘ_ｔの関数とみることが出来き、その結果、推論結果ｙ（ｘ，θ^＊）を、ｘ_ｔで微分可能となる。従って、後述するように、テストデータの推論結果に寄与した入力データの要素あるいは部分を求めることができる。多くの学習データに対して学習部４１が実行されることにより、モデルパラメータが得られ、機械学習モデル３６が生成される。 Therefore, learning the model parameters for N training data t (= (1, 2, ..., N)) is to find ^{the value θ * of the model parameter θ that minimizes the loss function.} It is expressed by the equation (1) of.

Here, equation (1) includes a regularization term of Λ (θ) and is minimized including the regularization term. The minimum value is obtained by using the gradient descent method or the like. If a specific function, such as L2norm (sum of squares of all θ), is selected for the regularization term, θ ^* can be regarded as a function of the input data x _t , and as a result, the inference result y (x, θ). ^*), and it is possible differentiated by _{x t.} Therefore, as will be described later, it is possible to obtain an element or part of the input data that contributed to the inference result of the test data. When the learning unit 41 is executed for a large amount of learning data, model parameters are obtained and a machine learning model 36 is generated.

As described above, the learning unit 41 is a program that executes the learning of the machine learning model 36 based on the learning data.
(inference)
The server 12 can realize an inference function for input data by using the machine learning model 36. Therefore, the monitoring device 11 transmits the image data to the server 12 and receives the inference result from the server 12, so that the monitoring system 1 can perform desired monitoring.

For example, when the monitoring system 1 is a system for monitoring animals, the image data of the camera 22 is transmitted from the monitoring device 11 to the server 12 in real time or at predetermined time intervals, and the server 12 transfers the image data to the image. The imaged animal is inferred, and the inferred result is transmitted to the monitoring device 11. The inference result information is displayed on the monitor 23 of the monitoring device 11, and the user of the monitoring system 1 can recognize the inference result information displayed on the monitor 23.
(Visualization)
When there is an error in the inference result, the user visualizes the elements of the learning data that influenced the inference result, corrects the learning data that caused the error in the inference result, and makes the inference result correct. It is necessary to modify the model parameters of the machine learning model 36 by so-called re-learning.

However, until now, the user can know which training data has a great influence on the inference result, but knows which element or part of the learning data has a great influence on the inference result. I couldn't. According to the present embodiment, by supplying the input data as test data to the server 12, it is possible to visualize to the user which element or part of the training data has a great influence on the inference result for the test data. Can be shown. As a result, the user can quickly correct and relearn the learning data by looking at the visualized information and evaluating the inference result. Therefore, the server 12 constitutes an inference result evaluation device.

FIG. 5 is a diagram showing an example of a process flow for analyzing which element or part of the training data of the test data has a great influence on the inference result and displaying the analysis result. The process of FIG. 5 is executed on the server 12.

The user wants to know which element or part of the training data in the plurality of training data has a great influence on the inference result for the image data. The image data is used as test data, and the server 12 together with the evaluation instruction command. Send to. For example, image data for which a correct inference result has not been obtained is selected as test data and transmitted to the server 12.

That is, the user specifies test data in the monitoring device 11 using the mouse 21a and the keyboard 21b, and transmits the test data to the server 12. Therefore, the mouse 21a and the keyboard 21b of the monitoring device 11 constitute an input data designation unit or an input data designation device for designating test data which is input data.

The processor 33 includes an inference / visualization unit 51 that performs inference processing and visualization processing. The inference / visualization unit 51 is an inference program and a visualization processing program. The inference program and the visualization processing program are stored in the program area 34a of the memory 34. The inference / visualization unit 51 includes a data acquisition unit 52, an inference unit 53, a calculation unit 54, and a result output unit 55.

The data acquisition unit 52 acquires the test data uploaded by the user from the monitoring device 11 and supplies it to the inference unit 53.

The inference unit 53 includes a data preprocessing unit 53a and an inference execution unit 53b. The data preprocessing unit 53a executes a predetermined preprocessing. In the data preprocessing section 53a, the padding process is not performed.

The inference execution unit 53b executes inference based on the test data using the machine learning model 36, and calculates the inference result, that is, the inference result y (x, θ ^* ). ^{The model parameter θ *} obtained in the learning phase described above is used for inference in the inference execution unit 53b. As described above, the inference unit 53 executes inference using the machine learning model 36 based on the test data.

FIG. 6 is a diagram showing an example of the processing flow of the inference execution unit 53b. The inference execution unit 53b includes a feature amount extraction unit 61, a gaze area extraction unit 62, and an inference processing unit 63. The test data from the data preprocessing unit 53a is input to the feature amount extraction unit 61.

The feature amount extraction unit 61 extracts a predetermined feature amount of the test data and outputs the feature amount information. The feature amount information is supplied to the gaze area extraction unit 62 and the inference processing unit 63.

The gaze area extraction unit 62 extracts the gaze area in the test data from the feature amount information and outputs the gaze area information. The gaze area extraction unit 62 generates gaze area information (hereinafter, referred to as inferred gaze area information) indicating which area should be emphasized based on each feature amount extracted from the test data. Based on the feature amount extracted from each area in the test data which is the image data, it is generated as the inference gaze area information based on the mask information of which area the feature amount is emphasized.

For example, from information such as the facial features in the upper left area of the image and the body features in the lower right area, the areas other than the important areas are masked as information for identifying the areas to be watched. Mask information is generated. In this case, the mask information is, for example, information for outputting the region of the face portion as it is and setting the feature amount of the region other than the face portion to zero. In other words, the mask information is represented by the multiplication of the feature amount map and the mask area map, the logical product, and the like. The generated inference gaze area information is supplied from the gaze area extraction unit 62 to the inference processing unit 63.

The inference processing unit 63 executes inference based on the feature amount information and the gaze area information by using the machine learning model 36, and outputs the inference result. The inference result includes, for example, each class and probability data for each class. For example, when the machine learning model 36 is a model that infers an animal in an image, the inference result is that the probability of the "dog" class is 90%, the probability of the "cat" class is 10%, and so on.

The processing of the inference unit 53 is the same as the inference processing executed in the above-mentioned monitoring processing.

The inference result output by the inference unit 53 is supplied to the calculation unit 54. The calculation unit 54 includes an important learning data extraction unit 54a and a learning data influence calculation unit 54b.

The important learning data extraction unit 54a is based on test data (x), all training data (x ₁ , x ₂ , ..., X _N ), test data and inference results of each training data, feature amount information, and the like. Extract important training data that contributed to the inference result about. In other words, the important learning data extraction unit 54a extracts one or two or more learning data that are considered to have had a great influence on the estimation result. Here, the important learning data extraction unit 54a extracts one or more learning data for each class related to the inference result. That is, the important learning data extraction unit 54a extracts one or more learning data that contributed to the inference result of the inference unit 53 for the test data that is the input data from a plurality of learning data as the extraction learning data. It constitutes an extraction unit.

The learning data can be extracted by the important learning data extraction unit 54a by using the degree of similarity of the data (for example, square error, inner product of feature amount, image similarity index, etc.).

For example, the important learning data extraction unit 54a calculates the similarity between the test data and each learning data, sorts them in descending order of similarity, and learns a predetermined number (1 or 2 or more) having a high similarity. Output data.

Further, the extraction of the training data in the important learning data extraction unit 54a may be performed by using (-(∂y / ∂y _t )) and the inner product of the feature amount of the test data and the feature amount of each training data. Other methods may be used as the extraction method.

Furthermore, the important learning data extraction unit 54a may output a predetermined number (1 or 2 or more) of learning data of a higher order that has a great influence on the inference result based on the influence function (Influence Function).

Alternatively, the important learning data extraction unit 54a may output a predetermined number (1 or 2 or more) of learning data having a close distance to the identification boundary from a plurality of learning data belonging to the class.

The important learning data extraction unit 54a outputs the training data when the inference result for the test data is correct as the data that has a good influence on the inference result, and the inference result for the test data is incorrect. The training data at that time may be output as data that adversely affects the inference result. That is, the important learning data extraction unit 54a extracts both one or two or more learning data that have a good influence on the inference result and one or two or more learning data that have a bad influence on the inference result. May be good.

The learning data influence calculation unit 54b determines which element affected the ^{inference result y (x, θ *} ) in each of the one or more learning data extracted by the important learning data extraction unit 54a. calculate.

Here, the learning data influence calculation unit 54b calculates the degree of influence for each pixel by partially differentiating the probability of the inference result with each pixel value ((∂y (x, θ ^* ) / ∂x _t). That is, the learning data influence calculation unit 54b uses the value obtained by partially differentiating the probability value included in the estimation result with the value of each element or each part of the learning data as the influence degree of each element or each part. Here, the learning data influence calculation unit 54b uses the value obtained by partially differentiating the probability value included in the estimation result with each pixel value of the image data as the influence degree of each pixel.

The degree of influence of each element will be described with a simple example. For example, suppose that there are two training data x1 = {x11, x12} and x2 = {x21, x22} having elements of a two-dimensional vector, and they have labels y1 and y2. Let y1 be "normal" and y2 be "abnormal". When the parameter θ of the machine learning classification model for discriminating the labels y1 and y2 is optimized by giving certain conditions so that the two training data x1 and x2 are classified into the labels y1 and y2, respectively, the parameter θ becomes. , Θ = θ (x11, x12, x21, x22).

Here, when unknown data ct is input to this classification model, the probability p = p (xt, θ) that seems to be y1 (for example, normal) is output. From the above function of θ, p (xt, θ) = p (xt, x11, x12, x21, x22), so p can be differentiated by each element x11, x12, x21, x22 of the training data. For example, if the value of (∂p / ∂x11) is a large positive value, increasing x11 a little increases the probability that the data xt is y1. On the contrary, when the value of (∂p / ∂x11) is small, the data xt has little relation to y1. In other words, it is possible to evaluate which element of the training data has influenced this inference result.

To give another example, if you can obtain an inference model of a sales forecasting system that classifies whether today's sales are high or low from the data of past business days, the sales of one day (for example, August 28) will be When it is predicted to be high, it can be evaluated that the temperature, which is an element of one of the training data (for example, August 14), has an influence.

さらになお、推論結果から逆伝搬演算により、各画素（すなわち各要素）の影響度を直接算出するようにしてもよく、学習データ影響算出には、他の方法を用いてもよい。よって、学習データ影響算出部５４ｂは、抽出された学習データの要素毎または所定の部分毎に、推論結果に対する影響度を算出する影響度算出部を構成する。 In the above-described embodiment, the degree of influence for each pixel is calculated by partially differentiating the probability of the inference result with each pixel value ((∂y (x, θ ^* ) / ∂x _t). For example, the derivative (that is, sensitivity) of the probability of the inference result for the test data ((∂y (x, θ ^* ) / ∂x) is further differentiated with the training data, and the following equation (2) is used. The degree of influence may be calculated.

Furthermore, the degree of influence of each pixel (that is, each element) may be directly calculated from the inference result by a back propagation calculation, and another method may be used for the learning data influence calculation. Therefore, the learning data influence calculation unit 54b constitutes an influence degree calculation unit that calculates the influence degree on the inference result for each element or a predetermined part of the extracted learning data.

The result output unit 55 generates GUI data including an image superimposed on the learning data of the information of the elements that have influenced the inference result calculated by the learning data influence calculation unit 54b, and monitors the information via the network 13. It is transmitted to the device 11.
(Correction and re-learning)
Next, the modification of the training data will be described. FIG. 7 is a diagram showing a flow of correction processing of learning data. In the present embodiment, the user can perform the correction processing of the learning data of the learning database 35 from the monitoring device 11 to the server 12.

As will be described later, the user can see the learning data that contributed to the inference and make corrections such as deleting the learning data and setting and changing the designated gaze area. Therefore, the server 12 has a modification program for the learning data of the learning database 35. The patch is stored in the program area 34a of the memory 34.

Corrections include deletion of training data that is inappropriate for proper inference, correction of mislabeling of training data, change of settings of designated gaze area, and the like. For example, learning data with ambiguous labels, learning data that is difficult for humans to recognize, learning data that includes many backgrounds that are easily mistaken, and the like can be deleted from the learning database 35. Further, for learning data or the like with incorrect labeling, the label information can be corrected to the correct label information. In addition, for learning data that includes a plurality of subjects and includes subjects other than the subject on which the correct label is attached, the area other than the subject on which the correct label is attached is set as the mask area. The designated gaze area can be set as such.

Therefore, the correction unit 42 is a processing unit for enabling the user to perform correction of the learning data. Then, the correction unit 42 can execute at least one of changing the label information of the learning data, setting the gaze area of the learning data, and deleting the learning data.

After the learning database 35 is modified, the learning unit 41 can regenerate the machine learning model 36 based on the modified learning database 35 when it receives the relearning instruction command from the user.

As described above, the result output unit 55 generates GUI data. FIG. 8 is a diagram showing an example of a GUI generated based on GUI data. FIG. 8 shows an example of GUI displayed on the screen 23a of the monitor 23 of the monitoring device 11 based on the GUI data generated by the result output unit 55. The GUI 1 of FIG. 8 includes a test data display area 71, an inference result display field 72, a first learning data display area 73, a display switching button 74 including two buttons 74a and 74b, a slider 75 including a slider knob 75a, and a third. 2 The training data display area 76 is included.

The test data display area 71 is a window in which test data is displayed. The inference result display field 72 displays the result of the inference made on the test data. In FIG. 8, “classification result: squirrel” is displayed in the inference result display field 72. That is, from the image of the test data displayed in the test data display area 71, it is shown that the machine learning model 36 outputs "squirrel" as an inference result.

In the first learning data display area 73, the learning data extracted by the important learning data extraction unit 54a and having a good influence on the inference result is displayed. Here, in the first learning data display area 73, the learning data of the class with the highest probability (here, “squirrel”) from the inference results is displayed. Further, in the first learning data display area 73, the learning data having the highest similarity to the test data is displayed as important learning data from among the plurality of learning data of the class having the highest probability (here, “squirrel”). Will be done.

In the first learning data display area 73, an area of one or two or more pixels whose degree of influence calculated by the learning data influence calculation unit 54b is equal to or more than a predetermined threshold value is emphasized with a predetermined color and is combined with another area. Is displayed identifiable. In FIG. 8, the two partial PAs (hatched portions) are shown as portions that are displayed to the user so that they can be distinguished from the other regions. Therefore, the result output unit 55 generates display data for displaying each element or each portion having an influence degree equal to or higher than a predetermined threshold value so as to be distinguishable from the other element or the other portion.

The user recognizes that the two partial PAs of the training data displayed in the first training data display area 73 have a great influence on the inference result "squirrel" for the test data displayed in the test data display area 71. Can be done.

The button 74a is clicked when displaying other learning data having a higher degree of similarity from the plurality of learning data in the first learning data display area 73. The button 74b is clicked when displaying the learning data having the next highest degree of similarity from the plurality of learning data in the first learning data display area 73.

Therefore, when the learning data having the highest degree of similarity is displayed in the first learning data display area 73, when the user clicks the button 74b using the mouse 21a, the degree of similarity is next to the learning data having the highest degree of similarity. The learning data having a high degree of similarity (that is, the learning data having the second highest degree of similarity) is displayed in the first learning data display area 73. Further, when the user clicks the button 74a in the display state, the learning data having the highest similarity is displayed in the first learning data display area 73, but when the user clicks the button 74b, the learning data having the highest similarity is the third highest. The learning data is displayed in the first learning data display area 73.

Therefore, the user can recognize the pixels (elements) or parts that have a great influence on the inference result displayed in the inference result display field 72 in each important learning data displayed in the first learning data display area 73. it can.

The slider 75 is used to change the above-mentioned influence threshold. The user can move the slider knob 75a to the left or right with the mouse 21a selected. When the threshold value is made small, the pixel is displayed in an identifiable manner even if the influence of the pixel value is small. When the threshold value is increased, only the pixels having a large influence are displayed in an identifiable manner. Therefore, the predetermined threshold value can be changed.

Instead of making it possible to identify only the pixels with a high degree of influence, a heat map that visualizes the magnitude of the degree of influence with colors, a contour line surrounding the pixels with a high degree of influence, etc. It may be displayed by using the display method.

For example, moving the slider knob 75a to the right increases the threshold width, and moving the slider knob 75a to the left decreases the threshold width. Therefore, the user can change the size of the identifiable portion PA by moving the slider knob 75a.

Further, in the second learning data display area 76 on the screen 23a, the important learning data extracted by the important learning data extraction unit 54a and having a bad influence on the inference result is displayed. The learning data of the class having the highest probability from the inference results is displayed in the first learning data display area 73, but the second learning data display area 76 has the second highest probability from the inference results. The learning data of the class (for example, "hamster" described later) is displayed. Further, in the first learning data display area 73, the learning data having the highest degree of similarity to the test data is displayed as important learning data from among the plurality of learning data of the class having the second highest probability.

By operating a scroll bar (not shown) or the like, the user can see the learning data in the second learning data display area 76 in the same manner as in the first learning data display area 73. Also in the vicinity of the second learning data display area 76, although not shown, a display switching button 74 including two buttons 74a and 74b relating to the second learning data display area 76, and a slider 75 including a slider knob 75a are displayed. To.

Therefore, the user can operate the display switching button 74 related to the second learning data display area 76 to display other important learning data that adversely affects the inference result. Further, the user can change the size of the identifiable partial PA by moving the slider 75 with respect to the second learning data display area 76. The higher the threshold with the slider 75, the more confident the underlying part (ie, the part with the higher degree of influence) is displayed in an identifiable manner.

Therefore, by displaying GUI1 of FIG. 8, the user can recognize which element or part of the important learning data that influenced the inference result influenced the inference result of the test data. Therefore, the result output unit 55 constitutes an influence degree visualization unit that visualizes the influence degree for each element or a predetermined part in the extracted learning data. In other words, the result output unit 55 constitutes an influence degree image generation unit that generates image data for visualizing the influence degree for each element or a predetermined part in the extracted learning data.

FIG. 9 is a diagram showing another example of the GUI generated by the result output unit 55. In FIG. 9, the same components as those in FIG. 8 are designated by the same reference numerals and description thereof will be omitted. The user can recognize which element or part of the training data influences the inference result of the test data by using GUI2, and can modify the training data. The GUI 2 of FIG. 9 includes a test data display area 71 and an inference result display field 72.

Further, the GUI 2 has a first learning data display area 81 and a second learning data display area 82, as shown by the alternate long and short dash line. The first learning data display area 81 includes a plurality of (two in this case) learning data display areas, a first display area 84a and a second display area 84b. The second learning data display area 82 also includes a plurality of (here, two) learning data display areas 85a and a second display area 85b.

In the first learning data display area 81, the learning data of the class with the highest probability (here, “squirrel”) from the inference results is displayed. In the second learning data display area 82, the learning data of the class having the second highest probability (here, “hamster”) from the inference results is displayed.

The GUI 2 includes a display number setting field 83 for setting the number of training data displayed in each of the first learning data display area 81 and the second learning data display area 82. The user can input and set a number in the display number setting field 83 by using the mouse 21a and the keyboard 21b. In FIG. 9, since “2” is set in the display number setting field 83, two learning data are displayed in each of the first learning data display area 81 and the second learning data display area 82.

The learning data having the highest degree of similarity is displayed in the first display area 84a of the first learning data display area 81. In the second display area 84b of the first learning data display area 81, the learning data having the next highest degree of similarity is displayed after the learning data having the highest degree of similarity.

The learning data having the highest degree of similarity is displayed in the first display area 85a of the second learning data display area 82. In the second display area 84b of the first learning data display area 81, the learning data having the next highest degree of similarity is displayed after the learning data having the highest degree of similarity.

A slider 75 and an inference result display field 86 are provided corresponding to each of the first display area 84a and the second display area 84b. In the inference result display field 86, label information of the training data of the corresponding display area is displayed. The user can change the label information by moving the cursor 91 to the desired inference result display field 86 using the mouse 21a. The label information is changed by the correction unit 42 described above.

Further, each display area 84a, 84b, 85a, 85b is provided with a deletion button 87 of a predetermined mark for deleting the corresponding learning data. When the delete button 87 is clicked, the learning data related to the delete button 87 is deleted from the learning database 35.

Further, the GUI 2 has a selection learning data display unit 88, a confirmation button 89, and a re-learning button 90.

Partial PAs are superimposed on the learning data displayed in the display areas 84a and 84b. Part PA indicates an element or part that has a positive influence on the inference result of the inference result display field 72.

Further, the partial PB is also superimposed on the learning data displayed in each of the display areas 84a and 84b. Part PB indicates an element or part that has a negative influence on the inference result of the inference result display field 72.

Similarly, the partial PA and PB are superimposed on the images displayed in the respective display areas 85a and 85b. By observing the states of the partial PAs and PBs, the user can estimate the influence on the estimation result. For example, in the case of the training data of the display area 84b, the partial PA at the tail of the squirrel determines that the squirrel is a squirrel, but the face of the squirrel in the partial PB is considered to question the estimation result.

In GUI2, partial PAs and PBs are also displayed for the test data displayed in the test data display area 71.

FIG. 9 shows a case where the inference result of the inference result display field 72 is displayed as “squirrel”, but the inference result of “hamster” is obtained with a probability lower than the probability of “squirrel”. Therefore, in each display area 84a, 84b, 85a, 85b, the user has a positive influence on the inference result of the inference result display field 72 and the inference result of the inference result display field 72. You can intuitively understand the part that has a negative influence.

The selective learning data display unit 88 is an area for enlarging and displaying the learning data selected by the user. The user can set the area to be watched in the learning data displayed on the selective learning data display unit 88. For example, when the user uses the mouse 21a to move the cursor 91 on the GUI 2 to select the learning data displayed in the display area 84b, the selected learning data is displayed on the selective learning data display unit 88. ..

After that, the user selects a predetermined command for the image displayed on the selective learning data display unit 88, and draws the surrounding line 93 using the virtual pen 92 in a mode in which freedom can draw a line. be able to. When the user clicks the confirm button 89, the area other than the inner area surrounded by the line 93 is set as the mask area. The designated gaze area based on this set mask area is registered in the meta information of the table data TBL. In FIG. 9, it is shown that the learning data displayed in the display area 84b is selected and the mask area is set by the line 93.

Further, when the inference result of the inference result display field 72 is incorrect, the user may know which learning data is affected and the inference result is incorrect. For example, when the test data displayed in the test data display area 71 is an image of "squirrel" but "hamster" is displayed in the inference result display field 72, the image of the display area 85a. May be found to affect the inference results.

In such a case, when the user selects the delete button 87 in the vicinity of the display area 85a with the cursor 91, the learning data itself displayed in the display area 85a can be deleted from the learning database 35. In the case of FIG. 9, since it is considered that the image of the face portion of the hamster image displayed in the display area 85a has a great influence on the inference result of the test data, the user displays it in the display area 85a. The learned data is deleted from the training database 35.

Commands such as deletion of inappropriate learning data, correction of incorrect labeling of learning data, change of setting of designated gaze area, etc. are transmitted from the monitoring device 11 to the server 12, and the correction unit 42 of the server 12 displays the display area. Correct the learning data displayed in 85a. After that, when the user moves the cursor 91 to the relearn button 90 and clicks it, a command instructing relearn is transmitted to the server 12.

As a result, the server 12 executes the learning program. The learning program performs learning using the modified learning database 35.

FIG. 10 is a diagram showing a display example of GUI2 after re-learning. Due to the deletion of the learning data, the learning data displayed in the display area 85b in FIG. 9 is displayed in the display area 85a, and the learning data having the next highest degree of similarity to the display area 85a is displayed in the display area 85b. To.

Further, in FIG. 10, since the area surrounded by the line 93 in FIG. 9 is designated as the designated gaze area, the squirrel tail portion PA is not displayed in the test data display area 71. Further, in FIG. 9, the partial PA is identified and displayed in the display area 84b, but the partial PA of the selective learning data display unit 88 in FIG. 9 is used as a mask area and is not used for learning. In the test data display area 71, it is not displayed as a partial PA as a judgment basis.

FIG. 11 is a flowchart showing a processing flow from supplying the test data to the server 12 to re-learning the learning data. When the monitoring system 1 is operating, inference is performed using the machine learning model 36 based on the image data from the monitoring device 11, and the monitoring process in which the inference result is transmitted to the monitoring device 11 is executed. (Step (hereinafter abbreviated as S) 1).

When the user wants to evaluate the inference result, such as when there is an error in the inference result, for example, the image data with the wrong inference result is transmitted to the server 12 together with the evaluation instruction command as test data.

If the test data is not transmitted (S2: NO), the monitoring system 1 continues the monitoring process. When the test data is transmitted (S2: YES), the server 12 executes the above-described learning data analysis process (S3). The analysis process is the process of the inference / visualization unit 51 shown in FIG.

When the user looks at the analysis result, for example, looks at the GUI of FIG. 8 or FIG. 9 and determines that re-learning is not necessary (S4: NO), the process returns to S1. When the user sees the analysis result and determines that re-learning is necessary (S4: YES), the user corrects the above-mentioned learning data and sends a re-learning command to the server 12. As a result, the server 12 executes re-learning using the modified learning data. To update the model parameters during learning, optimize the model parameters by stochastic gradient descent, etc., on the condition that the inference gaze area matches the specified gaze area so that the inference result matches the label information of the correct answer. Is done by. Since the machine learning model 36 is modified by the re-learning, the accuracy of the inference result is improved.

As described above, according to the above-described embodiment, the training data that contributed to the inference result of the test data was extracted, and which element or part of the training data affected the inference result of the test data. The visualization allows the user to intuitively understand the elements or parts of the training data that influenced the inference results of the test data. For example, as shown in each GUI1 and GUI2 described above, the user can intuitively understand which element or part of the training data has been learned to obtain the inference result. As a result, re-learning can be performed quickly.

As described above, according to the above-described embodiment, inference result evaluation capable of visualizing which element or part of the training data affected the inference result of the input input data in the machine learning model. A system, an inference result evaluation device, and an inference result evaluation method can be provided.

Although the above-described embodiment is an example applied to a monitoring system that performs inference using a machine learning model, it can also be applied to other systems other than the monitoring system.

As a computer program product, the program that executes the operations described above may be a portable medium such as a flexible disk or a CD-ROM, or a non-temporary computer-readable medium such as a hard disk, or the entire program. Some are recorded or remembered. The program is read by a computer and all or part of its operation is performed. Alternatively, the program in whole or in part can be distributed or provided via a communication network. The user can easily realize the inference result evaluation system of the present embodiment by downloading the program via the communication network and installing it on the computer, or installing it on the computer from the recording medium. it can.

Although some embodiments of the present invention have been described, these embodiments are exemplary by way of example and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other embodiments, and various omissions, replacements, and changes can be made without departing from the gist of the invention. These embodiments and modifications thereof are included in the scope and gist of the invention, and are also included in the scope of the invention described in the claims and the equivalent scope thereof.

1 monitoring system, 11 monitoring device, 12 server, 12a cloud, 13 network, 14 camera, 21a mouse, 21b keyboard, 22 camera, 22a detector, 23 monitor, 23a screen, 31 inference device, 32 communication interface, 33 processor, 34 Memory, 34a Program area, 35 Learning database, 36 Machine learning model, 41 Learning unit, 41a Data input unit, 41b Data preprocessing unit, 41c Inference execution unit, 41d Model parameter update unit, 42 Correction unit, 51 Visualization unit , 52 data acquisition unit, 53 inference unit, 53a data preprocessing unit, 53b inference execution unit, 54 calculation unit, 54a important learning data extraction unit, 54b learning data influence calculation unit, 55 result output unit, 61 feature quantity extraction unit, 62 Gaze area extraction unit, 63 Inference processing unit, 71 Test data display area, 72 Inference result display field, 73 Learning data display area, 74 Display switching button, 74a, 74b button, 75 slider, 76 Learning data display area, 81, 82 Learning data display area, 83 Display number setting field, 84a, 84b, 85a, 85b Display area, 86 Inference result display field, 87 Delete button, 88 Selective learning data display area, 89 Confirm button, 90 Relearn button, 91 Cursor , 92 virtual pen, 93 lines.

Claims (9)

Input data specification part that specifies input data and
An inference unit that executes inference using a machine learning model based on the input data,
A learning data extraction unit that extracts learning data that contributes to the inference result of the inference unit for the input data from a plurality of learning data as extraction learning data,
An influence degree calculation unit that calculates the influence degree on the inference result for each element or a predetermined part of the extracted learning data,
An influence degree visualization unit that visualizes the influence degree for each element or each predetermined part in the extracted learning data,
Inference result evaluation system with. The influence degree calculation unit uses the value obtained by partially differentiating the probability value included in the estimation result with the value of each element or each part of the extracted learning data as the influence degree of each element or each part. The inference result evaluation system according to claim 1. The inference result evaluation system according to claim 1, wherein the influence degree visualization unit displays each element or each part having an influence degree equal to or higher than a predetermined threshold value so as to be distinguishable from other elements or other parts. The inference result evaluation system according to claim 3, wherein the predetermined threshold value can be changed. The inference result evaluation system according to claim 1, further comprising a correction unit that corrects the learning data. The inference result evaluation system according to claim 5, wherein the correction unit can execute at least one of changing the label information of the learning data, setting the gaze area of the learning data, and deleting the learning data. The training data includes image data and includes image data.
Any of claims 1 to 6, wherein the influence degree calculation unit uses the value obtained by partially differentiating the probability value included in the estimation result with each pixel value of the image data as the influence degree of each pixel. The inference result evaluation system described in one. A learning data extraction unit that extracts learning data that contributes to the inference result of inference using a machine learning model based on input data as extraction learning data from a plurality of learning data.
An influence degree calculation unit that calculates the influence degree on the inference result for each element or a predetermined part of the extracted learning data,
An influence degree visualization unit that visualizes the influence degree for each element or each predetermined part in the extracted learning data,
Inference result evaluation device having. Specify the input data
Based on the input data, inference using a machine learning model is executed, and
The learning data that contributed to the inference result of the inference about the input data is extracted from a plurality of learning data as the extraction learning data.
The degree of influence on the inference result is calculated for each element or a predetermined part of the extracted learning data.
Visualize the degree of influence for each element or each predetermined part in the extracted learning data.
Inference result evaluation method.

Images