WO2023181232A1

WO2023181232A1 - Model analysis device, model analysis method, and recording medium

Info

Publication number: WO2023181232A1
Application number: PCT/JP2022/013815
Authority: WO
Inventors: 啓太佐久間; 智哉坂井; 竜太松野; 義男亀田
Original assignee: 日本電気株式会社
Priority date: 2022-03-24
Filing date: 2022-03-24
Publication date: 2023-09-28

Abstract

In this model analysis device, a predicted value acquisition means acquires a predicted value of a model with respect to input data. An output means outputs assessment information including: a graph that indicates predicted values and actual measured values; and a display area for displaying a determination criterion for prediction mistakes. A criterion acquisition means acquires the determination criterion. An extraction means extracts a predicted value corresponding to a prediction mistake and indicates the predicted value on the graph, on the basis of the determination criterion.

Description

Model analysis device, model analysis method, and recording medium

This disclosure relates to analysis of machine learning models.

In recent years, predictive models obtained through machine learning have been used in various fields. Patent Document 1 describes a method of predicting real estate prices using a prediction model.

International publication WO2020/004049

Patent Document 1 describes a method in which the method of removing predictions by a model (predicted value - actual value) is reversed, that is, data samples with different positive and negative prediction errors are displayed in pairs. However, the evaluation of the model changes depending on how the error is defined in the first place.

One objective of the present disclosure is to provide a model analysis device that can define errors in a prediction model and appropriately evaluate the model based on the defined errors.

In one aspect of the present disclosure, the model analysis device includes:
Predicted value obtaining means for obtaining a predicted value of the model for input data;
Output means for outputting evaluation information including a graph showing the predicted value and the actual measured value, and a display area for displaying a criterion for determining a prediction error;
Criterion acquisition means for acquiring the determination criteria;
Extracting means for extracting a predicted value that corresponds to a prediction error based on the judgment criterion and showing it on the graph;
Equipped with

In other aspects of the disclosure, the model analysis method includes:
Get the model's predicted value for the input data,
outputting evaluation information including a graph showing the predicted value and the actual measured value, and a display area for displaying a criterion for determining a prediction error;
obtain the judgment criteria;
Based on the determination criteria, predicted values corresponding to prediction errors are extracted and shown on the graph.

In yet another aspect of the present disclosure, the recording medium includes:
Get the model's predicted value for the input data,
outputting evaluation information including a graph showing the predicted value and the actual measured value, and a display area for displaying a criterion for determining a prediction error;
obtain the judgment criteria;
Based on the determination criteria, a predicted value corresponding to a prediction error is extracted, and a program is recorded that causes a computer to execute the processing shown on the graph.

According to the present disclosure, it is possible to set a criterion for error in a prediction model and appropriately evaluate the model using the error based on the set criterion.

FIG. 1 is a block diagram showing the overall configuration of a model generation system according to a first embodiment. FIG. 2 is a block diagram showing the hardware configuration of a model generation device. 1 is a block diagram showing a functional configuration of a model generation device according to a first embodiment; FIG. A first display example of evaluation information is shown. A second display example of evaluation information is shown. A third display example of evaluation information is shown. Another setting example of the threshold value is shown. 7 is a flowchart of model analysis processing performed by the model generation device. FIG. 1 is a block diagram showing a schematic configuration of a model generation system using a server and a terminal device. FIG. 2 is a block diagram showing the functional configuration of a model analysis device according to a second embodiment. It is a flowchart of processing by a model analysis device of a 2nd embodiment.

Hereinafter, preferred embodiments of the present disclosure will be described with reference to the drawings.
<First embodiment>
[overall structure]
FIG. 1 is a block diagram showing the overall configuration of a model generation system according to a first embodiment. The model generation system 1 includes a model generation device 100, a display device 2, and an input device 3. The model generation device 100 is an application of the model analysis device of the present disclosure, and is configured by, for example, a computer such as a personal computer (PC). The display device 2 is, for example, a liquid crystal display device, and displays the evaluation information generated by the model generation device 100. The input device 3 is, for example, a mouse, a keyboard, etc., and is used by the user to give instructions and input necessary when modifying a model or viewing evaluation information.

First, the operation of the model generation system 1 will be schematically explained. The model generation device 100 generates a machine learning model (hereinafter simply referred to as a "model") using training data prepared in advance. The model generation device 100 also evaluates the generated model. Specifically, the model generation device 100 performs prediction using a model using evaluation data and the like, detects a prediction error in the model based on the prediction result, and presents it to the user as evaluation information. The user can confirm prediction errors in the model and operate the input device 3 to input correction information for correcting the model. In particular, in this embodiment, the user can input the criterion for determining a prediction error and can further change it as necessary. Therefore, the user can appropriately evaluate the model by setting a criterion for determining a prediction error from a viewpoint that the user considers appropriate, and viewing evaluation information based on the criterion.

Here, the "model" is information representing the relationship between explanatory variables and objective variables. A model is, for example, a component for estimating a target result by calculating a target variable based on explanatory variables. A model is generated by executing a learning algorithm using as input learning data for which values of objective variables have already been obtained and arbitrary parameters. The model may be represented, for example, by a function c that maps an input x to a ground answer y. The model may be one that estimates a numerical value to be estimated, or may be one that estimates a label to be estimated. The model may output variables that describe the probability distribution of the target variable. A model is sometimes described as a "learning model," "analytical model," "AI (Artificial Intelligence) model," or "prediction formula."

[Hardware configuration]
FIG. 2 is a block diagram showing the hardware configuration of the model generation device 100. As illustrated, the model generation device 100 includes an interface (I/F) 111, a processor 112, a memory 113, a recording medium 114, and a database (DB) 115.

The I/F 111 inputs and outputs data to and from external devices. Specifically, training data, evaluation data, and instructions and inputs input by the user using the input device 3 are input to the model generation device 100 through the I/F 111. Furthermore, evaluation information of the model generated by the model generation device 100 is output to the display device 2 through the I/F 111.

The processor 112 is a computer such as a CPU (Central Processing Unit), and controls the entire model generation device 100 by executing a program prepared in advance. Note that the processor 112 may be a GPU (Graphics Processing Unit) or an FPGA (Field-Programmable Gate Array). The processor 112 executes model analysis processing, which will be described later.

The memory 113 is composed of ROM (Read Only Memory), RAM (Random Access Memory), and the like. Memory 113 is also used as a working memory while processor 112 executes various processes.

The recording medium 114 is a non-volatile, non-temporary recording medium such as a disk-shaped recording medium or a semiconductor memory, and is configured to be detachable from the model generation device 100. The recording medium 114 records various programs executed by the processor 112. When the model generation device 100 executes various processes, a program recorded on the recording medium 114 is loaded into the memory 113 and executed by the processor 112.

The DB 115 stores information regarding the model generated by the model generation device 100 (hereinafter referred to as "existing model") and the model after modification by retraining (hereinafter referred to as "modified model"). Further, the DB 115 stores training data input through the I/F 111, evaluation data, correction information input by the user, history of prediction error criteria input by the user, and the like, as necessary.

(Functional configuration)
FIG. 3 is a block diagram showing the functional configuration of the model generation device 100 of the first embodiment. The model generation device 100 functionally includes a training data DB 121, a model training section 122, a model DB 123, an evaluation data DB 124, a prediction error analysis section 125, and an evaluation information output section 126.

The training data DB 121 stores training data used for model generation. Training data D1 is input to model training section 122. Note that the training data D1 is composed of a plurality of combinations of input data and correct labels (teacher labels) for the input data.

The model training unit 122 trains a model using the training data D1 and generates a model. The model training unit 122 outputs model data M corresponding to the generated model to the model DB 123 and the prediction error analysis unit 125. Note that the model data M includes a plurality of parameter information constituting the model. The parameter information includes, for example, information on explanatory variables (or feature amounts) used as inputs of the model, information on weights for each explanatory variable, information on weights for each sample constituting input data, and the like.

Additionally, the model training unit 122 retrains the existing model to generate a modified model. In this case, the model training unit 122 corrects the parameters constituting the model based on the correction information D3 input by the user using the input device 3, and uses training data for retraining as necessary to improve the model. Perform retraining. The model training unit 122 stores the model data M of the corrected model obtained through retraining in the model DB 123 and outputs it to the prediction error analysis unit 125.

The evaluation data DB 124 stores evaluation data used to evaluate the generated model. The evaluation data includes various types of data that can be used to evaluate the model. The evaluation data is basically composed of a plurality of combinations of input data and correct labels (teacher labels) for the input data. Examples of evaluation data include the following.
(1) “Data not used for model generation” called validation data or test data
In this case, the evaluation data is basically a set of input data and correct answer labels.
(2) “Newly collected data after model generation” such as operational data
Note that if labeling is not performed immediately, the evaluation data may be input-only data.
(3) “Data that is generated by some method and is unknown to the model”
For example, if the feature amount in the input data is (day of the week, holiday, weather), it is possible to create pseudo future data using calendar information and weather forecasts.
(4) “Same data as training data”
The training data used to generate the model can be used as evaluation data. In this case, the same data as the training data may be stored in the evaluation data DB 124 as evaluation data.

The prediction error analysis unit 125 analyzes prediction errors of the existing model using the evaluation data. Specifically, the prediction error analysis unit 125 inputs the input data of the evaluation data into the existing model, performs prediction, and obtains the prediction result. Then, the prediction error analysis unit 125 extracts prediction errors caused by the existing model from the prediction results of the model based on the evaluation data used and the prediction results.

Here, the definition of a prediction error, that is, the criterion for determining a prediction error, is set by the user. The prediction error analysis unit 125 analyzes the prediction result by the model based on the criteria set by the user. Information D4 on the determination criteria set by the user is sent from the input device 3 to the prediction error analysis unit 125. The prediction error analysis unit 125 extracts prediction errors included in the prediction result according to the acquired criteria. Then, the prediction error analysis unit 125 outputs the prediction result by the model and the extracted prediction error to the evaluation information output unit 126. Note that, if necessary, the prediction error analysis unit 125 also outputs the used evaluation data to the evaluation information output unit 126. Note that the method for setting the criterion for prediction errors will be explained in detail later. The prediction error analysis unit 125 is an example of a predicted value acquisition means, a reference acquisition means, and an extraction means.

The evaluation information output unit 126 generates evaluation information D2 for evaluating the existing model based on the information input from the prediction error analysis unit 125. Specifically, the evaluation information D2 includes information indicating the relationship between the actual measurement value and the prediction result (prediction value) by the existing model, and the detected prediction error. Then, the evaluation information output unit 126 outputs the generated evaluation information D2 to the display device 2. The evaluation information output unit 126 is an example of an output means.

The display device 2 displays the evaluation information D2 output by the evaluation information output unit 126. Thereby, the user can evaluate the performance of the existing model by referring to the relationship between the measured value and the predicted value by the existing model, and information indicating prediction errors included in the predicted value by the model. Examples of information indicating a prediction error include information indicating a sample of a predicted value corresponding to a prediction error (hereinafter referred to as a "prediction error sample"). The user inputs modification information D3 into the input device 3 as necessary to modify the model so that prediction errors do not occur. The modification information D3 is information related to modification, such as information on explanatory variables used as inputs of the model, information on weights for each explanatory variable, and information on weights for each sample constituting the input data. The model training unit 122 corrects the model by retraining the model using the input correction information D3.

[Display example of evaluation information]
Next, a display example of evaluation information displayed on the display device 2 will be explained.
(First display example)
FIG. 4 shows a first display example of evaluation information. In this example, the prediction model is a model that predicts sales of a certain product. Note that FIG. 4 is a display example after the user has already set the prediction error criterion. The first display example 40 includes graphs 41a to 41c and an input area . The graph 41a shows predicted values by the model, and the graph 41b shows actual measured values. The horizontal axes of

graphs

41a and 41b indicate the date of a certain month, and the vertical axes indicate sales. A mark 41x indicating a prediction error sample is displayed on the predicted value graph 41a.

The graph 41c is a graph showing an error index for evaluating the error between the predicted value by the model and the actual measured value. In the example of FIG. 4, the graph 41c is a bar graph showing the absolute error between the predicted value by the model and the actual measured value. The horizontal axis of the graph 41c shows the date, and the vertical axis shows the absolute error. A threshold value 41d is shown on the graph 41c. The threshold is used to extract mispredicted samples based on the absolute error specified as the error index.

The input area 42 is an area for the user to set criteria for determining prediction errors. That is, by inputting necessary information into the input area 42, the user sets criteria for determining prediction errors that he/she wishes to extract. In the example of FIG. 4, an error index and a threshold are set as the determination criteria. In this case, the above-mentioned prediction error analysis unit 125 will extract, as a prediction error sample, a sample in which the error between the predicted value and the actual value based on the error index set by the user is larger than the threshold value set by the user.

As the "error index", for example, an error such as an absolute error or a squared error is set. The user operates the input area 43 to set the error index that he/she wishes to use. The "threshold" is defined by a threshold reference value and a threshold adjustment parameter. Note that in this example, the threshold value adjustment parameter is a magnification that indicates how many times the threshold value is the reference value. The reference value of the threshold value is defined by the type of data used and the average error according to the error specified as the error index. As the data to be used, for example, training data, validation data, training data for a predetermined period, validation data for a predetermined period, etc. can be used. The user operates the input area 44 to set the reference value of the threshold that he/she wishes to use. For example, when the user specifies absolute error as the error index and uses validation data, the user selects "validation data MAE" as shown in FIG. 4. "MAE" indicates mean absolute error (MAE). Note that when a squared error is used as an error index, a mean squared error (MSE) is usually used as the type of error.

The user also operates the input area 45 to set an arbitrary magnification as a threshold adjustment parameter. The threshold value is the product of the reference value of the threshold set in the input area 44 and the magnification set in the input area 45. In the example of FIG. 4, the user has set validation data MAE in the input area 44 and set the magnification "2" in the input area 45, so the threshold value is
Threshold=2×MAE_va
It is calculated as Note that "MAE_va" is a calculated value of the mean absolute error (MAE) of validation data.

An OK button 49 is displayed in the input area 42. The OK button 49 is a button used by the user to indicate that the determination criteria setting in the input area 42 has been completed. Further, in the input area 42, the number of extracted samples with prediction errors is displayed as an extraction result 48.

Note that, as described above, the display example 40 in FIG. 4 is a display example after the user has already set the prediction error criterion. In the initial state, that is, before the user sets the prediction error judgment criteria, the predicted value graph 41a and the actual value graph 41b are displayed, but the graph 41c and the mark 41x indicating a prediction error are not displayed. There is no input in each of the input areas 43 to 45 within the input area 42.

Then, when the user sets a judgment criterion for a prediction error and presses the OK button 49, the information D4 of the set judgment criterion, specifically, the information input in the input areas 43 to 45, is transmitted from the input device 3 to the prediction error. It is transmitted to the error analysis section 125. Based on the received judgment criterion information D4, the prediction error analysis section 125 extracts a sample corresponding to the judgment criterion from the predicted value of the model as a prediction error sample, and outputs it to the evaluation information output section 126. The evaluation information output unit 126 transmits information regarding the received prediction error sample to the display device 2, displays a mark 41x indicating the prediction error sample on the graph 41a, and displays the extraction result 48 of the prediction error sample in the input area 42. to be displayed. In this way, prediction error samples are extracted according to the criteria set by the user and displayed on the display device 2. As a result, a display as illustrated in FIG. 4 is performed.

Note that although the case where the user sets the determination criteria has been described here, the first display example is not limited thereto. For example, the display device 2 may display a predetermined value as the determination criterion. Alternatively, the display device 2 may display the criteria set by the user in the previous operation. Alternatively, the display device 2 may display recommended criteria for each user using a machine learning model learned from the input history of criteria.

(Second display example)
FIG. 5 shows a second display example of evaluation information. The second display example 40a differs from the first display example 40 in that the user can set or modify the prediction error criterion by specifying a sample on the graph 41a.

The user looks at the predicted value graph shown in the graph 41a and specifies a sample that is considered to be a prediction error. Specifically, in the example of FIG. 5, the user determines that the sample values of "8th" and "11th" correspond to prediction errors on the predicted value graph 41a, and the user makes a prediction error as shown in the mark 46. Click on these two samples to specify them. The user then presses the OK button 49 in the input area 42. As a result, information specifying the two samples indicated by the marks 46 is transmitted to the prediction error analysis unit 125. The prediction error analysis unit 125 corrects the judgment criteria so that the specified two samples are extracted as prediction errors, and outputs information on the prediction errors extracted based on the corrected judgment criteria to the evaluation information output unit 126. do. The evaluation information output unit 126 transmits evaluation information including the corrected prediction error information to the display device 2 and causes it to be displayed.

As a result, in the corrected display example 40a shown in FIG. 5, the magnification value in the input area 45 is changed to "1", and the corrected threshold 41d and the uncorrected threshold 41e are displayed. That is, in this example, the scaling factor is changed from "2" to "1" and the threshold value is decreased so that the two samples specified by the user are determined to be prediction errors, and the threshold value graph is changed accordingly. has been changed.

Note that in the above example, the determination criteria are modified by specifying samples on the predicted value graph 41a, but the determination criteria may be input using this method from the beginning. In this case, in the initial state where only the

graphs

41a and 41b are displayed, the user inputs the sample into the

input areas

43 and 44, and then selects the sample to be determined as a prediction error on the predicted value graph 41a without inputting the input into the input area 45. All you have to do is specify it above and press the OK button 49.

In this way, according to the second display example, when the user specifies a sample on the graph of predicted values, the prediction error analysis unit 125 sets or sets a criterion so that the specified sample is determined to be a prediction error. Fix it. Therefore, even a user who lacks knowledge and experience and has difficulty setting the input area 42, especially setting the threshold, can appropriately set and modify the criteria.

(Third display example)
FIG. 6 shows a third display example of evaluation information. The third display example 40b differs from the first display example 40 in that an option column 47 is provided in the input area 42. The option field 47 is an item for the user to specify display rules for prediction error samples. In the example of FIG. 6, display rule R1 ``Display all prediction error samples'', display rule R2 ``Display only consecutive prediction error samples'', and display rule R3 ``Display only the last day of consecutive prediction error samples'' are prepared. ing.

Specifically, when the user selects display rule R1, the prediction error analysis unit 125 extracts and displays all prediction error samples that correspond to the determination criteria. A graph 41a in FIG. 6 shows an example of this case.

When the user selects display rule R2, the prediction error analysis unit 125 extracts and displays only consecutive prediction error samples from among the plurality of prediction error samples that correspond to the determination criteria. Therefore, in this case, prediction error samples that are not consecutive in the horizontal axis (date) direction of the graph 41a are not displayed. For example, in the graph 41a of display example 40b, consecutive prediction error samples on the 18th and 19th are displayed, but if there are no consecutive prediction error samples before and after, such as on the 9th or 16th, the prediction Missed samples will not be displayed.

Furthermore, when the user selects display rule R3, the prediction error analysis unit 125 extracts only the last consecutive prediction error samples from among the plurality of prediction error samples that meet the criteria. Therefore, for example, if a prediction error sample occurs for two consecutive days, the prediction error sample is displayed only for the second day. Note that the display rules for prediction error samples are not limited to the above three, but can be set arbitrarily.

According to the third display example, the user can select a rule for displaying a prediction error sample on the display device 2 from the viewpoint of the purpose of model evaluation and the visibility of display contents.

(Other setting examples of threshold)
In the first to third display examples above, the threshold value is set as "threshold value = magnification * reference value", but the method of setting the threshold value is not limited to this, and various other methods can be used. . FIG. 7 shows another example of the input area included in the display example. The input area 42x shown in FIG. 7 includes

input areas

51 and 52, a histogram 53, and a threshold bar 54. Input area 51 is used by the user to specify a data set, and input area 52 is used by the user to specify an error measure. Histogram 53 shows the error calculated for each sample included in the data set based on user specifications. The threshold bar 54 is a bar that the user moves to arbitrarily set a threshold value.

In the example of FIG. 7, the user first operates input area 51 to specify a data set, and then operates input area 52 to specify an error index. Here, examples of error indicators that can be specified by the user include the following. Note that "y" indicates the actual value, and "y_pred" indicates the predicted value.
(Example 1) Error y－y_pred
(Example 2) Absolute error |y－y_pred|
(Example 3) Squared error (y-y_pred) ²
(Example 4) Error rate (y-y_pred)/y
(Example 5) Absolute error rate |y−y_pred|/y

The prediction error analysis unit 125 calculates the error index specified by the user for each sample in the data set specified by the user, and displays the obtained error of each sample as a histogram 53. The user can move the threshold value bar 54 while looking at the displayed histogram 53 and determine the threshold value intuitively. Note that instead of the user setting the threshold value, the prediction error analysis unit 125 may automatically set the threshold value so that a predetermined percentage (for example, 20%) of samples in the data set become prediction error samples. good.

[Model analysis processing]
Next, model analysis processing by the model generation device 100 will be explained. FIG. 8 is a flowchart of model analysis processing by the model generation device 100. The model analysis process is a process of extracting prediction errors of the existing model generated by the model training unit 122 and displaying them on the display device 2. This processing is realized by the processor 112 shown in FIG. 2 executing a program prepared in advance and operating as the element shown in FIG. 3.

First, the model generation device 100 inputs evaluation data into an existing model and obtains a predicted value by the existing model (step S10). Next, the model generation device 100 generates a graph showing the actual measured values included in the evaluation data and the predicted values by the existing model (step S11). The generated graph is displayed on the display device 2.

The user looks at the graph of actual measured values and predicted values displayed, for example, as shown in FIG. 4, and sets a criterion for determining a prediction error sample in the input area 42. The model generation device 100 acquires the set criterion for prediction error samples (step S12). Next, the model generation device 100 extracts a sample of the predicted value that corresponds to the acquired criterion as a prediction error sample and outputs it (step S13). The extracted prediction error samples are indicated by marks 41x on the graph of predicted values displayed on the display device 2, as shown in FIG.

Next, the model generation device 100 determines whether the user has input an instruction to modify the criteria (step S14). If an instruction to modify the determination criteria is input (step S14: Yes), the process returns to step S12, and the model generation device 100 acquires the revised determination criteria, extracts prediction error samples according to the criteria, and Output (step S13). In this way, the user can evaluate the model while repeatedly modifying the criteria as necessary.

On the other hand, if an instruction to modify the criterion has not been input (step S14: No), the model generation device 100 determines whether a termination instruction has been input by the user (step S15). If the termination instruction is not input (step S15: No), the process returns to step S14. On the other hand, if an end instruction is input (step S15: Yes), the model analysis process ends.

Note that the user may set or modify the determination criteria by specifying a sample on the graph of the predicted value, as in the second display example shown in FIG. In this case, the model generation device 100 obtains an input specifying a sample on the graph of the predicted value in step S12. Then, in step S13, the model generation device 100 corrects the criterion for a prediction error sample so that the sample is determined to be a prediction error, and then extracts the prediction error sample. Further, as in the third display example shown in FIG. 6, when the user specifies a display rule for prediction error samples in the option column 47, the model generation device 100 displays prediction error samples according to the specified display rule in step S13. A sample is selected and displayed on the display device 2.

[Modified example]
In the above embodiment, the model generation device 100 is configured as an independent device such as a PC, but instead, the model generation device may be configured with a server and a terminal device. FIG. 9 is a block diagram showing a schematic configuration of a model generation system 1x using a server and a terminal device. In FIG. 9, a server 100x includes the configuration of the model generation device 100 shown in FIG. Further, the display device 2x and input device 3x of the terminal device 7 used by the user are used as the display device 2 and input device 3 shown in FIG.

<Second embodiment>
FIG. 10 is a block diagram showing the functional configuration of the model analysis device 70 of the second embodiment. The model analysis device 70 includes a predicted value acquisition means 71, an output means 72, a reference acquisition means 73, and an extraction means 74.

FIG. 11 is a flowchart of processing by the model analysis device 70 of the second embodiment. The predicted value acquisition means 71 acquires the predicted value of the model for input data (step S71). The output means 72 outputs evaluation information including a graph showing the predicted value and the actual measured value, and a display area for displaying a criterion for determining a prediction error (step S72). The criterion acquisition means 73 acquires the determination criterion (step S73). The extraction means 74 extracts predicted values corresponding to prediction errors based on the determination criteria, and shows them on the graph (step S74).

According to the model analysis device 70 of the second embodiment, it is possible to set a criterion for the error of a predictive model and to appropriately evaluate the model using the error based on the set criterion.

Part or all of the above embodiments may be described as in the following additional notes, but are not limited to the following.

(Additional note 1)
Predicted value obtaining means for obtaining a predicted value of the model for input data;
Output means for outputting evaluation information including a graph showing the predicted value and the actual measured value, and a display area for displaying a criterion for determining a prediction error;
Criterion acquisition means for acquiring the determination criteria;
Extracting means for extracting a predicted value that corresponds to a prediction error based on the judgment criterion and showing it on the graph;
A model analysis device comprising:

(Additional note 2)
The model analysis device according to appendix 1, wherein the evaluation information is displayed on the graph of the predicted values and includes a mark indicating a predicted value corresponding to the prediction error.

(Additional note 3)
The model according to

appendix

1 or 2, wherein the determination criterion includes an error index that specifies the type of error between the predicted value and the actual measured value, and information that defines a threshold for determining the predicted value as a prediction error. Analysis equipment.

(Additional note 4)
The model analysis device according to appendix 3, wherein the information defining the threshold includes information indicating a reference value of the threshold and a parameter for adjusting the threshold.

(Appendix 5)
5. The model analysis device according to

appendix

3 or 4, wherein the evaluation information includes a graph showing an error between the predicted value and the measured value based on a specified error index, and the threshold value.

(Appendix 6)
The extraction means changes the determination criteria so that when a specific predicted value on the graph is designated by the user, the predicted value is determined to be a prediction error, and the extraction unit changes the determination criterion so that the predicted value is determined to be a prediction error based on the changed determination criterion. The model analysis device according to any one of Supplementary Notes 1 to 5, which extracts a predicted value corresponding to .

(Appendix 7)
The input area includes information on a plurality of rules for selecting a predicted value to be displayed from predicted values corresponding to a prediction error,
The model analysis device according to appendix 2, wherein the evaluation information includes a mark indicating a predicted value selected according to a rule selected by the user.

(Appendix 8)
Get the model's predicted value for the input data,
outputting evaluation information including a graph showing the predicted value and the actual measured value, and a display area for displaying a criterion for determining a prediction error;
obtain the judgment criteria;
A model analysis method for extracting predicted values corresponding to prediction errors based on the determination criteria and displaying the predicted values on the graph.

(Appendix 9)
Get the model's predicted value for the input data,
outputting evaluation information including a graph showing the predicted value and the actual measured value, and a display area for displaying a criterion for determining a prediction error;
obtain the judgment criteria;
A recording medium having recorded thereon a program for extracting predicted values corresponding to prediction errors based on the determination criteria and causing a computer to execute processing shown on the graph.

Although the present disclosure has been described above with reference to the embodiments and examples, the present disclosure is not limited to the above embodiments and examples. Various changes can be made to the structure and details of the present disclosure that can be understood by those skilled in the art within the scope of the present disclosure.

1, 1x

Model generation system

2,

2x Display device

3, 3x Input device 7 Terminal device 100 Model generation device 112 Processor 121 Training data DB
122 Model training department 123 Model DB
124 Evaluation data DB
125 Prediction error analysis section 126 Evaluation information output section

Claims

Predicted value obtaining means for obtaining a predicted value of the model for input data;
Output means for outputting evaluation information including a graph showing the predicted value and the actual measured value, and a display area for displaying a criterion for determining a prediction error;
Criterion acquisition means for acquiring the determination criteria;
Extracting means for extracting a predicted value that corresponds to a prediction error based on the judgment criterion and showing it on the graph;
A model analysis device comprising:
The model analysis device according to claim 1, wherein the evaluation information is displayed on the graph of the predicted values and includes a mark indicating the predicted value corresponding to the prediction error.
3. The determination criteria include an error index that specifies the type of error between the predicted value and the measured value, and information that defines a threshold for determining the predicted value as a prediction error. Model analysis equipment.
The model analysis device according to claim 3, wherein the information defining the threshold includes information indicating a reference value of the threshold and a parameter for adjusting the threshold.
5. The model analysis device according to claim 3, wherein the evaluation information includes a graph showing an error between the predicted value and the measured value based on a specified error index, and the threshold value.
The extraction means changes the determination criteria so that when a specific predicted value on the graph is designated by the user, the predicted value is determined to be a prediction error, and the extraction unit changes the determination criterion so that the predicted value is determined to be a prediction error based on the changed determination criterion. 6. The model analysis device according to claim 1, wherein the model analysis device extracts a predicted value corresponding to .
The display area includes information on a plurality of rules for selecting a predicted value to be displayed from predicted values corresponding to a prediction error,
The model analysis device according to claim 2, wherein the evaluation information includes a mark indicating a predicted value selected according to a rule selected by the user.
Get the model's predicted value for the input data,
outputting evaluation information including a graph showing the predicted value and the actual measured value, and a display area for displaying a criterion for determining a prediction error;
obtain the judgment criteria;
A model analysis method for extracting predicted values corresponding to prediction errors based on the determination criteria and displaying the predicted values on the graph.
Get the model's predicted value for the input data,
outputting evaluation information including a graph showing the predicted value and the actual measured value, and a display area for displaying a criterion for determining a prediction error;
obtain the judgment criteria;
A recording medium having recorded thereon a program for extracting predicted values corresponding to prediction errors based on the determination criteria and causing a computer to execute processing shown on the graph.