WO2023181320A1

WO2023181320A1 - Model processing device, model processing method, and recording medium

Info

Publication number: WO2023181320A1
Application number: PCT/JP2022/014228
Authority: WO
Inventors: 啓太佐久間; 智哉坂井; 竜太松野; 義男亀田
Original assignee: 日本電気株式会社
Priority date: 2022-03-25
Filing date: 2022-03-25
Publication date: 2023-09-28

Abstract

Provided is a model processing device in which an existing model acquisition means acquires an existing model. A first output means outputs first evaluation information indicating a relationship between data for evaluation and output of the existing model for the data for evaluation. A correction acquisition means acquires correction information inputted for the first evaluation information. A corrected model acquisition means acquires a corrected model that has been corrected on the basis of the correction information inputted for the first evaluation information. A second output means outputs second evaluation information indicating a relationship in output between the existing model and the corrected model for the data for evaluation.

Description

Model processing device, model processing method, and recording medium

This disclosure relates to evaluating and modifying machine learning models.

In recent years, predictive models obtained through machine learning have been used in various fields. If the accuracy of the created predictive model is not sufficient, or if time has passed since the model was originally created and the trends in the data used have changed, the predictive model will need to be retrained. Patent Document 1 describes a method of adjusting model learning processing based on evaluation information for a created model.

International Publication WO2021/161896 Publication

When determining whether or not relearning is necessary for a model in use and the policy for modification, it is desirable to obtain sufficient information regarding the data used to create the model and the output of the model in use.

One objective of the present disclosure is to provide a model processing device that enables appropriate modification of a model by presenting sufficient information regarding the data used to create the model and the output of the model in operation.

In one aspect of the present disclosure, the model processing device includes:
an existing model acquisition means for acquiring an existing model;
a first output means for outputting first evaluation information indicating a relationship between the evaluation data and the output of the existing model with respect to the evaluation data;
a modification acquisition means for acquiring modification information input to the first evaluation information;
modified model acquisition means for acquiring a modified model that is modified based on modification information input to the first evaluation information;
a second output means for outputting second evaluation information indicating a relationship between outputs of the existing model and the modified model with respect to evaluation data;
Equipped with

In another aspect of the disclosure, a model processing method includes:
Get an existing model,
outputting first evaluation information indicating a relationship between the evaluation data and the output of the existing model with respect to the evaluation data;
Obtaining correction information input for the first evaluation information,
obtaining a modified model that is modified based on modification information input to the first evaluation information;
Second evaluation information indicating a relationship between the outputs of the existing model and the modified model with respect to the evaluation data is output.

In yet another aspect of the present disclosure, the recording medium includes:
Get an existing model,
outputting first evaluation information indicating a relationship between the evaluation data and the output of the existing model with respect to the evaluation data;
Obtaining correction information input for the first evaluation information,
obtaining a modified model that is modified based on modification information input to the first evaluation information;
A program is recorded that causes a computer to execute a process of outputting second evaluation information indicating a relationship between outputs of the existing model and the modified model with respect to evaluation data.

According to the present disclosure, by presenting sufficient information regarding the data used to create the model and the output of the model in operation, it is possible to appropriately modify the model.

FIG. 1 is a block diagram showing the overall configuration of a model generation system according to a first embodiment. FIG. 2 is a block diagram showing the hardware configuration of a model generation device. 1 is a block diagram showing a functional configuration of a model generation device according to a first embodiment; FIG. A display example of evaluation information in the case of prediction of one-dimensional time series data is shown. A display example of evaluation information in the case of prediction of multidimensional time series data is shown. An example of a modification UI for inputting parameter modification information is shown. An example of a correction UI for inputting assignment correction information is shown. A display example of a model visualization UI and a correction UI in the case of regression analysis is shown. A display example of a model comparison UI and a correction UI in the case of regression analysis is shown. Shows how to input correction information. This is a display example of a model visualization UI and a modification UI in the case of a classification model. A display example of a model comparison UI and a modification UI in the case of a classification model is shown. A display example of a correction UI related to explanatory variables is shown. 7 is a flowchart of model correction processing performed by the model generation device. FIG. 1 is a block diagram showing a schematic configuration of a model generation system using a server and a terminal device. FIG. 2 is a block diagram showing the functional configuration of a model processing device according to a second embodiment. It is a flow chart of processing by a model processing device of a 2nd embodiment.

Hereinafter, preferred embodiments of the present disclosure will be described with reference to the drawings.
<First embodiment>
[Concept explanation]
First, the basic concept of this embodiment will be explained. In this embodiment, an interactive UI (User Interface) is used to present evaluation information regarding a predictive model (hereinafter also simply referred to as a "model") to the user, obtain model modification information from the user, and make necessary adjustments. perform model modifications. Note that the "user" is a person who modifies a model, such as a model developer or operator. Furthermore, the tasks of the prediction model include various tasks such as regression of time series data, regression of data other than time series, discrimination, and classification, and are not limited to specific tasks.

Here, the "model" is information representing the relationship between explanatory variables and objective variables. A model is, for example, a component for estimating a target result by calculating a target variable based on explanatory variables. A model is generated by executing a learning algorithm using as input learning data for which values of objective variables have already been obtained and arbitrary parameters. The model may be represented, for example, by a function c that maps an input x to a ground answer y. The model may be one that estimates a numerical value to be estimated, or may be one that estimates a label to be estimated. The model may output variables that describe the probability distribution of the target variable. A model is sometimes described as a "learning model," "analytical model," "AI (Artificial Intelligence) model," or "prediction formula."

Specifically, the above interactive UI includes a model visualization UI, a modification UI, and a model comparison UI. The "model visualization UI" is a UI for presenting to the user information indicating the relationship between training data used to generate a model and the output of the generated model. The "modification UI" is a UI for the user to input information for modifying the model (hereinafter referred to as "modification information"). In addition, the "model comparison UI" refers to the comparison between the model before being modified based on modification information (also referred to as "existing model") and the model after modification (also referred to as "post-modification model"). This is a UI for presenting information indicating relationships to the user. Note that the modification UI can be used in combination with the model visualization UI or the model comparison UI. In this way, in this embodiment, by using the interactive UI, it is possible to appropriately reflect the domain knowledge possessed by the user and modify the model.

[overall structure]
FIG. 1 is a block diagram showing the overall configuration of a model generation system according to a first embodiment. The model generation system 1 includes a model generation device 100, a display device 2, and an input device 3. The model generation device 100 is configured by, for example, a computer such as a personal computer (PC). The display device 2 is, for example, a liquid crystal display device, and displays the evaluation information generated by the model generation device 100. The input device 3 is, for example, a mouse, a keyboard, etc., and is used by the user to give instructions and input necessary when modifying a model.

First, the operation of the model generation system 1 will be schematically explained. The model generation device 100 generates a machine learning model (hereinafter simply referred to as a "model") using training data prepared in advance. Furthermore, the model generation device 100 displays evaluation information indicating the relationship between the training data used to generate the model and the output of the generated model on the display device 2. This evaluation information is provided to the user using the model visualization UI described above. This allows the user to evaluate the performance of the generated model in relation to the training data. The generated model is operated in the planned environment.

Now, if the performance of the existing model in operation becomes insufficient due to a change in the tendency of the data to be processed while the model is in operation, the user can use the input device 3 to input model correction information. Enter. This input is performed using the modification UI described above. The model generation device 100 modifies the existing model based on the input modification information and generates a modified model. Furthermore, the model generation device 100 displays evaluation information on the display device 2 indicating the relationship between the output of the existing model before modification and the output of the modified model. This evaluation information is presented to the user using the model comparison UI described above. This allows the user to evaluate the performance of the modified model in relation to the existing model before modification. In this way, the user can appropriately modify the model by referring to the evaluation information indicating the relationship between the training data and the created model, or the relationship between the model before and after modification.

Note that the model visualization UI and model comparison UI may present the relationship between data other than training data, that is, data that the model has not learned, and the created model. The data that the model has not learned includes, for example, validation data (test data) and operational data. Displaying the model's prediction results for data that the model has not learned is important when considering model issues or when comparing models.

[Hardware configuration]
FIG. 2 is a block diagram showing the hardware configuration of the model generation device 100. As illustrated, the model generation device 100 includes an interface (I/F) 111, a processor 112, a memory 113, a recording medium 114, and a database (DB) 115.

The I/F 111 inputs and outputs data to and from external devices. Specifically, training data used for model generation and modification information input by the user using the input device 3 are input to the model generation device 100 through the I/F 111. Further, evaluation information regarding the relationship between the training data and the output of the existing model, evaluation information regarding the relationship between the output of the existing model and the output of the corrected model, etc. are output to the display device 2 through the I/F 111.

The processor 112 is a computer such as a CPU (Central Processing Unit), and controls the entire model generation device 100 by executing a program prepared in advance. Note that the processor 112 may be a GPU (Graphics Processing Unit) or an FPGA (Field-Programmable Gate Array). The processor 112 executes model modification processing, which will be described later.

The memory 113 is composed of ROM (Read Only Memory), RAM (Random Access Memory), and the like. Memory 113 is also used as a working memory while processor 112 executes various processes.

The recording medium 114 is a non-volatile, non-temporary recording medium such as a disk-shaped recording medium or a semiconductor memory, and is configured to be detachable from the model generation device 100. The recording medium 114 records various programs executed by the processor 112. When the model generation device 100 executes various processes, a program recorded on the recording medium 114 is loaded into the memory 113 and executed by the processor 112.

The DB 115 stores information regarding existing models and modified models generated by the model generation device 100. In addition, the DB 115 also includes training data input through the I/F 111, correction information input by the user, evaluation information regarding the relationship between the training data and the output of the existing model, and information on the relationship between the output of the existing model and the corrected model. Stores evaluation information regarding the relationship with output.

(Functional configuration)
FIG. 3 is a block diagram showing the functional configuration of the model generation device 100 of the first embodiment. The model generation device 100 functionally includes a training data DB 121, a model training section 122, a model DB 123, an evaluation data DB 124, and an evaluation information output section 125.

The training data DB 121 stores training data used for model generation. Training data D1 is input to model training section 122. Note that the training data D1 is composed of a plurality of combinations of input data and correct labels (teacher labels) for the input data.

The model training unit 122 trains a model using training data and generates a model. The model training unit 122 outputs model data M corresponding to the generated model to the model DB 123 and the evaluation information output unit 125. Note that the model data M includes a plurality of parameter information constituting the model. The parameter information includes, for example, information on explanatory variables used as model inputs, information on weights for each explanatory variable, information on weights for each sample forming input data, and the like.

Additionally, the model training unit 122 retrains the existing model to generate a modified model. In this case, the model training unit 122 corrects the parameters constituting the model based on the correction information D3 input by the user using the input device 3, and uses training data for retraining as necessary to improve the model. Perform retraining. The model training unit 122 stores the model data M of the corrected model obtained through retraining in the model DB 123 and outputs it to the evaluation information output unit 125. The model training unit 122 is an example of a correction acquisition unit.

The evaluation data DB 124 stores evaluation data used to evaluate the generated model. The evaluation data includes various types of data that can be used to evaluate the model. Examples of evaluation data include the following.
(1) “Data not used for model generation” called validation data or test data
In this case, the evaluation data is basically a set of input data and correct answer labels.
(2) “Newly collected data after model generation” such as operational data
Note that if labeling is not performed immediately, the evaluation data may be input-only data.
(3) “Data that is generated by some method and is unknown to the model”
For example, if the feature amount in the input data is (day of the week, holiday, weather), it is possible to create pseudo future data using calendar information and weather forecasts.
(4) Training data Training data may be used as evaluation data to evaluate the model.

The evaluation information output unit 125 generates evaluation information D2 for evaluating each model based on the model data M, and outputs it to the display device 2. The evaluation information D2 includes information regarding the relationship between the evaluation data and the output of the generated model, information regarding the relationship between the output of the existing model and the output of the corrected model with respect to the evaluation data, and the like. Note that the evaluation information output unit 125 outputs data for displaying evaluation information to the display device 2 using the model visualization UI or model comparison UI described above. The evaluation information output unit 125 is an example of an existing model acquisition means, a modified model acquisition means, a first output means, and a second output means.

The display device 2 displays the evaluation information D2 output by the evaluation information output section 125. Thereby, the display device 2 presents evaluation information to the user using the model visualization UI or the model comparison UI. In addition, the input device 3 outputs modification information D3 input by the user using the modification UI described above to the model training unit 122.

In this way, the user refers to the evaluation information D2 displayed on the display device 2 using the model visualization UI and the model comparison UI, and evaluates the performance of the existing model and the modified model. Then, the user inputs the correction information D3 into the input device 3 using the correction UI as necessary. The model training unit 122 corrects the model by retraining the model using the input correction information D3. In this way, the user can modify the model at an appropriate time and with an appropriate policy.

[Example of interactive UI]
Next, an example of the above-mentioned interactive UI will be explained. As mentioned above, the interactive UI includes a model visualization UI, a modification UI, and a model comparison UI.

(Forecasting time series data)
First, a case where the model's task is to predict time series data will be explained.
(1) Prediction of one-dimensional time-series data FIG. 4 shows a display example of evaluation information when the task of the model is prediction of one-dimensional time-series data. Specifically, display example G1 shows training data and model output for a model that predicts daily sales of a certain product. In display example G1, the horizontal axis is date and the vertical axis is sales. Graph 11 shows the training data used to generate the model. Graph 12 shows the output of the existing model generated using the training data shown in graph 11 and future input data. Note that since the graph 12 is a predicted value by a model, correct label data corresponding to future input data is not necessarily required. Display example G1 is a display example using a model visualization UI that displays evaluation information indicating the relationship between training data, future input data, and the output of an existing model.

When operating a general regression model like a time series model, in order to output future predictions, prepare input data at a future point in time, input it to the model, and output a prediction at a future point in time. . Similarly, in the case of non-time-series regression or discrimination, in order to output the predicted value of a future target variable that is not in the training data, prepare input data at a future point in time, input it to the model, and then Output the predicted value at the time. As an exception, in the case of a time series model without explanatory variables, future input data is not necessary because future predicted values can be obtained by extending training data. Note that the time series model without explanatory variables refers to a model in which the latest predicted value is regressed from the past value of the objective variable, that is, an autoregressive model.

When the user looks at the display example G1 and determines that there is a problem with the existing model, the user inputs correction information. Specifically, the user inputs modification information into the input device 3 using the modification UI. In this embodiment, the modification information includes parameter modification information regarding training data and task modification information regarding the output of the existing model. The parameter modification information is information that specifies the adjustment location and adjustment amount of the weight given by the model to the training data. That is, the user can specify a particular period of training data and adjust the weight for the training data for that period.

In the example of FIG. 4, the user specifies the range (period) indicated by the line segment 21 among the training data indicated by the graph 11, and makes a correction indicating that the weight for the training data belonging to that period is to be increased by 1.5 times. Entering information. Similarly, a user can specify a particular period of training data and provide input to reduce the weight for training data for that period. In the example of FIG. 4, the user specifies the range (period) shown by the line segment 22 among the training data shown by the graph 11, and ignores the training data belonging to that period. Modification information to set the weight to "0" is input.

On the other hand, the task correction information is information that specifies the adjustment amount of the task location and task degree (hereinafter also referred to as "task degree") in the model output. The problem level indicates the extent to which there are problems with the output of the model. The level of the task can be specified by a plurality of levels (for example, 10 levels). The user can designate a specific period of model output as a task location and input the task level of that task location. In the example of FIG. 4, the user specifies the range (period) indicated by

line segments

23 and 24 among the outputs of the model indicated by the graph 12, and sets the problem level of the model output in that period to "4" and "5", respectively. ” is inputting the correction information to be set.

When the modification information is input, the model training unit 122 retrains the existing model based on the modification information and generates a modified model. The evaluation information output unit 125 generates evaluation information about the modified model and transmits it to the display device 2. FIG. 4 shows a display example G2 of evaluation information of the corrected model. In display example G2, a graph 11 of training data is displayed, similar to display example G1. In display example G2, a graph 12 showing the output of the existing model before modification is shown by a broken line, and a graph 13 showing the output of the model after modification is superimposed. This allows the user to easily compare the output of the model before and after modification.

Furthermore, if the user sees display example G2 and thinks that further modification is necessary, he or she can input modification information as in display example G1 to further modify the modified model. As described above, display examples G1 and G2 are display examples that use the model visualization UI and the model comparison UI at the same time. Note that in the example of FIG. 4, the model comparison UI compares two models before and after modification, but multiple models generated in the process of trial and error of model modification may be visualized at the same time. In this case, in the graph G2 of FIG. 4, a plurality of outputs of the existing model such as the graph 12 are displayed simultaneously.

(2) Prediction of multidimensional time series data Next, a case where the task of the model is prediction of multidimensional time series data will be explained. Prediction of multidimensional time-series data refers to a case where the model's task is to first predict a plurality of time-series data and then use those prediction results to predict the final objective variable. In this case as well, basically, the method for predicting one-dimensional time-series data described above can be applied to each dimension.

FIG. 5 shows a display example of evaluation information in the case of prediction of multidimensional time series data. In this example, the model first predicts the two-dimensional explanatory variables "humidity" and "temperature" and then predicts the one-dimensional target variable "sales" from the prediction results. Specifically, display example G3 is a display example of evaluation information regarding the task of predicting daily humidity. In display example G3, a graph 14 showing training data, a graph 15 showing the output of the existing model, and a graph 16 showing the output of the corrected model are displayed simultaneously. In display example G3, the user is inputting modification information for modifying the weight for training data using line segment 21.

Display example G4 is a display example of evaluation information regarding the task of predicting daily temperature. In display example G4, a graph 17 showing training data, a graph 18 showing the output of the existing model, and a graph 19 showing the output of the corrected model are displayed simultaneously. In display example G4, the user is inputting modification information for modifying the weight for the training data using the line segment 22.

Furthermore, display example G5 is a display example of evaluation information regarding a task of predicting daily sales from predicted humidity and temperature. In display example G5, similarly to FIG. 4, a graph 11 showing training data, a graph 12 showing the output of the existing model, and a graph 13 showing the output of the corrected model are displayed simultaneously. Note that in display example G5 as well, the user can input correction information as in FIG.

(3) Method of inputting modification information Next, a method of inputting modification information using the modification UI will be described in detail. As described above, the modification information includes parameter modification information regarding the training data and task modification information regarding the output of the existing model. The parameter modification information is information that specifies the adjustment location and adjustment amount of the weight given by the model to the training data.

FIG. 6 shows an example of a modification UI for inputting parameter modification information regarding training data. The user specifies a particular period of training data and provides input to adjust the weight for that period of training data. FIG. 6A shows an example of specifying a range using a cursor. Specifically, in FIG. 6A, the user specifies a range of adjustment points in the displayed training data using a cursor. In the example of FIG. 6A, the user specifies a rectangular range 31 with the cursor as the weight adjustment location. After specifying the range, the user inputs the amount of weight adjustment. The weight adjustment amount can be, for example, a magnification of the current weight value. For example, when the user inputs "1.5" as the weight adjustment amount, the model training unit 122 increases the weight for training data belonging to range 31 to 1.5 times the current value. By inputting a value less than "1" as the weight adjustment amount, the weight can be reduced. This method is effective when uniformly adjusting weights for a certain range of training data.

FIG. 6(B) shows an example in which the weight adjustment location and adjustment amount are input by clicking using a mouse. Specifically, the user selects either a weight increasing mode or a weight decreasing mode, and then clicks a data point on the training data graph that corresponds to the day on which he or she wishes to adjust. In the case of the mode that increases the weight, the weight increases according to the number of clicks by the user. Furthermore, in the case of a mode in which the weight is reduced, the weight is reduced according to the number of clicks by the user. This method is effective when the amount of weight adjustment is different for each date.

FIG. 6(C) shows an example in which the weight adjustment location and adjustment amount are input by specifying the length of the bar indicating the weight. In FIG. 6C, the weight values set by the existing model for the training data of each date are shown by weight bars. By changing the length of the weight bar on the day that the user wants to adjust, the user can adjust the weight for the training data on that day. That is, if the user wants to increase the weight, he/she can increase the length of the weight bar. As shown in FIG. 6C, the weight adjustment amount is displayed in an easy-to-read manner by displaying the weight bar 32a indicating the weight of the existing model and the portion 32b whose length has been changed by the user in different colors. be able to.

FIG. 7 shows an example of a modification UI for inputting problem modification information regarding the output of an existing model. The user specifies a specific period in the output of the existing model as a problem part, and inputs the problem level of the model output of the problem part. When the task location and task level are input as modification information, the model training unit 122 modifies the existing model so that the task at the specified task location is solved. For example, the model training unit 122 modifies the existing model so that the difference between the model output and the actual measurement value at the designated task location becomes smaller. In addition, the model training unit 122 increases the amount of modification of the existing model as the input task level increases.

Note that the model training unit 122 can also modify model parameters simply by inputting task modification information. In this case, there is no need to input parameter modification information. The method for modifying model parameters based on the input of problem modification information is as follows. First, when task correction information is specified for training data (for example, graph 11 in display example G1 in FIG. 4), the model training unit 122 changes the weight of the sample specified as a task location based on the task degree. Change according to predetermined rules. Furthermore, when the task correction information is specified for future data (for example, graph 12 in display example G1 in FIG. 4), the model training unit 122 selects a sample within the training data that is similar to the sample specified as the task location. The weight of the sample is changed according to a predetermined rule based on the task level. Here, as the predetermined rule, a "function or model indicating the relationship between the task level and the weight adjustment amount" prepared independently of past input logs can be used.

FIG. 7(A) shows an example of specifying a range using a cursor. Specifically, in FIG. 7A, the user uses a cursor to specify a range of adjustment points in the output of the displayed existing model. In the example of FIG. 7A, the user specifies a rectangular range 33 with the cursor as the task location. After specifying the range, the user inputs the level of assignment for that assignment location. The level of the task can be set to 10 levels, for example. It is assumed that the larger the numerical value of the problem level is, the larger the problem is. For example, when the user inputs "3" as the task level, the model training unit 122 sets the task level of the output of the existing model belonging to range 33 as "3" on a scale of 10. This method is effective when uniformly adjusting the task level for a certain range of existing model outputs.

FIG. 7(B) shows an example of inputting the task location and task level of the existing model output by clicking with a mouse. Specifically, the user selects either a mode for increasing the task level or a mode for decreasing the task level, and then clicks on the data point corresponding to the day on the graph of the existing model that he/she wants to adjust. In the mode of increasing the task level, the task level increases according to the number of clicks by the user. Furthermore, in the case of a mode in which the task level is reduced, the task level is reduced according to the number of clicks by the user. This method is effective when the amount of adjustment of the task level differs for each date.

FIG. 7(C) shows an example of inputting the task location and the adjustment amount of the task level by specifying the length of a bar indicating the task level (hereinafter referred to as the "assignment level bar"). In FIG. 7C, the problem level of the output of the existing model on each date is shown by a problem level bar. Note that the task level bar for each date is displayed by calculating the task level based on, for example, the difference between the output of an existing model on each date and the actual measurement value on that day. Note that when the actual measured value of the objective variable has already been obtained in this way, it is preferable to display the actual measured value on the model visualization UI and the model comparison UI. For example, the actual measured value of the target variable is displayed on the graph 12 of display examples G1 and G2 in FIG. This makes it easier for the user to find the location of the problem.

The user can adjust the issue level of the existing model output for that day by changing the length of the issue level bar on the day that the user wants to adjust. The amount of adjustment of the task level can be specified by the amount of change in the length of the task level bar. That is, if the user wants to increase the task level, he/she can increase the length of the task level bar. As shown in FIG. 7(C), by distinguishing and displaying the task level bar 34a indicating the task level of the existing model output and the portion 34b whose length has been changed by the user using different colors, the amount of adjustment of the task level can be adjusted. can be displayed easily.

(regression analysis)
Next, a case where the model task is regression analysis will be explained. When the model task is regression analysis of data other than time series data, the evaluation information is shown by a two-dimensional or three-dimensional scatter plot.

(1) Model visualization UI and correction UI
FIG. 8(A) shows a display example G11 of the model visualization UI and correction UI in the case of regression analysis. Display example G11 is an example of a model that predicts sales of hamburgers. The horizontal axis indicates the season from summer to winter, and the vertical axis indicates the time from day to night. Each data point in the diagram represents a predicted value of sales. Each data point is shown in a darker color (closer to black) as the sales value increases, and a lighter color (closer to white) as the value decreases. In display example G11, the model's predicted value for the training data used to train the existing model is shown as a circle, and the model's predicted value for the validation data used for evaluation and verification of the existing model is shown as a square. Note that the vertical and horizontal axes in Figure 8(A) are examples of feature quantities, but if there are three or more feature quantities, two feature quantities are randomly selected from the three or more feature quantities. Alternatively, a plurality of feature quantities may be integrated and converted into two feature quantities. This point also applies to the case of FIG. 8(B), which will be described later.

As in display example G11, by showing the distribution of predicted values of the model with respect to training data and validation data, the user can know the tendency of the existing model. For example, in display example G11, the predicted value for the training data and the predicted value for the validation data are different in the region of winter nights, indicating that the prediction accuracy of the existing model is low in the region of winter nights. In this case, the user can modify the existing model by inputting parameter modification information that increases the weight for the data point 37 that corresponds to the condition of winter night, for example in display example G11.

FIG. 8(B) shows another display example G12. Display example G12 simultaneously displays training data used for training the existing model and predicted values of the model for operational data used in actual operation. The training data shows actual measured sales. On the other hand, since actual measured values are not available for operational data, predicted values are shown that are the results of predictions made using existing models for operational data. Specifically, in display example G12, the outer periphery of the data points of the training data is shown by a solid line, and the outer periphery of the data points of the operational data is shown by a dotted line. Note that the vertical axis, horizontal axis, and colors indicating sales values are the same as in FIG. 8(A).

As in display example G12, by showing the distribution of training data and predicted values for operational data, the user can know the tendency of the existing model. For example, in display example G12, under the condition of a winter night, the predicted value for the operational data is large compared to the small value for the training data. Therefore, the user can determine that the prediction accuracy of the existing model is insufficient under the condition of a winter night and that correction is necessary. In this case as well, the user can modify the existing model by inputting parameter modification information that increases the weight for the data point 37 that corresponds to the condition of winter night in display example G12.

(2) Model comparison UI and correction UI
FIG. 9 shows a display example G13 of the model comparison UI and correction UI. Display example G13 is a plot of the error between the training data and the output of the existing model on a two-dimensional feature space. The vertical and horizontal axes indicate some feature amount that defines the feature space. That is, the vertical axis and the horizontal axis each indicate one feature amount. Note that when the feature quantities used by the model are three or more dimensional, the evaluation information output unit may randomly select two feature quantities from three or more feature quantities, or may integrate multiple feature quantities. It may also be converted into two feature quantities.

The color of each displayed data point indicates the error between the training data and the output of the existing model; the darker the color, the greater the error. The user looks at the display example G13, determines that the points located in the lower right area have a large error, and inputs correction information targeting those points. For example, the user uses a stylus pen or the like to draw a line segment 35 to enclose an area with a large error, thereby specifying a point belonging to the area surrounded by the line segment 35 as a task location. Further, the user inputs the task level for the points belonging to the area surrounded by the line segment 35. Display example G13 thus allows the user to input assignment modification information including the assignment location and assignment level. Note that the line segment 35 may be used to input parameter modification amounts such as sample weight modification information.

Note that in display example G13, the color of each data point indicates the error between the training data and the output of the existing model, but instead, the actual value or predicted value may be colored as shown in FIG. .

The model training unit 122 modifies the existing model based on the input modification information and generates a modified model. The evaluation information output unit 125 generates a new display example G14 shown in FIG. 9 based on the training data and the output of the corrected model, and displays it on the display device 2. In display example G14, the error at the points belonging to the lower right region is smaller, indicating that the performance of the model is improved. Note that the display example G14 shows an error between the training data and the output of the corrected model, and it is difficult to compare the existing model and the corrected model using only the display example G14. Therefore, in addition to display example G14, display example G13 showing the error between the training data and the output of the existing model may be displayed at the same time.

Display examples G13 and G14 above display the error between the training data and the model output as evaluation information, but instead, the error between the validation data and the model output may be displayed, or the actual value and predicted value may be displayed. Values may be shown in color.

(3) Method for inputting modification information Next, a method for inputting modification information in the modification UI will be described in detail. The modification UI is used for inputting parameter modification information in the display using the model visualization UI and modification UI, as shown in Figures 8 (A) and (B), and for inputting parameter modification information in the display using the model comparison UI and modification There are cases where assignment correction information is input in a display using a UI, but since the input method by the user is the same, they will be explained together.

FIG. 10(A) shows a method of inputting correction information by clicking on each point on the display of evaluation information. When inputting parameter modification information, the user specifies the weight adjustment point by clicking on the data point of the displayed training data whose weight is to be changed, and the amount of weight adjustment is determined by the number of clicks. Can be specified. In addition, when inputting issue correction information, the user can specify the issue location by clicking on a data point that has an issue among the displayed data points such as errors, and specify the issue level by the number of clicks. I can do it.

FIG. 10(B) shows a method of inputting correction information by surrounding data points in the display of evaluation information with line segments, as in display example G13 of FIG. 9. When inputting parameter correction information, the user specifies the weight adjustment point by enclosing the data point of the displayed training data whose weight is to be changed with a line segment, and separately inputs the weight adjustment amount. Or you can specify it. Furthermore, when inputting task correction information, the user may designate the task location by enclosing the data point having the task among the displayed data points with a line segment, and may separately input or specify the task level.

Similarly to the case of FIG. 10(B), FIG. 10(C) shows a method of inputting correction information by surrounding data points in the display of evaluation information with line segments or the like. However, in the case of Fig. 10(B), the same weight adjustment amount or task level is specified for all data points belonging to the area surrounded by the line segment, whereas in the case of Fig. 10(C) In the example, the gradation allows different weight adjustment amounts or task levels to be set for data points belonging to the area surrounded by the line segment. Specifically, when inputting parameter modification information, the user specifies the weight adjustment point by enclosing the data point whose weight is to be changed among the displayed training data points with a line segment, and then selects a different weight using the gradation display. All you have to do is specify the amount of adjustment. Furthermore, when inputting assignment correction information, the user can specify the assignment location by enclosing the data points with assignments among the displayed data points with a line segment, and specify a different assignment degree using the gradation display. .

(classification)
Next, a case where the model task is classification will be explained. When the task of the model is classification, evaluation information is shown by a two-dimensional or three-dimensional scatter plot, as in the case of regression analysis.

(1) Model visualization UI and correction UI
A display example G13 shown in FIG. 11 is a display example of a model visualization UI and a modification UI in the case of a classification model. The existing model is, for example, a binary classification model that classifies a certain image into two, "A" and "B." Display example G13, like display example G11 in FIG. 8, shows a feature amount space in which two specific feature amounts are plotted on the vertical and horizontal axes. Each data point is shown in a color corresponding to the misclassification rate calculated based on the training data and the output of the existing model, with the darker the color, the higher the misclassification rate.

As in display example G13, by showing the misclassification rate, the user can know the tendency of the existing model. For example, in display example G13, it can be seen that the misclassification rate is high in the lower right region of the feature space. Therefore, the user can specify the weight adjustment location by inputting the line segment 41 surrounding the data point that requires correction, and can input correction information by separately specifying the weight adjustment amount.

Note that any known method can be used to calculate the misclassification rate. For example, in the case of binary classification, if the labels of the predicted value by the model and the actual value do not match, the misclassification rate may be set to "1", and if they match, the misclassification rate may be set to "0". In this case, in display example G13 illustrated in FIG. 11, each data point has only two colors. As another example, when the classification model outputs a classification score, the difference between the actual value (correct value) and the classification score may be used as the misclassification rate. That is, if the actual measured values (correct values) are "0" and "1",
(Misclassification degree) = (actual value) - (classification score)
And it is sufficient.

Note that in display example G13 in FIG. 11, correction information is input by enclosing the data points with line segments 41, but in the case of a classification task, any of the methods shown in FIGS. You may also use

(2) Model comparison UI and correction UI
FIG. 12A shows a display example G14 of the model comparison UI and correction UI in the case of a classification model. Display example G14 shows the misclassification rate by the modified model modified based on the modification information input by the user in display example G13. In this case, the misclassification rate is calculated based on the training data and the output of the modified model. As can be seen from a comparison with display example G13 in FIG. 11, in the output of the corrected model, the misclassification rate has been improved for data points in the lower right region of the feature space. In the model comparison UI, data points for which the classification results have changed between the existing model and the revised model are distinguished and displayed, for example by emphasizing the outline, such as data point 42, so that the output of the old and new models can be compared. Relationships can be shown. Note that in display example G14, the model may be modified by further inputting modification information using any of the methods shown in FIGS. 10(A) to 10(C).

FIG. 12(B) shows another display example G15 of the model comparison UI. In display example G15, the color of each data point indicates the absolute value of the difference in classification scores between the existing model and the modified model. That is, the larger the difference between the classification results between the existing model and the modified model, the darker the color of the data point becomes. Even in this case, the relationship between the outputs of the old and new models can be shown. Note that in display example G15 as well, the model may be modified by further inputting modification information using any of the methods shown in FIGS. 10(A) to 10(C).

(Corrected UI regarding explanatory variables)
In the above example, the weight for each data point is adjusted as the parameter modification information, but instead, the weight may be adjusted for each explanatory variable. FIG. 13A shows an example of a modification UI for modifying the weight for each explanatory variable. This modification UI displays a list 51 of explanatory variables used by the existing model and a weight bar 52 set for each explanatory variable. In the explanatory variable list 51, explanatory variables whose checkboxes are checked are explanatory variables used by existing models. The user can increase or decrease the weight for each explanatory variable by changing the length of the weight bar 52 corresponding to each explanatory variable using the cursor 52x.

FIG. 13(B) shows an example of a modification UI for changing the explanatory variables themselves used by the existing model. This modification UI displays a list 53 of explanatory variables used by the existing model. If the user wants to delete an explanatory variable that is in use, he or she can remove the checkbox in the list 53. On the other hand, if the user wants to add an explanatory variable, when the user clicks on the "Add" tab in the list 53, an addition list 54 is displayed. The additional list 54 is a list of commonly used explanatory variables. The user can add a new explanatory variable by referring to the addition list 54, checking the checkbox of the explanatory variable he or she wishes to add, and pressing the "OK" button.

[Model correction processing]
Next, model correction processing by the model generation device 100 will be explained. FIG. 14 is a flowchart of model modification processing performed by the model generation device 100. The model modification process is a process for modifying an existing model, and is executed at an appropriate timing, such as during model operation, for example. This processing is realized by the processor 112 shown in FIG. 2 executing a program prepared in advance and operating as the element shown in FIG. 3.

First, the model generation device 100 acquires a target existing model (step S10). Next, the model generation device 100 displays the training data and the output of the existing model using the model visualization UI (step S11). The user looks at the relationship between the displayed training data and the output of the existing model and determines whether or not the existing model needs to be modified. If the user determines that the existing model needs to be modified, the user uses the modification UI to make the modification. Enter your information. The model generation device 100 determines whether the user has input correction information (step S12). If no modification information is input (step S12: No), the process ends.

On the other hand, if correction information is input (step S12: Yes), the model generation device 100 acquires the correction information. Specifically, when the user finds an output problem of an existing model and inputs problem correction information, the model generation device 100 acquires the problem correction information (step S13). Furthermore, when the user determines that parameters such as weights and explanatory variables need to be modified and inputs parameter modification information, the model generation device 100 acquires the parameter modification information (step S14).

Next, the model generation device 100 modifies the parameters of the existing model based on the input problem modification information and parameter modification information to generate a modified model (step S15). Then, the model generation device 100 displays the output of the existing model and the output of the corrected model using the model comparison UI (step S16). The process then returns to step S12. In this way, steps S13 to S16 are repeated while the user determines that the model needs to be modified, and when it is determined that the model does not need to be modified, the process ends.

In this way, the user can see the relationship between the training data displayed by the model visualization UI and the output of the existing model, and the relationship between the output of the existing model and the output of the modified model displayed by the model comparison UI. , determine model modifications. If it is determined that the model needs to be modified, the model can be modified by inputting modification information using the modification UI.

In the model correction process described above, the input of problem correction information in step S13 and the input of parameter correction information in step S14 are performed separately. This allows a single user to input task modification information and parameter modification information at different timings. Furthermore, it is also possible for different users to input the task modification information and the parameter modification information. For example, a model operator (AI operator) may input task modification information, and a model creator (AI creator) may input parameter modification information to modify the model. In this case, appropriate model modification can be achieved through smooth communication between the model creator and model operator.

In the above model modification process, the model generation device 100 acquires the parameter modification information input by the user in step S14, but if the problem modification information input by the user is acquired in step S13, the model generation device 100 The generation device 100 may function as a weight adjustment amount calculation means. Specifically, a function or Create a model in advance. When the model generation device 100 acquires the task correction information in step S13, it presents the recommended value of the weight adjustment amount to the user using the above function or model. The user may accept the presented recommended value, may change the presented recommended value, or may input the weight adjustment amount himself. Thereby, the input of parameter correction information in step S14 can be made more efficient. Furthermore, when a user with little parameter experience modifies a model, he or she can refer to the amount of weight adjustment made by many users in the past.

[Modified example]
(Modification 1)
In the above embodiment, the model generation device 100 generates the initial model based on training data, but this is not essential. For example, the model generation device 100 may generate a new modified model by acquiring an existing model from the outside and retraining the existing model.

(Modification 2)
Further, the function of the model training section 122 in the model generation device 100 may be provided externally. For example, a model training device having the functions of the model training unit 122 in FIG. 3 and a model evaluation device having the functions of the evaluation data DB 124 and the evaluation information output unit 125 are provided separately. In this case, the model evaluation device inputs the modification information D3 input from the input device 3 to the model training device, and acquires the modified model generated by the model training device from the model training device.

(Modification 3)
In the above embodiment, the model generation device 100 is configured as an independent device such as a PC, but instead, the model generation device may be configured with a server and a terminal device. FIG. 15 is a block diagram showing a schematic configuration of a model generation system 1x using a server and a terminal device. In FIG. 15, a server 100x includes the configuration of the model generation device 100 shown in FIG. Further, the display device 2x and input device 3x of the terminal device 7 used by the user are used as the display device 2 and input device 3 shown in FIG. With this configuration, it becomes possible to easily collect correction information and the like input by a plurality of users on the server 100x and share it.

<Second embodiment>
FIG. 16 is a block diagram showing the functional configuration of the model processing device according to the second embodiment. The information processing device 70 includes an existing model acquisition means 71 , a first output means 72 , a modification acquisition means 73 , a modified model acquisition means 74 , and a second output means 75 .

FIG. 17 is a flowchart of processing by the model processing device of the second embodiment. Existing model acquisition means 71 acquires an existing model (step S71). The first output means 72 outputs first evaluation information indicating the relationship between the evaluation data and the output of the existing model with respect to the evaluation data (step S72). The modification acquisition means 73 acquires modification information input for the first evaluation information (step S73). The modified model acquisition unit 74 acquires a modified model that has been modified based on the modification information input to the first evaluation information (step S74). The second output means 75 outputs second evaluation information indicating the relationship between the outputs of the existing model and the modified model with respect to the evaluation data (step S75).

According to the information processing device 70 of the second embodiment, by presenting sufficient information regarding the data used for model creation and the output of the model in operation, it is possible to appropriately modify the model.

Part or all of the above embodiments may be described as in the following additional notes, but are not limited to the following.

(Additional note 1)
an existing model acquisition means for acquiring an existing model;
a first output means for outputting first evaluation information indicating a relationship between the evaluation data and the output of the existing model with respect to the evaluation data;
a modification acquisition means for acquiring modification information input to the first evaluation information;
modified model acquisition means for acquiring a modified model that is modified based on modification information input to the first evaluation information;
a second output means for outputting second evaluation information indicating a relationship between outputs of the existing model and the modified model with respect to evaluation data;
A model processing device comprising:

(Additional note 2)
The modification acquisition means acquires modification information input to the second evaluation information,
The model processing device according to appendix 1, wherein the modified model acquisition means acquires a new modified model that is modified based on modification information input to the second evaluation information.

(Additional note 3)
The first evaluation information includes the evaluation data and the output of the existing model,
The model processing device according to appendix 1, wherein the second evaluation information includes an output of the existing model and an output of the modified model.

(Additional note 4)
The first evaluation information indicates a difference between the training data and the output of the existing model,
The model processing device according to supplementary note 1, wherein the second evaluation information indicates a difference in output between the existing model and the modified model.

(Appendix 5)
The model is a model that predicts time series values,
The first evaluation information simultaneously indicates the evaluation data and the output of the model,
The model processing device according to appendix 1, wherein the second evaluation information simultaneously indicates an output of the existing model and an output of the modified model.

(Appendix 6)
The model is a regression model,
The first evaluation information indicates the evaluation data and the output of the existing model,
The model processing device according to appendix 1, wherein the second evaluation information is a graph showing errors between the evaluation data and outputs of the existing model and the modified model.

(Appendix 7)
The model is a classification model,
The first evaluation information indicates a misclassification rate of the classification result by the existing model,
The model processing device according to supplementary note 1, wherein the second evaluation information indicates a degree of misclassification of the classification result by the modified model or a difference between the classification score of the existing model and the classification score of the modified model.

(Appendix 8)
8. The model processing device according to any one of Supplementary Notes 1 to 7, wherein the modification information includes designation of a problem location and a problem level in the output of the existing model.

(Appendix 9)
The model processing device according to supplementary note 8, wherein the modification information includes designation of adjustment points and adjustment amounts of weights given by the model to the evaluation data.

(Appendix 10)
The model processing device according to appendix 8 or 9, further comprising calculation means for calculating an adjustment amount of weight given by the model to the evaluation data based on the task level.

(Appendix 11)
Get an existing model,
outputting first evaluation information indicating a relationship between the evaluation data and the output of the existing model with respect to the evaluation data;
Obtaining correction information input for the first evaluation information,
obtaining a modified model that is modified based on modification information input to the first evaluation information;
A model processing method that outputs second evaluation information indicating a relationship between outputs of the existing model and the modified model with respect to evaluation data.

(Appendix 12)
Get an existing model,
outputting first evaluation information indicating a relationship between the evaluation data and the output of the existing model with respect to the evaluation data;
Obtaining correction information input for the first evaluation information,
obtaining a modified model that is modified based on modification information input to the first evaluation information;
A recording medium storing a program that causes a computer to execute a process of outputting second evaluation information indicating a relationship between outputs of the existing model and the modified model with respect to evaluation data.

Although the present disclosure has been described above with reference to the embodiments and examples, the present disclosure is not limited to the above embodiments and examples. Various changes can be made to the structure and details of the present disclosure that can be understood by those skilled in the art within the scope of the present disclosure.

1, 1x

Model generation system

2,

2x Display device

3, 3x Input device 7 Terminal device 100 Model generation device 112 Processor 121 Training data DB
122 Model training department 123 Model DB
124 Evaluation information output section

Claims

an existing model acquisition means for acquiring an existing model;
a first output means for outputting first evaluation information indicating a relationship between the evaluation data and the output of the existing model with respect to the evaluation data;
a modification acquisition means for acquiring modification information input to the first evaluation information;
modified model acquisition means for acquiring a modified model that is modified based on modification information input to the first evaluation information;
a second output means for outputting second evaluation information indicating a relationship between outputs of the existing model and the modified model with respect to evaluation data;
A model processing device comprising:
The modification acquisition means acquires modification information input to the second evaluation information,
2. The model processing device according to claim 1, wherein the modified model acquisition means acquires a new modified model that is modified based on modification information input to the second evaluation information.
The first evaluation information includes the evaluation data and the output of the existing model,
The model processing device according to claim 1, wherein the second evaluation information includes an output of the existing model and an output of the modified model.
The first evaluation information indicates a difference between the training data and the output of the existing model,
The model processing device according to claim 1, wherein the second evaluation information indicates a difference in output between the existing model and the modified model.
The model is a model that predicts time series values,
The first evaluation information simultaneously indicates the evaluation data and the output of the model,
The model processing device according to claim 1, wherein the second evaluation information simultaneously indicates an output of the existing model and an output of the modified model.
The model is a regression model,
The first evaluation information indicates the evaluation data and the output of the existing model,
The model processing device according to claim 1, wherein the second evaluation information is a graph showing errors between the evaluation data and outputs of the existing model and the modified model.
The model is a classification model,
The first evaluation information indicates a misclassification rate of the classification result by the existing model,
The model processing device according to claim 1, wherein the second evaluation information indicates a degree of misclassification of the classification result by the modified model or a difference between the classification score of the existing model and the classification score of the modified model. .
The model processing device according to any one of claims 1 to 7, wherein the modification information includes designation of a problem location and a problem level in the output of the existing model.
9. The model processing device according to claim 8, wherein the modification information includes designation of adjustment points and adjustment amounts of weights given by the model to the evaluation data.
The model processing device according to claim 8 or 9, further comprising calculation means for calculating an adjustment amount of weight given by the model to the evaluation data based on the task level.
Get an existing model,
outputting first evaluation information indicating a relationship between the evaluation data and the output of the existing model with respect to the evaluation data;
Obtaining correction information input for the first evaluation information,
obtaining a modified model that is modified based on modification information input to the first evaluation information;
A model processing method that outputs second evaluation information indicating a relationship between outputs of the existing model and the modified model with respect to evaluation data.
Get an existing model,
outputting first evaluation information indicating a relationship between the evaluation data and the output of the existing model with respect to the evaluation data;
Obtaining correction information input for the first evaluation information,
obtaining a modified model that is modified based on modification information input to the first evaluation information;
A recording medium storing a program that causes a computer to execute a process of outputting second evaluation information indicating a relationship between outputs of the existing model and the modified model with respect to evaluation data.