WO2023223533A1

WO2023223533A1 - Estimation device, estimation method, and program

Info

Publication number: WO2023223533A1
Application number: PCT/JP2022/020927
Authority: WO
Inventors: 竜太松野; 智哉坂井; 啓太佐久間; 義男亀田
Original assignee: 日本電気株式会社
Priority date: 2022-05-20
Filing date: 2022-05-20
Publication date: 2023-11-23

Abstract

An estimation device 100 according to the present invention comprises: a data selection means 121 for selecting, from first explanatory variable data prepared according to a preset prediction model and first objective variable data corresponding to the first explanatory variable data, first explanatory variable data and first objective variable data corresponding to the first explanatory variable data, on the basis of second explanatory variable data that is not associated with an objective variable; and a performance estimation means 122 for estimating the performance of a prediction model's prediction about the second explanatory variable data, on the basis of the comparison between prediction data obtained by inputting the first explanatory variable data selected according to the prediction model and the first objective variable data corresponding to the first explanatory variable data.

Description

Estimation device, estimation method and program

The present disclosure relates to an estimation device, an estimation method, and a program.

Create a prediction model that predicts the objective variable data from the explanatory variable data based on the explanatory variable data and the objective variable data that corresponds to the explanatory variable data, and put the created prediction model into practice. Methods are being considered. For example, in Patent Document 1, a plurality of predictive model candidates are created from explanatory variable data and new data generated from the explanatory variable data, and the prediction accuracy of the plural predictive model candidates is evaluated. A technique for selecting a predictive model is disclosed.

International Publication No. 2021/229648

However, depending on the situation in which the prediction model is operated, it may take time to obtain data on the objective variable that is the target of the prediction by the prediction model. In such a case, a problem arises in that the value of the predictive performance index of the predictive model cannot be calculated until the target variable is obtained.

In view of the above-mentioned problems, an object of the present disclosure is to provide an estimation device, an estimation method, and a program that can estimate the prediction performance of a prediction model without using data of a target variable.

An estimation device that is one form of the present invention includes:
A second explanatory variable to which an objective variable is not associated from first explanatory variable data prepared according to a preset prediction model and first objective variable data corresponding to the first explanatory variable data. data selection means for selecting the first explanatory variable data and the first objective variable data corresponding to the first explanatory variable data based on the data;
Based on a comparison between the prediction data obtained by inputting the first explanatory variable data selected for the prediction model and the first objective variable data corresponding to the first explanatory variable data, the performance estimating means for estimating the prediction performance of the prediction model for the second explanatory variable data;
has,
The structure is as follows.

Furthermore, an estimation method that is one form of the present invention is
A second explanatory variable to which an objective variable is not associated from first explanatory variable data prepared according to a preset prediction model and first objective variable data corresponding to the first explanatory variable data. selecting the first explanatory variable data and the first objective variable data corresponding to the first explanatory variable data based on the data;
Based on a comparison between the prediction data obtained by inputting the first explanatory variable data selected for the prediction model and the first objective variable data corresponding to the first explanatory variable data, the estimating the prediction performance of the prediction model for the second explanatory variable data;
The structure is as follows.

Further, a program that is one form of the present invention is
A second explanatory variable to which an objective variable is not associated from first explanatory variable data prepared according to a preset prediction model and first objective variable data corresponding to the first explanatory variable data. selecting the first explanatory variable data and the first objective variable data corresponding to the first explanatory variable data based on the data;
Based on a comparison between the prediction data obtained by inputting the first explanatory variable data selected for the prediction model and the first objective variable data corresponding to the first explanatory variable data, the estimating the prediction performance of the prediction model for the second explanatory variable data;
have a computer perform a process,
The structure is as follows.

According to the present disclosure, the predictive performance of the predictive model during operation can be estimated without using data of the objective variable.

FIG. 1 is a block diagram illustrating an example of a hardware configuration of an estimation device according to a first embodiment of the present disclosure. FIG. 1 is a block diagram illustrating an example of the configuration of an estimation device according to a first embodiment of the present disclosure. It is a flowchart which shows an example of operation of an estimation device in a 1st embodiment of this indication. It is a block diagram showing an example of composition of an estimation device in a 2nd embodiment of this indication. It is a block diagram showing an example of operation of an estimation device in a 2nd embodiment of this indication. FIG. 7 is a diagram illustrating an example of an estimation process performed by an estimation device according to a second embodiment of the present disclosure.

Preferred embodiments of the present disclosure will be described in detail with reference to the drawings.

<First embodiment>
First, an example of the first embodiment of the present disclosure will be described. FIG. 1 is a block diagram showing an example of the hardware configuration of an estimation device according to the present embodiment, and FIG. 2 is a block diagram showing an example of the configuration of the estimation device. FIG. 3 is a flowchart showing an example of the operation of the estimation device. Note that this embodiment shows an outline of the configuration of an estimation device and an estimation method that will be explained in a second embodiment described later.

First, with reference to FIG. 1, the hardware configuration of the estimation device 100 in this embodiment will be described. The estimation device 100 is constituted by a general information processing device, and is equipped with the following hardware configuration as an example.
・CPU (Central Processing Unit) 101 (arithmetic unit)
・ROM (Read Only Memory) 102 (storage device)
・RAM (Random Access Memory) 103 (storage device)
- Program group 104 loaded into RAM 103
- Storage device 105 that stores the program group 104
- A drive device 106 that reads and writes from and to a storage medium 110 external to the information processing device
-Communication interface 107 that connects to the communication network 111 outside the information processing device
・I/O interface 108 that inputs and outputs data
・Bus 109 connecting each component

Note that FIG. 1 shows an example of the hardware configuration of an information processing device that is the estimation device 100, and the hardware configuration of the information processing device is not limited to the above-mentioned case. For example, the information processing device may be configured from part of the configuration described above, such as not having the drive device 106. In addition, the information processing device uses GPU (Graphic Processing Unit), DSP (Digital Signal Processor), MPU (Micro Processing Unit), FPU (Float) instead of the above-mentioned CPU. ating point number Processing Unit), PPU (Physics Processing Unit) , a TPU (Tensor Processing Unit), a quantum processor, a microcontroller, or a combination thereof.

Then, the estimation device 100 can construct and be equipped with the data selection means 121 and the performance estimation means 122 shown in FIG. 2 by the CPU 101 acquiring the program group 104 and executing the program group 104. Note that the program group 104 is stored in advance in the storage device 105 or ROM 102, for example, and is loaded into the RAM 103 and executed by the CPU 101 as needed. Further, the program group 104 may be supplied to the CPU 101 via the communication network 111, or may be stored in the storage medium 110 in advance, and the drive device 106 may read the program and supply it to the CPU 101. However, the data selection means 121 and the performance estimating means 122 described above may be constructed of a dedicated electronic circuit for realizing such means.

First, the estimation device 100 has a function of acquiring a prediction model, first explanatory variable data, first objective variable data corresponding to the first explanatory variable data, and second explanatory variable data. . The prediction model has a function of outputting a prediction result of the data of the corresponding objective variable when data of the explanatory variable is input by being executed in the estimation device 100. At this time, the output data is called predicted data. The first explanatory variable data and the first objective variable data are data prepared according to the prediction model. For example, the first explanatory variable data and the first objective variable data are used as training data for generating the prediction model by machine learning. It includes first objective variable data, first explanatory variable data and first objective variable data for verifying the prediction model. In addition, the second explanatory variable data is explanatory variable data obtained when operating the prediction model in another task, and there is no objective variable data for such second explanatory variable data, and there is no correspondence. do not have.

Note that the estimation device 100 may acquire the prediction model and each data from outside the estimation device 100. For example, if the estimation device 100 is connected to another device via a network, the prediction model and each data may be acquired from another device that exists on the same network as the estimation device 100. In this case, estimation device 100 may include a communication device for communicating with other devices on the network. Furthermore, if the estimation device 100 includes a storage device (not shown), the prediction model and each data may be acquired from the storage device.

The data selection means 121 selects a partial or All the first explanatory variable data and the corresponding first objective variable data are selected. At this time, the data selection means 121 selects all or part of the data of the first explanatory variable that is more preferable for estimating the prediction performance of the prediction model for the data of the second explanatory variable, based on the content of the data of the second explanatory variable. Select section. More specifically, for the purpose of making the estimated value of the prediction performance index for the data of the second explanatory variable of the prediction model by the performance estimating means 122 (described later) closer to the actual value, the second explanation is All or part of the data of the first explanatory variable is selected based on the content of the data of the variable. For example, in estimating the predictive performance of a predictive model, it is generally desirable to use data on explanatory variables that are not used by the predictive model during training. Therefore, as a method for selecting all or part of the data of the first explanatory variable by the data selection means 121, a method of selecting data of an explanatory variable that is not used for training the prediction model may be used. As another example, in estimating the predictive performance of a prediction model for data on a second explanatory variable, data on the second explanatory variable, data on an explanatory variable that is close in the explanatory variable space, and data on the corresponding objective variable are used. is preferable. Therefore, as a method for selecting all or part of the data of the first explanatory variable by the data selection means 121, the spatial distribution (hereinafter simply referred to as distribution) of the explanatory variable of the data of the second explanatory variable is selected. Alternatively, a method may be used in which the data of the explanatory variables are selected so that the distribution of the data is close to each other. Note that the method for selecting the reference explanatory variable data may be determined more appropriately depending on the method for estimating performance by the performance estimating means 122, which will be described later. The explanatory variable data selected by the data selection means 121 is referred to as reference explanatory variable data. The data selection means 121 also selects objective variable data corresponding to the selected reference explanatory variable data, and this is referred to as reference objective variable data. The reference explanatory variable data and the corresponding objective variable data are collectively referred to as reference data.

The performance estimating means 122 performs the following based on the comparison between the predicted data obtained by inputting the data of the standard explanatory variable to the prediction model and the data of the standard objective variable corresponding to the data of the standard explanatory variable. Estimate the prediction performance of the prediction model for the second explanatory variable data. More specifically, for example, the performance estimation means 122 calculates a performance index calculated using prediction data obtained by inputting reference explanatory variable data into a prediction model and reference objective variable data. The value may be used as an estimate of the performance index for the data of the second explanatory variable of the predictive model. At this time, the performance estimating means 122 further takes into consideration the difference in trends between the data of the second explanatory variable and the data of the standard explanatory variable, so that the larger the difference in trends, the worse the estimated value of the performance index becomes. It may be adjusted to Here, the difference in tendency is information based on the result of comparing the contents of two data, for example, the difference in the contents of two data, that is, the numerical non-negative value that represents the extent to which the contents of two data differ. is the value of As an example of the difference in trends, a distance or an index defined between two data distributions may be used.

The performance index used by the performance estimating means 122 quantitatively evaluates the goodness or badness of the prediction performance of the prediction model. The actual value of the performance index is calculated based on a comparison between the predicted data obtained by inputting the data of the second explanatory variable into the prediction model and the data of the objective variable corresponding to the data of the second explanatory variable. Therefore, if data on the objective variable is not available, it is not possible to calculate the actual value of the performance index. Specific examples of performance indicators include mean absolute error, mean square error, coefficient of determination, etc. in the case of regression, and accuracy, F1 score, cross entropy, etc. in the case of discrimination. Of course, the performance indicators that can be used by the performance estimating means 122 are not limited to these, and any evaluation index can be used as the estimation target.

Then, the estimation device 100 configured as described above executes the estimation method shown in the flowchart of FIG. 3 by the functions of the data selection means 121 and performance estimation means 122 described above.

As shown in FIG. 3, the estimation device 100
A second explanatory variable to which an objective variable is not associated from first explanatory variable data prepared according to a preset prediction model and first objective variable data corresponding to the first explanatory variable data. Based on the data, select the first explanatory variable data and the first objective variable data corresponding to the first explanatory variable data (step S101),
Based on a comparison between the prediction data obtained by inputting the first explanatory variable data selected for the prediction model and the first objective variable data corresponding to the first explanatory variable data, the The prediction performance of the prediction model for the second explanatory variable data is estimated (step S102).

As described above, in the present disclosure, for example, the predictive model in operation and the explanatory variable data used to create the predictive model as the first explanatory variable data (i.e., the explanatory variable data used to train the predictive model) are described. data of the objective variable used to create the predictive model (i.e., the data of the objective variable used to train the predictive model) as the data of the objective variable corresponding to the data of the first explanatory variable. By using the data of the explanatory variables used in the operation of the prediction model as the data of the second explanatory variable (the data of the explanatory variables used in the operation of the prediction model), Predictive performance for data can be estimated. That is, the estimation device 100 can estimate the predictive performance of the predictive model during operation using the explanatory variable data during operation, without waiting for the data on the objective variable during operation to be obtained. I can do it.

Note that the configuration shown in FIG. 2 is just an example, and does not necessarily limit the configuration of the estimation device according to this embodiment. For example, some functions of the estimation device 100 may be realized by a plurality of devices working together. As a specific example, a first estimation device that obtains data on a first explanatory variable, data on an objective variable corresponding to the data on the first explanatory variable, and data on a second explanatory variable; It may be configured by a second estimating device that obtains the model. Further, in order to estimate the value of the performance index from multiple angles, the estimation device 100 may include a plurality of data selection means and a plurality of performance estimation means. Further, it may include a device for calculating a final estimated value by the estimating device 100 based on the estimated values by the plurality of performance estimating means. If the estimation device 100 has a function other than estimating the value of the performance index of the prediction model, it may include a device not shown in FIG. 2 .

<Second embodiment>
Next, an example of a second embodiment of the present disclosure will be described with reference to FIGS. 4 to 6. FIG. 4 is a block diagram for explaining an example of the configuration of the estimation device according to the present embodiment, and FIG. 5 is a flowchart for explaining an example of the operation of the estimation device. FIG. 6 is a diagram for explaining the situation when the estimation device is actually operated.

In this embodiment, the data of the explanatory variable used to create the prediction model is used as the data of the first explanatory variable. Further, as the data of the first objective variable corresponding to the data of the first explanatory variable, the data of the objective variable corresponding to the data of the explanatory variable used to create the prediction model is used. Furthermore, as the second explanatory variable data, explanatory variable data during operation of the prediction model is used. However, these are merely examples of the present embodiment as an example of the embodiment of the present disclosure, and do not limit the embodiment itself of the present disclosure. As an example not limited to each data of the present disclosure, for example, a predictive model created using explanatory variable data and objective variable data obtained in a certain task is applied to another task, and the explanatory variable data of another task is There are cases where we estimate the predictive performance of a predictive model for . At this time, as the second explanatory variable data, explanatory variable data of another task can be used instead of the explanatory variable data during operation of the prediction model, and it is possible to apply the estimation device according to the present disclosure. . In addition, data other than the data used to create the prediction model may be used as the data of the first explanatory variable and the data of the first objective variable corresponding to the data of the first explanatory variable. It is also possible to use explanatory variable data during operation of a prediction model for which objective variable data has already been obtained, and corresponding objective variable data.

The estimation device 1 in this embodiment is configured with one or more information processing devices including a calculation device and a storage device. As shown in FIG. 4, the estimation device 1 includes a second data acquisition section 10, an output section 20, a control section 30, and a storage section 40. The control unit 30 further includes a reference data selection unit 3 and a performance estimation unit 4. The functions of the second data acquisition unit 10, the output unit 20, the reference data selection unit 3, and the performance estimation unit 4 included in the control unit 30 are performed by the arithmetic unit using programs stored in the storage device to realize each function. This can be achieved by executing. Furthermore, the estimation device 1 includes a storage unit 40 that stores first data 41, a prediction model 42, and parameter information 43 in a storage device. Note that the control unit 30 performs data communication with each of the second data acquisition unit 10, the output unit 20, and the storage unit 40 via a communication network or the like.

Then, the estimation device 1 uses the above-described configurations to obtain the second explanatory variable data acquired by the second data acquisition unit 10 of the prediction model 42 stored in the storage unit 40, that is, the prediction model 42. It has the function of estimating predictive performance for explanatory variable data during operation. The configuration will be explained in detail below.

The second data acquisition unit 10 acquires explanatory variable data during operation of the prediction model as second explanatory variable data. The explanatory variable data during operation (that is, the second explanatory variable data) is data obtained during operation, and the prediction model predicts the corresponding objective variable data. For example, when the user of the estimation device 1 inputs explanatory variable data during operation, the second data acquisition unit 10 may include an interface that accepts user input, and specifically, includes a touch panel, buttons, and voice input. This includes input devices, etc. In addition, if the estimation device 1 itself has a function of collecting data on the second explanatory variable, for example, a sensor device such as a camera, a device that processes the information acquired from the sensor device, and a device that processes the information obtained from the sensor device and This corresponds to devices and the like that store data as explanatory variables in item 2. When acquiring data of the second explanatory variable from a device different from the estimation device 1, a communication device with the other device, etc. corresponds to the case.

The output unit 20 outputs the estimated value of the performance index calculated by the control unit 30. When outputting the estimated value to the user of the estimation device 1, a display for displaying the estimated value, a speaker for outputting sound, etc. are applicable. Furthermore, when the estimated value is used in a device other than the estimation device 1, a communication device or the like with the other device corresponds to the output unit 20. Furthermore, the estimation device 1 may have an alert function that notifies the deterioration of the prediction performance when the estimated value falls below a certain value, in which case the device that notifies the deterioration of the prediction performance outputs the This corresponds to Section 20.

The configuration of the storage unit 40 will be explained. As shown in FIG. 4, the storage unit 40 stores first data 41, a prediction model 42, and parameter information 43 in advance. The storage unit 40 transmits the first data 41, the prediction model 42, and the parameter information 43 to the control unit 30 as necessary.

The first data 41 includes explanatory variable data (first explanatory variable data) used to create the predictive model 42 and target variable data (first explanatory variable data) corresponding to the explanatory variable data used to create the predictive model 42. objective variable data). This first data 41 includes training data used for training the prediction model 42 (that is, explanatory variable data used for training the prediction model 42 and objective variable data corresponding to the explanatory variable data), and the prediction model 42. includes verification data used to verify the generalization performance of (that is, explanatory variable data used to verify the prediction model 42 and objective variable data corresponding to the explanatory variable data).

The prediction model 42 is a prediction model created using the first data 41. Specifically, the prediction model 42 is created by so-called supervised machine learning so that when explanatory variable data of the training data of the first data 41 is input, the data of the objective variable of the training data is predicted. may be done. Moreover, the generalization performance of the prediction model 42 may be verified using verification data of the first data 41. Examples of algorithms that implement the prediction model 42 include linear regression, decision trees, random forests, neural networks, and the like. These are just examples, and the algorithm for realizing the prediction model 42 is not limited as long as it is possible to predict the data of the objective variable based on the data of the explanatory variable.

The parameter information 43 is information on parameters necessary for estimating the value of the performance index. For example, it may include information regarding the performance index to be estimated, a reference data acquisition method described later, and a performance index estimation method.

In this embodiment, the prediction model, the data of the first explanatory variable, and the data of the first objective variable corresponding to the data of the first explanatory variable do not change for each estimation, so they are stored in the storage unit 40. By doing so, the processing required for acquisition can be reduced. On the other hand, the data of the second explanatory variables, that is, the data of the explanatory variables used during operation of the prediction model, change for each estimation. It becomes possible to estimate the value of the performance index for the data.

Next, the configuration of the control section 30 will be explained. As shown in FIG. 4, the control unit 30 includes a reference data selection unit 3 and a performance estimation unit 4. Further, the control section 30 includes a CPU, ROM, RAM, etc. (not shown), and performs various controls and calculations on the reference data selection section 3 and the performance estimating section 4.

The reference data selection unit 3 (data selection means) selects the second explanatory variable data acquired by the second data acquisition unit 10, the prediction model 42 acquired from the storage unit 40, the first data 41, and the parameter information 43. Based on this, all or part of the explanatory variable data of the first data 41 is selected. At this time, the reference data selection unit 3 also selects the data of the objective variable corresponding to the data of the explanatory variable of the selected first data 41. Here, the selected explanatory variable data is referred to as reference explanatory variable data. Moreover, the data of the objective variable of the first data 41 corresponding to the data of the standard explanatory variable is referred to as the data of the standard objective variable. The data of the standard explanatory variables and the data of the standard objective variables are collectively referred to as standard data.

As a specific example of a method for selecting reference data by the reference data selection unit 3, a method may be used in which only verification data of the first data 41 is used as reference data. Performance index estimation formula (1), which will be described later, uses the value of the performance index of the predictive model 42 with respect to the reference data as a standard for the estimated value of the performance index of the predictive model 42 with respect to the data of the second explanatory variable. . By not including the training data of the first data 41 in the reference data and using only the verification data, excessive evaluation of the performance index value due to overfitting with respect to the training data of the prediction model 42 is avoided, and the prediction model 42 The estimated value of the performance index can be calculated more suitably. Of course, all or part of the training data may be included in the reference data, and including the training data to increase the number of samples included in the reference data may be useful for optimally estimating the value of the performance index. be. For example, the reference data selection unit 3 may be unable to select related data for the first explanatory variable, as will be described later, such as when the attributes or characteristics of the data cannot be classified based on the content of the data for the second explanatory variable. In such cases, only the validation data or all data including training data may be selected.

In addition, as another example of the method for selecting reference data by the reference data selection unit 3, all or part of the first data 41 that is highly relevant to the prediction performance of the second explanatory variable data of the prediction model 42 may be selected. A method of selecting data as reference data may also be used. Here, "highly relevant" means that there is a positive correlation between the predictive performance of the predictive model 42 for the data of the second explanatory variable and the predictive performance of the predictive model 42 for the data of the reference explanatory variable. , refers to the fact that the two prediction performances are considered to be comparable. In other words, in this case, as a result of comparing the data of the second explanatory variable and the data of the first explanatory variable in the first data 41, the data of the second explanatory variable is determined based on the preset standard for the second explanatory variable data. This means that the content has a high number of relevant examples. Performance index estimation formula (1), which will be described later, uses the value of the performance index of the predictive model 42 with respect to the reference data as a standard for the estimated value of the performance index of the predictive model 42 with respect to the data of the second explanatory variable. . Therefore, by using some data highly related to the data of the second explanatory variable as reference data, the estimated value of the performance index of the prediction model 42 can be calculated more suitably. When selecting data highly relevant to the data of the second explanatory variable from the first data 41, empirical knowledge regarding the data (generally referred to as domain knowledge) or characteristics of the prediction model 42 may be used. As a specific method to obtain highly relevant data, for example, if the explanatory variable data includes date and time information, the same day of the same month as any sample containing the second explanatory variable data, or There is a method of acquiring data for the same day of the week or the same season from the first data 41. As another example, regarding an explanatory variable that the prediction model 42 particularly emphasizes during prediction, data whose value is close to that of the second explanatory variable may be acquired from the first data 41.

In addition, as another example of the method of selecting reference data by the reference data selection unit 3, the difference between the data of the second explanatory variable and the tendency among the first data 41, that is, the difference in the data content based on the preset standard. A method may be used in which a portion of data with a smaller difference compared to other data is selected as the reference data. Equation (1) for estimating the performance index described below shows that the larger the difference in tendency between the data of the second explanatory variable and the data of the reference explanatory variable, the greater the performance index of the predictive model 42 with respect to the data of the second explanatory variable. Calculate the estimated value so that the value becomes worse. The second explanatory variable data often includes only a limited amount of data and a limited range of data in the explanatory variable space compared to the first data 41. Under these circumstances, if the data of all the explanatory variables of the first data 41 are used as the data of the standard explanatory variable, the difference in tendency between the data of the second explanatory variable and the data of the standard explanatory variable becomes large, The value of the prediction performance index for the data of the second explanatory variable of the prediction model 42 is estimated poorly. However, the explanatory variable data of the first data 41 may include samples that are spatially close to each sample of the second explanatory variable data. In other words, the predictive model 42 has already been trained or verified using data for creating the predictive model 42 (first data 41) using data close to the explanatory variable data (second explanatory variable data) during operation. There are cases where In such a case, the actual value of the performance index of the prediction of the prediction model 42 for the data of the second explanatory variable becomes better. Therefore, by selecting some data that has a small difference in tendency from the data of the second explanatory variable among the explanatory variable data of the first data 41 and using it as the data of the standard explanatory variable, the predictive model 42 It is possible to estimate a value close to the actual value of the prediction performance index for the explanatory variable data. Further, the index of difference in tendency may be calculated using the same index as the index of difference in tendency used in performance index estimation formula (1) described later, or may be calculated using a different index. Further, calculation may be performed by combining a plurality of different indicators.

Here, as an example of a specific method for selecting, from the explanatory variable data of the first data 41, data with a small difference in tendency from the data of the second explanatory variable by the reference data selection unit 3, a selection method using a greedy method is used. May be used. First, one or more samples closest to the second explanatory variable data are obtained from the explanatory variable data of the first data 41 and used as temporary standard explanatory variable data. Next, from each sample of explanatory variable data of the first data 41 that is not included in the explanatory variable data of the temporary standard, the temporary standard is added to the data of the explanatory variable of the temporary standard. One or more samples that better reduce the difference in tendency between the data of the explanatory variable and the data of the second explanatory variable are obtained and added to the data of the temporary reference explanatory variable. The above addition process is repeated, for example, until the number of samples included in the data of the explanatory variable of the temporary standard exceeds a certain number, and finally the data of the explanatory variable of the temporary standard is used as the data of the explanatory variable of the standard. By doing so, it is possible to select data that has a small difference in tendency from the data of the second explanatory variable from the explanatory variable data of the first data 41.

The reference data selection unit 3 may use a selection method that is a combination of two or more examples of the reference data selection methods described above. For example, the reference data selection unit 3 first selects the verification data from the first data 41, then selects some data that is highly related to the data of the second explanatory variable, and then selects the data of the second explanatory variable. You may select some data that has a small difference between the data and the trend. Further, as the reference data selection method, a more suitable selection method may be used in consideration of a specific estimation method by the performance estimating section 4, which will be described later. In other words, if it is empirically or theoretically believed that the estimation method by the performance estimator 4 can more accurately calculate the performance estimated value with a specific reference data selection method, the above-mentioned selection method may be used. May be used.

It should be noted that the difference in trends between the explanatory variable data of the first data 41 and the data of the second explanatory variable is significantly different, and any sample of the data of the second explanatory variable is different from that of the explanatory variable of the first data 41. If the data has low relevance to the sample it contains, no matter how you select all or part of the explanatory variable data of the first data 41, the difference in trend from the second explanatory variable data will be 0. There's nothing wrong with that. Therefore, if the trends of the explanatory variable data of the first data 41 and the data of the second explanatory variable are significantly different, all or part of the explanatory variable data of the first data 41 is selected as described above. Even so, it is possible to estimate the deterioration in the prediction performance of the prediction model 42 for the data of the second explanatory variable.

The performance estimating unit 4 (performance estimating means) performs calculations on the data of the second explanatory variable based on the reference data, the data of the second explanatory variable, and the predictive model 42 and parameter information 43 acquired from the storage unit 40. The estimated value P of the performance index of the prediction model 42 is calculated using, for example, equation (1) described later. The value of the performance index estimated by the calculation is output to the output unit 20.
<Formula for calculating the estimated value of the performance index>
P=B+D×A…(1)
however,
P: estimated value of the performance index value of the predictive model 42 for the data of the second explanatory variable;
B: Value of the performance index of the predictive model for the data of the standard explanatory variable and the data of the standard objective variable,
D: a non-negative value indicating the difference in tendency between the data of the second explanatory variable and the data of the standard explanatory variable;
A: Calculated based on the standard explanatory variable data and the comparison between the predicted data obtained by inputting the standard explanatory variable data into the prediction model 42 and the standard objective variable data. rate of change in the value of the performance index,
It is.

B in equation (1) is the value of the performance index of the predictive model for the reference explanatory variable data and the reference objective variable data. In particular, when there is no difference in the trends between the data of the standard explanatory variable and the data of the second explanatory variable (that is, when the value of D in equation (1) is 0), the estimated value P of the performance index is value. In fact, if there is no difference in the trends between the data of the standard explanatory variable and the data of the second explanatory variable, the data of the second explanatory variable and the data of the standard explanatory variable can be considered the same, so the explanatory variable Unless a change in the relationship between the data of The performance is almost the same. Equation (1) allows estimation of the value of the performance index based on this fact. In other words, Equation (1) calculates the value of the performance index when no concept drift occurs and when the data of the standard explanatory variable can be selected so that it has the same tendency as the data of the second explanatory variable. , can be estimated accurately from the data of the standard explanatory variables and the data of the standard objective variables.

In addition, if there is a difference in tendency between the data of the standard explanatory variable and the data of the second explanatory variable (i.e., if the value of D in equation (1) is greater than 0), equation (1) is replaced by equation (1). ) is the best value, and the performance deterioration is estimated based on the values of D and A in equation (1). In general, the predictive performance of the predictive model will deteriorate for explanatory variable data that has a tendency different from that of the explanatory variable in the training data of the predictive model, but Equation (1) Based on this, it is possible to estimate the deterioration of the prediction performance of the prediction model 42 with respect to the data of the second explanatory variable.

D in equation (1) is a non-negative value that quantitatively indicates the difference in tendency between the data of the second explanatory variable and the data of the reference explanatory variable. D becomes 0 when there is no difference in tendency between the data of the second explanatory variable and the data of the standard explanatory variable, and takes a large value when the difference is large. As a specific example of D, the difference between the distribution of data of the second explanatory variable and the distribution of data of the standard explanatory variable can be calculated, and the calculated value can be used as D. Examples of indicators for measuring differences in data distribution include Kullback-Leibler information amount, Jensen-Shannon information amount, Wasserstein distance, Maximum Mean Discrepancy (hereinafter referred to as MMD), and the like. These indicators are 0 when there is no difference in data distribution, and take on larger values as the difference in data distribution becomes larger. In addition, as another example, statistics such as the mean and variance of the explanatory variables are calculated for each of the data of the second explanatory variable and the data of the standard explanatory variable, and D is calculated based on the difference between them. Good too. As another example, the proportion of samples in which there is no sample of the standard explanatory variable data within a certain distance from each sample among the data of the second explanatory variable may be calculated, and this value may be set as D. .

To calculate the value of D in the above example, all explanatory variables of the second explanatory variable data and the standard explanatory variable data may be used, or only some explanatory variables may be used. good. For example, only some explanatory variables that have a particularly strong influence on predictions made by the prediction model 42 may be used. In addition, to calculate the value of D in the above example, you may use all of the data of the second explanatory variable and the data of the standard explanatory variable, or you may use only some of the data. good. For example, when the data of the second explanatory variable consists of a large number of samples, the value of D may be calculated by sampling the data of the second explanatory variable. Thereby, the time required to calculate D in equation (1) can be shortened, and high-speed estimation becomes possible.

In addition, in calculating the value of D in equation (1), in order to adjust the degree of influence of each explanatory variable on D, it is necessary to compare the data of the second explanatory variable and the data of the standard explanatory variable before calculating D. The value of each explanatory variable may be converted. For example, in order to equalize the degree of influence of each explanatory variable on D, each explanatory variable may be converted by Min-Max normalization or Z-Score normalization. As another example, in order to strengthen the influence on D of some explanatory variables that have a particularly strong influence on the prediction result by the prediction model 42, explanation of the data of the second explanatory variable and the data of the reference explanatory variable For example, the value of the variable may be doubled.

A in formula (1) is calculated based on the data of the standard explanatory variable and the comparison between the predicted data obtained by inputting the data of the standard explanatory variable into the prediction model 42 and the data of the standard objective variable. is the rate of change in the value of the performance index according to D. If the higher the value of the performance index, the better the performance of the prediction model (for example, when using the coefficient of determination, accuracy, F1 score, etc. as the performance index), a negative value, and the lower the value of the performance index, the better the performance of the prediction model. If the predictive model is considered to have good performance (for example, if mean absolute error, mean squared error, cross entropy, etc. are used as performance indicators), a positive value is used. Thereby, it is possible to formulate the deterioration of the prediction performance of the prediction model 42 for the data of the second explanatory variable in accordance with the increase in the value of D in equation (1).

The value of A in formula (1) may be a constant, for example. More specifically, for example, if explanatory variable data and objective variable data corresponding to the explanatory variable data can be used in addition to the first data, the explanatory variable data is used as the second explanatory variable data. The value of A may be set so that the estimated value of P according to equation (1) when Further, the value of A may be suitably set based on, for example, theoretical analysis or empirical experimental results.

In addition, the value of A in equation (1) is based on the comparison between the standard explanatory variable data, the predicted data obtained by inputting the standard explanatory variable data into the prediction model 42, and the standard objective variable data. It may be calculated based on. As an example of a specific calculation method, the following formula (2) for calculating the predicted performance deterioration rate based on distribution robust optimization may be used. The calculation formula for the prediction performance deterioration rate based on this distribution robust optimization is calculated mathematically from the value of the prediction performance index for the data of the standard explanatory variable of the prediction model 42 and the distribution of the data of the standard explanatory variable. This is a method to calculate Note that when the arithmetic expression according to equation (2) is used, the value of A is positive, and therefore, as an index of prediction performance, an index is used in which the lower the value of the index, the better the performance of the prediction model.

<Formula for predicting performance deterioration rate based on distribution robust optimization>
A=(Σij Kij×Li×Lj)^(1/2)…(2)
however,
A: value of A in formula (1),
Σij: summation symbol when i and j are each changed from 1 to the number of data of the standard explanatory variable,
Kij: the value of the Laplace kernel calculation result of the i-th and j-th explanatory variables of the data of the standard explanatory variables,
Li: Value of the prediction performance index calculated based on the i-th value of the standard objective variable data and the output value when inputting the i-th sample of the standard explanatory variable data into the prediction model. ,
Lj: value of the prediction performance index calculated based on the j-th value of the standard objective variable data and the output value when inputting the j-th sample of the standard explanatory variable data into the prediction model ,
It is.

By using the MMD value based on the Laplace kernel for D in equation (1) and setting the value calculated by equation (2) as A in equation (1), the data of the standard explanatory variable and the second The value of the performance index when the prediction performance is the worst can be estimated based on the distance by MMD from the data of the explanatory variable.

<Explanation of operation>
Next, an example of the overall operation according to this embodiment will be described in detail with reference to the block diagram of FIG. 4 and the flowchart of FIG. 5. FIG. 5 is an example of a flowchart showing the processing procedure of the estimation device 1 in the first embodiment.

First, when the process of the estimation device 1 is started, the second data acquisition unit 10 acquires data of the second explanatory variable (step S1).
Next, the control unit 30 acquires the first data 41, the prediction model 42, and the parameter information 43 from the storage unit 40 (step S2).
The reference data selection unit 3 selects reference data based on the second explanatory variable data, the first data 41, and the parameter information 43 (step S3).
The performance estimating unit 4 calculates the estimated value of the performance index using equation (1) based on the reference data, the data of the second explanatory variable, the prediction model 42, and the parameter information 43 (step S4).
Finally, the estimation device 1 outputs the estimated value of the performance index obtained by the performance estimation section 4 through the output section 20 (step S5).

Note that the flow of operations shown in FIG. 5 is just an example of this embodiment, and does not necessarily limit the flow of operations according to this embodiment. As a specific example, the second explanatory variable acquisition process S1 may be executed after the first data 41, prediction model 42, and parameter information 43 acquisition process (step S2). Moreover, step S2 may be divided into a plurality of steps, and for example, the process of acquiring the prediction model 42 in step S2 may be executed after the process of selecting reference data (step S3).

<Example>
Next, as an example of the estimation device according to an embodiment of the present disclosure, an example of the operation of the system assuming a specific use case will be described.

(Example 1: Forecasting the demand for ice cream in a supermarket one month later)
First, as Example 1, in a physical store of a certain supermarket, the current month, day, day of the week, season, weather, temperature, humidity, number of visitors, number of sales of related products, etc. are used as explanatory variables, and ice cream one month later is An example of predicting the demand quantity using the objective variable will be explained. In this case, the actual number of sales cannot be known until one month has passed and the ice cream is sold. In other words, until one month has passed since the prediction was made, it is not possible to know the value of the objective variable, which is the number of ice creams in demand one month from now. For example, it is not possible to know the mean square error of the predicted values of the prediction model for the most recent week. Therefore, according to the present disclosure, the mean square error of the prediction of the prediction model for the most recent week is estimated.

The prediction model is based on a certain period of time in the past, for example, 3 years of data acquired in the past (i.e. 3 years of daily explanatory variable data and daily target variable data of ice cream 1 month after each day). (including sales numbers) as data for creating a predictive model. Here, the three years' worth of data is referred to as first explanatory variable data and objective variable data corresponding to the first explanatory variable data, and is collectively referred to as first data. The created prediction model and first data are held in a predetermined storage area.

Next, during actual operation, estimate the mean square error of the prediction model for the most recent week. That is, the explanatory variable data for the most recent week is used as the second explanatory variable data. Estimating prediction performance requires a method for selecting reference data, a method for calculating the value of D in equation (1), and a method for calculating the value of A in equation (1).

First, it is assumed that among the first data, data having a month and day that matches the month and day included in the data of the second explanatory variable is selected as reference data. This is based on the idea that the prediction performance of the prediction models in the same month of the same year will give roughly the same results, that is, the prediction performance of the prediction model with respect to the data of the second explanatory variable is highly related. These ideas can be derived, for example, from experience with supermarket sales or from detailed analysis of primary data (ie, domain knowledge). This makes it possible to more appropriately estimate the value of the mean squared error for the most recent week, which is the estimation target, based on data that is highly related to the data of the second explanatory variable.

Next, it is assumed that the MMD of the reference explanatory variable data and the second explanatory variable data is used as the value of D in equation (1). More specifically, MMD is calculated using a Laplace kernel with a sigma (parameter) of 1. Also, before calculating D, Z-Score normalization is performed. The method for calculating the value of D using MMD, together with the method for calculating the value of A in equation (1) using equation (3) described later, becomes a method for estimating the mean square error based on distribution robust optimization, and is suitable for theoretical analysis. The estimation method is based on

The value of A in equation (1) is a value calculated by the following equation (3) by embodying the above equation (2) based on the reference data and the prediction model. Calculating the value of A using equation (3) is an arithmetic method derived from analysis based on distribution robust optimization.

<Calculation formula for rate of change A>
A=(Σij Kij×(Yi-Pi)^2×(Yj-Pj)^2)^(1/2)...(3)
however,
A: value of A in formula (1),
Σij: summation symbol when i and j are each changed from 1 to the number of data of the standard explanatory variable,
Kij: The value of the calculation result of the Laplace kernel with the sigma (parameter) of the i-th and j-th explanatory variables of the Z-Score normalized standard explanatory variable data set to 1,
Yi: i-th value of the data of the standard objective variable,
Yj: j-th value of the standard objective variable data,
Pi: Output value when the i-th sample of data of the standard explanatory variable is input into the prediction model Pj: Output value when the j-th sample of the data of the standard explanatory variable is input into the prediction model,
It is.

By using the method of selecting reference data specified above, the method of calculating the value of D in equation (1), and the method of calculating the value of A in equation (1) using equation (3), the most recent It becomes possible to estimate the mean square error of the prediction model for one week.

FIG. 6 is a schematic diagram of a user interface that shows the estimation result of the mean square error according to the present example to the user in a graph. The graph in FIG. 6 shows the measured value of the mean square error and the predicted value according to the present example as a line graph, with the horizontal axis representing the date and the vertical axis representing the mean square error. In addition, FIG. 6 shows the breakdown of P in equation (1) of the predicted value of the mean square error according to the present example, based on the value of B in the first term on the right side of equation (1) and the value of D×A in the second term. It is divided and shown by a bar graph. Since it takes one month to know the value of the target variable, that is, the number of ice creams sold, the actual value of the mean squared error can be calculated and plotted only for predictions up to one month in advance. On the other hand, the prediction performance estimation according to the present disclosure can be calculated including today's prediction. Also, by displaying the breakdown at the same time, it is possible for the user to more easily understand the estimated value of the predicted performance and the change in the estimated value.

(Example 2: Disease diagnosis prediction based on initial symptoms)
Next, as a second embodiment, an example in which disease diagnosis prediction is performed based on initial symptoms will be described. More specifically, we use symptoms such as today's body temperature, yesterday's body temperature, the day before yesterday's temperature, the presence or absence of a sore throat, the presence or absence of nasal congestion, the presence or absence of a cough, and the presence or absence of malaise to determine the patient's illness, e.g. , cold, influenza, allergic rhinitis, streptococcal infection, acute bronchitis, measles, etc., are used as objective variables and predicted by a prediction model. Determining the actual disease requires diagnosis by a doctor, and it may take time to make the diagnosis, develop the symptoms necessary for diagnosis, and obtain the test results necessary for diagnosis. It is not possible to know the predictive accuracy of the predictive model for, for example, the most recent 30 patients until the diagnostic results for all patients are available. Therefore, the prediction accuracy of the prediction for the most recent 30 people is estimated using the estimation device according to the present disclosure.

First, the predictive model uses symptoms, which are explanatory variable data, and disease judgment results, which are objective variable data, for a certain number of people in the past, for example, 1000 people, as data for creating the predictive model. Suppose it is created with . Here, the data for the past 1000 people are defined as the data of the first explanatory variable and the data of the objective variable corresponding to the data of the first explanatory variable, and are collectively referred to as the first data. The created prediction model and first data are held in a predetermined storage area.

Next, during actual operation, the prediction accuracy of the prediction model for the most recent 30 people is estimated. That is, the explanatory variable data for the most recent 30 people is used as the second explanatory variable data. Estimating prediction accuracy requires a method for selecting reference data, a method for calculating the value of D in equation (1), and a method for calculating the value of A in equation (1).

In this example, it is assumed that all the first data is used as the reference data. In other words, all the first data is selected as the reference data. In this embodiment, the value of D in equation (1) is calculated based on whether the symptom is unknown compared to each symptom of the reference data, as will be described later. Actually known symptoms are all of the first data. Therefore, by selecting all of the first data as reference data, all of the first data can be treated as known symptoms, and more suitable predictions can be made. Estimation of accuracy becomes possible. Note that, depending on the method of calculating the value of D, the number of first data items, the presence or absence of domain knowledge regarding symptoms, the reference data may be selected using a more detailed method, but in this example, all data are selected for the reasons mentioned above. shall be selected.

In addition, the value of D in equation (1) is set within a certain distance from each sample of the Min-Max normalized second explanatory variable data, for example, within 1.0 in Euclidean distance. This is the percentage of samples in which neither sample exists. This is the proportion of samples that have explanatory variables that are unknown to the predictive model, such as those that are not included in the data of the first explanatory variable among the data of the second explanatory variable, that is, the predictive model has neither been trained nor verified. Calculate. If all samples of the data of the second explanatory variable are included in the data of the first explanatory variable, the value of D becomes 0.

Finally, the value of A in equation (1) is a constant of -1. This is an estimation method based on the assumption that, together with the calculation method of D, predictions made by a prediction model for samples on which the prediction model has not been trained or verified will generally be incorrect.

By using the above standard data selection method and setting the values of D and A in equation (1), the accuracy of the prediction model against the data for model creation is set as the upper limit, and the prediction model can be used for samples that have not been trained or verified. Assuming that the predictions made by the prediction model are incorrect, it is possible to estimate the prediction accuracy of the prediction model for the most recent 30 people.

Although the present disclosure has been described above with reference to the above-described embodiments, the present disclosure is not limited to the above-described embodiments. Various changes can be made to the structure and details of the present disclosure that can be understood by those skilled in the art within the scope of the present disclosure. Further, each of the means and functions described above may be executed by an information processing device installed and connected to any location on the network, that is, may be executed by so-called cloud computing.

Note that the above-mentioned programs can be stored and supplied to a computer using various types of non-transitory computer readable media. Non-transitory computer-readable media include various types of tangible storage media. Examples of non-transitory computer-readable media include magnetic recording media (e.g., flexible disks, magnetic tapes, hard disk drives), magneto-optical recording media (e.g., magneto-optical disks), CD-ROMs (Read Only Memory), CD-Rs, CD-R/W, semiconductor memory (eg, mask ROM, PROM (Programmable ROM), EPROM (Erasable PROM), flash ROM, RAM (Random Access Memory)). The program may also be supplied to the computer via various types of transitory computer readable media. Examples of transitory computer-readable media include electrical signals, optical signals, and electromagnetic waves. The temporary computer-readable medium can provide the program to the computer via wired communication channels, such as electrical wires and fiber optics, or wireless communication channels.

<Additional notes>
Part or all of the above embodiments may also be described as in the following additional notes. Hereinafter, the outline of the configuration of the estimation device, estimation method, and program in the present invention will be explained. However, the present invention is not limited to the following configuration.
(Additional note 1)
A second explanatory variable to which an objective variable is not associated from first explanatory variable data prepared according to a preset prediction model and first objective variable data corresponding to the first explanatory variable data. data selection means for selecting the first explanatory variable data and the first objective variable data corresponding to the first explanatory variable data based on the data;
Based on a comparison between the prediction data obtained by inputting the first explanatory variable data selected for the prediction model and the first objective variable data corresponding to the first explanatory variable data, the performance estimating means for estimating the prediction performance of the prediction model for the second explanatory variable data;
An estimation device having
(Additional note 2)
The estimation device according to supplementary note 1,
The performance estimating means is configured to calculate the performance estimation unit based on a comparison between the prediction data and the first objective variable data, and a comparison between the second explanatory variable data and the selected first explanatory variable data. estimating the prediction performance of the prediction model for the second explanatory variable data;
Estimation device.
(Additional note 3)
The estimation device according to appendix 2,
The performance estimating means calculates the prediction performance of the prediction model for the selected first explanatory variable data based on a comparison between the prediction data and the first objective variable data, and the prediction performance of the prediction model for the selected first explanatory variable data and the second explanatory variable data. estimating the prediction performance of the prediction model for the second explanatory variable data based on a difference between the selected first explanatory variable data and the data content based on a preset standard;
Estimation device.
(Additional note 4)
The estimation device according to appendix 2,
The performance estimating means may improve the second explanation of the prediction model as the difference between the second explanatory variable data and the selected first explanatory variable data based on a preset standard of data content increases. Estimating performance for variable data to deteriorate;
Estimation device.
(Appendix 5)
The estimation device according to appendix 4,
The performance estimating means is configured such that a prediction performance value of the prediction model for the selected first explanatory variable data based on a comparison between the prediction data and the first objective variable data is determined based on the second explanatory variable. Estimating that the larger the difference based on a preset standard of data content between the data and the selected first explanatory variable data, the worse the situation becomes;
Estimation device.
(Appendix 6)
The estimation device according to appendix 4,
The performance estimation means calculates a deterioration rate of performance of the prediction model with respect to the second explanatory variable data according to a difference between a distribution of the second explanatory variable data and a distribution of the selected first explanatory variable data. , using a calculation formula for a predicted performance deterioration rate based on distribution robust optimization, which is calculated from the selected first explanatory variable data, a comparison between the predicted data and the selected first objective variable data. and estimating the performance of the prediction model with respect to the second explanatory variable data based on the deterioration rate.
Estimation device.
(Appendix 7)
The estimation device according to supplementary note 1,
The data selection means selects data different from data used during training of the prediction model from among the first explanatory variable data.
Estimation device.
(Appendix 8)
The estimation device according to supplementary note 1,
The data selection means selects, from among the first explanatory variable data, data that has a preset relationship with the second explanatory variable data with respect to the prediction performance of the prediction model.
Estimation device.
(Appendix 9)
The estimation device according to supplementary note 1,
The data selection means selects, from among the first explanatory variable data, the first explanatory variable data that has a smaller difference between the second explanatory variable data and the data content based on a preset standard compared to other explanatory variable data. select,
Estimation device.
(Appendix 10)
A second explanatory variable to which an objective variable is not associated from first explanatory variable data prepared according to a preset prediction model and first objective variable data corresponding to the first explanatory variable data. selecting the first explanatory variable data and the first objective variable data corresponding to the first explanatory variable data based on the data;
Based on a comparison between the prediction data obtained by inputting the first explanatory variable data selected for the prediction model and the first objective variable data corresponding to the first explanatory variable data, the estimating the prediction performance of the prediction model for the second explanatory variable data;
Estimation method.
(Appendix 11)
A second explanatory variable to which an objective variable is not associated from first explanatory variable data prepared according to a preset prediction model and first objective variable data corresponding to the first explanatory variable data. selecting the first explanatory variable data and the first objective variable data corresponding to the first explanatory variable data based on the data;
Based on a comparison between the prediction data obtained by inputting the first explanatory variable data selected for the prediction model and the first objective variable data corresponding to the first explanatory variable data, the estimating the prediction performance of the prediction model for the second explanatory variable data;
A computer-readable storage medium that stores a program for causing a computer to execute processing.

1 Estimation device 3 Reference data selection section 4 Performance estimation section 10 Second data acquisition section 20 Output section 30 Control section 40 Storage section 41 First data 42 Prediction model 43 Parameter information 100 Estimation device 101 CPU
102 ROM
103 RAM
104 Program group 105 Storage device 106 Drive device 107 Communication interface 108 Input/output interface 109 Bus 110 Storage medium 111 Communication network 121 Data selection means 122 Performance estimation means

Claims

A second explanatory variable to which an objective variable is not associated from first explanatory variable data prepared according to a preset prediction model and first objective variable data corresponding to the first explanatory variable data. data selection means for selecting the first explanatory variable data and the first objective variable data corresponding to the first explanatory variable data based on the data;
Based on a comparison between the prediction data obtained by inputting the first explanatory variable data selected for the prediction model and the first objective variable data corresponding to the first explanatory variable data, the performance estimating means for estimating the prediction performance of the prediction model for the second explanatory variable data;
An estimation device having
The estimation device according to claim 1,
The performance estimating means is configured to calculate the performance estimation unit based on a comparison between the prediction data and the first objective variable data, and a comparison between the second explanatory variable data and the selected first explanatory variable data. estimating the prediction performance of the prediction model for the second explanatory variable data;
Estimation device.
The estimation device according to claim 2,
The performance estimating means calculates the prediction performance of the prediction model for the selected first explanatory variable data based on a comparison between the prediction data and the first objective variable data, and the prediction performance of the prediction model for the selected first explanatory variable data and the second explanatory variable data. estimating the prediction performance of the prediction model for the second explanatory variable data based on a difference between the selected first explanatory variable data and the data content based on a preset standard;
Estimation device.
The estimation device according to claim 2,
The performance estimating means may improve the second explanation of the prediction model as the difference between the second explanatory variable data and the selected first explanatory variable data based on a preset standard of data content increases. Estimating performance for variable data to deteriorate;
Estimation device.
The estimation device according to claim 4,
The performance estimating means is configured such that a prediction performance value of the prediction model for the selected first explanatory variable data based on a comparison between the prediction data and the first objective variable data is determined based on the second explanatory variable. Estimating that the larger the difference based on a preset standard of data content between the data and the selected first explanatory variable data, the worse the situation becomes;
Estimation device.
The estimation device according to claim 4,
The performance estimation means calculates a deterioration rate of performance of the prediction model with respect to the second explanatory variable data according to a difference between a distribution of the second explanatory variable data and a distribution of the selected first explanatory variable data. , using a calculation formula for a predicted performance deterioration rate based on distribution robust optimization, which is calculated from the selected first explanatory variable data, a comparison between the predicted data and the selected first objective variable data. and estimating the performance of the prediction model with respect to the second explanatory variable data based on the deterioration rate.
Estimation device.
The estimation device according to claim 1,
The data selection means selects data different from data used during training of the prediction model from among the first explanatory variable data.
Estimation device.
The estimation device according to claim 1,
The data selection means selects, from among the first explanatory variable data, data that has a preset relationship with the second explanatory variable data with respect to the prediction performance of the prediction model.
Estimation device.
The estimation device according to claim 1,
The data selection means selects, from among the first explanatory variable data, the first explanatory variable data that has a smaller difference between the second explanatory variable data and the data content based on a preset standard compared to other explanatory variable data. select,
Estimation device.
A second explanatory variable to which an objective variable is not associated from first explanatory variable data prepared according to a preset prediction model and first objective variable data corresponding to the first explanatory variable data. selecting the first explanatory variable data and the first objective variable data corresponding to the first explanatory variable data based on the data;
Based on a comparison between the prediction data obtained by inputting the first explanatory variable data selected for the prediction model and the first objective variable data corresponding to the first explanatory variable data, the estimating the prediction performance of the prediction model for the second explanatory variable data;
Estimation method.
A second explanatory variable to which an objective variable is not associated from first explanatory variable data prepared according to a preset prediction model and first objective variable data corresponding to the first explanatory variable data. selecting the first explanatory variable data and the first objective variable data corresponding to the first explanatory variable data based on the data;
Based on a comparison between the prediction data obtained by inputting the first explanatory variable data selected for the prediction model and the first objective variable data corresponding to the first explanatory variable data, the estimating the prediction performance of the prediction model for the second explanatory variable data;
A computer-readable storage medium that stores a program for causing a computer to execute processing.