WO2022168163A1

WO2022168163A1 - Generation device, method, and program

Info

Publication number: WO2022168163A1
Application number: PCT/JP2021/003735
Authority: WO
Inventors: 綜太朗前島; 和昭尾花; 啓介角田; 翠児玉; 直樹荒井
Original assignee: 日本電信電話株式会社
Priority date: 2021-02-02
Filing date: 2021-02-02
Publication date: 2022-08-11
Also published as: JPWO2022168163A1

Abstract

An acquisition unit (29) acquires, for each of a plurality of learning models that have performed machine learning using a plurality of types of feature quantities, the accuracy of the result of the prediction by the learning model and the degree of contribution of at least one type of feature quantity specified by a user to the prediction result; and a generation unit (30) generates, from the accuracy and the degree of contribution, an index for selecting a prescribed learning model from among the plurality of learning models.

Description

GENERATING DEVICE, METHOD AND PROGRAM

The disclosed technology relates to a generation device, a generation method, and a generation program.

Machine learning technology that automatically machine-learns input data patterns and creates learning models that predict the classification of video and audio, the transition of time-series data, etc. is becoming widespread. Also, the learning model can be created in various patterns by setting items related to the behavior of the learning model, such as the base structure, parameters, and input data shape. Therefore, when using machine learning technology, it is necessary to select a learning model with an appropriate pattern so that the prediction accuracy satisfies a predetermined standard.

As a technology related to learning model selection, there is a technology that generates various learning models that predict the same objective variable, but with different structures and feature values, and lists the generated learning models in descending order of prediction accuracy. proposed (Non-Patent Document 1).

In machine learning technology, not only the prediction accuracy of the learning model, but also the basis of the prediction may be important. For example, in a task that supports human decision-making, such as diagnosing a disease based on the prediction results of a learning model, it is necessary to look at not only the prediction results of the learning model, but also the grounds for the predictions, to determine the reliability of the prediction results. is judged. Therefore, it is necessary to appropriately select a learning model so that not only prediction accuracy but also prediction grounds satisfy predetermined standards.

In the prior art, learning models are listed based only on the accuracy of prediction results estimated based on predetermined indicators. There is a problem that they are not always listed in the correct order.

The disclosed technology has been made in view of the above points, and aims to generate an index for appropriately selecting a learning model that provides the desired prediction accuracy and basis for prediction.

A first aspect of the present disclosure is a generation device that generates an index for selecting a predetermined learning model from among a plurality of learning models machine-learned using a plurality of types of feature amounts, wherein the plurality of learning an acquisition unit that acquires, for each of the models, the accuracy of a prediction result by a learning model and the degree of contribution of at least one type of feature quantity specified by a user to the prediction result; and a generation unit that generates the

A second aspect of the present disclosure is a generation method for generating an index for selecting a predetermined learning model from among a plurality of learning models machine-learned using a plurality of types of feature quantities, wherein an acquisition unit comprises: For each of the plurality of learning models, the accuracy of the prediction result by the learning model and the degree of contribution of at least one type of feature quantity specified by the user to the prediction result are obtained, and the generation unit obtains the accuracy and the contribution It is a method of generating the index from degrees.

A third aspect of the present disclosure is a generation program for generating an index for selecting a predetermined learning model from among a plurality of learning models machine-learned using a plurality of types of feature quantities, wherein the computer comprises: an acquisition unit that acquires, for each of a plurality of learning models, the accuracy of a prediction result by the learning model and the degree of contribution of at least one type of feature quantity specified by a user to the prediction result; and the accuracy and the degree of contribution. and a program for functioning as a generation unit that generates the index from

According to the disclosed technology, it is possible to generate an index for appropriately selecting a learning model that provides the desired prediction accuracy and grounds for prediction.

It is a figure for explaining the outline of a 1st embodiment. 4 is a schematic diagram for explaining an example of a SHAP value; FIG. 2 is a block diagram showing the hardware configuration of a learning model selection device; FIG. 1 is a block diagram showing an example of a functional configuration of a learning model selection device according to a first embodiment; FIG. FIG. 10 is a sequence diagram showing the flow of learning model selection processing; FIG. 11 is a block diagram showing an example of a functional configuration of a learning model selection device according to a second embodiment; FIG. It is a figure which shows an example of material data. FIG. 4 is a diagram showing an example of a learning pattern set; It is a figure which shows an example of the column produced as the objective variable of learning data. It is a figure which shows an example of the series column produced as an explanatory variable of learning data. FIG. 10 is a diagram for explaining calculation of contribution for each feature identifier; FIG. 10 is a diagram showing an example of a contribution weight vector and an overall contribution weight; It is a figure which shows an example of the calculation result by an acquisition part. 4A and 4B are diagrams illustrating an example of an acquisition result by an acquisition unit and an example of a generation result by a generation unit; FIG. 7 is a flowchart illustrating an example of evaluation data set creation processing; 6 is a flowchart showing an example of learning model evaluation processing; FIG. 10 is a diagram for explaining differences in model evaluation functions due to differences in overall contribution weights; FIG. 10 is a diagram for explaining adjustment of overall contribution weight; FIG. 10 is a diagram showing an example of a prediction result when a learning pattern that maximizes a model evaluation function is selected when W2=1.0; FIG. 10 is a diagram showing an example of a prediction result when a learning pattern that maximizes the model evaluation function is selected when W2=1.5;

An example of an embodiment of the disclosed technology will be described below with reference to the drawings. In each drawing, the same or equivalent components and portions are given the same reference numerals. Also, the dimensional ratios in the drawings are exaggerated for convenience of explanation, and may differ from the actual ratios.

<First Embodiment>
FIG. 1 is a diagram for explaining the outline of the first embodiment. As shown in FIG. 1, input data is input to each of a plurality of learning models (learning models A and B in the example of FIG. 1), prediction processing is performed in each learning model, and output as a prediction result Data is output. The prediction accuracy of the learning model is obtained by comparing the output data, which is the prediction result, with the correct answer of the output data.

Conventionally, learning models are evaluated using this prediction accuracy. On the other hand, as described above, when it is desired to evaluate the learning model by emphasizing not only the prediction accuracy but also the basis of prediction, the desired feature amount contributes to obtaining the prediction result by the learning model. It is conceivable to use A desired feature amount may be selected based on prior knowledge. For example, when trying to predict the probability of developing lung cancer using a learning model, it is known that the probability of developing lung cancer is related to the amount of cigarettes consumed. A value extracted from the amount to be used may be used as a desired feature amount.

Based on the above points, in the first embodiment, in addition to the prediction accuracy, the contribution of the feature amount is also used to generate an index for selecting a learning model. The contribution of feature amount is a value indicating the contribution to the prediction result for each type of feature amount used in the learning model. In other words, it can be said that the greater the influence of a feature value on the prediction result, the greater the contribution of that feature value. Here, the type of feature amount is information for distinguishing each column existing in the learning data used for machine learning of the learning model. That is, the types of feature amounts are different between different columns. However, in the second embodiment, which will be described later, time series data is used as learning data, and columns of learning data include columns obtained by shifting the time series of a certain column. In this case, it is assumed that the original column and the time series-shifted column have the same feature amount type. Also, even if it is a column whose time series has been shifted, if the feature of the original column is the target variable and the feature of the column whose time series is shifted is the explanatory variable, the original column and the time series are shifted. Columns and columns are treated as different types of feature quantities.

An example of the contribution is the SHAP (SHAPley Additive exPlanations, reference 1) value.

Reference 1: Scott M. Lundberg and Su-In Lee, "A Unified Approach to Interpreting Model Predictions", Advances in Neural Information Processing Systems, pp. 4765-4774, 2017.

Fig. 2 shows an example of a schematic diagram of the SHAP value. In the example of FIG. 2 , the horizontal axis represents the magnitude of the SHAP value, and the contribution increases in the positive direction toward the right side, and the contribution increases in the negative direction toward the left side. Also, in the example of FIG. 2, the SHAP value (each point in FIG. 2) for each data of each feature amount is expressed in a histogram format for each type of feature amount. Note that in the example of FIG. 2, even if the types of feature amounts are the same, the feature amounts whose time series are shifted are distinguished from each other. Also, the darker the color density of each point, the larger the value of the feature value indicated by that point.

In the first embodiment, a learning model is selected such that a learning model with a high contribution of a desired type of feature quantity is selected from the contributions of each type of feature quantity, such as the SHAP value. Generate an index for The type of desired feature amount is designated by the user. Then, using the generated index, a learning model is selected such that the prediction accuracy and the degree of contribution of the specified type of feature amount satisfy predetermined criteria. A learning model selection device including a function of generating an index for selecting a predetermined learning model from among a plurality of learning models machine-learned using a plurality of types of feature values as described above will be described below.

FIG. 3 is a block diagram showing the hardware configuration of the learning model selection device 10 according to the first embodiment. As shown in FIG. 3, the learning model selection device 10 includes a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, a storage 14, an input unit 15, a display unit 16, and It has a communication I/F (Interface) 17 . Each component is communicably connected to each other via a bus 19 .

The CPU 11 is a central processing unit that executes various programs and controls each part. That is, the CPU 11 reads a program from the ROM 12 or the storage 14 and executes the program using the RAM 13 as a work area. The CPU 11 performs control of each configuration and various arithmetic processing according to programs stored in the ROM 12 or the storage 14 . In the first embodiment, the ROM 12 or storage 14 stores a learning model selection program for executing learning model selection processing, which will be described later.

The ROM 12 stores various programs and various data. The RAM 13 temporarily stores programs or data as a work area. The storage 14 is composed of storage devices such as HDD (Hard Disk Drive) and SSD (Solid State Drive), and stores various programs including an operating system and various data.

The input unit 15 includes a pointing device such as a mouse and a keyboard, and is used for various inputs.

The display unit 16 is, for example, a liquid crystal display, and displays various information. The display unit 16 may employ a touch panel system and function as the input unit 15 .

The communication I/F 17 is an interface for communicating with other devices outside the learning model selection device 10, and is, for example, a wired communication standard such as Ethernet (registered trademark) or FDDI, or 4G, 5G, or Wi-Fi. A wireless communication standard such as Fi (registered trademark) is used.

Next, the functional configuration of the learning model selection device 10 will be described. FIG. 4 is a block diagram showing an example of the functional configuration of the learning model selection device 10. As shown in FIG. As shown in FIG. 4, the learning model selection device 10 includes, as a functional configuration, a material data collection unit 21, a material data storage unit 22, a learning pattern transmission unit 23, an evaluation data set creation unit 24, an evaluation and data set storage unit 27 . The evaluation data set creation unit 24 further includes a verification data creation unit 25 and a learning model creation unit 26 . The learning model selection device 10 also includes a learning model evaluation unit 28 and a selected learning model storage unit 32 . The learning model evaluation unit 28 further includes an acquisition unit 29 , a generation unit 30 and a selection unit 31 . The acquisition unit 29 and the generation unit 30 are examples of the generation device of technology disclosed herein. Each functional configuration is realized by the CPU 11 reading out a learning model selection program including a generation program stored in the ROM 12 or storage 14, developing it in the RAM 13, and executing it.

The material data collection unit 21 collects material data used as materials for constructing a learning data set used to create a learning model. Material data is data that can be used as objective variables and explanatory variables, and is data that has a unique data identifier for distinguishing each data. The material data collection unit 21 receives an input of a data identifier designated by a user, and collects material data having the received data identifier. The material data collection unit 21 may collect, for example, sensor values output from sensors, data stored in an external or internal storage device, etc., as material data. The material data collection unit 21 stores the collected material data in the material data storage unit 22 .

The learning pattern transmission unit 23 accepts input of a learning pattern set specified by the user, and transmits the accepted learning pattern set to the evaluation data set creation unit 24 .

The behavior of the output data of the learning model changes due to differences in the base structure, parameters, input data shape, etc. Here, a "learning pattern set" is defined as a plurality of learning patterns, which are information settings that affect the behavior of the output data of the learning model, created in advance by the user and put together. Each learning pattern is given an index for identifying each learning pattern. Components of each learning pattern include, for example, a base model identifier, hyperparameters, and a learning data construction method.

The base model identifier is an identifier that identifies the model structure that is the basis of the learning model, and is, for example, the API (Application Programming Interface) name of the learning model. A hyperparameter is a parameter associated with a machine learning method that corresponds to a base model identifier. The value of each hyperparameter may be uniquely determined, or there may be multiple candidates. The learning data construction method is a method of processing material data and constructing learning data including explanatory variables and objective variables to be input to the learning model. As a learning data construction method, a feature identifier, which is an identifier indicating the type of feature quantity for each of the plurality of feature quantities that make up the learning data, a calculation method for material data to obtain the feature quantity for each feature identifier, etc. items are set. As described above, when time-series data is used as learning data, and the columns of the learning data include a column obtained by shifting the time series of a certain column, the original column and the column whose time series is shifted are The types of features are the same. Therefore, the same feature identifier is used for the feature amount of the original column and the feature amount of the column shifted in time series. However, if the feature of the original column is the objective variable and the feature of the column with the shifted time series is the explanatory variable, the feature of the original column and the column with the shifted time series will be different. Feature identifiers are used.

The evaluation data set creation unit 24 uses the verification data created by the verification data creation unit 25 and the learning model created by the learning model creation unit 26 as evaluation data. The evaluation data set creation unit 24 collects the evaluation data for the number of learning patterns, ie, the number of indexes included in the learning pattern set, as an evaluation data set, and stores the evaluation data set in the evaluation data set storage unit 27 .

The verification data creation unit 25 acquires material data from the material data storage unit 22 and creates learning data from the material data according to the learning data construction method included in the learning pattern. Also, the verification data creation unit 25 extracts a part of the learning data as verification data, and outputs the remaining learning data to the learning model creation unit 26 . Verification data is data used for verification of the created learning model, and is data not used for machine learning of the learning model.

The learning model creation unit 26 creates a learning model using the learning data output from the verification data creation unit 25 according to the base model identifier and hyperparameters included in the learning pattern.

The learning model evaluation unit 28 evaluates each of the plurality of learning models stored in the evaluation data set storage unit 27 based on the indices generated by the acquisition unit 29 and the generation unit 30, and the selection unit 31 selects the desired Choose a learning model. The learning model evaluation unit 28 then stores the learning model selected by the selection unit 31 in the selected learning model storage unit 32 .

The acquisition unit 29 acquires the prediction accuracy of each learning model and the degree of contribution of at least one type of feature quantity specified by the user to the prediction result of the learning model. Specifically, the acquisition unit 29 acquires the prediction accuracy based on the error between the output data of the learning model when the explanatory variables included in the verification data are input to the learning model and the objective variable included in the verification data. do. The prediction accuracy may be, for example, a value based on RMSE (Root Mean Square Error). In addition, the acquisition unit 29 acquires the degree of contribution for each type of feature quantity forming the learning data using the verification data. The contribution may be, for example, a value based on the SHAP value. The acquisition unit 29 outputs the prediction accuracy and the degree of contribution acquired for each learning model to the generation unit 30 .

The generation unit 30 uses the prediction accuracy and the degree of contribution output from the acquisition unit 29 to generate an index for selecting a learning model for each learning model. The generation unit 30 may generate the index using only the degree of contribution of the type of feature specified by the user. Further, the generation unit 30 may generate an index using the degree of contribution for each type of feature amount to which a weight specified by the user is added. This makes it possible to generate an index that considers the prediction accuracy and the contribution of the desired feature amount, that is, the grounds for the desired prediction. When adding weight to the degree of contribution, it is possible to further adjust which type of feature amount to generate the index with emphasis on the degree of contribution.

Also, the generation unit 30 may generate the index after matching the scale of the degree of contribution for each type of feature amount. A scale here is a measure of the magnitude of a value. This makes it possible to generate an index capable of fairly evaluating the relative degree of contribution for each type of feature quantity. Also, the generation unit 30 may generate an index by matching the scales of the prediction accuracy and the degree of contribution. As a result, it is possible to generate an index that can fairly evaluate the prediction accuracy and the degree of contribution. Further, the generator 30 may generate an index by adding weight to at least one of the prediction accuracy and the contribution. Thereby, when selecting a learning model, it is possible to adjust which of the prediction accuracy and the degree of contribution is emphasized. As described above, by matching the scales of the prediction accuracy and the degree of contribution and adding a weight to at least one of the prediction accuracy and the degree of contribution, it is possible to determine which of the prediction accuracy and the degree of contribution is more important. , can be adjusted more appropriately. The generation unit 30 outputs the index generated for each learning model to the selection unit 31 .

Based on the index of each learning model output from the generation unit 30, the selection unit 31 selects the learning model with the highest index from the plurality of learning models stored in the evaluation data set storage unit 27. Note that the selection unit 31 may select one or more learning models whose indices are equal to or greater than a predetermined value, or may select learning models whose indices are a predetermined number of higher ranks. When selecting a plurality of learning models, the selection unit 31 may present the selected learning models to the user together with the index, and accept the selection of the learning model to be adopted from the user.

Next, the action of the learning model selection device 10 will be described. FIG. 5 is a sequence diagram showing the flow of learning model selection processing by the learning model selection device 10. As shown in FIG. The learning model selection process is performed by the CPU 11 reading out a learning model selection program including a generating program from the ROM 12 or the storage 14, developing it in the RAM 13, and executing it.

In step S11, the CPU 11, as the material data collection unit 21, receives the input of the data identifier specified by the user, collects the material data having the received data identifier, and saves it in the material data storage unit 22. Next, in step S<b>12 , the CPU 11 serves as the learning pattern transmission unit 23 to receive an input of a learning pattern set designated by the user, and transmits the received learning pattern set to the evaluation data set creation unit 24 .

Next, in step S13, the CPU 11, acting as the evaluation data set creation section 24, acquires material data from the material data storage section 22. Next, in step S<b>14 , the CPU 11 performs evaluation data set creation processing as the evaluation data set creation unit 24 . Specifically, the CPU 11, as the verification data creation unit 25 of the evaluation data set creation unit 24, learns from the material data for each learning pattern included in the learning pattern set according to the learning data construction method included in the learning pattern. Create data. Further, CPU 11 , acting as verification data creation unit 25 , extracts part of the created learning data as verification data, and outputs the remaining learning data to learning model creation unit 26 . Furthermore, the CPU 11, as the learning model creation unit 26 of the evaluation data set creation unit 24, uses the learning data output from the verification data creation unit 25 according to the base model identifier and the hyperparameters included in the learning pattern, and performs learning. Create a model.

Next, in step S15, the CPU 11, as the evaluation data set creation unit 24, uses the verification data created by the verification data creation unit 25 and the learning model created by the learning model creation unit 26 as evaluation data. do. Then, the CPU 11, as the evaluation data set creation unit 24, collects the evaluation data for the number of learning patterns, that is, the number of indexes included in the learning pattern set, as an evaluation data set, and sets the evaluation data set storage unit 27. Save to

Next, in step S16, the CPU 11, acting as the learning model evaluation unit 28, acquires the evaluation data set stored in the evaluation data set storage unit 27. Further, in step S17, the CPU 11, as the learning model evaluation unit 28, receives an input of the type of feature specified by the user ("designation of feature" in FIG. 5). Note that the CPU 11, as the learning model evaluation unit 28, may receive a weight for each type of feature amount with respect to the contribution ("contribution degree weight" in FIG. 5) instead of specifying the feature amount. Furthermore, the CPU 11, as the learning model evaluation unit 28, may receive a weight ("total weight" in FIG. 5) to be given to at least one of the prediction accuracy and the degree of contribution when generating the index.

Next, in step S18, the CPU 11, as the learning model evaluation unit 28, executes learning model evaluation processing. Specifically, the CPU 11, as the acquisition unit 29 of the learning model evaluation unit 28, acquires prediction accuracy using verification data for each learning model included in the evaluation data set. In addition, the CPU 11, as the acquiring unit 29, acquires the degree of contribution for each type of feature amount for each learning model using the verification data. Then, the CPU 11, as the generation unit 30 of the learning model evaluation unit 28, uses the prediction accuracy acquired by the acquisition unit 29 and the contribution degree of the feature amount of the type specified by the user to perform learning for each learning model. Generate metrics for model selection. In addition, when the weight of the degree of contribution is accepted instead of the specification of the feature amount, the CPU 11, as the generation unit 30, adds the weight of the contribution degree to the degree of contribution for each type of feature amount to obtain an index. to generate Further, when the overall weight is accepted, the CPU 11, as the generation unit 30, adds the overall weight to at least one of the prediction accuracy and the degree of contribution to generate an index. Then, the CPU 11, as the selection unit 31 of the learning model evaluation unit 28, selects an index Choose the learning model with the highest .

Next, in step S19, the CPU 11, as the learning model evaluation unit 28, stores the learning model selected by the selection unit 31 ("selected learning model" in FIG. 5) in the selected learning model storage unit 32, and stores the learning model The selection process ends.

It should be noted that the processing executed by the CPU 11 as the acquisition unit 29 and the generation unit 30 in step S18 is an example of the generation processing performed by the CPU 11 executing the generation program included in the learning model selection program. Also, the generation process is an example of a generation method of technology disclosed herein.

Conventionally, in the evaluation of learning models, the magnitude and positive/negative of the contribution of each type of feature quantity to the prediction results are not considered. Therefore, when it is specified that the learning model is evaluated by the two indices of "prediction accuracy" and "contribution of each type of feature amount", it may not be possible to select an appropriate learning model. For example, as in the technology described in Non-Patent Document 1, even if multiple learning models with different structures and feature amounts to be used are listed in descending order of prediction accuracy, the desired prediction accuracy and the basis for prediction are satisfied. Not necessarily listed. This is because the conventional technology selects model candidates based only on the prediction accuracy estimated based on the predetermined index. The degree of contribution of each type of feature amount changes depending on the learning data, the nature of the learning model, the cost function designed after the learning model is selected, and the like. This point is not taken into consideration in the prior art, and learning that satisfies both the high prediction accuracy of the learning model and the distribution of the contribution of the feature value of the model being close to the desired distribution of the contribution of the feature value. Choosing a model is difficult. Therefore, in the conventional technology, it is necessary to manually search for the optimum learning model from the enumerated learning models. In particular, when the number of types of feature amounts increases, the number of candidate learning models also becomes enormous, and there is a problem that the work of learning model selection itself becomes difficult.

On the other hand, the learning model selection device according to the first embodiment generates an index for each of a plurality of learning models based on the prediction accuracy of the learning model and the degree of contribution of at least one type of feature specified by the user. do. In other words, this index takes into consideration both the contribution of the type of feature specified by the user and the prediction accuracy. Then, the learning model selection device uses this index to select a predetermined learning model from among a plurality of learning models machine-learned using a plurality of types of feature quantities. As a result, the learning model selection device according to the first embodiment can appropriately select a learning model from which desired prediction accuracy and grounds for prediction can be obtained.

<Second embodiment>
Next, a second embodiment will be described. 2nd Embodiment describes embodiment which actualized 1st Embodiment more. Specifically, in an air-conditioning unit inside a building that is separated from the outside by a door or the like, the goal is to optimize temperature comfort. This is an example of learning. An environment model that predicts changes in temperature according to the temperature setting of an air conditioner, which is an action of a reinforcement learning agent, is a learning model to be selected. Note that the hardware configuration of the learning model selection device according to the second embodiment is the same as the hardware configuration of the learning model selection device 10 according to the first embodiment shown in FIG. 3, so description thereof will be omitted.

A functional configuration of the learning model selection device 110 according to the second embodiment will be described. FIG. 6 is a block diagram showing an example of the functional configuration of the learning model selection device 110. As shown in FIG. As shown in FIG. 6 , the learning model selection device 110 includes, as a functional configuration, a material data collection unit 121, a material data storage unit 122, a learning pattern transmission unit 123, an evaluation data set creation unit 124, an evaluation and data set storage 127 . Evaluation data set creation unit 124 further includes verification data creation unit 125 and learning model creation unit 126 . Learning model selection device 110 also includes a learning model evaluation unit 128 and a selected learning model storage unit 132 . Learning model evaluation unit 128 further includes acquisition unit 129 , generation unit 130 , and selection unit 131 . The acquisition unit 129 and the generation unit 130 are examples of the generation device of technology disclosed herein. Each functional configuration is realized by the CPU 11 reading out a learning model selection program including a generation program stored in the ROM 12 or storage 14, developing it in the RAM 13, and executing it.

Note that the functional configuration of the learning model selection device 110 according to the second embodiment and the functional configuration of the learning model selection device 10 according to the first embodiment have the same functional configuration with the same last two digits of the code. A detailed description of the contents is omitted.

The material data collection unit 121 collects material data related to air conditioning control. For example, the material data collection unit 121 collects each material data having data identifiers represented by room temperature, outside temperature, people flow, air conditioning settings, and open flags. The room temperature is the temperature measured by the air-conditioning unit. The outside air temperature is the temperature measured outdoors. People flow is the unique number of people present in the air conditioning utilization department. The unique number of people is the number of people who exist in the air conditioning usage area per unit time. ). The unit time may be, for example, a data sampling interval. The air conditioning setting value is the temperature setting value of the air conditioner that exists in the air conditioning utilization unit. The open flag is a flag that indicates whether or not a person can enter or leave the building containing the air-conditioning section. For example, "1" may be set when it is possible to enter/exit the building, and "0" may be set when it is impossible to enter/exit the building.

Of the material data, room temperature is material data used as objective variables, and outside temperature, people flow, air conditioning set values, and open flags are material data used as explanatory variables. As in the second embodiment, when the use of the learning model is premised on operation as an environment model for reinforcement learning, the explanatory variables include the data itself corresponding to the action of the reinforcement learning agent, or the data corresponding to the action of the reinforcement learning agent. data must be included. This is because the actions of the reinforcement learning agent must change the output of the environment model. Here, explanatory variables using air conditioning set values are essential.

The material data collection unit 121 , for example, room temperature, outside temperature, air conditioning set values, and open flags may be collected from the BEMS (Building and Energy Management System) 201 . Also, the material data collection unit 121 may collect people flow from the people flow detection sensor 202 installed in the air conditioning utilization unit. All of these material data are time-series data. Specifically, each piece of material data is time-series data in which the date and time of a data sampling point are used as an index, and the index and the data value at the date and time indicated by the index are associated with each other. The material data collection unit 121 stores the collected material data in the material data storage unit 122 . FIG. 7 shows an example of material data stored in the material data storage unit 122. As shown in FIG.

The learning pattern transmission unit 123 accepts input of a learning pattern set specified by the user, and transmits the accepted learning pattern set to the evaluation data set creation unit 124 . FIG. 8 shows an example of a learning pattern set. In the example of FIG. 8, learning pattern (p) is the index of the learning pattern. Here p=1,2,3. Also, hereinafter, a learning pattern with an index of p (p=1, 2, 3) is referred to as a “learning pattern p”. Also, in the example of FIG. 8, the base model identifier of learning pattern 1 is Light GBM (Gradient Boosting Machine, Reference 2). Also, the base model identifier of

learning patterns

2 and 3 is XGBoost (eXtreme Gradient Boosting, reference 3). Learning pattern 2 and learning pattern 3 are partially different in the learning data construction method (details will be described later).

Reference 2: Ke et al., "LightGBM: A Highly Efficient Gradient Boosting Decision Tree", 2017.
Reference 3: Tianqi Chen, Carlos Guestrin, "XGBoost: A Scalable Tree Boosting System", 2016.

Here, in the second embodiment, the material data is time-series data, and can be expressed as table data including multiple columns, as shown in FIG. 7 above. Therefore, it is possible to create a new column as a feature amount that constitutes the learning data by performing calculations between different columns of table data, obtaining the time difference of the same column, or the like. Furthermore, in the case of explanatory variables, a column obtained by shifting the index of the new column created as described above, that is, a column whose time series is shifted can be created as a new column. Hereinafter, a column obtained by shifting the time series in this way will be referred to as a "series column". The method of creating this new column is defined in the learning data construction method of the learning pattern.

In the example of FIG. 8, feature identifiers F _i ^p , calculation formulas E _i ^p , and series parameters S1 _i ^p , S2 _i ^p , and S3 _i ^p are defined as the learning data construction method. The feature identifier F _i ^p is an identifier that indicates the type of feature quantity. i is the index of the feature identifier, where i=0,1,2,3,4,5. Therefore, the feature identifier F _i ^p represents the i-th kind of feature quantity of the learning pattern p. In addition, in the case of series columns, only the time series is shifted, and the feature values of the material data column or the newly created column that is the source of the shift and the series column are essentially the same. The same feature identifier F _i ^p is used for the feature amount of the type. For example, a column A and a new column A' created by shifting the data in column A backward by 30 minutes both have the same feature identifier of A. In the following description, in order to distinguish between series columns, a notation indicating the shifted time, such as F _i ^p −30min, is added to the feature identifier. Note that the feature quantity indicated by the feature identifier F _{i p with i=0 is set as the objective variable, and the feature quantity indicated by the feature identifier F i} ^p _with ⁱ ≧1 is set as the explanatory variable. For example, the room temperature difference (F ₀ ¹ ) on the ^first line in FIG _. It represents that. Further, for example, the room temperature difference (F ₁ ¹ ) in the _second ^row of FIG. It means that

The calculation formula E _i ^p is a formula for calculating a new column as a feature amount from the material data, and is defined using the data identifier of the material data. For example, the formula on the first line in FIG. ₈ is ^a formula E ₀ ¹ for calculating the room temperature difference (F ₀ ¹ ) at time t, and It is specified that the room temperature 60 minutes before is subtracted. Below, <data identifier> or <feature identifier> at time t is expressed as "<data identifier> or <feature identifier>(t)", for example, "room temperature (t)" or "room temperature difference (t)". write.

Series parameters S1 _i ^p , S2 _i ^p , and S3 _i ^p are parameters for creating a series column based on new column X created by calculation formula E _i ^p . S1 _i ^p is the number of sequences, S2 _i ^p is the start point, and S3 _i ^p is the end point. Specifically, it represents that the section from S2 _i ^p to S3 _i ^p is divided by the number of S1 _i ^p at equal intervals, and a sequence column is created by shifting X to each time point. For the feature identifier F ₀ ^p , ie for the target variable, no series parameters are defined since there is no need to create series columns.

The verification data creation unit 125 acquires material data from the material data storage unit 122 and creates learning data from the material data according to the learning data construction method included in the learning pattern. A specific example of learning data creation will be described using the example of the learning pattern set shown in FIG. The verification data creation unit 125 calculates the room temperature difference (t) indicated by the feature identifier F ₀ ¹ by E ₀ ¹ = room temperature (t)−room temperature (t−60min). For example, the verification data creation unit 125 determines [F ₀ ¹ at 2020-01-01 09:00:00]=[Room temperature at 2020-01-01 09:00:00]-[2020-01-01 08: room temperature at 00:00]. FIG. 9 shows an example of the room temperature difference (t) indicated by the feature identifier F ₀ ¹ created as the objective variable of the learning data.

Further, for example, the verification data creation unit 125 uses the calculation formula E ₁ ¹ and the sequence parameters S1 ₁ ¹ /S2 ₁ ¹ /S3 ₁ ¹ =6/−60 min/−360 min to calculate the feature identifier F ₁ ¹ Calculate the series column of the room temperature difference (t) denoted by . Specifically, the verification data creation unit 125 shifts F ₁ ¹ calculated by the formula E ₁ ¹ = room temperature (t)−room temperature (t−60 min) by −60 min to obtain F ₁ ¹ −60 min, Calculate as room temperature (t-60 min) - room temperature (t-120 min). For example, the verification data creation unit 125 calculates [F ₁ ¹ -60 min at 2020-01-01 09:00:00]=[Room temperature at 2020-01-01 08:00:00]-[2020-01-01 Room temperature at 07:00:00]. FIG. 10 shows an example of series columns of feature identifiers F ₁ ¹ to F ₅ ¹ created as explanatory variables of learning data. In addition, according to the example of FIG. 8, F ₁ ¹ in FIG. 10 is the room temperature difference, F ₂ ¹ is the outside temperature, F ₃ ¹ is the crowd flow difference, F ₄ ¹ is the air conditioning setting difference, and F ₅ ¹ is the open flag. is a feature identifier that indicates

Also, the verification data creation unit 125 extracts part of the learning data as verification data, and outputs the remaining learning data to the learning model creation unit 126 . For example, when one month's worth of material data is obtained, the verification data creating unit 125 creates one month's worth of learning data. Then, the verification data creation unit 125 extracts learning data for a predetermined one week out of one month as verification data.

The learning model creation unit 126 sets hyperparameters in the model structure indicated by the base model identifier included in the learning pattern, inputs explanatory variables of learning data to the learning model in which initial values are set for parameters to be adjusted, and outputs data. get Then, the learning model creation unit 126 executes machine learning of the learning model by updating the parameters so that the output data and the objective variable are closer to each other. Note that when hyperparameters are not uniquely defined and are specified as ranges, the learning model creation unit 126 may create a learning model while searching for hyperparameters that maximize the performance of the learning model.

For each learning pattern, the acquisition unit 129 converts the SHAP values of the same number of elements as the explanatory variables of the verification data (the number of columns of the explanatory variables×the number of indices of the verification data) into absolute values using the verification data. Calculate SHAP _i ^p . Then, as shown in FIG. 11, the average of SHAP _i ^p of each element in the verification data period of all columns belonging to the feature identifier F _i ^p (SHAP _i ^p ) (FIGS. 11, 13 and in the formula Now, calculate ``(overline)'' on ``SHAP _i ^p ''. The acquisition unit 129 calculates scale-converted c _i ^p such that all (SHAP _i ^p ) ̂ falls within 0 to 100 using the following equation (1). In addition, the acquisition unit 129 creates a contribution evaluation vector c ^p (shown in bold in the formula) in which c _i ^p are arranged, as shown in the following formula (2). In other words, c _i ^p corresponds to the degree of contribution for the type of feature amount indicated by the feature identifier F _i ^p .

The acquisition unit 129 also acquires the contribution weight vector W1 (shown in bold in the formula) specified by the user. As shown in FIG. 12, for example, the contribution weight vector W1 is represented by [w ₁ ^p , w ₂ ^p , w ₃ ^p , w ₄ ^p , w ₅ ^p ] ^T (T represents transposition). is a vector representing the weight corresponding to each element of the contribution evaluation vector ^cp . The contribution weight vector W1 is created in advance by the user, and the weight of each element may be set to a value corresponding to the degree of importance to be given when selecting a learning model. For example, as in the second embodiment, in the context of "environment model construction for reinforcement learning", a large weight may be specified for the "feature amount relating to the action of the reinforcement learning agent". Here, "air conditioning set value" corresponds. By using a binary value of 0 or 1 as the weight, it is possible to specify the feature quantity as described in the first embodiment. The acquisition unit 129 acquires the contribution evaluation α ^p by multiplying the contribution evaluation vector c ^p by the contribution weight vector W1 as shown in the following equation (3).

The acquisition unit 129 also calculates the mean square error RMSE ^p using the verification data, and acquires the reciprocal of the RMSE ^p as the accuracy evaluation β ^p , as shown in the following equation (4), for example. The acquiring unit 129 outputs the acquired contribution evaluation α ^p and accuracy evaluation β ^p to the generating unit 130 .

Using the contribution evaluation α ^{p and the accuracy evaluation β p output from the acquisition unit 129, the generating unit 130 matches the scales of the contribution evaluation α p} ^and ^the accuracy evaluation β ^p , for example, according to the following equation (5). Calculate the contribution scaling constant K for

Also, the generator 130 acquires the overall contribution weight W2 as shown in FIG. 12, for example. The overall contribution weight W2 is a value set in advance by the user, and if the prediction accuracy is set to 1, a value may be set according to how much weight is given to the contribution. The generation unit 130 generates the model evaluation function L ^p using the contribution evaluation α ^p , the accuracy evaluation β ^p , the contribution scaling constant K, and the overall contribution weight W2, for example, according to Equation (6) below.

Note that instead of the contribution scaling constant K, a prediction accuracy scaling constant K' obtained by reciprocating the equation (5) may be used. In this case, the β ^p term is multiplied by K′ in equation (6). Further, instead of the overall contribution weight W2, an overall prediction accuracy weight W2', which is a weight according to how much importance is placed on the prediction accuracy when the contribution is set to 1, may be used. In this case, in equation (6), the term β ^p is multiplied by W2′.

FIG. 13 shows an example of (SHAP _i ^p ) −, contribution evaluation vector c ^p , and RMSE ^p calculated by the acquisition unit 129 for each learning pattern. Further, FIG. 14 shows, for each learning pattern, the contribution evaluation α ^p and the accuracy evaluation β ^p obtained by the obtaining unit 129, the contribution scaling constant K calculated by the generating unit 130, and the contribution evaluation α after scale conversion. An example of ^p ×K and a model evaluation function L ^p generated by the generation unit 130 is shown.

The selection unit 131 selects a learning model that maximizes the model evaluation function L ^p generated by the generation unit 130 from a plurality of learning models stored in the evaluation data set storage unit 127 .

Next, the action of the learning model selection device 110 will be described. Also in the second embodiment, as in the first embodiment, the CPU 11 reads out a learning model selection program including a generation program from the ROM 12 or the storage 14, develops it in the RAM 13, and executes it, so that the learning shown in FIG. A model selection process is performed. Here, more detailed processing of each of the evaluation data creation processing in step S14 of FIG. 5 and the learning model evaluation processing in step S18 will be described using the flowcharts shown in FIGS. 15 and 16. FIG.

First, the evaluation data set creation process shown in FIG. 15 will be described.

In step S101, the CPU 11, as the evaluation data set creation unit 124, initializes the learning model set and the verification data set. Specifically, the CPU 11, as the evaluation data set creating unit 124, prepares an empty set for adding the created learning model and an empty set for adding the created verification data. Further, the CPU 11, as the evaluation data set creation unit 124, sets 1 to the variable p indicating the index of the learning pattern.

Next, in step S102, the CPU 11, as the evaluation data set creation unit 124, determines whether or not p is equal to or less than the maximum index value pmax (here, pmax=3). If p≤pmax, the process proceeds to step S103.

In step S103, the CPU 11, acting as the evaluation data set creation unit 124, initializes the learning data set. Specifically, the CPU 11, as the evaluation data set creation unit 124, prepares an empty set for adding the created learning data. Then, the CPU 11, as the verification data creation unit 125, sets 0 to the variable i indicating the index of the feature identifier.

Next, in step S104, the CPU 11, as the verification data creation unit 125, determines whether i is equal to or less than the maximum index value imax (here, imax=5). If i≤imax, the process proceeds to step S105.

In step S105, the CPU 11, as the verification data creation unit 125, acquires the material data used in the calculation formula E _i ^p from the material data storage unit 122, applies the calculation formula E _i ^p to the acquired material data, and creates a new Create column X.

Next, in step S106, the CPU 11, as the verification data creation unit 125, determines whether or not i is 0, that is, whether or not the feature quantity indicated by the feature identifier F _i ^p corresponding to column X is the objective variable. When i=0, the process proceeds to step S107, and the CPU 11, acting as the verification data creation unit 125, adds the column X to the learning data set. On the other hand, if i≧1, the process proceeds to step S108. In step S108, the CPU 11, as the verification data creation unit 125, reads the series parameters S1 _i ^p , S2 _i ^p , and S3 _i ^p , and divides the interval from S2 _i ^p to S3 _i ^p at equal intervals by the number of S1 _i ^p . Split and create series columns with X shifted to each time point. Then, the CPU 11, as the verification data creation unit 125, adds the created series column to the learning data set.

Next, in step S109, the CPU 11, acting as the verification data creation unit 125, increments i by 1, and the process returns to step S104. If i>imax is determined in step S104, the process proceeds to step S110.

In step S110, the CPU 11, as the verification data creation unit 125, extracts part of the learning data included in the learning data set as verification data and adds it to the verification data set. Further, CPU 11 outputs the remaining learning data to learning model creating section 126 as verification data creating section 125 .

Next, in step S111, the CPU 11, as the learning model creation unit 126, acquires the model structure specified by the base model identifier of the learning pattern p by calling an API from the base model identifier, and so on. Set parameters. Then, as the learning model creating unit 126, the CPU 11 uses the remaining learning data output from the verification data creating unit 125 to machine-learn a learning model in which hyperparameters are set in the model structure indicated by the base model identifier. to run. As the learning model creation unit 126, the CPU 11 executes machine learning while evaluating the learning model by grid search cross-validation, for example.

Next, in step S112, the CPU 11, as the learning model creation unit 126, adds the completed learning model to the learning model set. Next, in step S113, the CPU 11, acting as the evaluation data set creating unit 124, increments p by 1, and the process returns to step S102. If p>pmax is determined in step S102, the evaluation data set creation process ends.

The verification data set and the learning model set created by the evaluation data set creation process are stored in the evaluation data set storage unit 127 as evaluation data sets (S17 in FIG. 5).

Next, the learning model evaluation process shown in FIG. 16 will be described.

In step S121, the CPU 11, as the learning model evaluation unit 128, sets 1 to the variable p indicating the index of the learning pattern. Next, in step S122, the CPU 11, as the learning model evaluation unit 128, determines whether or not p is equal to or less than the maximum index value pmax (here, pmax=3). If p≤pmax, the process proceeds to step S123.

In step S123, the CPU 11, as the acquiring unit 129, uses the verification data of the learning pattern p to calculate SHAP _i ^p by converting the SHAP values of the same number of elements as the explanatory variables of the verification data into absolute values. Next, in step S124, the CPU 11, as the acquiring unit 129, sets 1 to the variable i indicating the index of the feature identifier.

Next, in step S125, the CPU 11, as the acquisition unit 129, determines whether or not i is equal to or less than the maximum index value imax (here, imax=5). If i≤imax, the process proceeds to step S126.

In step S126, the CPU 11, as the acquisition unit 129, calculates the average ( ^SHAP ^ip ₎ of _SHAP _ip of each element during the verification data period of all columns belonging to the feature identifier F ^ip . Next, in step S127, the CPU 11, as the acquiring unit 129, increments i by 1, and the process returns to step S125. If i>imax is determined in step S125, the process proceeds to step S128.

In step S128, the CPU 11, as the acquisition unit 129, calculates c _i ^p that is scale-converted so that all (SHAP _i ^p ) are within the range of 0 to 100, for example, according to equation (1). Next, in step S129, the CPU 11, as the acquiring unit 129, creates a contribution evaluation vector c ^p in which c _i ^p are arranged as shown in the equation (2). Further, the CPU 11, as the acquiring unit 129, acquires the contribution weighting vector W1 specified by the user, and multiplies the contribution evaluation vector ^cp by the contribution weighting vector W1 as shown in equation (3). , obtain a contribution estimate α ^p .

Next, in step S130, the CPU 11, as the acquiring unit 129, calculates the mean square error RMSE ^p using the verification data of the learning pattern p, and obtains the reciprocal of the RMSE ^p as shown in the equation (4), for example. Obtained as the accuracy estimate β ^p . Next, in step S131, the CPU 11, as the learning model evaluation unit 128, increments p by 1, and the process returns to step S122. If p>pmax is determined in step S122, the process proceeds to step S132.

In step S132, the CPU 11, as the generation unit 130, uses the contribution evaluation ^αp and the accuracy evaluation ^βp acquired by the acquisition unit 129, for example, by formula (5) to obtain the contribution evaluation ^αp and the accuracy evaluation ^βp Calculate a contribution scaling constant K to match the scale of .

Next, in step S133, the CPU 11, as the generator 130, acquires the overall contribution weight W2 specified by the user. Then, the CPU 11, as the generation unit 130, for each learning pattern, uses the contribution evaluation α ^p , the accuracy evaluation β ^p , the contribution scaling constant K, and the overall contribution weight W2, for example, by formula (6), the model Generate an evaluation function ^Lp . Then, the CPU 11, as the selection unit 131, selects a learning model that maximizes the model evaluation function ^Lp generated by the generation unit 130 from a plurality of learning models stored in the evaluation data set storage unit 127, and performs learning. The model evaluation process ends.

The learning model selected by the learning model evaluation process is stored in the selected learning model storage unit 132 (S19 in FIG. 5).

As described above, similarly to the learning model selection device according to the first embodiment, the learning model selection device according to the second embodiment appropriately selects a learning model that provides the desired prediction accuracy and basis for prediction. can be selected.

In addition, since the scales of the degree of contribution for each type of feature amount calculated for each learning model do not always match, if the scales are used as they are, the learning models cannot be compared fairly. For example, the sum of (SHAP _i ^p ) of each learning pattern does not match, and a learning model with a large sum of (SHAP _i ^p ) tends to be evaluated to have an unreasonably large contribution. In the second embodiment, the contribution of each type of feature quantity is scaled from 0 to 100 as shown in formula (1), for example, so that the total c ^p of each learning pattern becomes 100. It is possible to fairly evaluate the relative degree of contribution for each type of feature quantity.

In addition, in the second embodiment, by using a “contribution weighting vector” that represents individual weights for each type of feature amount, the contribution of a desired feature amount can be emphasized in evaluation. For example, by designing the contribution weighting vector W1 as shown in FIG. 12, it is possible to emphasize the air-conditioning set value or air-conditioning room temperature difference indicated by the fourth feature identifier F ₄ ^p with the largest weight. As a result, as shown in FIGS. 13 and 14, the learning pattern 1 whose c ₄ ^p corresponding to the feature identifier F ₄ ^p is the largest among c _i ^p has the largest contribution evaluation α ^p .

Also, since the scales of contribution and accuracy do not always match, either contribution evaluation or accuracy evaluation may be too large or too small. For example, in the example of FIG. 14, the contribution evaluation α ^p is generally larger than the accuracy evaluation β ^p . Therefore, if the model evaluation function L ^p is generated with K=1, the contribution evaluation becomes too large, and regardless of the user's intention, the difference in accuracy evaluation hardly affects the model evaluation function L ^p . In the second embodiment, for example, as shown in formula (5), by calculating a contribution scaling constant K for matching the scales of the contribution evaluation and the accuracy evaluation and multiplying the contribution evaluation α ^p , As shown in FIG. 14, the scales of the contribution degree evaluation α ^p and the accuracy evaluation β ^p are almost the same, and the evaluation can be performed fairly.

Also, by designing K as in the second embodiment, the user can intuitively design the overall contribution weight W2. For example, if the contribution evaluation and the accuracy evaluation are given equal importance, W2=1, if the contribution evaluation is given importance, W2>1, and if the accuracy evaluation is given importance, W2<1. can be done. For example, in the second embodiment, an example in which ^W2 =1.5 is shown. The model evaluation function ^Lp of the learning pattern 3 with the largest is the largest.

Here, a method for adjusting W2 will be described using a specific example similar to that of the second embodiment. As a premise, it is desirable that the room temperature fluctuates as predicted by the learning model in accordance with the air conditioning settings. Further, after a certain period of time has passed since the air conditioner was started, the room temperature becomes steady due to thermal equilibrium. I want to reproduce that behavior in the prediction of the learning model. Specifically, if the heating is started one hour earlier, the predicted room temperature will also rise one hour earlier. , that is, the behavior converges to the room temperature true value. After the air conditioner is started, the speed at which the predicted room temperature converges to the room temperature true value depends on the contribution of the air conditioning set value, and in the example of the second embodiment, depends on the contribution evaluation. In order to confirm that the air-conditioning set value contributes to room temperature prediction as expected, the behavior of the predicted room temperature is observed when the air-conditioning set value is shifted one hour earlier from the original data.

The procedure for adjusting W2 under the above assumptions is as follows. First, as shown in FIG. 18, FIG. 19 shows a prediction result when learning pattern 3 that maximizes the model evaluation function is selected when W2=1.0. Note that in FIG. 19 and FIG. 20 described later, temperature is the room temperature true value, and shifted_predict is the predicted room temperature when the air conditioning set value is shifted one hour earlier. Also, the air control temperature is the air conditioning set value or the air conditioning set value difference, and the shifted air control temperature is the air conditioning set value shifted one hour earlier or the air conditioning set value difference.

The learning model of learning pattern 3 has the highest accuracy, but even if the air conditioner is started at around 7:00, the predicted room temperature converges to the true room temperature at around 10:00 (broken line circle in FIG. 19). ), deviating from the ideal behavior. In this case, it is considered that the degree of contribution of the air conditioning setting value to the prediction of the learning model is insufficient, so W2 is increased to 1.5 and the learning model is selected again.

As shown in FIG. 18, when W2=1.5, FIG. 20 shows the prediction results when learning pattern 1 that maximizes the model evaluation function is selected. The accuracy of this learning model is slightly lower than that of the learning model of learning pattern 3, but after starting the air conditioner at around 7:00, the predicted room temperature converges to the true room temperature at around 11:00 (Fig. 20 The part circled by the dashed line in the middle) is close to the ideal behavior. Therefore, the contribution of the air conditioning set value is considered sufficient, and the learning model selected with W2=1.5 is adopted. If the time at which the predicted room temperature converges to the room temperature true value is late, such as 12:00, the degree of contribution is considered too high, and W2 is lowered to select the learning model again.

<Modification>
In the second embodiment, the case of handling time-series data such as room temperature and air conditioning set values and regression problems by decision tree models such as LightGBM has been described, but the disclosed technology is applicable to other types of data and problems. is also applicable. In the following, each modified example will be described mainly on the differences from the second embodiment.

The first modification is a stock price prediction regression problem using a deep learning model and stock trading time series data. In this problem, based on the stock trading data of a certain stock, the future "current price" is used as the objective variable. Data identifiers for material data include "current price", "maximum purchase price", "minimum selling price", "maximum selling price number", "maximum buying price number", "total number of buy orders", "total number of sell orders", etc. Used. The base model identifier may be selected from two types, for example, LSTM (Long Short-Term Memory, Reference 4) and QRNN (Quasi-Recurrent Neural Networks, Reference 5). The hyperparameters are the number of hidden layer nodes, the number of steps, the batch size, the dropout rate, and the number of layers for the layer of the base model identifier, the number and total number of nodes for the fully connected layer, the activation function, the number of nodes and the activation function for the layer, and an optimization function. In addition, since the size of transaction-related data varies greatly depending on the brand, normalization processing may be added to all feature identifiers in the calculation formula of the learning data construction method. Also, in the evaluation data set creation unit, the explanatory variables of the learning data become a two-dimensional matrix of [number of data x (number of features x number of steps)], which is set to [number of data x number of steps x number of features] Add processing to convert to a 3D matrix of

Also, as a second modification, there is a problem of classifying subscription status of paid membership services using a machine learning model and time-series data of member information. In this problem, the future "service subscription status (subscribed or unsubscribed)" is predicted based on time-series data such as the customer's usage history for a certain fee-based membership service. Data identifiers of material data include "customer ID", "sex", "service subscription date", "service withdrawal date (Nan value for those who have not withdrawn)", "service usage time of the day", "service subscription state” etc. are used. The indexes of the time-series data may overlap with the same index if the customer IDs are different. In addition, in the learning data constructing method, as a new feature identifier, "number of service usage days in the current month" may be created. The calculation formula for this feature identifier is to divide the data by "customer ID", group the data by the year and month of the index of the time-series data, and count the number of data whose "service usage time of the day" is 0 or more. It may be aggregated. Further, a process of converting the year and month into days may be added to the calculation formula for the "service subscription date". Also, a process of converting the Nan value of the "service withdrawal date" to -1 may be added. Since "customer ID" and "gender" are categorical variables, these formulas may be defined to perform label encoding. In creating series columns using series parameters in the evaluation data set creation unit, data is divided for each “customer ID” before processing.

A third modification is the regression problem of house prices using a machine learning model and house feature data. In this problem, the "price" of the house is regressed from various information of the house. Data identifiers for material data include "housing type (condominium, detached house, etc.)", "prefecture", "municipalities", "minutes on foot to nearest station", "building age", "floor layout (1K, 2LDK), etc. ”, “exclusive area”, “whether renovation”, “price”, etc. are used. In addition, in the calculation formula of the learning data construction method, categorical variables including characters such as "housing type", "prefecture", "city", "floor plan", "renovation", etc. may be specified. Since the material data in this problem is not time-series data, there is no need to define series parameters in the learning data construction method, and processing to create series columns is not performed.

A fourth variation is the problem of classifying iris varieties using a machine learning model and iris flower feature data. In this problem, we classify the 'varieties' of iris flowers based on various characteristic data. "Sepal length", "sepal width", "petal length", "petal width", "cultivar" and the like are used as data identifiers of the material data. The base model identifier may be selected from two types: support vector machine (reference 6) and logistic regression (reference 7). In the case of a support vector machine, the hyperparameters include kernel type, regularization method, evaluation function, whether to solve dual problems, algorithm termination conditions, severity of soft margins, and the like. For logistic regression, it is the regularization method, the strength of the regularization, and so on. In addition, in the learning data construction method, as new feature identifiers, "sepal information" calculated by a calculation formula ("sepal length" x "sepal width") and a calculation formula ("petal length ”×“Width of petal”). In the calculation formula of the learning data constructing method, since "cultivar" is a categorical variable, it may be specified to perform label encoding in the calculation formula. Since the material data in this problem is not time-series data, there is no need to define series parameters in the learning data construction method, and processing to create series columns is not performed.

Reference 4: S. Hochreiter, J. Schmidhuber, "Long short-term memory", Neural Computation 9 (8), pp. 1735-1780, 1997.
Reference 5: J. Bradbury, et al, "Quasi-Recurrent Neural Networks", ICLP, 2016.
Reference 6: Ioannis Tsochantaridis, Thorsten Joachims, Thomas Hofmann, Yasemin Altun, "Large Margin Methods for Structured and Interdependent Output Variables", The Journal of Machine Learning Research 6 (9), pp. 1453-1484, 2005.
Reference 7: D. R. Cox, "The regression analysis of binary sequences (with discussion)", Journal of the Royal Statistical Society, Series B (Methodological), Vol. 20, No. 2, pp. 215-242, 1958.

Note that the learning model selection process executed by the CPU by reading the software (program) in each of the above embodiments may be executed by various processors other than the CPU. In this case, the processor is a PLD (Programmable Logic Device) whose circuit configuration can be changed after manufacturing, such as an FPGA (Field-Programmable Gate Array), and an ASIC (Application Specific Integrated Circuit) to execute specific processing. A dedicated electric circuit or the like, which is a processor having a specially designed circuit configuration, is exemplified. In addition, the learning model selection process may be executed by one of these various processors, or a combination of two or more processors of the same or different type (for example, multiple FPGAs and a combination of a CPU and an FPGA). combination, etc.). More specifically, the hardware structure of these various processors is an electric circuit in which circuit elements such as semiconductor elements are combined.

Also, in each of the above-described embodiments, the mode in which the learning model selection program including the generation processing program is pre-stored (installed) in the ROM 12 or storage 14 has been described, but the present invention is not limited to this. Programs are stored in non-transitory storage media such as CD-ROM (Compact Disk Read Only Memory), DVD-ROM (Digital Versatile Disk Read Only Memory), and USB (Universal Serial Bus) memory. may be provided in the form Alternatively, the program may be downloaded from an external device via a network.

Regarding the above embodiments, the following additional remarks are disclosed.

(Appendix 1)
memory;
at least one processor connected to the memory;
including
The processor
For each of a plurality of learning models machine-learned using a plurality of types of feature quantities, the accuracy of the prediction result by the learning model and the contribution of at least one type of feature quantity specified by the user to the prediction result are obtained. death,
A generation device configured to generate an index for selecting a predetermined learning model from among the plurality of learning models from the accuracy and the degree of contribution.

(Appendix 2)
A non-transitory recording medium storing a program executable by a computer so as to execute the generating process,
The generation process includes
For each of a plurality of learning models machine-learned using a plurality of types of feature quantities, the accuracy of the prediction result by the learning model and the contribution of at least one type of feature quantity specified by the user to the prediction result are obtained. death,
A non-temporary recording medium comprising generating an index for selecting a predetermined learning model from among the plurality of learning models from the accuracy and the contribution.

10, 110 learning model selection device 11 CPU
12 ROMs
13 RAM
14 storage 15 input unit 16 display unit 17 communication I/F
19

buses

21, 121 material

data collection units

22, 122 material

data storage units

23, 123 learning

pattern transmission units

24, 124 evaluation data set creation units 25, 125 verification

data creation units

26, 126 learning

model creation units

27, 127 Evaluation data set

storage units

28 and 128 Learning

model evaluation units

29 and 129

Acquisition units

30 and 130

Generation units

31 and 131

Selection units

32 and 132 Selected learning model storage unit 201 BEMS
202 people flow detection sensor

Claims

A generation device for generating an index for selecting a predetermined learning model from among a plurality of learning models machine-learned using a plurality of types of feature quantities,
an acquisition unit that acquires, for each of the plurality of learning models, the accuracy of a prediction result by the learning model and the degree of contribution of at least one type of feature specified by a user to the prediction result;
a generation unit that generates the index from the accuracy and the contribution;
generator, including
The generation device according to claim 1, wherein the generation unit generates the index using the degree of contribution for each type of the feature amount to which a weight specified by the user is added.
3. The generation device according to claim 2, wherein the generation unit generates the index using the degree of contribution for each type of the feature amount with a scale of magnitude matched.
The generation device according to any one of claims 1 to 3, wherein the generation unit generates the index by matching the scale of the degree of accuracy and the degree of contribution.
The generation device according to any one of claims 1 to 4, wherein the generation unit adds weight to at least one of the accuracy and the degree of contribution to generate the index.
The learning model aims at optimizing the temperature of the air-conditioning unit and uses reinforcement learning to calculate the optimum control method for the temperature setting of the air-conditioning controller. is an environmental model that predicts changes in temperature using
The generation device according to any one of claims 1 to 5, wherein the feature amount is a feature amount related to air conditioning control including the temperature of the air conditioning use unit and the temperature setting value of the air conditioner.
A generation method for generating an index for selecting a predetermined learning model from among a plurality of learning models machine-learned using a plurality of types of feature quantities,
an acquisition unit acquiring, for each of the plurality of learning models, the accuracy of prediction results by the learning models and the contribution of at least one type of feature specified by a user to the prediction results;
A generation method, wherein a generation unit generates the index from the accuracy and the degree of contribution.
A generation program for generating an index for selecting a predetermined learning model from among a plurality of learning models machine-learned using a plurality of types of feature quantities,
the computer,
an acquisition unit that acquires, for each of the plurality of learning models, the accuracy of a prediction result by the learning model and the degree of contribution of at least one type of feature specified by a user to the prediction result;
A generation program for functioning as a generation unit that generates the index from the accuracy and the contribution.