JP2020144493A - Learning model generation support device and learning model generation support method - Google Patents

Abstract

To provide information which supports generation of a learning model.SOLUTION: A learning model generation support device generates, with respect to each of features available for generation of a learning model, a variation detection model for generating a variation score being a value indicative of a degree of variation of time-series data by using respective time-series data in a learning period of used features which were used for generation of an already generated learning model, as learning data, inputs respective new time-series data to respective variation detection models of the plurality of features to obtain individual variation scores being respective variation scores of the plurality of features in a new period, obtains an overall variation score being a value indicative of a degree of variation of the whole of the plurality of features in the new period on the basis of respective variation scores of the plurality of features, and identifies a contributing feature being a feature contributing to the overall variation score on the basis of respective individual variation scores of the plurality of features.SELECTED DRAWING: Figure 1

Description

The present invention relates to a learning model generation support device and a learning model generation support method.

Patent Document 1 describes an update device for a prediction model configured for the purpose of appropriately predicting a predetermined object using a model. The update device has an acquisition unit and an update unit. The acquisition unit acquires information about a prediction model that predicts a predetermined target, and the update unit updates the prediction model acquired by the acquisition unit when a predetermined condition is satisfied. Further, the update unit updates the prediction model by recalculating the weight of the prediction model when the prediction model corresponding to the predetermined union information satisfies the predetermined condition.

Non-Patent Document 1 describes a technique for detecting outliers and change points from a data stream, which is a stochastic model of the data source using an online discount learning algorithm that tracks the data source whose properties change over time. It describes the technique of learning.

Japanese Unexamined Patent Publication No. 2017-151867

A Unifying Framework for Detecting Outliers and Change Points from Time Series, IEEE Transactions on Knowledge and Data Engineering archive Volume 18 Issue 4, April 2006, Pages 482-492

In recent years, the introduction of IoT (Internet of Things) has been promoted at product manufacturing sites for the purpose of improving production efficiency, and application of machine learning to classify non-defective / defective products and predict equipment failures. Utilization of the advanced technology is being promoted. In such machine learning, learning data about various features is acquired from a data acquisition device such as a sensor device installed at a manufacturing site, and a learning model is generated (learned) based on the acquired learning data. Then, by inputting data newly acquired at the manufacturing site into the generated learning model, information such as classification results and prediction results is acquired, and product production is promoted using the acquired information.

By the way, the performance of the learning model (for example, classification accuracy and prediction accuracy) depends on the features used for its generation, and the selection of the features used for learning is an important task for ensuring the performance of the learning model. In addition, the performance of the learning model may deteriorate due to changes in the situation at the site while continuing to use the learning model, so the features used in the learning model should be reviewed as needed, and the learning model should be regenerated (updated) if necessary. There is a need. In addition, since the number of features to be examined at the site where the learning model is used is usually enormous, it is necessary to efficiently and appropriately select the features.

Patent Document 1 describes updating the prediction model by recalculating the weight of the prediction model. However, the update device described in Patent Document 1 only considers the features used in the first generated learning model even when the prediction model is updated. For example, in the generation of the first generated learning model. It is not possible to deal with the case where the performance of the learning model deteriorates due to changes in features other than the features used.

Non-Patent Document 1 describes a technique for detecting changes in the entire feature or changes in individual features by inputting time-series data of a plurality of features. However, the technique described in Non-Patent Document 1 cannot identify individual features that contribute to changes in the features as a whole. For example, in the field where the learning model is used, when updating the learning model, detailed information such as why the learning model needs to be updated and what kind of cause occurred in the field is the reason for updating the learning model. There is a need to grasp, but the technology described in Non-Patent Document 1 cannot meet such a need.

The present invention has been made based on such a background, and an object of the present invention is to provide a learning model generation support device and a learning model generation support method capable of providing information for supporting the generation of a learning model. To do.

One of the present inventions for achieving the above object is an information processing device that supports the generation of a learning model, and is a storage unit that stores time-series data of features that can be used for generating the learning model. For each, a change indicating the degree of change of the time-series data, using each time-series data in the learning period of the usage feature, which is the feature used for generating the generated learning model, as training data. A change detection model generation unit that generates a change detection model that is a machine learning model that generates a score, a data acquisition unit that acquires time-series data for a new period for each of the plurality of features, and each of the plurality of features. The individual change score calculation unit, which obtains the individual change score which is the change score of each of the plurality of features in the new period by inputting the new time series data into the change detection model of the above. The overall change score calculation unit, which obtains the overall change score, which is a value indicating the degree of change of the plurality of features as a whole in the new period, based on the individual change scores of each of the features of the plurality of features. Based on each of the individual change scores, a contribution feature specifying unit for specifying the contribution feature, which is the feature that contributes to the overall change score, is provided.

In addition, the problems disclosed in the present application and the solutions thereof will be clarified by the column of the form for carrying out the invention and the drawings.

According to the present invention, it is possible to provide information that supports the generation of a learning model.

It is a figure which shows the schematic structure of the machine learning system. It is a figure which shows the hardware configuration example of the learning model generation support apparatus. This is an example of training data. This is an example of learning model information. This is an example of change detection model information. This is an example of information related to feature changes. It is a flowchart explaining the learning model generation process. It is a flowchart explaining the change detection model generation process. It is a flowchart explaining the total change score calculation process. It is a figure explaining the calculation method of the total change score. It is a flowchart explaining the support information presentation process. This is an example of a support information presentation screen. It is a figure explaining the generation method of the change detection model by another change detection method. It is a flowchart explaining the change detection model generation process when the change detection model by another change detection method is used. It is a flowchart explaining the overall change score calculation process when the change detection model by another change detection method is used. This is an example of a change detection method selection screen. This is an example of the learning model setting screen. It is a flowchart explaining the time series data transmission control processing.

Hereinafter, embodiments will be described with reference to the drawings. In the following description, the same or similar configurations may be designated by the same reference numerals and duplicate description may be omitted.

In the embodiment shown below, a machine learning system introduced at a product manufacturing site for classifying non-defective products / defective products and predicting device failures will be described as an example. In the machine learning system, a learning model is generated using time-series data about a plurality of features acquired from sensors or the like provided at the manufacturing site as learning data. Here, in order to efficiently generate a learning model while constantly ensuring the performance of such a learning model (classification accuracy, failure prediction accuracy, etc.), the features used for generating the learning model can be easily and appropriately selected. There is a need. Therefore, the learning model generation support device 100 described in the present embodiment provides various information (hereinafter, referred to as “support information”) that supports selection of features used for learning the learning model.

For example, the learning model generation support device 100 uses a predetermined change detection method (“Change Finder”, “One Class”) for a change detection model, which is a machine learning model for detecting changes in each feature.
The individual change score, which is a value indicating the degree of change of each feature, is obtained by using the change detection model generated by SVM (etc.), and the overall change of each feature is calculated based on the obtained individual change score. The overall change score, which is a value indicating the degree, is obtained, and information on the individual change score and the overall change score is provided as support information.

Further, the learning model generation support device 100 provides, for example, information on a feature that contributes to the obtained overall change score (hereinafter, referred to as “contribution feature”) as support information. Further, the learning model generation support device 100 provides, for example, information on features other than the contribution features (hereinafter, referred to as “correlation features”) in which changes in the time-series data have a high correlation with the contribution features as support information. ..

In providing the above support information, the learning model generation support device 100 uses features that can be used for learning as features that are used for learning the learning model and features that are not used for learning the learning model. Classify as a certain unused feature. Then, when the amount of learning data (length (period) of time-series data) used for learning the learning model is sufficient (greater than or equal to a preset threshold value), the learning model generation support device 100 changes individually. When only unused features are used to calculate the score and overall change score, and the amount of training data used for learning the learning model is insufficient (less than the above threshold), the individual change score and overall change score are calculated. Both used and unused features are used for.

FIG. 1 shows a schematic configuration of the machine learning system 1. As shown in the figure, the machine learning system 1 includes a learning model generation support device 100 and a data providing device 3. These devices are communicably connected to each other via the communication network 5. The communication network 5 is, for example, a LAN (Local Area Network), a WAN (Wide Area Network), the Internet, a leased line, or a public communication network (public telephone network, mobile telephone network, etc.).

The data providing device 3 transmits time-series data acquired from various sensors and the like provided at the manufacturing site (environment) to the learning model generation support device 100 at any time via the communication network 5. As the data providing device 3, for example, a device that functions as an edge-side device in edge computing is assumed.

The learning model generation support device 100 generates a learning model using the time-series data sent from the data providing device 3 as learning data, and also provides the above-mentioned support information.

FIG. 2 shows an example of the hardware of the information processing device used to realize the learning model generation support device 100 and the data providing device 3. As shown in the figure, the illustrated information processing device 10 includes a processor 11, a main storage device 12, an auxiliary storage device 13, an input device 14, an output device 15, and a communication device 16. These are communicably connected to each other via a communication means such as a bus (not shown). The information processing device 10 is, for example, a personal computer, an office computer, a mainframe, or the like. The information processing device 10 may be realized by using virtual information processing resources such as a cloud server provided by a cloud system.

The processor 11 includes, for example, a CPU (Central Processing Unit), an MPU (Micro Processing Unit), a GPU (Graphics Processing Unit), an AI (Artificial Intelligence) chip, an FPGA (Field Programmable Gate Array), an ASIC (Application Specific Integrated Circuit), and the like. It is configured using.

The main storage device 12 is a device that stores programs and data, and is, for example, a ROM (Read).
Only Memory), RAM (Random Access Memory), non-volatile memory (NVRAM (Non Volatile RAM)) and the like.

The auxiliary storage device 13 is, for example, a hard disk drive or an SSD (Solid State Drive).
), Optical storage device (CD (Compact Disc), DVD (Digital Versatile Disc), etc.), storage system, IC card, reading / writing device for recording media such as SD card and optical recording medium, storage of cloud server Area etc. Programs and data can be read into the auxiliary storage device 13 via a reading device for a recording medium or a communication device 16. Programs and data stored (stored) in the auxiliary storage device 13 are read into the main storage device 12 at any time.
The input device 14 is an interface that accepts input from the outside, and is, for example, a keyboard, a mouse, a touch panel, a card reader, a voice input device, and the like.

The output device 15 is an interface that outputs various information such as processing progress and processing results. The output device 15 is, for example, a display device (liquid crystal monitor, LCD (Liquid Crystal Display), graphic card, etc.) that visualizes the above-mentioned various information, and a device (voice output device (speaker, etc.)) that visualizes the above-mentioned various information. , A device (printing device, etc.) that converts the above-mentioned various information into characters. In addition, for example, the information processing device 10 may be configured to input and output information to and from another device via the communication device 16. In the following, it is assumed that the output device 15 is a display device, and the term “output” means “display on the display device” in principle.

The communication device 16 is a device that realizes communication with other devices. The communication device 16 is a wired or wireless communication interface that realizes communication with other devices via communication means such as LAN (Local Area Network), WAN (Wide Area Network), and the Internet. , NIC (Network Interface Card), wireless communication module, USB (Universal Serial Bus) module, serial communication module and the like.

The functions of the learning model generation support device 100 and the data providing device 3 are performed by the processor 11 reading and executing a program stored in the main storage device 12, or by hardware (AI chip) constituting these devices. , FPGA, ASIC, etc.).

As shown in FIG. 1, the learning model generation support device 100 has each function of a storage unit 110, a learning data management unit 120, a learning model management unit 130, and a support information output unit 140. In addition to the above configuration, the learning model generation support device 100 may further include, for example, a device driver, an operating system, a file system, a DBMS (DataBase Management System), and the like.

Among the above functions, the storage unit 110 uses the main storage device 12, the auxiliary storage device 13, or an external storage device connected via the communication device 16 as a storage resource to obtain learning data 111 and learning model information 112. , Change detection model information 113, and feature change-related information 114 are stored. The storage unit 110 stores each of the above information as, for example, a table managed by the DBMS or a file managed by the file system.

The learning data management unit 120 has the functions of the data acquisition unit 121 and the data update unit 122. Of these, the data acquisition unit 121 performs various processes related to acquisition (reception) of learning data 111 (time-series data) from the data providing device 3. Further, the data acquisition unit 121 has a function of controlling the provision (transmission) of time series data from the data providing device 3 for each feature by communicating with the data providing device 3. The data update unit 122 performs various processes related to the learning data 111, such as updating the contents of the learning data 111.

The learning model management unit 130 generates (learns) and evaluates a learning model, manages the learning model, and the like. As shown in the figure, the learning model management unit 130 has the functions of the feature selection unit 131, the learning model generation unit 132, the learning model performance verification unit 133, and the learning model information management unit 134.

Of these, the feature selection unit 131 performs processing related to selection of features used for generating a learning model. The learning model generation unit 132 performs processing related to the generation of the learning model information 112. The learning model performance verification unit 133 performs processing related to evaluation of whether or not the performance of the learning model (classification accuracy, failure prediction accuracy, etc.) satisfies a preset standard. The learning model information management unit 134 performs processing related to management of the learning model information 112.

The support information output unit 140 generates the change detection model described above, calculates and outputs the individual change score and the overall change score described above based on the generated change detection model, identifies and outputs the contribution feature and the correlation feature described above, and describes the above. Performs processing such as output of the correlation coefficient. As shown in the figure, the support information output unit 140 includes a change detection model generation unit 141, an individual change score calculation unit 142, an overall change score calculation unit 143, a contribution feature identification unit 144, a correlation feature identification unit 145, and an information output unit. It has each function of 146 and execution control unit 147.

Of these, the change detection model generation unit 141 performs processing related to generation of the change detection model. The individual change score calculation unit 142 performs processing related to the calculation of the individual change score. The overall change score calculation unit 143 performs processing related to the calculation of the overall change score. Contribution feature identification unit 144 performs processing related to identification of contribution features, which are features that contribute to the overall change score. The correlation feature identification unit 145 performs processing related to the identification of the correlation feature, which is a feature similar to the contribution feature. The information output unit 146 performs processing related to generation of feature change-related information 114 (overall change score, contribution feature, correlation feature, correlation coefficient, etc.), which is the support information described above, and presentation of support information. The execution control unit 147 performs execution control for processing related to the calculation of the individual change score and processing related to the calculation of the overall change score.

FIG. 3 shows an example of the learning data 111. As shown in the figure, the learning data 111 includes one or more records having each item of the feature ID 1111 and the learning data 1112. One record of the training data 111 corresponds to one feature. A feature ID, which is an identifier assigned to each feature, is set in the feature ID 1111. The learning data 1112 stores the substance of the learning data (time series data).

FIG. 4 shows an example of the learning model information 112. Information about the learning model is managed in the learning model information 112. As shown in the figure, the learning model information 112 is one or more items having each item of learning model ID 1121, learning model parameter 1122, usage feature 1123, learning data amount 1124, performance verification information 1125, and overall change score 1126. Includes records. Each record of the learning model information 112 is specified by the learning model ID 1121.

Among the above items, the learning model ID 1121, which is an identifier of the learning model, is set in the learning model ID 1121. The parameters of the learning model are set in the learning model parameter 1122. In the usage feature 1123, a feature ID of one or more features (usage features) used for learning the learning model is set. Whether or not the amount of training data used for learning the training model (for example, the length of time series data) is sufficient for the training data amount 1124 (the amount of training data is equal to or more than a preset threshold or less than the threshold). Information indicating (?) Is set. In this example, when the amount of training data is sufficient, "sufficient" is set, and when the amount is not sufficient, "insufficient" is set in the training data amount 1124.

The performance verification information 1125 includes a verification necessity 11251 in which information indicating the necessity of verification of the performance of the learning model in advance is set when calculating the overall change score, and a determination threshold value which is a threshold value used for determining the necessity. Each item of the determination threshold value 11252 to be set is included.

In the overall change score 1126, information on the overall change score (time-series data of the overall change score in this example) or information indicating a storage destination (link destination) of the above information is set.

FIG. 5 shows an example of the change detection model information 113. Information about the change detection model is managed in the change detection model information 113. As shown in the figure, the change detection model information 113 has one item having each item of learning model ID 1131, feature ID 1132, change score type 1133, use / non-use during learning 1134, change detection model ID 1135, and change detection model parameter 1136. Includes the above records. Each record of change detection model information 113 is identified by a combination of learning model ID 1131, feature ID 1132, and change score type 1133.

Among the above items, the learning model ID is set in the learning model ID 1131. A feature ID is set in the feature ID 1132. The change score type 1133 is set to a change score type that is information (“Change Finder”, “One Class SVM”, etc.) indicating the type of change detection method used to generate the change detection model. Information (“used” or “unused” in this example) indicating whether the feature is a used feature or an unused feature is set in the use / non-use 1134 during learning. A change detection model ID, which is an identifier of the change detection model, is set in the change detection model ID 1135. The parameter of the change detection model is set in the change detection model parameter 1136.

FIG. 6 shows an example of the feature change related information 114. The feature change-related information (overall change score, contribution feature, correlation feature, correlation coefficient, etc.) described above is set in the feature change-related information 114. As shown in the figure, the feature change-related information 114 includes items of learning model ID 1141, feature ID 1142, change score type 1143, and related information 1144 (including a plurality of types of related information # 1 to related information #M). Consists of one or more records with. Each record of feature change related information 114 is identified by a combination of learning model ID 1141, feature ID 1142, and change score type 1143. In this example, M types of information can be held as the related information 1144, but it is not always necessary to use all the related information 1144.

Among the above items, the learning model ID is set in the learning model ID 1141. A feature ID is set in the feature ID 1142. A change score type is set for the change score type 1143. Characteristic change-related information is set in the related information 1144.

Subsequently, various processes performed by the learning model generation support device 100 will be described together with a flowchart.

FIG. 7 is a flowchart illustrating a process (hereinafter, referred to as a learning model generation process S700) performed by the learning model generation support device 100 when the learning model is generated. Hereinafter, the learning model generation process S700 will be described with reference to the figure. In addition, in the generation of the learning model, the generated learning model (hereinafter, also referred to as “existing learning model”) uses the learning data (time series data) of a new period acquired from the data providing device 3. Including when it is updated.

As shown in the figure, the learning model generation unit 132 of the learning model management unit 130 determines at any time whether or not the timing for generating the learning model has arrived (S711). When it is determined that the above timing has arrived (S711: YES), the learning model generation unit 132 performs the process from S712. The above timing comes, for example, when the learning model generation support device 100 receives a learning model generation instruction from the user. Further, the above timing comes, for example, when the learning data management unit 120 acquires new learning data (time-series data for a predetermined period, etc.) from the data providing device 3.

In S712, the feature selection unit 131 selects the feature to be used for generating the learning model. The above selection may be performed, for example, by storing the feature ID of the feature to be selected in the learning model generation support device 100 in advance and referring to the feature ID, or by displaying the contents of the learning data 111, for example. You may let the user choose. The user selects, for example, a feature to be used for generating the learning model with reference to the support information presented by the learning model generation support device 100. Further, the feature selection unit 131 may automatically select features based on predetermined information (features used for generating the learning model, history information about the accuracy of the learning model, etc.).

Subsequently, the learning model generation unit 132 generates a learning model based on the learning data 111 of the selected feature (S713). The learning model generation unit 132 generates a learning model using a predetermined machine learning algorithm (decision tree, SVM (Support Vector Machine), neural network, etc.).

Subsequently, the storage unit 110 stores the generated information about the learning model as the learning model information 112 (S714). Specifically, the storage unit 110 uses the generated learning model parameter as the learning model parameter 1122 of the learning model information 112, and the feature ID of the feature used for generating the learning model as the utilization feature 1123, and uses the learning model information 112. Store in. In this way, the feature ID of the feature used to generate the learning model is managed in the learning model information 112 as the usage feature 1123, so that the learning data of each feature is the learning data of the usage feature and the learning of the unused feature. It is classified and managed as data.

FIG. 8 is a flowchart illustrating a process of generating a change detection model of each feature (hereinafter, referred to as a change detection model generation process S800) for the learning model generated in the learning model generation process S700. The change detection model generation process S800 is executed, for example, after the learning model generation process S700 is executed. Hereinafter, the change detection model generation process S800 will be described with reference to the figure.

First, the range of features used by the learning model performance verification unit 133 to generate a change detection model based on the value of the learning data amount 1124 of the learning model information 112 of the learning model generated by the learning model generation process S700 (unused features only). Or both the used feature and the unused feature) (S811 to S813).

When "sufficient" is set in the training data amount 1124 (S811: sufficient), the learning model performance verification unit 133 determines that the range of features used for generating the change detection model is only the training data of unused features. (S812). On the other hand, when "insufficient" is set in the training data amount 1124 (S811: insufficient), the learning model performance verification unit 133 sets the range of features used for generating the change detection model to the learning data of unused features. And the learning data of the usage characteristics are both determined (S813).

The process from S814 is a loop process that is repeated while sequentially selecting the used or unused features in the range determined in S812 or S813.

First, the change detection model generation unit 141 selects one of the features in the determined range that has not been selected by the loop processing (S814).

Subsequently, the change detection model generation unit 141 refers to the selected feature as a period of learning data (time series data) used in the learning model generation process S700 executed in advance (hereinafter, referred to as a “learning period”. ), By inputting the learning data (time series data) about the selected feature into the change detection method (here, it is assumed to be "Change Finder").
A change detection model of the selected feature is generated (S815). Specifically, when the selected feature is an unused feature, the change detection model generation unit 141 inputs the learning data (time series data) of the unused feature in the learning period into the change detection method. Generates a change detection model for the selected feature. When the feature being selected is a usage feature, the change detection model generation unit 141 inputs the learning data (time series data) of the usage feature in the learning period into the change detection method, so that the feature being selected is selected. Generate a change detection model for.

Subsequently, the storage unit 110 stores the generated change detection model information as the change detection model information 113 (S816). Specifically, the storage unit 110 sets the parameters of the generated change detection model in the change detection model parameter 1136 of the change detection model information 113, the feature ID of the selected feature in the feature ID 1132, and the change detection used for generation. The change detection model information 113 in which the method (“Change Finder”) is set in the change score type 1133 is stored.

Subsequently, the individual change score calculation unit 142 obtains the maximum value of the individual change score in the learning period by applying the learning data (time series data) in the learning period to the generated change detection model (S817). .. This maximum value is used to calculate the overall change score.

Subsequently, the storage unit 110 stores the above maximum value as the feature change-related information 114 (S818). Specifically, the storage unit 110 generates the learning model ID of the learning model generated in the learning model generation process S700 executed in advance in the learning model ID 1141 and the feature ID of the selected feature in the feature ID 1142. The change detection method used in (here, "Change Finder") is set to change score type 1143, and the maximum value of the obtained individual change score is set to related information 114.
In 4, the feature change-related information 114 set respectively is stored.

Subsequently, the individual change score calculation unit 142 determines in S817 whether or not the parameters of the change detection model have been updated, and if updated, updates the change detection model information 113 to the updated contents (S819). ). The above update is performed when the change detection method used in the change detection model has a change detection model update function. Since the "Change Finder" taken up in this example has the above-mentioned update function, the change detection model information 113 is updated if the parameters of the change detection model are updated.

Subsequently, the individual change score calculation unit 142 determines whether or not all the features in the range determined in S812 or S813 have been selected in S814 (S820). If all the features have not been selected (S820: NO), the process returns to S814, and the loop process is repeatedly executed for the unselected features. When all the features have been selected (S820: YES), the change detection model generation process S800 ends.

FIG. 9 is a flowchart illustrating a process (hereinafter, referred to as “overall change score calculation process S900”) that is performed at any time for the existing learning model. Hereinafter, the overall change score calculation process S900 will be described with reference to the figure. In the following, one of the learning models (hereinafter, referred to as “target learning model”) will be focused on and described.

First, the execution control unit 147 refers to the verification necessity of the performance verification information 1125 of the target learning model of the learning model information 112, and determines whether or not the performance evaluation of the target learning model is necessary in advance (S911). The content of the performance verification information 1125 is manually set by the user, for example.

When it is determined that the performance evaluation in advance is unnecessary for the target learning model (S911: unnecessary), the execution control unit 147 starts the process from S913. On the other hand, when it is determined that the performance evaluation of the target learning model is necessary (S911: necessary), the execution control unit 147 has an index value (for example, classification accuracy) indicating the performance of the target learning model of the performance verification information 1125 of the target learning model. Whether or not it is less than the determination threshold value is determined at any time, and when it is determined that the index value is less than the determination threshold value (S912: YES), the process from S913 is started. The index value is obtained, for example, based on information provided from the data providing device 3 at any time.

In S913, the support information output unit 140 determines whether or not the amount of learning data used for learning the target learning model is sufficient based on the value of the learning data amount 1124 of the learning model information 112. When the support information output unit 140 determines that the amount of training data used for learning the target learning model is sufficient (S913: sufficient), the individual change score calculation unit 142 calculates the individual change score for each unused feature. (S914). On the other hand, when the support information output unit 140 determines that the amount of training data used for learning the target learning model is not sufficient (S913: insufficient), the individual change score calculation unit 142 determines each unused feature and each usage feature. The individual change score is calculated for both of the above (S915). In any of the processes of S914 and S915, the individual change score is obtained by using the change detection model generated for each feature.

In S916, the support information output unit 140 determines whether or not the parameters of the change detection model have been updated in the processing of S914 or S915, and if updated, updates the change detection model information 113 to the updated contents. To do. The above update is performed when the change detection method used in the change detection model has a change detection model update function. Since the "Change Finder" taken up in this example has the above-mentioned update function, the change detection model information 113 is updated if the parameters of the change detection model are updated.

Subsequently, the individual change score calculation unit 142 obtains the overall change score based on the maximum value obtained in S818 and the individual change score obtained in S914 or S915, and obtains the obtained overall change score as the overall change score of the learning model information 112. It is stored as 1126 (S917).

In the present embodiment, the overall change score calculation process S900 is repeatedly executed every time a predetermined amount of new learning data (time series data) is received from the data providing device 3, and when the overall change score is generated thereby. It is assumed that the series data is stored as the overall change score 1126.

FIG. 10 shows an example of a method of calculating the overall change score when the change detection method is “Change Finder”. As shown in the figure, in the case of this example, the individual change score calculation unit 142 obtains the individual change score of each feature based on the time series data of each feature, and the overall change score calculation unit 143 determines each individual change score. Calculate the overall change score based on this.

Specifically, the learning data (time-series data) acquired by the individual change score calculation unit 142 in a new period after the learning period of the learning data (time-series data) of the existing learning model for each feature. Is applied to each change detection model to obtain the individual change score in the above new period. Then, the overall change score calculation unit 143 compares the individual change score of each feature with the maximum value of each obtained in S817 for each feature, and the total number of features whose individual change score in the new period is equal to or more than the maximum value. Is calculated as the overall change score.

The method of calculating the overall change score is not necessarily limited to the above method. For example, each feature may be weighted and the total value may be used as the overall change score.

Returning to FIG. 9, the contribution feature identification unit 144 identifies the feature whose individual change score in the new period is equal to or higher than the maximum value obtained in S817 as the contribution feature, and stores the information on the identified contribution feature. (S918). More specifically, the contributing feature identification unit 144 sets information indicating that the feature is a contributing feature in the related information 1144 of the feature ID of the contributing feature of the feature change related information 114. In this example, since the overall change score is stored as time-series data as described above, the information indicating the contributing feature is also stored as time-series data.

Subsequently, the correlation feature identification unit 145 obtains the correlation (correlation coefficient) between the contributing feature and other features other than the contributing feature, and the degree of similarity (correlation coefficient) between the two is equal to or higher than a preset threshold value. Other features are specified as correlation features, and information indicating the identified correlation features is stored (S919). More specifically, the correlation feature identification unit 145 sets information indicating that the feature is a correlation feature in the related information 1144 of the feature ID of the contributing feature of the feature change related information 114. In this example, since the overall change score is stored as time-series data as described above, the information indicating the correlation feature is also stored as time-series data.

FIG. 11 is a flowchart illustrating a process in which the information output unit 146 presents support information for the designated learning model (hereinafter, referred to as “support information presentation process S1100”). Hereinafter, the support information presentation process S1100 will be described with reference to the figure.

First, the information output unit 146 accepts the designation of the learning model (learning model ID) from the user (S1111).

Subsequently, the information output unit 146 displays information on the overall change score (time-series data of the overall change score, etc.) for the designated learning model (S1112).

Subsequently, the information output unit 146 displays information on the contributing features (time-series data of the contributing features, etc.) for the designated learning model (S1113).

Subsequently, the information output unit 146 displays information on the correlation feature (time series data of the correlation feature, the correlation coefficient of the correlation feature, etc.) for the designated learning model (S1114).

For example, when the information output unit 146 receives the designation of the period from the user and the support information corresponding to the accepted period (time series data of the overall change score, time series data of the contribution feature, the correlation coefficient, and the correlation feature). Series data, etc.) may be displayed. Further, the information output unit 146 may display a list of contribution features and correlation features to prompt the user to make a selection, and display only information on the contribution features and correlation features selected by the user. In addition, the information output unit 146 preferentially displays the contribution features that have a high degree of contribution to the overall change score (for example, the change score exceeds the maximum value many times) (display by sorting, filtering, etc., displaying up to a predetermined number, etc.) You may try to do it. Further, the information output unit 146 may preferentially display (display by sorting, filtering, etc., display up to a predetermined number of higher ranks, etc.) the correlation features having a high correlation (large correlation coefficient).

FIG. 12 is an example of a screen (hereinafter referred to as “support information presentation screen 1200”) displayed by the information output unit 146 in the support information presentation process S1100 of FIG. As shown in the figure, the support information presentation screen 1200 has a change score type display field 1211, an overall change score display field 1212, a contribution feature display field 1213, and a correlation coefficient and correlation feature display field 1214. ..

Information indicating the type of change score type is displayed in the change score type display field 1211. Time-series data of the overall change score is displayed in the display column 1212 of the overall change score. In the contribution feature display column 1213, time-series data of the contribution feature is displayed. In the correlation coefficient and correlation feature display column 1214, time series data and correlation coefficient of the correlation feature are displayed.

[Other examples of change detection methods]
By the way, in the above, the case where "Change Finder" is used as the change detection method (change score type) is described as an example for generating the change detection model. For example, the change detection model is used by using "One Class SVM" as the change detection method. Can also be generated.

FIG. 13 is a diagram illustrating a method of generating a change detection model when "One Class SVM" is used. The figure is a graph of time series data for a certain feature, the horizontal axis is time, and the vertical axis is the index value (feature amount) of the feature. The learning period shown in the figure is the period of the learning data (time series data) used for learning the existing learning model.

When generating the change detection model, the change detection model generation unit 141 generates learning data to be used for generating the change detection model based on the time series data of the learning period. The example in the figure is the case where the vector size is "4", and by shifting the windows provided along the time axis with this size every unit time, [5, 1, 3, 2 ], [1,3,2,4], [3,2,4,4], [2,4,4,3] and the like can be obtained. The change detection model generation unit 141 generates a change detection model for each feature by applying these vectors obtained for each feature to the learning algorithm of “One Class SVM” as learning data.

When the generated conversion detection model is given a vector based on the new learning data acquired from the data providing device 3, the new training data is given a time series similar to the mode (scheme) of the time series data in the learning period. Outputs information indicating whether or not the aspect (outline form) of the data exists. Therefore, for example, the overall change score calculation unit 143 sets the value based on the above information to "0" when it is similar (for example, the similarity is equal to or higher than a preset threshold value), and dissimilar (the similarity is preset). If it is less than the above threshold value, it is set to "1"), and the total change score is obtained by summing the above values obtained for each feature.

FIG. 14 shows a process of generating a change detection model for the learning model generated by the learning model generation process S700 when “One Class SVM” is used as the change detection method (change score type) (hereinafter, change detection model). It is a flowchart explaining the generation process S1400). The change detection model generation process S1400 is executed, for example, after the learning model generation process S700 is executed. Hereinafter, the change detection model generation process S1400 will be described with reference to the figure.

Since the processes of S141 to S1413 shown in the figure are the same as the processes of S811 to S813 of FIG. 8, the description thereof will be omitted.

The process from S1414 is a loop process that is repeated while sequentially selecting the features in the range determined in S1412 or S1413.

In S1414, the change detection model generation unit 141 selects one of the features in the determined range that has not been selected by the loop processing.

Subsequently, the change detection model generation unit 141 sets the learning data (time series data) of the selected feature in the learning period of the learning data (time series data) in the learning model generation process S700 as “One Class SVM”. A change detection model is generated by applying to (S1415). Specifically, when the selected feature is an unused feature, the change detection model generation unit 141 applies the learning data (time series data) of the unused feature in the learning period to the "One Class SVM". By doing so, a change detection model for the selected feature is generated. When the selected feature is a usage feature, the change detection model generation unit 141 applies the learning data (time series data) of the usage feature in the learning period to the "One Class SVM" to select the feature. Generate a change detection model for features.

Subsequently, the storage unit 110 stores the information of the change detection model generated by the change detection model generation unit 141 as the change detection model information 113 (S1416). Specifically, the storage unit 110 uses the parameter of the change detection model generated by the change detection model generation unit 141 as the change detection model parameter 1136 of the change detection model information 113, and the feature ID of the selected feature as the feature ID 1132. The change detection model information 113 set in each case is stored with the change detection method "One Class SVM" used for generation as the change score type 1133.

In S1420, the individual change score calculation unit 142 determines whether or not all the features in the range determined in S1412 or S1413 have been selected in S1414. If all the features have not been selected (S1420: NO), the process returns to S1414, and the loop process is repeatedly executed for the unselected features. When all the features have been selected (S1420: YES), the change detection model generation process S1400 ends.

FIG. 15 shows a process (hereinafter, referred to as “overall change score calculation process S1500”) performed at any time for each existing learning model when “One Class SVM” is used as the change detection method (change score type). It is a flowchart to explain. Since the basic processing flow is the same as that of the overall change score calculation processing S900 in FIG. 9, the differences will be mainly described below.

Since the processes of S151 to S1513 shown in the figure are the same as the processes of S911 to S913 of FIG. 9, the description thereof will be omitted.

In S1514 or S1515, the individual change score calculation unit 142 describes each unused feature or the individual change score of each used feature together with FIG. 13 using the change point detection model generated by the change detection model generation process S1400 of FIG. It is obtained by the above-mentioned method.

In S1516, the overall change score calculation unit 143 obtains and stores the overall change score by the method described with reference to FIG. 13 based on the obtained individual change score. The method of calculating the overall change score is not necessarily limited. For example, each feature may be weighted and the total value may be used as the overall change score.

In S1517, the contribution feature identification unit 144 identifies a feature in which the obtained change score contributes to the overall change score as a contribution feature. For example, the contribution feature identification unit 144 identifies a feature having an individual change score of "1" (not similar) as a contribution feature.

In S1518, the correlation feature specifying unit 145 obtains and stores the correlation feature and the correlation coefficient in the same manner as in S919.

As described above, even when "One Class SVM" is used as the change detection method (change score type), the support information is displayed by the same method as the support information presentation process S1100 shown in FIG. In addition, another binary classification algorithm similar to "One Class SVM" may be used as a change detection method.

As described in detail above, according to the learning model generation support device 100 of the present embodiment, it is possible to provide information on the overall change score, which is a value indicating the degree of change of the plurality of features as a whole. Information about changes in multiple features as a whole can be obtained. In addition, it is possible to provide information on the contribution feature, which is a feature that contributes to the overall change score, and it is possible to grasp the feature that is a factor that changes the overall change score from the contribution feature. In addition, it is possible to discover features that contribute to improving the performance of the learning model from among unused features that were not used when learning the existing learning model. As described above, according to the learning model generation support device 100 of the present embodiment, the features used for generating the learning model can be efficiently and appropriately specified.

[Specify change detection method]
By the way, since the characteristics of the change detection model differ depending on the change detection method used to generate the change detection model, for example, the user may be able to select the change detection method to be used to generate the change detection model. Further, when presenting the support information, the user is made to select a plurality of change detection methods, and the change detection model generation unit 141 generates a plurality of change detection models based on each of the plurality of change detection methods selected by the user, and a plurality of changes. Support information may be generated and displayed using each of the detection models. By allowing the user to select the change detection method, the user can select an appropriate change detection model according to the situation of the product or the site and obtain useful support information. In addition, when the contribution features and correlation features are preferentially displayed (displayed by sorting, filtering, etc., displaying up to the upper predetermined number, etc.), the priority order is determined according to the selected change detection method, so that the user has a different viewpoint. Support information based on can be obtained.

FIG. 16 shows an example of a screen for allowing the user to specify a change detection method (hereinafter, referred to as “change detection method selection screen 1600”). As shown in the figure, the change detection method selection screen 1600 has a display field 1611 and a selection field 1612 for the name of the change detection method. The user can select the change detection method to be used for generating the change detection model by performing a selection operation on the selection field 1612 of the change detection method to be selected. The user can also select multiple change detection methods at the same time, in which case a change detection model is generated based on each of the selected multiple change detection methods, and support information is generated based on each change detection model. Will be done.

[Learning model settings]
For the setting of the learning model information 112 described above, for example, a predetermined screen (hereinafter, referred to as “learning model setting screen 1700”) is displayed and the setting is accepted from the user.

FIG. 17 shows an example of the learning model setting screen 1700. The learning model setting screen 1700 to be illustrated has a learning model ID setting field 1711, a learning data amount setting field 1712, and a performance verification information setting field 1713. The learning model generation support device 100 sets the setting contents of the learning model ID setting field 1711 in the learning model ID 1121, the setting contents of the learning data amount setting field 1712 in the learning data amount 1124, and the setting contents of the performance verification information setting field 1713. The contents are set in the performance verification information 1125, respectively. The contents set on the learning model setting screen 1700 are reflected in the learning model information 112.

[Time series data transmission control]
For example, when the machine learning system 1 takes an operation form (edge computing, etc.) in which the data providing device 3 is installed on the site (edge) side and the learning model generation support device 100 is installed on the cloud side, communication is performed. In order to reduce the amount of communication in the network 5, the data providing device 3 may restrict the transmission of time-series data used as learning data from the data providing device 3 to the learning model generation support device 100 during normal operation.

Therefore, a process using time-series data of unused features such as the change detection model generation process S800 and the overall change score calculation process S900 (the same applies to the change detection model generation process S1400 and the overall change score calculation process S1500) described above is executed. At that time, it is necessary to release the transmission restriction so that the learning model generation support device 100 can acquire the time series data of the unused features from the data providing device 3. In addition, after the use of the time-series data of the unused features is completed, the transmission of the unused features (excluding the features determined to be the used features as a result of the user's reference to the support information) is restricted again as usual. It is necessary to return to the operational state.

Therefore, for example, the learning data management unit 120 is provided with a function of transmitting a request for transmitting time-series data of a specified feature and a request for stopping transmission of time-series data of a specified feature to the data providing device 3, and has not been provided. Control the transmission of time-series data of usage features.

FIG. 18 shows a process (hereinafter, referred to as “time-series data transmission control process S1800”) performed by the learning data management unit 120 when making a time-series data transmission request or a transmission stop request to the data providing device 3. It is a flowchart to explain. Hereinafter, the time series data transmission control process S1800 will be described with reference to the figure.

First, the learning data management unit 120 transmits a transmission request for time-series data of unused features to the data providing device 3 (S1811).

Upon receiving the transmission request, the data providing device 3 starts transmitting the time-series data of the unused features, and the learning model generation support device 100 starts the time-series data of the unused features sent from the data providing device 3. Starts receiving (S1812).

The learning data management unit 120 executes the processes S181 to S1812 prior to the execution of the change detection model generation process S800 shown in FIG. 8 and the change detection model generation process S1400 shown in FIG. 14, for example.

After that, the learning data management unit 120 starts a process for stopping the reception of time-series data of unused features (excluding features determined to be used features as a result of the user referring to the support information). Whether or not it is determined at any time (S1813). The learning data management unit 120, for example,
When an instruction to stop receiving the time-series data of the unused feature is received from the user, it is determined that the reception of the time-series data of the unused feature is stopped. When the learning data management unit 120 determines that the reception of the time-series data of the unused feature is stopped (S1813: YES), the process proceeds to S1814.

In S1814, the learning data management unit 120 accepts the designation of an unused feature that stops receiving time-series data. The learning data management unit 120 may automatically determine an unused feature that stops receiving time-series data by referring to, for example, the change detection model information 113.

Subsequently, the learning data management unit 120 transmits a request for stopping transmission of the received feature time-series data to the data providing device 3 (S1815). Upon receiving the transmission stop request, the data providing device 3 stops the transmission of time-series data for the specified feature.

Although one embodiment has been described in detail above, the present invention is not limited to the embodiments described above, and includes various modifications and equivalent configurations within the scope of the appended claims. For example, the above-described embodiment has been described in detail in order to explain the present invention in an easy-to-understand manner, and the present invention is not necessarily limited to those having all the described configurations. Further, a part of the configuration of one embodiment may be replaced with the configuration of another embodiment. Further, the configuration of another embodiment may be added to the configuration of one embodiment. Further, a part of the configuration of each embodiment may be added, deleted, or replaced with another configuration.

For example, in the above embodiments, an identifier is used to identify various information, but for example, a UUID (Universally Unique Identifier) may be used as the identifier.

Further, each of the above-described configurations, functions, processing units, processing means, etc. may be realized by hardware by designing a part or all of them by, for example, an integrated circuit, and the processor realizes each function. It may be realized by software by interpreting and executing the program.

Information such as programs, tables, and files that realize each function is recorded in a memory, hard disk, storage device such as SSD (Solid State Drive), or IC (Integrated Circuit) card, SD card, DVD (Digital Versatile Disc). It can be stored in a medium.

The control lines and information lines indicate what is considered necessary for explanation, and do not necessarily indicate all the control lines and information lines necessary for implementation. In practice, it can be considered that almost all configurations are interconnected.

1 Machine learning system, 3 Data providing device, 5 Communication network, 100 Learning model generation support device, 110 Storage unit, 111 Learning data, 112 Learning model information, 113 Change detection model information, 114 Feature change related information, 120 Learning data management Unit, 121 Data acquisition unit, 122 Data update unit, 130 Learning model management unit, 131 Feature selection unit, 132 Learning model generation unit, 133 Learning model performance verification unit, 134 Learning model information management unit, 140 Support information output unit, 141 Change detection model generation unit, 142 individual change score calculation unit, 143 overall change score calculation unit, 144 contribution feature identification unit, 145 correlation feature identification unit, 146 information output unit, 147 execution control unit, S700 learning model generation process, S800 change Detection model generation processing, S900 overall change score calculation processing, 1200
Support information presentation screen, 1600 change detection method selection screen, 1700 learning model setting screen

Claims (15)

An information processing device that supports the generation of learning models.
A storage unit that stores time-series data of features that can be used to generate learning models,
For each of the features, a value indicating the degree of change of the time-series data, using each time-series data in the learning period of the feature-based feature used for generating the generated learning model as learning data. A change detection model generator that generates a change detection model, which is a machine learning model that generates a change score.
A data acquisition unit that acquires time-series data for a new period for each of the plurality of features.
By inputting the new time series data into the change detection model of each of the plurality of features, an individual change for obtaining an individual change score which is the change score of each of the plurality of features in the new period is obtained. Score calculation department,
An overall change score calculation unit, which obtains an overall change score which is a value indicating the degree of change of the plurality of features as a whole in the new period based on the individual change scores of the plurality of features.
Contribution feature identification unit, which identifies the contribution feature that is the feature that contributes to the overall change score based on the individual change score of each of the plurality of features.
A learning model generation support device equipped with. The learning model generation support device according to claim 1.
The individual change score calculation unit obtains the maximum value of the individual change score in the learning period by inputting each time-series data in the learning period into the change detection model for each of the plurality of features. Ask,
The overall change score calculation unit compares the individual change score in each of the new periods with the maximum value of each of the plurality of features, and the individual change score of the feature having the individual change score of the maximum value or more. The overall change score is calculated so that the larger the number, the larger the value.
Learning model generation support device. The learning model generation support device according to claim 2.
The contributing feature specifying unit identifies the feature whose individual change score is equal to or higher than the maximum value as the contributing feature.
Learning model generation support device. The learning model generation support device according to claim 2.
The change detection model generation unit generates the change detection model by using "Change Finder" which is an algorithm for detecting changes in time series data.
Learning model generation support device. The learning model generation support device according to claim 1.
For each of the plurality of features, the individual change score calculation unit has a degree of similarity between the mode of the time series data in the learning period and the mode of the time series data in the new period equal to or higher than a preset threshold value. The individual change score is calculated based on whether or not
The overall change score calculation unit obtains the change score so that the larger the number of features whose similarity is less than the threshold value, the larger the value.
Learning model generation support device. The learning model generation support device according to claim 5.
The contributing feature specifying unit identifies the feature whose similarity is less than the threshold value as the contributing feature.
Learning model generation support device. The learning model generation support device according to claim 5.
The change detection model generation unit generates the change detection model by using "One Class SVM" which is an algorithm for detecting changes in time series data.
Learning model generation support device. The learning model generation support device according to claim 1.
The individual change score calculation unit
When the amount of the training data used to generate the training model is less than a preset threshold value, the unused feature is a feature not used in the generation of the training model among the plurality of available features. By inputting the new time-series data into each of the change detection models, the individual change score was obtained.
When the amount of the training data used to generate the learning model is equal to or greater than the threshold value, the new time-series data is added to the change detection model for both the unused feature and the used feature. The individual change score is obtained by inputting.
Learning model generation support device. The learning model generation support device according to claim 1.
It further includes a support information presentation unit that outputs information on the overall change score and information on the contributing features.
Learning model generation support device. The learning model generation support device according to claim 9.
Further provided with a correlation feature identification unit for specifying the correlation feature which is the available feature in which the correlation coefficient of the time series data with the new time series data of the contribution feature is equal to or higher than a preset threshold value.
The support information presenting unit outputs information regarding the correlation feature.
Learning model generation support device. The learning model generation support device according to claim 10.
The support information presenting unit outputs the time-series data of the overall change score, the time-series data of the contribution feature, and the time-series data of the correlation feature.
Learning model generation support device. The learning model generation support device according to claim 9.
The change detection model generation unit generates a plurality of the change detection models based on each of a plurality of algorithms for detecting changes in time-series data.
The support information presenting unit accepts the designation of the algorithm and outputs information on the overall change score and information on the contribution feature by the change detection model based on the designated algorithm.
Learning model generation support device. The learning model generation support device according to claim 1.
A learning model performance verification unit that evaluates whether the performance of the learning model meets preset criteria, and
When the performance of the learning model does not meet the criteria, the individual change score calculation unit executes the individual change score calculation process and the overall change score calculation unit executes the overall change score calculation process. Control unit and
Further prepare
Learning model generation support device. The learning model generation support device according to claim 8.
A communication unit that is communicably connected to a data providing device that is an information processing device that provides time-series data of the available features.
A data acquisition unit that receives the available features from the data provider, and
The data acquisition unit
Prior to the calculation of the individual change score by the individual change score calculation unit, a transmission request for the unused feature is transmitted to the data providing device.
After the individual change score calculation unit calculates the individual change score, a transmission stop request for the unused feature is transmitted to the data providing device.
Learning model generation support device. A way to help generate learning models
Information processing device
Steps to store time series data of features that can be used to generate training models,
For each of the features, a value indicating the degree of change of the time-series data, using each time-series data in the learning period of the feature-based feature used for generating the generated learning model as learning data. Steps to generate a change detection model, which is a machine learning model that generates a change score,
Steps to acquire time series data for a new period for each of the plurality of above features,
A step of obtaining an individual change score, which is the change score of each of the plurality of features in the new period, by inputting the new time series data into the change detection model of each of the plurality of features.
A step of obtaining an overall change score, which is a value indicating the degree of change of the plurality of features as a whole in the new period, based on the individual change score of each of the plurality of features.
A step of identifying a contributing feature, which is a feature that contributes to the overall change score, based on the individual change score of each of the plurality of features.
A learning model generation support method that executes.

Images