TWI695327B - Device and method for managing predictive models - Google Patents

Abstract

A method for managing predictive models is provided, including: storing a first predictive model and a second predictive model, wherein the second predictive model corresponds to a historical data stream; loading data stream and recording a real outcome corresponding to the data stream; calculating a predictive outcome according to a current predictive model, wherein the current predictive model is the first predictive model; calculating accuracy of the predictive outcome according to the real outcome; calculating a correlation between the data stream and the historical data stream in response to the accuracy being lower than a first threshold; and switching the current predictive model from the first predictive model to the second predictive model in response to the correlation being higher than a second threshold.

Description

Device and method for managing prediction model

The present invention relates to a method of using an alternative algorithm, and in particular to an apparatus and method for managing a prediction model.

With the development of artificial intelligence technology, people began to apply the prediction model based on machine learning technology to various fields, such as image recognition, speech recognition, semantic analysis, language translation or prediction. Models produced by good machine learning algorithms can achieve very high prediction (or classification) accuracy. However, when the input data of the prediction model changes greatly, the prediction model generated by the fixed training data set often causes the prediction accuracy to decrease because the input data and the label data are too different. Therefore, how to dynamically adjust the prediction model according to the input data to maintain a better prediction accuracy is one of the goals of those skilled in the art.

The invention provides a device for managing a prediction model, including a storage unit and a processor. The storage unit stores multiple modules. The processor is coupled to the storage unit, and accesses and executes multiple modules. The multiple modules include a data dictionary module, an extraction module, a calculation module, and a selection module. The data dictionary module stores the first prediction model and the second prediction model, where the second prediction model corresponds to the historical data stream. The capture module loads the data stream and records the actual results corresponding to the data stream. The calculation module calculates the prediction result of the data stream according to the current prediction model, and calculates the accuracy of the prediction result according to the real result, where the current prediction model is the first prediction model. The selection module calculates the correlation between the data stream and the historical data stream in response to the accuracy being lower than the first threshold, and switches the current prediction model from the first prediction model to the second prediction model in response to the correlation being higher than the second threshold .

The present invention provides a method for managing a prediction model, including: storing a first prediction model and a second prediction model, where the second prediction model corresponds to a historical data stream; loading the data stream and recording the true results corresponding to the data stream; based on the current prediction The model calculates the prediction results of the data stream, where the current prediction model is the first prediction model; calculates the accuracy of the prediction results based on the actual results; calculates the correlation between the data stream and the historical data stream in response to the accuracy being lower than the first threshold; and The current prediction model is switched from the first prediction model to the second prediction model in response to the correlation being higher than the second threshold.

Based on the above, the present invention can select the best prediction model according to the correlation between the data stream and the historical data stream when the accuracy of the currently used prediction model is not good. In this way, no matter how the input data stream changes, the present invention can dynamically select the prediction model most suitable for the input data stream, thereby maintaining a high prediction accuracy. In addition, the present invention can record the relationship between the current data stream and the current prediction model when the current prediction model is applicable to the current data stream, so as to be a reference for selecting a prediction model in the future.

In order to make the above-mentioned features and advantages of the present invention more obvious and understandable, the embodiments are specifically described below in conjunction with the accompanying drawings for detailed description as follows.

In the prior art, the static data dictionary created along with the database cannot be adjusted in time with time, thereby affecting the predictive ability of the intelligent application machine learning model using the static data dictionary. In view of this, the present invention proposes a method that can dynamically adjust the data dictionary over time to help the user select a better prediction model.

FIG. 1 is a schematic diagram of an apparatus 10 for managing a prediction model according to an embodiment of the present invention. The device 10 may include a processor 100 and a storage unit 300. The storage unit 300 can store multiple modules. The processor 100 is coupled to the storage unit 300, and accesses and executes multiple modules stored in the storage unit 300, wherein the multiple modules include an extraction module 310, a calculation module 320, and a selection module 330. Training module 340 and data dictionary module 350.

The processor 100 may be, for example, a central processing unit (CPU), or other programmable general-purpose or special-purpose microprocessor (microprocessor), digital signal processor (DSP), or A programmable controller, an application specific integrated circuit (ASIC) or other similar components or a combination of the above components, the invention is not limited thereto.

The storage unit 300 may be, for example, any type of fixed or removable random access memory (RAM), read-only memory (ROM), flash memory (flash memory) ), hard disk drive (HDD), solid state drive (SSD) or similar components or a combination of the above components, the invention is not limited thereto.

The data dictionary module 350 can store multiple prediction models, and each of the multiple prediction models corresponds to a different historical data stream. Specifically, the data dictionary module 350 may record the associated prediction model in the model metadata table corresponding to the prediction model. For example, a model metadata table corresponding to the prediction model B of the historical data stream with the sequence number "201709" may be shown in Table 1 below (but the invention is not limited thereto). Table 1

The data dictionary module 350 may also record the associated prediction model and historical data stream in the relationship table between the prediction model and historical data stream, as shown in Table 2 below. For example, if the data dictionary module 350 stores "prediction model A", "prediction model B", and "prediction model C", if the "prediction model B" is used to calculate the corresponding to "historical data stream 201709" If the prediction result is the most accurate, the data dictionary module 350 can associate "prediction model B" with "historical data stream 201709", and record the associated "prediction model B" and "historical data stream 201709" in Table 2 The relationship table shown. In Table 2, "prediction model B" corresponds to "historical data stream 201709", "prediction model A" corresponds to "historical data stream 201708" and "prediction model C" corresponds to "historical data stream 201707". Table 2

The data dictionary module 350 can be implemented using cloud databases such as HBase, Live, Cassandra, MonogoDB, or related display databases Teradata, SQL server, etc. The present invention is not limited thereto.

The extraction module 310 can load a data stream and extract metadata from the data stream to generate a metadata table. For example, a metadata table of a data stream with the sequence number "201710" may be as shown in Table 3 below (but the invention is not limited thereto). In addition, the capture module 310 can be used to record the actual results corresponding to the data stream. For example, assume that "Data Stream 201710" is related to the set temperature of the air conditioner. If the user adjusts the set temperature of the air conditioner to "25.85 degrees", the capture module 310 can take "25.85 degrees" as the real result and record it in the metadata table corresponding to "Data Stream 201710", as shown in Table 3 "*4" in. In one embodiment, the data stream (and its corresponding metadata table) retrieved by the retrieval module 310 can be recorded in the data dictionary module 350 as a historical data stream (and its corresponding historical metadata table) Among. table 3

The process of loading and capturing using the capture module 310 can be implemented using programming languages such as Shell Scrip, Python, Java, R, or SparkSQL, or software tools such as Tableau, Highchart, etc. The present invention is not limited to this.

The calculation module 320 can be used to calculate the prediction result of the data stream according to the current prediction model, and calculate the accuracy of the prediction result according to the real result. For example, assume that the actual result of the set temperature corresponding to "Data Stream 201710" is "25.85 degrees" (as shown in Table 3), and the current prediction model used by the calculation module 320 is "Prediction Model A". If the calculation module 320 uses "prediction model A" as the current prediction model to calculate the prediction result corresponding to the data stream "201710" as "22 degrees", that is, the device 10 recommends that the user adjust the set temperature to "22 degrees" ". In this way, the calculation module 320 can calculate the prediction accuracy of the "prediction model A" for the data stream "201710" based on the actual result "25.85 degrees" and the prediction result "22 degrees". The closer the prediction result is to the true result, the higher the prediction accuracy. The calculation method of the accuracy can be adjusted by the user of the device 10 according to their needs, and the invention is not limited thereto.

The selection module 330 may calculate the correlation between the data stream and the historical data stream in response to the accuracy of the prediction being below a threshold. For example, suppose the threshold of accuracy corresponding to "prediction model A" is 80%, as shown in Table 1. When the calculation module 320 uses the "prediction model A" to calculate the accuracy based on the prediction result "22 degrees" and the actual result "25.85 degrees" corresponding to the "data stream 201710", if the calculated accuracy is less than The threshold of the accuracy of "prediction model A" is 80%, and the selection module 330 can start to calculate the correlation between the "data stream 201710" and each historical data stream. Taking the historical data stream recorded in Table 2 as an example, the selection module 330 can calculate the correlation between the "data stream 201710" and the "historical data stream 201707", and the "data stream 201710" and "historical data stream 201708" respectively. Correlation between "Data Stream 201710" and "Historical Data Stream 201709".

In an embodiment, different prediction models may have the same or different accuracy thresholds. For example, if the threshold of accuracy corresponding to "prediction model A" is 80%, the threshold of accuracy corresponding to "prediction model B" may be the same as or different from 80%.

In an embodiment, the selection module 330 may calculate the correlation between the data stream and the historical data stream according to the metadata table of the data stream and the historical metadata table of the historical data stream. For example, the selection module 330 may be based on at least one of the data field type, the data field distribution type, and the zero value ratio recorded in the metadata table of the "data stream 201710" and the "historical data stream 201709" "At least one of the data field type, data field distribution type, and zero value ratio recorded in the historical metadata table to calculate the correlation between "data stream 201710" and "historical data stream 201709" .

The correlation can be calculated based on the distance between the data groups. For example, the selection module 330 uses information including Euclidean distance, Manhattan distance, Mahalanobis distance, cosine distance, correlation distance, or message. Entropy (information entropy) and other ways to calculate the correlation between the data stream and the historical data stream, the present invention is not limited to this.

After calculating the correlation between the data stream and the historical data stream, the selection module 330 may switch the current prediction model from a prediction model to another corresponding to the historical data stream in response to the calculated correlation being higher than a threshold A prediction model. For example, if the calculation module 320 uses "prediction model A" as the current prediction model to calculate the prediction result of "data stream 201710", the selection module 330 may respond to "data stream 201710" and "historical data stream 201709" The correlation between them is higher than the correlation threshold and the current prediction model is converted from "prediction model A" to "prediction model B" corresponding to "historical data stream 201709". Due to the high correlation between "Data Stream 201710" and "Historical Data Stream 201709", "Prediction Model B" applicable to "Historical Data Stream 201709" is also more applicable to "Data Stream 201710" than other prediction models ". Therefore, using "Prediction Model B" to calculate the prediction result of "Data Stream 201710" will be more accurate than using "Prediction Model A" to calculate the prediction result of "Data Stream 201710".

In an embodiment, if the correlation between a plurality of historical data streams and data streams is higher than the correlation threshold, the selection module 330 may select the historical data stream with the highest correlation with the data stream The corresponding prediction model is used as the current prediction model.

In an embodiment, if the accuracy of the prediction result calculated by the calculation module 320 based on a prediction model and the actual result of a data stream is higher than a threshold, the selection module 330 may respond to the accuracy being higher than The threshold value records in the data dictionary module 350 that the data stream is associated with the prediction model. For example, if the accuracy of the prediction result calculated by the calculation module 320 according to the actual results of the "prediction model B" and the "data stream 201710" is higher than the accuracy threshold, the selection module 330 can Group 350 records that "data stream 201710" is associated with "prediction model B", and Table 2 is modified as shown in Table 4 below. Table 4

The training module 340 can be used to train a prediction model corresponding to the data stream according to the feature data of the data stream and the label data. Specifically, if the metadata table corresponding to a data stream includes many different types of data, the training module 340 may select the specific data as a feature for training the prediction model based on the degree of association between the specific data and the label data is greater than a threshold Data to train a prediction model corresponding to the data stream based on the feature data and the label data.

Taking the metadata table of “Data Stream 201710” described in Table 3 as an example, on the premise that the problem to be solved by the device 10 is “suggesting a suitable set temperature”, the training module 340 can set the “set temperature” as Label data used to train prediction models. Then, the training module 340 can change the "indoor temperature" and "indoor temperature" based on the degree of correlation between the content recorded in the "indoor temperature" and "outdoor temperature" fields and the content recorded in the "set temperature" field greater than a threshold. "Outdoor temperature" is defined as the characteristic data used to train the prediction model. For example, the training module 340 can calculate the degree of correlation between the "set temperature" and "indoor temperature" and the "set temperature" based on the "mean" values described in "set temperature", "indoor temperature", and "outdoor temperature" The degree of correlation with "outdoor temperature". Since the "mean" values described in "set temperature", "indoor temperature" and "outdoor temperature" are very close, the training module 340 can calculate that both "indoor temperature", "outdoor temperature" and "set temperature" have Highly correlated. Therefore, the training module 340 defines "indoor temperature" and "outdoor temperature" as the characteristic data for training the prediction model.

The selection module 330 may calculate the degree of correlation according to methods such as coefficient of correlation, coefficient of determination, regression analysis, or least square principle. The present invention is not limited to this .

In one embodiment, the training module 340 may determine the algorithm used by the prediction model according to the distribution types of the data fields of the feature data and the label data. Specifically, if the data field distribution types of the label data and the feature data are continuous, the training module 340 may decide to use a regression algorithm to train the prediction model. On the other hand, if the data field distribution types of the label data and the feature data are discrete, the training module 340 may decide to use a classification algorithm to train the prediction model. In an embodiment, if the label data of a data stream is not recorded in the metadata table of the data stream, the training module 340 may train the prediction model according to the clustering algorithm.

FIG. 2 is a schematic diagram of a method for managing a prediction model according to an embodiment of the present invention, where the method may be implemented by the apparatus 10 shown in FIG. 1. In step S21, the first prediction model and the second prediction model are stored, wherein the second prediction model corresponds to the historical data stream. In step S22, the data stream is loaded and the actual result corresponding to the data stream is recorded. In step S23, the prediction result of the data stream is calculated according to the current prediction model, where the current prediction model is the first prediction model. In step S24, the accuracy of the prediction result is calculated according to the real result. In step S25, it is determined whether the accuracy is lower than the first threshold. If the accuracy is lower than the first threshold, step S26 is entered. If the accuracy is higher than or equal to the first threshold, then return to step S22, load a new data stream and record the true result corresponding to the new data stream. In step S26, the correlation between the data stream and the historical data stream is calculated. In step S27, it is determined whether the correlation is higher than the second threshold. If the correlation is higher than the second threshold, step S28 is entered. If the correlation is lower than or equal to the second threshold, return to step S26 to calculate the correlation between the data stream and another historical data stream. In step S28, the current prediction model is switched from the first prediction model to the second prediction model.

In summary, the present invention can periodically store the data stream and its metadata, and dynamically adjust the prediction model used according to the changes in the data stream. If the accuracy of the currently used prediction model is not good, the present invention can select the best prediction model according to the correlation between the data stream and the historical data stream. In addition, the training module of the present invention can select the algorithm used by the prediction model based on the data field distribution type of the data stream. Therefore, the trained prediction model will have higher accuracy. On the other hand, the present invention can record the relationship between the current data stream and the current prediction model when the current prediction model is applicable to the current data stream, and serve as a reference for selecting a prediction model in the future.

Although the present invention has been disclosed as above with examples, it is not intended to limit the present invention. Any person with ordinary knowledge in the technical field can make some changes and modifications without departing from the spirit and scope of the present invention. The scope of protection of the present invention shall be subject to the scope defined in the appended patent application.

10: Device for managing prediction models

100: processor

300: storage unit

310: Capture module

320: computing module

330: Select module

340: Training module

350: Data dictionary module

S21, S22, S23, S24, S25, S26, S27, S28: steps

FIG. 1 is a schematic diagram of an apparatus for managing a prediction model according to an embodiment of the present invention. FIG. 2 is a schematic diagram of a method for managing a prediction model according to an embodiment of the present invention.

S21, S22, S23, S24, S25, S26, S27, S28: steps

Claims (8)

An apparatus for managing prediction models includes: a storage unit storing multiple modules; and a processor coupled to the storage unit, the processor accessing and executing the multiple modules, the multiple modules It includes: a data dictionary module that stores a first prediction model and a second prediction model, where the second prediction model corresponds to a historical data stream; an extraction module, loads a data stream, and records real results corresponding to the data stream And retrieve the metadata table of the data stream, where the metadata table includes feature data, label data and first data; the calculation module calculates the prediction result of the data stream according to the current prediction model, and according to the The true result calculates the accuracy of the prediction result, where the current prediction model is the first prediction model; a selection module that calculates the data flow and the data in response to the accuracy being lower than a first threshold Correlation of historical data streams, and in response to the correlation being higher than a second threshold, switching the current prediction model from the first prediction model to the second prediction model; and a training module, based on the The degree of association between the first data and the tag data is greater than a third threshold, the first data is selected as the feature data, and the corresponding to the data stream is trained according to the feature data and the tag data The first prediction model. The device according to item 1 of the patent application scope, wherein the metadata table includes data field distribution types, and the training module determines the first prediction model to use according to the data field distribution types Algorithm. The device according to item 2 of the patent application scope, wherein the training module determines that the algorithm is a regression algorithm based on the data field distribution type being continuous, and is based on the data field distribution type Instead of being discrete, the algorithm is determined to be a classification algorithm. The device according to item 1 of the patent application scope, wherein the selection module records in the data dictionary module that the data stream is associated with the first prediction in response to the accuracy being higher than a fourth threshold model. The device as described in item 1 of the patent application scope, wherein the data dictionary module further stores a historical metadata table of the historical data stream, wherein the selection module is further based on the metadata table and the historical metadata The data table calculates the correlation. The device according to item 5 of the patent application scope, wherein the selection module is further based on at least one of a data field type, a data field distribution type, and a zero value ratio in the metadata table Calculate the correlation. The device according to item 1 of the patent application scope, wherein the first threshold corresponds to the first prediction model. A method for managing a prediction model includes: storing a first prediction model and a second prediction model, wherein the second prediction model corresponds to a historical data stream; loading the data stream and recording the actual results corresponding to the data stream, and extracting Get a meta-data table of the data stream, where the meta-data table includes feature data, label data, and first data; calculate a prediction result of the data stream according to a current prediction model, where the current prediction model is the first A prediction model; calculating the accuracy of the prediction result based on the true result; calculating the correlation between the data stream and the historical data stream in response to the accuracy being lower than the first threshold; responding to the correlation Selects the current prediction model from the first prediction model to the second prediction model when the performance is higher than the second threshold; and selects based on the degree of association between the first data and the label data is greater than the third threshold The first data is used as the feature data, and the first prediction model corresponding to the data stream is trained based on the feature data and the tag data.

Images