TWI829895B - Model monitoring system based on health and method thereof - Google Patents

Abstract

The present invention relates to a model monitoring system based on health and a method thereof, the method comprises: providing a prediction model based on big data; receiving a test set generated by collecting test data of the big data; formulating model health weights and calculating various indicators of the prediction model and system; summing the various indicators based on the model health weights to generate the health of the prediction model; and monitoring the prediction model, determining the status of the prediction model or the need for reconstruction based on the health of the prediction model. The invention refers to various indicators of the system and the model, calculates the health of the model and monitors the model, and can more effectively and appropriately select the best model and provide the user with the best application service.

Description

Health-based model monitoring system and method

The present invention relates to model monitoring technology, and in particular, to a health-based model monitoring system and method.

Conventional model monitoring systems usually use model accuracy or other single evaluation indicators for model evaluation and monitoring to select models or decide to retrain models. However, the above-mentioned conventional system uses a single model evaluation index for model monitoring. In model evaluation, At the implementation level, specific model evaluation indicators are often used as the basis for measuring models. However, because multiple evaluation factors such as the system and model are not considered to monitor and evaluate the prediction model to decide whether to retain it in the system, it cannot adapt to a variety of different models. Based on the demand conditions, it provides application services for other users to call the model. Therefore, in complex situations, the commonly used single-index model evaluation system cannot effectively and appropriately select the best model from the model platform because it fails to refer to multiple indicators such as the system and the model. It is not a good design and is in urgent need of be improved.

It can be seen from the above that if a model monitoring mechanism can be found, especially one that can systemize multiple evaluation factors and automatically calculate health, so as to continuously monitor the model, this will become a goal that those in the technical field are eager to pursue a solution.

The purpose of the present invention is to propose a model monitoring mechanism that uses a broader model evaluation method and refers to a variety of systems and model-related important indicators to systematically and automatically calculate the health of the model and monitor the model accordingly, and is effective and appropriate in response to complex situations. Select the best model to improve the shortcomings of the existing technology that only monitors the model with a single evaluation index.

In order to achieve the above objects and other objects, the present invention proposes a health-based model monitoring system, including: a model management module to generate a prediction model based on big data; a data collection module to collect big data test data to Generate a test set; the model health module is used to receive the prediction model from the model management module and the test set from the data collection module, formulate model health weights and calculate various parameters of the prediction model and the system. An indicator is used to sum up the indicators according to the health degree weight of the model to generate the health degree of the prediction model; and the model monitoring module is connected to the model health degree module to receive the health degree of the prediction model, This is used to monitor the prediction model and determine the status or reconstruction needs of the prediction model.

In the above system, the model health weight is formulated based on other external data, or is formulated by itself, or a mixture of the above two methods is adopted.

In the above system, the various indicators include model evaluation indicators, system performance indicators, popular evaluation indicators or other evaluation indicators, and the model health module uses at least two of the above indicators to calculate health.

In the above system, the model evaluation index is to evaluate the prediction model through the prediction model and the test set using at least one method of control chart, accuracy or other methods to generate a model evaluation result; the system performance index is in the system Record and calculate at least one of the average response time, throughput, stability or other system performance of the prediction model to generate System performance evaluation results; the popularity evaluation indicator records and calculates at least one of the number of calls, number of users, service time or other related model popularity in the system to generate popularity evaluation results.

The present invention proposes a health-based model monitoring method, which includes: providing a prediction model based on big data; receiving a test set generated by collecting big data test data; formulating model health weights and calculating the prediction model and system Each indicator; sum up each indicator according to the health degree weight of the model to generate the health degree of the prediction model; and monitor the prediction model according to the health degree of the prediction model and determine the status or reconstruction needs of the prediction model.

In the above method, the step of formulating the model health weight includes formulating the model health weight of each indicator based on other external data, or formulating the model health weight of each indicator by oneself, or adopting the above two methods. A mixed method is used to formulate the model health weight of each indicator.

In the above method, the various indicators include model evaluation indicators, system performance indicators, popular evaluation indicators or other indicators, and the model health module uses at least two of the above indicators to calculate health.

In the above method, the model evaluation index is to evaluate the prediction model through the prediction model and the test set using at least one method of control chart, accuracy or other methods to generate a model evaluation result; the system performance index is based on the system Record and calculate at least one of the average response time, throughput, stability or other system performance of the prediction model to generate system performance evaluation results; the popular evaluation indicators are recorded in the system Record and calculate at least one of the number of calls, number of users, service time or other related model popularity of the prediction model to generate popularity evaluation results.

In summary, it can be seen that the health-based model monitoring system and method of the present invention generates a prediction model through the model management module and puts it online, and uses the data collection module to collect test sets. When sufficient test sets are accumulated, the model is formulated Health weight and calculation of various indicators, sum up the indicators according to the model health weight to generate the health of the prediction model, and then use the model monitoring module to monitor the model according to the health of the prediction model until a new Test set, and then perform the next round of health calculation and model monitoring. From the above, it can be seen that the present invention refers to various indicators such as the system and model to calculate the model health and monitor the model accordingly, which can more effectively and appropriately select the best model to provide users with the best application services. The present invention improves the shortcomings of the existing technology that only monitors the model with a single evaluation index. Furthermore, the present invention systematically and automatically calculates multiple evaluation factors while considering different requirements such as the system and the model. Healthy, based on which the model can be continuously monitored and the best model can be effectively and appropriately selected in response to complex situations.

1: Health-based model monitoring system

11:Model management module

12:Data collection module

13: Model health module

14: Model monitoring module

S21~S25: steps

S31~S38: Process

S41~S44: Process

Figure 1 is a system architecture diagram of the health-based model monitoring system of the present invention;

Figure 2 is a step diagram of the health-based model monitoring method of the present invention;

Figure 3 is a flow chart of the health-based model monitoring method in one embodiment of the present invention; and

Figure 4 is a flow chart of using control charts to present model evaluation results in the health-based model monitoring method of the present invention.

The following describes the technical content of the present invention through specific embodiments. Those familiar with the art can easily understand the advantages and effects of the present invention from the content disclosed in this specification. However, the present invention can also be implemented or applied through other different specific implementation forms.

Figure 1 is a system architecture diagram of the health-based model monitoring system of the present invention. As shown in the figure, the health-based model monitoring system 1 of the present invention can refer to a variety of systems and model-related important indicators to systematically and automatically calculate the model health and monitor the model accordingly. Among them, the health-based model monitoring system 1 The system includes a model management module 11, a data collection module 12, a model health module 13 and a model monitoring module 14.

The model management module 11 is used to generate prediction models based on big data. In short, the model management module 11 generates the model to be predicted and puts it online, and provides the functions of model training, evaluation and deployment.

The data collection module 12 is used to collect big data test data to generate a test set. The data collection module 12 is mainly used to collect big data test data and check whether sufficient test sets have been accumulated to determine the process. If enough test sets have been accumulated, the model health module 13 is started to perform subsequent procedures. Otherwise, if not, When enough test sets can be accumulated, the data collection module 12 continues to collect test sets.

The model health module 13 is used to receive the prediction model from the model management module 11 and the test set from the data collection module 12, and by formulating the model health weight and calculating various indicators of the prediction model and system, The various indicators are summed according to the healthiness weight of the model to generate the healthiness of the prediction model.

In one embodiment, the model health weight can be formulated based on other external data, or can be formulated by itself, or a mixture of the above two methods can be used. Specifically, the formulation of model health weights can include objective methods and preset methods. The objective method means that the model health weights are formulated based on actual conditions, such as other external data, such as referring to the statistical results of Google searches, and The default method is to set the model health weight by yourself.

In one embodiment, various indicators may include model evaluation indicators, system performance indicators, popular evaluation indicators or other evaluation indicators. The model health module 13 uses at least two of the above indicators to perform health calculations.

The aforementioned model evaluation indicators are used to evaluate the prediction model through the prediction model and the test set, using at least one method of control chart, accuracy or other methods to generate model evaluation results; the aforementioned system performance indicators are recorded in the system and Calculate at least one of the average response time, throughput, stability or other system performance of the prediction model to generate system performance evaluation results; in addition, the aforementioned popular evaluation indicators are recorded in the system and calculate the calls of the prediction model At least one of the number of times, number of users, service time or other related model popularity to generate popularity evaluation results.

The model monitoring module 14 is connected to the model health module 13 and is used to receive the health of the prediction model, monitor the prediction model, and determine the status or reconstruction needs of the prediction model.

It can be seen from the above that the health-based model monitoring system of the present invention mainly uses the model health module 13 and the model monitoring module 14 to evaluate the model health, wherein the model health module 13 transmits the model health to the model monitoring Module 14, when the model monitoring module 14 sends a request for health calculation, the model health module 13 will return the health calculation result, where, The model health weight can be formulated based on external data, or it can be formulated by itself, or it can be formulated using a mixture of the above two methods. The calculated indicators include various evaluation indicators of the system and model, such as model evaluation indicators, system performance indicators, popular Evaluation indicators or other evaluation indicators, and adopt at least two indicators for health calculation.

Figure 2 is a step diagram of the health-based model monitoring method of the present invention.

In step S21, a prediction model generated based on big data is provided. In this step, the prediction model can be generated based on big data through the model management module as shown in Figure 1 and put online.

In step S22, a test set generated by collecting big data test data is received. In this step, the big data test data can be collected through the data collection module as shown in Figure 1 to generate a test set.

In step S23, the model health weight is formulated and various indicators of the prediction model and system are calculated. In this step, the model health weight can be formulated through the model health module as shown in Figure 1, and various indicators of the prediction model and system can be calculated.

In one embodiment, the various indicators may be model evaluation indicators, system performance indicators or popular evaluation indicators, and the model health module uses at least two of the above indicators to perform health calculations. Specifically, the model evaluation index can use at least one method of control chart, accuracy or other methods to evaluate the prediction model through the prediction model and the test set to generate the model evaluation result; the system performance index can be recorded and calculated in the system. At least one of the average response time, throughput, stability or other system performance of the prediction model is used to generate system performance evaluation results; popular evaluation indicators can be recorded and calculated in the system such as the number of calls, number of users, and services of the prediction model At least one of time or other related model popularity to generate popularity evaluation results.

Specifically, formulating the model health weights described in this step may include formulating the model health weights for each indicator based on actual conditions (such as other external data), or formulating the model health weights for each indicator by ourselves. degree weight, or a mixture of the above two methods is used to formulate the model health weight of each indicator.

In step S24, the various indicators are summed according to the healthiness weight of the model to generate the healthiness of the prediction model. In this step, the model health is finally calculated by calculating various indicators of the system and model, summing up the model health weights, and using the model health calculation formula. The model health calculation formula will be described in detail later.

In step S25, the prediction model is monitored according to the health of the prediction model, and the status or reconstruction needs of the prediction model are determined. In this step, the model monitoring module as shown in Figure 1 is used to monitor the model according to its health, so as to achieve the purpose of monitoring the prediction model, or to further determine the model status and model reconstruction needs, so as to achieve effective and appropriate selection of the best model. The efficacy of the model.

In addition, the decision process depends on whether a new test set is obtained. When a new test set is obtained, the data collection module as shown in Figure 1 can be started to continuously collect the test set and perform new model evaluation.

Figure 3 is a flow chart of the health-based model monitoring method in one embodiment of the present invention.

In process S31, the model management module generates the model and goes online. This process generates the model to be predicted and provides the functions of model training, evaluation, and deployment.

In process S32, the data collection module collects the test set. This process collects big data test data, and then enters process S33. The process will be determined based on whether sufficient test sets have been accumulated. When enough test sets have been accumulated, the model health module will be started to perform subsequent model health. The formulation of degree weights and the calculation of various indicators. On the contrary, if insufficient test sets cannot be accumulated, the data collection module will be activated to continue collecting test sets.

In process S34, the model health module formulates the model health weight. In this process, the model health weight can be formulated by objectively formulating the model health weight of each indicator, or by formulating the health weight of each model by oneself, or by using a mixture of the above two methods to formulate the health weight of each model. Here The objective method described is based on the actual situation.

In process S35, the model health module calculates various indicators. In this process, each indicator refers to various evaluation indicators of the system and model, which can be model evaluation indicators, system performance indicators, popular evaluation indicators or other evaluation indicators, and at least two indicators are used for health calculation. By referring to various indicators such as systems and models, we can effectively and appropriately select the best model in response to complex situations, and provide users with the best application services.

In one embodiment, the aforementioned model evaluation indicators can be configured to receive a prediction model generated based on big data and a test set collected, evaluate the prediction model and generate a model evaluation result, where the model evaluation result can be The evaluation results are generated using at least one of control charts, accuracy, or other methods.

In another embodiment, the aforementioned system performance indicators can be configured to receive a prediction model generated based on big data, and the average response time, throughput, stability or other predictions related to this model can be recorded and calculated in the system. At least one of system performance.

In yet another embodiment, the aforementioned popular evaluation indicators can be configured to receive a prediction model generated based on big data, and the system can record and calculate the number of calls, number of users, service time or other related predictions predicted by this model. At least one of the model popularity levels.

In process S36, the model health module generates health based on weighted summation. In this step, various indicators of the system and model can be calculated, summed according to the health weight, and the model health calculation formula can be used to generate the model health.

In process S37, the model monitoring module monitors the model according to the health level. In this step, the model monitoring module is configured to receive the model health of the prediction model, monitor the prediction model accordingly, and determine the model status and model reconstruction requirements.

Among the above six processes, process S35 calculates various indicators such as system and model through the model health module, and cooperates with the formulating model health weight in process S34, and the health degree is automatically generated by process S36, because this model health is based on the reference A variety of system and model-related important indicators can be used to systematically and automatically calculate health in response to complex situations. Finally, through the model monitoring module of process S37, the model is monitored based on health, which can achieve the effect of effectively and appropriately selecting the best model. .

Finally, the above process can be executed in sequence until a new test set is obtained, and then the next round of health calculation and model monitoring is performed. That is, in process S38, the process will be decided based on whether a new test set has been obtained. When a new test set has been obtained, When the test set is obtained, the data collection module will be started to continuously collect the test set. When a new test set cannot be obtained, the entire process will be ended.

The following discloses an embodiment of the health-based model monitoring method of the present invention, please refer to Figure 3 together. This embodiment adopts a supervised machine learning algorithm and uses a decision tree algorithm to train the model and put it online. Taking into account demand conditions such as model evaluation, system performance, and popularity, model evaluation indicators, system performance indicators, and Three important indicators, including popular evaluation indicators, are aimed at improving prediction accuracy, reducing model response time, and mastering popular models. The health degree is obtained by linear weighting calculation, which is used to monitor the model in order to ensure that the model is accurate. Under complex combination conditions such as declining rate, declining system performance, declining popularity, or rising accuracy but declining system performance, inappropriate models can still be removed from the shelves, effectively improving the performance of model evaluation and monitoring.

First, as shown in process S31, the model management module generates a model and goes online, browses and manages the trained model, and then deploys and goes online.

Then, as in process S32, the test set is collected through the data collection module, new test set data is continuously collected, and the test set is uploaded in real time. For example, the evaluation algorithm can be set online in real time, and the system will immediately call the model health The degree module formulates the model health weight and calculates various indicators to facilitate health calculation and model monitoring. Among them, the real-time method can also be used to upload the test set to the periodic method, and the periodic method can be used to set the evaluation algorithm required for the system schedule. , evaluate the start and end time and frequency, and specify that the test set should be updated regularly, the system will also provide the above information to the model health module. In addition, as in process S33, the decision process will be based on whether sufficient test sets have been accumulated. When enough test sets have been accumulated, the model health module will be started to formulate model health weights and calculate various indicators. When sufficient test sets are not accumulated, When testing the set, the data collection module will continue to collect the test set.

The data in this embodiment belongs to the classification prediction in the supervised machine learning algorithm. Here, 12 sets of stable test sets ( S ₁ , S ₂ ,..., S ₁₂ ) are collected, in which each set of test sets S _i Contains a large amount of data, and the prediction model will be used in the future. Whenever a sufficient test set is accumulated, a model evaluation result Weighted F1 score will be obtained online in real time. At the same time, the system will immediately call the model health module to calculate the model evaluation index. To facilitate health calculation and model monitoring. Specifically, the Weighted F1 score is a high-quality characteristic, so when the Weighted F1 score is higher, it means that the model evaluation result is better. Accordingly, this embodiment will sequentially obtain 12 model evaluation results Weighted F1 score ( X ₁ , X ₂ ,..., X ₁₂ ) for subsequent calculation of model evaluation indicators.

Next, as in process S34, the model health module formulates the model health weight, is configured to receive a prediction model generated based on big data and the collected test set, and uses an objective method to formulate the model health weight of each indicator. , where each indicator value and each weight value are between 0 and 1, and the sum of each weight value is 1. The formulation method can be changed to the default method, that is, the health of each model is formulated according to the importance of each indicator. Weights, or a mixed method of objective and preset methods to determine the health weight of each model. By referring to various indicators such as the system and model, in response to complex situations, the effect of effectively and appropriately selecting the best model can be provided. the best application services.

The various indicators calculated by the model health module can include three indicators such as model evaluation indicators, system performance indicators, and popular evaluation indicators. There are three weight values W ₁ , W ₂ , and W ₃ for formulating the model health weight, and based on an objective method Develop a set of preset weight values ( W ₁ , W ₂ , W ₃ ), in which each weight value W ₁ , W ₂ , W ₃ is between 0 and 1 and the sum of their weight values is 1, which can be used for subsequent Healthiness is generated by linear summation based on model health weights. In this embodiment, the Google website is used to search using three keywords such as model monitor, average response time, and api call. The number of searches obtained are 665,000,000 times, 362,000,000 times, and 522,000,000 times respectively. The weight values after normalization are ( W ₁ , W ₂ , W ₃ )=(0.43,0.23,0.34), the model health can be calculated based on this combination of weight values later. The above is formulated in an objective way; in another embodiment, the system can formulate a set of presets in a preset way. Assume that the weight value ( W ₁ , W ₂ , W ₃ ) = (0.64, 0.12, 0.24); in another embodiment, the objective weight value and the default weight value can also be mixed, for example: the objective weight value ( W ₁ , W ₂ , W ₃ )=(0.43,0.23,0.34), the default weight value ( W ₁ , W ₂ , W ₃ )=(0.64,0.12,0.24), using the ratio=0.6:0.4, for the objective weight Values and preset weight values are weighted, that is, health weight = (0.43, 0.23, 0.34) × 0.6 + (0.64, 0.12, 0.24) × 0.4 = (0.51, 0.19, 0.30).

Then, as in process S35, the model health module calculates various indicators, among which the indicators are model evaluation indicators, system performance indicators, and popular evaluation indicators. By referring to various indicators such as the system and model, in response to complex situations, we can achieve Effectively and appropriately selecting the best model can provide users with the best application services. The calculation methods of the model evaluation index, system performance index, and popular evaluation index in this embodiment will be disclosed in sequence below.

When calculating model evaluation indicators, it can be configured to receive a prediction model generated based on big data and a collected test set, evaluate the prediction model and generate model evaluation results. The model evaluation results can use control charts, accuracy rates or other methods, at least one of which produces evaluation results.

The best model can be effectively selected by referring to the model evaluation metrics and calculating health based on them. The system will obtain model evaluation results based on the prediction model for a stable test set, use control charts and collect specific number of data, and select appropriate control charts for drawing based on Statistical Process Control (SPC). In addition, please refer to Figure 4 for regular correction of control limits. In process S41, the control chart interpretation rules are judged, that is, at least one item is selected to determine abnormal values according to the control chart interpretation rules. In process S42, abnormal values are screened out, that is, based on The control chart interpretation rules determine that the abnormal values should be eliminated. In process S43, the control limits are calculated, that is, the control limits are recalculated based on the model evaluation results after eliminating the outliers. In process S44, the control chart is drawn, that is, the model evaluation results and Regulatory boundaries are drawn on control charts. Then, according to whether the model evaluation result is a large quality characteristic or a small quality characteristic, the model evaluation index of this embodiment is calculated. The above-mentioned method of using a specific number of transactions can also adopt the number of transactions since the launch.

，其中R _i =|X _i -X _i-1|)，乘以E ₂(參照管制圖係數表，由於移動資料R _i 組內樣本數為2，故選用E ₂=2.66)，當作3個標準差的認定標準，即X-MR管制圖之中心線(CL)與管制上限(UCL)及管制下限(LCL)之距離皆為3個標準差。X-MR管制圖是以資料的平均值

做為管制圖的中心線(CL)，即

，如此，X-MR管制圖的管制上限(UCL)及管制下限(LCL)，分別為：

以及

。 When process S32 uses the data collection module to collect test sets and collects r sets of stable test sets, use the classification prediction model to obtain the model evaluation result Weighted F1 score ( X _i , where i=1,2,…,r), and Select the number n of data collected to calculate the control limit. According to the model evaluation results, the Weighted F1 score belongs to measurement data, and the sampling size is a single observation value. Refer to the SPC to select the X-MR control chart, with each forward and backward moving data absolute The average of the value differences (i.e.

, _where R _{i =} | _ _{_} _ _{_} _ _{_} The identification standard of three standard deviations is that the distance between the center line (CL) of the X-MR control chart and the upper control limit (UCL) and lower control limit (LCL) is 3 standard deviations. The X-MR control chart is based on the average of the data

As the center line (CL) of the control chart, that is

, so, the upper control limit (UCL) and lower control limit (LCL) of the X-MR control chart are respectively:

as well as

.

，且

。如此，X-MR管制圖的管制上限(UCL)及管制下限(LCL)，分別為：

0.73+2.66×0.074=0.927以及

0.533。 This embodiment collects 12 sets of stable test sets and uses a classification prediction model to obtain 12 model evaluation results. Weighted F1 score ( X _i ) are X ₁ =0.75, X ₂ =0.71, X ₃ = _0.63 , and , X ₅ =0.68, X ₆ =0.75, X ₇ =0.77, X ₈ = 0.68, X ₉ = _0.76 , X 10 _{= 0.78} , This _embodiment collects 10 model _evaluation results Weighted F1 score ( X _i ) and calculates R _{i as} follows _: R ₂ =| X ₂ |=|0.63-0.71| = 0.08, R ₄ =| X ₄ - X _{3 |} =|0.79-0.63 _| = _0.16 , R 5 =| R _{6 =} | _ _{_} _ _{_} _ _{_} _ _{_} _ _{_} _ _{_} _ _{_} 0.68-0.77| = 0.09, R ₉ =| X ₉ - X ₈ |= | _0.76-0.68 | _{= 0.08} , R 10 =|

,and

. In this way, the upper control limit (UCL) and lower control limit (LCL) of the X-MR control chart are respectively:

0.73+2.66×0.074=0.927 and

0.533.

，其中，

Subsequently, depending on the test set collection situation, the control limits are regularly corrected according to the control chart interpretation rules and the control chart is redrawn. At this time, the model health module calculates the model evaluation index I ₁ of this model based on the model evaluation results to maximize the quality characteristics. for:

,in,

，因而f(X ₁₁)=X ₁₁=0.41，再者X ₁₂=0.71>0.533=LCL，因而f(X ₁₂)=LCL=0.533。 Continuing the previous embodiment, since

, so f ( X _{11 ) =} _ _ _{_} _

，其中，

，

。 When the model evaluation result is a small quality characteristic, the model evaluation index I ₁ of this model is calculated as:

,in,

,

.

，故模型評估指標

The Weighted F1 score in this embodiment is a large target, where σ=

, so the model evaluation index

及

均小於0，模型評估指標I ₁可設定

或

，也 可以設定

或

、或其它轉換函數。 In addition, in this embodiment, the model evaluation index needs to be between 0 and 1, according to

and

are all less than 0, the model evaluation index I ₁ can be set

or

, you can also set

or

, or other conversion functions.

When calculating the system performance index, it can be configured to receive a prediction model generated based on big data, and record and calculate at least one of the average response time or other system performance predicted by the model in the system.

，其中，

，

Since the system performance index measures the average response time predicted by the model, it is more appropriate to select the best model by referring to the system performance index and calculating the health accordingly. That is, the system will record the average response time of all online models ( t _i , where i =1,2,…,m), select the acquisition period as a specific range or go online to date, set the upper specification limit of the average reaction time USL _{average reaction time} based on the historical data of the average reaction time, and calculate the process capability index Cpk _{average reaction time} , since the average reaction time only sets the upper limit of the specification, the USL _{average reaction time} is a unilateral specification, so Cpk _{average reaction time} =

,in,

,

，S _{平均反應時間} =

，因此，Cpk _{平均反應時間} =

This embodiment uses the model health module to calculate system performance indicators, and records and calculates 10 average response times related to this model prediction, which are t ₁ =13, t ₂ =17, t ₃ =16, and t ₄ =13 respectively. , t ₅ =21, t ₆ =18, t ₇ =17, t ₈ =20, t ₉ =14, t ₁₀ =15. Set the upper specification limit of average response time (unit: ms) USL _{average response time} = 17, so

, S _{average reaction time} =

, therefore, Cpk _{average reaction time} =

後，系統效能指標I ₂設定為： Develop process capability indicator thresholds

Afterwards, the system performance index I ₂ is set to:

，由於

，故系統效能指標I ₂

In this example, Cpk _{reaction time} = 0.073, set

,due to

, so the system performance index I ₂

In addition, in this embodiment, the system performance index needs to be between 0 and 1. According to the Cpk _{average response time} , which is between -∞ and ∞, the system performance index I ₂ can be set to the above function or other conversion function.

，由此可見，熱門評估指標之值介於0、1之間。其中選定採計時段可為特定範圍。 When calculating the popularity evaluation index, it can be configured to receive a prediction model generated based on big data, and record and calculate at least one of the number of calls predicted by this model or the popularity of other related models in the system. Since the popular evaluation indicators measure the number of calls predicted by the model, referring to the popular evaluation indicators and calculating the health based on them can more appropriately select the best model. The system will record the number of calls of all online models, and the selected acquisition period is the online to date. Comparison results The maximum value of the number of calls of all online models. The model health module calculates the popular evaluation index I ₃ of this model, which is the ratio of the number of calls predicted by this model to the above maximum value. In this embodiment, the number of calls predicted by this model is The number of calls is 1452, and the maximum number of calls for all online models is 1875, so the popular evaluation indicator

, it can be seen that the value of the popular evaluation index is between 0 and 1. The selected collection period can be a specific range.

，其中，健康度權重w ₁、w ₂、w ₃介於0、1之間，w ₁、w ₂、w ₃之總和為1，且I ₁、I ₂、I ₃亦介於0、1之間，推得模型健康度Health亦介於0、1之間。由上可知，本實施例之模型健康度權重組合(w ₁,w ₂,w ₃)=(0.51,0.19,0.30)，(I ₁,I ₂,I ₃)=(0.155,0.055,0.822)，則健康度

，顯然，健康度Health=0.34亦介於0、1之間。 Then, as in process S36, the model health module generates a health degree based on the weighted sum. According to the above embodiment, three indicators, namely the model evaluation index I ₁ , the system performance index I ₂ and the popular evaluation index I ₃ are collected to formulate the model health degree. There are three weight values w ₁ , w ₂ , and w ₃ , among which the sum of w ₁ , w ₂ , and w ₃ is 1. The health degree is generated by linear summation according to the model health weight, that is, according to the following health formula:

, where the health weights w ₁ , w ₂ , and w ₃ are between 0 and 1, the sum of w ₁ , w ₂ , and w ₃ is 1, and I ₁ , I ₂ , and I ₃ are also between 0 and 1. Between, the model health degree Health is also estimated to be between 0 and 1. It can be seen from the above that the model health weight combination of this embodiment ( w ₁ , w ₂ , w ₃ )=(0.51,0.19,0.30), ( I ₁ , I ₂ , I ₃ )=(0.155,0.055,0.822) , then the health degree

, obviously, the health degree Health =0.34 is also between 0 and 1.

This health degree Health =0.34 is the latest data. The historical data are sorted from oldest to newest: 0.21, 0.22, 0.24, 0.21, 0.25, 0.28, 0.28, 0.29, 0.32, 0.33, 0.33, 0.34.

In general, the indicators corresponding to the historical model health should reflect trends. Users can immediately grasp the health of the system and model from the latest data of the model health, and based on this, decide whether this prediction model will continue to be online to provide relevant applications. service. According to the above model health data analysis, the value has a slowly upward trend, indicating that the health of the system and model is getting better and better. This embodiment finally decided to continue to use this online model to provide application services.

This embodiment generates model health based on three indicators: model evaluation index, system performance index, and popular evaluation index. Linear summation generates model health. Since this model health reflects the three dimensions of model evaluation index, system performance index, and popular evaluation index, Compared with the traditional method that only refers to model accuracy or other single model evaluation indicators, the present invention refers to various indicators such as system performance and training model, and can more effectively and appropriately select the best model in response to complex situations, providing users with the best application services.

Then, as in process S37, the model monitoring module monitors the model according to the health degree, and uses the health degree value to monitor the health degree of the prediction model relative to the test set. The user can also use the model health degree value to understand whether the model has deviated from the latest test set, etc. status and decide whether to retain or retrain the model in the platform. At the same time, as in process S38, after the data collection module accumulates enough test sets, it will trigger the model health module and the model monitoring module to perform the next round of health calculation and model monitoring.

To sum up, the present invention formulates a model health calculation formula by summing up a variety of system and model-related important indicators according to weights, based on which the health of the model is monitored and evaluated, and the decision is made to select the prediction model or whether to retrain the model. requirements. Compared with the traditional method that only refers to model accuracy or other single model evaluation indicators, the present invention can select the best prediction model by referring to various indicators such as the system and model.

The system, module, etc. of the present invention include a microprocessor and a memory or are executed in a machine including a microprocessor and a memory. Algorithms, data, programs, etc. are stored in the memory or chip. The microprocessor Data can be loaded from the memory or algorithms or programs can be used to perform data analysis or calculations, which will not be described in detail here. For example, the model management module, data collection module, model health module and model monitoring module of the present invention include a microprocessor and a memory, and each module uses this to perform analysis operations. Therefore, the present invention The hardware details of the system can also be implemented in the same way.

The above embodiments are only illustrative to illustrate the principles and effects of the present invention, but are not intended to limit the present invention. Anyone skilled in this art can modify and change the above embodiments without departing from the spirit and scope of the invention. Therefore, the scope of protection of the present invention is It should be as listed in the patent application scope mentioned below.

S21~S25: steps

Claims (10)

A model monitoring system based on health, including: a model management module, used to generate a prediction model based on big data; a data collection module, used to collect big data test data to generate a test set; a model health module, To receive the prediction model from the model management module and the test set from the data collection module, by formulating the model health weight and calculating various indicators of the prediction model and system, the indicators are based on the model The health weights are summed to generate the health of the prediction model, in which the various indicators include model evaluation indicators, system performance indicators or popular evaluation indicators, and the model health module uses at least two of the above indicators to perform health assessment. Degree calculation; and the model monitoring module is connected to the model health module to receive the health of the prediction model, in order to monitor the prediction model and determine the status or reconstruction needs of the prediction model based on it. For example, in the health-based model monitoring system described in Item 1 of the patent application, the model health weight is formulated based on external data or is formulated by itself, or a mixture of the above two methods is adopted. A health-based model monitoring system as described in Item 1 of the patent application, wherein the model evaluation index is to evaluate the prediction model through the prediction model and the test set, using at least one method of control chart and accuracy to evaluate the prediction model. Produce model evaluation results. A health-based model monitoring system as described in item 1 of the patent application, wherein the system performance index is recorded in the system and calculated as the average response time, throughput, stability or various system performance of the prediction model. At least one to produce system performance evaluation results. A health-based model monitoring system as described in item 1 of the patent application, wherein the popularity evaluation index is one of the number of calls, number of users, service time or the popularity of the relevant model that is recorded and calculated in the system At least one, to produce a hot evaluation result. A health-based model monitoring method is executed by a computer device with a microprocessor and a memory to perform model monitoring. The method includes: the computer device provides a prediction model generated based on big data; the computer device receives The test set generated by collecting big data test data; the computer equipment formulates the model health weight and calculates various indicators of the prediction model and system; the computer equipment sums the indicators according to the model health weight to generate the The health of the prediction model, wherein the indicators include model evaluation indicators, system performance indicators or popular evaluation indicators, and the computer equipment uses at least two of the above indicators to calculate the health of the prediction model; and the computer equipment Monitor the prediction model according to the health of the prediction model, and determine the status or reconstruction needs of the prediction model. For example, in the health-based model monitoring method described in Item 6 of the patent application, the step of formulating the model health weight includes formulating the model health weight of each indicator based on external data, or formulating the model health weight by oneself. The model health weight of each indicator, or a mixture of the above two methods is used to formulate the model health weight of each indicator. The health-based model monitoring method described in Item 6 of the patent application scope, wherein the model evaluation index is to evaluate the prediction model through the prediction model and the test set, using at least one method of control chart and accuracy to evaluate the prediction model. Produce model evaluation results. The health-based model monitoring method described in item 6 of the patent application, wherein the system performance index is recorded in the system and calculated as the average response time, throughput, stability or various system performance of the prediction model At least one to produce system performance evaluation results. For example, in the health-based model monitoring method described in Item 6 of the patent application, the popularity evaluation indicator is recorded in the system and calculated among the number of calls, number of users, service time of the prediction model, or the popularity of the relevant model. At least one, to produce a hot evaluation result.

Images