TWI754446B - System and method for maintaining model inference quality - Google Patents

Abstract

The invention discloses a system and method for maintaining model inference quality. A comparison processing unit extracts an inference data set from an inference information database, compares corresponding feature fields of a training data set of an inference model and the inference data set to perform data distribution consistency verification, and determines whether the data distribution of the corresponding feature fields of the training data set of the inference model and the inference data set are consistent according to result of data distribution consistency verification. When the data distribution is inconsistent, the comparison processing unit finds a training data set with similar distribution to the inference data set from a historical training data set, deploys a trained historical model according the training data set with similar distribution to a model server unit to update or replace the inference model to provide inference services, thereby maintaining model inference quality.

Description

System and method for maintaining model inference quality

The present invention relates to a technology for model inference quality, and more particularly to a system and method for maintaining model inference quality.

Various applications of machine learning models have become more and more extensive with the improvement of data storage and commercial demand. In addition to knowledge in various industry fields, data analysts must also be familiar with various algorithms to train inference models suitable for industrial applications, and then Deploy the inference model to the server to provide inference services for the information system. Moreover, in the lengthy model supply chain, users have to face the selection of model training algorithms, the record management of model training parameters, and the comparison and update of models after they go online.

However, the training of the model in the conventional technology is mostly performed by the user selecting the algorithm, manually adjusting the parameters, repeating the training, and then selecting the best machine learning model based on the evaluation indicators, but the model training parameters and evaluation indicators are not recorded in the system. . At the same time, after the inference model is deployed and launched, when the information system is called for use, the performance of the model inference may be affected by the change of the data over time. Therefore, in addition to early warning of the insufficient inference performance of the model, there must be a method to replace or update the inference model to maintain Predictive ability for real situations.

In addition, most of the conventional techniques use the accuracy index of the inference model to determine whether the performance of the inference model meets the requirements, and the calculation of the accuracy of the inference model needs to wait for the actual value to be generated before it can be compared with the predicted value of the inference model. When the accuracy of the model decreases, the inference model is retrained with new data, and then the inference model is deployed online to provide applications. Therefore, the conventional technology is a passive process for the quality of model inference and cannot be maintained in time.

In addition, conventional technologies use model performance indicators or model health to test the performance of the inference model. When the performance of the inference model is lower than the set expectations, the inference model can only be retrained with new data, and then the inference model can be re-launched to provide inferences Serve.

Therefore, how to provide an innovative technology of model inference quality to solve one or more problems of the above-mentioned conventional technology has become a major research topic for those skilled in the art.

The present invention provides an innovative system and method for maintaining model inference quality, using the training data set of the inference model and the inference data set to perform consistent distribution comparison, or to compare the feature fields of the inference data set and the historical data set, or to improve the The timeliness of model quality judgment, or save the time of retraining the inference model.

The system for maintaining model inference quality in the present invention includes: an inference information base and a model version information base, which store an inference data set and a training data set respectively; a model server unit for deploying the inference model; and a comparison processing unit for setting A comparison period, so that when the data accumulation time of the inference data set reaches the set comparison period, the comparison processing unit retrieves the inference data set from the inference information database, so that the comparison processing unit can compare the inference model deployed in the model server unit The corresponding characteristics of the training data set and the inference data set extracted from the inference database Then, the comparison processing unit judges whether the data distribution of the corresponding feature fields of the training data set of the inference model and the inference data set are consistent according to the results of the data distribution consistency test. , when the data distribution is inconsistent, the comparison processing unit finds a training data set with a similar distribution to the inference data set from the historical training data set stored in the model version information database, and completes the training according to the similar distribution of the training data set The historical model of the model is deployed to the model server unit from the model version information database, and then the trained historical model is updated or replaced with the inference model in the model server unit to provide inference services, so as to maintain the model inference quality.

The method for maintaining model inference quality in the present invention includes: setting a comparison period for the comparison processing unit, so that when the data accumulation time of the inference data set reaches the set comparison period, the comparison processing unit retrieves the inference data set from the inference information database ; Let the comparison processing unit compare the corresponding feature fields of the training data set of the inference model deployed in the model server unit and the inference data set retrieved from the inference information database to perform a data distribution consistency check of the two, Then, the comparison processing unit judges whether the data distribution of the corresponding feature fields of the training data set of the inference model and the inference data set are consistent according to the result of the data distribution consistency check; Find a training data set with similar distribution to the inference data set from the historical training data set stored in the model version information database, so as to deploy the trained historical model from the model version information database to the model server according to the similar distribution of the training data set unit, and then update or replace the inference model in the model server unit with the historical model that has been trained to provide inference service, so as to maintain the model inference quality.

In order to make the above-mentioned features and advantages of the present invention more obvious and easy to understand, the following embodiments are given and described in detail with the accompanying drawings. Some aspects of the present invention will be set forth in the following description. Additional features and advantages will come, in part, from the description, or may be learned by practice of the present invention. It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not intended to limit the scope of the invention as claimed.

1: A system to maintain the quality of model inferences

10: Model Version Information Base

20: Compare Processing Units

30: Model servo unit

40: Model Application Unit

50: Inference Information Base

AX: training dataset

BX: Historical Data Collection

C: Inference data

CX: Inference Dataset

D: Model metadata

M: Inference model

N: Historical Model

S1 to S7: Steps

FIG. 1 is a schematic structural diagram of a system for maintaining model inference quality in the present invention; and

FIG. 2 is a schematic flowchart of the system and method for maintaining model inference quality in the present invention.

The embodiments of the present invention will be described below by means of specific specific embodiments. Those skilled in the art can understand other advantages and effects of the present invention from the contents disclosed in this specification, and can also be implemented by other specific and equivalent embodiments. or use.

FIG. 1 is a schematic structural diagram of a system 1 for maintaining model inference quality in the present invention. As shown in the figure, the system 1 for maintaining model inference quality at least includes a model version information base 10 (or model information base) that is connected or communicated with each other, a comparison processing unit 20 , and a model server unit 30 (or model inference unit) ), a model application unit 40 and an inference information base 50 .

In one embodiment, the comparison processing unit 20 may be a processor (such as a central processing unit/microprocessor), a processing chip, a processing circuit, a processing software (processing program), a comparator, a comparison circuit, a comparison software (comparing program) etc., the model server unit 30 may be a server host, a model server, etc., and the model application unit 40 may be a model application software (model application program), an application program interface, and the like. The model version repository 10 or the inference repository 50 may use a relational database Database), object-oriented database (Object-oriented Database), hierarchical database, network database and other databases to implement, or simply use the file system to implement. However, the present invention is not limited to this.

The comparison processing unit 20 sets a comparison period, so that when the data accumulation time of the inference data set CX reaches the set comparison period, the comparison processing unit 20 retrieves the inference data set CX from the inference information database 50 . The comparison processing unit 20 can also compare the corresponding feature fields of the training data set AX of the inference model M deployed in the model server unit 30 and the inference data set CX retrieved from the inference database 50 to perform the comparison between the two. In the data distribution consistency check, the comparison processing unit 20 determines whether the data distributions of the corresponding feature fields of the training data set AX and the inference data set CX of the inference model M are consistent according to the results of the data distribution consistency check. If the data distributions of the training data set AX and the inference data set CX of the inference model M are inconsistent, the comparison processing unit 20 can find the inference data set CX from the historical training data set AX stored in the model version information database 10 . The similarly distributed training data set AX is used to deploy the trained historical model N from the model version information database 10 to the model server unit 30 according to the similarly distributed training data set AX, and then the trained historical model N is updated or The inference service is provided in place of the inference model M in the model server unit 30 to maintain the model inference quality.

In other words, the model version information database 10 stores the trained historical model N (such as the historical inference model), the model metadata D, the training data set AX and the historical data set BX, etc., and the trained inference model M can be launched online. to the model server unit 30 (model inference unit) to provide inference services. The model application unit 40 transmits the inference data C (such as feature data) to the model server unit 30 through the application programming interface, so that the model server unit 30 stores the inference data C (such as the feature data) in the inference database 50 and the inference result. Reply to the model application unit 40 . Compare The processing unit 20 retrieves (eg periodically retrieves) the inference data set CX in the inference information database 50 to perform distribution comparison and data set integration of each feature data with the training data set AX and the historical data set BX in the model version information database 10 comparison, so as to decide whether to update the inference model M according to the comparison result.

After the inference model M is trained, the training data set AX (eg, the model training data set), the model metadata D and the training parameters are stored in the model version information database 10 . After the inference model M is deployed in the model server unit 30 (model inference unit), the model server unit 30 can continuously store the inference data C (such as characteristic data) to be predicted in the inference database 50, and the comparison processing unit 20 can periodically compare Historical training dataset AX (eg model training dataset). If the distribution of the data set used in the training of the inference data C (such as the feature data) to be predicted and the inference model M (such as the current inference model) is inconsistent, it means that the inference model M (such as the current inference model) must be updated, so The inference model M trained from a data set of suitable or similar distribution can be extracted from the model version information database 10 and deployed to the model server unit 30 (model inference unit). Therefore, the present invention can actively determine whether the inference model M (such as the current inference model) is suitable for the current inference data C (such as the characteristic data) without waiting for the actual value to be generated before calculating the difference between the actual value and the predicted value. At the same time, the present invention can reuse the trained historical model N (eg, historical inference model) without retraining the inference model M and deploy it in the model servo unit 30 in time. Moreover, the present invention actively judges the inference quality of the model and maintains it in time, which is also beneficial to save time for retraining the inference model M.

Since the inference model M is trained on the characteristic data of the training data set AX, if the inference data C (such as the characteristic data) to be inferred is different from the distribution of the training data set AX (that is, the data distribution is inconsistent), the inference model M is The inferred predicted value must deviate from the actual value (ie, the prediction is inaccurate and the accuracy rate is reduced). Therefore, the present invention can compare training materials The characteristic distribution of data and inference data C (such as characteristic data) can be used to judge whether the inference model M is suitable for new data, and then the conventional technology can be improved to use the predicted value and actual value to calculate the model accuracy rate or health related indicators to judge the quality of the inference model. Way.

After the inference model of the conventional technology is trained, if the performance index of the inference model reaches a certain or acceptable threshold, the inference model will be deployed and the online prediction service will be provided. However, the data used for training the model is based on the accumulated data collected in the past. , as the inference prediction continues to occur, the new data also continues to enter, but whether the inference model is suitable for the new data can only be known by comparing the actual value corresponding to the new data with the predicted value. For example, in the conventional technology, the factory predicts the tolerance of the finished product through the characteristic data of the processing machine, but it takes a period of time to process, and the error between the actual tolerance and the predicted tolerance of the finished product is not known until the actual tolerance is measured. , the model was replaced when it was found that it exceeded the allowable value. The production line of the factory has produced many semi-finished products with inconsistent tolerances, and a large amount of production costs (including materials and production time) have been accumulated accumulatively. Therefore, the present invention utilizes the data distribution test method to provide the ability to test whether the inference model M is suitable for new data without waiting for the actual value, thereby improving the timeliness of judging whether the inference model M is suitable. At the same time, the present invention tests the distribution of the inference model M and the training data. If the data distribution is inconsistent, it means that the inference result will have errors, that is, the quality of the inference model M will vary, and the training data needs to be searched from the historical model N (such as the historical inference model). Models that match the distribution of the inferred data are replaced to maintain the quality of the inferred model M.

The training process of the inference model is quite cumbersome and time-consuming. It is often necessary to repeatedly use different training data sets to find the model coefficients with the best performance. However, the conventional technology only stores the health index records of the trained model, so it can only remind The user achieves the warning that the accuracy rate drops, and cannot automatically find a historical model suitable for the inference data and further replace the inference model. therefore, The present invention compares the characteristic distribution of the historical training data set AX and the inference model M, and when the data characteristic difference of the inference model M currently deployed in the model server unit 30 is too large, the distribution can be found from the historical training data set AX For a similar trained inference model M, the trained inference model M is automatically deployed in the model server unit 30 and an online service is provided, so as to reduce the time cost of retraining the inference model M. In addition, after the inference model M is deployed and applied in the model server unit 30, the model server unit 30 of the present invention continuously captures the inference data C (such as characteristic data) to be predicted and records it in the inference information database 50, so as to periodically communicate with Compared with the historical training data set AX, if the accuracy of the inference model M drops below the threshold value, and the inference data C (such as feature data) to be predicted is similar to the past data set, it is extracted from the model version information database 10 The inference model M trained from a similar (suitable) data set is directly deployed to the model server unit 30, and the historical model N (eg, the historical inference model) can be reused without retraining.

FIG. 2 is a schematic flowchart of the system 1 and the method for maintaining model inference quality in the present invention, and is described with reference to FIG. 1 .

As shown in step S1 of FIG. 2 , the comparison period, the historical data interval and the threshold value (including the frequency threshold value) are set. The comparison processing unit 20 sets a comparison period in advance, and as the inference data C (such as characteristic data) is continuously stored in the inference information database 50, when the data accumulation time of the inference data set CX or the inference data C (such as the characteristic data) reaches the comparison period , the comparison processing unit 20 performs the comparison of the characteristic data through the subsequent steps. Since the inference model M will record the training data set AX in the model version information base 10 (model information base) during training, all comparisons require a lot of time, so the comparison processing unit 20 needs to set the comparison historical data interval first, so as to The set historical data interval finds the history most similar to the inference model M from the historical training data set AX training data. In addition, the comparison processing unit 20 needs to set a threshold value for comparison, that is, a significant level of whether the accepted data are consistent. The higher the threshold value, the higher the probability of rejecting the data distribution is consistent, and the lower the threshold value, the rejecting data distribution is consistent. The lower the probability is, in one embodiment, the threshold value is selected according to the degree of rigor, such as 0.05, 0.01, 0.001, etc., and the threshold value can be set according to the general principle of statistical verification.

As shown in step S2 of FIG. 2 , the training data set AX and the model metadata D are stored in the model version information base 10 . After the inference model M is trained, the training data set AX and model metadata D (such as model creation time, model training algorithm, model labels, model features, model evaluation indicators, trained models, etc.) are stored in the model version information The database 10 is used for subsequent query of the historical model N (eg, historical inference model), and the model version information database 10 can be implemented by using various types of databases suitable for the market. In addition, different from the prior art that only stores the model metadata of the inference model after training and does not store the training data set, the present invention also stores the training data set AX in the model version information database 10, and only needs to query and compare in the follow-up. The inference model M with similar data distribution can be deployed online in the model server unit 30 , and there is no need to retrain the inference model M, which can save time.

As shown in step S3 of FIG. 2 , the model server unit 30 (model inference unit) stores the inference data C (eg, characteristic data) to the inference information database 50 . The model server unit 30 can continuously store a plurality of inference data C (such as characteristic data) in the inference information database 50 to form the inference data CX. The inference data C is the characteristic data sent by the user to the model server unit 30 through the model application unit 40 . The model servo unit 30 then performs prediction inference. The purpose of storing inference data C (such as feature data) is to provide training data and historical models for follow-up and inference model M (such as the current inference model). The training data of N (such as historical inference model) are compared, so the present invention does not need to wait for the actual value to occur before comparing with the predicted result, but only needs to use the inference data C (such as characteristic data), so it is faster and more efficient than the conventional technology. Simple.

As shown in step S4 of FIG. 2 , the historical data set BX of the set interval and the inference data set CX in the inference information database 50 are extracted. When the data accumulation time of the inference data set CX reaches the comparison period set in the above step S1, the comparison processing unit 20 can retrieve the inference data set CX stored (periodically stored) in the above step S3 from the inference information database 50, The characteristic fields (such as each characteristic field) of the historical data set BX of the historical data interval set in the above step S1 can also be extracted from the model version information database 10, and then the comparison processing unit 20 compares the inference data set CX with The characteristic fields of both historical data sets BX (eg each characteristic field).

Because the distribution of each inference data (characteristic data) is different from each other, that is, not all the inference data C (such as characteristic data) belong to the normal distribution, and cannot be verified at the same time. Therefore, the present invention proposes a two-stage (or two-step) verification. way to solve this problem, as described in the following steps S5 and S6 in FIG. 2 .

As shown in step S5 of FIG. 2 , the distribution of each characteristic data is compared. The comparison processing unit 20 compares the corresponding features of the training data set AX of the inference model M (such as the current inference model) in the model servo unit and the inference data set CX extracted from the inference database 50 in the above step S4. field to carry out the data distribution consistency test (such as KS (Kolmogorov-Smirnov; Kolmogorov-Smirnov) test), and then the comparison processing unit 20 judges based on the results of the data distribution consistency test Or know whether the data distributions of the corresponding feature fields of the training data set AX of the inference model M (such as the current inference model) and the inference data set CX are consistent. The comparison processing unit 20 performs a data distribution consistency test (such as KS test) on the corresponding feature fields of the training data set AX and the inference data set CX of the inference model M, and obtains the _pi of the corresponding feature fields of the two value (such as a statistic), where i is a positive integer. If the p _i value (such as the statistical value) is less than the threshold value set in the above step S1, it means that the data distributions of the corresponding feature fields of the two are inconsistent; on the contrary, if the p _i value (such as the statistical value) is greater than or equal to the threshold value, indicating that the data distributions of the corresponding feature fields of the two are consistent.

Next, the comparison processing unit 20 performs a data distribution consistency check (eg KS test), so that the comparison processing unit 20 judges (learns) whether the data distributions of the corresponding feature fields of the historical training data set AX and the inference data set CX are consistent according to the result of the data distribution consistency test. If the p _i value (such as the statistical value) is less than the threshold value, it means that the data distributions of the corresponding feature fields of the two are inconsistent; on the contrary, if the p _i value (such as the statistical value) is greater than or equal to the threshold value, it means that the two correspond to each other. The distribution of data in the characteristic fields is consistent. However, the present invention is not limited to the use of the KS test method, other methods of data distribution consistency test, such as Mann-Whitney (Mann-Whitney; MW for short), Kruskal-Wallis (Klaska-Wallis; KW for short) ...and other data distribution verification methods can also be used.

As shown in step S6 of FIG. 2 , the data sets are integrated and compared. In the above step S5, the comparison processing unit 20 compares the consistency of the distribution of each feature data with respect to the inference model M (such as the current inference model) with the historical training data set AX and inference data set CX, and only knows each feature field at this time. Statistical value of the data distribution. Therefore, in step S6, the comparison processing unit 20 checks all the feature fields, and by obtaining the p _i values (such as statistical values) of the distribution consistency of each feature field in step S5, the comparison processing unit 20 then compares the inferences The p _i value (such as statistical value) and the threshold value of the corresponding feature fields of the training data set AX and the inference data set CX of the model M (such as the current inference model) are calculated to calculate the number of times that p _i is smaller than the threshold value, and then carry out The number of tests (such as binomial distribution test) to obtain the value of p _j (such as statistical times), where j is a positive integer. If p _j (such as the number of statistics) is less than the threshold value (such as the threshold value of times) set in the above step S1, it represents the whole of the training data set AX and the inference data set CX of the inference model M (such as the current inference model). The data distribution is inconsistent; on the contrary, if the value of p _j (such as the number of statistics) is greater than or equal to the threshold value (such as the threshold value of the number of times), it means that the training data set AX and the inference data set CX of the inference model M (such as the current inference model) The overall data distribution of the two is consistent. However, the present invention is not limited to the use of the binomial distribution test method, and other times test methods, such as symbol test, Wilcoxon symbol level test, etc., can also be used.

As can be seen from the description of the above steps S3 to S6, the present invention only needs the inference data C (such as characteristic data) to be inferred, and does not need to wait for the actual value. Therefore, the present invention can improve the conventional technology and rely on the calculation of the model predicted value and the actual value. The shortcoming of accuracy can also improve the timeliness in the judgment of inference quality.

As shown in step S7 of FIG. 2 , the inference model M is updated. If the data distributions of the training data set AX and the inference data set CX of the inference model M (such as the current inference model) are inconsistent, the comparison processing unit 20 selects the data from the historical training data set AX stored in the model version information database 10 . Find a training data set AX with a similar distribution to the inference data set CX, so as to deploy the trained historical model N (such as a historical inference model) from the model version information database 10 to the model server unit according to the similarly distributed training data set AX 30 (model inference unit), and then update or replace the inference model M (such as the current inference model) in the model server unit 30 with the trained historical model N to provide inference services to maintain the model inference quality. Therefore, the present invention can deploy a trained historical model N (such as historical Inference model), there is no need to retrain the inference model M with the newly advanced model, which is beneficial to save the time of retraining the model while maintaining the inference quality of the model.

Two examples are provided below to illustrate them.

The first embodiment: the conventional technology uses model performance indicators or model health to test the performance of the inference model. When the performance of the inference model is lower than the set expectations, the inference model can only be retrained with new data, and then the inference model can be re-launched to provide inference services. Relatively, the present invention can directly compare the distribution of the past historical training data and the new data. If the data distribution is consistent, the inference model M that has been trained can be directly obtained from the model version information database 10 (model database) to provide the data. On-line service, thereby saving the time for data scientists to retrain the inference model M.

The first embodiment assumes that a data scientist will train an inference model M for housing price prediction, and after performing multiple model trainings using a regression algorithm, provide inference services for the inference model M (eg, a regression prediction model).

As shown in step S1 of FIG. 2 , a comparison period, a historical data interval and a threshold value are set. Before comparing the inference data set CX with the training data set AX of the inference model M (such as the current inference model) and the historical data set BX (such as all historical data sets), it is necessary to set the comparison period, historical data interval and threshold value. For example, in the first embodiment, the comparison period is set so that the comparison is triggered when the inference model M accumulates 20 records; since there are many historical data sets BX, it is relatively time-consuming to compare one by one, so it is necessary to set the historical data interval for comparison in advance. The first embodiment Set and compare the historical data of the previously trained inference model M; then, set a threshold value, for example, the threshold value is 0.05.

As shown in step S2 of FIG. 2 , the training data set AX and the model metadata D are stored in the model version information base 10 . For example, the first embodiment will train an inference model M for housing price prediction, and there are twelve items of data on factors affecting housing prices, which are marked as X ₁ , X ₂ , X ₃ , X ₄ , X ₅ , X ₆ , X ₇ , X ₈ , X ₉ , X ₁₀ , X ₁₁ , X ₁₂ , so an inference model M is established based on the collected data. The data are shown in Table 1 below.

Table 1:

Corresponding house price AY=[24,21.6,34.7,33.4,36.2,28.7,22.9,27.1, 16.5, 18.9, 15, 18.9, 21.7, 20.4, 18.2, 19.9, 23.1, 17.5, 20.2, 18.2], the trained inference model M is called the inference model M1. The data scientist stores another trained inference model (called the inference model M2 ) in the model version database 10 , assuming that the inference model M2 is trained by machine learning through the data shown in Table 2 below. After the training process is completed, the inference model M1 is used in the model server unit 30 to provide the inference service online.

Table 2:

As shown in step S3 of FIG. 2 , the model server unit 30 (the model inference unit) can store the inference data C (eg, characteristic data) to the inference database 50 . After the inference model M1 is launched in the model server unit 30, the inference data C (such as characteristic data) are successively entered into the model server unit 30 through the model application unit 40 for inference, and these inference data C (such as characteristic data) are also continuously collected. The data are stored in the inference information database 50, and the data are listed in Table 3 below.

table 3:

As shown in step S4 of FIG. 2 , the historical data set BX of the set historical data interval is extracted from the model version information database 10 , and the current inference data set CX is extracted from the inference information database 50 . For example, in the first embodiment, the training data set AX of the on-line inference model M is [AX ₁ , AX ₂ , AX ₃ , AX ₄ , AX ₅ , AX ₆ , AX ₇ , AX ₈ , AX ₉ , AX ₁₀ ,AX ₁₁ ,AX ₁₂ ]; historical data set BX is [BX ₁ ,BX ₂ ,BX ₃ ,BX ₄ ,BX ₅ ,BX ₆ ,BX ₇ ,BX ₈ ,BX ₉ ,BX ₁₀ ,BX ₁₁ ,BX ₁₂ ] The inference data set CX is [CX ₁ , CX ₂ , CX ₃ , CX ₄ , CX ₅ , CX ₆ , CX ₇ , CX ₈ , CX ₉ , CX ₁₀ , CX ₁₁ , CX ₁₂ ] and other data.

As shown in step S5 of FIG. 2 , the distribution of each characteristic data is compared. The comparison processing unit 20 can retrieve the characteristic fields corresponding to the data sets and check whether their data distributions are similar, compare the training data set [AX _i ] and the inference data set [CX _i ], and pass the data distribution consistency check (eg, KS verification) p-values were obtained and the results are reported in Table 4 below.

Table 4:

The data distribution consistency test (eg, KS test) of the present invention can be obtained by using various statistical software or by program calculation. If the p value obtained by the data distribution consistency test (such as the KS test) is less than the threshold value (ie, the significant level), for example, the threshold value is 0.05, it means that the data distributions of the two mothers are inconsistent. In the first embodiment, the distribution of ten of the twelve features of the training data set AX of the inference model M and the inference data set CX is inconsistent, and it is determined that the phenomenon of feature drift occurs, that is, the inference that is currently online. Model M is no longer suitable for the current inference dataset CX.

Next, check the historical data set BX such as [BX ₁ , BX ₂ , BX ₃ , BX ₄ , BX ₅ , BX ₆ , BX ₇ , BX ₈ , BX ₉ , BX ₁₀ , BX ₁₁ , BX ₁₂ ] and the inference data set CX Such as [CX ₁ , CX ₂ , CX ₃ , CX ₄ , CX ₅ , CX ₆ , CX ₇ , CX ₈ , CX ₉ , CX ₁₀ , CX ₁₁ , CX ₁₂ ], the p-value results are shown in Table 5 below . In Table 5, after the twelve features are compared, all p-values are not less than 0.05, indicating that the data distributions of the twelve features of the inference data set CX and the historical data set BX are consistent.

table 5:

，透過計算或查表(如下表6)可得SB~B(12,2,0.5)=0.019281<0.05，因此可得知推論模型M(如目前的推論模型)的訓練資料集AX與推論資料集CX的各相對應特徵資料的分佈是不一致的，確定產生資料飄移的現象。而且，歷史資料集BX與推論資料集CX的特徵資料經資料分佈一致性檢定(如KS檢定)後，p值大於0.05的次數t=12，總特徵數n=12，計算統計值SB為二項分配SB~B(12,12,0.5)，其累積機率分佈為

q^n-x，透過計算或查表(如下表6)可得SB~B(12,12,0.5)=1>0.05，因此可得知歷史資料集BX與推論資料集CX的整體資料(各相對應特徵資料)的分佈是趨於一致的。 As shown in step S6 of FIG. 2 , the data sets are integrated and compared. Count and test the number of all feature fields. Since the training data set AX of the inference model M (such as the current inference model) and the feature data of the inference data set CX are tested for the consistency of the data distribution (such as the KS test), the p value The number of times greater than 0.05 is t=2, the total number of features is n=12, and the calculated statistical value SB is the binomial distribution SB~B(12,2,0.5), and its cumulative probability distribution is

, SB~B(12,2,0.5)=0.019281<0.05 can be obtained by calculation or table lookup (see Table 6 below), so the training data set AX and inference data of the inference model M (such as the current inference model) can be known The distribution of the corresponding characteristic data of the set CX is inconsistent, and the phenomenon of data drift is determined. Moreover, after the characteristic data of the historical data set BX and the inference data set CX are tested for the consistency of the data distribution (such as the KS test), the number of times the p value is greater than 0.05 is t=12, the total number of features is n=12, and the calculated statistical value SB is two The term is assigned SB~B(12,12,0.5), and its cumulative probability distribution is

q ^nx , SB~B(12,12,0.5)=1>0.05 can be obtained by calculation or table lookup (see Table 6 below), so the overall data of historical data set BX and inference data set CX can be known (each corresponding to The distribution of characteristic data) tends to be consistent.

Table 6:

As shown in step S7 of FIG. 2 , the inference model M is updated. Because the data distribution of the training data set [AX _i ] of the inference model M (such as the current inference model) is inconsistent with the data distribution of the inference data set [CX _i ], and the historical data set [BX _i ] and the inference data set [CX _i ] have different data distributions. The distribution of the corresponding feature data tends to be consistent, and the current inference data C (such as feature data) is more suitable for inference using the inference model M2 trained with historical data. Therefore, the inference model M2 is deployed from the model version information database 10 to the model The servo unit 30 (model inference unit) replaces the inference model M1, thereby completing the updating of the inference model M and providing it to the model application unit 40 for calling use.

According to the training data set AX such as [AX ₁ , AX ₂ , AX ₃ , AX ₄ , AX ₅ , AX ₆ , AX ₇ , AX ₈ , AX ₉ , AX ₁₀ , AX ₁₁ , AX ₁₂ ], the inference model M1 is Y _ModeA =21.34277+0.168947X ₁ +0.312266X ₂ +0.005807X ₃ +0X ₄ +8.46927X ₅ -0.06841X ₆ -2.80617X ₇ +3.069507X ₈ -0.14161X ₉ +0X _{10 -0.0087X} 12.0, ₁₁ -0. Historical data set BX such as [BX ₁ , BX ₂ , BX ₃ , BX ₄ , BX ₅ , BX ₆ , BX ₇ , BX ₈ , BX ₉ , BX ₁₀ , BX ₁₁ , BX ₁₂ ] The inference model M2 is Y _ModeB =50.18587+136.6193X ₁ -0.02561X ₂ -0.31037X ₃ -36.7141X ₄ +10.07272X ₅ -0.0793X ₆ -0.87524X ₇ +0.401028X ₈ -0.0216X _{9 -0.47391X 10 -1280.113} , the current new inference data set CX such as [CX ₁ , CX ₂ , CX ₃ , CX ₄ , CX ₅ , CX ₆ , CX ₇ , CX ₈ , CX ₉ , CX ₁₀ , CX ₁₁ , CX ₁₂ ] to infer the model The evaluation index such as MAPE (mean absolute percentage error; mean absolute percentage error) obtained by inference of new data in M1 was 0.4742, while the MAPE (mean absolute percentage error) obtained by inference model M2 was 0.1459. Therefore, it is actually verified that the MAPE (Mean Absolute Percent Error) of replacing the inference model M with the trained historical model N does indeed improve.

The first embodiment uses inference data C (such as characteristic data) to test the data distribution. If the data distribution is inconsistent, it means that the inference model M is not suitable for the current inference data C (such as characteristic data), and there is no need to wait for the real actual data to calculate the model. The correct rate is to replace the model. In addition, the present invention selects a model suitable for the current inference data distribution in the historical model N that has been trained in the past to directly update and deploy the inference model M without retraining a new model, which saves the cost of training the model and has timeliness.

Second Embodiment: The inference model M of the present invention can be used not only for regression prediction models (see the first embodiment), but also for binary classification models assuming that data scientists will train customer financial risk classification predictions (see the second embodiment) ), but not limited thereto.

As shown in step S1 of FIG. 2 , a comparison period, a historical data interval and a threshold value are set. Before comparing the inference data set CX with the training data set AX of the inference model M (such as the current inference model) and the historical data set BX (such as all historical data sets), it is necessary to set the comparison period, historical data interval and threshold value. For example, in the second embodiment, the comparison period is set to the inference mode Model M accumulates 20 records to trigger the comparison; since there are many historical data sets BX, it is relatively time-consuming to compare one by one, so it is necessary to set the historical data interval for comparison first, and the second embodiment sets and compares the historical data of the previously trained inference model; then , set the threshold value, for example, the threshold value is 0.05.

As shown in step S2 of FIG. 2 , the training data set AX and the model metadata D are stored in the model version information base 10 . For example, the second embodiment will train an inference model M for predicting customer financial risk classification, and the factors affecting the risk classification are eleven items of data collected from customer data, marked as X ₁ , X ₂ , X ₃ , X ₄ respectively , X ₅ , X ₆ , X ₇ , X ₈ , X ₉ , X ₁₀ , X ₁₁ , so an inference model M is established based on the collected data. The data are shown in Table 7 below.

Table 7:

The corresponding risk classification AY=[1,0,0,1,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0], the training is completed The inference model M is called the inference model M1. The data scientist stores another trained inference model M (called the inference model M2 ) in the model version database 10 . It is assumed that the inference model M2 is trained by machine learning through the data shown in Table 8 below. After the training process is completed, the inference model M1 is used in the model server unit 30 to provide the inference service online.

Table 8:

As shown in step S3 of FIG. 2 , the model server unit 30 (the model inference unit) can store the inference data C (eg, characteristic data) to the inference database 50 . After the inference model M1 goes online in the model server unit 30, inference data C (such as feature data) successively enters the model server unit 30 through the model application unit 40 for inference, and these inference data C (such as feature data) also continue to The data are collected and stored in the inference database 50, and the data are listed in Table 9 below.

Table 9:

As shown in step S4 of FIG. 2 , the historical data set BX of the set historical data interval is extracted from the model version information database 10 , and the current inference data set CX is extracted from the inference information database 50 . For example, in the second embodiment, the training data set AX of the on-line inference model M is [AX ₁ , AX ₂ , AX ₃ , AX ₄ , AX ₅ , AX ₆ , AX ₇ , AX ₈ , AX ₉ , AX ₁₀ ,AX ₁₁ ]; the historical data set BX is [BX ₁ , BX ₂ , BX ₃ , BX ₄ , BX ₅ , BX ₆ , BX ₇ , BX ₈ , BX ₉ , BX ₁₀ , BX ₁₁ ]; the inference data set CX is [CX ₁ , CX ₂ , CX ₃ , CX ₄ , CX ₅ , CX ₆ , CX ₇ , CX ₈ , CX ₉ , CX ₁₀ , CX ₁₁ ] and other information.

As shown in step S5 of FIG. 2 , the distribution of each characteristic data is compared. The comparison processing unit 20 performs regional data comparison, extracts the characteristic fields corresponding to the data sets and checks whether the data distributions are similar, compares the training data set [AX _i ] and the inference data set [CX _i ], and checks the consistency of the data distribution through the data distribution consistency check ( p-values were obtained as approved by KS) and the results are reported in Table 10 below.

Table 10:

The data distribution consistency test (eg, KS test) of the present invention can be obtained by using various statistical software or by program calculation. If the p value obtained by the data distribution consistency test (such as the KS test) is less than the threshold value (ie, the significant level), for example, the threshold value is 0.05, it means that the data distributions of the two mothers are inconsistent. In the second embodiment, ten of the eleven features of the training data set AX of the online inference model M and the inference data set CX are inconsistent in distribution, and it is determined that the phenomenon of feature drift occurs, that is, the inference that is currently online Model M is no longer suitable for the current inference dataset CX.

Next, check the historical data set BX such as [BX ₁ , BX ₂ , BX ₃ , BX ₄ , BX ₅ , BX ₆ , BX ₇ , BX ₈ , BX ₉ , BX ₁₀ , BX ₁₁ ] and the inference data set CX such as [CX ₁ , CX ₂ , CX ₃ , CX ₄ , CX ₅ , CX ₆ , CX ₇ , CX ₈ , CX ₉ , CX ₁₀ , CX ₁₁ ], the p-value results are shown in Table 11 below. In Table 11, after the eleven features are compared, only one feature's p-value is less than 0.05, indicating that the data distributions of the eleven features of the inference data set CX and the historical data set BX are consistent.

Table 11:

，透過計算或查表(如下表12)可得SB~B(11,1,0.5)=0.005859<0.05，因此可得知推論模型M(如目前的推論模型)的訓練資料集AX與推論資料集CX的整體資料分佈是不一致的，確定產生資料飄移的現象。而且，歷史資料集BX與推論資料集CX的特徵資料經資料分佈一致性檢定(如KS檢定)後，p值大於0.05的次數t=10，總特徵數n=11，計算統計值SB為二項分配SB~B(11,10,0.5)，其累積機率分佈為

，透過計算或查表(如下表12)可得SB~B(11,10,0.5)=0.99915>0.05，因此可得 知歷史資料集BX與推論資料集CX的整體資料(各相對應特徵資料)的分佈是趨於一致的。 As shown in step S6 of FIG. 2 , the data sets are integrated and compared. Count and test the number of all feature fields. Since the training data set AX of the inference model M (such as the current inference model) and the feature data of the inference data set CX are tested for the consistency of the data distribution (such as the KS test), the p value The number of times greater than 0.05 is t=1, the total number of features is n=11, and the calculated statistical value SB is the binomial distribution SB~B(11,1,0.5), and its cumulative probability distribution is

, SB~B(11,1,0.5)=0.005859<0.05 can be obtained by calculation or table lookup (see Table 12 below), so the training data set AX and inference data of the inference model M (such as the current inference model) can be obtained The overall data distribution of set CX is inconsistent, identifying the phenomenon that produces data drift. Moreover, after the characteristic data of the historical data set BX and the inference data set CX are tested for the consistency of the data distribution (such as the KS test), the number of times the p value is greater than 0.05 is t=10, the total number of features is n=11, and the calculated statistical value SB is two The term is assigned SB~B(11,10,0.5), and its cumulative probability distribution is

, SB~B(11,10,0.5)=0.99915>0.05 can be obtained by calculation or table lookup (see Table 12 below), so the overall data of historical data set BX and inference data set CX (each corresponding characteristic data can be obtained) ) distribution tends to be consistent.

Table 12:

As shown in step S7 of FIG. 2 , the inference model M is updated. Because the data distribution of the training data set [AX _i ] of the inference model M (such as the current inference model) is inconsistent with the data distribution of the inference data set [CX _i ], and the historical data set [BX _i ] and the inference data set [CX _i ] have different data distributions. The distribution of the corresponding feature data tends to be consistent, and the current inference data C (such as feature data) is more suitable for inference using the inference model M2 trained with historical data. Therefore, the inference model M2 is deployed from the model version information database 10 to the model The servo unit 30 (model inference unit) replaces the inference model M1, thereby completing the updating of the inference model M and providing it to the model application unit 40 for calling use.

According to the training data set AX such as [AX ₁ , AX ₂ , AX ₃ , AX ₄ , AX ₅ , AX ₆ , AX ₇ , AX ₈ , AX ₉ , AX ₁₀ , AX ₁₁ ], the inference is trained by the Loggers classification algorithm Model M1, historical data set BX such as [BX ₁ , BX ₂ , BX ₃ , BX ₄ , BX ₅ , BX ₆ , BX ₇ , BX ₈ , BX ₉ , BX ₁₀ , BX ₁₁ ] to train the inference model M2, which is currently new The inference data set CX is such as [CX ₁ , CX ₂ , CX ₃ , CX ₄ , CX ₅ , CX ₆ , CX ₇ , CX ₈ , CX ₉ , CX ₁₀ , CX ₁₁ ], and the inference model M1 is used to infer the new data The classification model accuracy evaluation index (Category accuracy; CA) is 0.7, and the CA obtained by classifying the new data with the inference model M2 is 0.8. Therefore, it is actually verified that the accuracy of replacing the inference model M with the trained historical model N is indeed improved.

The second embodiment describes whether the inference model M of the currently online service is suitable for the current inference data C (such as feature data) by comparing the distribution of data set features, and the inference quality of the model can be judged without waiting for the actual label data. If the training data set of the online inference model M is inconsistent with the characteristic data distribution of the inference data set CX, it means that the inference quality of the model has deteriorated, and the inference model M needs to be updated. In addition, the present invention can find an inference model M suitable for the current inference data C (such as characteristic data) from the trained historical model N. If the inference data is consistent with the training data distribution of the historical model N, the historical model N can be placed in The online service is deployed in the model server unit 30 without retraining a new model, so as to save time and training cost.

To sum up, the system and method for maintaining the quality of model inference in the present invention have at least the following features, advantages or technical effects.

1. The present invention uses the training data set of the inference model to compare the distribution of the inference data set to determine whether the inference model is suitable for the distribution of the new data, which is different from the conventional technology to judge the health of the inference model by the accuracy index .

2. The present invention is a (periodic) comparison of the characteristic column of the inference data set and the historical data set Bit, and the comparison method only needs to infer the data (characteristic data), and does not need to wait for the actual value, so it is faster and easier.

3. Because the distribution of each inference data (characteristic data) is different from each other, that is, not all inference data (characteristic data) belong to the normal distribution and cannot be verified at the same time, so the present invention proposes a two-stage (or two-step) verification method to resolve this issue.

Fourth, the present invention improves the disadvantage of the conventional technology that after the model goes online, the actual data needs to be recovered to judge the error degree of the model, and the timeliness of judging the quality of the model can be improved.

5. The present invention extracts an inference model suitable for new data from a historical training data set (such as a model training data set), and the inference model can be updated without retraining the inference model, saving the time of retraining the inference model.

6. The industries to which the present invention is applied are various industries such as manufacturing, finance, and service industries; meanwhile, the products to which the present invention is applied are various products such as machine learning platforms and deep learning platforms, but are not limited thereto.

The above-mentioned embodiments are only illustrative of the principles, features and effects of the present invention, and are not intended to limit the applicable scope of the present invention. Modifications and changes are made to the implementation form. Any equivalent changes and modifications made by using the contents disclosed in the present invention should still be covered by the scope of the patent application. Therefore, the protection scope of the present invention should be listed in the scope of the patent application.

1: A system to maintain the quality of model inferences

10: Model Version Information Base

20: Compare Processing Units

30: Model servo unit

40: Model Application Unit

50: Inference Information Base

AX: training dataset

BX: Historical Data Collection

C: Inference data

CX: Inference Dataset

D: Model metadata

M: Inference model

N: Historical Model

Claims (16)

A system for maintaining model inference quality, comprising: an inference information base and a model version information base, respectively storing an inference data set and a training data set; a model server unit for deploying an inference model; and a comparison processing unit for setting comparison a period, so that when the data accumulation time of the inference data set reaches the set comparison period, the comparison processing unit retrieves the inference data set from the inference information database for the comparison processing unit to compare the model servo unit The corresponding feature fields of the training data set of the inference model deployed in the inference model and the inference data set extracted from the inference information database are used to check the data distribution consistency between the two, and then the comparison processing unit According to the result of the data distribution consistency check, it is judged whether the data distribution of the corresponding feature fields of the training data set of the inference model and the inference data set are consistent. From the historical training data set stored in the model version information base, find a training data set with a similar distribution to the inference data set, so as to extract the trained historical model from the model version information base according to the similarly distributed training data set Deploying to the model server unit, and then updating or replacing the inference model in the model server unit with the historical model that has been trained to provide inference services to maintain model inference quality, wherein the comparison processing unit further sets a historical data interval , so as to retrieve the feature fields of the historical data set of the set historical data interval from the model version information database, so that the comparison processing unit compares the feature fields of the inference data set and the historical data set. The system of claim 1, wherein the inference repository or the model version repository is implemented using a relational database, an object-oriented database, a hierarchical database, a network-based database, or a file system . The system of claim 1, wherein the model version information base is used for storing historical data sets, training data sets and model metadata, and the inference data set stored in the inference information base consists of a plurality of inference data sets composition. The system of claim 1, wherein the comparison processing unit uses the KS (Kolmogorov-Smirnov) test method, the Mann-Whitney (Mann-Whitney) test method, or the Kruskal- The Wallis (Clarska-Wallis) test method is used to perform the data distribution consistency test with the corresponding feature fields of both the training data set of the inference model and the inference data set. A system for maintaining model inference quality, comprising: an inference information base and a model version information base, respectively storing an inference data set and a training data set; a model server unit for deploying an inference model; and a comparison processing unit for setting comparison a period, so that when the data accumulation time of the inference data set reaches the set comparison period, the comparison processing unit retrieves the inference data set from the inference information database for the comparison processing unit to compare the model servo unit The corresponding feature fields of the training data set of the inference model deployed in the inference model and the inference data set extracted from the inference information database are used to check the data distribution consistency between the two, and then the comparison processing unit According to the result of the data distribution consistency check, it is judged whether the data distribution of the corresponding feature fields of the training data set of the inference model and the inference data set are consistent. From the historical training data set stored in the model version information base, find a training data set with a similar distribution to the inference data set, so as to extract the trained historical model from the model version information base according to the similarly distributed training data set Deploy to the model server unit, and then update or replace the inference model in the model server unit with the historical model that has been trained to provide inference services, so as to maintain the model inference quality, Wherein, the comparison processing unit further sets a threshold value, so that the comparison processing unit obtains the data distribution consistency check on the corresponding feature fields of the training data set of the inference model and the inference data set by the comparison processing unit. The statistical value of the corresponding characteristic field of the The data distribution of the corresponding feature fields of the authors is consistent. The system according to claim 5, wherein the comparison processing unit further calculates the number of times the statistic value is smaller than the threshold value to perform a frequency test to obtain the statistic number, if the statistic number is less than the number of times the threshold value, it means the training of the inference model The overall data distribution of the data set and the inference data set is inconsistent. On the contrary, if the number of statistics is greater than or equal to the threshold of the number of times, it indicates the overall data distribution of the training data set of the inference model and the inference data set. is consistent. The system of claim 6, wherein the comparison processing unit uses a binomial distribution test method, a sign test method, or a Wilcoxon (Weiksen) sign level test method to perform the number of tests to obtain the statistical times. A system for maintaining model inference quality, comprising: an inference information base and a model version information base, respectively storing an inference data set and a training data set; a model server unit for deploying an inference model; and a comparison processing unit for setting comparison a period, so that when the data accumulation time of the inference data set reaches the set comparison period, the comparison processing unit retrieves the inference data set from the inference information database for the comparison processing unit to compare the model servo unit The corresponding feature fields of the training data set of the inference model deployed in the inference model and the inference data set extracted from the inference information database are used to check the data distribution consistency between the two, and then the comparison processing unit According to the The result of the data distribution consistency check determines whether the data distributions of the corresponding feature fields of the inference model training data set and the inference data set are consistent. Find a training data set with a similar distribution to the inference data set from the historical training data set stored in the version information base, and deploy the trained historical model from the model version information base to the model version information set according to the similarly distributed training data set. The model server unit further updates or replaces the inference model in the model server unit with the historical model that has been trained to provide inference services to maintain model inference quality, wherein the comparison processing unit is more accurate to the historical training data The data distribution consistency check is performed on the corresponding feature fields of the data set and the inference data set, so that the comparison processing unit judges the historical training data set and the inference data set according to the result of the data distribution consistency check. Whether the data distribution of the corresponding characteristic fields is consistent. A method for maintaining model inference quality, comprising: setting a comparison period by a comparison processing unit, so that when the data accumulation time of an inference data set reaches the set comparison period, the comparison processing unit retrieves the inference from an inference information database a data set; let the comparison processing unit compare the training data set of the inference model deployed in the model server unit with the corresponding feature fields of the inference data set retrieved from the inference information database to perform the two data distribution consistency check, the comparison processing unit then judges whether the data distributions of the corresponding feature fields of the training data set of the inference model and the inference data set are consistent according to the result of the data distribution consistency check; and when the data When the distributions are inconsistent, the comparison processing unit finds out a training data set with a similar distribution to the inference data set from the historical training data set stored in the model version information database, so as to classify the trained data set according to the similar distribution of the training data set. Historical models from this model version repository It is deployed in the model server unit, and then the historical model that has been trained is updated or replaced by the inference model in the model server unit to provide inference services, so as to maintain the model inference quality, wherein the comparison processing unit further sets the historical data an interval to retrieve the feature field of the historical data set of the set historical data interval from the model version information database, so that the comparison processing unit compares the feature field of the inference data set and the historical data set . The method of claim 9, wherein the inference repository or the model version repository is implemented using a relational database, an object-oriented database, a hierarchical database, a network-based database, or a file system . The method of claim 9, further comprising causing the model version information base to store historical data sets, training data sets and model metadata, and causing the inference information base to store a plurality of inference data to form the inference data set. The method of claim 9, wherein the comparison processing unit uses the KS (Kolmogorov-Smirnov) test method, the Mann-Whitney (Mann-Whitney) test method, or the Kruskal- The Wallis (Clarska-Wallis) test method is used to perform the data distribution consistency test with the corresponding feature fields of both the training data set of the inference model and the inference data set. A method for maintaining model inference quality, comprising: setting a comparison period by a comparison processing unit, so that when the data accumulation time of an inference data set reaches the set comparison period, the comparison processing unit retrieves the inference from an inference information database a data set; let the comparison processing unit compare the training data set of the inference model deployed in the model server unit and the corresponding feature fields of the inference data set retrieved from the inference information database to perform the two data Distribution consistency check, and then the comparison processing unit will distribute consistency according to the data The result of the test determines whether the data distribution of the corresponding feature fields of the inference model training data set and the inference data set is consistent; and when the data distribution is inconsistent, the comparison processing unit stores the data from the model version information database. A training data set with similar distribution to the inference data set is found in the historical training data set, so as to deploy the trained historical model from the model version information database to the model server unit according to the similarly distributed training data set, and then Updating or replacing the inference model in the model server unit with the trained historical model to provide inference service to maintain the model inference quality, wherein the comparison processing unit further sets a threshold value, so that the comparison processing unit will After the data distribution consistency check is performed on the corresponding feature fields of the training data set of the inference model and the inference data set, the statistical values of the corresponding feature fields of the two are obtained. If the statistical value is less than the threshold value, it means The data distributions of the corresponding feature fields of the two are inconsistent. On the contrary, if the statistic value is greater than or equal to the threshold value, it means that the data distributions of the corresponding feature fields of the two are consistent. The method as claimed in claim 13, further comprising causing the comparison processing unit to calculate the number of times the statistic value is smaller than the threshold value to perform a frequency test to obtain the statistic number, if the statistic number is less than the number of times the threshold value, indicating that the inference model is being trained The overall data distribution of the data set and the inference data set is inconsistent. On the contrary, if the number of statistics is greater than or equal to the threshold of the number of times, it indicates the overall data distribution of the training data set of the inference model and the inference data set. is consistent. The method of claim 14, wherein the comparison processing unit uses a binomial distribution test method, a sign test method, or a Wilcoxon (Weiksen) sign level test method to perform the number of tests to obtain the statistical times. A method of maintaining the quality of model inferences, including: Let the comparison processing unit set a comparison period, so that when the data accumulation time of the inference data set reaches the set comparison period, the comparison processing unit retrieves the inference data set from the inference information database; let the comparison processing unit compare the models The corresponding feature fields of the training data set of the inference model deployed in the servo unit and the inference data set retrieved from the inference database are used to check the consistency of the data distribution between the two, and then processed by the comparison The unit judges whether the data distributions of the corresponding feature fields of the training data set of the inference model and the inference data set are consistent according to the result of the data distribution consistency check; and when the data distributions are inconsistent, the comparison processing unit determines from Find out a training data set with a similar distribution to the inference data set from the historical training data set stored in the model version information base, and deploy the trained historical model from the model version information base according to the similarly distributed training data set to the model server unit, and then update or replace the inference model in the model server unit with the historical model that has been trained to provide inference services, so as to maintain the model inference quality, wherein the comparison processing unit is more sensitive to the historical training The data distribution consistency check is performed on the corresponding feature fields of the data set and the inference data set, so that the comparison processing unit judges both the historical training data set and the inference data set according to the result of the data distribution consistency check Whether the data distribution of the corresponding characteristic fields is consistent.

Images