CN113568820A - Method, apparatus, electronic device and medium for monitoring model - Google Patents
Method, apparatus, electronic device and medium for monitoring model Download PDFInfo
- Publication number
- CN113568820A CN113568820A CN202110813084.0A CN202110813084A CN113568820A CN 113568820 A CN113568820 A CN 113568820A CN 202110813084 A CN202110813084 A CN 202110813084A CN 113568820 A CN113568820 A CN 113568820A
- Authority
- CN
- China
- Prior art keywords
- model
- value
- negative
- positive
- contribution
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 31
- 238000012544 monitoring process Methods 0.000 title claims abstract description 26
- 238000009826 distribution Methods 0.000 claims description 17
- 238000012163 sequencing technique Methods 0.000 claims description 10
- 230000008859 change Effects 0.000 claims description 6
- 238000012821 model calculation Methods 0.000 claims description 4
- 238000012545 processing Methods 0.000 abstract description 21
- 230000010365 information processing Effects 0.000 abstract 1
- 238000011161 development Methods 0.000 description 9
- 238000010586 diagram Methods 0.000 description 8
- 238000012360 testing method Methods 0.000 description 6
- 230000000694 effects Effects 0.000 description 5
- 238000010801 machine learning Methods 0.000 description 5
- 238000011156 evaluation Methods 0.000 description 3
- 230000006870 function Effects 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 238000012549 training Methods 0.000 description 3
- 230000003044 adaptive effect Effects 0.000 description 2
- 238000004364 calculation method Methods 0.000 description 2
- 238000004590 computer program Methods 0.000 description 2
- 239000013307 optical fiber Substances 0.000 description 2
- 230000008569 process Effects 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 230000009471 action Effects 0.000 description 1
- 239000000654 additive Substances 0.000 description 1
- 230000000996 additive effect Effects 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 238000013473 artificial intelligence Methods 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 238000004140 cleaning Methods 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000007418 data mining Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 230000002452 interceptive effect Effects 0.000 description 1
- 238000002372 labelling Methods 0.000 description 1
- 238000003058 natural language processing Methods 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 230000003252 repetitive effect Effects 0.000 description 1
- 238000012954 risk control Methods 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/34—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
- G06F11/3438—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment monitoring of user actions
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/34—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
- G06F11/3452—Performance evaluation by statistical analysis
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N20/00—Machine learning
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Quality & Reliability (AREA)
- Computer Hardware Design (AREA)
- Software Systems (AREA)
- Medical Informatics (AREA)
- Computing Systems (AREA)
- Evolutionary Computation (AREA)
- Mathematical Physics (AREA)
- Data Mining & Analysis (AREA)
- Life Sciences & Earth Sciences (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Evolutionary Biology (AREA)
- Probability & Statistics with Applications (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Artificial Intelligence (AREA)
- Debugging And Monitoring (AREA)
Abstract
The invention relates to the field of computer information processing, and provides a method, a device, electronic equipment and a medium for monitoring a model, aiming at the situation that the existing model is unstable and can not be accurately predicted in network data processing to generate large fluctuation. The problem of providing the undulant influence factor of more accurate definite model is solved and then can be based on these accurate factors carry out accurate control update, need not the model retraining, high efficiency, accurate realization model attribution and model update effectively promote the monitoring performance.
Description
Technical Field
The present invention relates to the field of data processing, and in particular, to a method and an apparatus for monitoring a model, an electronic device, and a computer-readable storage medium.
Background
With the branching and implementation of artificial intelligence, machine learning is widely applied to the fields of vision processing, speech recognition, natural language processing, data mining, risk control and the like. And training the model according to the sample data by machine learning, and predicting and deciding the data by using the model. Particularly, in combination with big data and cloud processing, data processing, control, transmission (for example, voice recognition, image recognition, data security, data subscription, interactive access), and the like of various application scenarios are realized by deploying a machine learning-based model on a network line, and the application scenarios are also becoming more and more widespread.
Since the model is developed with sample data of a specific period, the population (test sample) of the online model test may change over time, for example, the development population (development sample) is white-collar population, and then the model is mainly applied to the test student population. Due to the fact that different crowd samples have distribution differences, the model with poor stability cannot well distinguish new crowds, and the model needs to be retrained to be updated by collecting new crowd sample data. However, the new process of collecting, cleaning, labeling and retraining the sample data of the crowd is time-consuming, labor-consuming and low in efficiency.
Thus, the monitoring process of the stability of the on-line deployed model is improved to control the adjustment of the model more efficiently, more accurately and more timely.
Disclosure of Invention
In view of the above defects in the prior art, the present invention provides a method, an apparatus, an electronic device and a computer-readable storage medium for monitoring a model, which can solve the technical problem of how to update a model in time and efficiently with a new sample having a distribution difference; furthermore, the technical problem of how to accurately position the factors of the model to be adjusted when the model is updated so as to accurately determine the model updating and maintaining the stability of the model according to the new sample can be solved, so that the monitoring performance of the model stability adjustment is improved, and the accuracy, timeliness and high efficiency of the updating are ensured.
In order to solve the above technical problem, a first aspect of the present invention provides a method for monitoring a model, including: acquiring directional contribution values of all characteristics in a current sample of the monitored model according to the received fluctuation information; converting the orientation contribution value to an orientation weight offset to control updating of the model.
According to a preferred embodiment of the present invention, the fluctuation information is received by finding a prompt given by a type corresponding to the fluctuation information; wherein, the fluctuation information is used for representing the fluctuation index change caused by the distribution difference of the samples; the directional contribution values include: a feature is a positive contribution that causes the model to become larger when in use, or a feature is a negative contribution that causes the model to become smaller when in use.
According to a preferred embodiment of the present invention, the model is enlarged when using time, indicating that the model calculates the sample of the day larger than the sample of the previous day; the model becomes smaller when time-varying is used, which means that the model calculates the sample of the day less than the sample of the previous day. Wherein the current day sample is a different sample from the previous day sample;
according to a preferred embodiment of the present invention, obtaining directional contribution values of each feature in a current sample of a monitored model according to received fluctuation information specifically includes: when the fluctuation index represented by the fluctuation information changes to: when the fluctuation index exceeds a preset threshold value, determining that the monitored model is in an unstable state; acquiring a forward contribution value corresponding to each feature of the current sample when the model calculation result is increased based on the unstable state; or acquiring a negative contribution value corresponding to each feature of the current sample when the model calculation result is reduced based on the unstable state.
According to a preferred embodiment of the present invention, converting the directional contribution value into a directional weight offset specifically includes: sorting the positive contribution values according to a specified rule to obtain corresponding positive group actual sorting values of the features, and/or sorting the negative contribution values according to the same specified rule to obtain corresponding negative group actual sorting values of the features; sorting according to the same specified rule through the reference contribution values corresponding to the features in the model to obtain the reference sorting values of the corresponding features; and calculating a positive weight offset and/or a negative weight offset by combining the reference ranking value according to the positive group actual ranking value and/or the negative group actual ranking value.
According to a preferred embodiment of the present invention, the fluctuation index is a model stability index, and when the stability index is greater than a preset threshold, it is determined that the monitored model is in an unstable state, and a corresponding model updating strategy needs to be executed; the directional contribution value is a positive or negative SHAP value; the directional weight offset is a positive or negative DNDPS value; the specified rule is that SHAP values are arranged from large to small.
According to a preferred embodiment of the present invention, calculating the positive weight offset and/or the negative weight offset specifically includes: sequencing the forward SHAP values according to a specified rule to obtain the forward group actual sequencing value rank of the ith characteristicact,i+(ii) a And/or sequencing the negative SHAP values according to the specified rule to obtain a negative group actual sequencing value rank of the ith characteristicact,i-(ii) a The basic SHAP value is sequenced according to the specified rule to obtain the basic sequencing value rank of the ith characteristicBZ,i(ii) a Obtaining a positive DNDPS value or a negative DNDPS value of the ith characteristic by the following formula:
wherein: n is the total number of features of the model, rankact,iIndicating positive or negative SHAP values rankact,i+Or rankact,i-。
According to a preferred embodiment of the present invention, controlling the update of the model specifically includes: and controlling the positive DNDPS value or the negative DNDPS value to be input into the model updating strategy according to the positive DNDPS value or the negative DNDPS value and the strategy of the model updating corresponding to the model, outputting the characteristics of the model to be adjusted, and adjusting the model according to the characteristic control to obtain the updated model.
In a second aspect, the present invention provides an apparatus for monitoring a model, including: the acquisition module is used for acquiring the directional contribution value of each feature in the current sample of the monitored model according to the received fluctuation information; and the control module is used for converting the orientation contribution value into an orientation weight offset so as to control the update of the model.
A third aspect of the present invention provides an electronic device, comprising: a processor; and a memory storing computer executable instructions that, when executed, cause the processor to perform the method of the aforementioned first aspect.
A fourth aspect of the present invention proposes a computer readable storage medium, wherein the computer readable storage medium stores one or more programs which, when executed by a processor, implement the method of the aforementioned first aspect.
In one embodiment of the present invention, by converting the SHAP index of each feature in the current sample of the model into a normalized weight offset D-NDPS (i.e., DNDPS) value of the feature representing the positive direction or the negative direction in different samples, the cause (one or more features) causing instability and large fluctuation of the model is located to realize attribution of the model, and the cause or factor of influence, such as one or more features corresponding to the model, is accurately found. Thus, the model is adjusted according to the DNDPS value control to improve stability. The improved monitoring mode can convert the direct NDPS value through the direction change SHAP index in the positive direction and the negative direction under the condition that the fluctuation of one or more monitored models is large, thereby realizing more accurate cause, accurately determining the factors influencing the characteristics of the models and the like, further eliminating the influence on the characteristic weight due to the sample distribution difference through the calculation of the DNDPS value, avoiding the complex and tedious new sample data acquisition process and the retraining of new samples to the models, efficiently determining the cause, rapidly controlling the model updating and adjusting the stability of the models, effectively improving the performance of the monitoring management network model, saving the manpower and the time, and improving the monitoring and updating efficiency of a plurality of models such as a distributed network platform and the user experience.
Drawings
In order to make the technical problems solved by the present invention, the technical means adopted and the technical effects obtained more clear, the following will describe in detail the embodiments of the present invention with reference to the accompanying drawings. It should be noted, however, that the drawings described below are only illustrations of exemplary embodiments of the invention, from which other embodiments can be derived by those skilled in the art without inventive step.
FIG. 1 is a principal flow diagram of one embodiment of a method of monitoring a model according to the present invention.
FIG. 2 is a block diagram of a functional module architecture of an embodiment of an apparatus for monitoring a model according to the present invention.
Fig. 3 is a block diagram of an exemplary embodiment of an electronic device according to the present invention.
FIG. 4 is a schematic diagram of one embodiment of a computer-readable storage medium according to the present invention.
Detailed Description
Exemplary embodiments of the present invention will now be described more fully hereinafter with reference to the accompanying drawings, in which exemplary embodiments of the invention may be embodied in many specific forms, and should not be construed as limited to the embodiments set forth herein. Rather, these exemplary embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the concept of the invention to those skilled in the art.
The structures, properties, effects or other characteristics described in a certain embodiment may be combined in any suitable manner in one or more other embodiments, while still complying with the technical idea of the invention.
In describing particular embodiments, specific details of structures, properties, effects, or other features are set forth in order to provide a thorough understanding of the embodiments by one skilled in the art. However, it is not excluded that a person skilled in the art may implement the invention in a specific case without the above-described structures, performances, effects or other features.
The flow chart in the drawings is only an exemplary flow demonstration, and does not represent that all the contents, operations and steps in the flow chart are necessarily included in the scheme of the invention, nor does it represent that the execution is necessarily performed in the order shown in the drawings. For example, some operations/steps in the flowcharts may be divided, some operations/steps may be combined or partially combined, and the like, and the execution order shown in the flowcharts may be changed according to actual situations without departing from the gist of the present invention.
The block diagrams in the figures generally represent functional entities and do not necessarily correspond to physically separate entities. I.e. these functional entities may be implemented in the form of software, or in one or more hardware modules or integrated circuits, or in different networks and/or processor means and/or microcontroller means.
The same reference numerals denote the same or similar elements, components, or parts throughout the drawings, and thus, a repetitive description thereof may be omitted hereinafter. It will be further understood that, although the terms first, second, third, etc. may be used herein to describe various elements, components, or sections, these elements, components, or sections should not be limited by these terms. That is, these phrases are used only to distinguish one from another. For example, a first device may also be referred to as a second device without departing from the spirit of the present invention. Furthermore, the term "and/or", "and/or" is intended to include all combinations of any one or more of the listed items.
In one embodiment of the invention, when the on-line models monitored when the sample difference distribution occurs have large model fluctuation, the factors causing the model fluctuation are accurately and efficiently found out by attribution processing according to the positive and negative directional characteristics, and the model is controlled to be adjusted or updated, so that the on-line monitoring data processing performance is improved. The method specifically analyzes and determines the change of indexes for the directional characteristic of the model with overlarge fluctuation, accurately monitors the characteristic which has a large influence on the model to determine the reason influencing the fluctuation of the model and controls the model to carry out corresponding adjustment. The problem of providing the influence factor of more accurate definite model fluctuation is helped to be solved, accurate monitoring updating can be carried out based on the accurate factors, model retraining is not needed, model attribution and model updating are efficiently and accurately achieved, and therefore monitoring performance is effectively improved.
The meaning and action of some terms or nouns possibly referred to in the present invention in various embodiments of the present invention are explained below:
an online model: such as machine learning models, etc., typically monitor and evaluate the on-line model operating conditions and update/adjust the stability of the model when it is unstable and fluctuates significantly.
Monitoring an online model: the method mainly includes calculating a PSI index (stability index) by periodically observing model scores and variable distributions, considering instability and large fluctuation, such as a value exceeding 0.1, when the PSI is greater than a certain threshold, or calculating a model and a KS (model risk-differentiation capability score) of variables by periodically using new samples, and considering failure when the KS is lower than a certain threshold. The data generated by the model at the corresponding time may be monitored individually for each model each day.
Throughput of the model: one type of fluctuation indicator is the current number of samples of the model.
Stability Index (PSI): a fluctuation index can measure the distribution difference of a test sample and a development sample, and is the most common model stability evaluation index.
SHAP: is a 'model interpretation' package developed by Python, which can interpret the output of any machine learning model, and under the inspiration of cooperative game theory, SHAP constructs an additive interpretation model, and all the characteristics are regarded as 'contributors'.
[ example 1 ]
Fig. 1 is a main flow diagram of an embodiment of a method according to the invention. As shown in fig. 1, the method of the present invention at least comprises the following steps:
and step S1, acquiring SHAP values of each feature in the current sample of the monitored model according to the fluctuation information of the distribution difference of the received samples.
In one embodiment, the difference in the distribution of the samples of the model is mainly the difference between the development sample and the test sample of the model. Assuming that the development sample is a white-collar population and the test sample is a blue-collar population, the feature distribution between the two populations will be different. And determining whether the sample has distribution difference or not by monitoring and analyzing the fluctuation indexes of the model, and if so, sending fluctuation information according to a preset rule.
The fluctuation index may adopt a stability index PSI of the model, or may adopt an index related to the service, such as throughput of the model.
In one embodiment, the fluctuation information may be classified into different types according to different fluctuation indexes by a preset rule. Example 1: (1) if the fluctuation index is a PSI absolute value and the fluctuation of the PSI absolute value is larger than a first preset fluctuation range, determining that the fluctuation information corresponds to a first type; the PSI absolute value fluctuation refers to the difference between the current PSI value of the model and the standard PSI value of the model; that is, when the difference between the current PSI value of the model and the standard PSI value of the model exceeds a first preset fluctuation range (for example, exceeds 20% of the standard PSI value), the fluctuation information corresponds to a first type, and for each type, there is a corresponding prompt of sound, light, vision, etc., that is, a prompt of the first type. (2) If the fluctuation index is a PSI relative value and the fluctuation of the PSI relative value is larger than a second preset fluctuation range, determining that the fluctuation information corresponds to a second type; the PSI relative value fluctuation refers to a difference value between a PSI value of the model in a first time period and a PSI value of the model in a second time period; for example, the difference between the PSI value of the model at yesterday and the PSI value of the model at the previous day exceeds a second preset fluctuation range (for example, exceeds the PSI value at yesterday by 15%), and the fluctuation information corresponds to a second type and has second type prompts which are distinguished from other types. (3) If the fluctuation index is a service index and the fluctuation of the service index is larger than a third preset fluctuation range, determining a third type corresponding to the fluctuation information; the service index fluctuation refers to a difference value between a service index value in the third time period and a service index value in the fourth time period. The third time period may be the same as or different from the first time period, and the fourth time period may be the same as or different from the second time period, which is not limited in the present invention. The service index is determined according to actual service, such as wind control service, and the model throughput can be selected as the service index. For example, the difference between the model throughput of yesterday and the model throughput of the previous day exceeds a third preset fluctuation range (for example, exceeds 10% of the model throughput of yesterday), and the fluctuation information corresponds to a third type and has a third type prompt which is different from other types.
Further, the model updating/adjusting mode which can be adopted is determined according to the corresponding relation between the type of the fluctuation information and the model updating (or model adjusting) strategy.
In this embodiment, the preset rule, the type of the fluctuation information, and the corresponding relationship of the model update policy may be configured in advance and stored in a specific location (for example, in a server). For example, the received fluctuation information is of a second type in terms of information prompt, that is, when PSI exceeds a threshold value as a result of comparison between the current day and yesterday, the model is updated according to a corresponding second strategy after the fluctuation reason is determined, so that the stability of the model is adjusted.
In one embodiment, the embodiment may eliminate the influence of the sample distribution difference on the feature weight by converting the SHAP finger of each feature in the current sample of the model into a DNDPS value capable of indicating the normalized weight offset of the positive or negative feature in different samples, and then adjust the model according to the DNDPS value, so that the correspondence between the type of the fluctuation information and the model updating policy of the present invention is configured according to the DNDPS value of the model feature. In one type of correspondence with model update policies, for example, the first policy of the update model corresponding to the first type is: comparing a reference SHAP value and a DNDSP value of the ith characteristic of the model, and taking the characteristic that the deviation between the reference SHAP value and the DNDPS value is greater than a first threshold value as the characteristic to be adjusted; the second policy for updating the model corresponding to the second type is as follows: comparing the DNDPS value of the ith model characteristic in the first time period with the DNDPS value in the second time period, and taking the characteristic that the deviation between the DNDPS value in the first time period and the DNSP value in the second time period is greater than a second threshold value as a characteristic to be adjusted; the third strategy for updating the model corresponding to the third type is as follows: and sorting the DNDPS values of the features in the descending order, and taking the feature M before sorting as the feature to be adjusted. The first threshold, the second threshold and the third threshold may be preset according to actual needs.
In one embodiment, for each prediction sample, the model generates a prediction value, and the SHAP value is the weight (i.e., contribution) assigned to the prediction value by each feature in the sample. Specifically, the model prediction value may be loaded into the SHAP packet, and the weight of each feature to the prediction value is output, where the range of each feature weight is from negative infinity to positive infinity, and the sum is a probability value between 0 and 1. For example: assuming that the predicted value is Y, the calculated SHAP value is expressed as the product of the feature ai and the corresponding weight wi, that is, the SHAP value corresponding to Y is the sum of the products of each feature weight: w1x1+ w2x2+ … … wixi + … … wnxn, wherein i and n are positive integers and represent the number, and wixi represents the contribution of the characteristic xi to the predicted value Y.
Example 1:
the online model is monitored. When receiving a second type prompt that the fluctuation information of a monitored certain model on the day is a fluctuation index, for example, the fluctuation index is PSI, the value of which is greater than the threshold value 0.1, and it indicates that the PSI difference between the PSI of yesterday and the PSI of the previous day exceeds the preset range or the preset threshold value, specifically, the average value of the predicted values on the day can be calculated for each monitored single model as the PSI calculation difference value of the average value of the PSI and the predicted value of the previous day. Further, it is considered to update the model with a corresponding policy.
Based on the fact that the PSI is greater than 0.1, a current SHAP value is calculated, such as the weight (i.e., contribution) wi to which each feature xi in the sample is assigned to the predicted value. The model prediction values may be loaded into the SHAP package and the weights wixi of the features xi to the prediction values may be output.
Further, positive or negative manifestations of each feature xi weight wixi are determined. It may be determined by detecting a difference between a preset reference value (e.g., a predicted value of each feature xi of the model versus development sample) and each predicted value, and since each predicted value corresponds to the feature xi, it may be determined whether the difference is greater than 0 or less than 0 or equal to 0. If the value is more than 0, the feature of the model ith is a positive feature, namely the current day is better than the previous day, if the value is less than 0, the feature of the model ith is a negative feature xi-, and if the value is equal to or less than 0, the feature is not considered, wherein i is a positive integer representing the number and the number of the features. The resulting SHAP value wixi for each feature xi is positive or negative.
Step S2, converting the SHAP value into a DNDPS value to control updating of the model.
In one embodiment, the NDPS values represent normalized weight offsets of the feature xi (or referred to as ith feature) in different samples for eliminating the effect of sample distribution differences on feature weights. Since each SHAP value is weighted with orientation at the time of conversion, i.e., is explicitly positive or negative, the converted value is referred to as oriented NDPS or DNDPS, and the first D represents the orientation direction.
In one embodiment, converting the SHAP value to an NDPS value may include:
firstly, sorting SHAP values (positive and negative weights) wixi of all features xi in a current sample according to a specified sequence, and respectively sorting the SHAP values in the positive direction and the negative direction to obtain an actual sorting value rank of each group of ith feature, namely the ith featureact,i(ii) a Forward direction, e.g. rankact,i+Negative, e.g. rankact,i-。
The order may be preferably in a descending order.
Secondly, obtaining the reference SHAP value of each feature in the model, and sorting the reference SHAP values of the features xi according to a specified sequence to obtain the reference sorting value rank of the ith featureBZ,i；
The reference SHAP value of each feature xi refers to the SHAP value of each feature xi in the model development sample. Namely, the model loads the predicted values of the development samples into the SHAP packet, and outputs the weight of each feature to the predicted values of the development samples. And the second step is in the same order as specified in the first step.
Thirdly, obtaining a DNDPS value of the ith characteristic through the following formula:
wherein: and N is the total number of the characteristics of the model.
The positive group and the negative group are respectively calculated to obtain a positive DNDPS value DNDPSi+ and negative DNDPS values DNDPSi-. Thus, the scoring of the model is realized through positive and negative directions, namely positive evaluation and negative evaluation, so that the reason of the larger fluctuation of the negative influence and the positive influence on the model can be determined more accurately. Such as why the predicted result becomes larger or smaller on the current day than on the previous day, etc., the influencing factors/factors or characteristics on the model can be accurately determined.
In one embodiment, the model may be adjusted based on the DNDPS value. Specifically, there are a set of DNDPS values for both positive and negative directions, respectively. DNDPS in forward groupi+ for example, the DNDPS with 60 Top characteristics xi can be takeniValue, further, the DNDPS of each feature xiiInputting the values into preset updating model strategies corresponding to the monitored target model, for example, a second strategy corresponding to a second type of fluctuation information in the embodiment, to obtain the characteristics to be adjusted corresponding to the updating strategy of the target model; and adjusting the characteristics to be adjusted. The negative bank is similar and will not be described again.
In the embodiment of the invention, the DNDPS value represents a DNDPS value of normalized weight offset of features in different samples to eliminate the influence of sample distribution difference on feature weights, the DNDPS value distinguished by orientation (i.e. positive and negative) can lock features causing model instability (such as taking Top60 features), accurately determine the reasons causing larger or smaller target model prediction (i.e. which features are factors causing larger or which are factors causing smaller), and accurately determine which factors are most influenced by the orientation distinguishing mode. Furthermore, the stability of the model can be effectively improved by adjusting the features (such as removing the features, re-processing the features and the like), and meanwhile, the mode of attributing the fluctuation of the score of the model by the orientation distinguishing mode is more accurate in adjustment, so that the performance of the whole monitoring treatment is improved.
According to the embodiment of the invention, the factor of the monitored model with larger fluctuation can be timely and efficiently determined, and particularly, under the condition that the group faced by the model changes, the model is rapidly updated without re-collecting and training a new sample; further, based on orientation, namely respectively attributing the larger fluctuation of the model by distinguishing positive grading from negative grading, the influence factors can be determined more accurately, so that the stability adjustment of the model is realized more accurately, and the adaptive model updating is completed.
[ example 2 ]
Similarly, an embodiment of the apparatus of the corresponding monitoring model corresponds to the method. As shown in fig. 2, the apparatus according to an embodiment of the present invention may specifically include:
the acquisition module is used for acquiring SHAP values of all characteristics in the current sample of the monitored model according to the fluctuation information of the distribution difference of the received samples; for specific functions, see specific steps and contents of S1, which are not described herein again.
A control module to convert the SHAP value to a DNDPS value to control updating of the model; for specific functions, see specific steps and contents of S2, which are not described herein again.
In the embodiment of the invention, the DNDPS value represents a DNDPS value of normalized weight offset of features in different samples, so as to eliminate the influence of sample distribution difference on feature weights, the DNDPS value distinguished by orientation (i.e. positive and negative) can lock features (such as taking Top60 features) which cause instability of the model, accurately determine the reasons (i.e. which features are factors causing larger prediction or smaller prediction) of the target model, and further, through adjustment of the features (such as removing the features, processing the features from new processing and the like), the stability of the model can be effectively improved, and meanwhile, due to the mode of orientation distinguishing, the mode of model score fluctuation is more accurate in adjustment, and the performance of the whole monitoring process is improved.
According to the embodiment of the invention, the factor of the monitored model with larger fluctuation can be timely and efficiently determined, and particularly, under the condition that the group faced by the model changes, the model is rapidly updated without re-collecting and training a new sample; further, based on orientation, namely respectively attributing the larger fluctuation of the model by distinguishing positive grading from negative grading, the influence factors can be determined more accurately, so that the stability adjustment of the model is realized more accurately, and the adaptive model updating is completed.
[ example 3 ]
In the following, embodiments of the electronic device of the present invention are described, which may be regarded as an implementation in physical form for the above-described embodiments of the method and apparatus of the present invention. Details described in the embodiments of the electronic device of the invention should be considered supplementary to the embodiments of the method or apparatus described above; for details which are not disclosed in embodiments of the electronic device of the invention, reference may be made to the above-described embodiments of the method or the apparatus.
Fig. 3 is a block diagram of an exemplary embodiment of an electronic device according to the present invention. The electronic device shown in fig. 3 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present invention.
As shown in fig. 3, the electronic apparatus 200 of the exemplary embodiment is represented in the form of a general-purpose data processing apparatus. The components of the electronic device 200 may include, but are not limited to: at least one processing unit 210, at least one memory unit 220, a bus 230 connecting different system components (including the memory unit 220 and the processing unit 210), a display unit 240, and the like.
The storage unit 220 stores a computer readable program, which may be a code of a source program or a read-only program. The program may be executed by the processing unit 210 such that the processing unit 210 performs the steps of various embodiments of the present invention. For example, the processing unit 210 may perform the steps as shown in fig. 1.
The memory unit 220 may include readable media in the form of volatile memory units, such as a random access memory unit (RAM)2201 and/or a cache memory unit 2202, and may further include a read only memory unit (ROM) 2203. The storage unit 220 may also include a program/utility 2204 having a set (at least one) of program modules 2205, such program modules 2205 including, but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which, or some combination thereof, may comprise an implementation of a network environment.
The electronic device 200 may also communicate with one or more external devices 300 (e.g., keyboard, display, network device, bluetooth device, etc.), enable a user to interact with the electronic device 200 via the external devices 300, and/or enable the electronic device 200 to communicate with one or more other data processing devices (e.g., router, modem, etc.). Such communication may occur via input/output (I/O) interfaces 250, and may also occur via network adapter 260 with one or more networks (e.g., a Local Area Network (LAN), a Wide Area Network (WAN) and/or a public network such as the Internet). The network adapter 260 may communicate with other modules of the electronic device 200 via the bus 230. It should be appreciated that although not shown in the figures, other hardware and/or software modules may be used in the electronic device 200, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, among others.
[ example 4 ]
FIG. 4 is a schematic diagram of one computer-readable medium embodiment of the present invention. As shown in fig. 4, the computer program may be stored on one or more computer readable media. The computer readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples (a non-exhaustive list) of the readable storage medium include: an electrical connection having one or more wires, a portable disk, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. The computer program, when executed by one or more data processing devices, enables the computer-readable medium to implement the above-described methods of the present invention.
Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments of the present invention described herein may be implemented by software, or by software in combination with necessary hardware. Therefore, the technical solution according to the embodiment of the present invention can be embodied in the form of a software product, which can be stored in a computer-readable storage medium (which can be a CD-ROM, a usb disk, a removable hard disk, etc.) or on a network, and includes several instructions to make a data processing device (which can be a personal computer, a server, or a network device, etc.) execute the above-mentioned method according to the present invention.
The computer readable storage medium may include a propagated data signal with readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A readable storage medium may also be any readable medium that is not a readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a readable storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
Program code for carrying out operations for aspects of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computing device (e.g., through the internet using an internet service provider).
In summary, the present invention determines the change of the index by analyzing the positive and negative characteristics of the model with excessive fluctuation, and accurately monitors the characteristics that have a large influence on the model to determine the cause of the model fluctuation and control the model to perform corresponding adjustment. The problem of providing the undulant influence factor of more accurate definite model is solved and then can be based on these accurate factors carry out accurate control update, need not the model retraining, high efficiency, accurate realization model attribution and model update effectively promote the monitoring performance.
While the foregoing embodiments have described the objects, aspects and advantages of the present invention in further detail, it should be understood that the present invention is not inherently related to any particular computer, virtual machine or electronic device, and various general-purpose machines may be used to implement the present invention. The invention is not to be considered as limited to the specific embodiments thereof, but is to be understood as being modified in all respects, all changes and equivalents that come within the spirit and scope of the invention.
Claims (11)
1. A method of monitoring a model, comprising:
acquiring directional contribution values of all characteristics in a current sample of the monitored model according to the received fluctuation information;
converting the orientation contribution value to an orientation weight offset to control updating of the model.
2. The method of claim 1,
receiving the fluctuation information through finding out a prompt given by the type corresponding to the fluctuation information;
wherein, the fluctuation information is used for representing the fluctuation index change caused by the distribution difference of the samples;
the directional contribution values include: a feature is a positive contribution that causes the model to become larger when in use, or a feature is a negative contribution that causes the model to become smaller when in use.
3. The method of claim 2,
the model is increased when time is used, so that the result of the model for calculating the sample of the day is larger than the result of calculating the sample of the previous day;
the model becomes smaller when time-varying is used, which means that the model calculates the sample of the day less than the sample of the previous day.
Wherein the current day sample is a different sample from the previous day sample.
4. The method according to claim 2 or 3, wherein obtaining the directional contribution value of each feature in the current sample of the monitored model according to the received fluctuation information specifically comprises:
when the fluctuation index represented by the fluctuation information changes to: when the fluctuation index exceeds a preset threshold value, determining that the monitored model is in an unstable state;
acquiring a forward contribution value corresponding to each feature of the current sample when the model calculation result is increased based on the unstable state; and/or acquiring a negative contribution value corresponding to each feature of the current sample when the model calculation result is reduced based on the unstable state.
5. The method according to any one of claims 2 to 4, wherein converting the directional contribution value into a directional weight offset comprises:
sorting the positive contribution values according to a specified rule to obtain corresponding positive group actual sorting values of the features, and/or sorting the negative contribution values according to the same specified rule to obtain corresponding negative group actual sorting values of the features;
sorting according to the same specified rule through the reference contribution values corresponding to the features in the model to obtain the reference sorting values of the corresponding features;
and calculating a positive weight offset and/or a negative weight offset by combining the reference ranking value according to the positive group actual ranking value and/or the negative group actual ranking value.
6. The method of claim 5,
the fluctuation index adopts a model stability index, when the stability index is larger than a preset threshold value, the monitored model is determined to be in an unstable state, and a corresponding model updating strategy needs to be executed;
the directional contribution value is a positive or negative SHAP value;
the directional weight offset is a positive or negative DNDPS value;
the specified rule is that SHAP values are arranged from large to small.
7. The method of claim 6, wherein calculating the positive-going weight offset and/or the negative-going weight offset comprises:
sequencing the forward SHAP values according to a specified rule to obtain the forward group actual sequencing value rank of the ith characteristicact,i+(ii) a And/or sequencing the negative SHAP values according to the specified rule to obtain a negative group actual sequencing value rank of the ith characteristicact,i-；
The basic SHAP value is sequenced according to the specified rule to obtain the basic sequencing value rank of the ith characteristicBZ,i；
Obtaining a positive DNDPS value or a negative DNDPS value of the ith characteristic by the following formula:
wherein: n is the total number of features of the model, rankact,iPositive or negative SHAP values rankact,i+Or rankact,i-。
8. The method according to any one of claims 1 to 7, wherein controlling the update of the model comprises:
and inputting the positive or negative DNDPS value into a model updating strategy according to the positive or negative DNDPS value and the strategy of the model updating corresponding to the model, outputting the characteristic of the model to be adjusted, and controlling the model to be adjusted according to the characteristic to obtain the updated model.
9. An apparatus for monitoring a model, comprising:
the acquisition module is used for acquiring the directional contribution value of each feature in the current sample of the monitored model according to the received fluctuation information;
and the control module is used for converting the orientation contribution value into an orientation weight offset so as to control the update of the model.
10. An electronic device, comprising:
a processor; and
a memory storing computer-executable instructions that, when executed, cause the processor to perform the steps of the method of any of claims 1 to 8.
11. A computer readable storage medium, wherein the computer readable storage medium stores one or more programs which, when executed by a processor, implement the method of any of claims 1-8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110813084.0A CN113568820A (en) | 2021-07-19 | 2021-07-19 | Method, apparatus, electronic device and medium for monitoring model |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110813084.0A CN113568820A (en) | 2021-07-19 | 2021-07-19 | Method, apparatus, electronic device and medium for monitoring model |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113568820A true CN113568820A (en) | 2021-10-29 |
Family
ID=78165621
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110813084.0A Pending CN113568820A (en) | 2021-07-19 | 2021-07-19 | Method, apparatus, electronic device and medium for monitoring model |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113568820A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114325454A (en) * | 2021-12-30 | 2022-04-12 | 东软睿驰汽车技术（沈阳）有限公司 | Method, device, equipment and medium for determining influence of multiple characteristics on battery health degree |
| CN117710066A (en) * | 2024-02-05 | 2024-03-15 | 厦门傲凡科技股份有限公司 | Financial customer recommendation method and system |
| WO2024169563A1 (en) * | 2023-02-17 | 2024-08-22 | 大唐移动通信设备有限公司 | Model monitoring method, and device and storage medium |
-
2021
- 2021-07-19 CN CN202110813084.0A patent/CN113568820A/en active Pending
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114325454A (en) * | 2021-12-30 | 2022-04-12 | 东软睿驰汽车技术（沈阳）有限公司 | Method, device, equipment and medium for determining influence of multiple characteristics on battery health degree |
| CN114325454B (en) * | 2021-12-30 | 2023-07-04 | 东软睿驰汽车技术（沈阳）有限公司 | Method, device, equipment and medium for determining influence of multiple characteristics on battery health |
| WO2024169563A1 (en) * | 2023-02-17 | 2024-08-22 | 大唐移动通信设备有限公司 | Model monitoring method, and device and storage medium |
| CN117710066A (en) * | 2024-02-05 | 2024-03-15 | 厦门傲凡科技股份有限公司 | Financial customer recommendation method and system |
| CN117710066B (en) * | 2024-02-05 | 2024-05-10 | 厦门傲凡科技股份有限公司 | Financial customer recommendation method and system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113568820A (en) | Method, apparatus, electronic device and medium for monitoring model | |
| CN109801705B (en) | Treatment recommendation method, system, device and storage medium | |
| CN111241287A (en) | Training method and device for generating generation model of confrontation text | |
| US20210319366A1 (en) | Method, apparatus and device for generating model and storage medium | |
| CN112270545A (en) | Financial risk prediction method and device based on migration sample screening and electronic equipment | |
| US20200293722A1 (en) | Word vector retrofitting method and apparatus | |
| JP7430820B2 (en) | Sorting model training method and device, electronic equipment, computer readable storage medium, computer program | |
| WO2020082734A1 (en) | Text emotion recognition method and apparatus, electronic device, and computer non-volatile readable storage medium | |
| CN112508580A (en) | Model construction method and device based on rejection inference method and electronic equipment | |
| US11281999B2 (en) | Predictive accuracy of classifiers using balanced training sets | |
| US11373760B2 (en) | False detection rate control with null-hypothesis | |
| WO2020090413A1 (en) | Classification device, classification method, and classification program | |
| CN113222149A (en) | Model training method, device, equipment and storage medium | |
| CN115063875A (en) | Model training method, image processing method, device and electronic equipment | |
| CN112883990A (en) | Data classification method and device, computer storage medium and electronic equipment | |
| KR20210066545A (en) | Electronic device, method, and computer readable medium for simulation of semiconductor device | |
| JPWO2019092931A1 (en) | Discriminant model generator, discriminant model generation method and discriminant model generation program | |
| CN115221516A (en) | Malicious application program identification method and device, storage medium and electronic equipment | |
| CN114462598A (en) | Deep learning model training method, and method and device for determining data category | |
| CN112766402A (en) | Algorithm selection method and device and electronic equipment | |
| CN111161238A (en) | Image quality evaluation method and device, electronic device, and storage medium | |
| CN111209930A (en) | Method and device for generating credit granting strategy and electronic equipment | |
| US11887003B1 (en) | Identifying contributing training datasets for outputs of machine learning models | |
| CN113919510A (en) | Sample feature selection method, device, equipment and medium | |
| CN114091570A (en) | Service processing system method, device and electronic equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |