CN110489314B - Model anomaly detection method and device, computer equipment and storage medium - Google Patents
Model anomaly detection method and device, computer equipment and storage medium Download PDFInfo
- Publication number
- CN110489314B CN110489314B CN201910603347.8A CN201910603347A CN110489314B CN 110489314 B CN110489314 B CN 110489314B CN 201910603347 A CN201910603347 A CN 201910603347A CN 110489314 B CN110489314 B CN 110489314B
- Authority
- CN
- China
- Prior art keywords
- model
- abnormal
- index
- sample data
- current
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/34—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
- G06F11/3409—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment for performance assessment
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/36—Preventing errors by testing or debugging software
- G06F11/3604—Software analysis for verifying properties of programs
- G06F11/3608—Software analysis for verifying properties of programs using formal methods, e.g. model checking, abstract interpretation
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Computer Hardware Design (AREA)
- Quality & Reliability (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Information Transfer Between Computers (AREA)
Abstract
The application relates to a model anomaly detection method, a model anomaly detection device, computer equipment and a storage medium, wherein the method comprises the following steps: acquiring a configuration file which is set aiming at the indexes of the model to be detected and comprises abnormal detection associated parameters; calling a general code corresponding to the index of the model to be detected, and extracting sample selection time corresponding to the current model from the abnormal detection associated parameters; acquiring sample data which accords with the sample selection time from a database through the universal code; determining the value of the model index to be tested through the general code and the sample data; when the model index to be detected of the current model is determined to be abnormal according to the value of the model index to be detected, triggering an alarm logic code to generate an abnormal reminding mail; and sending the abnormal reminding mail to a terminal. By adopting the method, the model abnormity detection efficiency can be improved.
Description
Technical Field
The present application relates to the field of computer technologies, and in particular, to a method and an apparatus for detecting a model anomaly, a computer device, and a storage medium.
Background
With the rapid development of science and technology, various machine learning models are produced. Machine learning models can provide great convenience in data processing and recognition. To ensure the accuracy of data processing, a stable machine learning model is of great importance.
In the traditional method, when monitoring the stability of the model, developers need to write monitoring codes and run the monitoring codes to monitor the stability of the model. However, monitoring the model involves a very large number of aspects, requiring a large amount of code to be written. This takes a lot of time and costs, resulting in very low efficiency.
Disclosure of Invention
In view of the above, it is necessary to provide a method, an apparatus, a computer device, and a storage medium for detecting a model abnormality, which can improve efficiency.
A method of model anomaly detection, the method comprising:
acquiring a configuration file which is set aiming at the indexes of the model to be detected and comprises abnormal detection associated parameters;
calling a general code corresponding to the index of the model to be detected, and extracting sample selection time corresponding to the current model from the abnormal detection associated parameters;
acquiring sample data which accords with the sample selection time from a database through the universal code;
determining the value of the model index to be tested through the general code and the sample data;
when the model index to be detected of the current model is determined to be abnormal according to the value of the model index to be detected, triggering an alarm logic code to generate an abnormal reminding mail;
and sending the abnormal reminding mail to a terminal.
In one embodiment, the sample selection time is a sample selection time period including a plurality of months; the determining the value of the model index to be tested through the general code and the sample data comprises:
when the model index to be measured is a group stability index, then
Selecting a current month from a first month in the sample selection time period in a circulating manner by month through the general code, and acquiring first sub-sample data corresponding to the current month and second sub-sample data corresponding to a month next to the current month in the sample data;
determining a first prediction result and a second prediction result which are obtained by inputting the first sub-sample data and the second sub-sample data into the current model respectively for prediction and outputting;
and analyzing the first prediction result and the second prediction result through the universal code to obtain a value of a population stability index corresponding to the current month.
In one embodiment, the method further comprises:
comparing the obtained values of the group stability indexes corresponding to the months in the sample selection time period with preset abnormal threshold values respectively;
determining an abnormal month corresponding to the value of the population stability index larger than the abnormal threshold;
and when the number of the abnormal months is larger than a preset number threshold, judging that the group stability index of the current model is abnormal.
In one embodiment, the determining, by the generic code and the sample data, the value of the model index under test includes:
when the model index to be tested is the information quantity index, extracting the independent variable field of the current model included in the configuration file through the general code;
determining a corresponding current information quantity index numerical value of each independent variable in the current model according to the sample data aiming at the independent variable represented by each independent variable field;
the method further comprises the following steps:
obtaining historical information quantity index values respectively corresponding to the independent variables in the historical models;
determining a variation coefficient of an information quantity index of the independent variable according to a historical information quantity index value and a current information quantity index value corresponding to the independent variable;
and when the variation coefficient is larger than or equal to the variation threshold value, judging that the information quantity index of the independent variable is abnormal.
In one embodiment, the triggering the alarm logic code to generate an exception alert mail comprises:
triggering an alarm logic code to obtain a type identifier of a type to which the abnormal model index belongs;
searching an abnormal reason set corresponding to the acquired type identifier according to a preset corresponding relation between the type identifier and the abnormal reason set;
and generating an abnormal reminding mail according to the mail content comprising the abnormal reason set.
In one embodiment, the method further comprises:
determining a target abnormal reason selected from an abnormal reason set positioned in the abnormal reminding mail;
sending a repair page corresponding to the target abnormal reason to the terminal;
receiving repair operation data which is sent by the terminal and is obtained based on the repair page and aims at the target abnormal reason;
and performing corresponding repair processing according to the repair operation data.
In one embodiment, the method further comprises:
after the repairing process is finished, obtaining sample data which accords with the sample selecting time from a database through the universal code again;
and iteratively adjusting the weight of each variable of the current model according to the newly acquired sample data until a training stopping condition is reached to obtain a normal model.
A model anomaly detection apparatus, the apparatus comprising:
the acquisition module is used for acquiring a configuration file which is set aiming at the index of the model to be detected and comprises an abnormal detection correlation parameter;
the anomaly detection module is used for calling a general code which is set corresponding to the index of the model to be detected and extracting sample selection time corresponding to the current model from the anomaly detection associated parameters; acquiring sample data which accords with the sample selection time from a database through the universal code; determining the value of the model index to be tested through the general code and the sample data; when the model index to be detected of the current model is determined to be abnormal according to the value of the model index to be detected, triggering an alarm logic code to generate an abnormal reminding mail;
and the abnormity reminding module is used for sending the abnormity reminding mail to a terminal.
A computer device comprising a memory storing a computer program and a processor implementing the following steps when the processor executes the computer program:
acquiring a configuration file which is set aiming at the indexes of the model to be detected and comprises abnormal detection associated parameters;
calling a general code corresponding to the index of the model to be detected, and extracting sample selection time corresponding to the current model from the abnormal detection associated parameters;
acquiring sample data which accords with the sample selection time from a database through the universal code;
determining the value of the model index to be tested through the general code and the sample data;
when the model index to be detected of the current model is determined to be abnormal according to the value of the model index to be detected, triggering an alarm logic code to generate an abnormal reminding mail;
and sending the abnormal reminding mail to a terminal.
A computer-readable storage medium, on which a computer program is stored which, when executed by a processor, carries out the steps of:
acquiring a configuration file which is set aiming at the indexes of the model to be detected and comprises abnormal detection associated parameters;
calling a general code corresponding to the index of the model to be detected, and extracting sample selection time corresponding to the current model from the abnormal detection associated parameters;
acquiring sample data which accords with the sample selection time from a database through the universal code;
determining the value of the model index to be tested through the general code and the sample data;
when the model index to be detected of the current model is determined to be abnormal according to the value of the model index to be detected, triggering an alarm logic code to generate an abnormal reminding mail;
and sending the abnormal reminding mail to a terminal.
According to the model anomaly detection method, the model anomaly detection device, the computer equipment and the storage medium, the configuration file comprising the anomaly detection associated parameters is set for the indexes of the model to be detected, the universal code is preset for the indexes of the model to be detected, and the sample selection time corresponding to the current model can be extracted from the anomaly detection associated parameters by calling the universal code; acquiring sample data which accords with the sample selection time from a database through the universal code; determining the value of the model index to be tested through the general code and the sample data; therefore, the abnormal detection of the model to be detected of the current model can be realized by the universal codes and the configuration files, the abnormal detection of the model can be realized without all codes, and the abnormal processing efficiency is improved. In addition, when the model index to be detected of the current model is determined to be abnormal according to the value of the model index to be detected, an alarm logic code is triggered to generate an abnormal reminding mail; and sending the abnormal reminding mail to a terminal. When the abnormity is detected, the abnormity reminding mail is directly triggered and generated, so that the abnormity condition is more efficiently notified, and the abnormity processing efficiency can also be improved.
Drawings
FIG. 1 is a diagram illustrating an exemplary application of a method for detecting anomalies in a model;
FIG. 2 is a schematic flow chart diagram illustrating a method for model anomaly detection in one embodiment;
FIG. 3 is a flow diagram illustrating the exception recovery step in one embodiment;
FIG. 4 is a block diagram showing the structure of a model abnormality detection apparatus according to an embodiment;
FIG. 5 is a block diagram showing the structure of a model abnormality detecting apparatus according to another embodiment;
FIG. 6 is a diagram illustrating an internal structure of a computer device according to an embodiment.
Detailed Description
In order to make the objects, technical solutions and advantages of the present application more apparent, the present application is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application.
The model anomaly detection method provided by the application can be applied to the application environment shown in fig. 1. Wherein the terminal 110 and the server 120 communicate through a network. The terminal 110 may be, but is not limited to, various personal computers, notebook computers, smart phones, tablet computers, and portable wearable devices. The servers 120 may be implemented as independent servers or as a server cluster composed of a plurality of servers.
The server 120 may obtain a configuration file including an anomaly detection associated parameter set for the model index to be detected; calling a general code which is set corresponding to the model index to be detected, and executing the general code to extract sample selection time from the abnormal detection associated parameters; acquiring sample data which accords with the sample selection time from a database through the universal code; analyzing the sample data through the general code to obtain the value of the model index to be detected; and when the value of the model index to be detected represents that the model index to be detected is abnormal, triggering an alarm logic code to generate an abnormal reminding mail. The server 120 may send the abnormality alerting mail to the terminal 110. The terminal 110 may display the abnormal reminding mail, and a technician may know that the current model is abnormal according to the abnormal reminding mail.
In one embodiment, as shown in fig. 2, a model anomaly detection method is provided, which is described by taking the method as an example applied to a computer device, which may be described by taking the server 120 in fig. 1 as an example, and includes the following steps:
s202, acquiring a configuration file which is set aiming at the indexes of the model to be detected and comprises the abnormal detection associated parameters.
The abnormality detection related parameter is a related parameter for performing abnormality detection on the model index. It is understood that the presence or absence of an abnormality in the model index may be detected based on the abnormality detection associated parameter.
The model index to be detected is the model index to be detected whether the abnormality exists or not. It can be understood that the model to be measured has at least one index. In one embodiment, the model index to be tested may include at least one of a population stability index and an information content index of the independent variable.
A Population Stability Index (PSI) for measuring the distribution difference of the scores of the test sample and the model development sample, and is an index for evaluating the stability of the model. In fact, the population stability index indicates whether population distribution changes for different samples or samples at different times after grading by scores, namely whether the proportion of the number in each score interval to the total number changes significantly. It can be appreciated that the population stability index may reflect the stability of the model to some extent.
And an information quantity Index (IV) used for measuring the prediction capability of the independent variable in the model.
Specifically, the server stores configuration files set for each model index to be measured in advance. The configuration file comprises an abnormal detection correlation parameter. The abnormal detection associated parameters in the configuration file can be used for detecting whether the model index to be detected corresponding to the configuration file is abnormal or not.
It should be noted that different configuration files may be set correspondingly for different model indexes to be detected, or one configuration file may be set correspondingly for different model indexes to be detected, and the abnormality detection associated parameters corresponding to the different model indexes to be detected are expressed in the configuration files in a differentiated manner by the identifiers of the model indexes.
In one embodiment, the anomaly detection associated parameter includes a sample selection time. The sample selecting time is used for indicating which time ranges generate data to be selected as sample data. In one embodiment, the sample selection time may be a single time, such as 1 month 1 day 2018. The sample selection time may also be a time period, i.e., a sample selection time period. In one embodiment, the sample selection time period may include a start time and an end time (e.g., 1/2018 to 31/10/2018).
Further, the anomaly detection related parameters may further include at least one of an independent variable field and a dependent variable field of the current model. The argument field of the current model is a field indicating an argument that is specified and requires information amount index abnormality detection. For example, a field indicating "gender", "age", etc. may be referred to as an argument field. Multiple argument fields may be specified in the configuration file.
It should be noted that the abnormality detection related parameters set corresponding to different model indexes may be different, and are specifically determined according to the model index to be calculated.
In one embodiment, before step S202, the method further comprises: and acquiring a detection request, analyzing a detection object field in the detection request, and extracting the identifier of the model index to be detected corresponding to the detection object field. Step S202 includes: and acquiring a configuration file storage address mapped with the identifier of the model index to be detected according to the mapping relation between the preset identifier of the model index and the configuration file storage address, and acquiring a configuration file comprising the abnormal detection associated parameters according to the configuration file storage address.
And S204, calling a general code which is set corresponding to the index of the model to be detected, and extracting the sample selection time corresponding to the current model from the abnormal detection associated parameters.
Wherein generic code, i.e. generic computer code, is used to calculate the values of the model indices.
In one embodiment, the generic code may be a JAR (Java archive) package. Among them, the JAR package is a Java (Java is a door-oriented object programming language) archive file in a software package format. The JAR package is equivalent to a general computing code template.
It can be understood that when the universal code is called, the abnormal detection associated parameter can be obtained from the configuration file, and then corresponding sample data is obtained according to the abnormal detection associated parameter, so that the numerical value of the model index is calculated.
The current model is a machine learning model obtained through current training. The current model is obtained by training sample data in advance, and the training process of the current model can be not considered in the model anomaly detection method. The sample selection time corresponding to the current model means that the sample selection time in the configuration file coincides with the generation time of the sample data used for training the current model. That is, the sample data used to train the current model in advance is consistent with the sample data that corresponds to the sample selection time in the configuration file.
It can be understood that the development of the machine learning model needs to go through a long period, and the machine learning model needs to be repeatedly trained through sample data collected at different periods, so that the machine learning model can be formally deduced after being ensured to be stable. Therefore, it is necessary to perform anomaly detection on the machine learning model trained by using the sample data acquired at each time period.
And S206, acquiring sample data which accords with the sample selection time from the database through the universal code.
Specifically, the computer device may select, as the sample data, data from the database that results from the sample selection time by executing the general-purpose code.
For example, the sample selection time is 1/10/31/2018, and the universal code can select data generated between 1/2018 and 31/10/31/2018 from the database as sample data.
And S208, determining the value of the model index to be measured through the universal code and the sample data.
Specifically, a calculation method for the model index to be measured is set in the general code, and the computer device can calculate the value of the model index to be measured by combining the calculation method set in the general code with the sample data.
It can be understood that the called universal code is set corresponding to the model index to be tested, and the universal codes corresponding to different model indexes to be tested may be different. Therefore, the calculation method performed by the general-purpose code for calculating the value of the model index to be measured may be different for different model indexes to be measured.
It should be noted that, for some model indexes to be measured, the computer device may directly calculate the value of the model index to be measured only through the common code and the sample data. For example, when the information amount index of the argument is calculated, the information amount index of the argument can be directly calculated by the common code and the sample data. However, for some model indices under test, the computer device may determine the value of the model index under test in combination with the current model, in addition to the code and sample data. For example, when calculating the group stability index, the value of the model index to be measured is determined according to the universal code, the current model and the sample data by combining the current model.
S210, when the abnormal model index of the current model is determined according to the value of the model index to be detected, an alarm logic code is triggered to generate an abnormal reminding mail.
Specifically, the computer device may determine whether the model index to be measured of the current model is abnormal according to the value of the model index to be measured. When the model index to be detected is judged to be abnormal, the computer equipment can trigger the alarm logic code to generate an abnormal reminding mail.
It is understood that the abnormality reminding mail may include only the description information of the abnormality model index. The description information of the abnormal model index, that is, the information describing the model index determined as abnormal, is used to inform which index model is abnormal. In other embodiments, the abnormality reminding mail may further include abnormality cause description information. And the abnormal reason description information is used for describing the reason of the abnormal occurrence.
The computer device may perform a test for at least one model index under test. When detecting that all the plurality of model indexes to be detected are abnormal, the computer equipment can generate a uniform abnormal reminding mail aiming at the plurality of detected abnormal model indexes. Namely, the description information of a plurality of abnormal model indexes is integrated and gathered, and the abnormal reminding mails are generated uniformly.
It should be noted that the computer device may determine whether the model index to be measured is abnormal only according to the value itself of the model index to be measured. For example, when the model index to be measured is a group stability index, the computer device may directly compare the value of the group stability index with a preset abnormal threshold, and when the value is greater than or equal to the preset abnormal threshold, it may determine that the group stability index is abnormal.
S212, sending an abnormal reminding mail to the terminal.
Specifically, the computer device may send an abnormality alert mail to the terminal. The terminal 110 may display the abnormal reminding mail, and a technician may know that the model index of the current model is abnormal according to the abnormal reminding mail.
In one embodiment, a technician may analyze an abnormal cause causing the abnormality, perform a repair operation for the abnormal cause based on the terminal, and the terminal acquires repair non-operation data and submits the repair operation data to the computer device. The computer device can perform corresponding repair processing according to the repair operation data.
In one embodiment, the cause of the anomaly comprises that the sample data is incorrect. The inaccuracy of the sample data includes inaccuracy of the sample data size and inaccuracy of the data content of the sample data. The causes of the anomaly that results in incorrect sample data may include a change in the source of the sample data and a change in the traffic information. The terminal can obtain repair operation data generated by technicians performing different repair operations for different abnormal reasons, so that the computer equipment can perform corresponding repair processing on the repair operation data. The service information change includes a service content change and a service flow change. The service content refers to the content of the service substantive.
For example, originally, data providing sample data is stored in the table a, and subsequently, a table B is added, and the data is separately stored in the two tables, namely the table a and the table B. Therefore, the sample data source is changed from one table to two tables, but before the repair, the sample data is still only acquired from the table a, so that the data volume of the acquired sample data is greatly reduced, and the current model is affected. It can be understood that the data volume of the sample data is greatly reduced, which may cause the training of the current model to be inaccurate due to the insufficient data volume.
Therefore, when the sample data source is determined to be changed according to the abnormal reminding mail, the technician can perform repairing operation on the acquired path of the sample data based on the terminal, and the terminal can send the repaired path information to the computer equipment. The computer equipment can carry out corresponding restoration processing according to the restored path information, so that after restoration is finished, correct sample data can be obtained according to the restored path information, and accuracy of the current model is guaranteed. In one embodiment, the computer device may update the repaired path information in a configuration file that includes the anomaly detection associated parameters to complete the repair process.
For another example, a previous service person binds 10 general users, and changes from 3 months in 2018 to bind 5 good users, which may belong to service content change. Of course, the service content change is not limited to this example. It can be understood that, since the service content changes may involve data changes, data before the service content changes and data after the service content changes may have differences, and sample data before the service content changes is still acquired before the repair, the acquired sample data may not conform to the changed service. Therefore, when the service flow is determined to be changed or the service content is determined to be changed according to the abnormal reminding mail, a technician can repair the sample selection time in the configuration file based on the terminal so as to modify the sample selection time into the time after the service content is changed, and the computer equipment can update the modified sample selection time into the configuration file, so that the data after the service content is changed can be selected as sample data according to the modified sample selection time in the configuration file, and the accuracy of the current model is improved. By combining the above example, sample data can be extracted from the data after 2018 and 3 months.
In one embodiment, the method further comprises: after the repairing process is finished, obtaining sample data which accords with the sample selection time from the database through the universal code again; and iteratively adjusting the weight of each variable of the current model according to the newly acquired sample data until a training stop condition is reached to obtain a normal model.
It can be understood that the sample data obtained again after the restoration is accurate, so that the weights of the variables of the current model are iteratively adjusted based on the accurate sample data until the training stopping condition is reached, and a relatively accurate normal model can be obtained.
According to the model anomaly detection method, the model anomaly detection device, the computer equipment and the storage medium, the configuration file comprising the anomaly detection associated parameters is set for the indexes of the model to be detected, the universal code is preset for the indexes of the model to be detected, and the sample selection time corresponding to the current model can be extracted from the anomaly detection associated parameters by calling the universal code; acquiring sample data which accords with the sample selection time from a database through the universal code; determining the value of the model index to be tested through the general code and the sample data; therefore, the abnormal detection of the model to be detected of the current model can be realized by the universal codes and the configuration files, the abnormal detection of the model can be realized without all codes, and the abnormal processing efficiency is improved. Moreover, the universal code has universality, and the code does not need to be written for each detection, so that the applicability is improved.
In addition, when the model index to be detected of the current model is determined to be abnormal according to the value of the model index to be detected, an alarm logic code is triggered to generate an abnormal reminding mail; and sending the abnormal reminding mail to a terminal. When the abnormity is detected, the abnormity reminding mail is directly triggered and generated, so that the abnormity condition is more efficiently notified, and the abnormity processing efficiency can also be improved.
In addition, for personnel who cannot write complex codes (for example, new staff or staff with slightly deficient technology), the abnormity detection work can be completed more simply and conveniently, the difficulty of abnormity detection is reduced, and the abnormity detection efficiency is improved.
In one embodiment, the sample selection time is a sample selection time period comprising a plurality of months; step S208 includes: when the model index to be measured is a group stability index, circularly selecting the current month from the first month in the sample selection time period by month through the universal code, and acquiring first sub-sample data corresponding to the current month and second sub-sample data corresponding to the next month of the current month in the sample data; determining a first prediction result and a second prediction result which are obtained by inputting the first sub-sample data and the second sub-sample data into the current model respectively for prediction and outputting; and analyzing the first prediction result and the second prediction result through the universal code to obtain a value of the group stability index corresponding to the current month.
Specifically, the sample selection period includes a plurality of months. The computer device can take the first month as the current month, extract the first subsample data corresponding to the first month and the second subsample data corresponding to the second month from the sample data, and then input the first subsample data and the second subsample data into the current model through the universal code for prediction to obtain a first prediction result and a second prediction result. The computer equipment can analyze the first prediction result and the second prediction result through the general code to obtain a value of the group stability index corresponding to the first month of the current month. The computer device may then proceed with the second month as the current month, and in the current iteration, the sample data for the second month, which may be referred to as the first subsample data, and then extract the second subsample data corresponding to the third month (i.e., the month following the second month) from the sample data. Since the predicted result corresponding to the sample data of the second month has been calculated in the previous iteration and may be referred to as the first predicted result in the current iteration, it may not be necessary to predict the sample data again. And the computer equipment can input the second subsample data into the current model through the universal code for prediction to obtain a second prediction result of the current iteration, and further analyzes the first prediction result and the second prediction result through the universal code to obtain a value of the group stability index corresponding to the second month. The computer device may continue to select the third month as the current month, and continue to process to obtain the value of the group stability indicator corresponding to the third month. And repeating the steps until the values of the group stability indexes corresponding to all months included in the sample selection time period are determined.
In one embodiment, the prediction result may be a class probability of each sample data of the prediction. Class probability, i.e. category probability, is used to indicate the concept that the sample data belongs to a certain category. The first prediction result may be a first class probability of each of the first subsampled data being predicted. The second prediction result may be a second type probability of each second subsample data being predicted.
The computer device may sort the first class probabilities corresponding to the first sub-sample data from small to large, equally divide a first sub-sample data set formed by the first sub-sample data according to a preset number of groups (the number of the first sub-sample data in each group is consistent, that is, equal-width groups), and determine the first class probability of the maximum and minimum prediction of each group. And according to the upper and lower boundaries (namely the first class probabilities of the upper and lower boundaries) of each equal division after the first sub-sample data set is equally divided, combining the second class probabilities of the second sub-sample data, and grouping the second sub-sample data to obtain equally divided groups which accord with the preset group number. The computer device may obtain the group stability index corresponding to the current month according to the following formula: PSI ═ sum ((actual to expected ratio) × ln). Wherein the actual ratio is the ratio of the second subsample data within each bisection limit demarcated by the first class probability by the second class probability, and the expected ratio is the ratio of each bisection of the first subsample data on the first subsample data set.
It is understood that the computer device may determine whether the population stability indicator of the current model is abnormal according to the calculated value of the population stability indicator corresponding to each month. If abnormal, step S210 is performed.
In one embodiment, the method further comprises: comparing the obtained values of the group stability indexes corresponding to the months in the sample selection time period with preset abnormal threshold values respectively; determining an abnormal month corresponding to the value of the population stability index larger than the abnormal threshold; and when the number of the abnormal months is larger than a preset number threshold, judging that the group stability index of the current model is abnormal.
Wherein the preset abnormal threshold is a preset empirical value. In one embodiment, the anomaly threshold may be 0.25.
Specifically, the computer device may compare the value of the group stability indicator corresponding to each month in the sample selection time period with a preset abnormal threshold, and when the value of the group stability indicator is greater than or equal to the preset abnormal threshold, determine that the month corresponding to the value of the group stability indicator is an abnormal month. The computer device may count the abnormal months, compare the number of the abnormal months with a preset number threshold, and determine that the population stability index of the current model is abnormal when the number of the abnormal months is greater than or equal to the preset number threshold. And when the number of the abnormal months is smaller than a preset number threshold, judging that the group stability index of the current model is normal.
It should be noted that, in other embodiments, the computer device may also directly determine that the population stability indicator of the current model is abnormal when determining that the value of the population stability indicator is greater than or equal to the preset abnormal threshold.
It can be understood that when it is determined that the population stability indicator of the current model is normal, it indicates that the data distribution is stable, that is, it indicates that the independent variable distribution is stable, and the independent variable and the weight of the current model can be used as the independent variable and the weight of the model detected next time.
In the embodiment, the sample selection time periods of the plurality of months are configured in the configuration file, and the general algorithm set in the general code is used, so that the value of the group stability index corresponding to each month is realized, the value of the group stability index can be quickly and conveniently calculated, and the anomaly detection efficiency is improved.
In one embodiment, step S208 includes: when the model index to be measured is the information quantity index, extracting the independent variable field of the current model included in the configuration file through the universal code; and determining the corresponding current information quantity index numerical value of the independent variable in the current model according to the sample data aiming at the independent variable represented by each independent variable field. In this embodiment, the method further includes: obtaining historical information quantity index values corresponding to the independent variables in the historical models respectively; determining the variation coefficient of the information quantity index of the independent variable according to the historical information quantity index value and the current information quantity index value corresponding to the independent variable; and when the variation coefficient is larger than or equal to the variation threshold value, judging that the information quantity index of the independent variable is abnormal.
The Coefficient of Variation (coeffient of Variation) is the ratio of the standard deviation of data to the mean of the data.
Specifically, the computer device may group the independent variable, each obtained independent variable group corresponds to a certain amount of sample data, and the summary of the sample data corresponding to each independent variable group is the obtained sample data conforming to the sample selection time. And aiming at each independent variable group, determining an evidence weight corresponding to the independent variable group according to the sub-sample data corresponding to the independent variable group, determining a sub-information quantity index value corresponding to the independent variable group according to the evidence weight of each independent variable group, and adding the sub-information quantity index values corresponding to the respective independent variable groups to obtain a current information quantity index value corresponding to the independent variable in the current model.
The computer equipment can directly acquire the historical information quantity index values respectively corresponding to the recorded independent variables in each historical model. The computer equipment can also determine the corresponding historical information quantity index numerical value of the independent variable in each historical model according to the historical sample data used for training each historical model. It can be understood that the method for determining the historical information quantity index value corresponding to the independent variable in each historical model by the computer device according to the historical sample data is the same as the above-mentioned method for obtaining the current information quantity index value, and is not described herein again.
The computer device may use each historical information amount index value and current information amount index value of the same independent variable as a group of data sets, calculate a standard deviation for the data sets, calculate an average value for the group of data sets, and obtain a variation coefficient of the information amount index of the independent variable by dividing the standard deviation by the average value. The computer device may compare the variation coefficient with a preset variation threshold, and determine that the information amount indicator of the independent variable is abnormal when the variation coefficient is greater than or equal to the variation threshold. On the contrary, when the variation coefficient is smaller than the variation threshold, it can be determined that the information quantity index of the independent variable is normal, and the prediction of the independent variable is stable, so that the independent variable can be screened out to be used as the independent variable of the model detected next time.
In one embodiment, the triggering the alarm logic code to generate an exception alert mail includes: triggering an alarm logic code to obtain a type identifier of a type to which the abnormal model index belongs; searching an abnormal reason set corresponding to the acquired type identifier according to a preset corresponding relation between the type identifier and the abnormal reason set; and generating an abnormal reminding mail according to the mail content comprising the abnormal reason set.
Specifically, when it is determined that the model index to be measured of the current model is abnormal, the general code may trigger an alarm logic code. The alarm logic code can directly acquire the type identification carried by the model index which is judged to be abnormal. The alarm logic code can also search the type identifier corresponding to the identifier of the model index according to the preset corresponding relationship between the identifier of the model index and the type identifier.
Further, the computer device may search the abnormal cause set corresponding to the obtained type identifier according to a preset correspondence between the type identifier and the abnormal cause set. The computer equipment can take the abnormal reason set as mail content to generate an abnormal reminding mail. The abnormality cause set may be all or part of the mail content, and is not limited to this.
In other embodiments, the computer device may obtain the recorded historical abnormal reasons, analyze the historical abnormal reasons, determine a high-frequency abnormal reason according to the occurrence frequency of each historical abnormal reason, generate abnormal reminding description information according to the high-frequency abnormal reason, and display the abnormal reminding description information in the abnormal reminding mail to prompt the abnormal reason.
In the embodiment, whether the information quantity index of the independent variable is abnormal or not is automatically judged by configuring the sample selection time and the independent variable field of the current model in the configuration file and the general algorithm set in the general code, so that the abnormality detection efficiency is improved. In addition, whether the information quantity index of the independent variable is abnormal or not is judged through the variation coefficient, and the accuracy of the abnormal judgment result is greatly improved, so that the accuracy of the abnormal detection is improved.
As shown in fig. 3, in an embodiment, the method further includes an abnormality repairing step, specifically including the following steps:
s302, determining a target abnormal reason selected from the abnormal reason set positioned in the abnormal reminding mail.
The target abnormality cause is the final abnormality cause determined.
Specifically, after receiving the abnormal reminding mail, the technician may analyze the final target abnormal reason based on the abnormal reason set reminded in the abnormal reminding mail. The technician can directly select a target abnormal reason from the abnormal reason set presented by the abnormal reminding mail. The terminal can report the selected target abnormal reason to the computer equipment.
S304, sending a repair page corresponding to the target abnormal reason to the terminal.
Specifically, the computer device sets a corresponding repair page in advance for each abnormal cause in the abnormal cause set. The computer device can acquire the repair page corresponding to the target abnormal reason and return the repair page to the terminal.
S306, the repair operation data which is sent by the terminal and is acquired based on the repair page and aims at the target abnormal reason is received.
In particular, the terminal may present the repair page. The technical staff can carry out the repairing operation aiming at the target abnormal reason based on the repairing page, and the terminal can obtain the corresponding repairing operation data and report the repairing operation data to the computer equipment.
And S308, performing corresponding repair processing according to the repair operation data.
In one embodiment, the repair operation data includes repaired path information. The computer equipment can carry out corresponding restoration processing according to the restored path information, so that after restoration is finished, correct sample data can be obtained according to the restored path information, and accuracy of the current model is guaranteed. In one embodiment, the computer device may update the repaired path information in a configuration file that includes the anomaly detection associated parameters to complete the repair process.
In one embodiment, the repair operation data includes a modified sample selection time. The computer equipment can update the modified sample selection time into the configuration file, so that the data after the service content is changed can be selected as sample data according to the modified sample selection time in the configuration file, and the accuracy of the current model is improved.
In one embodiment, the method further comprises: after the repairing process is finished, obtaining sample data which accords with the sample selection time from the database through the universal code again; and iteratively adjusting the weight of each variable of the current model according to the newly acquired sample data until a training stopping condition is reached to obtain a normal model.
In the embodiment, the technical personnel can directly trigger the entering of the repair page for repair operation based on the abnormal reminding mail, so that abnormal repair can be carried out very quickly, and the repair efficiency is improved.
It should be noted that the first and second embodiments in the present application are only used for distinction, and are not used as limitations in terms of size, dependency, or precedence.
As shown in fig. 4, there is provided a model anomaly detection apparatus 400, the apparatus 400 including: an obtaining module 402, an anomaly detection module 404, and an anomaly reminding module 406, wherein:
an obtaining module 402, configured to obtain a configuration file including an anomaly detection associated parameter, where the configuration file is set for an index of a model to be detected.
An anomaly detection module 404, configured to invoke a universal code set corresponding to the model index to be detected, and extract a sample selection time corresponding to the current model from the anomaly detection associated parameters; acquiring sample data which accords with the sample selection time from a database through the universal code; determining the value of the model index to be tested through the general code and the sample data; and when the model index to be detected of the current model is determined to be abnormal according to the value of the model index to be detected, triggering an alarm logic code to generate an abnormal reminding mail.
And an exception reminding module 406, configured to send the exception reminding email to the terminal.
In one embodiment, the sample selection time is a sample selection time period comprising a plurality of months; the anomaly detection module 404 is further configured to, when the model index to be detected is a group stability index, select a current month cyclically month by month from a first month in the sample selection time period through the common code, and obtain first sub-sample data corresponding to the current month and second sub-sample data corresponding to a next month of the current month in the sample data; determining a first prediction result and a second prediction result which are obtained by inputting the first sub-sample data and the second sub-sample data into the current model respectively for prediction and outputting; and analyzing the first prediction result and the second prediction result through the universal code to obtain a value of the group stability index corresponding to the current month.
In one embodiment, the anomaly detection module 404 is further configured to compare the obtained values of the population stability indicators corresponding to the respective months in the sample selection time period with preset anomaly thresholds respectively; determining an abnormal month corresponding to the value of the group stability index larger than the abnormal threshold; and when the number of the abnormal months is larger than a preset number threshold, judging that the group stability index of the current model is abnormal.
In one embodiment, the anomaly detection module 404 is further configured to, when the model index to be detected is the information quantity index, extract an argument field of the current model included in the configuration file through the common code; determining a corresponding current information quantity index numerical value of the independent variable in the current model according to the sample data aiming at the independent variable represented by each independent variable field; obtaining historical information quantity index values corresponding to the independent variables in the historical models respectively; determining the variation coefficient of the information quantity index of the independent variable according to the historical information quantity index value and the current information quantity index value corresponding to the independent variable; and when the variation coefficient is larger than or equal to the variation threshold value, judging that the information quantity index of the independent variable is abnormal.
In one embodiment, the exception alert module 406 is further configured to trigger the alarm logic code to obtain a type identifier of a type to which the model indicator for determining the exception belongs; searching an abnormal reason set corresponding to the acquired type identifier according to a preset corresponding relation between the type identifier and the abnormal reason set; and generating an abnormal reminding mail according to the mail content comprising the abnormal reason set.
In one embodiment, the apparatus 400 further comprises:
an exception recovery module 408, configured to determine a target exception cause selected from the exception cause set located in the exception alert e-mail; sending a repair page corresponding to the target abnormal reason to the terminal; the method comprises the steps of receiving repair operation data which are sent by a terminal and are acquired based on a repair page and aim at a target abnormal reason; and performing corresponding repair processing according to the repair operation data.
As shown in FIG. 5, in one embodiment, the apparatus 400 further includes an anomaly repair module 408 and a model adjustment module 410, wherein:
the model adjusting module 410 is configured to obtain sample data meeting the sample selection time from the database through the universal code again after the repair processing is completed; and iteratively adjusting the weight of each variable of the current model according to the newly acquired sample data until a training stopping condition is reached to obtain a normal model.
For the specific definition of the model anomaly detection device, reference may be made to the above definition of the model anomaly detection method, which is not described herein again. The modules in the model anomaly detection device can be wholly or partially realized by software, hardware and a combination thereof. The modules can be embedded in a hardware form or independent of a processor in the computer device, and can also be stored in a memory in the computer device in a software form, so that the processor can call and execute operations corresponding to the modules.
In one embodiment, a computer device is provided, which may be the server 120 in fig. 1, and its internal structure diagram may be as shown in fig. 6. The computer device includes a processor, a memory, and a network interface connected by a system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device comprises a nonvolatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of an operating system and computer programs in the non-volatile storage medium. The network interface of the computer device is used for communicating with an external terminal through a network connection. The computer program is executed by a processor to implement a model anomaly detection method.
Those skilled in the art will appreciate that the architecture shown in fig. 6 is merely a block diagram of some of the structures associated with the disclosed aspects and is not intended to limit the computing devices to which the disclosed aspects apply, as particular computing devices may include more or less components than those shown, or may combine certain components, or have a different arrangement of components.
In one embodiment, a computer device is provided, comprising a memory and a processor, the memory storing a computer program which, when executed by the processor, causes the processor to perform the steps of the above-described model anomaly detection method. Here, the steps of the model abnormality detection method may be steps in the model abnormality detection methods of the respective embodiments described above.
In one embodiment, a computer readable storage medium is provided, storing a computer program that, when executed by a processor, causes the processor to perform the steps of the above-described model anomaly detection method. Here, the steps of the model abnormality detection method may be steps in the model abnormality detection methods of the respective embodiments described above.
It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by hardware instructions of a computer program, which can be stored in a non-volatile computer-readable storage medium, and when executed, can include the processes of the embodiments of the methods described above. Any reference to memory, storage, database, or other medium used in the embodiments provided herein may include non-volatile and/or volatile memory, among others. Non-volatile memory can include read-only memory (ROM), Programmable ROM (PROM), Electrically Programmable ROM (EPROM), Electrically Erasable Programmable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDRSDRAM), Enhanced SDRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), Rambus Direct RAM (RDRAM), direct bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).
The technical features of the above embodiments can be arbitrarily combined, and for the sake of brevity, all possible combinations of the technical features in the above embodiments are not described, but should be considered as the scope of the present specification as long as there is no contradiction between the combinations of the technical features.
The above-mentioned embodiments only express several embodiments of the present application, and the description thereof is specific and detailed, but not to be understood as limiting the scope of the invention. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the concept of the present application, which falls within the scope of protection of the present application. Therefore, the protection scope of the present patent shall be subject to the appended claims.
Claims (10)
1. A method of model anomaly detection, the method comprising:
acquiring a configuration file which is set aiming at the indexes of the model to be detected and comprises abnormal detection associated parameters;
calling a general code corresponding to the index of the model to be detected, and extracting sample selection time corresponding to the current model from the abnormal detection associated parameters; the general codes are general computer codes provided with a calculation method aiming at the model indexes to be measured; the sample selection time is a sample selection time period comprising a plurality of months;
acquiring sample data which accords with the sample selection time from a database through the universal code;
determining the value of the model index to be tested through the universal code and the sample data comprises: when the model index to be tested is a group stability index, selecting a current month from a first month in the sample selection time period by circulating month by month through the universal code, and acquiring first sub-sample data corresponding to the current month and second sub-sample data corresponding to a next month of the current month in the sample data; determining a first prediction result and a second prediction result which are obtained by inputting the first sub-sample data and the second sub-sample data into the current model respectively for prediction and outputting; analyzing the first prediction result and the second prediction result through the universal code to obtain a value of a group stability index corresponding to the current month;
when the model index to be detected of the current model is determined to be abnormal according to the value of the model index to be detected, triggering an alarm logic code to generate an abnormal reminding mail;
and sending the abnormal reminding mail to a terminal.
2. The method of claim 1, further comprising:
comparing the obtained values of the group stability indexes corresponding to the months in the sample selection time period with preset abnormal threshold values respectively;
determining an abnormal month corresponding to the value of the population stability index larger than the abnormal threshold;
and when the number of the abnormal months is larger than a preset number threshold, judging that the group stability index of the current model is abnormal.
3. The method according to claim 1, wherein the abnormality detection related parameter is a related parameter for abnormality detection of a model index.
4. The method of claim 1, wherein said determining the value of the model under test metric by the generic code and the sample data further comprises:
when the model index to be tested is the information quantity index, extracting the independent variable field of the current model included in the configuration file through the general code;
determining a corresponding current information quantity index numerical value of the independent variable in a current model according to the sample data aiming at the independent variable represented by each independent variable field;
the method further comprises the following steps:
acquiring historical information quantity index values respectively corresponding to the independent variables in the historical models;
determining a variation coefficient of an information quantity index of the independent variable according to a historical information quantity index value and a current information quantity index value corresponding to the independent variable;
and when the variation coefficient is larger than or equal to the variation threshold value, judging that the information quantity index of the independent variable is abnormal.
5. The method of claim 1, wherein the triggering alarm logic code to generate an exception alert mail comprises:
triggering an alarm logic code to obtain a type identifier of a type to which the abnormal model index belongs;
searching an abnormal reason set corresponding to the acquired type identifier according to a preset corresponding relation between the type identifier and the abnormal reason set;
and generating an abnormal reminding mail according to the mail content comprising the abnormal reason set.
6. The method of claim 5, further comprising:
determining a target abnormal reason selected from an abnormal reason set positioned in the abnormal reminding mail;
sending a repair page corresponding to the target abnormal reason to the terminal;
receiving repair operation data which is sent by the terminal and is acquired based on the repair page and aims at the target abnormal reason;
and performing corresponding repair processing according to the repair operation data.
7. The method of claim 6, further comprising:
after the repairing process is finished, obtaining sample data which accords with the sample selecting time from a database through the universal code again;
and iteratively adjusting the weight of each variable of the current model according to the newly acquired sample data until a training stopping condition is reached to obtain a normal model.
8. A model anomaly detection apparatus, characterized in that the apparatus comprises:
the acquisition module is used for acquiring a configuration file which is set aiming at the index of the model to be detected and comprises an abnormal detection correlation parameter;
the anomaly detection module is used for calling a general code which is set corresponding to the index of the model to be detected and extracting sample selection time corresponding to the current model from the anomaly detection associated parameters; the general codes are general computer codes provided with a calculation method aiming at the model indexes to be measured; the sample selection time is a sample selection time period comprising a plurality of months; acquiring sample data which accords with the sample selection time from a database through the universal code; determining the value of the model index to be tested through the general code and the sample data, wherein the step of determining the value of the model index to be tested comprises the following steps: when the model index to be tested is a group stability index, circularly selecting a current month from a first month in the sample selection time period month by month through the general code, and acquiring first sub-sample data corresponding to the current month and second sub-sample data corresponding to a next month of the current month in the sample data; determining a first prediction result and a second prediction result which are obtained by inputting the first sub-sample data and the second sub-sample data into the current model respectively for prediction and outputting; analyzing the first prediction result and the second prediction result through the general code to obtain a value of a group stability index corresponding to the current month;
the abnormality detection module is also used for triggering an alarm logic code to generate an abnormality reminding mail when the abnormality of the model index to be detected of the current model is determined according to the value of the model index to be detected;
and the abnormity reminding module is used for sending the abnormity reminding mail to a terminal.
9. A computer device comprising a memory and a processor, the memory storing a computer program, wherein the processor implements the steps of the method of any one of claims 1 to 7 when executing the computer program.
10. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the method of any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910603347.8A CN110489314B (en) | 2019-07-05 | 2019-07-05 | Model anomaly detection method and device, computer equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910603347.8A CN110489314B (en) | 2019-07-05 | 2019-07-05 | Model anomaly detection method and device, computer equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110489314A CN110489314A (en) | 2019-11-22 |
| CN110489314B true CN110489314B (en) | 2022-07-12 |
Family
ID=68546769
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910603347.8A Active CN110489314B (en) | 2019-07-05 | 2019-07-05 | Model anomaly detection method and device, computer equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110489314B (en) |
Families Citing this family (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110928859A (en) * | 2019-11-26 | 2020-03-27 | 太平金融科技服务（上海）有限公司 | Model monitoring method and device, computer equipment and storage medium |
| CN110889117B (en) * | 2019-11-28 | 2022-04-19 | 支付宝(杭州)信息技术有限公司 | Method and device for defending model attack |
| CN111459778B (en) * | 2020-03-12 | 2024-05-07 | 平安科技（深圳）有限公司 | Operation and maintenance system abnormal index detection model optimization method, device and storage medium |
| CN112306814A (en) * | 2020-11-13 | 2021-02-02 | 苏州浪潮智能科技有限公司 | Network card temperature measuring method and device, computer equipment and storage medium |
| CN112508202B (en) * | 2021-02-07 | 2021-07-30 | 北京淇瑀信息科技有限公司 | Method and device for adjusting model stability and electronic equipment |
| CN113656389B (en) * | 2021-08-12 | 2022-05-27 | 北京可视化智能科技股份有限公司 | Intelligent factory abnormal data processing method, device and system and storage medium |
| CN114780400B (en) * | 2022-04-18 | 2023-06-27 | 南京安元科技有限公司 | Method for blocking cyclic call between services based on periodic data balance statistics |
| CN115757181B (en) * | 2022-12-05 | 2023-05-16 | 飞狐信息技术（天津）有限公司 | Code detection method, device, equipment and readable storage medium |
| CN115658550B (en) * | 2022-12-09 | 2023-04-07 | 合肥高维数据技术有限公司 | Automatic testing method and system for improving large-scale sample testing efficiency |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO1998006019A2 (en) * | 1996-08-07 | 1998-02-12 | Intrinsa Corporation | Simulated program execution error detection method and apparatus |
| CN103197663A (en) * | 2013-03-07 | 2013-07-10 | 北京信息科技大学 | Method and system of failure prediction |
| CN106708638A (en) * | 2015-11-12 | 2017-05-24 | 博雅网络游戏开发（深圳）有限公司 | System error detection method and apparatus |
| CN106951359A (en) * | 2017-02-28 | 2017-07-14 | 深圳市华傲数据技术有限公司 | A kind of system health degree determination method and device |
| CN109684118A (en) * | 2018-12-10 | 2019-04-26 | 深圳前海微众银行股份有限公司 | Detection method, device, equipment and the computer readable storage medium of abnormal data |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US11037080B2 (en) * | 2017-10-05 | 2021-06-15 | Aconex Limited | Operational process anomaly detection |
-
2019
- 2019-07-05 CN CN201910603347.8A patent/CN110489314B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO1998006019A2 (en) * | 1996-08-07 | 1998-02-12 | Intrinsa Corporation | Simulated program execution error detection method and apparatus |
| CN103197663A (en) * | 2013-03-07 | 2013-07-10 | 北京信息科技大学 | Method and system of failure prediction |
| CN106708638A (en) * | 2015-11-12 | 2017-05-24 | 博雅网络游戏开发（深圳）有限公司 | System error detection method and apparatus |
| CN106951359A (en) * | 2017-02-28 | 2017-07-14 | 深圳市华傲数据技术有限公司 | A kind of system health degree determination method and device |
| CN109684118A (en) * | 2018-12-10 | 2019-04-26 | 深圳前海微众银行股份有限公司 | Detection method, device, equipment and the computer readable storage medium of abnormal data |
Non-Patent Citations (1)
| Title |
|---|
| 基于Fisher判别分析法的间歇过程故障检测与诊断研究;郑晓慧;《中国优秀博硕士学位论文全文数据库(硕士)》;20180415(第04期);I140-972 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110489314A (en) | 2019-11-22 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110489314B (en) | Model anomaly detection method and device, computer equipment and storage medium | |
| CN110730107B (en) | Test data generation method and device, computer equipment and storage medium | |
| CN111177714B (en) | Abnormal behavior detection method and device, computer equipment and storage medium | |
| CN110083514B (en) | Software test defect evaluation method and device, computer equipment and storage medium | |
| CN111176990A (en) | Test data generation method and device based on data decision and computer equipment | |
| CN110929036A (en) | Electric power marketing inspection management method and device, computer equipment and storage medium | |
| CN110262939B (en) | Algorithm model operation monitoring method, device, computer equipment and storage medium | |
| CN112363920B (en) | Repair method and device for test cases, computer equipment and storage medium | |
| CN112559365A (en) | Test case screening method and device, computer equipment and storage medium | |
| CN110868241A (en) | Underground cable fault early warning method and device based on multiple models | |
| CN110941555A (en) | Test case recommendation method and device, computer equipment and storage medium | |
| CN112817843B (en) | Project management method and system | |
| CN116136950B (en) | Chip verification method, device, system, electronic equipment and storage medium | |
| CN110928859A (en) | Model monitoring method and device, computer equipment and storage medium | |
| CN111651358B (en) | Method for generating test case, software test method, device and server | |
| CN113282514A (en) | Problem data processing method and device, computer equipment and storage medium | |
| CN111124898B (en) | Question-answering system testing method and device, computer equipment and storage medium | |
| CN114610635A (en) | Interface test case generation method and device, computer equipment and storage medium | |
| CN112817847A (en) | Data processing task testing method and device, electronic equipment and storage medium | |
| CN115952081A (en) | Software testing method, device, storage medium and equipment | |
| CN113435664A (en) | Electricity charge abnormal data analysis method and device, terminal device and medium | |
| CN111767192A (en) | Service data detection method, device, equipment and medium based on artificial intelligence | |
| CN111444093A (en) | Method and device for determining quality of project development process and computer equipment | |
| CN114064497A (en) | Application program testing method and device, computer equipment and storage medium | |
| CN117391306A (en) | Homeland space planning result examination method, device, equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |