CN110458713B - Model monitoring method, device, computer equipment and storage medium - Google Patents
Model monitoring method, device, computer equipment and storage medium Download PDFInfo
- Publication number
- CN110458713B CN110458713B CN201910604966.9A CN201910604966A CN110458713B CN 110458713 B CN110458713 B CN 110458713B CN 201910604966 A CN201910604966 A CN 201910604966A CN 110458713 B CN110458713 B CN 110458713B
- Authority
- CN
- China
- Prior art keywords
- index
- target model
- current
- abnormal
- model
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q30/00—Commerce
- G06Q30/02—Marketing; Price estimation or determination; Fundraising
- G06Q30/0201—Market modelling; Market analysis; Collecting market data
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q40/00—Finance; Insurance; Tax strategies; Processing of corporate or income taxes
- G06Q40/08—Insurance
Landscapes
- Business, Economics & Management (AREA)
- Finance (AREA)
- Engineering & Computer Science (AREA)
- Accounting & Taxation (AREA)
- Development Economics (AREA)
- Strategic Management (AREA)
- General Business, Economics & Management (AREA)
- Entrepreneurship & Innovation (AREA)
- Economics (AREA)
- Marketing (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Game Theory and Decision Science (AREA)
- Data Mining & Analysis (AREA)
- Technology Law (AREA)
- Information Transfer Between Computers (AREA)
- Debugging And Monitoring (AREA)
Abstract
The embodiment of the invention discloses a model monitoring method, a model monitoring device, computer equipment and a storage medium. The method is applied to the field of model monitoring in model deployment. The method comprises the following steps: if the current time is detected to be the first preset time, acquiring a first current data index, a first current performance index and a first current effect index of the first target model; judging whether the first target model is abnormal or not according to the first current data index, the first current performance index and the first current effect index; if the first target model is abnormal, generating an abnormal message mail according to the number of the abnormal first target model, and sending the abnormal message mail to a preset mail address. By implementing the method of the embodiment of the invention, the data, the performance and the effect of the model can be monitored in all aspects, and the abnormal condition of the model can be found in time to inform monitoring personnel, so that the normal operation of the model is ensured.
Description
Technical Field
The present invention relates to the field of artificial intelligence technologies, and in particular, to a model monitoring method, a device, a computer device, and a storage medium.
Background
With the development of science and technology and the improvement of economy, the application of the model is more and more extensive, and various models gradually enter into daily life of people, so that great convenience is provided for the life of people. In the insurance industry, the user's behaviors are usually studied through a model to improve the sales of insurance, however, as the number of engineering project models of the user's behaviors increases, the number of running tasks of the model on production increases and the number of dependent tasks increases, and the limitation of resources affects the normal and effective running of the model.
Disclosure of Invention
The embodiment of the invention provides a model monitoring method, a device, computer equipment and a storage medium, which aim to solve the problem that the normal and effective operation of a model is influenced along with the increase of the number of operation tasks and the increase of dependent tasks of the model in production and the limitation of resources.
In a first aspect, an embodiment of the present invention provides a model monitoring method, including: detecting that the current time is a first preset time, and acquiring a first current data index, a first current performance index and a first current effect index of a first target model; judging whether the first target model is abnormal or not according to a preset interval in which the first current data index is located; comparing the first current performance index with the previous performance index to judge whether the first target model is abnormal or not; comparing the first current effect index with a previous effect index to judge whether the first target model is abnormal or not; if the first target model is abnormal, generating an abnormal message mail according to the number of the abnormal first target model, and sending the abnormal message mail to a preset mail address.
In a second aspect, an embodiment of the present invention further provides a model monitoring apparatus, including: the first acquisition unit is used for acquiring a first current data index, a first current performance index and a first current effect index of the first target model if the current time is detected to be a first preset time; the first judging unit is used for judging whether the first target model is abnormal or not according to a preset interval where the first current data index is located; the second judging unit is used for comparing the first current performance index with the previous performance index to judge whether the first target model is abnormal or not; the third judging unit is used for comparing the first current effect index with the previous effect index to judge whether the first target model is abnormal or not; the first generation unit is used for generating an abnormal message mail according to the number of the first target model with the abnormality if the first target model is abnormal, and sending the abnormal message mail to a preset mail address.
In a third aspect, an embodiment of the present invention further provides a computer device, which includes a memory and a processor, where the memory stores a computer program, and the processor implements the method when executing the computer program.
In a fourth aspect, embodiments of the present invention also provide a computer readable storage medium storing a computer program which, when executed by a processor, implements the above method.
The embodiment of the invention provides a model monitoring method, a model monitoring device, computer equipment and a storage medium. Wherein the method comprises the following steps: if the current time is detected to be the first preset time, acquiring a first current data index, a first current performance index and a first current effect index of the first target model; judging whether the first target model is abnormal or not according to a preset interval in which the first current data index is located; comparing the first current performance index with the previous performance index to judge whether the first target model is abnormal or not; comparing the first current effect index with a previous effect index to judge whether the first target model is abnormal or not; if the first target model is abnormal, generating an abnormal message mail according to the number of the abnormal first target model, and sending the abnormal message mail to a preset mail address. According to the embodiment of the invention, whether the first target model is abnormal or not is judged according to the first current data index, the first current performance index and the first current effect index, when the first target model is abnormal, an abnormal message mail is generated according to the abnormal first target model, so that the data, performance and effect of the model can be comprehensively monitored, the abnormal model can be timely found, and the effect of quick repair can be realized.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for the description of the embodiments will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a schematic flow chart of a model monitoring method according to an embodiment of the present invention;
FIG. 2 is a schematic sub-flowchart of a model monitoring method according to an embodiment of the present invention;
FIG. 3 is a schematic sub-flowchart of a model monitoring method according to an embodiment of the present invention;
FIG. 4 is a schematic sub-flowchart of a model monitoring method according to an embodiment of the present invention;
FIG. 5 is a flow chart of a model monitoring method according to another embodiment of the present invention;
FIG. 6 is a schematic block diagram of a model monitoring apparatus provided by an embodiment of the present invention;
FIG. 7 is a schematic block diagram of specific units of a model monitoring apparatus provided in an embodiment of the present invention;
FIG. 8 is a schematic block diagram of a model monitoring apparatus according to another embodiment of the present invention; and
fig. 9 is a schematic block diagram of a computer device according to an embodiment of the present invention.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and fully with reference to the accompanying drawings, in which it is evident that the embodiments described are some, but not all embodiments of the invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
It should be understood that the terms "comprises" and "comprising," when used in this specification and the appended claims, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.
It is also to be understood that the terminology used in the description of the invention herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used in this specification and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.
It should be further understood that the term "and/or" as used in the present specification and the appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes such combinations.
Referring to fig. 1, fig. 1 is a schematic flowchart of a model monitoring method according to an embodiment of the present invention. The model monitoring method is applied to the terminal, and the invention provides a model monitoring method, wherein the model is used for analyzing the behaviors of clients to improve the insurance sales, and the number of the models is multiple, such as an accurate marketing model, a preference model, a behavior analysis model and the like. As the product is changed more and more frequently along with the change of the times, the number of running tasks of the model on production is increased, the number of dependent tasks is increased, the resource is limited, the accuracy and the performance of the model are influenced, and the sales effect of the insurance product is reduced. Therefore, the model monitoring method provided by the invention is used for carrying out full-scale monitoring on the data, performance and effect of the model, and timely finding out the abnormality of the model to inform monitoring personnel so as to ensure the normal operation of the model.
Fig. 2 is a flow chart of a model monitoring method according to an embodiment of the present invention. As shown, the method includes the following steps S110-S150.
S110, if the current time is detected to be the first preset time, a first current data index, a first current performance index and a first current effect index of the first target model are obtained.
In an embodiment, the first target model refers to a daily model in which the run period of the model on production is one day, i.e. a model that is run once per day. The first preset time is a time at which the first target model is monitored periodically, for example, three weeks or every weekday. The first current data index, the first current performance index and the first current effect index are all calculated based on the first target model, the first target model can generate the three indexes every day, each time the first target model is operated, the calculated data index, the calculated performance index and the calculated effect index form records and are stored in a preset database, the three indexes are continuously overlapped and recorded every day in the preset database, and the preset database is used for storing data related to all models. Therefore, when the current time is detected to be the time of periodic monitoring, first, a first current data index, a first current performance index and a first current effect index corresponding to the first preset model are acquired from a preset database, and the current time is the latest index.
S120, judging whether the first target model is abnormal or not according to a preset interval where the first current data index is located.
Specifically, the first current data index refers to a data index obtained by calculating through a statistical method according to a current data set of the first target model, for example, a stability index PSI and an information value index IV. And evaluating and measuring an input data set of the first target model through the data index, and monitoring the first target model at the data level.
In one embodiment, as shown in fig. 2, the step S120 may include the steps of: S121-S122.
S121, judging the position of the first current data index in a preset interval, wherein the position of the preset interval comprises: an adjustment zone, a stabilization zone, and an observation zone.
S122, if the first current data index is in an adjustment area of a preset interval, judging that the first target model is abnormal.
In an embodiment, the preset interval refers to an interval divided according to the size of the data index, and is divided into an adjustment area, a stabilization area and an observation area. When the data index is positioned in the adjustment area, the current data set of the model is not applicable and needs to be adjusted; when the data index is positioned in the stable region, the current data set of the model is applicable, and adjustment is not needed; when the data index is located in the observation area, it is explained that the current data set of the model may need to be adjusted, but still further observation is needed. For example, the data index is a stability index PSI, which is used to measure the distribution difference between the current data set of the model and the initial data set when the model is built, and its specific formula is as follows:
If the PSI index is 0.3, the stable area of the preset area is 0-0.1, the observation area is 0.1-0.25, and the adjustment area is more than 0.25, then the current PSI index is 0.3 at the position of the adjustment area, which indicates that the stability of the current first target model is poor, so that the first target model is judged to be abnormal. For another example, if the IV index is 0.01, the adjustment area of the preset interval is 0-0.02, the observation area is 0.02-0.3, and the stability area is above 0.3, then the current IV index is 0.01 in the adjustment area, which indicates that the prediction ability of the current first target model is weak, so that it is determined that the first target model is abnormal. It will of course be appreciated that there are other data indicators, such as population count.
S130, comparing the first current performance index with a previous performance index to judge whether the first target model is abnormal or not.
Specifically, the first current performance index refers to performance indexes obtained by performing performance evaluation calculation on the current output of the first target model through the evaluation model, for example, hit rate, coverage rate and lifting degree of each fraction. And evaluating and measuring the accuracy of the output of the first target model through the performance index, and monitoring the first target model at the performance level.
In one embodiment, as shown in fig. 3, the step S130 may include the steps of: S131-S132.
S131, acquiring a previous performance index of the first target model and comparing the previous performance index with the first current performance index.
And S132, judging that the first target model is abnormal if the difference value between the previous performance index and the first current performance index exceeds a preset performance index threshold value.
In an embodiment, the previous performance index refers to a performance index obtained by performing performance evaluation calculation on the output of the first target model by an evaluation model when the first target model runs in the previous monitoring period, where the previous performance index is stored in a preset database after being obtained, and the evaluation model refers to a model for evaluating the performance of the first target model, for example, a hit rate model. Therefore, when monitoring is performed, the previous performance index is firstly obtained from a preset database, then the previous performance index is compared with the first current performance index, the change degree between the previous performance index and the first current performance index is judged, the abnormal condition of the first target model is judged according to whether the difference value between the previous performance index and the first current performance index exceeds a preset threshold value, if the difference value between the previous performance index and the first current performance index exceeds the preset threshold value, the performance change degree of the model is large, and the first target model is judged to be abnormal. For example, hit rate of each quantile, wherein the quantile is a quantile, such as a quarter, and clients are equally divided into four parts by three quantiles according to the number of clients after ordered according to rules; hit rate refers to the hit condition of model prediction. If the previous hit rate of the first bit is 30%, the current hit rate is 20%, and the preset threshold value is 5%, then it is determined that the first target model is abnormal. It will of course be appreciated that there are other performance metrics such as coverage and boost.
And S140, comparing the first current effect index with the previous effect index to judge whether the first target model is abnormal or not.
Specifically, the first current effect index refers to an effect index obtained by calculation according to the actual conversion condition of the current first target model, such as a conversion rate and a viewing rate. And evaluating and measuring the real marketing effect of the first target model through the effect index, and monitoring the first target model at the effect level.
In one embodiment, as shown in fig. 4, the step S140 may include the steps of: S141-S142.
S141, acquiring a previous effect index of the first target model and comparing the previous effect index with the first current effect index.
And S142, judging that the first target model is abnormal if the difference value between the previous effect index and the first current effect index exceeds a preset effect index threshold value.
In an embodiment, the previous effect index of the first target model refers to an effect index calculated according to a real conversion condition when the first target model runs in a previous monitoring period, and the previous effect index is stored in a preset database after being obtained. Therefore, when monitoring is performed, firstly, the previous effect index is obtained from a preset database, then, the previous effect index is compared with the first current effect index, the change degree between the previous effect index and the first current effect index is judged, the abnormal condition of the first target model is judged according to whether the difference value between the previous effect index and the first current effect index exceeds a preset threshold value, if the difference value between the previous effect index and the first current effect index exceeds the preset threshold value, the performance change degree of the model is large, and the first target model is judged to be abnormal. For example, the conversion rate may be that the customer who has not purchased the insurance purchases the insurance due to the effect of the model, or that the customer who has purchased the insurance purchases the insurance again due to the effect of the model, where the previous conversion rate is 20%, the current conversion rate is 10%, and the preset threshold is 5%, and then it is determined that the first target model is abnormal. It will be understood that other indicators of effectiveness, such as viewing rate, may also be used.
S150, if the first target model is abnormal, generating an abnormal message mail according to the number of the abnormal first target model, and sending the abnormal message mail to a preset mail address.
In one embodiment, each object model is pre-assigned a unique identification number, such as marking 001. When the monitoring finds that the first target model is abnormal, generating an abnormal message mail according to the number of the abnormal first target model, and reminding monitoring personnel in time. The subject of the abnormal message mail is the number of the first target model of the occurrence of the abnormality, and the content of the abnormal message mail comprises indexes of the occurrence of the abnormality and the degree of the occurrence of the abnormality, such as severity and slight. The preset mail address is the mail address of the monitoring personnel. Monitoring personnel find abnormal problem points through the model codes, repair the abnormal problem points rapidly and ensure the normal operation of the model. After knowing that a specific model is abnormal, monitoring personnel firstly check which indexes of the model are abnormal, and then check the indexes one by one according to the sequence of the data and the model according to the indexes of the abnormal occurrence, for example, when the crowd number in the age group is increased sharply, the abnormal occurrence of a data set can be found rapidly according to the checking sequence, and the abnormal occurrence of a target model is checked rapidly; if the data is not abnormal, the data is adjusted from the model, and the abnormal problem is solved by updating parameters of the model or replacing the model.
In an embodiment, as shown in fig. 5, the step S140 further includes the following steps: S160-S164.
And S160, if the current time is detected to be the second preset time, acquiring a second current data index, a second current performance index and a second current effect index of the second target model.
And S161, judging whether the second target model is abnormal according to the second current data index.
S162, judging whether the second target model is abnormal according to the second current performance index.
S163, judging whether the second target model is abnormal according to the second current effect index.
S164, if the second target model is abnormal, generating an abnormal message mail according to the number of the abnormal second target model, and sending the abnormal message mail to a preset mail address.
In an embodiment, the second target model refers to a month model in which the run time of the model on production is one month, i.e. a model that runs once a month. The second preset time is a time at which the second target model is monitored periodically, for example, monthly No. 1. The manner in which the second target model implements overall data, performance, and effect monitoring is similar to that of the first target model, and will not be described in detail herein.
The embodiment of the invention discloses a model monitoring method, which comprises the steps of obtaining a first current data index, a first current performance index and a first current effect index of a first target model through detecting that the current time is a first preset time; judging whether the first target model is abnormal or not according to a preset interval in which the first current data index is located; comparing the first current performance index with the previous performance index to judge whether the first target model is abnormal or not; comparing the first current effect index with a previous effect index to judge whether the first target model is abnormal or not; if the first target model is abnormal, generating an abnormal message mail according to the number of the first target model with the abnormality, and sending the abnormal message mail to a preset mail address, so that the data, performance and effect of the model can be monitored in all aspects, abnormal conditions of the model can be found in time, monitoring staff can be notified, and normal operation of the model is guaranteed.
Fig. 6 is a schematic block diagram of a model monitoring apparatus 200 according to an embodiment of the present invention. As shown in fig. 6, the present invention also provides a model monitoring apparatus 200 corresponding to the above model monitoring method. The model monitoring apparatus 200 includes a unit for performing the model monitoring method described above, and may be configured in a desktop computer, a tablet computer, a portable computer, or the like. Specifically, referring to fig. 6, the model monitoring apparatus 200 includes: the first acquisition unit 210, the first judgment unit 220, the second judgment unit 230, the third judgment unit 240, and the first generation unit 250.
The first obtaining unit 210 is configured to obtain a first current data index, a first current performance index, and a first current effect index of the first target model if the current time is detected to be a first preset time.
The first determining unit 220 is configured to determine whether the first target model is abnormal according to a preset interval in which the first current data index is located.
In one embodiment, as shown in fig. 7, the first determining unit 220 includes: a first determination subunit 221 and a first determination subunit 222.
A first judging subunit 221, configured to judge a position of the first current data indicator in a preset interval, where the position of the preset interval includes: an adjustment zone, a stabilization zone, and an observation zone;
the first determining subunit 222 is configured to determine that the first target model is abnormal if the first current data indicator is in the adjustment area of the preset interval.
A second judging unit 230, configured to compare the first current performance index with a previous performance index to judge whether the first target model is abnormal.
In one embodiment, as shown in fig. 7, the second determining unit 230 includes: the second comparing unit 231 and the second determining subunit 232.
A second comparing unit 231, configured to obtain a previous performance index of the first target model and compare the previous performance index with the first current performance index.
And a second determining subunit 232, configured to determine that the first target model is abnormal if the difference between the previous performance index and the first current performance index exceeds a preset performance index threshold.
A third judging unit 240, configured to compare the first current effect index with a previous effect index to judge whether the first target model is abnormal.
In one embodiment, as shown in fig. 7, the third determining unit 240 includes: the third comparing unit 241 and the third determining subunit 242.
And a third comparing unit 241, configured to obtain a previous effect index of the first target model and compare the previous effect index with the first current effect index.
And a third determining subunit 242, configured to determine that the first target model is abnormal if the difference between the previous effect index and the first current effect index exceeds a preset effect index threshold.
The first generating unit 250 is configured to generate an abnormal message mail according to the number of the first target model with the abnormality if the first target model is abnormal, and send the abnormal message mail to a preset mail address.
In one embodiment, as shown in fig. 8, the model monitoring apparatus 200 further includes: the second acquisition unit 260, the fourth judgment unit 261, the fifth judgment unit 262, the sixth judgment unit 263, and the second generation unit 264.
The second obtaining unit 260 is configured to obtain a second current data index, a second current performance index, and a second current effect index of the second target model if the current time is detected to be a second preset time.
A fourth judging unit 261, configured to judge whether the second target model is abnormal according to the second current data indicator.
And a fifth judging unit 262, configured to judge whether the second target model is abnormal according to the second current performance index.
A sixth judging unit 263, configured to judge whether the second target model is abnormal according to the second current effect index.
And a second generating unit 264, configured to generate an abnormal message mail according to the number of the abnormal second target model if the second target model is abnormal, and send the abnormal message mail to a preset mail address.
It should be noted that, as will be clearly understood by those skilled in the art, the specific implementation process of the model monitoring apparatus 200 and each unit may refer to the corresponding description in the foregoing method embodiments, and for convenience and brevity of description, the description is omitted herein.
The above-described model monitoring means may be implemented in the form of a computer program which can be run on a computer device as shown in fig. 9.
Referring to fig. 9, fig. 9 is a schematic block diagram of a computer device according to an embodiment of the present application. The computer device 500 may be a terminal, where the terminal may be an electronic device having a communication function, such as a smart phone, a tablet computer, a notebook computer, a desktop computer, a personal digital assistant, and a wearable device.
With reference to FIG. 9, the computer device 500 includes a processor 502, memory, and a network interface 505 connected by a system bus 501, where the memory may include a non-volatile storage medium 503 and an internal memory 504.
The non-volatile storage medium 503 may store an operating system 5031 and a computer program 5032. The computer program 5032 includes program instructions that, when executed, cause the processor 502 to perform a model monitoring method.
The processor 502 is used to provide computing and control capabilities to support the operation of the overall computer device 500.
The internal memory 504 provides an environment for the execution of a computer program 5032 in the non-volatile storage medium 503, which computer program 5032, when executed by the processor 502, causes the processor 502 to perform a model monitoring method.
The network interface 505 is used for network communication with other devices. It will be appreciated by those skilled in the art that the architecture shown in fig. 9 is merely a block diagram of some of the architecture relevant to the present inventive arrangements and is not limiting of the computer device 500 to which the present inventive arrangements may be implemented, as a particular computer device 500 may include more or fewer components than shown, or may combine some of the components, or have a different arrangement of components.
Wherein the processor 502 is configured to execute a computer program 5032 stored in a memory to implement the steps of: if the current time is detected to be the first preset time, acquiring a first current data index, a first current performance index and a first current effect index of the first target model; judging whether the first target model is abnormal or not according to a preset interval in which the first current data index is located; comparing the first current performance index with the previous performance index to judge whether the first target model is abnormal or not; comparing the first current effect index with a previous effect index to judge whether the first target model is abnormal or not; if the first target model is abnormal, generating an abnormal message mail according to the number of the abnormal first target model, and sending the abnormal message mail to a preset mail address.
In an embodiment, when the processor 502 determines whether the first target model has an abnormal step according to the preset interval in which the first current data index is located, the following steps are specifically implemented: judging the position of the first current data index in a preset interval, wherein the position of the preset interval comprises the following steps: an adjustment zone, a stabilization zone, and an observation zone; and if the first current data index is in an adjustment area of a preset interval, judging that the first target model is abnormal.
In an embodiment, when the processor 502 compares the first current performance index with the previous performance index to determine whether the first target model has an abnormal step, the following steps are specifically implemented: acquiring a previous performance index of the first target model and comparing the previous performance index with the first current performance index; and if the difference value between the previous performance index and the first current performance index exceeds a preset performance index threshold value, judging that the first target model is abnormal.
In an embodiment, when the processor 502 compares the first current effect index with the previous effect index to determine whether the first target model has an abnormal step, the following steps are specifically implemented: acquiring a previous effect index of the first target model and comparing the previous effect index with the first current effect index; and if the difference value between the previous effect index and the first current effect index exceeds a preset effect index threshold value, judging that the first target model is abnormal.
In an embodiment, after implementing the step of generating, by the processor 502, an abnormal message mail according to the number of the first target model in which the abnormality occurs if the first target model is abnormal, and sending the abnormal message mail to a preset mail address, the following steps are further implemented: if the current time is detected to be the second preset time, acquiring a second current data index, a second current performance index and a second current effect index of a second target model; judging whether the second target model is abnormal or not according to the second current data index; judging whether the second target model is abnormal or not according to the second current performance index; judging whether the second target model is abnormal or not according to the second current effect index; if the second target model is abnormal, generating an abnormal message mail according to the number of the abnormal second target model, and sending the abnormal message mail to a preset mail address.
It should be appreciated that in an embodiment of the application, the processor 502 may be a central processing unit (Central Processing Unit, CPU), the processor 502 may also be other general purpose processors, digital signal processors (Digital Signal Processor, DSPs), application specific integrated circuits (Application Specific Integrated Circuit, ASICs), off-the-shelf programmable gate arrays (Field-Programmable Gate Array, FPGAs) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or the like. Wherein the general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
Those skilled in the art will appreciate that all or part of the flow in a method embodying the above described embodiments may be accomplished by computer programs instructing the relevant hardware. The computer program comprises program instructions, and the computer program can be stored in a storage medium, which is a computer readable storage medium. The program instructions are executed by at least one processor in the computer system to implement the flow steps of the embodiments of the method described above.
Accordingly, the present invention also provides a storage medium. The storage medium may be a computer readable storage medium. The storage medium stores a computer program, wherein the computer program includes program instructions. The program instructions, when executed by the processor, cause the processor to perform the steps of: if the current time is detected to be the first preset time, acquiring a first current data index, a first current performance index and a first current effect index of the first target model; judging whether the first target model is abnormal or not according to a preset interval in which the first current data index is located; comparing the first current performance index with the previous performance index to judge whether the first target model is abnormal or not; comparing the first current effect index with a previous effect index to judge whether the first target model is abnormal or not; if the first target model is abnormal, generating an abnormal message mail according to the number of the abnormal first target model, and sending the abnormal message mail to a preset mail address.
In an embodiment, when the processor executes the program instruction to determine whether an abnormal step occurs in the first target model according to a preset interval in which the first current data index is located, the method specifically includes the following steps: judging the position of the first current data index in a preset interval, wherein the position of the preset interval comprises the following steps: an adjustment zone, a stabilization zone, and an observation zone; and if the first current data index is in an adjustment area of a preset interval, judging that the first target model is abnormal.
In an embodiment, when the processor executes the program instructions to implement the comparing the first current performance index with a previous performance index to determine whether an abnormal step occurs in the first target model, the method specifically includes the following steps: acquiring a previous performance index of the first target model and comparing the previous performance index with the first current performance index; and if the difference value between the previous performance index and the first current performance index exceeds a preset performance index threshold value, judging that the first target model is abnormal.
In an embodiment, when the processor executes the program instruction to implement the comparing the first current effect index with a previous effect index to determine whether an abnormal step occurs in the first target model, the method specifically includes the following steps: acquiring a previous effect index of the first target model and comparing the previous effect index with the first current effect index; and if the difference value between the previous effect index and the first current effect index exceeds a preset effect index threshold value, judging that the first target model is abnormal.
In an embodiment, after executing the program instruction to implement the step of generating an abnormal message mail according to the number of the first object model in which the abnormality occurs and sending the abnormal message mail to a preset mail address, the processor further implements the following steps: if the current time is detected to be the second preset time, acquiring a second current data index, a second current performance index and a second current effect index of a second target model; judging whether the second target model is abnormal or not according to the second current data index; judging whether the second target model is abnormal or not according to the second current performance index; judging whether the second target model is abnormal or not according to the second current effect index; if the second target model is abnormal, generating an abnormal message mail according to the number of the abnormal second target model, and sending the abnormal message mail to a preset mail address.
The storage medium may be a U-disk, a removable hard disk, a Read-Only Memory (ROM), a magnetic disk, or an optical disk, or other various computer-readable storage media that can store program codes.
Those of ordinary skill in the art will appreciate that the elements and algorithm steps described in connection with the embodiments disclosed herein may be embodied in electronic hardware, in computer software, or in a combination of the two, and that the elements and steps of the examples have been generally described in terms of function in the foregoing description to clearly illustrate the interchangeability of hardware and software. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the solution. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.
In the several embodiments provided by the present invention, it should be understood that the disclosed apparatus and method may be implemented in other manners. For example, the device embodiments described above are merely illustrative. For example, the division of each unit is only one logic function division, and there may be another division manner in actual implementation. For example, multiple units or components may be combined or may be integrated into another system, or some features may be omitted, or not performed.
The steps in the method of the embodiment of the invention can be sequentially adjusted, combined and deleted according to actual needs. The units in the device of the embodiment of the invention can be combined, divided and deleted according to actual needs. In addition, each functional unit in the embodiments of the present invention may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit.
The integrated unit may be stored in a storage medium if implemented in the form of a software functional unit and sold or used as a stand-alone product. Based on such understanding, the technical solution of the present invention is essentially or a part contributing to the prior art, or all or part of the technical solution may be embodied in the form of a software product stored in a storage medium, comprising several instructions for causing a computer device (which may be a personal computer, a terminal, a network device, etc.) to perform all or part of the steps of the method according to the embodiments of the present invention.
While the invention has been described with reference to certain preferred embodiments, it will be understood by those skilled in the art that various changes and substitutions of equivalents may be made and equivalents will be apparent to those skilled in the art without departing from the scope of the invention. Therefore, the protection scope of the invention is subject to the protection scope of the claims.
Claims (8)
1. A method of model monitoring, comprising:
if the current time is detected to be a first preset time, a first current data index, a first current performance index and a first current effect index of a first target model are obtained, wherein the first target model refers to a daily model with a production model running period of one day, and the first preset time is the time for periodically monitoring the first target model;
judging whether the first target model is abnormal or not according to a preset interval in which the first current data index is located, wherein the first current data index refers to a data index obtained by calculating a current data set of the first target model through a statistical method;
judging the position of the first current data index in a preset interval, wherein the preset interval is an interval divided according to the size of the data index, and the position of the preset interval comprises: an adjustment zone, a stabilization zone, and an observation zone;
if the first current data index is in an adjustment area of a preset interval, judging that the first target model is abnormal;
comparing the first current performance index with a previous performance index to judge whether the first target model is abnormal, wherein the first current performance index refers to performance evaluation calculation of the current output of the first target model through an evaluation model to obtain a performance index;
Comparing the first current effect index with a previous effect index to judge whether the first target model is abnormal, wherein the first current effect index is an effect index obtained by calculation according to the current real conversion condition of the first target model;
if the first target model is abnormal, generating an abnormal message mail according to the number of the abnormal first target model, and sending the abnormal message mail to a preset mail address.
2. The method of claim 1, wherein comparing the first current performance index with a previous performance index to determine whether the first target model is abnormal comprises:
acquiring a previous performance index of the first target model and comparing the previous performance index with the first current performance index;
and if the difference value between the previous performance index and the first current performance index exceeds a preset performance index threshold value, judging that the first target model is abnormal.
3. The method of claim 1, wherein comparing the first current effect index with a previous effect index to determine whether the first target model is abnormal comprises:
Acquiring a previous effect index of the first target model and comparing the previous effect index with the first current effect index;
and if the difference value between the previous effect index and the first current effect index exceeds a preset effect index threshold value, judging that the first target model is abnormal.
4. The method for monitoring a model according to claim 1, wherein if the first target model is abnormal, generating an abnormal message mail according to the number of the abnormal first target model, and sending the abnormal message mail to a preset mail address, further comprising:
if the current time is detected to be a second preset time, a second current data index, a second current performance index and a second current effect index of a second target model are obtained, wherein the second target model is a month model with one month of the operation period of the model on production, and the second preset time is the time for periodically monitoring the second target model;
judging whether the second target model is abnormal or not according to the second current data index;
judging whether the second target model is abnormal or not according to the second current performance index;
Judging whether the second target model is abnormal or not according to the second current effect index;
if the second target model is abnormal, generating an abnormal message mail according to the number of the abnormal second target model, and sending the abnormal message mail to a preset mail address.
5. A model monitoring device, characterized by comprising:
the first obtaining unit is configured to obtain a first current data index, a first current performance index, and a first current effect index of a first target model if the current time is detected to be a first preset time, where the first target model refers to a day model in which an operation period of a model on production is one day, and the first preset time is a time for periodically monitoring the first target model;
the first judging unit is used for judging whether the first target model is abnormal or not according to a preset interval where the first current data index is located, wherein the first current data index refers to a data index obtained by calculating a current data set of the first target model through a statistical method;
the first judging subunit is configured to judge a position of the first current data index in a preset interval, where the preset interval is an interval divided according to a size of the data index, and the position of the preset interval includes: an adjustment zone, a stabilization zone, and an observation zone;
The first judging subunit is used for judging that the first target model is abnormal if the first current data index is in an adjustment area of a preset interval;
the second judging unit is used for comparing the first current performance index with the previous performance index to judge whether the first target model is abnormal, wherein the first current performance index refers to performance evaluation calculation is carried out on the current output of the first target model through an evaluation model to obtain the performance index;
the third judging unit is used for comparing the first current effect index with the previous effect index to judge whether the first target model is abnormal, wherein the first current effect index is an effect index obtained by calculation according to the current real conversion condition of the first target model;
the first generation unit is used for generating an abnormal message mail according to the number of the first target model with the abnormality if the first target model is abnormal, and sending the abnormal message mail to a preset mail address.
6. The model monitoring apparatus according to claim 5, wherein the second judging unit includes:
The second comparison unit is used for acquiring a previous performance index of the first target model and comparing the previous performance index with the first current performance index;
and the second judging subunit is used for judging that the first target model is abnormal if the difference value between the previous performance index and the first current performance index exceeds a preset performance index threshold value.
7. A computer device, characterized in that it comprises a memory on which a computer program is stored and a processor which, when executing the computer program, implements the method according to any of claims 1-4.
8. A computer readable storage medium, characterized in that the storage medium stores a computer program which, when executed by a processor, implements the method according to any of claims 1-4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910604966.9A CN110458713B (en) | 2019-07-05 | 2019-07-05 | Model monitoring method, device, computer equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910604966.9A CN110458713B (en) | 2019-07-05 | 2019-07-05 | Model monitoring method, device, computer equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110458713A CN110458713A (en) | 2019-11-15 |
| CN110458713B true CN110458713B (en) | 2023-10-13 |
Family
ID=68482179
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910604966.9A Active CN110458713B (en) | 2019-07-05 | 2019-07-05 | Model monitoring method, device, computer equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110458713B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111457958A (en) * | 2020-03-10 | 2020-07-28 | 利维智能（深圳）有限公司 | Port machine equipment situation monitoring method and device, computer equipment and storage medium |
| CN111625437B (en) * | 2020-05-27 | 2024-01-05 | 北京互金新融科技有限公司 | Monitoring method and device for wind control model |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108090678A (en) * | 2017-12-19 | 2018-05-29 | 马上消费金融股份有限公司 | Data model monitoring method, system, equipment and computer storage medium |
| CN108847958A (en) * | 2018-05-24 | 2018-11-20 | 平安科技（深圳）有限公司 | Monitoring of tools management method, device, computer equipment and storage medium |
| CN109088775A (en) * | 2018-08-29 | 2018-12-25 | 阿里巴巴集团控股有限公司 | abnormality monitoring method, device and server |
| CN109149565A (en) * | 2018-09-04 | 2019-01-04 | 周翔 | Electric integrated management-control method, system, server and storage medium |
| CN109144820A (en) * | 2018-08-31 | 2019-01-04 | 新华三信息安全技术有限公司 | A kind of detection method and device of abnormal host |
| EP3451219A1 (en) * | 2017-08-31 | 2019-03-06 | KBC Groep NV | Improved anomaly detection |
| CN109446466A (en) * | 2018-09-05 | 2019-03-08 | 北京三快在线科技有限公司 | Method for detecting abnormality, device, electronic equipment and readable storage medium storing program for executing |
| CN109543956A (en) * | 2018-10-27 | 2019-03-29 | 平安医疗健康管理股份有限公司 | The detection method and relevant device of violation medical institutions based on data analysis |
| CN109684179A (en) * | 2018-09-03 | 2019-04-26 | 平安科技（深圳）有限公司 | Method for early warning, device, equipment and the storage medium of the system failure |
-
2019
- 2019-07-05 CN CN201910604966.9A patent/CN110458713B/en active Active
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP3451219A1 (en) * | 2017-08-31 | 2019-03-06 | KBC Groep NV | Improved anomaly detection |
| CN108090678A (en) * | 2017-12-19 | 2018-05-29 | 马上消费金融股份有限公司 | Data model monitoring method, system, equipment and computer storage medium |
| CN108847958A (en) * | 2018-05-24 | 2018-11-20 | 平安科技（深圳）有限公司 | Monitoring of tools management method, device, computer equipment and storage medium |
| CN109088775A (en) * | 2018-08-29 | 2018-12-25 | 阿里巴巴集团控股有限公司 | abnormality monitoring method, device and server |
| CN109144820A (en) * | 2018-08-31 | 2019-01-04 | 新华三信息安全技术有限公司 | A kind of detection method and device of abnormal host |
| CN109684179A (en) * | 2018-09-03 | 2019-04-26 | 平安科技（深圳）有限公司 | Method for early warning, device, equipment and the storage medium of the system failure |
| CN109149565A (en) * | 2018-09-04 | 2019-01-04 | 周翔 | Electric integrated management-control method, system, server and storage medium |
| CN109446466A (en) * | 2018-09-05 | 2019-03-08 | 北京三快在线科技有限公司 | Method for detecting abnormality, device, electronic equipment and readable storage medium storing program for executing |
| CN109543956A (en) * | 2018-10-27 | 2019-03-29 | 平安医疗健康管理股份有限公司 | The detection method and relevant device of violation medical institutions based on data analysis |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110458713A (en) | 2019-11-15 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107766299B (en) | Data index abnormity monitoring method and system, storage medium and electronic equipment | |
| US9392022B2 (en) | Methods and apparatus to measure compliance of a virtual computing environment | |
| CN112188531B (en) | Abnormality detection method, abnormality detection device, electronic apparatus, and computer storage medium | |
| JP7040851B2 (en) | Anomaly detection device, anomaly detection method and anomaly detection program | |
| CN107992410B (en) | Software quality monitoring method and device, computer equipment and storage medium | |
| US11605025B2 (en) | Automated quality check and diagnosis for production model refresh | |
| WO2014208002A1 (en) | System analysis device, system analysis method and system analysis program | |
| CN112231174A (en) | Abnormity warning method, device, equipment and storage medium | |
| JP5768983B2 (en) | Contract violation prediction system, contract violation prediction method, and contract violation prediction program | |
| CN108269189B (en) | Index data monitoring method and device, storage medium and computer equipment | |
| AU2019204000A1 (en) | Mitigating asset damage via asset data analysis and processing | |
| CN110458713B (en) | Model monitoring method, device, computer equipment and storage medium | |
| US11265232B2 (en) | IoT stream data quality measurement indicator and profiling method and system therefor | |
| CN108764290B (en) | Method and device for determining cause of model transaction and electronic equipment | |
| US20220222545A1 (en) | Generation method, non-transitory computer-readable storage medium, and information processing device | |
| CN112380073B (en) | Fault position detection method and device and readable storage medium | |
| Luo et al. | Condition-based maintenance policy for systems under dynamic environment | |
| US20220222580A1 (en) | Deterioration detection method, non-transitory computer-readable storage medium, and information processing device | |
| WO2021140942A1 (en) | Diagnosing device, diagnosing method, and program | |
| CN111651503B (en) | Power distribution network data anomaly identification method and system and terminal equipment | |
| KR20160053977A (en) | Apparatus and method for model adaptation | |
| CN106156470B (en) | Time series abnormity detection and labeling method and system | |
| CN111176931A (en) | Operation monitoring method, operation monitoring device, server and storage medium | |
| CN117972686B (en) | Data management method and related device | |
| US20220222579A1 (en) | Deterioration detection method, non-transitory computer-readable storage medium, and information processing device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |