CN111143102B - Abnormal data detection method and device, storage medium and electronic equipment - Google Patents
Abnormal data detection method and device, storage medium and electronic equipment Download PDFInfo
- Publication number
- CN111143102B CN111143102B CN201911286042.5A CN201911286042A CN111143102B CN 111143102 B CN111143102 B CN 111143102B CN 201911286042 A CN201911286042 A CN 201911286042A CN 111143102 B CN111143102 B CN 111143102B
- Authority
- CN
- China
- Prior art keywords
- data
- service
- model parameters
- abnormal
- target
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 230000002159 abnormal effect Effects 0.000 title claims abstract description 104
- 238000001514 detection method Methods 0.000 title claims abstract description 74
- 238000000034 method Methods 0.000 claims abstract description 28
- 238000004422 calculation algorithm Methods 0.000 claims description 13
- 230000009467 reduction Effects 0.000 claims description 10
- 238000012545 processing Methods 0.000 claims description 9
- 238000012549 training Methods 0.000 claims description 8
- 238000004364 calculation method Methods 0.000 claims description 6
- 238000004590 computer program Methods 0.000 claims description 6
- 230000005856 abnormality Effects 0.000 claims description 4
- 238000012423 maintenance Methods 0.000 abstract description 20
- 230000008569 process Effects 0.000 abstract description 8
- 238000004891 communication Methods 0.000 description 8
- 238000010586 diagram Methods 0.000 description 5
- 230000005236 sound signal Effects 0.000 description 4
- 238000011156 evaluation Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012544 monitoring process Methods 0.000 description 2
- 238000007781 pre-processing Methods 0.000 description 2
- 230000003068 static effect Effects 0.000 description 2
- KLDZYURQCUYZBL-UHFFFAOYSA-N 2-[3-[(2-hydroxyphenyl)methylideneamino]propyliminomethyl]phenol Chemical compound OC1=CC=CC=C1C=NCCCN=CC1=CC=CC=C1O KLDZYURQCUYZBL-UHFFFAOYSA-N 0.000 description 1
- BQCADISMDOOEFD-UHFFFAOYSA-N Silver Chemical compound [Ag] BQCADISMDOOEFD-UHFFFAOYSA-N 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 238000004140 cleaning Methods 0.000 description 1
- 201000001098 delayed sleep phase syndrome Diseases 0.000 description 1
- 208000033921 delayed sleep phase type circadian rhythm sleep disease Diseases 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 238000009499 grossing Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000000630 rising effect Effects 0.000 description 1
- 229910052709 silver Inorganic materials 0.000 description 1
- 239000004332 silver Substances 0.000 description 1
- 238000002759 z-score normalization Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/0703—Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation
- G06F11/0751—Error or fault detection not based on redundancy
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3003—Monitoring arrangements specially adapted to the computing system or computing system component being monitored
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Quality & Reliability (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computing Systems (AREA)
- Mathematical Physics (AREA)
- Debugging And Monitoring (AREA)
Abstract
The disclosure relates to an abnormal data detection method, an abnormal data detection device, a storage medium and electronic equipment, which are used for solving the technical problem of low flexibility in detecting abnormal data in the operation and maintenance process of a system. The method comprises the following steps: acquiring service characteristic data at the current moment, wherein the service characteristic data comprises service cycle type information; determining target model parameters corresponding to the service period type information from a plurality of groups of model parameters of the abnormal data detection model according to the service period type information, wherein each group of model parameters is suitable for carrying out abnormal detection on service characteristic data of different service period types; and inputting the business characteristic data into an abnormal data detection model, and outputting a result of whether abnormal data exists or not, which is calculated based on the target model parameters.
Description
Technical Field
The disclosure relates to the technical field of operation and maintenance monitoring, in particular to an abnormal data detection method, an abnormal data detection device, a storage medium and electronic equipment.
Background
IT operation and maintenance management is an important technical support part for ensuring the normal operation of an application system. In the prior art, in order to ensure service quality and data reliability, a silver-linked business system needs to monitor core business indexes in real time, and unstable factors of system operation can be found in time according to abnormal fluctuation conditions of data, so that huge economic loss is avoided. However, in the monitoring process, operation and maintenance personnel generally manually maintain each service index according to working experience, so that not only is the requirement on knowledge storage of the operation and maintenance personnel higher, but also the service indexes under different trade conditions cannot be flexibly and accurately evaluated.
Disclosure of Invention
The disclosure aims to provide an abnormal data detection method, an abnormal data detection device, a storage medium and electronic equipment, which are used for solving the technical problem of low flexibility in detecting abnormal data in the operation and maintenance process of a system.
In order to achieve the above object, a first aspect of the present disclosure provides an abnormal data detection method, the method comprising:
acquiring service characteristic data at the current moment, wherein the service characteristic data comprises service cycle type information;
determining target model parameters corresponding to the service period type information from a plurality of groups of model parameters of an abnormal data detection model according to the service period type information, wherein each group of model parameters is suitable for carrying out abnormal detection on service characteristic data of different service period types;
and inputting the business characteristic data into the abnormal data detection model, and outputting a result of whether abnormal data exists or not, which is calculated based on the target model parameters.
Optionally, the method further comprises a method of training the model parameters, comprising:
determining fluctuation trend data of a service period type based on historical key performance index KPI sequence data of the same service period type;
and calculating model parameters corresponding to the fluctuation trend data according to a preset algorithm, wherein the model parameters comprise a first threshold baseline and a second threshold baseline which are matched with the fluctuation trend data, and the first threshold baseline is higher than the second threshold baseline.
Optionally, the target model parameters include a target first threshold baseline and a target second threshold baseline;
the step of inputting the business feature data into the abnormal data detection model and outputting a result of whether abnormal data exists or not, which is calculated based on the target model parameters, comprises the following steps:
calculating the deviation value of the input business characteristic data and the corresponding moment value in the target first threshold baseline and/or the target second threshold baseline;
and outputting a result that the service characteristic data is abnormal data when the deviation value meets an abnormal condition.
Optionally, after calculating the deviation value of the input business feature data from the corresponding time value in the target first threshold baseline and/or the target second threshold baseline, the method further includes:
noise reduction processing is carried out on the deviation value through the following formula to obtain a new deviation value f (x):
wherein x is the deviation value, alpha is a denoising factor, and beta is a deviation reference parameter.
Optionally, the method further comprises:
and updating the target model parameters according to the business characteristic data and the output result of whether the business characteristic data is abnormal data or not.
A second aspect of the present disclosure provides an abnormal data detection apparatus, the apparatus comprising:
the data acquisition module is used for acquiring service characteristic data at the current moment, wherein the service characteristic data comprises service cycle type information;
the parameter determining module is used for determining target model parameters corresponding to the service cycle type information from a plurality of groups of model parameters of the abnormal data detection model according to the service cycle type information, wherein each group of model parameters is suitable for carrying out abnormal detection on service characteristic data of different service cycle types;
and the anomaly detection module is used for inputting the business characteristic data into the anomaly data detection model and outputting a result of whether the anomaly data exists or not, which is calculated based on the target model parameters.
Optionally, the device further comprises a parameter training module, configured to determine fluctuation trend data of the service period type based on historical key performance indicator KPI sequence data of the same service period type; and is combined with
And calculating model parameters corresponding to the fluctuation trend data according to a preset algorithm, wherein the model parameters comprise a first threshold baseline and a second threshold baseline which are matched with the fluctuation trend data, and the first threshold baseline is higher than the second threshold baseline.
Optionally, the target model parameters include a target first threshold baseline and a target second threshold baseline; the abnormality detection module is used for:
calculating the deviation value of the input business characteristic data and the corresponding moment value in the target first threshold baseline and/or the target second threshold baseline;
and outputting a result that the service characteristic data is abnormal data when the deviation value meets an abnormal condition.
Optionally, the device further includes a data denoising module, configured to perform denoising processing on the deviation value to obtain a new deviation value f (x) according to the following formula:
wherein x is the deviation value, alpha is a denoising factor, and beta is a deviation reference parameter.
Optionally, the device further includes a parameter updating module, configured to update the target model parameter according to the service feature data and an output result of whether the service feature data is abnormal data.
A third aspect of the present disclosure provides a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of the abnormal data detection method provided by the first aspect of the present disclosure.
A fourth aspect of the present disclosure provides an electronic device, comprising:
a memory having a computer program stored thereon;
a processor for executing the computer program in the memory to implement the steps of the abnormal data detection method provided in the first aspect of the present disclosure.
According to the technical scheme, for the service characteristic data of different service cycle types, different model parameters in the abnormal data detection model are adopted for calculation and evaluation so as to determine whether the service characteristic data belongs to abnormal data. Therefore, the abnormal state of the service characteristic data is automatically analyzed through the abnormal data detection model, so that the labor cost of system maintenance can be reduced, the workload of operation and maintenance personnel is reduced, the service characteristic data under different transaction conditions can be reasonably and accurately estimated in a self-adaptive manner, the judgment error caused by the insufficient experience of the operation and maintenance personnel is avoided, and the efficiency and the accuracy of abnormal data detection are improved.
Additional features and advantages of the present disclosure will be set forth in the detailed description which follows.
Drawings
The accompanying drawings are included to provide a further understanding of the disclosure, and are incorporated in and constitute a part of this specification, illustrate the disclosure and together with the description serve to explain, but do not limit the disclosure. In the drawings:
FIG. 1 is a flow chart of an abnormal data detection method provided by an embodiment of the present disclosure;
FIG. 2 is a flow chart of another method for detecting abnormal data provided by an embodiment of the present disclosure;
FIG. 3 is a graph of real-time business characteristic data, model parameters and corresponding bias values for a business cycle type provided by an embodiment of the present disclosure;
FIG. 4 is a block diagram of an abnormal data detection apparatus provided by an embodiment of the present disclosure;
FIG. 5 is a block diagram of another abnormal data detection apparatus provided by an embodiment of the present disclosure;
fig. 6 is a block diagram of an electronic device provided by an embodiment of the present disclosure.
Detailed Description
Specific embodiments of the present disclosure are described in detail below with reference to the accompanying drawings. It should be understood that the detailed description and specific examples, while indicating and illustrating the disclosure, are not intended to limit the disclosure.
In order to solve the technical problem of low flexibility in detecting abnormal data in the operation and maintenance process of a system, an embodiment of the disclosure provides an abnormal data detection method, an apparatus, a storage medium and an electronic device, and a technical scheme provided by the disclosure is described in detail below with reference to a specific embodiment.
Fig. 1 is a flowchart of an abnormal data detection method according to an embodiment of the present disclosure, as shown in fig. 1, the method includes the following steps:
s11, acquiring service characteristic data at the current moment.
The service characteristic data comprises a numerical value and service period type information to which the numerical value belongs. It should be noted that, the specific value of the business feature data may be a KPI (Key Performance Indicator ) value of a single dimension, or may be a KPI vector including multiple data dimensions, for example, the business feature data includes one or more of a core deposit proportion, a loan total to total property proportion, a liquidity to total property proportion, and a liability to total property proportion; the service period type represents various data scene types divided according to the distribution characteristics and the scale of the Unionpay service data, for example, the service period type comprises, but is not limited to, workdays, double holidays, minor holidays and typical dates. For example, for the service characteristic data a acquired at the time of 14:00 of 11.11.2019, the corresponding service period type information is a typical date.
S12, determining target model parameters corresponding to the service cycle type information from a plurality of groups of model parameters of the abnormal data detection model according to the service cycle type information.
For example, the multiple sets of model parameters of the abnormal data detection model refer to a set of a plurality of index threshold data under different data scene types, the index threshold data are arranged according to time sequence to form a parameter curve, each set of model parameters is suitable for performing abnormal detection on service characteristic data of different service cycle types, the index threshold data at any moment on a certain set of model parameters can be a KPI value of a single dimension or a KPI vector containing a plurality of data dimensions corresponding to the service characteristic data. According to the preset corresponding relation between the model parameters and the service period types of the service characteristic data, the target model parameters corresponding to the service period types of the service characteristic data at the current moment can be determined. For example, if a preset correspondence between the model parameter and the service cycle type of the service feature data is that when the service cycle type is a working day, the corresponding model parameter is a parameter curve B; when the service period type is double holidays, the corresponding model parameter is a parameter curve C; when the service period type is small and long, the corresponding model parameter is a parameter curve D; when the service period type is a typical date, the corresponding model parameter is a parameter curve E; in this way, for the service characteristic data a obtained at the time of 14:00 of 11.11.2019, the corresponding service period type information is the typical date, and the corresponding target model parameter is the parameter curve E.
In one possible implementation, each set of model parameters may include a first threshold baseline and a second threshold baseline. Wherein the first threshold baseline is higher than the second threshold baseline, that is, in the same coordinate system, for the same time t, the index threshold data of the first threshold baseline is greater than the index threshold data of the second threshold baseline. In the disclosed embodiments, model parameters may be trained by: and determining fluctuation trend data of the service period type based on the historical KPI sequence data of the same service period type, and calculating to obtain model parameters corresponding to the fluctuation trend data according to a preset algorithm, so as to obtain a first threshold baseline and a second threshold baseline which are matched with the fluctuation trend data of the service period type.
Specifically, the historical KPI sequence data of various service indexes collected by a system platform is used as a training data set, the historical KPI sequence data is classified according to service cycle types to obtain a historical KPI sequence data set of each service cycle type, and the historical KPI sequence data set of each service cycle type is subjected to data preprocessing, wherein the data preprocessing operation comprises abnormal data cleaning and data smoothing processing. And then calculating a plurality of historical KPI data at the same moment in a plurality of service periods of the same service period type to obtain the expected mu and standard deviation sigma of the plurality of historical KPI data at the moment, and repeating the calculation process to determine the expected mu and standard deviation sigma of the plurality of historical KPI data at each moment in the service period, namely the fluctuation trend data of the service index. Arranging the expected mu in the fluctuation trend data according to the time sequence can form a fluctuation trend curve of the service index, wherein the fluctuation trend curve can reflect the basic fluctuation trend of the traffic of the service index when the service system is in a steady running state, and mainly comprises three fluctuation trends: rising trend, falling trend, and no fluctuation.
Further, a first threshold baseline and a second threshold baseline corresponding to the service cycle type can be calculated according to the fluctuation trend data, and a fluctuation interval formed by the first threshold baseline and the second threshold baseline represents a normal floating interval of a data value of the service index in a steady running state of the service system.
For example, the first threshold baseline and the second threshold baseline of the fluctuation trend data may be calculated using a 3σ criterion algorithm, where the 3σ criterion is that the probability of the index value distribution in (μ - σ, μ+σ) is 0.6826, the probability of the value distribution in (μ -2σ, μ+2σ) is 0.9544, and the probability of the value distribution in (μ -3σ, μ+3σ) is 0.9974, so that it can be considered that the normal floating values of the traffic characteristic data are almost all concentrated in the (μ -3σ, μ+3σ) interval in the steady operation state of the system. In the embodiment of the disclosure, μ+kσ may be used as an upper limit of index threshold data of KPI data at a certain time, the upper limits of a plurality of index threshold data of the same service period type are arranged according to time sequence to form a first threshold baseline, μ -kσ is used as a lower limit of index threshold data of KPI data at a certain time, and the lower limits of a plurality of index threshold data of the same service period type are arranged according to time sequence to form a second threshold baseline. The k can be set autonomously according to human experience, and the larger k is, the larger the interval range between the first threshold base line and the second threshold base line is, namely the greater the possibility that the service characteristic data fall into the interval range is, and the size of the interval range represents the acceptability of the floating degree of the service characteristic data. And repeating the calculation process to obtain a first threshold baseline and a second threshold baseline of the model parameters corresponding to different service cycle types.
S13, inputting the business characteristic data into an abnormal data detection model, and outputting a result of whether abnormal data exists or not, which is calculated based on the target model parameters.
For the business feature data b input into the abnormal data detection model, determining a first threshold baseline and a second threshold baseline which are matched with model parameters corresponding to the business cycle type of the business feature data b, for example, k=1.96, taking mu+1.96 sigma as the upper limit of index threshold data of the business feature data b at the current moment, taking mu-1.96 sigma as the lower limit of index threshold data of the business feature data b at the current moment, and outputting the result that the business feature data b is normal data if the business feature data is detected to be in the interval range of [ mu-1.96 sigma, mu+1.96 sigma ]; and if the business characteristic data b is smaller than mu-1.96 sigma or larger than mu+1.96 sigma, outputting the business characteristic data b as an abnormal data result.
By adopting the method, for the service characteristic data of different service cycle types, different model parameters in the abnormal data detection model are adopted for calculation and evaluation so as to determine whether the service characteristic data belongs to abnormal data. Therefore, the abnormal state of the service characteristic data is automatically analyzed through the abnormal data detection model, so that the labor cost of system maintenance can be reduced, the workload of operation and maintenance personnel is reduced, the service characteristic data under different transaction conditions can be reasonably and accurately estimated in a self-adaptive manner, the judgment error caused by the insufficient experience of the operation and maintenance personnel is avoided, and the efficiency and the accuracy of abnormal data detection are improved. In addition, by predicting the interval range between the first threshold baseline and the second threshold baseline, tolerance to deviation caused by business feature data floating can be enhanced, so that the abnormal data detection model algorithm is more inclusive of reasonable fluctuation data, and robustness of the algorithm is improved.
Fig. 2 is a flowchart of another abnormal data detection method according to an embodiment of the present disclosure, as shown in fig. 2, the method includes the following steps:
s21, acquiring service characteristic data at the current moment.
The traffic characteristic data includes traffic cycle type information.
S22, determining target model parameters corresponding to the business cycle type information from a plurality of groups of model parameters of the abnormal data detection model according to the business cycle type information, wherein the target model parameters comprise a target first threshold baseline and a target second threshold baseline.
The method steps according to the previous embodiment may calculate a first threshold baseline and a second threshold baseline comprised by the model parameters corresponding to the traffic cycle type of the traffic feature data. For example, fig. 3 is a graph of real-time service characteristic data, model parameters and corresponding deviation values of the real-time service characteristic data and the model parameters of a service period type according to an embodiment of the present disclosure, as shown in fig. 3, a curve K0 is a real-time service characteristic data curve, a curve K1 is a first threshold baseline, and a curve K2 is a second threshold baseline.
S23, calculating the deviation value of the corresponding moment value in the input business characteristic data and the target first threshold baseline and/or the target second threshold baseline.
For example, the deviation value e (x) of the business feature data at the current time and the corresponding time value in the target first threshold baseline and/or the target second threshold baseline may be calculated by the following formula:
as shown in fig. 3, a curve e (x) is a graph of deviation values of real-time service characteristic data and corresponding moments of model parameters.
S25, outputting the result that the service characteristic data is the abnormal data when the deviation value meets the abnormal condition.
And if the deviation value is equal to 0, outputting a result that the service characteristic data is normal data, and if the deviation value is not 0, outputting a result that the service characteristic data is abnormal data. Therefore, the abnormal detection is carried out on the service characteristic data, the judging mode is simple and convenient, the inclusion of the abnormal data detection model is good, and the false alarm phenomenon caused by reasonable fluctuation of the data can be reduced. If the abnormal alarm severity level is to be judged, the abnormal alarm severity level can be determined according to the difference value between the non-0 deviation value and the manually set threshold value of each abnormal alarm severity level.
Alternatively, in another embodiment, the step S23 may be performed followed by performing a secondary process on the deviation value, as in the step S24: and carrying out noise reduction treatment on the deviation value to obtain a new deviation value.
Illustratively, the noise reduction processing step includes performing z-score normalization processing on the deviation value e (x) calculated in step S23, and obtaining a new deviation value f (x) by performing noise reduction processing on the calculation result by the following formula:
wherein x is the deviation value e (x) in step S23, α is the denoising factor, and β is the deviation reference parameter. For different service period types, the denoising factor alpha has different values, calibration can be carried out according to actual requirements, the deviation reference parameter beta represents a deviation value lower limit threshold, and through the formula, if the deviation value x is smaller, the final deviation value after the denoising treatment is smaller, and if the deviation value x is larger, the final deviation value after the denoising treatment is larger. For example, as shown in fig. 3, the first half of the curve e (x) has a large number of noise peaks N, and such noise peaks tend to interfere with the abnormal detection result of the data, so that most of the noise can be removed by the above noise reduction function, so that the detection result is more accurate.
S25, outputting the result that the service characteristic data is the abnormal data when the deviation value meets the abnormal condition.
For the deviation value f (x) calculated in step S24, the magnitude of the value can be directly used as an identifier of the severity level of the abnormal alarm, for example, each 1 increase of the deviation value f (x) indicates that the severity level of the abnormal alarm is raised.
Optionally, the embodiment of the disclosure may further update the target model parameter according to the service feature data and the output result of whether the service feature data is abnormal data, and support the user to correct and mark the output result, so as to continuously improve the accuracy of the training model parameter.
In summary, the abnormal state of the service characteristic data is automatically analyzed through the abnormal data detection model, so that the labor cost of system maintenance can be reduced, the workload of operation and maintenance personnel can be reduced, the service characteristic data under different transaction conditions can be adaptively and reasonably and accurately evaluated, the judgment error caused by the insufficient experience of the operation and maintenance personnel is avoided, and the efficiency and the accuracy of abnormal data detection are improved; by predicting the interval range between the first threshold baseline and the second threshold baseline, tolerance to deviation caused by business feature data floating can be enhanced, so that an abnormal data detection model algorithm is more contained for reasonable fluctuation data, and robustness of the algorithm is improved; through carrying out the noise reduction to the offset, can reduce the misinformation and the missing report condition of abnormal data that the interference of noise error item caused, strengthened the reliability of system, and need not the manual work to set up, maintain the warning threshold, abnormal data detection process is simple convenient.
Fig. 4 is a block diagram of an abnormal data detection apparatus provided in an embodiment of the present disclosure, where the apparatus 400 may be implemented as part or all of an electronic device by software, hardware, or a combination of both. Referring to fig. 4, the apparatus 400 includes:
a data acquisition module 41, configured to acquire service feature data at a current time, where the service feature data includes service period type information;
a parameter determining module 42, configured to determine, from a plurality of sets of model parameters of the abnormal data detection model according to the service period type information, a target model parameter corresponding to the service period type information, where each set of model parameters is adapted to perform abnormal detection on service feature data of different service period types;
the anomaly detection module 43 is configured to input the service feature data into the anomaly data detection model, and output a result of whether the anomaly data exists or not calculated based on the target model parameters.
Optionally, as shown in fig. 5, the apparatus 400 further includes a parameter training module 45, configured to determine fluctuation trend data of a service period type based on historical KPI (Key Performance Indicator ) sequence data of the same service period type, and calculate, according to a preset algorithm, model parameters corresponding to the fluctuation trend data, where the model parameters include a first threshold baseline and a second threshold baseline that are matched with the fluctuation trend data, and the first threshold baseline is higher than the second threshold baseline.
Specifically, the target model parameters include a target first threshold baseline and a target second threshold baseline, and the anomaly detection module 43 is configured to calculate a deviation value of the input service feature data from a corresponding time value in the target first threshold baseline and/or the target second threshold baseline, and output a result that the service feature data is anomaly data when the deviation value satisfies an anomaly condition.
Optionally, the apparatus further includes a data denoising module 44, configured to denoise the deviation value to obtain a new deviation value f (x) by the following formula:
wherein x is a deviation value, alpha is a denoising factor, and beta is a deviation reference parameter.
Optionally, the apparatus 400 further includes a parameter updating module 46, configured to update the target model parameter according to the service characteristic data and the output result of whether the service characteristic data is abnormal data.
By adopting the device, the abnormal state of the service characteristic data is automatically analyzed through the abnormal data detection model, so that the labor cost of system maintenance can be reduced, the workload of operation and maintenance personnel can be reduced, the service characteristic data under different transaction conditions can be reasonably and accurately estimated in a self-adaptive manner, the judgment error caused by the insufficient experience of the operation and maintenance personnel is avoided, and the efficiency and the accuracy of abnormal data detection are improved; the parameter training module predicts the interval range between the first threshold baseline and the second threshold baseline, so that tolerance to deviation caused by business feature data floating can be enhanced, an abnormal data detection model algorithm is more contained for reasonable fluctuation data, and robustness of the algorithm is improved; the data noise reduction module is used for carrying out noise reduction treatment on the deviation value, so that false alarm and missing report of abnormal data caused by interference of noise error items can be reduced, the reliability of the system is enhanced, the manual setting and maintenance of a warning threshold value are not needed, and the abnormal data detection process is simple and convenient.
The specific manner in which the various modules perform the operations in the apparatus of the above embodiments have been described in detail in connection with the embodiments of the method, and will not be described in detail herein.
Fig. 6 is a block diagram of an electronic device 600, according to an example embodiment. As shown in fig. 6, the electronic device 600 may include: a processor 601, a memory 602. The electronic device 600 may also include one or more of a multimedia component 603, an input/output (I/O) interface 604, and a communication component 605.
The processor 601 is configured to control the overall operation of the electronic device 600 to perform all or part of the steps in the abnormal data detection method described above. The memory 602 is used to store various types of data to support operations at the electronic device 600, which may include, for example, instructions for any application or method operating on the electronic device 600, as well as application-related data, such as contact data, transceived messages, pictures, audio, video, and the like. The Memory 602 may be implemented by any type or combination of volatile or nonvolatile Memory devices, such as static random access Memory (Static Random Access Memory, SRAM for short), electrically erasable programmable Read-Only Memory (Electrically Erasable Programmable Read-Only Memory, EEPROM for short), erasable programmable Read-Only Memory (Erasable Programmable Read-Only Memory, EPROM for short), programmable Read-Only Memory (Programmable Read-Only Memory, PROM for short), read-Only Memory (ROM for short), magnetic Memory, flash Memory, magnetic disk, or optical disk. The multimedia component 603 may include a screen and an audio component. Wherein the screen may be, for example, a touch screen, the audio component being for outputting and/or inputting audio signals. For example, the audio component may include a microphone for receiving external audio signals. The received audio signals may be further stored in the memory 602 or transmitted through the communication component 605. The audio assembly further comprises at least one speaker for outputting audio signals. The I/O interface 604 provides an interface between the processor 601 and other interface modules, which may be a keyboard, mouse, buttons, etc. These buttons may be virtual buttons or physical buttons. The communication component 605 is used for wired or wireless communication between the electronic device 600 and other devices. Wireless communication, such as Wi-Fi, bluetooth, near field communication (Near Field Communication, NFC for short), 2G, 3G or 4G, or a combination of one or more thereof, the corresponding communication component 605 may thus comprise: wi-Fi module, bluetooth module, NFC module.
In an exemplary embodiment, the electronic device 600 may be implemented by one or more application specific integrated circuits (Application Specific Integrated Circuit, abbreviated as ASIC), digital signal processors (Digital Signal Processor, abbreviated as DSP), digital signal processing devices (Digital Signal Processing Device, abbreviated as DSPD), programmable logic devices (Programmable Logic Device, abbreviated as PLD), field programmable gate arrays (Field Programmable Gate Array, abbreviated as FPGA), controllers, microcontrollers, microprocessors, or other electronic components for performing the abnormal data detection methods described above.
In another exemplary embodiment, a computer readable storage medium is also provided that includes program instructions that, when executed by a processor, implement the steps of the abnormal data detection method described above. For example, the computer readable storage medium may be the memory 602 including program instructions described above, which are executable by the processor 601 of the electronic device 600 to perform the abnormal data detection method described above.
The preferred embodiments of the present disclosure have been described in detail above with reference to the accompanying drawings, but the present disclosure is not limited to the specific details of the above embodiments, and various simple modifications may be made to the technical solutions of the present disclosure within the scope of the technical concept of the present disclosure, and all the simple modifications belong to the protection scope of the present disclosure.
In addition, the specific features described in the above embodiments may be combined in any suitable manner without contradiction. The various possible combinations are not described further in this disclosure in order to avoid unnecessary repetition.
Moreover, any combination between the various embodiments of the present disclosure is possible as long as it does not depart from the spirit of the present disclosure, which should also be construed as the disclosure of the present disclosure.
Claims (8)
1. A method of anomaly data detection, the method comprising:
acquiring service characteristic data at the current moment, wherein the service characteristic data comprises service cycle type information;
determining target model parameters corresponding to the service cycle type information from a plurality of groups of model parameters of an abnormal data detection model according to the service cycle type information, wherein the target model parameters comprise a target first threshold baseline and a target second threshold baseline, and each group of model parameters is suitable for carrying out abnormal detection on service characteristic data of different service cycle types;
calculating the deviation value of the input business characteristic data and the corresponding moment value in the target first threshold baseline and/or the target second threshold baseline;
noise reduction processing is carried out on the deviation value through the following formula to obtain a new deviation value f (x):
wherein x is the deviation value before noise reduction treatment, alpha is a denoising factor, and beta is a deviation reference parameter;
and when the deviation value meets an abnormal condition, outputting the new deviation value as an identification of the severity level of the abnormal alarm.
2. The method of claim 1, further comprising a method of training the model parameters, comprising:
determining fluctuation trend data of a service period type based on historical key performance index KPI sequence data of the same service period type;
and calculating model parameters corresponding to the fluctuation trend data according to a preset algorithm, wherein the model parameters comprise a first threshold baseline and a second threshold baseline which are matched with the fluctuation trend data, and the first threshold baseline is higher than the second threshold baseline.
3. The method according to any one of claims 1 to 2, further comprising:
and updating the target model parameters according to the business characteristic data and the output result of whether the business characteristic data is abnormal data or not.
4. An abnormal data detection apparatus, characterized in that the apparatus comprises:
the data acquisition module is used for acquiring service characteristic data at the current moment, wherein the service characteristic data comprises service cycle type information;
the parameter determining module is used for determining target model parameters corresponding to the service cycle type information from a plurality of groups of model parameters of the abnormal data detection model according to the service cycle type information, wherein the target model parameters comprise a target first threshold baseline and a target second threshold baseline, and each group of model parameters is suitable for carrying out abnormal detection on service characteristic data of different service cycle types;
the anomaly detection module is used for inputting the business characteristic data into the anomaly data detection model, outputting a result of whether the anomaly data exist or not obtained by calculation based on the target model parameters, and calculating a deviation value of the input business characteristic data and a corresponding moment value in a target first threshold base line and/or a target second threshold base line;
the data denoising module is used for denoising the deviation value to obtain a new deviation value f (x) through the following formula:
wherein x is a deviation value, alpha is a denoising factor, and beta is a deviation reference parameter;
and the abnormality detection module is also used for outputting the new deviation value as an identifier of the severity level of the abnormality alarm when the deviation value meets the abnormality condition.
5. The apparatus of claim 4, further comprising a parameter training module for determining fluctuating trend data for a same business cycle type based on historical key performance indicator KPI sequence data for the business cycle type; and is combined with
And calculating model parameters corresponding to the fluctuation trend data according to a preset algorithm, wherein the model parameters comprise a first threshold baseline and a second threshold baseline which are matched with the fluctuation trend data, and the first threshold baseline is higher than the second threshold baseline.
6. The apparatus according to claim 4 or 5, further comprising a parameter updating module for updating the target model parameters according to the service characteristic data and the output result of whether it is abnormal data.
7. A computer readable storage medium, on which a computer program is stored, characterized in that the program, when being executed by a processor, implements the steps of the method of any of claims 1 to 3.
8. An electronic device, comprising:
a memory having a computer program stored thereon;
a processor for executing the computer program in the memory to carry out the steps of the method of any one of claims 1 to 3.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911286042.5A CN111143102B (en) | 2019-12-13 | 2019-12-13 | Abnormal data detection method and device, storage medium and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911286042.5A CN111143102B (en) | 2019-12-13 | 2019-12-13 | Abnormal data detection method and device, storage medium and electronic equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111143102A CN111143102A (en) | 2020-05-12 |
| CN111143102B true CN111143102B (en) | 2024-01-19 |
Family
ID=70518554
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911286042.5A Active CN111143102B (en) | 2019-12-13 | 2019-12-13 | Abnormal data detection method and device, storage medium and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111143102B (en) |
Families Citing this family (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113746688B (en) * | 2020-05-29 | 2023-02-28 | 华为技术有限公司 | Method and device for updating anomaly detection model and computing equipment |
| CN111865407B (en) * | 2020-06-11 | 2021-11-30 | 烽火通信科技股份有限公司 | Intelligent early warning method, device, equipment and storage medium for optical channel performance degradation |
| CN111901171B (en) * | 2020-07-29 | 2023-10-20 | 腾讯科技（深圳）有限公司 | Anomaly detection and attribution method, apparatus, device, and computer-readable storage medium |
| CN112418509A (en) * | 2020-11-18 | 2021-02-26 | 青岛海尔科技有限公司 | Task data prediction method and device, storage medium and electronic device |
| CN112380100A (en) * | 2020-12-01 | 2021-02-19 | 北京威努特技术有限公司 | Method, apparatus and medium for detecting traffic anomaly based on directional deviation |
| CN112187555B (en) * | 2020-12-01 | 2021-03-19 | 北京蒙帕信创科技有限公司 | Real-time KPI data anomaly detection method and device based on machine learning |
| CN112613995A (en) * | 2020-12-30 | 2021-04-06 | 中国工商银行股份有限公司 | Abnormality diagnosis method and apparatus |
| CN112835769A (en) * | 2021-02-24 | 2021-05-25 | 北京顶象技术有限公司 | Service data abnormity diagnosis method, device, equipment and storage medium |
| CN113487407A (en) * | 2021-07-01 | 2021-10-08 | 中国工商银行股份有限公司 | Data processing method and device and electronic equipment |
| CN113705684B (en) * | 2021-08-30 | 2023-11-24 | 平安科技（深圳）有限公司 | Reverse iteration anomaly detection method and device, electronic equipment and medium |
| CN114615034B (en) * | 2022-03-01 | 2023-09-29 | 中铁第四勘察设计院集团有限公司 | Control method, device, processing equipment and storage medium for service transmission |
| CN115766507A (en) * | 2022-11-16 | 2023-03-07 | 支付宝(杭州)信息技术有限公司 | Service abnormity detection method and device |
| CN117827524B (en) * | 2024-03-06 | 2024-08-09 | 建信金融科技有限责任公司 | System operation and maintenance method and device |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1470964A (en) * | 2002-06-12 | 2004-01-28 | 欧姆龙株式会社 | Anomaly detection method and apparatus, and temperature regulator |
| CN104317681A (en) * | 2014-09-02 | 2015-01-28 | 上海交通大学 | Behavioral abnormality automatic detection method and behavioral abnormality automatic detection system aiming at computer system |
| CN107783942A (en) * | 2016-08-25 | 2018-03-09 | 中国移动通信集团上海有限公司 | A kind of anomaly detection method and device |
| CN108880841A (en) * | 2017-05-11 | 2018-11-23 | 上海宏时数据系统有限公司 | A kind of threshold values setting, abnormality detection system and the method for service monitoring system |
| CN109146236A (en) * | 2018-07-06 | 2019-01-04 | 东软集团股份有限公司 | Indexes Abnormality detection method, device, readable storage medium storing program for executing and electronic equipment |
| CN109213654A (en) * | 2018-07-05 | 2019-01-15 | 北京奇艺世纪科技有限公司 | A kind of method for detecting abnormality and device |
| CN109615226A (en) * | 2018-12-12 | 2019-04-12 | 焦点科技股份有限公司 | A kind of operation indicator method for monitoring abnormality |
-
2019
- 2019-12-13 CN CN201911286042.5A patent/CN111143102B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1470964A (en) * | 2002-06-12 | 2004-01-28 | 欧姆龙株式会社 | Anomaly detection method and apparatus, and temperature regulator |
| CN104317681A (en) * | 2014-09-02 | 2015-01-28 | 上海交通大学 | Behavioral abnormality automatic detection method and behavioral abnormality automatic detection system aiming at computer system |
| CN107783942A (en) * | 2016-08-25 | 2018-03-09 | 中国移动通信集团上海有限公司 | A kind of anomaly detection method and device |
| CN108880841A (en) * | 2017-05-11 | 2018-11-23 | 上海宏时数据系统有限公司 | A kind of threshold values setting, abnormality detection system and the method for service monitoring system |
| CN109213654A (en) * | 2018-07-05 | 2019-01-15 | 北京奇艺世纪科技有限公司 | A kind of method for detecting abnormality and device |
| CN109146236A (en) * | 2018-07-06 | 2019-01-04 | 东软集团股份有限公司 | Indexes Abnormality detection method, device, readable storage medium storing program for executing and electronic equipment |
| CN109615226A (en) * | 2018-12-12 | 2019-04-12 | 焦点科技股份有限公司 | A kind of operation indicator method for monitoring abnormality |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111143102A (en) | 2020-05-12 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111143102B (en) | Abnormal data detection method and device, storage medium and electronic equipment | |
| CN112162878B (en) | Database fault discovery method and device, electronic equipment and storage medium | |
| CN111199018B (en) | Abnormal data detection method and device, storage medium and electronic equipment | |
| CN108564181B (en) | Power equipment fault detection and maintenance method and terminal equipment | |
| CN105721187B (en) | A kind of traffic failure diagnostic method and device | |
| CN113556258B (en) | Anomaly detection method and device | |
| CN112188531B (en) | Abnormality detection method, abnormality detection device, electronic apparatus, and computer storage medium | |
| CN110995482B (en) | Alarm analysis method and device, computer equipment and computer readable storage medium | |
| CN111178760B (en) | Risk monitoring method, risk monitoring device, terminal equipment and computer readable storage medium | |
| CN107871190A (en) | A kind of operational indicator monitoring method and device | |
| CN109918279B (en) | Electronic device, method for identifying abnormal operation of user based on log data and storage medium | |
| CN110471821B (en) | Abnormality change detection method, server, and computer-readable storage medium | |
| CN113099476A (en) | Network quality detection method, device, equipment and storage medium | |
| CN115033463B (en) | System exception type determining method, device, equipment and storage medium | |
| CN110855484B (en) | Method, system, electronic device and storage medium for automatically detecting traffic change | |
| CN116756522A (en) | Probability forecasting method and device, storage medium and electronic equipment | |
| CN110134680B (en) | Space monitoring method and device, computer equipment and storage medium | |
| CN113780906A (en) | Machine management method and device and computer readable storage medium | |
| US10288547B2 (en) | Facility state analysis device, analysis method for facility state, storage medium, and facility management system | |
| CN116187510A (en) | Ammeter box fault prediction method, device, computer equipment and storage medium | |
| CN114331688A (en) | Method and device for detecting batch operation state of bank counter system business | |
| CN109508356B (en) | Data abnormality early warning method, device, computer equipment and storage medium | |
| CN113869373A (en) | Equipment abnormality detection method and device, computer equipment and storage medium | |
| CN113656452A (en) | Method and device for detecting abnormal index of call chain, electronic equipment and storage medium | |
| CN113411217A (en) | Method and device for monitoring and alarming call system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |