CN111639011B - Data monitoring method, device and equipment - Google Patents
Data monitoring method, device and equipment Download PDFInfo
- Publication number
- CN111639011B CN111639011B CN202010530114.2A CN202010530114A CN111639011B CN 111639011 B CN111639011 B CN 111639011B CN 202010530114 A CN202010530114 A CN 202010530114A CN 111639011 B CN111639011 B CN 111639011B
- Authority
- CN
- China
- Prior art keywords
- preset
- data
- alarm
- service
- service index
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3065—Monitoring arrangements determined by the means or processing involved in reporting the monitored data
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3089—Monitoring arrangements determined by the means or processing involved in sensing the monitored data, e.g. interfaces, connectors, sensors, probes, agents
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Quality & Reliability (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Alarm Systems (AREA)
- Debugging And Monitoring (AREA)
Abstract
The embodiment of the specification discloses a data monitoring method, a device and equipment, wherein the method comprises the following steps: acquiring historical service index data of a preset service index in a preset service scene based on log data generated in a service system; performing frequency analysis on the historical service index data to obtain frequency distribution corresponding to the preset service index, and performing noise processing on the frequency distribution corresponding to the preset service index to obtain processed frequency distribution; based on the processed frequency distribution, reversely analyzing the transformation from the time domain to the frequency domain of the historical service index data to obtain an alarm baseline aiming at the preset service index; and monitoring current service index data of the preset service index in the service system based on the alarm baseline.
Description
Technical Field
The present disclosure relates to the field of computer technologies, and in particular, to a data monitoring method, apparatus, and device.
Background
With the continuous development of computer technology, service providers can provide more and more service types for users, and accordingly, the amount of users is also increased, and how to monitor user business index data in the current scene has become a focus of attention of the service providers.
At present, the service index data in the service scene can be monitored and alarmed by manually setting an alarm baseline, for example, for the check-up amount (such as successful check-up amount) in the check-up table scene of the check-up body, the corresponding alarm baseline can be set through expert experience, if the set alarm baseline can be within 24 hours of the natural day, the alarm threshold corresponding to each time point is a, and if the current successful check-up amount is detected to be smaller than a, an alarm is sent.
However, because the number of the service indexes is large, the labor cost is increased by manually setting the alarm base line, the setting accuracy is low, the setting efficiency is reduced, and the monitoring efficiency is low. Therefore, there is a need to provide a data monitoring scheme with higher monitoring efficiency and monitoring accuracy.
Disclosure of Invention
An objective of the embodiments of the present disclosure is to provide a data monitoring method, apparatus, and device, so as to provide a data monitoring scheme capable of improving monitoring efficiency and monitoring accuracy.
In order to achieve the above technical solution, the embodiments of the present specification are implemented as follows:
in a first aspect, an embodiment of the present disclosure provides a data monitoring method, where the method includes: acquiring historical service index data of a preset service index in a preset service scene based on log data generated in a service system; performing frequency analysis on the historical service index data to obtain frequency distribution corresponding to the preset service index, and performing noise processing on the frequency distribution corresponding to the preset service index to obtain processed frequency distribution; based on the processed frequency distribution, reversely analyzing the transformation from the time domain to the frequency domain of the historical service index data to obtain an alarm baseline aiming at the preset service index, wherein the alarm baseline is reference data for judging whether to alarm or not; and monitoring current service index data of the preset service index in the service system based on the alarm baseline.
In a second aspect, embodiments of the present disclosure provide a data monitoring apparatus, the apparatus comprising: the data acquisition module is used for acquiring historical service index data of a preset service index in a preset service scene based on log data generated in the service system; the data processing module is used for carrying out frequency analysis on the historical service index data to obtain frequency distribution corresponding to the preset service index, and carrying out noise processing on the frequency distribution corresponding to the preset service index to obtain the processed frequency distribution; the base line acquisition module is used for carrying out reverse analysis on the processed frequency distribution to obtain an alarm base line aiming at the preset service index, wherein the alarm base line is reference data for judging whether to carry out alarm or not; and the data monitoring module is used for monitoring the current business index data of the preset business index in the business system based on the alarm baseline.
In a third aspect, embodiments of the present disclosure provide a data monitoring apparatus, including: a processor; and a memory arranged to store computer executable instructions that, when executed, cause the processor to: acquiring historical service index data of a preset service index in a preset service scene based on log data generated in a service system; performing frequency analysis on the historical service index data to obtain frequency distribution corresponding to the preset service index, and performing noise processing on the frequency distribution corresponding to the preset service index to obtain processed frequency distribution; based on the processed frequency distribution, reversely analyzing the transformation from the time domain to the frequency domain of the historical service index data to obtain an alarm baseline aiming at the preset service index, wherein the alarm baseline is reference data for judging whether to alarm or not; and monitoring current service index data of the preset service index in the service system based on the alarm baseline.
Drawings
In order to more clearly illustrate the embodiments of the present description or the technical solutions in the prior art, the drawings that are required in the embodiments or the description of the prior art will be briefly described below, it being obvious that the drawings in the following description are only some of the embodiments described in the present description, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a flow chart of an embodiment of a data monitoring method of the present disclosure;
FIG. 2 is a schematic diagram of an alarm baseline of the present description;
FIG. 3 is a flow chart of yet another embodiment of a data monitoring method of the present disclosure;
FIG. 4 is a schematic diagram illustrating a data monitoring apparatus according to another embodiment of the present disclosure;
fig. 5 is a schematic structural diagram of a data monitoring device according to the present disclosure.
Detailed Description
The embodiment of the specification provides a data monitoring method, a device and equipment.
In order to make the technical solutions in the present specification better understood by those skilled in the art, the technical solutions in the embodiments of the present specification will be clearly and completely described below with reference to the drawings in the embodiments of the present specification, and it is obvious that the described embodiments are only some embodiments of the present specification, not all embodiments. All other embodiments, which can be made by one of ordinary skill in the art without undue burden from the present disclosure, are intended to be within the scope of the present disclosure.
Example 1
As shown in fig. 1, the embodiment of the present disclosure provides a data monitoring method, where an execution body of the method may be a terminal device or a server, where the terminal device may be a terminal device of a host program, and the terminal device may be a device such as a personal computer, or a mobile terminal device such as a mobile phone, a tablet computer, or the like, and the server may be an independent server or a server cluster formed by multiple servers. The method specifically comprises the following steps:
in S102, historical business index data of a preset business index in a preset business scene is obtained based on log data generated in a business system.
The service system may be a system for providing a preset service for a user, for example, the service system may be a system for providing a service such as a transfer service and a payment service for the user, the log data may include behavior data and corresponding time data generated when the user uses the service provided by the service system, the preset service scenario may be a scenario for providing the preset service for the user, the preset service index may be any service index in the preset service scenario, for example, assuming that the preset service scenario is a scenario for providing the transfer service, the preset service index may include an index such as a check amount (including a check amount and a check failure amount) for checking the identity of the user, a user transfer amount, and the like, and the history service index data may be service index data of any history period, for example, the history service index data may be service index data of about 1 day, service index data about 3 days, service index data about 1 week, or all stored service index data, and the like.
In implementation, with the continuous development of computer technology, service providers can provide more and more service types for users, and accordingly, the amount of users is also increased, and how to monitor user business index data in the current scene has become a focus of attention of the service providers.
At present, the service index data in the service scene can be monitored and alarmed by manually setting an alarm baseline, for example, for the check-up amount (such as successful check-up amount) in the check-up table scene of the check-up body, the corresponding alarm baseline can be set through expert experience, if the set alarm baseline can be within 24 hours of the natural day, the alarm threshold corresponding to each time point is a, and if the current successful check-up amount is detected to be smaller than a, an alarm is sent. However, because the number of the service indexes is large, the labor cost is increased by manually setting the alarm base line, the setting accuracy is low, the setting efficiency is reduced, and the monitoring efficiency is low. Therefore, it is desirable to provide a data monitoring scheme that can improve monitoring efficiency and monitoring accuracy. In addition, in addition to the mode of manually setting the alarm baseline, the alarm baseline can be determined by methods such as a moving average algorithm and an exponential average algorithm, but the alarm baseline is determined by the moving average algorithm, so that the problem of larger noise exists in the service scene with higher corresponding mutation rate, the problem of larger calculation amount exists in the exponential average algorithm, and the monitoring accuracy is poor in the service scene with lower data fluctuation frequency.
The preset service scenario is a scenario of providing the transfer service for the user, the preset service index may include a verification state obtained after identity verification of the user using the transfer service, that is, verification success or verification failure, and the corresponding service index data may include verification effort and verification failure, and the historical service index data may be verification effort or verification failure per second in approximately 1 day, that is, each piece of historical index data corresponds to one time (that is, the historical service index data is time domain data), specifically, for example, the verification effort of 15:45:50 in 5 days of 6 months is 100, that is, 15:45:50 in 5 days of 100-6 months.
In S104, frequency analysis is performed on the historical service index data to obtain a frequency distribution corresponding to the preset service index, and noise processing is performed on the frequency distribution corresponding to the preset service index to obtain a processed frequency distribution.
The method for performing frequency analysis on the historical service index data may be various, for example, the frequency analysis may be performed on the historical service index data through a fast fourier transform algorithm, the frequency analysis may be performed on the historical service index data through a sinusoidal curve fitting algorithm, or the frequency analysis may be performed on the historical service index data through a wavelet transform method, a hilbert transform algorithm, or the like, and the frequency analysis method may be various and may be different according to different practical application scenarios, which is not particularly limited in the embodiment of the present invention.
In the implementation, by converting the historical business index data into the frequency domain (namely, performing frequency analysis on the historical business index data) and performing noise processing on the frequency distribution, the noise in the historical business index data can be removed more accurately, and the accuracy of an alarm baseline can be improved.
In S106, based on the processed frequency distribution, a reverse analysis is performed on the transformation from the time domain to the frequency domain of the historical business index data, so as to obtain an alarm baseline for the preset business index.
The alarm baseline may be reference data for determining whether to alarm.
In an implementation, the inverse analysis of the transformation from the time domain to the frequency domain of the historical traffic index data may be performed based on the processed frequency distribution, where the inverse analysis includes: the reverse analysis of the processed frequency distribution can be performed by the method for performing frequency analysis on the historical service index data, so that a corresponding alarm baseline can be obtained. For example, the inverse analysis of the time domain may be performed on the processed frequency distribution according to an inverse fourier transform algorithm, the inverse analysis may be performed on the change of the time domain to the frequency domain of the historical traffic index data based on the processed frequency distribution according to a wavelet transform method, or the inverse analysis may be performed on the change of the time domain to the frequency domain of the historical traffic index data based on the processed frequency distribution according to a sinusoidal curve fitting algorithm.
In S108, current business index data of a preset business index in the business system is monitored based on the alarm baseline.
In implementation, the alarm baseline may be a discrete time sequence, where there may be multiple time points on the sequence, where each time point may correspond to an alarm threshold, for example, as shown in fig. 2, the alarm baseline may be a discrete time sequence generated according to historical service index data of approximately 1 day, where the alarm baseline may be a discrete time sequence in seconds, where each time point may correspond to 1 alarm threshold, for example, as shown in fig. 2, the alarm threshold corresponding to 0:00 may be 50, the alarm threshold corresponding to 19:00 may be 80, and the service index data may be monitored according to the alarm baseline. For example, assuming that the current time point is 19:00, the corresponding alarm threshold is 80, and if the current business index data is 85 and is greater than the alarm threshold, it may be determined that there is an abnormality in the current business index data.
The embodiment of the specification provides a data monitoring method, which is characterized in that historical service index data of preset service indexes in a preset service scene is obtained based on log data generated in a service system, frequency analysis is carried out on the historical service index data to obtain frequency distribution corresponding to the preset service indexes, noise processing is carried out on the frequency distribution corresponding to the preset service indexes to obtain processed frequency distribution, reverse analysis is carried out on the conversion from a time domain to a frequency domain of the historical service index data based on the processed frequency distribution to obtain an alarm baseline aiming at the preset service indexes, the alarm baseline is reference data for judging whether to carry out alarm, and current service index data of the preset service indexes in the service system is monitored based on the alarm baseline. Therefore, the corresponding alarm baseline can be obtained through processing the historical service index data, and the current service index data is monitored, so that the problems of large workload and poor accuracy caused by a mode of manually setting the alarm baseline are avoided, namely the determination efficiency and the determination accuracy of the alarm baseline can be improved, and the monitoring efficiency and the monitoring accuracy are improved.
Example two
As shown in fig. 3, the embodiment of the present disclosure provides a data monitoring method, where an execution body of the method may be a terminal device or a server, where the terminal device may be a terminal device of a host program, and the terminal device may be a device such as a personal computer, or a mobile terminal device such as a mobile phone, a tablet computer, or the like, and the server may be an independent server or a server cluster formed by multiple servers. The method specifically comprises the following steps:
in S302, log data corresponding to a preset service scenario, which is generated in the service system, is obtained in a preset time period.
The preset time period may be any time period, such as about 1 day, about 3 days, or about 1 week.
In S304, first business index data in the log data is obtained for a preset business index.
In implementation, the first business index data may be index data corresponding to a preset business index in the log data, where each piece of first business index data corresponds to one time (i.e. the first business index data is time domain data), and specifically, the check effort is 10, i.e. 10-6 months, 6 days, 15:45:40.
In S304, the first business index data is preprocessed, so as to obtain historical business index data.
The preprocessing of the first business index data may include a missing value checking process, a missing value filling process, a numerical conversion process, and the like of the first business index data. For example, character type data in the first business index data may be converted into numerical type data, specifically, for example, "yes" may be converted into 1, and "no" may be converted into 0, that is, numerical conversion processing may be performed on the first business index data. Or, it may check whether the first business index data has a missing value according to time, that is, the missing value checking process. Or under the condition that the missing value exists in the first business index data, if the corresponding first business index data is absent at the time point of 15:45:41 of 6 months and 6 days, the first business index data 1 corresponding to the time point of 15:45:40 of 6 months and the first business index data 2 corresponding to the time point of 15:45:42 of 6 months and 6 days can be obtained, and the average value of the first business index data 1 and the first business index data 2 is used as the first business index data corresponding to the time point of 15:45:41 of 6 months and 6 days, namely the missing value filling processing is carried out.
The preprocessing is an optional and realizable preprocessing mode, and in an actual application scene, a plurality of different preprocessing modes can be adopted, and the preprocessing modes can be different according to different actual application scenes, so that the embodiment of the invention is not particularly limited.
In practice, historical business index data may be preprocessed to improve the accuracy of the alert baseline determination.
In S308, based on the fast fourier transform algorithm, frequency analysis is performed on the historical service index data to obtain a frequency distribution corresponding to the preset service index, and noise processing is performed on the frequency distribution corresponding to the preset service index to obtain a processed frequency distribution.
In practice, the amount of verification success per second may be substituted into the following equation
Obtaining k frequency distributions, wherein X (k) is the kth frequency distribution, N is the nth time point (i.e. nth second), N is the number of points of fourier transform (i.e. N is the time range of historical traffic index data, for example, the historical traffic index data is data of about 1 day, and the time point is second, then n=24×60×60), W N (kn) is a complex function of the variation,
where j is the imaginary part.
For example, when k=1,
and superposing the k frequency distributions to obtain a superposed frequency distribution, and carrying out noise treatment on the superposed frequency distribution to obtain a treated frequency distribution.
In S310, noise processing is performed on the frequency distribution corresponding to the preset service index by using a low-pass filtering algorithm, so as to obtain the processed frequency distribution.
The low-pass filtering algorithm may include various types, such as a first-order low-pass filtering algorithm, a second-order low-pass filtering algorithm, and the like, and the low-pass filtering algorithm in this embodiment may use the first-order low-pass filtering algorithm.
In implementation, noise in frequency distribution corresponding to a preset service index can be removed through a first-order low-pass filtering algorithm, so that the workload of subsequent processing is reduced, and the accuracy of determining an alarm baseline is improved.
In addition, in practical application, the selection of the low-pass filtering algorithm may be implemented by other low-pass filtering algorithms besides the first-order low-pass filtering algorithm, which may be different according to different practical application scenarios, and the embodiment of the present invention is not limited in particular.
In S312, based on the inverse fourier transform algorithm and the processed frequency distribution, inverse analysis is performed on the transformation from the time domain to the frequency domain of the historical business index data, so as to obtain an alarm baseline for the preset business index.
In practice, the processed frequency distribution may be converted into k frequency distributions, which are then substituted into the following formula
And obtaining alarm data points at each time point, and determining a corresponding alarm baseline by the n alarm data points.
In S314, current business index data of a preset business index in the business system is monitored based on the alarm baseline.
In implementation, the preset alarm information can be output when the current service index data of the preset service index in the service system is monitored and does not accord with the preset range corresponding to the alarm baseline. For example, as shown in fig. 2, if the current business index data exceeds the alarm threshold value at the corresponding time point (i.e. does not conform to the preset range corresponding to the alarm baseline), the current business index data may be considered to be abnormal, and preset alarm information may be output.
The alarm information may include information for prompting that the service provider currently has abnormal service index data, information for prompting that the service currently used by the user has abnormal service, and the like.
In addition, the user equipment corresponding to the current business index data which does not accord with the preset range corresponding to the alarm baseline can be obtained, and preset alarm information is output to the user equipment.
For example, application information installed in the user equipment may be acquired, an instant messaging application for outputting preset alert information is determined based on the application information, and the preset alert information is output through the instant messaging application. For example, an application program, such as an instant messaging application, capable of outputting preset alarm information can be determined through application program information installed in the user equipment, and the preset alarm information is output to the user through the determined one or more instant messaging applications.
In addition, the preset warning information can be output to the user equipment in a short message and/or email mode.
The embodiment of the specification provides a data monitoring method, which is characterized in that historical service index data of preset service indexes in a preset service scene is obtained based on log data generated in a service system, frequency analysis is carried out on the historical service index data to obtain frequency distribution corresponding to the preset service indexes, noise processing is carried out on the frequency distribution corresponding to the preset service indexes to obtain processed frequency distribution, reverse analysis is carried out on the conversion from a time domain to a frequency domain of the historical service index data based on the processed frequency distribution to obtain an alarm baseline aiming at the preset service indexes, the alarm baseline is reference data for judging whether to carry out alarm, and current service index data of the preset service indexes in the service system is monitored based on the alarm baseline. Therefore, the corresponding alarm baseline can be obtained through processing the historical service index data, and the current service index data is monitored, so that the problems of large workload and poor accuracy caused by a mode of manually setting the alarm baseline are avoided, namely the determination efficiency and the determination accuracy of the alarm baseline can be improved, and the monitoring efficiency and the monitoring accuracy are improved.
Example III
The data monitoring method provided in the embodiment of the present disclosure is based on the same concept, and the embodiment of the present disclosure further provides a data monitoring device, as shown in fig. 4.
The data monitoring device includes: a data acquisition module 401, a data processing module 402, a baseline acquisition module 403, and a data monitoring module 404, wherein:
the data acquisition module 401 is configured to acquire historical service index data of a preset service index in a preset service scenario based on log data generated in the service system;
the data processing module 402 is configured to perform frequency analysis on the historical service index data to obtain a frequency distribution corresponding to the preset service index, and perform noise processing on the frequency distribution corresponding to the preset service index to obtain a processed frequency distribution;
a baseline acquisition module 403, configured to reversely analyze a time domain to frequency domain transformation of the historical service index data based on the processed frequency distribution, to obtain an alarm baseline for the preset service index, where the alarm baseline is reference data for determining whether to perform an alarm;
and the data monitoring module 404 is configured to monitor current service indicator data of the preset service indicator in the service system based on the alarm baseline.
In the embodiment of the present disclosure, the data processing module 402 is configured to:
based on a fast Fourier transform algorithm, performing frequency analysis on the historical service index data to obtain frequency distribution corresponding to the preset service index;
the baseline acquisition module 403 is configured to:
and carrying out inverse analysis on the transformation from the time domain to the frequency domain of the historical business index data based on an inverse Fourier transform algorithm and the processed frequency distribution, and obtaining an alarm baseline aiming at the preset business index.
In this embodiment of the present disclosure, the data obtaining module 401 is configured to:
acquiring log data corresponding to the preset service scene, which is generated in the service system, in a preset time period;
acquiring first business index data in the log data aiming at the preset business index;
and preprocessing the first business index data to obtain the historical business index data.
In the embodiment of the present disclosure, the data processing module 402 is configured to:
and carrying out noise processing on the frequency distribution corresponding to the preset service index through a low-pass filtering algorithm to obtain the processed frequency distribution.
In an embodiment of the present disclosure, the apparatus further includes:
The information output module is used for outputting preset alarm information under the condition that the current service index data of the preset service index in the service system is monitored and does not accord with the preset range corresponding to the alarm baseline.
In this embodiment of the present disclosure, the information output module is configured to:
acquiring user equipment corresponding to the current business index data which does not accord with the preset range corresponding to the alarm baseline;
and outputting the preset alarm information to the user equipment.
In this embodiment of the present disclosure, the information output module is configured to:
acquiring application program information installed in the user equipment;
and determining an instant messaging application for outputting preset alarm information based on the application program information, and outputting the preset alarm information through the instant messaging application.
In this embodiment of the present disclosure, the information output module is configured to:
and outputting the preset alarm information to the user equipment in a short message and/or email mode.
The embodiment of the specification provides a data monitoring device, which is used for acquiring historical service index data of a preset service index in a preset service scene based on log data generated in a service system, performing frequency analysis on the historical service index data to obtain frequency distribution corresponding to the preset service index, performing noise processing on the frequency distribution corresponding to the preset service index to obtain processed frequency distribution, performing reverse analysis on the conversion from a time domain to a frequency domain of the historical service index data based on the processed frequency distribution to obtain an alarm baseline for the preset service index, wherein the alarm baseline is reference data for judging whether to alarm or not, and monitoring current service index data of the preset service index in the service system based on the alarm baseline. Therefore, the corresponding alarm baseline can be obtained through processing the historical service index data, and the current service index data is monitored, so that the problems of large workload and poor accuracy caused by a mode of manually setting the alarm baseline are avoided, namely the determination efficiency and the determination accuracy of the alarm baseline can be improved, and the monitoring efficiency and the monitoring accuracy are improved.
Example IV
Based on the same thought, the embodiment of the present disclosure further provides a data monitoring device, as shown in fig. 5.
The data monitoring device may be a terminal device or a server provided in the above embodiment.
The data monitoring device may vary considerably in configuration or performance and may include one or more processors 501 and memory 502, where the memory 502 may store one or more stored applications or data. Wherein the memory 502 may be transient storage or persistent storage. The application programs stored in memory 502 may include one or more modules (not shown), each of which may include a series of computer executable instructions in the data monitoring device. Still further, the processor 501 may be configured to communicate with the memory 502 and execute a series of computer executable instructions in the memory 502 on the data monitoring device. The data monitoring device may also include one or more power supplies 503, one or more wired or wireless network interfaces 504, one or more input/output interfaces 505, and one or more keyboards 504.
In particular, in this embodiment, the data monitoring device includes a memory, and one or more programs, wherein the one or more programs are stored in the memory, and the one or more programs may include one or more modules, and each module may include a series of computer-executable instructions in the data monitoring device, and the execution of the one or more programs by the one or more processors comprises instructions for:
Acquiring historical service index data of a preset service index in a preset service scene based on log data generated in a service system;
performing frequency analysis on the historical service index data to obtain frequency distribution corresponding to the preset service index, and performing noise processing on the frequency distribution corresponding to the preset service index to obtain processed frequency distribution;
based on the processed frequency distribution, reversely analyzing the transformation from the time domain to the frequency domain of the historical service index data to obtain an alarm baseline aiming at the preset service index, wherein the alarm baseline is reference data for judging whether to alarm or not;
and monitoring current service index data of the preset service index in the service system based on the alarm baseline.
Optionally, the performing frequency analysis on the historical service index data to obtain a frequency distribution corresponding to the preset service index includes:
based on a fast Fourier transform algorithm, performing frequency analysis on the historical service index data to obtain frequency distribution corresponding to the preset service index;
the inverse analysis of the transformation from the time domain to the frequency domain of the historical business index data based on the processed frequency distribution, to obtain an alarm baseline for the preset business index, includes:
And carrying out inverse analysis on the transformation from the time domain to the frequency domain of the historical business index data based on an inverse Fourier transform algorithm and the processed frequency distribution, and obtaining an alarm baseline aiming at the preset business index.
Optionally, the acquiring historical service index data of the preset service index in the preset service scene based on the log data generated in the service system includes:
acquiring log data corresponding to the preset service scene, which is generated in the service system, in a preset time period;
acquiring first business index data in the log data aiming at the preset business index;
and preprocessing the first business index data to obtain the historical business index data.
Optionally, the noise processing is performed on the frequency distribution corresponding to the preset service index to obtain a processed frequency distribution, which includes:
and carrying out noise processing on the frequency distribution corresponding to the preset service index through a low-pass filtering algorithm to obtain the processed frequency distribution.
Optionally, after the monitoring of the current business index data of the preset business index in the business system based on the alarm baseline, the method further includes:
And outputting preset alarm information under the condition that the current service index data of the preset service index in the service system is monitored and does not accord with the preset range corresponding to the alarm baseline.
Optionally, the outputting the preset alarm information includes:
acquiring user equipment corresponding to the current business index data which does not accord with the preset range corresponding to the alarm baseline;
and outputting the preset alarm information to the user equipment.
Optionally, the outputting the preset alarm information to the user equipment includes:
acquiring application program information installed in the user equipment;
and determining an instant messaging application for outputting preset alarm information based on the application program information, and outputting the preset alarm information through the instant messaging application.
Optionally, the outputting the preset alarm information to the user equipment includes:
and outputting the preset alarm information to the user equipment in a short message and/or email mode.
The embodiment of the specification provides a data monitoring device, which is used for acquiring historical service index data of a preset service index in a preset service scene based on log data generated in a service system, performing frequency analysis on the historical service index data to obtain frequency distribution corresponding to the preset service index, performing noise processing on the frequency distribution corresponding to the preset service index to obtain processed frequency distribution, performing reverse analysis on the conversion from a time domain to a frequency domain of the historical service index data based on the processed frequency distribution to obtain an alarm baseline for the preset service index, wherein the alarm baseline is reference data for judging whether to alarm or not, and monitoring current service index data of the preset service index in the service system based on the alarm baseline. Therefore, the corresponding alarm baseline can be obtained through processing the historical service index data, and the current service index data is monitored, so that the problems of large workload and poor accuracy caused by a mode of manually setting the alarm baseline are avoided, namely the determination efficiency and the determination accuracy of the alarm baseline can be improved, and the monitoring efficiency and the monitoring accuracy are improved.
Example five
The embodiments of the present disclosure further provide a computer readable storage medium, on which a computer program is stored, where the computer program when executed by a processor implements each process of the embodiments of the data monitoring method, and the same technical effects can be achieved, and for avoiding repetition, a detailed description is omitted herein. Wherein the computer readable storage medium is selected from Read-Only Memory (ROM), random access Memory (Random Access Memory, RAM), magnetic disk or optical disk.
The embodiment of the specification provides a computer readable storage medium, which is used for obtaining historical service index data of a preset service index in a preset service scene based on log data generated in a service system, performing frequency analysis on the historical service index data to obtain frequency distribution corresponding to the preset service index, performing noise processing on the frequency distribution corresponding to the preset service index to obtain processed frequency distribution, performing reverse analysis on the conversion from a time domain to a frequency domain of the historical service index data based on the processed frequency distribution to obtain an alarm baseline for the preset service index, wherein the alarm baseline is reference data for judging whether to perform alarm, and monitoring current service index data of the preset service index in the service system based on the alarm baseline. Therefore, the corresponding alarm baseline can be obtained through processing the historical service index data, and the current service index data is monitored, so that the problems of large workload and poor accuracy caused by a mode of manually setting the alarm baseline are avoided, namely the determination efficiency and the determination accuracy of the alarm baseline can be improved, and the monitoring efficiency and the monitoring accuracy are improved.
The foregoing describes specific embodiments of the present disclosure. Other embodiments are within the scope of the following claims. In some cases, the actions or steps recited in the claims can be performed in a different order than in the embodiments and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.
In the 90 s of the 20 th century, improvements to one technology could clearly be distinguished as improvements in hardware (e.g., improvements to circuit structures such as diodes, transistors, switches, etc.) or software (improvements to the process flow). However, with the development of technology, many improvements of the current method flows can be regarded as direct improvements of hardware circuit structures. Designers almost always obtain corresponding hardware circuit structures by programming improved method flows into hardware circuits. Therefore, an improvement of a method flow cannot be said to be realized by a hardware entity module. For example, a programmable logic device (Programmable Logic Device, PLD) (e.g., field programmable gate array (FieldProgrammable Gate Array, FPGA)) is an integrated circuit whose logic function is determined by the programming of the device by a user. A designer programs to "integrate" a digital system onto a PLD without requiring the chip manufacturer to design and fabricate application-specific integrated circuit chips. Moreover, nowadays, instead of manually manufacturing integrated circuit chips, such programming is mostly implemented by using "logic compiler" software, which is similar to the software compiler used in program development and writing, and the original code before the compiling is also written in a specific programming language, which is called hardware description language (Hardware Description Language, HDL), but not just one of the hdds, but a plurality of kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), lava, lola, myHDL, PALASM, RHDL (Ruby Hardware Description Language), etc., VHDL (Very-High-Speed Integrated Circuit Hardware Description Language) and Verilog are currently most commonly used. It will also be apparent to those skilled in the art that a hardware circuit implementing the logic method flow can be readily obtained by merely slightly programming the method flow into an integrated circuit using several of the hardware description languages described above.
The controller may be implemented in any suitable manner, for example, the controller may take the form of, for example, a microprocessor or processor and a computer readable medium storing computer readable program code (e.g., software or firmware) executable by the (micro) processor, logic gates, switches, application specific integrated circuits (Application Specific Integrated Circuit, ASIC), programmable logic controllers, and embedded microcontrollers, examples of which include, but are not limited to, the following microcontrollers: ARC 425D, atmel AT91SAM, microchip PIC18F24K20, and Silicone Labs C8051F320, the memory controller may also be implemented as part of the control logic of the memory. Those skilled in the art will also appreciate that, in addition to implementing the controller in a pure computer readable program code, it is well possible to implement the same functionality by logically programming the method steps such that the controller is in the form of logic gates, switches, application specific integrated circuits, programmable logic controllers, embedded microcontrollers, etc. Such a controller may thus be regarded as a kind of hardware component, and means for performing various functions included therein may also be regarded as structures within the hardware component. Or even means for achieving the various functions may be regarded as either software modules implementing the methods or structures within hardware components.
The system, apparatus, module or unit set forth in the above embodiments may be implemented in particular by a computer chip or entity, or by a product having a certain function. One typical implementation is a computer. In particular, the computer may be, for example, a personal computer, a laptop computer, a cellular telephone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.
For convenience of description, the above devices are described as being functionally divided into various units, respectively. Of course, the functionality of the units may be implemented in one or more software and/or hardware when implementing one or more embodiments of the present description.
It will be appreciated by those skilled in the art that embodiments of the present description may be provided as a method, system, or computer program product. Accordingly, one or more embodiments of the present description may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Moreover, one or more embodiments of the present description can take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.
Embodiments of the present description are described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the specification. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
In one typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.
The memory may include volatile memory in a computer-readable medium, random Access Memory (RAM) and/or nonvolatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM). Memory is an example of computer-readable media.
Computer readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of storage media for a computer include, but are not limited to, phase change memory (PRAM), static Random Access Memory (SRAM), dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), read Only Memory (ROM), electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium, which can be used to store information that can be accessed by a computing device. Computer-readable media, as defined herein, does not include transitory computer-readable media (transmission media), such as modulated data signals and carrier waves.
It should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article or apparatus that comprises the element.
It will be appreciated by those skilled in the art that embodiments of the present description may be provided as a method, system, or computer program product. Accordingly, one or more embodiments of the present description may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Moreover, one or more embodiments of the present description can take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.
One or more embodiments of the present specification may be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. One or more embodiments of the present description may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media including memory storage devices.
In this specification, each embodiment is described in a progressive manner, and identical and similar parts of each embodiment are all referred to each other, and each embodiment mainly describes differences from other embodiments. In particular, for system embodiments, since they are substantially similar to method embodiments, the description is relatively simple, as relevant to see a section of the description of method embodiments.
The foregoing is merely exemplary of the present disclosure and is not intended to limit the disclosure. Various modifications and alterations to this specification will become apparent to those skilled in the art. Any modifications, equivalent substitutions, improvements, or the like, which are within the spirit and principles of the present description, are intended to be included within the scope of the claims of the present description.
Claims (9)
1. A method of data monitoring, the method comprising:
acquiring historical service index data of a preset service index in a preset service scene based on log data generated in a service system;
performing frequency analysis on the historical service index data to obtain frequency distribution corresponding to the preset service index, and performing noise processing on the frequency distribution corresponding to the preset service index to obtain processed frequency distribution;
based on the processed frequency distribution, reversely analyzing the transformation from the time domain to the frequency domain of the historical service index data to obtain an alarm baseline aiming at the preset service index, wherein the alarm baseline is reference data for judging whether to alarm or not;
monitoring current service index data of the preset service index in the service system based on the alarm baseline;
under the condition that the current business index data of the preset business index in the business system is monitored and does not accord with the preset range corresponding to the alarm baseline, user equipment corresponding to the current business index data which does not accord with the preset range corresponding to the alarm baseline is obtained, and preset alarm information is output through instant messaging application determined by application program information installed in the user equipment;
The alarm baseline is a discrete time sequence including a plurality of time points, each time point corresponds to an alarm threshold, and the monitoring of the current service index data of the preset service index in the service system based on the alarm baseline includes:
and determining an alarm threshold corresponding to the current time point based on the current time point and the alarm baseline, and monitoring the current business index data based on the alarm threshold corresponding to the current time point.
2. The method of claim 1, wherein the performing frequency analysis on the historical traffic index data to obtain the frequency distribution corresponding to the preset traffic index comprises:
based on a fast Fourier transform algorithm, performing frequency analysis on the historical service index data to obtain frequency distribution corresponding to the preset service index;
the inverse analysis of the transformation from the time domain to the frequency domain of the historical business index data based on the processed frequency distribution, to obtain an alarm baseline for the preset business index, includes:
and carrying out inverse analysis on the transformation from the time domain to the frequency domain of the historical business index data based on an inverse Fourier transform algorithm and the processed frequency distribution, and obtaining an alarm baseline aiming at the preset business index.
3. The method of claim 1, wherein the obtaining historical service indicator data of the preset service indicator in the preset service scenario based on the log data generated in the service system comprises:
acquiring log data corresponding to the preset service scene, which is generated in the service system, in a preset time period;
acquiring first business index data in the log data aiming at the preset business index;
and preprocessing the first business index data to obtain the historical business index data.
4. The method of claim 1, wherein the noise processing is performed on the frequency distribution corresponding to the preset service index to obtain a processed frequency distribution, and the method comprises:
and carrying out noise processing on the frequency distribution corresponding to the preset service index through a low-pass filtering algorithm to obtain the processed frequency distribution.
5. The method of claim 1, the outputting the preset alert information, comprising:
and outputting the preset alarm information to the user equipment in a short message and/or email mode.
6. A data monitoring apparatus, the apparatus comprising:
the data acquisition module is used for acquiring historical service index data of a preset service index in a preset service scene based on log data generated in the service system;
The data processing module is used for carrying out frequency analysis on the historical service index data to obtain frequency distribution corresponding to the preset service index, and carrying out noise processing on the frequency distribution corresponding to the preset service index to obtain the processed frequency distribution;
the base line acquisition module is used for carrying out reverse analysis on the transformation from the time domain to the frequency domain of the historical service index data based on the processed frequency distribution to obtain an alarm base line aiming at the preset service index, wherein the alarm base line is reference data for judging whether to carry out alarm or not;
the data monitoring module is used for monitoring current service index data of the preset service index in the service system based on the alarm baseline;
the information output module is used for acquiring user equipment corresponding to the current business index data which does not accord with the preset range corresponding to the alarm baseline under the condition that the current business index data of the preset business index in the business system is monitored and does not accord with the preset range corresponding to the alarm baseline, and outputting preset alarm information through an instant messaging application determined by application program information installed in the user equipment;
The alarm baseline is a discrete time sequence comprising a plurality of time points, each time point corresponds to an alarm threshold value, and the data monitoring module is used for:
and determining an alarm threshold corresponding to the current time point based on the current time point and the alarm baseline, and monitoring the current business index data based on the alarm threshold corresponding to the current time point.
7. The apparatus of claim 6, the data processing module to:
based on a fast Fourier transform algorithm, performing frequency analysis on the historical service index data to obtain frequency distribution corresponding to the preset service index;
the baseline acquisition module is used for:
and carrying out inverse analysis on the processed frequency distribution based on an inverse Fourier transform algorithm to obtain an alarm baseline aiming at the preset service index.
8. The apparatus of claim 6, the data processing module to:
and carrying out noise processing on the frequency distribution corresponding to the preset service index through a low-pass filtering algorithm to obtain the processed frequency distribution.
9. A data monitoring device, the data monitoring device comprising:
A processor; and
a memory arranged to store computer executable instructions that, when executed, cause the processor to:
acquiring historical service index data of a preset service index in a preset service scene based on log data generated in a service system;
performing frequency analysis on the historical service index data to obtain frequency distribution corresponding to the preset service index, and performing noise processing on the frequency distribution corresponding to the preset service index to obtain processed frequency distribution;
based on the processed frequency distribution, reversely analyzing the transformation from the time domain to the frequency domain of the historical service index data to obtain an alarm baseline aiming at the preset service index, wherein the alarm baseline is reference data for judging whether to alarm or not;
monitoring current service index data of the preset service index in the service system based on the alarm baseline;
under the condition that the current business index data of the preset business index in the business system is monitored and does not accord with the preset range corresponding to the alarm baseline, user equipment corresponding to the current business index data which does not accord with the preset range corresponding to the alarm baseline is obtained, and preset alarm information is output through instant messaging application determined by application program information installed in the user equipment;
The alarm baseline is a discrete time sequence including a plurality of time points, each time point corresponds to an alarm threshold, and the monitoring of the current service index data of the preset service index in the service system based on the alarm baseline includes:
and determining an alarm threshold corresponding to the current time point based on the current time point and the alarm baseline, and monitoring the current business index data based on the alarm threshold corresponding to the current time point.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010530114.2A CN111639011B (en) | 2020-06-11 | 2020-06-11 | Data monitoring method, device and equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010530114.2A CN111639011B (en) | 2020-06-11 | 2020-06-11 | Data monitoring method, device and equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111639011A CN111639011A (en) | 2020-09-08 |
| CN111639011B true CN111639011B (en) | 2023-05-12 |
Family
ID=72332522
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010530114.2A Active CN111639011B (en) | 2020-06-11 | 2020-06-11 | Data monitoring method, device and equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111639011B (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112463834A (en) * | 2020-12-02 | 2021-03-09 | 中国建设银行股份有限公司 | Method and device for automatically realizing root cause analysis in streaming processing and electronic equipment |
| CN112416724B (en) * | 2020-12-04 | 2024-05-07 | 中国建设银行股份有限公司 | Alarm processing method, system, computer device and storage medium |
| CN113391981A (en) * | 2021-06-30 | 2021-09-14 | 中国民航信息网络股份有限公司 | Early warning method for monitoring index and related equipment |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107705149A (en) * | 2017-09-22 | 2018-02-16 | 平安科技（深圳）有限公司 | Data method for real-time monitoring, device, terminal device and storage medium |
| CN111158983A (en) * | 2019-12-27 | 2020-05-15 | 广东华保数据有限公司 | Integrated operation and maintenance management system |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102346745B (en) * | 2010-08-02 | 2014-04-02 | 阿里巴巴集团控股有限公司 | Method and device for predicting user behavior number for words |
| CN103854067B (en) * | 2012-12-06 | 2017-03-15 | 国家电网公司 | The detection method of load forecast precision and device |
-
2020
- 2020-06-11 CN CN202010530114.2A patent/CN111639011B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107705149A (en) * | 2017-09-22 | 2018-02-16 | 平安科技（深圳）有限公司 | Data method for real-time monitoring, device, terminal device and storage medium |
| CN111158983A (en) * | 2019-12-27 | 2020-05-15 | 广东华保数据有限公司 | Integrated operation and maintenance management system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111639011A (en) | 2020-09-08 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111639011B (en) | Data monitoring method, device and equipment | |
| CN110163417B (en) | Traffic prediction method, device and equipment | |
| CN107526667B (en) | Index abnormality detection method and device and electronic equipment | |
| CN110032698B (en) | Information display method and device, information processing method and device | |
| CN110457578B (en) | Customer service demand identification method and device | |
| CN110634030B (en) | Method, device and equipment for mining service indexes of applications | |
| CN111538869B (en) | Transaction abnormal group detection method, device and equipment | |
| CN110635962B (en) | Abnormity analysis method and device for distributed system | |
| CN109615171B (en) | Feature threshold determining method and device and problem object determining method and device | |
| CN110008394B (en) | Public opinion information identification method, device and equipment | |
| CN111324533A (en) | A/B test method and device and electronic equipment | |
| CN115712866B (en) | Data processing method, device and equipment | |
| CN110175084B (en) | Data change monitoring method and device | |
| CN116049761A (en) | Data processing method, device and equipment | |
| CN109597678B (en) | Task processing method and device | |
| CN117370034B (en) | Evaluation method and device of computing power dispatching system, storage medium and electronic equipment | |
| CN110245166B (en) | Data checking method and device | |
| CN116048977B (en) | Test method and device based on data reduction | |
| CN116485188A (en) | Risk identification method, device and equipment | |
| CN108470242B (en) | Risk management and control method, device and server | |
| CN112015570B (en) | Message reminding processing method and device | |
| CN111242195B (en) | Model, insurance wind control model training method and device and electronic equipment | |
| CN110046090B (en) | Page element positioning method and device | |
| CN110032495B (en) | Data anomaly detection method and device | |
| CN116304947A (en) | Service processing method and device and electronic equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| REG | Reference to a national code |
Ref country code: HK Ref legal event code: DE Ref document number: 40036413 Country of ref document: HK |
|
| GR01 | Patent grant | ||
| GR01 | Patent grant |