CN109981413B - Website monitoring index alarm method and system - Google Patents
Website monitoring index alarm method and system Download PDFInfo
- Publication number
- CN109981413B CN109981413B CN201910253059.4A CN201910253059A CN109981413B CN 109981413 B CN109981413 B CN 109981413B CN 201910253059 A CN201910253059 A CN 201910253059A CN 109981413 B CN109981413 B CN 109981413B
- Authority
- CN
- China
- Prior art keywords
- spectral peak
- peak
- frequency spectrum
- spectral
- abscissa
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000012544 monitoring process Methods 0.000 title claims abstract description 86
- 238000000034 method Methods 0.000 title claims abstract description 25
- 238000001228 spectrum Methods 0.000 claims abstract description 162
- 238000005311 autocorrelation function Methods 0.000 claims abstract description 37
- 238000001914 filtration Methods 0.000 claims abstract description 25
- 230000003595 spectral effect Effects 0.000 claims description 254
- 238000004364 calculation method Methods 0.000 claims description 19
- 238000010606 normalization Methods 0.000 claims description 7
- 230000000903 blocking effect Effects 0.000 claims description 2
- 238000005516 engineering process Methods 0.000 abstract description 8
- 238000012545 processing Methods 0.000 abstract description 7
- 230000000737 periodic effect Effects 0.000 description 19
- 238000001514 detection method Methods 0.000 description 9
- 230000009466 transformation Effects 0.000 description 8
- 238000010586 diagram Methods 0.000 description 7
- 238000004458 analytical method Methods 0.000 description 4
- 238000010183 spectrum analysis Methods 0.000 description 4
- 230000007547 defect Effects 0.000 description 3
- 230000000694 effects Effects 0.000 description 3
- 230000036541 health Effects 0.000 description 3
- 230000009467 reduction Effects 0.000 description 3
- 238000005070 sampling Methods 0.000 description 3
- 241001165575 Hylotelephium telephium subsp. maximum Species 0.000 description 2
- 230000002411 adverse Effects 0.000 description 2
- 230000001066 destructive effect Effects 0.000 description 2
- 238000007429 general method Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000008439 repair process Effects 0.000 description 2
- 238000012546 transfer Methods 0.000 description 2
- 230000002265 prevention Effects 0.000 description 1
- 238000005067 remediation Methods 0.000 description 1
- 230000011218 segmentation Effects 0.000 description 1
- 230000007704 transition Effects 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/08—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
Landscapes
- Engineering & Computer Science (AREA)
- Environmental & Geological Engineering (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Monitoring And Testing Of Transmission In General (AREA)
- Measurement Of Mechanical Vibrations Or Ultrasonic Waves (AREA)
Abstract
The invention discloses a method and a system for monitoring and alarming a website, wherein the method comprises the following steps: acquiring first time series data of the monitoring index; truncating the first time-series data using a window function; calculating an autocorrelation function of the intercepted first time-series data; filtering the autocorrelation function using a filter and removing a mean of the filtered autocorrelation function to obtain second time-series data; calculating a spectrum of the second time series data; and judging whether the frequency spectrum has periodicity or not, and if so, generating an alarm. According to the invention, the accurate frequency spectrum of the monitoring index can be obtained by carrying out window function processing, autocorrelation technology processing and filtering processing on the time sequence data of the monitoring index, so that frequency spectrum leakage and noise interference are prevented, and whether the monitoring index has a problem or not can be automatically and accurately analyzed by periodically judging and calculating the frequency spectrum, so that related personnel can be timely notified to avoid unnecessary loss.
Description
Technical Field
The invention relates to the technical field related to computer, internet and signal processing, in particular to a method and a system for alarming website monitoring indexes.
Background
Under a highly concurrent distributed environment, for services, interfaces and the like with large access amount, the health degree of a website needs to be monitored in time, slow access of the website is prevented, and even in special situations, scenes such as crash of an application server and the like occur, and in the face of a large amount of data, each website analysis tool has respective website monitoring indexes for different scenes, such as: the number of visitors to the website, the loyalty of the website users, the source of the website traffic, the attributes of the website visitors, and so forth. Monitoring indexes of a website are important references for measuring the health degree of business, application and a system, and the monitoring indexes are generally collected in a time sequence form and stored in a database. Due to the periodic fluctuation of the website monitoring indexes caused by the external destructive periodic access or the defects of the software system, attention needs to be paid in time and corresponding measures need to be taken for prevention or remediation. The traditional method basically depends on manual identification or can be found when an alarm is generated after indexes are deteriorated, the timeliness is insufficient, and no mature scheme can be used for reference from the current situation analysis of the internet industry and the periodic discovery of website monitoring indexes.
Disclosure of Invention
The invention provides a method and a system for alarming website monitoring indexes, aiming at overcoming the defect that the website monitoring indexes cannot be automatically, timely and accurately detected in the prior art.
The invention solves the technical problems through the following technical scheme:
the invention provides a method for alarming website monitoring indexes, which comprises the following steps:
acquiring first time series data of the monitoring index;
truncating the first time-series data using a window function;
calculating an autocorrelation function of the intercepted first time-series data;
filtering the autocorrelation function using a filter and removing a mean of the filtered autocorrelation function to obtain second time-series data;
calculating a spectrum of the second time series data;
and judging whether the frequency spectrum has periodicity or not, and if so, generating an alarm.
And acquiring first time sequence data of the monitoring index in real time at regular time intervals.
Preferably, the frequency spectrum of the second time series data is calculated by discrete fourier transform.
Preferably, the autocorrelation function is filtered using a low-pass butterworth filter.
Preferably, the step of determining whether the spectrum has periodicity includes:
Max-Min (a normalization method) normalizes the frequency spectrum and puts the frequency spectrum into a coordinate system with the ordinate of amplitude and the abscissa of frequency;
sorting the vertical coordinates of the spectral peaks of the frequency spectrum from large to small, wherein the sorted spectral peaks are sequentially as follows: a first spectral peak, a second spectral peak, a third spectral peak, a fourth spectral peak;
judging whether the ordinate of the second spectrum peak is smaller than a spectrum peak threshold value, if so, determining that the frequency spectrum has periodicity and the first spectrum peak is a target spectrum peak; if yes, judging whether the first spectral peak and the second spectral peak are continuous, if yes, executing step S1, and if not, executing step S2;
s1, judging whether the abscissa of the second spectral peak has a multiple relation with the abscissa of the first spectral peak, if so, determining that the frequency spectrum has periodicity and the first spectral peak is a target spectral peak;
s2, judging whether the third spectral peak is continuous with the first spectral peak and the second spectral peak, if not, executing a step S3, and if so, executing a step S4;
s3, judging whether the abscissa of the third spectral peak has a multiple relation with the abscissa of the first spectral peak or the second spectral peak, if so, determining that the frequency spectrum has periodicity, and determining the target spectral peak as a weighted average of the abscissa of the first spectral peak and the abscissa of the second spectral peak;
s4, judging whether the abscissa of the fourth spectral peak has a multiple relation with the abscissas of the first spectral peak, the second spectral peak or the third spectral peak, if not, the frequency spectrum has no periodicity, and if so, the frequency spectrum has periodicity; judging whether the second spectral peak and the third spectral peak are on the same side of the first spectral peak, if so, the first spectral peak is the target spectral peak, and if not, the target spectral peak is a weighted average of the abscissa of the second spectral peak and the third spectral peak;
the method for alarming the website monitoring index further comprises the following steps:
and calculating the period of the frequency spectrum according to the length of the first time sequence intercepted by the window function, the interval of the first time sequence and the abscissa of the target spectrum peak.
Wherein the multiple relation is an integer multiple relation, but a certain error is allowed.
Preferably, the method further comprises: if the frequency spectrum is judged to have periodicity, checking malicious IP through a network access log of the monitoring index, and blocking the malicious IP; or, detecting whether the application program itself is faulty by the application performance diagnostic tool.
The invention also provides a system for alarming the website monitoring index, which is characterized by comprising the following components:
the acquisition module is used for acquiring first time series data of the monitoring index;
a clipping module for clipping the first time-series data using a window function;
the first calculation module is used for calculating an autocorrelation function of the intercepted first time series data;
a filtering module, configured to filter the autocorrelation function by using a filter and remove a mean of the filtered autocorrelation function to obtain second time series data;
a second calculation module for calculating a spectrum of the second time-series data;
and the period judging module is used for judging whether the frequency spectrum has periodicity or not, and if so, generating an alarm.
Preferably, the second calculation module calculates a frequency spectrum of the second time-series data by discrete fourier transform.
Preferably, the filtering module filters the autocorrelation function using a low-pass butterworth filter.
Preferably, the period determination module includes:
the device comprises a normalization unit, a sorting unit, a first spectral peak judging unit, a second spectral peak judging unit, a third spectral peak judging unit, a fourth spectral peak judging unit, a fifth spectral peak judging unit and a sixth spectral peak judging unit;
the normalizing unit is used for normalizing the frequency spectrum through Max-Min; putting the frequency spectrum into a coordinate system with the ordinate as amplitude and the abscissa as frequency;
the sorting unit is used for sorting the vertical coordinates of the spectrum peaks of the frequency spectrum from large to small, and the sorted spectrum peaks sequentially comprise: a first spectral peak, a second spectral peak, a third spectral peak, a fourth spectral peak;
the first spectral peak judging unit is used for judging whether the ordinate of the second spectral peak is smaller than a spectral peak threshold value, if so, the frequency spectrum has periodicity, and the first spectral peak is a target spectral peak; if the first peak is larger than the second peak, executing a second peak judging unit, wherein the second peak judging unit is used for judging whether the first peak and the second peak are continuous, if so, executing a third peak judging unit, and if not, executing a fourth peak judging unit;
the third spectral peak judging unit is used for judging whether the abscissa of the second spectral peak has a multiple relation with the abscissa of the first spectral peak, if so, the frequency spectrum has periodicity, and the first spectral peak is a target spectral peak;
the fourth spectral peak judging unit is used for judging whether the third spectral peak is continuous with the first spectral peak and the second spectral peak, if not, executing a fifth spectral peak judging unit, and if so, executing a sixth spectral peak judging unit;
the fifth spectral peak judging unit is used for judging whether the abscissa of the third spectral peak has a multiple relation with the abscissa of the first spectral peak or the second spectral peak, if so, the frequency spectrum has periodicity, and the target spectral peak is a weighted average of the abscissa of the first spectral peak and the abscissa of the second spectral peak;
the sixth spectral peak judging unit is used for judging whether the abscissa of the fourth spectral peak has a multiple relation with the abscissas of the first spectral peak, the second spectral peak or the third spectral peak, if not, the frequency spectrum has no periodicity, and if so, the frequency spectrum has periodicity; judging whether the second spectral peak and the third spectral peak are on the same side of the first spectral peak, if so, the first spectral peak is the target spectral peak, and if not, the target spectral peak is a weighted average of the abscissa of the second spectral peak and the third spectral peak;
the system for alarming the website monitoring index further comprises:
and the third calculation module is used for calculating the period of the frequency spectrum according to the length of the first time sequence intercepted by the window function, the interval of the first time sequence and the abscissa of the target spectrum peak.
Preferably, the period judging module is further configured to check a malicious IP through a network access log of the monitoring index after judging that the spectrum has periodicity, and block the malicious IP; or, detecting whether the application program itself is faulty by the application performance diagnostic tool.
The positive progress effects of the invention are as follows:
according to the invention, the accurate frequency spectrum of the monitoring index can be obtained by carrying out window function processing, autocorrelation technology processing and filtering processing on the time sequence data of the monitoring index, so that frequency spectrum leakage and noise interference are prevented, and whether the monitoring index has a problem or not can be automatically and accurately analyzed by periodically judging and calculating the frequency spectrum, so that an alarm can be generated to a related operator in time, and unnecessary loss is avoided.
Drawings
Fig. 1 is a flowchart of a method for website monitoring index alarm according to embodiment 1.
Fig. 2 is a flowchart of a specific implementation of step 16 in embodiment 1.
Fig. 3 is a spectrum diagram of the first time-series data in example 1.
Fig. 4 is a spectrum diagram of the first time-series data after being truncated by using a window function in example 1.
Fig. 5 is a spectrum diagram of second time-series data in example 1.
Fig. 6 is a schematic diagram of the frequency amplitude characteristic of the butterworth low-pass filter in example 1.
Fig. 7 is a spectrum diagram obtained by FFT calculation in example 1.
FIG. 8 is a graph of the spectrum after Max-Min normalization in example 1.
Fig. 9 is a schematic block diagram of a website monitoring index alarm system according to embodiment 2.
Detailed Description
The invention is further illustrated by the following examples, which are not intended to limit the scope of the invention.
Example 1
The embodiment provides a method for alarming website monitoring indexes, as shown in fig. 1, the method includes:
In step 11, first time series data is acquired from a storage medium storing a website monitoring index at certain time intervals, where the first time series data may be defined as formula a:
wherein L is the length of the first time series data, n represents the nth data in the first time series data, and n ranges from 0 to L-1 in this embodiment.
In step 12, the first time series data is segmented and sliced by a Hamming function, and the first time series data is intercepted. Wherein, the definition of the Hamming function can be as formula b:
wherein α is a hyperparameter, and in formula b, others has the meaning: when n <0 or n > L-1, the Hamming window function takes a value of 0. Defining a Hamming window function vector as formula c, where n represents the nth vector:
the first time series data after being intercepted by the Hamming window function can be expressed as formula d:
wherein,
representing a point-to-point multiplication operation.
In order to prevent spectral leakage that may be caused by directly calculating DFT, in this embodiment, the hamming window function is used to truncate the first time series data in the time domain, so that spectral leakage can be prevented, and the subsequent spectral analysis is more accurate.
In this embodiment, the length of the first time series data is L, so the length of the autocorrelation function thereof is 2L-1, and the autocorrelation function obtained in step 13 is as shown in formula e:
any system has system noise, and the random body can be present on the monitoring index, so that certain random noise exists in the acquired time sequence. The random noise can be assumed to be white gaussian noise, and the amplitude-frequency characteristic of the white gaussian noise component in the time sequence is basically constant, that is, the energy of the noise is distributed from low frequency to high frequency, and occupies the whole frequency spectrum. Through traditional frequency domain filtering technology, can't filter cleanly well, however, gaussian white noise has a characteristic, is orthogonal to the signal of different moments, just so can carry out the filtering of making an uproar that falls through autocorrelation technique to the time series that contains the noise in the time domain, in this embodiment, through autocorrelation technique, thereby can restrain the gaussian white noise composition in the time series effectively and reach the effect of time domain filtering.
In this embodiment, by filtering the autocorrelation function and the low-pass butterworth filter, the difference equation of the filter can be expressed as a formula f:
where a (k) and b (k) are filter coefficients, N is filter order, M is equal to N in this embodiment, and the corresponding functions of a (k) and b (k) can be expressed in Z space as formula g:
the filter coefficients and the order need to be determined by an analog filter. The transfer function of the low-pass butterworth filter is an N-th order all-pole polynomial, which in this embodiment is the formula h:
wherein p is a normalized Las complex variable, p k Is the normalized pole on the complex plane located in the left half-plane. Determined by equation i:
the order N of the filter is determined by the formula j:
wherein k is sp And λ sp Respectively represented by formula k:
and formula l determines:
λ sp ＝Ω s /Ω p
the technical indexes of the filter are as follows:
maximum attenuation alpha of pass band p ＝3dB
Maximum attenuation alpha of stop band s ＝60dB
Cut-off frequency of pass band
Stop band cut-off frequency
The technical indexes are substituted into a solving formula of N to obtain
Therefore, in this embodiment, N is 3
By solving the coefficients in the denominator of the formula f, the following table can be obtained (only to order 8):
| N | b7 | b6 | b5 | b4 | b3 | | b1 | b0 | |
| 1 | 1.0000 | ||||||||
| 2 | 1.4142 | 1.0000 | |||||||
| 3 | 2.0000 | 2.0000 | 1.0000 | ||||||
| 4 | 2.6131 | 3.4142 | 2.6131 | 1.0000 | |||||
| 5 | 3.2361 | 5.2361 | 5.2361 | 3.2361 | 1.0000 | ||||
| 6 | 3.8637 | 7.4641 | 9.1416 | 7.4641 | 3.8637 | 1.0000 | |||
| 7 | 4.4940 | 10.0978 | 14.5918 | 14.5918 | 10.0978 | 4.4940 | 1.0000 | ||
| 8 | 5.1258 | 13.1371 | 21.8462 | 25.8462 | 21.8462 | 13.1371 | 5.1258 | 1.0000 |
to solve for the digital filter coefficients, the equation (formula f) needs to be mapped to Z-space by a bilinear transformation. The transformation formula is given by the formula m:
wherein omega c Is the cut-off frequency corresponding to-3 dB and T is the sampling rate 60 s. Maximum attenuation due to passband p 3dB, so Ω c ＝Ω p 。
Combining formula g, formula h, formula i and the table, the digital filter coefficients can be obtained as follows:
in step 14, the result y (n) is obtained by the formula f after the autocorrelation function passes through the filter, and the time series has a length of 1199. Removing the average value of y (n) to obtain the time sequence z (n), see formula n:
in this embodiment, in step 15, the spectrum of the second time-series data is calculated by Discrete Fourier Transform (DFT), see formula o:
in order to reduce the amount of computation and make the periodic computation faster, the above expression is implemented in such a way that the direct computation brings a large amount of computation, and the length of the time sequence is usually complemented to 1024 integral powers of 2, and then the result is computed by using FFT (fast fourier transform algorithm).
In this embodiment, when the frequency spectrum of the signal is obtained, the DFT is replaced by fast fourier transform, which is directly used for obtaining a larger calculation amount.
In the embodiment, the frequency spectrum leakage is prevented through the window function, noise reduction and filtering are performed on the noisy time sequence through the autocorrelation technology and the filter, interference of adverse factors is prevented, the calculation time is shortened through the FFT, and therefore the frequency spectrum of the second time sequence is calculated quickly and accurately.
In this embodiment, after the spectrum is calculated, step 16 is executed to determine whether the spectrum has periodicity, where a flowchart of specific implementation of the step is shown in fig. 2, and includes:
161, Max-Min normalizing the frequency spectrum, and putting the frequency spectrum into a coordinate system with the ordinate as amplitude and the abscissa as frequency;
if yes, executing step 165, judging whether the first spectral peak and the second spectral peak are continuous, if yes, executing step 166, and if not, executing step 167;
step 166, judging whether the abscissa of the second spectral peak has a multiple relation with the abscissa of the first spectral peak, if so, entering step 1662, wherein the frequency spectrum has periodicity and the first spectral peak is a target spectral peak; if not, go to step 1661, no periodicity exists;
1671, judging whether the abscissa of the third spectral peak has a multiple relation with the abscissa of the first spectral peak or the second spectral peak, if so, entering a step 1672, wherein the frequency spectrum has periodicity, and the target spectral peak is a weighted average of the abscissa of the first spectral peak and the abscissa of the second spectral peak; if not, go to step 1673, there is no periodicity;
step 1674, judging whether the abscissa of the fourth spectral peak has a multiple relation with the abscissas of the first spectral peak, the second spectral peak or the third spectral peak, if not, entering step 1676, if not, the spectrum has no periodicity, and if so, entering step 1675, the spectrum has periodicity; after executing step 1675, continuing to execute step 1677, determining whether the second spectral peak and the third spectral peak are on the same side of the first spectral peak, if so, executing step 1678, wherein the first spectral peak is the target spectral peak, and if not, executing step 1679, wherein the target spectral peak is a weighted average of abscissas of the second spectral peak and the third spectral peak;
wherein, according to experience, the threshold value of the second spectrum peak is 60% of the ordinate corresponding to the first spectrum peak.
The multiple relation is an integer multiple relation, but a certain error is allowed, in this embodiment, an error of 5% is allowed.
Wherein, by analyzing the frequency spectrum calculated by the formula o, the peak position of the spectrum in the amplitude-frequency characteristic is searched for detecting the periodicity contained in the original time sequence. Firstly, Max-Min normalization transformation is carried out on the frequency spectrum characteristics, and linear projection is carried out to [0,1], as the formula p:
the detection rule of the frequency spectrum after Max-Min normalized transformation is as follows:
if the ordinate corresponding to the first spectral peak is 0 or 1, using the second spectral peak as the first spectral peak, using the third spectral peak as the second spectral peak, and so on;
in this embodiment, the target spectral peak in step 1671 is a position weighted average of the first and second spectral peaks, and is calculated by formula q;
wherein index1 and index2 represent the positions of the first and second spectral peaks, respectively, index peak Representing the position of the target spectral peak.
In this embodiment, the target spectrum peak in step 1674 is a weighted average of abscissas of the second spectrum peak and the third spectrum peak, and is calculated by a formula r:
wherein index2 and index3 represent the positions of the second and third peaks, e.g. the positions of the first three peaks are 20, 19, 21, respectively, and the normalized spectral energy value is
And
according to the rule, index can be obtained peak ＝19.9974。
If the significant spectral peak exists in the amplitude-frequency characteristic, finding the position corresponding to the target spectral peak and marking as index peak The window length of the original monitoring time sequence is L, and the acquisition time interval is period x Then the period of the periodic sequence that can be detected according to equation s is:
in the embodiment, the problem of periodic automatic detection in the website monitoring index is solved, and the period in the monitoring index is calculated at the same time. A general method and a solution are provided for periodic discovery in website monitoring indexes, and the embodiment simplifies relative value operation in spectrum analysis by analyzing a frequency spectrum and normalizing amplitude-frequency characteristics; in addition, the accuracy and precision of periodic detection are improved through the detection rule of the target spectrum peak, so that whether the monitoring index of the website has a problem or not can be judged better.
In order to repair the problem of the website monitoring index, the embodiment may check the malicious IP through the network access log of the monitoring index, and block the malicious IP; or, whether the application program itself has a fault is detected by the application performance diagnostic tool, and garbage collection can be performed when the fault of the application program itself is detected.
In order to better understand the embodiment, the following further describes the website monitoring index alarming method according to the embodiment by using a specific example:
the monitoring indexes of the website are important references for measuring the health degree of services, applications and systems, most monitoring indexes are collected by taking the minute level as a sampling interval, the monitoring indexes are stored in a time sequence form after being collected, the periodic fluctuation of the monitoring indexes of the website caused by external destructive periodic access or software system defects needs to be paid attention in time and corresponding measures are taken for precaution or remedy, the traditional method basically depends on manual identification or can be found when an alarm is generated after the indexes are deteriorated, the timeliness is insufficient, therefore, the system needs to be capable of automatically finding the periodic characteristics in the monitoring indexes and further informing a system administrator of taking measures, therefore, in the embodiment, one index is collected every minute, namely the first time of obtaining the monitoring indexes of the website from a database every minute, and a frequency spectrum graph of the indexes collected every minute is shown in figure 3, i.e. a spectrogram of a first time series, wherein the abscissa represents the frequency of the first time series and the ordinate represents the amplitude of the corresponding frequency; after the spectrogram of the index is acquired, intercepting the frequency spectrum through a Hamming window function, in this example, the length parameter L of the window function is set to 600, and the hyperparameter α is set to 0.54, and fig. 4 shows the spectrogram of the first time series data to which the window function is added, wherein the abscissa represents the frequency of the first time series after the Hamming window is added, and the ordinate represents the amplitude of the corresponding frequency; then, calculating an autocorrelation function of the truncated first time series, and filtering the autocorrelation function by using a butterworth type low-pass filter and removing a mean value of the autocorrelation function to obtain second time series data, where in this example, the order of the filter is N ═ 3, and the cut-off frequency domain is:
the amplitude-frequency characteristics are shown in fig. 6, where the abscissa is frequency, the ordinate is amplitude, and the transition band is attenuation of-60 dB per 10 octaves, fig. 5 is a spectrogram of second time series data, where the abscissa represents frequency and the ordinate represents amplitude of the corresponding frequency, where the upper spectrum is a spectrogram of a time series calculated by an autocorrelation function, and the lower spectrum is a spectrogram of a low-pass filtered second time series; obtaining a second time sequence, and then obtaining a frequency spectrum through FFT calculation, wherein fig. 7 is a characteristic diagram of the frequency spectrum, in which the abscissa is frequency and the ordinate is amplitude; then, after normalizing the spectrum by Max-Min to calculate the detected target spectral peak, fig. 8 is a normalized spectrogram, wherein the abscissa is frequency and the ordinate is normalized amplitude corresponding to the frequency. As can be found from fig. 7, the positions of the first three spectral peaks are 20, 19, and 21, respectively, and the normalized spectral energy values are:
according to the above rule, it is possible to obtain:
index peak ＝19.9974
substituting into equation s, the period can be calculated as:
that is, the period component having a period of 1 hour exists in the monitoring timing index.
In the embodiment, the spectral characteristics of the monitoring index are accurately calculated, the spectral leakage is avoided through a window function, the white Gaussian noise of a time domain is filtered through an autocorrelation technology, the high-frequency component which is not concerned is removed through a frequency domain low-pass filter, and the spectral quality of the monitoring index is improved; by analyzing the frequency spectrum, the periodic characteristics of the frequency spectrum can be automatically, timely and accurately detected, so that the problem of the website monitoring index can be timely found, and the problem can be timely repaired.
Example 2
As shown in fig. 9, the present embodiment provides a system for website monitoring index alarm, including: the spectrum peak detection device comprises an acquisition module 21, an interception module 22, a first calculation module 23, a filtering module 24, a second calculation module 25, a third calculation module 26 and a period judgment module 27, wherein the period judgment module 27 further comprises a normalization unit, a sorting unit, a first spectrum peak judgment unit, a second spectrum peak judgment unit, a third spectrum peak judgment unit, a fourth spectrum peak judgment unit, a fifth spectrum peak judgment unit and a sixth spectrum peak judgment unit;
the obtaining module 21 is configured to obtain first time series data of the monitoring index;
the intercepting module 22 is used for intercepting the first time series data by using a window function;
the first calculating module 23 is configured to calculate an autocorrelation function of the intercepted first time-series data;
the filtering module 24 is configured to filter the autocorrelation function by using a filter and remove a mean of the filtered autocorrelation function to obtain second time series data;
the second calculating module 25 is configured to calculate a frequency spectrum of the second time-series data;
the period judging module 27 is configured to judge whether the frequency spectrum has periodicity, and if so, execute step 17 and generate an alarm; if not, continuing monitoring.
The third calculation module 26 is used to calculate the periodicity of the spectrum.
The obtaining module 21 obtains first time series data from a storage medium storing website monitoring indexes at certain time intervals, where the first time series data may be defined as formula a:
wherein L is the length of the first time series data, n represents the nth data in the first time series data, and n ranges from 0 to L-1 in this embodiment.
The truncation module 22 performs segmentation slicing on the first time-series data by a Hamming function, and truncates the first time-series data. Wherein, the definition of the Hamming function can be as formula b:
wherein α is a hyperparameter, and in formula b, others has the meaning: when n <0 or n > L-1, the Hamming window function takes a value of 0. Defining a Hamming window function vector as formula c, where n represents the nth vector:
the first time series data after being intercepted by the Hamming window function can be expressed as formula d:
wherein,
representing a point-to-point multiplication operation.
In order to prevent spectral leakage that may be caused by directly calculating DFT, in this embodiment, the hamming window function is used to truncate the first time series data in the time domain, so that spectral leakage can be prevented, and the subsequent spectral analysis is more accurate.
In this embodiment, the length of the first time series data is L, so the length of the autocorrelation function thereof is 2L-1, and the autocorrelation function obtained in step 13 is as shown in formula e:
any system has system noise, and the random body can be present on the monitoring index, so that certain random noise exists in the acquired time sequence. The random noise can be assumed to be white gaussian noise, and the amplitude-frequency characteristic of the white gaussian noise component in the time sequence is basically constant, that is, the energy of the noise is distributed from low frequency to high frequency, and occupies the whole frequency spectrum. Through traditional frequency domain filtering technique, can't well filter totally, however, gaussian white noise has a characteristic, is orthogonal to the signal of different moments, just so can fall the filtering of making an uproar to the time series that contains the noise in the time domain through autocorrelation technique, in this embodiment, through autocorrelation technique, thereby can restrain the gaussian white noise component in the time series effectively and reach the effect of time domain filtering.
In this embodiment, by filtering the autocorrelation function and the low-pass butterworth filter, the difference equation of the filter can be expressed as a formula f:
where a (k) and b (k) are filter coefficients, N is filter order, M is equal to N in this embodiment, and the corresponding functions of a (k) and b (k) can be expressed in Z space as formula g:
the filter coefficients and the order need to be determined by an analog filter. The transfer function of the low-pass Butterworth filter is an N-th order all-pole polynomial of formula h:
wherein p is a normalized Las complex variable, p k Is the normalized pole on the complex plane located in the left half-plane. Determined by equation i:
the order N of the filter is determined by the formula j:
wherein k is sp And λ sp Determined by equations k and l, respectively:
λ sp ＝Ω s /Ω p
the technical indexes of the filter are as follows:
maximum attenuation alpha of pass band p ＝3dB
Maximum attenuation alpha of stop band s ＝60dB
Cut-off frequency of pass band
Stop band cut-off frequency
The technical indexes are substituted into a solving formula of N to obtain
Therefore, in this embodiment, N is 3
By solving the coefficients of the terms in the denominator of the equation (formula f), the following table (calculated to order 8 only) can be obtained:
to solve for the digital filter coefficients, equation f needs to be mapped to Z-space by a bilinear transformation. The transformation formula is given by the formula m:
wherein omega c Is the cut-off frequency corresponding to-3 dB and T is the sampling rate 60 s. Maximum attenuation due to passband of alpha p 3dB, so Ω c ＝Ω p 。
Combining the formula g, the formula h, the formula i and the table, the digital filter coefficient can be obtained as follows:
after the autocorrelation function passes through the filter in the filtering module 24, the result y (n) is obtained by the formula f, and the time series has a length of 1199. Removing the average value of y (n) to obtain the time sequence z (n), see formula n:
in this embodiment, the second calculating module 25 calculates the frequency spectrum of the second time series data by Discrete Fourier Transform (DFT), see formula o, that is:
in order to reduce the amount of computation and make the periodic computation faster, the above expression is implemented in such a way that the direct computation brings a large amount of computation, and the length of the time sequence is usually complemented to 1024 integral powers of 2, and then the result is computed by using FFT (fast fourier transform algorithm).
In the embodiment, the frequency spectrum leakage is prevented through the window function, noise reduction and filtering are performed on the noisy time sequence through the autocorrelation technology and the filter, interference of adverse factors is prevented, the calculation time is shortened through the FFT, and therefore the frequency spectrum of the second time sequence is calculated quickly and accurately.
In this embodiment, in the period determining module 27, the normalizing unit 271 is configured to normalize the frequency spectrum by Max-Min, and place the frequency spectrum in a coordinate system with an amplitude on a vertical coordinate and a frequency on a horizontal coordinate;
the sorting unit 272 is configured to sort the vertical coordinates of the spectral peaks of the frequency spectrum from large to small, where the sorted spectral peaks sequentially are: a first spectral peak, a second spectral peak, a third spectral peak, a fourth spectral peak;
the first peak determining unit 273 is configured to determine whether a vertical coordinate of the second peak is smaller than a peak threshold, and if so, the frequency spectrum has periodicity and the first peak is a target peak; if the first peak is greater than the second peak, executing a second peak determining unit 274, where the second peak determining unit is configured to determine whether the first peak and the second peak are continuous, if so, executing a third peak determining unit 275, and if not, executing a fourth peak determining unit 276;
the third peak determining unit 275 is configured to determine whether there is a multiple relationship between the abscissa of the second peak and the abscissa of the first peak, if yes, the frequency spectrum has periodicity, and the first peak is a target peak;
the fourth peak determining unit 276 is configured to determine whether the third peak is consecutive to the first peak and the second peak, if not, execute the fifth peak determining unit 277, and if yes, execute the sixth peak determining unit 278;
the fifth spectral peak determining unit 277 is configured to determine whether an abscissa of the third spectral peak has a multiple relationship with an abscissa of the first spectral peak or an abscissa of the second spectral peak, if so, the frequency spectrum has periodicity, and the target spectral peak is a weighted average of the abscissas of the first spectral peak and the second spectral peak;
a sixth peak determining unit 278 is configured to determine whether an abscissa of the fourth peak has a multiple relationship with an abscissa of the first peak, the second peak, or the third peak, if the abscissa does not have a multiple relationship, the spectrum does not have periodicity, and if the abscissa does have periodicity, the spectrum has periodicity; judging whether the second spectral peak and the third spectral peak are on the same side of the first spectral peak, if so, the first spectral peak is the target spectral peak, and if not, the target spectral peak is a weighted average of the abscissa of the second spectral peak and the third spectral peak;
the system for alarming the website monitoring index further comprises:
and the third calculating module 26 is configured to calculate a period of the frequency spectrum according to the length of the first time series intercepted by the window function, the interval of the first time series, and the abscissa of the target spectrum peak.
The multiple relation is an integer multiple relation, but a certain error is allowed, in this embodiment, an error of 5% is allowed.
The frequency spectrum obtained by calculation through an analysis formula o is used for searching the position of a spectrum peak in amplitude-frequency characteristics for detecting the periodicity contained in an original time sequence. Firstly, Max-Min normalization transformation is carried out on the frequency spectrum characteristics, and the frequency spectrum characteristics are linearly projected to [0,1], such as a formula p
The detection rule of the frequency spectrum after Max-Min normalized transformation is as follows:
if the ordinate corresponding to the first spectral peak is 0 or 1, the second spectral peak is used as the first spectral peak, the third spectral peak is used as the second spectral peak, and so on;
wherein, according to experience, the threshold value of the second spectrum peak is 60% of the ordinate corresponding to the first spectrum peak.
In this embodiment, the target peak in the fifth peak determining unit 277 is a position weighted average of the first and second peaks, and is calculated by formula q;
wherein index1 and index2 represent the positions of the first and second spectral peaks, respectively, index peak Representing the position of the target spectral peak.
In this embodiment, the target spectrum peak in the sixth spectrum peak determining unit 278 is a weighted average of abscissas of the second spectrum peak and the third spectrum peak, and is calculated by a formula r:
wherein index2 and index3 represent the positions of the second and third spectral peaks, for exampleFor example, the positions of the first three spectral peaks are 20, 19 and 21 respectively, and the normalized spectral energy value is
And
according to the rule, index can be obtained peak ＝19.9974。
If the amplitude-frequency characteristic has obvious spectral peaks, the corresponding position of the target spectral peak is found and is marked as index peak The window length of the original monitoring time sequence is L, and the acquisition time interval is period x Then the period of the periodic sequence that can be detected according to equation s is:
in this embodiment, the third calculating module 26 is configured to calculate the period of the frequency spectrum according to the length of the first time series intercepted by the window function, the interval of the first time series, and the abscissa of the target spectral peak.
In this embodiment, the period determining module 27 is further configured to, after determining that the spectrum has periodicity, check a malicious IP through the network access log of the monitoring index, and block the malicious IP; or, detecting whether the application program itself is faulty by the application performance diagnostic tool.
In the embodiment, the problem of periodic automatic detection in the website monitoring index is solved, and the period in the monitoring index is calculated at the same time. A general method and a solution are provided for periodic discovery in website monitoring indexes, and the embodiment simplifies relative value operation in spectrum analysis by analyzing a frequency spectrum and normalizing amplitude-frequency characteristics; in addition, the accuracy and precision of periodic detection are improved through the detection rule of the target spectrum peak, so that whether the monitoring index of the website has a problem or not can be better judged.
In the embodiment, website monitoring indexes are segmented by adding a Hamming window, time domain noise reduction is carried out on the segmented time sequence through an autocorrelation technology, then frequency domain low-pass filtering is carried out on the noise-reduced result, amplitude-frequency characteristics are obtained through discrete Fourier transform calculation after high-frequency interference components are removed, finally whether periodicity exists in the corresponding monitoring indexes is judged through analysis on normalized frequency spectrum components, and if periodicity exists, the periodicity is calculated. Through the embodiment, the website can find the periodic characteristics of the monitoring indexes in time, so that the reason causing the periodicity can be found and dealt with in time
In order to repair the problem of the website monitoring index, the embodiment may check the malicious IP through the network access log of the monitoring index, and block the malicious IP; or, whether the application program itself has a fault is detected by the application performance diagnostic tool, and garbage collection can be performed when the fault of the application program itself is detected.
In the embodiment, the spectral characteristics of the monitoring index are accurately calculated, the spectral leakage is avoided through a window function, the white Gaussian noise of a time domain is filtered through an autocorrelation technology, the high-frequency component which is not concerned is removed through a frequency domain low-pass filter, and the spectral quality of the monitoring index is improved; by analyzing the frequency spectrum, the periodic characteristics of the frequency spectrum can be automatically, timely and accurately detected, so that the problem of the website monitoring index can be timely found, and the problem can be timely repaired.
While specific embodiments of the invention have been described above, it will be appreciated by those skilled in the art that this is by way of example only, and that the scope of the invention is defined by the appended claims. Various changes and modifications to these embodiments may be made by those skilled in the art without departing from the spirit and scope of the invention, and these changes and modifications are within the scope of the invention.
Claims (8)
1. A method for alarming website monitoring indexes is characterized by comprising the following steps:
acquiring first time series data of the monitoring index;
truncating the first time-series data using a window function;
calculating an autocorrelation function of the intercepted first time-series data;
filtering the autocorrelation function using a filter and removing a mean of the filtered autocorrelation function to obtain second time-series data;
calculating a spectrum of the second time series data;
judging whether the frequency spectrum has periodicity or not, and if so, generating an alarm;
the step of judging whether the frequency spectrum has periodicity comprises the following steps:
Max-Min normalizes the frequency spectrum, and puts the frequency spectrum into a coordinate system with the ordinate as amplitude and the abscissa as frequency;
sorting the vertical coordinates of the spectral peaks of the frequency spectrum from large to small, wherein the sorted spectral peaks are sequentially as follows: a first spectral peak, a second spectral peak, a third spectral peak, a fourth spectral peak;
judging whether the ordinate of the second spectrum peak is smaller than a spectrum peak threshold value, if so, determining that the frequency spectrum has periodicity and the first spectrum peak is a target spectrum peak; if yes, judging whether the first spectral peak and the second spectral peak are continuous, if yes, executing step S1, and if not, executing step S2;
s1, judging whether the abscissa of the second spectral peak has a multiple relation with the abscissa of the first spectral peak, if so, determining that the frequency spectrum has periodicity and the first spectral peak is a target spectral peak;
s2, judging whether the third spectral peak is continuous with the first spectral peak and the second spectral peak, if not, executing a step S3, and if so, executing a step S4;
s3, judging whether the abscissa of the third spectral peak has a multiple relation with the abscissa of the first spectral peak or the second spectral peak, if so, determining that the frequency spectrum has periodicity, and determining the target spectral peak as a weighted average of the abscissa of the first spectral peak and the abscissa of the second spectral peak;
s4, judging whether the abscissa of the fourth spectral peak has a multiple relation with the abscissas of the first spectral peak, the second spectral peak or the third spectral peak, if not, the frequency spectrum has no periodicity, and if so, the frequency spectrum has periodicity; judging whether the second spectral peak and the third spectral peak are on the same side of the first spectral peak, if so, the first spectral peak is the target spectral peak, and if not, the target spectral peak is a weighted average of the abscissa of the second spectral peak and the third spectral peak;
the method for alarming the website monitoring index further comprises the following steps:
and calculating the period of the frequency spectrum according to the length of the first time sequence intercepted by the window function, the interval of the first time sequence and the abscissa of the target spectrum peak.
2. The method for website monitoring indicator alerting as defined in claim 1, wherein the spectrum of the second time-series data is calculated by discrete fourier transform.
3. The method for website monitoring indicator alerting as defined in claim 1, wherein the autocorrelation function is filtered using a low-pass butterworth filter.
4. The method for website monitoring indicator alerting as defined in claim 1, further comprising: if the frequency spectrum is judged to have periodicity, checking malicious IP through a network access log of the monitoring index, and blocking the malicious IP; or, detecting whether the application program itself is faulty by the application performance diagnostic tool.
5. A system for alarming website monitoring indexes is characterized by comprising:
the acquisition module is used for acquiring first time series data of the monitoring index;
a clipping module for clipping the first time-series data using a window function;
the first calculation module is used for calculating an autocorrelation function of the intercepted first time series data;
a filtering module, configured to filter the autocorrelation function by using a filter and remove a mean of the filtered autocorrelation function to obtain second time series data;
a second calculation module for calculating a spectrum of the second time-series data;
the period judging module is used for judging whether the frequency spectrum has periodicity or not, and if so, generating an alarm;
the period judging module comprises: the device comprises a normalization unit, a sorting unit, a first spectral peak judging unit, a second spectral peak judging unit, a third spectral peak judging unit, a fourth spectral peak judging unit, a fifth spectral peak judging unit and a sixth spectral peak judging unit;
the normalizing unit is used for normalizing the frequency spectrum through Max-Min; putting the frequency spectrum into a coordinate system with the ordinate as amplitude and the abscissa as frequency;
the sorting unit is used for sorting the vertical coordinates of the spectrum peaks of the frequency spectrum from large to small, and the sorted spectrum peaks sequentially comprise: a first spectral peak, a second spectral peak, a third spectral peak, a fourth spectral peak;
the first spectral peak judging unit is used for judging whether the ordinate of the second spectral peak is smaller than a spectral peak threshold value, if so, the frequency spectrum has periodicity, and the first spectral peak is a target spectral peak; if the first peak is larger than the second peak, executing a second peak judging unit, wherein the second peak judging unit is used for judging whether the first peak and the second peak are continuous, if so, executing a third peak judging unit, and if not, executing a fourth peak judging unit;
the third spectral peak judging unit is used for judging whether the abscissa of the second spectral peak has a multiple relation with the abscissa of the first spectral peak, if so, the frequency spectrum has periodicity, and the first spectral peak is a target spectral peak;
the fourth spectral peak judging unit is used for judging whether the third spectral peak is continuous with the first spectral peak and the second spectral peak, if not, executing a fifth spectral peak judging unit, and if so, executing a sixth spectral peak judging unit;
the fifth spectral peak judging unit is used for judging whether the abscissa of the third spectral peak has a multiple relation with the abscissa of the first spectral peak or the second spectral peak, if so, the frequency spectrum has periodicity, and the target spectral peak is a weighted average of the abscissa of the first spectral peak and the abscissa of the second spectral peak;
the sixth spectral peak judging unit is used for judging whether the abscissa of the fourth spectral peak has a multiple relation with the abscissas of the first spectral peak, the second spectral peak or the third spectral peak, if not, the frequency spectrum has no periodicity, and if so, the frequency spectrum has periodicity; judging whether the second spectral peak and the third spectral peak are on the same side of the first spectral peak, if so, the first spectral peak is the target spectral peak, and if not, the target spectral peak is a weighted average of the abscissa of the second spectral peak and the third spectral peak;
the system for alarming the website monitoring index further comprises:
and the third calculation module is used for calculating the period of the frequency spectrum according to the length of the first time sequence intercepted by the window function, the interval of the first time sequence and the abscissa of the target spectrum peak.
6. The website monitoring indicator alert system of claim 5, wherein the second calculation module calculates a frequency spectrum of the second time series data by discrete Fourier transform.
7. The system for website monitoring indicator alerting as defined in claim 5, wherein the filtering module filters the autocorrelation function using a low-pass butterworth filter.
8. The system for website monitoring index alarming as recited in claim 5, wherein the period determination module is further configured to check malicious IPs through a network access log of the monitoring index and block the malicious IPs after determining that the spectrum has periodicity; or, detecting whether the application program itself is faulty by the application performance diagnostic tool.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910253059.4A CN109981413B (en) | 2019-03-29 | 2019-03-29 | Website monitoring index alarm method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910253059.4A CN109981413B (en) | 2019-03-29 | 2019-03-29 | Website monitoring index alarm method and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109981413A CN109981413A (en) | 2019-07-05 |
| CN109981413B true CN109981413B (en) | 2022-08-02 |
Family
ID=67081861
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910253059.4A Active CN109981413B (en) | 2019-03-29 | 2019-03-29 | Website monitoring index alarm method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109981413B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112526369B (en) * | 2019-09-18 | 2024-03-01 | 广州极飞科技股份有限公司 | Metering method for electric quantity based on voltage value, electric quantity determining method and device thereof |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN100472507C (en) * | 2004-11-24 | 2009-03-25 | 南京大学 | An estimating method for periodic signal period under strong background noise |
| EP2299650A1 (en) * | 2009-09-21 | 2011-03-23 | Siemens Aktiengesellschaft | Method for recognising anomalies in a control network |
| CN106850511B (en) * | 2015-12-07 | 2020-03-27 | 阿里巴巴集团控股有限公司 | Method and device for identifying access attack |
-
2019
- 2019-03-29 CN CN201910253059.4A patent/CN109981413B/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| CN109981413A (en) | 2019-07-05 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110535702B (en) | Alarm information processing method and device | |
| CA2417074C (en) | Method and system for trend detection and analysis | |
| CN111307438B (en) | Rotary machine vibration fault diagnosis method and system based on information entropy | |
| CN109238455B (en) | A kind of characteristic of rotating machines vibration signal monitoring method and system based on graph theory | |
| CN110717472B (en) | Fault diagnosis method and system based on improved wavelet threshold denoising | |
| CN107395608B (en) | Network access abnormity detection method and device | |
| CN110083507B (en) | Key performance index classification method and device | |
| CN117434337B (en) | Broadband voltage signal measurement method and system | |
| Shi et al. | The VMD-scale space based hoyergram and its application in rolling bearing fault diagnosis | |
| CN109981413B (en) | Website monitoring index alarm method and system | |
| CN118329197B (en) | Abnormality monitoring method and system for printing plate roller | |
| CN112597958B (en) | Automatic identification method and system for rolling bearing fault | |
| CN104111109B (en) | A kind of vibration condition recognition methods based on different order statistic and support vector machine | |
| Ekberg et al. | Network behaviour anomaly detection using Holt-Winters algorithm | |
| CN115979989A (en) | Terahertz detection method and device and computer readable storage medium | |
| CN112836583B (en) | Wind turbine fault diagnosis method | |
| Puchalski et al. | Stable distributions, generalised entropy, and fractal diagnostic models of mechanical vibration signals | |
| Jain et al. | A new framework for on-line change detection | |
| EP4345666A1 (en) | Anomaly detection using hash signature generation for model-based scoring | |
| CN113987482B (en) | IP first access detection method, system and equipment based on FM | |
| CN118509263B (en) | Information security scheduling method and system suitable for power operation and maintenance network | |
| CN118520402B (en) | Fault diagnosis system and method for large-sized variable-speed unit | |
| CN111985380B (en) | Bearing degradation process state monitoring method, system, equipment and storage medium | |
| CN117969681B (en) | Analysis method and system for internal damage and defects of titanium alloy pipe | |
| JPH10339664A (en) | Monitor and monitoring method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |