CN108509309A - A kind of system and method carrying out performance monitoring based on access log - Google Patents
A kind of system and method carrying out performance monitoring based on access log Download PDFInfo
- Publication number
- CN108509309A CN108509309A CN201810150710.0A CN201810150710A CN108509309A CN 108509309 A CN108509309 A CN 108509309A CN 201810150710 A CN201810150710 A CN 201810150710A CN 108509309 A CN108509309 A CN 108509309A
- Authority
- CN
- China
- Prior art keywords
- data
- calculated
- log
- time
- unit
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/34—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
- G06F11/3466—Performance evaluation by tracing or monitoring
- G06F11/3476—Data logging
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3065—Monitoring arrangements determined by the means or processing involved in reporting the monitored data
- G06F11/3068—Monitoring arrangements determined by the means or processing involved in reporting the monitored data where the reporting involves data format conversion
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/32—Monitoring with visual or acoustical indication of the functioning of the machine
- G06F11/321—Display for diagnostics, e.g. diagnostic result display, self-test user interface
- G06F11/322—Display of waveforms, e.g. of logic analysers
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/32—Monitoring with visual or acoustical indication of the functioning of the machine
- G06F11/324—Display of status information
- G06F11/327—Alarm or error message display
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Quality & Reliability (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Computer Hardware Design (AREA)
- Human Computer Interaction (AREA)
- Debugging And Monitoring (AREA)
Abstract
The invention discloses a kind of system and method carrying out performance monitoring based on access log, system includes：Access data collection module, data calculating and pushing module, real-time reception and display module.The present invention can carry out sensitive and intuitive performance evaluation, and versatile.The alarm algorithm requirement of each access log is identical, need not be respectively set；It is not influenced by abnormal data, helps long-term automatic study, ensure that index is accurate for a long time；Algorithm can be good at tradeoff wrong report and fail to report；Divide interface index that can be together as whole displaying according to what algorithm calculated, displayed page can automatic refresh data, and efficiency is relatively high, is conveniently operated, and can treat as live page and show；Daily record can be analyzed in real time, and supports distributed daily record.
Description
Technical field
The invention belongs to monitoring technology fields in computer digital animation, are related to a kind of based on access log progress performance prison
The system and method for control.
Background technology
Performance monitoring is a very important ring in system operation, and system fortune can be grasped at any time by performance monitoring
Market condition, even if so as to take processing means when being abnormal, and can be as the important reference that the later stage debugs.
The access log of Nginx, apache, tocmat etc. are frequently used to do performance monitoring.
At present when carrying out performance monitoring, it will usually the average response time for analyzing one day, when tapping the average response of mouth
Between, tap the mouth such as indexs such as hour statistical average response time according to the time period.According to average response time as index, simply
The coefficient that is multiplied by be used as, alert baseline.But this method has the following defects：
1. since each access log has difference, and continually changing feature at any time, the coefficient of index algorithm is caused to need
When adjusting by rule of thumb, inefficiency：This is because different access logs, needs to use different coefficients.If accessing day
Will amount is larger, then more times is needed to be arranged.In addition when time change (such as variation of dull and rush season), coefficient can lose
Effect, generates many wrong reports and fails to report situation.At this moment it needs to reset, it is time-consuming and laborious.
2. when average response time very little, setting coefficient can face adverse conditions to form the basic number of alarm：If setting
Small coefficient is influenced and little when flow is small, can be reported by mistake；If big coefficient is arranged, when flow is big, and the response time is less than
When baseline, big influence has at this moment been formd, can have been failed to report.So setting coefficient needs tradeoff demand (wrong report is failed to report), fish and
Bear's paw can not get both.
3. the intuitively access log of a monitoring page general system, it is only necessary to which a core curve is shown.But needle
To tapping the scheme of mouth difference coefficient, it is difficult to integrate, the core interface of a part can only be selected to show, can not be opened up comprehensively
Show its relevance.This just needs monitoring personnel to judge by rule of thumb, and whether system is whole problematic, it is difficult to refine.
Invention content
To solve the above problems, the invention discloses a kind of system and method carrying out performance monitoring based on access log,
Sensitive and intuitive performance evaluation can be carried out, and versatile.
In order to achieve the above object, the present invention provides the following technical solutions：
A kind of system that performance monitoring is carried out based on access log, including：Access data collection module, data are calculated and pushed away
Send module, real-time reception and display module；It accesses data collection module and is used to generate, convert and store the daily record of unified format simultaneously
Log Shipping is calculated to data and pushing module, data are calculated with pushing module for carrying out analysis calculating to daily record, and will
As a result it is pushed to real-time reception and display module, real-time reception and display module for obtaining data calculating and pushing module in real time
Analysis result simultaneously carries out related displaying.
Further, the access data collection module includes log producing unit and log storage unit, and daily record generates
Unit is used to be formatted processing to daily record, and log storage unit calculates w for storing daily record, for the daily record for needing to store
Value is not deposited if w is 1, and the calculation formula of w is as follows：
W=pre>baseline&&(pre-Cpre-Csqrt*p)
In above formula, pre is the average response time of interface current point in time, and baseline is Whole Response time reference line,
Baseline=Apre+m*Asqrt；Apre is the mean value for taking daylong data to calculate, and Asqrt is to take daylong number
According to the variance calculated, m is probability factor；Cpre is the mean value calculated according to storage data, and Csqrt is according to storage
The variance that data are calculated；
Data calculate and pushing module includes Data Computation Unit, index push unit and generation Alarm Unit, the number
Which, according to computing unit for identifying journal format according to data type and being belonging respectively to system, individually given birth to for each system
Script is calculated for carrying out collection of log data at a data, and current y values are then calculated according to following algorithm：
Y=w* (pre-Cpre)/Cpre/100*counts* ((pre>outline)+1)topNumber
Wherein, counts is the visit capacity of interface current point in time, and topNumber is ranking value range, and Outline is
User experience performance boundary；
Index push unit is used to the y values calculated being pushed to each browser and receive；
It generates Alarm Unit and is used to calculate the sum of the y values of tap mouth Ally, report is generated when Ally is more than some baseline value
Alert, Ally calculation formula are as follows：
Ally=Σ Yn* (1+coverage)
Wherein, Yn is the y values that ranking is n, and coverage is unnatural proportions；
Real-time reception and display module include real-time reception unit and curve display unit, and real-time reception unit is for obtaining
Data, the data received give curve displaying to be shown；Curve display unit is wrapped for showing curve on the page
Include the y curves of Ally curves and at least partly interface.
Further, processing procedure is formatted in the log producing unit includes：
1. the method for using canonical first judges whether to have in url " unified entrance ", if in the presence of if by question mark "" remove；
2. the method for then using canonical again will be similar to that " exception " of unified entrance identifies, they replaced
At sky；
3. then, changing number "/[0-9]+/ " into $ symbols successively, the content "/#.*/" after anchor note changes sky into；
4. again unify by question mark "" make " ＆ " symbol into；
5. finally according to demand, being derived from the number of parameters of oneself needs, remove remaining parameter.
Further, one day is divided into according to the time period when the log storage unit storage data a series of fixed
Point, and each to put one storage queue of each distribution.
Further, when calculating Cpre, the corresponding point of current slot and several several points of front are chosen, then take this
A little time points correspond to several data in queue.
Further, the index push unit places data into websocket message servers, in real time by it
It broadcasts and is received to each browser websocket receiving terminals.
Based on the method that access log carries out performance monitoring, specifically comprise the following steps：
Step 1, data collection is accessed
Step 1-1 is collected and is accessed data logging, and is formatted processing to daily record；
Step 1-2, passing through step 1-1, treated that formatted log stores, the daily record meter stored for needs
W values are calculated, are not deposited if w is 1, the calculation formula of w is as follows：
W=pre>baseline&&(pre-Cpre-Csqrt*p)
In above formula, pre is the average response time of interface current point in time, and baseline is Whole Response time reference line,
Baseline=Apre+m*Asqrt；Apre is the mean value for taking daylong data to calculate, and Asqrt is to take daylong number
According to the variance calculated, m is probability factor；Cpre is the mean value calculated according to storage data, and Csqrt is according to storage
The variance that data are calculated；
Step 2, data are calculated and are pushed
Step 2-1 automatically analyzes out how many daily record altogether, and identifies journal format and be belonging respectively to which system,
It is individually created a data for each system and calculates script, calculates script for carrying out collection of log data, according to following algorithm
Calculate current y values：
Y=w* (pre-Cpre)/Cpre/100*counts* ((pre>outline)+1)
Wherein, counts is the visit capacity of interface current point in time, and Outline is user experience performance boundary；
The y values calculated are pushed to each browser and received by step 2-2；
Step 2-3 calculates the sum of the y values of tap mouth Ally, alarm is generated when Ally is more than some baseline value；Ally is counted
It is as follows to calculate formula：
Ally=Σ Yn* (1+coverage)
Wherein, Yn is the y values that ranking is n, and coverage is unnatural proportions；The value range of n is 1-topNumber,
TopNumber is ranking value range；
Step 3, real-time reception and displaying
Step 3-1 obtains data；
Step 3-2 after receiving data, shows curve on the page, including the y of Ally curves and at least partly interface bent
Line.
Compared with prior art, the invention has the advantages that and advantageous effect：
The present invention provides a set of general sensitive and intuitive performance analysis system, the alarm algorithm of each access log is wanted
It asks identical, need not be respectively set；It is not influenced by abnormal data, helps long-term automatic study, ensure that index is accurate for a long time；It calculates
Method can be good at tradeoff wrong report and fail to report；Divide interface index that can be together as whole displaying according to what algorithm calculated, opens up
Show the page can automatic refresh data, and efficiency is relatively high, is conveniently operated, and can treat as live page and show；It can be real-time
Daily record is analyzed, and supports distributed daily record.
Description of the drawings
Fig. 1 is the system structure diagram provided by the invention that performance monitoring is carried out based on access log.
Fig. 2 is access analysis overall performance index curve graph.
Fig. 3 is the index curve graph of a certain interface.
Specific implementation mode
Technical solution provided by the invention is described in detail below with reference to specific embodiment, it should be understood that following specific
Embodiment is only illustrative of the invention and is not intended to limit the scope of the invention.In addition, step shown in the flowchart of the accompanying drawings
Suddenly it can be executed in the computer system of such as a group of computer-executable instructions, although also, showing in flow charts
Logical order, but in some cases, it can be with the steps shown or described are performed in an order that is different from the one herein.
As shown in Figure 1, the system provided by the invention that performance monitoring is carried out based on access log, including：Data are accessed to receive
Collect module, data calculating and pushing module, real-time reception and display module.Data collection module is accessed for generating, converting simultaneously
Store the daily record of unified format and by Log Shipping to data calculating and pushing module, data calculate and pushing module is for day
Will carries out analysis calculating, and result is pushed to real-time reception and display module, and real-time reception and display module for obtaining in real time
Access is according to calculating and pushing module analysis result and carry out related displaying.
Wherein, it includes log producing unit and log storage unit to access data collection module, and log producing unit is used for
Processing is formatted to daily record.The web containers such as nginx, tomcat, apache are all distributed deployments, they are produced
Distributed daily record.Due to carrying parameter in many interfaces, it is necessary to be formatted interface, variable replacement has been come at mark
At unique mark.
It needs to replace with lower part in these interfaces：Entrance url, the path of special rules, numerical character, # name anchor
Note, extra variable (for example only taking first variable, subsequent not take),Symbol.
The formatting procedure can use the language in logstash to realize, url is handled as follows：
1. the method for using canonical first identifies a part " unified entrance " in url, by url "" remove, it prevents not
Same interface blendes together an appearance.
2. the method for then using canonical again identifies that a part is also similar to that " exception " of unified entrance (answers root in url
The rule for belonging to exception is preset according to needs), it changes the mark of unified entrance into sky, prevents in fact the same interface quilt
Judge into an interface.
3. then, successively by digital " [0-9] ", the content " #.* " after anchor is remembered changes sky into, prevents them from being interfered.
4. next, again unify by question mark "" make " ＆ " symbol into.
5. finally according to demand, being derived from the number of parameters of oneself needs, for example only need to take first, then will
The content canonical of satisfaction " ＆.* " is removed, that is, removes other parameters below.
By handling above, interface can be converted to the mark of formatting.Each tap mouth is after formatting and converting, energy
It is enough to send out daily record according to prespecified format.
For storing daily record, technology used can be stored log storage unit with elasticsearch.Using multiple
Elasticsearch machines form a large amount of storages that cluster ensures data.The data stored externally provide api and access, this portion
Kibana can be referred to by dividing.When carrying out data calculating, these api interfaces will be used and complete data collection.Interface needs to divide
Period, for example, calculating for every 3 minutes primary.These data are stored into range, and range is related for targetedly storage
Data.
Log storage unit was divided into a series of fixed points by one day by the period above-mentioned, and each to put each distribution
One queue.W values are calculated for the data that should be stored into Each point in time, by the related data met all into queue, not on the same day
The data at same time point can be into the same queue.It is not deposited if w is 1.W indicates whether to belong to extremely, and it is 0 or 1, w to be worth
Data for 1 are abnormal datas, can ensure that data of the present invention are not influenced by abnormal data if not depositing, and are helped long-term automatic
Study ensures that index is accurate for a long time.The calculation formula of w is as follows：
W=pre>baseline&&(pre-Cpre-Csqrt*p)
In above formula, pre is the average response time of interface current point in time, and baseline is Whole Response time reference line,
Baseline=Apre+m*Asqrt.
Wherein, Apre is the mean value for taking daylong data to calculate, and Asqrt is that daylong data is taken to calculate
Variance, Apre and Asqrt that baseline is drawn according to daylong data determine, and developing programs for m is traditionally arranged to be
2. oneself is adjusted according to own situation.M is probability factor, can be false due to considering the data of historical time point
If current data fit is just distributed very much, so with reference to the data calculation formula being just distributed very much, different probability correspond to it is different because
Sub- size.Present invention acquiescence takes probability 1% to be used as benchmark.
Cpre is the mean value that the data in range are calculated.Csqrt is the variance that the data in range are calculated.
When calculating Cpre, the range taken is the corresponding point of current slot and several points (being usually 7) of front, then takes these
Time point corresponds to several data (being typically also 7) in queue, and it is many a normal to ensure that each period point has in this way
Data are as data from the sample survey.The Cpre and Csqrt for calculating these data can obtain average behavior and the side of current slot
Difference.
Data calculate and pushing module includes Data Computation Unit, index push unit and generation Alarm Unit.
Wherein, Data Computation Unit can automatically analyze out how many daily record altogether by elk interfaces, automatically according to
The data type of nginx, apache, tomcat identify journal format and are belonging respectively to which system.For each system list
It is only to calculate script at a data.Script is calculated at interval of the period, collection of log data is carried out, collects the access of tap mouth
These data are sent to access data collection module and are handled and stored, then according to algorithm by amount and response total duration
Calculate current y values.
Y=w* (pre-Cpre)/Cpre/100*counts* ((pre>outline)+1)
Wherein, y is final result index value, and counts is the visit capacity of interface current point in time.In view of performance is asked
Topic is generally concentrated in small part interface, and the interface for only calculating this part is substantially just enough, can save calculation amount in this way, subtract
Few unnecessary storage.Will be in a period, tapping mouth, respectively the response time calculates summation, by the inverted order ranking of summation,
TopNumber before taking.Outline is user experience performance boundary, and when performance is more than certain value, user experience can be non-
It is often poor, it can substantially cause flow to be lost in very much, so needing that this boundary value is arranged, be weighted to y.General website user
Boundary in 1-10 seconds ranges.It can give tacit consent to 2 seconds.The poor ratio of pre and Cpre, which is multiplied by counts, can take into account monitoring performance
The situation that poor small but visit capacity is big and poor performance is big but visit capacity is small.
The y values that index push unit is used to calculate, are placed into websocket message servers, in real time by it
Broadcast give each browser websocket receiving terminals receive.Here the technology of Websocket is all square with java and php etc.
Just it realizes.
Alarm Unit is generated, the sum of y values for calculating tap mouth Ally, Ally indicates overall performance index, when Ally is big
General alarm is just will produce when some baseline value.Baseline value is arranged as required to, such as 100, is more than 100 and is just triggered announcement
It is alert.
Ally=Σ Yn* (1+coverage)
Wherein, Yn is the y values that ranking is n, and coverage is unnatural proportions, refers to that w is abnormal connects in topNumber
Mouth accounting, the higher explanation of accounting, problem scope are wide.At this moment it is necessary to give mono- weight addition of Ally.The range of n is 1-
TopNumber, topNumber are ranking value range.
Real-time reception and display module include real-time reception unit and curve display unit.
Real-time reception unit is used to obtain the data of websocket message servers, since browser needs obtain in real time
The data of websocket message servers, this unit are taken to realize the receiving terminal of websocket using js.Message itself according to
System is grouped, and designated packet can obtain y the and Ally details of correspondence system.The data received give curve displaying
To be shown.
After curve display unit receives the data of real-time reception unit transmission, curve is shown on the page.Due to possessing
The detailed data of the y of Ally and tap mouth, so live page can assist the tap mouth of a part based on the curve of Ally
Y curves be shown.In Fig. 2 curve graphs, the line of bottom is exactly core index curve, and the protrusion that circle encloses is exactly to accuse
Displaying when alert.Ally shows that the achievement data of each interface is accumulated in together displaying spline curves with spline curves, can be with
More intuitively see the abnormal performance of total system.Fig. 3 curve graphs are the y curve graphs of a certain interface, and the line of bottom is y values
Figure, the protrusion that circle encloses is exactly abnormal data value.
Based on above system, the present invention also provides the method for carrying out performance monitoring based on access log, specifically include as
Lower step：
Step 1, data collection is accessed
Step 1-1 is collected and is accessed data logging, and is formatted processing to daily record, and formatting procedure is produced with reference to daily record
The function of being realized in raw unit.
Step 1-2, passing through step 1-1, treated that formatted log stores, and is divided by the period above-mentioned
A series of fixed points, and be their respectively one queues of distribution.By the related data calculated every time all into queue, if w is 1
It does not deposit then.Calculation formula of the w computational methods with reference to w in aforementioned log storage unit.
Step 2, data are calculated and are pushed
Step 2-1 automatically analyzes out how many daily record altogether, and identifies journal format and be belonging respectively to which system,
It is individually created a data for each system and calculates script, calculates script and be used to carry out collection of log data and push, then root
Current y values are calculated according to algorithm.Y values are calculated with reference to y calculation formula in Data Computation Unit.
Step 2-2, the y values that will be calculated, is placed into websocket message servers, broadcasted in real time by it to
Each browser websocket receiving terminals receive.
Step 2-3 calculates the sum of the y values of tap mouth Ally, and when Ally has value or big Mr. Yu's value just will produce generally
Alarm.As soon as can be defined here more than a baseline value, such as 100 triggering alarms.
Step 3, real-time reception and displaying
Step 3-1 is obtained the data of websocket message servers, the receiving terminal of websocket is realized using js.
Message itself is grouped according to system, and designated packet can obtain the y details of correspondence system.The data received, give song
Line shows to be shown.
Step 3-2 shows curve after receiving the data that real-time reception unit transmits on the page.With the curve of Ally
Based on, assist the y curves of the tap mouth of a part to be shown.As shown in Figure 2 and Figure 3.
The technical means disclosed in the embodiments of the present invention is not limited only to the technological means disclosed in the above embodiment, further includes
By the above technical characteristic arbitrarily the formed technical solution of combination.It should be pointed out that for those skilled in the art
For, various improvements and modifications may be made without departing from the principle of the present invention, these improvements and modifications are also considered as
Protection scope of the present invention.
Claims (7)
1. a kind of system carrying out performance monitoring based on access log, which is characterized in that including：Access data collection module, number
According to calculating and pushing module, real-time reception and display module；Data collection module is accessed for generating, converting and storing unified lattice
Log Shipping simultaneously is used to analyze daily record by the daily record of formula to data calculating and pushing module, data calculating and pushing module
It calculates, and result is pushed to real-time reception and display module, real-time reception and display module are calculated for obtaining data in real time
To pushing module analysis result and carry out related displaying.
2. the system according to claim 1 for carrying out performance monitoring based on access log, it is characterised in that：
The access data collection module includes log producing unit and log storage unit, and log producing unit is used for daily record
It is formatted processing, log storage unit calculates w values for storing daily record, for the daily record for needing to store, if w is 1 not
It deposits, the calculation formula of w is as follows：
W=pre>baseline&&(pre-Cpre-Csqrt*p)
In above formula, pre is the average response time of interface current point in time, and baseline is Whole Response time reference line,
Baseline=Apre+m*Asqrt；Apre is the mean value for taking daylong data to calculate, and Asqrt is to take daylong number
According to the variance calculated, m is probability factor；Cpre is the mean value calculated according to storage data, and Csqrt is according to storage
The variance that data are calculated；
Data calculate and pushing module includes Data Computation Unit, index push unit and generation Alarm Unit, the data meter
Unit is calculated for identifying journal format according to data type and being belonging respectively to which system, one is individually created for each system
A data calculate script for carrying out collection of log data, and current y values are then calculated according to following algorithm：
Y=w* (pre-Cpre)/Cpre/100*counts* ((pre>outline)+1)
Wherein, counts is the visit capacity of interface current point in time, and Outline is user experience performance boundary；
Index push unit is used to the y values calculated being pushed to each browser and receive；
It generates Alarm Unit and is used to calculate the sum of the y values of tap mouth Ally, alarm is generated when Ally is more than some baseline value,
Ally calculation formula are as follows：
Ally=Σ Yn* (1+coverage)
Wherein, Yn is the y values that ranking is n, and coverage is unnatural proportions, and the value range of n is 1-topNumber,
TopNumber is ranking value range；
Real-time reception and display module include real-time reception unit and curve display unit, and real-time reception unit is for obtaining number
According to the data received give curve displaying to be shown；Curve display unit for showing curve on the page, including
The y curves of Ally curves and at least partly interface.
3. the system according to claim 2 for carrying out performance monitoring based on access log, which is characterized in that the daily record production
Formatting processing procedure in raw unit includes：
1. the method for using canonical first judges whether to have in url " unified entrance ", if in the presence of if by question mark "" remove；
2. the method for then using canonical again will be similar to that " exception " of unified entrance identifies, they be substituted for sky；
3. then, changing number "/[0-9]+/ " into $ symbols successively, the content "/#.*/" after anchor note changes sky into；
4. again unify by question mark "" make " ＆ " symbol into；
5. finally according to demand, being derived from the number of parameters of oneself needs, remove remaining parameter.
4. the system according to claim 2 for carrying out performance monitoring based on access log, it is characterised in that：The daily record is deposited
It was divided into a series of fixed points according to the time period by one day when storage unit stores data, and each to put one storage team of each distribution
Row.
5. the system according to claim 2 for carrying out performance monitoring based on access log, which is characterized in that calculate Cpre
When, the corresponding point of current slot and several several points of front are chosen, then to take these time points to correspond to several in queue
Data.
6. the system according to claim 2 for carrying out performance monitoring based on access log, which is characterized in that the index pushes away
It send unit to place data into websocket message servers, is broadcasted by it and connect to each browser websocket in real time
Receiving end receives.
7. the method for carrying out performance monitoring based on access log, which is characterized in that specifically comprise the following steps：
Step 1, data collection is accessed
Step 1-1 is collected and is accessed data logging, and is formatted processing to daily record；
Step 1-2, passing through step 1-1, treated that formatted log stores, and w is calculated for the daily record for needing to store
Value is not deposited if w is 1, and the calculation formula of w is as follows：
W=pre>baseline&&(pre-Cpre-Csqrt*p)
In above formula, pre is the average response time of interface current point in time, and baseline is Whole Response time reference line,
Baseline=Apre+m*Asqrt；Apre is the mean value for taking daylong data to calculate, and Asqrt is to take daylong number
According to the variance calculated, m is probability factor；Cpre is the mean value calculated according to storage data, and Csqrt is according to storage
The variance that data are calculated；
Step 2, data are calculated and are pushed
Step 2-1 automatically analyzes out how many daily record altogether, and identifies journal format and be belonging respectively to which system, is every
A system is individually created a data and calculates script, calculates script for carrying out collection of log data, is calculated according to following algorithm
Current y values：
Y=w* (pre-Cpre)/Cpre/100*counts* ((pre>outline)+1)
Wherein, counts is the visit capacity of interface current point in time, and Outline is user experience performance boundary；
The y values calculated are pushed to each browser and received by step 2-2；
Step 2-3 calculates the sum of the y values of tap mouth Ally, alarm is generated when Ally is more than some baseline value；Ally calculates public
Formula is as follows：
Ally=Σ Yn* (1+coverage)
Wherein, Yn is the y values that ranking is n, and coverage is unnatural proportions, and the value range of n is 1-topNumber,
TopNumber is ranking value range；
Step 3, real-time reception and displaying
Step 3-1 obtains data；
Step 3-2 after receiving data, shows curve on the page, includes the y curves of Ally curves and at least partly interface.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810150710.0A CN108509309B (en) | 2018-02-13 | 2018-02-13 | System and method for performing performance monitoring based on access log |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810150710.0A CN108509309B (en) | 2018-02-13 | 2018-02-13 | System and method for performing performance monitoring based on access log |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN108509309A true CN108509309A (en) | 2018-09-07 |
| CN108509309B CN108509309B (en) | 2021-09-28 |
Family
ID=63375045
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810150710.0A Active CN108509309B (en) | 2018-02-13 | 2018-02-13 | System and method for performing performance monitoring based on access log |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108509309B (en) |
Cited By (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109857623A (en) * | 2018-12-13 | 2019-06-07 | 深圳壹账通智能科技有限公司 | Service performance analysis method, device and electronic equipment based on journal file |
| CN109885453A (en) * | 2019-02-18 | 2019-06-14 | 国家计算机网络与信息安全管理中心 | Big data platform monitoring system based on flow data processing |
| CN110442503A (en) * | 2019-07-29 | 2019-11-12 | 深圳数位传媒科技有限公司 | A kind of alarm method and device using log index |
| CN110493053A (en) * | 2019-08-22 | 2019-11-22 | 北京首都在线科技股份有限公司 | Merge monitoring method, device, terminal and the storage medium of content distributing network |
| CN110851234A (en) * | 2019-10-25 | 2020-02-28 | 中信梧桐港供应链管理有限公司 | Log processing method and device based on docker container |
| CN111082985A (en) * | 2019-12-16 | 2020-04-28 | 厦门大学附属第一医院 | API (application program interface) monitoring method based on open platform |
| CN112579418A (en) * | 2020-12-25 | 2021-03-30 | 泰康保险集团股份有限公司 | Method, device, equipment and computer readable medium for identifying access log |
| CN113282471A (en) * | 2021-05-17 | 2021-08-20 | 多点(深圳)数字科技有限公司 | Equipment performance testing method and device and terminal equipment |
| CN114244557A (en) * | 2021-11-09 | 2022-03-25 | 浪潮通用软件有限公司 | Development operation log isolation method and system based on user characteristics |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104598348A (en) * | 2015-02-28 | 2015-05-06 | 南京途牛科技有限公司 | Method and system for remotely analyzing performances of external system interfaces in real time |
| US9251035B1 (en) * | 2010-07-19 | 2016-02-02 | Soasta, Inc. | Load test charts with standard deviation and percentile statistics |
| CN106294091A (en) * | 2016-08-11 | 2017-01-04 | 福建富士通信息软件有限公司 | A kind of without intrusive mood daily record interception method for analyzing performance and system |
| CN106789189A (en) * | 2016-12-05 | 2017-05-31 | 深圳市彬讯科技有限公司 | A kind of SOA services monitor in real time is reported and performance metric system |
| CN107135119A (en) * | 2017-04-18 | 2017-09-05 | 国网福建省电力有限公司 | A kind of service response tracking and interface status monitor development system |
| CN107329877A (en) * | 2017-06-29 | 2017-11-07 | 南京途牛科技有限公司 | Air ticket business monitoring execution system and method |
-
2018
- 2018-02-13 CN CN201810150710.0A patent/CN108509309B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9251035B1 (en) * | 2010-07-19 | 2016-02-02 | Soasta, Inc. | Load test charts with standard deviation and percentile statistics |
| CN104598348A (en) * | 2015-02-28 | 2015-05-06 | 南京途牛科技有限公司 | Method and system for remotely analyzing performances of external system interfaces in real time |
| CN106294091A (en) * | 2016-08-11 | 2017-01-04 | 福建富士通信息软件有限公司 | A kind of without intrusive mood daily record interception method for analyzing performance and system |
| CN106789189A (en) * | 2016-12-05 | 2017-05-31 | 深圳市彬讯科技有限公司 | A kind of SOA services monitor in real time is reported and performance metric system |
| CN107135119A (en) * | 2017-04-18 | 2017-09-05 | 国网福建省电力有限公司 | A kind of service response tracking and interface status monitor development system |
| CN107329877A (en) * | 2017-06-29 | 2017-11-07 | 南京途牛科技有限公司 | Air ticket business monitoring execution system and method |
Cited By (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109857623A (en) * | 2018-12-13 | 2019-06-07 | 深圳壹账通智能科技有限公司 | Service performance analysis method, device and electronic equipment based on journal file |
| CN109885453A (en) * | 2019-02-18 | 2019-06-14 | 国家计算机网络与信息安全管理中心 | Big data platform monitoring system based on flow data processing |
| CN109885453B (en) * | 2019-02-18 | 2023-02-28 | 国家计算机网络与信息安全管理中心 | Big data platform monitoring system based on stream data processing |
| CN110442503A (en) * | 2019-07-29 | 2019-11-12 | 深圳数位传媒科技有限公司 | A kind of alarm method and device using log index |
| CN110493053A (en) * | 2019-08-22 | 2019-11-22 | 北京首都在线科技股份有限公司 | Merge monitoring method, device, terminal and the storage medium of content distributing network |
| CN110851234A (en) * | 2019-10-25 | 2020-02-28 | 中信梧桐港供应链管理有限公司 | Log processing method and device based on docker container |
| CN111082985A (en) * | 2019-12-16 | 2020-04-28 | 厦门大学附属第一医院 | API (application program interface) monitoring method based on open platform |
| CN112579418A (en) * | 2020-12-25 | 2021-03-30 | 泰康保险集团股份有限公司 | Method, device, equipment and computer readable medium for identifying access log |
| CN113282471A (en) * | 2021-05-17 | 2021-08-20 | 多点(深圳)数字科技有限公司 | Equipment performance testing method and device and terminal equipment |
| CN114244557A (en) * | 2021-11-09 | 2022-03-25 | 浪潮通用软件有限公司 | Development operation log isolation method and system based on user characteristics |
| CN114244557B (en) * | 2021-11-09 | 2023-06-16 | 浪潮通用软件有限公司 | Development operation log isolation method and system based on user characteristics |
Also Published As
| Publication number | Publication date |
|---|---|
| CN108509309B (en) | 2021-09-28 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108509309A (en) | A kind of system and method carrying out performance monitoring based on access log | |
| US11645293B2 (en) | Anomaly detection in big data time series analysis | |
| US20200258005A1 (en) | Unsupervised method for classifying seasonal patterns | |
| US10970891B2 (en) | Systems and methods for detecting and accommodating state changes in modelling | |
| US10248528B2 (en) | System monitoring method and apparatus | |
| US20190228022A1 (en) | System for detecting and characterizing seasons | |
| US10699211B2 (en) | Supervised method for classifying seasonal patterns | |
| CN105490854B (en) | Real-time logs collection method, system and application server cluster | |
| CN109684052B (en) | Transaction analysis method, device, equipment and storage medium | |
| CN111294217A (en) | Alarm analysis method, device, system and storage medium | |
| Wu et al. | Variation across scales: Measurement fidelity under twitter data sampling | |
| JPWO2007052327A1 (en) | Performance abnormality analyzer, method and program, and analysis result display method of performance abnormality analyzer | |
| CN111767951A (en) | Method for discovering abnormal data by applying isolated forest algorithm in residential electricity safety analysis | |
| CN113746703B (en) | Abnormal link monitoring method, system and device | |
| CN115022153B (en) | Fault root cause analysis method, device, equipment and storage medium | |
| CA3140769A1 (en) | Method and system for positioning fault root cause of service system | |
| CN116804957A (en) | System monitoring method and device | |
| CN111028862A (en) | Method, apparatus, computer device and storage medium for processing voice data | |
| RU2654167C2 (en) | Method for detecting hidden interactions on the internet of things | |
| CN116887340B (en) | Real-time pushing system for short message status report | |
| CA3083461A1 (en) | User interface system incorporating analysis of user interaction classification | |
| CN116910117A (en) | Multi-dimensional high-calculation-force microclimate sensor data analysis system and method | |
| CN116149947A (en) | Quality evaluation method and device for data model, electronic equipment and storage medium | |
| CN112800089B (en) | Intermediate data storage level adjusting method, storage medium and computer equipment | |
| US10812351B2 (en) | System and method for cyber sensing for power outage detection |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |