CN108509309A - A kind of system and method carrying out performance monitoring based on access log - Google Patents

A kind of system and method carrying out performance monitoring based on access log Download PDF

Info

Publication number
CN108509309A
CN108509309A CN201810150710.0A CN201810150710A CN108509309A CN 108509309 A CN108509309 A CN 108509309A CN 201810150710 A CN201810150710 A CN 201810150710A CN 108509309 A CN108509309 A CN 108509309A
Authority
CN
China
Prior art keywords
data
calculated
log
time
unit
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN201810150710.0A
Other languages
Chinese (zh)
Other versions
CN108509309B (en
Inventor
胡小敏
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Nanjing Tu Niu Science And Technology Ltd
Original Assignee
Nanjing Tu Niu Science And Technology Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Nanjing Tu Niu Science And Technology Ltd filed Critical Nanjing Tu Niu Science And Technology Ltd
Priority to CN201810150710.0A priority Critical patent/CN108509309B/en
Publication of CN108509309A publication Critical patent/CN108509309A/en
Application granted granted Critical
Publication of CN108509309B publication Critical patent/CN108509309B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00Error detection; Error correction; Monitoring
    • G06F11/30Monitoring
    • G06F11/34Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
    • G06F11/3466Performance evaluation by tracing or monitoring
    • G06F11/3476Data logging
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00Error detection; Error correction; Monitoring
    • G06F11/30Monitoring
    • G06F11/3065Monitoring arrangements determined by the means or processing involved in reporting the monitored data
    • G06F11/3068Monitoring arrangements determined by the means or processing involved in reporting the monitored data where the reporting involves data format conversion
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00Error detection; Error correction; Monitoring
    • G06F11/30Monitoring
    • G06F11/32Monitoring with visual or acoustical indication of the functioning of the machine
    • G06F11/321Display for diagnostics, e.g. diagnostic result display, self-test user interface
    • G06F11/322Display of waveforms, e.g. of logic analysers
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00Error detection; Error correction; Monitoring
    • G06F11/30Monitoring
    • G06F11/32Monitoring with visual or acoustical indication of the functioning of the machine
    • G06F11/324Display of status information
    • G06F11/327Alarm or error message display

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • General Engineering & Computer Science (AREA)
  • Quality & Reliability (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Hardware Design (AREA)
  • Human Computer Interaction (AREA)
  • Debugging And Monitoring (AREA)

Abstract

The invention discloses a kind of system and method carrying out performance monitoring based on access log, system includes:Access data collection module, data calculating and pushing module, real-time reception and display module.The present invention can carry out sensitive and intuitive performance evaluation, and versatile.The alarm algorithm requirement of each access log is identical, need not be respectively set;It is not influenced by abnormal data, helps long-term automatic study, ensure that index is accurate for a long time;Algorithm can be good at tradeoff wrong report and fail to report;Divide interface index that can be together as whole displaying according to what algorithm calculated, displayed page can automatic refresh data, and efficiency is relatively high, is conveniently operated, and can treat as live page and show;Daily record can be analyzed in real time, and supports distributed daily record.

Description

A kind of system and method carrying out performance monitoring based on access log
Technical field
The invention belongs to monitoring technology fields in computer digital animation, are related to a kind of based on access log progress performance prison The system and method for control.
Background technology
Performance monitoring is a very important ring in system operation, and system fortune can be grasped at any time by performance monitoring Market condition, even if so as to take processing means when being abnormal, and can be as the important reference that the later stage debugs.
The access log of Nginx, apache, tocmat etc. are frequently used to do performance monitoring.
At present when carrying out performance monitoring, it will usually the average response time for analyzing one day, when tapping the average response of mouth Between, tap the mouth such as indexs such as hour statistical average response time according to the time period.According to average response time as index, simply The coefficient that is multiplied by be used as, alert baseline.But this method has the following defects:
1. since each access log has difference, and continually changing feature at any time, the coefficient of index algorithm is caused to need When adjusting by rule of thumb, inefficiency:This is because different access logs, needs to use different coefficients.If accessing day Will amount is larger, then more times is needed to be arranged.In addition when time change (such as variation of dull and rush season), coefficient can lose Effect, generates many wrong reports and fails to report situation.At this moment it needs to reset, it is time-consuming and laborious.
2. when average response time very little, setting coefficient can face adverse conditions to form the basic number of alarm:If setting Small coefficient is influenced and little when flow is small, can be reported by mistake;If big coefficient is arranged, when flow is big, and the response time is less than When baseline, big influence has at this moment been formd, can have been failed to report.So setting coefficient needs tradeoff demand (wrong report is failed to report), fish and Bear's paw can not get both.
3. the intuitively access log of a monitoring page general system, it is only necessary to which a core curve is shown.But needle To tapping the scheme of mouth difference coefficient, it is difficult to integrate, the core interface of a part can only be selected to show, can not be opened up comprehensively Show its relevance.This just needs monitoring personnel to judge by rule of thumb, and whether system is whole problematic, it is difficult to refine.
Invention content
To solve the above problems, the invention discloses a kind of system and method carrying out performance monitoring based on access log, Sensitive and intuitive performance evaluation can be carried out, and versatile.
In order to achieve the above object, the present invention provides the following technical solutions:
A kind of system that performance monitoring is carried out based on access log, including:Access data collection module, data are calculated and pushed away Send module, real-time reception and display module;It accesses data collection module and is used to generate, convert and store the daily record of unified format simultaneously Log Shipping is calculated to data and pushing module, data are calculated with pushing module for carrying out analysis calculating to daily record, and will As a result it is pushed to real-time reception and display module, real-time reception and display module for obtaining data calculating and pushing module in real time Analysis result simultaneously carries out related displaying.
Further, the access data collection module includes log producing unit and log storage unit, and daily record generates Unit is used to be formatted processing to daily record, and log storage unit calculates w for storing daily record, for the daily record for needing to store Value is not deposited if w is 1, and the calculation formula of w is as follows:
W=pre>baseline&&(pre-Cpre-Csqrt*p)
In above formula, pre is the average response time of interface current point in time, and baseline is Whole Response time reference line, Baseline=Apre+m*Asqrt;Apre is the mean value for taking daylong data to calculate, and Asqrt is to take daylong number According to the variance calculated, m is probability factor;Cpre is the mean value calculated according to storage data, and Csqrt is according to storage The variance that data are calculated;
Data calculate and pushing module includes Data Computation Unit, index push unit and generation Alarm Unit, the number Which, according to computing unit for identifying journal format according to data type and being belonging respectively to system, individually given birth to for each system Script is calculated for carrying out collection of log data at a data, and current y values are then calculated according to following algorithm:
Y=w* (pre-Cpre)/Cpre/100*counts* ((pre>outline)+1)topNumber
Wherein, counts is the visit capacity of interface current point in time, and topNumber is ranking value range, and Outline is User experience performance boundary;
Index push unit is used to the y values calculated being pushed to each browser and receive;
It generates Alarm Unit and is used to calculate the sum of the y values of tap mouth Ally, report is generated when Ally is more than some baseline value Alert, Ally calculation formula are as follows:
Ally=Σ Yn* (1+coverage)
Wherein, Yn is the y values that ranking is n, and coverage is unnatural proportions;
Real-time reception and display module include real-time reception unit and curve display unit, and real-time reception unit is for obtaining Data, the data received give curve displaying to be shown;Curve display unit is wrapped for showing curve on the page Include the y curves of Ally curves and at least partly interface.
Further, processing procedure is formatted in the log producing unit includes:
1. the method for using canonical first judges whether to have in url " unified entrance ", if in the presence of if by question mark "" remove;
2. the method for then using canonical again will be similar to that " exception " of unified entrance identifies, they replaced At sky;
3. then, changing number "/[0-9]+/ " into $ symbols successively, the content "/#.*/" after anchor note changes sky into;
4. again unify by question mark "" make " & " symbol into;
5. finally according to demand, being derived from the number of parameters of oneself needs, remove remaining parameter.
Further, one day is divided into according to the time period when the log storage unit storage data a series of fixed Point, and each to put one storage queue of each distribution.
Further, when calculating Cpre, the corresponding point of current slot and several several points of front are chosen, then take this A little time points correspond to several data in queue.
Further, the index push unit places data into websocket message servers, in real time by it It broadcasts and is received to each browser websocket receiving terminals.
Based on the method that access log carries out performance monitoring, specifically comprise the following steps:
Step 1, data collection is accessed
Step 1-1 is collected and is accessed data logging, and is formatted processing to daily record;
Step 1-2, passing through step 1-1, treated that formatted log stores, the daily record meter stored for needs W values are calculated, are not deposited if w is 1, the calculation formula of w is as follows:
W=pre>baseline&&(pre-Cpre-Csqrt*p)
In above formula, pre is the average response time of interface current point in time, and baseline is Whole Response time reference line, Baseline=Apre+m*Asqrt;Apre is the mean value for taking daylong data to calculate, and Asqrt is to take daylong number According to the variance calculated, m is probability factor;Cpre is the mean value calculated according to storage data, and Csqrt is according to storage The variance that data are calculated;
Step 2, data are calculated and are pushed
Step 2-1 automatically analyzes out how many daily record altogether, and identifies journal format and be belonging respectively to which system, It is individually created a data for each system and calculates script, calculates script for carrying out collection of log data, according to following algorithm Calculate current y values:
Y=w* (pre-Cpre)/Cpre/100*counts* ((pre>outline)+1)
Wherein, counts is the visit capacity of interface current point in time, and Outline is user experience performance boundary;
The y values calculated are pushed to each browser and received by step 2-2;
Step 2-3 calculates the sum of the y values of tap mouth Ally, alarm is generated when Ally is more than some baseline value;Ally is counted It is as follows to calculate formula:
Ally=Σ Yn* (1+coverage)
Wherein, Yn is the y values that ranking is n, and coverage is unnatural proportions;The value range of n is 1-topNumber, TopNumber is ranking value range;
Step 3, real-time reception and displaying
Step 3-1 obtains data;
Step 3-2 after receiving data, shows curve on the page, including the y of Ally curves and at least partly interface bent Line.
Compared with prior art, the invention has the advantages that and advantageous effect:
The present invention provides a set of general sensitive and intuitive performance analysis system, the alarm algorithm of each access log is wanted It asks identical, need not be respectively set;It is not influenced by abnormal data, helps long-term automatic study, ensure that index is accurate for a long time;It calculates Method can be good at tradeoff wrong report and fail to report;Divide interface index that can be together as whole displaying according to what algorithm calculated, opens up Show the page can automatic refresh data, and efficiency is relatively high, is conveniently operated, and can treat as live page and show;It can be real-time Daily record is analyzed, and supports distributed daily record.
Description of the drawings
Fig. 1 is the system structure diagram provided by the invention that performance monitoring is carried out based on access log.
Fig. 2 is access analysis overall performance index curve graph.
Fig. 3 is the index curve graph of a certain interface.
Specific implementation mode
Technical solution provided by the invention is described in detail below with reference to specific embodiment, it should be understood that following specific Embodiment is only illustrative of the invention and is not intended to limit the scope of the invention.In addition, step shown in the flowchart of the accompanying drawings Suddenly it can be executed in the computer system of such as a group of computer-executable instructions, although also, showing in flow charts Logical order, but in some cases, it can be with the steps shown or described are performed in an order that is different from the one herein.
As shown in Figure 1, the system provided by the invention that performance monitoring is carried out based on access log, including:Data are accessed to receive Collect module, data calculating and pushing module, real-time reception and display module.Data collection module is accessed for generating, converting simultaneously Store the daily record of unified format and by Log Shipping to data calculating and pushing module, data calculate and pushing module is for day Will carries out analysis calculating, and result is pushed to real-time reception and display module, and real-time reception and display module for obtaining in real time Access is according to calculating and pushing module analysis result and carry out related displaying.
Wherein, it includes log producing unit and log storage unit to access data collection module, and log producing unit is used for Processing is formatted to daily record.The web containers such as nginx, tomcat, apache are all distributed deployments, they are produced Distributed daily record.Due to carrying parameter in many interfaces, it is necessary to be formatted interface, variable replacement has been come at mark At unique mark.
It needs to replace with lower part in these interfaces:Entrance url, the path of special rules, numerical character, # name anchor Note, extra variable (for example only taking first variable, subsequent not take),Symbol.
The formatting procedure can use the language in logstash to realize, url is handled as follows:
1. the method for using canonical first identifies a part " unified entrance " in url, by url "" remove, it prevents not Same interface blendes together an appearance.
2. the method for then using canonical again identifies that a part is also similar to that " exception " of unified entrance (answers root in url The rule for belonging to exception is preset according to needs), it changes the mark of unified entrance into sky, prevents in fact the same interface quilt Judge into an interface.
3. then, successively by digital " [0-9] ", the content " #.* " after anchor is remembered changes sky into, prevents them from being interfered.
4. next, again unify by question mark "" make " & " symbol into.
5. finally according to demand, being derived from the number of parameters of oneself needs, for example only need to take first, then will The content canonical of satisfaction " &.* " is removed, that is, removes other parameters below.
By handling above, interface can be converted to the mark of formatting.Each tap mouth is after formatting and converting, energy It is enough to send out daily record according to prespecified format.
For storing daily record, technology used can be stored log storage unit with elasticsearch.Using multiple Elasticsearch machines form a large amount of storages that cluster ensures data.The data stored externally provide api and access, this portion Kibana can be referred to by dividing.When carrying out data calculating, these api interfaces will be used and complete data collection.Interface needs to divide Period, for example, calculating for every 3 minutes primary.These data are stored into range, and range is related for targetedly storage Data.
Log storage unit was divided into a series of fixed points by one day by the period above-mentioned, and each to put each distribution One queue.W values are calculated for the data that should be stored into Each point in time, by the related data met all into queue, not on the same day The data at same time point can be into the same queue.It is not deposited if w is 1.W indicates whether to belong to extremely, and it is 0 or 1, w to be worth Data for 1 are abnormal datas, can ensure that data of the present invention are not influenced by abnormal data if not depositing, and are helped long-term automatic Study ensures that index is accurate for a long time.The calculation formula of w is as follows:
W=pre>baseline&&(pre-Cpre-Csqrt*p)
In above formula, pre is the average response time of interface current point in time, and baseline is Whole Response time reference line, Baseline=Apre+m*Asqrt.
Wherein, Apre is the mean value for taking daylong data to calculate, and Asqrt is that daylong data is taken to calculate Variance, Apre and Asqrt that baseline is drawn according to daylong data determine, and developing programs for m is traditionally arranged to be 2. oneself is adjusted according to own situation.M is probability factor, can be false due to considering the data of historical time point If current data fit is just distributed very much, so with reference to the data calculation formula being just distributed very much, different probability correspond to it is different because Sub- size.Present invention acquiescence takes probability 1% to be used as benchmark.
Cpre is the mean value that the data in range are calculated.Csqrt is the variance that the data in range are calculated. When calculating Cpre, the range taken is the corresponding point of current slot and several points (being usually 7) of front, then takes these Time point corresponds to several data (being typically also 7) in queue, and it is many a normal to ensure that each period point has in this way Data are as data from the sample survey.The Cpre and Csqrt for calculating these data can obtain average behavior and the side of current slot Difference.
Data calculate and pushing module includes Data Computation Unit, index push unit and generation Alarm Unit.
Wherein, Data Computation Unit can automatically analyze out how many daily record altogether by elk interfaces, automatically according to The data type of nginx, apache, tomcat identify journal format and are belonging respectively to which system.For each system list It is only to calculate script at a data.Script is calculated at interval of the period, collection of log data is carried out, collects the access of tap mouth These data are sent to access data collection module and are handled and stored, then according to algorithm by amount and response total duration Calculate current y values.
Y=w* (pre-Cpre)/Cpre/100*counts* ((pre>outline)+1)
Wherein, y is final result index value, and counts is the visit capacity of interface current point in time.In view of performance is asked Topic is generally concentrated in small part interface, and the interface for only calculating this part is substantially just enough, can save calculation amount in this way, subtract Few unnecessary storage.Will be in a period, tapping mouth, respectively the response time calculates summation, by the inverted order ranking of summation, TopNumber before taking.Outline is user experience performance boundary, and when performance is more than certain value, user experience can be non- It is often poor, it can substantially cause flow to be lost in very much, so needing that this boundary value is arranged, be weighted to y.General website user Boundary in 1-10 seconds ranges.It can give tacit consent to 2 seconds.The poor ratio of pre and Cpre, which is multiplied by counts, can take into account monitoring performance The situation that poor small but visit capacity is big and poor performance is big but visit capacity is small.
The y values that index push unit is used to calculate, are placed into websocket message servers, in real time by it Broadcast give each browser websocket receiving terminals receive.Here the technology of Websocket is all square with java and php etc. Just it realizes.
Alarm Unit is generated, the sum of y values for calculating tap mouth Ally, Ally indicates overall performance index, when Ally is big General alarm is just will produce when some baseline value.Baseline value is arranged as required to, such as 100, is more than 100 and is just triggered announcement It is alert.
Ally=Σ Yn* (1+coverage)
Wherein, Yn is the y values that ranking is n, and coverage is unnatural proportions, refers to that w is abnormal connects in topNumber Mouth accounting, the higher explanation of accounting, problem scope are wide.At this moment it is necessary to give mono- weight addition of Ally.The range of n is 1- TopNumber, topNumber are ranking value range.
Real-time reception and display module include real-time reception unit and curve display unit.
Real-time reception unit is used to obtain the data of websocket message servers, since browser needs obtain in real time The data of websocket message servers, this unit are taken to realize the receiving terminal of websocket using js.Message itself according to System is grouped, and designated packet can obtain y the and Ally details of correspondence system.The data received give curve displaying To be shown.
After curve display unit receives the data of real-time reception unit transmission, curve is shown on the page.Due to possessing The detailed data of the y of Ally and tap mouth, so live page can assist the tap mouth of a part based on the curve of Ally Y curves be shown.In Fig. 2 curve graphs, the line of bottom is exactly core index curve, and the protrusion that circle encloses is exactly to accuse Displaying when alert.Ally shows that the achievement data of each interface is accumulated in together displaying spline curves with spline curves, can be with More intuitively see the abnormal performance of total system.Fig. 3 curve graphs are the y curve graphs of a certain interface, and the line of bottom is y values Figure, the protrusion that circle encloses is exactly abnormal data value.
Based on above system, the present invention also provides the method for carrying out performance monitoring based on access log, specifically include as Lower step:
Step 1, data collection is accessed
Step 1-1 is collected and is accessed data logging, and is formatted processing to daily record, and formatting procedure is produced with reference to daily record The function of being realized in raw unit.
Step 1-2, passing through step 1-1, treated that formatted log stores, and is divided by the period above-mentioned A series of fixed points, and be their respectively one queues of distribution.By the related data calculated every time all into queue, if w is 1 It does not deposit then.Calculation formula of the w computational methods with reference to w in aforementioned log storage unit.
Step 2, data are calculated and are pushed
Step 2-1 automatically analyzes out how many daily record altogether, and identifies journal format and be belonging respectively to which system, It is individually created a data for each system and calculates script, calculates script and be used to carry out collection of log data and push, then root Current y values are calculated according to algorithm.Y values are calculated with reference to y calculation formula in Data Computation Unit.
Step 2-2, the y values that will be calculated, is placed into websocket message servers, broadcasted in real time by it to Each browser websocket receiving terminals receive.
Step 2-3 calculates the sum of the y values of tap mouth Ally, and when Ally has value or big Mr. Yu's value just will produce generally Alarm.As soon as can be defined here more than a baseline value, such as 100 triggering alarms.
Step 3, real-time reception and displaying
Step 3-1 is obtained the data of websocket message servers, the receiving terminal of websocket is realized using js. Message itself is grouped according to system, and designated packet can obtain the y details of correspondence system.The data received, give song Line shows to be shown.
Step 3-2 shows curve after receiving the data that real-time reception unit transmits on the page.With the curve of Ally Based on, assist the y curves of the tap mouth of a part to be shown.As shown in Figure 2 and Figure 3.
The technical means disclosed in the embodiments of the present invention is not limited only to the technological means disclosed in the above embodiment, further includes By the above technical characteristic arbitrarily the formed technical solution of combination.It should be pointed out that for those skilled in the art For, various improvements and modifications may be made without departing from the principle of the present invention, these improvements and modifications are also considered as Protection scope of the present invention.

Claims (7)

1. a kind of system carrying out performance monitoring based on access log, which is characterized in that including:Access data collection module, number According to calculating and pushing module, real-time reception and display module;Data collection module is accessed for generating, converting and storing unified lattice Log Shipping simultaneously is used to analyze daily record by the daily record of formula to data calculating and pushing module, data calculating and pushing module It calculates, and result is pushed to real-time reception and display module, real-time reception and display module are calculated for obtaining data in real time To pushing module analysis result and carry out related displaying.
2. the system according to claim 1 for carrying out performance monitoring based on access log, it is characterised in that:
The access data collection module includes log producing unit and log storage unit, and log producing unit is used for daily record It is formatted processing, log storage unit calculates w values for storing daily record, for the daily record for needing to store, if w is 1 not It deposits, the calculation formula of w is as follows:
W=pre>baseline&&(pre-Cpre-Csqrt*p)
In above formula, pre is the average response time of interface current point in time, and baseline is Whole Response time reference line, Baseline=Apre+m*Asqrt;Apre is the mean value for taking daylong data to calculate, and Asqrt is to take daylong number According to the variance calculated, m is probability factor;Cpre is the mean value calculated according to storage data, and Csqrt is according to storage The variance that data are calculated;
Data calculate and pushing module includes Data Computation Unit, index push unit and generation Alarm Unit, the data meter Unit is calculated for identifying journal format according to data type and being belonging respectively to which system, one is individually created for each system A data calculate script for carrying out collection of log data, and current y values are then calculated according to following algorithm:
Y=w* (pre-Cpre)/Cpre/100*counts* ((pre>outline)+1)
Wherein, counts is the visit capacity of interface current point in time, and Outline is user experience performance boundary;
Index push unit is used to the y values calculated being pushed to each browser and receive;
It generates Alarm Unit and is used to calculate the sum of the y values of tap mouth Ally, alarm is generated when Ally is more than some baseline value, Ally calculation formula are as follows:
Ally=Σ Yn* (1+coverage)
Wherein, Yn is the y values that ranking is n, and coverage is unnatural proportions, and the value range of n is 1-topNumber, TopNumber is ranking value range;
Real-time reception and display module include real-time reception unit and curve display unit, and real-time reception unit is for obtaining number According to the data received give curve displaying to be shown;Curve display unit for showing curve on the page, including The y curves of Ally curves and at least partly interface.
3. the system according to claim 2 for carrying out performance monitoring based on access log, which is characterized in that the daily record production Formatting processing procedure in raw unit includes:
1. the method for using canonical first judges whether to have in url " unified entrance ", if in the presence of if by question mark "" remove;
2. the method for then using canonical again will be similar to that " exception " of unified entrance identifies, they be substituted for sky;
3. then, changing number "/[0-9]+/ " into $ symbols successively, the content "/#.*/" after anchor note changes sky into;
4. again unify by question mark "" make " & " symbol into;
5. finally according to demand, being derived from the number of parameters of oneself needs, remove remaining parameter.
4. the system according to claim 2 for carrying out performance monitoring based on access log, it is characterised in that:The daily record is deposited It was divided into a series of fixed points according to the time period by one day when storage unit stores data, and each to put one storage team of each distribution Row.
5. the system according to claim 2 for carrying out performance monitoring based on access log, which is characterized in that calculate Cpre When, the corresponding point of current slot and several several points of front are chosen, then to take these time points to correspond to several in queue Data.
6. the system according to claim 2 for carrying out performance monitoring based on access log, which is characterized in that the index pushes away It send unit to place data into websocket message servers, is broadcasted by it and connect to each browser websocket in real time Receiving end receives.
7. the method for carrying out performance monitoring based on access log, which is characterized in that specifically comprise the following steps:
Step 1, data collection is accessed
Step 1-1 is collected and is accessed data logging, and is formatted processing to daily record;
Step 1-2, passing through step 1-1, treated that formatted log stores, and w is calculated for the daily record for needing to store Value is not deposited if w is 1, and the calculation formula of w is as follows:
W=pre>baseline&&(pre-Cpre-Csqrt*p)
In above formula, pre is the average response time of interface current point in time, and baseline is Whole Response time reference line, Baseline=Apre+m*Asqrt;Apre is the mean value for taking daylong data to calculate, and Asqrt is to take daylong number According to the variance calculated, m is probability factor;Cpre is the mean value calculated according to storage data, and Csqrt is according to storage The variance that data are calculated;
Step 2, data are calculated and are pushed
Step 2-1 automatically analyzes out how many daily record altogether, and identifies journal format and be belonging respectively to which system, is every A system is individually created a data and calculates script, calculates script for carrying out collection of log data, is calculated according to following algorithm Current y values:
Y=w* (pre-Cpre)/Cpre/100*counts* ((pre>outline)+1)
Wherein, counts is the visit capacity of interface current point in time, and Outline is user experience performance boundary;
The y values calculated are pushed to each browser and received by step 2-2;
Step 2-3 calculates the sum of the y values of tap mouth Ally, alarm is generated when Ally is more than some baseline value;Ally calculates public Formula is as follows:
Ally=Σ Yn* (1+coverage)
Wherein, Yn is the y values that ranking is n, and coverage is unnatural proportions, and the value range of n is 1-topNumber, TopNumber is ranking value range;
Step 3, real-time reception and displaying
Step 3-1 obtains data;
Step 3-2 after receiving data, shows curve on the page, includes the y curves of Ally curves and at least partly interface.
CN201810150710.0A 2018-02-13 2018-02-13 System and method for performing performance monitoring based on access log Active CN108509309B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201810150710.0A CN108509309B (en) 2018-02-13 2018-02-13 System and method for performing performance monitoring based on access log

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201810150710.0A CN108509309B (en) 2018-02-13 2018-02-13 System and method for performing performance monitoring based on access log

Publications (2)

Publication Number Publication Date
CN108509309A true CN108509309A (en) 2018-09-07
CN108509309B CN108509309B (en) 2021-09-28

Family

ID=63375045

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201810150710.0A Active CN108509309B (en) 2018-02-13 2018-02-13 System and method for performing performance monitoring based on access log

Country Status (1)

Country Link
CN (1) CN108509309B (en)

Cited By (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109857623A (en) * 2018-12-13 2019-06-07 深圳壹账通智能科技有限公司 Service performance analysis method, device and electronic equipment based on journal file
CN109885453A (en) * 2019-02-18 2019-06-14 国家计算机网络与信息安全管理中心 Big data platform monitoring system based on flow data processing
CN110442503A (en) * 2019-07-29 2019-11-12 深圳数位传媒科技有限公司 A kind of alarm method and device using log index
CN110493053A (en) * 2019-08-22 2019-11-22 北京首都在线科技股份有限公司 Merge monitoring method, device, terminal and the storage medium of content distributing network
CN110851234A (en) * 2019-10-25 2020-02-28 中信梧桐港供应链管理有限公司 Log processing method and device based on docker container
CN111082985A (en) * 2019-12-16 2020-04-28 厦门大学附属第一医院 API (application program interface) monitoring method based on open platform
CN112579418A (en) * 2020-12-25 2021-03-30 泰康保险集团股份有限公司 Method, device, equipment and computer readable medium for identifying access log
CN113282471A (en) * 2021-05-17 2021-08-20 多点(深圳)数字科技有限公司 Equipment performance testing method and device and terminal equipment
CN114244557A (en) * 2021-11-09 2022-03-25 浪潮通用软件有限公司 Development operation log isolation method and system based on user characteristics

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104598348A (en) * 2015-02-28 2015-05-06 南京途牛科技有限公司 Method and system for remotely analyzing performances of external system interfaces in real time
US9251035B1 (en) * 2010-07-19 2016-02-02 Soasta, Inc. Load test charts with standard deviation and percentile statistics
CN106294091A (en) * 2016-08-11 2017-01-04 福建富士通信息软件有限公司 A kind of without intrusive mood daily record interception method for analyzing performance and system
CN106789189A (en) * 2016-12-05 2017-05-31 深圳市彬讯科技有限公司 A kind of SOA services monitor in real time is reported and performance metric system
CN107135119A (en) * 2017-04-18 2017-09-05 国网福建省电力有限公司 A kind of service response tracking and interface status monitor development system
CN107329877A (en) * 2017-06-29 2017-11-07 南京途牛科技有限公司 Air ticket business monitoring execution system and method

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9251035B1 (en) * 2010-07-19 2016-02-02 Soasta, Inc. Load test charts with standard deviation and percentile statistics
CN104598348A (en) * 2015-02-28 2015-05-06 南京途牛科技有限公司 Method and system for remotely analyzing performances of external system interfaces in real time
CN106294091A (en) * 2016-08-11 2017-01-04 福建富士通信息软件有限公司 A kind of without intrusive mood daily record interception method for analyzing performance and system
CN106789189A (en) * 2016-12-05 2017-05-31 深圳市彬讯科技有限公司 A kind of SOA services monitor in real time is reported and performance metric system
CN107135119A (en) * 2017-04-18 2017-09-05 国网福建省电力有限公司 A kind of service response tracking and interface status monitor development system
CN107329877A (en) * 2017-06-29 2017-11-07 南京途牛科技有限公司 Air ticket business monitoring execution system and method

Cited By (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109857623A (en) * 2018-12-13 2019-06-07 深圳壹账通智能科技有限公司 Service performance analysis method, device and electronic equipment based on journal file
CN109885453A (en) * 2019-02-18 2019-06-14 国家计算机网络与信息安全管理中心 Big data platform monitoring system based on flow data processing
CN109885453B (en) * 2019-02-18 2023-02-28 国家计算机网络与信息安全管理中心 Big data platform monitoring system based on stream data processing
CN110442503A (en) * 2019-07-29 2019-11-12 深圳数位传媒科技有限公司 A kind of alarm method and device using log index
CN110493053A (en) * 2019-08-22 2019-11-22 北京首都在线科技股份有限公司 Merge monitoring method, device, terminal and the storage medium of content distributing network
CN110851234A (en) * 2019-10-25 2020-02-28 中信梧桐港供应链管理有限公司 Log processing method and device based on docker container
CN111082985A (en) * 2019-12-16 2020-04-28 厦门大学附属第一医院 API (application program interface) monitoring method based on open platform
CN112579418A (en) * 2020-12-25 2021-03-30 泰康保险集团股份有限公司 Method, device, equipment and computer readable medium for identifying access log
CN113282471A (en) * 2021-05-17 2021-08-20 多点(深圳)数字科技有限公司 Equipment performance testing method and device and terminal equipment
CN114244557A (en) * 2021-11-09 2022-03-25 浪潮通用软件有限公司 Development operation log isolation method and system based on user characteristics
CN114244557B (en) * 2021-11-09 2023-06-16 浪潮通用软件有限公司 Development operation log isolation method and system based on user characteristics

Also Published As

Publication number Publication date
CN108509309B (en) 2021-09-28

Similar Documents

Publication Publication Date Title
CN108509309A (en) A kind of system and method carrying out performance monitoring based on access log
US11645293B2 (en) Anomaly detection in big data time series analysis
US20200258005A1 (en) Unsupervised method for classifying seasonal patterns
US10970891B2 (en) Systems and methods for detecting and accommodating state changes in modelling
US10248528B2 (en) System monitoring method and apparatus
US20190228022A1 (en) System for detecting and characterizing seasons
US10699211B2 (en) Supervised method for classifying seasonal patterns
CN105490854B (en) Real-time logs collection method, system and application server cluster
CN109684052B (en) Transaction analysis method, device, equipment and storage medium
CN111294217A (en) Alarm analysis method, device, system and storage medium
Wu et al. Variation across scales: Measurement fidelity under twitter data sampling
JPWO2007052327A1 (en) Performance abnormality analyzer, method and program, and analysis result display method of performance abnormality analyzer
CN111767951A (en) Method for discovering abnormal data by applying isolated forest algorithm in residential electricity safety analysis
CN113746703B (en) Abnormal link monitoring method, system and device
CN115022153B (en) Fault root cause analysis method, device, equipment and storage medium
CA3140769A1 (en) Method and system for positioning fault root cause of service system
CN116804957A (en) System monitoring method and device
CN111028862A (en) Method, apparatus, computer device and storage medium for processing voice data
RU2654167C2 (en) Method for detecting hidden interactions on the internet of things
CN116887340B (en) Real-time pushing system for short message status report
CA3083461A1 (en) User interface system incorporating analysis of user interaction classification
CN116910117A (en) Multi-dimensional high-calculation-force microclimate sensor data analysis system and method
CN116149947A (en) Quality evaluation method and device for data model, electronic equipment and storage medium
CN112800089B (en) Intermediate data storage level adjusting method, storage medium and computer equipment
US10812351B2 (en) System and method for cyber sensing for power outage detection

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant