Communication network information system method for detecting abnormality based on Distributed Services daily record
Technical field
The present invention relates to communication network information abnormality detection technical field, more particularly to based on Distributed Services daily record
Communication network information system method for detecting abnormality.
Background technology
With the development of network, service when most of network service systems are required for offer 7*24 small, system occurs different
Chang Shi, if test problems and can not in time repair, it will cause huge economic loss.When system exception, developer
It would generally find and handle by log analysis and is abnormal.Diary record system operation information, it is that system exception analysis is most normal
One of method seen.But since system scale is excessive, system log is distributed on a different server, and server
Distribution is extensive and scattered, when developer has found that manual analyzing daily record will expend substantial amounts of to detect abnormal method
Between.In order to reduce the workload of developer, it is meaningful to study the method for detecting abnormality based on automatic log analysis.
For the abnormality detection of Distributed Services, center of gravity is placed on controlling stream graph of the exploitation based on time weighting by this patent
(TCFG) on model.The model development firstly generates controlling stream graph model (CFG), then partitioning boundary optimization structure, afterwards using K
Mean algorithm calculates time weighting.Finally by online log stream compared with the TCFG models excavated, realize that difference is examined
It is disconnected.
To understand the state of development of the prior art, retrieval is carried out to existing paper and patent, has compared and analyzes, has been screened
Go out the higher technical information of following degree related to the present invention:
Technical solution 1:Patent No. CN106209496A's《A kind of communication abnormality detection device and method》Patent is related to
A kind of method of abnormal detector is mainly completed by three steps of device:First, communication unit.Pass through the logical of communication unit
Believe that interface receives multiple data frames.The unit need to include at least one communication interface and at least one communication interface connects including first
Mouthful and second interface to meet the needs of different piece abnormality detection.Second, detection unit includes technical unit and time span
Determination unit.Counting unit is used for data of the mark received to the communication unit for frame end position to labeled as a frame
Data amount check until the data of start bit is counted.Count value of the time length determination unit for according to counting unit is true
Surely the time span between two frame data received.3rd, determination unit, by the time span detected and default time zone
Between compared, when the time span is in the time interval, the communication of the communication unit is normal, otherwise described logical
Believe the communication abnormality of unit and management server is sent to by transmitting element.
Technical solution 2:Patent No. CN106713012A's《A kind of detection method and device of communication abnormality》Patent relates to
And a kind of detection method and device of ATM communication abnormalities, mainly completed by four steps of device:First, according between default transmission
It is interposed between inside ATM and sends heartbeat packet;Second, the real-time Communication for Power load parameter of the corresponding heartbeat response bag of the heartbeat packet is detected,
The traffic load parameter is retardation rate or packet loss;3rd, working as inside the ATM, is determined according to the traffic load parameter
Preceding communications status, the communications status take Pyatyi grading system;4th, export the definite Current Communications Status.The invention
Have the advantages that can detect during no business datum abnormal.
Technical solution 3:Patent No. CN105323017A's《Communication abnormality detection device, communication abnormality detection method with
And program》Patent is related to a kind of communication to being detected in master device and multiple exceptions from the communication carried out between device
Abnormal detector is mainly completed by three steps of device:First, the test section of communication abnormality detection device observes main dress always
It puts and the communication voltage from the order wire between device;Second, notification unit is when communication voltage is less than setting value, except notice should
Outside alarm, ID of the identification as the slave device of communication object is informed about to master device;3rd, master device shows different in display unit
It often shows the maintenance warning of privileged site, and is shown the display unit by the ID slave devices determined, protected to show
Support advance notice.
Technical solution 1 employs a kind of by communication unit, detection unit, these three units of determination unit communicate
The method of abnormality detection, first, communication unit, for receiving multiple data frames;Second, detection unit is described logical for detecting
Believe the time span between the adjacent data frames in multiple data frames that unit receives;3rd, determination unit, for according to detection
The time span that unit detects is compared with default time interval, when the time span is in the time interval
When, the communication of the communication unit is normal, otherwise the communication abnormality of the communication unit.But the abnormality detection be merely able to by
Extremely detection is made caused by time, scope applicatory is also limited.
Technical solution 2 is employed to be detected during no business by sending heartbeat packet and detecting the real-time Communication for Power of its response bag
Mode.Heartbeat packet is sent according to prefixed time interval during no business and detects the real-time Communication for Power of its corresponding heartbeat response bag
Load parameter.Traffic load parameter can reflect the real-time communications status of ATM.The display of state uses grading system, has respectively just
Often, slight congestion, moderate congestion, heavy congestion, Lungs from Non-Heart-Beating this five states.By by traffic load parameter with more than five shapes
The requirement parameter of state, which is compared, would know that Current Communications Status.Finally by communications status output module by the communications status
Output.This method is simply effective, but canonical form is single, also not comprehensive enough with reference to factor, for the scale of the environment used
And condition is also conditional.
Technical solution 3 employs the mode of master and slave devices communication check in a communications system.Detection unit to master device with
At least one communication voltage between device is detected, i.e. rectangular wave buck/boost between reading master and slave devices in order wire
Value detects voltage;Notification unit is detecting that communication voltage is below setting voltage or expects that communication voltage can become
When smaller than setting voltage, in addition to the warning message, master device is informed about to determine the communication object of the communication
Identification information;Master device possesses communication unit, for being notified of warning to the slave device determined by identification information
Information;Possess display unit from device, be notified be notified of warning message when, display be notified of warning message.This side
Method be capable of the decline failure to master device and from the communication voltage in the order wire between device always generation be detected or
Prediction, but detectable failure is only as caused by voltage, functionality is more single.
The content of the invention
For the above situation, to overcome the defect of the prior art, the present invention's is designed to provide based on Distributed Services
The communication network information system method for detecting abnormality of daily record by excavating the service reference in network, finds automatic abnormality detection
When the case that compares, can automatically compare real-time logs stream and exploitation template, achieve the purpose that detection is abnormal.
Its technical solution solved is to detect including daily record parsing, generation TCFG and automatically abnormal three steps, first,
By parsing daily record, mapping and the template of original log and template are obtained, when mapping daily record template, record log template rope
To draw, then timestamp and daily record template generate TCFG, finally by online log stream compared with the TCFG models excavated,
Complete differential diagnostic, which is characterized in that described generation three steps of below TCFG,
S1, CFG side are excavated, in CFG or TCFG, node on behalf daily record template;For node there are one subset, one directly sub
Collection, a time weighting collection;Subset is one group of subsequent node in time window;Direct subset is the direct successor of node
Set;Time weighting collection is time interval set of the node directly between child node;It is the Candidate Set of side right weight;
Wherein there are two key point, one is time t, it is selected across the value of time window;The long time can be with too
More noises, and the short time can miss some correct descendant nodes;On the other hand how to calculate the frequency of child node;
Rule setting is that a child node is only calculated in a time window;
When calculating the frequency of child node, filtering rule is used to determine the direct subsequent node of node;Filtering rule 1
Be intended to based on the frequency of child node come noise filtering;Formula is described as follows;
TnRepresent every template node;N is the occurrence number of daily record every template node;Represent child node, MiIt is the generation of child node
Number;The author sets the filtering threshold of a child node as θ;If It will be by from sub- concentration filter and deleting
It removes;
Filter rules 2 are intended to select direct subsequent node according to the frequency of direct child node;Formula describes such as
Under;
Represent direct child node, QiIt is the frequency of direct child node;The author sets the filtering threshold of direct subset
η;If It will be by from direct sub- concentration filter and deletion;
Afterwards, formula (3) can be used to obtain direct subsequent node;
Ss=Sd∩Sc (3)
Ss,Sd,ScDirect subsequent node collection, subset and direct subset are represented respectively;
For each every template node, once direct subsequent node is calculated, it is confirmed that the side of node;Then, Suo Youjie
Point and its side can form CFG;
S2, conversion stream boundary segmentation, determines whether the time interval between adjacent node is stablized by calculating variance;It is public
Formula (4) is as follows;
When the variance of time interval between two adjacent nodes is more than threshold value, just delete between the two adjacent nodes
Side;Threshold value beta should be very big at this time;
Afterwards check CFG in whether cycling;Method is the variance for the variance for calculating all sides for forming Xun Huan, and deletes
The side of variance maximum;
S3, side time weighting calculate, and the time weighting on each side is represented between two daily record every template nodes in TCFG
Perform the time;Time weighting can be used for detection abnormal, using the time weighting algorithm based on K averages, input as time interval collection D
={ t1,t2,……,tm, cluster numbers K=3, output is time weighting;
The distribution of the time interval list on side is concentrated very much;Most of elements concentrate on a field, and other element is then
It is dispersed in other areas;Therefore time interval list is divided into 3 groups by this patent using K mean algorithms;The most group of quantity is adopted
It receives;Maximum element is time weighting in the group;
The basis of k mean algorithms is the quadratic sum of minimal error;Cost function is:
Wherein, k represents that cluster divides k groups, ciRepresent i-th group of element set, also referred to as x is ciIn arbitrary element, refer to one
A time interval,Represent cluster ciAverage value, E is variance.
Due to the use of above technical scheme, the present invention has the following advantages that compared with prior art;
The technology is studied automatic method for detecting abnormality, it is proposed that be based on based on network service benchmark is excavated
The automatic method for detecting abnormality of TCFG:Original log is analyzed first to generate the template with timestamp;Then, it is every by analyzing
The daily record of a service generates TCFG (time weight control flow chart), and TCFG is responsible for turning for the system that record is run in service
Change state;In addition, according to comparison TCFG, real-time logs throat floater can be diagnosed;Assessment shows the automatic abnormal inspection based on TCFG
Survey method has good accurate rate and recall rate, in true environment, averagely there is 80% accurate rate and 65% recall rate.
Description of the drawings
Fig. 1 is the flow chart of the communication network information system method for detecting abnormality the present invention is based on Distributed Services daily record.
Fig. 2 is the Hadoop days of the communication network information system method for detecting abnormality the present invention is based on Distributed Services daily record
The TCFG of will generation excavates figure.
Fig. 3 is the communication network information system method for detecting abnormality the present invention is based on Distributed Services daily record
The TCFG of Zookeeper daily records generation excavates figure.
Fig. 4 is the TCFG excavations of the communication network information system method for detecting abnormality the present invention is based on Distributed Services daily record
Precision and recall rate figure.
Fig. 5 is the abnormality detection of the communication network information system method for detecting abnormality the present invention is based on Distributed Services daily record
Precision and recall rate figure.
Specific embodiment
For the present invention foregoing and other technology contents, feature and effect, in following cooperation with reference to figures 1 through attached drawing 5
To in the detailed description of embodiment, can clearly present.The structure content being previously mentioned in following embodiment is with specification
Attached drawing is reference.
Embodiment one, the communication network information system method for detecting abnormality based on Distributed Services daily record, including daily record solution
Analysis, generation TCFG and abnormal three steps of automatic detection, first, by parse daily record (daily record parsing be using existing method,
First, by pretreatment, often row daily record according to space is segmented, and rejects the number of the inside, it then will by the method for cluster
Original log be divided into it is multigroup, finally, to one group of daily record according to its similar extraction daily record masterplate, the mapping of production master and daily record,
Obtain mapping and the template of original log and template, when mapping daily record template, record log template index, timestamp and daily record
Then template generates TCFG, finally by online log stream compared with the TCFG models excavated, complete differential diagnostic,
It is characterized in that, described generation three steps of below TCFG,
S1, CFG side are excavated, in CFG or TCFG, node on behalf daily record template;For node there are one subset, one directly sub
Collection, a time weighting collection;Subset is one group of subsequent node in time window;Direct subset is the direct successor of node
Set;Time weighting collection is time interval set of the node directly between child node;It is the Candidate Set of side right weight;
Wherein there are two key point, one is time t, it is selected across the value of time window;The long time can be with too
More noises, and the short time can miss some correct descendant nodes;On the other hand how to calculate the frequency of child node;
Rule setting is that a child node is only calculated in a time window;
When calculating the frequency of child node, filtering rule is used to determine the direct subsequent node of node;Filtering rule 1
Be intended to based on the frequency of child node come noise filtering;Formula is described as follows;
TnRepresent every template node;N is the occurrence number of daily record every template node;Represent child node, MiIt is the generation of child node
Number;The author sets the filtering threshold of a child node as θ;If It will be by from sub- concentration filter and deleting
It removes;
Filter rules 2 are intended to select direct subsequent node according to the frequency of direct child node;Formula describes such as
Under;
Represent direct child node, QiIt is the frequency of direct child node;The author sets the filtering threshold of direct subset
Value η;If It will be by from direct sub- concentration filter and deletion;
Afterwards, formula (3) can be used to obtain direct subsequent node;
Ss=Sd∩Sc (3)
Ss,Sd,ScDirect subsequent node collection, subset and direct subset are represented respectively;
For each every template node, once direct subsequent node is calculated, it is confirmed that the side of node;Then, Suo Youjie
Point and its side can form CFG;
S2, conversion stream boundary segmentation, determines whether the time interval between adjacent node is stablized by calculating variance;It is public
Formula (4) is as follows;
When the variance of time interval between two adjacent nodes is more than threshold value, just delete between the two adjacent nodes
Side;Threshold value beta should be very big at this time;
Afterwards check CFG in whether cycling;Method is the variance for the variance for calculating all sides for forming Xun Huan, and deletes
The side of variance maximum;
S3, side time weighting calculate, and the time weighting on each side is represented between two daily record every template nodes in TCFG
Perform the time;Time weighting can be used for detection abnormal, using the time weighting algorithm based on K averages, input as time interval collection D
={ t1,t2,……,tm, cluster numbers K=3, output is time weighting;
The distribution of the time interval list on side is concentrated very much;Most of elements concentrate on a field, and other element is then
It is dispersed in other areas;Therefore time interval list is divided into 3 groups by this patent using K mean algorithms;The most group of quantity is adopted
It receives;Maximum element is time weighting in the group;
The basis of k mean algorithms is the quadratic sum of minimal error;Cost function is:
Wherein, k represents that cluster divides k groups, ciRepresent i-th group of element set, also referred to as x is ciIn arbitrary element, refer to one
A time interval,Represent cluster ciAverage value, E is variance.
Present invention tool is in use, this patent is assessed using laboratory environment log data set and real log data set
TCFG is excavated and abnormity diagnosis, 70% data are developed, and 30% data are tested for assessing, and data details are as follows:
Hadoop daily record collection:Laboratory environment is Hadoop2.0 versions, two micro services is run, from one in system
The original log of ten days about 1.6GB by a definite date has been selected in micro services as data set;
Zookeeper daily record collection:Zookeeper systems are based on real world, the daily daily record for exporting about 20G, from being
The original log of eight hours about 8GB by a definite date has been selected in micro services in system as data set;
This patent assesses TCFG excavations and abnormity diagnosis, accuracy and recall rate definition by accuracy and recall rate
It is as follows.
Really really (TP) shows correct case being considered as correctly, and false correct (FP) shows wrong case being considered as correctly,
False error misses (FN) and shows correct case being considered as mistake.
Design parameter sets as shown in table 1.Depth is the depth of blog search tree, and similar threshold value is the similar of daily record template
Threshold value.
1 parameter setting of table
Fig. 2 and Fig. 3 is that have bifurcated and linear sequence from the TCFG of the Web log mining of Hadoop and Zookeeper, figure,
TCFG and abnormality diagnostic efficiency are assessed using precision/recall rate index of standard afterwards, this patent first passes through source generation
Code manually generates a benchmark TCFG structure, some abnormal logs are randomly with the addition of into test set;
It is the abnormity diagnosis side based on the mixing graph model (HGM) with daily record with the algorithm compared with this patent algorithm
Method, in this patent, for partitioning boundary, the execution for each step in ratio task that the interval time between two tasks is set
Time is all much longer, and therefore, the author highlights its advantage using HGM in an experiment;
From fig. 4, it can be seen that the precision of both algorithms is all very high, and recall rate is less than precision.This is because segmentation
Strategy is unable to divided ownership border, in order to filter all noises, the threshold value in boundary segmentation strategy be set it is larger, on the contrary,
If the too small normal side of threshold value can be also filtered, this patent algorithm has recalled 80% in Hadoop, higher than HGM algorithms
75%.Meanwhile this patent algorithm has recalled 75% in Zookeeper, higher than the 65% of HGM algorithms, thus, it will be seen that
Boundary segmentation strategy is better than HGM algorithms, in addition, the superiority of strategy is also be reflected in abnormity diagnosis;As shown in figure 5, this patent
Algorithm has 70% to recall in Hadoop, higher than the 65% of HGM algorithms, in addition, the algorithm is recalled in Zookeeper
65%, higher than the 50% of HGM algorithms.It may be concluded that the time weighting algorithm based on K averages also plays certain work
With.
The above is to combine specific embodiment further description made for the present invention, it is impossible to assert the present invention
Specific implementation is limited only to this;For belonging to the present invention and for those skilled in the technology concerned, based on skill of the present invention
Under the premise of art scheme thinking, the expansion made and operating method, the replacement of data, should all fall the scope of the present invention it
It is interior.