CN102147811A

CN102147811A - System performance analyzing method based on logs and device

Info

Publication number: CN102147811A
Application number: CN 201110069573
Authority: CN
Inventors: 余天舒
Original assignee: Hangzhou H3C Technologies Co Ltd
Current assignee: New H3C Technologies Co Ltd
Priority date: 2011-03-22
Filing date: 2011-03-22
Publication date: 2011-08-10
Anticipated expiration: 2031-03-22
Also published as: CN102147811B

Abstract

The invention provides a system performance analyzing method based on logs and a device. The method comprises the following steps: configuring log matching items for matching the logs in advance and inclusion relations among the log matching items, and generating a result set tree according to the inclusion relations among the log matching items; for each blog, using the log matching items corresponding to nodes in the tree for matching each log one by one, extracting system performance parameters in the log when the matching is successful, further carrying out performance analysis on a system according to the extracted system performance parameters and storing an analyzed result in the nodes, thereby realizing effective tracking on an online operation system.

Description

Log-based system performance analysis method and device

Technical Field

The invention relates to a computer technology, in particular to a system performance analysis method and device based on logs.

Background

It is a very complicated matter to perform performance analysis on a system, especially on an online running system, and it is necessary to analyze a large amount of log information recorded by the system and compare the analysis result with a normal situation, thereby analyzing an abnormal situation of performance change.

At present, the log information is mainly analyzed in a semi-manual mode, a log matching tool is adopted, matched initial logs and matched end logs are found out according to a matching rule of a user, and time-consuming logs are continuously searched in a slicing mode in the range of the initial logs and the end logs, so that the bottleneck of system performance is found out. The method mainly comprises the following steps:

1. according to the feedback of the user, determining which function is time-consuming, and finding the start and end logs of the function.

2. And gradually reducing the range in a step-by-step segmentation mode within the range of the starting log and the ending log of the function, roughly comparing the time consumption of each part with the time consumption of the part when the system normally runs, and finding the part with the largest time consumption increase, thereby positioning the performance bottleneck of the system.

As can be seen from the above steps, the analysis method for log information in the prior art is largely manual analysis, which is time-consuming, complex and likely to omit important information; and because it is done manually, it is difficult to compare large amounts of information with values under normal conditions, and thus obtain a situation of performance variation. Therefore, the currently adopted semi-manual mode is difficult to effectively track the online running system.

Disclosure of Invention

In view of the above, the present invention provides a log-based system performance analysis method, which can effectively track an online running system.

In order to achieve the above object, the present invention provides a log-based system performance analysis method, which pre-configures log matching items, configures inclusion relations among the log matching items, and generates a result set tree according to the inclusion relations among the log matching items; the method comprises the following steps:

A. b, reading a log from the log file, if the reading is successful, taking the first child node of the root node in the result set tree as the current node, then turning to the step B to execute, if the reading is failed, then saving the statistical result of each node in the result set tree, and ending the performance analysis of the system;

B. matching the read log with a log matching item corresponding to the current node, if the matching is successful, extracting system performance parameters in the log, carrying out system performance statistics, and turning to the step A for execution, and if the matching is not successful, directly turning to the step C for execution;

C. and C, taking the next node from the result set tree as a new current node, and turning to the step B for execution.

The log matching items comprise a starting log matching rule and an ending log matching rule;

the system performance parameter is time;

each node in the result set tree includes parameters: a processing record, the processing record comprising: start time, end time;

when the result set tree is generated according to the inclusion relation among the log matching items, further marking the initial log matching rule of the log matching item corresponding to each node in the result set tree as unmatched;

the step B comprises the following steps:

b1, judging whether the initial log matching rules of the log matching items corresponding to the current node are matched, if so, executing the step B3, otherwise, executing the step B2;

b2, matching the initial log matching rule of the log matching item corresponding to the current node with the read log, if the matching is successful, extracting the time in the log, marking the time as the initial time of the record being processed by the current node, marking the initial log matching rule of the log matching item corresponding to the current node as matched, turning to the step A for execution, and otherwise, turning to the step B3 for execution;

b3, matching the end log matching rule of the log matching item corresponding to the current node with the read log, if the matching is successful, extracting the time in the log, marking the time as the end time of the record being processed by the current node, carrying out system time consumption statistics according to the end time and the start time of the record being processed by the current node, marking the start log matching rule of the log matching item corresponding to the current node as unmatched, turning to the step A for execution, otherwise, turning to the step B3 for execution.

Each node in the result set tree further includes parameters: recording the number and total time spent;

when the result set tree is generated according to the inclusion relation among the log matching items, the record number and the total time consumption of each node in the result set tree are further cleared;

in step B3, the method for performing statistics on the system consumed time according to the start time and the end time of the record being processed by the current node includes: calculating the difference between the ending time and the starting time of the record being processed by the current node, and accumulating the difference into the total time consumption of the current node; after performing statistics on the system consumed time according to the start time and the end time of the record being processed by the current node, the step a is further performed: adding 1 to the number of records of the current node, and emptying the records processed by the current node; and when the initial log matching rules of the log matching items corresponding to the current node are marked as unmatched, further marking the initial log matching rules of the log matching items corresponding to all nodes in the subtree taking the current node as the root node as unmatched.

Each node in the result set tree further comprises: recording the most time-consuming; the most time consuming records include: a start time and an end time;

when the result set tree is generated according to the analysis rule, the most time-consuming record of each node in the result set tree is further emptied;

after adding 1 to the number of records of the current node and before clearing the records being processed by the current node, the method further includes: judging whether the record being processed by the current node is the most time-consuming record of the current node according to the starting time and the ending time of the record being processed by the current node, and if so, recording the record being processed by the current node as the most time-consuming record of the current node.

The method for judging whether the record processed by the current node is the most time-consuming record of the current node according to the start time and the end time of the record processed by the current node comprises the following steps: judging whether the starting time and the ending time in the most time-consuming record of the current node are empty or not, if so, recording the record being processed by the current node as the most time-consuming record of the current node, otherwise, comparing the difference value between the ending time and the starting time in the record being processed by the current node with the difference value between the ending time and the starting time in the most time-consuming record of the current node, and if the difference value between the ending time and the starting time in the record being processed by the current node is larger, recording the record being processed by the current node as the most time-consuming record of the current node.

The step C comprises the following steps:

c1, judging whether the initial log matching rules of the log matching items corresponding to the current node are matched, if so, executing the step C2, otherwise, executing the step C3;

c2, judging whether the current node has a child node, if yes, taking the first child node of the current node as a new current node, and turning to the step B, otherwise, turning to the step C4;

c3, judging whether the current node has a right brother node, if so, taking the first right brother node of the current node as a new current node, and turning to the step B, otherwise, turning to the step C4;

c4, judging whether the current node has a father node, if so, turning to the step C5, otherwise, obtaining the next node fails, and turning to the step A to execute;

c5, judging whether the father node of the current node has a right brother node, if so, taking the first right brother node of the father node of the current node as a new current node, and executing the step B, otherwise, setting the father node of the current node as the current node, and executing the step C4.

When the inclusion relation between the log matching items and the log matching items is configured in advance, a mark for judging whether the current statistical result is used as a baseline is further configured;

step A, before the system performance analysis is finished, the method further comprises the following steps: and judging whether the current statistical result is required to be used as the baseline according to the set mark of whether the current statistical result is used as the baseline, and if so, generating the baseline according to the current statistical result of each node in the result set tree.

The invention also provides a system performance analysis device based on the log, which can effectively track the online running system. The device includes: the device comprises a configuration unit, a log acquisition unit, a matching unit, a node acquisition unit and a result storage unit;

the configuration unit is used for configuring log matching items in advance, configuring the inclusion relationship among the log matching items and generating a result set tree according to the inclusion relationship among the configured log matching items;

the log acquiring unit is used for reading a log from the log file when receiving the log acquiring notification sent by the matching unit, and sending the read log to the matching unit if the log is successfully read; if the reading fails, the result storage unit is informed to store the statistical result of the performance analysis of the system;

the matching unit is used for informing the log obtaining unit to obtain the log when the system performance analysis is started; the node is used for taking the first child node of the root node in the result set tree as the current node after receiving the log sent by the log obtaining unit; the log matching item is used for matching the log matching item corresponding to the current node with the received log, if the matching is successful, the system performance parameters in the log are extracted, the system performance statistics is carried out, the log obtaining unit is informed to obtain the log, and if the matching is failed, the node obtaining unit is informed to obtain the next node; the node obtaining unit is used for obtaining a next node sent by the node obtaining unit; the log obtaining unit is used for obtaining the log after receiving a notice that the obtaining of the next node from the node obtaining unit fails;

the node obtaining unit is used for obtaining the next node from the result set tree after receiving the notification of obtaining the next node sent by the matching unit, if the obtaining is successful, the obtained next node is sent to the matching unit, and if the obtaining is failed, the matching unit is notified that the obtaining of the next node is failed;

and the result storage unit stores the statistical result of each node in the result set tree after receiving the notification of the log acquisition unit, and the system performance analysis is finished.

the system performance parameter is time;

when the configuration unit generates a result set tree according to the inclusion relationship among the log matching items, further marking the initial log matching rule of the log matching item corresponding to each node in the result set tree as unmatched;

when the matching unit matches the log matching item corresponding to the current node with the received log, the matching unit is configured to: judging whether the initial log matching rule of the log matching item corresponding to the current node is matched, if so, matching the end log matching rule of the log matching item corresponding to the current node with the read log, extracting the system performance parameter in the log under the condition of successful matching, marking the system performance parameter as the end time of the record being processed by the current node, carrying out system time consumption statistics according to the initial time and the end time of the record being processed by the current node, marking the initial log matching rule of the log matching item corresponding to the current node as unmatched, and informing a log acquisition unit of acquiring the log; under the condition of failed matching, informing a node acquisition unit to acquire a next node;

otherwise, matching the initial log matching rule of the log matching item corresponding to the current node with the read log, extracting the time in the log under the condition of successful matching, marking the time as the initial time of the record being processed by the current node, marking the initial log matching rule of the log matching item corresponding to the current node as matched, and informing a log acquisition unit of acquiring the log; in the case of a failure in matching, the node acquisition unit is notified to acquire the next node.

when the result set tree generating unit generates a result set tree according to the inclusion relation among the log matching items, the record number and the total time consumption of each node in the result set tree are further cleared;

the matching unit is used for performing system time consumption statistics according to the start time and the end time of the record being processed by the current node, and is used for: calculating the difference between the ending time and the starting time of the record being processed by the current node, and accumulating the difference into the total time consumption of the current node; after the system time consumption statistics is carried out according to the starting time and the ending time of the record being processed by the current node, the log reading unit is informed to further read the log: adding 1 to the number of records of the current node, and emptying the records processed by the current node; when the initial log matching rules of the log matching items corresponding to the current node are marked as unmatched, the initial log matching rules of the log matching items corresponding to all nodes in the subtree taking the current node as the root node are further marked as unmatched.

when the result set tree generating unit generates a result set tree according to the inclusion relation among the log matching items, the most time-consuming record of each node in the result set tree is further emptied;

the matching unit, after adding 1 to the number of records of the current node and before clearing the record being processed by the current node, is further configured to: and judging whether the record being processed by the current node is the most energy-consuming record of the current node according to the starting time and the ending time of the record being processed by the current node, and if so, recording the record being processed by the current node as the most time-consuming record of the current node.

The matching unit, when judging whether the record being processed by the current node is the most energy-consuming record of the current node according to the start time and the end time of the record being processed by the current node, is configured to: judging whether the starting time and the ending time in the most time-consuming record of the current node are empty or not, if so, recording the record being processed by the current node as the most time-consuming record of the current node, otherwise, comparing the difference value between the ending time and the starting time in the record being processed by the current node with the difference value between the ending time and the starting time in the most time-consuming record of the current node, and if the difference value between the ending time and the starting time in the record being processed by the current node is larger, recording the record being processed by the current node as the most time-consuming record of the current node.

The node obtaining unit is configured to, when obtaining a next node: judging whether the initial log matching rule of the log matching item corresponding to the current node is matched, if so, taking the first child node of the current node as the next acquired node when the current node has child nodes, otherwise, taking the first right brother node of the current node as the next acquired node when the current node has right brother nodes;

if the current node has neither child node nor right brother node, circularly judging whether the current node has a father node or not, and if the current node has no father node, acquiring the next node fails; otherwise, when there is a right brother node, the first right brother node of the father node of the current node is used as the next node to be obtained, and the cycle judgment is stopped, otherwise, the current node is set as the current node, and the current node is circularly judged again whether there is a father node until the father node of the current node has the right brother node, and the next node is successfully obtained, or the current node has no father node, and the next node is failed to be obtained.

When the configuration unit configures the inclusion relationship between the log matching items in advance, the configuration unit further configures a mark for judging whether the current statistical result is used as a baseline;

after the result saving unit saves the current statistical result of each node in the result set tree into the result set file, the result saving unit is further used for: and judging whether the current statistical result is required to be used as the baseline according to the set mark of whether the current statistical result is used as the baseline, and if so, generating a baseline file according to the current statistical result of each node in the result set tree.

According to the technical scheme, the log-based system performance analysis method generates a result set tree according to the inclusion relationship between the log matching items for matching the log and the log matching items by pre-configuring the inclusion relationship between the log matching items; and for each log, matching the log matching items corresponding to the nodes in the tree one by one, extracting system performance parameters in the log when the matching is successful, further performing system performance analysis according to the extracted system performance parameters, and storing an analysis result in the node, thereby realizing effective tracking of the online operating system.

Drawings

FIG. 1 is a flow chart of a log-based system performance analysis method of the present invention;

FIG. 2 is a schematic diagram of a result set tree generated from a parse rule definition file;

FIG. 3 is a flowchart of a log-based system performance analysis method in accordance with a preferred embodiment of the present invention;

FIG. 4 is a flowchart of a method for obtaining a next node in a result set tree in accordance with a preferred embodiment of the present invention;

fig. 5 is a schematic structural diagram of a log-based system performance analysis apparatus according to an embodiment of the present invention.

Detailed Description

In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention will be described in detail below with reference to the accompanying drawings and examples.

The method provided by the invention mainly comprises the following steps: predefining the inclusion relation between the log matching items and the log matching items, and generating a result set tree according to the inclusion relation between the log matching items; when system performance analysis is needed, reading logs one by one from a log file, matching each read log with a log matching item corresponding to each node in a result set tree one by one, if the log is successfully matched with the log matching item corresponding to one node, extracting system performance parameters in the log according to the log matching item successfully matched, carrying out system performance statistics, and if no log matching item capable of matching the log exists, skipping the log and continuously processing the next log until all logs are processed.

First, the log matching items in the technical solution of the present invention are explained.

The log matching item is a rule of matching log pre-configured based on the log content needing statistics, and can be stored in a matching rule definition file. Each log matching item corresponds to a matching rule of a complete log record, including a starting log matching rule and an ending log matching rule of the log. Here, a complete log record includes one or more logs related to the log, for example, when the system starts executing the function a, a start log indicating a start time for starting executing the function a is recorded, and when the system ends executing the function a, an end log indicating an end time for executing the function a is recorded, the start log and the end log are a complete log record related to the execution time of the function a.

The configured log matching item is used for matching the log recorded in the running process of the system, the log which accords with the log matching rule in the log matching item can be matched, and then various system performance parameters recorded in the log matched with the log matching item can be counted and analyzed. The log matching rule in the log matching item is expressed by a regular expression, and the log matching item in the matching rule definition file is explained by taking the starting time and the ending time of a complete log record as performance parameters of a system to be counted:

for example for the following logs:

2010-10-20 17:00:32[INFO][main][rmi::getRegistry]Start RMI registry。

2010-10-20 17:01:00[INFO][main][A::function]Start A。

2010-10-20 17:01:00[INFO][main][A::function]Stop A。

2010-10-20 17:02:32[INFO][main][rmi::getRegistry]Stop RMI registry。

the corresponding matching rule definition file content may be defined as follows:

in the matching rule definition file, the root node is defined as regexDefine and includes a plurality of define nodes. Each define node defines a log match, and the meaning of the attribute of the define node is shown in table one:

attribute name	Means of
		id	Log matching item ID

Watch 1

The define node may comprise a startRegex node and an endRegex node, wherein the startRegex node represents a start log matching rule of the log matching item defined by the define node, and the endRegex node represents an end log matching rule of the log matching item defined by the define node. startRegex and endRegex nodes contain the same attributes: regexString, the meaning of which is shown in Table two:

watch two

In this example, time is taken as a system performance statistic object, so that both the startRegex node and the endRegex node include a timeField node, the timeField node defines the position and format of a timestamp intercepted from a log after a log matching rule in the startRegex node or the endRegex node is successfully matched to the log, and the meaning of the attributes of the timeField node is as follows:

watch III

In the above example of the matching rule definition file, the log matching items are configured for the purpose of obtaining a statistical result of the time duration consumed by the system by taking the start time and the end time of each complete log record as statistical objects of system performance parameters. In fact, for different logs and different performance parameters to be counted in the logs, the child nodes that the define node may include in the corresponding matching rule definition file are different. For example, when cpu resources occupied by the system are taken as statistical objects of system performance parameters, only one Regex node is needed to represent the matching rules of the log matching items defined by the define node, and the log matching items only include one matching rule, and it is not necessary that each log matching item needs the start log matching rule and the end log matching rule to respectively match the start log record and the end log record of the log as in the previous example. Thus, the log matches in the matching rule definition file are configurable based on log differences and statistical system performance parameters.

In addition, in this example, the log matching rule is expressed by using a regular expression, and actually, other matching methods, for example, character string matching, may be used to express the log matching rule.

Next, the inclusion relationship between log matching items in the technical solution of the present invention will be described.

The inclusion relationship between log matching items can be configured in advance and stored in a parsing rule definition file for representing the inclusion relationship between logs corresponding to log matching items, for example, for the logs, in the matching rule definition file, a define node with attribute id being parent1 defines a log matching item parent1 corresponding to "[ INFO ] [ main ] [ rmi:: getRegistry ]", a define node with attribute id being child1 defines a log matching item child1 corresponding to "[ INFO ] [ main ] [ A:: function ]", and the log matching item parent1 includes a log matching item child 1.

In addition, it may also be indicated in the parsing rule definition file whether the current statistical result needs to be stored as a baseline in the baseline file, which is exemplified below.

Suppose that log matching items parent1, parent2, child1, child2 and child3 are defined in a matching rule definition file, and the inclusion relationship among the log matching items is as follows: if parent1 contains the journal matching items child1 and child2, and if the journal matching item parent2 contains the journal matching item child3, the corresponding parsing rule definition file can be defined as follows:

in the parsing rule definition file, the root node is defined as parsefine, and the meaning of the attribute of the parsefine node is shown in table four:

watch four

ParseDefine may contain a plurality of node nodes, each node representing a statistical object corresponding to a log match in the matching rule definition file. The node itself may contain a plurality of node nodes, and represents the inclusion relationship among log matching items, and the number of included layers is not limited. The attribute meaning of the node is shown in table five:

watch five

The above describes the log matching items and the inclusion relationship between the log matching items in the technical solution of the present invention, respectively, and the technical solution of the present invention is described below.

Referring to fig. 1, fig. 1 is a flow chart of a log-based system performance analysis method of the present invention, which includes the following steps:

step 101, pre-configuring log matching items, configuring the inclusion relationship among the log matching items, and generating a result set tree according to the inclusion relationship among the log matching items.

In this step, first, according to the format of the log to be subjected to system performance analysis, the corresponding log matching items and the inclusion relationship between the log matching items are configured in advance and stored in the matching rule definition file and the parsing rule definition file, respectively.

Here, a tree may be generated as a tree storing log statistics according to the inclusion relationship between log matches, and for convenience of description, the tree is referred to as a result set tree. And generating a result set tree according to the relation between the log matching items, wherein particularly, a parseDefine node in the parsing rule definition file is used as a root node of the tree, the nodes with direct inclusion relation are defined as a parent node and a child node, the nodes with parallel relation are defined as sibling nodes, and each node takes the attribute id of the corresponding node as a key word, so that each node corresponds to one log matching item in the matching rule definition file. For example, a tree generated according to the example of the above-described parsing rule definition file is shown in fig. 2, where each node uses the attribute id of its corresponding node as a key. Where parent1 is the first child node of the result set tree; parent1 is the left sibling of parent2 and parent2 is the right sibling of parent 1; parent1 is the parent node of child1 and child 2; child1 is the first child node of parent, child2 is the second child node of parent 1; parent2 is the parent of child3, child2 has no siblings.

In addition, according to the system performance parameter object to be counted in the log, in the result set tree, each node needs to be able to store the corresponding statistical data, and the time in the log is taken as the statistical object, each node needs to include the following parameters:

key words: corresponding to a log matching item ID;

recording number: the number of times that the log matching item corresponding to the node has been completely matched currently;

the total time consumption is as follows: the total time consumption obtained by statistics from the logs which completely match the log matching items corresponding to the node currently is long;

record being processed: the complete log record corresponding to the log matching item for indicating that matching is currently performed shall include a start time, a start log, an end time and an end log;

the most time-consuming recording: the log matching item is used for representing the most time-consuming record in the matched complete log records corresponding to the log matching item, wherein the record comprises a starting time, a starting log, an ending time and an ending log.

And in the log analysis process, continuously updating the current statistical result of each node according to the matching result, and obtaining the final statistical result corresponding to each log matching item after the log analysis is finished.

Step 102, reading a log from a log file.

And 103, judging whether the reading is successful or not, if so, executing the step 104, and otherwise, executing the step 107.

And step 104, matching the log with the log matching items corresponding to each node in the result set tree one by one.

In this step, nodes in the result set tree need to be traversed, the log matching items corresponding to the traversed nodes are matched with the read logs, if the matching is successful, the traversal of the result set tree is stopped, otherwise, the traversal of the result set tree is continued until all the nodes in the result set tree are traversed. The traversal method may be any one of the methods of traversing a tree in the prior art.

In addition, according to different statistical system performance parameters, the number of matching rules in a log matching item is different. Therefore, when log matching is carried out, if only one matching rule exists, the rule is directly used for matching; if there are two matching rules of the starting log matching rule and the ending log matching rule, it needs to first judge whether the starting log matching rule has been matched, if it has been matched, the ending log matching rule is used to match the log, if the starting log matching rule has not been matched, the starting log matching rule is used only to match the log, and the specific method will be given in the following preferred embodiment.

And 105, judging whether the matching with the log matching item corresponding to one node in the result set tree is successful, if so, executing the step 106, otherwise, returning to the step 102 for execution.

And 106, extracting system performance parameters in the log according to the log matching items corresponding to the successfully matched nodes, performing system performance statistics according to the extracted system performance parameters, storing the statistical result in the successfully matched nodes, and turning to the step 102 to continue reading the next log.

Here, since the matching rule is a method using a regular expression, the method of extracting the system performance parameter is the same as the method of extracting the parameter using the regular expression in the related art. Of course, the matching rule of the log may also use other expression modes as long as the matching and the extraction of the system performance parameters can be achieved, which is a problem in implementation.

In this step, if one log matching item includes a start log matching rule and an end log matching rule, only the system performance parameters in the log are extracted and stored when the log is successfully matched with the start log matching rule, and the system performance parameters in the log are firstly extracted and stored when the log is successfully matched with the end log matching rule, and system performance statistics is performed according to the system performance parameters respectively stored when the start log matching rule and the end log matching rule are matched.

And step 107, storing the statistical result in each node in the result set tree, and ending the performance statistics of the system.

If all the logs are processed, the statistical result of each node in the result set tree can be stored in a result set file, and the system performance analysis is directly finished; or continuing to wait until the system has a new log, and continuing to count the new log.

In this step, the statistical result in each node in the result set tree may be saved in the result set file in an xml document manner, for example, the result set file may be represented as follows:

the meaning of the contents of the result set file is explained below as follows:

the root node in the result set is defined as result and comprises one or more records, wherein each record corresponds to one node in the parsing rule definition file and one define node in the matching rule definition file. Specifically, the attribute id of the record node in the result set file, the attribute id of the node in the parsing rule definition file, and the attribute id of the define node in the matching rule definition file are in a one-to-one correspondence relationship. Each record node may contain three types of nodes:

a currentRecord node for indicating the records currently being processed.

And the mostteconsuminggrecording node is used for indicating the most time-consuming record in the log corresponding to the node.

And one or more record nodes corresponding to the inclusion relationship among the log matching items in the analysis rule definition file.

The attribute meaning of record node is shown in table six:

watch six

Each record contains a currentRecord node representing the record currently being processed, including the start record and the end record, and the attribute meaning of currentRecord is shown in table seven:

watch seven

Each record contains a motttieconsumingrecord node, which represents the most time-consuming record in the log corresponding to the node, and the meaning of the attribute is the same as that of the currentRecord.

In addition, if the result of the statistics needs to be used as a baseline for measuring the system performance, the result of the statistics can be stored as the baseline so as to detect the change of the system performance with reference to the baseline in the following. The base line can be counted for many times, and the result of each counting is weighted and averaged to be used as a new base line. The statistical baselines are stored in the form of XML files, and an example of storing a baseline result file is given below:

in the baseline result file, the root node is defined as baseline, the baseline data corresponding to each log matching item is stored in the result node, and the meaning of the result attribute is shown in table eight:

table eight

The technical solution of the present invention will be described in detail below with reference to preferred embodiments. Referring to fig. 3, fig. 3 is a flowchart of a log-based system performance analysis method according to a preferred embodiment of the present invention, which includes the following steps:

step 301, pre-configuring log matching items, configuring inclusion relations among the log matching items, and generating a result set tree according to the inclusion relations among the log matching items.

In this embodiment, the configured log matching items are stored in a matching rule definition file, and the inclusion relationship between the log matching items is stored in an analysis rule definition file. And each log matching item comprises a starting log matching rule and an ending log matching rule. In the result set tree generated according to the inclusion relationship among the log matching items, each node comprises the following parameters:

key words: uniquely identifying the node, corresponding to a log match ID;

marking position: the matching state of the log matching item corresponding to the node is marked to be 0, which represents that the initial log matching rule of the log matching item is not matched, and is 1, which represents that the initial log matching rule of the log matching item is matched;

recording number: representing the number of times that the log matching item corresponding to the node has been completely matched currently;

the total time consumption is as follows: for representing the total elapsed time that has been counted currently;

record being processed: and indicating that the log matching item corresponding to the node has not been matched with the record in the last time. If a record match is complete, i.e., a start log record and an end log record of a complete log record are found, then time is counted into total elapsed time and the number of records is accumulated, while the contents of the record are emptied for the next match. The record comprises a starting time, a starting log, an ending time and an ending log;

the most time-consuming recording: and indicating the record which takes the most time from the records of which the log matching item corresponding to the node has completed matching. The record is the same as the record being processed, including a start time, a start log, an end time, and an end log.

Here, the matched start log and end log are saved in the record being processed by the node and the most time-consuming record, so as to be used for the relevant personnel to view after the system performance analysis is performed.

Step 302, initializing parameters of each node in the result set tree except the keywords.

In this step, initializing parameters of each node includes: the marking bit, the number of records and the total time consumption are initialized to 0, and the contents of various parameters related to the records being processed and the most time-consuming records are set to be null.

Step 303, reading a log.

And step 304, judging whether the reading is successful, if so, executing step 305, otherwise, executing step 319.

In this step, if the log reading fails, it indicates that the log has been completely processed, and the system performance analysis is completed, so the process directly goes to step 319.

And 305, taking the first child node of the root node in the result set tree as a current node.

And step 306, acquiring a log matching item corresponding to the current node.

Here, the key of the current node is the log matching item ID corresponding to the node.

And 307, judging whether the marking bit of the current node is 0, if so, executing a step 308, and otherwise, executing a step 311.

Here, each log matching item includes a start log matching rule and an end log matching rule, which are respectively matched with a start log record and an end log record corresponding to one complete log record, when the log matching item corresponding to the node is used for matching the log, it is required to first judge whether the start log matching rule of the log matching item is matched, if so, the end log matching rule of the log matching item is directly used for matching the log, otherwise, only the start log matching rule of the log matching item is used for matching the log.

Step 308, matching the log with the initial log matching rule of the log matching item, and executing step 309.

Step 309, judging whether the initial log matching rule of the log matching item is successfully matched with the log, if so, executing step 310. Otherwise, step 313 is performed.

Step 310, extracting the recorded time, taking the time as the start time of the record being processed by the current node, and setting the flag bit of the current node to 1.

In this step, after the matching is successful, the system performance parameter needs to be extracted as the starting performance parameter of the record being processed by the current node. In addition, it is also necessary to record this log as the start log of the record being processed by the current node.

After matching is successful, the flag bit of the current node is set to 1, so as to flag that the initial log matching rule of the log matching item corresponding to the node is matched, and when a new log is subsequently read for matching, the end log matching rule of the log matching item is directly used for matching the new log, thereby realizing one-time complete matching of the log matching item.

And 311, matching the log by using the end log matching rule of the log matching item.

And step 312, judging whether the end log matching rule of the log matching item is successfully matched with the log, if so, executing step 314, otherwise, executing step 313.

And 313, acquiring the next node in the result set tree as the current node.

In this step, because the log matching item corresponding to the current node is not successfully matched with the log, a next node that is not matched with the log in the result set tree needs to be taken as a new current node, and the log matching item corresponding to the new current node is used to match the log.

The method of obtaining the next node in the result set tree is described in detail in fig. 4.

Step 314, extracting the time of the record, taking the time as the end time of the record being processed by the current node, and setting the flag bits of all nodes in the subtree taking the current node as the root node to 0.

In this step, the end log matching rule of the log matching item corresponding to the current node successfully matches the log, so the time in the log is extracted as the end time of the record being processed by the current node. In addition, it is also necessary to record this log as the end log of the record being processed by the current node. At this point, one complete matching of the log matching item corresponding to the node is completed.

After completing one complete matching of the log matching item corresponding to the node, the flag bit of the current node needs to be set to 0, because the log matching item corresponding to the current node is already completely matched once, and when a new log is subsequently read to match the log matching item corresponding to the current node, matching should be started again from the initial log matching rule of the log matching item. Meanwhile, the flag bits of all nodes in the subtree using the current node as the root node are also required to be set to 0, because when the start log matching rule and the end log matching rule of the log matching item corresponding to the current node are all matched, the log matching items contained in the subtree should be matched, and therefore, matching should be started from the start log matching rule of the log matching item corresponding to each node again.

Step 315, adding the difference between the ending time and the starting time of the record being processed by the current node into the total consumption of the current node, and adding 1 to the record number of the current node.

And step 316, judging whether the record currently processed by the current node is the most time-consuming record, if so, executing step 317, and otherwise, executing step 318.

The method for judging whether the record processed by the current node is the most time-consuming record comprises the following steps: judging whether the starting time and the ending time in the most time-consuming record of the current node are empty or not, if so, recording the record being processed by the current node as the most time-consuming record of the current node, otherwise, comparing the difference value between the ending time and the starting time in the record being processed by the current node with the difference value between the ending time and the starting time in the most time-consuming record of the current node, and if the difference value between the ending time and the starting time in the record being processed by the current node is greater than the difference value between the ending time and the starting time in the most time-consuming record of the current node, recording the record being processed by the current node as the most time-consuming record of the current node.

Step 317, storing the record being processed by the current node as the most time-consuming record of the current node.

Step 318, the record that the current node is processing is cleared.

The method for clearing the record being processed by the current node comprises the following steps: and setting parameters related to the record currently processed by the node to be null.

And 319, storing the statistical result in each node in the result set tree, and ending the performance analysis of the system.

It should be noted that after the log is processed and before step 319 is executed, it may be further determined whether to use the result of the current statistics as a baseline according to the value of the attribute isbasepine of the root node in the parsing rule definition file, and if so, a baseline is generated according to the statistical result of each node in the result set tree and is stored in the baseline file. The method for generating the baseline is related to the statistical system performance parameter object, and belongs to the problem of implementation, for example, in this embodiment, the total time consumption counted in each node is divided by the number of records to obtain the average time consumption, and the average time consumption is stored in the baseline file.

The method for obtaining the next node in the result set tree in step 313 will be described. Referring to fig. 4, fig. 4 is a flowchart of a method for obtaining a next node in a result set tree according to a preferred embodiment of the present invention, which includes the following steps:

step 401, determining whether the flag bit of the current node is 0, if yes, executing step 402, otherwise, executing step 403.

In this step, if the initial log matching rule of the log matching item corresponding to the current node is already matched, the next node is preferentially selected from the current child node, otherwise, the next node is preferentially selected from the sibling nodes.

Step 402, judging whether the current node has a right side brother node, if so, executing step 404, otherwise, executing step 406.

And 403, judging whether the current node has a child node, if so, executing the step 403, otherwise, executing the step 406.

Step 404, the first right sibling node of the current node is used as the next node to be obtained, and the next node is obtained successfully.

Step 405, taking the first child node of the current node as the next node to be acquired, and acquiring the next node successfully.

And 406, judging whether the current node has a parent node, if so, executing step 407, and otherwise, executing step 410.

Step 407, judging whether the father node of the current node has a right brother node, if so, executing step 408, otherwise, executing step 409.

And step 408, taking the first right side brother node of the father node of the current node as the next node to be obtained, and obtaining the next node successfully.

And 409, taking the father node of the current node as the current node, and returning to 406 for execution.

Step 410, acquiring the next node fails.

As can be seen from the above method of obtaining the next node, this method is actually a method of traversing each node in the tree. Here, other methods of traversing the tree may be used to obtain the next node.

The invention also provides a system performance analysis device based on the log, which can effectively track the system running on line.

Referring to fig. 5, fig. 5 is a schematic structural diagram of a log-based system performance analysis apparatus according to an embodiment of the present invention. The device includes: a configuration unit 501, a log acquisition unit 502, a matching unit 503, a node acquisition unit 504, and a result holding unit 505; wherein,

a configuration unit 501, configured to pre-configure the log matching items and the inclusion relationship between the log matching items, and generate a result set tree according to the configured inclusion relationship between the log matching items;

a log obtaining unit 502, configured to, when receiving a log obtaining notification sent by the matching unit 503, read a log from the log file, and if the log is successfully read, send the read log to the matching unit 503; if the reading fails, the notification result storage unit 505 stores the statistical result of the system performance analysis of this time;

a matching unit 503, configured to notify the log obtaining unit 502 to obtain a log when system performance analysis is started; the node is used for taking the first child node of the root node in the result set tree as the current node after receiving the log sent by the log obtaining unit 502; the system performance statistics module is used for matching the log matching item corresponding to the current node with the received log, if the matching is successful, extracting the system performance parameters in the log, performing system performance statistics, and notifying the log obtaining unit 502 to obtain the log, and if the matching is failed, notifying the node obtaining unit 504 to obtain the next node; when receiving the next node sent by the node obtaining unit 504, the node obtaining unit is configured to take the received next node as a new current node; the log obtaining unit 502 is notified to obtain the log after receiving a notification that obtaining the next node fails, which is sent by the node obtaining unit 504;

a node obtaining unit 504, configured to obtain a next node from the result set tree after receiving the notification of obtaining the next node sent by the matching unit 503, send the obtained next node to the matching unit 503 if the obtaining is successful, and notify the matching unit 503 that the obtaining of the next node is failed if the obtaining is failed;

after receiving the notification from log obtaining section 502, result saving section 505 saves the statistical result of each node in the result set tree, and this system performance analysis is completed.

the system performance parameter is time;

when the configuration unit 501 generates a result set tree according to the inclusion relationship between the log matching items, further marking the initial log matching rule of the log matching item corresponding to each node in the result set tree as unmatched;

when the matching unit 503 matches the log matching item corresponding to the current node with the received log, it is configured to: judging whether the initial log matching rule of the log matching item corresponding to the current node is matched, if so, matching the end log matching rule of the log matching item corresponding to the current node with the read log, extracting the system performance parameter in the log under the condition of successful matching, marking the system performance parameter as the end time of the record being processed by the current node, carrying out system time consumption statistics according to the initial time and the end time of the record being processed by the current node, marking the initial log matching rule of the log matching item corresponding to the current node as unmatched, and informing the log obtaining unit 502 to obtain the log; in the case of failure in matching, the node acquisition unit 504 is notified to acquire the next node;

otherwise, matching the initial log matching rule of the log matching item corresponding to the current node with the read log, extracting the time in the log under the condition of successful matching, marking the time as the initial time of the record being processed by the current node, marking the initial log matching rule of the log matching item corresponding to the current node as matched, and informing the log obtaining unit 502 to obtain the log; in the case where the matching fails, the node acquisition unit 504 is notified to acquire the next node.

the matching unit 503 is configured to count the system time consumption according to the start time and the end time of the record being processed by the current node, and is configured to: calculating the difference between the ending time and the starting time of the record being processed by the current node, and accumulating the difference into the total time consumption of the current node; after the system time consumption statistics is carried out according to the starting time and the ending time of the record being processed by the current node, the log reading unit is informed to further read the log: adding 1 to the number of records of the current node, and emptying the records processed by the current node; when the initial log matching rules of the log matching items corresponding to the current node are marked as unmatched, the initial log matching rules of the log matching items corresponding to all nodes in the subtree taking the current node as the root node are further marked as unmatched.

the matching unit 503, after adding 1 to the number of records of the current node, is further configured to: and judging whether the record being processed by the current node is the most energy-consuming record of the current node according to the starting time and the ending time of the record being processed by the current node, and if so, recording the record being processed by the current node as the most time-consuming record of the current node.

The matching unit 503, when determining whether the record being processed by the current node is the most energy-consuming record of the current node according to the start time and the end time of the record being processed by the current node, is configured to: judging whether the starting time and the ending time in the most time-consuming record of the current node are empty or not, if so, recording the record being processed by the current node as the most time-consuming record of the current node, otherwise, comparing the difference value between the ending time and the starting time in the record being processed by the current node with the difference value between the ending time and the starting time in the most time-consuming record of the current node, and if the difference value between the ending time and the starting time in the record being processed by the current node is larger, recording the record being processed by the current node as the most time-consuming record of the current node.

The node obtaining unit 504 is configured to, when obtaining a next node: judging whether the initial log matching rule of the log matching item corresponding to the current node is matched, if so, taking the first child node of the current node as the next acquired node when the current node has child nodes, otherwise, taking the first right brother node of the current node as the next acquired node when the current node has right brother nodes;

When the configuration unit 501 configures the inclusion relationship between the log matching items in advance, it further configures a flag indicating whether to use the current statistical result as a baseline;

after the result saving unit 505 saves the current statistical result of each node in the result set tree into the result set file, the current system performance analysis is further used to: and judging whether the current statistical result is required to be used as the baseline according to the set mark of whether the current statistical result is used as the baseline, and if so, generating a baseline file according to the current statistical result of each node in the result set tree.

The above description is only for the purpose of illustrating the preferred embodiments of the present invention and is not to be construed as limiting the invention, and any modifications, equivalents, improvements and the like made within the spirit and principle of the present invention should be included in the scope of the present invention.

Claims

1. A system performance analysis method based on logs is characterized in that log matching items are configured in advance, the inclusion relationship among the log matching items is configured, and a result set tree is generated according to the inclusion relationship among the log matching items; the method comprises the following steps:

C. and B, acquiring the next node from the result set tree as a new current node, and turning to the step B for execution.

2. The log-based system performance analysis method of claim 1, wherein the log matching entries comprise a start log matching rule and an end log matching rule;

the system performance parameter is time;

the step B comprises the following steps:

b3, matching the end log matching rule of the log matching item corresponding to the current node with the read log, if the matching is successful, extracting the time in the log, marking the time as the end time of the record being processed by the current node, carrying out system time consumption statistics according to the end time and the start time of the record being processed by the current node, marking the start log matching rule of the log matching item corresponding to the current node as unmatched, turning to the step A for execution, otherwise, turning to the step C for execution.

3. The log-based system performance analysis method of claim 2, wherein each node in the result set tree further comprises parameters: recording the number and total time spent;

4. The log-based system performance analysis method of claim 3, wherein each node in the result set tree further comprises: recording the most time-consuming; the most time consuming records include: a start time and an end time;

5. The log-based system performance analysis method of claim 4, wherein the method for determining whether the record being processed by the current node is the most time-consuming record of the current node according to the start time and the end time of the record being processed by the current node comprises: judging whether the starting time and the ending time in the most time-consuming record of the current node are empty or not, if so, recording the record being processed by the current node as the most time-consuming record of the current node, otherwise, comparing the difference value between the ending time and the starting time in the record being processed by the current node with the difference value between the ending time and the starting time in the most time-consuming record of the current node, and if the difference value between the ending time and the starting time in the record being processed by the current node is larger, recording the record being processed by the current node as the most time-consuming record of the current node.

6. The log-based system performance analysis method of any of claims 1 to 5, wherein step C comprises:

7. The log-based system performance analysis method of claim 6, wherein when the inclusion relationship between the log matching item and the log matching item is configured in advance, a flag indicating whether the current statistical result is used as a baseline is further configured;

8. An apparatus for log-based system performance analysis, the apparatus comprising: the device comprises a configuration unit, a log acquisition unit, a matching unit, a node acquisition unit and a result storage unit;

9. The apparatus of claim 8, wherein the log matching entries comprise a start log matching rule and an end log matching rule;

the system performance parameter is time;

10. The apparatus of claim 9, wherein each node in the result set tree further comprises a parameter: recording the number and total time spent;

11. The apparatus of claim 10, wherein each node in the result set tree further comprises: recording the most time-consuming; the most time consuming records include: a start time and an end time;

12. The log-based system performance analysis method of claim 11, wherein the matching unit, when determining whether the record being processed by the current node is the most energy consuming record of the current node according to the start time and the end time of the record being processed by the current node, is configured to: judging whether the starting time and the ending time in the most time-consuming record of the current node are empty or not, if so, recording the record being processed by the current node as the most time-consuming record of the current node, otherwise, comparing the difference value between the ending time and the starting time in the record being processed by the current node with the difference value between the ending time and the starting time in the most time-consuming record of the current node, and if the difference value between the ending time and the starting time in the record being processed by the current node is larger, recording the record being processed by the current node as the most time-consuming record of the current node.

13. The log-based system performance analysis method of any of claims 8 to 12, wherein the node obtaining unit, when obtaining a next node, is configured to: judging whether the initial log matching rule of the log matching item corresponding to the current node is matched, if so, taking the first child node of the current node as the next acquired node when the current node has child nodes, otherwise, taking the first right brother node of the current node as the next acquired node when the current node has right brother nodes;

14. The log-based system performance analysis method of claim 13, wherein when the configuration unit pre-configures the inclusion relationship between the log matching item and the log matching item, it further configures a flag indicating whether to use the current statistical result as a baseline;

after the result storage unit stores the current statistical result of each node in the result set tree, the result storage unit is further used for the following steps before the system performance analysis is finished: and judging whether the current statistical result is required to be used as the baseline according to the set mark of whether the current statistical result is used as the baseline, and if so, generating the baseline according to the current statistical result of each node in the result set tree.