CN114138977A

CN114138977A - Log processing method and device, computer equipment and storage medium

Info

Publication number: CN114138977A
Application number: CN202111516480.3A
Authority: CN
Inventors: 赵一帆
Original assignee: Industrial and Commercial Bank of China Ltd ICBC
Current assignee: Industrial and Commercial Bank of China Ltd ICBC
Priority date: 2021-12-13
Filing date: 2021-12-13
Publication date: 2022-03-04

Abstract

The application relates to a log processing method, a log processing device, computer equipment and a storage medium, which are applied to the technical field of artificial intelligence. The method comprises the following steps: performing information extraction processing on a log to be processed to obtain keyword information in the log to be processed; according to the middleware category corresponding to the log to be processed, carrying out extraction processing on the keyword information again to obtain target keyword information matched with the middleware category; the middleware category is a category of a server for deploying the application program corresponding to the log to be processed; determining fault information corresponding to the log to be processed based on the target keyword information; and searching a pre-constructed decision tree according to the fault information to obtain a decision result corresponding to the log to be processed. By adopting the method, the operation and maintenance emergency time can be shortened, and the risk of production accidents is reduced.

Description

Log processing method and device, computer equipment and storage medium

Technical Field

The present application relates to the field of artificial intelligence technologies, and in particular, to a log processing method, an apparatus, a computer device, a storage medium, and a computer program product.

Background

The operation and maintenance of the software are the last stage of the life cycle of the software and the longest duration, and are directly oriented to the terminal user, so that the operation and maintenance guarantee work of the software is of far-reaching significance, and the occurrence of production accidents in the operation and maintenance process of the software is often caused by the application of the software and is reflected in a log monitored on a non-system side. Therefore, the analysis and evaluation of the log become an important part of the operation and maintenance work.

However, most of the existing evaluation on log output is to acquire log analysis and evaluation by developers, and much time is consumed, so that the emergency timeliness is influenced.

Disclosure of Invention

In view of the foregoing, it is desirable to provide a log processing method, an apparatus, a computer device, a computer readable storage medium, and a computer program product for solving the above technical problems that a personalized log cannot be analyzed and responded in time by a conventional monitoring tool.

In a first aspect, the present application provides a log processing method. The method comprises the following steps:

performing information extraction processing on a log to be processed to obtain keyword information in the log to be processed;

according to the middleware category corresponding to the log to be processed, carrying out extraction processing on the keyword information again to obtain target keyword information matched with the middleware category; the middleware category is a category of a server for deploying the application program corresponding to the log to be processed;

determining fault information corresponding to the log to be processed based on the target keyword information;

and searching a pre-constructed decision tree according to the fault information to obtain a decision result corresponding to the log to be processed.

In one embodiment, the extracting information from the log to be processed to obtain the keyword information in the log to be processed includes:

extracting a plurality of target sample error keywords from a keyword information base based on the weight of the sample error keywords in a pre-constructed keyword information base; the keyword information base stores a plurality of sample error keywords and weights of the sample error keywords;

and according to each target sample error keyword, performing information extraction processing on the log to be processed to obtain keyword information in the log to be processed.

In one embodiment, the weight of the sample error key is determined by:

obtaining a plurality of sample error keywords in a sample log;

respectively counting the occurrence times of each sample error keyword in the sample log and the accumulated sum of the occurrence times of each sample error keyword;

and aiming at any sample error keyword, acquiring the ratio of the occurrence times of the sample error keyword to the accumulated sum of the occurrence times as the weight of the sample error keyword.

In one embodiment, the determining, based on the target keyword information, fault information corresponding to the log to be processed includes:

processing the target keyword information through the trained classification model to obtain fault information of the log to be processed; the classification model is obtained by training sample data which takes target keyword information as input and takes fault information as output.

In a second aspect, the present application further provides a log processing apparatus. The device comprises:

the first extraction module is used for extracting information from the log to be processed to obtain keyword information in the log to be processed;

the second extraction module is used for determining the type of the middleware corresponding to the log to be processed and extracting the keyword information again according to the type of the middleware to obtain target keyword information matched with the type of the middleware; the middleware category is a category of a server for deploying the application program corresponding to the log to be processed;

the determining module is used for determining fault information corresponding to the log to be processed based on the target keyword information;

and the searching module is used for searching a pre-constructed decision tree according to the fault information to obtain a decision result corresponding to the log to be processed.

In a third aspect, the present application also provides a computer device. The computer device comprises a memory storing a computer program and a processor implementing the following steps when executing the computer program:

In a fourth aspect, the present application further provides a computer-readable storage medium. The computer-readable storage medium having stored thereon a computer program which, when executed by a processor, performs the steps of:

In a fifth aspect, the present application further provides a computer program product. The computer program product comprising a computer program which when executed by a processor performs the steps of:

According to the log processing method, the log processing device, the computer equipment, the storage medium and the computer program product, the information extraction processing is firstly carried out on the log to be processed to obtain the keyword information in the log to be processed; and then extracting and processing the keyword information again according to the middleware category corresponding to the log to be processed to obtain target keyword information matched with the middleware category, determining fault information corresponding to the log to be processed based on the target keyword information, and finally searching a pre-constructed decision tree according to the fault information to obtain a decision result corresponding to the log to be processed. According to the method, the model is established at the terminal through the decision tree, the artificial intelligent scheduling of the log to be processed is realized, and developers do not need to analyze and judge the log by themselves, so that the operation and maintenance emergency time can be shortened, and the operation and maintenance efficiency is improved.

Drawings

FIG. 1 is a diagram of an application environment of a log processing method in one embodiment;

FIG. 2 is a flow diagram that illustrates a method for log processing, according to one embodiment;

FIG. 3 is a diagram of a middleware based knowledge graph in one embodiment;

FIG. 4 is a diagram illustrating a decision tree in a log processing method according to an embodiment;

FIG. 5 is a flowchart illustrating a log processing method according to another embodiment;

FIG. 6 is a flow chart illustrating a log processing method in an application example;

FIG. 7 is a block diagram showing the structure of a log processing apparatus according to an embodiment;

FIG. 8 is a diagram illustrating an internal structure of a computer device according to an embodiment.

Detailed Description

In order to make the objects, technical solutions and advantages of the present application more apparent, the present application is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application.

The operation and maintenance are taken as a stage of the software life cycle which is also the longest in duration at the end, and are directly oriented to the end user. Therefore, the importance degree of the operation and maintenance support system is obvious, and the operation and maintenance support system has a profound significance. For system operation and maintenance, it is currently common practice to deploy a monitoring program in advance by a technical means, and ensure the availability of services by actively sending an agreed message to a monitoring system in the production running process or check whether the system can be normally accessed by using a network detection mode, so as to isolate a fault server after an abnormality is found.

With the development of computer technology, software and hardware configuration is gradually upgraded and updated, and enterprises are also continuously promoting architecture transformation. However, after the basic platform is changed, the application side and the monitoring side need to re-agree a new set of rules to adapt to the new platform. The primary purpose of making relatively uncertain adaptations and operation and maintenance to achieve a new balance-stability comes in and goes out. Meanwhile, the monitoring in the aspects of the system and the network is only a part of the application operation and maintenance, and more production accidents are often caused by the application itself and are reflected in the log monitored by the non-system side. However, most of the existing evaluation on non-standardized log output is to acquire log analysis and evaluation by developers, and an automatic tool is lacked for intelligent judgment, so that the emergency timeliness is influenced, and the result is finally transmitted to the satisfaction and the trust degree of a terminal client to an enterprise.

Therefore, in order to solve the problem that the personalized log cannot be analyzed through a conventional monitoring tool and timely response is achieved, the application provides an intelligent operation and maintenance scheme for automatically evaluating and quickly feeding back to make a decision based on a decision tree algorithm: provided is a log processing method.

The log processing method provided by the embodiment of the application can be applied to the application environment shown in fig. 1. Wherein the terminal 102 communicates with the server 104 via a network. The data storage system may store data that the server 104 needs to process. The data storage system may be integrated on the server 104, or may be located on the cloud or other network server. In an application scenario of the application, the terminal 102 acquires a log to be processed from the server 104, performs information extraction processing on the log to be processed to obtain keyword information in the log to be processed, and performs extraction processing on the keyword information again according to a middleware category corresponding to the log to be processed to obtain target keyword information matched with the middleware category; determining fault information corresponding to the log to be processed based on the target keyword information; and searching a pre-constructed decision tree according to the fault information to obtain a decision result corresponding to the log to be processed.

In this application, the terminal 102 may be but is not limited to various personal computers, notebook computers, smart phones, tablet computers, internet of things devices and portable wearable devices, and the internet of things devices may be smart speakers, smart televisions, smart air conditioners, smart car-mounted devices, and the like. The portable wearable device can be a smart watch, a smart bracelet, a head-mounted device, and the like. The server 104 may be implemented as a stand-alone server or as a server cluster comprised of multiple servers.

In one embodiment, as shown in fig. 2, a log processing method is provided, which is described by taking the method as an example applied to the terminal 102 in fig. 1, and includes the following steps:

step S210, performing information extraction processing on the log to be processed to obtain keyword information in the log to be processed.

The log to be processed may represent a log generated when the application fails to operate.

The keyword information may be keyword information related to an application operation failure.

In the daily operation process of the application system, logs are continuously generated on the server. And calling a log collection program by deploying a timing task, and downloading the application side log to a special analysis platform within a certain interval time. Specifically, the required log files on the server can be downloaded through a garmed-ssh 2-build, jar, which is an existing jar (a computer file format) package in Java (a computer programming language). Among them, the gapymed-SSH 2 is a packet for implementing SSH-2 protocol (Secure Shell protocol) by java, and can be used to connect to the SSH server, and call sz instruction (a command for sending a file from the server to the client) of linux, so as to send a log file from the server 104 to the terminal 102.

After receiving the log file sent by the server 104, the terminal 102 uses the log file as a log to be processed, applies lexical and syntactic analysis of Natural Language Processing (NLP), splits the text in the log to be processed into independent participles, and extracts feature words related to error-reporting keywords according to a word frequency-inverse file frequency algorithm (TF-IDF algorithm) as keyword information. More specifically, the keyword information in the log to be processed is extracted by natural language processing, a plurality of target sample error keywords can be obtained by searching a pre-constructed keyword information base, and the information of the log to be processed is extracted according to each target sample error keyword to obtain the keyword information in the log to be processed.

Step S220, extracting the keyword information again according to the middleware category corresponding to the log to be processed to obtain target keyword information matched with the middleware category; the middleware category is a category of a server for deploying an application program corresponding to the log to be processed.

The middleware is a type of a server for deploying the application program corresponding to the log to be processed, and the middleware type may include types of wa, Liberty, Tomcat, TongWeb, and the like.

The target keyword information may include error reporting information, hidden trouble information, and the like.

In a specific implementation, since error information corresponding to different middleware may be different, as shown in fig. 3, the keyword information corresponding to the middleware Was is "com.ibm.ws.rsadapter.dsconfighelper $1.run (DSC onfigug helper.java:1273), and the keyword information corresponding to the middleware Liberty is" com.ibm.ejs.j2c.mcwr app.close (mcwrapp.java: 1680) ", therefore, in order to more accurately determine fault information corresponding to the log to be processed, after the keyword information is extracted from the log to be processed by a natural language processing manner, the category of the server where the application program corresponding to the log to be processed is deployed may be further determined, as the middleware category, the knowledge graph technology is used to extract the keyword information again according to the middleware category, so as to obtain error information and fault information matching with the middleware category corresponding to the log to be processed, as the hidden danger of the middleware, and the target algorithm may use the keyword information Error information and fault hidden trouble information form a structured keyword text, and unstructured information forms a structured data system.

The knowledge graph is a structured semantic knowledge base used for rapidly describing concepts and mutual relations in the physical world, is converted into simple and clear entity, relation and entity triplets by effectively processing, processing and integrating data of complex documents, and finally aggregates a large amount of knowledge, so that rapid response and reasoning of the knowledge are realized, and the knowledge graph is based on a middleware log as shown in fig. 3. Two nodes with relationship are connected together through a non-directional edge, the nodes are called entities, and the edge between the nodes is called relationship.

And step S230, determining fault information corresponding to the log to be processed based on the target keyword information.

The fault information may include information of a plurality of fault indicators, for example, the fault information may include information of a fault level, a production accident generation probability, and the like.

In the specific implementation, the error reporting information and the fault hidden danger information in the target keyword information are used as input, the fault grade and the production accident generation probability are used as output, a classification model is obtained through pre-training, the target keyword information and the influence range information of the fault are processed through the classification model, the fault grade and the production accident generation probability of the log to be processed are obtained, and the fault grade and the production accident generation probability are used as fault information of the log to be processed.

Step S240, according to the fault information, searching a pre-constructed decision tree to obtain a decision result corresponding to the log to be processed.

Wherein the decision tree represents a tree structure similar to the flow chart, each node inside the tree represents a test on a feature, a branch of the tree represents each test result of the feature, each leaf node of the tree represents a classification, the highest level of the tree is a root node, as shown in fig. 4, the tree is a schematic diagram of the decision tree, the internal nodes are represented by rectangles, and the leaf nodes are represented by ellipses.

The decision result may include that the application needs to be restarted and that the application does not need to be restarted.

In specific implementation, logs of different decision results can be respectively obtained according to a certain proportion to form training samples, each sample contains fault information of a plurality of fault indexes, for example, each sample contains fault levels and production accident occurrence probability, a decision tree is constructed based on the training samples and a decision tree component algorithm ID3 of information gain, and the fault levels of all the fault indexes in the fault information are determined. After the fault information of the log to be processed is obtained, the constructed decision trees are searched layer by layer according to the sequence of the fault grades of all fault indexes in the fault information from high to low, leaf nodes corresponding to the fault information in the decision trees are obtained, and the decision results corresponding to the leaf nodes are used as the decision results corresponding to the log to be processed.

For example, taking the decision tree shown in fig. 4 as an example, when searching for the decision tree, firstly, comparing the fault level indicators of the root node, and if the fault level of the log to be processed is greater than a, entering the leaf node of the left branch to obtain a decision result: a restart is required. If the fault level of the log to be processed is not greater than A, entering a node influence range node of a right branch, judging whether the influence range of the log to be processed is greater than B, if so, entering a leaf node of a left branch, and obtaining a decision result: a restart is required. If not, continuing to enter the production accident generation probability node of the right branch for judgment, and if the production accident generation probability of the log to be processed is greater than C, entering the leaf node of the left branch to obtain a decision result: a restart is required. Otherwise, entering a right branch, continuously judging whether the production accident generation probability of the log to be processed is smaller than D, if so, obtaining a decision result: restarting is not needed; if not, obtaining a decision result: a restart is required.

In the log processing method, the information extraction processing is firstly carried out on the log to be processed to obtain the keyword information in the log to be processed; and then extracting and processing the keyword information again according to the middleware category corresponding to the log to be processed to obtain target keyword information matched with the middleware category, determining fault information corresponding to the log to be processed based on the target keyword information, and finally searching a pre-constructed decision tree according to the fault information to obtain a decision result corresponding to the log to be processed. According to the method, the model is established at the terminal through the decision tree, the artificial intelligent scheduling of the log to be processed is realized, and developers do not need to analyze and judge the log by themselves, so that the operation and maintenance emergency time can be shortened, and the operation and maintenance efficiency is improved.

In an exemplary embodiment, the performing, in the step S210, information extraction processing on the log to be processed to obtain the keyword information in the log to be processed includes: extracting a plurality of target sample error keywords from a keyword information base based on the weights of the sample error keywords in a pre-constructed keyword information base; the keyword information base stores a plurality of sample error keywords and weights of the sample error keywords; and according to the error keywords of each target sample, performing information extraction processing on the log to be processed to obtain the keyword information in the log to be processed.

Wherein the weight may represent a frequency of occurrence of the error key in the fault log.

In specific implementation, sample error keywords in a sample log can be extracted in advance, the occurrence frequency of each sample error keyword is calculated to serve as the weight of each sample error keyword, an association relationship between each sample error keyword and the weight corresponding to each sample error keyword is established, and the association relationship is stored in a keyword information base. After the logs to be processed are obtained, the sample error keywords can be sequenced according to the sequence of the weighted values from high to low to obtain a sample error keyword sequence, then a plurality of target sample error keywords are extracted from the keyword information base according to the sequence of the weighted values from high to low, information extraction is carried out in the logs to be processed according to the target sample error keywords to obtain keyword information in the logs to be processed.

Further, in an exemplary embodiment, the weight of the sample error key is determined by: obtaining a plurality of sample error keywords in a sample log; respectively counting the occurrence times of each sample error keyword in the sample log and the cumulative sum of the occurrence times of each sample error keyword; and aiming at any sample error keyword, acquiring the ratio of the occurrence times of the sample error keyword to the accumulated sum of the occurrence times as the weight of the sample error keyword.

In specific implementation, a TF-IDF algorithm (Term Frequency-Inverse Document Frequency algorithm) can be used for extracting feature words related to sample log error-reporting keywords, and calculating the weights of the sample error keywords. More specifically, firstly, extracting keywords from the sample log to obtain a plurality of sample error keywords in the sample log, respectively counting the occurrence frequency of each sample error keyword in the sample log and the cumulative sum of the occurrence frequency of each sample error keyword, and calculating the ratio of the occurrence frequency of the sample error keyword to the cumulative sum of the occurrence frequency of any sample error keyword as the weight of the sample error keyword.

In the embodiment, the keyword information base is pre-constructed by acquiring the sample error keywords in the sample log and calculating the weight, so that the keyword information in the log to be processed can be better positioned, and the acquisition efficiency of the keyword information is improved.

In an exemplary embodiment, the determining, in step S230, the fault information corresponding to the log to be processed based on the target keyword information may be implemented by: processing the target keyword information through the trained classification model to obtain fault information of the log to be processed; the classification model is obtained by training sample data which takes target keyword information as input and takes fault information as output.

In the specific implementation, the classification of the target keyword information can be realized through a naive Bayes method, sample data is firstly obtained, the sample data is divided into training data and test data, the target keyword information is used as input, fault information is used as output, the Bayes model is trained through the training data, the Bayes model is tested through the test data, and the Bayes model with the accuracy meeting the preset conditions is obtained and used as the classification model after the training. After the target keyword information of the log to be processed is obtained, classifying the target keyword information through a Bayesian model to obtain the fault information of the log to be processed.

The naive Bayes algorithm is a classification method based on Bayes theorem and independent assumption of characteristic conditions, and the idea is that when the essence of an object cannot be accurately known, the probability of the essence attribute of the object can be judged according to the occurrence amount of events related to the specific essence of the object. For a given training set, the joint probability distribution of input and output is independently learned based on characteristic conditions, and then the output y with the maximum posterior probability is calculated on the given input x by using Bayes theorem on the model. The basic method of the algorithm is as follows:

setting an input space

For a set of n-dimensional vectors, the output space is the class label set Y ═ c1, c2 …, ck }. The input is a feature vector, the output is a label of a class, and the training set is as follows:

T＝{(x1,y1),(x2,y2),…,(xN,yN)}

learning the joint probability distribution P (X, Y) through the training set, the joint probability P (X, Y) calculation mode of one example:

P(X,Y)＝P(X|Y)·P(Y)＝P(Y|X)·P(X)

the general form of bayesian theory is obtained from the above equation:

naive bayes can also be expressed as:

after the same denominator is removed, the formula is simplified as:

in the embodiment, the incidence relation between the target keyword information and the fault information is established by constructing the automatic classification model with the learning capability, so that the prediction efficiency of the log fault to be processed can be improved.

In an exemplary embodiment, the determining, in step S230, the fault information corresponding to the log to be processed based on the target keyword information further includes: determining the influence range information of the log to be processed; and determining fault information corresponding to the log to be processed based on the influence range information and the target keyword information.

In the specific implementation, when the key word information is extracted again by using the knowledge graph technology, the influence range information of the fault can be identified at the same time, and the fault grade and the production accident generation probability corresponding to the log to be processed are determined according to the influence range information and the target key word information. More specifically, the fault reporting information, the fault hidden danger information and the fault influence range information in the target keyword information are used as input, the fault level and the production accident generation probability are used as output, a classification model is obtained through pre-training, the target keyword information and the fault influence range information are processed through the classification model, and the fault level and the production accident generation probability of the log to be processed are obtained and serve as the fault information of the log to be processed.

In this embodiment, the influence range information of the log to be processed is combined with the target keyword information to determine the fault information corresponding to the log to be processed, so that the accuracy of the determined fault information can be further improved.

In an exemplary embodiment, in the step S140, a pre-constructed decision tree is searched according to the fault information to obtain a decision result corresponding to the log to be processed, and the decision result may be implemented by: sequencing all fault indexes in the fault information according to a preset fault grade; the fault information comprises a plurality of fault indexes and index values of the fault indexes; searching the decision tree layer by layer according to the sequence of the fault grades of the fault indexes from high to low to obtain leaf nodes corresponding to the fault information in the decision tree; and determining a decision result corresponding to the leaf node as a decision result corresponding to the log to be processed.

In specific implementation, the sorting structure of the fault levels of each fault index in the fault information is the same as the sorting result of the fault levels of each fault index when the decision tree is constructed. After the fault information of the log to be processed is obtained, the decision trees can be searched layer by layer according to the sequence of the fault grades of all fault indexes in the fault information from high to low, the decision trees are compared with the conditions of all decision nodes until the leaf nodes are obtained, and the decision results corresponding to the leaf nodes are used as the decision results corresponding to the log to be processed.

The construction of the decision tree can be based on the decision tree construction algorithm ID3 of the information gain, and the following explains the construction of the decision tree by taking the application error reporting log information of Was middleware as an example:

assuming that the obtained samples are five samples shown in table 1 below, each sample has three attributes of a fault level, an influence range and a production accident generation probability, and finally, whether automatic restart is needed or not needs to be judged, various decision trees can be trained through the samples, as shown in fig. 5, which is one of the decision trees.

TABLE 1Was application error Log sample of middleware

By learning the data of table 1, a failure index such as A, B, C, D shown in fig. 5 can be set: and the fault grade, the influence range and the production accident generation probability, wherein the decision threshold value is used for minimizing the classification error rate.

The construction of the decision tree mainly comprises two steps of splitting a node and determining a threshold, wherein the splitting of the node indicates that when the attribute represented by one node cannot be judged, the node is generally selected to be divided into two child nodes (if the node is not a binary tree, the node is divided into n child nodes). The determination of the threshold requires selecting an appropriate threshold to ensure a high classification accuracy. The algorithm for generating the decision tree is a classification method for approximating a discrete function value, firstly, data is processed, readable rules and decision trees are generated by utilizing an inductive algorithm, and then new data is analyzed by using a decision. The decision tree algorithm constructs a decision tree to discover the classification rules embodied in the data. Taking the decision tree construction algorithm ID3 based on information gain as an example, the Entropy increase (Entropy) principle is used to decide which is the parent node and which needs to be split. For a set of data, a smaller entropy indicates a better classification result.

The definition for entropy is as follows:

wherein, p (x)_i) Is x_iThe probability of occurrence.

In case of two-class case, when each of class a and class B accounts for 50%,

Entropy＝-(0.5*log₂(0.5)+0.5*log₂(0.5))＝1

when there is only class a or only class B,

Entropy＝-(1*log₂(1)+0)＝0

so when control is at most 1, it is the state of the worst classification effect, and when it is at least 0, it is the state of the complete classification. The entropy equal to 0 is an ideal state, in general practical situations, the entropy is between 0 and 1, and the continuous minimization of the entropy is a process for improving the classification accuracy.

Such as 3 attributes in table 1: classified singularly by the following statements:

1) low fault level [ without restart ]: dividing by 1;

2) large influence range [ needs to restart ]: dividing by 1;

3) the probability of production accident is more than 50% [ needs to restart ]: no error is divided;

finally, the probability of production accident generation is found to be more than 50% [ restart is needed ], the error is the least, namely the entropy is the least, so the tree generation should be performed on the parent node by selecting the node. And when the father node is split, the reason is the same, and the classification error rate before splitting is compared, so that the selection capable of improving the accuracy rate is reserved.

After the decision tree is constructed, each leaf node has a corresponding execution scheme. Aiming at common system operation and maintenance work such as application server restart, directory cleaning and the like, the method can reach the standard of no manual intervention and is implemented by directly scheduling a deployed automation tool. The parts of the leaf nodes which need human intervention analysis are distributed with manual access processing in a short term, and operation and maintenance tools and decision tree modules of the parts are continuously perfected in a medium-long term so as to achieve higher-level intelligent operation and maintenance.

In the embodiment, the artificial intelligence scheduling of the automation tool is realized by establishing the model through the decision tree, the operation and maintenance emergency time is greatly shortened, and the risk of production accidents is reduced.

In one embodiment, as shown in fig. 5, a log processing method is provided, and in this embodiment, the method includes the following steps:

step S510, extracting a plurality of target sample error keywords from a keyword information base based on the weights of the sample error keywords in a pre-constructed keyword information base; the keyword information base stores a plurality of sample error keywords and weights of the sample error keywords;

step S520, according to each target sample error keyword, performing information extraction processing on the log to be processed to obtain keyword information in the log to be processed;

step S530, determining the influence range information and the middleware category of the log to be processed;

step S540, extracting the keyword information again according to the middleware category corresponding to the log to be processed to obtain target keyword information matched with the middleware category;

step S550, processing the target keyword information and the influence range information through the trained classification model to obtain fault information of the log to be processed; the classification model is obtained by training sample data which takes target keyword information as input and takes fault information as output;

step S560, sorting each fault index in the fault information according to a predetermined fault level; the fault information comprises a plurality of fault indexes and index values of the fault indexes;

step S570, searching the decision tree layer by layer according to the sequence of the fault grades of the fault indexes from high to low to obtain leaf nodes corresponding to the fault information in the decision tree;

step S580, determining a decision result corresponding to the leaf node as a decision result corresponding to the log to be processed.

It can be understood that the operation and maintenance work generally covers both the application and the system, wherein the system side is responsible for maintenance support by technical departments such as professional systems and networks, including monitoring CPU usage, disk capacity, table space usage, database session number, and the like. And the application side is supported by more indexes set by the application side, and whether the system has risks or not is judged by comparing the indexes with a preset threshold value. According to the method and the device, based on application layer logs, matching output information is analyzed by means of data mining, and then system layer emergency measures are intelligently scheduled according to a decision tree algorithm, so that the efficiency and flexibility of operation and maintenance can be improved.

In one embodiment, to facilitate understanding of embodiments of the present application by those skilled in the art, reference will now be made to the specific examples illustrated in the drawings. Referring to fig. 6, there is shown a flow chart diagram of a log processing method, including the following steps:

(1) acquiring file contents of error logs, and extracting information of the file contents of the error logs through Natural Language Processing (NLP) to obtain keyword information of the error logs;

natural Language Processing (NLP) refers to a study of processing and processing language texts (sentences, chapters, utterances, etc.) by using computer technology. The research content comprises various processing methods and implementation technologies such as recognition, classification, extraction, conversion and generation of information such as lexical, syntactic, semantic and pragmatic languages. In the invention, the text in the application log is divided into independent participles by using a lexical method and syntactic analysis of natural language processing, and characteristic words related to error-reporting keywords are extracted according to a word frequency-inverse file frequency algorithm (TF-IDF algorithm).

(2) Determining the middleware category and the influence range information of the error log by using a knowledge graph technology, and extracting the keyword information again based on the middleware category to obtain target keyword information;

(3) the method comprises the steps that a Bayesian algorithm in machine learning is utilized to sort and summarize the classes of the middleware, the influence range information and the target keyword information to form structured data;

(4) calculating the fault grade of the error log and the production accident generation probability based on the middleware category, the influence range information and the target keyword information by using a Bayesian algorithm again as fault information;

(5) and searching the decision tree shown in the figure 4 based on the fault grade and the production accident generation probability to obtain a decision result of whether the application needs to be restarted.

The TF-IDF algorithm (Term Frequency-Inverse Document Frequency algorithm) is a weighting technique for information retrieval and text mining, and is used to evaluate the importance of a word to a Document set or a Document in a corpus. The importance of a word increases in proportion to the number of times it appears in a document, but at the same time decreases in inverse proportion to the frequency with which it appears in the corpus. The main idea of TF-IDF is: if a word appears in an article with a high frequency TF and rarely appears in other articles, the word or phrase is considered to have a good classification capability and is suitable for classification.

The word frequency (TF) represents the frequency with which terms (keywords) appear in text.

The formula:

wherein n is_ijIs the number of times that the word appears in the file j, and the denominator is the sum of the number of times that all the words appear in the file j, that is:

inverse file frequency (IDF) is a statistic of how many documents in a document set a word occurs. If the documents containing the entry t are fewer, the IDF is larger, and the entry has good category distinguishing capability.

Where | D | is the total number of files in the corpus. I { j: t_i∈d_jDenotes the inclusion of the word t_iNumber of files (i.e., n)_i,jNumber of files not equal to 0). If the word is not in the corpus, it will result in a denominator of 0, so 1+ | { j: ti ∈ dj } |, i.e.:

wherein, the denominator is added with 1 to avoid the denominator being 0.

A high word frequency within a particular document, and a low document frequency for that word across the document collection, may result in a high weighted TF-IDF. Therefore, TF-IDF tends to filter out common words, retain important words, and derive the final formula: TF-IDF ═ TF × IDF.

Because the application is related to the application log, data mining work needs to be done in advance, documents recording error reporting information are integrated into a document set, and weights are given to commonly used characteristic words so as to better position keyword sentences.

In addition, the application supports programs which are operated on different middleware such as traditional nodes and cloud-entering nodes. And searching out matched corresponding relation from the library through the knowledge graph based on the difference of the log data structures for classification.

The application starts from operation and maintenance intellectualization, can serially communicate the application personalized log, the universality error reporting and the corresponding solution, and has the following advantages:

1. and the vacancy that the log monitoring at the application side is insufficient is filled. The pain point that the application running on different platforms is independently warfare is solved by using a unified intelligent operation and maintenance scheme, and a more reliable support guarantee is provided for application transformation.

2. The short plates in the current operation and maintenance work are fully considered, and the specialization of the operation and maintenance is strengthened. By automatically extracting and analyzing the logs, various problems caused by unclear work division duties or manual misoperation and the like are reduced, and the production management is promoted to develop towards a more refined direction.

3. The method combines DevOps (combination words of Development and Operations, which are general terms of a group of processes, methods and systems, and is used for promoting communication, cooperation and integration among Development (application program/software engineering), technical operation and Quality Assurance (QA) departments, and realizes artificial intelligent scheduling of an automation tool through a decision tree building model, thereby greatly shortening the emergency time of operation and maintenance and reducing the risk of production accidents.

It should be understood that, although the steps in the flowcharts related to the embodiments as described above are sequentially displayed as indicated by arrows, the steps are not necessarily performed sequentially as indicated by the arrows. The steps are not performed in the exact order shown and described, and may be performed in other orders, unless explicitly stated otherwise. Moreover, at least a part of the steps in the flowcharts related to the embodiments described above may include multiple steps or multiple stages, which are not necessarily performed at the same time, but may be performed at different times, and the execution order of the steps or stages is not necessarily sequential, but may be rotated or alternated with other steps or at least a part of the steps or stages in other steps.

Based on the same inventive concept, the embodiment of the present application further provides a log processing apparatus for implementing the log processing method. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the method, so specific limitations in one or more log processing device embodiments provided below can refer to the limitations on the log processing method in the foregoing, and details are not described herein again.

In one embodiment, as shown in fig. 7, there is provided a log processing apparatus including: a first extraction module 710, a second extraction module 720, a determination module 730, and a lookup module 740, wherein:

the first extraction module 710 is configured to perform information extraction processing on the log to be processed to obtain keyword information in the log to be processed;

the second extraction module 720 is configured to determine a middleware category corresponding to the log to be processed, and perform extraction processing on the keyword information again according to the middleware category to obtain target keyword information matched with the middleware category; the middleware category is a category of a server for deploying an application program corresponding to the log to be processed;

the determining module 730 is configured to determine, based on the target keyword information, fault information corresponding to the log to be processed;

the searching module 740 is configured to search a pre-constructed decision tree according to the fault information to obtain a decision result corresponding to the log to be processed.

In an embodiment, the first extracting module 710 is specifically configured to extract a plurality of target sample error keywords from a keyword information base based on weights of sample error keywords in a keyword information base constructed in advance; the keyword information base stores a plurality of sample error keywords and weights of the sample error keywords; and according to the error keywords of each target sample, performing information extraction processing on the log to be processed to obtain the keyword information in the log to be processed.

In one embodiment, the apparatus further includes a weight determination module, configured to obtain a plurality of sample error keywords in the sample log; respectively counting the occurrence times of each sample error keyword in the sample log and the cumulative sum of the occurrence times of each sample error keyword; and aiming at any sample error keyword, acquiring the ratio of the occurrence times of the sample error keyword to the accumulated sum of the occurrence times as the weight of the sample error keyword.

In an embodiment, the determining module 730 is specifically configured to process the target keyword information through the trained classification model to obtain fault information of the log to be processed; the classification model is obtained by training sample data which takes target keyword information as input and takes fault information as output.

In an embodiment, the determining module 730 is further configured to determine influence range information of the log to be processed; and determining fault information corresponding to the log to be processed based on the influence range information and the target keyword information.

In an embodiment, the searching module 740 is specifically configured to sort the fault indicators in the fault information according to a predetermined fault level; the fault information comprises a plurality of fault indexes and index values of the fault indexes; searching the decision tree layer by layer according to the sequence of the fault grades of the fault indexes from high to low to obtain leaf nodes corresponding to the fault information in the decision tree; and determining a decision result corresponding to the leaf node as a decision result corresponding to the log to be processed.

The respective modules in the log processing apparatus described above may be implemented in whole or in part by software, hardware, and a combination thereof. The modules can be embedded in a hardware form or independent from a processor in the computer device, and can also be stored in a memory in the computer device in a software form, so that the processor can call and execute operations corresponding to the modules.

In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as shown in fig. 8. The computer device includes a processor, a memory, a communication interface, a display screen, and an input device connected by a system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device comprises a nonvolatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of an operating system and computer programs in the non-volatile storage medium. The communication interface of the computer device is used for carrying out wired or wireless communication with an external terminal, and the wireless communication can be realized through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. The computer program is executed by a processor to implement a log processing method. The display screen of the computer equipment can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer equipment can be a touch layer covered on the display screen, a key, a track ball or a touch pad arranged on the shell of the computer equipment, an external keyboard, a touch pad or a mouse and the like.

Those skilled in the art will appreciate that the architecture shown in fig. 8 is merely a block diagram of some of the structures associated with the disclosed aspects and is not intended to limit the computing devices to which the disclosed aspects apply, as particular computing devices may include more or less components than those shown, or may combine certain components, or have a different arrangement of components.

In one embodiment, a computer device is further provided, which includes a memory and a processor, the memory stores a computer program, and the processor implements the steps of the above method embodiments when executing the computer program.

In an embodiment, a computer-readable storage medium is provided, on which a computer program is stored which, when being executed by a processor, carries out the steps of the above-mentioned method embodiments.

In an embodiment, a computer program product is provided, comprising a computer program which, when being executed by a processor, carries out the steps of the above-mentioned method embodiments.

It should be noted that, the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, presented data, etc.) referred to in the present application are information and data authorized by the user or sufficiently authorized by each party.

It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by hardware instructions of a computer program, which can be stored in a non-volatile computer-readable storage medium, and when executed, can include the processes of the embodiments of the methods described above. Any reference to memory, database, or other medium used in the embodiments provided herein may include at least one of non-volatile and volatile memory. The nonvolatile Memory may include Read-Only Memory (ROM), magnetic tape, floppy disk, flash Memory, optical Memory, high-density embedded nonvolatile Memory, resistive Random Access Memory (ReRAM), Magnetic Random Access Memory (MRAM), Ferroelectric Random Access Memory (FRAM), Phase Change Memory (PCM), graphene Memory, and the like. Volatile Memory can include Random Access Memory (RAM), external cache Memory, and the like. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM), among others. The databases referred to in various embodiments provided herein may include at least one of relational and non-relational databases. The non-relational database may include, but is not limited to, a block chain based distributed database, and the like. The processors referred to in the embodiments provided herein may be general purpose processors, central processing units, graphics processors, digital signal processors, programmable logic devices, quantum computing based data processing logic devices, etc., without limitation.

The technical features of the above embodiments can be arbitrarily combined, and for the sake of brevity, all possible combinations of the technical features in the above embodiments are not described, but should be considered as the scope of the present specification as long as there is no contradiction between the combinations of the technical features.

The above-mentioned embodiments only express several embodiments of the present application, and the description thereof is more specific and detailed, but not construed as limiting the scope of the present application. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the concept of the present application, which falls within the scope of protection of the present application. Therefore, the protection scope of the present application shall be subject to the appended claims.

Claims

1. A method of log processing, the method comprising:

2. The method according to claim 1, wherein the performing information extraction processing on the log to be processed to obtain the keyword information in the log to be processed comprises:

3. The method of claim 2, wherein the weight of the sample error key is determined by:

obtaining a plurality of sample error keywords in a sample log;

4. The method according to claim 1, wherein the determining fault information corresponding to the log to be processed based on the target keyword information includes:

5. The method according to claim 1, wherein the determining fault information corresponding to the log to be processed based on the target keyword information further comprises:

determining the influence range information of the log to be processed;

and determining fault information corresponding to the log to be processed based on the influence range information and the target keyword information.

6. The method of claim 1, wherein the fault information includes information for a plurality of fault indicators; the step of searching a pre-constructed decision tree according to the fault information to obtain a decision result corresponding to the log to be processed includes:

sorting all fault indexes in the fault information according to a preset fault grade; the fault information comprises a plurality of fault indexes and index values of the fault indexes;

searching the decision tree layer by layer according to the sequence of the fault grades of the fault indexes from high to low to obtain leaf nodes corresponding to the fault information in the decision tree;

and determining a decision result corresponding to the leaf node as a decision result corresponding to the log to be processed.

7. A log processing apparatus, characterized in that the apparatus comprises:

8. A computer device comprising a memory and a processor, the memory storing a computer program, wherein the processor implements the steps of the method of any one of claims 1 to 6 when executing the computer program.

9. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the method of any one of claims 1 to 6.

10. A computer program product comprising a computer program, characterized in that the computer program realizes the steps of the method of any one of claims 1 to 6 when executed by a processor.