CN114385452A - Intelligent cluster log monitoring and analyzing method - Google Patents
Intelligent cluster log monitoring and analyzing method Download PDFInfo
- Publication number
- CN114385452A CN114385452A CN202210032074.8A CN202210032074A CN114385452A CN 114385452 A CN114385452 A CN 114385452A CN 202210032074 A CN202210032074 A CN 202210032074A CN 114385452 A CN114385452 A CN 114385452A
- Authority
- CN
- China
- Prior art keywords
- log
- logs
- monitoring
- cluster
- server
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Withdrawn
Links
- 238000000034 method Methods 0.000 title claims abstract description 63
- 238000012544 monitoring process Methods 0.000 title claims abstract description 45
- 230000008569 process Effects 0.000 claims abstract description 16
- 230000002159 abnormal effect Effects 0.000 claims abstract description 12
- 230000006870 function Effects 0.000 claims description 13
- 238000012216 screening Methods 0.000 claims description 12
- 238000004364 calculation method Methods 0.000 claims description 11
- 239000013598 vector Substances 0.000 claims description 10
- 238000007726 management method Methods 0.000 claims description 9
- 230000007246 mechanism Effects 0.000 claims description 5
- 238000013500 data storage Methods 0.000 claims description 4
- 230000002085 persistent effect Effects 0.000 claims description 4
- 230000001360 synchronised effect Effects 0.000 claims description 3
- 238000004422 calculation algorithm Methods 0.000 claims 1
- 238000012423 maintenance Methods 0.000 abstract description 5
- 230000003252 repetitive effect Effects 0.000 abstract description 4
- 238000010586 diagram Methods 0.000 description 9
- 238000004590 computer program Methods 0.000 description 7
- 238000004458 analytical method Methods 0.000 description 6
- 238000012545 processing Methods 0.000 description 4
- 238000013507 mapping Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000004044 response Effects 0.000 description 2
- 238000013473 artificial intelligence Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000007405 data analysis Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000001914 filtration Methods 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000012163 sequencing technique Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3003—Monitoring arrangements specially adapted to the computing system or computing system component being monitored
- G06F11/3006—Monitoring arrangements specially adapted to the computing system or computing system component being monitored where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3065—Monitoring arrangements determined by the means or processing involved in reporting the monitored data
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/1805—Append-only file systems, e.g. using logs or journals to store data
- G06F16/1815—Journaling file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computing Systems (AREA)
- Databases & Information Systems (AREA)
- Quality & Reliability (AREA)
- Data Mining & Analysis (AREA)
- Mathematical Physics (AREA)
- Debugging And Monitoring (AREA)
Abstract
The invention discloses an intelligent cluster log monitoring and analyzing method, which belongs to the technical field of computer information and comprises the following steps: collecting server cluster log information, generating a knowledge base based on the log information, monitoring server cluster logs, and discovering abnormal logs and generating solutions. By using the intelligent cluster log monitoring and analyzing method disclosed by the invention, abnormal logs in the cluster running process can be automatically found, and a solution is automatically generated according to the knowledge base, so that the operation and maintenance efficiency of an administrator is improved, and the repetitive work is reduced. The invention also discloses an intelligent cluster log monitoring and analyzing method which can be rapidly deployed on the server and provides an easy-to-use and convenient operation interface for an administrator.
Description
Technical Field
The invention relates to an intelligent cluster log monitoring and analyzing method, and belongs to the technical field of computer information.
Background
The log contains the running state data of the objects such as equipment, resources, processes, services and the like in the server cluster, is an important basis for monitoring the running state of the software, diagnosing faults and removing faults, and is a main information set which should be concerned by operation and maintenance personnel.
At present, digital technologies represented by internet, big data, artificial intelligence and the like are comprehensively permeated into all fields of economic society, the world enters a digital economic era with the main characteristics of everything interconnection, data driving, software definition, platform support and intelligent domination, and in the digital economic era, a server cluster is used as a support platform of an application system to play an important role.
According to the W3C statistics, the number of Web servers rises exponentially in 28 years: from the units of the 90 s to over 950 million in the 2000 s, while in the 2019 the total number of Web servers has exceeded 17 hundred million 1700 million. In addition, according to IDC (Internet data center) China server market tracking report in the second quarter of 2020, the market delivery of the China servers is 149.89 thousands of servers in the first half of 2020, which is increased by 8.6% on year-by-year basis; the market size is 92.77 billion dollars (about 657.00 billion yuan RMB), and the same ratio is increased by 19.4%. Wherein the market delivery of the X86 server is 148.09 thousands of servers, and the increase of the market delivery is 7.5 percent on year-on-year basis; the market size is 89.44 billion dollars (about 633.41 billion yuan RMB), and the same time increases 19.3%.
The direct consequence of the sharp increase of the number of servers is the increase of the operation and maintenance complexity, the logs generated by the server cluster are also sharply increased along with the increase of the number of the servers, the mode that an administrator manually acquires and checks the server logs and finds abnormal logs is not suitable any more, and the automatic log analysis is needed. The traditional log analysis means is mainly based on a time axis, and the statistical theory and the method are utilized to collect the main characteristic information of the change of the running state along with the time, so that the efficiency is low, and the real-time requirement of the log data analysis in the new era of network development is difficult to adapt. Meanwhile, the traditional log analysis system can only perform simple analysis such as filtering, retrieval, sequencing and the like, and cannot provide a targeted solution according to the abnormal log.
Disclosure of Invention
The invention aims to provide an intelligent cluster log monitoring and analyzing method, which realizes automatic log collection, knowledge base generation, intelligent log monitoring and solution generation. The intelligent cluster log monitoring and analyzing method disclosed by the invention can automatically collect, process and analyze the server cluster logs in the machine room, automatically discover abnormal logs and generate a targeted solution.
In order to realize the purpose, the invention is realized by adopting the following technical scheme:
the invention provides an intelligent cluster log monitoring and analyzing method, which comprises the following steps:
an automatic log collection method, a knowledge base generation method, an intelligent log monitoring method and a solution generation method.
Further, the automatic log collection method comprises the steps of setting a log collection server, distributing log collection clients, caching log data and permanently storing the log data;
specifically, the log collection server is configured to deploy a log collection server, wherein the log collection server has the function of distributing a log collection client, receiving and storing log information from a host of the server to be collected, and a cache database service and a relational database service are deployed in the log collection server;
specifically, the log collection client distributes the log collection client software to each host in the cluster through an IP address, a user name and a password by the log collection server through a remote control protocol, and installs the log collection client, wherein the log collection client is installed to be a system service which is started along with the start of the system and has the function of acquiring log information and sending the log information to the log collection server;
specifically, the log data caching means that the log acquisition client sends log information to a cache database service in the log acquisition server. Because the number of the hosts in the cluster is possibly large, and the logs generated by each host in unit time are also possibly large, in order to ensure the storage efficiency and the data safety, the log data is not directly stored in the relational database in a persistent manner, but is firstly stored in the cache database based on the memory and then synchronized into the relational database.
Further, the log data cache uses a double-cache-area mechanism, and the process is that two queues (a queue A and a queue B) are established, the two queues are empty in the initial state, the maximum length of the queues is set according to the information such as the predicted number of logs, the server performance and the like, the logs are stored in the queue A one by one, if the queue A is full, the log data in the queue A are stored in the relational database one by one, meanwhile, new data are stored in the queue B one by one, when the queue B is full, the logs in the queue B are stored in the relational database one by one, meanwhile, the new data are stored in the queue A one by one, and the like.
The method for generating the knowledge base comprises the steps of log level grouping, log similarity calculation, common content and individual content screening and solution acquisition;
specifically, the grouping of the log level refers to grouping the logs according to the level, wherein the level of the logs is from low to high, namely INFO, WARNING and ERROR, and the INFO is generally used for informing a user that one operation or state is changed; WARNING is generally used to prompt current or future potential problems (e.g., slow response speed, insufficient memory, etc.); ERROR indicates that a problem occurs in operation and needs to be solved by the user. Generally, what needs to be addressed and addressed is an ERROR level log; several examples of logs are as follows:
time＝"2021-11-25T11:24:13.141263169+08:00"level＝error msg＝"Not continuing with pull after error:errors:\ndenied:requested access to the resource is denied\nunauthorized:authentication required\n"
time＝"2021-11-25T11:24:12.509190811+08:00"level＝info msg＝"Ignoring extra error returned from registry:unauthorized:authentication required"
the purpose of the log similarity calculation is to group the logs again according to the log content. Preferably, a method for calculating log similarity includes: mapping two log character strings to be compared into two vectors in a multi-dimensional space, calculating the cosine of an included angle between the two vectors, wherein the larger the other chord values are, the larger the character string similarity is;
specifically, the dimension of the multidimensional space is the number of the two character strings after the characters are deduplicated, the coordinate value in each dimension is the number of times of occurrence of the character corresponding to the dimension, and the formula for calculating the cosine value of the multidimensional vector A, B is as follows:
specifically, the common content and the individual content screening refers to acquiring common content and individual content in the logs, the common content refers to content existing in the grouped logs, the individual content is content only existing in part of the logs, the individual content is generally content such as a file path, a user name and the like, and represents certain states when the current logs are generated, the common content is generally real content to be expressed by the logs, and the common content of the logs needs to be paid attention when log analysis is performed;
preferably, for the log character string A and the log character string B, firstly removing punctuation marks from the log character string A, grouping the log character string A according to spaces, generating a list consisting of a plurality of words, traversing the list, if the word exists in the log character string B, the word is the common content, otherwise, the word is the individual content;
the solution acquisition means automatically retrieves solutions from the internet through a program according to the grouped and screened logs, and establishes a knowledge base of the logs and the solutions.
Specifically, the intelligent log monitoring means that a log collection client monitors all logs on a host, a user can set a monitoring level, and logs equal to or higher than the monitoring level are detected as abnormal logs;
specifically, the solution generation means that when an abnormal log is detected, the log in the knowledge base is retrieved through the log similarity calculation process, the log with the maximum similarity is found, then character string screening is performed, common content and individual content of the log are obtained, and then a corresponding solution is retrieved from the knowledge base and pushed to an administrator.
By using the intelligent cluster log monitoring and analyzing method disclosed by the invention, logs can be automatically collected and a knowledge base is generated, and meanwhile, the functions of intelligent log monitoring and solution generation are provided, so that the operation and maintenance efficiency of an administrator is improved, and the repetitive work of the administrator is reduced. In addition, the invention also provides an intelligent cluster log monitoring and analyzing system which can be rapidly deployed on the server and provides an easy-to-use and convenient operation interface for an administrator.
Compared with the prior art, the invention has the following beneficial effects:
the intelligent cluster log monitoring and analyzing method can realize automatic log collection, automatic knowledge base generation, intelligent cluster log monitoring and automatic solution generation. By using the method and the system disclosed by the invention, frequent repetitive work of an administrator is avoided, and quick and convenient server log management is realized.
Drawings
FIG. 1 is a flow chart illustrating the implementation of a double-caching mechanism according to an embodiment of the present invention;
FIG. 2 is a flow chart of the implementation of the commonality content and the personalized content screening provided by the embodiment of the invention;
FIG. 3 is a system architecture diagram provided by an embodiment of the present invention;
fig. 4 is a flow chart of a system according to an embodiment of the present invention.
Detailed Description
The present invention is further described with reference to the accompanying drawings, and the following examples are only for clearly illustrating the technical solutions of the present invention, and should not be taken as limiting the scope of the present invention.
The invention provides an intelligent cluster log monitoring and analyzing method, which comprises the following steps:
an automatic log collection method, a knowledge base generation method, an intelligent log monitoring method and a solution generation method.
The automatic log collection method comprises the steps of setting a log collection server, distributing log collection clients, caching log data and permanently storing the log data;
specifically, the log collection server is configured to deploy a log collection server, wherein the log collection server has the function of distributing a log collection client, receiving and storing log information from a host of the server to be collected, and a cache database service and a relational database service are deployed in the log collection server;
specifically, the log collection client distributes the log collection client software to each host in the cluster through an IP address, a user name and a password by the log collection server through a remote control protocol, and installs the log collection client, wherein the log collection client is installed to be a system service which is started along with the start of the system and has the function of acquiring log information and sending the log information to the log collection server;
specifically, the log data caching means that the log acquisition client sends log information to a cache database service in the log acquisition server. Because the number of the hosts in the cluster is possibly more and the logs generated by each host in unit time are also possibly more, in order to ensure the storage efficiency and the data safety, the log data are directly stored in the relational database in a persistent manner, but are firstly stored in the cache database based on the memory and then are synchronized into the relational database;
referring to fig. 1, the log data caching uses a double-cache-area mechanism, and the process is as follows:
101) initializing two queues, a primary queue and a secondary queue, and turning to 102);
102) if the collection process is finished, then the process is finished, otherwise, the process goes to 103);
103) receiving log data from the collection client, and turning to 104);
104) if the main queue is full, go to 105), otherwise go to 106);
105) switching the primary queue and the secondary queue, namely, the primary queue becomes the secondary queue, the primary queue becomes the primary queue, and turning to 106);
106) synchronizing the data in the secondary queue to a relational database (the process is executed in parallel with the subsequent process), and turning to 102);
107) deposit log information into the primary queue, go to 102).
The method for generating the knowledge base comprises the steps of log level grouping, log similarity calculation, common content and individual content screening and solution acquisition;
specifically, the grouping of the log level refers to grouping the logs according to the level, wherein the level of the logs is from low to high, namely INFO, WARNING and ERROR, and the INFO is generally used for informing a user that one operation or state is changed; WARNING is generally used to prompt current or future potential problems (e.g., slow response speed, insufficient memory, etc.); ERROR indicates that a problem occurs in operation and needs to be solved by the user. Generally, what needs to be addressed and addressed is an ERROR level log; several examples of logs are as follows:
time＝"2021-11-25T11:24:13.141263169+08:00"level＝error msg＝"Not continuing with pull after error:errors:\ndenied:requested access to the resource is denied\nunauthorized:authentication required\n"
time＝"2021-11-25T11:24:12.509190811+08:00"level＝info msg＝"Ignoring extra error returned from registry:unauthorized:authentication required"
the purpose of the log similarity calculation is to group the logs again according to the log content. Preferably, a method for calculating log similarity includes: mapping two log character strings to be compared into two vectors in a multi-dimensional space, calculating the cosine of an included angle between the two vectors, wherein the larger the other chord values are, the larger the character string similarity is;
specifically, the dimension of the multidimensional space is the number of the two character strings after the characters are deduplicated, the coordinate value in each dimension is the number of times of occurrence of the character corresponding to the dimension, and the formula for calculating the cosine value of the multidimensional vector A, B is as follows:
for example, for the strings ABCDEFG and abcijk, the similarity calculation process is to first obtain the union of two strings, i.e., { 'a', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I', 'J', 'K' }, and then map the two strings into multi-dimensional vectors according to the number of occurrences of the character, which are respectively:
ABCDEFG＝＝>[1,1,1,1,1,1,1,0,0,0,0]
ABCHIJK＝＝>[1,1,1,0,0,0,0,1,1,1,1]
and then calculating cosine values of the included angles of the two vectors, wherein the calculation formula is as follows:
the calculated result is 0.42857142857142855, and thus the similarity between the character string ABCDEFG and the character string abcijk is 0.42857142857142855.
Specifically, the common content and the individual content screening refers to acquiring common content and individual content in the logs, the common content refers to content existing in the grouped logs, the individual content is content only existing in part of the logs, the individual content is generally content such as a file path, a user name and the like, and represents certain states when the current logs are generated, the common content is generally real content to be expressed by the logs, and the common content of the logs needs to be paid attention when log analysis is performed;
preferably, referring to fig. 2, a method for screening the common content and the individual content is as follows:
201) screening and separating the input character string 1, removing symbols such as punctuations and the like in the character string 1, then separating according to spaces, wherein the separated result is a list, and turning to 202);
202) initializing a pointer, pointing to the first element of the list in the initial state, go to 203);
203) initializing result lists (a commonality list and a personality list), wherein the two lists are empty in an initial state and turning to 204);
204) go to 205) if the word pointed to by the current pointer exists in string 2), otherwise go to 207;
205) add the word to the commonality list, go to 206);
206) delete the word pointed to by the pointer in string 2, go to 208);
207) add the word to the personality list, go to 208);
208) if the pointer has pointed to the end of the list, go to 210), otherwise, go to 209);
209) moving the pointer backward by one bit, go to 204);
210) the content of string 2 is added to the personality list.
For example, for the following strings:
not continuing with pull after error errors denied requested access to the resource is denied unauthorized authentication required
not continuing with pull after error context canceled
the result after grouping string a is: [ 'Not', 'connecting', 'with', 'pull', 'after', 'access', 'to', 'the' resource ',' is ',' consumed ',' unauthorized ',' authentication ',' required ',' acquired ',' the list is traversed, if the word is present in the log string B, the word is the common content, and after the traversal, the common content is obtained as: not connecting with pull after error, the individual content is: (ii) errors condensed request access to the resource is condensed authenticated request and context authenticated
The solution acquisition means automatically retrieves solutions from the internet through a program according to the grouped and screened logs, and establishes a knowledge base of the logs and the solutions.
Specifically, the intelligent log monitoring means that a log collection client monitors all logs on a host, a user can set a monitoring level, and logs equal to or higher than the monitoring level are detected as abnormal logs;
specifically, the solution generation means that when an abnormal log is detected, the log in the knowledge base is retrieved through the log similarity calculation process, the log with the maximum similarity is found, then character string screening is performed, common content and individual content of the log are obtained, and then a corresponding solution is retrieved from the knowledge base and pushed to an administrator.
Referring to fig. 3, an implementation architecture of the method includes a management and control service, a cache service, a database service, and a server cluster.
Specifically, the server cluster is a host of logs to be collected, the cache service provides high-speed data storage, the database service provides persistent data storage, and the management and control service provides functions of remote operation of the server cluster, page display, database management and the like.
The management and control service records information such as IP addresses, user names and passwords of all hosts in the server cluster, can remotely operate the hosts in the server cluster through a network, distribute files to the hosts and execute commands, and can distribute the files to all the hosts through the management and control service, and automatically install and start the logs;
meanwhile, the management and control service provides a Web server, an administrator can access a page of the intelligent cluster log monitoring and analyzing system through a browser, the page can check logs on all hosts, when monitoring abnormal logs, the intelligent cluster log monitoring and analyzing system gives an alarm to the administrator to inform the administrator of host information, log information and a retrieved solution of the abnormal logs, and a remote window connected to the host is automatically opened;
by using the intelligent cluster log monitoring and analyzing method disclosed by the invention, logs can be automatically collected and a knowledge base is generated, and meanwhile, the functions of intelligent log monitoring and solution generation are provided, so that the operation and maintenance efficiency of an administrator is improved, and the repetitive work of the administrator is reduced;
as will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein;
the present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks;
these computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks;
these computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
The above description is only a preferred embodiment of the present invention, and it should be noted that, for those skilled in the art, several modifications and variations can be made without departing from the technical principle of the present invention, and these modifications and variations should also be regarded as the protection scope of the present invention.
Claims (8)
1. The intelligent cluster log monitoring and analyzing method is characterized by comprising an automatic log collecting method, a knowledge base generating method, an intelligent log monitoring method, a solution generating method and an intelligent cluster log monitoring and analyzing system;
the automatic log collection method comprises the steps of setting a log collection server, distributing log collection clients, caching log data and permanently storing the log data;
the method for generating the knowledge base comprises the steps of log level grouping, log similarity calculation, common content and individual content screening and solution acquisition;
the intelligent cluster log monitoring and analyzing system comprises a management and control service, a cache service, a database service and a server cluster.
2. The intelligent cluster log monitoring and analyzing method as claimed in claim 1, wherein the log collection server is configured to deploy a log collection server, the log collection server has a function of distributing a log collection client, receiving and storing log information from a host of the server to be collected, and the log collection server is deployed with a cache database service and a relational database service;
the log collection client distribution finger is used for distributing log collection client software to each host in the cluster by the log collection server and installing the log collection client;
the log data caching means that the log acquisition client sends log information to a cache database service in a log acquisition server, firstly, data is stored in a cache database based on a memory, and then, the data is synchronized to a relational database;
the log data permanent storage means that a double-cache mechanism is used for storing the data in the cache database into the relational database.
3. The intelligent cluster log monitoring and analyzing method of claim 1, wherein the log level grouping means that the logs are grouped according to levels;
the purpose of log similarity calculation is to group the logs again according to the log content, calculate the similarity between the logs through a log similarity calculation algorithm, and divide the logs with high similarity into one group;
the common content and the individual content screening refers to acquiring common content and individual content in the logs, wherein the common content refers to content existing in the grouped logs, and the individual content is content only existing in part of the logs;
and the solution acquisition means automatically retrieves the solution from the Internet through a program according to the grouped and screened logs, and establishes a knowledge base of the logs and the solution.
4. The intelligent cluster log monitoring and analyzing method as claimed in claim 1, wherein the intelligent log monitoring means that the log collection client monitors all logs on the host computer, and a user can set a monitoring level, and logs at or above the monitoring level are detected as abnormal logs.
5. The intelligent cluster log monitoring and analyzing method as claimed in claim 2, wherein the double-buffer mechanism comprises establishing two queues (queue a and queue B), wherein the queues are empty in an initial state, the maximum length of the queues is set according to the expected log number, server performance and other information, the logs are stored in the queue a one by one, if the queue a is full, the log data in the queue a is stored in the relational database one by one, and meanwhile, the new data is stored in the queue B one by one, and when the queue B is full, the logs in the queue B are stored in the relational database one by one, and meanwhile, the new data is stored in the queue a one by one, and so on.
6. The intelligent cluster log monitoring and analyzing method of claim 3, wherein the similarity calculation process is that firstly two log character strings of which the similarity needs to be compared are mapped into two vectors in a multidimensional space, the cosine of an included angle between the two vectors is calculated, and the greater the rest chord values are, the greater the character string similarity is.
7. The intelligent cluster log monitoring and analyzing method as claimed in claim 3, wherein the process of screening the common content and the individual content is that for the log character string A and the log character string B, the log character string A is firstly divided into groups according to blank spaces after punctuation marks are removed, a list consisting of a plurality of words is generated, the list is traversed, if the word exists in the log character string B, the word is the common content, otherwise, the word is the individual content.
8. The intelligent cluster log monitoring and analyzing method as claimed in claim 1, wherein the server cluster is a host of logs to be collected, the cache service provides high-speed data storage, the database service provides persistent data storage, and the management and control service provides functions of remotely operating the server cluster, displaying pages, managing databases, and the like.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210032074.8A CN114385452A (en) | 2022-01-12 | 2022-01-12 | Intelligent cluster log monitoring and analyzing method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210032074.8A CN114385452A (en) | 2022-01-12 | 2022-01-12 | Intelligent cluster log monitoring and analyzing method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114385452A true CN114385452A (en) | 2022-04-22 |
Family
ID=81202092
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210032074.8A Withdrawn CN114385452A (en) | 2022-01-12 | 2022-01-12 | Intelligent cluster log monitoring and analyzing method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114385452A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115757041A (en) * | 2022-11-28 | 2023-03-07 | 安超云软件有限公司 | Dynamically configurable multi-cluster log collection method and application |
| CN116627771A (en) * | 2023-07-18 | 2023-08-22 | 中移(苏州)软件技术有限公司 | Log acquisition method, device, electronic equipment and readable storage medium |
| CN117411732A (en) * | 2023-12-15 | 2024-01-16 | 国网四川省电力公司技能培训中心 | Monitoring method and system for network security event |
-
2022
- 2022-01-12 CN CN202210032074.8A patent/CN114385452A/en not_active Withdrawn
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115757041A (en) * | 2022-11-28 | 2023-03-07 | 安超云软件有限公司 | Dynamically configurable multi-cluster log collection method and application |
| CN115757041B (en) * | 2022-11-28 | 2023-11-21 | 安超云软件有限公司 | Method for collecting dynamically configurable multi-cluster logs and application |
| CN116627771A (en) * | 2023-07-18 | 2023-08-22 | 中移(苏州)软件技术有限公司 | Log acquisition method, device, electronic equipment and readable storage medium |
| CN116627771B (en) * | 2023-07-18 | 2023-10-13 | 中移(苏州)软件技术有限公司 | Log acquisition method, device, electronic equipment and readable storage medium |
| CN117411732A (en) * | 2023-12-15 | 2024-01-16 | 国网四川省电力公司技能培训中心 | Monitoring method and system for network security event |
| CN117411732B (en) * | 2023-12-15 | 2024-03-22 | 国网四川省电力公司技能培训中心 | Monitoring method and system for network security event |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN114385452A (en) | Intelligent cluster log monitoring and analyzing method | |
| US11238069B2 (en) | Transforming a data stream into structured data | |
| US10984010B2 (en) | Query summary generation using row-column data storage | |
| TWI406152B (en) | Storing log data efficiently while supporting querying | |
| US7107340B2 (en) | System and method for collecting and storing event data from distributed transactional applications | |
| US20070179934A1 (en) | Method and apparatus for performing bulk file system attribute retrieval | |
| CN107273267A (en) | Log analysis method based on elastic components | |
| KR20090100344A (en) | Storing log data efficiently while supporting querying to assist in computer network security | |
| CN102918534A (en) | Query pipeline | |
| CN113656807B (en) | Vulnerability management method, device, equipment and storage medium | |
| WO2020087082A1 (en) | Trace and span sampling and analysis for instrumented software | |
| US20140195502A1 (en) | Multidimension column-based partitioning and storage | |
| CN109669795A (en) | Crash info processing method and processing device | |
| US8090873B1 (en) | Methods and systems for high throughput information refinement | |
| CN111274218A (en) | Multi-source log data processing method for power information system | |
| US11934466B2 (en) | Direct cloud storage intake and upload architecture | |
| CN113672924A (en) | Data intrusion detection method and device of distributed cloud computing system | |
| CN108337100B (en) | Cloud platform monitoring method and device | |
| Cao et al. | LogKV: Exploiting key-value stores for event log processing | |
| US20230195554A1 (en) | Enriched high fidelity metrics | |
| CN109542913B (en) | Network asset safety management method in complex environment | |
| CN112003884A (en) | Network asset acquisition and natural language retrieval method | |
| Pape et al. | Restful correlation and consolidation of distributed logging data in cloud environments | |
| CN114422341A (en) | Industrial control asset identification method and system based on fingerprint characteristics | |
| CN112910842A (en) | Network attack event evidence obtaining method and device based on flow reduction |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WW01 | Invention patent application withdrawn after publication |
Application publication date: 20220422 |
|
| WW01 | Invention patent application withdrawn after publication |