CN114328395A - Log analysis method, log analysis device and computer-readable storage medium - Google Patents

Log analysis method, log analysis device and computer-readable storage medium Download PDF

Info

Publication number
CN114328395A
CN114328395A CN202111558366.7A CN202111558366A CN114328395A CN 114328395 A CN114328395 A CN 114328395A CN 202111558366 A CN202111558366 A CN 202111558366A CN 114328395 A CN114328395 A CN 114328395A
Authority
CN
China
Prior art keywords
log
request
log analysis
analysis
service
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202111558366.7A
Other languages
Chinese (zh)
Inventor
姚世荣
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Jiangsu Yuncongxihe Artificial Intelligence Co ltd
Original Assignee
Jiangsu Yuncongxihe Artificial Intelligence Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Jiangsu Yuncongxihe Artificial Intelligence Co ltd filed Critical Jiangsu Yuncongxihe Artificial Intelligence Co ltd
Priority to CN202111558366.7A priority Critical patent/CN114328395A/en
Publication of CN114328395A publication Critical patent/CN114328395A/en
Pending legal-status Critical Current

Links

Images

Landscapes

  • Debugging And Monitoring (AREA)

Abstract

The invention relates to the technical field of computer processing, in particular to a log analysis method, a log analysis device and a computer readable storage medium, and aims to solve the technical problem of improving the accuracy rate of log analysis. To this end, the method of the invention comprises: acquiring a log analysis request; using a search analysis engine, inquiring a service standard output log and a network request log related to a log analysis request from a database corresponding to the search analysis engine; and outputting the log and the network request log according to the service standard related to the log analysis request, and generating a log analysis result matched with the log analysis request. According to the invention, the service standard output log and the network request log are used for analysis at the same time, namely, the response condition of the business service to the user request can be combined with the context of the user request for analysis, and the accuracy of log analysis can be greatly improved.

Description

Log analysis method, log analysis device and computer-readable storage medium
Technical Field
The invention relates to the technical field of computers, and particularly provides a log analysis method and device and a computer-readable storage medium.
Background
In the technical field of computers, log analysis is an important way for knowing system hardware information, load conditions and safety states, and realizing problem troubleshooting and problem solving. The method generally adopted by developers and operation and maintenance personnel to realize log analysis is to log in a target node and check a specific log in a specific time interval. However, the method has a great disadvantage that if the operation and maintenance object is a cluster formed by tens, hundreds or even thousands of machines, machine logs are distributed at a plurality of nodes, so that the operation mode is very complicated and low-efficiency, a centralized log management platform is realized in an enterprise with a certain scale, a comprehensive cluster log management function of the system is provided, and the method is necessary and important for improving efficiency and reducing the work complexity.
The centralized log management platform realizes centralized management of a large amount of log data and also improves the complexity and difficulty of log analysis, so that how to improve the accuracy of log analysis becomes a technical problem to be solved.
Disclosure of Invention
In order to overcome the above-mentioned drawbacks, the present invention is proposed to provide a log analysis method, an apparatus and a computer-readable storage medium that solve or at least partially solve the technical problem of how to improve the accuracy rate of log analysis.
In a first aspect, the present invention provides a method of log analysis, the method comprising:
acquiring a log analysis request;
using a search analysis engine, inquiring a service standard output log and a network request log related to the log analysis request from a database corresponding to the search analysis engine, wherein the database stores a plurality of service standard output logs and a plurality of network request logs from service nodes, the plurality of service standard output logs are used for recording the response condition of business services of the service nodes to different user requests, and the plurality of network request logs are used for recording the context of different user requests;
and outputting a log and a network request log according to the service standard related to the log analysis request, and generating a log analysis result matched with the log analysis request.
In one technical solution of the log analysis method, the step of "calling a search analysis engine and querying a service standard output log and a network request log related to the log analysis request from a database corresponding to the search analysis engine" includes:
identifying a target user request related to the log analysis request, and inquiring a corresponding service standard output log and a corresponding network request log according to the unique code of the target user request by using a search analysis engine to serve as the service standard output log and the network request log related to the log analysis request;
and/or, further comprising:
when the service node receives a new user request, acquiring a unique code of the new user request from the context of the new user request, and adding the unique code into a service standard output log responding to the new user request;
and/or the step of "obtaining a log analysis request" specifically includes:
when the service node finds that the service node has an error, identifying a user request causing the error, and generating the log analysis request according to the unique code of the user request causing the error;
and/or, further comprising:
and generating a solution for eliminating errors according to the log analysis result.
In a second aspect, there is provided a log analysis apparatus, the apparatus comprising:
the request acquisition module is used for acquiring a log analysis request;
a log analysis module, which uses a search analysis engine to query a service standard output log and a network request log related to the log analysis request from a database corresponding to the search analysis engine, wherein the database stores a plurality of service standard output logs and a plurality of network request logs from service nodes, the plurality of service standard output logs are used for recording the response situation of the service of the service nodes to different user requests, and the plurality of network request logs are used for recording the context of different user requests;
and the result generation module is used for outputting a log and a network request log according to the service standard related to the log analysis request and generating a log analysis result matched with the log analysis request.
In one technical solution of the log analysis apparatus, the service standard output logs and the network request logs both carry unique codes of different user requests, the log analysis module identifies a target user request related to the log analysis request, and a search analysis engine is used to query corresponding service standard output logs and network request logs according to the unique code of the target user request, and the service standard output logs and the network request logs are used as service standard output logs and network request logs related to the log analysis request;
and/or, further comprising:
the code adding module is used for acquiring the unique code of the new user request from the context of the new user request and adding the unique code into a service standard output log responding to the new user request when the service node receives the new user request;
and/or
When finding that the service node has an error in the service, the request acquisition module identifies a user request causing the error, and generates the log analysis request according to a unique code of the user request causing the error;
and/or, further comprising:
and the error solving module generates a solution for eliminating errors according to the log analysis result.
In a third aspect, a control device is provided, which comprises a processor and a storage device, wherein the storage device is adapted to store a plurality of program codes, and the program codes are adapted to be loaded and run by the processor to perform the log analysis method according to any one of the above-mentioned technical aspects of the log analysis method.
In a fourth aspect, a computer readable storage medium is provided, in which a plurality of program codes are stored, the program codes being adapted to be loaded and executed by a processor to perform the above-mentioned log analyzing method according to any one of the above-mentioned log analyzing method.
One or more technical schemes of the invention at least have one or more of the following beneficial effects:
in one embodiment of the present invention, the log analysis method may include the following steps: acquiring a log analysis request; using a search analysis engine, inquiring a service standard output log and a network request log related to a log analysis request from a database corresponding to the search analysis engine, wherein the database stores a plurality of service standard output logs and a plurality of network request logs from service nodes, the plurality of service standard output logs are used for recording the response conditions of business services of the service nodes to different user requests, in the prior art, the logs are only output based on the service standard to carry out log analysis, and the obtained analysis conclusion has great uncertainty, while the invention simultaneously uses the network request logs to carry out analysis, the response condition of the business service to the user request can be combined with the context of the user request for analysis, and the accuracy of log analysis can be greatly improved.
Drawings
The disclosure of the present invention will become more readily understood with reference to the accompanying drawings. As is readily understood by those skilled in the art: these drawings are for illustrative purposes only and are not intended to constitute a limitation on the scope of the present invention. Wherein:
FIG. 1 is a flow diagram illustrating the main steps of a log analysis method according to an embodiment of the present invention;
FIG. 2 is a flow diagram illustrating the main steps of a log analysis method according to an embodiment of the present invention;
FIG. 3 is a schematic workflow diagram of a log analysis method according to one embodiment of the invention;
FIG. 4 is a schematic workflow diagram of a log analysis method according to one embodiment of the invention;
FIG. 5 is a block diagram illustrating the main structure of a log analysis device according to another embodiment of the present invention;
fig. 6 is a main block diagram of a log analysis apparatus according to another embodiment of the present invention.
Detailed Description
Some embodiments of the invention are described below with reference to the accompanying drawings. It should be understood by those skilled in the art that these embodiments are only for explaining the technical principle of the present invention, and are not intended to limit the scope of the present invention.
In the description of the present invention, a "module" or "processor" may include hardware, software, or a combination of both. A module may comprise hardware circuitry, various suitable sensors, communication ports, memory, may comprise software components such as program code, or may be a combination of software and hardware. The processor may be a central processing unit, microprocessor, image processor, digital signal processor, or any other suitable processor. The processor has data and/or signal processing functionality. The processor may be implemented in software, hardware, or a combination thereof. Non-transitory computer readable storage media include any suitable medium that can store program code, such as magnetic disks, hard disks, optical disks, flash memory, read-only memory, random-access memory, and the like. The term "a and/or B" denotes all possible combinations of a and B, such as a alone, B alone or a and B. The term "at least one A or B" or "at least one of A and B" means similar to "A and/or B" and may include only A, only B, or both A and B. The singular forms "a", "an" and "the" may include the plural forms as well.
The following describes the techniques used in the embodiments of the present invention, and it should be noted that the following specific techniques are only examples to describe the technical solutions of the embodiments of the present invention in detail, and do not limit the present invention, that is, the technical solutions of the present invention can be implemented by using the similar techniques of the following techniques.
Filebeat: filebeat is a lightweight transmitter used to forward and concentrate log data. It is installed as a proxy on the server, monitors the specified log files or locations, collects log events, and forwards them to the Elasticsearch or logstack for indexing.
Packetbeat: packetbeat is a real-time network packet analyzer that can be used with the Elasticsearch to provide an application monitoring and performance analysis system.
Logstash: logstash is an open server-side data processing pipeline that can collect data from multiple sources, transform the data, and then send the data into a "repository".
Elastic search: the Elasticsearch is a distributed search and analysis engine used for storing, analyzing and aggregating data sent by Filebeat, Packetbeat and Logstash.
Kibana: kibana is a source-opened front-end application program, which is based on an Elastic Stack and can provide searching and data visualization functions for data indexed in an Elastic search.
Kafka: kafka is an open source stream processing platform developed by the Apache software foundation, written in Scala and Java. Kafka is a high throughput distributed publish-subscribe messaging system that uses its high throughput feature primarily in the present invention to buffer data sent by filebeats and packetbeats.
Grafana: grafana is an open source metric analysis and visualization suite. A front-end tool developed by pure Javascript displays a self-defined report, a display chart and the like through accessing a library (such as InfluxDB). Most use is made in the monitoring of time series data, like Kibana. Grafana's UI is more flexible, has abundant plug-ins, and is powerful.
Referring to fig. 1, fig. 1 is a flow chart illustrating main steps of a log analysis method according to an embodiment of the invention. As shown in fig. 1, the log analysis method in the embodiment of the present invention mainly includes the following steps:
step S110, a log analysis request is acquired.
The log analysis request can be a request made manually or automatically by a machine, and the request can carry any analysis condition.
Step S120, using the search analysis engine to query a service standard output log and a network request log related to the log analysis request from a database corresponding to the search analysis engine, where the database stores a plurality of service standard output logs and a plurality of network request logs from the service node, the plurality of service standard output logs are used to record response conditions of the service of the service node to different user requests, and the plurality of network request logs are used to record contexts of the different user requests.
The present embodiment may adopt Filebeat as a proxy to run on a service node (i.e., a server) where a service component with a business service capability (which may be a web service) is located, collect service standard output logs, and distribute the service standard output logs to different Kafka topics according to different log categories (Topic is a basic unit of Kafka data write operation). Because Filebeat is focused on a 'log collection' thing, the method is simple and efficient, only occupies few resources of a host machine, and can ensure good performance and high throughput. In the embodiment, a port exposed by the packet interception service is used, and the request context of the user side is collected in real time in a packet capturing mode.
After the collection end collects the logs, the collection end needs to perform buffering and distribution through the message queue. The embodiment adopts Kafka as a buffering and distributing station, and can play a role in peak clipping. Although the Kafka cluster is in a shared state, the Topic is allocated to different logs according to the category to ensure information isolation, and the read-write of the Topic is authenticated and authorized in consideration of the security and privacy problems. Because logs from different sources have respective formats, and the contents are relatively messy and inconvenient to understand and analyze, before the logs are put in storage, the system can process the logs one by adopting Logstash, and uniformly preprocesses the logs by depending on a Filter of the Logstash to extract effective information from each log.
The present embodiment employs an Elasticsearch as a search analysis engine. The Elasticsearch is an extensible distributed full-text search and analysis engine, and by using innovative technologies such as hierarchical storage and off-heap memory management, the availability and the robustness of the system are greatly improved, and the influence of GC problems on the system can be effectively avoided.
Step S130, outputting the log and the network request log according to the service standard related to the log analysis request, and generating a log analysis result matched with the log analysis request.
The embodiment adopts Kibana as a log operation and maintenance interface of the user. Kibana provides a friendly visual analysis platform, and log retrieval and query are realized. A report can be established on the basis of the Kibana interface, log data are processed into different forms, charts and other information, so that advanced contents are displayed, and visual daily analysis of the log is realized.
In the prior art, when the backend service returns a response (i.e., an unsuccessful response) with a status code of not 200, a developer generally determines an error cause by querying a process log or an Nginx (a high-performance HTTP and a reverse proxy web server) log (both belonging to a service standard output log), which may have a very large uncertainty because a final conclusion obtained by a request context at a user side cannot be timely obtained, and may even ask for request data of the user during a debugging process to simulate a request of the user, thereby undoubtedly further increasing complexity and difficulty of debugging. In the embodiment, a port exposed by the Web service is intercepted by using the Packetbeat, the request context of the user side is collected in real time in a packet capturing mode, once the user request is wrong, the request data of the user can be checked in time, and the log is output by combining the service standard collected by the Filebeat, so that the log analysis is more easily and accurately carried out.
Referring to fig. 2, fig. 2 is a flow chart illustrating the main steps of a log analysis method according to an embodiment of the invention. As shown in fig. 2, the log analysis method in the embodiment of the present invention mainly includes the following steps:
step S210, when an error occurs in the service of the service node, identifying the user request causing the error, and generating a log analysis request according to the unique code of the user request causing the error.
When a business service error is found, the user request causing the error is automatically identified, and a log analysis request is generated based on the unique code of the user request, so that the error condition can be timely analyzed through the log when the business service error occurs.
Step S220, identifying a target user request related to a log analysis request, using a search analysis engine to query a corresponding service standard output log and a network request log according to a unique code of the target user request, wherein the service standard output log and the network request log are used as the service standard output log and the network request log related to the log analysis request, a plurality of service standard output logs and a plurality of network request logs from a service node are stored in a database, the plurality of service standard output logs are used for recording the response condition of the service of the service node to different user requests, the plurality of network request logs are used for recording the contexts of different user requests, and the plurality of service standard output logs and the plurality of network request logs all carry the unique codes of the different user requests.
As the operation time of the business system increases, the number of collected logs is more and more, and the occupied physical resources can increase in geometric multiples. More hardware is required to be invested for an enterprise to save log data, and extra energy is required for operation and maintenance personnel to migrate and archive old data. In the embodiment, the distributed elastic search is adopted as the data storage component to effectively solve the two pain points, firstly, the component naturally has distributed capability and is extremely easy to horizontally expand, once the operation and maintenance personnel with insufficient resources need to automatically balance data by calling an API (application programming interface) after adding a new software node, secondly, an ILM (embedded management module) mechanism of the component can automatically archive and delete old data to effectively save storage space, and the operation and maintenance personnel only need to make an archiving strategy according to log types.
The process output log and the network request log both contain unique codes (requestId), if the values of the unique codes are the same, the process output log and the network request log are generated by the same request, and the incidence relation can greatly improve the efficiency when being applied to a fault debugging scene.
Step S230, outputting the log and the network request log according to the service standard related to the log analysis request, and generating a log analysis result matched with the log analysis request.
When the log analysis result reflects that the user needs to be warned currently, the embodiment adopts Grafana as an abnormal log warning operation and maintenance system. Grafana provides a flexible and friendly configuration interface, so that the inquiry of an alarm report and the setting of an alarm rule are realized, and meanwhile, Grafana supports the setting of a notification mode which can be very conveniently integrated with a mainstream third-party system or a mainstream component. The user can customize the alarm rules, the alarm modes, the alarm notification personnel and the like through the interface of Grafana.
So far, the service standard output log transmission flow in this embodiment is shown in fig. 3: 1) collecting logs by using filebeat; 2) fileteam converts the log into an event and adds a custom field (including a uniquely encoded requestId); 3) fileteam sends the event to kafka; 4) topic of logstash consuming kafka takes a message (meaning valid log information); 5) logstack filter formats the message to generate a standard log event; 6) logstash outputs a standard log event to the elasticsearch; 7) kibana retrieves an elasticsearch drawing report; 8) grafana polls the elastic search to send an alarm notification according to a predetermined rule. The network request log transmission flow is shown in fig. 4: 1) collecting logs by packetbeat; 2) packetbeat converts the log into an event and adds a custom field; 3) packetbeat sends the event to kafka; 4) topic of logstash consuming kafka gets to message; 5) logstack filter formats the message to generate a standard log event; 6) logstash outputs a standard log event to the elasticsearch; 7) kibana retrieves an elasticsearch drawing report; 8) grafana polls the elastic search to send an alarm notification according to a predetermined rule.
Step S240, when the service node receives the new user request, the unique code of the new user request is obtained from the context of the new user request, and is added to the service standard output log responding to the new user request.
In a distributed cluster architecture, load balancing is usually realized by using Nginx as a front-end agent, and a front-end request is routed to a back-end service through configuring an upstream. In the embodiment, when the user request reaches Nginx, a unique code is generated and injected into the request context and is transmitted to the back-end business service, and when the business service outputs the process log, the unique code of the current request is taken out from the request context and is placed in the log content. The implementation manner of generating the unique request code in the embodiment supports two front-end load components, namely, Nginx and Ingress.
And step S250, generating a solution for eliminating errors according to the log analysis result.
The log analysis result of the embodiment can not only analyze the error, but also generate a technical scheme for solving the error based on the analysis result, and assist the user to quickly eliminate the error.
In the embodiment, a framework of filebed + Kafka + Logtash + elastic search + Kibana + Grafana is adopted, a convenient log query and analysis interface is provided for a user, log retrieval and statistical analysis are realized, a log-based early warning function is realized, the obstacle of performing advanced statistical analysis in a large-scale cluster by a traditional method for executing a Linux command is overcome, a visual monitoring platform is realized, and great convenience is brought to mass log analysis.
Referring to fig. 5, fig. 5 is a schematic diagram of a main structure of a log analysis apparatus according to an embodiment of the present invention. As shown in fig. 5, the log analysis apparatus in the embodiment of the present invention mainly includes the following steps:
the request obtaining module 510 obtains a log analysis request.
The log analysis request can be a request made manually or automatically by a machine, and the request can carry any analysis condition.
And a log analysis module 520, configured to query, by using the search analysis engine, a service standard output log and a network request log related to the log analysis request from a database corresponding to the search analysis engine, where the database stores a plurality of service standard output logs and a plurality of network request logs from the service node, the plurality of service standard output logs are used to record response conditions of the service of the service node to different user requests, and the plurality of network request logs are used to record contexts of the different user requests.
The present embodiment may adopt Filebeat as a proxy to run on a service node (i.e., a server) where a service component with a business service capability (which may be a web service) is located, collect service standard output logs, and distribute the service standard output logs to different Kafka topics according to different log categories (Topic is a basic unit of Kafka data write operation). Because Filebeat is focused on a 'log collection' thing, the method is simple and efficient, only occupies few resources of a host machine, and can ensure good performance and high throughput. In the embodiment, a port exposed by the packet interception service is used, and the request context of the user side is collected in real time in a packet capturing mode.
After the collection end collects the logs, the collection end needs to perform buffering and distribution through the message queue. The embodiment adopts Kafka as a buffering and distributing station, and can play a role in peak clipping. Although the Kafka cluster is in a shared state, the Topic is allocated to different logs according to the category to ensure information isolation, and the read-write of the Topic is authenticated and authorized in consideration of the security and privacy problems. Because logs from different sources have respective formats, and the contents are relatively messy and inconvenient to understand and analyze, before the logs are put in storage, the system can process the logs one by adopting Logstash, and uniformly preprocesses the logs by depending on a Filter of the Logstash to extract effective information from each log.
The present embodiment employs an Elasticsearch as a search analysis engine. The Elasticsearch is an extensible distributed full-text search and analysis engine, and by using innovative technologies such as hierarchical storage and off-heap memory management, the availability and the robustness of the system are greatly improved, and the influence of GC problems on the system can be effectively avoided.
The result generating module 530 outputs a log and a network request log according to the service standard related to the log analysis request, and generates a log analysis result matching the log analysis request.
The embodiment adopts Kibana as a log operation and maintenance interface of the user. Kibana provides a friendly visual analysis platform, and log retrieval and query are realized. A report can be established on the basis of the Kibana interface, log data are processed into different forms, charts and other information, so that advanced contents are displayed, and visual daily analysis of the log is realized.
In the prior art, when the backend service returns a response (i.e., an unsuccessful response) with a status code of not 200, a developer generally determines an error cause by querying a process log or an Nginx (a high-performance HTTP and a reverse proxy web server) log (both belonging to a service standard output log), which may have a very large uncertainty because a final conclusion obtained by a request context at a user side cannot be timely obtained, and may even ask for request data of the user during a debugging process to simulate a request of the user, thereby undoubtedly further increasing complexity and difficulty of debugging. In the embodiment, a port exposed by the Web service is intercepted by using the Packetbeat, the request context of the user side is collected in real time in a packet capturing mode, once the user request is wrong, the request data of the user can be checked in time, and the log is output by combining the service standard collected by the Filebeat, so that the log analysis is more easily and accurately carried out.
Referring to fig. 6, fig. 6 is a schematic diagram of a main structure of a log analysis apparatus according to an embodiment of the present invention. As shown in fig. 6, the log analysis apparatus in the embodiment of the present invention mainly includes the following modules:
the request obtaining module 610, when an error occurs in the service of the service node, identifies the user request causing the error, and generates a log analysis request according to the unique code of the user request causing the error.
When a business service error is found, the user request causing the error is automatically identified, and a log analysis request is generated based on the unique code of the user request, so that the error condition can be timely analyzed through the log when the business service error occurs.
The log analysis module 620 identifies a target user request related to the log analysis request, and uses the search analysis engine to query a corresponding service standard output log and a network request log according to a unique code of the target user request, wherein the service standard output log and the network request log are used as the service standard output log and the network request log related to the log analysis request, a plurality of service standard output logs and a plurality of network request logs from the service node are stored in the database, the plurality of service standard output logs are used for recording the response condition of the service of the service node to different user requests, the plurality of network request logs are used for recording the contexts of the different user requests, and the plurality of service standard output logs and the plurality of network request logs all carry the unique codes of the different user requests.
As the operation time of the business system increases, the number of collected logs is more and more, and the occupied physical resources can increase in geometric multiples. More hardware is required to be invested for an enterprise to save log data, and extra energy is required for operation and maintenance personnel to migrate and archive old data. In the embodiment, the distributed elastic search is adopted as the data storage component to effectively solve the two pain points, firstly, the component naturally has distributed capability and is extremely easy to horizontally expand, once the operation and maintenance personnel with insufficient resources need to automatically balance data by calling an API (application programming interface) after adding a new software node, secondly, an ILM (embedded management module) mechanism of the component can automatically archive and delete old data to effectively save storage space, and the operation and maintenance personnel only need to make an archiving strategy according to log types.
The process output log and the network request log both contain unique codes (requestId), if the values of the unique codes are the same, the process output log and the network request log are generated by the same request, and the incidence relation can greatly improve the efficiency when being applied to a fault debugging scene.
The result generating module 630 outputs a log and a network request log according to the service standard related to the log analysis request, and generates a log analysis result matched with the log analysis request.
When the log analysis result reflects that the user needs to be warned currently, the embodiment adopts Grafana as an abnormal log warning operation and maintenance system. Grafana provides a flexible and friendly configuration interface, so that the inquiry of an alarm report and the setting of an alarm rule are realized, and meanwhile, Grafana supports the setting of a notification mode which can be very conveniently integrated with a mainstream third-party system or a mainstream component. The user can customize the alarm rules, the alarm modes, the alarm notification personnel and the like through the interface of Grafana.
So far, the technical solution of this embodiment implements a centralized cluster log analysis service, which may be used for log analysis of basic components such as k8s, mysql, zk, consul, redis, etc., and may also process a process log and a network request log of programming languages such as java, golang, python, etc., and the flow thereof is as follows: 1, Filebeat collects service standard output logs, Packetbeat collects network request logs, and the Filebeat and the Packetbeat send collected data to a Kafka cluster; 2, Kafka receives data persistence storage sent by the beat component; logstash consumes Topic of Kafka, and sends to Elasticisearch after cleaning the conversion data; 4, the Elasticissearch receives data persistence storage sent by the Logstash; 5, Kibana draws a log report by using an elastic search as a data source; and 6, Grafana polls the elastic search to send an alarm notice according to a preset rule.
The code adding module 640 obtains the unique code of the new user request from the context of the new user request when the service node receives the new user request, and adds the unique code into a service standard output log responding to the new user request.
In a distributed cluster architecture, load balancing is usually realized by using Nginx as a front-end agent, and a front-end request is routed to a back-end service through configuring an upstream. In the embodiment, when the user request reaches Nginx, a unique code is generated and injected into the request context and is transmitted to the back-end business service, and when the business service outputs the process log, the unique code of the current request is taken out from the request context and is placed in the log content. The implementation manner of generating the unique request code in the embodiment supports two front-end load components, namely, Nginx and Ingress.
The error resolution module 650 generates a solution for eliminating errors according to the log analysis result.
The log analysis result of the embodiment can not only analyze the error, but also generate a technical scheme for solving the error based on the analysis result, and assist the user to quickly eliminate the error.
In the embodiment, a framework of filebed + Kafka + Logtash + elastic search + Kibana + Grafana is adopted, a convenient log query and analysis interface is provided for a user, log retrieval and statistical analysis are realized, a log-based early warning function is realized, the obstacle of performing advanced statistical analysis in a large-scale cluster by a traditional method for executing a Linux command is overcome, a visual monitoring platform is realized, and great convenience is brought to mass log analysis.
The log analysis apparatus shown in fig. 5 to 6 is used for executing the log analysis method embodiments shown in fig. 1 to 2, and the technical principles, the solved technical problems and the generated technical effects of the two are similar, and it can be clearly understood by those skilled in the art that, for convenience and simplicity of description, the specific working process and related description of the log analysis apparatus may refer to the content described in the log analysis method embodiments, and no further description is given here.
It will be understood by those skilled in the art that all or part of the flow of the method according to the above-described embodiment may be implemented by a computer program, which may be stored in a computer-readable storage medium and used to implement the steps of the above-described embodiments of the method when the computer program is executed by a processor. Wherein the computer program comprises computer program code, which may be in the form of source code, object code, an executable file or some intermediate form, etc. The computer-readable storage medium may include: any entity or device capable of carrying said computer program code, media, usb disk, removable hard disk, magnetic diskette, optical disk, computer memory, read-only memory, random access memory, electrical carrier wave signals, telecommunication signals, software distribution media, etc. It should be noted that the computer readable storage medium may contain content that is subject to appropriate increase or decrease as required by legislation and patent practice in jurisdictions, for example, in some jurisdictions, computer readable storage media that does not include electrical carrier signals and telecommunications signals in accordance with legislation and patent practice.
Furthermore, the invention also provides a control device. In an embodiment of the control device according to the present invention, the control device comprises a processor and a storage device, the storage device may be configured to store a program for performing the log analysis method of the above-mentioned method embodiment, and the processor may be configured to execute the program in the storage device, the program including but not limited to the program for performing the log analysis method of the above-mentioned method embodiment. For convenience of explanation, only the parts related to the embodiments of the present invention are shown, and details of the specific techniques are not disclosed. The control device may be a control device apparatus formed including various electronic apparatuses.
Further, the invention also provides a computer readable storage medium. In one computer-readable storage medium embodiment according to the present invention, a computer-readable storage medium may be configured to store a program that executes the log analysis method of the above-described method embodiment, and the program may be loaded and executed by a processor to implement the above-described log analysis method. For convenience of explanation, only the parts related to the embodiments of the present invention are shown, and details of the specific techniques are not disclosed. The computer readable storage medium may be a storage device formed by including various electronic devices, and optionally, the computer readable storage medium is a non-transitory computer readable storage medium in the embodiment of the present invention.
Further, it should be understood that, since the configuration of each module is only for explaining the functional units of the apparatus of the present invention, the corresponding physical devices of the modules may be the processor itself, or a part of software, a part of hardware, or a part of a combination of software and hardware in the processor. Thus, the number of individual modules in the figures is merely illustrative.
Those skilled in the art will appreciate that the various modules in the apparatus may be adaptively split or combined. Such splitting or combining of specific modules does not cause the technical solutions to deviate from the principle of the present invention, and therefore, the technical solutions after splitting or combining will fall within the protection scope of the present invention.
So far, the technical solutions of the present invention have been described in connection with the preferred embodiments shown in the drawings, but it is easily understood by those skilled in the art that the scope of the present invention is obviously not limited to these specific embodiments. Equivalent changes or substitutions of related technical features can be made by those skilled in the art without departing from the principle of the invention, and the technical scheme after the changes or substitutions can fall into the protection scope of the invention.

Claims (12)

1. A method of log analysis, the method comprising:
acquiring a log analysis request;
using a search analysis engine, inquiring a service standard output log and a network request log related to the log analysis request from a database corresponding to the search analysis engine, wherein the database stores a plurality of service standard output logs and a plurality of network request logs from service nodes, the plurality of service standard output logs are used for recording the response condition of business services of the service nodes to different user requests, and the plurality of network request logs are used for recording the context of different user requests;
and outputting a log and a network request log according to the service standard related to the log analysis request, and generating a log analysis result matched with the log analysis request.
2. The log analysis method according to claim 1, wherein the step of "calling a search analysis engine and querying a database corresponding to the search analysis engine for the service standard output logs and the network request logs related to the log analysis request" in the service standard output logs and the network request logs each carrying a unique code of the different user request "specifically comprises:
and identifying a target user request related to the log analysis request, and querying a corresponding service standard output log and a corresponding network request log according to the unique code of the target user request by using a search analysis engine to serve as the service standard output log and the network request log related to the log analysis request.
3. The log analysis method of claim 2, further comprising:
and when the service node receives a new user request, acquiring the unique code of the new user request from the context of the new user request, and adding the unique code into a service standard output log responding to the new user request.
4. The log analysis method according to claim 2, wherein the step of obtaining the log analysis request specifically comprises:
and when the service node finds that the business service has an error, identifying a user request causing the error, and generating the log analysis request according to the unique code of the user request causing the error.
5. The log analysis method of claim 4, further comprising:
and generating a solution for eliminating errors according to the log analysis result.
6. An apparatus for log analysis, the apparatus comprising:
the request acquisition module is used for acquiring a log analysis request;
a log analysis module, which uses a search analysis engine to query a service standard output log and a network request log related to the log analysis request from a database corresponding to the search analysis engine, wherein the database stores a plurality of service standard output logs and a plurality of network request logs from service nodes, the plurality of service standard output logs are used for recording the response situation of the service of the service nodes to different user requests, and the plurality of network request logs are used for recording the context of different user requests;
and the result generation module is used for outputting a log and a network request log according to the service standard related to the log analysis request and generating a log analysis result matched with the log analysis request.
7. The log analysis device of claim 6, wherein the service standard output logs and the network request logs each carry a unique code of the different user request, the log analysis module identifies a target user request related to the log analysis request, and the search analysis engine is used to query the corresponding service standard output log and network request log according to the unique code of the target user request, so as to obtain the service standard output log and network request log related to the log analysis request.
8. The log analysis device of claim 7, further comprising:
and the code adding module is used for acquiring the unique code of the new user request from the context of the new user request and adding the unique code into a service standard output log responding to the new user request when the service node receives the new user request.
9. The log analysis device of claim 7,
and when finding that the service of the service node has an error, the request acquisition module identifies the user request causing the error, and generates the log analysis request according to the unique code of the user request causing the error.
10. The log analysis device of claim 9, further comprising:
and the error solving module generates a solution for eliminating errors according to the log analysis result.
11. A control device comprising a processor and a storage device adapted to store a plurality of program codes, characterized in that said program codes are adapted to be loaded and run by said processor to perform the log analysis method according to any one of claims 1 to 5.
12. A computer-readable storage medium having stored therein a plurality of program codes, characterized in that the program codes are adapted to be loaded and executed by a processor to perform the log analysis method of any one of claims 1 to 5.
CN202111558366.7A 2021-12-16 2021-12-16 Log analysis method, log analysis device and computer-readable storage medium Pending CN114328395A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202111558366.7A CN114328395A (en) 2021-12-16 2021-12-16 Log analysis method, log analysis device and computer-readable storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202111558366.7A CN114328395A (en) 2021-12-16 2021-12-16 Log analysis method, log analysis device and computer-readable storage medium

Publications (1)

Publication Number Publication Date
CN114328395A true CN114328395A (en) 2022-04-12

Family

ID=81053511

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202111558366.7A Pending CN114328395A (en) 2021-12-16 2021-12-16 Log analysis method, log analysis device and computer-readable storage medium

Country Status (1)

Country Link
CN (1) CN114328395A (en)

Similar Documents

Publication Publication Date Title
CN110245078B (en) Software pressure testing method and device, storage medium and server
TWI564732B (en) A method and apparatus for monitoring user requests to run in a decentralized system
US8135827B2 (en) Distributed capture and aggregation of dynamic application usage information
CN109710615B (en) Database access management method, system, electronic device and storage medium
CN107370806B (en) HTTP status code monitoring method, device, storage medium and electronic equipment
US11093349B2 (en) System and method for reactive log spooling
CN101997925A (en) Server monitoring method with early warning function and system thereof
CN111752799A (en) Service link tracking method, device, equipment and storage medium
CN105207806A (en) Monitoring method and apparatus of distributed service
US11188443B2 (en) Method, apparatus and system for processing log data
CN111881011A (en) Log management method, platform, server and storage medium
CN108228322B (en) Distributed link tracking and analyzing method, server and global scheduler
EP3384391B1 (en) Real-time change data from disparate sources
US9600523B2 (en) Efficient data collection mechanism in middleware runtime environment
CN110737639A (en) Audit log method, device, computer equipment and storage medium
CN112631879A (en) Data acquisition method and device, computer readable medium and electronic equipment
CN113672452A (en) Method and system for monitoring operation of data acquisition task
CN110011845B (en) Log collection method and system
CN110309206B (en) Order information acquisition method and system
CN114328395A (en) Log analysis method, log analysis device and computer-readable storage medium
Alekseev et al. The BigPanDA self-monitoring alarm system for ATLAS
CN114610689A (en) Method for recording and analyzing request log in distributed environment
CN113392005A (en) Large file processing test method and system
Brim et al. Monitoring extreme-scale Lustre toolkit
CN113778777A (en) Log playback method and device

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination