CN108063685B - Log analysis method and device - Google Patents
Log analysis method and device Download PDFInfo
- Publication number
- CN108063685B CN108063685B CN201711278704.5A CN201711278704A CN108063685B CN 108063685 B CN108063685 B CN 108063685B CN 201711278704 A CN201711278704 A CN 201711278704A CN 108063685 B CN108063685 B CN 108063685B
- Authority
- CN
- China
- Prior art keywords
- log
- sequence number
- request
- user terminal
- server
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/06—Management of faults, events, alarms or notifications
- H04L41/069—Management of faults, events, alarms or notifications using logs of notifications; Post-processing of notifications
Abstract
The embodiment of the invention provides a log analysis method and a log analysis device, which are applied to a cluster system, wherein the method comprises the following steps: a target server for processing a user terminal service request in a cluster system sends a plurality of log files to a log analysis server in the cluster system, wherein the log files comprise at least one log, and each log comprises a log ID and a request serial number; the log analysis server sorts the logs based on the log ID and the request sequence number in each log to obtain sorted logs; the log analysis server analyzes the sorted logs. According to the scheme provided by the embodiment, the target server sends the log file to the log analysis server, the analysis server sorts the logs according to the log ID and the request sequence number of each log, and sorts the logs with the same log ID according to the request sequence number without manually searching and sorting, so that manpower and time are saved, and the analysis efficiency of the logs is improved.
Description
Technical Field
The present invention relates to the field of communications technologies, and in particular, to a log analysis method and apparatus.
Background
In order to improve the access performance of the system, in the prior art, most systems adopt cluster deployment, that is, the same service is deployed on many different servers, and the access capability of the whole system is improved by distributing the request of the user to the different servers. However, since most services are completed by many request interactions, the requests may be distributed to and logged on different servers. The generated logs cannot be managed in a unified manner, so that the logs are inconvenient to analyze and find the fault reason when the system fails.
Disclosure of Invention
The application describes a log analysis method and device.
In a first aspect, an embodiment of the present invention provides a log analysis method applied to a cluster system, where the method includes: a target server for processing a user terminal service request in the cluster system sends a plurality of log files to a log analysis server in the cluster system, wherein the log files comprise at least one log, and each log comprises a log ID and a request serial number; the log analysis server sorts the logs based on the log ID and the request serial number in each log to obtain sorted logs; the log analysis server analyzes the sorted logs. According to the scheme provided by the embodiment, the target server sends the log file to the log analysis server, the analysis server sorts the logs according to the log ID and the request sequence number of each log, and sorts the logs with the same log ID according to the request sequence number without manually searching and sorting, so that manpower and time are saved, and the analysis efficiency of the logs is improved.
Optionally, the method further comprises a step of generating a log, the step comprising: when processing a service request of a user terminal, the target server sends a request for generating a log ID and a request serial number to a log serial number generation server in the cluster system; and the log sequence number generation server generates the log ID and the request sequence number corresponding to the user terminal according to the user terminal information. The log sequence number generation server generates the request sequence number according to the sequence generated by the service request of the same user terminal, and each log is numbered on the basis of classifying the log ID generated by the service request of the same user terminal, so that the logs can be ordered according to the processing sequence, and the technical problem that the log sequence is different from the actual execution sequence due to the fact that the system time of a plurality of target servers is inconsistent is effectively solved.
Optionally, the step of generating, by the log sequence number generation server, the log ID and the request sequence number corresponding to the user terminal according to the user terminal information includes: the log sequence number generation server inquires whether a log ID corresponding to the user terminal exists according to the user terminal information; when no log ID exists, generating a log ID according to the identification information of the user terminal and the service request processing time, and setting a request serial number as an initial value, wherein the identification information comprises an IP address of the user terminal or an MAC address of the user terminal; and when the log ID exists, the request sequence number is increased according to a preset rule. The logs are numbered, so that the logs generated by the service requests of the same user terminal can be sequenced conveniently, and the technical problem of inaccurate analysis when the service requests of the same user terminal are processed by a plurality of target servers with inconsistent system time is solved. Meanwhile, whether the log serial number generation server generates logs for the service request of the user terminal is confirmed by searching the log ID, so that logs with continuous request serial numbers can be generated conveniently according to the log ID.
Optionally, the method further includes a step of sending, by the target server, the log file to a log analysis server, where the step is implemented by any one of the following ways:
the target server sends a log file to the log analysis server according to a preset time interval; or the target server sends the log file to a log analysis server when the capacity of the log file reaches a preset capacity threshold.
In a second aspect, an embodiment of the present invention further provides a log analysis method, which is applied to a log analysis server in a cluster system, where the method includes: the method comprises the steps that a log analysis server receives a plurality of log files sent by a target server for processing a user terminal service request in a cluster system, wherein the log files comprise at least one log, and each log comprises a log ID and a request serial number; the log analysis server sorts the logs based on the log ID and the request sequence number of each log to obtain sorted logs; the log analysis server analyzes the sorted logs. The analysis server sorts the logs according to the log IDs and the request sequence numbers, sorts the logs with the same log IDs according to the request sequence numbers, does not need to search and sort manually, is convenient to analyze the logs, and improves the analysis efficiency of the logs.
In a third aspect, an embodiment of the present invention further provides a log analysis method, which is applied to a log sequence number generation server in a cluster system, where the method includes: the log sequence number generation server receives a request for generating a log ID and a request sequence number corresponding to a service request sent by a target server in the cluster system when the target server processes the service request of a user terminal; the log sequence number generation server generates a log ID and a request sequence number according to user terminal information; and the log sequence number generation server sends the generated log ID and the request sequence number to the target server, and the target server records the generated log ID and the request sequence number into the current log, so that a log analysis server in the cluster system can conveniently analyze the log generated by the target server. The log sequence number generation server generates the log ID and the request sequence number according to the user terminal information, so that the log analysis server can conveniently perform subsequent log analysis processing, the log analysis efficiency is improved, and meanwhile, the corresponding request sequence number is generated according to the request sequence sent to the log sequence number generation server, so that the technical problem that log sorting errors according to time and the log analysis accuracy are influenced due to system time inconsistency of each target server can be solved.
In a fourth aspect, an embodiment of the present invention further provides a log analysis apparatus, which is applied to a log analysis server in a cluster system, where the apparatus includes: the system comprises a first receiving module, a second receiving module and a third receiving module, wherein the first receiving module is used for receiving a plurality of log files sent by a target server for processing a service request of a user terminal in the cluster system, the log files comprise at least one log, and each log comprises a log ID and a request serial number; the sorting module is used for sorting the logs based on the log ID and the request serial number of each log to obtain sorted logs; an analysis module to analyze the sorted logs. The analysis server sorts the logs according to the log IDs and the request sequence numbers, sorts the logs with the same log IDs according to the request sequence numbers, does not need to search and sort manually, is convenient to analyze the logs, and improves the analysis efficiency of the logs.
In a fifth aspect, an embodiment of the present invention further provides a log analysis device, which is applied to a log sequence number generation server in a cluster system, where the device includes: a second receiving module, configured to receive a request for generating a log ID and a request sequence number corresponding to a service request sent by a target server in the cluster system when the target server processes the service request of a user terminal; the generating module is used for generating a log ID and a request serial number according to the user terminal information; and the sending module is used for sending the generated log ID and the request serial number to the target server, and the target server records the generated log ID and the request serial number into the current log, so that a log analysis server in the cluster system can conveniently perform log analysis on the log generated by the target server. And the user terminal sends the log analysis result to a log analysis server in the cluster system for log analysis. The log sequence number generation server generates the log ID and the request sequence number according to the user terminal information, so that the log analysis server can conveniently perform subsequent log analysis processing, the log analysis efficiency is improved, and meanwhile, the corresponding request sequence number is generated according to the request sequence sent to the log sequence number generation server, so that the technical problem that log sorting errors according to time and the log analysis accuracy are influenced due to system time inconsistency of each target server can be solved.
According to the log analysis method and device provided by the embodiment of the invention, the logs are sequenced through the received log ID and the request serial number in each log, and the sequenced logs are obtained. The log analysis is convenient, the efficiency of log analysis is improved, and a guarantee is provided for rapidly solving system faults.
Additional features and advantages of the invention will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by the practice of the embodiments of the invention. The objectives and other advantages of the invention will be realized and attained by the structure particularly pointed out in the written description and claims hereof as well as the appended drawings.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings needed to be used in the embodiments will be briefly described below, it should be understood that the following drawings only illustrate some embodiments of the present invention and therefore should not be considered as limiting the scope, and for those skilled in the art, other related drawings can be obtained according to the drawings without inventive efforts.
Fig. 1 is a block diagram of a cluster system according to an embodiment of the present invention.
Fig. 2 is a flowchart of a log analysis method according to an embodiment of the present invention.
Fig. 3 is a partial flowchart of the log analysis method before step S240 in fig. 2.
Fig. 4 is a flowchart illustrating the sub-steps of step S230 in fig. 3.
Fig. 5 is a second flowchart of a log analysis method according to an embodiment of the present invention.
Fig. 6 is a third flowchart of a log analysis method according to an embodiment of the present invention.
Fig. 7 is a block diagram of a log analysis apparatus according to an embodiment of the present invention.
Fig. 8 is a second block diagram of the log analysis apparatus according to the embodiment of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. The components of embodiments of the present invention generally described and illustrated in the figures herein may be arranged and designed in a wide variety of different configurations. Thus, the following detailed description of the embodiments of the present invention, presented in the figures, is not intended to limit the scope of the invention, as claimed, but is merely representative of selected embodiments of the invention.
All other embodiments, which can be derived by a person skilled in the art from the embodiments of the present invention without making any creative effort, shall fall within the protection scope of the present invention. It should be noted that: like reference numbers and letters refer to like items in the following figures, and thus, once an item is defined in one figure, it need not be further defined and explained in subsequent figures. Meanwhile, in the description of the present invention, the terms "first", "second", and the like are used only for distinguishing the description, and are not to be construed as indicating or implying relative importance.
The inventor finds that: the traditional log record can disperse related information on different servers, log analysis is troublesome when system failure occurs, log information of all servers needs to be collected, and logs of all the servers can only correspond to each other by time. By analyzing in time sequence, when a plurality of service request flows are mixed together, the system time of a plurality of servers processing the service requests may be inconsistent, and thus, the problem of inaccurate log analysis exists. Meanwhile, different keywords are adopted for different service requests to distinguish, the problem that the same keyword is output by the same service request of different user terminals may exist, and logs cannot be classified accurately.
The above solutions have disadvantages that are the results of practical and careful study, and therefore, the discovery process of the above problems and the solutions proposed by the embodiments of the present invention below to the above problems should be the contributions of the inventors to the present invention in the course of the present invention.
In order to overcome the above-mentioned drawbacks of the prior art, the inventors have studied to provide the following embodiments to provide a solution.
Referring to fig. 1, fig. 1 is a block diagram of a cluster system 10 according to an embodiment of the present invention. The cluster system 10 includes: user terminal 100, balancing device 200, target server 300, log number generation server 400, and log analysis server 500.
In this embodiment, the balancing device 200 may be a single device, or a load balancing operation may be performed by a target server 300, and the log sequence number generating server 400 and the log analyzing server 500 may be respectively deployed on different servers or may be performed by the same server in actual settings.
Referring to fig. 2, a log analysis method provided in an embodiment of the present invention is applied to a cluster system 10, and the method includes the following steps.
In step S240, the target server 300 sends a plurality of log files to the log analysis server 500.
The log file includes at least one log, each log includes a log ID and a request sequence number, and the log ID and the request sequence number are generated by the log sequence number generation server 400. The log ID corresponds to the user terminal 100 that processes the service request, and the request sequence number is used to indicate that the log is the service request order corresponding to the user terminal 100. The log ID and request sequence number are sufficient to distinguish the different logs.
In step S250, the log analysis server 500 sorts the logs based on the log ID and the request sequence number of each log, and obtains the sorted logs.
The log analysis server 500 sorts the logs in the log file sent by the target server 300 according to the log IDs of the logs, and sorts the logs according to the request sequence numbers.
In step S260, the log analysis server 500 analyzes the sorted logs.
In this embodiment, through the sorting in step S250, the operation, analysis, and positioning of each user terminal 100 are clear at a glance, and when a system fails, the corresponding log can be quickly and accurately found, thereby saving the labor and time required for failure analysis.
Through the scheme, the target server 300 sends the log file to the log analysis server 500, the log analysis server 500 sorts the logs according to the log ID and the request sequence number in each log, and sorts the logs with the same log ID according to the request sequence number without manually searching and sorting, so that the logs are conveniently analyzed, and the analysis efficiency of the logs is improved.
Referring to fig. 3, in the present embodiment, the method further includes steps S210 to S230 of generating a log ID and a request sequence number.
In step S210, the balancing device 200 allocates the service request required for completing the service to at least one target server 300 for processing according to the service of the user terminal 100.
In this embodiment, a service of the user terminal 100 is sent to the balancing device 200 for processing, and the balancing device 200 allocates a service request corresponding to the service to at least one target server 300 for processing according to the number of service requests required for completing the service and the current load conditions of all the target servers 300.
For a service requiring multiple service requests to complete, the following service request may need to rely on the processing result of the previous service request, and for this reason, for multiple service requests of the service, the balancing device 200 may send the service requests to the target server 300 in sequence for processing.
In step S220, when processing the service request of the user terminal 100, the target server 300 sends a request for generating a log ID and a request number to the log number generation server 400 in the cluster system 10.
After the target server 300 receives the service request distributed by the balancing apparatus 200, the target server 300 sends a request for generating a log ID and a request sequence number corresponding to the service request to the log sequence number generating server 400.
In step S230, the log sequence number generation server 400 generates the log ID and the request sequence number corresponding to the user terminal 100 according to the user terminal 100 information.
In this embodiment, referring to fig. 4, the step S230 may include the following sub-steps:
in sub-step S231, the log sequence number generating server 400 queries whether a log ID corresponding to the user terminal 100 exists according to the information of the user terminal 100.
Optionally, the log sequence number generating server 400 may query, according to the information of the user terminal 100, whether the log sequence number generating server 400 generates the log ID for the user terminal 100.
If the log ID is generated for the user terminal 100, it is determined that the log ID and the request number corresponding to the user terminal 100 exist, and if the log ID is not generated for the user terminal 100, it is determined that the log ID and the request number corresponding to the user terminal 100 do not exist.
A substep S232, when there is no log ID, generating a log ID according to the identification information of the user terminal 100 and the service request processing time, and setting a request sequence number to an initial value (for example, to 1, or to 0 or any other value); when the log ID exists, the request sequence number is incremented according to a preset rule (for example, 1 may be added each time, or 2 may be added each time, which is not limited in the embodiment of the present invention), and the newly generated log ID and the request sequence number are recorded in the cache of the log sequence number generation server 400.
Wherein the identification information includes an IP address of the user terminal 100 or a MAC address of the user terminal 100. The method for generating the log ID according to the identification information of the user terminal 100 and the service request processing time may be to calculate the log ID by using an MD5 encryption algorithm or other algorithms.
When the log ID exists, the request sequence number is incremented according to a preset rule, and the request sequence number incremented according to the preset rule is recorded in the cache of the log sequence number generation server 400 to replace the previous request sequence number.
After generating the log ID and the request sequence number, the log sequence number generation server 400 also transmits the log ID and the request sequence number to the target server 300 that processes the service request, and saves the log ID and the request sequence number to the current log of the target server 300. The logs in each target server 300 may be saved to a log file to facilitate management of the logs by the target server 300.
In this embodiment, the manner of sending the log file to the log analysis server 500 in step S210 may be that the target server 300 sends the log file to the log analysis server 500 at preset time intervals, or the target server 300 sends the log file to the log analysis server 500 when the capacity of the log file reaches a preset capacity threshold.
In this embodiment, the log sequence number generation server 400 configures all log IDs and request sequence numbers in the log sequence number generation server 400 to be invalid after a preset time (for example, 24 hours). And the log ID is regenerated the next time it is requested. The timing failure log ID and the request serial number can prevent the request serial number from being too large to facilitate sequencing.
The following describes the log analysis method in this embodiment with a specific example, it should be understood that the following example is only for illustrating the implementation process of the log analysis method in this embodiment, and should not be construed as a limitation to this embodiment, and other implementations of this embodiment may also be implemented in other ways, and the specific example is as follows:
assume that the IP address of user terminal a is: 192.168.1.1, the MAC address of user terminal a is: 16-C6-42-BB-8E-26. The user terminal a needs 3 service requests to complete one service, and the service requests are sequentially and respectively loaded to the target server B, the target server a and the target server C through the balancing device 200.
When the first request of the user terminal a reaches the target server B, the target server B acquires the log ID and the request number from the log number generation server 400. The log sequence number generation server 400 queries the local cache according to the terminal information (IP (192.168.1.1)/MAC (16-C6-42-BB-8E-26)) of the user terminal a, and if there is no corresponding log ID at this time. Then the log ID is generated to KIUO09KLOP837L and the request sequence number is generated from the terminal information (terminal IP (192.168.1.1)/terminal MAC (16-C6-42-BB-8E-26)) and time (2017-10-10) using MD5 or other algorithm, setting the initial value to 1. The log sequence number generation server 400 records the log ID (KIUO09KLOP837L) and the request sequence number 1 to the cache, and returns this information to the target server B.
The target server B gets the log ID (KIUO09KLOP837L) and the request sequence number 1, and records this information in the current log. When the second request of the user terminal a reaches the target server a. The target server a acquires the log ID and the request sequence number from the log sequence number generation server 400. The log sequence number generation server 400 queries the local cache according to the terminal information (IP (192.168.1.1)/MAC (16-C6-42-BB-8E-26), when there has been a log ID of KIUO09KLOP837L and the request sequence number is 1. The log sequence number generation server 400 increments the request sequence number by 1 according to a preset rule, that is, the request sequence number is 2. The log sequence number generation server 400 updates the log ID (KIUO09KLOP837L) and the request sequence number 2 to the cache, and returns the log ID (KIUO09KLOP837L) and the request sequence number 2 to the target server a. The target server a gets the log ID of the record (KIUO09KLOP837L) and the request sequence number 2. This information is recorded in the current log.
When the third request of the user terminal a reaches the target server C. The target server C acquires the log ID and the request sequence number from the log sequence number generation server 400. The log sequence number generation server 400 queries the local cache according to the terminal information (IP (192.168.1.1)/MAC (16-C6-42-BB-8E-26), when there has been a log ID of KIUO09KLOP837L and the request sequence number is 2. The log sequence number generation server 400 increments the request sequence number by 1 according to a preset rule, that is, the request sequence number is 3. The log sequence number generation server 400 records the log ID (KIUO09KLOP837L) and the request sequence number 3 to the information before the cache overwriting, and returns the log ID (KIUO09KLOP837L) and the request sequence number 3 to the target server C. The target server C, gets the log ID of the record (KIUO09KLOP837L) and the request sequence number 3. This information is recorded in the current log.
Meanwhile, the terminal IP address of the user terminal B is assumed as follows: 192.168.1.5; the MAC address of the terminal is: 16-C6-42-DD-8E-27.
The user terminal B has 2 requests for completing one service, and the requests are sequentially and respectively loaded to the target server C and the target server a through the balancing device 200.
When the first request of the user terminal B reaches the target server C, the target server C acquires the log ID and the request sequence number from the log sequence number generation server 400. The log sequence number generation server 400 inquires the local cache according to the terminal information (IP (192.168.1.5)/MAC (16-C6-42-DD-8E-27), and there is no corresponding log ID at this time. Then the log ID is generated as LPOO09KLOP807A and the request sequence number is generated as 1 from the terminal information (terminal IP (192.168.1.5)/terminal MAC (16-C6-42-DD-8E-27)) and time (2017-10-10) using MD5 or other algorithm. The log sequence number generation server 400 records the log ID (LPOO09KLOP807A) and the request sequence number 1 to the cache, and returns this information to the target server C. The destination server C obtains the log ID (LPOO09KLOP807A) and the request sequence number 1. This information is recorded in the current log.
When the second request of the user terminal B reaches the target server a. The target server a acquires the log ID and the request sequence number from the log sequence number generation server 400. The log sequence number generation server 400 inquires the local cache according to the terminal information (IP (192.168.1.5)/MAC (16-C6-42-DD-8E-27), when the already existing log ID is LPOO09KLOP807A and the request sequence number is 1. The log sequence number generation server 400 increments the request sequence number by 1 according to a preset rule, that is, the request sequence number is 2. The log sequence number generation server 400 records the log ID (LPOO09KLOP807A) and the request sequence number 2 to the information before the cache overwriting, and returns the log ID (LPOO09KLOP807A) and the request sequence number 2 to the target server a. The target server a gets the log ID of the record (KIUO09KLOP837L) and the request sequence number 2. This information is recorded in the current log.
When the log file reaches a certain size (configurable), the target server a, the target server B, and the target server C upload the log file to the log analysis server 500. The log analysis server 500 sorts the logs according to the log IDs and the request sequence numbers, and the logs of each user can be displayed according to the operation sequence. The log of the user terminal a is extracted by the log ID (KIUO09KLOP837L) and sorted in order by request sequence number 1, 2, 3. The log of user terminal B is extracted by log ID (LPOO09KLOP807A) and sorted in order of request sequence number 1, 2.
Meanwhile, the log sequence number generation server 400 may be set to invalidate all log IDs and request sequence numbers after 24 hours (the time may be configured). Regenerated the next time it is requested.
The embodiment of the present invention further provides a log analysis method, which is different from the above embodiments, in that the log analysis method describes, from the side of the log analysis server 500, steps specifically executed by the log analysis server 500 in the log analysis process. It is understood that the steps involved in the log analysis method to be described next have been described in the above embodiment, and the detailed contents of the specific steps can be described with reference to the above embodiment, and only the steps performed on the log analysis server 500 side will be briefly described below. Referring to fig. 5, the log analysis method performed at the log analysis server 500 side may include the following steps.
In step S310, the log analysis server 500 receives a plurality of log files sent by the target server 300 for processing the service request of the user terminal in the cluster system 10, where the log files include at least one log, and each log includes a log ID and a request sequence number.
In step S320, the log analysis server 500 sorts the logs based on the log ID and the request sequence number of each log, and obtains the sorted logs.
In step S330, the log analysis server 500 analyzes the sorted logs.
The embodiment of the present invention further provides a log analysis method, which is different from the above embodiments, in the log analysis method, steps specifically executed by the log sequence number generation server 400 in the log analysis process are described from the log sequence number generation server 400 side. It is to be understood that the steps involved in the log analysis method to be described next have been described in the above embodiment, and specific details of each step can be described with reference to the above embodiment, and only the steps performed on the log sequence number generation server 400 side are briefly described below. Referring to fig. 6, the log analysis method performed at the log sequence number generation server 400 side may include the following steps.
Step S410, the log sequence number generating server 400 receives a request for generating a log ID and a request sequence number corresponding to a service request sent by the target server 300 in the cluster system 10 when the service request of the user terminal 100 is processed;
in step S420, the log sequence number generation server 400 generates a log ID and a request sequence number according to the user terminal information.
In this embodiment, step S420 may be implemented by:
the log sequence number generation server 400 queries whether a log ID corresponding to the user terminal 100 exists according to user terminal information;
when no log ID exists, generating a log ID according to the identification information of the user terminal 100 and the service request processing time, and setting a request serial number as an initial value, wherein the identification information comprises an IP address of the user terminal 100 or an MAC address of the user terminal;
and when the log ID exists, the request sequence number is increased according to a preset rule.
In step S430, the log sequence number generating server 400 sends the generated log ID and the request sequence number to the target server 300, and the target server 300 records the generated log ID and the request sequence number into the current log, so that the log analyzing server 500 in the cluster system 10 can analyze the log generated by the target server 300.
The embodiment of the present invention further provides a log analysis device 510, where the log analysis device 510 is applied to the log analysis server 500, and is used to correspondingly implement the method shown in fig. 5. Referring to fig. 7, the log analysis device includes a first receiving module 511, a sorting module 512, and an analyzing module 513.
A first receiving module 511, configured to receive multiple log files sent by a target server 300 that processes a service request of a user terminal in the cluster system, where the log files include at least one log, and each log includes a log ID and a request sequence number;
a sorting module 512, configured to sort the logs based on the log ID and the request sequence number of each log, and obtain sorted logs;
an analysis module 513 configured to analyze the sorted logs.
The embodiment of the present invention further provides a log analysis device 410, where the log analysis device 410 is applied to the log sequence number generation server 400, and is used to correspondingly implement the method shown in fig. 6. Referring to fig. 8, the log analysis apparatus includes a second receiving module 411, a generating module 412 and a sending module 413.
The second receiving module 411 is configured to receive a request for generating a log ID and a request sequence number corresponding to a service request sent by the target server 300 in the cluster system when the service request of the user terminal is processed.
The generating module 412 is configured to generate a log ID and a request serial number according to the user terminal information.
In this embodiment, the manner of generating the log ID and the request serial number by the generating module 412 according to the user terminal information includes:
inquiring whether a log ID corresponding to the user terminal 100 exists according to the user terminal information;
when no log ID exists, generating a log ID according to the identification information of the user terminal 100 and the service request processing time, and setting a request sequence number as an initial value, wherein the identification information comprises an IP address of the user terminal 100 or an MAC address of the user terminal 100;
and when the log ID exists, the request sequence number is increased according to a preset rule.
The sending module 413 is configured to send the generated log ID and the request sequence number to the target server 300, and the target server 300 records the generated log ID and the request sequence number into a current log, so that the log analysis server 500 in the cluster system performs log analysis on the log generated by the target server 300.
The embodiment of the invention provides a log analysis method and a log analysis device, which are applied to a cluster system, wherein the method comprises the following steps: a target server for processing a user terminal service request in the cluster system sends a plurality of log files to a log analysis server in the cluster system, wherein the log files comprise at least one log, and each log comprises a log ID and a request serial number; the log analysis server sorts the logs based on the log ID and the request serial number in each log to obtain sorted logs; the log analysis server analyzes the sorted logs. According to the scheme provided by the embodiment, the target server sends the log file to the log analysis server, the analysis server sorts the logs according to the log ID and the request sequence number of each log, and the logs with the same log ID are sorted according to the request sequence number without manually searching and sorting, so that the logs are conveniently analyzed, and the analysis efficiency of the logs is improved.
The above description is only for the specific embodiments of the present invention, but the scope of the present invention is not limited thereto, and any person skilled in the art can easily conceive of the changes or substitutions within the technical scope of the present invention, and all the changes or substitutions should be covered within the scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.
Claims (12)
1. A log analysis method is applied to a cluster system, and comprises the following steps:
a target server for processing a user terminal service request in the cluster system sends a plurality of log files to a log analysis server in the cluster system, wherein the log files comprise at least one log, and each log comprises a log ID and a request serial number; the log ID corresponds to the user terminal of the service request, and the request sequence number is used for representing the sequence of the service request of the user terminal corresponding to the log;
the log analysis server sorts the logs with the same log ID according to the request sequence number based on the log ID and the request sequence number in each log to obtain sorted logs;
the log analysis server analyzes the sorted logs.
2. The method of claim 1, further comprising the step of generating a log comprising:
when processing a service request of a user terminal, the target server sends a request for generating a log ID and a request serial number to a log serial number generation server in the cluster system;
and the log sequence number generation server generates the log ID and the request sequence number corresponding to the user terminal according to the user terminal information.
3. The method as claimed in claim 2, wherein the step of the log sequence number generation server generating the log ID and the request sequence number corresponding to the user terminal according to the user terminal information comprises:
the log sequence number generation server inquires whether a log ID corresponding to the user terminal exists according to the user terminal information;
when no log ID exists, generating a log ID according to the identification information of the user terminal and the service request processing time, and setting a request serial number as an initial value, wherein the identification information comprises an IP address of the user terminal or an MAC address of the user terminal;
and when the log ID exists, the request sequence number is increased according to a preset rule.
4. The method of claim 2 or 3, wherein the method further comprises:
and the log sequence number generation server sends the generated log ID and the request sequence number to a target server and records the log ID and the request sequence number into the current log.
5. The method of claim 4, wherein the method further comprises the step of the target server sending the log file to a log analysis server by any one of:
the target server sends a log file to the log analysis server according to a preset time interval; or
And the target server sends the log file to a log analysis server when the capacity of the log file reaches a preset capacity threshold value.
6. The method of claim 5, wherein the method further comprises:
and the log sequence number generation server configures all log IDs and request sequence numbers in the log sequence number generation server to be invalid after preset time.
7. A log analysis method is applied to a log analysis server in a cluster system, and comprises the following steps:
the method comprises the steps that a log analysis server receives a plurality of log files sent by a target server for processing a user terminal service request in a cluster system, wherein the log files comprise at least one log, and each log comprises a log ID and a request serial number; the log ID corresponds to the user terminal of the service request, and the request sequence number is used for representing the sequence of the service request of the user terminal corresponding to the log;
the log analysis server sorts the logs with the same log ID according to the request sequence number based on the log ID and the request sequence number of each log to obtain sorted logs;
the log analysis server analyzes the sorted logs.
8. A log analysis method is applied to a log sequence number generation server in a cluster system, and comprises the following steps:
the log sequence number generation server receives a request for generating a log ID and a request sequence number when a target server in the cluster system processes a service request of a user terminal;
the log sequence number generation server generates a log ID and a request sequence number according to user terminal information; the log ID corresponds to the user terminal of the service request, and the request sequence number is used for representing the sequence of the service request of the user terminal corresponding to the log;
and the log sequence number generation server sends the generated log ID and the request sequence number to the target server, and the target server records the generated log ID and the request sequence number into the current log, so that the log analysis server in the cluster system can sort the logs with the same log ID according to the request sequence number based on the log ID and the request sequence number in each log, obtain the sorted logs and analyze the sorted logs.
9. The method of claim 8, wherein the step of the log sequence number generation server generating the log ID and the request sequence number according to the user terminal information comprises:
the log sequence number generation server inquires whether a log ID corresponding to the user terminal exists according to the user terminal information;
when no log ID exists, generating a log ID according to the identification information of the user terminal and the service request processing time, and setting a request serial number as an initial value, wherein the identification information comprises an IP address of the user terminal or an MAC address of the user terminal;
and when the log ID exists, the request sequence number is increased according to a preset rule.
10. A log analysis device applied to a log analysis server in a cluster system, the device comprising:
the system comprises a first receiving module, a second receiving module and a third receiving module, wherein the first receiving module is used for receiving a plurality of log files sent by a target server for processing a service request of a user terminal in the cluster system, the log files comprise at least one log, and each log comprises a log ID and a request serial number; the log ID corresponds to the user terminal of the service request, and the request sequence number is used for representing the sequence of the service request of the user terminal corresponding to the log;
the sorting module is used for sorting the logs with the same log ID according to the request sequence number based on the log ID and the request sequence number of each log to obtain sorted logs;
an analysis module to analyze the sorted logs.
11. A log analysis device, applied to a log sequence number generation server in a cluster system, the device comprising:
the second receiving module is used for receiving a request for generating a log ID and a request serial number, which is sent by a target server in the cluster system when the target server processes a service request of a user terminal;
the generating module is used for generating a log ID and a request serial number according to the user terminal information; the log ID corresponds to the user terminal of the service request, and the request sequence number is used for representing the sequence of the service request of the user terminal corresponding to the log;
and the sending module is used for sending the generated log ID and the request sequence number to the target server, and the target server records the generated log ID and the request sequence number into the current log, so that the log analysis server in the cluster system can sort the logs with the same log ID according to the request sequence number based on the log ID and the request sequence number in each log, obtain the sorted logs and analyze the sorted logs.
12. The apparatus of claim 11, wherein the means for generating the log ID and the request sequence number according to the ue information comprises:
inquiring whether a log ID corresponding to the user terminal exists according to the user terminal information;
when no log ID exists, generating a log ID according to the identification information of the user terminal and the service request processing time, and setting a request serial number as an initial value, wherein the identification information comprises an IP address of the user terminal or an MAC address of the user terminal;
and when the log ID exists, the request sequence number is increased according to a preset rule.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711278704.5A CN108063685B (en) | 2017-12-06 | 2017-12-06 | Log analysis method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711278704.5A CN108063685B (en) | 2017-12-06 | 2017-12-06 | Log analysis method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN108063685A CN108063685A (en) | 2018-05-22 |
| CN108063685B true CN108063685B (en) | 2021-06-18 |
Family
ID=62135316
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201711278704.5A Active CN108063685B (en) | 2017-12-06 | 2017-12-06 | Log analysis method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108063685B (en) |
Families Citing this family (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110048899B (en) * | 2019-05-29 | 2022-03-04 | 北京奇艺世纪科技有限公司 | Log detection method and device, terminal and server |
| CN110569274A (en) * | 2019-08-02 | 2019-12-13 | 福建星网智慧软件有限公司 | Distributed real-time log analysis method and computer-readable storage medium |
| CN110647448A (en) * | 2019-08-09 | 2020-01-03 | 北京建筑大学 | Mobile application operation log data real-time analysis method, server and system |
| CN113938919B (en) * | 2021-09-03 | 2023-07-07 | 中国联合网络通信集团有限公司 | Data analysis method and device |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1637714A (en) * | 2003-11-26 | 2005-07-13 | 株式会社日立制作所 | Remote copy network |
| JP2009064125A (en) * | 2007-09-05 | 2009-03-26 | Fuji Electric Systems Co Ltd | Server device and program thereof |
| CN105824744A (en) * | 2016-03-21 | 2016-08-03 | 焦点科技股份有限公司 | Real-time log collection and analysis method on basis of B2B (Business to Business) platform |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20140236791A1 (en) * | 2013-02-15 | 2014-08-21 | Bank Of America Corporation | Image retrieval and transaction id capture |
-
2017
- 2017-12-06 CN CN201711278704.5A patent/CN108063685B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1637714A (en) * | 2003-11-26 | 2005-07-13 | 株式会社日立制作所 | Remote copy network |
| JP2009064125A (en) * | 2007-09-05 | 2009-03-26 | Fuji Electric Systems Co Ltd | Server device and program thereof |
| CN105824744A (en) * | 2016-03-21 | 2016-08-03 | 焦点科技股份有限公司 | Real-time log collection and analysis method on basis of B2B (Business to Business) platform |
Non-Patent Citations (2)
| Title |
|---|
| "海量网络流量日志分析系统的设计与实现";王家哲;《中国优秀硕士学位论文全文数据库 信息科技辑》;第I138-2043页;20170215;第1、13、23 、24、46、50页 * |
| "通过RequestID进行问题快速定位";佚名;《https://weibo.com/p/1001603728585759319483》;20140704;第1页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN108063685A (en) | 2018-05-22 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108063685B (en) | Log analysis method and device | |
| EP3169018B1 (en) | Method and device for establishing performance measurement task and processing performance measurement result | |
| CN108776616B (en) | Method for determining credible state of block chain node, block chain link point and system | |
| EP2869495B1 (en) | Node de-duplication in a network monitoring system | |
| CN107092686B (en) | File management method and device based on cloud storage platform | |
| US20160132520A1 (en) | Method and apparatus for finding file in storage device and router | |
| CN112602304A (en) | Identifying device types based on behavioral attributes | |
| CN112434039A (en) | Data storage method, device, storage medium and electronic device | |
| WO2020143181A1 (en) | Data storage method, apparatus, computer device and storage medium | |
| WO2018127005A1 (en) | Switching method and device of service node | |
| CN108322495B (en) | Method, device and system for processing resource access request | |
| CN107580052B (en) | Self-evolution network self-adaptive crawler method and system | |
| US10210351B2 (en) | Fingerprint-based configuration typing and classification | |
| CN108039960B (en) | Configuration information issuing method and server | |
| CN106648722B (en) | Method and device for processing Flume receiving terminal data based on big data | |
| CN114265927A (en) | Data query method and device, storage medium and electronic device | |
| CN107704494B (en) | User information collection method and system based on application software | |
| CN113987002A (en) | Data exchange method based on mass data analysis platform | |
| CN110955460B (en) | Service process starting method and device, electronic equipment and storage medium | |
| CN108229585B (en) | Log classification method and system | |
| CN101141469A (en) | Safety information retrieval server, system, method and a terminal | |
| CN106649678B (en) | Data processing method and system | |
| CN113055213A (en) | Alarm information management method, alarm information management system and server | |
| CN107656967B (en) | Scene information processing method and device | |
| CN114338794B (en) | Service message pushing method and device, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |