CN109951323B - Log analysis method and system - Google Patents
Log analysis method and system Download PDFInfo
- Publication number
- CN109951323B CN109951323B CN201910147163.5A CN201910147163A CN109951323B CN 109951323 B CN109951323 B CN 109951323B CN 201910147163 A CN201910147163 A CN 201910147163A CN 109951323 B CN109951323 B CN 109951323B
- Authority
- CN
- China
- Prior art keywords
- log
- log analysis
- analysis request
- center
- autonomous server
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Abstract
The invention provides a log analysis method, which comprises the following steps: the dispatching center receives a log analysis request, determines a target autonomous server corresponding to the log analysis request, and sends a log analysis instruction to the target autonomous server; each target autonomous server acquires and analyzes a target log file corresponding to the log analysis instruction, generates a sub-analysis result, and sends the sub-analysis result to the data processing center; and the data processing center processes all the sub-analysis results to generate processing results, and sends the processing results to the scheduling center so that the scheduling center responds to the log analysis request based on the processing results. The invention can effectively shorten the completion time of the log analysis task, further effectively reduce the response delay of the log analysis request and improve the service quality of the log analysis system.
Description
Technical Field
The invention relates to the technical field of internet, in particular to a log analysis method and a log analysis system.
Background
A CDN (Content Delivery Network) service deploys a plurality of node servers in the existing internet, so that a user can obtain data from a closer or better edge node server, and a Network acceleration service is provided for the user.
At present, a CDN service provider mainly performs unified analysis on logs of edge node servers in a CDN network through a central computing platform. Specifically, the edge node server may record service state information of the CDN service in the form of a log, and send the log file to the offline computing platform for storage. Therefore, when the off-line computing platform receives the log analysis request, all log files corresponding to the log analysis request can be searched in all stored log files, and all log files corresponding to the log analysis request are analyzed to generate an analysis result so as to respond to the log analysis request.
In the process of implementing the invention, the inventor finds that the prior art has at least the following problems:
when the log analysis request corresponds to a large number of logs, the process of searching all log files corresponding to the log analysis request in all stored log files by the central computing platform and the process of analyzing all searched log files by the central computing platform both need to consume a long time, and further the response delay of the log analysis request is high.
Disclosure of Invention
In order to solve the problems in the prior art, the embodiment of the invention provides a log analysis method and a log analysis system, and the technical scheme is as follows:
in a first aspect, a log analysis method is provided, where the method is applied to a log analysis system, where the log analysis system at least includes a scheduling center, a data processing center, and a plurality of distributed autonomous servers and edge node servers, where each autonomous server corresponds to a plurality of edge node servers, and the method includes:
the dispatching center receives a log analysis request, determines a target autonomous server corresponding to the log analysis request, and sends a log analysis instruction to the target autonomous server;
each target autonomous server acquires and analyzes a target log file corresponding to the log analysis instruction, generates a sub-analysis result, and sends the sub-analysis result to the data processing center;
and the data processing center processes all the sub-analysis results to generate processing results, and sends the processing results to the scheduling center so that the scheduling center responds to the log analysis request based on the processing results.
Further, the log analysis request carries a service indication identifier;
the determining of the target autonomous server corresponding to the log analysis request includes:
and if the log analysis request also carries a machine identifier of an autonomous server, determining a target autonomous server corresponding to the machine identifier, otherwise, determining the target autonomous server corresponding to the service indication identifier.
Further, the method further comprises:
each edge node server periodically sends the log file generated in the current period to the autonomous server corresponding to the edge node server;
and the autonomous server receives the log file sent by the edge node server and stores the log file according to the preset log storage duration of the autonomous server.
Further, the log analysis system further comprises a central computing platform, and the method further comprises:
each edge node server periodically sends the log files generated in the current period to the central computing platform;
and the central computing platform receives and stores all log files sent by all the edge node servers.
Further, after receiving the log analysis request, the scheduling center further includes:
if the target autonomous server corresponding to the log analysis request does not exist, the dispatching center forwards the log analysis request to the central computing platform;
the central computing platform acquires and analyzes the log to be analyzed corresponding to the log analysis request, generates an analysis result and sends the analysis result to the scheduling center;
the dispatching center responds to the log analysis request based on the analysis result.
Further, if there is no target autonomous server corresponding to the log analysis request, the scheduling center forwards the log analysis request to the central computing platform, including:
and if the interval duration of the earliest generation time and the current time of all the log files corresponding to the log analysis request is greater than the log storage duration, the scheduling center forwards the log analysis request to the central computing platform.
Further, after the data processing center processes all the sub-analysis results to generate a processing result, the method further includes:
the data processing center correspondingly stores the processing result and the log analysis request;
the dispatching center receives the log analysis request and determines a target autonomous server corresponding to the log analysis request, and the method comprises the following steps:
the scheduling center receives a log analysis request and inquires whether a target analysis result corresponding to the log analysis request is stored in the data processing center;
and if so, the scheduling center responds to the log analysis request based on the target analysis result, otherwise, the target autonomous server corresponding to the log analysis request is determined.
In a second aspect, a log analysis system is provided, where the system includes at least a dispatching center, a data processing center, and a plurality of distributed autonomous servers and edge node servers, where:
the dispatching center is used for receiving the log analysis request, determining a target autonomous server corresponding to the log analysis request and sending a log analysis instruction to the target autonomous server; and for responding to the log analysis request based on the processing result.
The autonomous server is used for acquiring and analyzing a target log file corresponding to the log analysis instruction, generating a sub-analysis result and sending the sub-analysis result to the data processing center;
and the data processing center is used for processing all the sub-analysis results to generate processing results and sending the processing results to the scheduling center.
Further, the dispatch center is further configured to:
and if the log analysis request also carries a machine identifier of an autonomous server, determining a target autonomous server corresponding to the machine identifier, otherwise, determining the target autonomous server corresponding to the service indication identifier.
Further, the edge node server is configured to:
periodically sending the log file generated in the current period to an autonomous server corresponding to the edge node server;
the autonomous server is further configured to:
and receiving the log file sent by the edge node server, and storing the log file according to the preset log storage duration of the autonomous server.
Further, the log analysis system further includes a central computing platform, and the edge node server is further configured to:
periodically sending the log file generated in the current period to the central computing platform;
the central computing platform to:
and receiving and storing all log files sent by all edge node servers.
The technical scheme provided by the embodiment of the invention has the following beneficial effects:
in the embodiment of the invention, the dispatching center receives a log analysis request, determines a target autonomous server corresponding to the log analysis request, and sends a log analysis instruction to the target autonomous server; each target autonomous server acquires and analyzes a target log file corresponding to the log analysis instruction, generates a sub-analysis result, and sends the sub-analysis result to the data processing center; and the data processing center processes all the sub-analysis results to generate processing results, and sends the processing results to the scheduling center so that the scheduling center responds to the log analysis request based on the processing results. Therefore, when the processing capacity of the log analysis task is large, the log analysis task can be split into a plurality of subtasks with small processing capacity, the subtasks are respectively executed by a plurality of autonomous servers, and the results are collected uniformly, so that the completion time of the log analysis task can be effectively shortened, the response delay of a log analysis request can be effectively reduced, and the service quality of the log analysis system can be improved.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the description below are only some embodiments of the present invention, and it is obvious for those skilled in the art that other drawings can be obtained according to the drawings without creative efforts.
FIG. 1 is a flow chart of a log analysis method according to an embodiment of the present invention;
fig. 2 is a schematic view of a scenario of a log analysis system according to an embodiment of the present invention.
Detailed Description
To make the objects, technical solutions and advantages of the present invention more apparent, embodiments of the present invention will be described in detail with reference to the accompanying drawings.
The embodiment of the invention provides a log analysis method which can be applied to a log analysis system, wherein the log analysis system at least comprises a dispatching center, a data processing center, a plurality of distributed autonomous servers and edge node servers. The dispatching center can receive the log analysis request, generate a log analysis instruction and send the log analysis instruction to the autonomous server; the autonomous server can acquire and analyze the log file based on the log analysis instruction and send the generated sub-analysis result to the data processing center; the data processing center can process the received sub-analysis results, generate final processing results and send the final processing results to the scheduling center, so that the scheduling center can respond to the log analysis request; the edge node server can record the service state information of the edge node server in a log mode, and sends the generated log file to the corresponding autonomous server for storage. The plurality of edge node servers may be divided into one or more server groups, and one autonomous server may be provided for each server group, and the autonomous server may communicate with all edge node servers in the group, that is, each autonomous server may correspond to a plurality of edge node servers in the group. The autonomous server may be any edge node server or may be a separate server.
It should be noted that the method may be applied to various scenes with the distributed network architecture, and this embodiment takes a scene applied to the CDN architecture as an example, and other scenes are similar to the scenario.
The following will describe in detail a processing flow of the log analysis method shown in fig. 1 with reference to the specific embodiment, and the contents may be as follows:
step 101: and the dispatching center receives the log analysis request, determines a target autonomous server corresponding to the log analysis request, and sends a log analysis instruction to the target autonomous server.
In implementation, the scheduling center may receive a log analysis request sent by a log analysis requester, and determine an autonomous server (which may be referred to as a target autonomous server) corresponding to the log analysis request according to information carried in the log analysis request, such as information of a service indication identifier, a machine identifier, and the like. And then, the dispatching center can generate a log analysis instruction based on the log analysis request and then send the log analysis instruction to the target autonomous server. Each log analysis request may correspond to one or more target autonomous servers, and generating a log analysis instruction based on the log analysis request may be understood as dividing a log analysis task corresponding to the log analysis request into a plurality of subtasks, where each subtask corresponds to one log analysis instruction.
Optionally, the log analysis requester may specify a target autonomous server corresponding to the log analysis request, and correspondingly, the log analysis request may carry a service indication identifier, and the processing of determining the target autonomous server corresponding to the log analysis request in step 101 may be as follows: and if the log analysis request also carries the machine identifier of the autonomous server, determining a target autonomous server corresponding to the machine identifier, and otherwise, determining the target autonomous server corresponding to the service indication identifier.
In implementation, when the log analysis requester needs to know the service state of a certain service, the log analysis requester may send a log analysis request carrying a service indication identifier to the scheduling center. Further, if the service state of the service in the partial region needs to be known specifically, the log analysis requester may further add a machine identifier of the corresponding autonomous server in the log analysis request, so that the scheduling center may determine the target autonomous server according to the service indication identifier carried in the log analysis request or the machine identifier of the autonomous server. Specifically, when a log analysis request carries a machine identifier of an autonomous server, a scheduling center may obtain the machine identifier of the autonomous server, and determine a target autonomous server corresponding to the log analysis request according to the machine identifier, where the machine identifier of the autonomous server carried in the log analysis request may be one or more, that is, one or more target autonomous servers corresponding to the log analysis request may be provided; when the log analysis request carries the service indication identifier but does not carry the machine identifier of the autonomous server, the scheduling center may determine the target autonomous server corresponding to the log analysis request according to the service indication identifier, and similarly, the target autonomous server corresponding to the service indication identifier may be one or multiple. For example, the service indication identifier may be a domain name a, the scheduling center may store a correspondence table between domain names and autonomous servers, and the correspondence table may record which domain names of each autonomous server store log files of domain name access services, so that the scheduling center may determine all target autonomous servers corresponding to the domain name a according to the correspondence table.
Optionally, the edge node server may actively send the generated log file to the autonomous server, and the autonomous server may store the log file based on a preset log storage duration, and accordingly, the method may further include the following processing: each edge node server periodically sends the log file generated in the current period to an autonomous server corresponding to the edge node server; and the autonomous server receives the log file sent by the edge node server and stores the log file according to the preset log storage duration of the autonomous server.
In implementation, each edge node server may record service state information of the local computer in the form of a log, where the log may be a user access log and a system log, the user access log may record a CDN user access to the edge node server or a mutual access record between the edge node servers, and may generally include information such as a user IP address, an accessed content object, whether access is hit, and the like; the system log may generally include information such as global configuration performed by the operation support system and the service support system on the edge node server, and performance index of the edge node server in operation. Each edge node server can send the generated log file to a corresponding autonomous server for storage. Specifically, in consideration of the accuracy of the log analysis result and the performance of the edge node servers, each edge node server may generally periodically send the log file generated in the current period to the corresponding autonomous server, where the sending period may be set to several seconds. Based on the consideration of the log storage cost, a limited storage space can be set on the autonomous servers, and after each autonomous server receives the log file sent by the corresponding edge node server, the log file can be stored according to the preset log storage duration. Taking a preset log storage time duration of 60 minutes as an example, assuming that the current time is 11.
Optionally, the log analysis system may further include a central computing platform, and each edge node server may further send the generated log file to the central computing platform for storage, and accordingly, the method may further include the following processing: each edge node server periodically sends the log file generated in the current period to a central computing platform; and the central computing platform receives and stores all the log files sent by all the edge node servers.
In an implementation, the log analysis system may further include a central computing platform, and the central computing platform may receive and store the log file sent by each edge node server in the log analysis system. Specifically, each edge node server may periodically send the log file generated in the current period to the central computing platform, for example, may be set to send the log file generated in the 10 seconds to the central computing platform every 10 seconds. After receiving the log file sent by each node server, the central computing platform may store the received log file. It should be noted that, a larger log storage space may be set in the central computing platform, so that the central computing platform may store a large number of log files sent by each edge node server.
Optionally, if there is no target autonomous server corresponding to the log analysis request, the scheduling center may forward the log analysis request to the central computing platform. Specifically, after the dispatch center receives the log analysis request in step 101, the following processing may be performed: if the target autonomous server corresponding to the log analysis request does not exist, the dispatching center forwards the log analysis request to the central computing platform; the computing platform acquires and analyzes the to-be-analyzed log corresponding to the log analysis request, generates an analysis result and sends the analysis result to the scheduling center; the dispatch center responds to the log analysis request based on the analysis result.
In implementation, after receiving the log analysis request, the scheduling center may determine the to-be-analyzed logs corresponding to the log analysis request, and then determine whether one or more target autonomous servers capable of providing all the to-be-analyzed logs exist. For example, if all autonomous servers only store log files of type a, and the to-be-analyzed log corresponding to the log analysis request includes a log file of type B, it may be determined that there is no target autonomous server corresponding to the log analysis request, where the type may be a type of a service to which the log file belongs, a type of an edge node server to which the log file belongs, or a type obtained by dividing the log file according to the generation time.
If the target autonomous server does not exist, namely the target autonomous server corresponding to the log analysis request does not exist, the dispatching center can forward the log analysis request to the central computing platform. The central computing platform may receive the log analysis request, and search and acquire all log files (which may be referred to as to-be-analyzed logs) corresponding to the log analysis request from all locally stored log files according to the log analysis request. And then, the central computing platform can analyze the acquired log to be analyzed, generate an analysis result corresponding to the log analysis request and send the analysis result to the dispatching center. So that the dispatching center can receive the analysis result and respond to the log analysis request based on the analysis result.
Optionally, the scheduling center may determine whether a corresponding target autonomous server exists according to an interval duration between an earliest generation time of a log file corresponding to the log analysis request and a current time, and correspondingly, if the target autonomous server corresponding to the log analysis request does not exist, the process of forwarding the log analysis request to the central computing platform by the scheduling center may be as follows: and if the interval duration between the earliest generating time and the current time of all the log files corresponding to the log analysis request is greater than the log storage duration, the scheduling center forwards the log analysis request to the central computing platform.
In implementation, the scheduling center may determine an interval duration between the earliest generation time and the current time of the log file corresponding to the log analysis request, and determine whether the target autonomous server corresponding to the log analysis request exists according to the interval duration. Specifically, still taking the preset storage time duration as 60 minutes and the current time as 11 00 as an example, assuming that the earliest generation time of the log file corresponding to the log analysis request is 09; assuming that the log file corresponding to the log analysis request is a historical log file before about 60 minutes, such as 09.
Step 102: and each target autonomous server acquires and analyzes a target log file corresponding to the log analysis instruction, generates a sub-analysis result and sends the sub-analysis result to the data processing center.
In an implementation, for any one target autonomous server, after receiving the log analysis instruction, the target autonomous server may determine a log file (which may be referred to as a target log file) corresponding to the log analysis instruction, that is, a log file required for executing the log analysis instruction. Then, the target autonomous server may obtain the target log file, analyze the obtained target log file, and generate a sub-analysis result corresponding to the log analysis instruction. Next, the target autonomous server may send the sub-analysis results to a data processing center for further processing.
Step 103: and the data processing center processes all the sub-analysis results to generate processing results, and the processing results are sent to the scheduling center so that the scheduling center responds to the log analysis request based on the processing results.
In implementation, after receiving the sub-analysis results sent by all the target autonomous servers, the data processing center may perform further processing on all the received sub-analysis results, such as performing further statistics, calculations, summarization, and the like, to generate a processing result corresponding to the log analysis request, and send the processing result to the scheduling center. In this way, the scheduling center may receive a processing result corresponding to the log analysis request sent by the data processing center, and then respond to the log analysis request based on the processing result.
Optionally, the data processing center may store the processing result generated in each past, and accordingly, the following processing may be performed after step 103: and the data processing center correspondingly stores the analysis result and the log analysis request.
In implementation, the data processing center may further store the processing result after generating the processing result, for example, the processing result may be stored according to the generation time of the log file corresponding to the log analysis request. Taking the generation time of the log file corresponding to the log analysis request as 10. Or the processing result may be added with a request identifier of the log analysis request and stored.
Optionally, when the data processing center stores the processing result, correspondingly, the scheduling center receives the log analysis request in step 101, and the processing for determining the target autonomous server corresponding to the log analysis request may also be as follows: the scheduling center receives the log analysis request and inquires whether a target analysis result corresponding to the log analysis request is stored in the data processing center; if so, the scheduling center responds to the log analysis request based on the target analysis result; otherwise, determining a target autonomous server corresponding to the log analysis request.
In implementation, when the data processing center generates a processing result and stores the processing result, after receiving the log analysis request, the scheduling center may first query whether there is a processing result corresponding to the log analysis request in the data processing center, and perform subsequent processing according to the query result. Specifically, after receiving the log analysis request, the scheduling center may obtain information, such as a service indication identifier, a machine identifier, and a generation time of a corresponding log file, carried in the log analysis request. Then, the scheduling center may query, according to the information, whether a processing result (which may be referred to as a target analysis result) corresponding to the log analysis request is stored in all processing results stored in the data processing center. If the log analysis request is stored, the scheduling center can obtain the target analysis result from the data processing center and respond to the log analysis request based on the target analysis result. Otherwise, the scheduling center can determine a target autonomous server corresponding to the log analysis request.
In the embodiment of the invention, the dispatching center receives a log analysis request, determines a target autonomous server corresponding to the log analysis request, and sends a log analysis instruction to the target autonomous server; each target autonomous server acquires and analyzes a target log file corresponding to the log analysis instruction, generates a sub-analysis result, and sends the sub-analysis result to the data processing center; and the data processing center processes all the sub-analysis results to generate processing results, and sends the processing results to the scheduling center so that the scheduling center responds to the log analysis request based on the processing results. Therefore, when the processing capacity of the log analysis task is large, the log analysis task can be split into a plurality of subtasks with small processing capacity, the subtasks are respectively executed by a plurality of autonomous servers, and the results are collected uniformly, so that the completion time of the log analysis task can be effectively shortened, the response delay of a log analysis request can be effectively reduced, and the service quality of the log analysis system can be improved.
Based on the same technical concept, an embodiment of the present invention further provides a log analysis system, as shown in fig. 2, the log analysis system may at least include a scheduling center, a data processing center, and a plurality of distributed autonomous servers and edge node servers, where:
the dispatching center is used for receiving the log analysis request, determining a target autonomous server corresponding to the log analysis request and sending a log analysis instruction to the target autonomous server; and for responding to the log analysis request based on the processing result.
The autonomous server is used for acquiring and analyzing a target log file corresponding to the log analysis instruction, generating a sub-analysis result and sending the sub-analysis result to the data processing center;
and the data processing center is used for processing all the sub-analysis results to generate processing results and sending the processing results to the scheduling center.
Further, the dispatch center is further configured to:
and if the log analysis request also carries a machine identifier of an autonomous server, determining a target autonomous server corresponding to the machine identifier, otherwise, determining the target autonomous server corresponding to the service indication identifier.
Further, the edge node server is configured to:
periodically sending the log files generated in the current period to an autonomous server corresponding to the edge node server;
the autonomous server is further configured to:
and receiving a log file sent by the edge node server, and storing the log file according to the preset log storage duration of the autonomous server.
Further, the log analysis system further includes a central computing platform, and the edge node server is further configured to:
periodically sending the log file generated in the current period to the central computing platform;
the central computing platform to:
and receiving and storing all log files sent by all edge node servers.
Further, the dispatch center is further configured to:
if the target autonomous server corresponding to the log analysis request does not exist, forwarding the log analysis request to the central computing platform; and for responding to the log analysis request based on the analysis result;
the central computing platform further to:
and acquiring and analyzing the to-be-analyzed log corresponding to the log analysis request, generating an analysis result, and sending the analysis result to the scheduling center.
Further, the dispatch center is further configured to:
and if the interval duration of the earliest generation time and the current time of all the log files corresponding to the log analysis request is greater than the log storage duration, forwarding the log analysis request to the central computing platform.
Further, the data processing center is further configured to:
correspondingly storing the processing result and the log analysis request;
the dispatch center is further configured to:
receiving a log analysis request, and inquiring whether a target analysis result corresponding to the log analysis request is stored in the data processing center;
if yes, responding to the log analysis request based on the target analysis result; otherwise, determining a target autonomous server corresponding to the log analysis request.
It should be noted that: the log analysis system and the log analysis method provided by the above embodiments belong to the same concept, and specific implementation processes thereof are described in detail in the method embodiments, which are not described herein again.
It will be understood by those skilled in the art that all or part of the steps for implementing the above embodiments may be implemented by hardware, or may be implemented by a program instructing relevant hardware, where the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a magnetic disk or an optical disk, etc.
The above description is only for the purpose of illustrating the preferred embodiments of the present invention and is not to be construed as limiting the invention, and any modifications, equivalents, improvements and the like that fall within the spirit and principle of the present invention are intended to be included therein.
Claims (11)
1. A log analysis method is applied to a log analysis system, the log analysis system at least comprises a dispatching center, a data processing center, a plurality of autonomy servers and edge node servers which are distributed and deployed, and each autonomy server corresponds to a plurality of edge node servers, and the method comprises the following steps:
the dispatching center receives a log analysis request, determines a target autonomous server corresponding to the log analysis request, generates a log analysis instruction based on the log analysis request, and sends the log analysis instruction to the target autonomous server; wherein generating a log analysis instruction based on the log analysis request comprises: dividing the log analysis task corresponding to the log analysis request into at least one subtask, wherein each subtask corresponds to a log analysis instruction;
each target autonomous server acquires and analyzes a target log file corresponding to the log analysis instruction, generates a sub-analysis result, and sends the sub-analysis result to the data processing center;
and the data processing center processes all the sub-analysis results to generate processing results, and the processing results are sent to the scheduling center so that the scheduling center responds to the log analysis request based on the processing results.
2. The method of claim 1, wherein the log analysis request carries a service indication identifier;
the determining of the target autonomous server corresponding to the log analysis request includes:
and if the log analysis request also carries a machine identifier of an autonomous server, determining a target autonomous server corresponding to the machine identifier, otherwise, determining the target autonomous server corresponding to the service indication identifier.
3. The method of claim 1, further comprising:
each edge node server periodically sends the log file generated in the current period to the autonomous server corresponding to the edge node server;
and the autonomous server receives the log file sent by the edge node server and stores the log file according to the preset log storage duration of the autonomous server.
4. The method of claim 3, wherein the log analysis system further comprises a central computing platform, the method further comprising:
each edge node server periodically sends the log files generated in the current period to the central computing platform;
and the central computing platform receives and stores all log files sent by all the edge node servers.
5. The method of claim 4, wherein after the dispatch center receives the log analysis request, further comprising:
if the target autonomous server corresponding to the log analysis request does not exist, the dispatching center forwards the log analysis request to the central computing platform;
the central computing platform acquires and analyzes the to-be-analyzed log corresponding to the log analysis request, generates an analysis result and sends the analysis result to the scheduling center;
the dispatch center responds to the log analysis request based on the analysis result.
6. The method according to claim 5, wherein if there is no target autonomous server corresponding to the log analysis request, the scheduling center forwards the log analysis request to the central computing platform, including:
and if the interval duration between the earliest generating time and the current time of all the log files corresponding to the log analysis request is greater than the log storage duration, the scheduling center forwards the log analysis request to the central computing platform.
7. The method of claim 1, wherein after the data processing center processes all the sub-analysis results to generate the processing results, the method further comprises:
the data processing center correspondingly stores the processing result and the log analysis request;
the dispatching center receives the log analysis request, determines a target autonomous server corresponding to the log analysis request, and comprises the following steps:
the scheduling center receives a log analysis request and inquires whether a target analysis result corresponding to the log analysis request is stored in the data processing center;
and if so, the dispatching center responds to the log analysis request based on the target analysis result, otherwise, a target autonomous server corresponding to the log analysis request is determined.
8. A log analysis system is characterized by at least comprising a dispatching center, a data processing center, a plurality of distributed autonomous servers and edge node servers, wherein:
the dispatching center is used for receiving the log analysis request, determining a target autonomous server corresponding to the log analysis request, generating a log analysis instruction based on the log analysis request, and sending the log analysis instruction to the target autonomous server; and for responding to the log analysis request based on the processing result; wherein generating a log analysis instruction based on the log analysis request comprises: dividing the log analysis task corresponding to the log analysis request into at least one subtask, wherein each subtask corresponds to a log analysis instruction;
the autonomous server is used for acquiring and analyzing a target log file corresponding to the log analysis instruction, generating a sub-analysis result and sending the sub-analysis result to the data processing center;
and the data processing center is used for processing all the sub-analysis results to generate processing results and sending the processing results to the scheduling center.
9. The system of claim 8, wherein the log analysis request carries a service indication identifier; the dispatch center is further configured to:
and if the log analysis request also carries a machine identifier of an autonomous server, determining a target autonomous server corresponding to the machine identifier, otherwise, determining the target autonomous server corresponding to the service indication identifier.
10. The system of claim 8, wherein the edge node server is configured to:
periodically sending the log file generated in the current period to an autonomous server corresponding to the edge node server;
the autonomous server is further configured to:
and receiving the log file sent by the edge node server, and storing the log file according to the preset log storage duration of the autonomous server.
11. The system of claim 10, wherein the log analysis system further comprises a central computing platform, and wherein the edge node server is further configured to:
periodically sending the log files generated in the current period to the central computing platform;
the central computing platform to:
and receiving and storing all log files sent by all edge node servers.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910147163.5A CN109951323B (en) | 2019-02-27 | 2019-02-27 | Log analysis method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910147163.5A CN109951323B (en) | 2019-02-27 | 2019-02-27 | Log analysis method and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109951323A CN109951323A (en) | 2019-06-28 |
| CN109951323B true CN109951323B (en) | 2022-11-08 |
Family
ID=67007000
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910147163.5A Active CN109951323B (en) | 2019-02-27 | 2019-02-27 | Log analysis method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109951323B (en) |
Families Citing this family (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110688359A (en) * | 2019-09-27 | 2020-01-14 | 南京天芯云数据服务有限公司 | Log analysis method and management platform |
| CN112685375B (en) * | 2019-10-18 | 2023-04-28 | 上海哔哩哔哩科技有限公司 | Log analysis method and device based on CDN |
| CN110768843B (en) * | 2019-10-31 | 2022-03-08 | 腾讯科技（深圳）有限公司 | Network problem analysis method, device, terminal and storage medium |
| CN111611058A (en) * | 2020-05-21 | 2020-09-01 | 北京明略软件系统有限公司 | Task execution method and device and electronic equipment |
| CN111740884B (en) * | 2020-08-25 | 2021-06-25 | 云盾智慧安全科技有限公司 | Log processing method, electronic equipment, server and storage medium |
| CN114513400A (en) * | 2021-12-30 | 2022-05-17 | 上海川源信息科技有限公司 | Log aggregation system and method for improving availability of log aggregation system |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101192227A (en) * | 2006-11-30 | 2008-06-04 | 阿里巴巴公司 | Log file analytical method and system based on distributed type computing network |
| KR20100036540A (en) * | 2008-09-30 | 2010-04-08 | 삼성네트웍스 주식회사 | Apparatus and method for managing a script for web log analysis on content delivery network |
| CN105357041A (en) * | 2015-10-30 | 2016-02-24 | 上海帝联信息科技股份有限公司 | Edge node server, and log file uploading method and system |
| CN108600300A (en) * | 2018-03-06 | 2018-09-28 | 北京思空科技有限公司 | Daily record data processing method and processing device |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105634845B (en) * | 2014-10-30 | 2019-01-22 | 任子行网络技术股份有限公司 | A kind of method and system for magnanimity DNS log progress multidimensional statistics analysis |
-
2019
- 2019-02-27 CN CN201910147163.5A patent/CN109951323B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101192227A (en) * | 2006-11-30 | 2008-06-04 | 阿里巴巴公司 | Log file analytical method and system based on distributed type computing network |
| KR20100036540A (en) * | 2008-09-30 | 2010-04-08 | 삼성네트웍스 주식회사 | Apparatus and method for managing a script for web log analysis on content delivery network |
| CN105357041A (en) * | 2015-10-30 | 2016-02-24 | 上海帝联信息科技股份有限公司 | Edge node server, and log file uploading method and system |
| CN108600300A (en) * | 2018-03-06 | 2018-09-28 | 北京思空科技有限公司 | Daily record data processing method and processing device |
Also Published As
| Publication number | Publication date |
|---|---|
| CN109951323A (en) | 2019-06-28 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109951323B (en) | Log analysis method and system | |
| CN106982236B (en) | Information processing method, device and system | |
| CN107395683B (en) | Method for selecting return path and server | |
| US8069224B2 (en) | Method, equipment and system for resource acquisition | |
| EP1825654B1 (en) | Routing a service query in an overlay network | |
| CN106951179B (en) | Data migration method and device | |
| CN110795029B (en) | Cloud hard disk management method, device, server and medium | |
| CN111258978B (en) | Data storage method | |
| CN105893542A (en) | Method and system for redistributing cold data files in cloud storage system | |
| US8914436B2 (en) | Data processing device and data retriever | |
| CN111556123A (en) | Self-adaptive network rapid configuration and load balancing system based on edge calculation | |
| CN113596078A (en) | Service problem positioning method and device | |
| CN108399175B (en) | Data storage and query method and device | |
| CN111400301A (en) | Data query method, device and equipment | |
| CN111600929A (en) | Transmission line detection method, route strategy generation method and proxy server | |
| CN110909072B (en) | Data table establishment method, device and equipment | |
| CN107888388B (en) | Charging method and system for network acceleration service | |
| US10812390B2 (en) | Intelligent load shedding of traffic based on current load state of target capacity | |
| CN115480934A (en) | Distributed data processing method, device, equipment and storage medium | |
| CN110581784A (en) | node health check method, device and equipment | |
| CN112507193A (en) | Data updating method, device, equipment and storage medium | |
| CN110866052A (en) | Data analysis method, device and equipment | |
| US11803448B1 (en) | Faster restart of task nodes using periodic checkpointing of data sources | |
| CN115426413B (en) | Edge node scheduling method, device, medium and equipment based on 5G | |
| CN114301709B (en) | Message processing method and device, storage medium and computing equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |