JP2013003950A - Decentralized processing system, log collection server, log collection method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To shorten the time for which an analyzer waits for a log to arrive if a failure occurs to a program.SOLUTION: A decentralized processing system of the present invention includes a plurality of service providing servers 10 and a log collection server 30 which collects logs from the service providing servers 10. The log collection server 30 includes: a log collection order list 33 showing the order of collection of logs determined according to dependence between programs and performed in the case of occurrence of a failure to each program; and a log collection part 31 which collects logs from the service providing servers 10 in the order shown in the log collection order list 33 with respect to a program on the service providing servers 10 to which a failure occurs in such a case.

Description

  The present invention relates to a technique for collecting logs from a server, and more particularly to a technique for collecting large-capacity logs from a plurality of servers.

  The distributed processing system is a system that provides a service by a plurality of servers operating in cooperation, and a plurality of programs operate on each server.

  Each program outputs a log (analysis log) for identifying the cause of the failure to each server in preparation for the occurrence of the failure. Logs are usually stored in each server because of their large capacity, and are collected from each server when a failure occurs and used by an analyst to identify the cause of the failure.

  As a technology for collecting logs from the server, Non-Patent Document 1, for example, provides an integrated log management environment that can be operated with simple operations, collects various logs necessary for system audits and fault investigations, and reduces management costs The technology to do is described.

  As another technique for collecting logs from the server, for example, Non-Patent Document 2 focuses on the dependency between each component of the server and various services. A technique for automatically extracting a log related to an event of interest is disclosed.

NEC, “WebSAM LogCollector”, [Search June 7, 2011], Internet <URL: http://www.nec.co.jp/middle/WebSAM/products/LogCollector/> Mikifumi Shikita, Hiroshi Goto, “Log Management Support Method Based on Component Dependency Between Large Servers”, March 15, 2008, Information Processing Society of Japan, IPSJ Transactions, Vol. 49, No. 3, p. .1081-1089, [Search June 7, 2011], Internet <URL: http://hdl.handle.net/10119/7763>

  However, in the techniques described in Non-Patent Documents 1 and 2, since the analysis is started after collecting the logs collectively, the analysis cannot be started until all the logs are collected.

  Collecting large volumes of logs from multiple servers that make up a distributed processing system takes a long time (in some cases it may take several days), so we have begun analysis after waiting for the arrival of all logs when a failure occurs. Will delay the identification of the cause of the failure.

  Accordingly, an object of the present invention is to provide a distributed processing system, a log collection server, a log collection method, and a program capable of reducing the time for an analyst to wait for the arrival of a log when a failure occurs in the program. is there.

The distributed processing system of the present invention
A distributed processing system comprising a plurality of service providing servers and a log collection server that collects logs from the service providing servers,
The log collection server
For each program, a first list that represents a sequence in which logs are collected when a failure occurs in the program, which is predetermined according to the dependency relationship between the programs;
And a log collection unit that collects logs from the service providing server according to the order represented in the first list when a failure occurs in the program on the service providing server.

The log collection server of the present invention
A log collection server that collects logs from a service providing server,
For each program, a first list that represents a sequence in which logs are collected when a failure occurs in the program, which is predetermined according to the dependency relationship between the programs;
And a log collection unit that collects logs from the service providing server according to the order represented in the first list when a failure occurs in the program on the service providing server.

The log collection method of the present invention includes:
A log collection method performed by a log collection server that collects logs from a service providing server,
For each program, register a first list representing a sequence in which logs are collected when a failure occurs in the program, which is determined in advance according to the dependency relationship between the programs,
When a failure occurs in the program on the service providing server, logs are collected from the service providing server according to the order shown in the first list for the program in which the failure has occurred.

The program of the present invention
This is for causing the log collection server to execute the log collection method.

  According to the present invention, for each program, the order of log collection when a failure occurs in the program is determined in advance according to the dependency relationship between the programs, and when a failure occurs, the logs are collected in that order.

  As a result, when a failure occurs, logs can be automatically collected in the order required by the analyst, and the time that the analyst waits for log collection can be shortened.

It is a figure which shows the example of the dependence relationship of a program. It is a figure which shows the example from which the occurrence location of a failure differs from a cause location. It is a figure which shows the example of a log collection order. It is a figure which shows the operation example of the server in a distributed processing system. It is a figure which shows the structural example of the distributed processing system of the 1st and 2nd embodiment of this invention. It is a figure which shows the example of a log collection order list. It is a figure which shows the example of a server list. It is a sequence chart explaining the operation example at the time of the failure generation of the distributed processing system of the 1st Embodiment of this invention. It is a flowchart explaining the operation example of the log collection operation | movement of the log collection server of the 1st Embodiment of this invention. It is a flowchart explaining the operation example of the log collection operation | movement of the log collection server of 2nd Embodiment of this invention. It is a figure which shows the structural example of the distributed processing system of the 3rd Embodiment of this invention. It is a sequence chart explaining the operation example at the time of the failure generation of the distributed processing system of the 3rd Embodiment of this invention.

(1) Outline of the Present Invention First, an outline of the present invention will be described.

  Here, a case where three programs P1, P2, and P3 operate on one server is considered. In this case, as shown in FIG. 1, the dependency relationship between P1, P2, and P3 is such that P1 uses P2 and P2 uses P3. Further, as shown in FIG. 2, it is assumed that a program in which a failure (alarm) has occurred is P1, and a program having a problem that causes a failure is P3.

  In this case, the analyst first analyzes the log of P1 in order to identify the cause of the failure of P1. As a result of analyzing the log of P1, if the cause of the failure is not P1, then the analyst analyzes the log of P2 used by P1. If the cause of the failure is not P2, as a result of analyzing the P2 log, the analyst then analyzes the P3 log used by P2. As a result of analyzing the log of P3, it is specified that the true cause of the failure is a malfunction of P3.

  In the above case, the order in which the analyst needs logs is the order of P1 → P2 → P3. Therefore, if logs are collected in the order of P1, P2, and P3, the time for the analyst to wait for the arrival of the log can be shortened.

  Therefore, in the present invention, as shown in FIG. 3, for each program, the order of log collection when a failure occurs in the program is determined in advance according to the dependency relationship between the programs. Collect logs in order.

  Thus, when a failure occurs, logs can be automatically collected in the order of P1 → P2 → P3 required by the analyst, and the time for the analyst to wait for log collection can be shortened.

  On the other hand, when logs are collected without considering the dependency relationship between programs, for example, a case where a log of P1 is collected first, then a log of P3, and finally a log of P2 is considered.

  In this case, the analyst tries to analyze the log of P2 after analyzing the log of P1, but there is a possibility that only the log of P3 can be collected at this point. In that case, the analyst needs to wait for the log of P2 to arrive, which hinders the progress of the analysis work.

  The above case is an example of a single server, but in a distributed processing system, as shown in FIG. 4, each program runs on a plurality of servers, and the plurality of servers cooperate. Works. Since the logs of each program have a large capacity and are distributed and accumulated in a plurality of servers, it takes a very long time to collect all the logs when a failure occurs.

  When a failure occurs, it is necessary to identify the cause as soon as possible, and the shorter the time required for log collection, the better. However, the analyst cannot analyze all of the large-capacity logs at the same time, so if the analysts can provide logs in the order in which they are analyzed, the time for the analysts to wait for the arrival of logs can be shortened, and There is no impediment to progress.

  Therefore, in the present invention, according to the log collection order of FIG. 3, first, the P1 log of the servers 1 to N, then the P2 log of the servers 1 to N, and finally the P3 log of the servers 1 to N. Collected and provided to analysts.

Thereby, it is possible to shorten the time for the analyst to wait for the arrival of the log.
(2) Embodiment of the Present Invention Next, an embodiment of the present invention will be described.
(2-1) First Embodiment FIG. 5 shows a configuration example of a distributed processing system of this embodiment.

  As shown in FIG. 5, the distributed processing system of the present embodiment has a plurality of (more specifically, several hundred or more) service providing servers (hereinafter referred to as “server providing servers”). 10, an alarm monitoring server 20, and a log collection server 30.

  The server 10 is classified for each group that provides the same service.

  The server 10 accumulates logs necessary for analysis in case a failure occurs.

  In FIG. 5, Pzi is the i-th program (z = A, B,..., I = 1, 2,...) Installed in each server 10 of the group z. Lzi is a log (z = A, B,..., I = 1, 2,...) Of the program Pzi (same in the following drawings).

  The server 10 includes an alarm transmission unit 11 that transmits an alarm when a failure occurs in a program on its service providing server. The alarm includes information for identifying the server 10 and information for identifying the program in which the failure has occurred.

  The alarm monitoring server 20 includes an alarm receiving unit 21 that receives an alarm from the server 10 in which a failure has occurred, and an alarm display unit 22 that displays an alarm received by the alarm receiving unit 21.

  The log collection server 30 includes a log collection unit 31 that collects logs from the server 10 when a failure occurs, a log storage unit 32 that stores logs collected from the server 10, a log collection order list 33, and a server list 34. Have.

  FIG. 6 shows an example of the log collection order list 33.

  As shown in FIG. 6, the log collection order list 33 is a first list representing the order in which logs are collected when a failure occurs in a program, which is predetermined for each program according to the dependency relationship between the programs. It is.

  FIG. 7 shows an example of the server list 34.

  As shown in FIG. 7, the server list 34 is a second list that represents information (server name, IP address, etc.) of the servers 10 belonging to the group for each group.

  In the distributed processing system, basically, a service is provided in units of groups, and a cooperative operation is performed between servers 10 belonging to the same group.

  Therefore, when a failure occurs in the server 10, the log collection unit 31 collects logs from all the servers 10 that belong to the same group as the server 10.

  The operation of the distributed processing system of this embodiment will be described below.

  FIG. 8 shows a sequence chart for explaining an operation example when a failure occurs in the program PA1 operating on the server 10-A1 in the distributed processing system of this embodiment.

  As shown in FIG. 8, each program Pzi always outputs the log Lzi to the server 10 (step A1).

  Here, it is assumed that a failure has occurred in the program PA1 on the server 10-A1 (step A2).

  Then, the alarm transmission unit 11 of the server 10-A1 transmits an alarm to the alarm monitoring server 20 (step A3).

  In the alarm monitoring server 20, when receiving an alarm, the alarm receiving unit 21 displays the alarm on the alarm display unit 22 (step A4).

  The operator 20A monitors the alarm display unit 22 of the alarm monitoring server 20 to confirm that a failure has occurred in the program PA1 on the server 10-A1 (step A5).

  Next, the operator 20A instructs the log collection unit 31 of the log collection server 30 to collect logs related to the program PA1 of the server 10-A1 (step A6).

  Then, the log collection unit 31 collects logs from all the servers 10 belonging to the same group as the server 10-A1 according to the flow shown in FIG. 9 described later, and stores the collected logs in the log storage unit 32 (step A7). ).

  The analyst 30A starts analysis as soon as the log LA1 is collected (step A8).

  Here, the logs are collected by the log collection unit 31 in the order of LA1 → LA2 → LA3... → LAX. Therefore, when the true cause of the failure is not PA1, the analyst 30A analyzes the logs in the order of LA2 → LA3.

  FIG. 9 shows a flowchart for explaining an operation example of the log collection operation of the log collection unit 31.

  As shown in FIG. 9, first, the log collection unit 31 initializes N and X to 1 respectively (step B1).

  Next, the log collection unit 31 specifies the Xth target log to be collected from the log collection order list 33 when the failure of the program PA1 occurs (step B2).

  Next, the log collection unit 31 specifies, from the server list 34, a target server to be collected in the Nth group A to which the failed server 10-A1 belongs (step B3).

  Note that the Nth server is determined based on, for example, the order of the numeric part of the server name described in the server list 34, the order of description in the server list 34, or the like.

  Next, the log collection unit 31 collects a target log from the target server (step B4).

  Next, the log collection unit 31 increments N (step B5), determines whether there is an uncollected server that has not collected the target log in the group A (step B6), and the uncollected server If there is (YES in step B6), the process returns to step B3.

  On the other hand, if there is no uncollected server in group A (NO in step B6), the log collecting unit 31 increments X and initializes N to 1 (step B7), and collects when a failure occurs in the program PA1. It is determined whether there is an uncollected log in the logs to be processed (step B8). If there is an uncollected log (YES in step B8), the process returns to step B2, and if there is no uncollected log (step B8). NO), the process is terminated.

  As described above, according to the present embodiment, for each program, the order of collecting logs when a failure occurs in the program is determined in advance according to the dependency relationship between the programs. Collect logs.

As a result, when a failure occurs, logs can be automatically collected in the order required by the analyst, and the time for the analyst to wait for log collection can be shortened.
(2-2) Second Embodiment The distributed processing system of the present embodiment has the same configuration as that of the first embodiment, but the operation is different from that of the first embodiment.

  That is, in the first embodiment, when logs are collected from the servers 10 belonging to the same group as the server 10 in which the failure has occurred, the logs are recorded in the order of the numerical parts of the server names or the order described in the server list 34. I was collecting.

  On the other hand, in the present embodiment, first, logs are collected from the server 10 in which the failure has occurred, and thereafter logs are collected from other servers 10 belonging to the same group as the server 10 in which the failure has occurred.

  Hereinafter, the operation of the distributed processing system of this embodiment will be described.

  Note that the present embodiment is different from the first embodiment in the log collection operation of the log collection unit 31, and the other operations are the same. Therefore, hereinafter, only an operation example of the log collection operation of the present embodiment will be described with reference to FIG.

  As illustrated in FIG. 10, first, the log collection unit 31 determines that the server 10-A1 in which the failure has occurred is a server that should first collect logs (step C1).

  Next, the log collection unit 31 initializes X to 1 (step C2).

  Next, the log collection unit 31 specifies the Xth target log to be collected from the log collection order list 33 when the failure of the program PA1 occurs (step C3).

  Next, the log collection unit 31 collects a target log from the server 10-A1 (step C4).

  Next, the log collection unit 31 increments X (step C5), determines whether there is an uncollected log in the logs to be collected when a failure occurs in the program PA1 (step C6), and uncollects If there is a log (YES in step C6), the process returns to step C3, and if there is no uncollected log (NO in step C6), the process proceeds to step C7.

  Thereafter, the same processes of steps C7 to C14 as steps B1 to B8 shown in FIG. 9 are performed.

  As described above, according to the present embodiment, since logs are first collected from the server 10 in which the failure has occurred, the logs of the server 10 in which the failure has occurred can be preferentially analyzed.

Other effects are the same as those of the first embodiment.
(2-3) Third Embodiment FIG. 11 shows a configuration example of a distributed processing system according to this embodiment.

  As shown in FIG. 11, the distributed processing system of this embodiment is different from the first embodiment shown in FIG. 5 in that the alarm monitoring server 20 is deleted, and the log collection server 30 has an alarm receiver 35. It is different from the point that was added.

  In the first embodiment, when the operator 20A monitors the alarm monitoring server 20 and confirms the alarm, the log collection server 30 is manually instructed to collect the log, and the log collection server 30 is instructed by the operator 20A. Was collected as a trigger.

  On the other hand, in the present embodiment, an alarm is directly transmitted from each server 10 to the log collection server 30, and the log collection server 30 automatically collects a log related to the alarm with the reception of the alarm as a trigger. .

  The operation of the distributed processing system of this embodiment will be described below.

  FIG. 12 shows a sequence chart for explaining an operation example when a failure occurs in the program PA1 operating on the server 10-A1 in the distributed processing system of this embodiment.

  As shown in FIG. 12, each program Pzi always outputs the log Lzi to the server 10 (step D1).

  Here, it is assumed that a failure has occurred in the program PA1 on the server 10-A1 (step D2).

  Then, the alarm transmission unit 11 of the server 10-A1 transmits an alarm to the log collection server 30 (step D3).

  In the log collection server 30, when receiving an alarm, the alarm receiving unit 35 instructs the log collecting unit 31 to collect a log related to the alarm (step D4).

  Then, the log collection unit 31 collects logs from all the servers 10 belonging to the same group as the server 10-A1 according to the flow shown in FIG. 9 or FIG. 10, and stores the collected logs in the log storage unit 32. (Step D5).

  The analyst 30A starts analysis as soon as the log LA1 is collected (step D6).

  Here, the logs are collected by the log collection unit 31 in the order of LA1 → LA2 → LA3... → LAX. Therefore, when the true cause of the failure is not PA1, the analyst 30A analyzes the logs in the order of LA2 → LA3.

  As described above, according to the present embodiment, the alarm is directly transmitted from each server 10 to the log collection server 30, so that the log related to the alarm can be automatically collected without manual instruction from the operator. it can.

  Other effects are the same as those of the first or second embodiment.

DESCRIPTION OF SYMBOLS 10 Service provision server 20 Alarm monitoring server 21 Alarm reception part 22 Alarm display part 30 Log collection server 31 Log collection part 32 Log storage part 33 Log collection order list 34 Server list 35 Alarm reception part

Claims (8)

A distributed processing system comprising a plurality of service providing servers and a log collection server that collects logs from the service providing servers,
The log collection server
For each program, a first list that represents a sequence in which logs are collected when a failure occurs in the program, which is predetermined according to the dependency relationship between the programs;
A distributed processing system comprising: a log collecting unit that collects logs from the service providing server according to the order represented in the first list when a failure occurs in the program on the service providing server; . The log collection server
Each group has a second list representing service providing servers belonging to the group,
The log collecting unit
The distributed processing system according to claim 1, wherein when a failure occurs in a program on the service providing server, logs are collected from all service providing servers belonging to the same group as the service providing server in which the failure has occurred. The log collecting unit
When a failure occurs in the program on the service providing server, the log is first collected from the service providing server in which the failure has occurred, and then another service providing server belonging to the same group as the service providing server in which the failure has occurred. The distributed processing system according to claim 2, wherein logs are collected from the system. The service providing server includes:
An alarm transmission unit that transmits an alarm indicating that a failure has occurred in the program on the service providing server;
The distributed processing system includes:
An alarm monitoring server for receiving and displaying the alarm from the service providing server;
The log collecting unit
4. The log according to claim 1, wherein logs are collected from the service providing server triggered by an input of an instruction to collect a log related to a program on the service providing server in which a failure has occurred. Distributed processing system. The service providing server includes:
An alarm transmission unit that transmits an alarm indicating that a failure has occurred in the program on the service providing server;
The log collection server
An alarm receiver for receiving the alarm from the service providing server;
The log collecting unit
4. The distributed processing system according to claim 1, wherein a log is collected from the service providing server, triggered by reception of the alarm. 5. A log collection server that collects logs from a service providing server,
For each program, a first list that represents a sequence in which logs are collected when a failure occurs in the program, which is predetermined according to the dependency relationship between the programs;
A log collection server having a log collection unit that collects logs from the service provision server according to the order represented in the first list when a failure occurs in the program on the service provision server; . A log collection method performed by a log collection server that collects logs from a service providing server,
For each program, register a first list representing a sequence in which logs are collected when a failure occurs in the program, which is determined in advance according to the dependency relationship between the programs,
A log collection method for collecting a log from the service providing server according to the order represented in the first list when a failure occurs in a program on the service providing server.   A program for causing the log collection server to execute the log collection method according to claim 7.

Images