JP2002366394A - Collection and management method for log data - Google Patents

Abstract

PROBLEM TO BE SOLVED: To manage log data on the basis of a common data form and also to manage log data based on the time of a manager in a system where the manager collects the log data from a plurality of agents through a network. SOLUTION: A manager 10 delivers various rules to an agent 20. The agent 20 receives log data from a monitor object log file group 30 in accordance with the rules and normalizes it and adds the correction time of the log output time to store it in a normalized log file 40. The agent 20 takes out normalized log data from the normalized log file 40 to transfer it to the manager 10 in response to a request from the manager 10. The manager 10 stores collected normalized log data in a normalized log database 50 in the correction time order.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

[0001] 1. Field of the Invention [0002] The present invention relates to a system in which a manager collects log data from an agent via a network, and in particular, the manager manages log data existing in the system based on a common log data format. The method of collecting and managing log data.

[0002]

2. Description of the Related Art An operating system (OS) and application programs executed in an information processing apparatus output various types of log information. Several methods for collecting output log information are known. For example, Japanese Patent Application Laid-Open No. 5-250229 discloses a log data collection method in which log data from a computer in an error state is preferentially transmitted by detecting an error code in the log data in collecting log data from a plurality of computers. Is disclosed. Japanese Patent Application Laid-Open No. Hei 5-28008 discloses an information processing system that collects a failure log by detecting that the number of pieces of log information stored in a storage unit has reached a certain number, and suppressing the log registration. A log information collection method for preventing loss of trouble information is disclosed. Japanese Patent Laid-Open No. 5-11
No. 1029 discloses that the time at which data from a lower-level equipment terminal is collected is added to the data and sent to a higher-level control device, so that the chronological relationship between the data from a plurality of terminals is impaired. Disclose a data collection scheme to prevent data collection.

[0003]

In a distributed processing system in which a plurality of computers are connected via a network and the computer performs processing while communicating with other computers, one user can access a plurality of computers and files over a wide range. Can access. Therefore, in order to detect unauthorized access to computers and files by analyzing log data, log data output by individual computers is aggregated in a central computer,
Need to accumulate in database. However, the log data output by various system programs and application programs have different data formats, so it is difficult to analyze the log data simply by collecting and aggregating the log data output by individual computer programs. is there. In addition, since the time held by each computer is not necessarily the same across all computers, the time assigned to each log data is generally shifted, and the aggregated log data is not correct. It is difficult to arrange according to the order of time.

The present invention has been made in view of the above circumstances, and has as its object to manage log data normalized to a common data format.

It is another object of the present invention to manage log data based on the time of a central computer that collects log data from computers at each site.

[0006]

In order to achieve the above object, the present invention provides a data item defined by extracting a value corresponding to a predefined data item from log data in a log file to be monitored. Means for generating and accumulating normalized log data in which values of the array are arranged, and means for transmitting the accumulated normalized log data to a computer executing a manager function via a network as processing means of the agent. It is characterized by a computer that collects log data.

Further, according to the present invention, log data normalized from a computer executing an agent function via a network according to a common data format defined in advance and corrected based on a time set as a reference of a manager. A computer that collects and manages log data, which has means for receiving normalized log data having a log output time and means for storing the normalized log data in a database in the order of correction time as a processing means of a manager. I do.

[0008]

DESCRIPTION OF THE PREFERRED EMBODIMENTS One embodiment of the present invention will be described below in detail with reference to the drawings.

FIG. 1 is a system configuration diagram showing an example of a communication network to which the present invention is applied. The network includes a plurality of LANs 1, 3, 4 and a WAN (wide area network).
Network 2). The manager 10 is an application program (AP) stored in the main storage device of the computer and running under the OS. The agent 20 is an AP that is stored in a main storage device of a computer such as a server and runs under the OS. Manager 1
0 can communicate with the agents 20-1, 20-2, 20-3, and 20-4 via the LANs 1, 3, and 4 or the WAN 2. As shown in the figure, Agent 2
The computer executing 0 can execute another AP in parallel with the agent 20. The computer that executes the manager 10 can also execute other APs in parallel with the manager 10, but it is desirable to allocate a dedicated computer to secure sufficient CPU performance and resources. Reference numeral 60 denotes a network management system that manages the network system, and includes a network management program and a network monitoring terminal (not shown). Network management system 60 provides information about network resources to manager 10.

FIG. 2 is a diagram showing an outline of processing operations performed by a manager and an agent. The manager 10 distributes various defined rules to the agent 20.
(). The agent 20 registers the received rule,
According to the rule, log data is input from the monitoring target log file group 30 (), the log data is normalized, and then stored in the normalized log file 40 (). At this time, the log output time in the normalized log data is corrected by the correction time based on the time of the manager 10. Agent 20
Extracts the normalized log data from the normalized log file 40 when there is a log collection request from the manager 10 (), and transfers it to the manager 10 (). The manager 10 stores the collected normalized log data in the normalized log database 50 in the order of the correction time (). Further, the manager 10 can also store the normalized log database 50 as necessary.
Extracts any normalized log data from and analyzes it.

FIG. 3 is a functional block diagram showing the configuration of the manager 10. As shown in FIG. The rules 110 are various defined rules and are stored in the storage device. The normalized log database 50 is a database that stores collected normalized log data, and is stored in a storage device. The network management system 60 is a system including a computer connected to the computer storing the manager 10 via the network and a terminal device monitoring the network. The network management program running on the same computer as the manager 10 is , The operating state of each communication line, router, repeater, various computers, computer programs, etc. (connected / disconnected, operating /
Outage, etc.), and the location of each network element. Hereinafter, an outline of the function of each functional module constituting the manager 10 will be described. (1) Rule definition 100 This is a program tool for an operator to define various rules, and provides a GUI (graphical user interface) that allows a user to easily set rules via an input device and a display device (not shown). provide. (2) The scheduler 101 is a program for notifying the rule distribution 106 and the log collection 107 of the timing of executing the rule distribution and the log collection, respectively. As an example of the opportunity, the start of execution is instructed, for example, at 5:00 pm every day and at 3:15 pm every Saturday. (3) Process management 102 This is a program for controlling start / stop of each functional module denoted by 100 to 109 of the manager 10. (4) Log analysis 103 This is a program for extracting a normalized log data group from the normalized log database 50 via the database management 109 and performing a predetermined analysis. Table 1 below shows an example of log data analysis when the log event is login. The log event and the normalization item will be described later.

[0012]

[Table 1] (5) Configuration management 104 This is a program for managing a list of agents 20 to be managed. Also, the network management system 60 makes an inquiry to the network management system 60 to acquire information such as the presence / absence of the operation of the computer on which the agent 20 operates and the response time of the ping, and transfers the information to the rule distribution 106 or the log collection 107. (6) Rule management 105 Various rules defined from the rule definition 100 are received and stored in the rule 110. The rule is read from the rule 110 and passed to the rule distribution 106. (7) Rule distribution 106 This is a program for acquiring various rules from the rule management 105 and distributing them to the agent 20 according to instructions from the scheduler 101. (8) Log collection 107 This is a program for collecting a normalized log file from the agent 20 according to the instruction of the scheduler 101. (9) Data communication 108 Rule distribution 106 and log collection 107 are Agent 2
This is a program for performing communication control when communicating with the “0”. (10) Database management 109 The normalized log data group (normalized log file) collected by the log collection 107 from the agent 20 is stored in the normalized log database 50, and the normalized log data is searched according to a request from the log analysis 103. Extract. Further, the normalized log data whose predetermined storage period has passed is deleted from the normalized log database 50 to create an unused storage area.

FIG. 4 is a functional block diagram showing the configuration of the agent 20. The rules 205 are various rules distributed and stored in a storage device. The monitoring target log file group 30 is a log file to be monitored,
Stored in a storage device. The normalized log file 40 is a file for storing normalized log data, and is stored in a storage device. The outline of the function of each function module constituting the agent 20 will be described below. (1) Data Communication 200 The data communication 200 is a program that performs communication control when the rule management 203 and the log file management 204 communicate with the manager 10. (2) Process management 201 This is a program for controlling the start / stop of each functional module to which the agents 20 to 204 are assigned. (3) Log input 202 A log file monitoring rule, a format rule, and the like are acquired from the rule management 203, and the monitoring target log file group 30 is acquired.
This is a program for normalizing log data input from the server and passing the normalized log data to the log file management 204. (4) Rule management 203 Various rules distributed from the manager 10 are stored in the rule 20
5 is a program that is stored in a storage device and provides a rule in response to a request from the log input 202 or the log file management 204. (5) Log file management 204 Acquires filtering rules from rule management 203,
The normalized log data received from the log input 202 is filtered, added with a correction time, and stored in the normalized log file 40. When a request to collect the normalized log file 40 is received from the log collection 107 of the manager 10, the normalized log file 40 is transferred to the manager 10.

FIGS. 5 to 14 show examples of the structure of normalized log data.

FIG. 5 is a diagram showing a schematic configuration of the normalized log data 300 stored in the normalized log file 40. The normalized log data 300 has a common information class 301
User information class 302, service information class 303, address information class 304,
File information class 305, traffic information class 30
6, individual information class 307 and the like. The common information class 301 is an information class indispensable for all the normalized log data 300, and the remaining information classes are selected according to the output log data.

FIG. 6 is a diagram showing the data structure of the common information class 301. The normalized version 310 is a number indicating the version of the normalization. Log type is log event 3
11, log event result 312, log output program 31
3. Data storage class 314 and log file name 315
including. The log output program 313 outputs the log
The log file name 315 is the name of the log file output by the log output program 313. The log event 311, the log event result 312, and the data storage class 314 will be described later. Manager
The host name 316 and the host IP address 317 of the computer on which the manager is mounted are stored. The agent is the host name of the computer on which the agent that entered the log is installed.
18 and the host IP address 319 are stored. The monitoring target is a host name 320 of a computer such as a server to be monitored.
And the host IP address 321 are stored. The time includes a log output time 322 and a correction time 323. The log output time 322 is a local time of the computer that outputs the log,
The correction time 323 is a time corrected based on the time of the computer on which the manager 10 is mounted. The filtering rule name 324 is the name of a filtering rule applied when normalizing log data.

FIG. 7 is a diagram showing the data structure of the user information class 302. The user information class 302 records information about the logged-in user, and includes a user name 330, a user ID (UID) 331, a user name 332 after a user change, a UID 333 after a change, a user security level 334, and a computer. And a file access right 335, an access result 336, and the name of the terminal device operated by the user (terminal name 337).

FIG. 8 is a diagram showing the data structure of the service information class 303. The service information class 303 is
Record information about services provided to the user.
The service is service name 340, service version 3
41, the name (process name 342) and the process ID 343 of the process started to provide the service are stored.

FIG. 9 is a diagram showing the data structure of the address information class 304. The address information class 304 is
It records information when a connection has been made with another computer. In addition to the host name, IP address, MAC address, and port number of the connection source and connection destination, the connection status, the connection start time and end time, and the other computer Stores access results to

FIG. 10 shows the data structure of the file information class 305. File information class 305
Records the file name, the access information before the change, and the access information after the change for the file created or changed by the user. The access information includes a file creation time, a last modification time, a last access time, and a file i-node.
e number, access permission, UID, group ID
(GID) and the size of the final file.

FIG. 11 is a diagram showing the data structure of the traffic information class 306. Traffic information class 30
Reference numeral 6 denotes log information output by a mail management program, a file transfer program, and the like, and records the number of received bytes and the number of transmitted bytes of data and messages via the network and the transfer time (processing time) of the data.

FIG. 12 is a diagram showing the data structure of the individual information class 307. The individual information class 307 is optional, and is the original text of the message text output by the program.

FIG. 13 shows the data structure of the data storage class 314. “T1” is data storage class 3
14, and “L1” indicates the length of an area V1 that specifies an existing information class. “V1”
Is an area for specifying an information class included in each normalized log data 300, and specifies a tag indicating the start of specification of the information class, the length of the information class identifier, and the information class identifier in order. The length of the information class identifier is variable. Assuming that a number for identifying each information class is x, Tx
(X ≧ 2) is a tag indicating the start of the information class, Lx
(X ≧ 2) is the length of the Vx portion, and Vx (x ≧ 2)
2) is an information class identifier. The manager 10 recognizes the information class of each normalized log data from the data storage class 314.

FIG. 14 is a diagram showing an example of a coding table 600 of the normalization items which can be coded. When the log event 311 of the normalized item is “login”, the code is coded as “1”, and when the log event 311 is the user change “su”, the code is coded as “2”. Connect (co
nnect) indicates a log event related to a connection when performing file transfer or inter-program communication between computers. A file is a log event related to a file whose content has been changed, and a job is a log event related to a start / stop / end state of a job. Email indicates log events related to email usage. The log event result 312 is a result of the log event, and classifies success or failure. The presence or absence of the access right 335 of the user information class 302 is classified. The access result of the user information class 302 and the address information class 304 is classified into success or failure. The access permission of the file information class 305 is classified into presence or absence.

FIGS. 15 to 20 are diagrams showing data formats of rules.

FIG. 15 is a diagram showing an example of the manager rule 450. DB_MAX 451 defines a storage period indicating a period during which the normalized log database 50 can store the normalized log data. RULE_MAX 452 defines the maximum number of communication paths that can be used for rule distribution. LOG
_MAX 453 defines the maximum number of communication paths that can be used for log collection. The rule distribution 106 and the log collection 107 can use the communication path by this multiplicity. However, if the processing request is larger than this multiplicity, the communication paths are used in order, and after waiting until the communication path becomes empty, the remaining processing is performed. Execute

FIG. 16 is a diagram showing an example of the operation condition rule 470 of the agent 20. MANAGER_A
DDRESS 471 defines the IP address of the manager, and FILE_MAXSIZE 472 defines the maximum storage capacity that the normalized log file 40 can use.

FIG. 17 shows a log file monitoring rule 500.
It is a figure showing an example of. TARGET_LOG defines the log file name to be monitored, FORMAT defines the file format (SEQ: sequential format, WRAP: wraparound format), and INTERVAL defines the monitoring interval time (for example, 10 minute intervals). FMT_NAME defines a rule to be applied when normalizing the log file. For example, TARGET_LOG: /usr/adm/syslog.log,FORMAT=SEQ,INTERVAL
= 10m, FMT_NAME = abc; is a sequential file / usr / adm /
syslog. This indicates that the log is monitored at 10-minute intervals, and that normalization processing is performed according to the format rule abc.
Agent 2 if FMT_NAME is not specified
Normalization is performed between 0 using a common format rule.

FIGS. 18 and 19 show format rules 5
It is a figure which shows an example of 10 and 515. If the log data is in text format, FMT_T is applied, and if the log data is in binary format, FMT_B is applied. REGTEXT = "character string n" indicates a condition for selecting log data. If a character string n exists in log data, it indicates that log data is normalized according to the following rules. && indicates a logical product (AND), and the presence of a plurality of character strings can be used as a selection condition. | Means then, and means that normalization items are sequentially picked up from the character string in accordance with the character string sequence of the log data. The normalization item is a data item defined in each information class. :
Is a delimiter for cutting out the value corresponding to the normalized item by shifting the pointer in order from the beginning of the log data. The character specified in [] following the normalization item specifies the length of a character string recognized from an arbitrary pointer, or the last character recognized when the character string is variable length.
SKIP means that when the pointer is shifted in order from the beginning of the log data, if there is a character string unrelated to the normalization item, the character string is skipped.
Specify the number of characters to be skipped or “separator character” as the last character to be recognized. If a delimiter is specified, skip to that delimiter. The following are examples of format rules. (A) Format rule A FMT_T: REGTEXT == “SU” && REGTEXT == “+” | Log event == “2” | Log event result == “0”: SKIP [“”]:
Log output time [10]: SKIP [3]: Terminal name [“”]: User name
[“-”]: User name after change [“”]; (b) Format rule B FMT_T: REGTEXT == “connect” && REGTEXT == “refus
ed "| Log event ==“ 3 ”| Log event result ==“ 1 ”: Log output time [15]: Host name of connection destination [“ ”]: Process name
[“["]: Process ID [“]”]: SKIP [“from”]: Connection source host name [“”]; FIG. 21 is a diagram illustrating an example of the original text of the message text stored in the log file. The message texts 551 and 552 are examples of the message text regarding the user change output by the OS. The message texts 553 to 555 are message texts that the OS outputs at the time of connection. The message texts 556 and 557 are message texts output by OS job management.

When the message text 551 is normalized by the format rule A, log event 311 = 2 (su) log event result 312 = 0 (success) log output time 322 = 1/30 epoch time of 11:18 The terminal name 337 = ttyp5, the user name 330 = fujino, and the changed user name 332 = root.

When the message text 554 is normalized according to the format rule B, log event 311 = 3 (connect) log event result 312 = 1 (failure) log output time 322 = Jan 12 13: 12: 1
Epoch time of 5 ・ Destination host name 354 = hosta ・ Destination IP address 355 = Value converted from hosta to IP address ・ Process name 342 = ftpd ・ Process ID 343 = 1111 ・ Connection source host name 350 = hostb ・ Connection source IP Address 351 = value obtained by converting hostb into an IP address.

FIG. 20 is a diagram showing an example of the filtering rule 520, which is a rule for the log file management 204 to store only the corresponding normalized log data. FLT indicates a filtering rule,
If the normalized item is a specified character string or code, or is normalized log data in a specified time zone, only the normalized log data that meets such conditions is extracted and stored. == is equal! == not equal, &
& Means a logical AND, || means a logical OR, and-means a time interval.

FIG. 22 is a diagram illustrating a data structure of the normalized log database 50. The normalized log database 50 stores the normalized log data arranged in order of the correction time 630. From a certain correction time 630, the log data of the common information class 631 is chained. In addition, the common information class 631 is successively chained to the log data of the information class existing subsequently. The search key indicated corresponding to the information class indicates a normalized item used as a key when searching for normalized log data. A chain is extended from the correction time 630 to the next correction time 630. The user can efficiently extract the target normalized log data from the normalized log database 50 by the correction time 630 and by specifying the search key of the corresponding information class.

FIG. 23 shows the rule distribution 1 of the manager 10.
It is a PAD figure showing the flow of processing of 06. Rule distribution 1
06 loops after the initial setting (step 701) until a termination request is received from the process management 102 (step 70).
2) Have an event (step 703). The events include a rule distribution request from the scheduler 101 (step 704) and a termination request from the process management 102 (step 712).

When a rule distribution request (step 704) is received, a list of rules to be distributed and a list of agents 20 as distribution destinations are acquired via the rule management 105 (step 705). Acquisition of status and response time information (step 70)
6). The acquired ping response times of the agents 20 are rearranged in the order of the smallest response time (step 70).
7). At this time, the response time of the agent 20 not operating is interpreted as infinity. Looping is performed by the number of agents 20 that distribute rules (step 70).
8) If the agent 20 is operating (YES in step 709), rules are distributed in ascending order of response time (step 710). If the agent 20 is not operating, a rule distribution failure message is output (step 710). 711). RULE_M for rule distribution
Apply AX452. The distributed rules are transferred to the rule management 203 of the agent 20.

When an end request is received (step 71)
2), exit the loop and perform end processing (step 71)
3).

FIG. 24 shows the log collection 10 of the manager 10.
7 is a PAD diagram showing the flow of the processing of FIG. Log collection 107
After the initial setting (step 801), the process management 10
Loop until the end request comes from step 2 (step 80).
2) Have an event (step 803). The events include a log collection request from the scheduler 101 (step 804), a start notification from the agent 20 (step 811), and a termination request from the process management 102 (step 816).

When a log collection request is received (step 804), a list of agents 20 that collect logs is acquired from the scheduler 101 (step 805), and the operation status and response time of these agents 20 are acquired from the configuration management 104. The information is obtained and the agents 20 are sorted in the order of the shortest response time (step 80).
6). A loop is performed for several minutes of the agent 20 that collects logs (step 807). If the agent is operating (step 808 YES), logs are collected (step 8).
09) If the agent is not operating (step 808 NO), a log collection failure message is output (step 810). For log collection, LOG_MAX
Apply 453. Log collection 107 is Agent 2
The normalized log file 40 is collected via the log file management 204 of 0.

When the agent start notification is received (step 811), the response time of the ping (ICMP echo request) with the agent 20 is acquired from the configuration management 104 (step 812). ICMP (Int
ernet ControlMessage Prot
ocol) is one of the international standards for the management of communication networks, IAB: IAB.
Internet Activities Board)
Is a management standard. By using ICMP, it is possible to confirm whether an IP node (for example, a computer) can communicate with another IP node. If you use ping,
It is possible to acquire the operation state and the response time of whether or not communication with any IP node is possible. If the ping response time has been obtained (step 813 YES), the agent 20 that issued the activation notification is notified of the time obtained by adding the communication time obtained from the response time to the current time of the manager 10 (step 814). In order to inform the manager of the time to the agent, the time of the manager plus the communication time from the manager to the agent may be notified. The response time of the ping is a time interval obtained by adding the communication time from the manager to the agent and the agent to the manager, that is, the going and returning communication times. Therefore, 1/2 of the ping response time is used as the communication time. That is, the manager estimates the time of the agent using the following formula and notifies the agent: (time of the agent) エ ー ジ ェ ン ト
If (manager time) + (ping response time) / 2 response time could not be acquired (step 813 NO),
That is, when information cannot be obtained from the network management system 60, the current time of the manager 10 is simply notified to the agent 20 (step 815). The agent 20 acquires the current time of the manager 10 and applies it to the corrected time 323 of the normalized log data.

When an end request is received (step 81
6), exit the loop and perform end processing (step 81)
7).

FIG. 25 shows the configuration management 10 of the manager 10.
FIG. 14 is a PAD diagram showing the flow of the processing of No. 4; Configuration management 104
After the initial setting (step 901), the process management 10
Loop until the end request comes from step 2 (step 90).
2) Have an event (step 903). Events include an agent information storage request from the rule distribution 106 (step 904), a rule distribution 106 and a log collection 10
Agent information acquisition request from PC7 (Step 909)
And a termination request from the process management 102 (step 91
There is 3).

When an agent information storage request is received (step 904), information on the agent 20 that has distributed the rule is acquired from the rule distribution 106 (step 905). The information of the agent 20 includes the distribution time of the rule, the distributed rule name, and the like. When communication with the network management system 60 is possible (step 906Y)
ES) passes the obtained agent information to the network management system 60 (step 907). The network management system 60 can use the received rule distribution history information for network management. When communication with the network management system 60 is not possible (step 9
(NO in step 06), the agent information is stored in a file held by the configuration management 104 (step 908).

When an agent information acquisition request is received (step 909), the network management system 60
If the communication with the network management system 60 is possible (YES in step 910), a list of the agents 20, the presence / absence of operation, the response time of ping, and the like are obtained from the network management system 60, and these information are returned to the request source (step 911). When communication with the network management system 60 is not possible (step 910)
In the case of NO), information about the list of agents is acquired from the file of the configuration management 104 and returned to the request source (step 912).

When an end request is received (step 91)
3), exit the loop and perform end processing (step 914).

FIG. 26 is a PAD diagram showing a processing flow of the database management 109 of the manager 10. The database management 109 performs an initial setting (step 1001).
Then, a request is made to confirm the storage period of the normalized log data stored in the normalized log database 50 (step 100).
After 2), the process loops until an end request is received from the process management 102 (step 1003), and has an event (step 1004). In the event, a normalized log data storage notification from the log collection 107 (step 1005), a normalized log data extraction request from the log analysis 103 (step 1008), and the database management 109 itself confirms the storage period of the normalized log data. Request (Step 1)
010) and an end request from the process management 102 (step 1014).

When a notification of storing the normalized log data is received (step 1005), the normalized log data is acquired from the log collection 107 and stored in the normalized log database 50 (step 1006). For storage, the normalized log data is stored according to the data structure of FIG. Since the normalized log data is arranged in the order of the correction time 630, the normalized log data is merged in this order. Also, a request is made to confirm the storage period of the normalized log data (step 100).
7).

When a normalized log data extraction request is received (step 1008), the normalized log database 50 is searched using the specified search key, and the data extracted as a result is returned to the request source (step 1009).

When the storage period confirmation request (step 1010) is received, the difference between the oldest corrected time 630 of the normalized log data stored in the normalized log database 50 and the current time is compared with DB_MAX 451. (Step 1011) When the normalized log data older than the storage period is stored, the old normalized log data is deleted (Step 1012), and a deletion message is output to notify the operator (Step 1013).

When an end request is received (step 10
14), exit from the loop and end processing (step 1015)
I do.

FIG. 27 shows the log input 2 of the agent 20.
FIG. 22 is a PAD diagram showing the flow of the process No. 02. Log input 20
2 is initialized (Step 1101), and the log file monitoring rule 5 is set from the rule management 203 of the agent 20.
After acquiring 00 and format rules 510 and 515 (step 1102), the process loops until an end request is received from the process management 201 (step 1103), and has an event (step 1104). The event includes a log input interruption request from the log file management 204 (step 110).
5), a log input restart request (step 1107), an end request from the process management 201 (step 1109), and a timer interrupt due to time monitoring.

When a log input interruption request is received (step 1105), the log input is interrupted (step 110).
6). The interruption occurs when the capacity of the normalized log file 40 reaches FILE_MAXSIZE 472.

When a log input restart request is received (step 1107), log input is restarted (step 110).
8). The cause of the restart is when the normalized log file is transferred to the manager 10.

When an end request is received (step 11
09), exits from the loop and ends (step 1117)
I do.

When the monitoring time of the log file comes according to the monitoring interval set in the log file monitoring rule 500, the monitoring target log file group 30 is opened (step 1111), and the file management information of this log file is obtained. It is confirmed whether or not the file management information is the same as the one obtained last time (step 1112). If they are the same (YES in step 1112), the log data is input from the offset of the previous file because the content is the same as the file opened last time (step 1113). The offset of the previous file points to the record next to the previously input record for the file. If they are different (step 1112 NO), it is interpreted as a new file (the previously opened file has been deleted, etc.), and log data is input from the beginning (step 1114). Thereafter, the input log data is normalized (step 111).
5) Notify the log file management 204 of the normalized log data (step 1116). Normalization of log data is
This is performed by applying the format rules 510 and 515 as described above. Log file management 20 for normalized log data
Then, the log file is closed, and a timer is set at the next monitoring time according to the monitoring interval.

FIG. 28 is a PAD diagram showing the flow of processing of the log file management 204 of the agent 20. The log file management 204 initializes (step 120
1) Obtain the operation condition rule 470 and the filtering rule 520 from the rule management 203 (step 1202)
Then, the process loops until an end request is received from the process management 201 (step 1203), and has an event (step 1204). Events include a normalized log data storage notification from the log input 202 (step 1205), a log collection request from the log collection 107 of the manager 10 (step 1209), and a normalized log file capacity confirmation request from the log file management 204 itself. (Step 1211), a notification of the manager time from the log collection 107 of the manager 10 (Step 1216), and an end request from the process management 201 (Step 1218).

When the storage notification of the normalized log data is received (step 1205), the normalized log data is obtained from the log input 202 (step 1206), and the time difference between the agent and the manager (processing result of step 1217).
From the log output time 322, the correction time 323 is calculated.
The corrected time 323 is added to the normalized log data acquired from the log input 202 and stored in the normalized log file 40 (step 1207). By applying the filtering rule 520 to the obtained normalized log data, only the normalized log data that matches the conditions is stored in the normalized log file 40. Next, a normalized log file capacity confirmation request is issued (step 1208).

When a log collection request is received (step 1209), the normalized log data in the normalized log file 40 is sorted in the order of the correction time 323, and
Manager 10 shown in ER_ADDRESS 471
(Step 1210).

When a normalized log file capacity confirmation request is received (step 1211), the used capacity of the normalized log file is compared with FILE_MAXSIZE 472 (step 1212). An interruption notification is issued to the input 202 (step 1213). If the maximum size has not been reached (step 1212 NO), it is checked whether or not a previous interruption request has been issued (step 1214). If it has been issued (step 1214 YES), a log input restart request is notified to the log input 202 ( Step 1215).

When the notification of the manager time is received (step 1216), the difference between the computer time of the agent and the computer time of the manager is calculated (step 1217).

When an end request is received (step 12
18), exit from the loop and end processing (step 1219)
I do.

[0061]

As described above, according to the present invention, after an agent monitors a plurality of log files and inputs log data output in various formats, it normalizes and converts the log data to a common data format. I do. In addition, only the necessary log data is extracted, and the corrected time according to the clock of the manager is used as the output time of the log data. And the time can be analyzed.

Since the log data stored by the manager is stored for a predetermined period, the total amount of log information can be regulated by deleting old log data in order.

Further, since the collected log data can be searched by the correction time and the normalized item, the operator can easily obtain necessary log information.

[Brief description of the drawings]

FIG. 1 is a configuration diagram of a network system according to an embodiment.

FIG. 2 is a diagram schematically illustrating a processing operation performed by a manager and an agent according to the embodiment;

FIG. 3 is a functional block diagram illustrating a configuration of a manager 10 according to the embodiment.

FIG. 4 is a functional block diagram illustrating a configuration of an agent 20 according to the embodiment.

FIG. 5 is a diagram illustrating a schematic configuration of normalized log data according to the embodiment.

FIG. 6 is a diagram illustrating a data configuration of a common information class according to the embodiment.

FIG. 7 is a diagram illustrating a data configuration of a user information class according to the embodiment.

FIG. 8 is a diagram illustrating a data configuration of a service information class according to the embodiment.

FIG. 9 is a diagram illustrating a data configuration of an address information class according to the embodiment.

FIG. 10 is a diagram illustrating a data configuration of a file information class according to the embodiment.

FIG. 11 is a diagram illustrating a data configuration of a traffic information class according to the embodiment.

FIG. 12 is a diagram illustrating a data configuration of an individual information class according to the embodiment.

FIG. 13 is a diagram illustrating a data structure of a data storage class according to the embodiment.

FIG. 14 is a diagram showing an example of a coding table of normalization items.

FIG. 15 is a diagram illustrating an example of a manager rule.

FIG. 16 is a diagram illustrating an example of an operation condition rule of an agent.

FIG. 17 is a diagram illustrating an example of a log file monitoring rule of an agent.

FIG. 18 is a diagram illustrating an example of a format rule (1) of an agent.

FIG. 19 is a diagram illustrating an example of a format rule (2) of an agent.

FIG. 20 is a diagram illustrating an example of an agent filtering rule.

FIG. 21 is a diagram illustrating an example of a message text that is log data of a monitoring target log file.

FIG. 22 is a diagram illustrating a data structure of a normalized log database according to the embodiment.

FIG. 23 is a PAD diagram illustrating a flow of a rule distribution process performed by the manager according to the embodiment.

FIG. 24 is a PAD diagram illustrating a flow of a log collection process performed by the manager according to the embodiment.

FIG. 25 is a PAD diagram illustrating a flow of a configuration management process performed by the manager according to the embodiment.

FIG. 26 is a PAD diagram illustrating a flow of a database management process performed by the manager according to the embodiment.

FIG. 27 is a PAD diagram illustrating a flow of a log input process performed by the agent according to the embodiment.

FIG. 28 is a PAD illustrating a flow of a log file management process performed by the agent according to the embodiment.

[Explanation of symbols]

10 ... manager, 20 ... agent, 40
... Normalized log file, 50 ... Normalized log database, 60 ... Network management system, 10
4 ・ ・ ・ Configuration management, 106 ・ ・ ・ Rule distribution, 107 ・
..Log collection, 109 ... database management, 110
... rules, 202 ... log input, 204 ... log file management, 205 ... rules

 ──────────────────────────────────────────────────続 き Continued on the front page (72) Inventor Akihiro Urano 1099 Ozenji Temple, Aso-ku, Kawasaki City, Kanagawa Prefecture Inside System Development Laboratory, Hitachi, Ltd. (72) Inventor Hideki Nakano 2-5-2, Otemachi, Chiyoda-ku, Tokyo No. Within Hitachi Information Network Co., Ltd. (72) Inventor Shinji Morita 5030 Totsuka-cho, Totsuka-ku, Yokohama-shi, Kanagawa Prefecture Inside Software Development Headquarters, Hitachi, Ltd. Address: Hitachi, Ltd. Software Development Division (72) Inventor: Miki Niimura 5030 Totsuka-cho, Totsuka-ku, Yokohama-shi, Kanagawa Prefecture F-term in Hitachi Software Corporation Software Development Division 5B042 GA12 GC15 MA08 MA09 MC35 MC40 5B085 AC11 AC14 AE02 AE03 BA06 BC01 BG02 5B089 GB08 JA36 JB07 KA13 M C03

Claims (4)

[Claims] 1. A method for collecting and managing log information in a system to which a computer is connected, wherein a log analysis item and a data item for normalizing log information are associated with each other and stored as log data analysis information in a first computer, The first computer stores the log data analysis information in the second
The second computer acquires log information using the distributed log data analysis information, and transmits the acquired log information to the first computer. Collection management method. 2. The log data collection management method according to claim 1, wherein the log analysis item is information related to an event that triggers log information acquisition. 3. When the event that triggers the collection of log information is a log-in process, the information related to the event that triggers the acquisition of the log information includes at least a log-in from a place other than a specified log-in area and succeeds. That it has been detected,
Failure to log in outside the specified time, failure of the same user to log in from different regions at the same time, failure of login due to more than the specified number of logins in a certain period, user name and password more than the specified number of times 3. The log data collection management method according to claim 2, further comprising any one of a user change in which the combination of the log data has failed. 4. A data item for normalizing log information stored in association with information related to the event which triggers the log information acquisition, when the event triggering the log information collection is a login process. At least the user name, the connection source host name, the connection source IP address, the connection start time, the log event, the result of the log event, the user name when the user is changed,
3. The log data collection management method according to claim 2, wherein any one of the correction times or a combination thereof is included.