JP5746585B2 - Log management system, log management method, application server, and log server - Google Patents

Description

  The present invention relates to a log management system, a log management method, an application server, and a log server that manage logs in a cloud environment.

  In recent years, a technology called a cloud computing system has attracted attention. In a cloud computing system, a cloud service vendor provides a virtual server (VM), storage, and various services on the Internet. An application developer pays a usage fee to a cloud service vendor and uses those services to develop a cloud application.

  Here, a log management method in a cloud computing system is considered. In the cloud service, services are provided on a plurality of servers (VMs), and a log is output from each service. In Patent Document 1, a client mounted on a server transmits a log output on each server to a log management server. As a result, unified management of logs is possible, and performance and efficiency of operations such as log aggregation, display, and output can be improved.

JP 2010-072984 A

  Consider a system that can be used by many users and logged in using different user IDs (accounts) or the like, such as a cloud service. At this time, since each user can process the same event (for example, printing), the collected log overlaps with the log that is already held and managed only from the item specifying the event. Judgment is not possible.

  In addition, when a certain client falls into a state in which logs cannot be transmitted, if the server performs log aggregation, logs to be collected from a certain client cannot be collected, and log recording during that time occurs.

  In order to solve the above problems, the present invention has the following configuration. That is, a log management system including an application server that provides a service in response to a request from a user and a log server that accumulates a log of the service provided by the application server, the application server receiving a request from the user Log file output means for outputting a log of processing performed according to the file, and the log server that collects the log from the files to be collected among the files output by the log file output means One or more log data collecting means for transmitting to the log, and an providing means for generating an identifier for uniquely identifying the one or more log data collecting means and providing the identifier to the log server. The server receives the log transmitted from the application server and stores it in the storage unit. And the storage means receives an identifier provided by the application server associated with the received log and an identifier associated with the log already stored in the storage unit, and receives The other information for uniquely identifying the log included in the recorded log matches the other information for uniquely identifying the log included in the log already stored in the storage unit The log management system is characterized in that the received log is not stored in, and the received log is stored in other cases.

  According to the present invention, in a system that performs processing in parallel in response to requests from a plurality of users, it is possible to prevent duplicate logs from being saved and avoid omission of log recording for logs as a result of processing. It becomes possible.

System Configuration. The hardware block diagram of the application server 102. FIG. Functional block diagram. The figure which shows the example of a description of the setting file of a log collection part. The figure which shows the definition example of I / F of a log filter, and a LogData class. The figure which shows the example of a description of ID file. The figure which shows the example of a description of the record file at the time of initial registration of a record. The figure which shows the structural example of log DB. The figure which shows the structural example of status DB. The figure which shows the example of a description of the setting file of an archive application. The flowchart regarding starting of a log collection part, and preparation of ID file. The flowchart regarding the log collection of a log collection part. The flowchart regarding the log output of a batch application server. Application server processing flowchart for failover

  Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

[System configuration]
FIG. 1 shows the overall configuration of a log management system according to an embodiment of the present invention. A PC 101, application server 102, log server 103, storage server 104, and batch application server 105 are connected via the network 100. The application server 102 receives a request transmitted from the PC 101 via the network 100 and performs processing. The log server 103 has a function of receiving a log transmitted from the application server 102 and storing it in the storage server 104, and a function of receiving a log data output request from the batch application server 105 and performing a corresponding process. . The storage server 104 stores the logs collected by the log server 103.

  The network 100 functions as a communication line for exchanging information between the devices described above, and the form of wired, wireless, and line is not limited. The network 100 is configured by combining an external network such as the Internet and an internal network such as a LAN.

  In FIG. 1, two application servers 102 are shown, but the present invention is not limited to this. Similarly, each server is not limited to a single configuration, and a plurality of physical configurations may be provided. In this embodiment, each server is shown as a separate device for each function, but may be realized by a single device. For example, the log server 103, the storage server 104, and the batch application server 105 may be configured by a single device. In addition, a plurality of VMs may be configured to physically operate in one device, or a plurality of VMs may operate in cooperation with a plurality of physical devices.

[Hardware configuration]
FIG. 2 shows a hardware configuration according to an embodiment of the present invention. It is assumed that each device shown in FIG. 1 has the same configuration, and here, the application server 102 will be described as an example.

  The application server 102 includes a CPU (Central Processing Unit) 201, a direct storage unit 202, an indirect storage unit 203, and a network interface 204. The CPU 201 is a unit that executes a predetermined program stored in each storage unit and instructs various controls of the application server 102. The direct storage unit 202 is a work memory used when the CPU 201 executes a program, and the program executed by the CPU 201 is loaded into the direct storage unit 202. The direct storage unit 202 is realized by a RAM (Random Access Memory). The indirect storage unit 203 stores application programs and various programs including an OS (Operating System).

  Various programs stored in the indirect storage unit 203 are read directly to the storage unit 202 when the CPU 201 executes the programs. The indirect storage unit 203 is realized by a ROM (Read Only Memory), a HDD (Hard Disc Drive), or an SSD (Solid State Drive). Note that the CPU 201 may be a multiprocessor. Each component in the application server 102 is connected to be communicable with each other via an internal bus (not shown). The network interface 204 is connected to the network 100 and can communicate with other devices connected to the network 100. Since the PC 101, the log server 103, the storage server 104, and the batch application server 105 have the same hardware configuration, description thereof is omitted. In addition, each device may include components not shown here, and can be realized by a general-purpose computer.

Software configuration
FIG. 3 is a functional block diagram (software configuration) of each device constituting the system according to the embodiment of the present invention. First, the PC 101 will be described. The PC 101 includes a network communication unit 313, a user interface 312, and a web browser 311. The user interface 312 is an interface for the user to use various applications (not shown) provided by the PC 101. The PC 101 executes various applications in accordance with user operations input via this. A web browser 311, which is a type of application, displays various types of information according to information transmitted and received from the network communication unit 313 via the network 100.

  The web browser 311 is realized by a program stored in the indirect storage unit 203 in FIG. 2 being loaded into the direct storage unit 202 and executed by the CPU 201. In this embodiment, a user who operates the PC 101 communicates with the request processing unit 3211 of the application server 102 via the user interface 312 and the network communication unit 313 to transmit and receive data.

  Next, the request processing unit 3211, the log file output unit 3212, and the request processing unit state monitoring unit 3213 included in the application unit 321 mounted on the application server 102 will be described. The request processing unit 3211 receives a request from the PC 101 and performs processing according to the received content. For example, assume that the application server 102 implements an online print service. The user selects a document to be printed using the PC 101, designates print settings and the like, and transmits a print processing request to the application server 102. The application server 102 performs processing such as converting the document into a printable format in accordance with the received request and transmitting the document to a print server (not shown).

  Here, an example of a print system has been described as an example, but there is no limitation on the functions realized by the application server 102. The request processing unit 3211 has a function equivalent to that of a general Web server and can perform synchronization processing. The request processing unit 3211 can communicate with another application server 102 via the network 100. In addition, the request processing unit 3211 periodically transmits a packet indicating that it is operating to the request processing unit state monitoring unit 3213.

  The log file output unit 3212 outputs the processing log to a file according to the processing content of the request processing unit 3211. At this time, the output log needs to record the time at which the request was processed. Regarding other items (for example, a user count and a log relating to the contents of processing), arbitrary items may be output. There are no particular restrictions on the log format, and output in various formats such as a comma-delimited or tab-delimited format is possible. In addition, logs can be output to different files for each processing content and application.

  The request processing unit state monitoring unit 3213 monitors whether or not a packet is periodically transmitted from the request processing unit 3211. If a packet has not been transmitted, the request processing unit state monitoring unit 3213 assumes that the request processing unit 3211 is in an operation continuation impossible state and performs failover. The failover flow will be described later with reference to FIG.

  Next, the log collection agent unit 322 implemented on the application server 102 will be described. The log collection agent unit 322 includes a plurality of processing units (a log data collection unit 3221, a log data transmission unit 3222, a status transmission unit 3223, a log filter unit 3224, and an ID management unit 3225). Below, description of each process part is described.

  First, the log data collection unit 3221 reads a log from the log file output to the file by the log file output unit 3212. As described above, the log file output unit 3212 outputs a log described in an arbitrary format to a file. In order to save the log in accordance with the schema defined in the log DB unit 341 of the storage server 104, it is necessary to convert the log described in an arbitrary format into a format according to the schema. Therefore, the log filter unit 3224 is used to convert the log format described in the file into a format suitable for the schema.

  The log filter unit 3224 converts the log passed from the log data collection unit 3221 into a format according to the schema, and returns it to the log data collection unit 3221. By converting the log filter unit 3224 into an appropriate format, it is possible to cope with an arbitrary format. Details of the log filter unit 3224 will be described with reference to FIG. Here, a plurality of log collection agent units 322 can be operated in one application server 102. The log collection agent unit 322 includes setting files and can define the name of the log file to be collected for each.

  The log filter unit 3224 can also be set for each. Thus, for example, the case where the log file output unit 3212 outputs a plurality of log files in different formats can be handled.

  The log data collection unit 3221 passes the log data converted by the log filter unit 3224 to the log data transmission unit 3222. The log data transmission unit 3222 transmits the log data passed from the log data collection unit 3221 to the log DB request processing unit 331. The log DB request processing unit 331 stores the received log data in the log DB unit 341. The log data stored in the log DB unit 341 includes information such as the time and the user ID that performed the process for generating the log, the contents of the process, and the service based on the information. It is also possible to make a charge according to the usage amount. The schema of the log DB unit 341 will be described later.

  In addition, when the log data transmission process ends normally, the log data collection unit 3221 notifies the status transmission unit 3223 of the time when the log is collected from the log file. The status transmission unit 3223 transmits the time to the status DB request processing unit 333. At this time, as information for identifying the log collection agent unit 322, an ID assigned to each log collection agent unit is transmitted in association with each other. The status DB request processing unit 333 stores the received ID and time pair in the status DB unit 342 of the storage server 104. This pair of ID and time is treated as a status, and the status is information indicating the time until which the log collection agent unit 322 having the ID included in the status is stored.

  The log collection agent unit 322 includes an ID management unit 3225 as a processing unit for issuing an ID. The ID management unit 3225 starts processing simultaneously with the activation of the log collection agent unit 322, issues a unique ID to the log collection agent unit, and notifies the log data collection unit 3221 of the ID information. This ID is managed as a file (ID file) on the application server 102. The format, file name, storage path, etc. of the ID file will be described later with reference to FIG.

  Next, each processing unit included in the batch application server 105 will be described. On the batch application server 105, an application (archive application) for outputting a log stored in the log DB unit 341 is implemented. First, the acquisition date / time specifying unit 351 specifies a time range to be acquired from the log DB unit 341 and notifies the log data acquisition unit 353 of the range.

  Next, the status confirmation unit 352 transmits a status acquisition request to the status DB request processing unit 333 of the log server 103. Also, the status confirmation unit 352 transmits an acquisition request for the time of the log having the oldest time among the logs stored in the log DB unit 341 to the log DB request processing unit 331 of the log server 103. . When receiving the request, the log DB request processing unit 331 acquires the oldest time from the time of the log stored in the log DB unit 341 and transmits the acquired time to the status confirmation unit 352. Thereafter, the status confirmation unit 352 notifies the log data acquisition unit 353 of the log time of the oldest received time and the time recorded in the status.

  The log data acquisition unit 353 compares the times notified from the acquisition date / time specifying unit 351 and the status confirmation unit 352. The comparison logic will be described in detail in the explanation of the flowchart described later. As a result of the comparison, if it is determined that the log data in the range notified by the acquisition date / time specifying unit 351 can be acquired, a log acquisition request is transmitted to the log DB request processing unit 331. When receiving the request, the log DB request processing unit 331 acquires a log from the log DB unit 341 and transmits the log to the log data acquisition unit 353. The log data acquisition unit 353 passes the received data to the log data output unit 354. The log data output unit 354 formats the format of the received log data and outputs it to a file.

  Next, the log server 103 will be described. First, the log DB request processing unit 331 receives a request regarding processing to the log DB unit 341 of the storage server 104 transmitted from the application server 102 and the batch application server 105. Then, the log DB request processing unit 331 performs processing according to the received request. In addition, when receiving the request, the log DB request processing unit 331 transmits a processing request to the log DB I / O unit 332. The log DB I / O unit 332 executes processing for the log DB unit 341 according to the request received from the log DB request processing unit 331. Similarly, the status DB request processing unit 333 receives a request regarding processing for the status DB unit 342 transmitted from the application server 102 and the batch application server 105. Then, the status DB request processing unit 333 performs processing according to the received request. Further, when receiving the request, the status DB request processing unit 333 makes a processing request to the status DB I / O unit 334. The status DB I / O unit 334 executes processing corresponding to the request received from the status DB request processing unit 333 with respect to the status DB unit 342.

  Next, the storage server 104 will be described. The log DB unit 341 accumulates and stores each log collected via the log server 103. The status DB unit 342 holds the status and ID of each log collection agent unit included in the application server 102 in association with each other. The data structure here will be described later with reference to FIG. Each DB (database) unit is realized by an indirect storage unit provided in the storage server 104.

[Setting file for log collection agent section]
The items that can be set in the setting file included in each log collection agent unit will be described with reference to FIG. First, the filter name 4011 and the assembly name 4012 which are setting items related to the log filter unit 3224 will be described. In the present embodiment, the application server 102 can provide a plurality of virtually different servers in one physical device using the VM technology. In this case, a plurality of setting files are held in one apparatus. As described above, the log collection agent unit 322 includes the log filter unit 3224. Since the log filter unit 3224 can change depending on the format of the log file to be collected, it is desirable that the log filter unit 3224 can be changed without changing the log data collection unit 3221. Therefore, the log filter unit 3224 is mounted in a format that can be linked at the time of execution (for example, a DLL (Dynamic Link Library) file in the case of Windows (registered trademark)).

  In addition, the assembly path is described in the setting file so that the log data collection unit 3221 can dynamically read it at the time of execution, so that the log filter unit 3224 can be easily changed. Here, a plurality of logFilter tags in the setting file can be created, and settings related to a plurality of log filter units can be described in one setting file. At that time, the filter name 4011 assigned to each log Filter tag needs to be set to be unique.

  Next, the agent tag will be described. In the agent tag, settings related to the log data collection unit 3221 are described. The following describes the setting items. The Agent name 4013 is a name that uniquely identifies the agent. A plurality of agent tags can be described, and the agent name 4013 assigned to each agent tag needs to be unique in order to identify the log data collection unit 3221.

  Next, in interval 4014, the activation interval of the agent (log data collection unit 3221) is set. In accordance with the specified interval, the log data collection unit 3221 performs log reading processing from the log file of each service. The dir 4015 describes a folder path where a log file to be collected by the log data collection unit 3221 exists. Further, filenameRegEx 4017 describes a regular expression of a collection target file name. The log data collection unit 3221 collects all files that exist in the dir 4015 and match the filenameRegEx 4017.

  In the includeSubDir 4016, it is set whether or not the logs included in the subfolders existing under the dir 4015 are to be collected. A log name 4011 to be used is set in logFilter-ref 4018. The assembly that matches the logFilter name set here is loaded, and the log filter unit 3224 is created. In logServer 4019 and statusServer 4020, the addresses of the log DB request processing unit 331 and the status DB request processing unit 333 to which the log data collection unit 3221 is connected are set. The log collection agent unit 322 reads this setting file at the time of activation and performs processing according to the setting. A detailed processing flow will be described later with reference to FIG.

[Log Filter I / F]
FIG. 5A shows a definition example of the log filter I / F. When creating the log filter unit 3224 of the log collection agent unit 322, it is necessary to create a module in which the log filter I / F 501 is mounted. In the example shown in FIG. 5A, C # is selected as the programming language, and an example of I / F definition based on C # is described. However, the programming language is not limited, and any programming language may be used as long as a module that can be loaded by the log collection agent unit 322 can be created.

  When C # is used as the program language, the interface definition portion is composed of a class 5011, a return value 5012, a function 5013, and an argument 5014 as shown in FIG. The creator of the log filter unit 3224 creates a class in which a class that inherits the class 5011 is created, and implements a function including the function 5013 in the class. In this function, a one-line character string read from the log file by the log collection agent unit 322 is given to the argument 5014. The creator of the log filter unit 3224 analyzes the argument 5014, creates a class having properties as shown in FIG. 5B, and implements a function that is returned as a return value 5012.

  FIG. 5B shows a definition example of properties of the LogData class. Similarly, the definition of this property shows a definition example when C # is used as a programming language.

  A description of each property is given below. TimeStamp 5021 is a property for setting the time (time stamp) described in the log. TenantId 5022 is a property for setting an ID for identifying the tenant (company, organization) to which the user who performed the operation that triggered the log output belongs. UserId 5023 is a property for setting an ID for identifying the user who performed the operation that caused the log to be output. ServiceId 5024 is an ID for representing the type of operation performed by the user, for example, the type of operation such as Print or Scan. Finally, ObjectID 5025 is a property for setting a processing target, for example, in the case of printing, the name of a printed document.

  In this definition example in FIG. 5B, the above five properties are defined, but the present invention is not limited to this. When other items are to be recorded as a log, a property is added to the LogData class 502, and the log filter unit 3224 is implemented so that a value is set for the item when the function 5013 is implemented. If items other than TimeStamp 5021 are not recorded as a log, the property may be deleted from the class definition.

[ID file]
The ID file will be described with reference to FIG. The ID file according to the present embodiment is configured by a schema as shown in FIG. The ID file is composed of an agentId 6011 tag, and an identifier (GUID: Global Unique IDentifier) for uniquely identifying the log collection agent unit 322 is set in the attribute id. The GUID is created by the log collection agent unit 322 itself when the log collection agent unit 322 is activated. The created ID file is stored in the folder indicated by dir 4015 in the setting file of the log collection agent unit 322 described above. If the folder does not exist, the log collection agent unit 322 is not activated. The file name of the ID file uses a character string generated from filenameRegEx 4017 of the setting file.

  Since a regular expression is set in filenameRegEx 4017 as described above, a character string that cannot be specified in a file name such as “?” Or “¥” may be set. Therefore, a conversion process is performed from a character string specified in filenameRegEx 4017 to a character string that can be used as a file name. Specifically, character string conversion using Base64 encoding or replacement of a character string such as “¥” with a character string such as “bs” can be given. Here, as a condition at the time of conversion, there is a mapping relationship between the character string before conversion and the character string after conversion.

  In this embodiment, when a plurality of VMs (application servers) are operated in one physical device, an ID file is generated for each log collection agent unit 322 of each VM. It will be.

[Record file]
As described above, the log data collection unit 3221 periodically reads a log file. Then, the log data collection unit 3221 shapes only the newly added log using the log filter unit 3224, and stores the log data in the log DB unit 341 of the storage server 104. In this way, it is possible to improve performance by targeting only newly added logs. In order to target only newly added logs, it is necessary to record the location of the files collected by the log data collection unit 3221 and the time of collection at the time of the previous collection. The data is described in a file called a record file indicating the log collection status. Here, the record file is dynamically created in the folder where the log collection agent unit 322 exists. The file name of the record file is a file name in which “.xml” is added to the end of the Agent name 4013 in the setting file of the log collection agent unit 322.

  The record file according to the present embodiment will be described with reference to FIG. The record file is a file having a format as shown in FIG. The record tag is composed of four attributes: filename 7011, updateTime 7012, offset 7013, and lineNumber 7014. The record tags are created for the number of files to be collected by the log data collection unit 3221. The file name 7011 is set with the name of the log file collected by the log data collection unit 3221. In updateTime 7012, the log data collection unit 3221 sets the time when the previous collection was performed. If the file indicated by filename 7011 has not been collected in the past, “1970/01/01 00:00:00” is set as the initial value.

  In offset 7013, the number of bytes (data size) from the beginning of the file to the location where the log data collection unit 3221 has completed collection is set. Finally, the line number 7014 is set with the line number of the log that has been collected. When collection processing is not performed on the file indicated by filename 7011, “0” is set in offset 7013 and lineNumber 7014. The log data collection unit 3221 is activated for each interval 4014 described above and reads a record file. Thereafter, the update time of filename 7011 is checked, and the time is compared with updateTime 7012. If the updateTime 7012 is older, the log data collection unit 3221 moves the file by the number of bytes indicated by the offset 7013 from the beginning of the file, and starts collecting logs. When log collection is completed, each item is updated. In this way, only newly added logs can be collected. Although each information is described in FIG. 7 in an XML (extensible Markup Language) format, it is not necessary to be in the XML format. There is no problem even if each item is described in comma delimited (CSV format) or the like.

[Log DB]
The log DB 801 held in the log DB unit 341 according to the present embodiment will be described with reference to FIG. First, there are four items required for the log DB 801: TimeStamp 8011, AgentID 8012, FileName 8013, and LineNumber 8014. TimeStamp 8011 stores the time when the event recorded in the log has occurred. The Agent ID 8012 stores an ID described in the ID file of the log collection agent unit 322 that collected the log. FileName 8013 describes the file path of the log file in which the log is recorded. Finally, LineNumber 8014 stores a line number that can be placed in the log file of the log.

  These four items are necessary to prevent the same log from being recorded twice or more in the log DB 801 of the log DB unit 341 (hereinafter referred to as double recording). For example, when the record file described above cannot be read for some reason, the log data collection unit 3221 collects again from the beginning of all the files existing in the collection target folder. At this time, since the already collected logs are stored again in the log DB unit 341, double recording occurs as a result. Therefore, in the present embodiment, the log DB 801 is restricted so that the above four items are unique. For example, in the case of RDBMS (Relativistic DataBase Management System), it is possible to prevent double recording by restricting the above four items to be Unique. In the case of the key-value store, by defining the combination of the above four character strings as a key, data having the same key cannot be stored, and it is possible to prevent duplicate recording.

  Items other than the above four items can be arbitrarily added depending on the items that need to be left as a log. For example, in the case of FIG. 8, Tenant ID 8015, User ID 8016, Service ID 8017, and Object ID 8018 are listed as log items. In the TenantID 8015, an organization (for example, a company) to which the user who performed the process belongs is recorded. In UserID 8016, information for specifying the user who performed the process is recorded. Here, the user does not need to be specified only by UserID. For example, it is sufficient that the user can be uniquely specified by a combination with TenantID.

  Service ID 8017 records the ID of the service used by the user. For example, a service ID of a function such as a user addition function, Print, or Scan is recorded. Finally, in ObjectID 8018, the target to be processed is recorded. For example, in the case of a user addition process (Add User), the added user ID is recorded. In the case of Print or Scan, the name of the target document is recorded. When an item other than the items listed above is recorded, a new item can be recorded by adding a column of the item.

  In this embodiment, it is determined whether or not to store a log using four items of TimeStamp 8011, AgentID 8012, FileName 8013, and LineNumber 8014 as keys. However, the present invention is not limited to this, and any item other than AgentID 8012 may be any of these items, or another item may be added. That is, it is possible to specify the log by using AgentID 8012 as an essential key item and further using information of one or more other key items. In addition, when changing the item used here, it is necessary to also change the information acquired in the LogData class 502 shown in FIG.5 (b).

[Status DB]
The status DB held by the status DB unit 342 according to the present embodiment will be described with reference to FIGS. In the status DB unit 342 of the storage server 104, the ID of the log collection agent unit 322 (AgentID 9011, 9021) and the time when the log collection agent unit 322 collected the log (LastCrawlTime 9012, 9022) are recorded as the status DB. The The time recorded in the LastCrawlTime is a value for guaranteeing that all logs that have been output to the log file up to this time are stored in the log DB unit 341.

  First, the contents registered in the status DB unit 342 at the time of initial registration will be described with reference to FIG. The log collection agent unit 322 reads (or creates) an ID file at the time of activation. Therefore, by transmitting the created ID to the status DB unit 342, data corresponding to its own ID is registered in the status DB unit 342. At that time, the LastCrawlTime 9012 is registered with a time (1970/01/01 00:00:00) representing the initial registration status. In this embodiment, “1970/01/01 00:00:00” is used as the initial registration time, but there is no particular problem even if, for example, a value more than one year past the current time is used. The condition as the value of the initial registration used here is that the date and time when this system is used is not equal.

  After initial registration of the log collection agent unit 322, the log data collection unit 3221 performs log collection processing. The log collection agent unit 322 transmits the ID and the current time in association with each other to the status DB request processing unit 333 after the log storage processing in the log DB unit 341 is successful. The status DB request processing unit 333 extracts the ID from the received data, and updates it with the time when the LastCrawlTime 9022 of the entity having Agent IDs 9011 and 9021 equal to the ID is received. Thus, the log collection agent unit 322 updates the status DB unit 342.

[Archive application settings file]
The definition of the setting file 10000 of the archive application (not shown) that operates on the batch application server 105 according to the present embodiment will be described with reference to FIG. Here, the archive application is an application having a function of outputting log data stored in the log DB unit 341 to a file. The setting file of the archive application is described in XML format as described in FIG. Here, the format of the setting file is not limited to the XML format but may be, for example, a CSV format (Comma Separated Values) in which each item is separated by a comma or the like.

  The archive application first has an archiveSetting tag in the setting file 10000. This tag is composed of saveDirectory 10001, logDbUri10002, statusDbUri10003, startDate10004, and endDate10005. The saveDirectory 10001 indicates a directory path for saving a file output by the archive application. The archive application acquires a log for each date and creates a file with the acquired date as a file name.

  For example, when the log of January 1, 2011 is acquired from the DB, a file having a file name such as “201110101.csv” is stored in the folder indicated by the saveDirectory 10001. In logDbUri10002, the address of the log DB request processing unit 331 is described. In statusDbUri10003, the address of the status DB request processing unit 333 is described. The startDate 10004 describes the start date of the period for storing the log. In the case of the example illustrated in FIG. 10, the log whose TimeStamp 8011 of the log stored in the log DB unit 341 is “January 5, 2011 0: 0: 0” or later is targeted. The endDate 10005 describes the end date of the period for storing the log. In the case of the example illustrated in FIG. 10, a log whose TimeStamp 8011 of the log stored in the log DB unit 341 is less than “01.20.2011 0: 0: 0” is targeted.

  The log stored in the log DB unit 341 can be stored outside using an archive application. Since the stored log can be deleted from the log DB unit 341, it is possible to secure storage capacity and reduce costs related to storage specifications. Although not shown in FIG. 10, an Agent ID may be specified in the setting file 10000. In this case, it is possible to output only the logs collected by a specific log collection agent unit.

[flowchart]
The flow of processing of the application server 102, log server 103, batch application server 105, and storage server 104 in the present invention will be described in detail with reference to FIGS.

[Start Log Collection Agent]
First, the flow of processing when the log collection agent unit 322 is activated will be described with reference to FIG. In the present embodiment, this processing flow is realized by the CPU provided in each of the application server 102 and the log server 103 reading and executing the program stored in the indirect storage unit directly into the storage unit.

  In step S <b> 1101, the OS (not shown) of the application server 102 activates the log collection agent unit 322.

  In S1102, the log collection agent unit 322 reads a setting file as shown in FIG. If reading of the setting file has failed (NO in S1103), the process proceeds to S1123. If the setting file has been successfully read (YES in S1103), the process proceeds to S1104.

  In S1104, the log collection agent unit 322 checks whether all agents have been generated based on the settings of all agent tags included in the agents tag described in the setting file. That is, the log collection agent unit 322 determines whether all the log data collection units defined in the agents tab of the setting file have been generated. If an agent is generated based on all agent tags (YES in S1104), the process proceeds to S1118. If not all agents have been generated (NO in S1104), the process proceeds to S1105.

  In step S1105, the log collection agent unit 322 checks whether there is a folder to be collected by the agent (log data collection unit) indicated by dir4015 described in the setting file. If the folder does not exist (NO in S1105), the process proceeds to S1123. If the folder does not exist (YES in S1105), the process proceeds to S1106. In S1106, the log collection agent unit 322 checks whether an ID file having a file name generated from filenameRegEx 4017 exists in the existing folder. If the ID file does not exist (NO in S1106), the process proceeds to S1109. If the ID file exists (YES in S1106), the process proceeds to S1107.

  In S1109, the log collection agent unit 322 creates an ID file and writes the ID in the file. At this time, if the file creation fails (NO in S1110), the process proceeds to S1123. If the file has been successfully created (YES in S1110), the process proceeds to S1111.

  In step S1107, the log collection agent unit 322 reads ID information corresponding to the log collection agent unit from the ID file as illustrated in FIG. At this time, if the ID file reading process has failed (NO in S1108), the process proceeds to S1123. If the ID file reading process is successful (YES in S1108), the process proceeds to S1111.

  In S1111, the log collection agent unit 322 loads the module indicated by the assembly name 4012 described in the setting file of FIG. 4 and creates the log filter unit 3224 shown in FIG.

  In S1112, the log collection agent unit 322 generates a log data collection unit 3221. In S1113, the log collection agent unit 322 checks whether a file with “.xml” appended to the end of the Agent name 4013 of the setting file in FIG. 4, for example, a record file as shown in FIG. To do. If the record file exists (YES in S1113), the process proceeds to S1114. If the record file does not exist (NO in S1113), the process proceeds to S1115.

  In S1114, the log collection agent unit 322 reads the record file. In S1115, the log collection agent unit 322 creates a record file. Here, in the record file, all log files that exist in the folder path specified in the dir 4015 of the setting file and have a file name that matches the filenameRegEx 4017 are initially registered in the record file. That is, as many elements as the record tab shown in the record file 701 in FIG. 7 are registered for the number of log files.

  In S <b> 1116, the log collection agent unit 322 transmits its own agent ID as illustrated in FIG. 6 to the status DB request processing unit 333 of the log server 103. In S1112, the log server 103 performs initial registration in the status DB unit 342 of the storage server 104 using the received status ID.

  In S1118, the log collection agent unit 322 confirms whether or not the processing of all the log data collection units 3221 generated from the setting file has been started. If all log data collection units 3221 have not started processing (NO in S1118), the process proceeds to S1119. If log data collection unit 3221 generated from all agent tags has already started processing (YES in S1118), the process proceeds to S1123.

  In step S <b> 1119, the log collection agent unit 322 starts processing of the log data collection unit 3221. In S <b> 1120, the log collection agent unit 322 collects logs and transmits them to the log server 103. Details of this processing will be described with reference to the flowchart of FIG. In S <b> 1121, the log server 103 stores the received data in the log DB unit 341 of the storage server 104.

  In S1122, the log collection agent unit 322 sets a timer so that the log data collection unit 3221 restarts the process after the number of seconds set in the interval 4014 of the setting file, and waits. Then, when a predetermined time has elapsed, the log data collection unit 3221 resumes the processing. In S1123, the log collection agent unit 322 assumes that an error has occurred during the processing, and stops the processing.

[Processing flow of the log data collection unit]
The processing flow of the log data collection unit 3221 will be described with reference to FIG. In the present embodiment, this processing flow is realized by the CPU provided in each of the application server 102 and the log server 103 reading and executing the program stored in the indirect storage unit directly into the storage unit.

  In step S <b> 1201, the OS / timer of the application server 102 issues a process restart instruction for the log data collection unit 3221. Here, the case where the restart command is issued corresponds to the case where the processing of the log data collection unit 3221 in the standby state is restarted as indicated by 1122 in FIG. In step S <b> 1202, the log data collection unit 3221 checks whether there is a stop instruction for the log collection agent unit 322. If there is a stop command (YES in S1202), the process proceeds to S1217. If there is no stop command (NO in S1202), the process proceeds to S1203.

  In step S1203, the log data collection unit 3221 acquires a list of files that exist in the folder path described in the dir 4015 of the setting file illustrated in FIG. 4 and have file names that match the filenameRegEx 4017. In S1204, the log data collection unit 3221 initially registers, in the record file, a file having a file name that is not recorded in the record file in the file list acquired in S1203.

  In step S1205, the log data collection unit 3221 compares updateTime 7012 recorded in the record file with the update time of the file, and acquires a list of files updated after the time indicated by updateTime 7012. In step S1206, the log data collection unit 3221 confirms the number of file lists acquired in step S1205. If the number of files is “0”, that is, if an updated log file does not exist (NO in S1206), the process proceeds to S1213. If the number of files is 1 or more, that is, if an updated file exists (YES in S1206), the process proceeds to S1207.

  In step S1207, the log data collection unit 3221 confirms whether the processing in steps S1208 to S1211 has been performed on the file list acquired in step S1205. That is, the log data collection unit 3221 determines whether or not all log files have been collected. If the processes of S1208 to S1211 are not performed (NO in S1207), the process proceeds to S1208. If the processing from S1208 to S1211 has been performed on all the files included in the file list (YES in S1207), the process proceeds to S1212.

  In step S1208, the log data collection unit 3221 reads a log from one file included in the file list acquired in step S1205. Then, the log data collection unit 3221 shapes the log using the log filter unit 3224 and transmits the shaped log to the log DB request processing unit 331 of the log server 103. At the time of transmission, the data shaped by the log filter unit 3224 is associated with the file name read by the log data collection unit 3221, the number of log lines, and the ID assigned to itself.

  In S1209, the log server 103 receives the log data transmitted from the log data collection unit 3221 in S1208, and stores the data in the log DB unit 341 of the storage server 104. Then, the log server 103 transmits to the log data collection unit 3221 whether the storage in the log DB unit 341 is successful.

  In step S <b> 1210, the log data collection unit 3221 receives the processing result transmitted from the log server 103 and determines whether the log storage by the log server 103 is successful. If the processing result is a normal end indicating that the storage was successful (YES in S1210), the process proceeds to S1211. If the processing result indicates abnormal termination (NO in S1210), the process proceeds to S1207.

  In S1211, the log data collection unit 3221 updates the record file. Specifically, updateTime 7012 is the current time, offset 7013 is the number of bytes at the end of the file, and lineNumber 7014 is the line number of the last line of the file. After updating the record file, the process proceeds to S1207.

  In step S1212, the log data collection unit 3221 checks whether the processing in steps S1208 to S1211 has been successful for all the files acquired in step S1205. If the processing of S1208 to S1211 has succeeded for all files (YES in S1212), the process proceeds to S1213. If the processes in S1208 to S1211 have failed for one or more log files (NO in S1212), the process proceeds to S1215.

  In step S <b> 1213, the log data collection unit 3221 transmits data that associates the time representing the LastCrawlTime with its own ID to the status DB request processing unit 333 of the log server 103. Here, the time representing the LastCrawlTime to be transmitted is the time acquired before reading the log from the log file in S1208.

  In S <b> 1214, the log server 103 receives data associated with the ID and the LastCrawlTime transmitted from the log data collection unit 3221. After receiving, the log server 103 updates the LastCrawlTime included in the entity of the status DB having the Agent ID equal to the received ID with the received LastCrawlTime.

  In step S1215, the log data collection unit 3221 checks whether the log collection agent unit 322 has a stop command. If there is a stop command (YES in S1215), the process proceeds to S1217. If there is no stop command (NO in S1215), the process proceeds to S1216.

  In S1216, the log data collection unit 3221 sets a timer so that the process of S1201 is called after the number of seconds described in the interval 4014 of the setting file. In S1217, the log data collection unit 3221 stops the process.

[Processing flow of batch application server 105]
A processing flow of an archive application (not shown) existing in the batch application server 105 will be described with reference to FIG. In the present embodiment, this processing flow is realized by the CPU included in each of the log server 103 and the batch application server 105 reading and executing the program stored in the indirect storage unit directly into the storage unit.

  In S1301, the archive application obtains a log storage period from StartDate 10004 and EndDate 10005 described in the setting file shown in FIG. In the following description, the time indicated by StartDate 10004 is referred to as “log acquisition start date”, and the time indicated by EndDate 10005 is referred to as “log acquisition end date”. In S <b> 1302, the archive application issues a request for acquiring the end date of the log acquirable range to the status DB request processing unit 333 of the log server 103.

  In S <b> 1303, the status DB request processing unit 333 of the log server 103 acquires a list of LastCrawlTime 9022 from the status DB unit 342 of the storage server 104. In step S1304, the status DB request processing unit 333 checks whether an element is included in the acquired list of LastCrawlTime. If no element is included in the acquired LastCrawlTime list (NO in S1304), the process proceeds to S1316. If an element is included (YES in S1304), the process proceeds to S1305.

  In S1305, the status DB request processing unit 333 checks whether or not an initial value (“1970/01/01” in the present embodiment) exists in the list of LastCrawlTime acquired in S1303. If an initial value exists in the list (YES in S1305), the process proceeds to S1316. If there is no initial value in the list (NO in S1305), the process proceeds to S1306.

  In S1306, the status DB request processing unit 333 acquires the oldest time in the list of LastCrawlTime acquired in S1303. Then, the status DB request processing unit 333 transmits the acquired time to the archive application of the batch application server 105.

  In step S1307, the archive application compares the received time with the log acquisition end date. If the time acquired in S1306 is newer than the time indicated by the log acquisition end date (YES in S1308), the process proceeds to S1321. If the time acquired in S1306 is older than the time indicated by the log acquisition end date (NO in S1308), the process proceeds to S1309.

  In S <b> 1309, the archive application issues a request for acquiring a TimeStamp of a log having the oldest TimeStamp among logs existing in the log DB unit 341 of the storage server 104 to the log DB request processing unit 331 of the log server 103.

  In step S <b> 1310, the log DB request processing unit 331 acquires the TimeStamp of the log having the oldest TimeStamp among the log DB units 341, and transmits it to the batch application server 105.

  In S1311, the archive application compares the time acquired from the log server 103 with the log acquisition start date. If the log acquisition start date is newer (YES in SS 1312), the process proceeds to S1313. If the log acquisition start date is older than the acquired time (NO in S1312), the process proceeds to S1321.

  In S1313, the archive application issues a log acquisition request between the log acquisition start date and the log acquisition end date to the log DB request processing unit 331 of the log server 103. In S <b> 1314, the log DB request processing unit 331 of the log server 103 acquires all the logs having the TimeStamp corresponding to the log acquisition end date from the received log acquisition start date from the log DB unit 341.

  In S <b> 1314, the log DB request processing unit 331 of the log server 103 sets a value indicating normal termination to the termination code. Here, there is no particular value restriction on the code indicating normal termination. In other words, any code may be defined as long as it indicates normal termination. In S1316, the log DB request processing unit 331 sets a code indicating abnormal termination as the termination code.

  In S1317, the log DB request processing unit 331 transmits a set of all log data acquired in S1314 and the end code set in S1315 or S1316 to the archive application of the batch application server 105.

  In S <b> 1318, the archive application of the batch application server 105 receives a set of the log data set and the end code transmitted from the log DB request processing unit 331 of the log server 103. In S1319, the archive application confirms whether or not the received end code is an error code. If the end code indicates an abnormal end (NO in S1319), the process proceeds to S1321. If the end code is a normal end code (YES in S1319), the process proceeds to S1320.

  In S1320, the archive application formats the received log data into the CSV format and outputs it to a file. Here, the format of the output file need not be the CSV format, and may be output in an arbitrary format. In S1321, the archive application ends the process.

[Flow of processing during failover]
A process flow of the application server 102 at the time of failover will be described. Here, failover refers to a mechanism of a form generally called cold standby or hot standby. In the above method, for example, when a certain application server is unable to continue processing, the application server is restarted, and the state is not recovered, but the processing is transferred to a different application server. As a result, the processing of the entire system is continued and the service provision is restored.

  In this system, as shown in FIG. 9, the status DB unit 342 of the storage server 104 records a log collection agent unit 322 having the same ID as the Agent ID, using the Agent ID as a key. Further, the date and time when the collection is completed is recorded in the LastCrawlTime in association with the log collection agent unit 322.

  Here, it is assumed that the log collection agent unit 322 is activated on a different application server. The log collection agent unit 322 creates a new ID in S1109 of FIG. 11, transmits the created ID to the status DB request processing unit 333 of the log server 103 in S1116, and performs registration work. In this case, as shown in FIG. 9C, a new entity (AgentID “CE2099797...”) Is registered in the status DB 903 of the storage server 104.

  Here, the LastCrawlTime 9032 of the entity having the Agent ID “BE209997...” Is not updated and retains the old value. The archive application of the batch application server 105 can output a log having a TimeStamp up to the oldest time among all LastCrawlTimes recorded in the status DB unit 342 of the storage server 104. In the case of FIG. 9C, the oldest time is “2011/01/03 01:00:00”, and since this value is not updated, the archive application is permanently “2011/01/03 01:00. : 0 "only can be output.

  Therefore, in this embodiment, even when processing is delegated to a different application server at the time of failover, the application server flow for updating the LastCrawlTime after the ID is taken over before and after the failover is as follows. Describe. In other words, even when the process is delegated at the time of failover, a new Agent ID is not added as shown in FIG. 9C, and it corresponds to the Agent ID of the delegation source as shown in FIG. 9B. LastCrawlTime is updated. Also, in the following flow, the application server 1 is a server that is inoperable and fails over, and the application server 2 is a server to which processing is transferred. The character strings of Dir 4015 and FilenameRegEx 4017 described in the setting file included in the log collection agent unit of the application server 1 and the application server 2 are the same.

  In the present embodiment, this processing flow is realized by the CPU included in each application server reading and executing the program stored in the indirect storage unit directly into the storage unit.

  In step S1401, the request processing unit state monitoring unit 3213 on the application server 1 detects that the operation of the application server 1 cannot be continued. In S1402, the application server 1 copies all files existing in the folder where the log file was output to a folder existing on the same folder path on the application server 2. Here, since the ID file corresponding to the application server 1 also exists in the same folder, the ID file is similarly copied.

  In S1403, the application server 2 notifies the application server 1 of the success or failure of copying. In step S <b> 1404, the application server 1 confirms the success / failure of the copy notified from the application server 1. If the copying has succeeded (YES in S1404), the process proceeds to S1405. If the copy has failed (No in S1404), the process proceeds to S1407, and a failover failure is assumed.

In step S1405, the application server 2 activates the log collection agent unit 322. Since the flow after the log collection agent unit 322 is activated is the same as the flow shown in FIG. 11, the description thereof is omitted. In S1406, the application server 2 starts the application unit 321. As described above, the application server and the log server are independently constructed by performing processing according to FIGS. 11 to 14, and requests from a plurality of users are received. In a large-scale system that processes in parallel, it is possible to prevent the logs from being recorded in duplicate. Furthermore, it is possible to prevent duplicate logs from being output when a log for a certain period is output. In addition, when a process is delegated to a different application server due to a failover or the like, and another log collection agent unit operates, it is possible to collect and manage logs without omission by using an ID corresponding to the delegation source. I can do it.

<Other embodiments>
The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, or the like) of the system or apparatus reads the program. It is a process to be executed.

Claims (16)

A log management system including an application server that provides a service in response to a request from a user, and a log server that accumulates a log of the service provided by the application server,
The application server is
Log file output means for outputting a log of processing performed in response to a request from a user to a file;
One or more log data collection means for collecting the logs from the files to be collected out of the files whose logs are output by the log file output means, and transmitting the logs to the log server;
Providing means for generating an identifier for uniquely identifying the one or more log data collection means, and providing the identifier to the log server;
The log server
Receiving a log transmitted from the application server, and storing the storage unit in a storage unit;
The storage means includes an identifier provided from the application server associated with the received log and an identifier associated with the log already stored in the storage unit, and is included in the received log. The log received when the other information for uniquely identifying the log matches the other information for uniquely identifying the log included in the log already stored in the storage unit A log management system that stores the received log in other cases without storing   Other information for uniquely identifying the log includes at least the name of the log file to be collected, the line number where the log was output in the file, and the time stamp when the log was output The log management system according to claim 1, wherein one piece of information is included.   When the application server is unable to continue processing, when performing a failover to another application server, the other application server operates its own log data collection means, and the other application server The log management system according to claim 1, wherein the providing means includes a new identifier for uniquely identifying the operated log data collecting means and provides the identifier to the log server. .   The log data collection unit further includes log collection status information, and collects logs that have not been collected according to the log collection status information. Log management system.   The log data collecting means holds the information on the location of the folder storing the log file to be collected, the name of the log file to be collected, and the interval of the collection time as a setting file. The log management system according to claim 1 or 2, characterized by the above.   The said provision means stores the data which specify the produced | generated said identifier in the folder where the file of the log used as the collection object designated by the said setting file is stored. Log management system.   When the data specifying the identifier is already stored in the folder in which the log file to be collected is stored in the setting file, the providing unit uses the identifier specified by the data as the identifier. The log management system according to claim 6, wherein the log management system is provided to a log server. When the application server fails over to another application server when processing cannot be continued, it is included in the folder in which the log file to be collected by the log data collecting means is stored. And copy means for copying the existing data together with data specifying the identifier stored in the providing means to the same folder location of the other application server,
8. The log management system according to claim 7, wherein the providing unit of the other application server provides the log server with an identifier specified by the copied data.   The log collection status information is information indicating the time at which the previous log was collected, the data size from the beginning of the log file at the end of collection, and the line number at which collection in the log file was completed. The log management system according to any one of claims 4 to 8.   The log management system according to any one of claims 5 to 9, wherein a name of the log file to be collected is specified by a regular expression. The application server further includes a status transmission unit that transmits, to the log server, information that associates the identifier of the log data collection unit with the time when the log data collection unit last collected the log. ,
The log management system according to claim 1, wherein the storage unit further stores information received from the status transmission unit in the storage unit. The log management system further includes output means for outputting a log stored in the storage unit for a specified period,
The output means outputs the log when the log data collection means stored in the storage unit last collects the log when the specified period has passed. The log management system according to claim 11. An application server in a log management system including an application server that provides a service in response to a request from a user and a log server that accumulates a log of the service provided by the application server,
Log file output means for outputting a log of processing performed in response to a request from a user to a file;
One or more log data collection means for collecting the logs from the files to be collected out of the files whose logs are output by the log file output means, and transmitting the logs to the log server;
Providing means for generating an identifier for uniquely identifying the one or more log data collection means, and providing the identifier to the log server;
The log data collection unit includes an identifier of the log data collection unit and other information for uniquely identifying the log in the data of the log.   When another application server becomes unable to continue processing and receives a failover request from the other application server, the log data collection unit is operated, and the providing unit operates the log that has been operated. 14. The application server according to claim 13, wherein an identifier for uniquely identifying the data collection unit is newly generated and the identifier is provided to the log server. One or more log data collection means for transmitting a log of processing executed when a service is provided in response to a request from a user to a log server, and an identifier for uniquely identifying the one or more log data collection means A log server in a log management system comprising: an application server having a providing means for providing to a log server; and the log server for accumulating logs of services provided by the application server,
Receiving a log transmitted from the application server and storing in a storage unit;
The storage means includes an identifier provided from the application server associated with the received log and an identifier associated with the log already stored in the storage unit, and is included in the received log. The log received when the other information for uniquely identifying the log matches the other information for uniquely identifying the log included in the log already stored in the storage unit A log server characterized by storing the received log in other cases without storing the log. A log management method in a log management system including an application server that provides a service in response to a request from a user and a log server that accumulates a log of the service provided by the application server,
In the application server,
A log file output step in which the log file output means outputs a log of processing performed in response to a request from the user to a file;
One or more log data collection means each collects the log from a file to be collected out of the files output by the log file output means, and sends the log data to the log server; and
Providing means for generating an identifier for uniquely identifying the one or more log data collection means, and providing the identifier to the log server;
In the log server,
The storage means has a storage step of receiving the log transmitted from the application server and storing it in the storage unit,
In the storage step, the identifier provided from the application server associated with the received log matches the identifier associated with the log already stored in the storage unit, and is included in the received log. The log received when the other information for uniquely identifying the log matches the other information for uniquely identifying the log included in the log already stored in the storage unit A log management method characterized by storing the received log in other cases without storing the log .

Images