JP2001229052A - Log processor, log processing method and log processing program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To safety collect logs from various servers, and to preserve the logs in an easily utilizable. SOLUTION: This log processor is provided with a first means for collecting the logs of a plurality of applications and a second means for storing the log of the applications as a single record by correlating the logs of the applications according to a prescribed condition.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a technology for collecting logs from devices on a network and storing the collected logs.

[0002]

2. Description of the Related Art Various services in electronic commerce (electronic commerce: EC) and the like are provided on the Internet using a system including a server. A log is output from the server along with its operation, and the log is generally stored in the server. The log is a record indicating what processing the server has performed, and is referred to when it is necessary to track what processing the system has performed in the past in providing the service.

[0003]

Since logs are continuously generated during operation of the system and become large, there is a problem that if the logs are stored on the server side, the management thereof is not easy. Therefore, for example, when it is obliged for EC site operators to keep logs of e-commerce and the like, the burden on the server administrator may become extremely heavy. It is also conceivable to send logs to another device and save them collectively,
There has been no method to prevent the log including the transaction data from being tampered with and safely transmit it to another device and store it.

In electronic commerce and the like, a service is generally provided by a plurality of various servers. When it is necessary to track logs, the logs of each server are individually referred to and the logs are referred to. Since the log must be tracked based on a specific session or the like, there is a problem that it takes much time and effort.

[0005] For example, from a user who purchases a product using an EC site, a complaint such as "I do not remember the purchase", "I do not remember the membership registration", "I am not satisfied with the explanation of the business operator" operates the EC site. EC
The site examines the action taken by the user in the site, that is, the web server access log and the mail server log for sales confirmation, and discloses information serving as evidence to the user. The investigation is extremely troublesome because it must be performed over logs of a plurality of servers.

The present invention has been made in view of the above points, and has as its object to collect logs safely from various servers and store them in a format that is easy to use. That is, a method, a device, and a program for collecting and storing logs output from a plurality of servers while preventing tampering, and linking and storing logs from each server so that log analysis is facilitated. The purpose is to provide.

[0007]

In order to achieve the above object, the present invention can be configured as follows.

According to a first aspect of the present invention, there is provided a log processing apparatus for processing logs of a plurality of applications generated in a device on a network, wherein the first means collects logs of the plurality of applications; Second means for associating logs of a plurality of applications according to predetermined conditions and storing the logs of the plurality of applications as one record.

According to a second aspect of the present invention, in the first aspect, when the predetermined key data exists in a log of a certain application and a log of another application, It is assumed that there is provided a means for using the log of (1) as one record.

[0010] According to the above invention, the collected logs are associated (also referred to as “linking”) according to the application and stored as one record, so that it is not necessary to track individual logs when analyzing logs across a plurality of applications. , Log analysis becomes easier.

According to a third aspect of the present invention, in the first aspect, the first means includes: means for receiving a log from a device on a network; calculating a hash value from the received log; It has means for comparing the value with a hash value received from a device on the network, and means for applying an electronic signature to the received log when the hash values match.

According to the present invention, it is possible to receive a log while preventing tampering of the log, and to prevent tampering of the log with an electronic signature even after receiving the log.

According to a fourth aspect of the present invention, there is provided a log processing apparatus for processing a log generated in a device on a network, comprising: means for receiving a log from a device on the network; Means for calculating the hash value and a hash value received from a device on the network; and
Means for applying a digital signature to the received log and storing the log with the digital signature.

The invention according to claims 5 and 6 is a log processing method in a log processing device.

According to a seventh aspect of the present invention, in the fifth aspect, a step in which the device on the network opens a log file and records a unique number of the log file; Sequentially reading the log file and transmitting the read log to the log processing device; and a device on the network checks a unique number of the log file, and checks the unique number and the recorded unique number. And transmitting the hash value of the log sequentially read and transmitted to the log processing device when they do not match, and the log processing device extracts the hash value from the received log. Calculating and comparing the hash value with a hash value received from a device on the network; and If, it is assumed that a step of applying a digital signature to the received log.

According to the present invention, in addition to the effect of linking, a device on a network that generates a log can transmit a log to a log processing device while preventing tampering.

An eighth aspect of the present invention is a log processing method for processing a log generated in a device on a network, wherein the device on the network opens a log file and assigns a unique number to the log file. Recording, a device on the network sequentially reads the log file, and transmitting the read log to the log processing device; and a device on the network checks a log file unique number, Comparing the unique number with the recorded unique number, and, if they do not match, transmitting a hash value of the log sequentially read and transmitted to the log processing apparatus; A processing device calculates a hash value from the received log and compares the hash value with a hash value received from a device on the network. And-up, the log processing apparatus, if the hash values match, applying an electronic signature to the received log, and a step of storing the log which has been subjected to electronic signature.

According to the present invention, it becomes possible for a device on a network that generates a log to transmit a log to a log processing device while preventing tampering. Also, since the log processing device applies the digital signature, tampering can be prevented even after the log is stored.

The invention according to claims 9 to 13 is a program suitable for executing the above method on a computer.

[0020]

DESCRIPTION OF THE PREFERRED EMBODIMENTS First, an outline of the present invention is shown in FIG.
This will be described with reference to FIG.

As shown in FIG. 1, in e-commerce such as online shopping, a Web server 1 that receives access from a user, a mail server 2 that transmits a purchase confirmation mail and the like, and an EC that performs processing such as ordering and purchasing goods Services are provided by a server 3, a settlement server 4 for paying the price of the product, an authentication server 5 for performing various authentications, and the like. These are networks 6 including the Internet
Connected to. In the embodiment of the present invention, the logs of these servers are collected by a log collection / storage device, and the logs are stored in a linked state according to a predetermined condition. The stored log is provided to the user in response to a request from the user.

An outline of linking logs will be described with reference to FIG.

FIG. 2 shows various servers used for providing services related to electronic commerce. In addition, logs output from Web servers and mail servers, E
Logs output from the C server, the settlement server, and the authentication server are shown, and a record 8 in which these are linked is shown.

For example, when a certain user accesses the Web, selects a book, inputs predetermined items, and presses a purchase button on the screen to complete the purchase, the order number is displayed on the Web screen and the order number is displayed. Is sent. In response to such a series of processing, each server outputs the above log.

After that, for example, when the user inquires whether he / she purchased the book, if there is data associated with the order number, the IP address, the card number, the transmitted message of the mail, and the like, By referring to the data, it is easy to determine whether the user really purchased the book.

That is, the user of the predetermined IP address places an order, and it can be seen from the record 8 that the user has paid for the goods by card. If there is no such linked record, the logs of various servers must be individually checked and the logs must be checked while judging the relation between the logs, which is very troublesome.

FIG. 2 shows that an Internet connection application to an ISP (ISP signup), an Internet connection history of an end user, and a log in a hosting service application (hosting signup) can also be collected and linked and stored. Is shown.

Next, the system configuration and operation in the embodiment of the present invention will be described.

FIG. 3 is a diagram showing a system configuration according to an embodiment of the present invention. A user service site and a log backup server to be described later are installed in a data center.
Fig. 2 shows an example of a configuration in which a log storage center for storing logs is installed at a location remote from a data center. FIG. 4 is a diagram showing an outline of the operation of the system shown in FIG.

The configuration and operation of the system will be described below with reference to FIGS.

The data center 9 has user sites 10-1.
2. It has a log backup server 13-16, a reporting server 17, a firewall 18, and a router 19.

Each user site comprises one or more servers for providing a user with services such as electronic commerce,
Connected to the log backup server via LAN.

The log backup server collects raw logs from each server at the user site. It also collects relevant logs via the Internet as needed.
The logs to be collected include Web server access logs,
There are a DB authentication log, an electronic commerce server transaction log, a settlement server log, a mail delivery log, a login history log, a firewall session log, an authentication server log, and the like.

The log backup server performs a mining process, which will be described later, and copies a raw log, a primary processed (shaped) log, and a linked log to a tape by a tape drive. The tape is transported by logistics to a log storage center. By using two methods, log data is stored more safely. Also, since logs are stored in the data center and the log storage center, high security can be obtained.

The log processed by the log backup server is transmitted to the reporting server 17, and the reporting server 17 performs generation management of the log and makes a backup to a tape or the like. Further, the processed log data is transmitted to a log storage center via a network such as a dedicated line.
The reporting server 17 receives the system resource management measurement data and the application log of the server alone from the user server group, accumulates the data, and processes the data so that it can be provided to the user. Thereby, for example, the transition of the load on the server can be graphically shown to the user.

The log storage center 26 has a database server 20, a report view server 21, a log mining server 22, firewalls 23 and 24, and a tape library 25. The database server 20 stores and manages log data and system resource data transmitted from the data center. The stored data can be searched and browsed by the user accessing the report view server 21 via the Web or the like.

For a complicated log analysis beyond the range that the user can analyze and track by search, the operator can analyze and track using the log data mining server 22 and notify the user of the result. .
The tape library 25 stores a tape on which a log is recorded.

The configuration for collecting, processing, and storing logs is not limited to the above configuration. For example, in a state where the user servers are distributed and arranged on the network, logs may be collected and linked using only the log backup server, and the processed logs may be stored there. .

The functions of each server described above are executed by a program for performing the processing according to the present invention. By storing the program on a recording medium such as a CD-ROM and installing the program on a computer serving as a server, the log processing according to the present invention can be executed. The program can also be transferred via a network.

Next, main processing in each server will be described in detail.

First, the process in which the log backup server collects logs from the user's server will be described with reference to the flowchart of FIG. This process is an example of a process when a commonly used UNIX-based server is used. That is, when an application log is generated in the server, the log is written to a log file. Log files are generated (rotated) as needed, and each log file is assigned an i-node number corresponding to each file on a one-to-one basis.

First, a log file is opened on the user's server (step 1). Next, the i-node number of the log file is recorded as the i-node number of the currently opened log (step 2). The contents of the opened file are sequentially read (step 3), and while the log can be read, the read data is transmitted to the log backup server (step 4), and the hash value of the contents of the log file is updated (step 3). Step 5).

When the log file cannot be read, for example, when all the logs included in the log file are read, the i-node number of the log file and the i-node number of the currently open log recorded Are compared (step 6). As a result of the comparison, if they match, there is a possibility that a log may be added to the original log file, so the process returns to reading the log file (step 1). If they do not match, it is determined that the log has been rotated, and the log file is closed (step 7).

Then, the end of the log is notified to the log backup server, and the calculated hash value is transmitted.
The process moves to the next log file (step 8, step 1).

The log backup server compares the hash value generated from the received log with the received hash value (step 9). If they match, an electronic signature is applied to the received log (step 10). If they do not match, the log backup server administrator is notified of the log file name (step 11).

If the user server attempts to transmit the log to the log backup server when the log has been accumulated to some extent without performing the above processing, a time lag occurs between the log generation and transmission from the application. In some cases, the contents of the log may be falsified. In the above-described processing according to the present invention, there is no time lag from when a log is generated to when the log is transmitted to the log backup server, so that tampering can be prevented. Since tampering is checked with a hash value, tampering can be reliably prevented. Further, since the log file is digitally signed by the log backup server, the log backup server can check the log for tampering.

Next, the processing of the log backup server will be described with reference to the flowchart of FIG.

In the log backup server, the log acquisition program (1) is started periodically (step 21).
The log acquisition program includes the processing described in FIG. 5 and acquires various logs from the user server group (step 2).
2). If the acquisition has failed, the log backup server administrator is notified of the type of log that failed to acquire (step 23).

Next, a log primary machining program is started (step 24). The log primary processing program performs primary statistical processing on the obtained various logs, and saves the primary processed application log subjected to the primary statistical processing (step 25). In the primary statistical processing, for example, a shaping processing for dividing a log into a time width of 30 minutes is performed, and further, a statistical processing of how many times a certain server has accessed a specific access destination is performed.

Subsequently, the log mining process is started (step 26). In the log mining process, first, various logs are divided into division units of each log such as a session or a job, and are stored as intermediate data (step 2).
7). The session unit is, for example, a unit from a time when a specific user first accesses for shopping to a time when a commodity is purchased and finished. That is,
The collected logs are only recorded in the order in which the server was accessed, and when logs of a plurality of users are mixed up, the logs can be divided into users by dividing them into sessions. In order to divide a session, a specific session is identified by a user ID, an IP address, a mail address, and the like in the log, and a log of the session is extracted. The job unit is a unit of processing of the server. For example, the unit is a unit in which an e-mail for merchandise purchase confirmation is transmitted to the user, and a reception confirmation is received.

Next, a key for association is extracted from the intermediate data of various logs, and an index is created (step 28). That is, for the logs cut out in the above-mentioned division unit, for example, focusing on the IP address or the like of the access source of the user, the log including the same IP address or the like is extracted as a key to create an index. What is used as a key depends on the application and is determined by hearing with the user or the like.

After the above processing is completed for all the logs (step 29), a linking rule is applied to the extracted keys (logs) to link various log data, and a mining table is created. Data is created (step 30). The linking rule is a method of connecting logs of each intermediate data.

A specific example of linking logs will be described with reference to FIGS.

FIG. 7 is a diagram showing an example in which a Web log and a mail log are linked. In the Web log shown in FIG. 7, the IP address of the user (client) is 172.2.
The log which is 1.217.1 is extracted as a key, and the mail log shown in FIG. 7 is extracted by using a log whose mail destination address is 10.129.240.2 as a key. Then, the Web log and the mail log are combined using time as a key.

As a result, a series of records linked to each other and provided to the user is generated. This linked record indicates that the specific client has accessed the Web server and made an order, and has received a confirmation email. If there is no such linked data,
Since it is necessary to search each of the Web log and the mail log and check the relation, it takes time and effort, but by performing the linking of the present invention, the linked log corresponding to the specific user can be referred to. Just do it.

FIG. 8 shows another example. FIG. 8 shows a Web log,
It is a figure showing an example at the time of linking a mail log and an EC server log. In the example shown in FIG. 8, first, the Web server log and the EC server log are linked using the Cookie as a key (step 1). As a result, the client IP address serving as the access source of the orderer and the mail address are linked. Next, the EC server log and the mail server log are linked using the mail address as a key (step 2). Thus, the transmission confirmation of the mail of the order receipt is linked to the orderer.

Finally, the linking results in steps 1) and 2) are merged with the duplicate key as an axis to form a series of records. As a result, I, which is the access source of the orderer,
From the P address, order time, order number, product (UR
I), the mail address of the orderer, and the confirmation of the mail transmission of the order receipt are linked as a series of records.
In addition to a record in a server log, a customer ID stored in a DB in an EC server can be used as a key record.

Further, if there is no key for linking each server log, the linking can be performed in a time series from the time recorded in the record of each server log and the predicted linking management can be performed.

Actually, the above processing is executed by a mining program installed in the log backup server. Since the server configuration differs for each EC site operator,
The types of logs output from this server are also different, and it is necessary to ask the EC site operator in advance how to manage a series of processing transactions as a series of records. (Step 1) to (Step 3)).

If there is no log data serving as a key for association in the log, the customer ID and the mail address are associated from the customer management database or the like.

In FIG. 6, when the generation of the mining table data (linked records) is completed, the intermediate data and the index become unnecessary and are deleted (step 31).

Thereafter, the primary processed application log, the generated mining table, and the raw log are set and stored on a tape using a tape device (step 32). If the storage has failed, the log backup server administrator is notified of the type of log that failed to be stored (step 33).

Next, the processing in the reporting server will be described with reference to the flowchart of FIG.

In the reporting server, the log acquisition program (2) (which operates differently from the log acquisition program (1) in the log backup server) is started periodically (step 41).

Then, the system resource management measurement data is obtained from the user server group (step 42), and the mining table data,
And a primary processed application log is acquired (step 43). In order to consider providing data to the user, the reporting server performs generation management, deletes the oldest generation of various data files that are generation-managed on the disk, and replaces the acquired various data with the latest data files. It is stored as a generation (step 44).

Then, a backup of the obtained various data is made by a tape device (step 45), and the reporting server administrator is notified of the completion of the processing and information of the obtained various data (step 46).

If the processing has failed, the reporting server administrator is notified of the failed processing and an error event (step 47).

In this embodiment, data processed by the reporting server is transmitted to the database server of the log storage center and stored. The user can search for processed logs by accessing the report view server of the log storage center. The user may access the reporting server and perform a search.

FIG. 10 shows a screen image on the user side when the user accesses the report view server. The search is performed, for example, in the following procedure.

First, the user inputs search conditions (step 51). For example, the user inputs a time, a user ID, a mail address, and the like.

The report view server searches for logs stored in the database according to the search conditions, and displays logs meeting the conditions on the user terminal (step 52).
When a specific record is selected from the list display, the record is displayed in detail (step 53). here,
By selecting "display log", that portion of the raw log corresponding to the mining data can be displayed and can be downloaded (step 54).

The present invention is not limited to the above embodiments, but can be variously modified and applied within the scope of the claims.

[0073]

As described above, according to the present invention, it is possible to collect and store logs while preventing tampering.
In addition, since logs from various servers are linked and stored, it is easy to confirm a series of processing contents over a plurality of servers in electronic commerce or the like.

[0074]

[Brief description of the drawings]

FIG. 1 is a diagram for explaining an outline of the present invention.

FIG. 2 is a diagram for explaining an outline of linking logs;

FIG. 3 is a diagram illustrating a system configuration according to an embodiment of the present invention.

FIG. 4 is a diagram showing an outline of the operation of the system according to the embodiment of the present invention.

FIG. 5 is a flowchart illustrating a process of collecting logs.

FIG. 6 is a flowchart showing processing in the log backup server.

FIG. 7 is a diagram illustrating an example of a case where a Web log and a mail log are linked;

FIG. 8 is a diagram illustrating an example in which a Web log, a mail log, and a log of an EC server are associated with each other.

FIG. 9 is a flowchart showing processing in the reporting server.

FIG. 10 is a diagram showing a screen image on the user side when the user accesses the report view server.

[Explanation of symbols]

 DESCRIPTION OF SYMBOLS 1 Web server 2 Mail server 3 EC server 4 Payment server 5 Authentication server 6 Network 7 Log collection / storage device 8 Linked record 9 Data center 10-12 User site 13-16 Log backup server 17 Reporting server 18 Firewall 19 Router 20 Database server 21 Report view server 22 Log mining server 23, 24 Firewall 25 Tape library 26 Log storage center

 ──────────────────────────────────────────────────続 き Continuation of the front page (71) Applicant 501103181 Shinji Yoneda 6-1-3 Maison Kure 202, Higashimaeharahigashi, Funabashi-shi, Chiba 202 (72) Inventor Yoko Ogawa 2-5-11-401 Maenocho, Itabashi-ku, Tokyo (72) Inventor Yoichi Serizawa No. 3-8-1-2-202, Yashiki, Narashino City, Chiba Prefecture (72) Inventor Tomomi Ishizawa 2-7-21, Nukii Minamimachi, Koganei City, Tokyo (72) Inventor Shinji Yoneda Chiba 6-1-3 Maison Higashimaehara, Funabashi City, Miyagi Prefecture

Claims (13)

[Claims] 1. A log processing device for processing logs of a plurality of applications generated in a device on a network, comprising: first means for collecting logs of a plurality of applications; A second means for associating application logs and storing the logs of the plurality of applications as one record. 2. The apparatus according to claim 1, wherein said second means includes means for, when predetermined key data exists in a log of a certain application and a log of another application, making those logs one record. A log processing device according to claim 1. 3. The first means includes: means for receiving a log from a device on a network; calculating a hash value from the received log; and calculating the hash value and the hash value received from the device on the network. 2. The log processing device according to claim 1, further comprising: means for comparing; and means for applying an electronic signature to the received log when the hash values match. 4. A log processing device for processing a log generated in a device on a network, comprising: means for receiving a log from a device on the network; calculating a hash value from the received log; And a means for comparing a hash value received from a device on the network with a hash value, and, when the hash values match, applying a digital signature to the received log and storing the digitally signed log. Characteristic log processing device. 5. A log processing method for causing a log processing device to perform processing on logs of a plurality of applications generated in a device on a network, wherein the log processing device collects logs of the plurality of applications. And a second step in which the log processing device associates logs of a plurality of applications according to a predetermined condition and stores the logs of the plurality of applications as one record. . 6. The method according to claim 5, wherein the second step includes the step of, when a predetermined key data exists in a log of a certain application and a log of another application, setting those logs as one record. Log processing method described in. 7. A step in which a device on the network opens a log file and records a unique number of the log file; and a device on the network sequentially reads the log file and sends the log file to the log processing device. Transmitting the read log; and a device on the network examining a unique number of the log file, and comparing the unique number with the recorded unique number. Transmitting a hash value of a log sequentially read and transmitted to the log processing apparatus; and the log processing apparatus calculates a hash value from the received log, and receives the hash value and the hash value from a device on the network. Comparing the received log with a digital signature when the hash values match. Claim 5 having bets
Log processing method described in. 8. A log processing method for processing a log generated in a device on a network, wherein the device on the network opens a log file and records a unique number of the log file; The above device sequentially reads the log file, and transmits the read log to the log processing device. The device on the network checks the log file unique number, and checks the unique number and the record. Comparing the log numbers with the unique numbers obtained, and when they do not match, transmitting the hash values of the logs sequentially read and transmitted to the log processing apparatus; and Calculating a hash value from the hash value and comparing the hash value with a hash value received from a device on the network; A step of, when the hash values match, applying a digital signature to the received log and storing the log with the digital signature. 9. A program for causing a computer to execute a process for logs of a plurality of applications generated in a device on a network, the program comprising: a first procedure for collecting logs of the plurality of applications; A second procedure for associating the application logs with each other and storing the logs of the plurality of applications as one record. 10. The method according to claim 9, wherein, when predetermined key data exists in a log of a certain application and a log of another application, the log is used as one record. The program described in. 11. The first procedure is a procedure of receiving a log from a device on a network, calculating a hash value from the received log, and calculating the hash value and the hash value received from the device on the network. The program according to claim 9, further comprising: a step of comparing; and a step of, when the hash values match, applying an electronic signature to the received log. 12. A program for causing a computer to execute a process of transmitting a log generated in a device on a network to a log processing device, comprising the steps of: opening a log file and recording a unique number of the log file; A step of sequentially reading the file and transmitting the read log to the log processing device; a step of examining a log file unique number and comparing the unique number with the recorded unique number; And a program for causing a computer to execute a procedure of transmitting a hash value of a log sequentially read and transmitted to the log processing apparatus when the log processing apparatus does not perform the processing. 13. A program for causing a computer to execute a process for a log generated in a device on a network, comprising the steps of: receiving a log from a device on the network; calculating a hash value from the received log; A program for causing a computer to execute a procedure of comparing a value with a hash value received from a device on the network, and a step of applying an electronic signature to a received log when the hash values match.