JP2002251500A - System and method for collecting log information - Google Patents

Abstract

PROBLEM TO BE SOLVED: To manage and control collection of log information on a service providing server in a unified way. SOLUTION: A service providing sever 1 is provided with a service log memory means 11, which stores service log information, and a log file collecting means 13, which collects uncollected log information from the service log memory means 11 and generates a data file. A log information collecting server 2 is provided with an information collection control means 22, which requests the service providing server 1 to generate a data file, and a log file receiving means 23, which retrieves a data file from the service providing server 1. In addition, the log information collecting server 2 is also provided with a log file editing means 24, which selects data files of the same service from a plurality of data files and permutes them as a one file to output data, which is included in the file, in a time sequence.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

The present invention relates to a log information collecting system and a log information collecting method for collecting log information of each service providing server in a service network system including a plurality of service providing servers connected by an electronic network. .

[0002]

2. Description of the Related Art In recent years, various service providers have provided services using server computers connected to an electronic network such as the Internet, and a plurality of such server computers (hereinafter referred to as service providing servers) have been provided. Service network systems configured by connection have also appeared. In such a service network system, log information, which is a history of various information related to the provided service recorded by each service providing server, may be collected, and billing processing, marketing analysis, usage analysis, and the like may be performed.

In this case, the log information is collected by, for example, providing a module that individually collects the log information in the service providing server, and a log information collecting apparatus that connects the log information collected in each service providing server to the service network system. It was done by collecting. In addition, the collection of log information by the log information collection device has been performed mechanically at regular intervals determined for each service providing server.

[0004]

However, in the conventional service network system, since the log information collecting device collects the log information for each service providing server, there is a problem that the collection of the log information cannot be centrally managed and controlled. was there. For example, when a plurality of service providing servers provide the same service, log information relating to the same service is distributed and provided on the plurality of service providing servers, so that these cannot be collectively handled as one piece of information. In addition, there is a problem in that the collection time of the log information cannot be set arbitrarily, which is inconvenient. Further, there is a problem that log information of services flowing on the network is large and network use efficiency is reduced.

SUMMARY OF THE INVENTION The present invention has been made to solve the above problems, and has as its object to provide a log information collection method and apparatus for centrally managing and controlling the collection of log information of a service providing server. I do. It is another object of the present invention to make it possible to arbitrarily set a collection time and a collection interval of log information, thereby improving convenience. It is another object of the present invention to reduce the amount of log information of a service flowing on a network and improve the use efficiency of the network.

[0006]

In order to solve the above-mentioned problems, a log information collecting system according to the present invention comprises a plurality of service providing servers and a log information collecting system for collecting log information of services from these service providing servers. An electronic network connecting the service providing server and the log information collecting device. The service providing server comprises: a service log storage unit storing service log information; and a log not collected from the service log storage unit. Log file collection processing means for collecting information and generating a data file; the log information collecting apparatus retrieves the data file from the service providing server; and information collection control means for requesting the service providing server to generate the data file. Log file reception processing means and data of the same service from multiple data files Select Airu, combined as a single file, characterized by having a log file editing processing means for outputting sort the data in this file in the order of time series.

One example of the configuration of this log information collecting system is as follows.
The log information collecting apparatus further includes means for processing data output from the log file editing processing means and converting the data into a predetermined format. In this case, one configuration example of the log information collection device includes a setting information storage unit that stores changeable setting information regarding a collection target, and the information collection control unit performs service at a predetermined collection time or collection interval based on the setting information. It is configured to request the providing server to generate a data file. In another configuration example of the log information collecting system, the data file generated by the log file collecting unit of the service providing server is compressed, and the data file extracted by the log file receiving unit of the log information collecting device is decompressed. It is configured as follows.

A log information collecting method according to the present invention is a log information collecting method for collecting log information of a service from a plurality of service providing servers connected via an electronic network. A first step of making a request to collect the log information of the service, and a second step of collecting the uncollected log information from the service log storage unit storing the log information of the service and generating a data file by the service providing server receiving the request. And the third step of retrieving a data file from the service providing server that made the request
And a fourth step of selecting data files of the same service from a plurality of data files, combining them into one file, and rearranging and outputting the data in this file in chronological order. Characterized.

One example of the configuration of this log information collecting method is described in the fourth section.
After the step (b), there is further provided a fifth step of processing the data and converting the data into a predetermined format. In one configuration example of the first step, a request is made to the service providing server to collect uncollected log information at a predetermined collection time or a collection interval based on changeable setting information on a collection target. In another configuration example of the log information collection method, the data file generated in the second step is further compressed, and the data file extracted in the third step is decompressed.

[0010]

Embodiments of the present invention will be described below with reference to the drawings. FIG. 1 is a conceptual diagram showing an embodiment of a log information collection system according to the present invention, and shows components of the system and a flow of processing. In the figure,
The log information collecting system of this embodiment includes a plurality of service providing servers 1 and a log information collecting server 2 which is a log information collecting device, and these are mutually connected via an electronic network such as the Internet (not shown). It is configured to be able to communicate. Also, the log information collection server 2
At least one external system 3 is configured to be able to communicate with each other via an electronic network such as the Internet (not shown).

The service providing server 1 is a server computer for a service provider to provide a service via an electronic network such as the Internet. The service providing server 1 includes a service log storage unit 11, a setting information storage unit 12, and a log file collection unit. Processing means 13 and transmission log storage means 14
And In the service providing server 1, a service log, which is a record of the provided service, is written in the service log storage unit 11 by a service providing unit (not shown). Here, each service providing server 1
Are the same except for the type and number of services to be provided. When the service providing server 1 provides a plurality of services, a service log storage unit 11 and a transmission log storage unit 14 are provided for each service.

A log information collection server 2 collects service logs from a plurality of service providing servers 1 via an electronic network such as the Internet, and collects the log in at least one external system 3 via an electronic network such as the Internet. A setting information storage unit 21 and an information collection control unit 2
2, a log file reception processing unit 23, a log file editing processing unit 24, and a filtering / conversion processing unit 2.
5 and an external system transmission processing means 26.

The operation of this embodiment will be described below with reference to FIG. In the log information collection server 2, the information collection control unit 22 requests the collection target service providing server 1 to perform log file collection processing according to a predetermined schedule. Upon receiving the request, the service providing server 1 activates the log file collection processing unit 13. The log file collection processing unit 13 generates a file including a service log after the previous collection from the service log file stored in the predetermined service log storage unit 11, adds the server identification information, and compresses the data. Is stored in a predetermined transmission log storage unit 14 as a transmission log file. In this way, a data file in which uncollected service logs are recorded for each service providing server 1 to be collected on a predetermined schedule is prepared.

Next, the information collection control means 22 activates the log file reception processing means 23. The log file reception processing unit 23 extracts a transmission log file from the service providing server 1 to be collected, decompresses the transmission log file, and individually retains it as a reception log file. Next, the information collection control means 22 activates the log file editing processing means 24.
The log file editing processing means 24 specifies a reception log file to be subjected to format conversion, and
Retrieve the reception log file of the format conversion target held in, combine (merge) the reception log files of the same service into one file, perform chronological sorting, arrange the service logs in chronological order and edit A log file is stored for each service type.

Next, filtering / conversion processing means 25
Input the service logs in the edit log file held in the log file edit processing means 24 sequentially from the top of the file, and perform filtering such as deletion of unnecessary information;
Perform the conversion process to the file format of the predetermined format,
Store as a conversion log file. Here, it is converted into a CSV format file. Next, the external system transmission processing means 26 transmits a reception log file or a conversion log file suitable for the type of service and the file format to be reported to the destination according to a predetermined transmission schedule to the predetermined external system. In this manner, the service record relating to the service to be collected is reported in a format suitable for the external system of the report destination on a predetermined schedule.

Next, the service providing server 1 of this embodiment will be described in detail. Service log storage means 1
Reference numeral 1 denotes a storage device provided with a predetermined directory corresponding to a service, and stores a service log file in this directory. When there are a plurality of services to be provided, a directory is provided for each service, and a service log file corresponding to each directory is stored. The setting information storage unit 12 is a storage device that stores setting information used for processing of the log file collection processing unit 13, and stores processing directory names in which service log files are stored, identification information for identifying servers, and the like. I remember.

The log file collection processing means 13 receives a request from the log information collection server 2, collects uncollected service logs from a service log file of a predetermined service, and obtains server identification information for identifying the service providing server. After performing data compression, the data is stored in the transmission log storage unit 14 as a transmission log file.
The transmission log storage unit 14 is a storage device that stores a transmission log file, and is configured to be readable from the log information collection server 2 via an electronic network.

Next, the operation of the log file collection processing means 13 in this embodiment will be described with reference to FIG. FIG. 2 is a flowchart showing the operation of the log file collection processing means 13. First, the setting information storage unit 12
With reference to the setting information, a process directory name is acquired (step S01), and a collection target file is specified (step S02). Next, the specified service log file is read from the processing directory of the service log storage unit 11 and compared with the service log file stored in the setting information storage unit 12 at the previous collection, and a difference file from the previous collection is stored. It is created (step S03). The specified service log file is stored in the setting information storage unit 12 for the next time, and the service log file collected at the previous time is deleted.

Next, the server identification information stored in the setting information storage means 12 is read out and added to the difference file, and the server identification setting is performed on the difference file (step S0).
4). Next, the difference file is stored in a predetermined transmission log storage unit 14.
Into a working directory (step S0
5) The data of the copied difference file is compressed (step S06). Next, the completion of the processing is notified to the log information collection server 2 (step S07), and the processing ends.

Next, the log information collection server 2 of this embodiment will be described in detail. Setting information storage means 21
Are information collection control means 22, log file reception processing means 23, log file editing processing means 24, filtering / conversion processing means 25, and external system transmission processing means 26
This is a storage device that stores the setting information used for the processing of the service log, various settings related to the service providing server of the service log collection target, settings related to the retry interval and the number of retries when an error occurs when the service log is received and the collection result is transmitted,
Stores settings related to the external system of the collection result transmission destination.

The various settings relating to the service providing server include, for example, the network address of each service providing server, the directory to which the transmission log file is output, the collection time or collection interval of the transmission log file, the presence or absence of a format conversion target, and the service identification information. and so on. The settings relating to the external system include, for example, the network address of each external system, the type of the report target service, the file format of the report information, the report time or the report interval. These setting information can be arbitrarily changed by a data input device (not shown) provided in the log information collection server 2 or a terminal device (not shown) connected via a network.

The information collection control means 22 comprises a scheduling section 22a and a processing control section 22b, and is activated at a predetermined cycle to perform processing. Scheduling unit 22
“a” refers to the setting information on the service providing server of the service log collection target stored in the setting information storage unit 21 and centrally manages the time or collection interval at which the service log of the target service providing server is collected. Has a schedule management function of issuing a processing instruction to the processing control unit 22b at a predetermined time.

The processing control unit 22b has a function of transmitting a command to activate the log file collection processing means 13 to the corresponding service providing server based on a processing instruction from the scheduling unit 22a, and a function of receiving a file to perform file reception processing. A function of designating a transmission log file on the service providing server to activate the log file reception processing means 23 and a function of activating the log file editing processing means 24 when the designated transmission log file is set as a format conversion target Having.

Next, the operation of the information collection control means 22 in this embodiment will be described with reference to FIG. FIG.
5 is a flowchart showing the operation of the information collection control means 22. First, the scheduling unit 22a checks whether a predetermined information collection time has been reached for the service providing server from which the service log is to be collected (step S11). If it is not the predetermined information collection time, the process is temporarily ended. If there is a service providing server that has reached the predetermined information collection time, the processing control unit 22b transmits a command to activate the log file collection processing unit 13 to the corresponding service providing server (step S12).

Next, a transmission log file on the service providing server to be received is specified and the log file reception processing means 2
3 is started (step S13). Here, before the log file reception processing unit 23 is activated, the log file collection processing unit 13 in the service providing server that is to receive the log file reception processing unit 23.
It is desirable to confirm that the processing has been completed or to provide a predetermined waiting time. Next, it is checked whether the specified transmission log file is a format conversion target (step S14),
If not, the process ends. If it is the format conversion target, after the log file reception processing means 23 ends, the log file editing processing means 24 is started (step S15), and the processing ends.

The log file reception processing means 23 comprises a log file reception processing unit 23a and a reception log storage unit 23b. The log file reception processing unit 23a receives the specified transmission log file, decompresses the received transmission log file, and stores it in the reception log storage unit 23b as a reception log file, and determines whether there is an abnormality based on the file reception result and the decompression processing result. And a function of performing re-reception based on the setting of the retry interval and the number of retries in the setting information storage unit 21 when an abnormality occurs. In this embodiment, the transmission log file is received by FTP (File).
(Transport Protocol) is used.

Here, the log file reception processing unit 23a is provided for each service providing server, and the reception log storage unit 2
3b is provided for each transmission log file to be received. The log file reception processing unit 23a may process one service providing server for all reception targets, or may process a plurality of service providing servers. Further, the information may be provided for each transmission log file to be received.

Next, the operation of the log file reception processing means 23 in this embodiment will be described with reference to FIG. FIG. 4 is a flowchart showing the operation of the log file reception processing means 23. First, the log file reception processing unit 23a accesses the designated transmission log file,
This file is received (step S21), and the received file is decompressed (step S22). Next, the file reception result and the decompression processing result are determined (step S2).
3) In the event of an abnormality, the process returns to step S21 based on the setting of the retry interval and the number of retries in the setting information storage unit 21 to perform re-reception. If the retry is over, error information is output to an error file (step S24).
If the determination result is normal, it is stored as a reception log file in the reception log storage unit 23b (step S25).

The log file editing unit 24 includes a log file editing unit 24a and an edited log storage unit 24b. The log file editing processing unit 24a reads the reception log file set as the format conversion target from the reception log storage unit 23b, combines (merges) the reception log files of the same service, and chronologically converts the data in this file. (Time-series sorting), and storing the edited log file in the edited log storage unit 24b as an edited log file. In this case, whether the service is the same or not is determined based on the setting of the service providing server 1 in the setting information storage unit 21. Note that although it is set as a format conversion target, if there is no other received log file to be combined, only the time-series sorting is performed. The edit log storage unit 24b is provided for each edit log generated by the log file edit processing unit 24a.

Next, the operation of the log file editing processing means 24 in this embodiment will be described with reference to FIG. FIG. 5 is a flowchart showing the operation of the log file editing processing means 24. First, the log file editing processing unit 24a reads a reception log file set as a format conversion target from each reception log storage unit 23b, and refers to the setting related to the service providing server 1 of the setting information storage unit 21 to provide a service provider. The processing of combining (merging) the reception log files having the same identification information of the service provided by the service log into one file is performed for each service type (step S31). Next, the data in the generated file is rearranged in chronological order (time-series sorting) (step S3).
2) Store the edited log file in the edited log storage unit 24b (step S33).

The filtering / conversion processing unit 25 includes a filtering / conversion processing unit 25a and a conversion log storage unit 25.
b. Filtering / conversion processing unit 2
5a is a function of reading out the edit log file stored in the edit log storage unit 24b, performing filtering such as deletion of unnecessary information, converting the file into a file of a predetermined format, and storing it in the conversion log storage unit 25b. It has. In this embodiment, the data is converted into a CSV format. The filtering condition and the format of the file before and after the conversion may be set based on the setting of the service providing server in the setting information storage unit 21 or may be incorporated in the filtering / conversion processing unit 25a as a processing specification in advance. . Further, the filtering / conversion processing unit 25a and the conversion log storage unit 25b are provided for each edit log storage unit 24b, but the filtering / conversion processing unit 25a is a single unit for storing the files of all the edit log storage units 24b. Processing may be performed.

Next, the operation of the filtering / conversion processing means 25 in this embodiment will be described with reference to FIG. FIG. 6 is a flowchart showing the operation of the filtering / conversion processing means 25. First, the filtering / conversion processing unit 25a inputs one record from the edit log file stored in the edit log storage unit 24b (step S41). In this case, the service logs are read one by one from the beginning of the edit log file. Next, filtering is performed on one record based on the setting of the service providing server in the setting information storage unit 21, and unnecessary information is deleted (step S42). Next, the format of this record is converted based on the setting of the service providing server in the setting information storage means 21,
The record is set in a CSV format (step S43). Next, the presence or absence of the next record is confirmed, and steps S41 to S43 are repeated until there is no more record (step S44). When there is no record and format conversion is completed, the format is stored in the conversion log storage unit 25b as a conversion log file. (Step S45)

The external system transmission processing means 26, based on the setting regarding the external system of the collection result transmission destination of the setting information storage means 21, according to a predetermined transmission schedule, stores a predetermined file in the reception log storage unit 23b or the conversion log storage unit 25b. To a predetermined external system, and a function of judging the presence or absence of an abnormality based on a file transmission result, and performing a retransmission based on the setting of the retry interval and the number of retries in the setting information storage unit 21 when an abnormality occurs. ing.

Next, the operation of the external system transmission processing means 26 in this embodiment will be described with reference to FIG. FIG. 7 is a flowchart showing the operation of the external system transmission processing means 26. First, the setting information storage unit 21
In step S51, a reception log file or a conversion log file suitable for the type and file format of the service to be reported to the transmission destination is read and transmitted according to a predetermined transmission schedule with reference to the setting regarding the external system of the collection result transmission destination (step S51). Next, the presence or absence of an abnormality is determined based on the file transmission result (step S52). If an abnormality occurs, the process returns to step S51 based on the setting of the setting information storage unit 21 regarding the retry interval and the number of retries, and performs retransmission. If the retry is over, error information is output to an error file (step S53). If the result of the determination is normal, the process ends.

The above-described log information collecting system according to the present embodiment includes a server computer connected to an electronic network such as the Internet and a computer program stored in the server computer. The functions are realized by being executed by a computer. The server computer includes a communication interface unit (not shown) for performing data communication with an electronic network, a CPU (not shown) for executing a predetermined program, a main storage memory (not shown), a computer program, Hardware similar to a general network server, such as an external storage device (hard disk, not shown) for storing various information, is provided.

In cooperation with the hardware resources and the software, in the service providing server 1, the service log storage unit 11, the setting information storage unit 1
2. A log file collection processing unit 13, a transmission log storage unit 14, and a service providing unit are realized. In the log information collection server 2, a setting information storage unit 21, an information collection control unit 22, a log file reception processing unit 23, a log A file editing processing unit 24, a filtering / conversion processing unit 25, and an external system transmission processing unit 26 are realized.
The hardware that constitutes the server computer is
The description is omitted because it is a well-known technique.

As described above, according to this embodiment, even when a plurality of service providing servers provide the same service, the log information collecting server collects the log information related to the same service into one information. So,
The collection and management of log information can be centrally managed and controlled. Furthermore, since the log information can be processed according to the external system and converted to a predetermined format,
Convenience is improved. Further, the information collection control means of the log information collection server requests the service providing server to be collected log file collection processing according to the schedule of the arbitrarily changeable setting information stored in the setting information storage means. The collection time or the collection interval can be set arbitrarily, and the effect of further improving the convenience is obtained. Also,
The log file collection processing means of each service providing server provides log information as differential data, compresses the data into a transmission log file, and decompresses the transmission log file extracted by the log file reception processing means of the log information collection server. Since the reception log file is used, the log information of the service flowing on the network is reduced, and the effect of improving the use efficiency of the network is obtained.

The log information collecting system of the present invention is not limited to this embodiment, but can be variously modified. For example, the configuration may be such that each service providing server and log information collecting server are connected to each other via a closed network separately from a network used for external services. In this case, it is expected that the communication amount between each service providing server and the log information collecting server is not so large as to cause a decrease in network efficiency. Decompression processing in the log file reception processing means of the information collection server can be omitted. As a result, effects such as simplification of a processing program and saving of hardware resources can be obtained.

[0039]

As described above, according to the present invention,
The log information collection server has an effect that the collection of log information can be centrally managed and controlled. Also,
Regardless of the type and number of services and the format of the log information, log information can be collected at a desired collection time or collection interval, thereby improving convenience. In addition, since the log information is provided by the service providing server as the differential data and the data is compressed, the information amount of the log information of the service flowing on the network is reduced, and the effect of improving the use efficiency of the network is obtained.

[Brief description of the drawings]

FIG. 1 is a conceptual diagram showing an embodiment of a log information collection system according to the present invention.

FIG. 2 is a flowchart showing an operation of a log file collection processing unit in FIG. 1;

FIG. 3 is a flowchart showing the operation of the information collection control means of FIG. 1;

FIG. 4 is a flowchart showing an operation of the log file reception processing means of FIG. 1;

FIG. 5 is a flowchart showing the operation of the log file editing processing means of FIG. 1;

FIG. 6 is a flowchart showing the operation of the filtering / conversion processing means of FIG. 1;

FIG. 7 is a flowchart showing the operation of the external system transmission processing means of FIG. 1;

[Explanation of symbols]

1 ... service providing server, 2 ... log information collecting server, 3
... external system, 11 ... service log storage means, 12 ...
Setting information storage means, 13 log file collection processing means,
14: transmission log storage means, 21: setting information storage means, 2
2 ... information collection control means, 22a ... scheduling unit,
22b: processing control unit, 23: log file reception processing unit, 23a: log file reception processing unit, 23b: reception log storage unit, 24: log file editing processing unit, 24a ...
Log file edit processing unit, 24b ... edit log storage unit, 2
5 filtering / conversion processing means, 25a filtering / conversion processing unit, 25b conversion log storage unit, 26
External system transmission processing means.

Claims (8)

[Claims] 1. A server comprising: a plurality of service providing servers; a log information collecting apparatus for collecting log information of a service from the service providing servers; and an electronic network connecting the service providing server and the log information collecting apparatus. A log information collection system, wherein the service providing server includes: a service log storage unit that stores service log information; and a log file collection that collects uncollected log information from the service log storage unit and generates a data file. Processing means, the log information collecting device, information collection control means for requesting the service providing server to generate the data file, log file receiving processing means for extracting the data file from the service providing server, From a plurality of the data files, the Select Le, combined as a single file, the log information collection system characterized by having a log file editing processing means for outputting sort the data in this file in the order of time series. 2. The log information collection device according to claim 1, wherein the log information collection device further includes a unit that processes the data output by the log file editing processing unit and converts the data into a predetermined format. system. 3. The log information collecting apparatus includes a setting information storage unit that stores changeable setting information relating to a collection target, and the information collection control unit includes a predetermined collection time or a collection interval based on the setting information. 3. The log information collection system according to claim 1 or 2, wherein the server requests the service providing server to generate the data file. 4. The log file collection processing means compresses the data file, and the log file reception processing means decompresses the extracted data file. Log information collection system described in Crab. 5. A log information collection method for collecting log information of a service from a plurality of service providing servers connected via an electronic network, wherein a request to collect uncollected log information from the plurality of service providing servers is provided. A first step of performing; a second step in which the service providing server that receives the request collects uncollected log information from a service log storage unit that stores service log information and generates a data file; A third step of retrieving the data file from the service providing server that has performed the above, and selecting the data files of the same service from a plurality of the data files, combining them into one file, and extracting the data in the file. And a fourth step of rearranging the sequence and outputting the sequence. 6. The log information collecting method according to claim 5, further comprising a fifth step of processing the data and converting the data into a predetermined format after the fourth step. 7. The method according to claim 1, wherein the first step issues a request to the service providing server to collect uncollected log information at a predetermined collection time or a collection interval based on changeable setting information on a collection target. The log information collection method according to claim 5 or 6, wherein the log information is collected. 8. The method according to claim 5, wherein the second step further compresses the generated data file, and the third step decompresses the extracted data file. Log information collection method described in any of them.