JP5595957B2 - Access log processing system and method, program, and access log storage / retrieval device - Google Patents

Description

  The present invention relates to an access log processing system, method and program, and an access log storage / retrieval apparatus. In particular, the present invention provides an access log for high-speed search of an access log including a compression storage technique capable of high-speed search using the characteristics of access log search and a unique character string (data that uniquely identifies a user). It relates to storage and retrieval technology.

In recent years, with the increase in Web access, surveys and Web access logs related to Web access are also increasing. When conducting a survey on web access, the web access log stored using the identification information of the inquired user as a key is searched to extract the corresponding web access log. The web access log required for the inquiry from the user includes the identification information of the user, and as a feature of the web access, the URL data is continuously hosted data such as host data. It is done.
Patent Document 1 registers entries linked to the same entity file for files with the same data content on a file server, and treats them as logically separate files on the management table. An information processing apparatus that realizes saving is described.

JP 2009-237799 A

Conventional techniques are mainly based on general-purpose file compression techniques or database storage techniques. When searching for a Web access log corresponding to a specific user's Web access, using these conventional technologies, one of the storage methods is not specialized for the Web access log. There is a problem that the search processing time due to may decrease.
In addition, in the conventional technology, in order to improve the storage size of the Web access log data, the data may be compressed and stored. However, in the search process, it takes time to restore the Web access log, and the search response time is long. There was a problem that it might be slow.

In view of the above, an object of the present invention is to search for access logs at high speed by creating search metadata in which position information is recorded and compressing and storing access log data.

In the present invention, in order to solve the above-mentioned problem, similar continuous data continues in the access log such as Web collected for each user, and the user's inquiry can be confirmed by extracting the access log of the user. Thus, a high-speed search process is provided in which the data storage method specialized for the access log search work such as the Web is used and the compression and the memory access such as the HDD at the time of the search are reduced. In the present invention, the access log of the Web or the like is stored and stored for each user, so that when the Web access log of a unique user is extracted, the metadata is read and the disk access of the access log of the corresponding Web or the like is performed. The number of times can be reduced.
According to one aspect of the present invention, a Web access log storage / retrieval system includes a Web access log storage function unit, a Web access log storage HDD, and a Web access log search function unit. The Web access log storage function unit processes and compresses the Web access log and creates metadata in consideration of the search characteristics of Web access log search, and stores it in the Web access log storage HDD. The Web access log search function unit reads metadata from the Web access log storage HDD and calls a location based on user information. In addition to the normal Web access log search result, only the difference data can be output as the search result.

According to the first solution of the present invention,
Multiple servers providing web or content or other information;
A terminal of a user accessing the server;
In an access log processing system comprising an access log storage / retrieval device for processing an access log recording access to the server from a user,
The access log storage / retrieval device includes:
Corresponding to the user identification information, a plurality of accesses divided by one or more acronyms of the user identification information for storing an access log including time and access identification information of the web or content or other information Log files and
Corresponding to the user identification information, a metadata file that stores metadata including an access log file name and access log data storage location information;
An access log processing system, and an access log processing method in the access log processing system,

The access log storage / retrieval device includes:
Collecting access logs about information output to the terminal by each server by access from each user,
If duplicate data exists in each of the user identification information, time, and access access identification information among a plurality of access logs, the duplicate data is deleted, and one or more accesses for each user identification information Create log data,
Based on one or more access log data for each user identification information, create metadata including access log file name and data storage location information for each user identification information,
Storing one or more access log data for each user identification information in the access log file for each initial of the user identification information, storing metadata for each user identification information in the metadata file,
as well as,
When a search processing request specifying user identification information is received, the metadata file is searched based on the user identification information included in the received search processing request, and the access log file name and data storage location information of the metadata Extract
Read the access log from the access log file of the extracted access log file name according to the extracted data storage location information,
Restore the deleted duplicate data and create an access log, or create an access log with blanks or symbols or characters indicating duplication without restoring the deleted duplicate data. An access log processing system and an access log processing method that display on a display unit or store in a storage unit are provided.

According to the second solution of the present invention,
Multiple servers providing web or content or other information;
A terminal of a user accessing the server;
An access log storage / retrieval device for processing an access log recording access to the server from a user;
Corresponding to the user identification information, a plurality of accesses divided by one or more acronyms of the user identification information for storing an access log including time and access identification information of the web or content or other information Log files,
Corresponding to the user identification information, a metadata file that stores metadata including an access log file name and access log data storage location information;
An access log processing program in an access log processing system comprising:
A procedure of collecting an access log about information output to the terminal by each server by access from each user;
If duplicate data exists in each of the user identification information, time, and access access identification information among a plurality of access logs, the duplicate data is deleted, and one or more accesses for each user identification information The procedure to create log data,
A procedure for creating metadata including an access log file name and data storage location information for each user identification information based on one or a plurality of access log data for each user identification information;
Procedure for storing one or a plurality of access log data for each user identification information in the access log file for each initial of the user identification information, and storing metadata for each user identification information in the metadata file When,
as well as,
When a search processing request specifying user identification information is received, the metadata file is searched based on the user identification information included in the received search processing request, and the access log file name and data storage location information of the metadata Steps to extract
The procedure for reading the access log from the access log file with the extracted access log file name according to the extracted data storage location information,
Restore the deleted duplicate data and create an access log, or create an access log with blanks or symbols or characters indicating duplication without restoring the deleted duplicate data. There is provided an access log processing program for causing the access log storage / retrieval apparatus to execute a procedure for displaying on a display unit or storing in a storage unit.

According to the third solution of the present invention,
An access log storage and retrieval device for processing an access log recording access from a user to a plurality of servers that provide web or content or other information,

Corresponding to the user identification information, a plurality of accesses divided by one or more acronyms of the user identification information for storing an access log including time and access identification information of the web or content or other information Log files,
Corresponding to the user identification information, a metadata file that stores metadata including an access log file name and access log data storage location information;
An access log storage function unit for processing the access log and storing the access log and the metadata in the access log file and the metadata file, respectively;
In accordance with the access log search request, the access log search function unit for searching the access log and metadata from the access log file and the metadata file, respectively,
The access log storage function unit
Collecting access logs about information output to the terminal by each server by access from each user,
If duplicate data exists in each of the user identification information, time, and access access identification information among a plurality of access logs, the duplicate data is deleted, and one or more accesses for each user identification information Create log data,
Based on one or more access log data for each user identification information, create metadata including access log file name and data storage location information for each user identification information,
Storing one or more access log data for each user identification information in the access log file for each initial of the user identification information, storing metadata for each user identification information in the metadata file,
as well as,
The access log search function unit
When a search processing request specifying user identification information is received, the metadata file is searched based on the user identification information included in the received search processing request, and the access log file name and data storage location information of the metadata Extract
Read the access log from the access log file of the extracted access log file name according to the extracted data storage location information,
Restore the deleted duplicate data and create an access log, or create an access log with blanks or symbols or characters indicating duplication without restoring the deleted duplicate data. An access log storage / retrieval device that displays on a display unit or stores in a storage unit is provided.

According to the present invention, it is possible to search an access log at high speed by creating search metadata in which position information is recorded and compressing and storing the data of the access log.
In addition, according to the present invention, comparison is made with a method in which all data is read and output each time by creating metadata in which position information is recorded and reading corresponding data from a file or a disk based on the position information. In such a case, it is possible to obtain the search result at high speed. In addition, according to the present invention, it is possible to efficiently store and call access log data to the HDD by storing only the difference data, particularly for accesses such as Web access in which the same data continues. it can.

The system configuration figure and block diagram of the actual form of the Web access log storage / retrieval system to which the present invention is applied. The sequence diagram (data storage process) of embodiment of the web access log storage process to which this invention is applied. The sequence diagram (data search process) of embodiment of the web access log search process to which this invention is applied. FIG. 3 is an explanatory diagram (data block diagram) of a Web access log storage HDD 45 to which the Web access log storage process of FIG. 2 is applied. The figure which showed the search result screen (1) to which the web access log search process of FIG. 4 was applied. The figure (CSV file) which showed the search result screen (2) to which the web access log search process of FIG. 4 was applied. The flowchart about the Web access log duplication data deletion process 37. FIG. The flowchart about the web access log metadata preparation process 41. FIG. The figure which shows the example of the status code of a web server. Explanatory drawing of a metadata file. Explanatory drawing of an access log file.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.
FIG. 1 is a system configuration diagram / block diagram showing the actual form of a Web access log storage / retrieval system to which the present invention is applied. 1, this system includes a user terminal 1, a web server 2 accessed from the user terminal 1, a web access log storage / retrieval device 100, a web access log searcher terminal 6, a terminal 1 and web access. A network such as the Internet for connecting the log storage / retrieval device 100 is provided. Note that the terminal 6 of the Web access log searcher may also be connected to the Web access log storage / search apparatus 100 via a network such as the Internet.
The web access log storage / retrieval apparatus 100 includes a web access log storage function unit 3, a web access log storage HDD 4, and a web access log search function unit 5. The web access log storage function 3 processes a web access log recording access from a user and stores it in the web access log storage HDD 4. The Web access log search function unit 5 receives a Web access log search request from the Web access log searcher 6 for each user, and receives the metadata file 15 and the Web access log storage function unit 3 from the Web access log storage HDD 4. The metadata and the web access log respectively stored in the web access log files 16 and 17 are read and search processing is performed. The web access log storage function unit 3 includes a web access log collection process 7, a division process 8 for each web access log key, a merge process 9 using web access log key data, a web access log sort process 10, and a web access log duplicate data deletion process. 11, compression processing 12 for each Web access log key, Web access log metadata creation processing 13, and Web access log HDD storage processing 14. The Web access log storage HDD 4 includes a metadata file 15 and Web access log files 16 and 17 generated by the Web access log storage function unit 3. The apparatus for recording the Web access log is not limited to the HDD, and an appropriate memory device such as a flash memory or SD can be used. The Web access log search function 5 includes a Web access log search reception process 18, a search key metadata reading process 19, a corresponding Web access log reading part specifying process 20, a Web access log corresponding part reading process 21, and a corresponding data decompression process. Processing 22, duplicate data restoration processing 23, and search result response processing 24 are included.
Further, the Web access log storage / retrieval device 100 and / or the searcher terminal 6 includes a display unit for displaying a search result by the Web access log search function 5.

FIG. 10 is an explanatory diagram of the metadata file.
The metadata stored in the metadata file is represented by the access log file name of the access log file storing the access log data, the data start position and the data size of the corresponding user, corresponding to the user identification information. Access log data storage location. The user identification information is information such as a character string that can uniquely identify the user, such as a telephone number, an e-mail address, a login ID, and a user ID. In addition to this example, the data storage position of the access log may be appropriate position information such as a data storage start position and a data storage end position.

FIG. 11 is an explanatory diagram of the access log file.
A plurality of access log files are provided for each initial of the user identification information, and an access log file name is given. The “initial” used as a unit may be English, Japanese, other languages or symbols, or a single initial (eg, “a”, “b”... "I" ... etc.), or a plurality of initials (e.g., "aa", "ab"... The access log includes time, URL, user IP address, and status code corresponding to the user identification information. The URL is not limited to this, and may be appropriate access identification information that is web, content, or other information. The user IP address can be the address of the user's terminal. The access log file has the same format in the appropriate temporary storage area in the Web access log storage / retrieval apparatus 100 as described in FIG. 2 and the like in addition to the Web access log storage HDD 4.

FIG. 9 is a diagram illustrating an example of a status code of the Web server.
As shown in the figure, the status is indicated by the status code.

FIG. 2 shows a sequence diagram of an embodiment of a Web access log storage process to which the present invention is applied. This process is executed by the Web access log storage function unit 3.
In FIG. 2, the Web access log storage function unit 3 is “Web access log collection process 29 (7)” (reference numbers in parentheses correspond to those in FIG. 1), and each Web server receives a Web access from each user. The output web access log 28 is collected. Next, the Web access log storage function unit 3 performs “divide processing 31 (8) by Web access log key unit” for each user identification information (hereinafter, key) unit included in the Web access log 30. The web access log 30 collected from the web server is divided into files. For example, when there is a web access from the user “ab102”, “ab105”, “cy292”, “cy353”, each web server logs each file in a file such as a file with an initial letter “a” or “c” for each web access log. Divide the server log. Next, the Web access log storage function unit 3 uses the “Web access log key data merge process 33 (9)” to key the Web access log (file division in units of keys) 32 divided for each Web server log. Merge into units. For example, if the Web server 1 has four files, the file with the initial letter “a”, the file “c”, the file with the initial letter “a”, and the file “c” in the Web server 2, The process of merging the files 1 and 2 into the two files with the initial “a” and the initial “b” is performed. The Web access log storage function unit 3 sorts the Web access log (merged) 34 in units of keys by “Web access log sorting process 35 (10)”. For example, if the file with the initial “a” of the key includes the web access logs of the users “ab102” and “ab105”, the “ab105” is sorted after the web access log of “ab102” by sorting by the key unit. "Is the order of the web access log.

Next, in the “Web access log duplicate data deletion process 37 (11)”, the Web access log storage function unit 3 duplicates if there is duplicate data in each column of the Web access log (CSV file format). Delete duplicate data. For example, if there are four Web access logs for the user “ab102”, the user identification information (key) is duplicated, so the “ab102” data in the second and subsequent rows is deleted. As for the time, when accessing at the same time, the time information at the same time is deleted. As for the URL, for example, data that is divided and overlapped into schema, authority, and path is deleted, and data is rewritten to data of only the difference portion (backward match). The user IP address and status code columns are also deleted if duplicate data continues. Next, the Web access log storage function unit 3 performs compression (ZIP compression etc.) by compressing data (ZIP compression or the like) from which duplicate data is deleted in units of user identification information in “Compression process 39 (12) in Web access log key units”. Processing (BASE64 etc.) is performed. Next, the Web access log storage function unit 3 performs the “Web access log metadata creation process 41 (13)”, the value of each key of the data compressed in units of keys, the name of the stored Web access log file, the compression Create metadata with the data start position and compressed data size as a set. For example, when the data of the user “ab102” is recorded in the access log file (the key starts with “a”), the key value “ab102” and the target file name “access log file (the key start”) are stored in the metadata. “File-a”) with the character “a”), the start position of the web access log data (compressed) of the user “ab102” at the start position, and the web access log data of the user “ab102” (compressed) at the size ) Size is added. Next, the Web access log storage function unit 3 uses the “Web access log HDD storage process 44 (14)” to generate the Web access log (key unit compression) generated by the “Web access log key unit compression process 39 (12)”. ) 40 and the metadata created by the “Web access log metadata creation process 41 (13)” are stored in the access log file and the metadata file of the web access log storage HDD 4 respectively.
In each process from “Web access log collection process 29 (7)” to “Web access log metadata creation process 41 (13)”, an access log and a meta file in the access log file and meta file in the Web access log storage HDD 4 are stored. The data may be directly stored. In this case, the “Web access log HDD storage process 44 (14)” can be omitted.

FIG. 4 is an explanatory diagram of the Web access log storage HDD 45 to which the Web access log storage process of FIG. 2 is applied.
The metadata 68 records information for identifying a user indicating a data storage position in units of users, a target file name, a data storage start position, and a data storage size. Access log files 69 and 74 are files divided in units of user identification information. In this example, when the user identification information (key) is “ab102”, the metadata 68 stores the target file name “file-a”, the start position “0”, and the size “520”. The target file “file-a” 69 is a Web access log file whose user identification information (key) starts with “a”. By reading the size “520” from the start position “0”, the user identification information “ab102” is stored. The web access log 70 can be extracted. When decompressed, Web access log (duplicate data deleted) data 72 can be acquired. When there are four Web access logs of the user identification information “ab102”, the user identification information is duplicated, so the second and later are deleted. If the time is the same, it is deleted as well. The URL is decomposed into a schema, authority, and path so as to be duplicated, and duplicate data is similarly deleted. When only a part is different, data is stored only at the changed part (with backward matching), and other values are deleted.

FIG. 7 shows a flowchart for the duplicate Web access log data deletion process 37.
The Web access log storage function unit reads the collected Web access logs by the Web access log duplicate data deletion process 37 (S101). Next, the Web access log duplication data deletion process 37 compares the access log data read this time with the access log data read last time, and the same data as the previous access log (user identification information, time, URL, usage) (IP address, status code, etc.) are converted into duplicate identifiers of predetermined unique characters (eg, “.”, Blank, etc.) (S103). At this time, for the URL, a difference character string is extracted from the rear for each of the schema, authority, and path data. Next, by the Web access log duplication data deletion processing 37, the access log is accessed with an initial corresponding to the user identification information in the appropriate temporary storage area or the Web access log storage HDD 4 in the Web access log storage / retrieval apparatus. Store in the access log file with the log file name (S105). The above processing is repeated until there is no access log to be read next (S107).

FIG. 8 shows a flowchart of the Web access log metadata creation process 41.
The Web access log storage function unit reads the Web access log in units of user identification information (keys) from the temporary storage area or the accelerator log file of the Web access log storage HDD 4 by the Web access log metadata creation processing 41 (S201). ). Next, data storage position information such as the start position and size of the compressed data in key units is obtained by the Web access log metadata creation processing 41 (S203). Then, the web access log metadata creation processing 41 converts the media data including the access log file name and the data storage position information such as the data start position and data size of the corresponding user into the meta corresponding to the user identification information. Writing to the data file (S205). The above processing is repeated until there is no access log to be read next (S207).

FIG. 3 shows a sequence diagram of an embodiment of a Web access log search process to which the present invention is applied. This process is executed by the Web access log search function unit 5.
In FIG. 3, the Web access log search function unit 5 uses the “Web access log search reception process 53 (18)” (reference numbers in parentheses correspond to those in FIG. 1) to request a search from the Web access log searcher 51. And accepts the search process. When accepting a search, user identification information (key) that is a key of search data is designated. For example, when searching the Web access log of the user “ab102”, the Web access log searcher 51 inputs the data of the user “ab102” in the “Web access log search reception process 53 (18)”. Next, the Web access log search function unit 5 reads the metadata 55 from the metadata file in the “search key metadata reading process 54 (19)”. Since the metadata 55 is only data for the number of users, the size is smaller than that of the Web access log 58. Next, the Web access log search function unit 5 extracts the user's metadata from the metadata 55 by “the corresponding Web access log read location specification 56 (20)”. For example, when the reading location of the user “ab102” is specified, the corresponding line that matches the user identification information (key) of the metadata is searched, and the target file name, start position, and size of the corresponding line are extracted. . Next, the Web access log search function unit 5 uses the Web access log (key unit compression) 58 to search for the corresponding data (Web access log) of the user in the “Web access log corresponding portion reading process 57 (21)”. ) 59 is read. In this reading process, the Web access log file (key unit compression) 58 to be searched is designated from the target file name extracted in the “reading process 56 (20) of the corresponding Web access log reading”, and the corresponding from the start position and size. Data (Web access log) 59 is read. Next, the Web access log search function unit 5 performs decoding and decompression processing of the read corresponding data (Web access log) 59 in “corresponding data decompression processing 60 (22)”. After the decompression process, the Web access log (duplicate data deleted) (CSV file) 61 of the corresponding user from which the duplicate data has been deleted is output. Next, the Web access log search function unit 5 restores the deleted duplicate data in the “duplicate data restoration process 62 (23)”. At this time, the “duplicate data restoration process 62 (23)” determines that the data is duplicated based on the duplicate identifier (unique character “.”, Etc.) as described in the flowchart of FIG. 7, and restores the corresponding data. be able to. For example, when the time of the web access log is 10:00, the time after the second line is deleted, but the value of 10:00 is set in the restoration process. In the case of a URL, the same host and path often continue, and deletion processing for the second and subsequent lines and restoration processing of data in which only differences are recorded are performed. Next, the Web access log search function unit 5 sends a search result 66 response to the Web access log searcher 51 in the “search result response process 64 (24)”. The Web access log search function unit 5 may store the search result in the Web access log storage HDD 4.

  FIG. 5 is a diagram showing a search result screen (1) to which the Web access log search process of FIG. 4 is applied. In FIG. 5A, a normal web access result screen 79 displays a web access log result obtained by restoring duplicate data. In FIG. 5B, the change point is displayed to the searcher by using the Web access result (for difference investigation) 80 that is a screen in a format that does not restore duplicate data so that the Web access can be easily separated. Is possible. In FIG. 5C, the change point is emphasized by using the Web access result (for difference investigation) 81 in which the duplicate data is emphasized with the font, thickness, color, etc. so that the Web access can be easily separated in the same manner. Can be displayed to the searcher.

  FIG. 6 is a view showing a search result screen (2) to which the Web access log search process of FIG. 4 is applied. As shown in FIG. 6A, the search result can be displayed as a normal Web access log result CSV file 82. Further, as shown in FIG. 6B, it is possible to output a web access log result (for difference investigation) CSV file 83 for difference investigation.

As described above, according to the Web access log accelerated search technology as in the present embodiment, by creating metadata in which position information is recorded and reading the corresponding data based on the position information from the HDD. When compared with a method in which all data is read and output each time, search results can be acquired at high speed. In addition, Web access often follows links on the Web, and the same data, such as URL host data and path data, is continuous. By storing only the difference data, Web access log data is efficiently stored in the HDD. And call.

In the above, an example in which a Web server is accessed from a user terminal and various Webs are provided to the terminal by the Web server has been described. However, the terminal is not limited to the Web, and the server from the terminal regarding appropriate content or other information. And the server may provide the information to the terminal.
An access log processing method or system and an access log storage / retrieval apparatus according to the present invention include an access log processing program for causing a computer to execute each procedure, a computer-readable recording medium recording the access log processing program, and an access log process The program product can be provided by a program product that can be loaded into the internal memory of the computer, a computer such as a server that includes the program, and the like.

DESCRIPTION OF SYMBOLS 1 User's terminal 2 Web server 100 Web access log storage / retrieval apparatus 6 Web access log searcher's terminal 3 Web access log storage function part 4 Web access log storage HDD
5 Web access log search function part

Claims (10)

Multiple servers providing web or content or other information;
A terminal of a user accessing the server;
In an access log processing system comprising an access log storage / retrieval device for processing an access log recording access to the server from a user,
The access log storage / retrieval device includes:
Corresponding to the user identification information, a plurality of accesses divided by one or more acronyms of the user identification information for storing an access log including time and access identification information of the web or content or other information Log files,
Corresponding to the user identification information, a metadata file that stores metadata including an access log file name and access log data storage location information;
With

The access log storage / retrieval device includes:
Collecting access logs about information output to the terminal by each server by access from each user,
Among a plurality of access logs, if duplicate data to each of the user identification information and the time及beauty access identification information is present, remove each of the duplicate data, for each user identification information one or more Create access log data,
Based on one or more access log data for each user identification information, create metadata including access log file name and data storage location information for each user identification information,
Storing one or more access log data for each user identification information in the access log file for each initial of the user identification information, storing metadata for each user identification information in the metadata file,
as well as,
When a search processing request specifying user identification information is received, the metadata file is searched based on the user identification information included in the received search processing request, and the access log file name and data storage location information of the metadata Extract
Read the access log from the access log file of the extracted access log file name according to the extracted data storage location information,
Restore the deleted duplicate data and create an access log, or create an access log with blanks or symbols or characters indicating duplication without restoring the deleted duplicate data. An access log processing system that displays on a display unit or stores in a storage unit. The access log processing system according to claim 1,
The access log storage / retrieval device includes:
An access log storage function unit for processing the access log and storing the access log and the metadata in the access log file and the metadata file, respectively;
With
The access log storage function unit
Collect access logs about the information that each server outputs to the terminal by access from each user,
The access log collected from each server is divided into a plurality of files for each initial of the user identification information provided for each server .
Merge multiple access logs of multiple files for each initial of multiple servers according to user identification information,
Sort the merged multiple access logs by user identification information unit for each time and access identification information,
The sorted user identification information or time of a plurality of access logs or if there is overlap data access identification information, delete the duplicate data, by inserting a duplicate identifier indicating that delete, Create access log data for each user identification information, store it in the access log file for each initial of the user identification information,
An access log processing system that creates metadata including user identification information, a stored access log file name, and data storage location information based on access log data for each user identification information, and stores the metadata in a metadata file. In the access log processing system according to claim 1 or 2,
The access log storage / retrieval device includes:
In accordance with the access log search request, the access log search function unit for searching the access log and metadata from the access log file and the metadata file, respectively,
The access log search function unit
A search processing request specifying user identification information is received from the access log searcher terminal.
Based on the user identification information included in the received search processing request, search the metadata file, extract the access log file name and data storage location information in the metadata,
Read the access log from the access log file of the extracted access log file name according to the extracted data storage location information,
In accordance with the duplicate identifier of the access log, restore the deleted duplicate data and create an access log, or create an access log that contains blanks or symbols or characters indicating duplication without restoring the duplicate data deleted, An access log processing system that displays the access log on a display unit or stores it in a storage unit. In the access log processing system according to any one of claims 1 to 3,
The access log processing system, wherein the data storage position information is a data storage start position and a data storage size, or a data storage start position and a data storage end position. In the access log processing system according to any one of claims 1 to 4,
The access log processing system, wherein the user identification information includes any one of a user-specific ID, a login ID, a user telephone number, and a user mail address. In the access log processing system according to any one of claims 1 to 5,
2. The access log processing system according to claim 1, wherein the access identification information is a URL, and duplication is deleted for each scheme data, authority data, host data, port data, and path data. The access log processing system according to any one of claims 1 to 6,
Performs compression processing and encoding processing of data from which duplicate data has been deleted for each user identification information unit, and stores it in an access log file.
An access log processing system which performs decoding processing and decompression processing of an access log read from an access log file.
Multiple servers providing web or content or other information;
A terminal of a user accessing the server;
In an access log processing system comprising an access log storage / retrieval device for processing an access log recording access to the server from a user,
The access log storage / retrieval device includes:
Corresponding to the user identification information, a plurality of accesses divided by one or more acronyms of the user identification information for storing an access log including time and access identification information of the web or content or other information Log files,
Corresponding to the user identification information, a metadata file that stores metadata including an access log file name and access log data storage location information;
An access log processing method in the access log processing system comprising:

The access log storage / retrieval device includes:
Collecting access logs about information output to the terminal by each server by access from each user,
Among a plurality of access logs, if duplicate data to each of the user identification information and the time及beauty access identification information is present, remove each of the duplicate data, for each user identification information one or more Create access log data,
Based on one or more access log data for each user identification information, create metadata including access log file name and data storage location information for each user identification information,
Storing one or more access log data for each user identification information in the access log file for each initial of the user identification information, storing metadata for each user identification information in the metadata file,
as well as,
When a search processing request specifying user identification information is received, the metadata file is searched based on the user identification information included in the received search processing request, and the access log file name and data storage location information of the metadata Extract
Read the access log from the access log file of the extracted access log file name according to the extracted data storage location information,
Restore the deleted duplicate data and create an access log, or create an access log with blanks or symbols or characters indicating duplication without restoring the deleted duplicate data. An access log processing method for displaying on a display unit or storing in a storage unit. Multiple servers providing web or content or other information;
A terminal of a user accessing the server;
An access log storage / retrieval device for processing an access log recording access to the server from a user;
Corresponding to the user identification information, a plurality of accesses divided by one or more acronyms of the user identification information for storing an access log including time and access identification information of the web or content or other information Log files,
Corresponding to the user identification information, a metadata file that stores metadata including an access log file name and access log data storage location information;
An access log processing program in an access log processing system comprising:
A procedure of collecting an access log about information output to the terminal by each server by access from each user;
Among a plurality of access logs, if duplicate data to each of the user identification information and the time及beauty access identification information is present, remove each of the duplicate data, for each user identification information one or more Create access log data,
A procedure for creating metadata including an access log file name and data storage location information for each user identification information based on one or a plurality of access log data for each user identification information;
Procedure for storing one or a plurality of access log data for each user identification information in the access log file for each initial of the user identification information, and storing metadata for each user identification information in the metadata file When,
as well as,
When a search processing request specifying user identification information is received, the metadata file is searched based on the user identification information included in the received search processing request, and the access log file name and data storage location information of the metadata Steps to extract
The procedure for reading the access log from the access log file with the extracted access log file name according to the extracted data storage location information,
Restore the deleted duplicate data and create an access log, or create an access log with blanks or symbols or characters indicating duplication without restoring the deleted duplicate data. An access log processing program for causing the access log storage / retrieval apparatus to execute a procedure of displaying on a display unit or storing in a storage unit. An access log storage and retrieval device for processing an access log recording access from a user to a plurality of servers that provide web or content or other information,

Corresponding to the user identification information, a plurality of accesses divided by one or more acronyms of the user identification information for storing an access log including time and access identification information of the web or content or other information Log files,
Corresponding to the user identification information, a metadata file that stores metadata including an access log file name and access log data storage location information;
An access log storage function unit for processing the access log and storing the access log and the metadata in the access log file and the metadata file, respectively;
In accordance with the access log search request, the access log search function unit for searching the access log and metadata from the access log file and the metadata file, respectively,
The access log storage function unit
Collecting access logs about information output to the terminal by each server by access from each user,
Among a plurality of access logs, if duplicate data to each of the user identification information and the time及beauty access identification information is present, remove each of the duplicate data, for each user identification information one or more Create access log data,
Based on one or more access log data for each user identification information, create metadata including access log file name and data storage location information for each user identification information,
Storing one or more access log data for each user identification information in the access log file for each initial of the user identification information, storing metadata for each user identification information in the metadata file,
as well as,
The access log search function unit
When a search processing request specifying user identification information is received, the metadata file is searched based on the user identification information included in the received search processing request, and the access log file name and data storage location information of the metadata Extract
Read the access log from the access log file of the extracted access log file name according to the extracted data storage location information,
Restore the deleted duplicate data and create an access log, or create an access log with blanks or symbols or characters indicating duplication without restoring the deleted duplicate data. An access log storage / retrieval device that displays on a display unit or stores in a storage unit.

Images