JP6971053B2 - Data management equipment, data management methods, and programs - Google Patents

Description

The present invention relates to a data management device, a data management method, and a program.

Conventionally, in a system that provides a web page, an operation of acquiring an access log showing a history of a user accessing a web page using a terminal device and storing it in a storage device has been performed. Further, a method has been proposed in which the stored access log is analyzed to grasp the number of times a web page is accessed, information on a user who has accessed the web page, and the like (see, for example, Patent Document 1).

Japanese Unexamined Patent Publication No. 2014-22821

When the above-mentioned access log is stored for a long period of time, the capacity of the storage device for storing the access log may become tight. As a countermeasure, access logs are deleted regularly. However, even an access log that is determined to be unnecessary at the time of deletion may be required in the analysis process to be performed in the future. Therefore, there is a demand for a method of storing access logs required for analysis while reducing the capacity.

Further, in the conventional access log analysis process, the access log itself is targeted for processing. However, when processing an access log having a large capacity, the load on the processing device increases, and the analysis process may take time. Therefore, there is a demand for a method of storing access logs in a format that is easy to analyze.

The present invention has been made in consideration of such circumstances, and provides a data management device, a data management method, and a program capable of storing log data in a format suitable for analysis processing while suppressing the capacity. One of the purposes is to do.

One aspect of the present invention is an acquisition unit that acquires log data acquired in response to access by a terminal device, and data included in the log data acquired by the acquisition unit for which the first period has elapsed. On the other hand, the first sampling process focusing on the first item included in the log data is performed to generate the first sampling log, and the second item focusing on the second item included in the log data is generated. The sampling unit that performs sampling processing to generate a second sampling log, and the data included in the log data acquired by the acquisition unit that has passed the first period are invalidated and the sampling unit is used. Invalidates the first sampling log and the second sampling log that have passed a second period longer than the first period among the first sampling log and the second sampling log generated by. It is a data management device equipped with a conversion unit.

According to one aspect of the present invention, it is possible to manage log data in a format suitable for analysis processing while suppressing the capacity.

It is a block diagram which shows an example of a data management system 1. It is a figure which shows an example of the log data L. It is a figure which shows the concept of data management by a data management apparatus 7. It is a block diagram which shows an example of the functional structure of a data management apparatus 7. It is a flowchart which shows an example of the flow of processing by a data management apparatus 7. It is a figure which shows an example of a sampling log SL (behavior log). It is a figure which shows an example of a sampling log SL (content log). It is a figure which shows an example of edit data ED (attribute data). It is a figure which shows an example of edit data ED (metadata).

Hereinafter, embodiments of the data management apparatus, data management method, and program of the present invention will be described with reference to the drawings. The data management device of the present invention performs data storage period, for example, by performing sampling processing, editing processing, invalidation processing, etc. on log data acquired in response to access to an electronic page such as a web page. Manage log data by appropriately setting the contents of the data to be stored. The electronic page may include a web page referenced by a browser as well as an app page referenced by an application program. In the following explanation, the explanation will be focused on the web page.

[overall structure]
FIG. 1 is a configuration diagram showing an example of a data management system 1. The data management system 1 includes, for example, one or more terminal devices 3, one or more service providing devices 5, and one or more data management devices 7. The terminal device 3, the service providing device 5, and the data management device 7 are connected to each other by a network NW, and communicate with each other via the network NW. The network NW includes, for example, a WAN (Wide Area Network), a LAN (Local Area Network), the Internet, a dedicated line, a wireless base station, a provider, and the like.

[Terminal device]
The terminal device 3 is operated by a user who uses the service provided by the service providing device 5. The terminal device 3 is, for example, a personal computer, a mobile phone such as a smartphone, a tablet terminal, or a computer device such as a PDA (Personal Digital Assistant).

When the terminal device 3 receives a predetermined operation from the user, the terminal device 3 accesses the web page provided by the service providing device 5 via a browser installed in advance. For example, the web page provided by the service providing device 5 is a page constituting a news site, a shopping site, a search site, an auction site, an SNS (Social Networking Service) site, and the like.

[Service provider]
The service providing device 5 may be a web server device that provides a web page such as the above-mentioned news site or shopping site on the Internet, or communicates with the terminal device 3 in which the application is started to obtain various information. It may be an application server device that performs delivery. The service providing device 5 outputs the log data L acquired in response to the access by the terminal device 3 to the web page provided by the service providing device 5.

FIG. 2 is a diagram showing an example of log data L. The log data L indicates, for example, a "user ID 10" that identifies a user who uses the terminal device 3, a "content ID 11" that identifies content associated with a web page, and a date when access from the terminal device 3 is processed. "Date 12", "target URL 13" indicating the URL (Uniform Resource Locator) of the web page accessed from the terminal device 3, "transition source URL 14" indicating the URL of the transition source web page to the target URL, and the like. included.

The "user ID 10" includes, for example, a login ID when the user is logged in when accessing the service providing device 5 using the terminal device 3. Alternatively, the "user ID 10" includes information on a cookie (HTTP cookie) managed for each web browser provided in the terminal device 3, an IP address of the terminal device 3, and the like.

The "content ID 11" is, for example, an identifier that identifies a product, service, or the like that is content associated with a web page. For example, when the web page is a shopping site for purchasing "product A", the web page is associated with a content ID indicating "product A". Further, for example, when the web page is a news site that provides news of "sports A", the web page is associated with a content ID indicating "sports A". The data indicating the association between the web page and the content may be stored in, for example, various masters stored in the service providing device 5 or an external storage device.

For example, one line of data is added to the log data L each time the terminal device 3 accesses the web page. The log data L is text data created at an arbitrary timing such as every day or every week.

[Data management device]
The data management device 7 acquires, for example, the log data L of the web page provided by the service providing device 5 or the application page referenced by the application program, and manages the acquired log data L.

FIG. 3 is a diagram showing the concept of data management by the data management device 7. The data management device 7 manages the data included in the log data L in three stages. These three stages include, for example, a short-term storage stage, a long-term storage stage, and an indefinite storage stage.

The short-term storage stage is a stage in which the log data L itself is stored. In this short-term storage stage, for example, log data L for the past year is stored.

The long-term storage stage is a stage in which the sampling log SL obtained by performing a predetermined sampling process on the log data L is stored. In this long-term storage stage, for example, sampling log SL for the past three years, which is longer than the short-term storage stage described above, is stored. In this sampling log SL, for example, the "behavior log AL" and the "content" obtained as a result of sampling processing focusing on the items related to the "user" for the data included in the log data L. The "content log CL" obtained as a result of performing sampling processing focusing on the items related to is included.

The indefinite period storage stage is a stage in which the edited data ED obtained by performing a predetermined editing process on the log data L is stored. In this indefinite storage stage, for example, no storage period is provided, and all past edited data EDs are stored. The edited data ED includes, for example, "attribute data AD" and "content" obtained as a result of editing the data included in the log data L by focusing on the items related to the "user". The "metadata MD" obtained as a result of performing the editing process focusing on the items related to the above is included.

FIG. 4 is a block diagram showing an example of the functional configuration of the data management device 7. The data management device 7 includes, for example, an acquisition unit 20, a sampling unit 22, an editing unit 24, an invalidation unit 26, and a storage unit 28. The storage unit 28 includes, for example, a log data storage unit 30, a sampling log storage unit 32, and an edit data storage unit 34. Each functional unit included in the data management device 7 may be distributed to a plurality of devices. For example, the sampling unit 22 and other functional units may be realized by a separate device. The storage unit 28 may be a storage device such as NAS (Network Attached Storage).

The acquisition unit 20, the sampling unit 22, the editing unit 24, and the invalidation unit 26 are realized by, for example, a processor such as a CPU (Central Processing Unit) executing a program (software) stored in the storage unit 28. NS. The program may be downloaded from the application server via the network NW, or may be pre-installed in the data management device 7 in advance. Further, these functional units may be realized by hardware such as LSI (Large Scale Integration), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), or cooperation between software and hardware. May be realized by.

The storage unit 28 may be realized by, for example, a RAM (Random Access Memory), a ROM (Read Only Memory), an HDD (Hard Disk Drive), a flash memory, or a hybrid storage device in which a plurality of these are combined. ..

The acquisition unit 20 acquires the log data L from the service providing device 5 via the network NW. The acquisition unit 20 stores the log data L acquired from the service providing device 5 in the log data storage unit 30.

The sampling unit 22 performs sampling processing on the log data L for which the first period has passed among the data included in the log data L acquired by the acquisition unit 20, and generates a sampling log SL. The sampling unit 22 stores the generated sampling log SL in the sampling log storage unit 32. For example, when the log data L is daily text data created daily by the service providing device 5, the sampling unit 22 performs sampling processing on the text data one year or more after the creation.

(Sampling process)
The sampling unit 22 performs data sampling (data thinning) on the log data L for which the first period has elapsed. This sampling process is, for example, a first sampling process focusing on the first item related to the "user" and a second sampling process focusing on the second item related to the "content". including.

In the first sampling process focusing on the first item related to the "user", for example, in the log data L, the log data in which the data of the item of the "user ID 10" related to the "user" exists is extracted. Sampling process to be performed, sampling process to extract log data of "user ID 10" in a specific format, and the like are included.

The second sampling process focusing on the second item related to the "content" includes, for example, a sampling process for extracting data including the "content ID 11" associated with the specific content in the log data L. include.

Further, the sampling unit 22 is set with a ratio of log data sampled by the sampling process (hereinafter, referred to as “sampling rate”). For example, the sampling unit 22 adjusts the amount of the sampling log SL based on the sampling rate. In addition to the above-mentioned sampling of log data in row units, the sampling unit 22 may perform sampling processing for each item in column units.

Further, the sampling unit 22 may perform sampling processing on the log data L step by step. For example, the sampling unit 22 may perform the first sampling process on the log data L for which the first period has elapsed, and then perform the second sampling process after the elapse of a predetermined period. Further, the sampling unit 22 may perform a third sampling process on the data on which the second sampling process has been performed after a predetermined period of time has elapsed. By such two or more stages of sampling processing, it is possible to set a stepwise storage period such as short-term storage, medium-term storage, and long-term storage of the sampling log SL.

The editing unit 24 performs an editing process on the log data L acquired by the acquisition unit 20 to generate the edited data ED. The editing unit 24 stores the generated editing data ED in the editing data storage unit 34. For example, if the log data L is daily text data created daily by the service providing device 5, the editorial unit 24 edits the text data.

(Editing process)
The above-mentioned editing process is, for example, a first editing process focusing on the first item related to the "user" and a second editing process focusing on the second item related to the "content". And include.

In the first editing process focusing on the first item related to the "user", for example, in the log data L, for each item of the "user ID 10" related to the "user", other items are semantically based. Aggregate or replace in the attributed columns (attributes) of, sort and invalidate other items, sort log data, etc., and generate edit data ED (attribute data AD) in a predetermined format according to the purpose. do.

The second editing process focusing on the second item related to the "content" is, for example, in the log data L, for each "content ID 11" related to the "content", other items are meaning-based attributes. It aggregates or replaces the converted columns (attributes), sorts and invalidates other items, sorts log data, etc., and generates edit data in a predetermined format according to the purpose.

The invalidation unit 26 invalidates the data contained in the log data L stored in the log data storage unit 30 for which the first period has passed. Disabling the log data L means, for example, physically or logically deleting the log data L from the log data storage unit 30. Here, the log data L to be invalidated is the log data to be sampled by the sampling unit 22. The first period is set to, for example, "1 year". Here, when the log data L is daily text data created daily by the service providing device 5, the invalidation unit 26 invalidates the text data one year or more after the creation.

Further, the invalidation unit 26 invalidates the sampling log SL in which the second period has elapsed among the sampling SLs stored in the sampling log storage unit 32. This second period is set to be longer than the first period described above. The second period is set to, for example, "3 years". In this case, the invalidation unit 26 invalidates the sampling log SL that has been generated by the sampling unit 22 for 3 years or more. When the sampling unit 22 performs the stepwise sampling process as described above, the invalidation unit 26 invalidates the above for each sampling log S according to the period individually set for the sampling log SL. You may perform the conversion process.

The log data storage unit 30 stores the log data L acquired by the acquisition unit 20. The sampling log storage unit 32 stores the sampling log SL generated by the sampling unit 22. The editing data storage unit 34 stores the editing data ED generated by the editing unit 24.

[Processing of data management device]
Next, the operation of the data management device 7 will be described with reference to FIGS. 5 to 9. FIG. 5 is a flowchart showing an example of the processing flow by the data management device 7. The process shown in the flowchart of FIG. 5 is started at a predetermined timing such as daily, weekly, or monthly. Alternatively, this process may be started in response to an instruction from the user of the data management device 7.

First, the acquisition unit 20 acquires the log data L from the service providing device 5 via the network NW (step S101). The acquisition unit 20 stores the acquired log data L in the log data storage unit 30.

Next, either one or both of the sampling process (step S103) by the sampling unit 22 and the editing process (step S105) by the editing unit 24 are performed. The process to be executed is determined according to a preset schedule or an instruction by the user.

(Sampling process)
The sampling unit 22 reads out the log data L for which the first period has passed out of the log data L acquired by the acquisition unit 20 and stored in the log data storage unit 30, performs sampling processing, and performs sampling log SL. Generate (step S103). The sampling unit 22 stores the generated sampling log SL in the sampling log storage unit 32.

Here, the sampling unit 22 performs the first sampling process focusing on the item of the "user ID 10" related to the "user" among the items included in the log data L, and performs the first sampling process, and the action log AL (as the sampling log SL). First sampling log) is generated. For example, among the data included in the log data L, the sampling unit 22 samples the data in which the data of the item of the user ID 10 exists.

FIG. 6 is a diagram showing an example of a sampling log SL in which log data in which the data of the item of the user ID 10 exists is sampled from the log data L shown in FIG. As shown in FIG. 6, in the sampling log SL, among the data included in the log data L, the data in which the data of the item of the user ID 10 in the third row and the sixth row does not exist is invalidated.

Further, the sampling unit 22 performs a second sampling process focusing on the item of the "content ID 11" related to the "content" among the items of the data included in the log data L, and performs the second sampling process, and the content log CL as the sampling log SL. (Second sampling log) is generated. The sampling unit 22 samples, for example, data included in the log data L in which the data of the item of the content ID 11 has a specific format.

FIG. 7 is a diagram showing an example of a sampling log SL obtained by sampling log data in which the data of the item of the content ID 11 is in the range of “00001” to “00099” from the log data L shown in FIG. As shown in FIG. 7, in the sampling log SL, among the data included in the log data L, the data of the item of the content ID 11 is not in the range of "00001" to "00099", and the data in the 5th to 7th lines. Has been disabled.

Further, the sampling unit 22 adjusts the amount of data included in the above-mentioned sampling log SL based on a preset sampling rate. For example, when "30%" is set as the sampling rate, adjustment is made so that 30% of the data included in the sampling log SL remains (70% of the data is invalidated).

(Editing process)
The editing unit 24 performs an editing process on the log data L acquired by the acquisition unit 20 and stored in the log data storage unit 30, and generates an edited data ED (step S105). The editing unit 24 stores the generated editing data ED in the editing data storage unit 34.

The editorial unit 24 performs the first editing process focusing on the item of the "user ID 10" related to the "user" among the items included in the log data L, and performs the first editing process, and the attribute data AD (first) as the editing data ED. Edit data) is generated. The editing unit 24 performs the first editing process according to the master data or a predetermined processing logic, and generates the editing data ED. For example, in the log data L, the editorial unit 24 aggregates the tendency of the content to be accessed, whether the linked page of the target URL associated with each user ID is for men or women, and the user ID. Is used as a key, and other items are aggregated or replaced with a meaning-based attributed column (attribute) such as "gender". The correspondence between the "user ID" and the "attribute" may be performed by referring to the data input by the user when registering as a member of the web page.

FIG. 8 is a diagram showing an example of edit data ED (behavior history data) obtained by performing the first edit process on the log data L shown in FIG. 2. FIG. 8 shows data in which "attribute 1 (gender)" is "male" and "attribute 2 (hobby)" is "sports" with respect to data in which "user ID" is "bbbbbb". ing. In this way, it is possible to compress the data capacity by performing the editing process focusing on the item of the "user ID 10" related to the "user". Further, such edited data ED is in a format suitable for analysis processing in which it is easy to grasp statistical data of user behavior. The editorial unit 24 may add various attribute information such as age and access frequency to the editing data ED in addition to or in place of the above-mentioned "attribute 1 (gender)" and "attribute 2 (hobby)". ..

Further, the editorial unit 24 performs a second editing process focusing on the item of the "content ID 11" related to the "content" among the items included in the log data L, and performs the second editing process, and the metadata MD (the first) as the editing data ED. 2 edit data) is generated. The editing unit 24 performs a second editing process according to the master data or a predetermined processing logic, and generates the editing data ED. For example, the editorial unit 24 aggregates or replaces other items with a meaning-based attributed column (attribute) such as "product, service" using the content ID included in the log data L as a key. In addition, the editorial unit 24, for example, aggregates the tendencies of users who have accessed each content ID, and determines the attribute of the "target user".

FIG. 9 is a diagram showing an example of edit data ED (access history data) obtained by performing a second edit process on the log data L shown in FIG. 2. In FIG. 9, data in which "attribute 1 (product, service)" is associated with "sports" and "attribute 2 (target user)" is associated with "male" with respect to data in which "content ID" is "00001". It is shown. In this way, it is possible to compress the data capacity by performing the editing process focusing on the item of "content ID 11" related to "content". Further, such an edited data ED is in a format suitable for analysis processing, which makes it easy to grasp the user's access status to the web page associated with the content ID. In addition, the editorial unit 24 adds various attribute information such as the number of accesses to the edit data ED in addition to or instead of the above "attribute 1 (product, service)" and "attribute 2 (target user)". good.

When information related to the image displayed on each web page is stored in the log data storage unit 30, the editorial unit 24 determines the feature amount such as the color of the image and the type of the image (wind image scene). The edited data ED (metadata MD) may be generated based on (whether it is a person image or not).

Next, the invalidation unit 26 invalidates the log data L whose first period has passed among the log data L stored in the log data storage unit 30, and the sampling SL stored in the sampling log storage unit 32. Of these, the sampling log SL for which the second period has elapsed is invalidated (step S107). This completes the processing of this flowchart.

According to the embodiment described above, it is possible to manage log data in a format suitable for analysis processing while suppressing the capacity. Since the capacity of the sampling log SL is smaller than that of the log data L, it is possible to reduce the capacity of the data stored in the storage unit while retaining the necessary information. This sampling log SL can be used when generating a desired model using the log data L. Further, the edited data ED has a format suitable for analysis, and it is possible to reduce the load on the processing device in the analysis and also to shorten the processing time. Further, since the edited data ED is not invalidated, it is possible to leave necessary information while suppressing the capacity.

In the above-described embodiment, each of the acquisition unit 20, the sampling unit 22, and the editing unit 24 stores the log data L, the sampling log SL, and the editing data ED in the data management device 7. The configuration to be stored in 28 was explained. However, each of the acquisition unit 20, the sampling unit 22, and the editing unit 24 stores the log data L, the sampling log SL, and the editing data ED in a region (for example, a foreign country, a region, etc.) where the electricity bill is cheap. It may be sent to the device. Further, each of the acquisition unit 20, the sampling unit 22, and the editing unit 24 may output the log data L, the sampling log SL, and the editing data ED to an external storage medium such as a magnetic tape.

Further, in the above-described embodiment, the configuration in which the sampling unit 22 performs the sampling process based on a predetermined sampling rate has been described. However, the sampling unit 22 may check the free space of the storage unit 28 and dynamically change the sampling rate according to the free space.

Further, in the above-described embodiment, the sampling unit 22 performs sampling processing based on a predetermined "first period", and the invalidation unit 26 performs a predetermined "first period" and "second period". ], The configuration for performing invalidation processing was explained. However, the sampling unit 22 or the invalidation unit 26 confirms the free space of the storage unit 28 and dynamically changes the "first period" and the "second period" according to the free space. May be good.

Although the embodiments for carrying out the present invention have been described above using the embodiments, the present invention is not limited to these embodiments, and various modifications and substitutions are made without departing from the gist of the present invention. Can be added.

1 Data management system, 3 Terminal equipment, 5 Service provision equipment, 7 Data management equipment, 20 Acquisition unit, 22 Sampling unit, 24 Editing department, 26 Invalidation unit, 28 Storage unit, 30 Log data storage unit, 32 sampling log storage unit, 34 editing data storage unit, NW network

Claims (9)

An acquisition unit that acquires log data acquired in response to access by the terminal device,
Of the data included in the log data acquired by the acquisition unit, the data for which the first period has passed is subjected to the first sampling process focusing on the first item included in the log data. A sampling unit that generates a sampling log of 1 and performs a second sampling process focusing on the second item included in the log data to generate a second sampling log.
Of the data included in the log data acquired by the acquisition unit, the data for which the first period has elapsed is invalidated, and the first sampling log and the second sampling log generated by the sampling unit are used. out e Bei and said invalidating unit for invalidating the first sampling logs and second sampling logs longer second period of time than the first time period,
The first sampling process is a process of extracting data in which the data of the first item is present in the data for which the first period has elapsed, or a process of extracting the data in which the data of the first period has elapsed, or the first in the data for which the first period has elapsed. Item data is the process of extracting data in a specific format.
The second sampling process is a process of extracting data in which the data of the second item is associated with a specific content in the data after the first period has passed.
Data management device. The data included in the log data acquired by the acquisition unit is subjected to the first editing process focusing on the first item to generate the first editing data, and the second item is focused on. second editing process further example Bei editing unit for generating a second edited data by that,
The first editing process is a process of aggregating or replacing the data of other items for each data of the first item in the log data.
The second editing process is a process of aggregating or replacing the data of other items for each data of the second item in the log data.
The data management device according to claim 1. The first item is an item for identifying a user who uses the terminal device.
The second item is an item for identifying the content associated with the electronic page accessed by the terminal device.
The data management device according to claim 1 or 2. The invalidation unit does not invalidate the first edit data and the second edit data generated by the editorial unit.
The data management device according to claim 2. The sampling unit performs the first sampling process and the second sampling process based on a predetermined sampling rate.
The data management device according to any one of claims 1 to 4. Each of the acquisition unit, the sampling unit, and the editing unit arranges the log data, the sampling log, and the editing data in an area where the electricity bill is cheaper than the area where the data management device is arranged. Send to the storage device,
The data management device according to claim 2. An acquisition unit that acquires log data acquired in response to access by the terminal device,
Of the data included in the log data acquired by the acquisition unit, the data for which the first period has passed is subjected to the first editing process focusing on the first item included in the log data. e Bei and editing unit configured to generate a second editing data by the second editing process that focuses on the second item included in the log data to generate a first edited data,
The first editing process is a process of aggregating or replacing the data of other items for each data of the first item in the log data.
The second editing process is a process of aggregating or replacing the data of other items for each data of the second item in the log data.
Data management device. The computer
Acquires the log data acquired in response to access by the terminal device,
Of the acquired data included in the log data, the data for which the first period has passed is subjected to the first sampling process focusing on the first item included in the log data, and the first sampling is performed. A log is generated and a second sampling process focusing on the second item included in the log data is performed to generate a second sampling log.
Of the acquired data included in the log data, the data for which the first period has passed is invalidated, and the first sampling log and the second sampling log generated are said to have the first period. Disable the first sampling log and the second sampling log after a longer second period .
The first sampling process is a process of extracting data in which the data of the first item is present in the data for which the first period has elapsed, or a process of extracting the data in which the data of the first period has elapsed, or the first in the data for which the first period has elapsed. Item data is the process of extracting data in a specific format.
The second sampling process is a process of extracting data in which the data of the second item is associated with a specific content in the data after the first period has passed.
Data management method. On the computer
Get the log data acquired according to the access by the terminal device,
Of the acquired data included in the log data, the data for which the first period has passed is subjected to the first sampling process focusing on the first item included in the log data, and the first sampling is performed. A log is generated and a second sampling process focusing on the second item included in the log data is performed to generate a second sampling log.
Of the acquired data included in the log data , the data for which the first period has passed is invalidated, and the first sampling log and the second sampling log generated are said to have the first period. Disable the first sampling log and the second sampling log after a longer second period .
The first sampling process is a process of extracting data in which the data of the first item is present in the data for which the first period has elapsed, or a process of extracting the data in which the data of the first period has elapsed, or the first in the data for which the first period has elapsed. Item data is the process of extracting data in a specific format.
The second sampling process is a process of extracting data in which the data of the second item is associated with a specific content in the data after the first period has passed.
program.

Images