JP2018181234A - Data management apparatus, data management method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a data management apparatus which can manage log data in a format suitable to analysis, while suppressing the volume, a data management method, and a program.SOLUTION: A data management apparatus includes: an acquisition unit which acquires log data to be acquired in accordance with an access from a terminal device; a sampling unit which executes first sampling processing on data over a first period included in the log data, focusing on a first item included in the log data, to generate a first sampling log, and executes second sampling processing focusing a second item to generate a second sampling log; and an invalidation unit which invalidates the data over the first period included in the log data, and invalidates the first sampling log and second sampling log over a second period longer than the first period.SELECTED DRAWING: Figure 4

Description

  The present invention relates to a data management device, a data management method, and a program.

  2. Description of the Related Art Conventionally, in a system for providing a web page, an operation is performed in which an access log indicating a history of access to a web page by a user using a terminal device is acquired and stored in a storage device. Further, there has been proposed a method of grasping the access count of a web page, the information of the user who accessed the web page, and the like by analyzing the stored access log (see, for example, Patent Document 1).

JP 2014-22821 A

  If the above access log is stored for a long time, the capacity of the storage device storing the access log may be exhausted. As a countermeasure against this, an operation to delete access logs regularly is performed. However, even an access log determined to be unnecessary at the time of deletion may be required in analysis processing to be performed in the future. Therefore, there is a need for a method of storing access logs necessary for analysis while suppressing the capacity.

  Further, in the conventional access log analysis processing, the access log itself is a processing target. However, when an access log with a large capacity is processed, the load on the processing apparatus may increase, which may require time for analysis processing. Therefore, there is a need for a method of storing access logs in a format that is easy to analyze.

  The present invention has been made in consideration of such circumstances, and provides a data management apparatus, data management method, and program capable of storing log data in a format suitable for analysis processing while suppressing the capacity. One of the goals.

  One aspect of the present invention is an acquisition unit that acquires log data acquired according to an access by a terminal device, and data in which a first period has elapsed among data included in the log data acquired by the acquisition unit. And a first sampling process focusing on the first item included in the log data to generate a first sampling log and a second focusing on the second item included in the log data. A sampling unit that performs sampling processing to generate a second sampling log, invalidates data of which the first period has elapsed among data included in the log data acquired by the acquisition unit, and the sampling unit A second of the first sampling log and the second sampling log generated by During a data management device and a disabling unit for disabling the first sampling logs and second sampling log has passed.

  According to one aspect of the present invention, it is possible to manage log data in a format suitable for analysis processing while suppressing the capacity.

FIG. 1 is a block diagram showing an example of a data management system 1; It is a figure which shows an example of log data L. FIG. FIG. 6 is a diagram showing the concept of data management by the data management device 7; FIG. 6 is a block diagram showing an example of a functional configuration of a data management device 7; 5 is a flowchart showing an example of the flow of processing by the data management device 7; It is a figure which shows an example of sampling log SL (action log). It is a figure which shows an example of sampling log SL (content log). It is a figure which shows an example of edit data ED (attribute data). It is a figure which shows an example of edit data ED (metadata).

  Hereinafter, embodiments of a data management apparatus, a data management method, and a program of the present invention will be described with reference to the drawings. The data management apparatus according to the present invention performs, for example, sampling processing, editing processing, invalidation processing, and the like on log data acquired in response to access to an electronic page such as a web page, whereby the data storage period, Set the contents of stored data, etc. as appropriate, and manage log data. The electronic page may include, in addition to the web page referenced by the browser, the application page referenced by the application program. The following description focuses on web pages.

[overall structure]
FIG. 1 is a block diagram showing an example of the data management system 1. The data management system 1 includes, for example, one or more terminal devices 3, one or more service providing devices 5, and one or more data management devices 7. The terminal device 3, the service providing device 5, and the data management device 7 are connected to one another by a network NW, and communicate with each other via the network NW. The network NW includes, for example, a wide area network (WAN), a local area network (LAN), the Internet, a dedicated line, a wireless base station, a provider, and the like.

[Terminal device]
The terminal device 3 is operated by the user who uses the service provided by the service providing device 5. The terminal device 3 is, for example, a computer device such as a personal computer, a mobile phone such as a smart phone, a tablet terminal, or a PDA (Personal Digital Assistant).

  When the terminal device 3 receives a predetermined operation from the user, the terminal device 3 accesses a web page provided by the service providing device 5 through a browser installed in advance. For example, the web page provided by the service providing apparatus 5 is a page configuring a news site, a shopping site, a search site, an auction site, a social networking service (SNS) site, and the like.

[Service Providing Device]
The service providing device 5 may be a web server device that provides web pages such as the above-described news site and shopping site on the Internet, and communicates with the terminal device 3 on which the application has been started to communicate various information It may be an application server device that performs delivery. The service providing device 5 outputs log data L acquired in response to access by the terminal device 3 to a web page provided by itself.

  FIG. 2 is a diagram showing an example of the log data L. As shown in FIG. The log data L indicates, for example, "user ID 10" for identifying a user who uses the terminal device 3, "content ID 11" for identifying content associated with a web page, and date when access from the terminal device 3 was processed. "Date 12", "Target URL 13" indicating the URL (Uniform Resource Locator) of the web page accessed by the terminal device 3, "Transition source URL 14" indicating the URL of the web page of the transition source to the target URL, etc. included.

  The “user ID 10” includes, for example, a login ID in the case where the user is logging in when accessing the service providing apparatus 5 using the terminal device 3. Alternatively, the “user ID 10” includes information on a cookie (HTTP cookie) managed for each web browser provided in the terminal device 3, an IP address of the terminal device 3, and the like.

  The “content ID 11” is, for example, an identifier that identifies a product, a service, or the like that is content associated with the web page. For example, in the case where the web page is a shopping site for purchasing "Product A", a content ID indicating "Product A" is associated with this web page. Also, for example, when the web page is a news site that provides the news of "sports A", a content ID indicating "sports A" is associated with this web page. Data indicating the association between the web page and the content may be stored, for example, in various masters stored in the service providing device 5 or an external storage device.

  For example, one line of data is added to the log data L each time the terminal device 3 accesses a web page. The log data L is, for example, text data created at an arbitrary timing such as every day or every week.

[Data management device]
The data management device 7 acquires, for example, log data L of a web page provided by the service providing device 5 or an application page referenced by an application program, and manages the acquired log data L.

  FIG. 3 is a view showing the concept of data management by the data management device 7. The data management device 7 manages data included in the log data L in three stages. The three phases include, for example, a short term storage phase, a long term storage phase, and an indefinite storage phase.

  The short-term storage phase is a phase where the log data L itself is stored. At this short-term storage stage, for example, log data L of the past one year is stored.

  The long-term storage step is a step of storing a sampling log SL obtained by performing predetermined sampling processing on the log data L. In this long-term storage phase, for example, sampling logs SL of the past three years longer than the above-mentioned short-term storage phase are stored. In this sampling log SL, for example, “action log AL” obtained as a result of performing sampling processing on data included in the log data L, focusing on items related to “user”, “content” And “content log CL” obtained as a result of performing sampling processing focusing on items related to

  The indefinite period storage step is a step of storing edited data ED obtained by performing predetermined editing processing on the log data L. In this indefinite storage stage, for example, no storage period is provided, and all past edit data ED is stored. In this editing data ED, for example, “attribute data AD” obtained as a result of performing editing processing on data included in the log data L, focusing on items related to the “user”, “content” And “metadata MD” obtained as a result of performing the editing process paying attention to the item related to.

  FIG. 4 is a block diagram showing an example of a functional configuration of the data management device 7. The data management device 7 includes, for example, an acquisition unit 20, a sampling unit 22, an editing unit 24, an invalidation unit 26, and a storage unit 28. The storage unit 28 includes, for example, a log data storage unit 30, a sampling log storage unit 32, and an editing data storage unit 34. Each functional unit included in the data management device 7 may be distributed to a plurality of devices. For example, the sampling unit 22 and another functional unit may be realized by a separate device. The storage unit 28 may be a storage device such as NAS (Network Attached Storage).

  The acquisition unit 20, the sampling unit 22, the editing unit 24, and the invalidation unit 26 are realized, for example, by a processor such as a central processing unit (CPU) executing a program (software) stored in the storage unit 28. Ru. The program may be downloaded from the application server via the network NW, for example, or may be preinstalled on the data management device 7 in advance. Also, these functional units may be realized by hardware such as LSI (Large Scale Integration), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), etc. Cooperation between software and hardware It may be realized by

  The storage unit 28 may be realized by, for example, a random access memory (RAM), a read only memory (ROM), a hard disk drive (HDD), a flash memory, or a hybrid storage device in which a plurality of these are combined. .

  The acquisition unit 20 acquires log data L from the service providing device 5 via the network NW. The acquisition unit 20 stores the log data L acquired from the service providing device 5 in the log data storage unit 30.

  The sampling unit 22 performs sampling processing on the log data L for which the first period has elapsed among the data included in the log data L acquired by the acquisition unit 20, and generates a sampling log SL. The sampling unit 22 stores the generated sampling log SL in the sampling log storage unit 32. For example, when the log data L is daily text data created on a daily basis by the service providing device 5, the sampling unit 22 performs sampling processing on text data one year or more after creation.

(Sampling process)
The sampling unit 22 performs data sampling (data thinning) on the log data L for which the first period has elapsed. The sampling process includes, for example, a first sampling process performed focusing on a first item related to the “user”, and a second sampling process performed based on a second item related to the “content”. including.

  The first sampling process performed focusing on the first item related to the “user” extracts, for example, log data in which data of the item “user ID 10” related to the “user” exists in the log data L Sampling processing, sampling processing for extracting log data of “user ID 10” of a specific format, and the like.

  The second sampling process performed focusing on the second item related to “content” is, for example, a sampling process for extracting data including “content ID 11” associated with a specific content in the log data L. Including.

  Further, in the sampling unit 22, a ratio of log data sampled by the sampling process (hereinafter, referred to as "sampling rate") is set. For example, the sampling unit 22 adjusts the amount of the sampling log SL based on the sampling rate. The sampling unit 22 may perform sampling processing for each item in column units in addition to sampling of log data in row units described above.

  In addition, the sampling unit 22 may perform sampling processing on the log data L in stages. For example, the sampling unit 22 may perform the second sampling process after a predetermined period has elapsed after performing the first sampling process on the log data L in which the first period has elapsed. Furthermore, the sampling unit 22 may perform the third sampling process on the data subjected to the second sampling process after a predetermined period has elapsed. Such two or more stages of sampling processing make it possible to set a phased storage period such as short-term storage, medium-term storage, and long-term storage of the sampling log SL.

  The editing unit 24 performs editing processing on the log data L acquired by the acquisition unit 20 to generate edited data ED. The editing unit 24 stores the generated editing data ED in the editing data storage unit 34. For example, when the log data L is daily text data created on a daily basis by the service providing device 5, the editing unit 24 performs editing processing on the text data.

(Editing process)
The above-mentioned editing process is performed, for example, by performing a first editing process focusing on the first item related to the "user" and a second editing process focusing on the second item related to the "content". And.

  The first editing process performed focusing on the first item related to “user” is, for example, in the log data L, meaning-based other items for each item of “user ID 10” related to “user”. Consolidate or replace in the attributeized column (attribute), rearrange and invalidate other items, rearrange log data, etc., and generate edit data ED (attribute data AD) in a predetermined format according to the purpose. Do.

  The second editing process performed focusing on the second item related to “content” is, for example, in the log data L, attribute-based attributes of other items for each “content ID 11” related to “content”. It consolidates or replaces in the converted columns (attributes), rearranges and invalidates other items, rearranges log data, etc., and generates editing data of a predetermined format according to the purpose.

  The invalidation unit 26 invalidates, of the data included in the log data L stored in the log data storage unit 30, the data for which the first period has elapsed. Invalidation of the log data L means, for example, physically or logically deleting the log data L from the log data storage unit 30. Here, the log data L to be invalidated is the log data to be subjected to the sampling process by the sampling unit 22. The first period is set to, for example, "one year". Here, when the log data L is daily text data created by the service providing device 5 on a daily basis, the invalidation unit 26 invalidates text data one year or more after creation.

  Further, the invalidation unit 26 invalidates the sampling log SL of which the second period has elapsed among the sampling SLs stored in the sampling log storage unit 32. The second period is set to be longer than the above-described first period. The second period is set to, for example, "3 years". In this case, the invalidation unit 26 invalidates the sampling log SL that has been generated by the sampling unit 22 and has passed three years or more. As described above, when the sampling unit 22 performs the stepwise sampling process, the invalidation unit 26 invalidates the above-described invalidation for each sampling log S according to the period individually set for the sampling log SL. Processing may be performed.

  The log data storage unit 30 stores the log data L acquired by the acquisition unit 20. The sampling log storage unit 32 stores the sampling log SL generated by the sampling unit 22. The editing data storage unit 34 stores the editing data ED generated by the editing unit 24.

[Process of data management device]
Next, the operation of the data management device 7 will be described with reference to FIGS. 5 to 9. FIG. 5 is a flow chart showing an example of the flow of processing by the data management device 7. The process shown in the flowchart of FIG. 5 is started at a predetermined timing such as daily, weekly or monthly. Alternatively, this process may be started in response to an instruction from the user of the data management device 7.

  First, the acquiring unit 20 acquires log data L from the service providing apparatus 5 via the network NW (step S101). The acquisition unit 20 stores the acquired log data L in the log data storage unit 30.

  Next, one or both of the sampling process (step S103) by the sampling unit 22 and the editing process (step S105) by the editing unit 24 are performed. The processing to be performed is determined according to a preset schedule or an instruction from the user.

(Sampling process)
The sampling unit 22 reads the log data L for which the first period has elapsed from the log data L acquired by the acquisition unit 20 and stored in the log data storage unit 30, performs sampling processing, and outputs the sampling log SL. It generates (step S103). The sampling unit 22 stores the generated sampling log SL in the sampling log storage unit 32.

  Here, the sampling unit 22 performs the first sampling process focusing on the item of “user ID 10” related to “user” among the items included in the log data L, and performs an action log AL (a sampling log SL) Generate the first sampling log). The sampling unit 22 samples, for example, data in which data of an item of the user ID 10 exists among data included in the log data L.

  FIG. 6 is a view showing an example of a sampling log SL obtained by sampling log data in which data of the item of the user ID 10 exists from the log data L shown in FIG. As shown in FIG. 6, in the sampling log SL, among the data included in the log data L, data in which the data of the item of the user ID 10 in the third and sixth rows do not exist is invalidated.

  Further, the sampling unit 22 performs second sampling processing focusing on the item of “content ID 11” related to “content” among the items of data included in the log data L, and the content log CL is used as a sampling log SL. Generate (second sampling log). The sampling unit 22 samples, for example, data of which the data of the item of the content ID 11 has a specific format among the data included in the log data L.

  FIG. 7 is a view showing an example of a sampling log SL obtained by sampling log data in the range of “00001” to “00099” in the data of the item of the content ID 11 from the log data L shown in FIG. As shown in FIG. 7, in the sampling log SL, data of the item of the content ID 11 among the data included in the log data L is the data of the fifth to seventh rows which are not in the range of “00001” to “00009”. Is disabled.

  In addition, the sampling unit 22 adjusts the amount of data included in the above-described sampling log SL based on a preset sampling rate. For example, when “30%” is set as the sampling rate, adjustment is performed so that 30% of the data included in the sampling log SL remains (70% of data is invalidated).

(Editing process)
The editing unit 24 performs editing processing on the log data L acquired by the acquiring unit 20 and stored in the log data storage unit 30, and generates editing data ED (step S105). The editing unit 24 stores the generated editing data ED in the editing data storage unit 34.

  Among the items included in the log data L, the editing unit 24 performs a first editing process paying attention to the item of “user ID 10” related to “user”, and the attribute data AD (first Generate editing data). The editing unit 24 performs first editing processing according to master data or predetermined processing logic, and generates editing data ED. For example, in the log data L, the editing unit 24 counts whether the page of the link destination of the target URL associated with each user ID is for men or women, and counts the tendency of the content to be accessed. The other items are aggregated or replaced into a semantic-based attributeized column (attribute) such as "gender" with the key as a key. The association between the “user ID” and the “attribute” may be performed by referring to data input by the user at the time of membership registration of the web page.

  FIG. 8 is a view showing an example of edit data ED (action history data) obtained by performing the first editing process on the log data L shown in FIG. In FIG. 8, data in which "attribute 1 (gender)" is associated with "male" and "attribute 2 (hobby)" is associated with "sport" is shown for data in which "user ID" is "bbbbb". ing. Thus, it is possible to compress the data volume by performing the editing process focusing on the item of “user ID 10” related to “user”. Further, such editing data ED has a format suitable for analysis processing in which statistical data of the user's behavior can be easily grasped. Note that the editing unit 24 may add various attribute information such as age and access frequency to the editing data ED in addition to or instead of the above “attribute 1 (sex)” and “attribute 2 (hobby)”. .

  Further, the editing unit 24 performs the second editing process paying attention to the item “content ID 11” related to “content” among the items included in the log data L, and generates the editing data ED as the metadata MD Generate 2) edit data. The editing unit 24 performs second editing processing according to master data or predetermined processing logic, and generates editing data ED. For example, with the content ID included in the log data L as a key, the editing unit 24 aggregates or replaces other items into a semantic-based attributeized column (attribute) such as "goods, services". Also, the editing unit 24 counts the tendency of the user who has accessed each content ID, for example, to determine the attribute of the “target user”.

  FIG. 9 is a view showing an example of edited data ED (access history data) obtained by performing the second editing process on the log data L shown in FIG. In FIG. 9, data in which “sport” for “attribute 1 (goods and services)” and “male” for “attribute 2 (target user)” are associated with data for which “content ID” is “00001”. It is shown. Thus, it is possible to compress the data volume by performing the editing process focusing on the item of "content ID 11" related to "content". Further, such edit data ED has a format suitable for analysis processing in which it is easy to grasp the user's access state to the web page associated with the content ID. Note that the editing unit 24 adds various attribute information such as the number of accesses to the editing data ED in addition to or instead of the above “attribute 1 (goods and services)” and “attribute 2 (target user)”. Good.

  When the information related to the image displayed on each web page is stored in the log data storage unit 30, the editing unit 24 determines the feature amount such as the color of the image, the type of the image (the wind image landscape The edit data ED (metadata MD) may be generated on the basis of a human image or the like.

  Next, the invalidation unit 26 invalidates, of the log data L stored in the log data storage unit 30, the log data L for which the first period has elapsed, and the sampling SL stored in the sampling log storage unit 32. Among them, the sampling log SL for which the second period has elapsed is invalidated (step S107). Thus, the process of this flowchart is ended.

  According to the embodiment described above, it is possible to manage log data in a format suitable for analysis processing while suppressing the capacity. Since the capacity of the sampling log SL is smaller than the log data L, it is possible to reduce the capacity of data stored in the storage unit while leaving necessary information. The sampling log SL can be used when generating a desired model using the log data L. Further, the editing data ED has a format suitable for analysis, and it is also possible to reduce the processing time as well as reduce the processing load on the analysis. Further, since the edited data ED is not invalidated, it is possible to keep necessary information while reducing the capacity.

  In the above embodiment, each of the acquisition unit 20, the sampling unit 22, and the editing unit 24 is a storage unit in which the log data L, the sampling log SL, and the editing data ED are provided in the data management device 7. The configuration to be stored in 28 has been described. However, each of the acquisition unit 20, the sampling unit 22, and the editing unit 24 stores the log data L, the sampling log SL, and the editing data ED in an inexpensive area (for example, a foreign country, a region, etc.) It may be transmitted to the device. Further, each of the acquisition unit 20, the sampling unit 22, and the editing unit 24 may output the log data L, the sampling log SL, and the editing data ED to an external storage medium such as a magnetic tape.

  Moreover, in the above-mentioned embodiment, the structure which the sampling part 22 performs a sampling process based on the predetermined | prescribed sampling rate was demonstrated. However, the sampling unit 22 may check the free space of the storage unit 28 and dynamically change the sampling rate according to the free space.

  Further, in the above-described embodiment, the sampling unit 22 performs the sampling process based on the predetermined “first period”, and the invalidation unit 26 performs the predetermined “first period” and the “second period”. The configuration for performing the invalidation processing based on the above has been described. However, the sampling unit 22 or the invalidation unit 26 checks the free space of the storage unit 28 to dynamically change the “first period” and the “second period” according to the free space. It is also good.

  As mentioned above, although the form for carrying out the present invention was explained using an embodiment, the present invention is not limited at all by such an embodiment, and various modification and substitution within the range which does not deviate from the gist of the present invention Can be added.

  1. data management system 3. terminal device 5. service providing device 7. data management device 20 acquisition unit 22 sampling unit 24 editing unit 26 invalidation unit 28 storage unit 30 .. log data storage unit, 32 .. sampling log storage unit, 34 .. edit data storage unit, NW .. network

Claims (9)

An acquisition unit that acquires log data acquired according to access by the terminal device;
The first sampling process focusing on the first item included in the log data is performed on the data included in the log data acquired by the acquisition unit and for which the first period has elapsed, by performing first sampling processing A sampling unit that generates a sampling log and generates a second sampling log by performing a second sampling process focusing on a second item included in the log data while generating a sampling log of 1;
Among the data included in the log data acquired by the acquisition unit, data in which the first period has elapsed is invalidated and the first sampling log and the second sampling log generated by the sampling unit A data management apparatus, comprising: an invalidation unit that invalidates the first sampling log and the second sampling log of which the second period longer than the first period has elapsed. A first editing process focusing on the first item is performed on data included in the log data acquired by the acquisition unit to generate a first editing data, and an attention is paid to the second item. Further comprising an editing unit which performs the second editing process described above to generate second editing data;
The data management device according to claim 1. The first item is an item identifying a user who uses the terminal device,
The second item is an item that identifies content associated with an electronic page accessed by the terminal device.
The data management device according to claim 1. The invalidation unit does not invalidate the first editing data and the second editing data generated by the editing unit.
The data management device according to claim 2. The sampling unit performs the first sampling process and the second sampling process based on a predetermined sampling rate.
The data management device according to any one of claims 1 to 4. Each of the acquisition unit, the sampling unit, and the editing unit transmits the log data, the sampling log, and the editing data to a storage device arranged in an inexpensive area of an electricity bill.
The data management device according to claim 2. An acquisition unit that acquires log data acquired according to access by the terminal device;
The first editing process focusing on the first item included in the log data is performed on the data included in the log data acquired by the acquisition unit and for which the first period has elapsed, by performing a first editing process A data management apparatus comprising: an editing unit that generates first editing data and performs second editing processing focusing on a second item included in the log data to generate second editing data. The computer is
Acquire log data acquired according to access by a terminal device,
A first sampling process focusing on a first item included in the log data is performed on data in which a first period has elapsed among data included in the acquired log data, and a first sampling is performed Generating a second sampling log by performing a second sampling process focusing on a second item included in the log data while generating a log;
Of the data included in the acquired log data, data in which the first period has elapsed is invalidated, and the first period of the generated first sampling log and second sampling log is invalidated. Invalidating the first sampling log and the second sampling log after a second period longer than the second period has elapsed,
Data management method. On the computer
Have the log data acquired in response to the access by the terminal device be acquired,
A first sampling process focusing on a first item included in the log data is performed on data in which a first period has elapsed among data included in the acquired log data, and a first sampling is performed Generating a second sampling log by performing a second sampling process focusing on a second item included in the access log while generating a log;
Among the data included in the acquired log data log data, data in which the first period has elapsed is invalidated, and the first sampling log and the second sampling log that are generated are generated. Invalidating the first sampling log and the second sampling log after the second period longer than the second period has elapsed,
program.

Images