JP6319694B2 - Data cache method, node device, and program - Google Patents

Description

  The present invention relates to a data cache method for performing data analysis in a structured overlay system, a node device that implements the data cache method, and a program thereof.

  Structured overlay is a technique for sharing and managing a single logical ID space by a plurality of nodes, and is used in a distributed database such as Key Value Store. When data is arranged in a structured overlay, the name of the data is used as a key, and the data is arranged for a node in charge of the key. When acquiring data from an overlay, the name of the data to be acquired is used as a key, and a node in charge of the key is searched from the overlay to acquire the data. This method is a data storage method used in large-scale data analysis infrastructures such as Hadoop (Non-Patent Document 1) and Jubatus (Non-Patent Document 3). These analysis infrastructures can perform data analysis at high speed and in a distributed manner by performing MapReduce processing (Non-Patent Document 2) on stored data. In the existing method (Non-Patent Document 6), all analysis procedures and results are analyzed in a system called Pig (Non-Patent Document 7) that projects an analysis procedure described by an analyst using an SQL-like language to MapReduce processing. By saving the pair, the calculation of analysis results for a new analysis procedure is accelerated.

K. Shvachko, H .; Kuang, S .; Radia, and R.R. Chansler, “The Hadoop Distributed File System” in Proc. IEEE MSST, 2010. J. et al. Dean and S.M. Ghemawat, “MapReduce: Simply Data Processing on Large Clusters”, in Proc. OSDI 2004: 6th Symposium on Operating Systems Design and Implementation, 2004. Masanori Han, Hiroyuki Makino, Hiroki Kumazaki, “Real-time large-scale data processing infrastructure Jubatus and its use cases”, Management Science 57 (12), pp. 689-694, 2012. L. Avinash and M.M. Prashant, “Cassandra: A Decentralized Structured Storage System” in ACM SIGOPS Operating Systems Review, vol. 44, Issue. 2, pp. 35-40, 2010. A. Thusoo, J. et al. S. Sarma, N .; Jain, Z .; Shao, P.A. Chakka, N .; Zhang, S.H. Antony, H.C. Liu, and R.R. Murthy, “Hive-A Petabyte Scale Datawarehouse Usage Hadoop”, in Proc. ICDE, 2010. I. Elghandour and A.M. Aboolaga, “ReStore: Reusing Results of MapReduce Jobs” in Proc. VLDB Endowment, vol. 5, Issue. 6, pp. 586-597, 2012. O. Christopher et al, “Pig Latin: A Not-So-Foreign Language for Data Processing” in Proc. SIGMOD 2008, pp. 1099-1110, 2008. Yosuke Takei, Kohei Ota, Nei Kato, Yoshiaki Nemoto, “Unauthorized Access Detection and Tracking Method Using Traffic Patterns”, IEICE Transactions B, vol. J84-B, no. 8, pp. 1464-1473, 2001.

  However, in the high-speed technique such as Non-Patent Document 7, since all pairs of analysis procedures and analysis results are stored, the storage amount (cache) of analysis results greatly increases due to the execution of complicated and diverse analysis, and the hard disk There is a problem that the capacity may be depleted.

  Accordingly, an object of the present invention is to provide a data cache method, a node device, and a program that can increase the speed of data analysis and reduce the possibility that a hard disk capacity is exhausted in order to solve the above-described problems.

  The data cache method according to the present invention records the input analysis procedure and the analysis result, extracts a frequently used partial analysis procedure from the recorded analysis procedure every predetermined period, and then extracts the extracted analysis procedure. When an analysis procedure including a partial analysis procedure is input, the analysis result of the partial analysis procedure is saved, and the analysis result stored when a new analysis procedure including the partial analysis procedure is input is reused. It was decided.

Specifically, a data cache method according to the present invention is a data cache method for a tree-structured structured overlay system comprising a plurality of node devices,
An analysis procedure for analyzing time-series data stored in the structured overlay system, input to the node device, and an entire analysis recording step for recording an analysis result analyzed by the analysis procedure;
A partial operation sequence extracting step for extracting a partial common operation sequence from a plurality of the analysis procedures recorded in the entire analysis recording step every predetermined period, and recording the partial operation sequence;
Partial analysis result recording that records the analysis result analyzed in the partial operation sequence as a partial analysis result when a new analysis procedure including the partial operation sequence recorded in the partial operation sequence extraction step is input to the node device Steps,
When a new analysis procedure including the partial operation sequence is further input to the node device after the partial analysis result recording step, an analysis is performed using the partial analysis result recorded in the partial analysis result recording step. Analysis result reuse step to be performed;
It is characterized by performing.

A node device according to the present invention is a node device that constitutes a structured overlay system having a tree structure,
Cache storage,
Data storage to store time series data;
A path control module for searching for specific time series data by following a tree structure on the structured overlay system;
An analysis procedure for analyzing time-series data accumulated by the structured overlay system and an analysis result analyzed by the analysis procedure are recorded in the cache storage;
Extracting a partial operation sequence that is partially common to a plurality of the analysis procedures recorded by the cache storage for each predetermined period, and recording the partial operation sequence in the cache storage;
Recording an analysis result analyzed in the partial operation sequence in the cache storage as a partial analysis result when a new analysis procedure including the partial operation sequence is input; and a new analysis procedure including the partial operation sequence A controller that performs analysis using the partial analysis result recorded by the cache storage,
It is characterized by providing.

  The present invention is an analysis procedure that is frequently used from past analysis procedures when analyzing time-series data such as traffic data and sensor data stored in a structured overlay of a tree structure constructed by a large number of servers. (Partial operation sequence) and the result thereof are recorded, and when a new analysis procedure is input, the analysis time is shortened by reusing the result of the past analysis procedure (partial operation sequence). Further, since the present invention records only the frequently used partial operation sequence and its result, the cache capacity can be reduced.

  Therefore, the present invention can provide a data cache method and a node device that can speed up data analysis and reduce the possibility of the hard disk capacity being exhausted.

  The data cache method according to the present invention further includes a higher transfer step of transferring the partial operation sequence extracted in the partial operation sequence extraction step to an upper node device above the node device.

The data cache method according to the present invention includes:
Inter-node common operation for extracting a partially common inter-node operation sequence from the plurality of partial operation sequences transferred in the upper transfer step from a plurality of the node devices at a certain period in the upper node device. A column extraction step;
A subordinate that transfers the inter-node common operation sequence from the upper node device to the node device in order to cause the node device to record the inter-node common operation sequence extracted in the inter-node common operation sequence extraction step as the partial operation sequence A transfer step;
It is characterized by performing.

  Further, the controller of the node device according to the present invention transfers the extracted partial operation sequence to a higher-level node device that is higher than the own device.

The controller of the node device according to the present invention includes:
The upper node apparatus receives an inter-node common operation sequence that is partially common to a plurality of the partial operation sequences extracted every predetermined period, and stores the inter-node common operation sequence as the partial operation sequence in the cache storage. Is further performed.

  The present invention takes into account the proximity that is characteristic of time-series data (data of close time has a strong correlation), and is an analysis procedure frequently used between servers that store time-series data of close time By sharing (partial operation sequence), the reusability of each analysis procedure (partial operation sequence) is improved.

  The program according to the present invention is a program for causing a computer to execute the data cache method. The node device can be realized by a computer and the program, and the program can be recorded on a recording medium or provided through a network.

  The present invention can provide a data cache method, a node device, and a program that can increase the speed of data analysis and reduce the possibility of the hard disk capacity being exhausted.

It is a figure explaining the structure of the node apparatus which concerns on this invention. It is a figure explaining the example of the structured overlay with a tree structure. It is a figure explaining the data cache method which concerns on this invention. It is a figure explaining the data cache method which concerns on this invention. It is a figure explaining the data cache method which concerns on this invention. It is a figure explaining the data cache method which concerns on this invention.

  Embodiments of the present invention will be described with reference to the accompanying drawings. The embodiments described below are examples of the present invention, and the present invention is not limited to the following embodiments. In the present specification and drawings, the same reference numerals denote the same components.

FIG. 1 is a diagram illustrating the configuration of the node device according to the present embodiment. This node device is a node device constituting a structured overlay system of a tree structure,
Cache storage 30,
A data storage 40 for storing time-series data;
A path control module 10 for searching for specific time-series data by tracing a tree structure on the structured overlay system;
An analysis procedure for analyzing the input time series data stored in the structured overlay system and an analysis result analyzed by the analysis procedure are recorded in the cache storage 30;
Extracting a partial operation sequence that is partially common to the plurality of analysis procedures recorded by the cache storage 30 every predetermined period, and causing the cache storage 30 to record the partial operation sequence;
Recording an analysis result analyzed in the partial operation sequence in the cache storage 30 as a partial analysis result when a new analysis procedure including the partial operation sequence is input; and a new analysis procedure including the partial operation sequence Cache storage controller 20 that performs analysis using the partial analysis result recorded by the cache storage 30 when
It is characterized by providing.

The data cache method performed by the node device is as follows. The data cache method is a data cache method for a structured tree overlay system comprising a plurality of node devices,
An analysis procedure for analyzing time-series data stored in the structured overlay system, input to the node device, and an entire analysis recording step for recording an analysis result analyzed by the analysis procedure;
A partial operation sequence extracting step for extracting a partial common operation sequence from a plurality of the analysis procedures recorded in the entire analysis recording step every predetermined period, and recording the partial operation sequence;
Partial analysis result recording that records the analysis result analyzed in the partial operation sequence as a partial analysis result when a new analysis procedure including the partial operation sequence recorded in the partial operation sequence extraction step is input to the node device Steps,
When a new analysis procedure including the partial operation sequence is further input to the node device after the partial analysis result recording step, an analysis is performed using the partial analysis result recorded in the partial analysis result recording step. Analysis result reuse step to be performed;
It is characterized by performing.

The routing control module 10 is a routing control module in a structured overlay having a tree structure, and has a function of searching for data having a specific ID on the structured overlay by tracing the parent-child relationship of the tree structure. ing. The node at the end of the tree has an ID, and the ID of node i (0 ≦ i <ID_Space_Size) is X _i . The ID space is connected by 0 and ID_Space_Size and has a ring structure. At this time, the node clockwise from X _i is called the successor (ID: suc _i ) of node i, and the area in charge of node i is [X _i , suc _i ). [A, B] means A or more and less than B, and (A, B] means larger than A and less than or equal to B.

  The node corresponding to the node of the tree records the area in charge of the node existing under the node. Examples of these tree-structured overlays are shown in FIG. In FIG. 2, devices indicated by reference signs A to G are node devices. In this embodiment, since it is assumed that the time series data is managed by the structured overlay, the size of the ID space is [0, 86400), and the time series data is stored in the data storage 40 using time (seconds) as a key. Suppose that it is preserved.

  The cache storage 30 is a storage for storing analysis procedures and analysis results for the data in its own area, and the cache storage controller 20 determines whether or not to store the analysis procedures and analysis results in the cache storage 30. Since the proposed method is achieved by the cache storage 30 and the cache storage controller 20, these details will be described below.

  The cache storage controller 20 includes a partial operation sequence extraction unit 21, an analysis procedure execution unit 22, and an information sharing unit 23. Here, it is considered that the analysis procedure {ABCD} is performed on data in a time range of (p, q] p ≠ q. Here, A and B are “a certain feature”. "X, Y, Z" indicates that the operation is performed in the order of X, Y, Z. This analysis procedure includes (p, q]. This is executed by the cache storage controller 20 of the node having the area in charge.

といった形（キャッシュ）で、キャッシュストレージ３０に対して解析結果を保存する（Ｓ０３）。ステップＳ０１、Ｓ０２及びＳ０３は前記全解析記録ステップである。 1. Analysis Procedure Execution Unit FIG. 3 is a diagram for explaining the operation of the analysis procedure execution unit 22.
First, when the analysis procedure {ABCD} is input, the analysis procedure execution unit 22 determines whether the same analysis procedure has been input in the past (S01). If the input analysis procedure has not been performed in the past, data corresponding to the analysis procedure is read from the data storage 40 (S02), and the result of the analysis is displayed.

In this form (cache), the analysis result is stored in the cache storage 30 (S03). Steps S01, S02 and S03 are the all analysis recording steps.

  Thereafter, the analysis procedure execution unit 22 confirms with the cache storage 30 about the analysis procedure input to the node device (S04). Then, when the analysis procedure is stored in the cache storage 30, the analysis procedure execution unit 22 reads the cache (S05), and thereby the analysis procedure overlaps with the past (data analysis is performed and an analysis result is obtained). ) Is omitted.

  Further, when an analysis procedure including a partial operation sequence that is determined to be high in frequency by the partial operation sequence extraction unit 21 to be described later is input, the analysis procedure execution unit 22 analyzes the partial operation sequence (partial analysis result). ) Is written to the cache storage 30.

2. Partial Operation Sequence Extraction Unit FIG. 4 is a diagram for explaining the operation of the partial operation sequence extraction unit 21.
If the analysis procedure and the analysis result are continuously stored, the capacity of the hard disk may be exhausted by the cache. For this reason, in this embodiment, the partial operation sequence extraction unit 21 extracts, as a partial operation sequence, a procedure that is frequently analyzed from the above analysis procedure and analysis results every predetermined period (S06), and writes it to the cache storage 30. (S07). Steps S06 and S07 are the partial operation sequence extraction steps.

Specifically, partial procedures appearing k (> 0) times or more (procedure length is 1 or more) are collectively extracted. The procedure length indicates the number of data manipulations existing in the analysis procedure, and the procedure length of the analysis procedure of {ABC} is 3. FIG. 5 is a diagram illustrating a partial operation sequence extraction method performed by the partial operation sequence extraction unit 21.
Step 1:
From all analysis procedures, create a trie tree and summarize sharing operations. Also, how many times each operation has been performed is stored.
Step 2:
A partial operation sequence that appears k times or more and has a procedure length of 1 or more is acquired from the created trie tree.
In the example of FIG. 5, the partial operation sequences {AB} and {CA} could be extracted.

  The partial operation sequence extraction unit 21 stores the extracted partial operation sequence in the cache storage 30, and then deletes the analysis procedure and the analysis result stored in the cache storage 30 by the analysis procedure execution unit 22 within the predetermined period. To do.

  The analysis procedure execution unit 22 checks the cache storage 30 for the analysis procedure when a new analysis procedure is input (S08), and includes a partial operation sequence with a high execution frequency from which the analysis procedure is extracted (S09). ), The partial analysis result of the partial operation sequence is stored in the cache storage 30 (S10). Steps S08 to S10 are the partial analysis result recording steps.

というキャッシュをキャッシュストレージ３０に保存しておく。 For example, when an analysis procedure for a certain range (r, s) is input, the analysis procedure execution unit 22 stores {A-B} and {C-A} from the top in the analysis procedure in the cache storage 30. The analysis result (partial analysis result) is stored, for example, when the analysis procedure C-A-D is input, the analysis procedure execution unit 22

Is stored in the cache storage 30.

  By this data cache method, it is possible to reuse frequently used analysis results and achieve high speed analysis results (analysis result reuse step). Also, instead of saving all analysis procedures and their results, saving a partial sequence of frequently used analysis procedures can improve the cache reuse rate and deplete hard disk space. The possibility can be reduced.

3. Information Sharing Unit FIGS. 4 and 6 are diagrams for explaining the operation of the information sharing unit 23.
The cache storage controller 20 transfers the extracted partial operation sequence to an upper node device that is higher than the own device, and the upper node device is partially common to the plurality of partial operation sequences extracted at regular intervals. The inter-node common operation sequence is received, and the inter-node common operation sequence is stored in the cache storage as the partial operation sequence.

The data cache method performed by the node device is as follows. This data caching method is
An upper transfer step of transferring the partial operation sequence extracted in the partial operation sequence extraction step to an upper node device above the node device;
Inter-node common operation for extracting a partially common inter-node operation sequence from the plurality of partial operation sequences transferred in the upper transfer step from a plurality of the node devices at a certain period in the upper node device. A column extraction step;
A subordinate that transfers the inter-node common operation sequence from the upper node device to the node device in order to cause the node device to record the inter-node common operation sequence extracted in the inter-node common operation sequence extraction step as the partial operation sequence A transfer step;
Do further.

  Time series data is generally considered to have a specific correlation between close times (Non-Patent Document 8), and it is considered that similar analysis is performed on similar data. For this reason, the information sharing unit 23 shares a partial operation sequence frequently examined by each node device with other node devices.

4 and 6 show examples of sharing methods.
The partial operation sequence extracting unit 21 delivers the extracted partial operation sequence having a high frequency to the information sharing unit 23 (S11). The information sharing unit 23 of each node device transmits the partial operation sequence to the information sharing unit 23 of the upper node device (parent node) that is higher than the own device (S12). Steps S11 and S12 are the upper transfer steps.

  The parent node stores the partial operation sequence in the cache storage 30 of the parent node (S13). Then, the partial operation sequence extraction unit 21 of the parent node reads the partial operation sequence every fixed period, and finds a partial operation sequence having a high frequency among them (S14, S15). Steps S14 and S15 are the inter-node common operation sequence extraction step. For example, the partial operation sequence extraction unit 21 searches for a partial operation sequence that satisfies the conditions of l and k of the parent node itself.

  Then, the partial operation sequence extraction unit 21 of the parent node passes the found partial operation sequence to the information sharing unit 23 of the parent node (S16), and the information sharing unit 23 shares the partial operation sequence among subordinate child nodes ( S17). The child node information sharing unit 23 also stores the partial operation sequence received from the parent node in the cache storage 30. Steps S16 and S17 are the lower transfer steps.

をキャッシュストレージ３０に保存しておき、当該部分操作列を含む解析手順が投入された場合は、当該解析結果を利用する。 The child node holds the partial operation sequence extracted by itself and the partial operation sequence transferred from the parent node. When an analysis procedure for a certain range (v, w) is input, {A -B}, {CA}, and {KA}, the partial analysis result is stored in the cache storage 30. For example, when the analysis procedure {CAB} is input,

Is stored in the cache storage 30, and when an analysis procedure including the partial operation sequence is input, the analysis result is used.

  At this time, for example, when an analysis procedure that includes {KA} but does not match from the beginning is input as in the analysis procedure {CKA}, the analysis procedure execution unit 22 performs the partial operation. The partial analysis result of the column {KA} is not saved.

  The partial operation sequence extracted by each parent node may be shared by a higher-order parent node.

Note that the entire analysis recording step of this data cache method corresponds to S01 to S03 in FIG. 3, the partial operation sequence extraction step corresponds to S06 to S07 in FIG. 4, and the partial analysis result recording step corresponds to S08 to S08 in FIG. This corresponds to S10.
Also, the upper transfer step corresponds to S11 to S12 in FIG. 4, the inter-node common operation sequence extraction step corresponds to S14 to S15 in FIG. 4, and the lower transfer step corresponds to S16 to S17 in FIG.

4). Program This program is a program for causing a computer to execute the data cache method. The data cache method can be realized by causing a plurality of computers connected via a network or the like to execute the program.

(effect)
The present invention is an analysis procedure that is frequently used from past analysis procedures when analyzing time-series data such as traffic data and sensor data stored in a structured overlay of a tree structure constructed by a large number of servers. And a result thereof, and when a new analysis procedure is input, the present invention relates to a technique for shortening the analysis time by reusing the result of a similar analysis procedure in the past. In addition, considering the proximity that is characteristic of time series data (short time data has a strong correlation), frequently used analysis procedures are shared between servers that store time series data of near time This improves the reusability of each other's analysis procedures.

10: Path control module 20: Cache storage controller 21: Partial operation sequence extraction unit 22: Analysis procedure execution unit 23: Information sharing unit 30: Cache storage 40: Data storage

Claims (7)

A data cache method for a tree structured overlay system comprising a plurality of node devices, comprising:
An analysis procedure for analyzing time-series data stored in the structured overlay system, input to the node device, and an entire analysis recording step for recording an analysis result analyzed by the analysis procedure;
A partial operation sequence extracting step for extracting a partial common operation sequence from a plurality of the analysis procedures recorded in the entire analysis recording step every predetermined period, and recording the partial operation sequence;
Partial analysis result recording that records the analysis result analyzed in the partial operation sequence as a partial analysis result when a new analysis procedure including the partial operation sequence recorded in the partial operation sequence extraction step is input to the node device Steps,
When a new analysis procedure including the partial operation sequence is further input to the node device after the partial analysis result recording step, an analysis is performed using the partial analysis result recorded in the partial analysis result recording step. Analysis result reuse step to be performed;
And a data cache method. An upper transfer step of transferring the partial operation sequence extracted in the partial operation sequence extraction step to an upper node device above the node device;
The data cache method according to claim 1, further comprising: Inter-node common operation for extracting a partially common inter-node operation sequence from the plurality of partial operation sequences transferred in the upper transfer step from a plurality of the node devices at a certain period in the upper node device. A column extraction step;
A subordinate that transfers the inter-node common operation sequence from the upper node device to the node device in order to cause the node device to record the inter-node common operation sequence extracted in the inter-node common operation sequence extraction step as the partial operation sequence A transfer step;
The data cache method according to claim 2, wherein: A node device constituting a structured overlay system having a tree structure,
Cache storage,
Data storage to store time series data;
A path control module for searching for specific time series data by following a tree structure on the structured overlay system;
An analysis procedure for analyzing time-series data accumulated by the structured overlay system and an analysis result analyzed by the analysis procedure are recorded in the cache storage;
Extracting a partial operation sequence that is partially common to a plurality of the analysis procedures recorded by the cache storage for each predetermined period, and recording the partial operation sequence in the cache storage;
Recording an analysis result analyzed in the partial operation sequence in the cache storage as a partial analysis result when a new analysis procedure including the partial operation sequence is input; and a new analysis procedure including the partial operation sequence A controller that performs analysis using the partial analysis result recorded by the cache storage,
A node device comprising:   The node device according to claim 4, wherein the controller transfers the extracted partial operation sequence to a higher-level node device that is higher than the own device. The controller is
The upper node apparatus receives an inter-node common operation sequence that is partially common to a plurality of the partial operation sequences extracted every predetermined period, and stores the inter-node common operation sequence as the partial operation sequence in the cache storage. The node device according to claim 5, further comprising:   The program for making a computer perform the data cache method in any one of Claim 1 to 3.

Images