JP2013235515A - Data distribution management system, apparatus, method and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To enable shortening of an analysis time and analysis of continuous data on a structured overlay.SOLUTION: A system having a plurality of nodes on a structured overlay network performs control so that a management module of a structured overlay makes loads of data that the nodes have uniform between nodes; performs structuring so that connection between nodes becomes a balance tree in order to analyze the uniformed data; accumulates data transmitted to the overlay network in a management module of MapReduce processing; manages an analysis scheme for analyzing the data accumulated in data accumulation means; transmits the analysis result of the analysis scheme to other nodes; and combines the analysis results.

Description

  The present invention relates to a data distribution management system, apparatus, method, and program, and in particular, on a structured overlay network that distributes and manages a plurality of metadata (action history, RFID (Radio Frequency Identification), sensor information). The present invention relates to a data distribution management system, apparatus, method, and program for maximizing parallel management of continuous data such as series data and parallel analysis of the continuous data.

  In recent years, by analyzing vast web information and sensor information called big data, we can personalize search advertisements from web browsing history, and predict traffic congestion by collecting and analyzing automobile sensor information. (For example, see Non-Patent Document 1). However, big data is so huge that it is impossible to analyze it with a single computer. Therefore, in order to efficiently analyze big data, distributed processing is indispensable, and many distributed processing models have been proposed (for example, see Non-Patent Documents 2 and 3). Among them, the concept called MapReduce has been widely applied to big data analysis because it is highly scalable and has a flexible definition of schemes for distributed processing (for example, Non-Patent Documents 4 and 5). MapReduce is broadly divided into Map, Shuffle, and Reduce processes, and the operations in each process are independent of each other, so analysis can be performed in parallel. In map processing, big data is divided into certain calculation units, shuffle processing integrates the divided data into arbitrary nodes (computer resources), and reduce processing performs statistical processing of aggregated data.

  However, there are two major problems with the existing information analysis system using MapReduce. One is that the continuity of the divided data is lost by performing the Shuffle processing using the hash value of the data divided by the Map processing. In MapReduce, considering the possibility that data with continuity will be concentrated on a specific node, the aim is to eliminate processing bias in subsequent Reduce operations by performing a Shuffle using the hash value of each data There is. For this reason, it is difficult to perform analysis that aggregates continuous data and refers to the context.

  Another problem is that it is difficult to do recursive MapReduce. When MapReduce is performed again on the results of data aggregation / analysis by MapReduce for other nodes, an aggregation node for aggregating the analysis results must be selected. In order to maximize the computational efficiency (parallelism) of recursive MapReduce, all nodes must maintain a balanced tree aggregation structure. However, by providing an aggregated structure of balanced trees, the load for performing recursive analysis can be made uniform among nodes, but there is a problem that the maintenance cost of the balanced tree becomes very high. Specifically, when building a balanced tree, there is a problem that the tree must be rebuilt many times in order to balance the left and right sides of the tree.

  In the following, among the existing information analysis systems using MapReduce, "Jubatus" (registered trademark) that can perform MapReduce recursively and a structured overlay are used to achieve an information analysis system with excellent scalability. The reason why each system cannot satisfy recursive analysis and continuous data management at the same time is described (for example, see Non-Patent Documents 7 and 8).

  “Jubatus” (registered trademark) is an information analysis system using MapReduce provided as Open Source. The architecture of “Jubatus” (registered trademark) is shown in FIG. "Jubatus" (registered trademark) provides a Map function called "JubaKeeper" and a Shuffle function to clients that perform the Map function. When a client sends a job such as morphological analysis to "JubaKeeper", "JubaKeeper" performs Map processing and Shuffle processing, and transfers data to "JubaKeeper" in charge of Reduce. In “Jubatus” (registered trademark), since the analysis scheme in “JubaClassifier” can be described by the client, recursive analysis can be performed by aggregating Reduce results (mix processing) between “JubaClassifier”. When performing MIX processing, the resource of the node subject to MIX processing is locked through “ZooKeeper” which manages consistency and maintenance in all nodes in a unified manner, and then data MIX processing is performed. . Note that data sent to the node during the lock is discarded. Since management of the aggregation node in “Jubatus” (registered trademark) is left to “ZooKeeper”, there is a problem that “ZooKeeper” can be a single point of failure, and the scalability of the scale is low. Although continuous data can be handled as analysis data, the selection of an aggregation node when performing MIX processing on continuous data depends on the analysis scheme of the client. Unless the state of each node and the distribution of the entire key are known, it is difficult to handle the aggregate structure as a balanced tree, and the parallelism of analysis cannot always be maximized.

  In contrast to “Jubatus” (registered trademark), “ddd” improves the scale extensibility of MapReduce by managing all nodes with a structured overlay. The architecture of “ddd” is shown in FIG. In “ddd”, shuffle processing is performed on the hashed data, and the data is arranged at a node on the structured overlay based on the consistent hashing method, so that continuous data cannot be handled (see, for example, Non-Patent Document 9). However, it is possible to handle continuous data with "ddd". When handling continuous data, there is a problem that the load is biased depending on the data. In order to solve this problem, “ddd” solves the load imbalance between nodes using a virtual node. In load balancing using virtual nodes, each node on the structured overlay generates a large number of its own virtual nodes and fills the ID space on the structured overlay with virtual nodes, thereby reducing the load on each node. Make uniform. When load distribution by virtual nodes is used, even if the input data is continuous data, the load can be made uniform on the structured overlay. However, since the number of virtual nodes needs to be determined in advance, the load is not necessarily averaged depending on the distribution of continuous data. Furthermore, when performing recursive Reduce processing, the user must describe Reduce recursive processing, and it is difficult to construct a balanced tree structure for the same reason as “Jubatus” (registered trademark).

S4 Project, http://incubator.apache.org/s4/ Leslie G Valiant, "A Bridging Model for Parallel Computation," Communications of the ACM, Volume 33 Issue 8, Aug. 1990. http://www.cs.berkeley.edu/matei/spark/ Hadoop, http://lucene.apache.org/hadoop/, 2007. J. Dean and S. Ghemawat, "MapReduce: Simplified Data Processing on Large Clusters," in OSDI 2004: 6th Symposium on Operating Systems Design and Implementation, 2004. P. William, "Skip Lists: A Probabilistic Alternative to Balanced Trees," Communications of the ACM, 1990. Jubatus, http://jubat.us/ ddd, http://www.iij.ad.jp/development/tech-activities/detail/ddd.html D. Karger, et al, "Consistent Hashing and Random Trees: Distributed Caching Protocols for Relieving Hot Spots on the World Wide Web," in Proc. ACM Symposium on Theory of Computing, 1997.

  As described above, basic studies on scale extensibility and recursive Reduce processing in MapReduce have been performed, but no optimization method for these processing is mentioned. In other words, it is considered difficult to build an information analysis system that is rich in scale and parallelism and that can achieve efficient operation management even for continuous data.

  The present invention has been made in view of the above points, and an object thereof is to provide a data distribution management system, apparatus, method, and program capable of reducing analysis time and analyzing continuous data on a structured overlay. To do.

In order to solve the above problems, the present invention (Claim 1) is a distributed data management system in a structured overlay network that distributes and manages a plurality of metadata.
Having multiple nodes on a structured overlay network;
The node is
Having structured overlay management means and MapReduce processing management means,
The structured overlay management means includes:
Load balancing means for controlling the data held by the plurality of nodes so that the load is uniform among the nodes;
Structuring means for structuring the connections between the nodes to be balanced trees in order to analyze the data that has been made uniform by the load balancing means,
The MapReduce process management means
Data storage means for storing data transmitted to the overlay network;
Analysis management means for managing an analysis scheme for analyzing data stored in the data storage means;
Analysis result integration means for transmitting an analysis result according to the analysis scheme to another node and combining the analysis results.

The present invention (Claim 2) provides the MapReduce process management means,
Maintaining the tree structure of the balanced tree for the connection relationship between nodes, Map processing for dividing a plurality of metadata into data of calculation units according to the tree structure of the balanced tree, Shuffle processing for transferring the divided data to the node in charge, It includes MapReduce processing means for performing Reduce processing for collecting and processing the analysis results of each node device.

Further, the present invention (Claim 3) provides the load balancing means,
When the own node is a root node, a node having a low load or a node having a high load is determined by a predetermined discriminant based on the load information of each node aggregated according to the tree structure of the balanced tree, and the load is high. A means for selecting a hash ID (HashID) that makes the load uniform among the nodes from the ID space managed by the node, and adding the selected HashID to a node having a low load;

  As described above, according to the present invention, the data capacity of each node device can be made uniform with respect to continuous data such as time-series data, as well as being highly scalable and efficient. An information analysis system that can achieve operational management can be realized.

It is an architecture of “Jubatus” (registered trademark). "ddd" architecture. It is an operation | movement outline | summary of MapReduce on the structured overlay using SkipList in one embodiment of this invention. It is a node block diagram in one embodiment of this invention. It is the aggregation structure of the analysis data in one embodiment of this invention. It is a sequence chart of the construction process of the routing table in one embodiment of this invention. It is the outline | summary of the load distribution method in one embodiment of this invention. It is a sequence chart of load distribution in an embodiment of the present invention. It is a search procedure of Shuffle processing in one embodiment of the present invention. It is a figure which shows the distribution and execution procedure of the analysis scheme in one embodiment of this invention. It is a sequence chart of the load equalization process in one embodiment of the present invention. It is a sequence chart of the analysis process in one embodiment of the present invention.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  The present invention is similar to the Publisher / Subscriber system. The Publisher / Subscriber system is a system used in a delivery service such as a mail magazine. A client (Subscriber) registers a mail magazine to be subscribed to a server (Publisher) through a subscription registration interface. When the server issues a mail magazine, the mail magazine is issued to the subscriber of the mail magazine using push-type distribution. When the Publisher / Subscriber system is applied to the present invention, data that is constantly transmitted (Publish) from various sensors and monitors is continuously accumulated (Shuffle) on the structured overlay using SkipList. When performing some kind of analysis on the data accumulated by the user, the user obtains the result of performing parallel analysis temporarily and periodically by subscribing the analysis scheme described by the user to the SkipList. Can do.

  FIG. 3 shows an outline of MapReduce operation on a structured overlay using SkipList. A large amount of data arrives at a node operating on SkipList, and the data is shuffled (stored) on SkipList. Further, the accumulated data is recursively reduced according to the balanced tree in SkipList.

  FIG. 4 shows a device configuration necessary to achieve MapReduce using SkipList.

  The MapReduce processing management unit (MapReduce processing management module) 100 is an analysis scheme buffer (Analysis Scheme Buffer) 110 for storing an analysis scheme issued by a user, an analysis result synchronization unit (Analysis Scheme Coordinator) 120, and data for storing continuous data A buffer (Data Buffer) 130 is mounted. Note that an API (Scatter API) provided by the structured overlay management unit 200 is used for propagation of an analysis scheme issued from the MapReduce processing management unit 100.

  In the structured overlay management unit (structured overlay management module) 200, a SkipList manager (SkipListManager) 210 having a function for building and maintaining a SkipList and a load balancing unit (Load Balancing Executor having a function for performing load distribution on the SkipList) ) 220 is implemented. The communication unit 300 includes a data transmission unit (Sender) 310 and a reception unit (Receiver) 320 using the TCP protocol. Note that the communication unit 300 is a module only for transmitting and receiving data using a TCP socket installed in the OS, and thus description thereof is omitted.

  In the following, the aggregation structure using SkipList and its construction procedure are described, and a method for equalizing the load on continuous data is also described. We will also briefly describe how to subscribe analysis schemes.

  First, the structured overlay management unit 200 will be described.

  The structured overlay management unit 200 includes a SkipList manager 210 and a load distribution unit 220.

  The SkipList manager 210 includes a routing buffer 211.

SkipList is a type of balanced tree, and unlike AVL trees and B-trees, SkipList does not perform processing (tree reconstruction) to keep the tree depth constant (see Non-Patent Document 6, for example). Therefore, since a balanced tree can be easily constructed, parallelism of analysis using MapReduce can be easily improved. In order to apply this structure to the aggregate structure between nodes, first, each node has two IDs. One is a hash value, which is used to perform consistent hashing like "ddd". Another ID is a level, and the node aggregation structure is hierarchized according to this level, and the relationship between nodes is made into a tree structure. For example, when a balanced N-ary tree is constructed, the probability that each node exists at level i is (1 / N) ⁱ , and belongs to the levels below i at the same time. Therefore, the level of each node uses a random number RAND (0 ≦ RAND <1),

と計算できる。[]はガウス記号で、[a]はaを超えない最大の整数を意味する。なお以下では[a， b]や[a， b)のような表現が頻出するが、[a， b]はa以上b以下を意味し、[a， b)はa以上bより小さいを意味する。さらに、(a， b]はaより大きくb以下を意味し、(a， b)はaより大きくbより小さいことを意味する。

Can be calculated. [] Is a Gaussian symbol, and [a] means the largest integer that does not exceed a. In the following, expressions such as [a, b] and [a, b) appear frequently, but [a, b] means from a to b, and [a, b) means from a to less than b. To do. Furthermore, (a, b] means greater than a and less than b, and (a, b) means greater than a and less than b.

The hash ID (HashID: p) of the node i at level i is the routing buffer for the node information (HashID and IP Address) as the successor and predecessor in the clockwise and counterclockwise adjacent nodes at each level below the level i. Record in the routing table in 211. In the following, it is assumed that the HashID of the successor at the level k (0 ≦ k ≦ i) of the node p is sp _{, k,} and the HashID of the predecessor at the level k is p _{p, k} . The ID space handled by the node p is (p _{p, 0} , p] when p _{p, 0} <p is satisfied, and the ID assigned by the node p when p <p _{p, 0} is satisfied. The space is ( p _{p, 0} , L), [0, p] (L is the size of the entire ID space, HashID: L is equivalent to HashID: 0, and the ID space is a ring-type structure. In the following, the above ranges are summarized as (p _{p, 0} , p].

  Note that the ID space handled by the node p means a range of HashIDs of data arranged in the Shuffle process. For example, the ID space handled by the node p is 10 to 100. At this time, data with HashID 45 and 76 is held by the node p.

Furthermore, at level k (1 ≦ k ≦ i), the node information of level k−1 satisfying [p, s _{p, k} ) whose HashID satisfies [p, s _{p, k} ) is held as a child node, and its own HashID at level i Node information is held in the routing buffer 211 with a node holding p as a child as a parent. The reason why the ID space handled by the node p and the ID space of the child node are determined in the opposite direction is to prevent the ID space handled by the child node and the ID space handled by the node p from overlapping and redundant. is there. When the number of nodes is N ^q (0 <N, 0 ≦ q), the level is q if LEVEL> q is satisfied. Note that the node at level q is the root node and plays the role of overseeing distributed computation on SkipList. If there are multiple root nodes belonging to level q, the node that has been judged that the successor's HashID at level q is smaller than its own HashID will belong to level q + 1, and will have a node group of level q as its children . These child nodes, parent nodes, and routing tables (information of successor and predecessor nodes) are stored in the routing buffer 211.

  An example of the aggregated structure of analysis data as described above is shown in FIG.

  FIG. 5 shows a SkipList structure composed of 10 nodes, and the number of each node represents a HashID. Node 2 records nodes 1 and 3 at level 0, and level 1 also records nodes 0 and 4 as successors and predecessor nodes in the routing table. Further, node 2 records level 0 node 3 as a child node, and records level 2 node 0 as a parent node. Similar to the node 2, the node 0 also holds the routing table, parent node, and child node information. Note that at level 2, there is no node adjacent to node 0, so that node is the root node.

  Next, a specific method for constructing SkipList will be described. Note that a process in which a node joins a SkipList structure is defined as a “Join process”. The Join process means a procedure for determining the ID space handled by the node p, determining a parent node and a child node, and constructing a routing table in the routing buffer 211.

  FIG. 6 is a sequence chart of a routing table construction process according to an embodiment of the present invention.

First, the SkipList manager 210 (P) of the node P sends a message (Join message) for searching for a node in charge of p, which is its HashID, to any node existing on the SkipList. (Step 101). Upon receiving the message, the SkipList manager 210 (a) of the node a at level a receives (b, s _{b, i} ) (0 ≦ i) for all successor node information recorded in its routing table. It is checked whether p is included in the range of ≦ a) (checked from the top of the routing table) (step 102). If p is included in [b, s _{b, i} ), the message is forwarded to the child node with the HashID that is closest to p and does not exceed p among the child nodes in the range ( Step 103). If not included, the message is transferred to the parent node, and the join processing is entrusted to the parent node (step 104).

  This transfer is performed in the clockwise direction. By repeating this transfer processing between nodes recursively, the message reaches the node in charge of p. When the node that manages p is node m, node m forwards its parent, child node, and routing table information to node p, and at the same time, p becomes its routing table, parent, and child node information. Reflected by transferring to the other. By the above procedure, each node completes the Join process for the SkipList structure, and the SkipList structure is constructed.

  Next, the load distribution unit 220 will be described.

  In order to distribute the load of continuous data on the structured overlay, each node throws a query that randomly walks on the overlay (Sampling Query), and randomly selects a node on the overlay and collects load information on that node. There is a method for estimating the total load (see, for example, AR Bharamde, M. Agrawal, and S. Sechan, “Mercury: Scalable Routing for Range Queries,” in ACM SIGCOMM, 2004.). As a problem in this method, it is necessary to appropriately specify the number of times the Sampling Query is transmitted according to the scale, which may affect the overall estimation accuracy based on the collected load information.

Compared to the existing technology, the load estimation technology of the present invention can accurately estimate the entire load without depending on the scale by using the SkipList aggregation structure. FIG. 7 shows an outline of the load balancing method. Specifically, the load distribution unit 220 of the structured overlay management unit 200 of each node transmits its own load information to the parent node, and the load distribution unit 220 of the parent node creates a load ranking. Ranking sorts node information in descending order of load from the load information transmitted from the child node, and node information up to the upper R (1 ≦ R) and node information up to the lower R (1 ≦ R) The extraction process is recursively transferred to the root node repeatedly. The load distribution unit 220 of the root node selects, from the routing table of the routing buffer 211, a hash ID that makes the load uniform from the ID space managed by the high load node to the low load node with reference to the load information, Instruct the low load node to add a HashID. Furthermore, the node is instructed to send a Join message using the added HashID to the node having a high load, and the load is made uniform. A discriminant for distinguishing between a high load node and a low load node is shown below. The i (1 ≦ i ≦ R) -th load information of the ranking received by the root node is L _i , the average value of the entire loads is AVG, the i-th load value of the ranking is Q _i , and a constant α (0 <α <1), Q _i is

と表すことにする。なお、以下で説明するランキングとは、集約した上位・下位のランキングを一体化した、1から2Rまでの順位を持つランキングと見なす。

It will be expressed as Note that the ranking described below is regarded as a ranking having a rank from 1 to 2R, which is a combination of the aggregated upper and lower rankings.

FIG. 8 shows a sequence chart of the load distribution operation. In the load distribution unit 220 of the structured overlay management unit 200 of the root node, first, the load values Q ₁ and Q _2R of the load information L ₁ ranked _first and the load information L _2R ranked 2R are calculated using Equation (2). look, if satisfying Q ₁ = 1, Q _2R = 0, root node, to a node L ₁ (hereinafter referred to as "node 1"), L _2R node (hereinafter, referred to as "node 2R" ) Instructs the node 2R to make an ID space division request (step 201). As a result, the load distribution unit 220 of the node _2R is informed of the IP address of the node 1 from the root node, and notifies the node 1 of its own load information L _2R (step 202). The node 1 that knows L _2R divides its ID space so that its load L ₁ ′ becomes (L ₁ + L _2R ) / 2 in the load balancer 220 (step 203), and the node 1R The ID space divided and the data group related to the ID space are transmitted to the node 2R, and the node 2R accumulates in the data buffer 130 of the MapReduce process management unit 100 (step 204).

  Note that after the root node notifies the node 1 and the node 2R of the ID space division request, the same processing is performed for the node 2 and the node 2R-1. This process is repeated until the load values of the high load node and the low load node are no longer 1 or 0. If each load value does not become 1 or 0, the root node ends load distribution, and the entire load distribution is completed. Note that this load distribution processing is performed at a certain time interval in the root node, and a command for transmitting load information to the parent node of each node is transmitted from the root node, and the child node from each node It is added to the stabilization message issued to the node and transferred from the upper layer to the lower layer node. Thereby, the number of messages required for the load distribution process can be suppressed.

  Next, Shuffle processing will be described.

  It is assumed that data n (HashID: n) issued from a sensor or monitor has arrived at node p. When performing the Shuffle process on the data n, the node p must transfer the data to the node in charge of the data. The transfer method is the same transfer method as the construction method shown in FIG.

  A specific search procedure will be described with reference to FIG. It is assumed that the ID space is managed by 10 nodes (0 to 9), and each node manages an ID between adjacent nodes (Predecessor) counterclockwise. Further, the ID space is assumed to manage traffic data using time as an ID, node 4 manages data at time 3:45, and node 6 manages data at time 5:54. At this time, consider a procedure of processing in which data at time 8:37 arrives at the node 4 and is transferred (Shuffle) to the node that manages the data.

  First, the SkipList manager 210 (4) of the node 4 refers to its own routing table and checks whether 8:47 is included between 4 and all of its own route information from the top of the routing table. At this time, it can be determined that 8:47 does not belong to any Successor on the routing table. The SkipList manager 210 (4) of the node 4 transfers a LookUp message for searching for a node that manages 8:47 data to the parent node Node 0. Node 0 that has received the LookUp message reads 8:47, the destination of the LookUp message, and, from the top of its routing table, examines the Successor including 8:47, and as a result, can find the node that manages 8:47. Suppose there wasn't. Next, it is checked whether there is a node that manages 8:47 from the parent and child nodes, and it is determined that the node 8 is managing, and the LookUp message is transferred to the node 8. Upon receiving the LookUp message, node 8 can check its own routing table from the top and determine that node 9 is managing 8:47 data. To transmit the IP address of the node 9. The node 4 that knows the IP address of the node 9 transfers the data of 8:47 to the node 9, thereby completing the shuffle process. When the Shuffle processing is completed, the data is automatically stored in the data buffer 130 in the upper layer MapReduce processing management unit 100 from the SkipList manager 210 (9) Manager of the node 9.

  Note that the number of communications when performing shuffle processing on data issued from sensors and monitors is 2 logs (n) per data, where n is the number of nodes, and even if the number of nodes increases, shuffle processing Since the rate of increase in communication costs when performing, the Shuffle with excellent scalability is realized.

  Next, a method for maintaining the SkipList in the SkipList manager 210 will be described.

  In order to maintain the SkipList, the SkipList manager 210 of each node periodically transmits a Stabilization message to each level of Successor, Predecessor, and parent node, and confirms the existence / extraction of the node. If there is no response, it is determined that the node has left, and the node that is a candidate for Successor, Predecessor, or Parent is searched again, and the information of the alternative node is obtained to update the SkipList structure.

  In addition to this procedure, the SkipList structure can be easily maintained by increasing the number of Successors and Predecessors. Specifically, each node has information on Successor of its own Successor and Predecessor of Predecessor. By doing so, even if the successor leaves, the structure of the skip list can be maintained simply by establishing a connection relationship with the successor of the successor instantly.

  Next, the processing of the MapReduce process management unit 100 will be described.

  A method of reflecting analysis contents defined by the user, called an analysis scheme, is described for SkipList. FIG. 10 shows an outline of the distribution and execution procedure of the analysis scheme. The analysis scheme can be input to any node, and the user inputs the analysis scheme with an analysis scheme identifier (AID). In order to reflect the analysis scheme to all nodes, the MapReduce process management unit 100 uses an API provided by the structured overlay management unit 200 called Scatter (AnalysisScheme). When the Scatter API is used to request the structured overlay management unit 200 to distribute the analysis scheme, first, the SkipList manager 210 of the node to which the analysis scheme is input searches for the root node. When the root node is found, the ScatterAnalizer message is transferred to the root node, and the root node transfers the message to its child node. ScatterAnalizer messages are recursively forwarded to each node according to a SkipList tree structure with a user-defined analysis scheme. Each node that receives the ScatterAnalizer message registers the AID of the scheme and the scheme in its own synchronization unit 120 (Scheme Coordinator). The synchronization unit 120 synchronizes the analysis results from the child nodes when saving the registered analysis scheme and executing the analysis scheme.

  An example of analysis is shown below.

  Assume that the daily traffic data is stored for all nodes. It is assumed that the traffic data is time series data, and there is little daytime traffic data, but there is a huge amount of nighttime traffic data. These data are transferred to the node in charge of each data by Shuffle processing, and each node in charge manages the traffic data. It is assumed that the traffic data is composed of the time and the IP address of the communication terminal. It is assumed that an analysis scheme of contents such as specifying IP addresses of communication terminals up to 100th with the largest number of communication is reflected in the nodes on SkipList from this traffic data.

-Load equalization Here, the load balancing of ID spaces of high load nodes and low nodes will be described.

  FIG. 11 is a sequence chart of the load equalization process according to the embodiment of the present invention.

  At regular time intervals, the SkipList manager 210 of the root node structured overlay management unit 200 requests its child nodes to transfer load information (step 301). Thereafter, the child node transfers the request to its child node (step 302). By performing these processes recursively, a load information transfer request arrives at the level 0 node group (step 303). The level 0 node transmits the total amount of traffic data as its own load information to the parent node (step 304). The parent node that has received the load information from each parent node creates a ranking of the load information (step 305) and transfers it again to its own parent node (step 306). By performing this process recursively, the ranking is transmitted to the root node (step 307). The root node that has received the ranking instructs the ID statement in which the high load node is responsible for the low load node to perform the join process in the load statement s section 220 (step 308). For example, if the load of a node with a high load is 100 and the load of a node with a low load is 10, a node with a low load has an ID space from the node with a high load so that the load is 55, which is between 10 and 100. , And join process is performed on the ID space. By repeating this, it is possible to eliminate the gap in the load of continuous data.

・ Analysis process
The operation of the analysis process in the MapReduce process management unit 100 is shown in FIG.

  First, each node creates a ranking of the number of communications for each IP of the communication terminal from the traffic data managed in the data buffer 130 in its own MapReduce processing management unit 100 (step 401), and communication to the top 100 The terminal IP and the communication count of the terminal are transferred to the parent node (step 402). Each parent node waits until rankings are sent from all the child nodes managed by itself (step 403). When the ranking results are complete (steps 404 and 405), ranking up to the 100th place is performed again. It is created from the received data (step 406), and the ranking is transferred to its parent node (step 407). By recursively creating this ranking, the ranking arrives at the root node. After the ranking is reconfigured in the root node (step 408), the analysis result is transferred to the node that requested the analysis registered in the AID (step 409). At this time, the load of the traffic data, which is continuous data, is smoothed by the load distribution unit 220, and each analysis is performed recursively on the structured overlay having a balanced tree structure. The parallelism of the analysis can be maximized.

  Each component of the mapreduce processing management unit 100, structured overlay management unit 200, and communication unit 300 of the node shown in FIG. 4 is constructed as software (program), and the mapreduce processing management module and structured overlay management. Modules and communication modules can be installed on a computer used as a node and executed, or distributed via a network.

  The present invention is not limited to the above-described embodiments, and various modifications and applications can be made within the scope of the claims.

100 MapReduce Processing Management Unit 110 Analysis Scheme Buffer 120 Synchronization Unit (Analysis Scheme Coordinator)
130 Data Buffer 200 Structured Overlay Management Unit 210 Ship Manager 211 Routing Buffer 220 Load Balancing Executor
300 communication unit 310 transmission unit 320 reception unit

Claims (8)

A distributed data management system in a structured overlay network that manages a plurality of metadata in a distributed manner,
Having multiple nodes on a structured overlay network;
The node is
Having structured overlay management means and MapReduce processing management means,
The structured overlay management means includes:
Load balancing means for controlling the data held by the plurality of nodes so that the load is uniform among the nodes;
Structuring means for structuring the connections between the nodes to be balanced trees in order to analyze the data that has been made uniform by the load balancing means,
The MapReduce process management means
Data storage means for storing data transmitted to the overlay network;
Analysis management means for managing an analysis scheme for analyzing data stored in the data storage means;
A data distribution management system comprising: an analysis result integration unit that transmits an analysis result according to the analysis scheme to another node and combines the analysis results. The MapReduce process management means
Maintaining the tree structure of the balanced tree for the connection relationship between nodes, Map processing for dividing a plurality of metadata into data of calculation units according to the tree structure of the balanced tree, Shuffle processing for transferring the divided data to the node in charge, 2. The data distribution management system according to claim 1, further comprising MapReduce processing means for performing Reduce processing for collecting and processing analysis results of each node device. The load balancing means includes
When the own node is a root node, a node having a low load or a node having a high load is determined by a predetermined discriminant based on the load information of each node aggregated according to the tree structure of the balanced tree, and the load is high. The data according to claim 1, further comprising means for selecting a hash ID (HashID) that makes the load uniform among nodes from an ID space managed by the node, and adding the selected HashID to a node having a low load. Distributed management system. A plurality of data distribution management devices (nodes) in a data distribution management system in a structured overlay network that manages a plurality of metadata in a distributed manner,
Having structured overlay management means and MapReduce processing management means,
The structured overlay management means includes:
Load balancing means for controlling the data held by the plurality of nodes so that the load is uniform among the nodes;
Structuring means for structuring the connections between the nodes to be balanced trees in order to analyze the data that has been made uniform by the load balancing means,
The MapReduce process management means
Data storage means for storing data transmitted to the overlay network;
Analysis management means for managing an analysis scheme for analyzing data stored in the data storage means;
A data distribution management device comprising: analysis result integration means for transmitting an analysis result according to the analysis scheme to another node and combining the analysis results. The MapReduce process management means
Maintaining the tree structure of the balanced tree for the connection relationship between nodes, Map processing for dividing a plurality of metadata into data of calculation units according to the tree structure of the balanced tree, Shuffle processing for transferring the divided data to the node in charge, 5. The data distribution management device according to claim 4, further comprising MapReduce processing means for performing a Reduce processing for collecting and processing the analysis results of the respective node devices. The load balancing means includes
When the own node is a root node, a node having a low load or a node having a high load is determined by a predetermined discriminant based on the load information of each node aggregated according to the tree structure of the balanced tree, and the load is high. 5. The distributed data management apparatus according to claim 4, further comprising means for selecting a Hash ID that makes the load uniform among nodes from an ID space managed by the node, and adding the selected Hash ID to a node having a low load. . A data distribution management method in a data distribution management system in a structured overlay network that distributes and manages a plurality of metadata,
In a system having multiple nodes on a structured overlay network,
The structured overlay management means controls the data held by the plurality of nodes so that the load is uniform among the nodes, and in order to analyze the uniformed data, the connection between the nodes becomes a balanced tree. A structured overlay management step that is structured into
The MapReduce processing management unit stores data transmitted to the overlay network, manages an analysis scheme for analyzing the data stored in the data storage unit, and analyzes the analysis result by the analysis scheme to other nodes MapReduce process management step that combines the analysis results sent to
And
In the structured overlay management step, when equalizing the load,
When the own node is a root node, a node having a low load or a node having a high load is determined by a predetermined discriminant based on the load information of each node aggregated according to the tree structure of the balanced tree, and the load is high. A distributed data management method characterized by selecting a HashID that makes the load uniform among nodes from an ID space managed by the node, and adding the selected HashID to a node with a low load. Computer
A data distribution management program for functioning as each unit of the data distribution management apparatus according to claim 4.

Images