KR102054068B1 - Partitioning method and partitioning device for real-time distributed storage of graph stream - Google Patents

Abstract

In a graph composed of one master node and several slave nodes, a distributed storage management technique of graph streams for improving query processing performance of graphs in a situation where a large graph is divided based on vertex truncation is disclosed.
The partitioning method for real-time distributed storage for the graph stream is divided into the partitioning method for real-time distributed storage for the graph stream. Determining a selected node having the highest score, among the plurality of nodes, and connecting a vertex associated with the new graph stream by a trunk to a lower graph belonging to the selected node, at the selected node. And processing the new graph stream.

Description

PARTITIONING METHOD AND PARTITIONING DEVICE FOR REAL-TIME DISTRIBUTED STORAGE OF GRAPH STREAM}

The present invention relates to a distributed storage management technique of a graph stream for improving query processing performance for graphs in a situation where a large graph is divided based on vertex truncation in a graph composed of one master node and several slave nodes. will be.

The background technology of the present invention is disclosed in the following documents.
1) Registration No.: 10-1285078 (2013-07-05) "Discrete MapReduce based Distributed Parallel Processing System and Method for Stream Data"
2) Registration No.: 10-1678149 (2016-11-15) "Data search method of database and its device and computer program for it"
Recently, as services such as social networks, the semantic web, the Internet of things, etc. are actively used, a large number of graph data are used to express relationships or interactions between users and things. In addition, the generalization of mobile devices is increasing the amount of graph data generated by the rapid proliferation of SNS such as Twitter and Facebook.

In order to efficiently store and manage a large amount of graphs, development of a technique for dividing a large graph into smaller graphs is required.

Conventional Metis or PowerGraph are representative graph segmentation techniques. Metis proposes a method of minimizing the number of cutting edges connecting several divided subgraphs in order to minimize the communication cost between subgraphs using the edge cutting method. On the other hand, PowerGraph proposes a method of eliminating edges connected between each subgraph by performing vertex splitting and a programming model for this.

Changes in graphs, such as the addition / deletion of edges representing relationships between objects in graph data or the addition / deletion of vertices representing objects, occur irregularly, and such changes may be referred to as graph streams.

The existing segmentation technique for static graphs requires a management policy for dynamic graphs, that is, graph streams, in which the shape of the graph changes, for irregular graph changes.

In order to manage the graph stream, it is necessary to consider a criterion for selecting a location to store additional data in a situation where a large static graph is statically stored and a problem that may occur due to deletion of existing data.

To set the criteria for storing the graph stream, it is necessary to consider the state of the nodes constituting the current cluster. In addition, in setting the criterion, it is necessary to consider the rate of replication of vertices caused by the placement of new data, assuming that the segmentation is based on vertex truncation. In addition, the setting of the criterion should consider the minimization of the cutting edge if the division of the cutting edge is made.

When the throughput of a particular node is high or the data memory space is insufficient, the performance of the entire system may be degraded. Therefore, the variation in throughput or the appropriate utilization of memory should always be appropriately related.

Therefore, there is a demand for a method of managing the state information of the entire cluster and using the information whenever a data is generated in real time to find and solve a problem caused by calculating or deleting a new data partitioning criterion.

Existing inventions for graph stream segmentation include vertex truncation and edge truncation according to the criteria of segmentation, and various considerations are made for graph stream segmentation.

S-PowerGraph applies the programming model proposed by Power Graph to real-time graph stream segmentation and uses vertex truncation based segmentation technique.

The invention of other conventional graph streams has proposed a trunk cutting scheme that minimizes the number of cutting edges caused by the addition of new data. The invention of another conventional graph stream proposes a method of reducing the variation in the storage amount by arranging more data in a node having less memory than a node having high memory utilization in consideration of the memory space of each node constituting the cluster at the same time. .

In the conventional invention of another graph stream, vertex truncation is used, and both storage and throughput are considered as states of nodes in a cluster.

Existing techniques, however, consider only the replication ratio of the vertices in the segmentation criteria of the additional streams, not the state of the servers in the cluster, and the storage capacity in order to consider the state of the servers in the cluster. Did not do it.

In addition, the existing techniques consider storage and processing performance, but define processing performance as the amount of data stored in one node for a certain time. This can be efficient for operations such as PageRank, but for queries looking for a specific graph pattern, The problem is that it is not appropriate.

In addition, in the case of PageRank operation, the number of vertices stored in one node can be directly related to throughput because PageRank values for all vertices must be calculated.However, when only a specific vertex is used for calculation, the total storage is calculated as throughput. It is inappropriate to do.

Also, like the PageRank algorithm, we need to calculate the linkage of all vertices and the values that each vertex has, so the amount of vertices can be directly linked to throughput. However, in the case of a subgraph search query, throughput may be directly related to how many vertices included in the searched subgraph exist in a specific node, rather than the amount of vertices stored in the node. Accordingly, the view of load considered in existing techniques is not common.

Accordingly, the emergence of a new graph stream segmentation technique is urgently required to improve query processing performance for large graphs.

In order to solve the above problems, the present invention uses a vertex truncation-based graph stream segmentation technique in consideration of load balancing, the replication rate of the vertices due to the data added for distributed storage management of the graph stream, addition The object of the present invention is to improve the query processing performance for a large amount of data in the form of graphs by proposing a partitioning criterion that weights each element in consideration of the storage capacity and throughput of nodes in a cluster to which data is to be arranged.

In addition, the present invention proposes a partitioning criterion that weights each element in consideration of the replication ratio of the vertices due to the added data, the storage capacity of the nodes in the cluster where the added data is to be placed, and the throughput for distributed storage management of the graph stream. The purpose of this is to appropriately adjust the load balancing of the cluster to improve the processing performance of the entire system.

In addition, the present invention aims to improve throughput of the overall system by more efficiently adjusting load balancing by defining throughput as a direct query processing frequency.

In addition, the present invention considers the new data as hot data when the new data having connectivity with the hot data used in the graph stream occurs, and puts a heavy load on one node by placing a high weight on the consideration of throughput. It is aimed at preventing becoming.

In order to achieve the above object, a segmentation method for real-time distributed storage of a graph stream includes: calculating a score for a plurality of nodes that individually process each of a plurality of subgraphs obtained by dividing the graph; Determining a selection node having the highest score, and connecting the vertex associated with the new graph stream to the lower graph belonging to the selection node by edge, thereby connecting the new graph stream to the selection node. Processing may be included.

In addition, as a technical configuration for achieving the above object, in a social network, when a new graph stream is generated in accordance with the interaction between things, for a plurality of nodes that individually process each of a plurality of subgraphs divided into a graph A vertex associated with the new graph stream in a score calculator for calculating a score, a node determiner for determining a selected node having the highest score among the plurality of nodes, and a lower graph belonging to the selected node; By connecting by trunk lines, a splitting apparatus for real time distributed storage of a graph stream including a split connection unit for processing the new graph stream at the selected node may be configured.

According to the present invention, it is possible to provide a stream graph segmentation technique based on vertex truncation considering query processing performance and load balancing.

In addition, according to the present invention, it is possible to propose a criterion for arranging a new subgraph in a situation where a large graph is stored.

In addition, according to the present invention, in order to partition and store the graph stream in a server in real time, the state of the server and the replication rate of vertices such as memory utilization and throughput may be considered.

In addition, according to the present invention, when hot data is inserted, load balancing is performed by giving a higher weight to the throughput, thereby solving the problem that load is concentrated on a specific node.

1 is a diagram illustrating a detailed configuration of a splitting apparatus for real time distributed storage of a graph stream according to an embodiment of the present invention.
2 is a view for explaining the overall processing structure of the segmentation scheme for real-time distributed storage for the graph stream according to the present invention.
3 is a view for explaining a process of calculating the throughput and memory utilization in accordance with the present invention.
4 is a view for explaining an example in which the performance of the system is degraded due to hot data.
5 is a diagram illustrating an example of a subgraph query according to the present invention.
6 is a view for explaining a segmentation process of a graph stream according to the present invention.
7 is a diagram illustrating a neighboring trunk line according to the present invention.
8 and 9 are diagrams for explaining an example of a graph stream segmentation added according to the present invention.
FIG. 10 is a detailed flowchart illustrating a partitioning method for real-time distributed storage of a graph stream according to an embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. However, the present invention is not limited or limited by the embodiments. Like reference numerals in the drawings denote like elements.

1 is a diagram illustrating a detailed configuration of a splitting apparatus for real time distributed storage of a graph stream according to an embodiment of the present invention.

The splitting apparatus 100 for real-time distributed storage of the graph stream of the present invention may include a score calculator 110, a node determiner 120, and a split connection unit 130.

First, in a social network, when a new graph stream is generated according to interactions between objects in the social network, the score calculator 110 scores a score for a plurality of nodes that individually process each of a plurality of subgraphs in which the graph is divided. Calculate That is, the score calculator 110 functions to indicate the state of each node as a numerical value in order to select nodes that can be optimal for processing the newly generated graph stream.

As a parameter used for score calculation by the score calculator 110, a storage amount of graph data owned by an individual node, a throughput of processing the graph data in a previous operation, and the like may be exemplified.

In calculating the score for each node, the score calculator 110 checks the storage amount and the throughput for each of the plurality of nodes with reference to the load table 140, and the lower the storage amount, or the The lower the throughput, the higher the score can be calculated.

The load table 140 records the amount of the subgraphs held by each node and the throughput for each cycle of the graph data of the predetermined subgraph. As shown in Table 1 below, the load table 140 can record, for example, the throughput of a node as the number of vertices of the lower graph, and can also record the storage of the node as a memory utilization.

That is, the score calculating unit 110 may give a high score to a node having sufficient capacity for processing the graph data, even if a new graph stream is additionally connected due to the small amount of storage at the present time based on the information in the load table 140. It is also possible to assign high scores to nodes that do not have much throughput up to the previous point in time.

As an example, the score calculator 110 may calculate a total score (TSi) calculated for each of the plurality of nodes as a score by applying the equation (1).

로 표현할 수 있으며,Equation 1 is

Can be expressed as

Here, RSi (Replication Score) may be defined as a vertex replication ratio score of a node, Storage Utilization Score (USi) may be defined as a storage score per node, and CSi (Computation Size Score) may be defined as a throughput score per node. In addition, α, β, and γ may be weights assigned to each score.

The replication score (RSi) may calculate a higher RSi as the number of neighboring edges existing in a node increases, thereby minimizing a replication ratio of a vertex in the graph.

The storage utilization score (USi) may calculate the USi higher as the amount of data stored by the node is smaller, so that the new graph stream is arranged in a node having a relatively low storage amount.

The Computation Size Score (CSi) may calculate the CSi higher as the throughput decreases during the previous processing, thereby minimizing the arrangement of the new graph stream in a node having a relatively large storage amount.

That is, the score calculating unit 110 calculates TSi by applying weights α, β, and γ to the values of RSi, USi, and CSi and adds them, and as a result, each graph to be a reference for arranging a new graph stream. The score of the node can be calculated.

In particular, in the case of γ which is the weight of the CSi, the score calculator 110 identifies whether the data included in the first vertex associated with the new graph stream is hot data used in excess of a predetermined frequency, In the case of hot data, the gamma can be adjusted to a positive value. That is, the score calculator 110 may adjust γ to a larger value so that CSi is calculated higher as the new graph stream includes hot data frequently used.

In another example of calculating the score for each node, the score calculator 110 identifies the first vertex associated with the new graph stream and the second vertex having a common pattern from the graph, and identifies the second vertex. The score can be calculated by giving a weight to a node processing a lower graph having a structure of. That is, the score calculator 110 recognizes the arrangement / configuration of the vertices of the new graph stream as a pattern, finds the vertices in the existing graph that have the recognized patterns similarly, and relatively to the related nodes. High scores can be given.

In another example of calculating the score for each node, the score calculator 110 identifies whether the data included in the first vertex associated with the new graph stream is hot data used in excess of a predetermined frequency, In the case of the hot data, the highest score may be calculated for the node having the lowest throughput among the plurality of nodes with reference to the load table 140. That is, the score calculating unit 110 is a node that processes the vertex in the new graph stream that is generated as the hot data is used frequently, so that the node that records the lowest throughput is determined. More scores can be calculated.

The node determination unit 120 determines a selected node having the highest calculated score among the plurality of nodes. In other words, the node determiner 120 determines the node that has the highest score obtained by quantifying the state of the node as a node capable of optimally processing the new graph stream, and determines the node as the selected node.

The segment connection unit 130 connects the vertices associated with the new graph stream with the trunk to the lower graph belonging to the selected node. That is, the split connection unit 130 may connect the new graph stream to a lower graph maintained by the selected node and update the new graph stream to one lower graph, thereby allowing the selected node to process the new graph stream.

According to the present invention, it is possible to provide a stream graph segmentation technique based on vertex truncation considering query processing performance and load balancing.

In addition, according to the present invention, it is possible to propose a criterion for arranging a new subgraph in a situation where a large graph is stored.

In addition, according to the present invention, in order to partition and store the graph stream in a server in real time, the state of the server and the replication rate of vertices such as memory utilization and throughput may be considered.

In addition, according to the present invention, when hot data is inserted, load balancing is performed by giving a higher weight to the throughput, thereby solving the problem that load is concentrated on a specific node.

In an environment where graph data changes rapidly in real time due to the rapid spread of SNS due to the widespread use of mobile devices, an improved invention on distributed storage management of graph streams is required.

In the improved invention, for static graph segmentation techniques such as Metis or PowerGraph, it is possible to identify which problem occurs when a new vertex or edge is added to the segmented graph, or when deletion continues to occur. There is a need to manage it.

In the present invention, in a cluster composed of one master node and several slave nodes, a distributed storage management scheme of graph streams for improving query processing performance of graphs in a situation where a large graph is divided based on vertex truncation is proposed. do.

In addition, the present invention solves the problems caused by the placement criteria of the added data and the deleted data, and aims to improve the processing performance of the entire system.

In the arrangement of additional data, partitioning criteria considering storage and throughput of each node constituting the cluster are applied. For example, as a partitioning criterion, a criterion for placing more data in a node with low storage or a node with low throughput may be set.

In this case, an additional factor for considering throughput may be hot data frequently used in a query, and the hot data is managed and used as statistical information. In the present invention, the splitting criteria may be changed depending on whether or not the data to be added is hot data, so that hot data may be arranged at a node having low throughput, for example.

2 is a view for explaining the overall processing structure of the segmentation scheme for real-time distributed storage for the graph stream according to the present invention.

As shown in FIG. 2, each slave node stores sub-graph data divided by a segmentation scheme of a vertex truncation scheme proposed by PowerGraph.

The statistical information management module 210 stores information on throughput and storage for each node according to the query processing in the load table 212. In addition, the statistical information management module 210 stores information about frequently requested data in the hot data table 214 based on the query.

Thereafter, when the graph stream occurs, the stream splitting module 220 first refers to the hot data table 214 and analyzes whether the generated graph stream is hot data (Stream Analysis). For example, if the newly generated graph stream is data associated with a vertex frequently requested for a query, the stream dividing module 220 may determine that the hot stream is likely to be used together in the query request after being added.

When it is determined whether hot data for the graph stream is determined, the stream dividing module 220 checks the state of each node with reference to the load table 212 (Node Analysis). In addition, the stream dividing module 220 determines the node having the highest score after calculating the score for each node in consideration of the state of each node according to a predetermined criterion. Finally, the stream splitting module 220 transmits a new graph stream to the determined node (Data Transmission).

In order to increase the performance of the system by performing proper load balancing, the graph stream should be partitioned in consideration of the state of each node constituting the cluster. To this end, the present invention monitors the load of each road in the cluster through load management, and utilizes the information collected according to the result in the segmentation of the graph stream.

In the present invention, the statistical information management module 210 of the master node of FIG. 2 may be configured to manage storage and throughput for each node. The statistical information management module 210 suppresses the performance degradation of the overall system by avoiding storing additional data in a node that has already stored a large amount of data or storing the additional data in a large amount of nodes in comparison with other nodes. can do. That is, the statistical information management module 210 may reflect the result of the load management in the processing of the graph stream in order to prevent such performance degradation.

When a query for searching for a specific subgraph pattern is requested to the master node, the query processing module 230 determines a node to be searched and transmits a pattern to be found.

Subsequently, in the slave node to be searched, the query processing result is transmitted back to the query processing module 230 of the master node, and the number and ID of vertices to be searched included in each node among the requested patterns, The memory usage rate of the corresponding node may be transmitted to the statistical information management module 210. Here, the number of vertices to be searched and the memory utilization are maintained in the load table 212.

Table 1 is an example for a load table.

Table 1 exemplifies maintaining the number of vertices as the throughput of the node and maintaining the memory utilization as the storage. In this case, the throughput is calculated as a ratio relative to the amount processed in the whole system.

3 is a view for explaining a process of calculating the throughput and memory utilization in accordance with the present invention.

3 (a) shows data stored in each node and data included in a query among them.

Throughput may refer to the amount of data included in the query among the data stored in each node.

In FIG. 3A, Node # 1 includes eight vertices, and 5 of the vertices 'a, b, c, e, d' included in the query can be calculated as throughput. According to a similar calculation method, the throughput of Node # 2 is 2, the throughput of Node # 3 is 5, and the throughput of Node # 4 is 2.

When these throughputs are converted on a split basis, the converted throughputs can be calculated as 5/14, 2/14, 5/14, and 2/14 for each node.

3B illustrates a randomly generated subgraph search query.

As shown in (b) of FIG. 3, one subgraph query can be calculated as a path connecting vertices included in the query for each node. For example, with respect to Node # 1, we can create a subgraph query of b-c-e and a-b-c-d around vertex 'c'.

In the present invention, hot data refers to a subgraph frequently used for query processing. Assuming that there are two nodes that store the same amount of graphs, a node with hot data requires a relatively higher throughput for query processing, which can lead to poor overall system performance.

Therefore, when a newly generated subgraph has hot data connectivity, it is regarded as hot data. That is, when the added subgraph is connected to the existing hot data by the edge, the performance of the query processing will be increased for the subgraph additionally connected during the query processing due to the graph connectivity.

Thus, nodes associated with hot data give high weight to throughput to store new subgraphs.

4 is a view for explaining an example in which the performance of the system is degraded due to hot data.

4 is a diagram illustrating a performance degradation situation of the entire system due to hot data. In FIG. 4A, the entire graph is divided into three subgraphs and stored, and the first subgraph (partition # 1) includes hot data vertices s, t, and u.

In this case, when each subgraph is processed in one slave node, FIG. 4B shows the execution time for each partition. In (b) of FIG. 4, all three slave nodes process the same amount of subgraphs, but in the case of slave node 1 including the subgraph 1, the processing time is relatively increased. Accordingly, the other slave nodes (partition # 3, # 2) must wait for the end of processing at the slave node 1 even after the processing has already been completed.

In order to solve the problem that the load increases to a specific node due to the hot data, it is necessary to manage the information of the hot data and to prevent a phenomenon in which the hot data is concentrated on one node.

To this end, the statistical information management module 210 of the present invention manages the subgraphs used for query processing by the hot data table 214, and the stream partitioning module 220 based on the information stored in the hot data table 214. Determines whether the generated graph stream is hot data.

In the present invention, the information on the hot data is collected together in the load monitoring step. The query processing module 230 in the master node may manage the hot data table 214 by ordering the number of vertices appearing in the randomly generated subgraph search query as shown in Table 2.

Table 2 counts and records the number of times the vertex is requested, based on the vertex as the center of the subgraph query in FIG. 5 described later.

For example, in Table 2, the highest number of vertices b with the number of requests 4 may be recorded as the first rank.

5 is a diagram illustrating an example of a subgraph query according to the present invention.

In FIG. 5, a subgraph query formed by connecting vertices a, d, and c around vertex b is Query 1, and a subgraph query formed by connecting vertices b, f, and i around vertex h is Query 2. For example, a subgraph query formed by connecting vertices g, n, and m around vertex b is Query 3, and a subgraph query formed by connecting vertices b, x, and z around vertex q is Query 4 do.

Based on the example of FIG. 5, when the data is summarized in Table 2, for example, since the number of times vertex b is requested in FIG. 5 is 4 in total, the number of requests for vertex ID b in Table 2 is recorded as 4. In addition, by applying a similar counting method, the number of requests for the vertex ID a and the like are recorded as one. In Table 2, the highest rank can be assigned to the vertex ID b having the highest number of requests.

In the present invention, the amount of storage, throughput, and the rate of replication of vertices can be considered to determine which node to store the added graph.

Here, the replication rate of the vertices is a consideration for minimizing the replication rate of the vertices in order to reduce the communication cost of the entire system since communication between the partitions is made through the divided vertices.

In addition, the storage amount is an element for making the amount of subgraphs stored evenly because a large number of subgraphs stored in a node have more throughput in an operation that requires operations on all vertices.

In addition, throughput is a consideration for improving the performance of the entire system by evenly balancing the loads as the value of a specific node increases.

In order to apply these considerations, the present invention makes three considerations into a formula, and when an additional subgraph occurs in the added graph stream, the value of the formula is calculated and the subgraph is added to the node having the highest value. To be deployed.

6 is a view for explaining a segmentation process of a graph stream according to the present invention.

When a new graph stream occurs, the statistical information management module 610 checks the hot data table 614 to determine whether the graph stream corresponds to hot data. In addition, the statistical information management module 610 may adjust a weight for calculating a split criterion according to whether the graph stream is hot data.

The statistical information management module 610 utilizes the information in the load table 614 to calculate the splitting criteria, and the stream splitting module 620 transmits data to the node having the highest score for the score calculated for each node. do.

In calculating the score for each node by the stream splitting module 620, when there are k nodes constituting the cluster, the score of each node is the score of the i (i is 1, 2, ..., k) th node. It is calculated by TSi (Total Score) which means.

Equation 1 is an equation for calculating the Ti value.

Equation 1 takes a method of adding and adding weights α, β, and γ to the values of RSi, USi, and CSi. The value calculated by Equation 1 means the score of each node which is a reference for arranging the added subgraph. Through Equation 1, the present invention calculates values for all nodes constituting the cluster, and arranges subgraphs at nodes having the highest values.

The first element in Equation 1 regarding the partitioning criterion, RSi (Replication Score), is to minimize the replication of the vertex, and refers to the vertex replication ratio score of the i-th node.

The vertex replication ratio, one of the important factors in the vertex splitting scheme, indicates how many nodes a vertex is divided and stored. The higher the vertex replication rate, the more vertices a single vertex is stored in multiple nodes. This means that the cost of communication between nodes to synchronize the vertices of a vertex in an algorithm such as PageRank that calculates a specific value based on the vertices. It means to generate more.

Therefore, in the present invention, when a new subgraph e is added to minimize the replication rate of the vertices, a large score can be given to a node having a large number of neighbor edges existing in the node.

(2) is an expression for calculating RSi (Replication Score) as a vertex replication ratio score of a node. In Equation 2, Pi denotes the i-th partition, and in the present invention, since one node stores one partition. N (e) means a set of neighboring edges of the edge e.

7 is a diagram illustrating a neighboring trunk line according to the present invention.

7 illustrates a concept of neighboring edges. In FIG. 7, e _kl denotes an _edge connecting a vertex k and vertex l, and neighboring edges of e _kl are e _ij , e _lm , and e _ln .

As a result, RSi is a factor to consider which node has the most neighbor edges for the newly added subgraph e, that is, the node's replication rate can be minimized when placed on which node.

의 값은 가장 큰 값을 갖는 노드의 값으로 나누어 해당요소를 0~1사이의 값으로 정규화하기 위한 것이다. In Equation 2, the denominator

The value of is divided by the value of the node with the largest value to normalize the element to a value between 0 and 1.

The Storage Utilization Score (USi) of Equation 1 is an element for considering storage amount for each node. If a large number of subgraphs are stored in a particular node, query processing will require a relatively large number of operations.

In order to prevent this, the added subgraph should be placed in a node having a relatively small storage amount. For this purpose, in the present invention, the storage amount score is calculated using Equation 3.

According to Equation 3, it is possible to give a high score to a node having a small stored amount.

값은 i번째 노드에 저장된 서브그래프의 크기를 의미한다.Pi represented by Equation 3 means the i-th node described above, and

The value represents the size of the subgraph stored in the i th node.

Ci means the total memory size of the i-th node, and consequently USi represents the memory utilization of the i-th node.

Computation Size Score (CSi) of Equation 1 is an element for considering throughput. Continued addition of subgraphs to high-throughput nodes results in high throughput on only one node, resulting in load imbalance and poor system performance.

In order to prevent this, in the present invention, a throughput score is calculated as in Equation 4 so that a node having a relatively low throughput during a previous query processing has a high score.

In Equation 1, γ, which is a weight of the throughput score, varies in weight depending on whether or not the added subgraph is hot data. When it is determined that the hot data is determined, γ is adjusted to a relatively large value, thereby increasing the throughput score at a node with low throughput to induce hot data to be placed.

Si in Equation 4 represents the number of vertices included in the i-th node among the vertices included in the query. The rightmost side of Equation 4 may be calculated by summing all the number of vertices included in each node among the vertices included in the query, and dividing the number for each node by the total sum.

Subsequently, in the present invention, by subtracting the result value of the rightmost side of Equation 4 from 1, a node having a high throughput can receive a small amount of graphs.

In the formula of division criteria (Equations 3 and 4), each value has a value between 0 and 1 so that the entire division criteria is not affected by only one value. In addition, α, β, and γ suggest appropriate values through comparison of query processing performance in performance evaluation.

If the deletion of the subgraph continues on a particular node, the whole system will experience load imbalances in terms of storage. As a result, the variation in throughput may be large, and even if a small amount of hot data is deleted, there is a possibility of an imbalance in terms of throughput.

An imbalance problem that occurs from the storage point of view is that if the storage value of one node is lower than the threshold value for each node, the subgraph inputted by setting β to a high value in Equation 2 applied at the time of insertion has priority This allows them to be deployed on nodes with less storage.

In terms of throughput, when a node's throughput drops below a threshold value, γ is set to a high value for the next input so that a node having low throughput has priority.

8 and 9 are diagrams for explaining an example of a graph stream segmentation added according to the present invention.

As shown in FIG. 8, the subgraphs are stored in each of slave nodes 1 to 4.

In the slave node 1, when a trunk line connecting vertices 3 and 7 is newly inserted, the present invention first determines whether vertices belonging to hot data are among vertices 3 and 7. For example, in the present invention, since vertex 7 belongs to the first position on the hot data table of FIG. 9, vertex 7 may be determined as hot data.

Since the added subgraph has been determined to be hot data, the present invention adjusts the weight of Equation 1 to be calculated.

As shown in (a) and (b) of FIG. 9, the score calculated for each vertex has a higher score as the memory usage rate, which is a storage amount, and as the throughput is low.

In (c) of FIG. 9, the value of the last column represents the finally calculated score for each node. The weighting of the throughput is increased by the occurrence of hot data, so node 1 has the highest value and a new subgraph is placed in node 1. However, it is important to note that only the throughput is not considered even when hot data occurs, because the scores for the replication rate and storage did not differ significantly.

The present invention proposes an efficient graph stream partitioning scheme considering the load of nodes constituting the cluster to improve query processing.

We considered the replication rate, storage, and throughput of the vertices as factors considered in the stream segmentation criteria, and defined the throughput as the amount of data used for query processing to be able to clearly distinguish it from the storage.

In addition, the partitioning criteria are changed according to whether hot data is generated to prevent hot data from being concentrated on one node.

The present invention can be applied to a technique for distributing or analyzing graph data generated in real time. In addition, the present invention is utilized as a method for effectively storing a large amount of data in the field of services for storing and managing object relationships in real time, such as social networks and the Internet of Things.

Hereinafter, a detailed description will be given of a workflow of a partitioning method for real time distributed storage of a graph stream according to an embodiment of the present invention.

FIG. 10 is a detailed flowchart illustrating a partitioning method for real-time distributed storage of a graph stream according to an embodiment of the present invention.

The division method for real time distributed storage of the graph stream according to the present embodiment may be performed by the division apparatus 100 described above.

First, when a new graph stream is generated according to interaction between objects in the social network, the dividing apparatus 100 calculates a score for a plurality of nodes that individually process each of a plurality of subgraphs that have divided the graph. 1010. Step 1010 may be a process of numerically representing the state of each node to select nodes that may be optimal for processing the newly generated graph stream.

As a parameter used to calculate the score in the dividing apparatus 100, a storage amount of graph data owned by an individual node, a throughput of processing the graph data in a previous operation, and the like may be exemplified.

In calculating the score for each node, the division apparatus 100 confirms the storage amount and the throughput for each of the plurality of nodes with reference to the load table, and the lower the storage amount or the lower the throughput, The score can be calculated higher.

The load table records the amount of subgraphs held by each node and the throughput for each cycle of graph data of a given subgraph. The load table can, for example, record the throughput of a node as the number of vertices that a subgraph has, and can also record the storage of a node in memory utilization.

In operation 1010, the partitioning apparatus 100 may give a high score to a node having sufficient capacity for processing the graph data, even if a new graph stream is additionally connected due to the small amount of storage at the present time, based on the information in the load table. In addition, a high score can be given to a node that does not have much throughput up to the previous point in time.

As an example, the dividing apparatus 100 may calculate a total score (TSi) calculated for each of the plurality of nodes as a score by applying the equation (1).

로 표현할 수 있으며,Equation 1 is

Can be expressed as

Here, RSi (Replication Score) may be defined as a vertex replication ratio score of a node, Storage Utilization Score (USi) may be defined as a storage score per node, and CSi (Computation Size Score) may be defined as a throughput score per node. In addition, α, β, and γ may be weights assigned to each score.

The replication score (RSi) may calculate a higher RSi as the number of neighboring edges existing in a node increases, thereby minimizing a replication ratio of a vertex in the graph.

The storage utilization score (USi) may calculate the USi higher as the amount of data stored by the node is smaller, so that the new graph stream is arranged in a node having a relatively low storage amount.

The Computation Size Score (CSi) may calculate the CSi higher as the throughput decreases during the previous processing, thereby minimizing the arrangement of the new graph stream in a node having a relatively large storage amount.

That is, the division apparatus 100 calculates TSi by applying weights α, β, and γ to RSi, USi, and CSi values, and adds them, and as a result, each node to be a reference for arranging a new graph stream. The score of can be calculated.

In particular, in the case of γ which is the weight of the CSi, the dividing apparatus 100 identifies whether the data included in the first vertex associated with the new graph stream is hot data used over a predetermined frequency, and the hot In the case of data, the gamma can be adjusted to a positive value. That is, the division apparatus 100 may adjust γ to a larger value so that CSi is calculated higher as the new graph stream includes hot data that is frequently utilized.

In another example of calculating the score per node 1010, the segmentation apparatus 100 identifies from the graph a first vertex associated with the new graph stream and a second vertex having a common pattern, A weight may be given to the node processing the lower graph constituting the second vertex to calculate the score higher. That is, the segmentation apparatus 100 recognizes the arrangement / configuration of the vertices of the new graph stream as a pattern, finds the vertices in the existing graph having similarly recognized patterns, and has a relatively high level with respect to the related nodes. A score can be given.

In another example of calculating the score per node 1010, the dividing apparatus 100 determines whether the data included in the first vertex associated with the new graph stream is hot data that is used over a predetermined frequency. In the case of hot data, the highest score may be calculated for the node having the lowest throughput among the plurality of nodes with reference to the load table. In other words, the segmentation apparatus 100 is a node that processes the vertices in the new graph stream, which is generated frequently, as the vertices associated with the hot data are utilized, so that the node that records the lowest throughput until then is determined. Many scores can be calculated.

In addition, the division apparatus 100 determines a selected node having the highest calculated score among the plurality of nodes (1020). Step 1020 may be a process of determining a node having the highest numerical value of the state of the node as a node capable of optimally processing the new graph stream and determining it as a selection node.

In operation 1030, the segmentation apparatus 100 connects the vertices associated with the new graph stream to the lower graph belonging to the selected node by an edge. Step 1030 may be a process of processing the new graph stream at the selection node by connecting the new graph stream to a lower graph maintained by the selection node and updating the data with one subgraph.

According to the present invention, it is possible to provide a stream graph segmentation technique based on vertex truncation considering query processing performance and load balancing.

In addition, according to the present invention, it is possible to propose a criterion for arranging a new subgraph in a situation where a large graph is stored.

In addition, according to the present invention, in order to partition and store the graph stream in a server in real time, the state of the server and the replication rate of vertices such as memory utilization and throughput may be considered.

In addition, according to the present invention, when hot data is inserted, load balancing is performed by giving a higher weight to the throughput, thereby solving the problem that load is concentrated on a specific node.

The method according to an embodiment of the present invention can be implemented in the form of program instructions that can be executed by various computer means and recorded in a computer readable medium. The computer readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the media may be those specially designed and constructed for the purposes of the embodiments, or they may be of the kind well-known and available to those having skill in the computer software arts. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tape, optical media such as CD-ROMs, DVDs, and magnetic disks, such as floppy disks. Magneto-optical media, and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine code generated by a compiler, but also high-level language code that can be executed by a computer using an interpreter or the like. The hardware device described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

Although the embodiments have been described by the limited embodiments and the drawings as described above, various modifications and variations are possible to those skilled in the art from the above description. For example, the described techniques may be performed in a different order than the described method, and / or components of the described systems, structures, devices, circuits, etc. may be combined or combined in a different manner than the described method, or other components. Or even if replaced or replaced by equivalents, an appropriate result can be achieved.

Therefore, other implementations, other embodiments, and equivalents to the claims are within the scope of the claims that follow.

100: partitioning device for real time distributed storage of graph stream
110: score calculation unit 120: node determination unit
130: split connection

Claims (14)

A partitioning method for real-time distributed storage for a graph stream implemented by a splitting device for real-time distributed storage for a graph stream,
In social networks, when a new graph stream occurs due to interaction between things,
Calculating, by the score calculator in the splitting device, a score for a plurality of nodes that individually process each of the plurality of subgraphs obtained by dividing the graph;
Determining, by the node determining unit in the dividing device, a selected node having the highest calculated score among the plurality of nodes; And
Processing the new graph stream at the selection node by connecting a vertex associated with the new graph stream to the lower graph belonging to the selection node by a trunk at a division connection unit in the division apparatus;
Including,
The step of calculating the score,

Calculating a total score (TSi) calculated for each of the plurality of nodes as the score;
The Replication Score (RSi) is a peak replication ratio score of the node, the Storage Utilization Score (USi) is a storage score for each node, and the Computation Size Score (CSi) is a throughput score for each node,-The α is the Is a weight of RSi, β is a weight of the USi, and γ is a weight of the CSi
Identifying whether data included in a first vertex associated with the new graph stream is hot data used over a predetermined frequency; And
In the case of the hot data, the CSi is calculated to be high by adjusting the gamma to a positive value.
Partitioning method for real-time distributed storage for the graph stream comprising a. The method of claim 1,
The step of calculating the score,
Identifying a storage amount and a throughput for each of the plurality of nodes with reference to a load table; And
Calculating the score higher as the storage amount is lower or the throughput is lower.
Partitioning method for real-time distributed storage for the graph stream further comprising. The method of claim 1,
The step of calculating the score,
Identifying from the graph a first vertex associated with the new graph stream and a second vertex having a common pattern; And
Calculating a high score by assigning a weight to a node processing a lower graph constituting the second vertex
Partitioning method for real-time distributed storage for the graph stream further comprising. The method of claim 1,
The step of calculating the score,
In the case of the hot data, calculating a highest score at a node having the lowest throughput among the plurality of nodes by referring to a load table.
Partitioning method for real-time distributed storage for the graph stream further comprising. delete The method of claim 1,
The step of calculating the score,
Calculating the RSi higher as the number of neighboring edges existing in a node increases, thereby minimizing a replication ratio of the vertices in the graph
Partitioning method for real-time distributed storage for the graph stream further comprising. The method of claim 1,
The step of calculating the score,
Computing the USi higher as the amount of data stored by a node is smaller, so that the new graph stream is placed in a node having a relatively small amount of storage.
Partitioning method for real-time distributed storage for the graph stream further comprising. The method of claim 1,
The step of calculating the score,
Computing the CSi higher as the throughput is lower during the previous processing, thereby minimizing the new graph stream being placed in a node with a relatively large amount of storage.
Partitioning method for real-time distributed storage for the graph stream further comprising. delete In social networks, when a new graph stream occurs due to interaction between things,
A score calculator configured to calculate a score for a plurality of nodes that individually process each of the plurality of subgraphs obtained by dividing the graph;
A node determination unit that determines a selected node having the highest calculated score among the plurality of nodes; And
A segmentation connector for processing the new graph stream at the selected node by connecting the vertices associated with the new graph stream with edges to a lower graph belonging to the selected node.
Including,
The score calculation unit,

Applying, calculates TSi (Total Score) calculated for each of the plurality of nodes as the score,
The Replication Score (RSi) is a peak replication ratio score of a node, the Storage Utilization Score (USi) is a storage score for each node, and the Computation Size Score (CSi) is a throughput score for each node,-The α is the Is a weight of RSi, β is a weight of the USi, and γ is a weight of the CSi
Identifying whether the data included in the first vertex associated with the new graph stream is hot data used over a predetermined frequency,
In the case of the hot data, the CSi is calculated to be high by adjusting the gamma to a positive value.
Partitioning device for real-time distributed storage of graph streams. The method of claim 10,
The score calculation unit,
With reference to the load table, for each of the plurality of nodes, the storage amount and the throughput are checked, and the lower the storage amount or the lower the throughput, the higher the score is calculated.
Splitting device. The method of claim 10,
The score calculation unit,
Identify from the graph a first vertex associated with the new graph stream and a second vertex having a common pattern,
A weight is given to a node processing a lower graph constituting the second vertex, and the score is calculated to be high.
Splitting device. The method of claim 10,
The score calculation unit,
In the case of the hot data, the highest score is calculated for a node having the lowest throughput among the plurality of nodes with reference to a load table.
Splitting device. delete

Images