WO2013145310A1

WO2013145310A1 - Data stream parallel processing program, method, and system

Info

Publication number: WO2013145310A1
Application number: PCT/JP2012/058731
Authority: WO
Inventors: エメリックヴィエル; 晴康上田
Original assignee: 富士通株式会社
Priority date: 2012-03-30
Filing date: 2012-03-30
Publication date: 2013-10-03

Abstract

Provided is a system for allocating a plurality of queries used to process data streams to a plurality of nodes. Each of the plurality of nodes has a capability of outputting a result obtained by executing an allocated query to a subsequent stage. The system includes: a query information acquisition section that acquires the plurality of queries and the connection relationships among the plurality of queries; a distributed-key-set extraction section that extracts a distributed key set which can be used to execute each of the plurality of queries in parallel, in association with each of the plurality of queries; a distributed-segment generation section that generates a distributed segment to which a query subset of the plurality of queries belongs and which has a common distributed key set that the query subset has in common; and a parallel distribution section that distributes nodes in parallel corresponding to the distributed segment and allocates the query subset that belongs to the distributed segment to each of the one or more nodes distributed in parallel.

Description

Data stream parallel processing program, method, and system

The present invention relates to a data stream parallel processing program, method, and system.

In recent years, there has been an increasing demand for services that collect and utilize big data, which is a large amount of data provided from various information sources, devices, sensors, etc. connected to a network. If a large amount of data generated in the real world can be sequentially processed, information can be obtained in a state close to real time. For example, there is a need for a technique that can sequentially process a large amount of data stream that is constantly provided from various sensors.

As a technology that meets such requirements, complex event processing that handles big data is known. However, the spread of smartphones and tablet terminals in recent years has dramatically increased the amount of communication. Further, in the future, when not only humans but also a large number of devices are connected to the network, it is expected that the amount of communication will increase further. Therefore, further development of such technology is required.

In this case, for example, the data (event sequence) obtained from the data stream can be temporarily stored in the database, and then the data can be extracted and processed. However, from the viewpoint of easily obtaining accurate information in real time, the measures as described above often do not provide sufficient results for needs. Therefore, there is a need for a technique for processing and analyzing a large amount of data streams in real time (or near real time). In order to satisfy this need, a technique for sequentially processing data streams in parallel is required.

FIG. 1 shows an example of data stream processing. As shown, the stream processing system 140 sequentially processes the data streams for the three input streams (110, 120, 130). The stream processing system 140 provides two output streams (150, 160). For example, in the input stream 110, a plurality of events (111, 112, 113) are input to the stream processing system 140 in time series. The stream processing system 140 includes a plurality of queries (142, 144, 146, 148, 149). These queries are similar to those used in static database processing. However, queries for stream processing systems have different aspects than queries for databases in that they always operate on input information and provide the required output. Also, the output of one query is the input of another query, which is different from a database query. Thus, it should be noted that the term “query” as used herein also has a different function than a database query. Each query is connected as indicated by a plurality of arrows. These arrows indicate the flow of data. The stream 150 output from the stream processing system 140 includes, for example, a plurality of processing results (151 and 152). In the present specification, a graph indicating a connection relationship between a plurality of queries included in the stream processing system 140 is referred to as a query graph. A processing program including a set of queries represented by a query graph and the relationship between each query is referred to as a data stream program.

In FIG. 2A, two queries Q1 and Q2 are connected by an intermediate stream 240. Then, the input stream 210 is processed by the query Q1 and the query Q2, and the query Q2 outputs the output stream 270. The query Q1 has K1 and K2 as distribution keys, and the query Q2 has K2 and K3 as distribution keys. Note that the distribution key corresponds to a distribution key that can be applied to a hash function used in parallel hash join processing in a static database. A set of distributed keys input to the hash function is called a distributed key set. In other words, for example, a distributed key means a key used for joining in a join operator constituting a query. The distributed key used in this specification has the same meaning as the above definition. In this specification, the hash join method using the distribution key is used to appropriately distribute the data stream processed by the query to the subsequent nodes, as described below with reference to FIG. It should be noted that this is different from the hash join method in the database technology that handles tables.

FIG. 2B shows an example in which the data stream program shown in FIG. 2A is executed in parallel in order to process the input stream 210 according to the parallel hash join method.

The input stream 210 is expressed using a notation according to the table handled in the database. The input stream 210 has a plurality of keys (K1, K2, K3). Further, a plurality of specific events (212, 214, 216, 218) are arranged in time series to form the input stream 210. The query Q1 is deployed to the node 232 and the node 234. Here, specifically, the node may be, for example, a physical machine or a virtual machine. The query Q1 is distributedly processed by the two nodes (232, 234). At point 220, in order to distribute the input stream 210 to the

nodes

232 and 234, the distribution key set (K 1, K 2) is applied to an appropriate hash function, and the input stream 210 is divided into the stream 221 and the stream 222. By the stream 221, the event 212a and the event 214a arrive at the node 232 sequentially and are processed. Event 216a and event 218a sequentially arrive at node 234 and are processed by stream 222. A technique used in a static database can be used as the hash function. Specifically, a hash table or the like may be used. In this case, various hash functions for separating the data stream 210 into two streams (221, 222) using the distributed key set (K1, K2) are applicable.

In FIG. 2B, the query Q2 is further deployed in the node 252 and the node 254. Q2 has a distributed key set (K2, K3) and is different from Q1. Therefore, for example, the events to be processed by Q2 in the node 252 are the event 212b from the node 232 and the event 216b from the node 234. Similarly, events to be processed by Q2 in the node 254 are an event 214b from the node 232 and an event 218b from the node 234.

Therefore, in the node 232, the output is distributed to the stream 242 and the stream 244 according to the distributed key set (K2, K3) of Q2 and an appropriate hash function, and an appropriate event of the output is assigned to the node 252 and the node 252 respectively. 254 need to be sent. Similarly, the node 234 distributes the output to the stream 246 and the stream 248 according to the distributed key set (K2, K3) of Q2 and an appropriate hash function, and the appropriate event of the output is assigned to the node 252 and the node 245, respectively. 254 need to be sent.

In this way, in the example shown in FIG. 2B, four nodes (232, 234, 252, 254) are provided, and the queries Q1 and Q2 are executed in parallel by the parallel hash join method. It can be seen that communication between the four nodes (242, 244, 246, 248) occurs between the two nodes (232, 234, 252, 254). This communication consumes network resources between nodes.

In this specification, for easy understanding of the description, the data flow between the nodes is described using each event of the input stream. In the actual processing, the processing result of the query belonging to the preceding node is provided to the succeeding node at the succeeding node located after the certain preceding node.

The above-mentioned network resource consumption increases as the number of nodes for parallel distribution increases, which is a problem in appropriately realizing parallel distributed processing.

Conventionally, when multiple nodes that process databases are stored, data is distributed and stored in each node, and related source and related destination data are stored in the same node, queries including related operations There is a technology of a distributed database system that executes and processes. When the relation source and the relation destination are stored in the same node, the related operation does not cross the nodes, and the related operation process becomes an asynchronous process for each node and can be executed in parallel. There is a technique in which the results processed for each node are collected by sorting and merging the processing results (see Patent Document 1).

In addition, when data necessary for processing is stored in the computer that has received the data in the stream data processing, a query is executed, and when the necessary data is not stored, other data is stored. There is a technique for receiving data to be processed from a computer and executing a query (see Patent Document 2).

Also, in the distributed database, the input inquiry request is analyzed, and a plurality of database calculation requests corresponding to the number of key ranges of the database calculation keys and a plurality of data retrieval requests corresponding to the database calculation requests are generated. . A technique for distributing the generated database calculation requests to a plurality of nodes provided corresponding to the key ranges, receiving the database calculation processing results from the plurality of nodes, and outputting the processing results of the inquiry request. Yes (see Patent Document 3).

Also, in a distributed database consisting of three or more devices, when performing a join between three or more tables, N keys are sent from node 1 to node 2, receiving matching data, and matching data There is a technique for sending a key to node 3, receiving matching data, and combining the two results and outputting the entire match (see Patent Document 4).

Japanese Patent Laid-Open No. 10-240753 JP 2011-34255 A Japanese Patent No. 3538322 JP 2010-272030 A

In one aspect, an object of the present invention is to reduce the amount of data communication between queries. *

The embodiment is a program for deploying a plurality of queries for processing a data stream to a plurality of nodes, and each of the plurality of nodes outputs a result of executing the query deployed to the nodes to a subsequent node or Used to obtain a plurality of queries and a connection relationship between each of the plurality of queries, hash the data stream, and execute each of the plurality of queries in parallel. A distributed key set that is associated with each of the plurality of queries and is a distributed segment to which a query subset of the plurality of queries belongs, and a common distributed key set that the query subset has in common A distributed segment having a node, and correspondingly distributing the node in parallel to each of the one or more parallel-distributed nodes. The query subset belonging to the segment is deployed, processes, provides a program for causing a computer to execute.

According to the embodiment, it is possible to reduce the amount of data communication between queries.

It is a figure explaining the concept of the data stream process using a query. It is a figure which shows the example which performs a query in parallel by the parallel hash joining method. It is a figure which shows the outline of one Embodiment of this invention. It is a figure which shows the processing flow of one Embodiment. It is a figure which shows the detail of the extraction flow of the dispersion | distribution segment in one Embodiment. It is a figure which shows the specific example of extraction of the dispersion | distribution segment in one Embodiment. It is a figure which shows the specific example of extraction of the dispersion | distribution segment in one Embodiment. It is a figure which shows the detail of the extraction flow of the dispersion | distribution segment in one Embodiment. It is a figure which shows the result of the extraction of the dispersion | distribution segment in one Embodiment. It is a figure which shows the example of prioritization of the distribution segment in one Embodiment. FIG. 6 illustrates node redistribution in one embodiment. FIG. 4 is a diagram illustrating extraction of distributed segments in an embodiment. FIG. 6 illustrates the deployment of queries to nodes distributed in parallel according to one embodiment. It is a functional block diagram of one embodiment. It is a figure which shows the hardware structural example of one Embodiment.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. It should be noted that the following embodiments are for understanding the invention and are not intended to limit the scope of the present invention. Further, the following embodiments are not mutually exclusive. Therefore, it should be noted that the elements of the embodiments are also intended to be combined unless a contradiction arises. Further, in the invention according to the method and the program described in the claims, the order of the processes may be changed as long as there is no contradiction, or a plurality of processes may be performed simultaneously. It goes without saying that these embodiments are also included in the technical scope of the invention described in the claims. In addition, it should be noted that similar components may be denoted by the same reference numerals in a plurality of drawings.

FIG. 3 shows an outline of an embodiment according to the present invention. FIG. 3A shows a query graph of a data stream program including a query Q1 and a query Q2 similar to those in FIG. A different point from FIG. 2 is that a distributed segment 310 in which the query Q1 and the query Q2 are integrated is introduced. The distributed segment 310 has a distributed key set K2, which is a common distributed key set for the queries Q1 and Q2. Among a series of queries, queries having a common distributed key set belong to the same distributed segment, and the distributed segment has this common distributed key set (common distributed key set).

Referring back to FIG. 2, only one query was considered for each node. In the processing of the data stream, in the case where the distributed key sets do not completely match in each query (that is, query Q1 and query Q2), complicated communication has occurred between the nodes where each query is deployed.

FIG. 3B shows an example of a measure for reducing the above-mentioned complicated communication between nodes (between queries) that occurred in FIG. 2B. As shown in the figure, a query Q1 and a query Q2 belonging to the distributed segment 310 are arranged in each of the four nodes (312, 314, 316, 318). The data stream 210 is then distributed at point 320 by applying the distributed key set K2 to the appropriate hash function. That is, the event (212c, 214c, 216c, 218c) is given to the node (312, 314, 316, 318) via the stream (321, 322, 323, 324), respectively. In this case, the distributed key set K2 that is a shared distributed key set of the query Q1 and the query Q2 is applied to the hash function, and the stream 210 is distributed and distributed. For this reason, for example, in the node 312, the output of the query Q1 may be given to the query Q2 in the node. The same applies to the other nodes (314, 316, 318). Therefore, in this case, complicated communication (242, 244, 246, 248) between nodes as seen in FIG. 2B does not occur. Then, the outputs of the nodes are finally combined to obtain an output 270.

Further, refer to FIG. As described above, when a common distributed key set exists in continuous queries, these are combined as one distributed segment in the query graph. Then, one or more queries included in the same distributed segment are deployed to one node. As a result, even if the nodes are distributed in parallel, the occurrence of complicated communication between the nodes is prevented. Note that the number of distributed segments and the number of nodes that distribute queries in parallel may be determined depending on the weight of query processing, the number of physical machines that can be adopted as nodes, the amount of streams, and the like. An appropriate hash function to which the distributed key set of the distributed segment should be applied may be defined according to the determined number of distributed processing nodes. As for the hash function, an appropriate hash function may be used so that the events of the data stream are as uniform as possible.

FIG. 4A shows the processing flow of one embodiment.

In step 402, a data stream program including a query is read. This data stream program is converted into a query graph. The query graph is used for the following processing.

In step 404, the distribution key set of each query is extracted. For example, in the SQL language, a key connected by a join can be an element of a set of distributed keys.

In step 406, a distributed segment is generated. For example, when a common distributed key set exists between two adjacent query distributed key sets, a distributed segment having the common distributed key set may be generated. These two queries can belong to the generated segment. Details of this processing will be described later (see FIGS. 5 to 7).

In step 408, priorities are assigned to the extracted plurality of distributed segments according to a predetermined rule. Details of the priority assignment processing will be described later (see FIG. 8). And the process which selects from the dispersion | distribution segment with a high priority is performed. In the process of selecting a distributed segment, if all the queries are covered by the selected plurality of distributed segments, this selection ends.

In step 410, a query is assigned to each distributed segment.

In step 412, each distributed segment is distributed in parallel to each node. When distributing in parallel, an appropriate hash function is specified in consideration of the number of available nodes, the weight of query processing, and the like. Then, a distributed segment is assigned to each node, and a query belonging to the distributed segment is deployed.

Through the above processing, appropriate queries are deployed to a plurality of nodes for performing parallel and distributed processing of queries, and connection relationships between the nodes are constructed.

Refer to FIG. 4 (B). The processing starting from step 430 shows processing for further tuning when the data stream is actually processed.

Processing is started in step 430. As a trigger for this start, for example, a timer interrupt may be used and executed periodically. Or you may perform suitably according to an operator's instruction | indication.

In step 432, execution profiling of each query and / or each node is performed. By this processing, the processing status of each query and / or each node is acquired as a profile. Various information including the number of events per unit time in each query and / or each node, CPU load, memory usage, processing capability, and the like may be acquired in the profile.

In step 433, this profile is evaluated. For example, a heavily loaded query and / or node is identified using a predetermined rule. This rule may use the number of events per fixed time, memory usage, and the like. This rule corresponds to the second rule. This identified query and / or node is determined as a load balancing recommendation target in a later process, and if it is a recommendation target, a process for load balancing is performed [0].

In step 434, it is determined whether or not there is a candidate distributed segment that can be used for the above-described load distribution from the unused distributed segments that are not yet used. For example, it is checked whether there is an unused distributed segment having a query that overlaps with the query specified in step 432 and / or the query set included in the node. If such an unused distributed segment exists, there is a possibility that a query belonging to the unused distributed segment can be deployed to a new node.

In Step 436, if there is another unused distributed segment, an unused distributed segment is selected in consideration of the priority of the distributed segment. Attach the query to the selected distributed segment. Thereafter, the process returns to step 412. In step 412, as described above, a query belonging to the selected distributed segment is deployed to the new node so that the selected distributed segment can be used. If necessary, a plurality of new nodes may be prepared and distributed in parallel. A specific example will be described later (see FIG. 11).

Note that the distribution of data streams by hash functions may be checked by profiling. In general, the range of values for each distribution key is greater than the number of nodes distributed in parallel. For this reason, as long as there is no deviation in the distribution by the hash function, the event is transferred to all the nodes distributed in parallel. In the profiling, when a deviation is found in the distribution of the data stream by the hash function, a hash function that is conscious of eliminating the deviation may be selected again. Further, the deployment plan related to the number of parallel distributions of nodes may be changed.

FIG. 5 shows a detailed flow of the process of step 406 in FIG. In this process, each query existing in the query graph acquired in step 402 is comprehensively processed in order from the input side. Therefore, processing is looped by the number of queries.

In step 502, the process is started. As described above, the query is processed by paying attention to one query in order from the query close to the input. One query that is attracting attention in the processing is referred to as “under consideration query”.

In step 504, a list of distributed segments extracted up to the previous query (that is, a query connected to the input side of the query under consideration) is acquired. If a plurality of queries are connected to the input side of the query under consideration, it is necessary that the distributed segment extraction process has been completed for all of the plurality of queries. A specific example of this process will be described later with reference to FIG.

In step 506, a distributed key set is obtained from the query under consideration.

In step 508, a new distributed segment is created from the acquired distributed key set. Only the query under consideration will belong to this new distributed segment. Further, the distribution key set of the new distribution segment is the same as the distribution key set of the query under consideration belonging to the new distribution segment.

In step 510, the distributed key set of the new distributed segment is matched with the distributed key set of the distributed segment extracted up to the previous query. As a result of this matching, if there is no common distributed key set (complete match), the new distributed segment is registered as an official distributed segment. In this case, it is divided into the following two processes. (1) If there is a distributed segment including a partially matching distributed key set, a new distributed segment having a partially matching distributed key set is created and registered to the input side. (2) If there is no partially consistent distribution key, the new distribution segment is formally registered as it is. A specific example of this process will be described later with reference to FIG. If there is a distributed segment having a common distributed key set (exact match), the new distributed segment is discarded, and a process of adding the under-considered distributed segment to the distributed segment extracted up to the previous query is performed. A specific example of this process will be described with reference to FIG.

If there is an unprocessed query in step 512, the process returns to step 512. If the processing has been completed for all queries, this processing ends.

FIG. 6 shows a specific example of the processing in step 504 in FIG. Assume that the query Q3 is a query under consideration. In this case, in order to check the query Q3, the generation (extraction) of the distributed segments in the queries Q1 and Q2, which are the queries before the query Q3, must be completed.

FIG. 6A shows the integration processing of the same distributed segment. Here, in the notation S1 (K1) shown in the query Q1 before processing in FIG. 6A, Q1 belongs to the distributed segment S1, and the distributed key set of the distributed segment S1 is K1. Indicates. Q2 belongs to the distributed segment S2 and indicates that the common distributed key set is K1. In this case, the distributed segment S1 and the distributed segment S2 have the same distributed key set. For this reason, the distributed segment S2 is discarded, and the query Q2 also belongs to the distributed segment S1. Therefore, both the query Q1 and the query Q2 belong to the distributed segment S1 and have the distributed key set K1. This state is shown after processing. Thus, it is shown that the query Q1 and the query Q2 that are not adjacent to each other may belong to the same distributed segment S1. As shown in the distributed segment S1 (K1) (610), the query Q2 also belongs to the distributed segment S1.

FIG. 6B illustrates a case where the distributed segment S1 to which the query Q1 belongs and the distributed segment S2 to which the query Q2 belongs have a common distributed key set. The distributed segment S1 and the distributed segment S2 have a common distributed key set K1. However, there is no distributed segment having the distributed key set K1. In this case, a new distributed segment S3 having a distributed key set K1 is created. After the processing of FIG. 6B, it can be seen that the distributed segment S3 (K1) (620) and the distributed segment S3 (K1) (621) are generated. Therefore, in this case, the query Q1 belongs to the distributed segment S1 and also belongs to the distributed segment S3. Similarly, the query Q2 belongs to the distributed segment S2 and also belongs to the distributed segment S3.

FIG. 6C shows an example in which the distributed segment S1 to which the query Q1 belongs is extended to the query Q2. That is, as shown in the distributed segment S1 (K1) (630), it can be seen that the query Q2 also belongs to the distributed segment S1 after the processing.

FIG. 7 shows a specific example of the processing of step 510 in FIG. Assume that the query Q3 is a query under consideration.

In FIG. 7A, the query Q3 has a distributed key set K1. The query Q2 before the query Q3 belongs to the distributed segment S1 having a common distributed key set K1. In this case, as shown after processing, the query Q3 also belongs to the distributed segment S1 having the common distributed key set K1. As a result, the distributed segment S1 is extended to the query Q3 (see S1 (K1) (710)).

FIG. 7B illustrates a case where the query Q2 before the query Q3 under consideration does not belong to a distributed segment having the same distributed key set as the distributed key set of Q3. In this case, the query Q1, the query Q2, and the query Q3 have a common distributed key set K1. Therefore, a new distributed segment S2 is generated, and the query Q1, the query Q2, and the query Q3 are assigned to this. This point is shown as distributed segment S2 (K1) (720), distributed segment S2 (K1) (721), and distributed segment S2 (K1) (722).

FIG. 8 shows a detailed flow of step 408 in FIG. It should be noted that the flow shown in FIG. 8 performs recursive processing.

In step 802, one or more distributed segments to be processed are checked. In other words, a predetermined evaluation function is applied to the distributed segment to be processed, and sorting is performed in order of high evaluation (priority order). As the predetermined evaluation function, the longer the distributed segment length (the number of queries to which it belongs), the higher the evaluation for the distributed segment may be. Or you may give high evaluation, so that there are many common distributed key sets which a distributed segment has. Or you may give high evaluation, so that the weight of the process of the query which belongs to a distributed segment is heavy. The processing weight may be acquired by execution profiling as described in FIG. The present invention is not limited to the above evaluation function. This evaluation function corresponds to the first rule.

In step 804, it is determined whether there is only one distributed segment to be processed. If this determination is “No”, the process proceeds to Step 806. If this determination is “Yes”, the process proceeds to Step 820.

In step 820, an appropriate order is assigned to the last one distributed segment. As an example of giving priority, in the range of the last one distributed segment, if there is one or more distributed segments that have already been assigned an order, the last of the one or more distributed segments The next order of the order may be assigned to the processing target distributed segment. In addition, this invention is not limited to provision of this order. In this case, since the last processing target distributed segment has been processed, the processing ends.

In step 806, a distributed segment with the highest priority is acquired from among the distributed segments remaining as candidates for processing. As will be described later, when a subgraph is created in the subsequent step 8012, a distributed segment having the highest priority among the distributed segments belonging to the subgraph is acquired. It should be noted that this processing flow makes a recursive call, so that subgraphs may be nested. A method for creating a subgraph will be described in step 812.

In step 808, the acquired distributed segments are given priorities in the order of the distributed segments. As an example of giving priority, if there is one or more distributed segments that have already been given priority in the range of the obtained distributed segment, the last priority among the one or more distributed segments The next priority may be given to the obtained distributed segment. In addition, this invention is not limited to provision of this priority.

In step 810, the distributed segment to which the priority is given in the immediately preceding step 808 may be excluded from the processing target. If priority is given to a part of a distributed segment, the distributed segment is deleted if priority is not given to all parts (all queries) including other parts of the distributed segment. You may leave without. This is because the plurality of distributed segments may overlap partially.

In step 812, a subgraph is created in the range of the distributed segment deleted in the immediately preceding step 810. The subgraph refers to a distributed segment that exists in the range of the deleted distributed segment among the distributed segments that remain as processing targets. If a part of the distributed segment is included in this range, a part of the distributed segment that is included in this range is included in the subgraph. If the subgraph to be created already exists, there is no need to create a new subgraph. If the already created subgraph is an empty set, the subgraph is deleted. As already mentioned, it should be noted that this process is called recursively, so multiple subgraphs may be nested. When a subgraph is deleted, the subgraph may exist in a shallower nest. In this case, what is necessary is just to continue a process regarding the distributed segment which exists in the subgraph. If there is no subgraph, the process moves to step 814.

In step 814, in order to perform distributed segment prioritization processing for other ranges, the priority to be assigned to the distributed segment is reset, and step 802 is recursively called for the remaining distributed segments.

Priority is given to distributed segments by the above processing. The result of specific prioritization of the distributed segments will be described with reference to FIG. The present invention is not limited to the above prioritization.

FIG. 9 shows an embodiment relating to extraction of distributed segments.

FIG. 9A shows a query graph having a query Q1, a query Q2, and a query Q3. The query Q1 has K1 and K2 as distribution keys. The query Q2 has K1, K2, and K3 as distribution keys. Query Q3 has K1 and K3 as distribution keys.

FIG. 9B is a table in which distributed segments extracted by applying the already described distributed segment extraction method are arranged in correspondence with queries. In this table, the segment column to which the segment belongs is lined up with segments with higher priority in order from the left.

FIG. 9C is a table in which queries are arranged in correspondence with the extracted distributed segments. The information in the table of FIG. 9B and the table of FIG. 9C is the same. The distributed segment S1 constitutes a query subset including the query Q1 and the query Q2. The distributed segment S2 includes a query Q2. The distributed segment S3 includes a query Q2 and a query Q3. The distributed segment S4 includes a query Q1, a query Q2, and a query Q3.

FIG. 10 illustrates the deployment of queries to nodes that are distributed in parallel using distributed segments, according to one embodiment.

FIG. 10A shows the distributed segments (S1, S2, S3, S4) arranged in order of priority. Regarding priorities, the distributed segment S4 is the highest and the distributed segment S2 is the lowest. For this prioritization, an evaluation function is used in which the priority becomes higher as the segment length (number of queries) becomes longer.

FIG. 10B shows an example in which the distributed segment S4 (1000) having the highest priority is adopted and the queries (Q1, Q2, Q3) belonging to the distributed segment S4 are deployed.

FIG. 10C shows an example in which a query (Q1, Q2, Q3) is arranged in each node in units of the distributed segment S4, and the nodes 1000-1, 1000-2 to 1000-N are distributed in parallel. Yes. The number of parallel distributions is N. At point 1020, the input stream 1010 is distributed to N nodes. In this distribution, the input stream may be distributed by applying K1 which is a common distributed key set of the distributed segment S4 to a hash function that outputs N hash values. The distributed stream is processed in the order of the query Q1, the query Q2, and the query Q3 in each node. The processing results of the nodes are combined at point 1030 to obtain a processing result 1040.

FIG. 11 shows an example in which the processing load of the query Q1 (1110) of the node 1 and the query 1130 of the node 3 is large as a result of actually profiling the input stream according to the embodiment, and redistribution is performed. Show. As shown in the figure, node 1.2 is added and node 1.1 is redistributed. In addition, node 3.2 is added and redistribution of node 3.1 is performed. Details thereof will be described below.

In this specification, for the sake of easy understanding, the data flow between each query is described using each event of the input stream. In actual processing, the processing result of the query belonging to the preceding query can be provided to the succeeding query for the succeeding query located after the certain preceding query.

When performing re-distribution, the redistribution key is different, so re-hashing occurs and communication increases by a certain amount.However, this does not occur at all nodes, but at a specific node (or to a specific hash value) The impact of this communication is local.
For this reason, when considering redistribution, it is desirable to confirm that there is a sufficient bandwidth in the network to which the redistributed nodes are directly connected.

Consider the case of distributing the processing of query Q1 (1110). Search for the segment with the next highest priority, including query Q1. Since it can be seen that the segment S1 meets this condition, the queries Q1 and Q2 belonging to the segment S1 are deployed to the new node 1.2 and redistributed.

When performing redistribution, it is desirable that the node management mechanism establishes a virtual node called node 1 so as not to affect the hash function H1 that distributes the stream to N at the point 1190. Then, the virtual node may be distributed to the physical nodes of node 1.1 and node 1.2. In the node 1.2, a query Q1 (1115) and a query Q2 (1125) belonging to the segment S1 are provided. As a result, the processing of the query Q1 (1110) and the query Q2 (1120) of the node 1.1 is distributed. Since the number of nodes has increased, it is necessary to perform distribution at the point 1195 by applying the distributed key set (K1, K2) to the hash H2 that outputs two hash values. The output of the query Q2 (1125) needs to be given to the query Q3 (1121). Therefore, it can be seen that the communication amount increases to some extent by adding the node 1.2.

Also, in order to reduce the processing load of the query Q3 (1130), a segment including the query Q2 and having the next highest priority is searched. Since it can be seen that the segment S3 meets this condition, the queries Q2 and Q3 belonging to the segment S3 are deployed to the new node 3.2 and redistribution is performed. In the node 3.2, a query Q2 (1135) and a query Q3 (1345) belonging to the segment S3 are provided. As a result, the processing of the query Q2 (1130) and the query Q3 (1140) of the node 3.1 is distributed. Since this increases the number of nodes, it is necessary to distribute the hash function H2 that outputs two hash values by applying the distributed key set (K1, K3). The outputs of the query Q3 (1140) and the query Q3 (1145) need to be given to the point 1196. Therefore, it can be seen that the communication amount increases to some extent by adding the node 3.2.

FIG. 11B shows an example of a hash function. Note that K1% N means that the distributed key set K1 is applied to a hash function that outputs N hash values.

As described above, it can be seen that by performing re-distribution, the amount of communication increases to some extent, but more appropriate load distribution can be performed.

FIG. 12 illustrates the extraction of distributed segments in one embodiment.

FIG. 12A shows a query graph, which indicates that the queries Q1 to Q6 each have the illustrated distributed key set.

FIG. 12B is a table showing the distributed segments to which each query belongs.

FIG. 12C is a table showing queries belonging to each segment.

FIG. 13 illustrates the deployment of queries to parallel distributed nodes according to one embodiment.

FIG. 13 (A) shows the distributed segments (S1, S2, S3, S4, S5) arranged in order of priority. Regarding priorities, the distributed segments S1 and S5 are the highest and the distributed segment S4 is the lowest. For this prioritization, an evaluation function is used in which the priority increases as the segment length (number of queries) increases. After assigning S1, the longest segment in the subgraph (Q5, Q6) excluding the subgraph (Q1, Q2, Q3, Q4) of S1 is S5 instead of S4, so S5 is assigned to Q5 and Q6. It is desirable to assign priority. Therefore, the priority of S5 is high.

FIG. 13B employs distributed segments S1 and S5 having the highest priority, and queries (Q1, Q2, Q3, Q4) belonging to the distributed segment S1 and queries (Q5, Q5) belonging to the distributed segment S5. An example in which Q6) is deployed is shown.

In FIG. 13C, in the distributed segment S1, a query (Q1, Q2, Q3, Q4) is allocated to node 1 to node N / 2, and a query (Q5, Q6) is assigned to node N / 2 + 1 to node N. An example of deployment and parallel distribution is shown.

In this case, for example, the output result of Q4 (1301) of node 1 needs to be distributed to node N / 2 + 1 or node N by applying distributed key set K3 to a hash function that outputs N / 2 hash values. is there. This is because the distributed key set K1 of the segment S1 and the distributed key set K3 of the segment S5 are different. In this way, communication occurs between nodes at the segment boundary, but the amount of communication can be drastically reduced as compared with the case where a node is assigned for each query.

FIG. 14 shows a functional block diagram of an embodiment, which includes elements constituting the system.

This embodiment includes a query information acquisition unit 1420, a distributed key set extraction unit 1430, a distributed segment generation unit 1440, a parallel distribution unit 1450, a profile acquisition unit 1460, a profile evaluation unit 1465, and a query / node specification unit 1470.

First, the query information acquisition unit 1420 accepts the data stream program 1410. The query information acquisition unit 1420 recognizes a plurality of queries and a connection relationship between each of the plurality of queries. Then, the result is passed to the distributed key set extraction unit 1430.

The distributed key set extraction unit 1430 extracts a distributed key set that can be used when hashing the data stream and executing each of the plurality of queries in parallel with each of the plurality of queries.

The distributed segment generation unit 1440 generates a distributed segment to which a series of queries having a common distributed key set belong. In principle, at least one query belonging to the distributed segment is contiguous. Exceptionally, it may be desirable to have multiple queries that send output to the same query belong to the same segment, as in the case shown in FIG.

The distributed segment generation unit 1440 may include a distributed segment priority assigning unit 1445 and a distributed segment selection unit 1446.

The distributed segment priority assigning unit 1445 assigns a priority to each distributed segment using a predetermined evaluation function. As the predetermined evaluation function, the longer the distributed segment length (the number of queries to which it belongs), the higher the evaluation for the distributed segment may be. Or you may give high evaluation, so that there are many common distributed key sets which a distributed segment has. Or you may give high evaluation, so that the weight of the process of the query which belongs to a distributed segment is heavy.

The distributed segment selection unit 1446 selects a distributed segment to be used from one or more distributed segments based on the priority assigned to the distributed segment.

The parallel distribution unit 1450 distributes a plurality of nodes in parallel based on a hash function applied to a common distribution key set corresponding to the distributed segment, in order to execute a query belonging to the distributed segment to be used in parallel. Each of the plurality of nodes distributed in parallel is provided with a query belonging to the distributed segment. Note that an appropriate number of nodes may be determined based on the number of existing physical machines.

Through the above processing, a query is deployed to a plurality of nodes distributed in parallel.

Furthermore, the profile acquisition unit 1460 performs profiling when the data stream is processed. Through this profiling, an execution profile of each query and / or each node is obtained.

The profile evaluation unit 1465 can check whether a load is concentrated on a specific query and / or a specific node by evaluating the profile. As a measure of load concentration, various information including the number of events per unit time in each query and / or each node, CPU load, memory usage, processing capability, and the like may be acquired. Note that the present invention is not limited to the above exemplary description.

The query / node specifying unit 1470 uses a predetermined evaluation function to specify a query or node where the load is concentrated. This result is given to the re-parallel distribution unit 1455.

The re-parallel distribution unit 1455 may exist in the parallel distribution unit 1450. The re-parallel distribution unit 1455 includes a new node in which a new segment to which all or part of a specific query or a query deployed in the specific node belongs, a node in which the specific query exists, or the specific Redistribute the nodes again in parallel.

FIG. 15 shows a configuration example of hardware (computer) according to the embodiment of the present invention. The hardware includes a CPU 1510, a memory 1515, an input device 1520, an output device 1525, an external storage device 1530, a portable recording medium drive device 1535, and a network connection device 1545. Each device is connected by a bus 1550. Further, the portable recording medium driving device 1535 can read and write the portable recording medium 1540. A network 1560 is connected to the network connection device 1545.

It should be noted that all or part of the present embodiment can be implemented by a program. This program can be stored in the portable recording medium 1540. The portable recording medium 1540 refers to one or more non-transitory, tangible storage media having a structure. Illustrative examples of the portable recording medium 1540 include a magnetic recording medium, an optical disk, a magneto-optical recording medium, and a nonvolatile memory. Magnetic recording media include HDDs, flexible disks (FD), magnetic tapes (MT) and the like. Optical disks include DVD (Digital Versatile Disc), DVD-RAM, CD-ROM (Compact Disc-Read Only Memory), CD-R (Recordable) / RW (ReWritable), and the like. Magneto-optical recording media include MO (Magneto-Optical disk). All or part of the embodiments of the present invention can be implemented by reading a program stored in a portable recording medium and executing it by a CPU.

1410 Data stream program 1420 Query information acquisition unit 1430 Distributed key set extraction unit 1440 Distributed segment generation unit 1445 Distributed segment priority assignment unit 1446 Distributed segment selection unit 1450 Parallel distribution unit 1455 Re-parallel distribution unit 1460 Profile acquisition unit 1465 Profile evaluation unit 1470 Query / node identification part

Claims

A program for deploying a plurality of queries for processing a data stream to a plurality of nodes, each of the plurality of nodes having an ability to provide a result of executing the deployed query to a subsequent node or output. And
Obtaining a connection relationship between each of the plurality of queries and the plurality of queries;
Extracting a distributed key set that can be used when hashing the data stream and executing each of the plurality of queries in parallel with each of the plurality of queries;
A distributed segment to which a query subset of the plurality of queries belongs, having a common distributed key set that the query subset has in common;
A node is distributed in parallel corresponding to the distributed segment, and the query subset belonging to the segment is deployed in each of one or more of the parallel distributed nodes.
A program that causes a computer to execute processing.
The process of generating the distributed segment is as follows:
Giving each one or more of the distributed segments a priority based on a predetermined first rule;
Selecting a distributed segment for use from one or more of the distributed segments based on the priority;
The program according to claim 1 including processing.
Obtaining a profile of each of the plurality of queries and / or each of the plurality of nodes as the plurality of queries deployed on the plurality of nodes process the data stream;
Evaluating the profile based on a predetermined second rule;
Identify a specific query or a specific node based on the evaluation;
Let the computer execute the process further,
The parallel distributed processing is:
A new node in which a new segment to which all or a part of the specific query or a query deployed in the specific node belongs and a node in which the specific query exists or the specific node is parallel again Distributed, including processing,
The program according to claim 1 or 2. (Fig. 4)
The program according to any one of claims 1 to 3, wherein the query subset includes a series of queries that are continuously processed.
A method for deploying a plurality of queries for processing a data stream to a plurality of nodes, each of the plurality of nodes having an ability to provide a result of executing the deployed query to a subsequent node or output. And
Obtaining a connection relationship between each of the plurality of queries and the plurality of queries;
Extracting a distributed key set that can be used when hashing the data stream and executing each of the plurality of queries in parallel with each of the plurality of queries;
A distributed segment to which a query subset of the plurality of queries belongs, having a common distributed key set that the query subset has in common;
A node is distributed in parallel corresponding to the distributed segment, and the query subset belonging to the segment is deployed in each of one or more of the parallel distributed nodes.
A method having processing.
The process of generating the distributed segment is as follows:
Giving each one or more of the distributed segments a priority based on a predetermined first rule;
Selecting a distributed segment for use from one or more of the distributed segments based on the priority;
6. The method of claim 5, comprising processing.
Obtaining a profile for each of the plurality of queries and / or each of the plurality of nodes as the plurality of queries deployed on the plurality of nodes process the data stream;
Evaluating the profile based on a predetermined second rule;
Identify a specific query or a specific node based on the evaluation;
Have processing,
The parallel distributed processing is:
A new node in which a new segment to which all or a part of the specific query or a query deployed in the specific node belongs and a node in which the specific query exists or the specific node is parallel again Distributed, including processing,
The method according to claim 5 or 6.
The method according to any one of claims 5 to 7, wherein the query subset includes a series of queries that are continuously processed.
A system for deploying a plurality of queries for processing a data stream to a plurality of nodes, each of the plurality of nodes having an ability to provide a result of executing the deployed query to a subsequent node or output. And
A query information acquisition unit for acquiring a connection relationship between each of the plurality of queries and the plurality of queries;
A distributed key set extraction unit that extracts a distributed key set that can be used when hashing the data stream and causing each of the plurality of queries to be executed in parallel; and
A distributed segment generation unit that generates a distributed segment to which a query subset of the plurality of queries belongs, and has a common distributed key set that the query subset has in common;
A parallel distribution unit in which nodes are distributed in parallel corresponding to the distributed segments, and each of the one or more parallel distributed nodes is provided with the query subset belonging to the segment; and
Having a system.
The distributed segment generation unit
A distributed segment priority assigning unit that assigns a priority to each of the one or more distributed segments based on a predetermined first rule;
A distributed segment selector that selects, for use, a distributed segment from one or more of the distributed segments based on the priority;
10. The system of claim 9, comprising:
A profile acquisition unit that acquires a profile of each of the plurality of queries and / or each of the plurality of nodes when the plurality of queries deployed in the plurality of nodes process the data stream;
A profile evaluation unit that evaluates the profile based on a predetermined second rule;
A query / node identification unit that identifies a specific query or a specific node based on the evaluation;
Have
The parallel distribution unit is:
A new node in which a new segment to which all or a part of the specific query or a query deployed in the specific node belongs and a node in which the specific query exists or the specific node is parallel again Including a re-parallel distribution unit,
The system according to claim 9 or 10.
The system according to any one of claims 9 to 11, wherein the query subset includes a series of queries in which processing is continued.