JP2015011484A - Data processing method and computer - Google Patents

Abstract

PROBLEM TO BE SOLVED: To improve the processing efficiency of stream data processing for coping with array type data.SOLUTION: There is provided a data processing method in a computer for executing stream data processing. The computer includes: a query analysis unit for generating the information of a query graph indicating the configuration of an operator in stream data processing; a query construction unit for generating an operator execution unit on the basis of the information of the query graph; and a query execution unit for processing the input stream data. The method includes: a step in which the query construction unit selects a processing object operator; a step in which the query construction unit analyzes the processing content of the operator, and determines a data structure in a data storage region allocated to an operator execution unit corresponding to the operator on the basis of the result of the analysis; a step in which the query construction unit generates the operator execution unit on the basis of the result of the analysis and the information of the data structure; and a step in which the query construction unit registers the operator execution unit in the query execution unit.

Description

  The present invention relates to a data processing method for improving the processing efficiency of stream data processing that handles array type data.

  In recent years, the amount of data to be processed has increased, and a stream data processing system that can aggregate and analyze data in real time has attracted attention. The stream data processing system targets stream data. Here, the stream data is a data sequence in chronological order that arrives without interruption.

  In the stream data processing system, predetermined stream data processing is executed according to a predefined query. A query is a scenario indicating the data to be processed and the processing content. An example of a computer language that defines a query is disclosed in Non-Patent Document 1.

  In a stream data processing application, there are cases where numerical calculation such as time series prediction calculation by regression analysis is required. Since such calculation is vector calculation or matrix calculation, it is defined as a process using array type data.

  Although it is not stream data, SQL99, which is a standard query language for stocked data, defines an array type column specification that represents array type data as a single column value. This specification can also be applied to a query language for stream processing. Similarly, a technique described in Non-Patent Document 2 is known as a technique for efficiently processing stocked multidimensional array data. Non-Patent Document 2 describes SciDB, which is a data management system that manages a large scale of array data.

A. Arasu, S .; Babu and J.M. Widom, “The CQL Continuous Query Language: Semantic Foundations and Query Execution”, 2003 Paul G. Brown, “Overview of sciDB: large scale array storage, processing and analysis”, In Proc. of SIGMOD 2010, 963-968, 2010

  Conventionally, in order to realize high-speed stream data processing that handles array-type data, the cost of array search processing and the efficiency of use of memory are problematic. Here, a conventional problem will be described using an example of calculation processing for calculating the product of two matrices.

  FIG. 11 is a diagram illustrating an example of a query for realizing a calculation process for calculating a product of two conventional matrices. In the query shown in FIG. 11, an arithmetic process for calculating the product of the matrix A and the matrix B is executed.

  In the query shown in FIG. 11, matrix data is managed in a data structure including a plurality of tuples configured from row columns, column columns, and matrix component value columns. Therefore, there is a problem in that the use efficiency of the memory is low because the amount of data increases as compared with general matrix representation data.

  In addition, in the arithmetic processing based on the query, it is necessary to search for a combination in which the column value of the matrix A and the column value of the row of the matrix B match. Therefore, as the number of rows and columns increases, the cost of search processing increases, and there is a problem in that the performance of arithmetic processing decreases accordingly.

  The following conventional techniques are known as methods for solving the above two problems.

  One method is a method using an index that specifies a combination of a column value of a column of the matrix A and a column value of a row of the matrix B in advance. By using the index, it is possible to suppress an increase in the cost of the search process described above. However, since it is necessary to secure a memory area for storing the index, there is a problem in that the memory utilization efficiency is further reduced.

  The other method is a method using an array type column similar to the above-mentioned SQL99. FIG. 12 is a diagram illustrating an example of a query using an array type column. In the query shown in FIG. 12, the matrix components themselves are managed as an array. A query using an array type column has the following problems.

  One problem is that the arithmetic processing associated with the data update of one matrix component cannot be performed efficiently. For example, when one of the matrix components is updated after the matrix product is calculated, the product component that is changed by the update is limited to a component whose row or column matches the updated component. Thus, in order to obtain the updated matrix product, it is only necessary to execute the arithmetic processing of the changed component. However, in the array type column, the entire matrix corresponds to one column, and therefore the array type column (matrix) itself is updated as one matrix component is updated. It is necessary to calculate the matrix product anew. Therefore, there is a problem that the arithmetic processing related to the updating of some values cannot be performed efficiently.

  Another problem is the need to define built-in functions for calculating matrix products. When using an array type column, it is necessary to define a new function necessary for the operation. In some cases, a function that is not prepared in advance as a library is required. Therefore, there is a problem that development efficiency is poor.

  The present invention has been made in view of the above-described problems. That is, an object of the present invention is to provide a data processing method and computer capable of realizing high-speed stream data processing for handling array type data.

  A typical example of the invention disclosed in the present application is as follows. That is, a data processing method in a computer that executes stream data processing, wherein the computer includes a processor, a memory connected to the processor, and a network interface connected to the processor. The stream data that is the target of the stream data processing includes a plurality of tuples composed of a plurality of columns, and the computer has at least one for realizing the stream data processing. Based on the information of the query graph, the query analysis unit that generates the information of the query graph indicating the configuration of the operator in the stream data processing by receiving the analysis scenario including the query of, and analyzing the analysis scenario, Corresponding to the operator A query construction unit that generates an operator execution unit that executes processing, and a query execution unit that processes input stream data based on the operator execution unit generated by the query construction unit, and the method A first step in which the query construction unit selects an operator to be processed from the plurality of operators included in the stream data processing, and the query construction unit, based on information in the query graph, A second step of analyzing a processing content of the selected operator and determining a data structure in a data storage area allocated to the operator execution unit corresponding to the selected operator based on a result of the analysis; The query construction unit selects the selection based on the analysis result and the information on the determined data structure. A third step of generating an operator execution unit corresponding to the generated operator, and a fourth step of the query construction unit registering the generated operator execution unit in the query execution unit. And

  According to the present invention, since the data structure of the array data can be optimized for each operator included in the stream data processing, the calculation processing cost can be reduced and the memory utilization efficiency can be improved. Therefore, it is possible to speed up the stream data processing.

  Problems, configurations, and effects other than those described above will be clarified by the following description of embodiments.

It is a block diagram which shows the structural example of the stream data processing system in Example 1 of this invention. It is a block diagram which shows the structural example of the stream data processing part in Example 1 of this invention. It is explanatory drawing which shows an example of the analysis scenario of Example 1 of this invention. It is explanatory drawing which shows an example of the query graph structure information of Example 1 of this invention. It is a flowchart explaining the process which the stream data processing part of the Example of this invention performs at the time of registration of an analysis scenario. It is a flowchart explaining the process which the Partition window optimization part of Example 1 of this invention performs. It is a flowchart explaining the process which the Join operator optimization part of Example 1 of this invention performs. It is a flowchart explaining the process which the GroupBy operator optimization part of Example 1 of this invention performs. It is explanatory drawing which shows the flow of the data between operators of Example 1 of this invention. It is explanatory drawing which shows the flow of the data between operators in Example 1 of this invention. It is a figure which shows an example of the query for implement | achieving the arithmetic processing which calculates the product of the conventional two matrices. It is a figure which shows an example of the query using the conventional array type | mold column.

  Embodiments of the present invention will be described below with reference to the accompanying drawings.

  FIG. 1 is a block diagram showing a configuration example of a stream data processing system in Embodiment 1 of the present invention.

  The stream data processing system includes a stream data processing server 100 and a plurality of host computers 120, 130, and 140.

  The stream data processing server 100 is connected to a plurality of host computers 120, 130, and 140 via a network 150. Note that the network 150 may be a wide area network (WAN), a local area network (LAN), or the like, but the present invention is not limited to the connection type of the network 150.

  The stream data processing server 100 receives the stream data transmitted from the host computer 120, processes the stream data according to the instructed query, and outputs the processing result as an output tuple 141.

  The stream data includes a plurality of input tuples 121. In this embodiment, the input tuple 121 includes a value column and an index column for specifying the value. It is assumed that the input tuple 121 includes one or more index columns. The input tuple 121 may include a time stamp column.

  The stream data processing server 100 includes a processor 101, a memory 102, a network interface 104, and a storage device 105, and each component is connected via a bus 103.

  The processor 101 executes various processes by executing a program stored in the memory 102. In the following description, when processing is described mainly with a program, it indicates that the program is being executed by the processor 101.

  The memory 102 stores a program executed by the processor 101 and information necessary for executing the program. Specifically, the memory 102 stores a program for realizing the stream data processing unit 110.

  The stream data processing unit 110 processes stream data. Details of the stream data processing unit 110 will be described later with reference to FIG.

  The storage device 105 stores stream data, queries, and other information. The storage device 105 may be, for example, a storage medium such as an HDD (Hard Disk Drive) or an SSD (Solid State Drive). The present invention is not limited to the type of storage medium.

  Note that the program and information stored in the memory 102 may be stored in the storage device 105. In this case, the processor 101 reads a program and information from the storage device 105 and loads the read program and information into the memory 102.

  The host computers 120, 130, and 140 are computers used by users who use the stream data processing server 100, and include a processor (not shown), a memory (not shown), and a network interface (not shown).

  The host computer 120 transmits stream data (input tuple 121) to be processed by the stream data processing server 100. The input tuple 121 may be transmitted from a system including a plurality of host computers 120.

  The host computer 140 receives the output tuple 141 processed by the stream data processing server 100. The output tuple 141 may be received by a system including a plurality of host computers 140.

  The host computer 130 executes a program for realizing the query registration interface 131. The query registration interface 131 is an interface for registering the analysis scenario 132 and instructing execution of the analysis scenario 132.

  Here, the analysis scenario 132 includes one or more queries. The query also includes one or more operators. The operator indicates the minimum unit processing in the stream data processing. The operator corresponds to, for example, data extraction processing, data aggregation processing, and the like.

  The analysis scenario 132 of this embodiment includes stream definition information that defines one or more input streams, and query definition information that defines one or more queries. Details of the analysis scenario 132 will be described later with reference to FIG.

  When the analysis scenario 132 is input, the stream data processing server 100 analyzes the analysis scenario 132 and configures a query graph for executing predetermined stream data processing. The stream data processing server 100 processes the stream data (input tuple 121) according to the query graph.

  FIG. 2 is a block diagram illustrating a configuration example of the stream data processing unit 110 according to the first embodiment of this invention.

  The stream data processing unit 110 includes a query parser 201, a query construction unit 203, a query execution unit 210, a tuple input unit 221, and a tuple output unit 222. In addition, the stream data processing unit 110 holds query graph configuration information 202.

  The query parser 201 receives an input of the analysis scenario 132, analyzes the received analysis scenario 132, and generates a query graph based on the analysis result. Information about the generated query graph is held as query graph configuration information 202. Details of the query graph configuration information 202 will be described later with reference to FIG.

  As described above, the analysis scenario includes one or more operators. Therefore, the query graph is composed of one or more operators.

  In the example illustrated in FIG. 2, a query graph including an operator 1, an operator 2, an operator 3, an operator 4, an operator 5, and an operator 6 is illustrated. The present invention is not limited to the processing content of the operator.

  The query construction unit 203 generates an operator execution unit 211 based on the query graph configuration information 202, and outputs the generated operator execution unit 211 to the query execution unit 210.

  The operator execution unit 211 is a module that executes specific processing of the operator included in the query graph. The operator execution unit 211 includes setting information necessary for execution of processing, such as processing target tuples, columns included in the tuples, and functions used for processing. In the present embodiment, setting information of the operator's data storage area is further included.

  Here, the data storage area is a storage area in which the operator execution unit 211 corresponding to the operator temporarily holds tuples. The data storage area setting information is information for setting the structure of data held in the data storage area. For example, information for setting the data structure of a multidimensional array can be considered. An example of the setting information of the data storage area will be described later with reference to FIG.

  The query construction unit 203 includes a query optimization unit 204. The query optimization unit 204 analyzes the analysis scenario 132 based on the query graph configuration information 202 and determines a management method for the data storage area of each operator.

  In the present embodiment, as operators included in the query graph, an operator corresponding to a Partition window operation, a Join operator, and a GroupBy operator are assumed. Therefore, the query optimization unit 204 includes a Partition window optimization unit 205, a Join operator optimization unit 206, and a GroupBy operator optimization unit 207.

  The partition window optimization unit 205 determines a management method of the data storage area of the operator corresponding to the partition window calculation. The Join operator optimization unit 206 determines a management method for the data storage area of the Join operator. The GroupBy operator optimization unit 207 determines a method for managing the data storage area of the GroupBy operator.

  In this embodiment, the query optimization unit 204 determines a method for managing the data storage area of each operator and outputs the processing result. The query construction unit 203 generates an operator execution unit 211 corresponding to each operator based on the processing result.

  Details of processing executed by the query optimization unit 204 will be described later with reference to FIGS.

  The tuple input unit 221 receives input of stream data (input tuple 121). Further, the tuple input unit 221 outputs the received stream data (input tuple 121) to the query execution unit 210.

  The query execution unit 210 executes stream data processing using the input tuple 121 and outputs the processing result to the tuple output unit 222.

  The query execution unit 210 includes an operator execution unit for each operator constituting the query graph. In the example illustrated in FIG. 2, the query execution unit 210 includes an operator 1 execution unit 211-1, an operator 2 execution unit 211-2, an operator 3 execution unit 211-3, an operator 4 execution unit 211-4, and an operator 5 execution unit. 211-5 and an operator 6 execution unit 211-6.

  In the following description, the operator execution units 211-1, 211-2, 211-3, 211-4, 211-5, and 211-6 are also referred to as operator execution units 211 when not distinguished.

  The tuple output unit 222 transmits the processing result output from the query execution unit 210 to the host computer 140 as an output tuple 141.

  FIG. 3 is an explanatory diagram illustrating an example of the analysis scenario 132 according to the first embodiment of this invention.

  The analysis scenario 132 shown in FIG. 3 is composed of stream definition information expressing each component of the matrix A and the matrix B, and one query definition information for calculating the product of both matrices.

  The column “i” represents an index corresponding to a matrix row, the column “j” represents an index corresponding to a matrix column, and the column “v” represents a matrix component value. The column “A.i” represents the column “i” of the matrix A, the column “A.j” represents the column “j” of the matrix A, and the column “A.v” represents the column “v” of the matrix A. The same applies to the matrix B.

  In the query C shown in FIG. 3, the following matrix product operations are defined.

  First, only one value of the column “v” is extracted for the combination of the column “i” and the column “j” of the stream data A according to the designation of the PARTION window. Similarly, only one value of the column “v” is extracted for the combination of the column “i” and the column “j” of the stream data B.

  Next, tuples matching the columns “A.j” and “B.i” are combined according to the specification of the WHERE clause.

  Next, tuples are grouped for each combination of the columns “A.i” and “B.j” according to the specification of the GROUP BY clause, and the tuple columns “A.v” within the group according to the specification of the SELECT clause. ”And the column“ Bv ”is calculated.

  In this embodiment, a character string “ARRAY_INDEX” that specifies that the column corresponding to the index is an array index type column is used in the REGISTER STREAM clause that defines the input stream.

  Here, the array index type column represents a column whose value range is an integer that continues from 1. In the following description, columns other than the array index type column are also referred to as normal type columns.

  In the example illustrated in FIG. 3, the column “i” and the column “j” of the input stream A and the column “i” and the column “j” of the input stream B are designated as the array index type columns.

  FIG. 4 is an explanatory diagram illustrating an example of the query graph configuration information 202 according to the first embodiment of this invention.

  The query graph configuration information 202 includes a plurality of information (tables). Specifically, the query graph configuration information 202 includes operator management information 410, Partition window management information 420, Join operator management information 430, and GroupBy operator management information 440.

  The operator management information 410 is information for managing an overview of all operators included in the query graph. The operator management information 410 includes an operator ID 411, a type 412, and an input 413.

  The operator ID 411 is identification information for uniquely identifying an operator included in the query graph.

  Type 412 is information indicating the type of operator. In this embodiment, information of “Partition”, “Join”, or “GroupBy” is stored in the type 412.

  The input 413 is data identification information input to the operator.

  The partition window management information 420 is detailed information of an operator corresponding to the partition window calculation. The Partition window management information 420 includes an operator ID 421, an input column 422, a Grouping column 423, an Array Index column 424, and a window size 425.

  The operator ID 421 is the same as the operator ID 411.

  The input column 422 is column identification information included in the tuple input to the operator corresponding to the Partition window calculation.

  A Grouping column 423 is identification information of a column to be grouped in the Partition window calculation. In the Partition window operation, a plurality of subsets (groups) are generated from a set of a plurality of tuples.

  The Array Index column 424 is column identification information corresponding to the sequence index type column.

  The window size 425 indicates the number of elements included in the subset. The example shown in FIG. 4 shows that only one tuple can survive at the same time for the combination of the column “i” and the column “j”. Therefore, after a tuple of “1” in column “i” and “2” in column “j” is input, a new tuple of “1” in column “i” and “2” in column “j” is newly created. When entered, only new tuples can exist.

  Join operator management information 430 is detailed information of the Join operator. The join operator management information 430 includes an operator ID 431, an input column 432, a join column 433, and an array index column 434.

  The operator ID 431 is the same as the operator ID 411.

  The input column 432 is column identification information included in the tuple input to the Join operator.

  The Join column 433 is identification information of a column to be combined.

  The Array Index column 434 is column identification information corresponding to the sequence index type column.

  The GroupBy operator management information 440 is detailed information of the GroupBy operator. The GroupBy operator management information 440 includes an operator ID 441, an input column 442, a Grouping column 443, and an Array Index column 444.

  The operator ID 441 is the same as the operator ID 411.

  The input column 442 is column identification information included in the tuple input to the GroupBy operator.

  The Grouping column 443 is identification information of a column to be grouped by the GroupBy operator.

  The Array Index column 444 is column identification information corresponding to the sequence index type column.

  FIG. 5 is a flowchart for describing processing executed by the stream data processing unit 110 according to the first embodiment of this invention when the analysis scenario 132 is registered.

  Upon receiving the analysis scenario 132 (step S501), the stream data processing unit 110 generates the query graph configuration information 202 based on the analysis scenario 132 (step S502). Specifically, the following processing is executed. Here, it is assumed that the analysis scenario 132 shown in FIG. 3 has been received.

  First, the query parser 201 identifies an operator included in the query C based on the analysis scenario 132. In this embodiment, the query parser 201 specifies an operator corresponding to two Partition window operations, one Join operator, and one GroupBy operator from the definition information of the query C.

  The query parser 201 initializes the operator management information 410, the partition window management information 420, the join operator management information 430, and the GroupBy operator management information 440.

  The query parser 201 gives identification numbers “1” to “4” to the specified operator. For example, the query parser 201 gives an identification number to the execution order of the operators.

  The query parser 201 generates an entry in the operator management information 410, stores the identification number assigned to the operator ID 411 of the generated entry, and stores the identification information of each operator in the type 412.

  Further, the query parser 201 analyzes the input / output relationship of each operator from the definition information of the query C, and stores identification information of data input to the input 413 based on the analysis result. For example, an entry whose operator ID 411 is “1” is an operator corresponding to a Partition window operation in which the input stream A is input. Therefore, “Partition” is stored in the type 412, and “Stream A” is stored in the input 413. Stored.

  Next, the query parser 201 generates management information for each operator. In the present embodiment, Partition window management information 420, Join operator management information 430, and GroupBy operator management information 440 are generated.

  The partition window management information 420 is generated as follows.

  The query parser 201 specifies the columns included in the tuple based on the definition information of the input stream A. In addition, the query parser 201 specifies an array index type column based on the definition information of the input stream A. In this embodiment, the query parser 201 knows that the column i and the column j are array index type columns.

  Further, the query parser 201 specifies grouping conditions and a window size based on the FROM clause of the definition information of the query C.

  The query parser 201 generates an entry in the Partition window management information 420 and stores information necessary for the generated entry based on the analysis result described above. The same processing is executed for the operator corresponding to the Partition window calculation to which the input stream B is input.

  Join operator management information 430 is generated as follows.

  The query parser 201 specifies the columns included in the tuple and the join condition based on the definition information of the input stream A, the definition information of the input stream B, and the WHERE clause of the definition information of the query C.

  In addition, the query parser 201 identifies an array index type column from among the columns included in the tuple based on the definition information of the input stream A and the definition information of the input stream B.

  In this embodiment, since the column “i” and the column “j” are designated as the array index type column, the query parser 201 has the column “Ai”, the column “Aj”, the column “Bi”, It can also be seen that the column “Bj” is also an array index type column.

  The query parser 201 generates an entry in the join operator management information 430, and stores information necessary for the generated entry based on the analysis result described above.

  The GroupBy operator management information 440 is generated as follows.

  The query parser 201 specifies the columns included in the tuple and the grouping conditions based on the definition information of the input stream A, the definition information of the input stream B, and the GROUP BY clause of the definition information of the query C. In addition, the query parser 201 identifies an array index type column from among columns included in the tuple based on the definition information of the input stream A and the definition information of the stream data B.

  In this embodiment, since the column “i” and the column “j” are designated as the array index type column, the query parser 201 has the column “Ai”, the column “Aj”, the column “Bi”, It can also be seen that the column “Bj” is also an array index type column.

  The query parser 201 generates an entry in the GroupBy operator management information 440, and stores information necessary for the generated entry based on the analysis result.

  The above is the description of the process in step S502. The above-described processing is an example, and the present invention is not limited to this.

  Next, the stream data processing unit 110 starts an operator loop process (step S503). Specifically, the following processing is executed.

  The stream data processing unit 110 calls the query construction unit 203 and instructs the start of processing. The query construction unit 203 calls the query optimization unit 204 and instructs the start of processing.

  The query optimization unit 204 refers to the operator management information 410 of the query graph configuration information 202 and selects one operator to be processed from the operators included in the analysis scenario 132. Here, it is assumed that the entries are selected in order from the entry on the operator management information 410. The present invention is not limited to the operator selection method.

  The query optimization unit 204 determines whether or not the selected operator is an operator corresponding to the Partition window calculation (step S504).

  Specifically, the query optimization unit 204 determines whether or not the entry type 412 corresponding to the selected operator is “Partition”. When the type 412 is “Partition”, the query optimization unit 204 determines that the selected operator is an operator corresponding to the Partition window calculation.

  If it is determined that the selected operator is not an operator corresponding to the Partition window calculation, the query optimization unit 204 proceeds to step S506.

  When it is determined that the selected operator is an operator corresponding to the Partition window calculation, the query optimization unit 204 calls the Partition window optimization unit 205 and instructs the start of processing (step S505). The processing executed by the partition window optimization unit 205 will be described later with reference to FIG.

  The query optimization unit 204 waits until the processing result is output from the partition window optimization unit 205.

  The query optimization unit 204 determines whether or not the selected operator is a Join operator (step S506).

  Specifically, the query optimization unit 204 determines whether or not the entry type 412 corresponding to the selected operator is “Join”. When the type 412 is “Join”, the query optimization unit 204 determines that the selected operator is a Join operator.

  If it is determined that the selected operator is not a Join operator, the query optimization unit 204 proceeds to step S508.

  When it is determined that the selected operator is a Join operator, the query optimization unit 204 calls the Join operator optimization unit 206 and instructs the start of processing (step S507). The process executed by the Join operator optimization unit 206 will be described later with reference to FIG.

  The query optimization unit 204 waits until a processing result is output from the Join operator optimization unit 206.

  The query optimization unit 204 determines whether or not the selected operator is a GroupBy operator (step S508).

  Specifically, the query optimization unit 204 determines whether or not the type 412 of the entry corresponding to the selected operator is “GroupBy”. When the type 412 is “GroupBy”, the query optimization unit 204 determines that the selected operator is a GroupBy operator.

  If it is determined that the selected operator is not a GroupBy operator, the query optimization unit 204 proceeds to step S510.

  If it is determined that the selected operator is a GroupBy operator, the query optimization unit 204 calls the GroupBy operator optimization unit 207 and instructs the start of processing (step S509).

  The query optimization unit 204 waits until the processing result is output from the GroupBy operator optimization unit 207. After the processing result is output from the GroupBy operator optimization unit 207, the query optimization unit 204 displays the processing result output from the Partition window optimization unit 205, the Join operator optimization unit 206, or the GroupBy operator optimization unit 207. The data is output to the query construction unit 203, and the process ends.

  The query construction unit 203 generates the operator execution unit 211 based on the query graph configuration information 202 and the processing result output from the query optimization unit 204 (step S510). Specifically, the following processing is executed.

  The query construction unit 203 generates setting information necessary for the execution of processing based on the query graph configuration information 202, and based on the processing result output from the query optimization unit 204, the data storage area of the operator is stored. Generate configuration information.

  The query construction unit 203 generates an operator execution unit 211 based on the generated setting information. In addition, the query construction unit 203 registers the generated operator execution unit 211 in the query execution unit 210.

  The above is the description of step S510.

  Next, the query optimization unit 204 determines whether or not the processing has been completed for all operators included in the analysis scenario 132 (step S511).

  When it is determined that the processing has not been completed for all operators included in the analysis scenario 132, the query optimization unit 204 returns to step S503 and executes the same processing.

  When it is determined that the processing has been completed for all operators included in the analysis scenario 132, the query optimization unit 204 ends the processing.

  Note that the stream data processing unit 110 starts processing upon reception of the analysis scenario 132, but the processing start trigger is not limited to this. For example, the stream data processing unit 110 may store the received analysis scenario 132 in the storage apparatus 105 or the like, and may start the process when a process start instruction is input from a user or the like.

  FIG. 6 is a flowchart illustrating processing executed by the partition window optimization unit 205 according to the first embodiment of this invention.

  When the partition window optimization unit 205 is called from the query optimization unit 204, the partition window optimization unit 205 starts processing described below. At this time, the query optimization unit 204 outputs the identification information (operator ID) of the operator to be processed.

  The partition window optimization unit 205 refers to the partition window management information 420 included in the query graph configuration information 202, and determines whether or not an array index column is included in the grouping target column (step S601). Specifically, the following processing is executed.

  The partition window optimization unit 205 refers to the partition window management information 420 and searches for an entry that matches the operator identification information to which the operator ID 421 is input.

  The partition window optimization unit 205 compares the grouping column 423 and the array index column 424 of the searched entry, and the array index column 424 has one or more columns that match the columns stored in the grouping column 423. It is determined whether or not.

  The Grouping column 423 corresponds to the grouping target column, and the Array Index column 424 corresponds to the array index type column.

  If there is at least one column in the Array Index column 424 that matches the column stored in the Grouping column 423, the Partition window optimization unit 205 determines that the column to be grouped includes an array index type column. .

  The above is the description of the process in step S601.

  When it is determined that the column to be grouped does not include an array index type column, the Partition window optimization unit 205 outputs to the query optimization unit 204 as a processing result that the setting information of the data storage area is not necessary (Step S606), and the process ends.

  If it is determined that the column to be grouped includes an array index type column, the Partition window optimization unit 205 sets the column to be grouped as an array index type column and a normal type column based on the Partition window management information 420. (Step S602).

  The partition window optimization unit 205 determines whether or not the normal type column is included in the grouping target column based on the classification result (step S603).

  For example, when the identification information of all the columns stored in the Grouping column 423 is stored in the Array Index column 424, the Partition window optimization unit 205 includes the normal type column among the columns to be grouped. Judge that there is no.

  In the example illustrated in FIG. 4, the partition window optimization unit 205 determines that the normal column is not included in the grouping target columns.

  If it is determined that the normal column is not included in the grouping target column, the partition window optimization unit 205 determines the data structure in the data storage area (step S604). Thereafter, the partition window optimization unit 205 outputs information on the data structure in the determined data storage area as a processing result (step S606), and ends the processing.

  Specifically, the partition window optimization unit 205 determines to hold the tuple as an array having the same number of dimensions as the number of array index type columns.

  In the example illustrated in FIG. 4, the array index column 424 stores identification information of two columns of columns “i” and “j”. Therefore, the partition window optimization unit 205 determines to hold the tuple as a two-dimensional array.

  If it is determined that the normal type column is included in the grouping target column, the partition window optimization unit 205 determines the data structure in the data storage area (step S605). Thereafter, the partition window optimization unit 205 outputs information on the data structure in the determined data storage area as a processing result (step S606), and ends the processing.

  Specifically, the partition window optimizing unit 205 determines to hold the tuple as a hash map having a normal type column as a key and an array having the same number of dimensions as the number of array index type columns.

  For example, in the case of a tuple including an array index type column “i”, “j”, and a normal type column “k”, the partition window optimization unit 205 uses the normal type column “k” as a key and a two-dimensional array as a value. Decide to keep the tuple as a hash map.

  The data structure determined as described above is held in the operator execution unit 211 as setting information of the data storage area. The operator execution unit 211 temporarily stores data of each tuple according to the setting information when executing a query.

  The setting information includes a data storage area form (that is, a multi-dimensional array or a hash map) and information related to the multi-dimensional array or the hash map. In the case of a multidimensional array, the information about the multidimensional array or hash map includes the name of the array index type column of each dimension. In the case of a hash map, the name of the normal type column that is the key of the hash map, and the hash map Contains the name of the array index type column for each dimension of the multidimensional array that is the value.

  For example, the setting information in the above two examples includes (form = multidimensional array, array index column = “i”, “j”), and (form = hash map, hash key column = “k”, array index column = “I”, “j”).

  FIG. 7 is a flowchart illustrating processing executed by the Join operator optimizing unit 206 according to the first embodiment of this invention.

  When called from the query optimization unit 204, the Join operator optimization unit 206 starts processing described below. At this time, the query optimization unit 204 outputs the identification information (operator ID) of the operator to be processed.

  The Join operator optimizing unit 206 refers to the Join operator management information 430 included in the query graph configuration information 202, and determines whether or not an array index type column is included in the input schema (Step S701). Specifically, the following processing is executed.

  The Join operator optimizing unit 206 refers to the operator management information 410 and searches for an entry that matches the operator identification information to which the operator ID 411 is input.

  The Join operator optimizing unit 206 refers to the input 413 of the searched entry and acquires the identification information of the input destination operator. Also, the Join operator optimizing unit 206 specifies management information to be referenced based on the type 412 of the retrieved entry.

  In the example shown in FIG. 4, since the operator ID 411 is “1” and the type 412 “2” is “Partition”, the Join operator optimizing unit 206 refers to the Partition window management information 420.

  The Join operator optimization unit 206 refers to the Partition window management information 420 and searches for an entry that matches the acquired operator identification information. The Join operator optimizing unit 206 obtains column identification information stored in the Array Index column 424 of the searched entry.

  The Join operator optimizing unit 206 refers to the Join operator management information 430 and searches for an entry that matches the operator identification information to which the operator ID 431 is input.

  The Join operator optimization unit 206 acquires the identification information of the input schema column from the input column 432 of the searched entry.

  The Join operator optimization unit 206 compares the column identification information acquired from the Array Index column 424 with the column identification information acquired from the input column 432, and matches the column stored in the Array Index column 424. It is determined whether one or more columns exist in the input column 432.

  If there is at least one column in the input column 432 that matches the column stored in the Array Index column 424, the Join operator optimizing unit 206 determines that the input schema includes an array index type column.

  The above is the description of the processing in step S701.

  When it is determined that the array index type column is not included in the input schema, the Join operator optimizing unit 206 outputs to the query optimizing unit 204 as a processing result that the setting information of the data storage area is not necessary (Step (Step)). S704), the process ends.

  When it is determined that the array index type column is included in the input schema, the Join operator optimization unit 206 determines whether or not the join condition in the Join operator is a join condition related to the array index type column (step S702). Specifically, the following processing is executed.

  The Join operator optimizing unit 206 refers to the Array Index column 424 of the entry retrieved from the Partition window management information 420 and the Join column 433 of the entry retrieved from the Join operator management information 430.

  The Join operator optimization unit 206 determines whether or not the join condition stored in the Join column 433 is a join condition related to the array index type column.

  In the example illustrated in FIG. 4, the join column 433 stores a join condition related to the column “A.j” and the column “Bi” which are array index type columns. Therefore, the Join operator optimizing unit 206 determines that the join condition in the Join operator is a join condition related to the array index type column.

  The above is the description of the processing in step S702.

  When it is determined that the join condition in the Join operator is not a join condition related to the array index type column, the Join operator optimization unit 206 outputs to the query optimization unit 204 as a processing result that the setting information of the data storage area is not necessary. (Step S704), and the process ends.

  When it is determined that the join condition in the Join operator is a join condition related to the array index type column, the Join operator optimizing unit 206 determines the data structure in the data storage area (step S703). Thereafter, the Join operator optimizing unit 206 outputs information on the data structure in the determined data storage area as a processing result (step S704), and ends the processing.

  Specifically, the Join operator optimization unit 206 determines to hold the tuple as an array having the same number of dimensions as the number of array index type columns.

  In the example shown in FIG. 4, the identification information of the two columns “i” and “j” is stored in the Array Index column 424 of the stream data A. Therefore, the Join operator optimization unit 206 determines to hold the tuples included in the input stream A as a two-dimensional array. The same applies to the stream data B.

  FIG. 8 is a flowchart illustrating processing executed by the GroupBy operator optimization unit 207 according to the first embodiment of this invention.

  When called from the query optimization unit 204, the GroupBy operator optimization unit 207 starts processing described below. At this time, the query optimization unit 204 outputs the identification information (operator ID) of the operator to be processed.

  The GroupBy operator optimization unit 207 refers to the GroupBy operator management information 440 included in the query graph configuration information 202, and determines whether or not an array index type column is included in the grouping target column (step S801). Specifically, the following processing is executed.

  The GroupBy operator optimizing unit 207 refers to the GroupBy operator management information 440 and searches for an entry that matches the operator identification information to which the operator ID 441 is input.

  The GroupBy operator optimizing unit 207 compares the Grouping column 443 and the Array Index column 444 of the searched entry, and whether there is one or more columns in the Array Index column 444 that match the columns stored in the Grouping column 443. Determine whether or not.

  If there is one or more columns that match the columns stored in the Grouping column 443, the GroupBy operator optimization unit 207 determines that the column to be grouped includes an array index type column.

  The above is the description of the process in step S801.

  When it is determined that the column to be grouped does not include an array index type column, the GroupBy operator optimization unit 207 outputs to the query optimization unit 204 as a processing result that the setting information of the data storage area is not necessary (Step S806), and the process ends.

  When it is determined that the column to be grouped includes an array index type column, the GroupBy operator optimization unit 207 determines that the column to be grouped is an array index type column and a normal type column based on the GroupBy operator management information 440. (Step S802).

  The GroupBy operator optimization unit 207 determines whether or not the normal type column is included in the grouping target column based on the classification result (step S803).

  For example, when the identification information of all the columns stored in the Grouping column 443 is stored in the Array Index column 444, the GroupBy operator optimization unit 207 includes the normal type column in the grouping target columns. Judge that there is no.

  In the example shown in FIG. 4, since all the columns “A.i” and “B.j” stored in the Grouping column 443 are stored in the Array Index column 444, the GroupBy operator optimizing unit 207 It is determined that the normal column is not included in the columns to be divided.

  When it is determined that the normal column is not included in the grouping target column, the GroupBy operator optimization unit 207 determines the data structure in the data storage area (step S804). Thereafter, the GroupBy operator optimizing unit 207 outputs information on the data structure in the determined data storage area as a processing result (step S806), and ends the processing.

  Specifically, the GroupBy operator optimizing unit 207 determines to hold the aggregate value as an array with an array index type and an array with the same number of dimensions as the number of columns that are the grouping target columns.

  In the example illustrated in FIG. 4, the column “A.i”, the column “A.j”, the column “B.i”, and the column “B.j” are stored in the Array Index column 444. Stores a column “A.i” and a column “B.j”. Therefore, the number of columns that are array index type columns and grouping columns is “2”. Accordingly, the GroupBy operator optimizing unit 207 determines to hold the aggregate value as a two-dimensional array.

  If it is determined that the normal column is included in the grouping target column, the GroupBy operator optimization unit 207 determines the data structure in the data storage area (step S805). Thereafter, the GroupBy operator optimizing unit 207 outputs information on the data structure in the determined data storage area as a processing result (step S806), and ends the processing.

  Specifically, the GroupBy operator optimizing unit 207 holds the tuple as a hash map having the normal type column as a key, an array index type column, and an array having the same number of dimensions as the number of grouping targets. To decide.

  For example, in the case of a tuple including an array index type column “A.i”, “B.j”, and a normal type column “A.k”, the GroupBy operator optimization unit 207 uses the normal type column “A.k” as a key. And the total value is determined to be held as a hash map having a two-dimensional array as a value.

  The above is description of the process which each optimization part performs. In this embodiment, each optimization unit determines a data structure optimal for processing based on information such as an array index type column. In addition, the query construction unit 203 generates setting information for the data storage area of the operator based on the processing results described with reference to FIGS. Accordingly, the operator execution unit 211 can manage data having a data structure optimum for processing based on the setting information of the data storage area when executing a query.

  By managing the data of the multidimensional array, it is possible to improve the processing efficiency in each operator. When it is not necessary to manage as a multidimensional array, it is handled as a normal tuple. As a result, unnecessary memory consumption can be reduced. That is, the memory utilization efficiency can be improved.

  This is because holding all tuples as a multidimensional array may increase memory consumption by generating unnecessary indexes. Therefore, in this embodiment, the query optimization unit 204 determines a data management method that can improve the efficiency of the search process and reduce the memory consumption based on the analysis result of the query graph.

  Here, an example of data handling during query execution will be described.

  In the following description, it is assumed that operator execution units 211 of all operators included in the query graph are registered in the query execution unit 210.

  FIG. 9 is an explanatory diagram illustrating a data flow between operators according to the first embodiment of this invention.

  The query graph shown in FIG. 9 is a query graph corresponding to the analysis scenario 132 shown in FIG. The query graph shown in FIG. 9 includes four operators, a Partition window 901-1 and 901-2, a Join operator 902, and a GroupBy operator 903. Therefore, four operator execution units 211 are registered in the query execution unit 210.

  In the following description, the operator execution unit 211 corresponding to the Partition window 901-1 is described as a first operator execution unit, and the operator execution unit 211 corresponding to the Partition window 901-2 is described as a second operator execution unit. The operator execution unit 211 corresponding to the Join operator 902 is referred to as a third operator execution unit, and the operator execution unit 211 corresponding to the GroupBy operator 903 is referred to as a fourth operator execution unit.

  In the Partition window 901-1, input tuples 921, 922, and 923 including a column “i”, a column “j”, and a column “v” are input.

  At this time, since the first operator execution unit is set to hold the input tuple as a two-dimensional array, the input tuples 921, 922, and 923 are held in the data storage area 911-1 as a two-dimensional array. .

  Further, the first operator execution unit extracts one value for each combination of the column “i” and the column “j”, and includes the column “i”, the column “j”, and the column of the extracted value. Tuples 931, 932, 933 are output to Join operator 902.

  Similarly, the second operator execution unit holds the input tuples 924, 925, and 926 in the data storage area 911-2 as a two-dimensional array.

  The third operator execution unit is set to hold tuples output from the Partition window 901-1 and the Partition window 901-2 as a two-dimensional array. Therefore, the third operator execution unit holds the tuples 931, 932, and 933 as a two-dimensional array in the data storage area 912, and holds the tuples 934, 935, and 936 as a two-dimensional array in the data storage area 912.

  The third operator execution unit combines the tuples in which the columns “A.j” and “B.j” match, so that the column position of the input stream A and the row position of the input stream B are Combine identical elements.

  The third operator execution unit sets the position of the element of the input stream A to the column “A.i” and the column “A.j”, and sets the position of the element of the input stream B to the column “B.i” and the column “B. j ”, and further, tuples 941, 942, and 943 are output in which the elements of the combined input stream A are the column“ A.v ”and the elements of the combined input stream B are the column“ B.v ”.

  In this embodiment, 18 tuples are output.

  The fourth operator execution unit includes column “A.i” and column [B. j] and the tuples output from the Join operator 902 are divided into a plurality of groups. The fourth operator execution unit calculates the product of the elements of the tuples for each group, and further calculates the total value of the products of the tuples.

  Since the fourth operator execution unit is set to hold the total result as a two-dimensional array, the total value of each group is held in the data storage area 913 as a two-dimensional array.

  In addition, the fourth operator execution unit processes the tuples 951, 952, and 953 in which the position where the total value is stored is the column “A.i” and the column “B.j” and the total value is the column “v”. Output as a result.

  As described above, it is possible to improve the processing efficiency of the operator by managing data as a multidimensional array. In addition, since an optimal data format is determined for each operator's data storage area, memory consumption can be reduced.

  FIG. 10 is an explanatory diagram showing the flow of data between operators in the first embodiment of the present invention.

  FIG. 10 shows the flow of data accompanying the updating of the value of one tuple.

  When a new tuple 1001 is input to the Partition window 901-2, the second operator execution unit selects data that matches the index included in the input tuple based on the position of the two-dimensional array of the stream data B. Search for.

  In the Partition window 901-2, only one value can exist for the combination of the columns “i” and “j”, and thus the retrieved data is deleted. Therefore, the second operator execution unit outputs a tuple 1002 of data to be deleted and a tuple 1003 of data to be updated to the Join operator 902.

  The third operator execution unit combines the tuples related to the data to be deleted, outputs the combined result as tuples 1004, 1006, and 1008, combines the tuples related to the updated data, and displays the combined result. Output as tuples 1005, 1007, and 1009.

  The fourth operator execution unit refers to the column “A.i” and the column “B.j” included in the tuple input from the Join operator, and specifies the position of the total value to be updated. The fourth operator execution unit reads a value from the position of the specified total value, and executes a total process regarding the specified total value.

  Further, the fourth operator execution unit outputs the tuples 1010, 1012, 1014 of the aggregate values to be deleted, and outputs the tuples 1011, 1013, 1015 of the aggregate values to be updated.

  In the present embodiment, by managing data in a multidimensional array, it is possible to execute processing accompanying updating of some elements at high speed.

  In this embodiment, one stream data processing server 100 executes processing of stream data, but the present invention is not limited to this.

  For example, a computer system composed of a plurality of servers may be used. In this case, a form in which each of the plurality of servers includes the stream data processing unit 110 is conceivable. In addition, each functional unit of the stream data processing unit 110 may be distributed and arranged in a plurality of servers.

  A plurality of virtual servers may be generated on one server, and the same configuration may be realized using the virtual servers.

  As described above, according to the first embodiment, the optimum data structure in the data storage area of the operator can be determined based on the analysis of the query graph. This makes it possible to speed up the search process and reduce the memory usage. In addition, since it is not necessary to define a special built-in function in the matrix operation, the development efficiency of the user can be improved.

  The various software illustrated in the present embodiment can be stored in various recording media (for example, non-temporary storage media) such as electromagnetic, electronic, and optical, and through a communication network such as the Internet. It can be downloaded to a computer.

  Furthermore, in the present embodiment, the example using the control by software has been described, but a part thereof may be realized by hardware.

  Although the present invention has been described in detail with reference to the accompanying drawings, the present invention is not limited to such specific configurations, and various modifications and equivalents within the spirit of the appended claims Includes configuration.

100 stream data processing server 101 processor 102 memory 103 bus 104 network interface 105 storage device 110 stream data processing unit 120 host computer 121 input tuple 130 host computer 131 query registration interface 132 analysis scenario 140 host computer 141 output tuple 150 network 201 query parser 202 query Graph configuration information 203 Query construction unit 204 Query optimization unit 205 Partition window optimization unit 206 Join operator optimization unit 207 GroupBy operator optimization unit 210 Query execution unit 211 Operator execution unit 221 Tuple input unit 222 Tuple output unit 410 Operator management information 420 Partition Window Management Information 430 J in the operator management information 440 GroupBy operator management information 901 Partition window 902 Join operator 903 GroupBy operator 911, 912, 913 data storage area

Claims (10)

A data processing method in a computer that executes stream data processing,
The computer has a processor, a memory connected to the processor, a network interface connected to the processor,
The stream data processing includes a plurality of operators as minimum processing units,
The stream data that is the target of the stream data processing includes a plurality of tuples composed of a plurality of columns,
The calculator is
Query analysis that receives an analysis scenario including one or more queries for realizing the stream data processing and generates information of a query graph indicating the configuration of the operator in the stream data processing by analyzing the analysis scenario And
A query construction unit that generates an operator execution unit that executes processing corresponding to the operator based on the information of the query graph;
A query execution unit that processes the input stream data based on the operator execution unit generated by the query construction unit;
Have
The method
A first step in which the query construction unit selects an operator to be processed from the plurality of operators included in the stream data processing;
The query construction unit analyzes the processing content of the selected operator based on the information of the query graph, and is assigned to the operator execution unit corresponding to the selected operator based on the result of the analysis. A second step of determining a data structure in the data storage area;
A third step in which the query construction unit generates an operator execution unit corresponding to the selected operator based on the result of the analysis and information on the determined data structure;
A fourth step in which the query construction unit registers the generated operator execution unit in the query execution unit;
A data processing method comprising: A data processing method according to claim 1, comprising:
The method
The query analysis unit analyzing the analysis scenario to identify an array index type column that is an integer continuous from 1 among the columns constituting the tuple that is the target of the stream data processing;
The query analysis unit generating information of the query graph including information of the identified sequence index type column;
Including
The two steps are:
A fifth step of referring to the information of the query graph and determining whether a tuple processed by the selected operator includes the column of the array index type;
When it is determined that the tuple to be processed by the selected operator includes the array index type column, the tuple in the data storage area is managed as a multidimensional array defined based on the array index type. And a sixth step of generating setting information for managing the tuple as a multidimensional array. A data processing method according to claim 2, wherein
The stream data processing includes a first operator that extracts a predetermined number of the tuples from the stream data;
The sixth step includes
Classifying columns constituting a tuple to be processed by the selected operator into an array index type column and a normal type column that is a column other than the array index type;
Determining whether the tuple processed by the selected operator includes the normal type column based on the classification result;
If it is determined that the tuple processed by the selected operator does not include the normal type column, the tuple for holding the tuple as a multidimensional array having the same number of dimensions as the number of columns of the array index type Generating configuration information;
When it is determined that the tuple processed by the selected operator includes the normal type column, the normal type column is used as a key, and a multidimensional array having the same number of dimensions as the number of the array index type columns is provided. Generating the setting information for holding the tuple as a hash map as a value;
A data processing method comprising: A data processing method according to claim 2, wherein
The stream data processing includes a second operator that combines the tuples input from the plurality of operators based on a predetermined column condition,
The sixth step includes
Referring to the information of the query graph, it is determined whether or not the sequence index type column included in the tuple processed by the selected operator is related to the condition of the predetermined column in the second operator. And steps to
When it is determined that the array index type column included in the tuple processed by the selected operator is related to the condition of the predetermined column in the second operator, the number of the array index type columns Generating the setting information for holding the tuple as a multidimensional array of the same number of dimensions;
A data processing method comprising: A data processing method according to claim 2, wherein
The stream data processing is a third operator that divides the plurality of tuples into a plurality of groups based on a predetermined condition, and aggregates the values of the plurality of tuples for the plurality of groups based on a predetermined arithmetic processing. Including
The sixth step includes
Classifying columns constituting a tuple to be processed by the selected operator into an array index type column and a normal type column that is a column other than the array index type;
Determining whether the tuple processed by the selected operator includes the normal type column based on the classification result;
If it is determined that the tuple processed by the selected operator does not include the normal type column, the tuple for holding the tuple as a multidimensional array having the same number of dimensions as the number of columns of the array index type Generating configuration information;
When it is determined that the tuple processed by the selected operator includes the normal type column, the normal type column is used as a key, and a multidimensional array having the same number of dimensions as the number of the array index type columns is provided. Generating the setting information for holding the tuple as a hash map as a value;
A data processing method comprising: A computer having a processor, a memory connected to the processor, a network interface connected to the processor, and executing stream data processing;
The stream data processing includes a plurality of operators as minimum processing units,
The stream data that is the target of the stream data processing includes a plurality of tuples composed of a plurality of columns,
The calculator is
Query analysis that receives an analysis scenario including one or more queries for realizing the stream data processing and generates information of a query graph indicating the configuration of the operator in the stream data processing by analyzing the analysis scenario And
A query construction unit that generates an operator execution unit that executes processing corresponding to the operator based on the information of the query graph;
A query execution unit that processes the input stream data based on the operator execution unit generated by the query construction unit;
Have
The query building unit
Select an operator to be processed from the plurality of operators included in the stream data processing,
Based on the information of the query graph, the processing contents of the selected operator are analyzed, and based on the analysis result, the data structure in the data storage area allocated to the operator execution unit corresponding to the selected operator Decide
Based on the result of the analysis and the information of the determined data structure, an operator execution unit corresponding to the selected operator is generated,
A computer, wherein the generated operator execution unit is registered in the query execution unit. The computer according to claim 6, wherein
The query analysis unit
By analyzing the analysis scenario, an array index type column that is an integer continuous from 1 among the columns constituting the tuple that is the target of the stream data processing is specified,
Generating information of the query graph including information of the column of the identified sequence index type;
The query building unit
When determining the data structure in the data storage area, refer to the information of the query graph, determine whether the tuple processed by the selected operator includes the array index type column,
When it is determined that the tuple to be processed by the selected operator includes the array index type column, the tuple in the data storage area is managed as a multidimensional array defined based on the array index type. Decided on
A computer that generates setting information for managing the tuple as a multidimensional array. The computer according to claim 7,
The stream data processing includes a first operator that extracts a predetermined number of the tuples from the stream data;
The query building unit
When determining the data structure in the data storage area, the columns constituting the tuple processed by the selected operator are the column of the array index type and the normal type column which is a column other than the array index type And categorized as
Based on the result of the classification, determine whether the tuple processed by the selected operator includes the normal type column;
If it is determined that the tuple processed by the selected operator does not include the normal type column, the tuple for holding the tuple as a multidimensional array having the same number of dimensions as the number of columns of the array index type Generate configuration information,
When it is determined that the tuple processed by the selected operator includes the normal type column, the normal type column is used as a key, and a multidimensional array having the same number of dimensions as the number of the array index type columns is provided. A computer that generates the setting information for holding the tuple as a hash map to be a value. The computer according to claim 7,
The stream data processing includes a second operator that combines the tuples input from the plurality of operators based on a predetermined column condition,
The query building unit
When determining the data structure in the data storage area, the array index type column included in the tuple processed by the selected operator is referred to the information in the query graph, and the second operator has the array index type column. Determine if it is relevant to the given column condition,
When it is determined that the array index type column included in the tuple processed by the selected operator is related to the condition of the predetermined column in the second operator, the number of the array index type columns A computer that generates the setting information for holding the tuple as a multidimensional array having the same number of dimensions. The computer according to claim 7,
The stream data processing is a third operator that divides the plurality of tuples into a plurality of groups based on a predetermined condition, and aggregates the values of the plurality of tuples for the plurality of groups based on a predetermined arithmetic processing. Including
The query building unit
When determining the data structure in the data storage area, the columns constituting the tuple processed by the selected operator are the column of the array index type and the normal type column which is a column other than the array index type And categorized as
Based on the result of the classification, determine whether the tuple processed by the selected operator includes the normal type column;
If it is determined that the tuple processed by the selected operator does not include the normal type column, the tuple for holding the tuple as a multidimensional array having the same number of dimensions as the number of columns of the array index type Generate configuration information,
When it is determined that the tuple processed by the selected operator includes the normal type column, the normal type column is used as a key, and a multidimensional array having the same number of dimensions as the number of the array index type columns is provided. A computer that generates the setting information for holding the tuple as a hash map to be a value.

Images