WO2015139670A1 - System and method for column-specific materialization scheduling - Google Patents
System and method for column-specific materialization scheduling Download PDFInfo
- Publication number
- WO2015139670A1 WO2015139670A1 PCT/CN2015/074819 CN2015074819W WO2015139670A1 WO 2015139670 A1 WO2015139670 A1 WO 2015139670A1 CN 2015074819 W CN2015074819 W CN 2015074819W WO 2015139670 A1 WO2015139670 A1 WO 2015139670A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- rel
- statement
- dag
- forest
- parallel
- Prior art date
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/221—Column-oriented storage; Management thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2453—Query optimisation
- G06F16/24534—Query rewriting; Transformation
- G06F16/24542—Plan optimisation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/284—Relational databases
- G06F16/285—Clustering or classification
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/80—Information retrieval; Database structures therefor; File system structures therefor of semi-structured data, e.g. markup language structured data such as SGML, XML or HTML
- G06F16/83—Querying
- G06F16/835—Query processing
- G06F16/8365—Query optimisation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/95—Retrieval from the web
- G06F16/951—Indexing; Web crawling techniques
Definitions
- the present disclosure is generally directed to relational database management systems (RDBMSs) , and more specifically to a system and method for column-specific materialization in a column oriented RDBMS.
- RDBMSs relational database management systems
- a column oriented RDBMS is a DBMS that stores data tables as sections of columns of data, rather than as rows of data.
- materialization is an important factor in determining query performance in a column oriented RDBMS.
- Existing column oriented RDBMSs typically employ either fixed early materialization or fixed late materialization. In early materialization, columns referenced in a query are fetched at the leaf nodes of an operator graph and they are transmitted from a child operator to a parent operator if required by up-stream operators. In late materialization, columns needed by an operator are fetched from their sources just before processing and discarded afterwards. For most column oriented RDBMSs, the column materialization strategy is hard coded.
- This disclosure is directed to determining an optimal materialization schedule for each column in a query execution in a column oriented RDBMS.
- One example embodiment includes a method of dynamically establishing a materialization schedule in a RDBMS.
- the method includes receiving a query text, transforming the query text into a Rel directed acyclic graph (DAG) , performing a bottom-up transversal of the Rel DAG to create a parallel Rel DAG, and computing a column specific materialization schedule of the parallel Rel DAG.
- the parallel Rel DAG is transformed into a DAG of function calls and data re-shuffling actions to create a parallel statement forest.
- a coordinator statement forest is generated that invokes the function calls and the data re-shuffling actions according to the parallel statement forest.
- the parallel statement forest and the coordinator statement forest are transformed into a forest of binary association table (BAT) operator lists to compute an optimal materialization schedule for each column of a table.
- BAT binary association table
- a RDBMS is configured to dynamically establish a materialization schedule.
- a relational data base management system configured to dynamically establish a materialization schedule.
- the RDBMS includes: a receiving means configured to receive a query text, a transforming means configured to transform the query text into a Rel directed acyclic graph (DAG) , a performing means configured to perform a bottom-up transversal of the Rel DAG to create a parallel Rel DAG, and a computing means configured to compute a column specific materialization schedule of the parallel Rel DAG.
- the parallel Rel DAG is transformed into a DAG of function calls and data re-shuffling actions to create a parallel statement forest.
- a coordinator statement forest is generated that invokes the function calls and the data re-shuffling actions according to the parallel statement forest.
- the parallel statement forest and the coordinator statement forest are transformed into a forest of binary association table (BAT) operator lists to compute an optimal materialization schedule for each column of a table.
- BAT binary association table
- FIGURE 1 illustrates a table schema from the Transaction Processing Performance Council (TPC) Benchmark H (TPC-H) ;
- FIGURE 2 illustrates a syntax tree of a query plan for a structured query language (SQL) statement
- FIGURE 3 illustrates an example column-specific materialization algorithm in accordance with this disclosure
- FIGURE 4 illustrates an example method for parallel query optimization in accordance with this disclosure
- FIGURE 5 illustrates an example parallel statement forest
- FIGURE 6 illustrates an example coordinator statement forest
- FIGURE 7 illustrates an example of a computing device for parallel query optimization according to this disclosure.
- FIGURES 1 through 7, discussed below, and the various embodiments used to describe the principles of the present invention in this patent document are by way of illustration only and should not be construed in any way to limit the scope of the invention. Those skilled in the art will understand that the principles of the invention may be implemented in any type of suitably arranged device or system.
- Embodiments of this disclosure provide a method and apparatus for dynamic column-specific materialization scheduling in a distributed column oriented RDBMS.
- the materialization schedule is optimized by selecting a materialization strategy based on an execution cost including central processing unit (CPU) , disk, and network costs for each individual exchange operator.
- the disclosed embodiments use dynamic programming techniques to determine the optimal materialization schedule. Dynamic programming is computationally feasible for the disclosed embodiments because the optimal schedule for a sub-plan is path independent.
- TPC Transaction Processing Performance Council
- TPC-H Benchmark H
- join selectivity is approximately 50%.
- join selectivity is a measure of how much variation (i.e., how many different values) exists between records in a join result. Low selectivity means that there is not a lot of variation in the values in a column, while high selectivity means there is substantial variation in the values in the column.
- the join selectivity can be examined to determine the cost of early materialization and late materialization. After computing the cost, it is found that early materialization (i.e., stitch l_suppkey with l_partkey, and then shuffle) is better for Query 1. This is because the communication/CPU cost of sending 50% of Row IDs and l_suppkey column data would be more costly than re-shuffling the entire l_suppkey column data.
- example Query 2 which is also based on the TPC-H table schema shown in FIGURE 1.
- a mixed materialization scheme in accordance with this disclosure can be used. For example, consider the example Query 3, which is also based on the TPC-H table schema shown in FIGURE 1.
- FIGURE 2 illustrates a syntax tree of a query plan for example Query 3.
- the syntax tree 200 includes a node for each operator in Query 3.
- the syntax tree 200 is a directed acyclic graph (DAG) that includes a plurality of nodes 201-207.
- the node 201 represents the LINEITEM table in Query 3.
- the node 201 is at the bottom of the syntax tree 200 because LINEITEM is the first operator in Query 3.
- the node 202 is an exchange node that represents a data shuffling or redistribution operation in Query 3.
- the data in the LINEITEM table is shuffled before its join to the PART table.
- the node 203 represents the first instance (instance A) of the PART table.
- the node 204 represents the join of the shuffled LINEITEM table and instance A of the PART table, as well as the SELECT statement.
- the node 205 is another exchange node that represents a data shuffling of the result set from the node 204.
- the node 206 represents the second instance (instance B) of the PART table.
- the node 207 represents the join of shuffled result set from the node 205 and instance B of the PART table.
- a column-specific materialization algorithm is part of a parallel query optimization compilation process that transforms a structured query language (SQL) statement into a parallel execution plan.
- SQL structured query language
- a decision to using early materialization or late materialization can be based on the following recursive reasoning.
- the parallel execution plan is represented as a DAG of exchange nodes (such as the syntax tree 200 shown in FIGURE 2)
- the best way to materialize a column C at an exchange node E depends on whether the column is materialized at E’s child exchange node, E_1 (e.g., the exchange node 202) .
- the cost to materialize column C at exchange node E would be the cost to materialize column C at exchange node E_1 plus the communication/CPU cost to re-shuffle column C at exchange node E. If column C is not materialized at exchange node E_1, the cost would be the cost of late materialization at exchange node E, which is the communication/CPU cost of sending Row IDs and column C’s data.
- M [L-1] is the materialization choice at level L-1
- E [L-1] is the Exchange node at level L-1.
- a method for computing the optimal materialization schedule for a column is provided.
- the disclosed method assumes that the distributed execution plan is represented by a DAG of exchange nodes and relational operators, such as the syntax tree 200 shown in FIGURE 2.
- the parent exchange node materializes this column if it has a lower cost than fetching the column from Row ID in the parent exchange node.
- the column is materialized at the current exchange node if it is required at the current exchange node or if the cost at (a. i. 1) is smaller than cost at (a. i. 2) .
- FIGURE 3 illustrates an example column-specific materialization algorithm 300 in accordance with this disclosure.
- the algorithm 300 is pseudocode of an acyclic algorithm that can represent (or be represented by) a DAG.
- the algorithm 300 can be part of a parallel query optimization compilation process that transforms a structured query language (SQL) statement into a parallel execution plan.
- the algorithm 300 can be used to perform a SQL operation that may include mixed materialization, such as Query 3.
- the algorithm 300 may be performed using a computing device capable of RDBMS operations, such as the computing device 700 of FIGURE 7 (described below) .
- the algorithm 300 includes three inputs: Exchange Node E, level L, and column C.
- the exchange node is a database operator that is used to shuffle records in one or more tables.
- the level L is provided by the system and is used to identify a level in a query tree.
- the levels in the query tree are numbered such that the lowest levels of the query tree are shown or indicated at the bottom of the tree, and the highest levels of the query tree are shown or indicated at the top of the tree.
- Array K in the algorithm 300 contains materialization costs for the different levels. That is, each element of the array K corresponds to a materialization cost for one level.
- the IF-THEN-ELSE argument indicated at 301 in the algorithm 300 determines if an early schedule or a late schedule will be used for level 1 based on the materialization cost for that level. Thus, the schedule is determined first for level 1.
- the SET COST operation indicated at 302 is a recursive function that calls the Materialization Schedule algorithm 300 to be performed on a next lower level using Exchange Node E’s child as in input. For example, if a query tree includes four levels, and the algorithm 300 is being performed for level 4, then the SET COST operation 302 is used to call the algorithm 300 to be performed for level 3.
- the SET SCHEDULE operation indicated at 303 is used to set the schedule (early schedule or late schedule) for levels other than level 1 by selecting the minimum cost between (a) the cost of the next lower level + the early materialization cost, and (b) the cost of the next lower level + the late materialization cost. Then, the operation indicated at 304 sets the cost at level L based on the cost of the next lower level (L –1) and the materialization cost at level L.
- the algorithm 300 is based on dynamic programming principles. Dynamic programming is a technique for solving complex problems by breaking them down into simpler sub-problems. Dynamic programming is often used in mathematics, computer science, economics, and in other fields. One classic example of a complex problem for which dynamic programming is frequently used is determining the shortest path between two cities or locations on a map, taking into account the different roads and intermediate points available in the area.
- the complex problem In order to be able to use dynamic programming to solve a complex problem, the complex problem itself must possess certain properties. First, the complex problem must include overlapping sub-problems. Second the complex problem must have an optimal substructure. If a problem does not possess these properties, then use of dynamic programming to solve the problem may either be impossible or lead to a sub-optimal solution.
- the materialization algorithm 300 includes overlapping sub-problems. For example, the optimal schedule at level L is determined based on the optimal schedule at level L-1, while the optimal schedule at level L-1 is determined based on the optimal schedule at level L-2, and so on. Thus, the determinations of the different levels can be considered to overlap.
- the materialization algorithm 300 includes an optimal substructure. For example, the IF-THEN-ELSE argument at 301 in the algorithm 300 determines an optimal schedule for level 1 based on the materialization cost for that level, and then the algorithm 300 determines the optimal schedule for higher levels based on the schedule for the next lower level. Thus, materialization algorithm 300 includes an optimal substructure based on a lowest performance cost (i.e., a fastest execution time) .
- Dynamic programming is different from a greedy algorithm.
- a greedy algorithm may find a local optimal solution to a sub-problem, but often may arrive at a globally sub-optimal solution. For example, considering the shortest path between two cities problem, a greedy algorithm may find a locally optimal solution to a traffic jam at one intersection, but the local solution may be optimal for only that intersection, and may result in a sub-optimal route overall when the total route between the two cities is considered as a global solution.
- the algorithm 300 can be part of a parallel query optimization compilation process that transforms a SQL statement into a parallel execution plan.
- One approach to this query-text to-parallel-plan transformation process to anchor the context within which the column specific algorithm 300 is performed can be summarized in the following method described in FIGURE 4.
- FIGURE 4 illustrates an example method for parallel query optimization in accordance with this disclosure.
- the method 400 is described as being used with the algorithm 300 of FIGURE 3.
- the method 400 could be used with any suitable algorithm and in any suitable system.
- the method 400 may be performed using a computing device capable of RDBMS operations, such as the computing device 700 of FIGURE 7 (described below) , or may be performed by another suitable device or system.
- a query text is transformed into a syntax tree.
- the syntax tree is checked for semantic correctness.
- the syntax tree is transformed into a DAG of relational operators (rels) , which may be referred to as a Rel DAG, as known in the art.
- the leaf nodes of the Rel DAG are annotated with clustering information.
- an exchange node is inserted between a parent Rel and a child Rel when the clustering properties of the output of the child Rel is incompatible with the clustering properties of the input of the parent node.
- the resulting DAG is called a parallel Rel DAG.
- the parallel Rel DAG may be similar to the DAG 200 shown in FIGURE 2.
- a column-specific materialization algorithm (e.g., the algorithm 300) is performed to compute the optimal materialization schedule for each column.
- the parallel Rel DAG is transformed into a DAG of function calls and data re-shuffling actions according to the following details.
- Each function corresponds to a fragment of the parallel Rel DAG between two adjacent exchange nodes.
- Each data re-shuffling action corresponds to an exchange node.
- Each function is transformed into a statement forest, where a statement represents a logical BAT operator. The logical BAT operator produces an expression based on expressions produced by its children statements.
- the logical BAT operator makes a depth-first traversal of the Rel DAG fragment. Then, for each Rel, for each expression exported by the Rel, and for each combination of the source tables’ partitions, a statement DAG is generated for the expression.
- Each function takes the columns’ data exported from its children exchange nodes as an input.
- the outputs of each function are expressions exported by the top Rel of the function.
- the output of the function becomes the input of the data re-shuffling action of its parent exchange node. Note that Row IDs are always exported by a Rel.
- Each data re-shuffling action re-shuffles columns to be materialized at this exchange node. Columns to be materialized but not exported by a child exchange node are fetched by using Row IDs. Each data re-shuffling action is transformed into a statement forest containing one statement DAG for each re-shuffled column. The resulting statement forest is called the parallel statement forest.
- An example parallel statement forest 500 is shown in FIGURE 5.
- a statement DAG is generated that invokes the functions and data re-shuffling actions according to the depth-first traversal sequence of the parallel statement forest.
- the resulting statement DAG is called the coordinator statement forest.
- An example coordinator statement forest 600 is shown in FIGURE 6.
- the parallel statement forest and the coordinator statement forest are transformed into a forest of BAT operator lists.
- Each list corresponds to a function, a data re-shuffling action, or the coordinator program.
- FIGURE 4 illustrates one example of a method 400 for parallel query optimization
- various changes may be made to FIGURE 4.
- steps in FIGURE 4 could overlap, occur in parallel, occur in a different order, or occur any number of times.
- FIGURE 7 illustrates an example of a computing device 700 for performing the materialization algorithm 300 of FIGURE 3 or the parallel query optimization method 400 of FIGURE 4.
- the computing device 700 includes a computing block 703 with a processing block 705 and a system memory 707.
- the processing block 705 may be any type of programmable electronic device for executing software instructions, but will conventionally be one or more microprocessors.
- the system memory 707 may include both a read-only memory (ROM) 709 and a random access memory (RAM) 711. As will be appreciated by those of skill in the art, both the read-only memory 709 and the random access memory 711 may store software instructions for execution by the processing block 705.
- the processing block 705 and the system memory 707 are connected, either directly or indirectly, through a bus 713 or alternate communication structure, to one or more peripheral devices.
- the processing block 705 or the system memory 707 may be directly or indirectly connected to one or more additional memory storage devices 715.
- the memory storage devices 715 may include, for example, a “hard” magnetic disk drive, a solid state disk drive, an optical disk drive, and a removable disk drive.
- the processing block 705 and the system memory 707 also may be directly or indirectly connected to one or more input devices 717 and one or more output devices 719.
- the input devices 717 may include, for example, a keyboard, a pointing device (such as a mouse, touchpad, stylus, trackball, or joystick) , a touch screen, a scanner, a camera, and a microphone.
- the output devices 719 may include, for example, a display device, a printer and speakers. Such a display device may be configured to display video images.
- one or more of the peripheral devices 715-719 may be internally housed with the computing block 703. Alternately, one or more of the peripheral devices 715-719 may be external to the housing for the computing block 703 and connected to the bus 713 through, for example, a Universal Serial Bus (USB) connection or a digital visual interface (DVI) connection.
- USB Universal Serial Bus
- DVI digital visual interface
- the computing block 703 may also be directly or indirectly connected to one or more network interfaces cards (NIC) 721, for communicating with other devices making up a network.
- the network interface cards 721 translate data and control signals from the computing block 703 into network messages according to one or more communication protocols, such as the transmission control protocol (TCP) and the Internet protocol (IP) .
- TCP transmission control protocol
- IP Internet protocol
- the network interface cards 721 may employ any suitable connection agent (or combination of agents) for connecting to a network, including, for example, a wireless transceiver, a modem, or an Ethernet connection.
- computing device 700 is illustrated as an example only, and it not intended to be limiting. Various embodiments of this disclosure may be implemented using one or more computing devices that include the components of the computing device 700 illustrated in FIGURE 7, or which include an alternate combination of components, including components that are not shown in FIGURE 7. For example, various embodiments of the invention may be implemented using a multi-processor computer, a plurality of single and/or multiprocessor computers arranged into a network, or some combination of both.
- the algorithm described in this disclosure computes the best materialization schedule for each column on every exchange operator within a query. This is advantageous over existing materialization scheduling algorithms that employ either fixed early materialization or fixed late materialization for all exchange operators in a query.
- the algorithm disclosed herein can be implemented by traversing the parallel execution graph from the top down, identifying columns that have not been scheduled. For each such column, dynamic programming is applied to compute the materialization schedule in a recursive (or bottom up) fashion.
- the minimum materialization costs at level L-1 do not change with the choice of materialization at levels greater or equal to L.
- the computation complexity is linearly proportional to the height of the parallel execution graph and number of columns.
- Embodiments of this disclosure has been demonstrated in simulation tests to reduce the interconnect bandwidth requirement for distributed query processing by an average of 10% -30%. Assuming that the inter-node communication cost is about 25% of the total query processing cost, this reduces the total cost of distributed query processing by 2.5% -7.5%.
- a computer program that is formed from computer readable program code and that is embodied in a computer readable medium.
- computer readable program code includes any type of computer code, including source code, object code, and executable code.
- computer readable medium includes any type of medium capable of being accessed by a computer, such as read only memory (ROM) , random access memory (RAM) , a hard disk drive, a compact disc (CD) , a digital video disc (DVD) , or any other type of memory.
Abstract
Description
Claims (21)
- A method of establishing a materialization schedule using a dynamic programming technique in a relational data base management system (RDBMS) , the method comprising:receiving a query text;transforming the query text into a Rel directed acyclic graph (DAG) ;performing a bottom-up transversal of the Rel DAG to create a parallel Rel DAG;computing a column specific materialization schedule of the parallel Rel DAG;transforming the parallel Rel DAG into a DAG of function calls and data re-shuffling actions to create a parallel statement forest;generating a coordinator statement forest that invokes the function calls and the data re-shuffling actions according to the parallel statement forest; andtransforming the parallel statement forest and the coordinator statement forest into a forest of binary association table (BAT) operator lists to compute an optimal materialization schedule for each column of a table.
- The method as specified in Claim 1, wherein:each function call corresponds to a fragment of the parallel Rel DAG between two adjacent exchange nodes; andeach data re-shuffling action corresponds to an exchange node.
- The method as specified in Claim 1 or Claim 2, wherein leaf nodes of the Rel DAG are annotated with clustering information.
- The method as specified in any one of Claims 1-3, wherein the bottom-up traversal is performed by inserting an exchange node between a parent Rel node and a child Rel node when a clustering property of an output of the child Rel node is incompatible with a clustering property of an input of the parent Rel node.
- The method as specified in any one of Claims 1-4, wherein each function call is transformed into a statement forest, wherein a statement represents a BAT operator, to produce an expression produced by a child statement of the function call.
- The method as specified in Claim 5, wherein the expression is produced by:making a depth-first traversal of a corresponding Rel DAG fragment, andfor each Rel, for each expression exported by the Rel, and for each combination of a source tables’ partitions, generate a statement DAG for the expression.
- The method as specified in any one of Claims 1-6, wherein:each function call takes columns’ data exported from its child Rel nodes as an input,an output of each function call is expression exported by a top Rel of the function call, andthe output of the function call is the input of the data re-shuffling action of its parent node, wherein Row IDs are exported by a Rel.
- The method as specified in Claim 4, wherein each data re-shuffling action re-shuffles columns to be materialized at a respective Rel node, and columns materialized but not exported by a child Rel node are fetched by using Row IDs.
- The method as specified in Claim 8, wherein each data re-shuffling action is transformed into a statement forest containing one statement DAG for each re-shuffled column to create the parallel statement forest.
- The method as specified in any one of Claims 1-9, further comprising generating a statement DAG that invokes the function calls and data re-shuffling actions according to a depth-first traversal sequence of the parallel statement forest to create the coordinator statement forest.
- A relational data base management system (RDBMS) configured to:receive a query text;transform the query text into a Rel directed acyclic graph (DAG) ;perform a bottom-up transversal of the Rel DAG to create a parallel Rel DAG;compute a column specific materialization schedule of the parallel Rel DAG;transform the parallel Rel DAG into a DAG of function calls and data re-shuffling actions to create a parallel statement forest;generate a coordinator statement forest that invokes the function calls and the data re-shuffling actions according to the parallel statement forest; andtransform the parallel statement forest and the coordinator statement forest into a forest of binary association table (BAT) operator lists to compute an optimal materialization schedule for each column of a table.
- The RDBMS as specified in Claim 11, wherein:each function call corresponds to a fragment of the parallel Rel DAG between two adjacent exchange nodes; andeach data re-shuffling action corresponds to an exchange node.
- The RDBMS as specified in Claim 11 or claim 12, wherein leaf nodes of the Rel DAG are configured to be annotated with clustering information.
- The RDBMS as specified in any one of Claims 11-13, wherein the bottom-up traversal is performed by inserting an exchange node between a parent Rel node and a child Rel node when a clustering property of an output of the child Rel node is incompatible with a clustering property of an input of the parent Rel node.
- The RDBMS as specified in any one of Claims 11-14, wherein each function call is configured to be transformed into a statement forest, wherein a statement represents a BAT operator, to produce an expression produced by a child statement of the function call.
- The RDBMS as specified in Claim 15, wherein the expression is configured to be produced by:making a depth-first traversal of a corresponding Rel DAG fragment, andfor each Rel, for each expression exported by the Rel, and for each combination of a source tables’ partitions, generating a statement DAG for the expression.
- The RDBMS as specified in any one of Claims 11-16, wherein:each function call takes columns’ data exported from its child Rel nodes as an input,an output of each function call is expression exported by a top Rel of the function call, andthe output of the function call is the input of the data re-shuffling action of its parent node, wherein Row IDs are exported by a Rel.
- The RDBMS as specified in Claim 14, wherein each data re-shuffling action re-shuffles columns to be materialized at a respective Rel node, and columns materialized but not exported by a child Rel node are fetched by using Row IDs.
- The RDBMS as specified in Claim 18, wherein each data re-shuffling action is transformed into a statement forest containing one statement DAG for each re-shuffled column to create the parallel statement forest.
- A method of establishing a materialization schedule using a dynamic programming technique in a relational database management system (RDBMS) , the method comprising:receiving a query text;transforming the query text into a Rel directed acyclic graph (DAG) ;performing a bottom-up transversal of the Rel DAG to create a parallel Rel DAG;computing a column specific materialization schedule of the parallel Rel DAG;transforming the parallel Rel DAG into a DAG of function calls and data re-shuffling actions to create a parallel statement forest, wherein:each function call takes columns’ data exported from its child Rel nodes as an input,output of each function call is expression exported by a top Rel of the function call, andthe output of the function call is the input of a data re-shuffling action of its parent node, wherein Row IDs are exported by a Rel;generating a coordinator statement forest that invokes the function calls and the data re-shuffling actions according to the parallel statement forest; andtransforming the parallel statement forest and the coordinator statement forest into a forest of binary association table (BAT) operator lists to compute an optimal materialization schedule for each column of a table.
- A relational data base management system (RDBMS) configured to dynamically establish a materialization schedule, wherein the RDBMS comprises:a receiving means, configured to receive a query text;a transforming means, configured to transform the query text into a Rel directed acyclic graph (DAG) ;a performing means, configured to perform a bottom-up transversal of the Rel DAG to create a parallel Rel DAG;a computing means, configured to compute a column specific materialization schedule of the parallel Rel DAG;the transforming means, configured to transform the parallel Rel DAG into a DAG of function calls and data re-shuffling actions to create a parallel statement forest;a generating means, configured to generate a coordinator statement forest that invokes the function calls and the data re-shuffling actions according to the parallel statement forest; andthe transforming means, configured to transform the parallel statement forest and the coordinator statement forest into a forest of binary association table (BAT) operator lists to compute an optimal materialization schedule for each column of a table.
Priority Applications (3)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
BR112016021702A BR112016021702A8 (en) | 2014-03-21 | 2015-03-21 | SYSTEM, DEVICE AND METHOD FOR SCHEDULE OF SPECIFIC MATERIALIZATION TO THE COLUMN. |
CN201580013931.8A CN106462585B (en) | 2014-03-21 | 2015-03-21 | System and method for particular column materialization scheduling |
EP15764647.2A EP3108388A4 (en) | 2014-03-21 | 2015-03-21 | System and method for column-specific materialization scheduling |
Applications Claiming Priority (4)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US201461968793P | 2014-03-21 | 2014-03-21 | |
US61/968.793 | 2014-03-21 | ||
US14/663,210 US10073873B2 (en) | 2014-03-21 | 2015-03-19 | System and method for column-specific materialization scheduling |
US14/663,210 | 2015-03-19 |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2015139670A1 true WO2015139670A1 (en) | 2015-09-24 |
Family
ID=54142311
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/CN2015/074819 WO2015139670A1 (en) | 2014-03-21 | 2015-03-21 | System and method for column-specific materialization scheduling |
Country Status (5)
Country | Link |
---|---|
US (1) | US10073873B2 (en) |
EP (1) | EP3108388A4 (en) |
CN (1) | CN106462585B (en) |
BR (1) | BR112016021702A8 (en) |
WO (1) | WO2015139670A1 (en) |
Families Citing this family (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN106354829A (en) * | 2016-08-31 | 2017-01-25 | 天津南大通用数据技术股份有限公司 | Physic-chemical method and device of column storage database |
US10970284B2 (en) * | 2017-05-12 | 2021-04-06 | Oracle International Corporation | Dynamic self-reconfiguration of nodes in a processing pipeline |
US11132366B2 (en) * | 2019-04-01 | 2021-09-28 | Sap Se | Transforming directed acyclic graph shaped sub plans to enable late materialization |
CN110321210A (en) * | 2019-06-28 | 2019-10-11 | 京东数字科技控股有限公司 | Data processing method, device, computer-readable medium and electronic equipment |
CN112380286B (en) * | 2020-11-17 | 2022-03-18 | 平安科技(深圳)有限公司 | Method, device, equipment and medium for generating data object relation map of database |
CN116737763B (en) * | 2023-08-16 | 2023-11-21 | 腾讯科技(深圳)有限公司 | Structured query statement execution method, device, computer equipment and storage medium |
Citations (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
EP2040180A1 (en) * | 2007-09-24 | 2009-03-25 | Hasso-Plattner-Institut für Softwaresystemtechnik GmbH | ETL-less zero-redundancy system and method for reporting OLTP data |
US20090150413A1 (en) * | 2007-12-06 | 2009-06-11 | Oracle International Corporation | Virtual columns |
US20140075350A1 (en) * | 2012-09-10 | 2014-03-13 | Sap Ag | Visualization and integration with analytics of business objects |
Family Cites Families (8)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US7519577B2 (en) * | 2003-06-23 | 2009-04-14 | Microsoft Corporation | Query intermediate language method and system |
US8468151B2 (en) * | 2010-06-29 | 2013-06-18 | Teradata Us, Inc. | Methods and systems for hardware acceleration of database operations and queries based on multiple hardware accelerators |
US8356027B2 (en) * | 2010-10-07 | 2013-01-15 | Sap Ag | Hybrid query execution plan generation and cost model evaluation |
CN102567527A (en) * | 2011-12-30 | 2012-07-11 | 华东师范大学 | Materialized view layout in distributive system under column-orientated storage environment and maintaining method of materialized view layout |
CN102609451B (en) * | 2012-01-11 | 2014-12-17 | 华中科技大学 | SQL (structured query language) query plan generation method oriented to streaming data processing |
CN103324765B (en) * | 2013-07-19 | 2016-08-17 | 西安电子科技大学 | A kind of multi-core synchronization data query optimization method based on row storage |
US20150220571A1 (en) * | 2014-01-31 | 2015-08-06 | Futurewei Technologies, Inc. | Pipelined re-shuffling for distributed column store |
WO2015120603A1 (en) * | 2014-02-13 | 2015-08-20 | Sap Ag | Database calculation using parallel-computation in directed acyclic graph |
-
2015
- 2015-03-19 US US14/663,210 patent/US10073873B2/en active Active
- 2015-03-21 WO PCT/CN2015/074819 patent/WO2015139670A1/en active Application Filing
- 2015-03-21 EP EP15764647.2A patent/EP3108388A4/en not_active Ceased
- 2015-03-21 CN CN201580013931.8A patent/CN106462585B/en active Active
- 2015-03-21 BR BR112016021702A patent/BR112016021702A8/en not_active Application Discontinuation
Patent Citations (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
EP2040180A1 (en) * | 2007-09-24 | 2009-03-25 | Hasso-Plattner-Institut für Softwaresystemtechnik GmbH | ETL-less zero-redundancy system and method for reporting OLTP data |
US20090150413A1 (en) * | 2007-12-06 | 2009-06-11 | Oracle International Corporation | Virtual columns |
US20140075350A1 (en) * | 2012-09-10 | 2014-03-13 | Sap Ag | Visualization and integration with analytics of business objects |
Non-Patent Citations (1)
Title |
---|
See also references of EP3108388A4 * |
Also Published As
Publication number | Publication date |
---|---|
EP3108388A1 (en) | 2016-12-28 |
CN106462585A (en) | 2017-02-22 |
BR112016021702A2 (en) | 2018-07-10 |
US10073873B2 (en) | 2018-09-11 |
US20150269202A1 (en) | 2015-09-24 |
EP3108388A4 (en) | 2017-02-22 |
CN106462585B (en) | 2019-10-22 |
BR112016021702A8 (en) | 2023-01-10 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US10073873B2 (en) | System and method for column-specific materialization scheduling | |
US11693839B2 (en) | Parser for schema-free data exchange format | |
US10133778B2 (en) | Query optimization using join cardinality | |
US10268638B2 (en) | Limiting plan choice for database queries using plan constraints | |
US7630967B1 (en) | Join paths across multiple databases | |
US8166022B2 (en) | System, method, and apparatus for parallelizing query optimization | |
Borkar et al. | Hyracks: A flexible and extensible foundation for data-intensive computing | |
US10762087B2 (en) | Database search | |
US10706052B2 (en) | Method for performing in-memory hash join processing in relational database systems | |
US8965918B2 (en) | Decomposed query conditions | |
JP5791149B2 (en) | Computer-implemented method, computer program, and data processing system for database query optimization | |
US20140075161A1 (en) | Data-Parallel Computation Management | |
US9244950B2 (en) | Method for synthetic data generation for query workloads | |
US8037057B2 (en) | Multi-column statistics usage within index selection tools | |
US20180365294A1 (en) | Artificial intelligence driven declarative analytic platform technology | |
Schlegel et al. | Balloon fusion: SPARQL rewriting based on unified co-reference information | |
ten Wolde et al. | DuckPGQ: Efficient property graph queries in an analytical RDBMS | |
Kim et al. | Type-based semantic optimization for scalable RDF graph pattern matching | |
CN107818181A (en) | Indexing means and its system based on Plcient interactive mode engines | |
Kim et al. | RG-index: An RDF graph index for efficient SPARQL query processing | |
US11625398B1 (en) | Join cardinality estimation using machine learning and graph kernels | |
Sharma et al. | Indexer++ workload-aware online index tuning with transformers and reinforcement learning | |
Werner et al. | Automated composition and execution of hardware-accelerated operator graphs | |
Pipita | Dynamic query optimization in spark | |
이태휘 | Join Processing with Filtering Techniques on MapReduce Cluster |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 15764647 Country of ref document: EP Kind code of ref document: A1 |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
REEP | Request for entry into the european phase |
Ref document number: 2015764647 Country of ref document: EP |
|
WWE | Wipo information: entry into national phase |
Ref document number: 2015764647 Country of ref document: EP |
|
REG | Reference to national code |
Ref country code: BR Ref legal event code: B01A Ref document number: 112016021702 Country of ref document: BR |
|
ENP | Entry into the national phase |
Ref document number: 112016021702 Country of ref document: BR Kind code of ref document: A2 Effective date: 20160921 |