CN102323946A - Implementation method for operator reuse in parallel database - Google Patents
Implementation method for operator reuse in parallel database Download PDFInfo
- Publication number
- CN102323946A CN102323946A CN201110259524A CN201110259524A CN102323946A CN 102323946 A CN102323946 A CN 102323946A CN 201110259524 A CN201110259524 A CN 201110259524A CN 201110259524 A CN201110259524 A CN 201110259524A CN 102323946 A CN102323946 A CN 102323946A
- Authority
- CN
- China
- Prior art keywords
- operator
- plan
- thread
- materialization
- multiplexing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses an implementation method for operator reuse in a parallel database, comprising the following steps of: step 1, generating a serial query plan for query through a normal query planning method, wherein the query plan is a binary tree structure; step 2, executing the query plane by scanning from top to bottom, searching materialized reusable operators, changing the query plane structure, and changing thread level materialized operators into global reusable materialized operators; step 3, parallelizing the query plan changed in the step 2, and generating a plan forest for parallel execution of a plurality of threads; step 4, executing global reusable operator combination on the plan forest generated in the step 3, and generating a directed graph plan for the materialized reusable operators capable of being executed by the plurality of threads in parallel; step 5, executing own plan part in the directed graph by each thread in parallel, wherein the thread which executes the global reusable operator firstly is called a main thread, the main thread locks the global reusable operator and truly executes the operator and the plan of the operator, and other threads wait; step 6, unlocking the global reusable operator by the main thread after execution, wherein other threads start to read data from the global reusable operator and continue to execute own plan tree; and step 7, releasing the materialized data of the operator by the main thread after all the plans read the data of the global reusable operator.
Description
Technical field
The present invention relates to a kind of Database Systems, especially relate to a kind of multiplexing implementation method of operator of parallel database.
Background technology
Along with the development of infotech with popularize, data expand rapidly with index speed, and handling mass data more and more becomes the major issue that computer realm faces.The research to OLAP, DSS, data mining etc. that database field is risen all is the research to mass data processing in essence.
Solving the popular technology of mass data processing problem at present is parallel query technology and Clustering.The parallel query technology all is the research focus of database field all the time, and academia has proposed the architectural framework of multiple parallel query: Share-Everything (shared fully) framework, Share-Memory (shared drive) framework, Share-Disk (shared disk) framework and Share-Nothing (do not have and share) framework.Share-Memory framework and Share-Everything framework all can shared main storages, and process or thread can pass through the Memory Exchange data.But the at present popular Share-Memory and the parallel database system of Share-Everything framework all are to use shared drive to communicate and exchanges data, and do not utilize the operator between a plurality of concurrent processes or the thread multiplexing.Under parallel architecture based on subregion; A plurality of threads or process are independently carried out task separately; Parallel a plurality of often processes or the almost completely identical statement of thread execution structure in the inquiry of same query statement, the different part table that just wherein relates to difference.
In this case, each process or thread are all carried out separately inquiry one time, and do not consider operator multiplexing be significant wastage to resource.
Summary of the invention
In order to address the above problem, the invention provides a kind of in the parallel database of SM framework through realizing that the operator multiplex technique improves the resource utilization and the performance of system.
The present invention adopts following technical scheme:
Step 1, the common inquiry planing method of use are the inquiry plan that inquiry generates serial, and said inquiry plan is a binary tree structure;
Said inquiry plan is carried out in step 2, top-down scanning, seeks reusable materialization class operator, and change inquiry plan structure, changes thread-level materialization operator into the overall situation multiplexing materialization operator;
Inquiry plan after step 3, the change that step 2 is generated carries out parallelization to be handled, and generates to be used for the plan forest that a plurality of thread parallels are carried out;
Step 4, the plan forest that step 3 is generated carry out overall multiplexing operator merging processing, generate the digraph plan that is used for supplying a plurality of thread parallels execution and reusable materialization operator;
Step 5, each thread parallel are carried out the plan part separately in the said digraph; First thread that implements overall multiplexing operator is referred to as main thread; Pin the multiplexing operator of this overall situation and real this operator and following plan thereof, other thread waits carried out by main thread;
Step 6, said main thread execute the release afterwards of this operator, and other threads begin reading of data and continuation plan tree separately from the multiplexing operator of this overall situation;
Step 7, said main thread wait for that all plans all read the data that discharge this operator materialization after the data of the overall multiplexing operator that finishes.
The present invention is the optimization of the parallel data library inquiry of SM framework being carried out flow process, and key features is materialization operator identical in a plurality of threads is shared.Carry out flow process with common parallel query and compare, not only save CPU and memory source, also will lack IO and read, do not increase any cost.
Description of drawings
Below in conjunction with accompanying drawing and embodiment the present invention is further described.
Fig. 1 shows the synoptic diagram of the plan tree construction of step 1 generation;
Fig. 2 shows the synoptic diagram of the plan tree construction of step 2;
Fig. 3 shows the synoptic diagram of the plan forest structure of step 3;
Fig. 4 shows the synoptic diagram of the digraph proposed figures for the plan of step 4;
Fig. 5 shows the data flowchart in planning execution stage.
Embodiment
Below in conjunction with accompanying drawing and specific embodiment the present invention is done further detailed description:
Can supply multiplexing operator through the parallel plan of multithreading being scanned therefrom to seek in query optimization stage of database; Can supply multiplexing operator to be revised as the overall situation and share operator; And the change proposed figures for the plan, will plan tree and become the plan forest, and further be rewritten into digraph.Through this digraph plan of executed in parallel, the intermediate result of multiplexing materialization operator in the planning execution process.
This method mainly may further comprise the steps:
The plan generation phase:
Step 1:
Use common inquiry planing method, be the inquiry plan of inquiry generation serial, this inquiry plan is a binary tree structure.Wherein inquiry relates to partition table, so the part leaf node is the scanning to partition table.As shown in Figure 1, some inquiries are select*from A, B, P where A.a=B.b and A.a=P.p; Wherein A, B are not partition tables, and P is a partition table, and child partition is P1, P2.This plan tree is illustrated in this inquiry, at first the B table is created the Hash table, is connected through the HashJoin mode with the B table by A, then the result of the connection of A, B is created Hash and shows, and shows to be connected through the HashJoin mode with P again.
Step 2:
Reusable materialization class operator is sought in top-down scanning executive plan, and the change proposed figures for the plan, changes thread-level materialization operator into the overall situation multiplexing materialization operator.When finding after one, no longer continue scanning subtree downwards.The criterion of reusable materialization operator is: if certain materialization operator and following plan tree thereof do not comprise partition table, this materialization operator can be re-used so.Like the plan in this example, as shown in Figure 2, top-down scan plan can find at first that it is reusable materialization operator that the connection result to A, B creates the Hash table, and therefore revising this Hash operator is the GlobalHash operator.If can find that it also is a materialization operator that can be re-used that the B table is created the Hash table though continue downward scanning; But because the B table is created the Hash table is the subtree of the connection result of A, B being created the Hash table; We stop to continue downward scanning after searching out a reusable operator, therefore the B table being created the Hash table will not be re-used.
Step 3:
The inquiry plan that step 2 is generated carries out the parallelization processing, generates to be used for the plan that a plurality of thread parallels are carried out.Concrete grammar is a partition table in the scan plan, and partitioned mode is identical if all relate to partition table, so just can duplicate plan with the identical mark of the number of partitions, and replace the subregion master meter in each plan with the subregion sublist, forms a plan forest.As shown in Figure 3, the partition table in this example is P, and the number of partitions is 2.Therefore the plan in the step 2 is copied as 2 parts, and the P table in each plan tree is revised as P1 and P2 respectively.
Step 4:
The plan forest that step 3 is generated carries out overall multiplexing operator merging processing, generates the digraph plan that is used for supplying a plurality of thread parallels execution and reusable materialization operator.Concrete grammar is each plan tree in the scan plan forest, runs into overall materialization operator and just is merged into one to the overall materialization operator of the same position in each plan tree.As shown in Figure 4, the plan forest in this example is merged into following digraph plan.
The planning execution stage:
The execution of parallel plan is actually the concurrent execution of a plurality of threads, and carries out the process of data transmission mutually.The work that materialization operator multiplex technique is done in the planning execution process be exactly have only the actual execution of thread this operator with and under plan, other threads all not have execution, and the result that has been multiplexing.Concrete method is: each thread parallel is carried out the plan part separately in the digraph; First thread that implements overall multiplexing operator is referred to as main thread; Pin the multiplexing operator of this overall situation and real this operator and following plan thereof, other thread waits carried out by main thread.Main thread executes the release afterwards of this operator, and other threads begin reading of data and continuation plan tree separately from the multiplexing operator of this overall situation.Main thread waits for that all plans all read the data that discharge this operator materialization after the data of the overall multiplexing operator that finishes.DFD is as shown in Figure 5 in this example.As can be seen from the figure, two thread parallels are carried out HashJoin separately, and wherein main thread will be accomplished the connection of P1, A, three tables of B, and another thread will be accomplished the connection of the same A of P2, B table.Create the Hash table but have only main thread to carry out the HashJoin of A and B and A, B are connected the result, another thread directly in the GlobalHash operator reading of data be Hashjoin with the P2 table.Obviously, multiplexing through to operator created the Hash table to the B table and only done once with the operation that the result creates the Hash table that is connected to A, B, saved internal memory and cpu resource; Attended operation to A, B has only been done once, has saved cpu resource; The data of A, B table are read only done once, saved the IO resource.
The present invention is the optimization of the parallel data library inquiry of SM framework being carried out flow process, and key features is materialization operator identical in a plurality of threads is shared.Carry out flow process with common parallel query and compare, not only save CPU and memory source, also will lack IO and read, do not increase any cost.Through test, in TPC-H100G benchmark test, use the operator multiplex technique can the cpu resource utilization rate be reduced by 16%, internal memory uses and reduces by 32%, and the IO amount reduces by 35%.
Certainly; The present invention also can have other various embodiments; Under the situation that does not deviate from spirit of the present invention and essence thereof; Those of ordinary skill in the art work as can make various corresponding changes and distortion according to the present invention, but these changes believed and distortion all should belong to the protection domain of the appended claim of the present invention.
Claims (5)
1. the multiplexing implementation method of operator in the parallel database comprises the steps:
Step 1, the common inquiry planing method of use are the inquiry plan that inquiry generates serial, and said inquiry plan is a binary tree structure;
Said inquiry plan is carried out in step 2, top-down scanning, seeks reusable materialization class operator, and change inquiry plan structure, changes thread-level materialization operator into the overall situation multiplexing materialization operator;
Inquiry plan after step 3, the change that step 2 is generated carries out parallelization to be handled, and generates to be used for the plan forest that a plurality of thread parallels are carried out;
Step 4, the plan forest that step 3 is generated carry out overall multiplexing operator merging processing, generate the digraph plan that is used for supplying a plurality of thread parallels execution and reusable materialization operator;
Step 5, each thread parallel are carried out the plan part separately in the said digraph; First thread that implements overall multiplexing operator is referred to as main thread; Pin the multiplexing operator of this overall situation and real this operator and following plan thereof, other thread waits carried out by main thread;
Step 6, said main thread execute the release afterwards of this operator, and other threads begin reading of data and continuation plan tree separately from the multiplexing operator of this overall situation;
Step 7, said main thread wait for that all plans all read the data that discharge this operator materialization after the data of the overall multiplexing operator that finishes.
2. the multiplexing implementation method of operator in the parallel database as claimed in claim 1, wherein step 1 specifically comprises: said inquiry relates to partition table, and the part leaf node is the scanning to partition table.
3. the multiplexing implementation method of operator in the parallel database as claimed in claim 1, wherein step 2 specifically comprises: after finding a reusable materialization class operator, no longer continue scanning subtree downwards; The criterion of reusable materialization operator is: if certain materialization operator and following plan tree thereof do not comprise partition table, this materialization operator can be re-used so.
4. the multiplexing implementation method of operator in the parallel database as claimed in claim 1; Wherein step 3 specifically comprises: partition table in the scan plan; Partitioned mode is identical if all relate to partition table; So just can duplicate plan with the identical mark of the number of partitions, and replace the subregion master meter in each plan with the subregion sublist, form a plan forest.
5. the multiplexing implementation method of operator in the parallel database as claimed in claim 1; Wherein step 4 specifically comprises: each plan tree in the scan plan forest runs into overall materialization operator and just is merged into one to the overall materialization operator of the same position in each plan tree.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN 201110259524 CN102323946B (en) | 2011-09-05 | 2011-09-05 | Implementation method for operator reuse in parallel database |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN 201110259524 CN102323946B (en) | 2011-09-05 | 2011-09-05 | Implementation method for operator reuse in parallel database |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN102323946A true CN102323946A (en) | 2012-01-18 |
| CN102323946B CN102323946B (en) | 2013-03-27 |
Family
ID=45451689
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN 201110259524 Active CN102323946B (en) | 2011-09-05 | 2011-09-05 | Implementation method for operator reuse in parallel database |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN102323946B (en) |
Cited By (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2014040426A1 (en) * | 2012-09-14 | 2014-03-20 | 华为技术有限公司 | Query processing method and device |
| CN103678619A (en) * | 2013-12-17 | 2014-03-26 | 北京国双科技有限公司 | Database index processing method and device |
| CN103748578A (en) * | 2012-07-26 | 2014-04-23 | 华为技术有限公司 | Data distribution method, device, and system |
| CN105630789A (en) * | 2014-10-28 | 2016-06-01 | 华为技术有限公司 | Query plan converting method and device |
| WO2018054221A1 (en) * | 2016-09-23 | 2018-03-29 | Huawei Technologies Co., Ltd. | Pipeline dependent tree query optimizer and scheduler |
| WO2018108000A1 (en) * | 2016-12-16 | 2018-06-21 | Huawei Technologies Co., Ltd. | Database system and method for compiling serial and parallel database query execution plans |
| CN108369591A (en) * | 2015-12-07 | 2018-08-03 | 华为技术有限公司 | System and method for caching and parameterizing ir |
| CN108829735A (en) * | 2018-05-21 | 2018-11-16 | 上海达梦数据库有限公司 | Synchronous method, device, server and the storage medium of parallel executive plan |
| CN110909023A (en) * | 2018-09-17 | 2020-03-24 | 华为技术有限公司 | Query plan acquisition method, data query method and data query device |
| CN112270412A (en) * | 2020-10-15 | 2021-01-26 | 北京百度网讯科技有限公司 | Network operator processing method and device, electronic equipment and storage medium |
| CN112270413A (en) * | 2020-10-22 | 2021-01-26 | 北京百度网讯科技有限公司 | Operator merging method and device, electronic equipment and storage medium |
| CN116644090A (en) * | 2023-07-27 | 2023-08-25 | 天津神舟通用数据技术有限公司 | Data query method, device, equipment and medium |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2002029643A1 (en) * | 2000-10-06 | 2002-04-11 | Whamtech, L.P. | Enhanced boolean processor with parallel input |
| CN101120340A (en) * | 2004-02-21 | 2008-02-06 | 数据迅捷股份有限公司 | Ultra-shared-nothing parallel database |
| CN101187937A (en) * | 2007-10-30 | 2008-05-28 | 北京航空航天大学 | Mode multiplexing isomerous database access and integration method under gridding environment |
-
2011
- 2011-09-05 CN CN 201110259524 patent/CN102323946B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2002029643A1 (en) * | 2000-10-06 | 2002-04-11 | Whamtech, L.P. | Enhanced boolean processor with parallel input |
| CN101120340A (en) * | 2004-02-21 | 2008-02-06 | 数据迅捷股份有限公司 | Ultra-shared-nothing parallel database |
| CN101187937A (en) * | 2007-10-30 | 2008-05-28 | 北京航空航天大学 | Mode multiplexing isomerous database access and integration method under gridding environment |
Non-Patent Citations (2)
| Title |
|---|
| 许新华等: "并行数据库查询优化技术的最新进展", 《计算机工程与设计》, vol. 30, no. 16, 31 December 2009 (2009-12-31), pages 3814 - 3819 * |
| 金树东等: "并行数据库系统PARO的数据装载方法", 《计算机研究与发展》, vol. 35, no. 7, 31 July 1998 (1998-07-31), pages 616 - 621 * |
Cited By (23)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103748578A (en) * | 2012-07-26 | 2014-04-23 | 华为技术有限公司 | Data distribution method, device, and system |
| WO2014040426A1 (en) * | 2012-09-14 | 2014-03-20 | 华为技术有限公司 | Query processing method and device |
| CN103678368B (en) * | 2012-09-14 | 2017-02-08 | 华为技术有限公司 | query processing method and device |
| CN103678619A (en) * | 2013-12-17 | 2014-03-26 | 北京国双科技有限公司 | Database index processing method and device |
| CN103678619B (en) * | 2013-12-17 | 2017-06-30 | 北京国双科技有限公司 | Database index treating method and apparatus |
| CN105630789A (en) * | 2014-10-28 | 2016-06-01 | 华为技术有限公司 | Query plan converting method and device |
| CN105630789B (en) * | 2014-10-28 | 2019-07-12 | 华为技术有限公司 | A kind of inquiry plan method for transformation and device |
| CN108369591B (en) * | 2015-12-07 | 2021-08-13 | 华为技术有限公司 | System and method for caching and parameterizing IR |
| CN108369591A (en) * | 2015-12-07 | 2018-08-03 | 华为技术有限公司 | System and method for caching and parameterizing ir |
| CN109791492A (en) * | 2016-09-23 | 2019-05-21 | 华为技术有限公司 | Assembly line association tree query optimizer and scheduler |
| US10671607B2 (en) | 2016-09-23 | 2020-06-02 | Futurewei Technologies, Inc. | Pipeline dependent tree query optimizer and scheduler |
| WO2018054221A1 (en) * | 2016-09-23 | 2018-03-29 | Huawei Technologies Co., Ltd. | Pipeline dependent tree query optimizer and scheduler |
| WO2018108000A1 (en) * | 2016-12-16 | 2018-06-21 | Huawei Technologies Co., Ltd. | Database system and method for compiling serial and parallel database query execution plans |
| CN108829735A (en) * | 2018-05-21 | 2018-11-16 | 上海达梦数据库有限公司 | Synchronous method, device, server and the storage medium of parallel executive plan |
| CN110909023A (en) * | 2018-09-17 | 2020-03-24 | 华为技术有限公司 | Query plan acquisition method, data query method and data query device |
| WO2020057369A1 (en) * | 2018-09-17 | 2020-03-26 | 华为技术有限公司 | Query plan acquisition method, and data query method and apparatus |
| CN110909023B (en) * | 2018-09-17 | 2021-11-19 | 华为技术有限公司 | Query plan acquisition method, data query method and data query device |
| CN112270412B (en) * | 2020-10-15 | 2023-10-27 | 北京百度网讯科技有限公司 | Network operator processing method and device, electronic equipment and storage medium |
| CN112270412A (en) * | 2020-10-15 | 2021-01-26 | 北京百度网讯科技有限公司 | Network operator processing method and device, electronic equipment and storage medium |
| CN112270413A (en) * | 2020-10-22 | 2021-01-26 | 北京百度网讯科技有限公司 | Operator merging method and device, electronic equipment and storage medium |
| CN112270413B (en) * | 2020-10-22 | 2024-02-27 | 北京百度网讯科技有限公司 | Operator merging method, device, electronic equipment and storage medium |
| CN116644090A (en) * | 2023-07-27 | 2023-08-25 | 天津神舟通用数据技术有限公司 | Data query method, device, equipment and medium |
| CN116644090B (en) * | 2023-07-27 | 2023-11-10 | 天津神舟通用数据技术有限公司 | Data query method, device, equipment and medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN102323946B (en) | 2013-03-27 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN102323946B (en) | Implementation method for operator reuse in parallel database | |
| Hueske et al. | Opening the black boxes in data flow optimization | |
| Chu et al. | From theory to practice: Efficient join query evaluation in a parallel database system | |
| US8984515B2 (en) | System and method for shared execution of mixed data flows | |
| Wang et al. | Multi-query optimization in mapreduce framework | |
| CN103761080A (en) | Structured query language (SQL) based MapReduce operation generating method and system | |
| Giannakouris et al. | MuSQLE: Distributed SQL query execution over multiple engine environments | |
| CN104111936A (en) | Method and system for querying data | |
| US20190179832A1 (en) | Relational database storage system and method for supporting quick query processing with low data redundancy, and method for processing query on basis of relational database storage method | |
| CN104871153A (en) | System and method for flexible distributed massively parallel processing (mpp) database | |
| Guo et al. | Exploiting reuse for gpu subgraph enumeration | |
| Tatemura et al. | Partiqle: An elastic SQL engine over key-value stores | |
| CN108009270A (en) | A kind of text searching method calculated based on distributed memory | |
| Gao et al. | GLog: A high level graph analysis system using MapReduce | |
| Cai et al. | Fusion insight librA: huawei's enterprise cloud data analytics platform | |
| An et al. | Using index in the mapreduce framework | |
| CN105630778A (en) | DB data migration method and system | |
| Munir et al. | Resilientstore: A heuristic-based data format selector for intermediate results | |
| Nam et al. | A parallel query processing system based on graph-based database partitioning | |
| CN103593401A (en) | Code conversion method and device | |
| Suryajaya et al. | A fast large-size production data transformation scheme for supporting smart manufacturing in semiconductor industry | |
| Kläbe et al. | Patched Multi-Key Partitioning for Robust Query Performance. | |
| Douglas et al. | Blind men and an elephant coalescing open-source, academic, and industrial perspectives on BigData | |
| Novak et al. | Proposed architecture for ETL workflow generator | |
| Holland et al. | Distributing an SQL Query over a Cluster of Containers |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant |