CN104391918B - The implementation method of distributed networks database query priority management based on equity deployment - Google Patents
The implementation method of distributed networks database query priority management based on equity deployment Download PDFInfo
- Publication number
- CN104391918B CN104391918B CN201410663305.0A CN201410663305A CN104391918B CN 104391918 B CN104391918 B CN 104391918B CN 201410663305 A CN201410663305 A CN 201410663305A CN 104391918 B CN104391918 B CN 104391918B
- Authority
- CN
- China
- Prior art keywords
- task
- priority
- query
- resource
- queue
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2453—Query optimisation
- G06F16/24534—Query rewriting; Transformation
- G06F16/24549—Run-time optimisation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
Abstract
The present invention provides the implementation method of the distributed networks database query priority management based on equity deployment, comprising：For each query execution node of distributed data base, defined according to the set priority of identical cluster and divided its resource according to certain ratio；The consistent task queue based on cluster priority is established on each execution node, each task queue can manage the query task of certain amount, and cluster provides globally unique task ID for query task, and query task sorts according to task ID in task queue；Each execution node is for task queue, and using identical scheduling method, the task that high priority is fallen out can be more than low priority.The present invention has the advantages and positive effects of：It is other to different priorities that resource is divided in proportion to each execution node unification, to ensure that high priority inquiry can obtain more execution resources；Only need each execution node can be without unified resource management center according to same policy deployment.
Description
Technical field
The invention belongs to distributed networks database query administrative skill field, more particularly, to a kind of point based on equity deployment
The implementation method and device of cloth data base querying priority management.
Background technology
With the fast development of informationization technology, large-scale database system needs processing and the data volume stored increasingly
Greatly, calculating becomes increasingly complex, and the challenge for performance is also increasing, performance, reliability, and the demand of scalability will be more next
Stronger, this when, one centralized database obviously can not meet demand.In order to adapt to the hair of applied business
Exhibition is needed, and distributed data base system is distributed data across on the different nodes of computer network, and these data are logically
Belong to same system, the system can be described as distributed experiment ＆ measurement system.In distributed experiment ＆ measurement system with traditional data
Storehouse is equally also required to set different priority according to different user or user's group, and the user of high priority can possess more moneys
Source, to ensure that it can obtain more preferable service, and low priority user then can be used for execution efficiency no requirement (NR) in some tasks
Perform, can prevent it from excessively taking resource.
For distributed experiment ＆ measurement system, because its resource distribution is in different nodes, and some inquiry also can be possible
Many steps are broken down into, and is assigned to different nodes and performs, then are then often needed for traditional priority implementation method
Resource management that will be unified and task scheduling center, as shown in figure 1, such case can solve the realization of priority, but in fact
It is existing relative complex, and some system resources are often consumed, such as it will collect all node resource states with for different excellent
First level divides, and control centre then needs to dispatch all query tasks of whole cluster, can exist in execution efficiency higher
Realize difficulty.
In summary, existing distributed experiment ＆ measurement system is difficult to realize priority management under the premise of guarantee is efficient.
The content of the invention
In view of the above-mentioned problems, it is an object of the invention to provide a kind of distributed networks database query based on equity deployment is preferential
The implementation method and device of level management, realize efficient Query priority management in distributed experiment ＆ measurement system, to be adapted to
The demand of the other user of different priorities is established in distributed data base.
The design philosophy that the present invention uses is：According to reciprocity deployment mode, in each execution node using same by collection
The principle and task scheduling strategy of group's priority division resource, and appointed by the globally unique task ID of cluster with ensuring to inquire about
The order that business performs.
In order to solve the above technical problems, one aspect of the present invention provides a kind of distributed networks database query based on equity deployment
The implementation method of priority management, comprising：
For each query execution node of distributed data base, defined according to the set priority of identical cluster by it
Resource is divided according to certain ratio, and the high resource ratio of priority is big；
The consistent task queue based on cluster priority is established on each execution node, each task queue can manage
The query task of certain amount, cluster provide globally unique task ID for query task, and query task is in office according to task ID
Sorted in business queue；
Each execution node is for task queue, and using identical scheduling method, priority is from high to low from task queue
In or take different number of task to go to perform, the task that high priority is fallen out can be more than low priority.
Preferably, for each query execution node of distributed data base, its resource is carried out according to certain ratio
It is divided into multiple resource management groups, the corresponding priority of each resource management group, the inquiry of different priorities can be articulated to pair
In the resource management group answered.
Preferably, the task ID can create according to non recounting pattern, and the inquiry for ensureing to arrive first can obtain less task
ID。
Preferably, the task dispatching queue length of each priority allows cluster configuration as parameter, accesses from high to low
Task queue, the task of setting is obtained, as number is less than task dispatching queue length in task queue, then all taken out.
Preferably, the task dispatching queue length is the length of the summation of the once task of falling out of all priority.
Preferably, the resource management group realizes the selection that need to control system resource and its setting using parameter.
Another aspect of the present invention provides a kind of reality of the distributed networks database query priority management based on equity deployment
Existing device, comprising：
Resource control unit, for each query execution node for distributed data base, according to identical cluster both
Fixed priority, which defines, is divided its resource according to certain ratio, and the high resource ratio of priority is big；
Role management unit, for establishing the consistent task queue based on cluster priority on each execution node,
Each task queue can manage the query task of certain amount, and cluster provides globally unique task ID for query task, inquire about
Task sorts according to task ID in task queue；
Task scheduling unit, realize each execution node for task queue, using identical scheduling method, priority from
High to Low from task queue or to take different number of task to go to perform, the task that high priority is fallen out can be more than low priority.
Preferably, role management unit also realizes that the task ID creates according to non recounting pattern, ensures the inquiry arrived first
Less task ID can be obtained.
Preferably, task scheduling unit allows to collect flock mating using the task dispatching queue length of each priority as parameter
Put, access task queue from high to low, obtain the task of setting.
Preferably, the resource control unit realizes the selection that need to control system resource and its setting using parameter.
The present invention has the advantages and positive effects of：
Resource is not divided in proportion to different priorities, to ensure that high priority inquiry can obtain more execution resources；Only
Each execution node is needed to realize that structure is relatively simple without unified resource management center according to same policy deployment
Easily；
Only need each node that performs can be without in unified task scheduling according to same task scheduling strategy deployment
The heart, realize that structure is relatively simple；
Implementation method of the present invention and device are compared with the implementation method of existing centralized management on the premise of effect is ensured
Realize relatively easy, effectively increase use control of the cluster to resource and done with search efficiency and to the execution sequence of query task
Effective management.
Brief description of the drawings
Fig. 1 is prior art distributed data base priority resources and task scheduling situation schematic diagram；
Fig. 2 is the implementation method of distributed networks database query priority management of the one embodiment of the invention based on equity deployment
Flow chart；
Fig. 3 is that distributed data base performs node priority resource dividing condition schematic diagram in one embodiment of the invention；
Fig. 4 is that priority tasks queue enters to list intention in distributed data base in one embodiment of the invention；
Fig. 5 is that priority tasks queue is fallen out schematic diagram in distributed data base in one embodiment of the invention；
Fig. 6 sends node for one embodiment of the invention inquiry and issues querying flow figure；
Fig. 7 is that one embodiment of the invention performs node priority task queue and fallen out flow chart.
Embodiment
The specific embodiment of the present invention is elaborated below in conjunction with the accompanying drawings.
Fig. 2 is the implementation method of distributed networks database query priority management of the one embodiment of the invention based on equity deployment
Flow chart, the present invention based on equity deployment distributed networks database query priority management implementation method, as shown in Fig. 2
Comprise the following steps：
Step 1, each query execution node for distributed data base, determine according to the set priority of identical cluster
Justice is divided its resource according to certain ratio, and the high resource ratio of priority is big；
In step 1, for each query execution node of distributed data base, its resource is entered according to certain ratio
Row is divided into multiple resource management groups, the corresponding priority of each resource management group, and the inquiry of different priorities can be articulated to
In corresponding resource management group；
Resource management group described in the embodiment of the present invention realizes the selection that need to control system resource and its setting using parameter；Control
Parameter processed such as CPU using weight can the technology as similar LINUX CGROUP help to realize；
Fig. 3 is that distributed data base performs node priority resource dividing condition schematic diagram in one embodiment of the invention, this
An embodiment is invented by taking the execution node resource in Fig. 3 as an example, the resource category divided in the present embodiment has CPU and DISKIO,
Not limited to this when realizing, individually can also only control CPU, and one shares 8 CPU and DISK in the present embodiment, and cluster refers to
Fixed priority is divided into 0-3 levels, then the division of its priority resources group refers to Fig. 2, and each node can be according to the ratio
Divide resource.
Wherein, the name and division of node resource management group, name can be according to cluster instance names and tenant and priority
It Zu He not name, with careful difference different user.
Step 2, establish the consistent task queue based on cluster priority, each task queue on each execution node
The query task of certain amount can be managed, cluster provides globally unique task ID for query task, and query task is according to task
ID sorts in task queue；Its task ID can be obtained by the ID generators of a cluster；Wherein described in the embodiment of the present invention
Task ID can create according to non recounting pattern, and the inquiry for ensureing to arrive first can obtain less task ID；
Each priority tasks can be put into its corresponding task queue, and be sorted according to its task ID, wait scheduled
Perform；Query task sorts according to the ID in task queue, to ensure that first initiating for task first carries out and each execution node
For the task from same inquiry, position is unanimous on the whole in its queue, synchronously completes as far as possible；
Fig. 4 is that priority query's task is entered to list intention in distributed data base in one embodiment of the invention；The present embodiment
The priority specified is divided into 0-3 levels, a certain inquiry in figure using priority as the 1 and query task ID USER1 users for being 3
Exemplified by, the corresponding tagmeme of the queue of priority 1 can be put into each node that performs by describing the task of the inquiry.
Step 3, each execution node are for task queue, and using identical scheduling method, priority is from high to low from appointing
In business queue or different number of task is taken to go to perform, the task that high priority is fallen out can be more than low priority；
Fig. 5 is that priority query's task is fallen out schematic diagram in distributed data base in one embodiment of the invention, implementation process
In, there is an execution to send queue on each execution node and be used for loading ready execution task, each priority is once
The task number that poll is fallen out during taking task is different, and high-priority queue can go out on missions number lower priority can be more,
To ensure that high-priority task can faster perform completion.
The task dispatching queue length of each priority of the embodiment of the present invention allows cluster configuration as parameter, from high to low
Task queue is accessed, obtains the task of setting, as number is less than task dispatching queue length in task queue, then all taking-ups；
Task dispatching queue length described in the present embodiment is the length of the summation of the once task of falling out of all priority.
Fig. 6 sends node for present invention inquiry and issues querying flow figure, in the inventive method implementation process, as shown in fig. 6,
Inquiry, which sends node and issues inquiry, to be comprised the following steps：
Step 501, be according to query statement task resolution and formulation and implementation plan, i.e., the inquiry divide how many steps execution with
And performed to those execution nodes；
Step 502, globally unique all tasks of task ID dispensing are obtained；
Step 503, transfer query task corresponding to this step and node is performed corresponding to；
Step 504, tasks carrying is waited to complete；
Step 505, untill query task is fully completed；
Wherein step 502-505 is realized issues task to execution node step by step, and waits its completion.
Fig. 7 is that query execution node priority queue task is fallen out flow chart, as shown in fig. 7, step 601-604 is traversal
All priority queries take out the process of the tasks carrying of respective number.
The realization device of distributed networks database query priority management of the present invention based on equity deployment, comprising：
Resource control unit, for each query execution node for distributed data base, according to identical cluster both
Fixed priority, which defines, is divided its resource according to certain ratio, and the high resource ratio of priority is big；
Role management unit, for establishing the consistent task queue based on cluster priority on each execution node,
Each task queue can manage the query task of certain amount, and cluster provides globally unique task ID for query task, inquire about
Task sorts according to task ID in task queue；
Task scheduling unit, realize each execution node for task queue, using identical scheduling method, priority from
High to Low from task queue or to take different number of task to go to perform, the task that high priority is fallen out can be more than low priority.
Role management unit of the present invention also realizes that the task ID creates according to non recounting pattern, and ensure to arrive first looks into
Inquiry can obtain less task ID.
The task dispatching queue length of each priority is allowed cluster by task scheduling unit of the present invention
Configuration, accesses task queue, obtains the task of setting from high to low.
Resource control unit of the present invention realizes the selection that need to control system resource and its setting using parameter.
One embodiment of the present of invention is described in detail above, but the content is only the preferable implementation of the present invention
Example, it is impossible to be considered as the practical range for limiting the present invention.All equivalent changes made according to the present patent application scope and improvement
Deng, all should still belong to the present invention patent covering scope within.
Claims (10)
1. the implementation method of the distributed networks database query priority management based on equity deployment, it is characterised in that include：
For each query execution node of distributed data base, defined according to the set priority of identical cluster by its resource
Divided according to certain ratio, the high resource ratio of priority is big；
The consistent task queue based on cluster priority is established on each execution node, each task queue can manage necessarily
The query task of number, cluster provide globally unique task ID for query task, and query task is according to task ID in task team
Sorted in row；
Each execution node is for task queue, and using identical scheduling method, priority obtains from task queue from high to low
Different number of task is taken to go to perform, the task that high priority is fallen out can be more than low priority.
2. the implementation method of the distributed networks database query priority management according to claim 1 based on equity deployment,
It is characterized in that：For each query execution node of distributed data base, its resource is divided according to certain ratio
For multiple resource management groups, the corresponding priority of each resource management group, corresponding to the inquiries of different priorities can be articulated to
In resource management group.
3. the implementation method of the distributed networks database query priority management according to claim 1 based on equity deployment,
It is characterized in that：The task ID can create according to non recounting pattern, and the inquiry for ensureing to arrive first can obtain less task ID.
4. the implementation method of the distributed networks database query priority management according to claim 1 based on equity deployment,
It is characterized in that：The task dispatching queue length of each priority allows cluster configuration as parameter, accesses task from high to low
Queue, the task of setting is obtained, as number is less than task dispatching queue length in task queue, then all taken out.
5. the implementation method of the distributed networks database query priority management according to claim 4 based on equity deployment,
It is characterized in that：The task dispatching queue length is the length of the summation of the once task of falling out of all priority.
6. the implementation method of the distributed networks database query priority management according to claim 2 based on equity deployment,
It is characterized in that：The resource management group realizes the selection that need to control system resource and its setting using parameter.
7. the realization device of the distributed networks database query priority management based on equity deployment, it is characterised in that include：
Resource control unit, it is set according to identical cluster for each query execution node for distributed data base
Priority, which defines, is divided its resource according to certain ratio, and the high resource ratio of priority is big；
Role management unit, for establishing the consistent task queue based on cluster priority on each execution node, each
Task queue can manage the query task of certain amount, and cluster provides globally unique task ID, query task for query task
Sorted according to task ID in task queue；
Task scheduling unit, realize each execution node for task queue, using identical scheduling method, priority from height to
Low different number of task is obtained from task queue to go to perform, the task that high priority is fallen out can be more than low priority.
8. the realization device of the distributed networks database query priority management according to claim 7 based on equity deployment,
It is characterized in that：Role management unit also realizes that the task ID creates according to non recounting pattern, and the inquiry for ensureing to arrive first can obtain
Obtain less task ID.
9. the realization device of the distributed networks database query priority management according to claim 7 based on equity deployment,
It is characterized in that：The task dispatching queue length of each priority is allowed cluster configuration by task scheduling unit, from
High to Low access task queue, obtain the task of setting.
10. the realization device of the distributed networks database query priority management according to claim 7 based on equity deployment,
It is characterized in that：The resource control unit realizes the selection that need to control system resource and its setting using parameter.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410663305.0A CN104391918B (en) | 2014-11-19 | 2014-11-19 | The implementation method of distributed networks database query priority management based on equity deployment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410663305.0A CN104391918B (en) | 2014-11-19 | 2014-11-19 | The implementation method of distributed networks database query priority management based on equity deployment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN104391918A CN104391918A (en) | 2015-03-04 |
| CN104391918B true CN104391918B (en) | 2018-01-19 |
Family
ID=52609822
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201410663305.0A Active CN104391918B (en) | 2014-11-19 | 2014-11-19 | The implementation method of distributed networks database query priority management based on equity deployment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN104391918B (en) |
Families Citing this family (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105511950A (en) * | 2015-12-10 | 2016-04-20 | 天津海量信息技术有限公司 | Dispatching management method for task queue priority of large data set |
| CN105550038A (en) * | 2015-12-12 | 2016-05-04 | 天津南大通用数据技术股份有限公司 | Equivalently deployed distributed database resource management and load adjustment method |
| US10382348B2 (en) * | 2016-11-23 | 2019-08-13 | General Motors Llc | Time ordered message serializer |
| CN106991164A (en) * | 2017-03-31 | 2017-07-28 | 北京京东金融科技控股有限公司 | Method, device and electronic equipment that finance data is handled are used for based on block chain |
| CN107239342A (en) * | 2017-05-31 | 2017-10-10 | 郑州云海信息技术有限公司 | A kind of storage cluster task management method and device |
| CN108156086B (en) * | 2017-12-19 | 2022-04-22 | 北京奇艺世纪科技有限公司 | Policy rule issuing method and device |
| CN109976910A (en) * | 2019-03-20 | 2019-07-05 | 跬云(上海)信息科技有限公司 | Querying method and device based on precomputation OLAP model |
| CN110750350B (en) * | 2019-10-29 | 2022-08-16 | 广东浪潮大数据研究有限公司 | Large resource scheduling method, system, device and readable storage medium |
| CN111736965A (en) * | 2019-12-11 | 2020-10-02 | 西安宇视信息科技有限公司 | Task scheduling method and device, scheduling server and machine-readable storage medium |
| CN111190932B (en) * | 2019-12-16 | 2023-08-18 | 北京淇瑀信息科技有限公司 | Privacy cluster query method and device and electronic equipment |
| CN113364825B (en) * | 2020-03-06 | 2022-12-06 | 联通系统集成有限公司 | Distributed resource integration system |
| CN113781063A (en) * | 2020-12-30 | 2021-12-10 | 北京京东振世信息技术有限公司 | User resource processing method and device |
| CN113495923A (en) * | 2021-02-09 | 2021-10-12 | 深圳市云网万店科技有限公司 | Scheduling management method and system for distributed database executor |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102243598A (en) * | 2010-05-14 | 2011-11-16 | 深圳市腾讯计算机系统有限公司 | Task scheduling method and system in distributed data warehouse |
| CN102387173A (en) * | 2010-09-01 | 2012-03-21 | 中国移动通信集团公司 | MapReduce system and method and device for scheduling tasks thereof |
| CN102567086A (en) * | 2010-12-30 | 2012-07-11 | 中国移动通信集团公司 | Task scheduling method, equipment and system |
| CN103123652A (en) * | 2013-03-14 | 2013-05-29 | 曙光信息产业(北京)有限公司 | Data query method and cluster database system |
| CN103902646A (en) * | 2013-12-27 | 2014-07-02 | 北京天融信软件有限公司 | Distributed task managing system and method |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| GB0108077D0 (en) * | 2001-03-30 | 2001-05-23 | British Telecomm | Database management system |
| US8712972B2 (en) * | 2009-09-22 | 2014-04-29 | Sybase, Inc. | Query optimization with awareness of limited resource usage |
-
2014
- 2014-11-19 CN CN201410663305.0A patent/CN104391918B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102243598A (en) * | 2010-05-14 | 2011-11-16 | 深圳市腾讯计算机系统有限公司 | Task scheduling method and system in distributed data warehouse |
| CN102387173A (en) * | 2010-09-01 | 2012-03-21 | 中国移动通信集团公司 | MapReduce system and method and device for scheduling tasks thereof |
| CN102567086A (en) * | 2010-12-30 | 2012-07-11 | 中国移动通信集团公司 | Task scheduling method, equipment and system |
| CN103123652A (en) * | 2013-03-14 | 2013-05-29 | 曙光信息产业(北京)有限公司 | Data query method and cluster database system |
| CN103902646A (en) * | 2013-12-27 | 2014-07-02 | 北京天融信软件有限公司 | Distributed task managing system and method |
Also Published As
| Publication number | Publication date |
|---|---|
| CN104391918A (en) | 2015-03-04 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN104391918B (en) | The implementation method of distributed networks database query priority management based on equity deployment | |
| CN104461740B (en) | A kind of cross-domain PC cluster resource polymerization and the method for distribution | |
| Wang et al. | Multi-resource fair allocation in heterogeneous cloud computing systems | |
| Razaque et al. | Task scheduling in cloud computing | |
| CN102387173B (en) | MapReduce system and method and device for scheduling tasks thereof | |
| US20200174844A1 (en) | System and method for resource partitioning in distributed computing | |
| CN107710200A (en) | System and method for the operator based on hash in parallelization SMP databases | |
| CN110383764A (en) | The system and method for usage history data processing event in serverless backup system | |
| US10027596B1 (en) | Hierarchical mapping of applications, services and resources for enhanced orchestration in converged infrastructure | |
| CN103023980B (en) | A kind of method and system of cloud platform processes user service request | |
| CN107450977A (en) | The resource management dispatching method towards GPGPU clusters based on YARN | |
| CN106557471A (en) | Method for scheduling task and device | |
| CN105450684B (en) | Cloud computing resource scheduling method and system | |
| CN110187960A (en) | A kind of distributed resource scheduling method and device | |
| CN108427602B (en) | Distributed computing task cooperative scheduling method and device | |
| CN106790332B (en) | Resource scheduling method, system and main node | |
| CN104881322A (en) | Method and device for dispatching cluster resource based on packing model | |
| CN110311965A (en) | Method for scheduling task and system under a kind of cloud computing environment | |
| Wang et al. | Task scheduling algorithm based on improved Min-Min algorithm in cloud computing environment | |
| CN104112049A (en) | P2P (peer-to-peer) architecture based cross-data-center MapReduce task scheduling system and P2P architecture based cross-data-center MapReduce task scheduling method | |
| Ke et al. | Aggregation on the fly: Reducing traffic for big data in the cloud | |
| Wang et al. | Dependency-aware network adaptive scheduling of data-intensive parallel jobs | |
| CN103617083B (en) | Store dispatching method and system, job scheduling method and system and management node | |
| Shu-Jun et al. | Optimization and research of hadoop platform based on fifo scheduler | |
| Fiore et al. | A cluster-based data-centric model for network-aware task scheduling in distributed systems |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |