WO2020211300A1 - 资源分配方法、装置、计算机设备和存储介质 - Google Patents

资源分配方法、装置、计算机设备和存储介质 Download PDF

Info

Publication number
WO2020211300A1
WO2020211300A1 PCT/CN2019/109355 CN2019109355W WO2020211300A1 WO 2020211300 A1 WO2020211300 A1 WO 2020211300A1 CN 2019109355 W CN2019109355 W CN 2019109355W WO 2020211300 A1 WO2020211300 A1 WO 2020211300A1
Authority
WO
WIPO (PCT)
Prior art keywords
current
resource
resource consumption
query request
consumption value
Prior art date
Application number
PCT/CN2019/109355
Other languages
English (en)
French (fr)
Inventor
汪金忠
Original Assignee
苏宁云计算有限公司
苏宁易购集团股份有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 苏宁云计算有限公司, 苏宁易购集团股份有限公司 filed Critical 苏宁云计算有限公司
Priority to CA3177212A priority Critical patent/CA3177212A1/en
Publication of WO2020211300A1 publication Critical patent/WO2020211300A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/24—Querying
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
    • G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
    • G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
    • H04L41/08—Configuration management of networks or network elements
    • H04L41/0896—Bandwidth or capacity management, i.e. automatically increasing or decreasing capacities
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/10—Protocols in which an application is distributed across nodes in the network
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/50—Network services
    • H04L67/56—Provisioning of proxy services
    • H04L67/562—Brokering proxy services

Definitions

  • the present invention relates to the field of database technology, in particular to a resource allocation method, device, computer equipment and storage medium.
  • Online analysis and processing is mainly a fast software technology that uses distributed systems to share multi-dimensional information and target specific problems on-line data access and analysis.
  • FAIR First Input First Output
  • some queries consume extremely large resources and take extremely long execution time. If several such query requests are executed concurrently, it is easy to fill up the resources of the cluster. This will make other queries unable to obtain sufficient resources, hinder the execution of these query requests (even if the resource occupied by this query request is very small, the execution is also very fast), and affect the user experience; the serious one may cause the application memory to run out (Out Of Memory, OOM) or the entire cluster crashes.
  • OOM Out Of Memory
  • a resource allocation method includes:
  • resources are allocated to the current query request.
  • the foregoing obtaining the expected resource consumption value of the current query request includes:
  • determining the expected resource consumption value according to the execution plan tree described above includes:
  • determining the expected resource consumption value according to the execution plan tree described above includes:
  • the preset table statistics, preset field statistics, and execution plan tree determine the number of query rows and data input and output costs of the target data table.
  • the target data table is the data that needs to be used in the current query request table;
  • preset table statistics determine the connection cost of the current query request, the number of rows after the connection, the grouping cardinality, and the CPU cost;
  • the foregoing obtaining the expected resource consumption value of the current query request includes:
  • the execution plan tree is sent to the cost analysis server to obtain the expected resource consumption value returned by the cost analysis server, and the expected resource consumption is determined according to the execution plan tree.
  • the foregoing resource allocation for the current query request according to the current resource consumption level includes:
  • the current resource consumption level determine the current query request queue, where query requests of different resource consumption levels are allocated to different queues
  • the current query request is allocated resources.
  • the above method further includes:
  • the foregoing resource allocation for the current query request according to the current resource consumption level includes:
  • the current query request is allocated to the cluster corresponding to the current resource consumption level, wherein each cluster uses a fair scheduling method to allocate resources to the allocated query request.
  • the above method further includes:
  • the above method further includes:
  • the execution log obtain the actual resource consumption value, actual execution time, scheduling time and waiting time of the current query request;
  • actual resource consumption value actual execution time, scheduling time and waiting time, adjust the correspondence between the expected resource consumption value and the resource consumption level, or adjust the calculation method of the expected resource consumption value.
  • a resource allocation device which includes:
  • the consumption value determination module is used to receive the current query request and obtain the expected resource consumption value of the current query request;
  • the consumption level determination module is used to determine the current resource consumption level of the current query request according to the expected resource consumption value
  • the resource allocation module is used to allocate resources for the current query request according to the current resource consumption level.
  • a computer device includes a memory, a processor, and a computer program stored on the memory and running on the processor.
  • the processor implements the following steps when the processor executes the computer program:
  • resources are allocated to the current query request.
  • the computer program is executed by a processor, the following steps are implemented:
  • resources are allocated to the current query request.
  • the above resource allocation method, device, computer equipment and storage medium receive the current query request, obtain the expected resource consumption value of the current query request, determine the current resource consumption level of the current query request according to the expected resource consumption value, and according to the current resource consumption level, Allocate resources for the current query request. In this way, the isolated allocation of resources is realized.
  • query requests with relatively low resource consumption will not fail to obtain resources due to query requests with relatively high resource consumption, while query requests with relatively high resource consumption are in most cases It can also be executed normally, and can improve the stability and overall performance of the system when the resource-intensive query requests appear in a concentrated manner.
  • Figure 1 is an application environment diagram of a resource allocation method in an embodiment
  • FIG. 2 is a schematic flowchart of a resource allocation method in an embodiment
  • FIG. 3 is a schematic flowchart of steps for obtaining an expected resource consumption value in an embodiment
  • FIG. 4 is a schematic flowchart of steps for obtaining an expected resource consumption value in another embodiment
  • FIG. 5 is a schematic flowchart of a step of determining an expected resource consumption value in an embodiment
  • FIG. 6 is a schematic flowchart of a step of determining an expected resource consumption value in another embodiment
  • FIG. 7 is a schematic flowchart of a resource allocation step in an embodiment
  • FIG. 8 is a schematic flowchart of a corresponding relationship adjustment step or a calculation method adjustment step in an embodiment
  • FIG. 9 is a schematic structural principle diagram of a resource consumption analysis model in an embodiment
  • FIG. 10 is a sequence diagram of a resource allocation method in an embodiment
  • FIG. 11 is a schematic flowchart of query distribution in an embodiment
  • Figure 12 is a structural block diagram of a resource allocation method in an embodiment
  • Figure 13 is an internal structure diagram of a computer device in an embodiment.
  • the resource allocation method provided by the present invention can be applied to the application environment shown in FIG. 1.
  • the terminal 102 and the server 104 communicate through a network
  • the server 104 and the cluster 106 communicate through a wired or wireless manner.
  • the terminal 102 can be, but is not limited to, various personal computers, notebook computers, smart phones, tablet computers, and portable wearable devices.
  • the server 104 can be implemented as an independent server or a server cluster composed of multiple servers.
  • the cluster 106 is To provide resources, the number may be one or more.
  • the cluster 106 generally includes multiple computer devices (machines).
  • the resource in the resource allocation method provided by the present invention may refer to a certain cluster in multiple clusters, or a certain machine in a cluster.
  • the resource allocation method provided by the present invention is generally applied to an online analysis and processing system.
  • a resource allocation method is provided.
  • the method is applied to the server in FIG. 1 as an example for description, including the following steps:
  • Step 202 Receive the current query request, and obtain the expected resource consumption value of the current query request;
  • the current query request generally refers to an online analytical processing query, and may also be an online transaction processing (OLTP) request and other types of connection query requests.
  • OTP online transaction processing
  • the expected resource consumption value refers to the resource value expected to be consumed to complete the query task corresponding to the current query request.
  • the server obtains the current query request (for example, receives the current query request sent by the terminal), analyzes the current query request, and obtains the expected resource consumption of the current query request according to the analysis result of the analysis and analysis. value. It can also be that the server obtains the current query request and obtains the expected resource consumption value of the current query request from the preprocessing server (for example, the cost analysis server), where the preprocessing server is used to perform resource consumption values for various query requests analysis.
  • the preprocessing server for example, the cost analysis server
  • Step 204 Determine the current resource consumption level of the current query request according to the expected resource consumption value
  • the number of resource consumption levels and the corresponding relationship with the resource consumption value can be set according to actual conditions and actual needs.
  • the first resource consumption threshold value and the second resource consumption threshold value three resource consumption levels can be divided into high resource consumption level, medium resource consumption level and low resource consumption level.
  • the first resource consumption threshold is greater than the second resource consumption threshold, and when the expected resource consumption value is greater than the first resource consumption threshold, it is a high resource consumption level; when the expected resource consumption value is within the first resource consumption threshold When the limit value is between the second resource consumption threshold value, it is a medium resource consumption level; when the expected resource consumption value is less than the second resource consumption threshold value, it is a low resource consumption level.
  • Step 206 Perform resource allocation to the current query request according to the current resource consumption level.
  • the current query request is allocated to the resource cluster or queue corresponding to the current resource consumption level, where query requests of different resource consumption levels are allocated to different resource clusters or queues.
  • the current query request is received, the expected resource consumption value of the current query request is obtained, the current resource consumption level of the current query request is determined according to the expected resource consumption value, and the current query request is resourced according to the current resource consumption level.
  • Allocation in this way, can realize the isolated allocation of resources.
  • the query request with relatively low resource consumption will not be unable to obtain resources due to the query request with relatively high resource consumption, and the query request with relatively high resource consumption can be executed normally in most cases, and can be executed in most cases.
  • resource-intensive query requests appear in a centralized manner, the stability and overall performance of the system are improved.
  • the above-mentioned obtaining the expected resource consumption value of the current query request may include the following steps:
  • Step 302 parse the current query request to obtain the execution plan tree of the current query request
  • the current query request generally includes query statements, for example, structured query statements, that is, SQL (Structured Query Language, structured query language) statements.
  • structured query statements that is, SQL (Structured Query Language, structured query language) statements.
  • cost-based optimization (Cost-Based Logical Optimizer, CBO) is used to optimize the execution plan.
  • CBO Cost-Based Logical Optimizer
  • the main purpose of the cost-based optimization method is to generate an efficient execution path by detecting the tables and conditions used in the query, thereby reducing resource utilization and reducing query execution time.
  • a query statement can be parsed into an execution plan tree, and each node on the execution plan tree represents an operation in the query, such as scanning, projection, or filtering.
  • Step 304 Determine the expected resource consumption value according to the execution plan tree.
  • the server may first calculate the input/output (i/o) cost, central processing unit (CPU) cost, and storage cost of each operation on the execution plan tree. (Memory) cost and network (NETWORK) cost; then, the input and output cost, central processing unit cost, storage cost, and network cost of each operation are summed to obtain the resource consumption value of each operation. Finally, for each operation The resource consumption value of is summed to obtain the expected resource consumption value of the current query request.
  • i/o input/output
  • CPU central processing unit
  • NETWORK network cost
  • the input/output cost refers to the cost required to read data from disk into memory (this cost is the most important for query, so a basic principle in optimization is to reduce the total number of input/output);
  • CPU cost It refers to the cost of processing the data in the memory. Once the data is read into the memory, when the memory recognizes the required data, it will perform a sort or join on the data, which requires central processing.
  • the storage cost is the cost required to store data. For accessing remote nodes, the cost of networking is also very high.
  • the expected resource consumption value of the current query request is determined, which is efficient and fast.
  • the above-mentioned determining the expected resource consumption value according to the execution plan tree may include the following steps:
  • Step 402 Determine the resource consumption value of each node in the execution plan tree according to preset table statistical information and preset field statistical information;
  • the table statistical information may include table information such as the number of rows in the table, the size of the table counted in bytes, the size of the table's partition, and the number of partitions, but it is not limited to this information.
  • the field statistical information can include field information such as the base of the field (count distinct), the maximum value of the field, the lowest value of the field, the number of records with empty content (Null count), and the average length of the field, but it is not limited to this information. .
  • the collected information includes but is not limited to the various table statistical information and various field statistical information mentioned above.
  • the data table is the target data table below.
  • the preset table statistics and preset field statistics determine the table statistics and field statistics of the target data table; finally, calculate the execution plan tree based on the table statistics and field statistics of the target data table.
  • the resource consumption value of each node is also the resource consumption value of each operation mentioned above.
  • Step 404 Determine the expected resource consumption value according to the resource consumption value of each node.
  • the resource consumption value of each node in each execution plan tree can be summed to obtain the expected resource consumption value of the current query request.
  • the table statistics and field statistics are collected in advance, and the expected resource consumption value of the current query request is determined based on the execution plan tree, which can improve the calculation efficiency of the expected resource consumption value.
  • the above-mentioned determining the expected resource consumption value according to the execution plan tree may include the following steps:
  • Step 502 Determine the number of query rows and data input and output costs of the target data table according to the preset table statistics, preset field statistics, and execution plan tree.
  • the target data table is required for the current query request.
  • the data table and conditions used in the query of the current query request can be checked in a cost-based optimization manner, and the conditions include filter conditions.
  • the number of query rows and data input and output costs of each data table that need to be used in the query of the current query request can be determined according to the preset table statistical information, the preset field statistical information, and the filter condition.
  • Step 504 Determine the connection cost of the current query request, the number of rows after the connection, the grouping base, and the central processing unit cost according to the preset table statistics, the preset field statistics, and the execution plan tree;
  • connection cost can be calculated according to the join relationship that needs to be used in the query of the current query request; it can be based on the group by clause and the level of the dimension in the execution plan tree Relations, as well as preset table statistics and preset field statistics, calculate the aggregated cardinality (carndinality), that is, the grouping cardinality; the CPU cost can be determined according to the summary function and sorting information in the execution plan tree.
  • Step 506 Determine the expected resource consumption value according to the number of query rows, the data input and output cost, the connection cost, the number of connected rows, the grouping cardinality, and the central processing unit cost;
  • the number of query rows, data input and output costs, connection costs, number of connected rows, grouping cardinality, and central processing unit cost can be aggregated and analyzed to obtain the expected resource consumption value of the current query request.
  • one is to perform resource consumption statistics based on nodes in the execution plan tree, and the other is to perform resource consumption statistics based on various types of costs. According to needs, you can choose any one of them to calculate the expected resource consumption value. Calculations can also be used to calculate the expected resource consumption value by combining these two methods.
  • the table statistics and field statistics are collected in advance, and the expected resource consumption value of the current query request is determined based on the execution plan tree, which can improve the calculation efficiency of the expected resource consumption value.
  • the above-mentioned determining the expected resource consumption value according to the execution plan tree may include the following steps:
  • Step 602 parse the current query request to obtain the execution plan tree of the current query request
  • step 602 reference may be made to the description of step 302, which is not repeated here.
  • Step 604 Send the execution plan tree to the cost analysis server to obtain the expected resource consumption value returned by the cost analysis server, and the expected resource consumption is determined according to the execution plan tree;
  • the method for the cost analysis server to determine the expected resource consumption value may adopt the method of determining the expected resource consumption value according to the execution plan tree in the above two embodiments.
  • the execution plan tree is obtained by parsing the current query request, and the execution plan tree is sent to the cost analysis server to calculate the expected resource consumption value. In this way, the expected resource can be improved when the query request is highly concurrent. Consumption acquisition efficiency.
  • the above-mentioned resource allocation to the current query request according to the current resource consumption level may include the following steps:
  • Step 702 Determine the queue of the current query request according to the current resource consumption level, where query requests with different resource consumption levels are allocated to different queues;
  • capacity refers to the number (or maximum number) of query requests waiting to be scheduled for execution. For example, for a situation including a high resource consumption level, a medium resource consumption level, and a low resource consumption level, the low resource consumption level has the largest queue capacity, and the high resource consumption level has the smallest capacity.
  • Step 704 Perform resource allocation for the current query request according to the resource scheduling policy of the queue of the current query request.
  • each queue is configured with a scheduling process to schedule query requests in the queue.
  • the scheduling processes of different queues have different scheduling strategies.
  • the scheduling strategy of a queue with a high resource consumption level allows only one query to be scheduled at the same time. request.
  • the higher the resource consumption level that is, the higher the average resource consumption value of each query request
  • the level is negatively correlated.
  • query requests are placed in different queues according to resource consumption levels. When the queue is full, subsequent query requests are generally set to be discarded. With the solution of this embodiment, it can be ensured that sufficient resources are always available for query requests with relatively low resource consumption, and resources will not be lost due to query requests with relatively high resource consumption. In addition, query requests with relatively high resource consumption can be executed normally in most cases, which can ensure the smooth and orderly operation of the entire system.
  • the resource allocation method of the present application may further include the step of obtaining the resource usage rate of each queue, and when the resource usage rate of the current queue is lower than a preset resource usage rate threshold, the resources of the current queue Adjust to the remaining queues in each of the queues.
  • the current queue can be any queue.
  • adjusting the resources of the current queue to the remaining queues in each of the queues may be adjusting part of the resources of the current queue to one or more of the remaining queues in each of the queues.
  • the resource dynamic adjustment strategy can be set according to actual needs, and the resource dynamic adjustment strategy can specifically include information such as adjustment time period, resource adjustment ratio, and adjustment to several queues.
  • the resource allocation method of the present invention may further include the step of: obtaining the resource demand rate of each queue, and when the resource demand rate of the current queue is higher than a preset resource demand rate threshold, pass the preset The dynamic resource expansion plan adds resources to the current queue.
  • the current queue can be any queue.
  • the resource demand rate is equal to the ratio of the resource demand value to the actual value of the resource.
  • the resource dynamic expansion plan may specifically include information such as the capacity expansion period and the resource value to be increased.
  • the aforementioned resource allocation for the current query request based on the current resource consumption level may include the following steps: according to the preset correspondence between the resource consumption level and the cluster, the current query request is allocated to the current resource consumption The clusters corresponding to the levels, where each cluster uses a fair scheduling method to allocate resources to the allocated query requests.
  • each cluster (node) is preset with different grade labels, such as high, medium, and low.
  • Resource clusters of different levels execute different levels of query requests.
  • the server allocates the current query request to the cluster corresponding to the current resource consumption level according to the preset correspondence between the resource consumption level and the cluster, and the cluster executes the current query request, that is, executes the query task of the current query request.
  • resource clusters of different levels execute different levels of query requests, and query requests with relatively low resource consumption can ensure that sufficient resources are always available, and no query requests with relatively high resource consumption will be obtained. In most cases, query requests that lack resources and consume relatively high resources can be executed normally, which can ensure the smooth and orderly operation of the entire system.
  • the resource allocation method of the present invention may further include the step of obtaining the resource usage rate of each cluster, and when the resource usage rate of the current cluster is lower than a preset resource usage rate threshold, the resources of the current cluster Adjust to the remaining clusters in each of the clusters.
  • the current cluster can be any cluster.
  • adjusting the resources of the current cluster to the remaining clusters in each of the clusters may be adjusting part of the resources of the current cluster to one or more columns of the remaining clusters in each of the clusters.
  • the resource dynamic adjustment strategy can be set according to actual needs, and the resource dynamic adjustment strategy can specifically include information such as adjustment time period, resource adjustment ratio, and adjustment to several clusters.
  • the resource allocation method of the present invention may further include the step of obtaining the resource demand rate of each of the clusters, and when the current resource demand rate of the cluster is higher than a preset resource demand rate threshold, pass the preset The dynamic resource expansion plan adds resources to the current cluster.
  • the current cluster can be any cluster.
  • the resource demand rate is equal to the ratio of the resource demand value to the actual value of the resource.
  • the resource dynamic expansion plan may specifically include information such as the capacity expansion period and the resource value to be increased.
  • the resource allocation method of the present invention may further include the steps:
  • Step 802 Obtain the execution log of the current query request
  • the execution log of the current query request may be recorded by the system, and the execution log refers to the log generated by the actual execution of the current query request.
  • Step 804 According to the execution log, obtain the actual resource consumption value, actual execution duration, scheduling time, and waiting time of the current query request;
  • Step 806 Adjust the correspondence between the expected resource consumption value and the resource consumption level, or adjust the calculation method of the expected resource consumption value according to the actual resource consumption value, the actual execution duration, the scheduling time and the waiting time.
  • algorithms such as machine learning or artificial intelligence can be used to continuously learn and adjust the correspondence between the expected resource consumption value and the resource consumption level, as well as the calculation method of the expected resource consumption value, to improve accuracy and reduce manslaughter or mistakes. Judging the situation.
  • the online analysis and processing engine (for example, the server in the above embodiment) mainly uses the data query system (druid) as the storage of summary statistical data, and can push down the query to the data query system for execution as much as possible, reducing the programming language (spark) The overhead of itself.
  • the data query system druid
  • spark programming language
  • Figure 9 for the consumption analysis model (cost model) of the data query system.
  • the cost model Before query request (SQL statement) scheduling and command distribution, the cost model can be called to analyze the expected resource consumption value of the query request. If it is found that the expected resource consumption value of the query request exceeds the threshold, it can be specially processed to achieve resource isolation and service degradation, and protect the entire online analysis and processing engine.
  • Storage must also have resource isolation and flow control, and now it is mainly to control the resources of aggregate storage.
  • the history node distinguishes different groups (tier).
  • the high-priority group has more resources, and the low-priority group has fewer resources.
  • a data source needs to be allocated to different groups, and the number of high-priority copies is higher. Many, low priority copy data is less
  • Brokers also have different priorities. The number of agents with different priorities is different, and they must be routed to different groups.
  • Postgresql uses the citus distributed cluster solution.
  • the management node (master node) and processing node (worker node) have no concept of priority, but we can artificially distinguish the management nodes: divided into high priority management nodes and low priority Management node.
  • the number of resources of management nodes at each level is also different. Different priority management nodes handle different levels of query requests.
  • the management (master) layer of the online analysis processing engine needs to perceive the priority level of the management nodes in the data query system cluster and the logical structure cluster.
  • the management node information needs to be passed to the management node cluster and the management node Only operate on the resources under your management.
  • Different management node clusters manage different levels, but the same priority level can be managed by multiple management node clusters.
  • the resource here refers to a certain cluster in multiple clusters, or a certain machine in a cluster.
  • the specified resources can only handle certain specific query requests.
  • Resource classification can be achieved by tagging.
  • an online analytical processing engine consists of three clusters
  • the query can be returned within 1-5s (expected resource consumption value is between 100-500);
  • classification method is not limited to the method in this example.
  • Each query request will go through the routing component (route component).
  • the routing component calculates the expected resource consumption value of the query request, and calculates the expected resource consumption value of each query request.
  • the routing component will send the query request to the appropriate cluster according to the configuration value of the expected resource consumption value and the cluster label. For example, for a summary query in a large area, the expected resource consumption value is calculated as 50, and then forwarded to the high-speed cluster; while the query request for calculating the top ten sales of goods is expensive, for example, the expected resource consumption value is 1000, then forwarded to the slow cluster.
  • a dynamic resource expansion plan can be used to increase resources and relieve the tension of resources.
  • a resource allocation device including: a consumption value determination module 1202, a consumption level determination module 1204, and a resource allocation module 1206, wherein:
  • the consumption value determining module 1202 is configured to receive the current query request and obtain the expected resource consumption value of the current query request;
  • the consumption level determining module 1204 is configured to determine the current resource consumption level of the current query request according to the expected resource consumption value
  • the resource allocation module 1206 is configured to allocate resources for the current query request according to the current resource consumption level.
  • the consumption value determination module 1202 may parse the current query request, obtain the execution plan tree of the current query request, and determine the expected resource consumption value according to the execution plan tree.
  • the consumption value determination module 1202 can determine the resource consumption value of each node in the execution plan tree according to preset table statistics and preset field statistics, and determine the expected resource consumption value of each node Resource consumption value.
  • the consumption value determination module 1202 can determine the number of query rows and data input and output costs of the target data table according to preset table statistics, preset field statistics, and execution plan tree.
  • the target data table is The data tables that need to be used in the query of the current query request, according to the preset table statistics, preset field statistics and execution plan tree, determine the connection cost of the current query request, the number of rows after the connection, and the grouping cardinality
  • the central processor cost according to the number of query rows, data input and output cost, connection cost, number of rows after connection, grouping cardinal number and central processor cost, determine the expected resource consumption value.
  • the consumption value determination module 1202 may analyze the current query request, obtain the execution plan tree of the current query request, send the execution plan tree to the cost analysis server, and obtain the expected resource consumption value returned by the cost analysis server. The expected resource consumption is determined according to the execution plan tree.
  • the resource allocation module 1206 can determine the queue of the current query request according to the current resource consumption level, wherein query requests of different resource consumption levels are allocated to different queues, according to the resource scheduling strategy of the queue of the current query request To allocate resources for the current query request.
  • the above-mentioned apparatus may further include a first resource adjustment module or a first resource expansion module:
  • the first resource adjustment module is configured to obtain the resource usage rate of each queue, and when the resource usage rate of the current queue is lower than a preset resource usage rate threshold, adjust the resources of the current queue to the remaining queues in each queue;
  • the first resource expansion module is used to obtain the resource demand rate of each queue, and when the resource demand rate of the current queue is higher than the preset resource demand rate threshold, add resources to the current queue through the preset resource dynamic expansion plan.
  • the resource allocation module 1206 can allocate the current query request to the cluster corresponding to the current resource consumption level according to the preset correspondence between the resource consumption level and the cluster, wherein each cluster adopts a fair scheduling method. Allocate resources for the allocated query request.
  • the above-mentioned apparatus may further include a second resource adjustment module or a second resource expansion module:
  • the second resource adjustment module is used to obtain the resource usage rate of each cluster, and when the resource usage rate of the current cluster is lower than the preset resource usage rate threshold, adjust the resources of the current cluster to the remaining clusters in each cluster;
  • the second resource expansion module is used to obtain the resource demand rate of each cluster, and when the resource demand rate of the current cluster is higher than the preset resource demand rate threshold, add resources to the current cluster through the preset resource dynamic expansion plan.
  • the above-mentioned device may also dynamically adjust the module, which is used to obtain the execution log of the current query request, and obtain the actual resource consumption value, actual execution duration, and scheduling time of the current query request according to the execution log. And waiting time, according to the actual resource consumption value, actual execution time, scheduling time and waiting time, adjust the correspondence between the expected resource consumption value and the resource consumption level, or adjust the calculation method of the expected resource consumption value.
  • Each module in the above-mentioned resource allocation device may be implemented in whole or in part by software, hardware, and a combination thereof.
  • the foregoing modules may be embedded in the form of hardware or independent of the processor in the computer device, or may be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the foregoing modules.
  • a computer device is provided.
  • the computer device may be a server, and its internal structure diagram may be as shown in FIG. 13.
  • the computer equipment includes a processor, a memory, and a network interface connected through a system bus.
  • the processor of the computer device is used to provide calculation and control capabilities.
  • the memory of the computer device includes a non-volatile storage medium and an internal memory.
  • the non-volatile storage medium stores an operating system, a computer program, and a database.
  • the internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium.
  • the network interface of the computer device is used to communicate with an external terminal through a network connection.
  • the computer program is executed by the processor to realize a resource allocation method.
  • FIG. 13 is only a block diagram of part of the structure related to the solution of the present invention, and does not constitute a limitation on the computer device to which the solution of the present invention is applied.
  • the specific computer device may Including more or fewer parts than shown in the figure, or combining some parts, or having a different arrangement of parts.
  • a computer device including a memory, a processor, and a computer program stored in the memory and capable of running on the processor, and the processor implements the following steps when the processor executes the computer program:
  • resources are allocated to the current query request.
  • the processor executes the computer program to implement the above-mentioned step of obtaining the expected resource consumption value of the current query request
  • the following steps are specifically implemented: parsing the current query request to obtain the execution plan tree of the current query request; According to the execution plan tree, determine the expected resource consumption value.
  • the processor when the processor executes the computer program to implement the above-mentioned step of determining the expected resource consumption value according to the execution plan tree, the following steps are specifically implemented: according to preset table statistics and preset field statistics, Determine the resource consumption value of each node in the execution plan tree; determine the expected resource consumption value according to the resource consumption value of each node.
  • the processor executes the computer program to implement the above-mentioned step of determining the expected resource consumption value according to the execution plan tree
  • the following steps are specifically implemented: according to preset table statistics, preset field statistics, and Execute the plan tree to determine the number of query rows and data input and output costs of the target data table.
  • the target data table is the data table that needs to be used in the current query request; according to the preset table statistics and preset field statistics Information and execution plan tree to determine the connection cost of the current query request, the number of rows after the connection, the grouping cardinality and the CPU cost; according to the number of query rows, data input and output costs, connection costs, the number of rows after connection, grouping cardinality and The CPU cost determines the expected resource consumption value.
  • the processor executes the computer program to implement the above-mentioned step of obtaining the expected resource consumption value of the current query request
  • the following steps are specifically implemented: parsing the current query request to obtain the execution plan tree of the current query request;
  • the execution plan tree is sent to the cost analysis server to obtain the expected resource consumption value returned by the cost analysis server, and the expected resource consumption is determined according to the execution plan tree.
  • the processor executes the computer program to implement the above-mentioned step of resource allocation for the current query request according to the current resource consumption level
  • the following steps are specifically implemented: determine the queue of the current query request according to the current resource consumption level , Where query requests of different resource consumption levels are allocated to different queues; according to the resource scheduling strategy of the current query request queue, the current query request is allocated resources.
  • the processor further implements the following steps when executing the computer program:
  • the processor executes the computer program to implement the above-mentioned step of resource allocation to the current query request according to the current resource consumption level
  • the following steps are specifically implemented: according to the correspondence between the preset resource consumption level and the cluster , Allocate the current query request to the cluster corresponding to the current resource consumption level, wherein each cluster uses a fair scheduling method to allocate resources to the allocated query request.
  • the processor further implements the following steps when executing the computer program:
  • the processor further implements the following steps when executing the computer program: obtaining the execution log of the current query request; according to the execution log, obtaining the actual resource consumption value, actual execution time, scheduling time, and waiting time of the current query request; Actual resource consumption value, actual execution time, scheduling time and waiting time, adjust the correspondence between expected resource consumption value and resource consumption level, or adjust the calculation method of expected resource consumption value.
  • a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:
  • resources are allocated to the current query request.
  • the following steps are specifically implemented: parsing the current query request to obtain the execution plan tree of the current query request ; According to the execution plan tree, determine the expected resource consumption value.
  • the following steps are specifically implemented: according to preset table statistics and preset field statistics , Determine the resource consumption value of each node in the execution plan tree; determine the expected resource consumption value according to the resource consumption value of each node.
  • the following steps are specifically implemented: according to preset table statistics, preset field statistics As well as the execution plan tree, determine the number of query rows and data input and output costs of the target data table.
  • the target data table is the data table that needs to be used in the current query request; according to the preset table statistics and preset fields Statistical information and execution plan tree to determine the connection cost of the current query request, the number of connected rows, the grouping cardinality, and the central processing unit cost; according to the number of query rows, data input and output costs, connection costs, connected rows, and grouping cardinals And the CPU cost to determine the expected resource consumption value.
  • the following steps are specifically implemented: parsing the current query request to obtain the execution plan tree of the current query request ; Send the execution plan tree to the cost analysis server to obtain the expected resource consumption value returned by the cost analysis server, and the expected resource consumption is determined according to the execution plan tree.
  • the following steps are specifically implemented: determine the current query request based on the current resource consumption level Queue, in which query requests of different resource consumption levels are allocated to different queues; according to the resource scheduling strategy of the current query request queue, the current query request is allocated resources.
  • the computer program further implements the following steps when being executed by the processor:
  • the following steps are specifically implemented: corresponding to the cluster according to the preset resource consumption level Relationship, the current query request is allocated to the cluster corresponding to the current resource consumption level, wherein each cluster uses a fair scheduling method to allocate resources to the allocated query request.
  • the computer program further implements the following steps when being executed by the processor:
  • the following steps are further implemented: obtaining the execution log of the current query request; according to the execution log, obtaining the actual resource consumption value, actual execution duration, scheduling time, and waiting time of the current query request; According to the actual resource consumption value, actual execution time, scheduling time and waiting time, adjust the correspondence between the expected resource consumption value and the resource consumption level, or adjust the calculation method of the expected resource consumption value.
  • Non-volatile memory may include read only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory.
  • Volatile memory may include random access memory (RAM) or external cache memory.
  • RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous chain Channel (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Signal Processing (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Software Systems (AREA)
  • Computational Linguistics (AREA)
  • Data Mining & Analysis (AREA)
  • Databases & Information Systems (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
  • Management, Administration, Business Operations System, And Electronic Commerce (AREA)

Abstract

本发明涉及一种资源分配方法、装置、计算机设备和存储介质。所述方法包括：接收当前查询请求，获取所述当前查询请求的预期资源消耗值；根据所述预期资源消耗值，确定所述当前查询请求的当前资源消耗等级；根据所述当前资源消耗等级，对所述当前查询请求进行资源分配。采用本方法能够提升系统的稳定性和总体性能。

Description

资源分配方法、装置、计算机设备和存储介质 技术领域
本发明涉及数据库技术领域，特别是涉及一种资源分配方法、装置、计算机设备和存储介质。
背景技术
随着数据库技术的发展，出现了联机分析处理(Online Analytical Processing，OLAP)技术。联机分析处理主要是采用分布式系统进行共享多维信息的、针对特定问题的联机数据访问和分析的快速软件技术。
对于联机分析处理查询，当前分布式系统一般采用先进先出(First Input First Output)或者公平(FAIR)等调度算法来进行资源分配。然而，在联机分析处理查询中，有些查询对资源的消耗特别大，执行时间也特别长。如果同时有几个这样的查询请求并发执行，很容易就会把集群的资源占满。这样轻者会使得其它查询无法获得足够资源，阻碍这些查询请求的执行(即使这个查询请求占用的资源特别小，执行也特别快)，影响用户体验；重者可能导致应用程序内存耗尽(Out Of Memory，OOM)或者整个集群崩溃。可见，传统的先进先出的资源分配方式或者公平的资源分配方式，在对资源消耗大的查询请求集中出现时，会影响系统的稳定性和总体性能。
发明内容
基于此，有必要针对上述技术问题，提供一种能够提升系统的稳定性和总体性能的资源分配方法、装置、计算机设备和存储介质。
一种资源分配方法，该方法包括：
接收当前查询请求，获取当前查询请求的预期资源消耗值；
根据预期资源消耗值，确定当前查询请求的当前资源消耗等级；
根据当前资源消耗等级，对当前查询请求进行资源分配。
在其中一个实施例中，上述的获取当前查询请求的预期资源消耗值，包括：
对当前查询请求进行解析，获得当前查询请求的执行计划树；
根据执行计划树，确定预期资源消耗值。
在其中一个实施例中，上述的根据执行计划树，确定预期资源消耗值，包括：
根据预设的表统计信息和预设的字段统计信息，确定执行计划树中各个节点的资源消耗值；
根据各节点的资源消耗值，确定预期资源消耗值。
在其中一个实施例中，上述的根据执行计划树，确定预期资源消耗值，包括：
根据预设的表统计信息、预设的字段统计信息以及执行计划树，确定目标数据表的查询行数和数据输入输出代价，目标数据表为在进行当前查询请求的查询时需要使用的各数据表；
根据预设的表统计信息、预设的字段统计信息以及执行计划树，确定当前查询请求的连接代价、连接后的行数、分组基数和中央处理器代价；
根据查询行数、数据输入输出代价、连接代价、连接后的行数、分组基数和中央处理器代价，确定预期资源消耗值。
在其中一个实施例中，上述的获取当前查询请求的预期资源消耗值，包括：
对当前查询请求进行解析，获得当前查询请求的执行计划树；
将执行计划树发送给代价分析服务器，获取代价分析服务器返回的预期资源消耗值，预期资源消耗根据执行计划树确定。
在其中一个实施例中，上述的根据当前资源消耗等级，对当前查询请求进行资源分配，包括：
根据当前资源消耗等级，确定当前查询请求的队列，其中，不同资源消耗等级的查询请求分配到不同队列上；
按照当前查询请求的队列的资源调度策略，对当前查询请求进行资源分配。
在其中一个实施例中，上述的方法还包括：
获取各队列的资源使用率，在当前队列的资源使用率低于预设的资源使用率阈值时，将当前队列的资源调整到各队列中的剩余队列；
或者
获取各队列的资源需求率，在当前队列的资源需求率高于预设的资源需求率阈值时，通过预设的资源动态扩容方案为当前队列增加资源。
在其中一个实施例中，上述的根据当前资源消耗等级，对当前查询请求进行资源分配，包括：
根据预设的资源消耗等级与集群的对应关系，将当前查询请求分配给当前资源消耗等级对应的集群，其中，各集群中分别采用公平调度的方式为分配到的查询请求分配资源。
在其中一个实施例中，上述的方法还包括：
获取各集群的资源使用率，在当前集群的资源使用率低于预设的资源使用率阈值时，将当前集群的资源调整到各集群中的剩余集群；
或者
获取各集群的资源需求率，在当前集群的资源需求率高于预设的资源需求率阈值时，通过预设的资源动态扩容方案为当前集群增加资源。
在其中一个实施例中，上述的方法还包括：
获取当前查询请求的执行日志；
根据执行日志，获取当前查询请求的实际资源消耗值、实际执行时长、调度时间和等待时间；
根据实际资源消耗值、实际执行时长、调度时间和等待时间，调整预期资源消耗值和资源消耗等级的对应关系，或者调整预期资源消耗值的计算方式。
一种资源分配装置，该装置包括：
消耗值确定模块，用于接收当前查询请求，获取当前查询请求的预期资源消耗值；
消耗等级确定模块，用于根据预期资源消耗值，确定当前查询请求的当前 资源消耗等级；
资源分配模块，用于根据当前资源消耗等级，对当前查询请求进行资源分配。
一种计算机设备，包括存储器、处理器及存储在存储器上并可在处理器上运行的计算机程序，处理器执行计算机程序时实现以下步骤：
接收当前查询请求，获取当前查询请求的预期资源消耗值；
根据预期资源消耗值，确定当前查询请求的当前资源消耗等级；
根据当前资源消耗等级，对当前查询请求进行资源分配。
一种计算机可读存储介质，其上存储有计算机程序，计算机程序被处理器执行时实现以下步骤：
接收当前查询请求，获取当前查询请求的预期资源消耗值；
根据预期资源消耗值，确定当前查询请求的当前资源消耗等级；
根据当前资源消耗等级，对当前查询请求进行资源分配。
上述资源分配方法、装置、计算机设备和存储介质，接收当前查询请求，获取当前查询请求的预期资源消耗值，根据预期资源消耗值，确定当前查询请求的当前资源消耗等级，根据当前资源消耗等级，对当前查询请求进行资源分配。如此，实现了资源的隔离分配，采用该方案，使得资源消耗比较低的查询请求不会因为资源消耗比较高的查询请求而获取不到资源，而资源消耗比较高的查询请求在大多数情况下也是可以正常执行，能够在对资源消耗大的查询请求集中出现时，提升系统的稳定性和总体性能。
附图说明
图1为一个实施例中资源分配方法的应用环境图；
图2为一个实施例中资源分配方法的流程示意图；
图3为一个实施例中预期资源消耗值的获取步骤的流程示意图；
图4为另一个实施例中预期资源消耗值的获取步骤的流程示意图；
图5为一个实施例中预期资源消耗值的确定步骤的流程示意图；
图6为另一个实施例中预期资源消耗值的确定步骤的流程示意图；
图7为一个实施例中资源分配步骤的流程示意图；
图8为一个实施例中对应关系调整步骤或者计算方式调整步骤的流程示意图；
图9为一个实施例中的资源消耗分析模型的结构原理示意图；
图10为一个实施例中的资源分配方法的时序图；
图11为一个实施例中的查询分发的流程示意图；
图12为一个实施例中资源分配方法的结构框图；
图13为一个实施例中计算机设备的内部结构图。
具体实施方式
为了使本发明的目的、技术方案及优点更加清楚明白，以下结合附图及实施例，对本发明进行进一步详细说明。应当理解，此处描述的具体实施例仅仅 用以解释本发明，并不用于限定本发明。
本发明提供的资源分配方法，可以应用于如图1所示的应用环境中。其中，终端102和服务器104通过网络进行通信，服务器104与集群106通过有线方式或者无线方式进行通信。其中，终端102可以但不限于是各种个人计算机、笔记本电脑、智能手机、平板电脑和便携式可穿戴设备，服务器104可以用独立的服务器或者是多个服务器组成的服务器集群来实现，集群106用于提供资源，其数量可以是一个或者多个，集群106一般包括多台计算机设备(机器)。本发明提供的资源分配方法中的资源可以指的是多集群中的某个集群，或者一个集群中的某台机器。本发明提供的资源分配方法一般应用于联机分析处理系统中。
在一个实施例中，如图2所示，提供了一种资源分配方法，以该方法应用于图1中的服务器为例进行说明，包括以下步骤：
步骤202，接收当前查询请求，获取当前查询请求的预期资源消耗值；
这里，当前查询请求一般是指联机分析处理查询，也可以是联机事务处理(on-line transaction processing，OLTP)请求等其他类型的联结查询请求。
这里，预期资源消耗值是指完成该当前查询请求对应的查询任务，预期所需消耗的资源值。
具体地，可以是服务器在获取到当前查询请求(例如，接收到终端发送的当前查询请求)，对该当前查询请求进行解析分析，根据该解析分析的分析结果获得该当前查询请求的预期资源消耗值。也可以是服务器在获取到当前查询请求，从预处理服务器(例如，代价分析服务器)获取该该当前查询请求的预期资源消耗值，其中，预处理服务器用于对各种查询请求进行资源消耗值分析。
步骤204，根据预期资源消耗值，确定当前查询请求的当前资源消耗等级；
其中，资源消耗等级的数量以及与资源消耗值的对应关系，可以根据实际情况以及实际需要设定。例如，可以根据第一资源消耗门限值和第二资源消耗门限值，划分成高资源消耗级、中资源消耗级和低资源消耗三个资源消耗等级。其中，第一资源消耗门限值大于第二资源消耗门限值，在预期资源消耗值大于第一资源消耗门限值时，为高资源消耗级；在预期资源消耗值处于第一资源消耗门限值和第二资源消耗门限值之间时，为中资源消耗级；在预期资源消耗值小于第二资源消耗门限值时，为低资源消耗级。
步骤206，根据当前资源消耗等级，对当前查询请求进行资源分配。
具体地，根据当前资源消耗等级，将当前查询请求分配给当前资源消耗等级对应的资源集群或者队列，其中，不同资源消耗等级的查询请求被分配给不同的资源集群或者队列。
上述资源分配方法中，是接收当前查询请求，获取当前查询请求的预期资源消耗值，根据预期资源消耗值，确定当前查询请求的当前资源消耗等级，根据当前资源消耗等级，对当前查询请求进行资源分配，如此，可以实现资源的隔离分配。采用实施例的方案，使得资源消耗比较低的查询请求不会因为资源消耗比较高的查询请求而获取不到资源，而资源消耗比较高的查询请求在大多数 情况下也是可以正常执行，能够在对资源消耗大的查询请求集中出现时，提升系统的稳定性和总体性能。
在其中一个实施例中，如图3所示，上述的获取当前查询请求的预期资源消耗值，可以包括如下步骤：
步骤302，对当前查询请求进行解析，获得当前查询请求的执行计划树；
这里，当前查询请求一般包括查询语句，例如，结构化查询语句，也即SQL(Structured Query Language，结构化查询语言)语句。
一般地，对于成熟的联机分析处理系统，都会使用基于代价的优化方式(Cost-Based Logical Optimizer，CBO)来对执行计划进行优化。基于代价的优化方式的主要目的也就是通过检测在查询中使用到的表和条件，来产生高效的执行路径，从而降低资源利用率，减少查询执行时间。一个查询语句可以解析成执行计划树，执行计划树上的每个节点代表查询中的一个操作，例如，扫描、投影或者过滤等。
步骤304，根据执行计划树，确定预期资源消耗值。
具体地，服务器可以首先对该执行计划树上的每个操作，分别计算每个操作的输入/输出(input/output，i/o)代价、中央处理器(Central Processing Unit，CPU)代价、存储(Memory)代价和联网(NETWORK)代价；接着，分别对每个操作的输入输出代价、中央处理器代价、存储代价和联网代价求和，得到每个操作的资源消耗值，最后，对各个操作的资源消耗值进行求和处理，得到当前查询请求的预期资源消耗值。
其中，输入/输出代价是指把数据从磁盘读入内存时所需代价(该代价是查询所需最主要的，所以在优化时一个基本原则就是降低输入/输出总次数)；中央处理器代价是指处理内存中数据所需的代价，其中，数据一旦读入内存，当内存识别出所要的数据后，会在这些数据上执行排序操作(sort)或连接(join)，这需要消耗中央处理器资源；存储代价是值对数据进行存储所需的代价，对于访问远程节点来说，联网代价的花费也是很大的。
本实施例中，通过解析当前查询请求获得执行计划树的方式，确定当前查询请求的预期资源消耗值，高效快捷。
在其中一个实施例中，如图4所示，上述的根据执行计划树，确定预期资源消耗值，可以包括如下步骤：
步骤402，根据预设的表统计信息和预设的字段统计信息，确定执行计划树中各个节点的资源消耗值；
这里，表统计信息可以包括表的行数，以字节统计的表的大小，表的分区大小和分区数目等表信息，但也不限于这些信息。
这里，字段统计信息可以包括字段的基数(count distinct)、字段的最大值、字段的最下值、内容为空的记录数(Null count)和字段平均长度等字段信息，但也不限于这些信息。
具体地，可以在进行资源消耗分析之前，预先对管理的每个表进行统计信息收集，收集的信息包括但不限于是上述提及的各种表统计信息和各种字段统计信息。在进行当前查询请求的资源消耗分析时，可以首先根据该当前查询请 求的执行计划树确定在该当前查询请求中需要使用到的数据表，为了便于描述，以下将该数据表成为目标数据表，其次，从预设的表统计信息和预设的字段统计信息中，确定目标数据表的表统计信息和字段统计信息；最后，根据目标数据表的表统计信息和字段统计信息，计算执行计划树中各个节点的资源消耗值。这里，各个节点的资源消耗值也即至上述的各个操作的资源消耗值。
步骤404，根据各节点的资源消耗值，确定预期资源消耗值。
具体地，可以对各执行计划树中各个节点的资源消耗值进行求和处理，得到当前查询请求的预期资源消耗值。
本实施例中，预先采集了表统计信息和字段统计信息，且基于执行计划树确定当前查询请求的预期资源消耗值，可以提升预期资源消耗值的计算效率。
在其中一个实施例中，如图5所示，上述的根据执行计划树，确定预期资源消耗值，可以包括如下步骤：
步骤502，根据预设的表统计信息、预设的字段统计信息以及执行计划树，确定目标数据表的查询行数和数据输入输出代价，目标数据表为在进行当前查询请求的查询时需要使用的各数据表；
这里的表统计信息和字段统计信息可以参照上一个实施例中的描述，在此不予赘述。
具体地，可以基于代价的优化方式检查当前查询请求的查询中使用到的数据表和条件，该条件包括过滤条件。可以根据预设的表统计信息、预设的字段统计信息以及该过滤条件，确定在进行当前查询请求的查询时需要使用的各数据表的查询行数和数据输入输出代价。
步骤504，根据预设的表统计信息、预设的字段统计信息以及执行计划树，确定当前查询请求的连接代价、连接后的行数、分组基数和中央处理器代价；
具体他，可以根据在进行当前查询请求的查询时需要使用的连接(join)关系，计算连接代价和连接后的行数；可以根据执行计划树中的分组(group by)子句和维度的层级关系，以及预设的表统计信息和预设的字段统计信息，计算汇聚后的基数(carndinality)，也即分组基数；可以根据执行计划树中汇总函数和是否排序等信息确定中央处理器代价。
步骤506，根据查询行数、数据输入输出代价、连接代价、连接后的行数、分组基数和中央处理器代价，确定预期资源消耗值；
具体地，可以对查询行数、数据输入输出代价、连接代价、连接后的行数、分组基数和中央处理器代价进行汇总分析，得到当前查询请求的预期资源消耗值。
上述两个实施例，一种是基于执行计划树中的节点进行资源消耗统计，一种是基于各种类型代价进行资源消耗统计，根据需要，可以选择其中任何一种方式进行预期资源消耗值的计算，也可以通过将这两种方式有机结合的方式进行预期资源消耗值的计算。
本实施例中，预先采集了表统计信息和字段统计信息，且基于执行计划树确定当前查询请求的预期资源消耗值，可以提升预期资源消耗值的计算效率。
在其中一个实施例中，如图6所示，上述的根据执行计划树，确定预期资 源消耗值，可以包括如下步骤：
步骤602，对当前查询请求进行解析，获得当前查询请求的执行计划树；
该步骤602可以参照上述步骤302的描述，在此不予赘述。
步骤604，将执行计划树发送给代价分析服务器，获取代价分析服务器返回的预期资源消耗值，预期资源消耗根据执行计划树确定；
其中，代价分析服务器确定预期资源消耗值的方式，可以采用上述两个实施例中的根据执行计划树，确定预期资源消耗值的方式。
本实施例中，通过解析当前查询请求获得执行计划树的方式，并将执行计划树发送给代价分析服务器进行预期资源消耗值的计算，这种方式，在查询请求高并发时，可以提升预期资源消耗的获取效率。
在其中一个实施例中，如图7，上述的根据当前资源消耗等级，对当前查询请求进行资源分配，可以包括如下步骤：
步骤702，根据当前资源消耗等级，确定当前查询请求的队列，其中，不同资源消耗等级的查询请求分配到不同队列上；
一般地，不同资源消耗等级的查询请求放到不同的队列上，不同资源消耗等级队列的容量也不一样，这里，容量是指等待调度执行的查询请求的数目(或者最大数目)。例如，对于包括高资源消耗等级、中资源消耗等级和低资源消耗等级的情况，低资源消耗等级队列容量最大，高资源消耗等级的容量最小。
步骤704，按照当前查询请求的队列的资源调度策略，对当前查询请求进行资源分配。
其中，每个队列上都配置有调度进程，对队列中的查询请求进行调度，不同队列的调度进程有不同的调度策略，例如，高资源消耗等级的队列的调度策略最多只允许同时调度一个查询请求。一般地，资源消耗等级越高(即每个查询请求的平均资源消耗值越高)的队列，同一时段允许调度的查询请求的数量越少，即同一时段允许调度的查询请求的数量与资源消耗等级负相关。
本实施例的方案，查询请求根据资源消耗等级放到不同的队列中，当队列满了以后，后续的查询请求一般设置为被抛弃掉。采用本实施例的方案，对资源消耗比较低的查询请求能够保证总是可以获得很充足的资源，不会因为有资源消耗比较高的查询请求获取不到资源。且资源消耗比较高的查询请求在大多数情况下也是可以正常执行，可以保障整个系统平稳有序的运行。
在其中一个实施例中，本申请的资源分配方法，还可以包括步骤：获取各队列的资源使用率，在当前队列的资源使用率低于预设的资源使用率阈值时，将当前队列的资源调整到各所述队列中的剩余队列。
这里，当前队列可以是任意一队列。这里，将当前队列的资源调整到各所述队列中的剩余队列，可以是将当前队列的部分资源调整到各所述队列中的剩余队列中的一个或者多个多列中去。
具体地，在当前队列在指定时间内的资源使用率低于预设的资源使用率阈值时，通过预设的资源动态调整策略，将该队列中的一部分资源调整到资源使用率较高的队列上。这里，资源动态调整策略可以根据实际需要设定，资源动态调整策略具体可以包括调整时段、资源调整比率以及向几个队列调整等信息。
采用本实施例的方案，可以实现资源动态调整，进一步提升系统的稳定性和总体性能。
在其中一个实施例中，本发明的资源分配方法，还可以包括步骤：获取各所述队列的资源需求率，在当前队列的资源需求率高于预设的资源需求率阈值时，通过预设的资源动态扩容方案为当前队列增加资源。
这里，当前队列可以是任意一队列。这里，资源需求率等于资源需求值与资源实际值的比值。
具体地，可以在当前队列的资源需求率高于预设的资源需求率阈值时，判定该当前队列的资源处于紧张状态，通过预设的资源动态扩容方案为当前队列增加资源，如此，可以缓解当前队列的资源紧张状况。其中，资源动态扩容方案具体可以包括增容时段以及所需增加的资源值等信息。
采用本实施例的方案，可以实现资源动态扩容，进一步提升系统的稳定性和总体性能。
在其中一个实施例中，上述的根据当前资源消耗等级，对当前查询请求进行资源分配，可以包括如下步骤：根据预设的资源消耗等级与集群的对应关系，将当前查询请求分配给当前资源消耗等级对应的集群，其中，各集群中分别采用公平调度的方式为分配到的查询请求分配资源。
具体地，各集群(节点)被预先设置不同的等级标签，例如高、中、低。不同等级的资源集群执行不同等级的查询请求。服务器根据预设的资源消耗等级与集群的对应关系，将当前查询请求分配给当前资源消耗等级对应的集群，由该集群进行当前查询请求的执行，即执行当前查询请求的查询任务。
本实施例的方案中，不同等级的资源集群执行不同等级的查询请求，对资源消耗比较低的查询请求能够保证总是可以获得很充足的资源，不会因为有资源消耗比较高的查询请求获取不到资源，且资源消耗比较高的查询请求在大多数情况下也是可以正常执行，可以保障整个系统平稳有序的运行。
在其中一个实施例中，本发明的资源分配方法，还可以包括步骤：获取各集群的资源使用率，在当前集群的资源使用率低于预设的资源使用率阈值时，将当前集群的资源调整到各所述集群中的剩余集群。
这里，当前集群可以是任意一集群。这里，将当前集群的资源调整到各所述集群中的剩余集群，可以是将当前集群的部分资源调整到各所述集群中的剩余集群中的一个或者多个多列中去。
具体地，在当前集群在指定时间内的资源使用率低于预设的资源使用率阈值时，通过预设的资源动态调整策略，将该集群中的一部分资源调整到资源使用率较高的集群上。这里，资源动态调整策略可以根据实际需要设定，资源动态调整策略具体可以包括调整时段、资源调整比率以及向几个集群调整等信息。
采用本实施例的方案，可以实现资源动态调整，进一步提升系统的稳定性和总体性能。
在其中一个实施例中，本发明的资源分配方法，还可以包括步骤：获取各所述集群的资源需求率，在当前集群的资源需求率高于预设的资源需求率阈值时，通过预设的资源动态扩容方案为当前集群增加资源。
这里，当前集群可以是任意一集群。这里，资源需求率等于资源需求值与资源实际值的比值。
具体地，可以在当前集群的资源需求率高于预设的资源需求率阈值时，判定该当前集群的资源处于紧张状态，通过预设的资源动态扩容方案为当前集群增加资源，如此，可以缓解当前集群的资源紧张状况。其中，资源动态扩容方案具体可以包括增容时段以及所需增加的资源值等信息。
采用本实施例的方案，可以实现资源动态扩容，进一步提升系统的稳定性和总体性能。
在其中一个实施例中，本发明的资源分配方法，如图8所示，还可以包括步骤：
步骤802，获取当前查询请求的执行日志；
具体地，可以通过系统记录日志当前查询请求的执行日志，该执行日志指实际执行当前查询请求所产生的日志。
步骤804，根据执行日志，获取当前查询请求的实际资源消耗值、实际执行时长、调度时间和等待时间；
步骤806，根据实际资源消耗值、实际执行时长、调度时间和等待时间，调整预期资源消耗值和资源消耗等级的对应关系，或者调整预期资源消耗值的计算方式。
具体地，可以采用机器学习或者人工智能等算法，对预期资源消耗值和资源消耗等级的对应关系，以及预期资源消耗值的计算方式进行不断地学习和调整，以提升准确度，减少误杀或者误判情况。
为了便于理解本发明的方案，以下通过具体示例进行说明。
联机分析处理引擎(例如如上实施例中的服务器)主要使用数据查询系统(druid)作为汇总统计数据的存储，可以尽可能会将查询下压到数据查询系统中来执行，减少编程语言(spark)本身的开销。数据查询系统的消耗分析模型(cost模型)可以参考图9。
在进行查询请求(SQL语句)调度和命令分发前，可以调用cost模型对查询请求的预期资源消耗值进行分析。如果发现该条查询请求的预期资源消耗值超过阈值，可以对其进行特殊化处理，从而达到资源隔离以及服务降级的作用，对整个联机分析处理引擎起到保护作用。
存储也要有资源隔离和流量管控，现在主要是对汇总性存储的资源进行管控。
对于数据查询系统：
历史节点区分不同的组(tier)，高优先级的组的资源较多，低优先级组的资源较少，一个数据源(datasource)要分配到不同的组中，高优先级的副本数较多，低优先级的副本数据较少
代理(Broker)也要划分不同优先级。不同优先级代理资源数也不相同，并且要路由到不同的组中。
对于关系型数据库(例如postgresql)；
postgresql采用citus的分布式集群方案，管理节点(master节点)和处 理节点(worker节点)没有优先级的概念，但是我们可以人为的对管理节点进行区分：分为高优先级的管理节点和低优先级的管理节点。各个级别的管理节点的资源数也是不同的。不同优先级管理节点处理不同级别的查询请求。
联机分析处理引擎的管理(master)层需要感知数据查询系统集群和逻辑结构集群中的管理节点的优先级别，在创建和管理管理节点集群时，需要将管理节点信息传递给管理节点集群，管理节点只对自己管理下的资源进行操作。不同的管理节点集群管理不同级别，但是同一优先级别可以由多个管理节点集群管理。
关于资源分级：
这里的资源指的是多集群中的某个集群，或者一个集群中的某台机器。通过对特定资源进行分级，使得指定资源只处理某些特定的查询请求。资源分级可以通过打标签来实现。
例如，一个联机分析处理引擎由三个集群组成
高速：要求查询能在1s内返回。(预期资源消耗值(cost)在100以下)；
中速：要求查询能在1-5s内返回(预期资源消耗值在100-500之间)；
慢速：查询性能在5s外返回(预期资源消耗值在在500以上)。
当然，分级方式并不限于该举例中的方式。
关于预期资源消耗值的计算：
各查询请求都会经过路由组件(route组件)。路由组件会对查询请求进行预期资源消耗值的计算，算出各查询请求的预期资源消耗值。
关于查询请求分发，参见图10：
路由组件会根据预期资源消耗值的配置值和集群的标签，将查询请求发到合适的集群中。例如，对于大区的汇总查询，预期资源消耗值的计算值为50，则转发到高速集群中；而计算商品销售排行前十的查询请求,则资源代价很大，例如，预期资源消耗值为1000，则转发到慢速集群中。
关于资源动态调整：
当某一集群(队列)中在指定时间内资源使用率在一定阈值下，可以通过资源动态调整策略，将一部分资源调整到资源使用率较高的集群(队列)上。
关于资源动态扩容：
当某一集群(队列)中的资源在一定时间内处于紧张状态，可以通过资源动态扩容方案，增加资源，缓解资源紧张状况。
应该理解的是，虽然图2-8的流程图中的各个步骤按照箭头的指示依次显示，但是这些步骤并不是必然按照箭头指示的顺序依次执行。除非本文中有明确的说明，这些步骤的执行并没有严格的顺序限制，这些步骤可以以其它的顺序执行。而且，图2-8中的至少一部分步骤可以包括多个子步骤或者多个阶段，这些子步骤或者阶段并不必然是在同一时刻执行完成，而是可以在不同的时刻执行，这些子步骤或者阶段的执行顺序也不必然是依次进行，而是可以与其它步骤或者其它步骤的子步骤或者阶段的至少一部分轮流或者交替地执行。
在一个实施例中，如图12所示，提供了一种资源分配装置，包括：消耗值确定模块1202、消耗等级确定模块1204和资源分配模块1206，其中：
消耗值确定模块1202，用于接收当前查询请求，获取当前查询请求的预期资源消耗值；
消耗等级确定模块1204，用于根据预期资源消耗值，确定当前查询请求的当前资源消耗等级；
资源分配模块1206，用于根据当前资源消耗等级，对当前查询请求进行资源分配。
在其中一个实施例中，消耗值确定模块1202可以对当前查询请求进行解析，获得当前查询请求的执行计划树，根据执行计划树，确定预期资源消耗值。
在其中一个实施例中，消耗值确定模块1202可以根据预设的表统计信息和预设的字段统计信息，确定执行计划树中各个节点的资源消耗值，根据各节点的资源消耗值，确定预期资源消耗值。
在其中一个实施例中，消耗值确定模块1202可以根据预设的表统计信息、预设的字段统计信息以及执行计划树，确定目标数据表的查询行数和数据输入输出代价，目标数据表为在进行当前查询请求的查询时需要使用的各数据表，根据预设的表统计信息、预设的字段统计信息以及执行计划树，确定当前查询请求的连接代价、连接后的行数、分组基数和中央处理器代价，根据查询行数、数据输入输出代价、连接代价、连接后的行数、分组基数和中央处理器代价，确定预期资源消耗值。
在其中一个实施例中，消耗值确定模块1202可以对当前查询请求进行解析，获得当前查询请求的执行计划树，将执行计划树发送给代价分析服务器，获取代价分析服务器返回的预期资源消耗值，预期资源消耗根据执行计划树确定。
在其中一个实施例中，资源分配模块1206可以根据当前资源消耗等级，确定当前查询请求的队列，其中，不同资源消耗等级的查询请求分配到不同队列上，按照当前查询请求的队列的资源调度策略，对当前查询请求进行资源分配。
在其中一个实施例中，上述的装置还可以包括第一资源调整模块或者第一资源扩容模块：
第一资源调整模块，用于获取各队列的资源使用率，在当前队列的资源使用率低于预设的资源使用率阈值时，将当前队列的资源调整到各队列中的剩余队列；
第一资源扩容模块，用于获取各队列的资源需求率，在当前队列的资源需求率高于预设的资源需求率阈值时，通过预设的资源动态扩容方案为当前队列增加资源。
在其中一个实施例中，资源分配模块1206可以根据预设的资源消耗等级与集群的对应关系，将当前查询请求分配给当前资源消耗等级对应的集群，其中，各集群中分别采用公平调度的方式为分配到的查询请求分配资源。
在其中一个实施例中，上述的装置还可以包括第二资源调整模块或者第二资源扩容模块：
第二资源调整模块，用于获取各集群的资源使用率，在当前集群的资源使用率低于预设的资源使用率阈值时，将当前集群的资源调整到各集群中的剩余集群；
第二资源扩容模块，用于获取各集群的资源需求率，在当前集群的资源需求率高于预设的资源需求率阈值时，通过预设的资源动态扩容方案为当前集群增加资源。
在其中一个实施例中，上述的装置还可以动态调整模块，该动态调整模块用于获取当前查询请求的执行日志，根据执行日志，获取当前查询请求的实际资源消耗值、实际执行时长、调度时间和等待时间，根据实际资源消耗值、实际执行时长、调度时间和等待时间，调整预期资源消耗值和资源消耗等级的对应关系，或者调整预期资源消耗值的计算方式。
关于资源分配装置的具体限定可以参见上文中对于资源分配方法的限定，在此不再赘述。上述资源分配装置中的各个模块可全部或部分通过软件、硬件及其组合来实现。上述各模块可以硬件形式内嵌于或独立于计算机设备中的处理器中，也可以以软件形式存储于计算机设备中的存储器中，以便于处理器调用执行以上各个模块对应的操作。
在一个实施例中，提供了一种计算机设备，该计算机设备可以是服务器，其内部结构图可以如图13所示。该计算机设备包括通过系统总线连接的处理器、存储器和网络接口。其中，该计算机设备的处理器用于提供计算和控制能力。该计算机设备的存储器包括非易失性存储介质、内存储器。该非易失性存储介质存储有操作系统、计算机程序和数据库。该内存储器为非易失性存储介质中的操作系统和计算机程序的运行提供环境。该计算机设备的网络接口用于与外部的终端通过网络连接通信。该计算机程序被处理器执行时以实现一种资源分配方法。
本领域技术人员可以理解，图13中示出的结构，仅仅是与本发明方案相关的部分结构的框图，并不构成对本发明方案所应用于其上的计算机设备的限定，具体的计算机设备可以包括比图中所示更多或更少的部件，或者组合某些部件，或者具有不同的部件布置。
在一个实施例中，提供了一种计算机设备，包括存储器、处理器及存储在存储器上并可在处理器上运行的计算机程序，处理器执行计算机程序时实现以下步骤：
接收当前查询请求，获取当前查询请求的预期资源消耗值；
根据预期资源消耗值，确定当前查询请求的当前资源消耗等级；
根据当前资源消耗等级，对当前查询请求进行资源分配。
在其中一个实施例中，在处理器执行计算机程序实现上述的获取当前查询请求的预期资源消耗值的步骤时，具体实现以下步骤：对当前查询请求进行解析，获得当前查询请求的执行计划树；根据执行计划树，确定预期资源消耗值。
在其中一个实施例中，在处理器执行计算机程序实现上述的根据执行计划树，确定预期资源消耗值的步骤时，具体实现以下步骤：根据预设的表统计信息和预设的字段统计信息，确定执行计划树中各个节点的资源消耗值；根据各节点的资源消耗值，确定预期资源消耗值。
在其中一个实施例中，在处理器执行计算机程序实现上述的根据执行计划树，确定预期资源消耗值的步骤时，具体实现以下步骤：根据预设的表统计信 息、预设的字段统计信息以及执行计划树，确定目标数据表的查询行数和数据输入输出代价，目标数据表为在进行当前查询请求的查询时需要使用的各数据表；根据预设的表统计信息、预设的字段统计信息以及执行计划树，确定当前查询请求的连接代价、连接后的行数、分组基数和中央处理器代价；根据查询行数、数据输入输出代价、连接代价、连接后的行数、分组基数和中央处理器代价，确定预期资源消耗值。
在其中一个实施例中，在处理器执行计算机程序实现上述的获取当前查询请求的预期资源消耗值的步骤时，具体实现以下步骤：对当前查询请求进行解析，获得当前查询请求的执行计划树；将执行计划树发送给代价分析服务器，获取代价分析服务器返回的预期资源消耗值，预期资源消耗根据执行计划树确定。
在其中一个实施例中，在处理器执行计算机程序实现上述的根据当前资源消耗等级，对当前查询请求进行资源分配的步骤时，具体实现以下步骤：根据当前资源消耗等级，确定当前查询请求的队列，其中，不同资源消耗等级的查询请求分配到不同队列上；按照当前查询请求的队列的资源调度策略，对当前查询请求进行资源分配。
在一个实施例中，处理器执行计算机程序时还实现以下步骤：
获取各队列的资源使用率，在当前队列的资源使用率低于预设的资源使用率阈值时，将当前队列的资源调整到各队列中的剩余队列；
或者
获取各队列的资源需求率，在当前队列的资源需求率高于预设的资源需求率阈值时，通过预设的资源动态扩容方案为当前队列增加资源。
在其中一个实施例中，在处理器执行计算机程序实现上述的根据当前资源消耗等级，对当前查询请求进行资源分配的步骤时，具体实现以下步骤：根据预设的资源消耗等级与集群的对应关系，将当前查询请求分配给当前资源消耗等级对应的集群，其中，各集群中分别采用公平调度的方式为分配到的查询请求分配资源。
在一个实施例中，处理器执行计算机程序时还实现以下步骤：
获取各集群的资源使用率，在当前集群的资源使用率低于预设的资源使用率阈值时，将当前集群的资源调整到各集群中的剩余集群；
或者
获取各集群的资源需求率，在当前集群的资源需求率高于预设的资源需求率阈值时，通过预设的资源动态扩容方案为当前集群增加资源。
在一个实施例中，处理器执行计算机程序时还实现以下步骤：获取当前查询请求的执行日志；根据执行日志，获取当前查询请求的实际资源消耗值、实际执行时长、调度时间和等待时间；根据实际资源消耗值、实际执行时长、调度时间和等待时间，调整预期资源消耗值和资源消耗等级的对应关系，或者调整预期资源消耗值的计算方式。
在一个实施例中，提供了一种计算机可读存储介质，其上存储有计算机程序，计算机程序被处理器执行时实现以下步骤：
接收当前查询请求，获取当前查询请求的预期资源消耗值；
根据预期资源消耗值，确定当前查询请求的当前资源消耗等级；
根据当前资源消耗等级，对当前查询请求进行资源分配。
在其中一个实施例中，在计算机程序被处理器执行实现上述的获取当前查询请求的预期资源消耗值的步骤时，具体实现以下步骤：对当前查询请求进行解析，获得当前查询请求的执行计划树；根据执行计划树，确定预期资源消耗值。
在其中一个实施例中，在计算机程序被处理器执行实现上述的根据执行计划树，确定预期资源消耗值的步骤时，具体实现以下步骤：根据预设的表统计信息和预设的字段统计信息，确定执行计划树中各个节点的资源消耗值；根据各节点的资源消耗值，确定预期资源消耗值。
在其中一个实施例中，在计算机程序被处理器执行实现上述的根据执行计划树，确定预期资源消耗值的步骤时，具体实现以下步骤：根据预设的表统计信息、预设的字段统计信息以及执行计划树，确定目标数据表的查询行数和数据输入输出代价，目标数据表为在进行当前查询请求的查询时需要使用的各数据表；根据预设的表统计信息、预设的字段统计信息以及执行计划树，确定当前查询请求的连接代价、连接后的行数、分组基数和中央处理器代价；根据查询行数、数据输入输出代价、连接代价、连接后的行数、分组基数和中央处理器代价，确定预期资源消耗值。
在其中一个实施例中，在计算机程序被处理器执行实现上述的获取当前查询请求的预期资源消耗值的步骤时，具体实现以下步骤：对当前查询请求进行解析，获得当前查询请求的执行计划树；将执行计划树发送给代价分析服务器，获取代价分析服务器返回的预期资源消耗值，预期资源消耗根据执行计划树确定。
在其中一个实施例中，在计算机程序被处理器执行实现上述的根据当前资源消耗等级，对当前查询请求进行资源分配的步骤时，具体实现以下步骤：根据当前资源消耗等级，确定当前查询请求的队列，其中，不同资源消耗等级的查询请求分配到不同队列上；按照当前查询请求的队列的资源调度策略，对当前查询请求进行资源分配。
在一个实施例中，计算机程序被处理器执行时还实现以下步骤：
获取各队列的资源使用率，在当前队列的资源使用率低于预设的资源使用率阈值时，将当前队列的资源调整到各队列中的剩余队列；
或者
获取各队列的资源需求率，在当前队列的资源需求率高于预设的资源需求率阈值时，通过预设的资源动态扩容方案为当前队列增加资源。
在其中一个实施例中，在计算机程序被处理器执行实现上述的根据当前资源消耗等级，对当前查询请求进行资源分配的步骤时，具体实现以下步骤：根据预设的资源消耗等级与集群的对应关系，将当前查询请求分配给当前资源消耗等级对应的集群，其中，各集群中分别采用公平调度的方式为分配到的查询请求分配资源。
在一个实施例中，计算机程序被处理器执行时还实现以下步骤：
获取各集群的资源使用率，在当前集群的资源使用率低于预设的资源使用率阈值时，将当前集群的资源调整到各集群中的剩余集群；
或者
获取各集群的资源需求率，在当前集群的资源需求率高于预设的资源需求率阈值时，通过预设的资源动态扩容方案为当前集群增加资源。
在一个实施例中，计算机程序被处理器执行时还实现以下步骤：获取当前查询请求的执行日志；根据执行日志，获取当前查询请求的实际资源消耗值、实际执行时长、调度时间和等待时间；根据实际资源消耗值、实际执行时长、调度时间和等待时间，调整预期资源消耗值和资源消耗等级的对应关系，或者调整预期资源消耗值的计算方式。
本领域普通技术人员可以理解实现上述实施例方法中的全部或部分流程，是可以通过计算机程序来指令相关的硬件来完成，所述的计算机程序可存储于一非易失性计算机可读取存储介质中，该计算机程序在执行时，可包括如上述各方法的实施例的流程。其中，本发明所提供的各实施例中所使用的对存储器、存储、数据库或其它介质的任何引用，均可包括非易失性和/或易失性存储器。非易失性存储器可包括只读存储器(ROM)、可编程ROM(PROM)、电可编程ROM(EPROM)、电可擦除可编程ROM(EEPROM)或闪存。易失性存储器可包括随机存取存储器(RAM)或者外部高速缓冲存储器。作为说明而非局限，RAM以多种形式可得，诸如静态RAM(SRAM)、动态RAM(DRAM)、同步DRAM(SDRAM)、双数据率SDRAM(DDRSDRAM)、增强型SDRAM(ESDRAM)、同步链路(Synchlink)DRAM(SLDRAM)、存储器总线(Rambus)直接RAM(RDRAM)、直接存储器总线动态RAM(DRDRAM)、以及存储器总线动态RAM(RDRAM)等。
以上实施例的各技术特征可以进行任意的组合，为使描述简洁，未对上述实施例中的各个技术特征所有可能的组合都进行描述，然而，只要这些技术特征的组合不存在矛盾，都应当认为是本说明书记载的范围。
以上所述实施例仅表达了本发明的几种实施方式，其描述较为具体和详细，但并不能因此而理解为对发明专利范围的限制。应当指出的是，对于本领域的普通技术人员来说，在不脱离本发明构思的前提下，还可以做出若干变形和改进，这些都属于本发明的保护范围。因此，本发明专利的保护范围应以所附权利要求为准。

Claims (13)

  1. 一种资源分配方法，其特征在于，所述方法包括：
    接收当前查询请求，获取所述当前查询请求的预期资源消耗值；
    根据所述预期资源消耗值，确定所述当前查询请求的当前资源消耗等级；
    根据所述当前资源消耗等级，对所述当前查询请求进行资源分配。
  2. 根据权利要求1所述的方法，其特征在于，所述获取所述当前查询请求的预期资源消耗值，包括：
    对所述当前查询请求进行解析，获得当前所述查询请求的执行计划树；
    根据所述执行计划树，确定所述预期资源消耗值。
  3. 根据权利要求2所述的方法，其特征在于，所述根据所述执行计划树，确定所述预期资源消耗值，包括：
    根据预设的表统计信息和预设的字段统计信息，确定所述执行计划树中各个节点的资源消耗值；
    根据各所述节点的资源消耗值，确定所述预期资源消耗值。
  4. 根据权利要求2所述的方法，其特征在于，所述根据所述执行计划树，确定所述预期资源消耗值，包括：
    根据预设的表统计信息、预设的字段统计信息以及所述执行计划树，确定目标数据表的查询行数和数据输入输出代价，所述目标数据表为在进行当前查询请求的查询时需要使用的各数据表；
    根据预设的表统计信息、预设的字段统计信息以及所述执行计划树，确定所述当前查询请求的连接代价、连接后的行数、分组基数和中央处理器代价；
    根据所述查询行数、所述数据输入输出代价、所述连接代价、所述连接后的行数、所述分组基数和所述中央处理器代价，确定所述预期资源消耗值。
  5. 根据权利要求1所述的方法，其特征在于，所述获取所述当前查询请求的预期资源消耗值，包括：
    对所述当前查询请求进行解析，获得所述当前查询请求的执行计划树；
    将所述执行计划树发送给代价分析服务器，获取所述代价分析服务器返回的预期资源消耗值，所述预期资源消耗根据所述执行计划树确定。
  6. 根据权利要求1所述的方法，其特征在于，所述根据所述当前资源消耗等级，对所述当前查询请求进行资源分配，包括：
    根据所述当前资源消耗等级，确定所述当前查询请求的队列，其中，不同资源消耗等级的查询请求分配到不同队列上；
    按照所述当前查询请求的队列的资源调度策略，对所述当前查询请求进行资源分配。
  7. 根据权利要求6所述的方法，其特征在于，所述方法还包括：
    获取各队列的资源使用率，在当前队列的资源使用率低于预设的资源使用率阈值时，将当前队列的资源调整到各所述队列中的剩余队列；
    或者
    获取各所述队列的资源需求率，在当前队列的资源需求率高于预设的资源需求率阈值时，通过预设的资源动态扩容方案为当前队列增加资源。
  8. 根据权利要求1所述的方法，其特征在于，所述根据所述当前资源消耗 等级，对所述当前查询请求进行资源分配，包括：
    根据预设的资源消耗等级与集群的对应关系，将所述当前查询请求分配给所述当前资源消耗等级对应的集群，其中，各集群中分别采用公平调度的方式为分配到的查询请求分配资源。
  9. 根据权利要求8所述的方法，其特征在于，所述方法还包括：
    获取各集群的资源使用率，在当前集群的资源使用率低于预设的资源使用率阈值时，将当前集群的资源调整到各所述集群中的剩余集群；
    或者
    获取各所述集群的资源需求率，在当前集群的资源需求率高于预设的资源需求率阈值时，通过预设的资源动态扩容方案为当前集群增加资源。
  10. 根据权利要求1所述的方法，其特征在于，所述方法还包括：
    获取所述当前查询请求的执行日志；
    根据所述执行日志，获取所述当前查询请求的实际资源消耗值、实际执行时长、调度时间和等待时间；
    根据所述实际资源消耗值、所述实际执行时长、所述调度时间和所述等待时间，调整预期资源消耗值和资源消耗等级的对应关系，或者调整预期资源消耗值的计算方式。
  11. 一种资源分配装置，其特征在于，所述装置包括：
    消耗值确定模块，用于接收当前查询请求，获取所述当前查询请求的预期资源消耗值；
    消耗等级确定模块，用于根据所述预期资源消耗值，确定所述当前查询请求的当前资源消耗等级；
    资源分配模块，用于根据所述当前资源消耗等级，对所述当前查询请求进行资源分配。
  12. 一种计算机设备，包括存储器、处理器及存储在存储器上并可在处理器上运行的计算机程序，其特征在于，所述处理器执行所述计算机程序时实现权利要求1至10中任一项所述方法的步骤。
  13. 一种计算机可读存储介质，其上存储有计算机程序，其特征在于，所述计算机程序被处理器执行时实现权利要求1至10中任一项所述的方法的步骤。
PCT/CN2019/109355 2019-04-16 2019-09-30 资源分配方法、装置、计算机设备和存储介质 WO2020211300A1 (zh)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CA3177212A CA3177212A1 (en) 2019-04-16 2019-09-30 Resource allocating method, device, computer equipment, and storage medium

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201910305271.0 2019-04-16
CN201910305271.0A CN110166282B (zh) 2019-04-16 2019-04-16 资源分配方法、装置、计算机设备和存储介质

Publications (1)

Publication Number Publication Date
WO2020211300A1 true WO2020211300A1 (zh) 2020-10-22

Family

ID=67639608

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2019/109355 WO2020211300A1 (zh) 2019-04-16 2019-09-30 资源分配方法、装置、计算机设备和存储介质

Country Status (3)

Country Link
CN (1) CN110166282B (zh)
CA (1) CA3177212A1 (zh)
WO (1) WO2020211300A1 (zh)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116401465A (zh) * 2023-06-06 2023-07-07 支付宝(杭州)信息技术有限公司 资源分组分配方法、调整值确定方法和装置

Families Citing this family (15)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110166282B (zh) * 2019-04-16 2020-12-01 苏宁云计算有限公司 资源分配方法、装置、计算机设备和存储介质
CN110569257B (zh) * 2019-09-16 2022-04-01 上海达梦数据库有限公司 数据处理方法、相应装置、设备及存储介质
CN111190927A (zh) * 2019-12-17 2020-05-22 未鲲(上海)科技服务有限公司 Sql查询命令处理方法、装置、设备及可读存储介质
CN111198767B (zh) * 2020-01-07 2024-10-18 平安科技（深圳）有限公司 大数据资源处理方法、装置、终端及存储介质
CN111367656B (zh) * 2020-02-20 2023-09-26 深圳震有科技股份有限公司 一种分配媒体资源的方法、计算机设备及存储介质
CN113535348A (zh) * 2020-04-21 2021-10-22 华为技术有限公司 一种资源调度方法及相关装置
CN111694668B (zh) * 2020-06-11 2023-06-30 广州市百果园信息技术有限公司 查询任务路由方法、装置、设备及存储介质
CN111475584B (zh) * 2020-06-19 2021-01-22 阿里云计算有限公司 数据处理方法、系统以及装置
CN112131267B (zh) * 2020-08-14 2023-10-03 北京达佳互联信息技术有限公司 计数处理方法、装置、服务器和计数处理系统
CN112579324A (zh) * 2020-12-29 2021-03-30 中盈优创资讯科技有限公司 一种基于cost模型的商品汇总统计方法
CN113111083B (zh) * 2021-03-31 2024-07-16 北京沃东天骏信息技术有限公司 数据查询的方法、装置、设备、存储介质和程序产品
CN113806068B (zh) * 2021-07-30 2023-12-12 上海晶赞融宣科技有限公司 业务系统的扩容方法及装置、可读存储介质、终端
CN113553361A (zh) * 2021-07-30 2021-10-26 北京东方国信科技股份有限公司 资源管理方法及装置
CN115145953A (zh) * 2021-10-22 2022-10-04 上海卓辰信息科技有限公司 数据查询方法
CN117806778B (zh) * 2024-02-29 2024-06-07 济南浪潮数据技术有限公司 资源管理方法、系统、设备及介质

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20120110592A1 (en) * 2004-11-08 2012-05-03 International Business Machines Corporation Autonomic Self-Tuning Of Database Management System In Dynamic Logical Partitioning Environment
CN107797863A (zh) * 2017-09-29 2018-03-13 西安交通大学 一种云计算平台中细粒度资源匹配方法
CN108845874A (zh) * 2018-06-25 2018-11-20 腾讯科技（深圳）有限公司 资源的动态分配方法及服务器
CN110166282A (zh) * 2019-04-16 2019-08-23 苏宁易购集团股份有限公司 资源分配方法、装置、计算机设备和存储介质

Family Cites Families (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102567086B (zh) * 2010-12-30 2014-05-07 中国移动通信集团公司 一种任务调度的方法、设备和系统
CN103064875B (zh) * 2012-10-30 2017-06-16 中国标准化研究院 一种服务化空间数据分布式查询方法
CN107168977B (zh) * 2016-03-08 2020-07-28 阿里巴巴集团控股有限公司 一种数据查询的优化方法及装置
WO2017190058A1 (en) * 2016-04-28 2017-11-02 Snowflake Computing Inc. Multi-cluster warehouse
CN106383864B (zh) * 2016-09-02 2019-08-27 北京百度网讯科技有限公司 一种时序数据库的查询请求处理方法和装置
US11537615B2 (en) * 2017-05-01 2022-12-27 Futurewei Technologies, Inc. Using machine learning to estimate query resource consumption in MPPDB
CN108989583B (zh) * 2018-08-10 2021-08-31 携程旅游信息技术（上海）有限公司 呼叫中心的压力测试方法、系统、电子设备和存储介质

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20120110592A1 (en) * 2004-11-08 2012-05-03 International Business Machines Corporation Autonomic Self-Tuning Of Database Management System In Dynamic Logical Partitioning Environment
CN107797863A (zh) * 2017-09-29 2018-03-13 西安交通大学 一种云计算平台中细粒度资源匹配方法
CN108845874A (zh) * 2018-06-25 2018-11-20 腾讯科技（深圳）有限公司 资源的动态分配方法及服务器
CN110166282A (zh) * 2019-04-16 2019-08-23 苏宁易购集团股份有限公司 资源分配方法、装置、计算机设备和存储介质

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116401465A (zh) * 2023-06-06 2023-07-07 支付宝(杭州)信息技术有限公司 资源分组分配方法、调整值确定方法和装置
CN116401465B (zh) * 2023-06-06 2023-10-20 支付宝(杭州)信息技术有限公司 资源分组分配方法、调整值确定方法和装置

Also Published As

Publication number Publication date
CN110166282B (zh) 2020-12-01
CN110166282A (zh) 2019-08-23
CA3177212A1 (en) 2020-10-22

Similar Documents

Publication Publication Date Title
WO2020211300A1 (zh) 资源分配方法、装置、计算机设备和存储介质
US11888702B2 (en) Intelligent analytic cloud provisioning
US9672064B2 (en) Dynamically adaptive, resource aware system and method for scheduling
US8082273B2 (en) Dynamic control and regulation of critical database resources using a virtual memory table interface
RU2675054C2 (ru) Балансировка нагрузки для больших баз данных в оперативной памяти
US8392404B2 (en) Dynamic query and step routing between systems tuned for different objectives
CN108595254B (zh) 一种查询调度方法
US9235590B1 (en) Selective data compression in a database system
US20090132611A1 (en) Closed-loop system management method and process capable of managing workloads in a multi-system database environment
CN106294757B (zh) 一种基于超图划分的分布式数据库及其集群分区方法
Li et al. Real-time scheduling based on optimized topology and communication traffic in distributed real-time computation platform of storm
US9110947B1 (en) Column-oriented task execution in a row-partitioned database system
US8392461B2 (en) Virtual data maintenance
US20190266154A1 (en) High performance data profiler for big data
US20160274940A1 (en) Regulating enterprise database warehouse resource usage
CN109918450A (zh) 基于分析类场景下的分布式并行数据库及存储方法
CN116089414B (zh) 基于海量数据场景的时序数据库写入性能优化方法及装置
US20160034528A1 (en) Co-processor-based array-oriented database processing
CN107395446B (zh) 日志实时处理系统
CN114756629A (zh) 基于sql的多源异构数据交互分析引擎及方法
CN113568931A (zh) 一种数据访问请求的路由解析系统及方法
CN105608138A (zh) 一种优化阵列数据库并行数据加载性能的系统
CN110928649A (zh) 资源调度的方法和装置
US20130013824A1 (en) Parallel aggregation system
CN115982230A (zh) 数据库的跨数据源查询方法、系统、设备及存储介质

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19925306

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 19925306

Country of ref document: EP

Kind code of ref document: A1

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205 DATED 22/04/2022)

ENP Entry into the national phase

Ref document number: 3177212

Country of ref document: CA

122 Ep: pct application non-entry in european phase

Ref document number: 19925306

Country of ref document: EP

Kind code of ref document: A1