WO2020211300A1 - 资源分配方法、装置、计算机设备和存储介质 - Google Patents
资源分配方法、装置、计算机设备和存储介质 Download PDFInfo
- Publication number
- WO2020211300A1 WO2020211300A1 PCT/CN2019/109355 CN2019109355W WO2020211300A1 WO 2020211300 A1 WO2020211300 A1 WO 2020211300A1 CN 2019109355 W CN2019109355 W CN 2019109355W WO 2020211300 A1 WO2020211300 A1 WO 2020211300A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- current
- resource
- resource consumption
- query request
- consumption value
- Prior art date
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/08—Configuration management of networks or network elements
- H04L41/0896—Bandwidth or capacity management, i.e. automatically increasing or decreasing capacities
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/56—Provisioning of proxy services
- H04L67/562—Brokering proxy services
Definitions
- the present invention relates to the field of database technology, in particular to a resource allocation method, device, computer equipment and storage medium.
- Online analysis and processing is mainly a fast software technology that uses distributed systems to share multi-dimensional information and target specific problems on-line data access and analysis.
- FAIR First Input First Output
- some queries consume extremely large resources and take extremely long execution time. If several such query requests are executed concurrently, it is easy to fill up the resources of the cluster. This will make other queries unable to obtain sufficient resources, hinder the execution of these query requests (even if the resource occupied by this query request is very small, the execution is also very fast), and affect the user experience; the serious one may cause the application memory to run out (Out Of Memory, OOM) or the entire cluster crashes.
- OOM Out Of Memory
- a resource allocation method includes:
- resources are allocated to the current query request.
- the foregoing obtaining the expected resource consumption value of the current query request includes:
- determining the expected resource consumption value according to the execution plan tree described above includes:
- determining the expected resource consumption value according to the execution plan tree described above includes:
- the preset table statistics, preset field statistics, and execution plan tree determine the number of query rows and data input and output costs of the target data table.
- the target data table is the data that needs to be used in the current query request table;
- preset table statistics determine the connection cost of the current query request, the number of rows after the connection, the grouping cardinality, and the CPU cost;
- the foregoing obtaining the expected resource consumption value of the current query request includes:
- the execution plan tree is sent to the cost analysis server to obtain the expected resource consumption value returned by the cost analysis server, and the expected resource consumption is determined according to the execution plan tree.
- the foregoing resource allocation for the current query request according to the current resource consumption level includes:
- the current resource consumption level determine the current query request queue, where query requests of different resource consumption levels are allocated to different queues
- the current query request is allocated resources.
- the above method further includes:
- the foregoing resource allocation for the current query request according to the current resource consumption level includes:
- the current query request is allocated to the cluster corresponding to the current resource consumption level, wherein each cluster uses a fair scheduling method to allocate resources to the allocated query request.
- the above method further includes:
- the above method further includes:
- the execution log obtain the actual resource consumption value, actual execution time, scheduling time and waiting time of the current query request;
- actual resource consumption value actual execution time, scheduling time and waiting time, adjust the correspondence between the expected resource consumption value and the resource consumption level, or adjust the calculation method of the expected resource consumption value.
- a resource allocation device which includes:
- the consumption value determination module is used to receive the current query request and obtain the expected resource consumption value of the current query request;
- the consumption level determination module is used to determine the current resource consumption level of the current query request according to the expected resource consumption value
- the resource allocation module is used to allocate resources for the current query request according to the current resource consumption level.
- a computer device includes a memory, a processor, and a computer program stored on the memory and running on the processor.
- the processor implements the following steps when the processor executes the computer program:
- resources are allocated to the current query request.
- the computer program is executed by a processor, the following steps are implemented:
- resources are allocated to the current query request.
- the above resource allocation method, device, computer equipment and storage medium receive the current query request, obtain the expected resource consumption value of the current query request, determine the current resource consumption level of the current query request according to the expected resource consumption value, and according to the current resource consumption level, Allocate resources for the current query request. In this way, the isolated allocation of resources is realized.
- query requests with relatively low resource consumption will not fail to obtain resources due to query requests with relatively high resource consumption, while query requests with relatively high resource consumption are in most cases It can also be executed normally, and can improve the stability and overall performance of the system when the resource-intensive query requests appear in a concentrated manner.
- Figure 1 is an application environment diagram of a resource allocation method in an embodiment
- FIG. 2 is a schematic flowchart of a resource allocation method in an embodiment
- FIG. 3 is a schematic flowchart of steps for obtaining an expected resource consumption value in an embodiment
- FIG. 4 is a schematic flowchart of steps for obtaining an expected resource consumption value in another embodiment
- FIG. 5 is a schematic flowchart of a step of determining an expected resource consumption value in an embodiment
- FIG. 6 is a schematic flowchart of a step of determining an expected resource consumption value in another embodiment
- FIG. 7 is a schematic flowchart of a resource allocation step in an embodiment
- FIG. 8 is a schematic flowchart of a corresponding relationship adjustment step or a calculation method adjustment step in an embodiment
- FIG. 9 is a schematic structural principle diagram of a resource consumption analysis model in an embodiment
- FIG. 10 is a sequence diagram of a resource allocation method in an embodiment
- FIG. 11 is a schematic flowchart of query distribution in an embodiment
- Figure 12 is a structural block diagram of a resource allocation method in an embodiment
- Figure 13 is an internal structure diagram of a computer device in an embodiment.
- the resource allocation method provided by the present invention can be applied to the application environment shown in FIG. 1.
- the terminal 102 and the server 104 communicate through a network
- the server 104 and the cluster 106 communicate through a wired or wireless manner.
- the terminal 102 can be, but is not limited to, various personal computers, notebook computers, smart phones, tablet computers, and portable wearable devices.
- the server 104 can be implemented as an independent server or a server cluster composed of multiple servers.
- the cluster 106 is To provide resources, the number may be one or more.
- the cluster 106 generally includes multiple computer devices (machines).
- the resource in the resource allocation method provided by the present invention may refer to a certain cluster in multiple clusters, or a certain machine in a cluster.
- the resource allocation method provided by the present invention is generally applied to an online analysis and processing system.
- a resource allocation method is provided.
- the method is applied to the server in FIG. 1 as an example for description, including the following steps:
- Step 202 Receive the current query request, and obtain the expected resource consumption value of the current query request;
- the current query request generally refers to an online analytical processing query, and may also be an online transaction processing (OLTP) request and other types of connection query requests.
- OTP online transaction processing
- the expected resource consumption value refers to the resource value expected to be consumed to complete the query task corresponding to the current query request.
- the server obtains the current query request (for example, receives the current query request sent by the terminal), analyzes the current query request, and obtains the expected resource consumption of the current query request according to the analysis result of the analysis and analysis. value. It can also be that the server obtains the current query request and obtains the expected resource consumption value of the current query request from the preprocessing server (for example, the cost analysis server), where the preprocessing server is used to perform resource consumption values for various query requests analysis.
- the preprocessing server for example, the cost analysis server
- Step 204 Determine the current resource consumption level of the current query request according to the expected resource consumption value
- the number of resource consumption levels and the corresponding relationship with the resource consumption value can be set according to actual conditions and actual needs.
- the first resource consumption threshold value and the second resource consumption threshold value three resource consumption levels can be divided into high resource consumption level, medium resource consumption level and low resource consumption level.
- the first resource consumption threshold is greater than the second resource consumption threshold, and when the expected resource consumption value is greater than the first resource consumption threshold, it is a high resource consumption level; when the expected resource consumption value is within the first resource consumption threshold When the limit value is between the second resource consumption threshold value, it is a medium resource consumption level; when the expected resource consumption value is less than the second resource consumption threshold value, it is a low resource consumption level.
- Step 206 Perform resource allocation to the current query request according to the current resource consumption level.
- the current query request is allocated to the resource cluster or queue corresponding to the current resource consumption level, where query requests of different resource consumption levels are allocated to different resource clusters or queues.
- the current query request is received, the expected resource consumption value of the current query request is obtained, the current resource consumption level of the current query request is determined according to the expected resource consumption value, and the current query request is resourced according to the current resource consumption level.
- Allocation in this way, can realize the isolated allocation of resources.
- the query request with relatively low resource consumption will not be unable to obtain resources due to the query request with relatively high resource consumption, and the query request with relatively high resource consumption can be executed normally in most cases, and can be executed in most cases.
- resource-intensive query requests appear in a centralized manner, the stability and overall performance of the system are improved.
- the above-mentioned obtaining the expected resource consumption value of the current query request may include the following steps:
- Step 302 parse the current query request to obtain the execution plan tree of the current query request
- the current query request generally includes query statements, for example, structured query statements, that is, SQL (Structured Query Language, structured query language) statements.
- structured query statements that is, SQL (Structured Query Language, structured query language) statements.
- cost-based optimization (Cost-Based Logical Optimizer, CBO) is used to optimize the execution plan.
- CBO Cost-Based Logical Optimizer
- the main purpose of the cost-based optimization method is to generate an efficient execution path by detecting the tables and conditions used in the query, thereby reducing resource utilization and reducing query execution time.
- a query statement can be parsed into an execution plan tree, and each node on the execution plan tree represents an operation in the query, such as scanning, projection, or filtering.
- Step 304 Determine the expected resource consumption value according to the execution plan tree.
- the server may first calculate the input/output (i/o) cost, central processing unit (CPU) cost, and storage cost of each operation on the execution plan tree. (Memory) cost and network (NETWORK) cost; then, the input and output cost, central processing unit cost, storage cost, and network cost of each operation are summed to obtain the resource consumption value of each operation. Finally, for each operation The resource consumption value of is summed to obtain the expected resource consumption value of the current query request.
- i/o input/output
- CPU central processing unit
- NETWORK network cost
- the input/output cost refers to the cost required to read data from disk into memory (this cost is the most important for query, so a basic principle in optimization is to reduce the total number of input/output);
- CPU cost It refers to the cost of processing the data in the memory. Once the data is read into the memory, when the memory recognizes the required data, it will perform a sort or join on the data, which requires central processing.
- the storage cost is the cost required to store data. For accessing remote nodes, the cost of networking is also very high.
- the expected resource consumption value of the current query request is determined, which is efficient and fast.
- the above-mentioned determining the expected resource consumption value according to the execution plan tree may include the following steps:
- Step 402 Determine the resource consumption value of each node in the execution plan tree according to preset table statistical information and preset field statistical information;
- the table statistical information may include table information such as the number of rows in the table, the size of the table counted in bytes, the size of the table's partition, and the number of partitions, but it is not limited to this information.
- the field statistical information can include field information such as the base of the field (count distinct), the maximum value of the field, the lowest value of the field, the number of records with empty content (Null count), and the average length of the field, but it is not limited to this information. .
- the collected information includes but is not limited to the various table statistical information and various field statistical information mentioned above.
- the data table is the target data table below.
- the preset table statistics and preset field statistics determine the table statistics and field statistics of the target data table; finally, calculate the execution plan tree based on the table statistics and field statistics of the target data table.
- the resource consumption value of each node is also the resource consumption value of each operation mentioned above.
- Step 404 Determine the expected resource consumption value according to the resource consumption value of each node.
- the resource consumption value of each node in each execution plan tree can be summed to obtain the expected resource consumption value of the current query request.
- the table statistics and field statistics are collected in advance, and the expected resource consumption value of the current query request is determined based on the execution plan tree, which can improve the calculation efficiency of the expected resource consumption value.
- the above-mentioned determining the expected resource consumption value according to the execution plan tree may include the following steps:
- Step 502 Determine the number of query rows and data input and output costs of the target data table according to the preset table statistics, preset field statistics, and execution plan tree.
- the target data table is required for the current query request.
- the data table and conditions used in the query of the current query request can be checked in a cost-based optimization manner, and the conditions include filter conditions.
- the number of query rows and data input and output costs of each data table that need to be used in the query of the current query request can be determined according to the preset table statistical information, the preset field statistical information, and the filter condition.
- Step 504 Determine the connection cost of the current query request, the number of rows after the connection, the grouping base, and the central processing unit cost according to the preset table statistics, the preset field statistics, and the execution plan tree;
- connection cost can be calculated according to the join relationship that needs to be used in the query of the current query request; it can be based on the group by clause and the level of the dimension in the execution plan tree Relations, as well as preset table statistics and preset field statistics, calculate the aggregated cardinality (carndinality), that is, the grouping cardinality; the CPU cost can be determined according to the summary function and sorting information in the execution plan tree.
- Step 506 Determine the expected resource consumption value according to the number of query rows, the data input and output cost, the connection cost, the number of connected rows, the grouping cardinality, and the central processing unit cost;
- the number of query rows, data input and output costs, connection costs, number of connected rows, grouping cardinality, and central processing unit cost can be aggregated and analyzed to obtain the expected resource consumption value of the current query request.
- one is to perform resource consumption statistics based on nodes in the execution plan tree, and the other is to perform resource consumption statistics based on various types of costs. According to needs, you can choose any one of them to calculate the expected resource consumption value. Calculations can also be used to calculate the expected resource consumption value by combining these two methods.
- the table statistics and field statistics are collected in advance, and the expected resource consumption value of the current query request is determined based on the execution plan tree, which can improve the calculation efficiency of the expected resource consumption value.
- the above-mentioned determining the expected resource consumption value according to the execution plan tree may include the following steps:
- Step 602 parse the current query request to obtain the execution plan tree of the current query request
- step 602 reference may be made to the description of step 302, which is not repeated here.
- Step 604 Send the execution plan tree to the cost analysis server to obtain the expected resource consumption value returned by the cost analysis server, and the expected resource consumption is determined according to the execution plan tree;
- the method for the cost analysis server to determine the expected resource consumption value may adopt the method of determining the expected resource consumption value according to the execution plan tree in the above two embodiments.
- the execution plan tree is obtained by parsing the current query request, and the execution plan tree is sent to the cost analysis server to calculate the expected resource consumption value. In this way, the expected resource can be improved when the query request is highly concurrent. Consumption acquisition efficiency.
- the above-mentioned resource allocation to the current query request according to the current resource consumption level may include the following steps:
- Step 702 Determine the queue of the current query request according to the current resource consumption level, where query requests with different resource consumption levels are allocated to different queues;
- capacity refers to the number (or maximum number) of query requests waiting to be scheduled for execution. For example, for a situation including a high resource consumption level, a medium resource consumption level, and a low resource consumption level, the low resource consumption level has the largest queue capacity, and the high resource consumption level has the smallest capacity.
- Step 704 Perform resource allocation for the current query request according to the resource scheduling policy of the queue of the current query request.
- each queue is configured with a scheduling process to schedule query requests in the queue.
- the scheduling processes of different queues have different scheduling strategies.
- the scheduling strategy of a queue with a high resource consumption level allows only one query to be scheduled at the same time. request.
- the higher the resource consumption level that is, the higher the average resource consumption value of each query request
- the level is negatively correlated.
- query requests are placed in different queues according to resource consumption levels. When the queue is full, subsequent query requests are generally set to be discarded. With the solution of this embodiment, it can be ensured that sufficient resources are always available for query requests with relatively low resource consumption, and resources will not be lost due to query requests with relatively high resource consumption. In addition, query requests with relatively high resource consumption can be executed normally in most cases, which can ensure the smooth and orderly operation of the entire system.
- the resource allocation method of the present application may further include the step of obtaining the resource usage rate of each queue, and when the resource usage rate of the current queue is lower than a preset resource usage rate threshold, the resources of the current queue Adjust to the remaining queues in each of the queues.
- the current queue can be any queue.
- adjusting the resources of the current queue to the remaining queues in each of the queues may be adjusting part of the resources of the current queue to one or more of the remaining queues in each of the queues.
- the resource dynamic adjustment strategy can be set according to actual needs, and the resource dynamic adjustment strategy can specifically include information such as adjustment time period, resource adjustment ratio, and adjustment to several queues.
- the resource allocation method of the present invention may further include the step of: obtaining the resource demand rate of each queue, and when the resource demand rate of the current queue is higher than a preset resource demand rate threshold, pass the preset The dynamic resource expansion plan adds resources to the current queue.
- the current queue can be any queue.
- the resource demand rate is equal to the ratio of the resource demand value to the actual value of the resource.
- the resource dynamic expansion plan may specifically include information such as the capacity expansion period and the resource value to be increased.
- the aforementioned resource allocation for the current query request based on the current resource consumption level may include the following steps: according to the preset correspondence between the resource consumption level and the cluster, the current query request is allocated to the current resource consumption The clusters corresponding to the levels, where each cluster uses a fair scheduling method to allocate resources to the allocated query requests.
- each cluster (node) is preset with different grade labels, such as high, medium, and low.
- Resource clusters of different levels execute different levels of query requests.
- the server allocates the current query request to the cluster corresponding to the current resource consumption level according to the preset correspondence between the resource consumption level and the cluster, and the cluster executes the current query request, that is, executes the query task of the current query request.
- resource clusters of different levels execute different levels of query requests, and query requests with relatively low resource consumption can ensure that sufficient resources are always available, and no query requests with relatively high resource consumption will be obtained. In most cases, query requests that lack resources and consume relatively high resources can be executed normally, which can ensure the smooth and orderly operation of the entire system.
- the resource allocation method of the present invention may further include the step of obtaining the resource usage rate of each cluster, and when the resource usage rate of the current cluster is lower than a preset resource usage rate threshold, the resources of the current cluster Adjust to the remaining clusters in each of the clusters.
- the current cluster can be any cluster.
- adjusting the resources of the current cluster to the remaining clusters in each of the clusters may be adjusting part of the resources of the current cluster to one or more columns of the remaining clusters in each of the clusters.
- the resource dynamic adjustment strategy can be set according to actual needs, and the resource dynamic adjustment strategy can specifically include information such as adjustment time period, resource adjustment ratio, and adjustment to several clusters.
- the resource allocation method of the present invention may further include the step of obtaining the resource demand rate of each of the clusters, and when the current resource demand rate of the cluster is higher than a preset resource demand rate threshold, pass the preset The dynamic resource expansion plan adds resources to the current cluster.
- the current cluster can be any cluster.
- the resource demand rate is equal to the ratio of the resource demand value to the actual value of the resource.
- the resource dynamic expansion plan may specifically include information such as the capacity expansion period and the resource value to be increased.
- the resource allocation method of the present invention may further include the steps:
- Step 802 Obtain the execution log of the current query request
- the execution log of the current query request may be recorded by the system, and the execution log refers to the log generated by the actual execution of the current query request.
- Step 804 According to the execution log, obtain the actual resource consumption value, actual execution duration, scheduling time, and waiting time of the current query request;
- Step 806 Adjust the correspondence between the expected resource consumption value and the resource consumption level, or adjust the calculation method of the expected resource consumption value according to the actual resource consumption value, the actual execution duration, the scheduling time and the waiting time.
- algorithms such as machine learning or artificial intelligence can be used to continuously learn and adjust the correspondence between the expected resource consumption value and the resource consumption level, as well as the calculation method of the expected resource consumption value, to improve accuracy and reduce manslaughter or mistakes. Judging the situation.
- the online analysis and processing engine (for example, the server in the above embodiment) mainly uses the data query system (druid) as the storage of summary statistical data, and can push down the query to the data query system for execution as much as possible, reducing the programming language (spark) The overhead of itself.
- the data query system druid
- spark programming language
- Figure 9 for the consumption analysis model (cost model) of the data query system.
- the cost model Before query request (SQL statement) scheduling and command distribution, the cost model can be called to analyze the expected resource consumption value of the query request. If it is found that the expected resource consumption value of the query request exceeds the threshold, it can be specially processed to achieve resource isolation and service degradation, and protect the entire online analysis and processing engine.
- Storage must also have resource isolation and flow control, and now it is mainly to control the resources of aggregate storage.
- the history node distinguishes different groups (tier).
- the high-priority group has more resources, and the low-priority group has fewer resources.
- a data source needs to be allocated to different groups, and the number of high-priority copies is higher. Many, low priority copy data is less
- Brokers also have different priorities. The number of agents with different priorities is different, and they must be routed to different groups.
- Postgresql uses the citus distributed cluster solution.
- the management node (master node) and processing node (worker node) have no concept of priority, but we can artificially distinguish the management nodes: divided into high priority management nodes and low priority Management node.
- the number of resources of management nodes at each level is also different. Different priority management nodes handle different levels of query requests.
- the management (master) layer of the online analysis processing engine needs to perceive the priority level of the management nodes in the data query system cluster and the logical structure cluster.
- the management node information needs to be passed to the management node cluster and the management node Only operate on the resources under your management.
- Different management node clusters manage different levels, but the same priority level can be managed by multiple management node clusters.
- the resource here refers to a certain cluster in multiple clusters, or a certain machine in a cluster.
- the specified resources can only handle certain specific query requests.
- Resource classification can be achieved by tagging.
- an online analytical processing engine consists of three clusters
- the query can be returned within 1-5s (expected resource consumption value is between 100-500);
- classification method is not limited to the method in this example.
- Each query request will go through the routing component (route component).
- the routing component calculates the expected resource consumption value of the query request, and calculates the expected resource consumption value of each query request.
- the routing component will send the query request to the appropriate cluster according to the configuration value of the expected resource consumption value and the cluster label. For example, for a summary query in a large area, the expected resource consumption value is calculated as 50, and then forwarded to the high-speed cluster; while the query request for calculating the top ten sales of goods is expensive, for example, the expected resource consumption value is 1000, then forwarded to the slow cluster.
- a dynamic resource expansion plan can be used to increase resources and relieve the tension of resources.
- a resource allocation device including: a consumption value determination module 1202, a consumption level determination module 1204, and a resource allocation module 1206, wherein:
- the consumption value determining module 1202 is configured to receive the current query request and obtain the expected resource consumption value of the current query request;
- the consumption level determining module 1204 is configured to determine the current resource consumption level of the current query request according to the expected resource consumption value
- the resource allocation module 1206 is configured to allocate resources for the current query request according to the current resource consumption level.
- the consumption value determination module 1202 may parse the current query request, obtain the execution plan tree of the current query request, and determine the expected resource consumption value according to the execution plan tree.
- the consumption value determination module 1202 can determine the resource consumption value of each node in the execution plan tree according to preset table statistics and preset field statistics, and determine the expected resource consumption value of each node Resource consumption value.
- the consumption value determination module 1202 can determine the number of query rows and data input and output costs of the target data table according to preset table statistics, preset field statistics, and execution plan tree.
- the target data table is The data tables that need to be used in the query of the current query request, according to the preset table statistics, preset field statistics and execution plan tree, determine the connection cost of the current query request, the number of rows after the connection, and the grouping cardinality
- the central processor cost according to the number of query rows, data input and output cost, connection cost, number of rows after connection, grouping cardinal number and central processor cost, determine the expected resource consumption value.
- the consumption value determination module 1202 may analyze the current query request, obtain the execution plan tree of the current query request, send the execution plan tree to the cost analysis server, and obtain the expected resource consumption value returned by the cost analysis server. The expected resource consumption is determined according to the execution plan tree.
- the resource allocation module 1206 can determine the queue of the current query request according to the current resource consumption level, wherein query requests of different resource consumption levels are allocated to different queues, according to the resource scheduling strategy of the queue of the current query request To allocate resources for the current query request.
- the above-mentioned apparatus may further include a first resource adjustment module or a first resource expansion module:
- the first resource adjustment module is configured to obtain the resource usage rate of each queue, and when the resource usage rate of the current queue is lower than a preset resource usage rate threshold, adjust the resources of the current queue to the remaining queues in each queue;
- the first resource expansion module is used to obtain the resource demand rate of each queue, and when the resource demand rate of the current queue is higher than the preset resource demand rate threshold, add resources to the current queue through the preset resource dynamic expansion plan.
- the resource allocation module 1206 can allocate the current query request to the cluster corresponding to the current resource consumption level according to the preset correspondence between the resource consumption level and the cluster, wherein each cluster adopts a fair scheduling method. Allocate resources for the allocated query request.
- the above-mentioned apparatus may further include a second resource adjustment module or a second resource expansion module:
- the second resource adjustment module is used to obtain the resource usage rate of each cluster, and when the resource usage rate of the current cluster is lower than the preset resource usage rate threshold, adjust the resources of the current cluster to the remaining clusters in each cluster;
- the second resource expansion module is used to obtain the resource demand rate of each cluster, and when the resource demand rate of the current cluster is higher than the preset resource demand rate threshold, add resources to the current cluster through the preset resource dynamic expansion plan.
- the above-mentioned device may also dynamically adjust the module, which is used to obtain the execution log of the current query request, and obtain the actual resource consumption value, actual execution duration, and scheduling time of the current query request according to the execution log. And waiting time, according to the actual resource consumption value, actual execution time, scheduling time and waiting time, adjust the correspondence between the expected resource consumption value and the resource consumption level, or adjust the calculation method of the expected resource consumption value.
- Each module in the above-mentioned resource allocation device may be implemented in whole or in part by software, hardware, and a combination thereof.
- the foregoing modules may be embedded in the form of hardware or independent of the processor in the computer device, or may be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the foregoing modules.
- a computer device is provided.
- the computer device may be a server, and its internal structure diagram may be as shown in FIG. 13.
- the computer equipment includes a processor, a memory, and a network interface connected through a system bus.
- the processor of the computer device is used to provide calculation and control capabilities.
- the memory of the computer device includes a non-volatile storage medium and an internal memory.
- the non-volatile storage medium stores an operating system, a computer program, and a database.
- the internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium.
- the network interface of the computer device is used to communicate with an external terminal through a network connection.
- the computer program is executed by the processor to realize a resource allocation method.
- FIG. 13 is only a block diagram of part of the structure related to the solution of the present invention, and does not constitute a limitation on the computer device to which the solution of the present invention is applied.
- the specific computer device may Including more or fewer parts than shown in the figure, or combining some parts, or having a different arrangement of parts.
- a computer device including a memory, a processor, and a computer program stored in the memory and capable of running on the processor, and the processor implements the following steps when the processor executes the computer program:
- resources are allocated to the current query request.
- the processor executes the computer program to implement the above-mentioned step of obtaining the expected resource consumption value of the current query request
- the following steps are specifically implemented: parsing the current query request to obtain the execution plan tree of the current query request; According to the execution plan tree, determine the expected resource consumption value.
- the processor when the processor executes the computer program to implement the above-mentioned step of determining the expected resource consumption value according to the execution plan tree, the following steps are specifically implemented: according to preset table statistics and preset field statistics, Determine the resource consumption value of each node in the execution plan tree; determine the expected resource consumption value according to the resource consumption value of each node.
- the processor executes the computer program to implement the above-mentioned step of determining the expected resource consumption value according to the execution plan tree
- the following steps are specifically implemented: according to preset table statistics, preset field statistics, and Execute the plan tree to determine the number of query rows and data input and output costs of the target data table.
- the target data table is the data table that needs to be used in the current query request; according to the preset table statistics and preset field statistics Information and execution plan tree to determine the connection cost of the current query request, the number of rows after the connection, the grouping cardinality and the CPU cost; according to the number of query rows, data input and output costs, connection costs, the number of rows after connection, grouping cardinality and The CPU cost determines the expected resource consumption value.
- the processor executes the computer program to implement the above-mentioned step of obtaining the expected resource consumption value of the current query request
- the following steps are specifically implemented: parsing the current query request to obtain the execution plan tree of the current query request;
- the execution plan tree is sent to the cost analysis server to obtain the expected resource consumption value returned by the cost analysis server, and the expected resource consumption is determined according to the execution plan tree.
- the processor executes the computer program to implement the above-mentioned step of resource allocation for the current query request according to the current resource consumption level
- the following steps are specifically implemented: determine the queue of the current query request according to the current resource consumption level , Where query requests of different resource consumption levels are allocated to different queues; according to the resource scheduling strategy of the current query request queue, the current query request is allocated resources.
- the processor further implements the following steps when executing the computer program:
- the processor executes the computer program to implement the above-mentioned step of resource allocation to the current query request according to the current resource consumption level
- the following steps are specifically implemented: according to the correspondence between the preset resource consumption level and the cluster , Allocate the current query request to the cluster corresponding to the current resource consumption level, wherein each cluster uses a fair scheduling method to allocate resources to the allocated query request.
- the processor further implements the following steps when executing the computer program:
- the processor further implements the following steps when executing the computer program: obtaining the execution log of the current query request; according to the execution log, obtaining the actual resource consumption value, actual execution time, scheduling time, and waiting time of the current query request; Actual resource consumption value, actual execution time, scheduling time and waiting time, adjust the correspondence between expected resource consumption value and resource consumption level, or adjust the calculation method of expected resource consumption value.
- a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:
- resources are allocated to the current query request.
- the following steps are specifically implemented: parsing the current query request to obtain the execution plan tree of the current query request ; According to the execution plan tree, determine the expected resource consumption value.
- the following steps are specifically implemented: according to preset table statistics and preset field statistics , Determine the resource consumption value of each node in the execution plan tree; determine the expected resource consumption value according to the resource consumption value of each node.
- the following steps are specifically implemented: according to preset table statistics, preset field statistics As well as the execution plan tree, determine the number of query rows and data input and output costs of the target data table.
- the target data table is the data table that needs to be used in the current query request; according to the preset table statistics and preset fields Statistical information and execution plan tree to determine the connection cost of the current query request, the number of connected rows, the grouping cardinality, and the central processing unit cost; according to the number of query rows, data input and output costs, connection costs, connected rows, and grouping cardinals And the CPU cost to determine the expected resource consumption value.
- the following steps are specifically implemented: parsing the current query request to obtain the execution plan tree of the current query request ; Send the execution plan tree to the cost analysis server to obtain the expected resource consumption value returned by the cost analysis server, and the expected resource consumption is determined according to the execution plan tree.
- the following steps are specifically implemented: determine the current query request based on the current resource consumption level Queue, in which query requests of different resource consumption levels are allocated to different queues; according to the resource scheduling strategy of the current query request queue, the current query request is allocated resources.
- the computer program further implements the following steps when being executed by the processor:
- the following steps are specifically implemented: corresponding to the cluster according to the preset resource consumption level Relationship, the current query request is allocated to the cluster corresponding to the current resource consumption level, wherein each cluster uses a fair scheduling method to allocate resources to the allocated query request.
- the computer program further implements the following steps when being executed by the processor:
- the following steps are further implemented: obtaining the execution log of the current query request; according to the execution log, obtaining the actual resource consumption value, actual execution duration, scheduling time, and waiting time of the current query request; According to the actual resource consumption value, actual execution time, scheduling time and waiting time, adjust the correspondence between the expected resource consumption value and the resource consumption level, or adjust the calculation method of the expected resource consumption value.
- Non-volatile memory may include read only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory.
- Volatile memory may include random access memory (RAM) or external cache memory.
- RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous chain Channel (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Signal Processing (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computer Networks & Wireless Communication (AREA)
- Software Systems (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
Description
Claims (13)
- 一种资源分配方法,其特征在于,所述方法包括:接收当前查询请求,获取所述当前查询请求的预期资源消耗值;根据所述预期资源消耗值,确定所述当前查询请求的当前资源消耗等级;根据所述当前资源消耗等级,对所述当前查询请求进行资源分配。
- 根据权利要求1所述的方法,其特征在于,所述获取所述当前查询请求的预期资源消耗值,包括:对所述当前查询请求进行解析,获得当前所述查询请求的执行计划树;根据所述执行计划树,确定所述预期资源消耗值。
- 根据权利要求2所述的方法,其特征在于,所述根据所述执行计划树,确定所述预期资源消耗值,包括:根据预设的表统计信息和预设的字段统计信息,确定所述执行计划树中各个节点的资源消耗值;根据各所述节点的资源消耗值,确定所述预期资源消耗值。
- 根据权利要求2所述的方法,其特征在于,所述根据所述执行计划树,确定所述预期资源消耗值,包括:根据预设的表统计信息、预设的字段统计信息以及所述执行计划树,确定目标数据表的查询行数和数据输入输出代价,所述目标数据表为在进行当前查询请求的查询时需要使用的各数据表;根据预设的表统计信息、预设的字段统计信息以及所述执行计划树,确定所述当前查询请求的连接代价、连接后的行数、分组基数和中央处理器代价;根据所述查询行数、所述数据输入输出代价、所述连接代价、所述连接后的行数、所述分组基数和所述中央处理器代价,确定所述预期资源消耗值。
- 根据权利要求1所述的方法,其特征在于,所述获取所述当前查询请求的预期资源消耗值,包括:对所述当前查询请求进行解析,获得所述当前查询请求的执行计划树;将所述执行计划树发送给代价分析服务器,获取所述代价分析服务器返回的预期资源消耗值,所述预期资源消耗根据所述执行计划树确定。
- 根据权利要求1所述的方法,其特征在于,所述根据所述当前资源消耗等级,对所述当前查询请求进行资源分配,包括:根据所述当前资源消耗等级,确定所述当前查询请求的队列,其中,不同资源消耗等级的查询请求分配到不同队列上;按照所述当前查询请求的队列的资源调度策略,对所述当前查询请求进行资源分配。
- 根据权利要求6所述的方法,其特征在于,所述方法还包括:获取各队列的资源使用率,在当前队列的资源使用率低于预设的资源使用率阈值时,将当前队列的资源调整到各所述队列中的剩余队列;或者获取各所述队列的资源需求率,在当前队列的资源需求率高于预设的资源需求率阈值时,通过预设的资源动态扩容方案为当前队列增加资源。
- 根据权利要求1所述的方法,其特征在于,所述根据所述当前资源消耗 等级,对所述当前查询请求进行资源分配,包括:根据预设的资源消耗等级与集群的对应关系,将所述当前查询请求分配给所述当前资源消耗等级对应的集群,其中,各集群中分别采用公平调度的方式为分配到的查询请求分配资源。
- 根据权利要求8所述的方法,其特征在于,所述方法还包括:获取各集群的资源使用率,在当前集群的资源使用率低于预设的资源使用率阈值时,将当前集群的资源调整到各所述集群中的剩余集群;或者获取各所述集群的资源需求率,在当前集群的资源需求率高于预设的资源需求率阈值时,通过预设的资源动态扩容方案为当前集群增加资源。
- 根据权利要求1所述的方法,其特征在于,所述方法还包括:获取所述当前查询请求的执行日志;根据所述执行日志,获取所述当前查询请求的实际资源消耗值、实际执行时长、调度时间和等待时间;根据所述实际资源消耗值、所述实际执行时长、所述调度时间和所述等待时间,调整预期资源消耗值和资源消耗等级的对应关系,或者调整预期资源消耗值的计算方式。
- 一种资源分配装置,其特征在于,所述装置包括:消耗值确定模块,用于接收当前查询请求,获取所述当前查询请求的预期资源消耗值;消耗等级确定模块,用于根据所述预期资源消耗值,确定所述当前查询请求的当前资源消耗等级;资源分配模块,用于根据所述当前资源消耗等级,对所述当前查询请求进行资源分配。
- 一种计算机设备,包括存储器、处理器及存储在存储器上并可在处理器上运行的计算机程序,其特征在于,所述处理器执行所述计算机程序时实现权利要求1至10中任一项所述方法的步骤。
- 一种计算机可读存储介质,其上存储有计算机程序,其特征在于,所述计算机程序被处理器执行时实现权利要求1至10中任一项所述的方法的步骤。
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CA3177212A CA3177212A1 (en) | 2019-04-16 | 2019-09-30 | Resource allocating method, device, computer equipment, and storage medium |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201910305271.0 | 2019-04-16 | ||
CN201910305271.0A CN110166282B (zh) | 2019-04-16 | 2019-04-16 | 资源分配方法、装置、计算机设备和存储介质 |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2020211300A1 true WO2020211300A1 (zh) | 2020-10-22 |
Family
ID=67639608
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/CN2019/109355 WO2020211300A1 (zh) | 2019-04-16 | 2019-09-30 | 资源分配方法、装置、计算机设备和存储介质 |
Country Status (3)
Country | Link |
---|---|
CN (1) | CN110166282B (zh) |
CA (1) | CA3177212A1 (zh) |
WO (1) | WO2020211300A1 (zh) |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN116401465A (zh) * | 2023-06-06 | 2023-07-07 | 支付宝(杭州)信息技术有限公司 | 资源分组分配方法、调整值确定方法和装置 |
Families Citing this family (15)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN110166282B (zh) * | 2019-04-16 | 2020-12-01 | 苏宁云计算有限公司 | 资源分配方法、装置、计算机设备和存储介质 |
CN110569257B (zh) * | 2019-09-16 | 2022-04-01 | 上海达梦数据库有限公司 | 数据处理方法、相应装置、设备及存储介质 |
CN111190927A (zh) * | 2019-12-17 | 2020-05-22 | 未鲲(上海)科技服务有限公司 | Sql查询命令处理方法、装置、设备及可读存储介质 |
CN111198767B (zh) * | 2020-01-07 | 2024-10-18 | 平安科技(深圳)有限公司 | 大数据资源处理方法、装置、终端及存储介质 |
CN111367656B (zh) * | 2020-02-20 | 2023-09-26 | 深圳震有科技股份有限公司 | 一种分配媒体资源的方法、计算机设备及存储介质 |
CN113535348A (zh) * | 2020-04-21 | 2021-10-22 | 华为技术有限公司 | 一种资源调度方法及相关装置 |
CN111694668B (zh) * | 2020-06-11 | 2023-06-30 | 广州市百果园信息技术有限公司 | 查询任务路由方法、装置、设备及存储介质 |
CN111475584B (zh) * | 2020-06-19 | 2021-01-22 | 阿里云计算有限公司 | 数据处理方法、系统以及装置 |
CN112131267B (zh) * | 2020-08-14 | 2023-10-03 | 北京达佳互联信息技术有限公司 | 计数处理方法、装置、服务器和计数处理系统 |
CN112579324A (zh) * | 2020-12-29 | 2021-03-30 | 中盈优创资讯科技有限公司 | 一种基于cost模型的商品汇总统计方法 |
CN113111083B (zh) * | 2021-03-31 | 2024-07-16 | 北京沃东天骏信息技术有限公司 | 数据查询的方法、装置、设备、存储介质和程序产品 |
CN113806068B (zh) * | 2021-07-30 | 2023-12-12 | 上海晶赞融宣科技有限公司 | 业务系统的扩容方法及装置、可读存储介质、终端 |
CN113553361A (zh) * | 2021-07-30 | 2021-10-26 | 北京东方国信科技股份有限公司 | 资源管理方法及装置 |
CN115145953A (zh) * | 2021-10-22 | 2022-10-04 | 上海卓辰信息科技有限公司 | 数据查询方法 |
CN117806778B (zh) * | 2024-02-29 | 2024-06-07 | 济南浪潮数据技术有限公司 | 资源管理方法、系统、设备及介质 |
Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20120110592A1 (en) * | 2004-11-08 | 2012-05-03 | International Business Machines Corporation | Autonomic Self-Tuning Of Database Management System In Dynamic Logical Partitioning Environment |
CN107797863A (zh) * | 2017-09-29 | 2018-03-13 | 西安交通大学 | 一种云计算平台中细粒度资源匹配方法 |
CN108845874A (zh) * | 2018-06-25 | 2018-11-20 | 腾讯科技(深圳)有限公司 | 资源的动态分配方法及服务器 |
CN110166282A (zh) * | 2019-04-16 | 2019-08-23 | 苏宁易购集团股份有限公司 | 资源分配方法、装置、计算机设备和存储介质 |
Family Cites Families (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN102567086B (zh) * | 2010-12-30 | 2014-05-07 | 中国移动通信集团公司 | 一种任务调度的方法、设备和系统 |
CN103064875B (zh) * | 2012-10-30 | 2017-06-16 | 中国标准化研究院 | 一种服务化空间数据分布式查询方法 |
CN107168977B (zh) * | 2016-03-08 | 2020-07-28 | 阿里巴巴集团控股有限公司 | 一种数据查询的优化方法及装置 |
WO2017190058A1 (en) * | 2016-04-28 | 2017-11-02 | Snowflake Computing Inc. | Multi-cluster warehouse |
CN106383864B (zh) * | 2016-09-02 | 2019-08-27 | 北京百度网讯科技有限公司 | 一种时序数据库的查询请求处理方法和装置 |
US11537615B2 (en) * | 2017-05-01 | 2022-12-27 | Futurewei Technologies, Inc. | Using machine learning to estimate query resource consumption in MPPDB |
CN108989583B (zh) * | 2018-08-10 | 2021-08-31 | 携程旅游信息技术(上海)有限公司 | 呼叫中心的压力测试方法、系统、电子设备和存储介质 |
-
2019
- 2019-04-16 CN CN201910305271.0A patent/CN110166282B/zh active Active
- 2019-09-30 WO PCT/CN2019/109355 patent/WO2020211300A1/zh active Application Filing
- 2019-09-30 CA CA3177212A patent/CA3177212A1/en active Pending
Patent Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20120110592A1 (en) * | 2004-11-08 | 2012-05-03 | International Business Machines Corporation | Autonomic Self-Tuning Of Database Management System In Dynamic Logical Partitioning Environment |
CN107797863A (zh) * | 2017-09-29 | 2018-03-13 | 西安交通大学 | 一种云计算平台中细粒度资源匹配方法 |
CN108845874A (zh) * | 2018-06-25 | 2018-11-20 | 腾讯科技(深圳)有限公司 | 资源的动态分配方法及服务器 |
CN110166282A (zh) * | 2019-04-16 | 2019-08-23 | 苏宁易购集团股份有限公司 | 资源分配方法、装置、计算机设备和存储介质 |
Cited By (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN116401465A (zh) * | 2023-06-06 | 2023-07-07 | 支付宝(杭州)信息技术有限公司 | 资源分组分配方法、调整值确定方法和装置 |
CN116401465B (zh) * | 2023-06-06 | 2023-10-20 | 支付宝(杭州)信息技术有限公司 | 资源分组分配方法、调整值确定方法和装置 |
Also Published As
Publication number | Publication date |
---|---|
CN110166282B (zh) | 2020-12-01 |
CN110166282A (zh) | 2019-08-23 |
CA3177212A1 (en) | 2020-10-22 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
WO2020211300A1 (zh) | 资源分配方法、装置、计算机设备和存储介质 | |
US11888702B2 (en) | Intelligent analytic cloud provisioning | |
US9672064B2 (en) | Dynamically adaptive, resource aware system and method for scheduling | |
US8082273B2 (en) | Dynamic control and regulation of critical database resources using a virtual memory table interface | |
RU2675054C2 (ru) | Балансировка нагрузки для больших баз данных в оперативной памяти | |
US8392404B2 (en) | Dynamic query and step routing between systems tuned for different objectives | |
CN108595254B (zh) | 一种查询调度方法 | |
US9235590B1 (en) | Selective data compression in a database system | |
US20090132611A1 (en) | Closed-loop system management method and process capable of managing workloads in a multi-system database environment | |
CN106294757B (zh) | 一种基于超图划分的分布式数据库及其集群分区方法 | |
Li et al. | Real-time scheduling based on optimized topology and communication traffic in distributed real-time computation platform of storm | |
US9110947B1 (en) | Column-oriented task execution in a row-partitioned database system | |
US8392461B2 (en) | Virtual data maintenance | |
US20190266154A1 (en) | High performance data profiler for big data | |
US20160274940A1 (en) | Regulating enterprise database warehouse resource usage | |
CN109918450A (zh) | 基于分析类场景下的分布式并行数据库及存储方法 | |
CN116089414B (zh) | 基于海量数据场景的时序数据库写入性能优化方法及装置 | |
US20160034528A1 (en) | Co-processor-based array-oriented database processing | |
CN107395446B (zh) | 日志实时处理系统 | |
CN114756629A (zh) | 基于sql的多源异构数据交互分析引擎及方法 | |
CN113568931A (zh) | 一种数据访问请求的路由解析系统及方法 | |
CN105608138A (zh) | 一种优化阵列数据库并行数据加载性能的系统 | |
CN110928649A (zh) | 资源调度的方法和装置 | |
US20130013824A1 (en) | Parallel aggregation system | |
CN115982230A (zh) | 数据库的跨数据源查询方法、系统、设备及存储介质 |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 19925306 Country of ref document: EP Kind code of ref document: A1 |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
122 | Ep: pct application non-entry in european phase |
Ref document number: 19925306 Country of ref document: EP Kind code of ref document: A1 |
|
32PN | Ep: public notification in the ep bulletin as address of the adressee cannot be established |
Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205 DATED 22/04/2022) |
|
ENP | Entry into the national phase |
Ref document number: 3177212 Country of ref document: CA |
|
122 | Ep: pct application non-entry in european phase |
Ref document number: 19925306 Country of ref document: EP Kind code of ref document: A1 |