CN112698943A - Resource allocation method, device, computer equipment and storage medium - Google Patents

Resource allocation method, device, computer equipment and storage medium Download PDF

Info

Publication number
CN112698943A
CN112698943A CN202011601345.4A CN202011601345A CN112698943A CN 112698943 A CN112698943 A CN 112698943A CN 202011601345 A CN202011601345 A CN 202011601345A CN 112698943 A CN112698943 A CN 112698943A
Authority
CN
China
Prior art keywords
resource
task
resources
resource pool
executed
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202011601345.4A
Other languages
Chinese (zh)
Inventor
吴浩然
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing Shunda Technology Co ltd
Original Assignee
Beijing Shunda Technology Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing Shunda Technology Co ltd filed Critical Beijing Shunda Technology Co ltd
Priority to CN202011601345.4A priority Critical patent/CN112698943A/en
Publication of CN112698943A publication Critical patent/CN112698943A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00Arrangements for program control, e.g. control units
    • G06F9/06Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46Multiprogramming arrangements
    • G06F9/50Allocation of resources, e.g. of the central processing unit [CPU]
    • G06F9/5005Allocation of resources, e.g. of the central processing unit [CPU] to service a request
    • G06F9/5027Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
    • G06F9/5038Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals considering the execution order of a plurality of tasks, e.g. taking priority or time dependency constraints into consideration
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00Arrangements for program control, e.g. control units
    • G06F9/06Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46Multiprogramming arrangements
    • G06F9/48Program initiating; Program switching, e.g. by interrupt
    • G06F9/4806Task transfer initiation or dispatching
    • G06F9/4843Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
    • G06F9/4881Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F2209/00Indexing scheme relating to G06F9/00
    • G06F2209/48Indexing scheme relating to G06F9/48
    • G06F2209/484Precedence
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F2209/00Indexing scheme relating to G06F9/00
    • G06F2209/50Indexing scheme relating to G06F9/50
    • G06F2209/5011Pool
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F2209/00Indexing scheme relating to G06F9/00
    • G06F2209/50Indexing scheme relating to G06F9/50
    • G06F2209/5021Priority

Abstract

The application provides a resource allocation method, a resource allocation device, a computer device and a storage medium, wherein the method comprises the following steps: acquiring a target task to be executed; determining total demand resources corresponding to the target task; when the residual resources in the first resource pool corresponding to the target task are smaller than the total demand resources, determining demand gap resources corresponding to the target task; and allocating available resources in a second resource pool based on the demand gap resources to obtain an outer area borrowing resource, wherein the outer area borrowing resource and the residual resources are used for executing the target task, and the second resource pool and the first resource pool have the same role attributes and different area attributes. By adopting the method, not only can the resource utilization rate be improved, but also the task execution efficiency can be improved.

Description

Resource allocation method, device, computer equipment and storage medium
Technical Field
The present application relates to the field of resource scheduling technologies, and in particular, to a resource allocation method, an apparatus, a computer device, and a storage medium.
Background
The resource scheduling is a resource balance strategy in the field of big data computing, and the main purpose of the resource scheduling is to perform unified control and allocation of resources on different computing tasks, so that the resources of the different tasks can be isolated and do not interfere with each other, the utilization of the resources is maximized, and the task execution efficiency is higher.
However, the existing resource scheduling method generally adopts a first-in first-out strategy, and it is very easy for a person to submit a plurality of sql (structured Query language) in a short time, which results in the situation that the whole cluster resource is exclusively occupied by one person. Meanwhile, since it cannot be determined which of the two SQL, i.e., A, B resource requirements is more matched with the current cluster resource, the task execution after resource allocation can be performed only in the order of SQL submission time, and therefore the following situations are very likely to occur: because the resource requirement of the task submitted earlier is not matched with the current cluster, the task can not be executed, and the rest of the tasks submitted later and matched with the resource requirement and the current cluster are blocked, so that the task processing efficiency is reduced.
Therefore, the existing resource scheduling method has the technical problem of low resource utilization rate due to unreasonable resource scheduling.
Disclosure of Invention
Therefore, it is necessary to provide a resource allocation method, a resource allocation device, a computer device, and a storage medium for reasonably allocating resources, improving resource utilization, and improving task processing efficiency.
In a first aspect, the present application provides a resource allocation method, including:
acquiring a target task to be executed;
determining total demand resources corresponding to the target task;
when the residual resources in the first resource pool corresponding to the target task are smaller than the total demand resources, determining demand gap resources corresponding to the target task;
and allocating available resources in a second resource pool based on the demand gap resources to obtain an outer area borrowing resource, wherein the outer area borrowing resource and the residual resources are used for executing the target task, and the second resource pool and the first resource pool have the same role attributes and different area attributes.
In some embodiments of the present application, the step of obtaining a target task to be executed includes:
acquiring task processing information of a task to be executed in a preset time period, wherein the task processing information comprises read-in data information, memory consumption information, core number occupation information, shuffle data information and aggregation frequency information;
calculating a weight according to preset time length, and performing weighted summation processing on the task processing information to obtain the resource occupation time length of the task to be executed, wherein the time length calculation weight is determined according to the task processing information of the historical task;
and acquiring the target task to be executed according to the resource occupation duration of the task to be executed.
In some embodiments of the present application, the task to be executed includes at least two tasks to be executed, and the step of obtaining the target task to be executed according to the resource occupation duration of the task to be executed includes:
determining the resource occupation duration corresponding to each task to be executed;
based on the resource occupation duration, performing ascending arrangement on the tasks to be executed to obtain a task sequence to be executed;
determining the task priority of each task to be executed based on the task sequence to be executed, wherein the task priority is reduced in sequence in the task sequence to be executed;
determining the task state of each task to be executed based on the task priority, wherein the task state comprises a state to be executed and a waiting state;
and acquiring a target task to be executed according to the task state of each task to be executed, wherein the task state of the target task is the task to be executed.
In some embodiments of the present application, after determining the task state of each task to be executed based on the task priority, the method further includes:
acquiring current time;
if the current time is within a preset idle time period, acquiring a first residual resource in a resource pool to which each task to be executed belongs and/or a second residual resource in a candidate resource pool based on the task priority of each task to be executed, so as to execute each task to be executed through the first residual resource and/or the second residual resource, wherein the candidate resource pool is an available resource pool except the resource pool;
and if the current time is out of the preset idle time period, executing the step of acquiring the target task to be executed according to the task state of each task to be executed.
In some embodiments of the present application, after the obtaining of the target task to be executed, the method further includes:
determining a user account corresponding to the target task;
acquiring at least one task associated with the user account, wherein the at least one task has a corresponding task state respectively, and the task state comprises an executing state;
if the at least one task comprises the task in the executing state, stopping executing the target task until the at least one task does not comprise the task in the executing state;
and if the at least one task does not comprise the task in the executing state, executing the step of determining the total demand resource corresponding to the target task.
In some embodiments of the present application, when the remaining resources in the first resource pool to which the target task belongs are less than the total required resources, the step of determining the required gap resources corresponding to the target task includes:
acquiring a residual resource in a first resource pool to which the target task belongs, wherein the residual resource is a difference between a first total resource in the first resource pool and a used resource in the first resource pool;
when the residual resource is smaller than the total demand resource, calculating the difference between the total demand resource and the residual resource to obtain a resource difference value;
and determining the resource difference value as a demand gap resource corresponding to the target task.
In some embodiments of the present application, the allocating available resources in the second resource pool based on the demand gap resource to obtain an outer zone lent resource includes:
determining a large area corresponding to the first resource pool as a target large area, wherein the target large area is preset and associated with at least one regional resource pool, the role attributes of the regional resource pools are the same, and the role attributes at least comprise one of the following: service attribute, research and development attribute;
determining a regional resource pool in each regional resource pool except the first resource pool as the second resource pool;
acquiring a preset resource splitting weight corresponding to the second resource pool and available resources in the second resource pool, wherein the available resources are determined according to the available resources in the second resource pool;
and allocating the available resources in the second resource pool based on the demand gap resources and the resource splitting weight to obtain the split resources of the outer zone.
In some embodiments of the present application, the step of obtaining available resources in the second resource pool includes:
acquiring a second total resource, a reserved resource and an in-use resource in the second resource pool, wherein the in-use resource is a resource occupied by a task in the second resource pool, and the reserved resource is a preset resource which cannot be allocated in the second resource pool;
if the active resources are larger than the reserved resources, determining a difference value between the second total resources and the active resources as the available resources;
and if the used resources are smaller than the reserved resources, determining a difference value between the second total resources and the reserved resources as the available resources.
In some embodiments of the present application, the allocating available resources in the second resource pool based on the demand gap resource to obtain an outer zone lent resource further includes:
acquiring the sum of the split resources of the outer area and the residual resources to obtain the available total resources corresponding to the target task;
if the available total resources are less than the total demand resources, determining candidate large areas except the target large area in each large area according to the role attributes of the first resource pool, wherein the candidate large areas are preset and associated with at least one candidate regional resource pool;
determining a target regional resource pool in each candidate regional resource pool based on a preset large region arrangement sequence, wherein the candidate large region to which the target regional resource pool belongs is arranged behind the target large region;
and acquiring a target available resource in the target area resource pool according to a resource difference value between the total demand resource and the available total resource and a corresponding preset resource splitting weight of the target area resource pool, wherein the target available resource is used for executing the target task by combining the external area borrowing resource and the residual resource.
In a second aspect, the present application provides an apparatus for resource allocation, the apparatus comprising:
the task acquisition module is used for acquiring a target task to be executed;
the resource determining module is used for determining total demand resources corresponding to the target task;
a gap determining module, configured to determine a required gap resource corresponding to the target task when a remaining resource in a first resource pool to which the target task belongs is smaller than the total required resource;
and the resource allocation module is used for allocating available resources in a second resource pool based on the demand gap resource to obtain an outer area borrowing resource, the outer area borrowing resource and the residual resource are used for executing the target task, and the second resource pool and the first resource pool have the same role attribute and different area attributes.
In a third aspect, the present application further provides a server, including:
one or more processors;
a memory; and
one or more application programs, wherein the one or more application programs are stored in the memory and configured to be executed by the processor to implement the resource allocation method.
In a fourth aspect, the present application further provides a computer-readable storage medium, on which a computer program is stored, the computer program being loaded by a processor to perform the steps of the resource allocation method.
According to the resource allocation method, the resource allocation device, the computer equipment and the storage medium, the total demand resource corresponding to the target task is determined by obtaining the target task to be executed, so that when the residual resource in the first resource pool corresponding to the target task is smaller than the total demand resource, the available resource in the second resource pool can be allocated based on the demand gap resource corresponding to the target task, and therefore the external area borrowing resource required by executing the target task is obtained. The method and the device have the advantages that the resources available for the tasks are subjected to the resource isolation division of role attributes and region attributes in advance, so that the tasks can use the resources in the exclusive resource pool in the execution process, and the resource isolation and mutual noninterference of different tasks are realized. Meanwhile, a resource allocation scheme between the first resource pool and the second resource pool is set, so that the resource utilization rate can be improved, and the task execution efficiency can be improved.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.
Fig. 1 is a schematic view of a scenario of a resource allocation method in an embodiment of the present application;
FIG. 2 is a diagram illustrating a result of partitioning resource pools in an embodiment of the present application;
FIG. 3 is a flowchart illustrating a resource allocation method according to an embodiment of the present application;
FIG. 4(a) is a schematic diagram of resource partitioning of a resource pool in the embodiment of the present application;
FIG. 4(b) is a schematic diagram of resource allocation priorities of resource pools in the embodiment of the present application;
FIG. 5 is a flowchart illustrating a resource allocation method according to an embodiment of the present application;
fig. 6 is a schematic structural diagram of a resource allocation apparatus in an embodiment of the present application;
fig. 7 is a schematic structural diagram of a computer device in an embodiment of the present application.
Detailed Description
The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
In the description of the present application, it is to be understood that the terms "first", "second" are used for descriptive purposes only and are not to be construed as indicating or implying relative importance or implying any number of technical features indicated. Thus, features defined as "first", "second", may explicitly or implicitly include one or more of the described features. In the description of the present application, "a plurality" means two or more unless specifically limited otherwise.
In the description of the present application, it is to be understood that the word "for example" is used to mean "serving as an example, instance, or illustration. Any embodiment described herein as "for example" is not necessarily to be construed as preferred or advantageous over other embodiments. The following description is presented to enable any person skilled in the art to make and use the invention. In the following description, details are set forth for the purpose of explanation. It will be apparent to one of ordinary skill in the art that the present invention may be practiced without these specific details. In other instances, well-known structures and processes are not shown in detail to avoid obscuring the description of the invention with unnecessary detail. Thus, the present invention is not intended to be limited to the embodiments shown, but is to be accorded the widest scope consistent with the principles and features disclosed herein.
In the embodiment of the present application, it should be noted that, because the resource allocation method provided in the present application is executed in a computer device, processing objects of each computer device all exist in the form of data or information, for example, time, which is substantially time information, and it should be understood that, in the following embodiments, if resources, quantity, size, and the like are mentioned, corresponding data exist so as to be processed by the computer device, and details are not described herein.
In the embodiment of the present application, it should be further described that the resource allocation method provided in the embodiment of the present application may be applied to a resource allocation system as shown in fig. 1. Wherein the resource allocation system comprises a terminal 100, a server 200 and a database 300, the terminal 100 may be a device comprising both receiving and transmitting hardware, i.e. a device having receiving and transmitting hardware capable of performing a bi-directional communication over a bi-directional communication link. Such a device may include: a cellular or other communication device having a single line display or a multi-line display or a cellular or other communication device without a multi-line display. The terminal 100 may specifically be a desktop terminal or a mobile terminal, and the terminal 100 may also specifically be one of a mobile phone, a tablet computer, a notebook computer, and the like. The server 200 may be an independent server, or may be a server network or a server cluster composed of servers, which includes but is not limited to a computer, a network host, a single network server, a plurality of network server sets, or a cloud server composed of a plurality of servers. Among them, the Cloud server is constituted by a large number of computers or web servers based on Cloud Computing (Cloud Computing). The database 300 may be a relational database management system (MySQL), and stores data expressed in sql (structured Query language), such as task request data. In terms of WEB applications, MySQL is one of the best RDBMS (Relational Database Management System) application software, and the SQL language used by MySQL is the most common standardized language for accessing databases.
Those skilled in the art will appreciate that the application environment shown in fig. 1 is only one application scenario related to the present application, and does not constitute a limitation to the application scenario of the present application, and that other application environments may further include more or less computer devices than those shown in fig. 1, for example, only 1 server 200 is shown in fig. 1, and it is understood that the resource allocation system may further include one or more other servers, and is not limited herein. In addition, as shown in fig. 1, the resource allocation system may further include a memory for storing data, such as SQL record data.
It should be further noted that the scenario diagram of the resource allocation system shown in fig. 1 is only an example, and the resource allocation system and the scenario described in the embodiment of the present invention are for more clearly illustrating the technical solution of the embodiment of the present invention, and do not form a limitation on the technical solution provided in the embodiment of the present invention.
Finally, it should be noted that, in the resource allocation method provided in the embodiment of the present application, in order to allocate each resource and improve the resource utilization rate, role division, large area division, and regional division are performed on the resources in advance. As shown in fig. 2, that is, a schematic diagram of resource pool division provided in the embodiment of the present application, where "business" and "research and development" represent role attributes of a resource pool, "east China" and "south China" represent area attributes (large area attributes) of the resource pool, and "Nanjing" and "Guangdong" represent area attributes (regional attributes) of the resource pool. By the resource dividing mode based on the role, the large area and the region attributes, the role of the user can be isolated, the resource occupation of the user such as research and development, business and the like can be avoided, and the resource utilization rate can be further improved.
Referring to fig. 3, an embodiment of the present application provides a resource allocation method, which is mainly exemplified by applying the method to the server 200 in fig. 1, and the method includes steps S301 to S304, which are specifically as follows:
s301, acquiring a target task to be executed.
The tasks related to the embodiment of the application can be SQL query tasks, and are used for querying SQL statements which are being executed or are executed; the target task may be a target task currently to be executed by the server 200, i.e., an SQL query task currently to be parsed and submitted by the server 200.
Specifically, the resource allocation system provided in the embodiment of the present application includes a MySQL database, which may be disposed in the server 200 or in another server or platform, and is configured to store the SQL query tasks (hereinafter, referred to as SQL) submitted by the user, so that the server 200 can obtain each SQL analysis and allocate resources in time, thereby smoothly executing the tasks. It should be noted that the SQL stored in the MySQL database is not a target task, and the determination of the target task needs to be based on the analysis result of the server 200, so that the server 200 can obtain one or more SQL through the MySQL database, and then analyze each SQL to determine the target task to be executed. The target task acquisition step involved in the present embodiment will be described in detail below.
In one embodiment, this step includes: acquiring task processing information of a task to be executed in a preset time period, wherein the task processing information comprises read-in data information, memory consumption information, core number occupation information, shuffle data information and aggregation frequency information; calculating a weight according to preset time length, and performing weighted summation processing on the task processing information to obtain the resource occupation time length of the task to be executed, wherein the time length calculation weight is determined according to the task processing information of the historical task; and acquiring the target task to be executed according to the resource occupation duration of the task to be executed.
The preset time period refers to a historical time period for recording the SQL submission time, for example, within the previous 30 minutes, within the previous 2 hours, and the like. It can be understood that the preset time period in this embodiment may be a time period accumulated by taking the current computer time as the termination time, or may be a time period defined by a preset start-stop time. For example, the time period accumulated by taking the current computer time as the termination time may be within the last 30 minutes by taking the computer time at the moment as the ending time; the time period defined at the preset start-stop time may be 10 to 11 points yesterday, etc.
The task to be executed may be an SQL query task stored in the MySQL database, and all tasks are tasks to be executed, but not necessarily target tasks that the server 200 will currently process.
The task processing information can be obtained through an SQL execution plan, and comprises the following steps: the read-in data information can be memory consumption information, core number occupation information, shuffle data information and aggregation frequency information; the read-in data information may refer to the size of the read-in (input) data, denoted as "InputData"; the Memory consumption information may refer to the size of consumed Memory, denoted as "Memory"; the Core number occupation information may refer to the number of cores occupying a Central Processing Unit (CPU), and is denoted as "Core"; the shuffle data information may refer to a shuffle data size, denoted as "shuffle data"; the aggregation number information may refer to the number of aggregation calculations, denoted as "groupByData".
The duration calculation weight may be a preset duration calculation weight corresponding to different task processing information, specifically, a calculation weight of the resource occupation duration, and the weight may be obtained by performing linear regression analysis on historical data, or may be manually set. If the historical data is obtained through analysis, the historical SQL query task processing information can be determined through analysis, and if the historical SQL query task processing information is manually set, the historical SQL query task processing information can be manually set through experience.
The resource occupation duration may refer to a Time period for occupying resources in the resource pool, and is denoted as "Time". In the embodiment of the application, the larger the "Time" is, the longer the Time that the task occupies the resource is, and conversely, the smaller the "Time" is, the shorter the Time that the task occupies the resource is. In the embodiment of the present application, the determination of the priority of the task, that is, the determination of the target task with the highest priority, substantially depends on the specific size of the resource occupation duration. See the examples below for details of the analysis.
Specifically, before the server 200 obtains the target task to be executed, the task to be executed in the preset time period needs to be obtained, that is, the SQL query tasks stored in the MySQL database and submitted in accordance with the preset time period are obtained, so as to obtain the task processing information of each SQL query task, including read-in data information, memory consumption information, core number occupation information, shuffle data information, and aggregation number information. After the server 200 obtains the processing information of each task, the time consumption evaluation can be performed on the SQL, and the priority of each SQL is determined, so that the SQL with the highest priority is defined as the target task to be executed immediately. Aiming at the time-consuming evaluation mode of SQL, the weight is calculated according to the preset duration, and the weighted summation processing is carried out on each item of task processing information, wherein the calculation formula is as follows:
A×Core+B×Memory+C×InputData+D×ShuffleData+E×groupByData=Time
a, B, C, D, E are all the time length calculation weights of different task processing information.
More specifically, the server 200 obtains the SQL within the preset time period, and after analyzing the corresponding resource occupation duration according to the task processing information of each SQL, compares the resource occupation duration of each task to be executed, so as to obtain the target task to be executed by using the comparison result. The resource occupation duration analysis step in the present embodiment will be described in detail below.
In one embodiment, the task to be executed includes at least two tasks to be executed, and the step of obtaining the target task to be executed according to the resource occupation duration of the task to be executed includes: determining the resource occupation duration corresponding to each task to be executed; based on the resource occupation duration, performing ascending arrangement on the tasks to be executed to obtain a task sequence to be executed; determining the task priority of each task to be executed based on the task sequence to be executed, wherein the task priority is reduced in sequence in the task sequence to be executed; determining the task state of each task to be executed based on the task priority, wherein the task state comprises a state to be executed and a waiting state; and acquiring a target task to be executed according to the task state of each task to be executed, wherein the task state of the target task is the task to be executed.
Specifically, the server 200 obtains the resource occupation Time of each to-be-executed task SQL, and may perform ascending arrangement according to the size of the resource occupation Time to obtain a sequence in which the Time values are increased one by one, and further obtain the to-be-executed task sequence.
For example, the server 200 obtains 3 tasks to be executed: the method comprises SQL1, SQL2 and SQL3, wherein the resource occupation time of SQL1 is 30 minutes, the resource occupation time of SQL2 is 15 minutes, and the resource occupation time of SQL3 is 18 minutes, and then the task sequences to be executed finally obtained by the server 200 are SQL2, SQL3 and SQL 1.
More specifically, as SQL that consumes more resources is longer in the theoretical execution time of hardware resources under a certain condition, SQL that is more complex occupies more resources, and complex SQL may affect the rationality of resource scheduling, that is, the subsequent simple SQL may be blocked and cannot be executed because the preamble SQL is too complex and slow to execute. Therefore, the embodiment of the application provides priority division for the tasks to be executed, namely SQL with relatively short resource occupation duration is set as high priority, and SQL with relatively long resource occupation duration is set as low priority. The server 200 may determine the priority of each task to be executed in the sequence after acquiring the sequence of the sequenced tasks to be executed.
For example, the sequence of tasks to be executed currently obtained by the server 200 is: SQL2, SQL3, SQL1, based on the ordering of the tasks in the sequence, found that SQL2 has the highest priority, SQL3 has the lowest priority after SQL2, and SQL1 has the lowest priority.
Furthermore, the purpose of dividing the task priority is to determine the sequence of execution of the tasks to be executed, so that the simple SQL is processed preferentially, the complex SQL is processed later, and the resource utilization rate is improved. Therefore, after the server 200 determines the task priority of each task to be executed based on the task sequence to be executed, in order to clarify the processing timing of each task to be executed, the task state of each task to be executed may be modified, that is, the task state of the task to be processed with the highest priority is set as: a state to be executed; setting the task state of the task to be processed with the later priority as follows: and waiting for the state, and updating the task state of each SQL in the MySQL database, so that other platforms, servers or terminals can inquire in real time. Finally, the server 200 may set the SQL in the to-be-executed state as the target task, and if the target task is executed, the server 200 may sequentially modify the task state of each to-be-executed task according to the to-be-executed task sequence, so as to execute each to-be-executed task in sequence.
In one embodiment, after the determining the task state of each task to be executed based on the task priority, the method further includes: acquiring current time; if the current time is within a preset idle time period, acquiring a first residual resource in a resource pool to which each task to be executed belongs and/or a second residual resource in a candidate resource pool based on the task priority of each task to be executed, so as to execute each task to be executed through the first residual resource and/or the second residual resource, wherein the candidate resource pool is an available resource pool except the resource pool; and if the current time is out of the preset idle time period, executing the step of acquiring the target task to be executed according to the task state of each task to be executed.
The current time may be a network time queried by the server 200 through the internet, and the time is limited by time zones, that is, there is a difference between times in different time zones, and the application takes the beijing time as an example, for example, the current time is 1 month 1 am 10 am of 2020 year of beijing. It is understood that, although the current time in the embodiment of the present application is the beijing time, it is not excluded that other time zones may be used in other embodiments, or the time obtained by the server 200 based on the preset timing rule, and the setting of the timing rule depends on the actual business requirement.
The idle time period refers to a low peak time period occupied by the resource, for example, 21 to 24 points (the second day, point 0), 0 to 7 points, and the like.
The first residual resource refers to a residual resource in a resource pool corresponding to the currently executed task; the second remaining resource refers to a remaining resource in the other resource pools except the resource pool corresponding to the current task. It can be understood that each resource pool has an upper resource limit, that is, the total resource that can be stored in the resource pool is limited, and the remaining resource is the total resource — the used resource, and the used resource may be a resource that is occupied by other parallel services in use, or a resource that is consumed by a historical task and cannot be used again.
Specifically, after the server 200 evaluates the time consumption of each SQL, determines the priority, modifies the task state, and updates the database record, the task execution time may be analyzed to provide different task execution resource scheduling manners, that is, because the task execution generally has a peak and a valley, if the task in the peak period wants to be executed smoothly and effectively, it is necessary to ensure that a resource pool corresponding to the task itself can be reserved with a bottom-preserved resource for standby, the bottom-preserved resource is called as a reserved resource, and the reserved resource is not sharable and is dedicated to the task source of the divided and determined role, large area, and region. However, the task at the low peak time does not have the condition of heap bundling, and the condition that the resource in the resource pool a is continuously idle and the resource pool B cannot meet the requirement of task execution often exists.
Therefore, for the tasks submitted in the peak period and the low peak period, the application provides two resource allocation modes: (1) if the current execution time of the SQL task is within the preset idle time period, the server 200 may not need to consider the limitation of "reserved resources" in each resource pool, and if the remaining resources in the resource pool corresponding to the task (corresponding to the role, the large area, and the region) are insufficient, the remaining resources in other resource pools may be removed to obtain enough resources to execute the task. For example, if the service party in south-south beige of china submits an SQL task, the current time for executing the task is in the idle time period, and the server 200 finds that the remaining resources in the resource pool corresponding to the service party in south-south beige of china are seriously insufficient during the execution of the task, the resources in the resource pool corresponding to the service party in south-south beige of china can be lent in a detachable manner, and at this time, it is not necessary to care whether the lent resource pool has the undelible reserved resources. (2) If the current time for the execution of the SQL task is outside the preset idle time period, the server 200 may need to take into account the limitation of "reserved resources" in each resource pool. For the resource allocation manner (2), the details will be specifically analyzed below.
In one embodiment, after the obtaining of the target task to be executed, the method further includes: determining a user account corresponding to the target task; acquiring at least one task associated with the user account, wherein the at least one task has a corresponding task state respectively, and the task state comprises an executing state; if the at least one task comprises the task in the executing state, stopping executing the target task until the at least one task does not comprise the task in the executing state; and if the at least one task does not comprise the task in the executing state, executing the step of determining the total demand resource corresponding to the target task.
Specifically, the SQL submitted by the user should be associated with the user account thereof, and the server 200 may obtain the user account corresponding to each SQL by obtaining each SQL. In order to prevent a resource from being occupied by an individual and reducing the resource utilization rate due to the fact that the individual submits SQL for multiple times, a resource exclusive prevention scheme is set, and the scheme is described in detail in the embodiment: acquiring whether the user account has other SQL which is being executed or not by determining the user account to which the currently executed target task belongs, and if so, suspending the execution of the target task until the SQL submitted by the preorder is completely executed and then starting the target task; otherwise, the target task can be continuously executed.
S302, determining the total demand resource corresponding to the target task.
The total demand resource refers to the total amount of resources required for executing the target task.
Specifically, the total demand resources of the target task may be directly obtained through the server 200, or may be obtained through analysis by the server 200, that is, the total demand resources corresponding to the target task are determined according to the index required to be achieved by the target task and the preset resource usage prediction table.
S303, when the residual resource in the first resource pool corresponding to the target task is smaller than the total demand resource, determining the demand gap resource corresponding to the target task.
The first resource pool is a resource pool to which the target task belongs, for example, the target task is SQL submitted by business personnel in the major area of east china and the area of south beijing, and then the resource pool of south beijing in the east china under the business role shown in fig. 2 is the first resource pool.
Wherein, the remaining resource in the first resource pool is (the first total resource in the first resource pool) - (the used resource in the first resource pool), because the remaining resource is the total resource-the used resource. For example, if the first total resource in the first resource pool is "10G", the used resource in the first resource pool is "7G", and the remaining resource in the first resource pool is "3G".
Specifically, the SQL execution needs to ensure sufficient resources, and if the resources are insufficient, the resources need to be reasonably scheduled and allocated, but in order to isolate various resources, so as to ensure that the execution of each SQL is not interfered with each other, and promote task execution efficiency, the present application performs role + region division on the resource pools in advance to obtain each independent resource pool as shown in fig. 2, so that when the remaining resources in the first resource pool to which the target task corresponds are less than the total required resources, it is indicated that the remaining available resources in the resource pool to which the target task corresponds are insufficient, and the required gap resources corresponding to the target task can be determined, so as to schedule the resources in other resource pools, and promote the target task to be smoothly executed. The requirement gap resource determination procedure in this embodiment will be described in detail below.
In one embodiment, this step includes: acquiring a residual resource in a first resource pool to which the target task belongs, wherein the residual resource is a difference between a first total resource in the first resource pool and a used resource in the first resource pool; when the residual resource is smaller than the total demand resource, calculating the difference between the total demand resource and the residual resource to obtain a resource difference value; and determining the resource difference value as a demand gap resource corresponding to the target task.
Specifically, when the remaining resources in the first resource pool to which the target task belongs are smaller than the total demand resources, the difference between the total demand resources and the remaining resources may be calculated as demand gap resources corresponding to the target task.
For example, if the first total resource in the first resource pool is "10G", the used resource in the first resource pool is "7G", the remaining resource in the first resource pool is "3G", and the total required resource of the target task is "5G", then the remaining resource in the first resource pool is smaller than the total required resource, and the required gap resource corresponding to the target task is "5-3 ═ 2G". It is to be understood that, although the resource unit is expressed as "G" in the present embodiment and the above embodiments, it is not excluded that "MB", "KB", and the like may be used in an actual application scenario.
S304, based on the demand gap resource, allocating the available resource in the second resource pool to obtain the external area borrowing resource.
Wherein, the second resource pool refers to other resource pools except the first resource pool.
The available resources refer to other resources except reserved resources in the resource pool, and the reserved resources are exclusively used by tasks corresponding to the resource pool and do not need to be shared to fixed resources of the tasks corresponding to the other resource pools.
Specifically, if the server 200 analyzes and determines the demand gap resource corresponding to the target task, the available resource in the second resource pool needs to be allocated to obtain the external area lending resource provided by the second resource pool, so that the external area lending resource combines with the remaining resource in the first resource pool, and the target task can be smoothly executed. The external region lending resource acquiring step involved in the present embodiment will be described in detail below.
In one embodiment, this step includes: determining a large area corresponding to the first resource pool as a target large area, wherein the target large area is preset and associated with at least one regional resource pool, the role attributes of the regional resource pools are the same, and the role attributes at least comprise one of the following: service attribute, research and development attribute; determining a regional resource pool in each regional resource pool except the first resource pool as the second resource pool; acquiring a preset resource splitting weight corresponding to the second resource pool and available resources in the second resource pool, wherein the available resources are determined according to the available resources in the second resource pool; and allocating the available resources in the second resource pool based on the demand gap resources and the resource splitting weight to obtain the split resources of the outer zone.
The regional resource pool refers to both a regional exclusive resource pool corresponding to each role and a regional exclusive resource pool in a certain large area, for example, a Nanjing regional resource pool and a Guangdong regional resource pool corresponding to a "business" role; the resource pool in Nanjing area is a resource pool special for Nanjing in the large area of east China; the resource pool in the Guangdong region is a resource pool special for the Guangdong in the large area of south China.
The resource splitting weight refers to a preset weight for splitting resources in proportion, for example, 2, 5, and the like.
Specifically, as shown in fig. 2, a schematic diagram of resource pool partitioning provided in the embodiment of the present application is shown, and a specific partitioning manner has been described in detail above, but it is not described above that, when a resource pool is partitioned, the total number of resources corresponding to different roles is limited, and the total resources corresponding to different macrocells are also limited, so that the total resources obtained by each macrocells under different roles can be partitioned according to resource partitioning weights of each macrocell, for example, the total resources corresponding to a "service" role is "20G", and four macrocells currently exist: east China, south China, West China and North China, and the resource division weight of each large area is as follows in sequence: 1. 3, 2 and 4, the total resource of the great zone in east China is '2G', the total resource of the great zone in south China is '6G', the total resource of the great zone in west China is '4G' and the total resource of the great zone in north China is '8G'. Similarly, the total resources obtained in each area under different large areas can be divided according to the resource division weight of each area, for example, the total resources of the north China area under the role of "business" is "8G", and 2 areas exist in the north China area: beijing and Harbin, the resource partition weight of Beijing is 7, the resource partition weight of Harbin is 3, the total resource of Beijing area under the role of 'business' is 5.6G, and the total resource of Harbin area under the role of 'business' is 2.4G.
More specifically, after the server 200 analyzes and determines the demand gap resource corresponding to the target task, the available resource in the second resource pool may be allocated to obtain the external area loan resource provided by the second resource pool, and the specific allocation manner is as follows: if the demand gap resource of the target task needs to be acquired, the resources of other resource pools belonging to the same large area as the first resource pool can be preferentially borrowed, and the large area may only comprise the first resource pool or may comprise other resource pools except the first resource pool, so that the resources of the resource pools at the same level can be preferentially scheduled and allocated due to the insufficient resources of the first resource pool.
For example, in the resource pool division diagram shown in fig. 4(a), if the first resource pool is a east-shanghai resource pool under the role of "business" and the remaining resources of the first resource pool are not enough to execute the target task at the moment, the required demand gap resources can be preferentially borrowed from the state resource pool and the Nanjing resource pool belonging to the same east district of China, where the east district of China is the target district and the Hangzhou resource pool and the Nanjing resource pool are the second resource pools. After determining all the second resource pools, the server 200 may obtain and determine an external region split resource according to the preset resource split weight corresponding to the second resource pool and the available resources in each second resource pool. For example, when the resource splitting weight of the Nanjing resource pool is "6" and the resource splitting weight of the Hangzhou resource pool is "4", and at this time, a memory of "10G" needs to be split (the required gap resource is "10G"), an outer region split resource with a memory size of 6 ÷ (6+4) × 10 ═ 6G needs to be split from the Nanjing resource pool, and an outer region split resource with a memory size of "4G" needs to be split from the Hangzhou resource pool. If the available resources of the Nanjing resource pool are more than 6G and the available resources of the Hangzhou resource pool are more than 4G, the split resources of the outer region can be obtained smoothly; if the available resource of the Nanjing resource pool is less than 6G and/or the available resource of the Hangzhou resource pool is less than 4G, other resource allocation schemes are adopted to obtain enough external region split resources so as to meet the demand gap resource. Other resource allocation schemes involved in the present embodiment will be described in detail below.
In one embodiment, the step of obtaining the available resources in the second resource pool includes: acquiring a second total resource, a reserved resource and an in-use resource in the second resource pool, wherein the in-use resource is a resource occupied by a task in the second resource pool, and the reserved resource is a preset resource which cannot be allocated in the second resource pool; if the active resources are larger than the reserved resources, determining a difference value between the second total resources and the active resources as the available resources; and if the used resources are smaller than the reserved resources, determining a difference value between the second total resources and the reserved resources as the available resources.
Specifically, the above embodiment has described in detail how to allocate the available resources of the second resource pool, but does not describe how to acquire the available resources of the second resource pool, so the embodiment will describe in detail how to acquire the available resources of the second resource pool: before the server 200 allocates the available resources of the second resource pool, it needs to first obtain the second total resources, the reserved resources and the in-use resources in the second resource pool, and then compare the in-use resources with the reserved resources. If the in-use resource is greater than the reserved resource, the available resource may be determined as a difference between the second total resource and the in-use resource; conversely, if the in-use resources are less than the reserved resources, the available resources may be determined as a difference between the second total resources and the reserved resources.
For example, if the second total resource in the second resource pool is "10G", the reserved resource is "5G", and the active resource is "7G", the active resource at this time is larger than the reserved resource, and the available resource is "10-7 ═ 3G". For another example, if the second total resource in the second resource pool is "10G", the reserved resource is "5G", and the active resource is "3G", the active resource at this time is smaller than the reserved resource, and the available resource is "10-5 ═ 5G". It is to be understood that, although it is clear how to determine the available resources when the in-use resources are equal to the reserved resources in the present embodiment, it can be seen from the above example that, when the in-use resources are equal to the reserved resources, the available resources are at most "10-5 ═ 5G". That is, the task corresponding to each resource pool preferentially uses the reserved resource in the resource pool to which the task belongs, and can borrow and split the resources in the resource pool except the reserved resource under the condition that the reserved resource is used and idle resources still exist; when the reserved resources are not idle or even insufficient, the redundant resources in the resource pool can be only borrowed and split.
In one embodiment, the method further comprises: acquiring the sum of the split resources of the outer area and the residual resources to obtain the available total resources corresponding to the target task; if the available total resources are less than the total demand resources, determining candidate large areas except the target large area in each large area according to the role attributes of the first resource pool, wherein the candidate large areas are preset and associated with at least one candidate regional resource pool; determining a target regional resource pool in each candidate regional resource pool based on a preset large region arrangement sequence, wherein the candidate large region to which the target regional resource pool belongs is arranged behind the target large region; and acquiring a target available resource in the target area resource pool according to a resource difference value between the total demand resource and the available total resource and a corresponding preset resource splitting weight of the target area resource pool, wherein the target available resource is used for executing the target task by combining the external area borrowing resource and the residual resource.
The candidate regional resource pool is a regional resource pool of a candidate large area; the candidate large area is a large area of a large area to which the non-first resource pool belongs; the target regional resource pool refers to a regional resource pool of a candidate large region arranged behind the target large region, for example, the large region to which the first resource pool belongs is "north china", the candidate large region sequentially includes "east china", "south china" and "west china", and then the candidate regional resource pool includes "east china-hang state resource pool", "east china-south jing resource pool", "south china-Shenzhen resource pool", "south china-Guangzhou resource pool", "west china-Yunnan resource pool" and "west china-Guiyang resource pool", and then the target regional resource pool should be east china-Hangzhou resource pool "and" south china-Nanjing resource pool ", because the east china is arranged behind the north china". It is to be understood that, although the present embodiment proposes that the resource allocation may be performed according to the large region ranking order, it is not excluded that the resource allocation may also be performed by specifying the resource pool in other embodiments.
Specifically, the above embodiments have described in detail how to allocate the available resources of the second resource pool, and it is described that when there is a resource allocation requirement, resource borrowing should be preferentially performed on resource pools in different regions of the same large area, but it does not describe how to allocate resources from where resource pools in different regions of the same large area still cannot meet the target task. Therefore, this embodiment will further describe that when the sum of the split resources of the outer area corresponding to the resource pool of each area and the remaining resources in the first resource pool in the target large area, that is, the sum of the resources currently matchable to the target task is still smaller than the total required resources needed by the target task to execute the task, the server 200 may continue to sequentially perform resource borrowing and splitting to the resource pools of different large areas according to the preset resource allocation rule. However, it should be noted that the regional resource pool that can be used as a candidate resource should be a resource pool that belongs to the same role attribute as the first resource pool, and based on this rule, the resource allocation priority of each second resource pool can be determined.
For example, as shown in fig. 4(b), when the remaining resources of the first resource pool "east china-shanghai resource pool" are insufficient, the resource allocation priorities of the second resource pools are sequentially decreased to: "China east-Hangzhou resource pool", "China east-Nanjing resource pool" and "China south-Guangzhou resource pool".
In the resource allocation method in the foregoing embodiment, the total required resource corresponding to the target task is determined by obtaining the target task to be executed, so that when the remaining resource in the first resource pool to which the target task belongs is smaller than the total required resource, the available resource in the second resource pool can be allocated based on the requirement gap resource corresponding to the target task, thereby obtaining the external area lent resource required by executing the target task. The resources available for the tasks are subjected to resource isolation division of role attributes and region attributes in advance, so that the tasks can use the resources in the exclusive resource pool in the execution process, and the resource isolation and mutual noninterference of different tasks are realized. Meanwhile, a resource allocation scheme between the first resource pool and the second resource pool is set, so that the resource utilization rate can be improved, and the task execution efficiency can be improved.
In order to fully understand the resource allocation scheme proposed by the present application, the present application further provides an application scenario applying the resource allocation method described above. Specifically, as shown in fig. 5, the application of the resource allocation method in the application scenario is to obtain a target task to be executed, so as to determine a total required resource corresponding to the target task, so that when a remaining resource in a first resource pool to which the target task belongs is smaller than the total required resource, the available resource in a second resource pool can be allocated based on a requirement gap resource corresponding to the target task, thereby obtaining an external area leased resource required for executing the target task. The resources available for the tasks are subjected to resource isolation division of role attributes and region attributes in advance, so that the tasks can use the resources in the exclusive resource pool in the execution process, and the resource isolation and mutual noninterference of different tasks are realized. Meanwhile, a resource allocation scheme between the first resource pool and the second resource pool is set, so that the utilization rate of resources can be improved, and the task execution efficiency can be improved.
In order to better implement the resource allocation method in the embodiment of the present application, on the basis of the resource allocation method, a resource allocation apparatus is further provided in the embodiment of the present application, as shown in fig. 6, the resource allocation apparatus 600 includes:
a task obtaining module 610, configured to obtain a target task to be executed;
a resource determining module 620, configured to determine a total required resource corresponding to the target task;
a gap determining module 630, configured to determine a required gap resource corresponding to the target task when a remaining resource in the first resource pool to which the target task belongs is smaller than the total required resource;
a resource allocation module 640, configured to allocate an available resource in a second resource pool based on the demand gap resource, to obtain an outer area lent resource, where the outer area lent resource and the remaining resource are used to execute the target task, and the second resource pool and the first resource pool have the same role attribute and different area attributes.
In some embodiments of the present application, the task obtaining module 610 is further configured to obtain task processing information of a task to be executed in a preset time period, where the task processing information includes read-in data information, memory consumption information, core number occupation information, shuffle data information, and aggregation number information; calculating a weight according to preset time length, and performing weighted summation processing on the task processing information to obtain the resource occupation time length of the task to be executed, wherein the time length calculation weight is determined according to the task processing information of the historical task; and acquiring the target task to be executed according to the resource occupation duration of the task to be executed.
In some embodiments of the present application, the tasks to be executed include at least two tasks to be executed, and the task obtaining module 610 is further configured to determine a resource occupation duration corresponding to each task to be executed; based on the resource occupation duration, performing ascending arrangement on the tasks to be executed to obtain a task sequence to be executed; determining the task priority of each task to be executed based on the task sequence to be executed, wherein the task priority is reduced in sequence in the task sequence to be executed; determining the task state of each task to be executed based on the task priority, wherein the task state comprises a state to be executed and a waiting state; and acquiring a target task to be executed according to the task state of each task to be executed, wherein the task state of the target task is the task to be executed.
In some embodiments of the present application, the task obtaining module 610 is further configured to obtain a current time; if the current time is within a preset idle time period, acquiring a first residual resource in a resource pool to which each task to be executed belongs and/or a second residual resource in a candidate resource pool based on the task priority of each task to be executed, so as to execute each task to be executed through the first residual resource and/or the second residual resource, wherein the candidate resource pool is an available resource pool except the resource pool; and if the current time is out of the preset idle time period, executing the step of acquiring the target task to be executed according to the task state of each task to be executed.
In some embodiments of the present application, the resource allocation apparatus 600 further includes an account verification module, configured to determine a user account to which the target task corresponds; acquiring at least one task associated with the user account, wherein the at least one task has a corresponding task state respectively, and the task state comprises an executing state; if the at least one task comprises the task in the executing state, stopping executing the target task until the at least one task does not comprise the task in the executing state; and if the at least one task does not comprise the task in the executing state, executing the step of determining the total demand resource corresponding to the target task.
In some embodiments of the present application, the gap determining module 630 is further configured to obtain a remaining resource in the first resource pool to which the target task belongs, where the remaining resource is a difference between a first total resource in the first resource pool and a used resource in the first resource pool; when the residual resource is smaller than the total demand resource, calculating the difference between the total demand resource and the residual resource to obtain a resource difference value; and determining the resource difference value as a demand gap resource corresponding to the target task.
In some embodiments of the present application, the resource allocation module 640 is further configured to determine a large area to which the first resource pool belongs as a target large area, where the target large area is preset and associated with at least one regional resource pool, and role attributes of the regional resource pools are the same, where the role attributes at least include one of: service attribute, research and development attribute; determining a regional resource pool in each regional resource pool except the first resource pool as the second resource pool; acquiring a preset resource splitting weight corresponding to the second resource pool and available resources in the second resource pool, wherein the available resources are determined according to the available resources in the second resource pool; and allocating the available resources in the second resource pool based on the demand gap resources and the resource splitting weight to obtain the split resources of the outer zone.
In some embodiments of the present application, the resource allocation module 640 is further configured to obtain a second total resource, a reserved resource, and an in-use resource in the second resource pool, where the in-use resource is a resource occupied by a task in the second resource pool, and the reserved resource is a preset resource that is not allocable in the second resource pool; if the active resources are larger than the reserved resources, determining a difference value between the second total resources and the active resources as the available resources; and if the used resources are smaller than the reserved resources, determining a difference value between the second total resources and the reserved resources as the available resources.
In some embodiments of the present application, the resource allocation module 640 is further configured to obtain a sum of the split resources in the outer zone and the remaining resources, so as to obtain an available total resource corresponding to the target task; if the available total resources are less than the total demand resources, determining candidate large areas except the target large area in each large area according to the role attributes of the first resource pool, wherein the candidate large areas are preset and associated with at least one candidate regional resource pool; determining a target regional resource pool in each candidate regional resource pool based on a preset large region arrangement sequence, wherein the candidate large region to which the target regional resource pool belongs is arranged behind the target large region; and acquiring a target available resource in the target area resource pool according to a resource difference value between the total demand resource and the available total resource and a corresponding preset resource splitting weight of the target area resource pool, wherein the target available resource is used for executing the target task by combining the external area borrowing resource and the residual resource.
In the above embodiment, the total demand resource corresponding to the target task is determined by obtaining the target task to be executed, so that when the remaining resource in the first resource pool to which the target task belongs is smaller than the total demand resource, the available resource in the second resource pool can be allocated based on the demand gap resource corresponding to the target task, thereby obtaining the external area borrowing resource required by executing the target task. The resources available for the tasks are subjected to resource isolation division of role attributes and region attributes in advance, so that the tasks can use the resources in the exclusive resource pool in the execution process, and the resource isolation and mutual noninterference of different tasks are realized. Meanwhile, a resource allocation scheme between the first resource pool and the second resource pool is set, so that the resource utilization rate can be improved, and the task execution efficiency can be improved.
In some embodiments of the present application, the resource allocation apparatus 600 may be implemented in the form of a computer program that is executable on a computer device as shown in fig. 7. The memory of the computer device may store various program modules constituting the resource allocation apparatus 600, such as the task obtaining module 610, the resource determining module 620, the gap determining module 630, and the resource allocating module 640 shown in fig. 6. The computer program constituted by the respective program modules causes the processor to execute the steps in the resource allocation method of the respective embodiments of the present application described in the present specification.
For example, the computer device shown in fig. 7 may execute step S201 through the task obtaining module 610 in the resource allocation apparatus 600 shown in fig. 6. The computer device may perform step S202 through the resource determination module 620. The computer device may perform step S203 through the gap determination module 630. The computer device may perform step S204 through the resource allocation module 640. The computer device includes a processor, a memory, and a network interface connected by a system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of an operating system and computer programs in the non-volatile storage medium. The network interface of the computer device is used for communicating with an external computer device through a network connection. The computer program is executed by a processor to implement a resource allocation method.
Those skilled in the art will appreciate that the architecture shown in fig. 7 is merely a block diagram of some of the structures associated with the disclosed aspects and is not intended to limit the computing devices to which the disclosed aspects apply, as particular computing devices may include more or less components than those shown, or may combine certain components, or have a different arrangement of components.
In some embodiments of the present application, there is provided a computer device comprising one or more processors; a memory; and one or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the processor to perform the steps of the above-described resource allocation method. Here, the steps of the resource allocation method may be steps in the resource allocation methods of the above-described embodiments.
In some embodiments of the present application, a computer-readable storage medium is provided, in which a computer program is stored, which is loaded by a processor, so that the processor performs the steps of the above-mentioned resource allocation method. Here, the steps of the resource allocation method may be steps in the resource allocation methods of the above-described embodiments.
The foregoing detailed description is directed to a resource allocation method, apparatus, computer device, and storage medium provided in the embodiments of the present application, and specific examples are applied in the present application to explain the principles and implementations of the present invention, and the descriptions of the foregoing embodiments are only used to help understand the method and the core ideas of the present invention; meanwhile, for those skilled in the art, according to the idea of the present invention, there may be variations in the specific embodiments and the application scope, and in summary, the content of the present specification should not be construed as a limitation to the present invention.

Claims (12)

1. A method for resource allocation, the method comprising:
acquiring a target task to be executed;
determining total demand resources corresponding to the target task;
when the residual resources in the first resource pool corresponding to the target task are smaller than the total demand resources, determining demand gap resources corresponding to the target task;
and allocating available resources in a second resource pool based on the demand gap resources to obtain an outer area borrowing resource, wherein the outer area borrowing resource and the residual resources are used for executing the target task, and the second resource pool and the first resource pool have the same role attributes and different area attributes.
2. The resource allocation method of claim 1, wherein the step of obtaining the target task to be executed comprises:
acquiring task processing information of a task to be executed in a preset time period, wherein the task processing information comprises read-in data information, memory consumption information, core number occupation information, shuffle data information and aggregation frequency information;
calculating a weight according to preset time length, and performing weighted summation processing on the task processing information to obtain the resource occupation time length of the task to be executed, wherein the time length calculation weight is determined according to the task processing information of the historical task;
and acquiring the target task to be executed according to the resource occupation duration of the task to be executed.
3. The resource allocation method according to claim 2, wherein the tasks to be executed include at least two tasks to be executed, and the step of obtaining the target task to be executed according to the resource occupation duration of the tasks to be executed includes:
determining the resource occupation duration corresponding to each task to be executed;
based on the resource occupation duration, performing ascending arrangement on the tasks to be executed to obtain a task sequence to be executed;
determining the task priority of each task to be executed based on the task sequence to be executed, wherein the task priority is reduced in sequence in the task sequence to be executed;
determining the task state of each task to be executed based on the task priority, wherein the task state comprises a state to be executed and a waiting state;
and acquiring a target task to be executed according to the task state of each task to be executed, wherein the task state of the target task is the task to be executed.
4. The method of claim 3, wherein after the determining the task status for each of the tasks to be performed based on the task priority, the method further comprises:
acquiring current time;
if the current time is within a preset idle time period, acquiring a first residual resource in a resource pool to which each task to be executed belongs and/or a second residual resource in a candidate resource pool based on the task priority of each task to be executed, so as to execute each task to be executed through the first residual resource and/or the second residual resource, wherein the candidate resource pool is an available resource pool except the resource pool;
and if the current time is out of the preset idle time period, executing the step of acquiring the target task to be executed according to the task state of each task to be executed.
5. The method of resource allocation according to claim 1, wherein after said obtaining a target task to be performed, the method further comprises:
determining a user account corresponding to the target task;
acquiring at least one task associated with the user account, wherein the at least one task has a corresponding task state respectively, and the task state comprises an executing state;
if the at least one task comprises the task in the executing state, stopping executing the target task until the at least one task does not comprise the task in the executing state;
and if the at least one task does not comprise the task in the executing state, executing the step of determining the total demand resource corresponding to the target task.
6. The method of claim 1, wherein the step of determining the demand gap resource corresponding to the target task when the remaining resource in the first resource pool to which the target task belongs is less than the total demand resource comprises:
acquiring a residual resource in a first resource pool to which the target task belongs, wherein the residual resource is a difference between a first total resource in the first resource pool and a used resource in the first resource pool;
when the residual resource is smaller than the total demand resource, calculating the difference between the total demand resource and the residual resource to obtain a resource difference value;
and determining the resource difference value as a demand gap resource corresponding to the target task.
7. The method of claim 1, wherein the step of allocating the available resource in the second resource pool based on the demand gap resource to obtain an outer zone leased resource comprises:
determining a large area corresponding to the first resource pool as a target large area, wherein the target large area is preset and associated with at least one regional resource pool, the role attributes of the regional resource pools are the same, and the role attributes at least comprise one of the following: service attribute, research and development attribute;
determining a regional resource pool in each regional resource pool except the first resource pool as the second resource pool;
acquiring a preset resource splitting weight corresponding to the second resource pool and available resources in the second resource pool, wherein the available resources are determined according to the available resources in the second resource pool;
and allocating the available resources in the second resource pool based on the demand gap resources and the resource splitting weight to obtain the split resources of the outer zone.
8. The method for resource allocation according to claim 7, wherein the step of obtaining the available resources in the second resource pool comprises:
acquiring a second total resource, a reserved resource and an in-use resource in the second resource pool, wherein the in-use resource is a resource occupied by a task in the second resource pool, and the reserved resource is a preset resource which cannot be allocated in the second resource pool;
if the active resources are larger than the reserved resources, determining a difference value between the second total resources and the active resources as the available resources;
and if the used resources are smaller than the reserved resources, determining a difference value between the second total resources and the reserved resources as the available resources.
9. The method of resource allocation according to claim 7, wherein the method further comprises:
acquiring the sum of the split resources of the outer area and the residual resources to obtain the available total resources corresponding to the target task;
if the available total resources are less than the total demand resources, determining candidate large areas except the target large area in each large area according to the role attributes of the first resource pool, wherein the candidate large areas are preset and associated with at least one candidate regional resource pool;
determining a target regional resource pool in each candidate regional resource pool based on a preset large region arrangement sequence, wherein the candidate large region to which the target regional resource pool belongs is arranged behind the target large region;
and acquiring a target available resource in the target area resource pool according to a resource difference value between the total demand resource and the available total resource and a corresponding preset resource splitting weight of the target area resource pool, wherein the target available resource is used for executing the target task by combining the external area borrowing resource and the residual resource.
10. An apparatus for resource allocation, the apparatus comprising:
the task acquisition module is used for acquiring a target task to be executed;
the resource determining module is used for determining total demand resources corresponding to the target task;
a gap determining module, configured to determine a required gap resource corresponding to the target task when a remaining resource in a first resource pool to which the target task belongs is smaller than the total required resource;
and the resource allocation module is used for allocating available resources in a second resource pool based on the demand gap resource to obtain an outer area borrowing resource, the outer area borrowing resource and the residual resource are used for executing the target task, and the second resource pool and the first resource pool have the same role attribute and different area attributes.
11. A computer device, characterized in that the computer device comprises:
one or more processors;
a memory; and
one or more application programs, wherein the one or more application programs are stored in the memory and configured to be executed by the processor to implement the resource allocation method of any of claims 1-9.
12. A computer-readable storage medium, having stored thereon a computer program which is loaded by a processor for performing the steps of the resource allocation method according to any one of claims 1 to 9.
CN202011601345.4A 2020-12-29 2020-12-29 Resource allocation method, device, computer equipment and storage medium Pending CN112698943A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202011601345.4A CN112698943A (en) 2020-12-29 2020-12-29 Resource allocation method, device, computer equipment and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202011601345.4A CN112698943A (en) 2020-12-29 2020-12-29 Resource allocation method, device, computer equipment and storage medium

Publications (1)

Publication Number Publication Date
CN112698943A true CN112698943A (en) 2021-04-23

Family

ID=75512176

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202011601345.4A Pending CN112698943A (en) 2020-12-29 2020-12-29 Resource allocation method, device, computer equipment and storage medium

Country Status (1)

Country Link
CN (1) CN112698943A (en)

Cited By (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113220452A (en) * 2021-05-10 2021-08-06 北京百度网讯科技有限公司 Resource allocation method, model training method, device and electronic equipment
CN113391928A (en) * 2021-08-17 2021-09-14 上海燧原科技有限公司 Hardware resource allocation method and device, electronic equipment and storage medium
CN113449037A (en) * 2021-06-29 2021-09-28 未鲲(上海)科技服务有限公司 AI-based SQL engine calling method, device, equipment and medium
CN113507441A (en) * 2021-06-08 2021-10-15 中国联合网络通信集团有限公司 Security resource expansion method, security protection management platform and data node
CN114356586A (en) * 2022-03-17 2022-04-15 飞腾信息技术有限公司 Processor and electronic equipment
CN114884900A (en) * 2022-06-09 2022-08-09 中国联合网络通信集团有限公司 Resource allocation method, device, equipment and storage medium

Citations (12)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20090320033A1 (en) * 2008-06-19 2009-12-24 Parag Gokhale Data storage resource allocation by employing dynamic methods and blacklisting resource request pools
CN102831012A (en) * 2011-06-16 2012-12-19 日立(中国)研究开发有限公司 Task scheduling device and task scheduling method in multimode distributive system
CN105827448A (en) * 2016-03-31 2016-08-03 乐视控股(北京)有限公司 Resource distribution method and apparatus
CN105912397A (en) * 2016-03-31 2016-08-31 乐视控股(北京)有限公司 Resources management method and device
CN107291545A (en) * 2017-08-07 2017-10-24 星环信息科技(上海)有限公司 The method for scheduling task and equipment of multi-user in computing cluster
CN107402807A (en) * 2017-07-18 2017-11-28 浪潮金融信息技术有限公司 Method, system and the processor of multitask execution efficiency are effectively lifted in computer systems
CN108768871A (en) * 2018-05-16 2018-11-06 阿里巴巴集团控股有限公司 Service traffics distribution method, device and equipment
CN110300184A (en) * 2019-07-10 2019-10-01 深圳市网心科技有限公司 Fringe node distribution method, device, dispatch server and storage medium
CN110673944A (en) * 2018-07-03 2020-01-10 杭州海康威视数字技术股份有限公司 Method and device for executing task
CN111104222A (en) * 2019-12-16 2020-05-05 上海众源网络有限公司 Task processing method and device, computer equipment and storage medium
CN111158879A (en) * 2019-12-31 2020-05-15 上海依图网络科技有限公司 System resource scheduling method, device, machine readable medium and system
CN111813524A (en) * 2020-07-09 2020-10-23 北京奇艺世纪科技有限公司 Task execution method and device, electronic equipment and storage medium

Patent Citations (12)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20090320033A1 (en) * 2008-06-19 2009-12-24 Parag Gokhale Data storage resource allocation by employing dynamic methods and blacklisting resource request pools
CN102831012A (en) * 2011-06-16 2012-12-19 日立(中国)研究开发有限公司 Task scheduling device and task scheduling method in multimode distributive system
CN105827448A (en) * 2016-03-31 2016-08-03 乐视控股(北京)有限公司 Resource distribution method and apparatus
CN105912397A (en) * 2016-03-31 2016-08-31 乐视控股(北京)有限公司 Resources management method and device
CN107402807A (en) * 2017-07-18 2017-11-28 浪潮金融信息技术有限公司 Method, system and the processor of multitask execution efficiency are effectively lifted in computer systems
CN107291545A (en) * 2017-08-07 2017-10-24 星环信息科技(上海)有限公司 The method for scheduling task and equipment of multi-user in computing cluster
CN108768871A (en) * 2018-05-16 2018-11-06 阿里巴巴集团控股有限公司 Service traffics distribution method, device and equipment
CN110673944A (en) * 2018-07-03 2020-01-10 杭州海康威视数字技术股份有限公司 Method and device for executing task
CN110300184A (en) * 2019-07-10 2019-10-01 深圳市网心科技有限公司 Fringe node distribution method, device, dispatch server and storage medium
CN111104222A (en) * 2019-12-16 2020-05-05 上海众源网络有限公司 Task processing method and device, computer equipment and storage medium
CN111158879A (en) * 2019-12-31 2020-05-15 上海依图网络科技有限公司 System resource scheduling method, device, machine readable medium and system
CN111813524A (en) * 2020-07-09 2020-10-23 北京奇艺世纪科技有限公司 Task execution method and device, electronic equipment and storage medium

Cited By (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113220452A (en) * 2021-05-10 2021-08-06 北京百度网讯科技有限公司 Resource allocation method, model training method, device and electronic equipment
CN113507441A (en) * 2021-06-08 2021-10-15 中国联合网络通信集团有限公司 Security resource expansion method, security protection management platform and data node
CN113449037A (en) * 2021-06-29 2021-09-28 未鲲(上海)科技服务有限公司 AI-based SQL engine calling method, device, equipment and medium
CN113449037B (en) * 2021-06-29 2024-04-12 云之富(上海)数据服务有限公司 AI-based SQL engine calling method, device, equipment and medium
CN113391928A (en) * 2021-08-17 2021-09-14 上海燧原科技有限公司 Hardware resource allocation method and device, electronic equipment and storage medium
CN113391928B (en) * 2021-08-17 2021-11-16 上海燧原科技有限公司 Hardware resource allocation method and device, electronic equipment and storage medium
CN114356586A (en) * 2022-03-17 2022-04-15 飞腾信息技术有限公司 Processor and electronic equipment
CN114884900A (en) * 2022-06-09 2022-08-09 中国联合网络通信集团有限公司 Resource allocation method, device, equipment and storage medium
CN114884900B (en) * 2022-06-09 2023-10-31 中国联合网络通信集团有限公司 Resource allocation method, device, equipment and storage medium

Similar Documents

Publication Publication Date Title
CN112698943A (en) Resource allocation method, device, computer equipment and storage medium
US9442760B2 (en) Job scheduling using expected server performance information
US9973512B2 (en) Determining variable wait time in an asynchronous call-back system based on calculated average sub-queue wait time
US8627325B2 (en) Scheduling memory usage of a workload
CN110262901B (en) Data processing method and data processing system
US10686728B2 (en) Systems and methods for allocating computing resources in distributed computing
CN103617087A (en) MapReduce optimizing method suitable for iterative computations
CN103927225A (en) Multi-core framework Internet information processing and optimizing method
CN111597048B (en) Micro-service scheduling method and system based on service quality and electronic equipment
CN111897654A (en) Method and device for migrating application to cloud platform, electronic equipment and storage medium
CN108519917A (en) A kind of resource pool distribution method and device
US20220156115A1 (en) Resource Allocation Method And Resource Borrowing Method
US9547576B2 (en) Multi-core processor system and control method
CN111798113A (en) Resource allocation method, device, storage medium and electronic equipment
CN112783659A (en) Resource allocation method and device, computer equipment and storage medium
CN107515781A (en) A kind of multicomputer system deterministic schedule and load-balancing method
CN111459622B (en) Method, device, computer equipment and storage medium for scheduling virtual CPU
CN111625339A (en) Cluster resource scheduling method, device, medium and computing equipment
WO2020108337A1 (en) Cpu resource scheduling method and electronic equipment
Liu et al. KubFBS: A fine‐grained and balance‐aware scheduling system for deep learning tasks based on kubernetes
EP4357916A1 (en) Job processing method and apparatus, computer device, and storage medium
CN116010093A (en) Data processing method, apparatus, computer device and readable storage medium
WO2020086053A1 (en) Dynamic allocation of computing resources for electronic design automation operations
US20210149746A1 (en) Method, System, Computer Readable Medium, and Device for Scheduling Computational Operation Based on Graph Data
CN114090234A (en) Request scheduling method and device, electronic equipment and storage medium

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination