CN117389715A - Resource allocation method, resource allocation device, computer device, and storage medium - Google Patents
Resource allocation method, resource allocation device, computer device, and storage medium Download PDFInfo
- Publication number
- CN117389715A CN117389715A CN202311108136.XA CN202311108136A CN117389715A CN 117389715 A CN117389715 A CN 117389715A CN 202311108136 A CN202311108136 A CN 202311108136A CN 117389715 A CN117389715 A CN 117389715A
- Authority
- CN
- China
- Prior art keywords
- resource
- allocated
- task
- processed
- resources
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000013468 resource allocation Methods 0.000 title claims abstract description 112
- 238000000034 method Methods 0.000 title claims abstract description 49
- 238000005516 engineering process Methods 0.000 claims abstract description 28
- 238000004590 computer program Methods 0.000 claims abstract description 25
- 230000008569 process Effects 0.000 claims description 6
- 238000012544 monitoring process Methods 0.000 claims description 4
- 238000004364 calculation method Methods 0.000 description 10
- 239000010410 layer Substances 0.000 description 5
- 238000010586 diagram Methods 0.000 description 4
- 238000004891 communication Methods 0.000 description 3
- 238000012545 processing Methods 0.000 description 3
- 230000008859 change Effects 0.000 description 2
- 230000003068 static effect Effects 0.000 description 2
- 239000002699 waste material Substances 0.000 description 2
- OKTJSMMVPCPJKN-UHFFFAOYSA-N Carbon Chemical compound [C] OKTJSMMVPCPJKN-UHFFFAOYSA-N 0.000 description 1
- 241001481833 Coryphaena hippurus Species 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 229910021389 graphene Inorganic materials 0.000 description 1
- 238000007726 management method Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 239000002344 surface layer Substances 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5011—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resources being hardware resources other than CPUs, Servers and Terminals
- G06F9/5016—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resources being hardware resources other than CPUs, Servers and Terminals the resource being the memory
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
- G06F9/5038—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals considering the execution order of a plurality of tasks, e.g. taking priority or time dependency constraints into consideration
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/50—Indexing scheme relating to G06F9/50
- G06F2209/5021—Priority
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/50—Indexing scheme relating to G06F9/50
- G06F2209/508—Monitor
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Abstract
The present invention relates to the field of big data technology, and in particular, to a resource allocation method, a resource allocation apparatus, a computer device, a storage medium, and a computer program product. The method comprises the steps of obtaining a task to be processed, determining a matched resource allocation strategy according to task attribute information, determining initial resources of the task to be processed according to the matched resource allocation strategy, and then distributing the initial resources to the task to be processed, so that the task to be processed can operate according to the initial resources. Because the initial resources are allocated to the task to be processed according to the attribute information of the task to be processed, the accuracy of resource allocation can be improved, reasonable allocation of resources is realized, and the problem of inaccuracy of manually allocated resources is avoided.
Description
Technical Field
The present invention relates to the field of big data technology, and in particular, to a resource allocation method, a resource allocation apparatus, a computer device, a storage medium, and a computer program product.
Background
With the development of big data technology, the application of big data scheduling management systems, such as a distributed scheduling framework plastic-job, a distributed workflow task scheduling system easysschedulers, dolphin schedulers and the like, is becoming more common. The big data scheduling program is operated through cluster resources, the resources required by the operation are estimated by a developer before the program is operated, and the relevant resource configuration is completed on a big data platform.
However, when the developer estimates that the resource is far greater than the resource required by the scheduler, other tasks cannot be executed due to lack of the resource, so that the resource utilization rate is low; when the estimated resources of the developer are smaller than the resources required by the program operation, the program operation fails, and then the developer is required to readjust the configuration resources, so that the workload is high. It can be seen that the accuracy of the conventional way of predicting resources by developers is not high.
Disclosure of Invention
In view of the foregoing, it is desirable to provide a resource allocation method, a resource allocation apparatus, a computer device, a computer-readable storage medium, and a computer program product that are capable of improving the accuracy of resource allocation.
In a first aspect, the present application provides a resource allocation method, where the method includes:
acquiring a task to be processed, wherein the task to be processed has corresponding task attribute information;
determining a matched resource allocation strategy according to the task attribute information, and determining initial resources of the task to be processed according to the matched resource allocation strategy;
and distributing the initial resources to the task to be processed so that the task to be processed runs according to the initial resources.
In one embodiment, after the allocating the initial resource to the task to be processed, the method further includes:
monitoring the resource utilization rate in the running process of the task to be processed;
and when the resource utilization rate reaches a set threshold, allocating additional resources to the task to be processed, wherein the additional resources are determined according to the initial resources and the set ratio.
In one embodiment, the task attribute information includes a data type and a data amount corresponding to the task to be processed; the resource allocation strategy comprises resources to be allocated, which are determined in advance according to a set data amount and correspond to the data types;
the determining a matched resource allocation strategy according to the task attribute information, and determining the initial resource of the task to be processed according to the matched resource allocation strategy comprises the following steps:
identifying a target data type corresponding to the task to be processed and a target data amount under the target data type;
determining the resources to be allocated, which are matched with the target data types, and calculating target resources to be allocated according to the matched resources to be allocated and the target data quantity;
and taking the target resource to be allocated as the initial resource of the task to be processed.
In one embodiment, the data type includes a service type, a technology type, and a computational complexity, and the resource allocation policy further includes a first weight corresponding to the service type, a second weight corresponding to the technology type, and a third weight corresponding to the computational complexity;
the determining the resource to be allocated matched with the target data type, and calculating the target resource to be allocated according to the matched resource to be allocated and the target data amount, including:
determining a first resource to be allocated matched with the service type, a second resource to be allocated matched with the technical type and a third resource to be allocated matched with the computational complexity according to the data volume;
and calculating the target resource to be allocated according to the first resource to be allocated, the first weight, the second resource to be allocated, the second weight, the third resource to be allocated and the third weight.
In one embodiment, the data types include a service type, a technology type and a computational complexity, and the resource allocation policy further includes priority conditions for different data types;
the determining the resource to be allocated matched with the target data type, and calculating the target resource to be allocated according to the matched resource to be allocated and the target data amount, including:
determining a target priority condition corresponding to the target data type;
and determining the resources to be allocated which are matched with the target data type according to the target priority condition, and calculating the target resources to be allocated according to the matched resources to be allocated and the target data volume under the target data type.
In one embodiment, the data types include a service type, a technology type and a computational complexity, and the resource allocation policy further includes priority conditions for different data types;
the determining the resource to be allocated matched with the target data type, and calculating the target resource to be allocated according to the matched resource to be allocated and the target data amount, including:
determining a target priority condition corresponding to the target data type;
determining resources to be allocated, which are matched with the target data type, according to the target priority condition;
determining corresponding weights according to the target data quantity under the target data type;
and calculating the target resource to be allocated according to the target data quantity under the target data type, the resource to be allocated and the weight.
In a second aspect, the present application provides a resource allocation apparatus, the apparatus comprising:
the task acquisition module is used for acquiring a task to be processed, wherein the task to be processed has corresponding task attribute information;
the initial resource determining module is used for determining a matched resource allocation strategy according to the task attribute information and determining initial resources of the task to be processed according to the matched resource allocation strategy;
and the resource allocation module is used for allocating the initial resources to the task to be processed so that the task to be processed runs according to the initial resources.
In a third aspect, the present application also provides a computer device comprising a memory storing a computer program and a processor implementing the steps of the above method when the processor executes the computer program.
In a fourth aspect, the present application also provides a computer readable storage medium having stored thereon a computer program which when executed by a processor performs the steps of the above-described method.
In a fifth aspect, the present application also provides a computer program product comprising a computer program which, when executed by a processor, implements the steps of the method described above.
According to the resource allocation method, the resource allocation device, the computer equipment, the storage medium and the computer program product, the task to be processed is obtained, the matched resource allocation strategy is determined according to the task attribute information, the initial resource of the task to be processed is determined according to the matched resource allocation strategy, and then the initial resource is allocated to the task to be processed, so that the task to be processed can operate according to the initial resource. Because the initial resources are allocated to the task to be processed according to the attribute information of the task to be processed, the accuracy of resource allocation can be improved, reasonable allocation of resources is realized, and the problem of inaccuracy of manually allocated resources is avoided.
Drawings
FIG. 1 is a flow chart of a method of resource allocation in one embodiment;
FIG. 2 is a flow diagram of the steps for determining initial resources in one embodiment;
FIG. 3 is a flowchart illustrating steps for calculating a target resource to be allocated in one embodiment;
FIG. 4 is a flowchart illustrating a step of calculating a target resource to be allocated according to another embodiment;
FIG. 5 is a flow chart of a method for allocating resources according to another embodiment;
FIG. 6 is a block diagram of a resource allocation device in one embodiment;
fig. 7 is an internal structural diagram of a computer device in one embodiment.
Detailed Description
In order to make the objects, technical solutions and advantages of the present application more apparent, the present application will be further described in detail with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the present application.
It should be noted that the method and the device for allocating resources of the present invention may be used in the financial field, and in particular, may be used in a big data scheduling system in the financial field, and may also be used in a big data scheduling system in any field other than the financial field.
In one embodiment, as shown in fig. 1, a resource allocation method is provided, and the method of this embodiment may be applied to a scheduling system, and may specifically include the following steps:
step 102, obtaining a task to be processed.
The task to be processed may be a code program to be executed, which may be manually started or scheduled to be executed. Specifically, the task to be processed has corresponding task attribute information, and the task attribute information includes, but is not limited to, a data type and a data amount corresponding to the task.
In this embodiment, in order to avoid the problem of low accuracy caused by manually predicting the resources, the scheduling system may automatically allocate the resources for the task to be processed. Thus, the scheduling system may first acquire the task to be processed.
Step 104, determining a matched resource allocation strategy according to the task attribute information, and determining the initial resources of the task to be processed according to the matched resource allocation strategy.
The resource allocation policy may be a corresponding resource allocation method set in advance based on different task attribute information. The initial resources are resources which are preliminarily determined to be allocated to the task to be processed according to the resource allocation strategy matched with the task attribute information based on the task attribute information of the task to be processed.
In this embodiment, the scheduling system may determine a matched resource allocation policy according to the task attribute information, and further determine an initial resource of the task to be processed according to the matched resource allocation policy.
And step 106, the initial resources are allocated to the task to be processed.
Specifically, the scheduling system may allocate the determined initial resources to the task to be processed, so that the task to be processed may run according to the initial resources.
In the resource allocation method, the task to be processed is acquired, the matched resource allocation strategy is determined according to the task attribute information, the initial resource of the task to be processed is determined according to the matched resource allocation strategy, and then the initial resource is allocated to the task to be processed, so that the task to be processed can operate according to the initial resource. Because the initial resources are allocated to the task to be processed according to the attribute information of the task to be processed, the accuracy of resource allocation can be improved, reasonable allocation of resources is realized, and the problem of inaccuracy of manually allocated resources is avoided.
In one embodiment, the task attribute information may include a data type and a data amount corresponding to the task to be processed, where the data type may include a service type of data (such as a debit card type, a credit card type, a scene type corresponding to each service scene, etc. of banking service), a technical type of data (such as a source skin layer, a detail data layer, a base data layer, a public model layer, etc.), and a computation complexity of data (such as the number and quantity of join, case white, in, group by, etc.), etc.
The resource allocation policy may include resources to be allocated corresponding to the data type, which is determined in advance according to the set data amount. For example, the resource allocation policy may include, for each million data amount, a credit card type in which the data type is the service type: the execution core 4 cores, the execution memory 10g, the execution 2 cores, the driver core 4 cores, the driver memory 12g and the driver 1.
The resource allocation policy may include, for each million data amounts, a debit card type in which the data type is the traffic type: the execution core 2 cores, the execution memory 5g, the execution 2 cores, the driver core 4 cores, the driver memory 6g and the driver 1.
The resource allocation strategy can include: the execution core 2 cores, the execution memory 4g, the execution 2 cores, the driver core 2 cores, the driver memory 4g and the driver 1.
The resource allocation policy in the source layer with data type as technology type and data volume as per million can include: the execution core 2 cores, the execution memory 4g, the execution 2 cores, the driver core 2 cores, the driver memory 12g and the driver 1.
The resource allocation strategy can include: the execution core 6 cores, the execution memory 12g, the execution 3 cores, the driver core 4 cores, the driver memory 10g and the driver 1 core.
The resource allocation policy may include, for each million pieces of data, an inner join in computational complexity by data type: the execution core 8 cores, the execution memory 16g, the execution 4 cores, the driver core 4 cores, the driver memory 12g and the driver 1.
The resource allocation strategy may include: the execution core 4 cores, the execution memory 8g, the execution 2 cores, the driver core 2 cores, the driver memory 6g and the driver 1.
The resource allocation strategy can include: the execution core 4 cores, the execution memory 6g, the execution 2 cores, the driver core 2 cores, the driver memory 6g and the driver 1 core.
Specifically, the corresponding resource allocation policy may be set in advance according to the set data amount and the different data types.
In one embodiment, as shown in fig. 2, in step 104, determining a matched resource allocation policy according to the task attribute information, and determining an initial resource of the task to be processed according to the matched resource allocation policy may specifically include:
step 202, identifying a target data type corresponding to the task to be processed and a target data amount under the target data type.
The target data type may be a specific data type corresponding to the task to be processed, and the target data amount may be a data amount corresponding to the target data type. Specifically, the scheduling system determines initial resources through subsequent steps by identifying a target data type corresponding to the task to be processed and a target data amount under the target data type.
Step 204, determining the resources to be allocated, which are matched with the target data types, and calculating the target resources to be allocated according to the matched resources to be allocated and the target data amount.
The resource to be allocated may be a resource to be allocated corresponding to the resource allocation policy. The target resource to be allocated is the resource allocated for the task to be processed, which is obtained through calculation. Specifically, the scheduling system may determine the matched resource allocation policy according to the identified target data type of the task to be processed, so as to determine the resource to be allocated, and further calculate the target resource to be allocated according to the target data amount and the resource to be allocated.
For example, if a task to be processed includes 500 ten thousand debit card types of data, it may be determined that the target data type corresponding to the task to be processed is a debit card type, and the corresponding target data amount is 500 ten thousand debit card types. The target resource allocation policy matched with the debit card type can be determined based on the preset resource allocation policy, so that the resource to be allocated corresponding to the policy is determined, for example, the resource to be allocated of each million pieces of debit card type data comprises: the execution core 2 cores, the execution memory 5g, the execution 2 cores, the driver core 4 cores, the driver memory 6g and the driver 1. Furthermore, the target resources to be allocated, that is, the executor core, may be calculated from 500 ten thousand target data amounts and the above resources to be allocated, that is, the executor memory is 2×5 cores, the executor memory is 5×5 cores, the executor memory is 2×5 cores, the driver core is 4×5 cores, the driver memory is 6×5 cores, and the driver is 1×5 cores.
And 206, taking the target resource to be allocated as an initial resource of the task to be processed.
Specifically, the scheduling system may take the determined target resource to be allocated as an initial resource of the task to be processed.
The method comprises the steps of identifying a target data type corresponding to a task to be processed and a target data amount under the target data type, determining resources to be allocated, which are matched with the target data type, calculating the resources to be allocated according to the matched resources to be allocated and the target data amount, and taking the resources to be allocated as initial resources of the task to be processed. In the embodiment, when the initial resource of the task to be processed is determined, the target data type corresponding to the task to be processed and the target data amount under the target data type are fully considered, so that the determined initial resource is more reasonable, and the waste of the resource and the underallocation of the resource can be avoided.
In one embodiment, the data type corresponding to the task to be processed includes a service type, a technology type and a computational complexity, and the resource allocation policy may further include a first weight corresponding to the service type, a second weight corresponding to the technology type and a third weight corresponding to the computational complexity. Then, as shown in fig. 3, in step 204, a resource to be allocated that matches the target data type is determined, and a target resource to be allocated is calculated according to the matched resource to be allocated and the target data amount, which may specifically further include:
step 302, determining a first resource to be allocated matched with the service type, a second resource to be allocated matched with the technology type and a third resource to be allocated matched with the calculation complexity according to the data quantity.
In this embodiment, the corresponding target resources to be allocated may be determined based on all dimensions of the data type of the task to be processed. Specifically, a resource allocation policy that each data type matches may be determined first, so as to determine a corresponding resource to be allocated. For example, a resource allocation policy matching a service type may be determined, and a corresponding first resource to be allocated may be determined according to an amount of data under the service type, a resource allocation policy matching a technology type may be determined, and a corresponding second resource to be allocated may be determined according to an amount of data under the technology type, and a resource allocation policy matching a computational complexity may be determined, and a corresponding third resource to be allocated may be determined according to an amount of data under the computational complexity type.
Step 304, calculating the target resource to be allocated according to the first resource to be allocated and the first weight, the second resource to be allocated and the second weight, and the third resource to be allocated and the third weight.
For example, if the first weight corresponding to the service type is 0.3, the second weight corresponding to the technology type is 0.3, and the third weight corresponding to the computation complexity is 0.4. If the service type corresponding to the task to be processed is a credit card type, the technology type is a source surface layer, the calculation complexity is inner join (internal connection), and if the data size is 100 ten thousand, the required target resource to be allocated can be calculated based on the resource allocation strategy corresponding to each data type:
exechamor core kernel number= (4×0.3) + (2×0.3) + (8×0.4) =5 kernels
The number of executor memory= (10×0.3) + (4×0.3) + (16×0.4) =10.6≡11g
Other computing methods of resources (such as executor and driver core) are the same as above, and detailed description thereof is omitted in this embodiment.
According to the method and the device, the target resources to be allocated are determined through calculation based on different dimensions and weights of the data types corresponding to the tasks to be processed and corresponding resource allocation strategies, and the accuracy of the target resources to be allocated can be further improved.
In one embodiment, the data types corresponding to the task to be processed include a service type, a technology type and a computational complexity, and the resource allocation policy may further include priority conditions of different data types. In step 204, the resource to be allocated that matches the target data type is determined, and the target resource to be allocated is calculated according to the matched resource to be allocated and the target data amount, which may specifically further include: determining a target priority condition corresponding to the target data type; and determining the resources to be allocated, which are matched with the target data type, according to the target priority condition, and calculating the target resources to be allocated according to the matched resources to be allocated and the target data quantity under the target data type. The target data type may be a specific data type corresponding to the task to be processed. The target priority condition is then the priority that it actually meets.
For example, the priority condition may be a priority of a particular data type. For example, the priority condition may be a condition that prioritizes the traffic type, reconsiders the technology type, and finally reconsiders the computational complexity. And if the task to be processed meets the service type specified in the priority, directly calculating the target resource to be allocated of the task to be processed according to the resource allocation strategy corresponding to the service type. If the task to be processed does not meet the service type specified in the priority, considering whether the task to be processed meets the technical type specified in the priority, and if so, calculating target resources to be allocated of the task to be processed according to a resource allocation strategy corresponding to the technical type. If the task to be processed does not meet the technical type specified in the priority, whether the task to be processed meets the calculation complexity type specified in the priority is considered, and if so, the target resource to be allocated of the task to be processed is calculated according to the resource allocation strategy corresponding to the calculation complexity type.
In one embodiment, the data types corresponding to the task to be processed may include a service type, a technology type, and a computational complexity. The resource allocation policy may also include priority conditions for different data types. Then, as shown in fig. 4, in step 204, a resource to be allocated that matches the target data type is determined, and a target resource to be allocated is calculated according to the matched resource to be allocated and the target data amount, which may specifically further include:
step 402, determining a target priority condition corresponding to the target data type.
The target data type may be a specific data type corresponding to the task to be processed. The target priority condition is then the priority that it actually meets. Specifically, the scheduling system may determine the target priority condition corresponding to the target data type based on the priority conditions of the different data types.
And step 404, determining the resources to be allocated matched with the target data type according to the target priority condition.
Specifically, a resource allocation policy that matches the target data type may be determined based on the target priority condition, and a corresponding resource to be allocated may be determined.
Step 406, determining the corresponding weight according to the target data amount under the target data type.
Wherein the weights may be determined based on the amount of data under each data type. I.e. the weight of the target data type is determined from the target data amount under the target data type.
In step 408, the target resource to be allocated is calculated according to the target data amount under the target data type, the resource to be allocated and the weight.
Specifically, based on the determined resource to be allocated, which is matched with the target data type, and the target data amount and the weight under the target data type, the target resource to be allocated can be calculated.
For example, there may be both credit card type data and debit card type data in a task to be processed, with the computational complexity of both inner joint and case wire. The data in the task to be processed can be determined to be respectively of which data type according to the priority condition, and then different subclasses in the corresponding data types are calculated according to different weights. For example, a task to be processed does not satisfy a given service type or a data technology type, but uses two kinds of calculation of inner join and left join, and the corresponding weights of the task to be processed can be respectively determined according to given data volumes of the inner join and the left join. For example, if the task to be processed includes 500 ten thousand pieces of data, wherein the inner joint has 300 ten thousand pieces of data and the left joint has 200 ten thousand pieces of data, the weight of the corresponding inner joint is 0.6, and the weight of the corresponding left joint is 0.4.
Based on the above description, the required target resources to be allocated can be calculated:
exechamor core number= (8×3) ×0.6+ (4×2) ×0.4=17.6≡18 core
exechamor memory number= (16×3) ×0.6+ (8×2) ×0.4=35.2≡35g
Other computing methods of resources (such as executor and driver core) are the same as above, and detailed description thereof is omitted in this embodiment.
In one scenario, resources may also be allocated randomly according to traffic type, technology type, and computational complexity type. If the first 100 ten thousand data in the task to be processed meets the credit card service type, calculating the corresponding first resource to be allocated according to the credit card service type; and the last 100 pieces of data are calculated to be corresponding to the second resources to be allocated according to the inner joint when the calculation complexity of the inner joint is used for the last 100 pieces of data in the tasks to be processed, and finally, the sum of the first resources to be allocated and the second resources to be allocated is used as the target resources to be allocated of the tasks to be processed.
In the above embodiment, the resource allocation policy is matched based on different modes, and the target resources to be allocated of the task to be processed are calculated, so that the flexibility of resource allocation is realized, the accuracy of resource allocation can be improved, the resource utilization rate is improved, and the resource waste is avoided.
In one embodiment, as shown in fig. 5, after the initial resources are allocated to the task to be processed in step 106, the method may further include:
step 502, monitoring the resource utilization rate in the running process of the task to be processed.
Where resource usage may be a ratio of consumed resources. Specifically, after the scheduling system allocates initial resources for the task to be processed, the task to be processed can run according to the initial resources, and in the process, the scheduling system can monitor the resource utilization rate in the running process of the task to be processed.
And step 504, when the resource utilization rate reaches a set threshold value, additional resources are allocated to the task to be processed.
The additional resource may be a resource that needs to be allocated to the task to be processed again, which is determined according to the initial resource and the set ratio. For example, each additional resource for a task to be processed may be 20% of the initial resource for the corresponding task to be processed. Specifically, when the monitored resource usage rate reaches the set threshold (e.g., 80%), it indicates that the initial resources allocated to the task to be processed may not be sufficient, so the scheduling system may further allocate additional resources to the task to be processed, i.e., dynamically adjust the resource allocation of the task to be processed, so that the task to be processed can be successfully executed.
It should be understood that, although the steps in the flowcharts related to the embodiments described above are sequentially shown as indicated by arrows, these steps are not necessarily sequentially performed in the order indicated by the arrows. The steps are not strictly limited to the order of execution unless explicitly recited herein, and the steps may be executed in other orders. Moreover, at least some of the steps in the flowcharts described in the above embodiments may include a plurality of steps or a plurality of stages, which are not necessarily performed at the same time, but may be performed at different times, and the order of the steps or stages is not necessarily performed sequentially, but may be performed alternately or alternately with at least some of the other steps or stages.
Based on the same inventive concept, the embodiment of the application also provides a resource allocation device for realizing the above-mentioned resource allocation method. The implementation of the solution provided by the apparatus is similar to the implementation described in the above method, so the specific limitation in the embodiments of one or more resource allocation apparatus provided below may refer to the limitation of the resource allocation method hereinabove, and will not be repeated here.
In one embodiment, as shown in fig. 6, there is provided a resource allocation apparatus, including: a task acquisition module 602, an initial resource determination module 604, and a resource allocation module 606, wherein:
the task obtaining module 602 is configured to obtain a task to be processed, where the task to be processed has corresponding task attribute information;
an initial resource determining module 604, configured to determine a matched resource allocation policy according to the task attribute information, and determine an initial resource of the task to be processed according to the matched resource allocation policy;
and the resource allocation module 606 is configured to allocate the initial resource to the task to be processed, so that the task to be processed runs according to the initial resource.
In one embodiment, the device further comprises a utilization rate monitoring module, configured to monitor a utilization rate of resources during the running process of the task to be processed; and when the resource utilization rate reaches a set threshold, allocating additional resources to the task to be processed, wherein the additional resources are determined according to the initial resources and the set ratio.
In one embodiment, the task attribute information includes a data type and a data amount corresponding to the task to be processed; the resource allocation strategy comprises resources to be allocated, which are determined in advance according to a set data amount and correspond to the data types; the resource allocation module is further configured to: identifying a target data type corresponding to the task to be processed and a target data amount under the target data type; determining the resources to be allocated, which are matched with the target data types, and calculating target resources to be allocated according to the matched resources to be allocated and the target data quantity; and taking the target resource to be allocated as the initial resource of the task to be processed.
In one embodiment, the data type includes a service type, a technology type, and a computational complexity, and the resource allocation policy further includes a first weight corresponding to the service type, a second weight corresponding to the technology type, and a third weight corresponding to the computational complexity; the resource allocation module is further configured to: determining a first resource to be allocated matched with the service type, a second resource to be allocated matched with the technical type and a third resource to be allocated matched with the computational complexity according to the data volume; and calculating the target resource to be allocated according to the first resource to be allocated, the first weight, the second resource to be allocated, the second weight, the third resource to be allocated and the third weight.
In one embodiment, the data types include a traffic type, a technology type, and a computational complexity, and the resource allocation policy further includes priority conditions for different data types; the resource allocation module is further configured to: determining a target priority condition corresponding to the target data type; and determining the resources to be allocated which are matched with the target data type according to the target priority condition, and calculating the target resources to be allocated according to the matched resources to be allocated and the target data volume under the target data type.
In one embodiment, the data types include a traffic type, a technology type, and a computational complexity, and the resource allocation policy further includes priority conditions for different data types; the resource allocation module is further configured to: determining a target priority condition corresponding to the target data type; determining resources to be allocated, which are matched with the target data type, according to the target priority condition; determining corresponding weights according to the target data quantity under the target data type; and calculating the target resource to be allocated according to the target data quantity under the target data type, the resource to be allocated and the weight.
The respective modules in the above-described resource allocation apparatus may be implemented in whole or in part by software, hardware, and combinations thereof. The above modules may be embedded in hardware or may be independent of a processor in the computer device, or may be stored in software in a memory in the computer device, so that the processor may call and execute operations corresponding to the above modules.
In one embodiment, a computer device is provided, which may be a server, the internal structure of which may be as shown in fig. 7. The computer device includes a processor, a memory, an Input/Output interface (I/O) and a communication interface. The processor, the memory and the input/output interface are connected through a system bus, and the communication interface is connected to the system bus through the input/output interface. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The database of the computer device is for storing data related to the resource allocation policy. The input/output interface of the computer device is used to exchange information between the processor and the external device. The communication interface of the computer device is used for communicating with an external terminal through a network connection. The computer program is executed by a processor to implement a resource allocation method.
It will be appreciated by those skilled in the art that the structure shown in fig. 7 is merely a block diagram of some of the structures associated with the present application and is not limiting of the computer device to which the present application may be applied, and that a particular computer device may include more or fewer components than shown, or may combine certain components, or have a different arrangement of components.
In one embodiment, a computer device is provided comprising a memory having a computer program stored therein and a processor that implements the steps of the method described above when the computer program is executed.
In one embodiment, a computer readable storage medium is provided having a computer program stored thereon, which when executed by a processor, implements the steps of the above method.
In an embodiment, a computer program product is provided comprising a computer program which, when executed by a processor, implements the steps of the above method.
It should be noted that, the user information (including, but not limited to, user equipment information, user personal information, etc.) and the data (including, but not limited to, data for analysis, stored data, presented data, etc.) referred to in the present application are information and data authorized by the user or sufficiently authorized by each party, and the collection, use and processing of the related data are required to comply with the related laws and regulations and standards of the related countries and regions.
Those skilled in the art will appreciate that implementing all or part of the above described methods may be accomplished by way of a computer program stored on a non-transitory computer readable storage medium, which when executed, may comprise the steps of the embodiments of the methods described above. Any reference to memory, database, or other medium used in the various embodiments provided herein may include at least one of non-volatile and volatile memory. The nonvolatile Memory may include Read-Only Memory (ROM), magnetic tape, floppy disk, flash Memory, optical Memory, high density embedded nonvolatile Memory, resistive random access Memory (ReRAM), magnetic random access Memory (Magnetoresistive Random Access Memory, MRAM), ferroelectric Memory (Ferroelectric Random Access Memory, FRAM), phase change Memory (Phase Change Memory, PCM), graphene Memory, and the like. Volatile memory can include random access memory (Random Access Memory, RAM) or external cache memory, and the like. By way of illustration, and not limitation, RAM can be in the form of a variety of forms, such as static random access memory (Static Random Access Memory, SRAM) or dynamic random access memory (Dynamic Random Access Memory, DRAM), and the like. The databases referred to in the various embodiments provided herein may include at least one of relational databases and non-relational databases. The non-relational database may include, but is not limited to, a blockchain-based distributed database, and the like. The processors referred to in the embodiments provided herein may be general purpose processors, central processing units, graphics processors, digital signal processors, programmable logic units, quantum computing-based data processing logic units, etc., without being limited thereto.
The technical features of the above embodiments may be arbitrarily combined, and all possible combinations of the technical features in the above embodiments are not described for brevity of description, however, as long as there is no contradiction between the combinations of the technical features, they should be considered as the scope of the description.
The above examples only represent a few embodiments of the present application, which are described in more detail and are not to be construed as limiting the scope of the present application. It should be noted that it would be apparent to those skilled in the art that various modifications and improvements could be made without departing from the spirit of the present application, which would be within the scope of the present application. Accordingly, the scope of protection of the present application shall be subject to the appended claims.
Claims (10)
1. A method of resource allocation, the method comprising:
acquiring a task to be processed, wherein the task to be processed has corresponding task attribute information;
determining a matched resource allocation strategy according to the task attribute information, and determining initial resources of the task to be processed according to the matched resource allocation strategy;
and distributing the initial resources to the task to be processed so that the task to be processed runs according to the initial resources.
2. The method of claim 1, wherein after the assigning of the initial resources to the task to be processed, the method further comprises:
monitoring the resource utilization rate in the running process of the task to be processed;
and when the resource utilization rate reaches a set threshold, allocating additional resources to the task to be processed, wherein the additional resources are determined according to the initial resources and the set ratio.
3. The method according to claim 1, wherein the task attribute information includes a data type and a data amount corresponding to the task to be processed; the resource allocation strategy comprises resources to be allocated, which are determined in advance according to a set data amount and correspond to the data types;
the determining a matched resource allocation strategy according to the task attribute information, and determining the initial resource of the task to be processed according to the matched resource allocation strategy comprises the following steps:
identifying a target data type corresponding to the task to be processed and a target data amount under the target data type;
determining the resources to be allocated, which are matched with the target data types, and calculating target resources to be allocated according to the matched resources to be allocated and the target data quantity;
and taking the target resource to be allocated as the initial resource of the task to be processed.
4. The method of claim 3, wherein the data type comprises a traffic type, a technology type, and a computational complexity, and wherein the resource allocation policy further comprises a first weight corresponding to the traffic type, a second weight corresponding to the technology type, and a third weight corresponding to the computational complexity;
the determining the resource to be allocated matched with the target data type, and calculating the target resource to be allocated according to the matched resource to be allocated and the target data amount, including:
determining a first resource to be allocated matched with the service type, a second resource to be allocated matched with the technical type and a third resource to be allocated matched with the computational complexity according to the data volume;
and calculating the target resource to be allocated according to the first resource to be allocated, the first weight, the second resource to be allocated, the second weight, the third resource to be allocated and the third weight.
5. The method of claim 3, wherein the data types include traffic types, technology types, and computational complexity, and wherein the resource allocation policy further includes priority conditions for different ones of the data types;
the determining the resource to be allocated matched with the target data type, and calculating the target resource to be allocated according to the matched resource to be allocated and the target data amount, including:
determining a target priority condition corresponding to the target data type;
and determining the resources to be allocated which are matched with the target data type according to the target priority condition, and calculating the target resources to be allocated according to the matched resources to be allocated and the target data volume under the target data type.
6. The method of claim 3, wherein the data types include traffic types, technology types, and computational complexity, and wherein the resource allocation policy further includes priority conditions for different ones of the data types;
the determining the resource to be allocated matched with the target data type, and calculating the target resource to be allocated according to the matched resource to be allocated and the target data amount, including:
determining a target priority condition corresponding to the target data type;
determining resources to be allocated, which are matched with the target data type, according to the target priority condition;
determining corresponding weights according to the target data quantity under the target data type;
and calculating the target resource to be allocated according to the target data quantity under the target data type, the resource to be allocated and the weight.
7. A resource allocation apparatus, the apparatus comprising:
the task acquisition module is used for acquiring a task to be processed, wherein the task to be processed has corresponding task attribute information;
the initial resource determining module is used for determining a matched resource allocation strategy according to the task attribute information and determining initial resources of the task to be processed according to the matched resource allocation strategy;
and the resource allocation module is used for allocating the initial resources to the task to be processed so that the task to be processed runs according to the initial resources.
8. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that the processor implements the steps of the method of any of claims 1 to 6 when the computer program is executed.
9. A computer readable storage medium, on which a computer program is stored, characterized in that the computer program, when being executed by a processor, implements the steps of the method of any of claims 1 to 6.
10. A computer program product comprising a computer program, characterized in that the computer program, when being executed by a processor, implements the steps of the method of any of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311108136.XA CN117389715A (en) | 2023-08-30 | 2023-08-30 | Resource allocation method, resource allocation device, computer device, and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311108136.XA CN117389715A (en) | 2023-08-30 | 2023-08-30 | Resource allocation method, resource allocation device, computer device, and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117389715A true CN117389715A (en) | 2024-01-12 |
Family
ID=89463810
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311108136.XA Pending CN117389715A (en) | 2023-08-30 | 2023-08-30 | Resource allocation method, resource allocation device, computer device, and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117389715A (en) |
-
2023
- 2023-08-30 CN CN202311108136.XA patent/CN117389715A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108492005B (en) | Project data processing method and device, computer equipment and storage medium | |
| CN109189572B (en) | Resource estimation method and system, electronic equipment and storage medium | |
| CN111352736A (en) | Method and device for scheduling big data resources, server and storage medium | |
| CN116719646A (en) | Hot spot data processing method, device, electronic device and storage medium | |
| US11954419B2 (en) | Dynamic allocation of computing resources for electronic design automation operations | |
| CN116260876A (en) | AI application scheduling method and device based on K8s and electronic equipment | |
| CN112860387A (en) | Distributed task scheduling method and device, computer equipment and storage medium | |
| CN116578410A (en) | Resource management method, device, computer equipment and storage medium | |
| CN112286623A (en) | Information processing method and device and storage medium | |
| CN117389715A (en) | Resource allocation method, resource allocation device, computer device, and storage medium | |
| CN107391262B (en) | Job scheduling method and device | |
| CN111475277A (en) | Resource allocation method, system, equipment and machine readable storage medium | |
| CN113742059B (en) | Task allocation method, device, computer equipment and storage medium | |
| CN111796934B (en) | Task issuing method and device, storage medium and electronic equipment | |
| CN114493598A (en) | Computing resource management method, device, computer equipment and storage medium | |
| CN113504988A (en) | Control method, system, medium and device for system computing power resource scheduling | |
| CN116032928B (en) | Data collaborative computing method, device, system, electronic device and storage medium | |
| CN117314067A (en) | Work order distribution method, apparatus, device, storage medium and program product | |
| CN111290850B (en) | Data storage method, device and equipment | |
| CN115951988B (en) | Job scheduling method, computing equipment and storage medium | |
| CN115391042B (en) | Resource allocation method and device, electronic equipment and storage medium | |
| CN115543568A (en) | Timed task processing method and device, computer equipment and storage medium | |
| CN116401150A (en) | Method and device for testing real-time operating system, computer equipment and storage medium | |
| CN117851071A (en) | Application resource allocation method and device, computer equipment and storage medium | |
| CN116881003A (en) | Resource allocation method, device, service equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |