CN115391047A - Resource scheduling method and device - Google Patents
Resource scheduling method and device Download PDFInfo
- Publication number
- CN115391047A CN115391047A CN202211137218.2A CN202211137218A CN115391047A CN 115391047 A CN115391047 A CN 115391047A CN 202211137218 A CN202211137218 A CN 202211137218A CN 115391047 A CN115391047 A CN 115391047A
- Authority
- CN
- China
- Prior art keywords
- resources
- jobs
- job
- resource
- optimal
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/48—Program initiating; Program switching, e.g. by interrupt
- G06F9/4806—Task transfer initiation or dispatching
- G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
- G06F9/4881—Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5011—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resources being hardware resources other than CPUs, Servers and Terminals
- G06F9/5016—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resources being hardware resources other than CPUs, Servers and Terminals the resource being the memory
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
- G06F9/5038—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals considering the execution order of a plurality of tasks, e.g. taking priority or time dependency constraints into consideration
Abstract
The invention discloses a resource scheduling method and a device, relating to a distributed mode, wherein the method comprises the following steps: determining the requirements of different types of jobs on various resources, wherein the requirements of the jobs of various types on the various resources are obtained by analyzing a sample data set constructed according to the quantity of the various resources consumed by processing the jobs of different types by each working node in a service system, and the service system is a distributed service system for realizing batch job distribution through an intelligent batch scheduling mechanism; under the total number of various resources which can be provided by a service system, constructing an optimal resource allocation problem of service system profit maximization based on a multidimensional bounded knapsack algorithm, and obtaining an optimal operation allocation strategy by solving the optimal resource allocation problem; and scheduling the resources of the service system according to the optimal operation allocation strategy. The invention can maximize the utilization rate of various resources of the system, thereby improving the processing efficiency of batch operation and finally improving the performance of the whole distributed system.
Description
Technical Field
The present invention relates to the field of distributed technologies, and in particular, to a resource scheduling method and apparatus.
Background
In the distributed unitized financial service system, the reasonable distribution of batch jobs is realized through an intelligent batch scheduling mechanism, so that the utilization rate of various resources of the financial service system is maximized, and the problems of low utilization rate of the resources of the financial service system and low job processing efficiency in the traditional job distribution mode are solved.
The existing financial business system mostly adopts a random job distribution mode, or performs job scheduling around single resources according to various single rules such as client regions or number sections.
The defects of the prior art are as follows: the resource is unevenly distributed and the utilization rate is low.
Disclosure of Invention
The embodiment of the invention provides a resource scheduling method, which is used for solving the problems of uneven resource distribution and low utilization rate and comprises the following steps:
determining the requirements of different types of jobs on various resources, wherein the requirements of the jobs of various types on the various resources are obtained by analyzing a sample data set constructed according to the quantity of the various resources consumed by processing the jobs of different types by each working node in a service system, and the service system is a distributed service system for realizing batch job distribution through an intelligent batch scheduling mechanism;
under the total number of various resources which can be provided by a service system, constructing an optimal resource allocation problem of service system profit maximization based on a multidimensional bounded knapsack algorithm, and obtaining an optimal operation allocation strategy by solving the optimal resource allocation problem;
and scheduling the resources of the service system according to the optimal operation allocation strategy.
The embodiment of the present invention further provides a resource scheduling apparatus, which is used to solve the problems of uneven resource allocation and low utilization rate, and the apparatus includes:
the system comprises a demand module, a service system and a management module, wherein the demand module is used for determining the demands of different types of jobs on various resources, the demands of the jobs of various types on the various resources are obtained by analyzing a sample data set constructed according to the quantity of the various resources consumed by processing the jobs of different types by each working node in the service system, and the service system is a distributed service system for realizing batch job distribution through an intelligent batch scheduling mechanism;
the strategy module is used for constructing a resource optimal allocation problem of business system profit maximization based on a multidimensional bounded knapsack algorithm under the total number of various resources which can be provided by a business system, and obtaining an optimal operation allocation strategy by solving the resource optimal allocation problem;
and the scheduling module is used for scheduling the resources of the service system according to the optimal operation allocation strategy.
The embodiment of the present invention further provides a computer device, which includes a memory, a processor, and a computer program stored in the memory and capable of running on the processor, and when the processor executes the computer program, the resource scheduling method is implemented.
An embodiment of the present invention further provides a computer-readable storage medium, where a computer program is stored, and when the computer program is executed by a processor, the method for scheduling resources is implemented.
An embodiment of the present invention further provides a computer program product, where the computer program product includes a computer program, and when the computer program is executed by a processor, the method for scheduling resources is implemented.
In the embodiment of the invention, compared with the technical scheme of random operation distribution or distribution around various single rules of single resources in the prior art, the method comprises the steps of firstly determining the types of the operations, then analyzing the resources consumed by various operations, taking the resources as the basis, scheduling the resources according to a multidimensional bounded knapsack algorithm, and solving the optimal distribution of the resources due to the multidimensional bounded knapsack problem, so that the utilization rate of various resources of the system can be maximized, the batch operation processing efficiency is improved, and the performance of the whole distributed system is finally improved;
furthermore, when the resources are scheduled, scheduling is performed in sequence according to the types of the jobs, and based on the current jobs to be processed and the total number of the resources, the problem of optimal allocation of the resources is solved through iteration until all types of jobs are processed, so that the processing efficiency of batch jobs can be further improved, and the performance of the whole distributed system is improved.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts. In the drawings:
FIG. 1 is a schematic flow chart illustrating a resource scheduling method according to an embodiment of the present invention;
FIG. 2 is a diagram illustrating a batch scheduling system according to an embodiment of the present invention;
FIG. 3 is a schematic diagram illustrating an implementation flow of resource scheduling in an embodiment of the present invention;
FIG. 4 is a schematic diagram of a resource scheduling apparatus according to an embodiment of the present invention;
FIG. 5 is a schematic diagram of a computer device according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention more apparent, the embodiments of the present invention are further described in detail below with reference to the accompanying drawings. The exemplary embodiments and descriptions of the present invention are provided to explain the present invention, but not to limit the present invention.
The inventor notices in the process of invention that:
in the distributed unitized financial service system, the reasonable distribution of batch jobs is realized through an intelligent batch scheduling mechanism, so that the utilization rate of various resources of the financial service system is maximized, and the problems of low utilization rate of the resources of the financial service system and low job processing efficiency in the traditional job distribution mode are solved. The existing financial service system mostly adopts a random job distribution mode, or performs job scheduling around single resources according to various single rules such as client regions or number sections, so that the total resource distribution of the whole distributed financial service system is uneven, and the utilization rate is low.
Based on the above, a scheme is needed, starting from the multi-dimensional total resource amount of the distributed system, the batch scheduling module performs optimal segmentation on batch jobs according to the demand of various jobs on various resources, so that the throughput benefit of the system is maximized, the resource utilization rate of the system is maximized, and the overall job processing performance of the system is improved.
The following description will be given with reference to examples.
Fig. 1 is a schematic flowchart of an implementation of a resource scheduling method, as shown in fig. 1, the method may include:
step 101, determining the requirements of different types of jobs on various resources, wherein the requirements of the jobs of various types on the various resources are obtained by analyzing a sample data set constructed according to the quantity of the various resources consumed by processing the jobs of different types by each working node in a service system, and the service system is a distributed service system for realizing batch job distribution through an intelligent batch scheduling mechanism;
102, under the total number of various resources which can be provided by a service system, constructing an optimal resource allocation problem of service system profit maximization based on a multidimensional bounded knapsack algorithm, and obtaining an optimal operation allocation strategy by solving the optimal resource allocation problem;
and 103, scheduling the resources of the service system according to the optimal operation allocation strategy.
In the scheme, for the multi-dimensional bounded knapsack problem, the knapsack problem is a combined optimization problem. Given a set of items, each item has a corresponding weight and value. A strategy is given to maximize the total value of items loaded into the backpack, within the total weight defined by the backpack. When the group of articles has multiple attributes, a multidimensional bounded knapsack problem is formed.
When the method is implemented, sample data sets of various resource consumption conditions of different types of jobs need to be constructed; analyzing the resource demand of any type of unit-size operation; constructing an optimal scheduling problem model based on a multidimensional bounded knapsack algorithm, and solving; and (5) iteratively analyzing the scheduling strategy until all the jobs are scheduled.
Specifically, firstly, a sample data set is constructed for subsequent analysis by recording the quantity of various resources consumed by all batch working nodes in the distributed system to process different types of batch jobs.
And then analyzing the sample data set, and calculating the demand of different types of jobs with unit size on various resources.
And then, a system profit maximization model is constructed based on all the current to-be-processed jobs and the total number of various resources of the distributed system, and the problem is solved to obtain an optimal job scheduling strategy, so that the utilization of system resources is maximized.
And finally, scheduling the jobs according to the solved optimal allocation strategy, removing the scheduled jobs from the job set to be processed, then performing optimal problem solving again based on the current job set to be processed and the total amount of system resources, and repeating the steps until all jobs are processed.
The following more specifically describes the present invention.
1. Analyzing the resource consumption conditions of various operations processed by the nodes in batch, and constructing a sample data set.
And classifying the batch jobs according to the text content of the batch jobs, recording system resource allocation when each batch work node in the distributed system processes different types of jobs, such as the number of distributed CPUs (central processing units), memory capacity, disk capacity and the like, and recording the text size for processing the types of jobs.
2. And analyzing the sample data set to acquire the quantity of resources required by processing each type of batch operation.
And analyzing the sample data set information, and calculating a weighted average value of the quantity of the resources required by any type of operation to obtain the quantity of the resources required by the type of operation. Assuming that all batch working nodes in the distributed system process nB-sized A-type jobs together within a certain time threshold value and consume mB memory together, the demand of unit-sized A-type jobs on memory resources is m/n.
3. And constructing a multi-dimensional resource allocation problem model based on a multi-dimensional bounded knapsack problem algorithm, and analyzing an optimal resource allocation strategy.
In implementation, business system revenue maximization is determined as follows:
the system throughput benefit Pk of job k versus resource demand is inversely proportional to the average resource consumption of job k:
wherein, P k System throughput gain, W, representing Job k versus resource demand i，k Representing the demand of job k for resources of type i, c i Representing the number of i-type resources, and m represents the resource type;
the throughput gain maximization of the distributed system is expressed as:
a k ≤j k ，k＝1，......，n，
wherein, P k System throughput gain, a, representing job k versus resource demand k Indicates the number of jobs k, W i，k Representing the demand of job k on resources of type i, j k Representing the number of k types of batch jobs, m representing the total number of resource categories, n representing the total number of batch job categories, a k Is an integer.
Specifically, from the resource allocation perspective, various resource demands are analyzed around the operation, the intelligent allocation of the batch operation is realized by maximizing the total resource utilization rate of all the working nodes in the distributed system, and the optimal problem is limited by the quantity of various resources, namely, the optimal problem is similar to the multidimensional bounded knapsack problem.
Assume that the bulk job resource requirement is R (R) 1 ，r 2 ，r i ，…，r m ) Where m represents the kind of resource required, r i Representing the amount of demand of the batch job for the i-type resource. The quantity of resources provided by all the batch working nodes in the distributed system is C (C) 1 ，c 2 ，c i ，…，c m ) Wherein c is m Indicating the number of m classes of resources. The set of batch jobs to be processed is J (J) 1 ，j 2 ，j i ，…，j n ) Wherein n represents a batchTotal number of categories j of measurement work i Denotes the number of i-type batch jobs, and the demand of job k for resources is denoted as Wq (W) 1k ，W 2k ，W ik ，…，W mk )，W ik Representing the demand of job k for resources of type i.
Using a k The number of jobs k is indicated, and a resource allocation policy A (a) is finally obtained by analyzing the resource allocation problem 1 ，a 2 ，a i ，…，a n ) Wherein a is i Indicates to call a i A j i And (6) operating.
System throughput gain P using average resource consumption for a given job K as a measure of job K versus resource demand k Namely, the following formula (1):
the distributed system throughput yield optimization problem is expressed by the following equation (2):
a k ≤j k ，k＝1，......，n
wherein a is k Is an integer. The optimization problem is solved using a depth first search method.
4. And performing job scheduling based on the solution set of the multidimensional resource allocation problem model.
In an implementation, the method may further include:
after determining a queue formed by various types of jobs to be scheduled, determining the requirements of different types of jobs in the queue on various types of resources;
selecting a category of operation in the queue, constructing an optimal resource allocation problem of maximizing the profit of the service system based on a multidimensional bounded knapsack algorithm under the condition of total number of various resources provided by the service system, and obtaining an optimal operation allocation strategy of the category of operation by solving the optimal resource allocation problem;
for the operation of the category, after scheduling the resources of the service system according to the optimal operation allocation strategy, moving the operation of the category out of the queue;
and after scheduling the jobs of each category in the queue, removing the queue until no job to be scheduled exists in the queue.
Specifically, an optimal job scheduling strategy A1 is obtained by solving a formula (2), the batch scheduling module performs job allocation according to A1, removes the jobs with the corresponding number of categories contained in A1 from the job set to be processed, performs optimal scheduling strategy analysis again based on the remaining job set, and repeats the steps until all job scheduling is completed, namely, the batch job task is completed.
The following is an example.
Firstly, a sample data set is constructed, and the demand of various operations on various resources of the system is recorded. And analyzing the resource demand of batch operation of a specific category of unit size according to the sample data set, and storing according to the category. And constructing a multi-dimensional resource optimal distribution problem based on a multi-dimensional bounded knapsack algorithm, and solving the problem to obtain an optimal operation distribution strategy. And scheduling the jobs according to the calculated optimal allocation strategy, updating the number of jobs to be processed and the total number of resources, reconstructing and solving a problem model, and repeating the steps until all jobs are processed.
FIG. 2 is a schematic diagram of a batch scheduling system architecture, and as shown in FIG. 2, at least one functional architecture that can implement resource scheduling may include:
1. the operation resource demand analysis module: and analyzing the sample data set within the specified time threshold, and calculating the demand of unit-size batch operation of any category on various resources.
2. The operation resource demand storage module: and storing the job resource demand amount calculated by the job resource demand analysis module into a Map (mapping) data structure according to the job type.
3. A to-be-processed job storage module: analyzing the category of the operation to be processed, acquiring the quantity of the resources required by the operation of the type of unit size from the operation resource requirement storage module according to the category of the operation, calculating according to the actual size of the current operation, and storing the mapping relation between the operation to be processed and the quantity of the corresponding required resources for the scheduling strategy calculation module to use.
4. A scheduling policy calculation module: a multidimensional bounded knapsack algorithm problem model is constructed based on the total quantity of various resources of the current system and the demand of the operation to be processed on the resources, and then the problem is solved by using a depth search priority method to obtain an optimal resource allocation strategy.
5. The job scheduling module: and performing job scheduling according to the optimal allocation strategy calculated by the scheduling strategy calculation module. And feeds back the data to the job storage module to be processed, and removes the distributed jobs from the module.
6. A system resource statistics module: and counting the number of various types of resources currently available in the distributed system.
Fig. 3 is a schematic diagram illustrating an implementation flow of resource scheduling, and as shown in fig. 3, according to a distributed batch job scheduling scheme based on a multidimensional bounded back-pack problem, at least the batch scheduling system shown in fig. 2 can be implemented as follows:
and 301, constructing a sample data set, and recording the resource consumption condition when all batch working nodes in the distributed system process various batch jobs.
And classifying the jobs according to the text content of the batch jobs, and recording resources such as a CPU (Central processing Unit), a memory, a disk and the like consumed when each batch work node of the distributed system processes the jobs with a specific size and a specific type in real time from the viewpoint of job type.
And step 302, analyzing the demand of batch jobs of any category on various resources according to the sample data set, and storing according to categories.
In implementation, the demands of different types of jobs on various types of resources are stored in a map structure.
The Object (Object) of avaScript is essentially a collection of key-value pairs (Hash structure), but can only use strings as keys. This imposes a great limitation on its use. To address this issue, a Map data structure is provided. It is similar to an object and is also a set of key-value pairs, but the scope of "key" is not limited to character strings, and various types of values (including objects) can be regarded as keys. That is, the Object structure provides a "string-value" correspondence, and the Map structure provides a "value-value" correspondence, which is a more sophisticated implementation of the Hash structure.
Specifically, a sample data set within a specified time threshold is analyzed, and the consumption of various resources in the system by a unit size (Byte is taken as a basic unit of the job size) job within the time is calculated, namely the consumption represents the resource demand of the job. The resource requirement of various types of operation stored by using map structure, for example, unit size A type operation, requires 3 types of resources, and the quantity of each type of resource is r 1 ，r 2 ，r 3 Then store as map (A, { r) 1 ，r 2 ，r 3 }), when processing jobs of size 100B, the resources needed are {100r1, 100r2, 100r3}.
And 303, constructing a multidimensional bounded knapsack algorithm model, and analyzing an allocation strategy with a maximized resource utilization rate.
Under the limitation of limited system resources, the maximization of system benefits is realized through intelligent operation allocation, and a multidimensional bounded knapsack algorithm model is built accordingly. System throughput benefit P of Job k relative to resource demand k Inversely proportional to the average resource consumption of job k, i.e.:
wherein, W ik Representing the demand of job k for resources of type i, c i Indicating the number of i-type resources and m representing the resource type. The throughput benefit maximization of the distributed system is expressed as:
a k ≤j k ，k＝1，......，n
wherein, P k System throughput gain, a, representing job k versus resource demand k Indicates the number of jobs k, W ik Representing the amount of demand of job k on resources of type i, j k Representing the number of k types of batch jobs, m representing the total number of resource categories, n representing the total number of batch job categories, a k Are integers.
In implementation, the optimal job allocation strategy is solved by utilizing a depth-first search method according to the throughput gain maximization.
Specifically, the above maximization problem may be solved by using a depth-first search method, so as to obtain the optimal resource allocation policy a.
And step 304, repeatedly constructing and solving an optimization problem based on the existing resource iterative analysis, and scheduling resources based on the solution set.
And (3) obtaining a scheduling strategy A through the analysis of the optimal problem in the step 303, then performing job allocation by the batch scheduling module according to the strategy, removing the allocated jobs from the job set, modeling and solving the optimal problem again based on the existing total number of resources and job conditions, and repeating the operation until all jobs are scheduled.
The embodiment of the invention also provides a resource scheduling device, which is described in the following embodiment. Because the principle of the device for solving the problems is similar to the resource scheduling method, the implementation of the device can refer to the implementation of the resource scheduling method, and repeated details are not repeated.
Fig. 4 is a schematic structural diagram of a resource scheduling apparatus, as shown in fig. 4, the resource scheduling apparatus may include:
the demand module 401 is configured to determine demands of different types of jobs on various resources, where the demands of the various types of jobs on the various resources are obtained by analyzing a sample data set constructed according to the number of various resources consumed by each work node in a service system when processing the different types of jobs, and the service system is a distributed service system that implements batch job allocation through an intelligent batch scheduling mechanism;
a strategy module 402, configured to construct an optimal resource allocation problem of maximizing revenue of the service system based on a multidimensional bounded knapsack algorithm in the total number of various resources that can be provided by the service system, and obtain an optimal operation allocation strategy by solving the optimal resource allocation problem;
and a scheduling module 403, configured to schedule resources of the service system according to the optimal job allocation policy.
In implementation, the demand module is further configured to determine the demands of different types of jobs in a queue on various types of resources after determining the queue formed by various types of jobs to be scheduled each time;
the strategy module is further used for selecting a category of operation in the queue, constructing an optimal resource allocation problem with maximized business system income based on a multidimensional bounded knapsack algorithm under the total number of various resources provided by the business system, and obtaining an optimal operation allocation strategy of the category of operation by solving the optimal resource allocation problem;
the scheduling module is further used for moving the jobs of the category out of the queue after scheduling the resources of the business system according to the optimal job allocation strategy aiming at the jobs of the category;
the scheduling module is further used for removing the queue after scheduling the jobs of each category in the queue until no job to be scheduled exists in the queue.
In implementation, the demand module is further configured to store demands of different types of jobs for various types of resources in a map structure.
In implementation, the policy module is further configured to determine that business system revenue is maximized in the following manner:
system throughput benefit P for Job k versus resource demand k Inversely proportional to the average resource consumption of job k:
wherein, P k System throughput gain, W, representing job k versus resource demand i，k Representing the demand of job k for resources of type i, c i Representing the number of i-type resources, and m represents the resource type;
the throughput gain maximization of the distributed system is expressed as:
a k ≤j k ，k＝1，......，n，
wherein, P k System throughput gain, a, representing job k versus resource demand k Indicates the number of jobs k, W i，k Representing the demand of job k on resources of type i, j k Representing the number of k types of batch jobs, m representing the total number of resource categories, n representing the total number of batch job categories, a k Are integers.
In implementation, the strategy module is further configured to solve the optimal job allocation strategy by utilizing a depth-first search method according to the throughput gain maximization.
When the technical scheme provided by the embodiment of the invention is implemented, the implementation can be carried out as follows.
Fig. 5 is a schematic diagram of a computer device, as shown in fig. 5, the computer device includes:
the processor 500, which is used to read the program in the memory 520, executes the following processes:
determining the requirements of different types of jobs on various resources, wherein the requirements of the jobs of various types on the various resources are obtained by analyzing a sample data set constructed according to the quantity of the various resources consumed by processing the jobs of different types by each working node in a service system, and the service system is a distributed service system for realizing batch job distribution through an intelligent batch scheduling mechanism;
under the total number of various resources which can be provided by a service system, constructing an optimal resource allocation problem of service system profit maximization based on a multidimensional bounded knapsack algorithm, and obtaining an optimal operation allocation strategy by solving the optimal resource allocation problem;
scheduling resources of the service system according to the optimal operation allocation strategy;
a transceiver 510 for receiving and transmitting data under the control of the processor 500.
In an implementation, the method further comprises the following steps:
after determining a queue formed by various types of jobs to be scheduled, determining the requirements of different types of jobs in the queue on various types of resources;
selecting a category of operation in the queue, constructing an optimal resource allocation problem of maximizing the income of the service system based on a multidimensional bounded knapsack algorithm under the total number of various resources which can be provided by the service system, and obtaining an optimal operation allocation strategy of the category of operation by solving the optimal resource allocation problem;
for the operation of the category, after scheduling the resources of the service system according to the optimal operation allocation strategy, moving the operation of the category out of the queue;
and removing the queue after scheduling the jobs of each category in the queue until no job to be scheduled exists in the queue.
In implementation, the demands of different classes of jobs on various types of resources are stored in a map structure.
In implementation, business system revenue maximization is determined as follows:
system throughput benefit P of Job k relative to resource demand k Inversely proportional to the average resource consumption of job k:
wherein, P k Representing Job k relative resource needsCalculated system throughput gain, W i，k Representing the amount of demand of job k on resources of type i, c i Representing the number of i-type resources, and m represents the resource type;
the throughput gain maximization of the distributed system is expressed as:
a k ≤j k ，k＝1，......，n，
wherein, P k System throughput gain representing Job k versus resource demand, a k Indicates the number of jobs k, W i，k Representing the demand of job k on resources of type i, j k Representing the number of k types of batch jobs, m representing the total number of resource types, n representing the total number of batch job types, a k Is an integer.
In implementation, the optimal job allocation strategy is solved by utilizing a depth-first search method according to the throughput gain maximization.
Wherein in fig. 5, the bus architecture may include any number of interconnected buses and bridges, with one or more processors, represented by processor 500, and various circuits, represented by memory 520, being linked together. The bus architecture may also link together various other circuits such as peripherals, voltage regulators, power management circuits, and the like, which are well known in the art, and therefore, will not be described any further herein. The bus interface provides an interface. The transceiver 510 may be a number of elements including a transmitter and a receiver that provide a means for communicating with various other apparatus over a transmission medium. The processor 500 is responsible for managing the bus architecture and general processing, and the memory 520 may store data used by the processor 500 in performing operations.
An embodiment of the present invention further provides a computer-readable storage medium, where a computer program is stored, and when the computer program is executed by a processor, the method for scheduling resources is implemented.
An embodiment of the present invention further provides a computer program product, where the computer program product includes a computer program, and when the computer program is executed by a processor, the method for scheduling resources is implemented.
According to the technical scheme provided by the embodiment of the invention, resources required by various batch jobs are recorded, the demand of any type of batch jobs on the resources in unit size is analyzed and calculated, the throughput gain of the jobs is measured by utilizing the inverse ratio of the average resource consumption of the jobs, the resource allocation strategy of maximizing the system gain is solved, and then the optimal resource allocation problem is iteratively solved based on the current jobs to be processed and the total number of the resources until all jobs are processed.
In the scheme, the batch job scheduling of the distributed unitized financial service system is intelligently scheduled by the scheduling node according to the resource demand of the batch job to be processed and the total resource quantity of the distributed system, so that the utilization rate of various resources of the system is maximized, the batch job processing efficiency is greatly improved, and the performance of the whole distributed system is finally improved.
According to the technical scheme, the data acquisition, storage, use, processing and the like meet relevant regulations of national laws and regulations, and various types of data such as personal identity data, operation data, behavior data and the like related to individuals, clients, crowds and the like are authorized.
As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
The above-mentioned embodiments are provided to further explain the objects, technical solutions and advantages of the present invention in detail, and it should be understood that the above-mentioned embodiments are only examples of the present invention and should not be used to limit the scope of the present invention, and any modifications, equivalents, improvements and the like made within the spirit and principle of the present invention should be included in the scope of the present invention.
Claims (13)
1. A method for scheduling resources, comprising:
determining the requirements of different types of jobs on various resources, wherein the requirements of the jobs of various types on the various resources are obtained by analyzing a sample data set constructed according to the quantity of the various resources consumed by processing the jobs of different types by each working node in a service system, and the service system is a distributed service system for realizing batch job distribution through an intelligent batch scheduling mechanism;
under the total number of various resources which can be provided by a service system, constructing an optimal resource allocation problem of service system profit maximization based on a multidimensional bounded knapsack algorithm, and obtaining an optimal operation allocation strategy by solving the optimal resource allocation problem;
and scheduling the resources of the service system according to the optimal operation allocation strategy.
2. The method of claim 1, further comprising:
after determining a queue formed by various types of jobs to be scheduled, determining the requirements of different types of jobs in the queue on various types of resources;
selecting a category of operation in the queue, constructing an optimal resource allocation problem of maximizing the income of the service system based on a multidimensional bounded knapsack algorithm under the total number of various resources which can be provided by the service system, and obtaining an optimal operation allocation strategy of the category of operation by solving the optimal resource allocation problem;
for the operation of the category, after scheduling the resources of the service system according to the optimal operation allocation strategy, moving the operation of the category out of the queue;
and after scheduling the jobs of each category in the queue, removing the queue until no job to be scheduled exists in the queue.
3. The method of claim 1, wherein the demands of different classes of jobs for various classes of resources are stored in a map mapping structure.
4. A method according to any one of claims 1 to 3, wherein the business system benefit maximization is determined as follows:
system throughput benefit P for Job k versus resource demand k Inversely proportional to the average resource consumption of job k:
wherein, P k System throughput gain, W, representing job k versus resource demand i，k Representing the demand of job k for resources of type i, c i Representing the number of i-type resources, and m represents the resource type;
the throughput benefit maximization of the distributed system is expressed as follows:
a k ≤j k ，k=1，......，n，
wherein, P k System throughput gain representing Job k versus resource demand, a k Indicates the number of jobs k, W i ， k Representing the amount of demand of job k on resources of type i, j k Representing the number of k types of batch jobs, m representing the total number of resource types, n representing the total number of batch job types, a k Are integers.
5. The method of claim 4, wherein the optimal job allocation strategy is solved using a depth first search method according to throughput gain maximization.
6. A resource scheduling apparatus, comprising:
the system comprises a demand module, a service system and a management module, wherein the demand module is used for determining the demands of different types of jobs on various resources, the demands of the jobs of various types on the various resources are obtained by analyzing a sample data set constructed according to the quantity of the various resources consumed by processing the jobs of different types by each working node in the service system, and the service system is a distributed service system for realizing batch job distribution through an intelligent batch scheduling mechanism;
the strategy module is used for constructing a resource optimal allocation problem of business system profit maximization based on a multidimensional bounded knapsack algorithm under the total number of various resources which can be provided by a business system, and obtaining an optimal operation allocation strategy by solving the resource optimal allocation problem;
and the scheduling module is used for scheduling the resources of the service system according to the optimal operation allocation strategy.
7. The apparatus of claim 6,
the demand module is further used for determining the demands of different types of jobs in the queue on various types of resources after determining the queue formed by various types of jobs to be scheduled each time;
the strategy module is further used for selecting a category of operation in the queue, constructing an optimal resource allocation problem with maximized business system income based on a multidimensional bounded knapsack algorithm under the total number of various resources provided by the business system, and obtaining an optimal operation allocation strategy of the category of operation by solving the optimal resource allocation problem;
the scheduling module is further used for moving the jobs of the category out of the queue after scheduling the resources of the business system according to the optimal job allocation strategy aiming at the jobs of the category;
the scheduling module is further used for removing the queue after scheduling the jobs of each category in the queue until no job to be scheduled exists in the queue.
8. The apparatus of claim 6, wherein the requirements module is further to store requirements for various types of resources for different types of jobs in a map structure.
9. The apparatus of any of claims 6 to 8, wherein the policy module is further configured to determine business system revenue maximization by:
system throughput benefit P for Job k versus resource demand k Inversely proportional to the average resource consumption of job k:
wherein, P k System throughput gain, W, representing Job k versus resource demand i ， k Representing the demand of job k for resources of type i, c i Representing the number of i-type resources, and m represents the resource type;
the throughput gain maximization of the distributed system is expressed as:
a k ≤j k ，k=1，......，n，
wherein, P k System throughput gain, a, representing job k versus resource demand k Indicates the number of jobs k, W i，k Representing the demand of job k on resources of type i, j k Representing the number of k types of batch jobs, m representing the total number of resource categories, n representing the total number of batch job categories, a k Are integers.
10. The apparatus of claim 9, wherein the policy module is further configured to solve the optimal job assignment policy using a depth first search method based on throughput gain maximization.
11. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the method of any of claims 1 to 5 when executing the computer program.
12. A computer-readable storage medium, characterized in that it stores a computer program which, when executed by a processor, implements the method of any one of claims 1 to 5.
13. A computer program product, characterized in that the computer program product comprises a computer program which, when being executed by a processor, carries out the method of any one of claims 1 to 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211137218.2A CN115391047A (en) | 2022-09-19 | 2022-09-19 | Resource scheduling method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211137218.2A CN115391047A (en) | 2022-09-19 | 2022-09-19 | Resource scheduling method and device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115391047A true CN115391047A (en) | 2022-11-25 |
Family
ID=84126252
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211137218.2A Pending CN115391047A (en) | 2022-09-19 | 2022-09-19 | Resource scheduling method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115391047A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116302500A (en) * | 2023-02-22 | 2023-06-23 | 北京乐研科技股份有限公司 | CPIE resource sharing scheduling method and system based on X86 platform |
-
2022
- 2022-09-19 CN CN202211137218.2A patent/CN115391047A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116302500A (en) * | 2023-02-22 | 2023-06-23 | 北京乐研科技股份有限公司 | CPIE resource sharing scheduling method and system based on X86 platform |
| CN116302500B (en) * | 2023-02-22 | 2024-01-30 | 北京乐研科技股份有限公司 | CPIE resource sharing scheduling method and system based on X86 platform |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107038069B (en) | Dynamic label matching DLMS scheduling method under Hadoop platform | |
| Zhao et al. | Parallel k-means clustering based on mapreduce | |
| CN110390345B (en) | Cloud platform-based big data cluster self-adaptive resource scheduling method | |
| Yang et al. | Intermediate data caching optimization for multi-stage and parallel big data frameworks | |
| CN107908536B (en) | Performance evaluation method and system for GPU application in CPU-GPU heterogeneous environment | |
| US11709834B2 (en) | Method and database system for sequentially executing a query and methods for use therein | |
| You et al. | Comprehensive workload analysis and modeling of a petascale supercomputer | |
| CN102393839A (en) | Parallel data processing system and method | |
| CN113037877A (en) | Optimization method for time-space data and resource scheduling under cloud edge architecture | |
| Wu et al. | A deadline-aware estimation of distribution algorithm for resource scheduling in fog computing systems | |
| Li et al. | Intermediate data placement and cache replacement strategy under Spark platform | |
| CN114066073A (en) | Power grid load prediction method | |
| CN110705716A (en) | Multi-model parallel training method | |
| CN114895773A (en) | Energy consumption optimization method, system and device of heterogeneous multi-core processor and storage medium | |
| Ramazanli et al. | Adaptive sampling distributed stochastic variance reduced gradient for heterogeneous distributed datasets | |
| CN115391047A (en) | Resource scheduling method and device | |
| CN113568759B (en) | Cloud computing-based big data processing method and system | |
| CN115827237A (en) | Storm task scheduling method based on cost performance | |
| CN110532071B (en) | Multi-application scheduling system and method based on GPU | |
| CN110929121B (en) | Log analysis-based seismic data processor time refinement calculation method and system | |
| CN202838322U (en) | Parallel data processing system | |
| Harris et al. | Monte carlo based server consolidation for energy efficient cloud data centers | |
| CN110689320A (en) | Large-scale multi-target project scheduling method based on co-evolution algorithm | |
| CN110058812B (en) | Scientific workflow data placement method in cloud environment | |
| Wright et al. | Usher: Utility-based Scheduling Algorithms for Polymorphic Applications |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |