CN111343288B - Job scheduling method and system and computing device - Google Patents
Job scheduling method and system and computing device Download PDFInfo
- Publication number
- CN111343288B CN111343288B CN202010436985.8A CN202010436985A CN111343288B CN 111343288 B CN111343288 B CN 111343288B CN 202010436985 A CN202010436985 A CN 202010436985A CN 111343288 B CN111343288 B CN 111343288B
- Authority
- CN
- China
- Prior art keywords
- job
- jobs
- time
- time period
- node
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/02—Protocols based on web technology, e.g. hypertext transfer protocol [HTTP]
- H04L67/025—Protocols based on web technology, e.g. hypertext transfer protocol [HTTP] for remote control or remote monitoring of applications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/60—Scheduling or organising the servicing of application requests, e.g. requests for application data transmissions using the analysis and optimisation of the required network resources
- H04L67/62—Establishing a time schedule for servicing the requests
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/60—Scheduling or organising the servicing of application requests, e.g. requests for application data transmissions using the analysis and optimisation of the required network resources
- H04L67/63—Routing a service request depending on the request content or context
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The invention discloses a job scheduling method, which comprises the following steps: receiving a job running request of a client, and adding the job to a job queue, wherein the job queue comprises one or more jobs; determining the node demand quantity of each job in the job queue; acquiring one or more jobs with the node demand quantity exceeding a threshold value from the job queue as delayed jobs based on the idle node quantity interval of the first time period; and allocating the one or more delayed jobs to a plurality of idle nodes in the first time period of the next job run cycle so as to run the one or more delayed jobs in the first time period of the next job run cycle. The invention also discloses a corresponding job scheduling system and a corresponding computing device. The job scheduling method is beneficial to improving the utilization rate of cluster node resources and improving the overall efficiency of operation.
Description
Technical Field
The invention relates to the technical field of internet and computers, in particular to a job scheduling method, a job scheduling system and computing equipment.
Background
Because a single computer provides limited computing power, when a large-scale computing job needs to be processed, a cluster is usually used for computing. A cluster is a supercomputer consisting of multiple computers interconnected by a high-speed network. A plurality of users apply for computing resources as required in a cluster sharing mode, and therefore computing work of the users is completed. Although the cluster can allocate available computing resources according to the application of the user, the resources of the cluster are limited and cannot meet the resource requirements of all users at any time, so that the cluster needs to use a job scheduler to reasonably schedule and allocate the computing resources, on one hand, the full utilization of the resources is ensured, and the utilization efficiency is improved; on the other hand, the fair use principle of the users is ensured, and the situation that resources are occupied by a small number of users for a long time and other users cannot use the resources is avoided.
The job schedulers are various in types, but the main functions are different and different, and the problem of scheduling and allocating resources is solved. Closely related to resource scheduling is a scheduling policy, i.e. a rule for allocating resources. In different scenarios, different scheduling strategies are usually selected for different purposes. In the prior art, the most common scheduling strategies include: FIFO, first-in first-out, according to the sequence, the first submitted job, preferentially using the resources; backfill, a Backfill strategy, and preferentially selecting a job which can fill in the resource vacancy for running, and for the job which needs more resources, the scheduling strategy has lower running probability because the job can run after waiting for a larger resource vacancy. Comparing the two scheduling strategies, the FIFO strategy is more fair for users to use, and the backhaul is more favorable for improving the utilization rate of resources, but is likely to cause the phenomena of larger job size and longer queuing time, and is not friendly to some users to use.
For this reason, a job scheduling method is needed to solve the problems in the above technical solutions.
Disclosure of Invention
To this end, the present invention provides a job scheduling method and a job scheduling system in an attempt to solve or at least alleviate the above-existing problems.
According to an aspect of the present invention, there is provided a job scheduling method, executed in a computing device coupled to a computer cluster, the computer cluster including a plurality of computing nodes adapted to run jobs and adapted to run jobs based on job run cycles, each job run cycle including a first time period and a second time period, the number of idle nodes of the first time period being greater than the number of idle nodes of the second time period, the method including the steps of: receiving a job running request of a client, and adding the job to a job queue, wherein the job queue comprises one or more jobs; determining the node demand quantity of each job in the job queue; acquiring one or more jobs with the node demand quantity exceeding a threshold value from the job queue as delayed jobs based on the idle node quantity interval of the first time period; and allocating the one or more delayed jobs to a plurality of idle nodes within the first time period of the next job run cycle so that the one or more delayed jobs are run during the first time period of the next job run cycle.
Optionally, in the job scheduling method according to the present invention, the step of allocating the one or more delayed jobs to a plurality of idle nodes in a first time period of a next job run cycle includes: setting the running time of the one or more deferred jobs to be in a first time period of a next job running cycle so as to run the one or more deferred jobs based on a plurality of idle nodes in the first time period.
Optionally, in the job scheduling method according to the present invention, the step of setting the running time of the one or more delayed jobs to be within the first time period of the next job running cycle includes: setting the start run times of the one or more deferred jobs to each predetermined time point of the first period of the next job run cycle so as to run the one or more deferred jobs from the predetermined time point of the next job run cycle.
Optionally, in the job scheduling method according to the present invention, the step of setting the running time of the one or more delayed jobs to be within the first time period of the next job running cycle includes: determining the running time of the one or more delayed jobs according to the number of idle nodes at each time point in the first time period.
Optionally, in the job scheduling method according to the present invention, the step of determining the running time of the one or more delayed jobs includes: determining the number of idle nodes at each time point of a first time period, sequencing the one or more delay jobs based on the number of node demands, and determining the time point of running of each delay job based on the sequence of the number of node demands and the number of idle nodes at each time point.
Optionally, in the job scheduling method according to the present invention, the step of acquiring one or more jobs whose node demand number exceeds a threshold from the job queue includes: scanning the jobs in the job queue once every preset time in a second time period of the current job running cycle to determine the jobs with the node demand quantity exceeding a threshold value; and sorting the jobs with the node demand quantity exceeding a threshold value based on the height of the node demand quantity, and selecting one or more jobs with the highest node demand quantity, wherein the sum of the node demand quantities of the selected one or more jobs is within the idle node quantity interval of the first time period.
Optionally, in the job scheduling method according to the present invention, the step of selecting one or more jobs with the highest node demand number includes: after the operation that the node demand quantity exceeds the threshold value is determined by scanning each time, the operation that the node demand quantity determined this time exceeds the threshold value and one or more operations with the highest node demand quantity selected last time are combined together for sequencing; and re-selecting one or more jobs with the highest node requirement quantity from all the jobs which are combined together and sequenced as the delay jobs.
Optionally, in the job scheduling method according to the present invention, the method further includes: and in each job running period, sequentially allocating one or more corresponding idle nodes to the jobs in the job queue based on the adding sequence so as to run the jobs in the job queue.
Optionally, in the job scheduling method according to the present invention, the method further includes: monitoring the number of idle nodes of the computer cluster in real time, and acquiring idle node data of one or more previous operation running periods; and determining a first time period and a second time period of a job operation cycle based on the idle node data, and determining an idle node number interval of the first time period.
Optionally, in the job scheduling method according to the present invention, the idle node data includes the number of idle nodes at each time point of the previous job running period.
According to an aspect of the present invention, there is provided a job scheduling system including: a scheduling server adapted to execute the job scheduling method as described above; the client is coupled with the scheduling server and is suitable for responding to the job running request of a user and sending the job running request to the scheduling server; and the computer cluster is coupled with the scheduling server, comprises a plurality of computing nodes suitable for running the jobs, and is suitable for running the jobs based on job running cycles, wherein each job running cycle comprises a first time period and a second time period, and the number of idle nodes in the first time period is greater than that in the second time period.
Optionally, in the job scheduling system according to the present invention, further comprising: and the monitoring server is respectively coupled with the scheduling server and the computer cluster and is suitable for monitoring the number of idle nodes of the computer cluster in real time.
According to an aspect of the present invention, there is provided a computing device comprising: at least one processor; and a memory storing program instructions, wherein the program instructions are configured to be executed by the at least one processor, the program instructions comprising instructions for performing the job scheduling method as described above.
According to an aspect of the present invention, there is provided a readable storage medium storing program instructions which, when read and executed by a computing device, cause the computing device to perform the method as described above.
According to the technical scheme, the invention provides a job scheduling method, which comprises the steps of selecting a plurality of large-scale jobs with the highest node demand quantity from a job queue as delay jobs, delaying the delay jobs to run at the first time period of the next job running period, and running the delay jobs by using idle nodes with more first time periods of the next job running period. These delayed jobs are allocated to the idle node more quickly during the first time period of the next job run cycle than the lack of idle nodes during the current second time period. In addition, the problem that other jobs in the current job queue are hindered to run due to the fact that a large number of idle nodes are needed for running of large-scale jobs can be avoided, and the other jobs in the current job queue can be distributed to the idle nodes to run as soon as possible.
Therefore, according to the technical scheme of the invention, the busy-time operation and the idle-time operation in each operation cycle can be balanced, and the large-scale operation submitted in the busy time is delayed to be operated in the idle time of the next operation cycle, so that the waste of idle-time node resources caused by queuing the large-scale operation for waiting the node resources in the busy time is avoided, the utilization rate of cluster node resources can be improved, and the overall efficiency of operation is improved.
Drawings
To the accomplishment of the foregoing and related ends, certain illustrative aspects are described herein in connection with the following description and the annexed drawings, which are indicative of various ways in which the principles disclosed herein may be practiced, and all aspects and equivalents thereof are intended to be within the scope of the claimed subject matter. The above and other objects, features and advantages of the present disclosure will become more apparent from the following detailed description read in conjunction with the accompanying drawings. Throughout this disclosure, like reference numerals generally refer to like parts or elements.
FIG. 1 shows a schematic diagram of a job scheduling system 100 according to one embodiment of the present invention;
FIG. 2 shows a schematic diagram of a computing device 200, according to one embodiment of the invention; and
FIG. 3 shows a flow diagram of a job scheduling method 300 according to one embodiment of the present invention.
Detailed Description
Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be embodied in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.
FIG. 1 shows a schematic diagram of a job scheduling system 100 according to one embodiment of the present invention.
As shown in FIG. 1, the system 100 may include one or more clients 110, a dispatch server 120, and at least one computer cluster 130. Wherein, the dispatch server 120 is communicatively connected to the client 110 and the computer cluster 130, respectively. It should be understood that the job scheduling system 100 shown in FIG. 1 is merely exemplary, and that there may be different numbers of clients 110, scheduling servers 120, and computer clusters 130 in a particular implementation, and the invention is not limited to the number or deployment of these devices.
The client 130 may be implemented as a personal computer including desktop and notebook configurations, and may also be implemented as a mobile device such as a cell phone, tablet, notebook, television box, wearable device, and the like. A corresponding application is resident in the client 130, and a user can transmit a job execution request to the scheduling server 120 through the application.
The client 110 transmits a job execution request to the scheduling server 120 in response to a request of a user. Dispatch server 120 may be deployed in multiple geographic locations, respectively, which may be implemented as a single computing device or as a cluster. Accordingly, the client 110 can transmit a job execution request to one scheduling server 120 connected thereto. The job execution request includes the number of computing nodes (required number of nodes) required to execute the job.
After receiving the job execution request from the client 110, the scheduling server 120 adds the job to the job queue. Also, the scheduling server 120 may allocate a corresponding number of compute nodes for the job according to the number of node demands of the job so that the job is run through the compute nodes. The scheduling server 120 is adapted to sequentially allocate a corresponding number of computing nodes to the jobs in the current job queue according to the arranging order in the job queue, so that the jobs in the current job queue are executed through the computing nodes, and the jobs in the job queue are queued for execution.
It should be noted that during the operation of jobs through the computing nodes of the computer cluster 130, the usage of the cluster node resources has a periodic tidal pattern, and each job operation cycle includes a peak period and a valley period of the cluster resource usage. Therefore, in each operation cycle, the number of idle nodes in each time period is different, and each operation cycle is divided into two time periods, namely a first time period (idle time) with a large number of idle nodes and a second time period (busy time) with a small number of idle nodes. That is, the number of idle nodes in the first period of time is larger than the number of idle nodes in the second period of time per job operation cycle as a whole.
In one embodiment, dispatch server 120 may determine the number of node requirements for each job in the current job queue. Based on the idle node number interval of the first time period, one or more jobs with the node demand number exceeding a threshold value are selected from the job queues of the second time period of the current job operation cycle, and the jobs exceeding the threshold value are extracted from the job queues to serve as delay jobs. Here, the free node number interval of the first period is a range in which the number of free nodes of the first period suitable for each job operation cycle determined by evaluation is evaluated based on the node operation job condition at each time point within the first period of the previous job operation cycle.
In one embodiment, the dispatch server 120 may monitor the number of free nodes of the computer cluster 130 in real time each job run cycle. By acquiring idle node data of a previous job running cycle from the monitored data, the idle node data includes the number of idle nodes at each time point of the previous job running cycle, so that the scheduling server 120 can determine a first time period in which the number of idle nodes is generally large and a second time period in which the number of idle nodes is small in the job running cycle according to the acquired idle node data. And, according to the number of idle nodes at each time point of the previous job execution cycle, the idle node number interval determining the first time period of the job execution cycle may be evaluated.
In another embodiment, the system 100 may further provide a separate monitoring server 140, and the monitoring server 140 is connected to the scheduling server 120 and the computer cluster 130, respectively, so as to monitor the number of idle nodes of the computer cluster 130 in real time through the monitoring server 140.
In one embodiment, the scheduling server 120 allocates the selected one or more delayed jobs to a plurality of idle nodes in the first time period of the next job running cycle, i.e., the delayed jobs are delayed to run in the first time period of the next job running cycle, and the delayed jobs are run by using more idle nodes in the first time period of the next job running cycle.
Specifically, the scheduling server 120 runs the deferred jobs by a plurality of idle nodes within the first period of the next job run cycle by setting the run time of one or more deferred jobs to be within the first period of the next job run cycle. Here, the present invention does not limit a specific time point at which each of the delayed jobs is executed in the first period of the next job execution cycle.
In one embodiment, the scheduling server 120 is adapted to perform the job scheduling method 300 of the present invention. The job scheduling method 300 of the present invention will be described in detail below.
According to an embodiment of the present invention, the various components in the job scheduling system 100 described above may communicate over one or more networks, such as a Local Area Network (LAN) or a Wide Area Network (WAN), such as the internet. Each of the computing nodes in the computer cluster 130, the scheduling server 120, the monitoring server 140, and the client 110 may be implemented by a computing device 200 as described below.
According to the job scheduling system, the busy-time job operation and the idle-time job operation in each job operation cycle can be balanced, and the large-scale job submitted in the busy time is delayed to be operated in the idle time of the next operation cycle, so that the waste of idle-time node resources caused by queuing the large-scale job for waiting the node resources in the busy time is avoided, the utilization rate of cluster node resources can be improved, and the overall efficiency of operation job can be improved.
FIG. 2 shows a schematic diagram of a computing device 200, according to one embodiment of the invention.
As shown in FIG. 2, in a basic configuration 202, a computing device 200 typically includes a system memory 206 and one or more processors 204. A memory bus 208 may be used for communication between the processor 204 and the system memory 206.
Depending on the desired configuration, the processor 204 may be any type of processing, including but not limited to: a microprocessor (μ P), a microcontroller (μ C), a digital information processor (DSP), or any combination thereof. The processor 204 may include one or more levels of cache, such as a level one cache 210 and a level two cache 212, a processor core 214, and registers 216. Example processor cores 214 may include Arithmetic Logic Units (ALUs), Floating Point Units (FPUs), digital signal processing cores (DSP cores), or any combination thereof. The example memory controller 218 may be used with the processor 204, or in some implementations the memory controller 218 may be an internal part of the processor 204.
Depending on the desired configuration, system memory 206 may be any type of memory, including but not limited to: volatile memory (such as RAM), non-volatile memory (such as ROM, flash memory, etc.), or any combination thereof. System memory 106 may include an operating system 220, one or more applications 222, and program data 224. The application 222 is actually a plurality of program instructions that direct the processor 204 to perform corresponding operations. In some embodiments, application 222 may be arranged to cause processor 204 to operate with program data 224 on an operating system.
Computing device 200 may also include an interface bus 240 that facilitates communication from various interface devices (e.g., output devices 242, peripheral interfaces 244, and communication devices 246) to the basic configuration 202 via the bus/interface controller 230. The example output device 242 includes a graphics processing unit 248 and an audio processing unit 250. They may be configured to facilitate communication with various external devices, such as a display or speakers, via one or more a/V ports 252. Example peripheral interfaces 244 can include a serial interface controller 254 and a parallel interface controller 256, which can be configured to facilitate communications with external devices such as input devices (e.g., keyboard, mouse, pen, voice input device, touch input device) or other peripherals (e.g., printer, scanner, etc.) via one or more I/O ports 258. An example communication device 246 may include a network controller 260, which may be arranged to facilitate communications with one or more other computing devices 262 over a network communication link via one or more communication ports 264.
A network communication link may be one example of a communication medium. Communication media may typically be embodied by computer readable instructions, data structures, program modules, and may include any information delivery media, such as carrier waves or other transport mechanisms, in a modulated data signal. A "modulated data signal" may be a signal that has one or more of its data set or its changes made in such a manner as to encode information in the signal. By way of non-limiting example, communication media may include wired media such as a wired network or private-wired network, and various wireless media such as acoustic, Radio Frequency (RF), microwave, Infrared (IR), or other wireless media. The term computer readable media as used herein may include both storage media and communication media.
In an embodiment according to the invention, the computing device 200 is configured to perform a job submission method according to the invention. Included in the application 222 of the computing device 200 are a plurality of program instructions for performing the job scheduling method 300 of the present invention, which may instruct the processor 204 to perform the job scheduling method 300 of the present invention. The computing device 200 enables a user-submitted job to be distributed to one or more computing nodes for execution by executing the job scheduling method 300 of the present invention.
FIG. 3 shows a flow diagram of a job scheduling method 300 according to one embodiment of the present invention. The method 300 is suitable for execution in a computing device, such as the computing device 200 described above.
In an embodiment in accordance with the invention, computing device 200 is coupled to a computer cluster 130, the computer cluster 130 including a plurality of computing nodes, the plurality of computing nodes operable to run jobs. Also, the computer cluster 130 runs the job based on the job run period.
It should be noted that during the operation of jobs through the computing nodes of the computer cluster 130, the usage of the cluster node resources has a periodic tidal pattern, and each job operation cycle includes a peak period and a valley period of the cluster resource usage. In this way, in each job running cycle, the number of idle nodes (node resources) in each time period is different, and each job running cycle is divided into two time periods, namely a first time period (idle time) with a large number of idle nodes and a second time period (busy time) with a small number of idle nodes. That is, the number of idle nodes (node resources) in the first period of time per job run cycle is generally larger than the number of idle nodes in the second period of time as a whole.
As shown in FIG. 3, the job scheduling method 300 begins at step S310.
In step S310, a job execution request sent by the client 110 is received, and a job is added to the job queue. Here, the job queue includes one or more jobs that have been submitted by one or more clients 110 and that have not been assigned to a computing node to run, the one or more jobs being arranged in chronological order of addition to the job queue. It should be noted that the number of computing nodes required to run the job may be included in the job running request, so that the computing device 200 may allocate a corresponding number of computing nodes to the job according to the number of node requirements of the job and run the job through the computing nodes.
Subsequently, in step S320, the computing device 200 determines the node demand number of each job in the current job queue according to the job execution request.
Subsequently, in step S330, one or more jobs with a node demand quantity exceeding a threshold are selected from the job queue in busy time (i.e. in the second time period of the current job running cycle) based on the idle node quantity interval in the first time period, and the jobs exceeding the threshold are extracted from the job queue as delayed jobs. Here, the free node number interval of the first period is a range in which the number of free nodes of the first period suitable for each job operation cycle determined by evaluation is evaluated based on the node operation job condition at each time point within the first period of the previous job operation cycle.
It should be noted that, in the embodiment according to the present invention, at each time point of each job running cycle, the computing device 200 sequentially allocates one or more corresponding free nodes to the jobs in the current job queue (not including the selected delayed jobs, where the selected delayed jobs are not already in the current job queue) according to the arranged order in the job queue, and the jobs in the job queue are executed by the corresponding free nodes, so as to queue up for execution. It should also be noted that the selected deferred jobs are not queued up in the job queue and are delayed by the computing device 200 to be assigned to the corresponding computing nodes.
According to one embodiment, the computing device 200 may monitor the number of free nodes of the computer cluster 130 in real time each job run cycle. In addition, the computing device 200 may also be connected to the monitoring server 140 and monitor the number of idle nodes of the computer cluster 130 in real time through the monitoring server 140.
Prior to performing method S330, the computing device 200 may first obtain idle node data for one or more previous job run periods from the monitored data. The idle node data includes the number of idle nodes at each time point of the previous job run period. In this way, the computing device 200 can determine, from the acquired free node data, a first time period (idle time) during which there are typically a large number of free nodes and a second time period (busy time) during which there are a small number of free nodes in a job run cycle.
For example, each job run cycle is one day (24 hours), the number of idle nodes of each whole point in each day is obtained from the monitored data, the tide rule of the cluster node resource usage is determined according to the obtained idle node data, and the first time period (idle time) and the second time period (busy time) of the job run cycle are determined. Here, the first period of time of each job operation cycle determined from the acquired idle node data is 0 point to 8 points, and the second period of time is 8 points to 24 points.
And, according to the number of idle nodes at each time point of the previous job execution cycle, the idle node number interval determining the first time period of the job execution cycle may be evaluated.
Finally, in step S340, the selected one or more delayed jobs are allocated to a plurality of idle nodes in the first time period of the next job running cycle, that is, the delayed jobs are delayed to the first time period of the next job running cycle for running, and the delayed jobs are run by using more idle nodes in the first time period of the next job running cycle.
According to one embodiment, the run time of one or more deferred jobs is set to be within the first period of the next job run cycle so that the deferred jobs are run by a plurality of idle nodes within the first period of the next job run cycle. Here, the present invention does not limit a specific time point at which each of the delayed jobs is executed in the first period of the next job execution cycle.
According to one embodiment, the sum of the node demand numbers of the jobs whose node demand numbers exceed the threshold value, which are acquired from the job queue in step S330, may be within the free node number interval of the first period of time and close to the maximum value of the free node number interval. It should be appreciated that the selected job or jobs are jobs in the job queue that require a relatively greater number of nodes (as compared to other unselected jobs).
According to one embodiment, during the second time period of each job run cycle, computing device 200 may scan the jobs in the job queue once every predetermined time, from which one or more jobs in the current job queue whose node demand number exceeds the threshold may be determined.
Subsequently, the computing device 200 sorts the jobs with the determined number of node demands exceeding the threshold value according to the level of the number of node demands, selects one or more jobs with the highest number of node demands, and extracts the jobs from the job queue. Here, the sum of the node demand numbers of the one or more jobs extracted from the job queue is within the determined free node number interval of the first period of time and is close to the maximum value of the free node number interval.
According to one embodiment, after each time of scanning and determining the jobs with the node demand quantity exceeding the threshold value in the current job queue, the jobs with the node demand quantity exceeding the threshold value determined by the current scanning and one or more jobs with the node demand quantity highest selected last time are merged together and sorted again. And then, one or more jobs with the highest node requirement quantity are reselected from all the jobs which are combined together and sequenced as the current delay jobs determined after the current scanning. It should be noted that, after the jobs with the node demand number exceeding the threshold determined by the current scanning and the jobs selected last time are merged together and then sorted again, some of the jobs selected last time may no longer be the job or jobs with the highest node demand number after merging and sorting, and these jobs are not reselected this time, so that these jobs are continuously added to the current job queue, and these jobs are no longer processed as delayed jobs, but continue to wait for the distributed computing nodes to run according to the order in the job queue in the current job running period.
According to one embodiment, the computing device 200 may uniformly set the start running times of the selected plurality of delayed jobs to a predetermined time point within the first time period of the next job running cycle, for example, uniformly set the start time to 0 point of the next job running cycle. In this way, the computing device 200 allocates free nodes for the delayed jobs from the predetermined point of time of the next job execution cycle, and causes the delayed jobs to be executed from the predetermined point of time of the next job execution cycle.
It should be noted that, because the number of idle nodes at each time point in the first time period is different, the number of idle nodes at the predetermined time point may be lower than the sum of the required number of nodes of the plurality of delayed jobs, so that at the predetermined time point of the next job operation cycle, the plurality of delayed jobs may also need to be queued for operation.
According to yet another embodiment, the computing device 200 may determine the run time of each deferred job based on the number of idle nodes at various points in time within the first time period. That is, the computing device 200 does not set the run times uniformly for all of the deferred jobs, and the start run times of a plurality of deferred jobs may be set to different time points, respectively.
Specifically, when the running time of the plurality of delay jobs is determined, the plurality of delay jobs are sorted based on the level of the node demand number by determining the idle node number of each time point of the first time period, and then the time point at which each delay job starts to run is reasonably set according to the order of the node demand number of the plurality of delay jobs and the idle node number of each time point. For example, according to the order of the node demand quantities of the plurality of delay jobs, the starting running time of the delay job with the highest node demand quantity is set as the time point with the highest number of idle nodes in the first time period.
According to the job scheduling method, several large-scale jobs with the highest node demand quantity are selected from the job queue as delay jobs in the current second time period, the delay jobs are delayed to be operated in the first time period of the next job operation period, and the delay jobs are operated by using idle nodes with more first time periods in the next job operation period. These delayed jobs are allocated to the idle node more quickly during the first time period of the next job run cycle than the lack of idle nodes during the current second time period. In addition, the problem that other jobs in the current job queue are hindered to run due to the fact that a large number of idle nodes are needed for running of large-scale jobs can be avoided, and the other jobs in the current job queue can be distributed to the idle nodes to run as soon as possible. Therefore, according to the technical scheme of the invention, the busy-time operation and the idle-time operation in each operation cycle can be balanced, and the large-scale operation submitted in the busy time is delayed to be operated in the idle time of the next operation cycle, so that the waste of idle-time node resources caused by queuing the large-scale operation for waiting the node resources in the busy time is avoided, the utilization rate of cluster node resources can be improved, and the overall efficiency of operation is improved. The various techniques described herein may be implemented in connection with hardware or software or, alternatively, with a combination of both. Thus, the methods and apparatus of the present invention, or certain aspects or portions thereof, may take the form of program code (i.e., instructions) embodied in tangible media, such as removable hard drives, U.S. disks, floppy disks, CD-ROMs, or any other machine-readable storage medium, wherein, when the program is loaded into and executed by a machine, such as a computer, the machine becomes an apparatus for practicing the invention.
In the case of program code execution on programmable computers, the computing device will generally include a processor, a storage medium readable by the processor (including volatile and non-volatile memory and/or storage elements), at least one input device, and at least one output device. Wherein the memory is configured to store program code; the processor is configured to execute the multilingual spam-text recognition method of the present invention according to instructions in said program code stored in the memory.
By way of example, and not limitation, readable media may comprise readable storage media and communication media. Readable storage media store information such as computer readable instructions, data structures, program modules or other data. Communication media typically embodies computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media. Combinations of any of the above are also included within the scope of readable media.
In the description provided herein, algorithms and displays are not inherently related to any particular computer, virtual system, or other apparatus. Various general purpose systems may also be used with examples of this invention. The required structure for constructing such a system will be apparent from the description above. Moreover, the present invention is not directed to any particular programming language. It is appreciated that a variety of programming languages may be used to implement the teachings of the present invention as described herein, and any descriptions of specific languages are provided above to disclose the best mode of the invention.
In the description provided herein, numerous specific details are set forth. It is understood, however, that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures and techniques have not been shown in detail in order not to obscure an understanding of this description.
Similarly, it should be appreciated that in the foregoing description of exemplary embodiments of the invention, various features of the invention are sometimes grouped together in a single embodiment, figure, or description thereof for the purpose of streamlining the disclosure and aiding in the understanding of one or more of the various inventive aspects. However, the disclosed method should not be interpreted as reflecting an intention that: that the invention as claimed requires more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive aspects lie in less than all features of a single foregoing disclosed embodiment. Thus, the claims following the detailed description are hereby expressly incorporated into this detailed description, with each claim standing on its own as a separate embodiment of this invention.
Those skilled in the art will appreciate that the modules or units or components of the devices in the examples disclosed herein may be arranged in a device as described in this embodiment or alternatively may be located in one or more devices different from the devices in this example. The modules in the foregoing examples may be combined into one module or may be further divided into multiple sub-modules.
Those skilled in the art will appreciate that the modules in the device in an embodiment may be adaptively changed and disposed in one or more devices different from the embodiment. The modules or units or components of the embodiments may be combined into one module or unit or component, and furthermore they may be divided into a plurality of sub-modules or sub-units or sub-components. All of the features disclosed in this specification (including any accompanying claims, abstract and drawings), and all of the processes or elements of any method or apparatus so disclosed, may be combined in any combination, except combinations where at least some of such features and/or processes or elements are mutually exclusive. Each feature disclosed in this specification (including any accompanying claims, abstract and drawings) may be replaced by alternative features serving the same, equivalent or similar purpose, unless expressly stated otherwise.
Furthermore, those skilled in the art will appreciate that while some embodiments described herein include some features included in other embodiments, rather than other features, combinations of features of different embodiments are meant to be within the scope of the invention and form different embodiments. For example, in the following claims, any of the claimed embodiments may be used in any combination.
Furthermore, some of the described embodiments are described herein as a method or combination of method elements that can be performed by a processor of a computer system or by other means of performing the described functions. A processor having the necessary instructions for carrying out the method or method elements thus forms a means for carrying out the method or method elements. Further, the elements of the apparatus embodiments described herein are examples of the following apparatus: the apparatus is used to implement the functions performed by the elements for the purpose of carrying out the invention.
As used herein, unless otherwise specified the use of the ordinal adjectives "first", "second", "third", etc., to describe a common object, merely indicate that different instances of like objects are being referred to, and are not intended to imply that the objects so described must be in a given sequence, either temporally, spatially, in ranking, or in any other manner.
While the invention has been described with respect to a limited number of embodiments, those skilled in the art, having benefit of this description, will appreciate that other embodiments can be devised which do not depart from the scope of the invention as described herein. Furthermore, it should be noted that the language used in the specification has been principally selected for readability and instructional purposes, and may not have been selected to delineate or circumscribe the inventive subject matter. Accordingly, many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the appended claims. The present invention has been disclosed in an illustrative rather than a restrictive sense, and the scope of the present invention is defined by the appended claims.
Claims (14)
1. A job scheduling method executed in a computing device coupled to a computer cluster, the computer cluster including a plurality of computing nodes adapted to run jobs and adapted to run jobs based on job run cycles, each job run cycle including a first time period and a second time period, the first time period having a greater number of idle nodes than the second time period, the method comprising the steps of:
receiving a job running request of a client, and adding the job to a job queue, wherein the job queue comprises one or more jobs;
determining the node demand quantity of each job in the job queue;
acquiring one or more jobs with the node demand quantity exceeding a threshold value from the job queue as delayed jobs in a second time period of a current job operation cycle based on the idle node quantity interval of the first time period, wherein the sum of the node demand quantities of the acquired one or more jobs is within the idle node quantity interval of the first time period; and
and allocating the one or more delayed jobs to a plurality of idle nodes in the first time period of the next job run cycle so as to run the one or more delayed jobs in the first time period of the next job run cycle.
2. The job scheduling method according to claim 1, wherein the step of assigning the one or more delayed jobs to a plurality of idle nodes within the first time period of the next job run cycle comprises:
setting the running time of the one or more deferred jobs to be in a first time period of a next job running cycle so as to run the one or more deferred jobs based on a plurality of idle nodes in the first time period.
3. The job scheduling method according to claim 2, wherein the step of setting the running time of the one or more delayed jobs to be within the first period of time of the next job running cycle comprises:
setting the start run times of the one or more deferred jobs to each predetermined time point of the first period of the next job run cycle so as to run the one or more deferred jobs from the predetermined time point of the next job run cycle.
4. The job scheduling method according to claim 2, wherein the step of setting the running time of the one or more delayed jobs to be within the first period of time of the next job running cycle comprises:
determining the running time of the one or more delayed jobs according to the number of idle nodes at each time point in the first time period.
5. The job scheduling method of claim 4, wherein determining the run time of the one or more deferred jobs comprises:
determining the number of idle nodes at each time point of the first time period,
ordering the one or more delay jobs based on the level of the number of node demands,
and determining the time point of running each delayed job based on the sequence of the node demand quantity and the number of idle nodes at each time point.
6. The job scheduling method according to any one of claims 1 to 5, wherein the step of obtaining one or more jobs from the job queue for which the number of node demands exceeds a threshold value comprises:
scanning the jobs in the job queue once every preset time in a second time period of the current job running cycle to determine the jobs with the node demand quantity exceeding a threshold value;
and sorting the jobs with the node demand quantity exceeding a threshold value based on the height of the node demand quantity, and selecting one or more jobs with the highest node demand quantity, wherein the sum of the node demand quantities of the selected one or more jobs is within the idle node quantity interval of the first time period.
7. The job scheduling method according to claim 6, wherein the step of selecting one or more jobs with the highest node demand number comprises:
after the operation that the node demand quantity exceeds the threshold value is determined by scanning each time, the operation that the node demand quantity determined this time exceeds the threshold value and one or more operations with the highest node demand quantity selected last time are combined together for sequencing;
and re-selecting one or more jobs with the highest node requirement quantity from all the jobs which are combined together and sequenced as the delay jobs.
8. The job scheduling method according to any one of claims 1 to 5, further comprising the steps of:
and in each job running period, sequentially allocating one or more corresponding idle nodes to the jobs in the job queue based on the adding sequence so as to run the jobs in the job queue.
9. The job scheduling method according to any one of claims 1 to 5, further comprising the steps of:
monitoring the number of idle nodes of the computer cluster in real time, and acquiring idle node data of one or more previous operation running periods;
and determining a first time period and a second time period of a job operation cycle based on the idle node data, and determining an idle node number interval of the first time period.
10. The job scheduling method according to claim 9,
the idle node data includes the number of idle nodes at each time point of the previous job run period.
11. A job scheduling system comprising:
a scheduling server adapted to perform a job scheduling method according to any one of claims 1 to 10;
the client is coupled with the scheduling server and is suitable for responding to the job running request of a user and sending the job running request to the scheduling server; and
the computer cluster is coupled with the scheduling server, comprises a plurality of computing nodes suitable for running the jobs, and is suitable for running the jobs based on job running cycles, each job running cycle comprises a first time period and a second time period, and the number of idle nodes in the first time period is larger than that in the second time period.
12. The job scheduling system according to claim 11, further comprising:
and the monitoring server is respectively coupled with the scheduling server and the computer cluster and is suitable for monitoring the number of idle nodes of the computer cluster in real time.
13. A computing device, comprising:
at least one actuator; and
a memory storing program instructions, wherein the program instructions are configured to be executed by the at least one actuator, the program instructions comprising instructions for performing the method of any of claims 1-10.
14. A readable storage medium storing program instructions that, when read and executed by a computing device, cause the computing device to perform the method of any of claims 1-10.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010436985.8A CN111343288B (en) | 2020-05-21 | 2020-05-21 | Job scheduling method and system and computing device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010436985.8A CN111343288B (en) | 2020-05-21 | 2020-05-21 | Job scheduling method and system and computing device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111343288A CN111343288A (en) | 2020-06-26 |
| CN111343288B true CN111343288B (en) | 2020-09-01 |
Family
ID=71187607
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010436985.8A Active CN111343288B (en) | 2020-05-21 | 2020-05-21 | Job scheduling method and system and computing device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111343288B (en) |
Families Citing this family (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112131001B (en) * | 2020-09-23 | 2022-10-18 | 山石网科通信技术股份有限公司 | Sandbox-based resource processing method and device, storage medium and processor |
| CN112965817B (en) * | 2021-02-22 | 2023-08-11 | 网易（杭州）网络有限公司 | Resource management method and device and electronic equipment |
| CN113641505B (en) * | 2021-10-14 | 2022-01-04 | 阿里云计算有限公司 | Resource allocation control method and device for server cluster |
| CN113946430B (en) * | 2021-12-20 | 2022-05-06 | 北京并行科技股份有限公司 | Job scheduling method, computing device and storage medium |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103761147A (en) * | 2014-01-15 | 2014-04-30 | 清华大学 | Method and system for managing calculation examples in cloud platforms |
| CN105159769A (en) * | 2015-09-11 | 2015-12-16 | 国电南瑞科技股份有限公司 | Distributed job scheduling method suitable for heterogeneous computational capability cluster |
| CN107203422A (en) * | 2016-08-28 | 2017-09-26 | 深圳晶泰科技有限公司 | A kind of job scheduling method towards high-performance calculation cloud platform |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104268018B (en) * | 2014-09-22 | 2017-11-24 | 浪潮(北京)电子信息产业有限公司 | Job scheduling method and job scheduler in a kind of Hadoop clusters |
| US11010193B2 (en) * | 2017-04-17 | 2021-05-18 | Microsoft Technology Licensing, Llc | Efficient queue management for cluster scheduling |
-
2020
- 2020-05-21 CN CN202010436985.8A patent/CN111343288B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103761147A (en) * | 2014-01-15 | 2014-04-30 | 清华大学 | Method and system for managing calculation examples in cloud platforms |
| CN105159769A (en) * | 2015-09-11 | 2015-12-16 | 国电南瑞科技股份有限公司 | Distributed job scheduling method suitable for heterogeneous computational capability cluster |
| CN107203422A (en) * | 2016-08-28 | 2017-09-26 | 深圳晶泰科技有限公司 | A kind of job scheduling method towards high-performance calculation cloud platform |
Non-Patent Citations (2)
| Title |
|---|
| On the idle time control for multistep scheduling algorithm in parallel and distributed processing;Hitoshi Yamazaki等;《IEEE》;20141231;全文 * |
| 基于延迟调度策略的reduce调度优化算法;石义龙等;《计算机应用研究》;20170731;第34卷(第7期);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111343288A (en) | 2020-06-26 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111343288B (en) | Job scheduling method and system and computing device | |
| US10223166B2 (en) | Scheduling homogeneous and heterogeneous workloads with runtime elasticity in a parallel processing environment | |
| US8020161B2 (en) | Method and system for the dynamic scheduling of a stream of computing jobs based on priority and trigger threshold | |
| US8434085B2 (en) | Scalable scheduling of tasks in heterogeneous systems | |
| Venkatesh et al. | A case for application-oblivious energy-efficient MPI runtime | |
| CN111104211A (en) | Task dependency based computation offload method, system, device and medium | |
| TWI503742B (en) | Multiprocessors systems and processes scheduling methods thereof | |
| JP2010267025A (en) | Program, device and method for scheduling job | |
| CN109086135B (en) | Resource scaling method and device, computer equipment and storage medium | |
| CN107341041B (en) | Cloud task multidimensional constraint backfill scheduling method based on priority queue | |
| CN103150213A (en) | Method and device for balancing load | |
| JP4912927B2 (en) | Task allocation apparatus and task allocation method | |
| CN114579285B (en) | Task running system and method and computing device | |
| CN110908782A (en) | Genetic algorithm optimization-based packaging type distributed job task scheduling method and system | |
| Lin et al. | A heuristic task scheduling algorithm for heterogeneous virtual clusters | |
| Qureshi et al. | Grid resource allocation for real-time data-intensive tasks | |
| Syed | HAMM: A hybrid algorithm of Min-Min and Max-Min task scheduling algorithms in cloud computing | |
| CN111625367B (en) | Method for dynamically adjusting read-write resources of file system | |
| WO2021212965A1 (en) | Resource scheduling method and related device | |
| Wu et al. | Abp scheduler: Speeding up service spread in docker swarm | |
| CN110175078B (en) | Service processing method and device | |
| CN104731662B (en) | A kind of resource allocation methods of variable concurrent job | |
| CN116069500A (en) | Model training task processing method and device, electronic equipment and readable medium | |
| Daoud et al. | Processor allocation algorithm based on frame combing with memorization for 2d mesh cmps | |
| CN113946430A (en) | Job scheduling method, computing device and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| PE01 | Entry into force of the registration of the contract for pledge of patent right |
Denomination of invention: Job scheduling method, system and computing equipment Effective date of registration: 20210129 Granted publication date: 20200901 Pledgee: Beijing ustron Tongsheng financing Company limited by guarantee Pledgor: BEIJING PARATERA TECHNOLOGY Co.,Ltd. Registration number: Y2021990000134 |
|
| PE01 | Entry into force of the registration of the contract for pledge of patent right | ||
| PC01 | Cancellation of the registration of the contract for pledge of patent right |
Date of cancellation: 20230303 Granted publication date: 20200901 Pledgee: Beijing ustron Tongsheng financing Company limited by guarantee Pledgor: BEIJING PARATERA TECHNOLOGY Co.,Ltd. Registration number: Y2021990000134 |
|
| PC01 | Cancellation of the registration of the contract for pledge of patent right |