WO2023273502A1 - 作业处理的方法、装置、计算机设备以及存储介质 - Google Patents

作业处理的方法、装置、计算机设备以及存储介质 Download PDF

Info

Publication number
WO2023273502A1
WO2023273502A1 PCT/CN2022/086057 CN2022086057W WO2023273502A1 WO 2023273502 A1 WO2023273502 A1 WO 2023273502A1 CN 2022086057 W CN2022086057 W CN 2022086057W WO 2023273502 A1 WO2023273502 A1 WO 2023273502A1
Authority
WO
WIPO (PCT)
Prior art keywords
job
category
amount
resources
resource
Prior art date
Application number
PCT/CN2022/086057
Other languages
English (en)
French (fr)
Inventor
胡守峰
朱波
郭晋
董栋
Original Assignee
华为技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority claimed from CN202111062578.6A external-priority patent/CN115543552A/zh
Application filed by 华为技术有限公司 filed Critical 华为技术有限公司
Priority to EP22831333.4A priority Critical patent/EP4357916A1/en
Publication of WO2023273502A1 publication Critical patent/WO2023273502A1/zh
Priority to US18/400,186 priority patent/US20240231901A9/en

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/48—Program initiating; Program switching, e.g. by interrupt
    • G06F9/4806—Task transfer initiation or dispatching
    • G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
    • G06F9/4881—Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
    • G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
    • G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
    • G06F9/505—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals considering the load
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
    • G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
    • G06F9/5011—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resources being hardware resources other than CPUs, Servers and Terminals
    • G06F9/5016—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resources being hardware resources other than CPUs, Servers and Terminals the resource being the memory
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F2209/00—Indexing scheme relating to G06F9/00
    • G06F2209/50—Indexing scheme relating to G06F9/50
    • G06F2209/5019—Workload prediction

Definitions

  • the present application relates to the technical field of job scheduling, and in particular to a job processing method, device, computer equipment and storage medium.
  • High performance computing is a cross-industry and cross-application computing discipline, which is widely used in the calculation of large-scale scientific problems and the processing of massive data, such as weather forecasting, automotive simulation, biopharmaceuticals, gene sequencing, nuclear Explosion simulation, and chip design and manufacturing.
  • a computer cluster capable of providing HPC services can be called an HPC system.
  • HPC performance the HPC system can handle a large number of jobs submitted by users to complete the computing services of each user.
  • the process of job processing can be as follows: the scheduling node in the HPC system supports multiple users to submit jobs to the HPC system at the same time, and when each user submits their own jobs, they also specify the real-time resources required by their respective jobs during execution quantity. For a certain job submitted by the user, the scheduling node can assign the job to a computing node with sufficient free resources in the HPC system according to the amount of resources required by the job specified by the user, and the computing node will execute the job.
  • the amount of resources consumed during the execution of a job may be constantly changing, and the amount of resources required by the job specified by the user is often the maximum amount of consumed resources (also called the peak amount of resources).
  • the present application provides a job processing method, device, computer equipment and storage medium, which can improve the effective utilization of resources in computing nodes and improve the efficiency of job processing during the process of processing jobs in a high-performance computing system.
  • the technical solution is as follows:
  • a method for job processing is provided, which can be executed by a scheduling node in a high-performance computing system.
  • the scheduling node After receiving a job to be processed by the high-performance computing system, the scheduling node first determines the category of the job, and then According to the type of the job, the predicted resource amount of the job is predicted. The predicted resource amount is the real-time resource consumption during the predicted job execution process. After that, the scheduling node schedules the computing nodes in the high-performance computing system to execute according to the predicted resource amount of the job. Operation.
  • the scheduling node in the high-performance computing system after the scheduling node in the high-performance computing system receives the job to be processed, it predicts the amount of resources consumed in real time during the execution of the job according to the type of the job.
  • the amount of consumed resources is relatively close. Therefore, compared with the specified amount of resources, the predicted amount of resources is closer to the actual amount of resources consumed by the job.
  • the scheduling node schedules the computing nodes in the high-performance computing system to execute the task according to the predicted amount of resources. During the operation process, the effective utilization rate of resources in the computing nodes can be improved, and the efficiency of job processing can be improved.
  • the scheduling node predicts the resource amount of the job according to the job type, including: the scheduling node predicts the resource amount consumed in real time during the job execution process according to the reference resource amount corresponding to the job type, and obtains the predicted A resource amount, wherein the reference resource amount is obtained based on the actual consumed resource amount of the historical jobs of this category.
  • the scheduling node when the scheduling node predicts the amount of resources consumed in real time during the execution of the job according to the type of the job, it can calculate the amount of resources consumed in real time during the execution of the job according to the amount of reference resources corresponding to the type. Forecasting, since the reference resource amount is obtained from the actual resource consumption of historical jobs in this category, the predicted resource amount predicted based on the reference resource amount corresponding to this category is closer to the actual resource consumption amount of the job, then the scheduling node is based on When predicting the amount of resources and scheduling computing nodes in the high-performance computing system to execute jobs, the effective utilization of resources in the computing nodes can be further improved.
  • the method before the scheduling node determines the type of the job, the method further includes: the scheduling node receives a mode adjustment instruction, and enables the prediction mode according to the mode adjustment instruction, wherein the mode adjustment instruction is used to indicate to enable the prediction mode, which provides the ability to predict the amount of resources required by a job in forecast mode.
  • the scheduling node predicts the amount of resources consumed in real time during the job execution process according to the reference resource amount corresponding to the job type, including: in the prediction mode, the scheduling node predicts the job execution according to the reference resource amount corresponding to the job type Predict the amount of resources consumed in real time during the process.
  • the prediction mode is a working mode of the scheduling node.
  • the scheduling node can adjust its own working mode according to the mode adjustment instruction, so that the technician can trigger the scheduling node to adjust the working mode by issuing the mode adjustment instruction to the scheduling node. , to suit different application scenarios.
  • the scheduling node predicts the amount of resources consumed in real time during job execution according to the reference resource amount corresponding to the job category, including: in the prediction mode, the scheduling node first obtains The number of historical jobs of the job category. If the number of historical jobs of this category is greater than or equal to the number threshold, the scheduling node will then predict the amount of resources consumed in real time during the execution of the job based on the reference resource amount corresponding to the category .
  • the amount of real-time resources required during the execution of historical jobs of this category is the sample data of this category, and the amount of reference resources corresponding to this category is the actual resource consumption of historical jobs in this category.
  • the number of historical operations of this category is greater than or equal to the number threshold, it means that the number of samples (that is, the number of historical operations) of this category is sufficient at this time, and it is considered that the amount of reference resources corresponding to this category at this time is the same as the number of this category.
  • the error between the actual resource consumption of the job is relatively small.
  • the scheduling node predicts that the error between the predicted resource amount of the job and the actual resource consumption amount of the job is relatively small, so that The accuracy of the forecasted resource amount for this job has been further improved.
  • the job includes the specified amount of resources for the job
  • the scheduling node predicts the amount of resources consumed in real time during the job execution process according to the reference resource amount corresponding to the job type, including: the scheduling node specifies based on the job The amount of resources, the amount of reference resources corresponding to this category, and the number of historical jobs of this category are used to predict the average amount of resources consumed in real time during the execution of each job of this category.
  • the specified resource amount of the job is the real-time resource amount predicted by the user during the execution of the job based on experience, and the reference resource amount is obtained from the actual resource consumption amount of the historical jobs in this category, therefore, Based on the specified amount of resources, the amount of reference resources corresponding to this category, and the number of historical jobs of this category, the predicted amount of resources is closer to the actual resource consumption of the job, and the scheduling node schedules the high-performance computing system based on the amount of predicted resources.
  • the effective utilization of resources in computing nodes can be further improved.
  • the method before the scheduling node predicts the predicted resource amount of the job according to the type of the job, the method further includes: the scheduling node obtains the average value of the actual resource consumption of the historical jobs of the type as the The amount of reference resources of the category; and record the corresponding relationship between the category and the amount of reference resources of this category.
  • the job includes a job attribute of the job, and the job attribute indicates the attribute of the job; the scheduling node determines the category of the job, including: the scheduling node determines the category of the job from multiple categories based on the job attribute of the job , jobs of the same category share at least one attribute.
  • the scheduling node can quickly query jobs from multiple categories based on the job attribute of the job category, which improves the efficiency of determining the job category.
  • the job attributes include the user type of the user to which the job belongs and the queue ID of the job queue where the job is located; the scheduling node determines the job category from multiple categories based on the job attributes of the job, including: based on multiple categories The corresponding relationship with the job attribute determines the category of the job by the category corresponding to the job attribute of the job among the multiple categories, wherein each category in the multiple categories corresponds to a job attribute.
  • the scheduling node can quickly query the category corresponding to the job attribute of the job from multiple categories based on the correspondence between multiple categories and job attributes, further improving the efficiency of determining the job category. efficiency.
  • the method further includes: the scheduling node receives the actual consumed resource amount of the job from the computing node , based on the actual consumed resource amount and the number of historical jobs of the job category, updating the reference resource amount corresponding to the category, and recording the correspondence between the updated reference resource amount and the job category.
  • an apparatus for job processing which is used for executing the above method for job processing.
  • the apparatus for processing a job includes a functional module for executing the method for processing a job provided in the above first aspect or any optional manner of the above first aspect.
  • a computer device includes a processor, and the processor is configured to execute program codes, so that the computer device executes operations performed by the above job processing method.
  • a computer-readable storage medium in which at least one piece of program code is stored, and the program code is read by a processor to enable a computer device to perform the operations performed by the above job processing method.
  • a computer program product or computer program includes program code, the program code is stored in a computer-readable storage medium, and a processor of a computer device reads the program code from the computer-readable storage medium. Taking the program code, the processor executes the program code, so that the computer device executes the method provided in the above first aspect or various optional implementation manners of the first aspect.
  • Fig. 1 is the schematic diagram of a kind of HPC system provided by the present application.
  • FIG. 2 is a schematic diagram of interaction between a scheduling node and a computing node provided by the present application
  • FIG. 3 is a flowchart of a job processing method provided by the present application.
  • FIG. 4 is a schematic diagram of processor resource occupation provided by the present application.
  • Fig. 5 is a schematic diagram of a job processing flow provided by the present application.
  • Fig. 6 is a flow chart of predicting resource amounts of different threads and updating reference resource amounts provided by the present application
  • Fig. 7 is a schematic structural diagram of a job processing device provided by the present application.
  • FIG. 8 is a schematic structural diagram of a computer device provided by the present application.
  • Job A collection of program instances that need to be executed to complete a specific computing service, usually corresponding to a group of processes, containers, or other runtime entities on one or more computers, where a group of programs An instance includes at least one piece of program code.
  • Historical jobs jobs that have been processed, or jobs that have been executed.
  • Specified resource amount the real-time resource amount required by the user-specified job during execution, or it can also be understood as the real-time resource amount required by the user-forecasted job during execution.
  • Predicted resource amount the amount of resource consumed in real time during the execution of the job predicted by the scheduling node.
  • Reference resource amount the average value of the actual consumed resource amount of historical jobs of the same type, which can also be called the baseline value (baseline) of the real-time resource amount consumed by each job of the same type during execution.
  • processor resources at least one of processor resources, memory resources, storage resources, and network resources required when a computer program runs (or executes).
  • the processor resource includes at least one of a central processing unit (central processing unit, CPU) resource, a graphics processing unit (graphics processing unit, GPU) resource, and an embedded neural network processor (neural-network processing unit, NPU) resource.
  • CPU central processing unit
  • GPU graphics processing unit
  • NPU embedded neural network processor
  • Storage resources include hard disk resources, and of course other types of storage resources may also be included.
  • Network resources include network bandwidth, and of course other types of network resources may also be included.
  • Idle resources resources that have not been used, or can also be understood as resources that are in an idle state.
  • Resource Amount The amount of resources consumed by the job. Taking the processor resource as CPU as an example, the resource amount of CPU is the number of CPU cores consumed by the job. Taking memory resources as an example, the amount of memory resources consumed by a job is the size of the memory consumed by the job, such as 1GB or 512MB. Taking storage resources as an example, the amount of storage resources consumed by a job is the size of the hard disk consumed by the job, such as 1GB or 512MB. Taking network resources as an example, the amount of network bandwidth is the size of the network bandwidth consumed by the job.
  • FIG. 1 is a schematic diagram of an HPC system provided by the present application.
  • the system 100 can also be called an HPC cluster, and the system 100 includes a scheduling node 101 and a computing node 102, wherein the computing node 101 in the system 100 There is at least one.
  • the scheduling node 101 or computing node 102 is a server cluster composed of one or more servers.
  • the terminal submits a job including a specified amount of resources to the scheduling node 101 .
  • the terminal is user equipment, which may also be referred to as a terminal station, user terminal, user device, access device, subscriber station, subscriber unit, mobile station, user agent, user equipment, portable terminal, laptop terminal, desktop terminal, etc. name.
  • the terminal can be a personal computer (such as a notebook computer, a tablet computer, a desktop computer, an ultrabook, etc.), a device with a web product interface (Website user interface, Web UI) function, a smart phone, a smart TV, Smart wearable devices, artificial intelligence (AI) products, smart cars, smart instruments, or Internet of things (IoT) terminals, etc.
  • a personal computer such as a notebook computer, a tablet computer, a desktop computer, an ultrabook, etc.
  • Website user interface, Web UI web product interface
  • smart phone a smart TV, Smart wearable devices, artificial intelligence (AI) products, smart cars, smart instruments, or Internet of things (
  • the scheduling node 101 is used to manage resource allocation of each computing node 102 in the system 100 .
  • the scheduling node 101 runs a scheduler based on the specified resource amount of the job submitted by the user, and queries the computing nodes 102 in the system 100 whose idle resource amount is greater than or equal to the specified resource amount.
  • the scheduling node 101 sends the job to the computing node 102, and the computing node 102 executes the job.
  • the present application provides a technical solution in which a scheduling node predicts the amount of resources consumed by a job in real time during execution according to the type of the job, and schedules a computing node to execute the job according to the predicted amount of resources. Since the amount of resources consumed by jobs of the same type in real time is relatively close during execution, the predicted amount of resources is closer to the actual amount of resources consumed by the job than the specified amount of resources, thereby improving the effective utilization of resources in computing nodes .
  • the scheduling node 101 is configured to provide a master service and an intelligent prediction service for the terminal, wherein the master service includes allocating resources for a job submitted by the terminal and scheduling the computing node 102 to execute the job submitted by the terminal.
  • the intelligent prediction service is a newly added service of the scheduling node 101, including predicting the resource amount of the job submitted by the terminal according to the type of the job.
  • the scheduling node 101 includes a job management module 11 and a resource scheduling module 12 to provide the main service.
  • the scheduling node 101 also includes a resource consumption prediction module 13 and a reference amount management module 14 to provide the intelligent prediction service.
  • the computing node 102 is configured to provide an agent service, where the agent service includes executing a job and counting the actual resource consumption of the job.
  • the proxy service is automatically provided by the job execution module 21 in the computing node 102 .
  • the job management module 11 When the job management module 11 receives the job submitted by the terminal, it requests the resource consumption prediction module 13 to predict the amount of resources consumed by the job in real time during execution. After receiving the request from the job management module 11, the resource consumption prediction module 13 determines the category of the job, and predicts the predicted resource amount of the job according to the category of the job, for example, the resource consumption prediction module 13 according to the reference resource amount and the specified resource amount for the job, forecast the forecasted resource amount for the job. The resource consumption prediction module 13 returns the predicted resource amount of the job to the job management module 11 as a prediction result. The job management module 11 requests the resource scheduling module 12 to allocate resources for the job according to the predicted resource amount of the job.
  • the resource scheduling module 12 allocates resources (that is, assigns computing nodes) to the job based on the predicted resource amount of the job, and sends the assignment result to the job management module 11.
  • the job management module 11 issues the job to the computing node 102 indicated by the allocation result.
  • the job execution module 21 in the computing node 102 executes the assigned job, and monitors the actual resource consumption of the job during the execution of the job, and returns the actual resource consumption of the job to the job management module 11 after the job is completed. The amount of resources consumed.
  • the job management module 11 sends the actual consumed resource amount of the job to the reference amount management module 14 .
  • the reference quantity management module 14 updates the reference resource quantity corresponding to the category of the job based on the actual resource consumption of the job, so that the resource consumption prediction module 13 acquires the updated reference resource of the category from the reference quantity management module 14 Amount, and then based on the updated reference resource amount of this category, the predicted resource amount of new jobs under this category can be predicted.
  • Step 301 the terminal sends a job to a scheduling node in a high-performance computing system.
  • the high-performance computing system is used to provide users with services such as job submission, scheduling, execution, and monitoring.
  • the process of the terminal sending the job to the scheduling node is also the process of the terminal submitting the job to the scheduling node.
  • the job sent by the terminal is any job to be executed in the high-performance computing system, that is, a job to be processed by the high-performance computing system.
  • the job includes a specified amount of resources for the job, where the specified amount of resources for the job includes a specified amount of resources for at least one resource required to execute the job, such as processor resources, memory resources
  • the designated resource amount of at least one of resources, storage resources, and network resources, processor resources, memory resources, storage resources, and network resources have been introduced above, and will not be repeated here.
  • the job further includes a job attribute of the job, where the job attribute indicates an attribute of the job.
  • the job attribute includes the queue identifier of the job queue where the job is located and the user type of the user to which the job belongs, that is, the queue identifier and the user type are attributes of the job respectively.
  • the jobs submitted by each user of the same user type are used to complete similar computing services, so the actual resource consumption of the jobs submitted by each user of the same user type has the same characteristics, for example, the jobs submitted by each user of the same user type All jobs consume more processor resources in real time during execution, for example, more CPU cores are consumed in real time.
  • the user type of the user indicates the business department in which the user belongs.
  • the jobs of each user belonging to the same user type are submitted to the same job queue in the scheduling node. Therefore, the actual resource consumption of each job in the same job queue has the same characteristics.
  • the job attribute further includes a user ID of the user to whom the job belongs, so as to indicate which user submitted the job.
  • the job attribute further includes a job tag (tag) of the job, wherein the job tag includes at least one of a service type, a service identifier, and a computing node tag of a computing service implemented by the job.
  • the compute node label indicates the user's preferred compute node for executing the job.
  • each item included in the job attribute is an attribute of the job, for example, the user type is an attribute, and the queue ID is an attribute.
  • the user issues a service execution instruction to the terminal, where the service execution instruction is used to instruct execution of the job, for example, the job execution instruction includes the job.
  • the terminal After receiving the service execution instruction, the terminal sends the job to the scheduling node based on the service execution instruction.
  • Step 302 the scheduling node receives the job.
  • the scheduling node maintains multiple job queues, each job queue corresponds to at least one user type, and each job queue is used to store jobs submitted by users of corresponding user types, that is, jobs provided by users of different user types May be submitted to the same job queue. Since there may be at least one user belonging to the same user type, correspondingly, each job queue corresponds to at least one user.
  • the scheduling node after the scheduling node receives the job, if the job includes the queue identifier of the job queue, the scheduling node submits the job to the job queue. If the job does not include the queue identifier or user type of the job queue, the scheduling node submits the job to a default job queue, where the default job queue may be a job queue corresponding to the user type of the user to which the job belongs. For example, if the job does not include a queue identifier, the scheduling node stores the job in the job queue corresponding to the user type based on the correspondence between the user type and the job queue.
  • the scheduling node determines the user type to which the user belongs based on the correspondence between the user ID and the user type, and sends the job Stored in the job queue corresponding to the user type.
  • the default job queue can also be a dedicated job queue set by the scheduling node for the user to which the job belongs, instead of the job queue corresponding to the user type of the user to which the job belongs.
  • the job does not include the queue of the job queue, but includes the user ID, then The scheduling node submits the job to the dedicated professional queue corresponding to the user identifier.
  • the scheduling node polls the read jobs from the plurality of job queues according to a preset polling rule.
  • the preset polling rule includes polling in descending order of the priorities of the job queues, and each job queue corresponds to a priority. The computing service priority corresponding to the jobs in the job queue with higher priority is higher, and the computing service priority corresponding to the job in the job queue with lower priority is lower.
  • the preset polling rule includes that each job queue has no priority, and each job queue is polled sequentially.
  • the scheduling node After receiving the job A, store the job A in the job queue 1. After polling the job queue 1, and the job A is at the beginning of the job queue 1, the scheduling node reads from the job queue 1 Take the assignment A.
  • the scheduling node includes a communication module and a job management module, the communication module receives the job, and the job management module submits the job to a job queue and reads the job from the job queue.
  • Step 303 the scheduling node receives a mode adjustment instruction, and the mode adjustment instruction is used to instruct to turn on the prediction mode, and the function of predicting the amount of resources required by the job is provided in the prediction mode.
  • the forecasting mode refers to a working mode in which the forecasted resource amount of the job is used as the job scheduling resource.
  • the mode adjustment instruction includes a forecasting parameter, and the forecasting parameter is used to instruct the scheduling node to A function of providing resource scheduling for jobs in a working mode, wherein the predictive mode is a working mode of a scheduling node.
  • the prediction parameter is a first prediction parameter, and the first prediction parameter indicates the function of providing the resource amount required by the prediction job in the prediction mode, and the mode adjustment instruction including the first prediction parameter is used to instruct to turn on the prediction mode, In this forecast mode, the function of forecasting the amount of resources required by the job is provided.
  • the working mode of the scheduling node includes not only the predictive mode, but also a non-predictive mode, wherein the non-predictive mode refers to a working mode in which resources are scheduled for a job through the specified resource amount of the job .
  • the prediction parameter is a second prediction parameter, and the second prediction parameter indicates that the function of resource scheduling is provided in the non-prediction mode. Provide resource scheduling function in predictive mode.
  • the first prediction parameter and the second prediction parameter may have different representations, for example, the first prediction parameter is “true” and the second prediction parameter is “false”.
  • the technician inputs a mode adjustment command in the configuration interface of the scheduling node, and issues the input mode scheduling command to the scheduling node through the configuration interface, and accordingly, the scheduling node receives the Mode dispatch instructions.
  • the mode adjustment instruction including the first prediction parameter is recorded as the first mode adjustment instruction
  • the mode adjustment instruction including the second prediction parameter is recorded as the second mode adjustment instruction.
  • the embodiment shown in FIG. 3 is described by taking the mode adjustment instruction received by the scheduling node as the first mode adjustment instruction as an example.
  • Step 304 the scheduling node starts the prediction mode according to the received mode adjustment instruction.
  • the prediction mode and the non-prediction mode are realized by turning on or off the prediction software, and the prediction software is used to provide a function of predicting the amount of resources required by the job.
  • the forecasting software is turned on, the scheduling node works in the forecasting mode, and when the forecasting software is turned off, the scheduling node works in the non-predicting mode.
  • the scheduling node enables (enable) the application programming interface (application programming interface) of the prediction software , API), the scheduling node works in predictive mode. If the prediction parameter in the received mode adjustment instruction is the second prediction parameter, it means that the mode adjustment instruction is the second mode adjustment instruction, then the scheduling node disables (disable) the API of the prediction software, and the scheduling node works in the non-prediction mode .
  • the scheduling node when the scheduling node is initialized, the predictive mode has been enabled, and there is no need for a technician to enable it through a mode adjustment command.
  • the processes shown in the following steps 305-308 are processes in which the scheduling node acquires the predicted resource amount of the job in the prediction mode. If the received mode adjustment instruction is the second adjustment instruction, the scheduling node does not execute the following steps 305-308 after receiving the job, but executes the following step 309. In addition, the scheduling node may also execute steps 303-304 first, and then execute step 302. This application does not limit the execution order of steps 302 and 303-304. Only one working mode of the scheduling node can be turned on, and there is no need to turn it on repeatedly.
  • Step 305 the scheduling node determines the type of the job.
  • the scheduling node records correspondences between multiple categories and job attributes, and each category is indicated by a category identifier.
  • Each category corresponds to a job attribute
  • the job attributes corresponding to each category include the same attributes of each job under the same category, or include all attributes of each job under the same category.
  • the job attributes corresponding to each category include at least one of queue identifier, user type, user identifier and job label. Taking category 1 as an example, jobs 1-2 belong to category 1.
  • the job attributes of job 1 include queue ID 1, user type 1, and user ID 1.
  • the job attributes of job 2 include queue ID 1, user type 1, and user ID 2. .
  • the job attributes corresponding to category 1 include queue ID 1 and user type 1.
  • the job attributes corresponding to category 1 include the same attributes of job 1 and job 2. If the job attributes of the new job include queue ID 1 and user type 1 , the category of the new job is category 1. Or, the job attributes corresponding to category 1 include queue ID 1, user type 1, and user IDs 1-10. At this time, the job attributes corresponding to category 1 include all attributes of job 1 and job 2. If the job attributes of the new job include queue ID 1, user type 1, and user ID 3, the category of the new job is category 1.
  • the same category corresponds to at least one job queue
  • different users under the same user type can submit jobs with the same job label to the same job queue, and submit jobs with different job labels to different job queues. Queues, therefore, jobs submitted by different users with different job labels may belong to different categories.
  • the jobs in the same job queue may come from users of different user types, therefore, the queue identifiers in the job attributes corresponding to different categories may also be the same.
  • a first category relationship table is stored in the scheduling node, and the first category relationship table records correspondences between multiple categories and job attributes, as shown in Table 1 below.
  • the job attributes corresponding to category 1 include job queue 1, user type 1, user IDs 1-1 to 1-5, and job labels 1-A to 1-C.
  • job attributes corresponding to category 1 includes five users, namely users 1-1 to 1-5.
  • job tags 1-A to 1-C There are three types of job tags for job submissions submitted by users under user type 1, which are job tags 1-A to 1-C. Each job submission under user type 1 is submitted to job queue 1. Jobs submitted by users under user type 1 belong to category 1.
  • the scheduling node determines the category of the job from multiple categories based on the job attribute of the job, and at least one attribute of each job of the same category is the same. For example, the scheduling node determines the category corresponding to the job attribute of the job among the multiple categories as the category of the job based on the correspondence between the multiple categories and job attributes. Wherein, for the category of the job, the job attributes of the category include at least one attribute of the job attributes of the job.
  • the job attribute of the job is recorded as the first job attribute
  • the job attributes corresponding to the multiple categories of records are recorded as the second job attribute.
  • the scheduling node queries the second job attributes including attributes in the first job attributes among the plurality of second job attributes. If at least one second job attribute is found to include attributes in the first job attribute, the scheduling node takes the second job attribute that includes the most attributes in the first job attribute among the at least one second job attribute as the first job attribute.
  • the second job attribute corresponding to the job attribute, and the type corresponding to the second job attribute corresponding to the first job attribute is used as the type of the job.
  • the job is job A submitted by user 1-1 in user type 1, and the correspondence between types and job attributes is shown in Table 1.
  • the scheduling node queries the above-mentioned table 1, and finds that category 1 and category 2 in table 1 correspond The job properties of all include job queue 1.
  • the job attributes corresponding to category 1 in Table 1 include user type 1
  • the job attributes excluding category 2 include user type 1, that is, the category in table 1
  • the job attribute corresponding to 1 includes the most attributes among the job attributes of the job A, then the scheduling node takes category 1 as the category of job A.
  • the first category relationship table is stored in the memory of the scheduling node, and the scheduling node may read the first category relationship table from the memory, and then perform step 304 .
  • the scheduling node when the job attribute of the job also includes the job category identifier, the scheduling node does not need to determine the job category based on the correspondence between the job category and the job attribute, but based on the job The category ID in the attribute from which to determine the category of this job.
  • Step 306 the scheduling node obtains the corresponding relationship between the multiple types of records and the amount of reference resources.
  • the actual resource consumption amount of each job under each category has the same characteristics. For example, each job under the same category consumes more/less processor resources in real time, or consumes more/less storage resources in real time.
  • the reference computing amount corresponding to each category is obtained based on the actual resource consumption of historical jobs of each category.
  • Each category corresponds to a reference resource amount group, and a reference resource amount group includes reference resource amounts of at least one resource, such as reference resource amounts of at least one resource among processor resources, memory resources, storage resources, and network resources.
  • the reference resource amount of any one resource in the at least one resource is obtained based on the actual consumed resource amount of any one resource of a category of historical jobs.
  • a second category relationship table is stored in the scheduling node, and the second category relationship table records the correspondence between the multiple categories and the amount of reference resources.
  • Table 2 shows that the reference resource amount group corresponding to category 1 includes reference resource amount 1 of processor resources, reference resource amount 2 of storage resources, and reference resource amount 3 of network resources.
  • the second category relationship table is stored in the memory of the scheduling node, and the scheduling node can read the second category relationship table from the memory, so that the scheduling node reads the second category relationship table from the category relationship table , to obtain the corresponding relationship between multiple categories of records and the amount of reference resources.
  • the second category relationship table and the first category relationship table are combined into one category relationship table and stored in the memory, and the scheduling node can read the combined category relationship table from the memory, and from the combined The corresponding relationship between multiple categories recorded in the category relationship table and the amount of reference resources is obtained.
  • the category identifier of each category may be the queue identifier of the job queue corresponding to each category, or other identifiers other than the queue identifier used to identify the category.
  • the embodiment of the present application There is no restriction on the representation of the category.
  • the scheduling node may record the correspondence between the multiple categories and the reference calculation amount through the process shown in the following steps A-B.
  • Step A For each category, the scheduling node determines the average value of the actual resource consumption of historical jobs of each category as the reference resource quantity of each category.
  • the scheduling node obtains the average value of the actual resource consumption of the resource in the historical jobs of the category as the reference resource amount of the various resources in the category.
  • Step B The scheduling node records the correspondence between each category and the amount of reference resources of each category.
  • the scheduling node records the category identification of each category and the reference resource amount of each category in the second category relationship table, and stores the second category relationship table in the database.
  • the scheduling node can also The second category relationship table is loaded from the database into the memory, so that the subsequent scheduling nodes can directly and quickly read the second category relationship table from the memory.
  • the scheduling node may first store the second category relationship table in the memory, and then the scheduling node may also store the second category relationship table in the database, so as to avoid data loss in the memory.
  • Step 307 the scheduling node determines the reference resource amount corresponding to the job type based on the recorded correspondence between multiple categories and the reference resource amount.
  • the scheduling node acquires the amount of reference resources corresponding to the category from the second category relationship table.
  • Step 308 the scheduling node predicts the real-time resource consumption during the execution of the job based on the specified resource amount and the reference resource amount corresponding to the job type, and obtains the predicted resource amount of the job.
  • the predicted resource amount of the job includes a predicted resource amount of at least one resource, for example, includes a predicted resource amount of at least one resource among processor resources, storage resources, and network resources.
  • the scheduling node if the preset prediction parameter indicates that the function of predicting the amount of resources required by the job is provided in the prediction mode, the scheduling node, based on the specified amount of resources and the amount of reference resources corresponding to the type of the job, Predict the amount of resources consumed in real time during the execution of the job. That is, if the preset prediction parameter indicates that the function of predicting the amount of resources required by the job is provided in the prediction mode, the scheduling node executes step 307 .
  • the amount of real-time resources required during the execution of historical jobs of this category is the sample data of this category, and the amount of reference resources corresponding to this category is determined by the actual resource consumption of historical jobs in this category, if the If the number of samples (that is, the number of historical jobs) is sufficient, it is considered that the reference resource amount obtained based on the actual resource consumption amount of sufficient samples is closer to the actual resource consumption amount of the new job. Therefore, in a possible implementation manner, when predicting the predicted resource amount of a job, the number of historical jobs is also considered.
  • the scheduling node obtains the number of historical jobs of the job category, and if the number of historical jobs of the job category is greater than or equal to the number threshold, the scheduling node based on the specified amount of resources and the reference resources corresponding to the job category To predict the amount of resources consumed in real time during the execution of the job.
  • the scheduling node executes step 308.
  • the scheduling node does not execute this step 308, but executes the following step 309.
  • the scheduling node based on the specified amount of resources, the amount of reference resources corresponding to the type of the job, and the number of historical jobs of the type, The average amount of resources consumed in real time is predicted. Wherein, the predicted average amount of resources consumed in real time during execution of each job of the category is also the predicted resource amount of the job.
  • the scheduling node is based on the specified resource quantity of the any kind of resource, the reference resource quantity of the any kind of resource corresponding to the category, and the historical job of the category number, the resource amount of any type of resource consumed in real time during the execution of the job is predicted to obtain the predicted resource amount of any type of resource for the job.
  • the predicted resource amount of any resource of the job is also a predicted average resource amount of any resource consumed in real time during execution of each job of the category.
  • the scheduling node takes the number of historical jobs of the category, the specified resource amount of any type of resource, and the reference resource amount of any type of resource corresponding to the type as input parameters, and enters the following formula (1):
  • P is the predicted resource quantity of any resource of the job
  • baseline is the reference resource quantity of any resource of the job
  • REQ is the specified resource quantity of any resource of the job
  • N is the category of the job
  • the number of historical jobs, N is greater than the number threshold.
  • the formula (1) is also the function realized by the forecasting software. For example, the scheduling node inputs the input parameters into the API of the forecasting software, and the forecasting software calculates the formula (1) based on the input parameters, and reports to the scheduling The node outputs the predicted resource amount P of any resource of the job.
  • the scheduling node can directly use the formula (1) to calculate the predicted resource amount of any resource, At this time, N may also be smaller than the number threshold.
  • step 308 is an implementation manner in which the scheduling node predicts the amount of resources consumed in real time during the execution of the job according to the amount of reference resources corresponding to the category.
  • the computer device uses the reference resource amount corresponding to the category as the predicted resource amount of the job.
  • the scheduling node calculates the real-time resource consumption during the execution of the job according to the reference resource amount corresponding to the type, that is, the scheduling node predicts the predicted resource amount of the job according to the type of the job.
  • Step 309 the scheduling node takes the specified resource amount as the predicted resource amount of the job.
  • Step 310 the scheduling node schedules computing nodes in the high-performance computing system to execute the job based on the predicted amount of resources.
  • the scheduling node records idle resource amounts of multiple computing nodes, wherein the multiple computing nodes belong to the high-performance computing system.
  • the idle resource amount of a computing node is the resource amount of the idle resource of the computing node at the current moment.
  • the idle resource amount of a computing node includes the idle resource amount of at least one resource.
  • the scheduling node predicts the computing node with the best performance in executing the job in the high-performance computing system based on the predicted resource amount and the recorded idle resource amount of each computing node, and uses the predicted compute nodes to execute the job. For example, the process shown in the following steps 3101-3105.
  • Step 3101 the scheduling node updates the specified resource amount in the job to the predicted resource amount of the job.
  • this step 3101 is an optional step. For example, if the scheduling node executes the above step 308, then execute this step 3101. If the scheduling node does not execute the above step 308, it means that the predicted resource amount of the job is , the scheduling node does not execute step 3101.
  • Step 3102 the scheduling node determines a target computing node from multiple computing nodes, and the idle resource amount of the target computing node is greater than or equal to the predicted resource amount of the job.
  • the target computing node is the computing node scheduled by the scheduling node for the job, or it can also be understood that the computing node allocated by the scheduling node for the job, that is, the effect of executing the job in the high-performance computing system predicted by the scheduling node The best compute nodes.
  • the scheduling node inquires whether there is a computing node whose idle resource amount is greater than or equal to the predicted computing resource amount of the job in the resource amount statistics table, and if it exists, the scheduling node will query to The compute node of is determined as the target compute node. Wherein, the amount of idle resources of the plurality of computing nodes is recorded in the resource amount statistics table.
  • the scheduling node determines any computing node as the target computing node.
  • the scheduling node may also perform step 3102 first, and then step 3101.
  • this embodiment of the present application does not limit the execution order of steps 3101 and 3102.
  • Step 3103 the scheduling node sends the updated job to the target computing node.
  • Step 3104 the target computing node receives the updated job.
  • Step 3105 the target computing node executes the updated job.
  • Step 311 during the execution of the job, the scheduled computing node acquires the actual resource consumption of the job.
  • the scheduled computing node is also the target computing node.
  • the actual consumed resource amount of the job includes the actual consumed resource amount of at least one resource of the job. For example, at least one of the actual resource consumption of at least one resource among processor resources, storage resources, and network resources.
  • this step 311 is implemented by the process shown in the following steps 3111-3113.
  • Step 3111 For any one of the at least one resource, during the execution of the job, the computing node collects the resource amount of any resource consumed by the job in real time at intervals of preset time intervals to obtain the Multiple sample resource quantities of any kind of resource.
  • the preset duration may be set according to an actual application scenario, and here, the embodiment of the present application does not limit the preset duration.
  • Each sampled resource amount of any resource is the resource amount of any resource consumed by the job at a moment. If any one of the resources is a CPU resource in the processor resource, a sampling resource amount of the CPU resource is the number of CPU cores consumed by the job at a time; if the any one resource is a memory resource in the storage resource, A sampling resource amount of the memory resource is the memory size consumed by the job at one moment. If any one of the resources is network bandwidth, a sample resource amount of the network bandwidth is the size of the network bandwidth consumed by the job at one moment.
  • the computing node periodically samples the CPU usage of the job at preset intervals to obtain multiple CPU resources. occupancy rate, the computing node multiplies each CPU occupancy rate by 100 to obtain a sampled amount of CPU resources (that is, the number of CPU cores) of the job.
  • the number of CPU cores is an integer value. Therefore, if a value obtained by multiplying a CPU usage rate by 100 is an integer value, the computing node uses this value as a sample resource amount of the CPU resource. If the value obtained by multiplying a CPU usage rate by 100 is not an integer value, the computing node adds 1 to the integer value of the value to obtain a sample resource amount of the CPU resource.
  • FIG. 4 is a schematic diagram of processor resource occupation provided by the present application.
  • the CPU usage of the job is 200% in one sampling, and the sampling resources of the job sampled this time are about It is 2 CPU cores (that is, 2 cores of the CPU).
  • the computing node uses 16 threads to execute a certain job, the CPU usage rate of the job is 395.3% in one sampling, then the sampling resource amount of the job sampled this time is About 4 CPU cores.
  • the computing node binds 2 CPU cores for the job, that is, two specific CPU cores are used to provide computing services for the job.
  • the computing node uses 16 threads to execute the job, During one sampling, the CPU usage of the job is always 200%, so the sampling resources of the job sampled this time are about 2 CPU cores.
  • Step 3112 the computing node obtains the average sampled resource amount and the maximum sampled resource amount of any one resource based on the multiple sampled resource amounts of any resource.
  • the average sampled resource amount of any kind of resource is the average value of multiple sampled resource amounts of any kind of resource
  • the maximum sampled resource amount of any kind of resource is the largest among the multiple sampled resource amounts of any kind of resource. value.
  • Step 3113 the computing node obtains the actual consumed resource amount of any resource of the job based on the average sampled resource amount and the maximum sampled resource amount.
  • the actual consumed resource amount of any resource is greater than or equal to the average sampled resource amount and less than or equal to the maximum sampled resource amount, that is, the actual consumed resource amount of any resource is between the average sampled resource amount and the maximum sampled resource amount. between sample resource volumes.
  • the weight corresponding to any resource on each computing node is greater than Any value equal to 0 and less than or equal to 1.
  • the weight indicates a preference for the average sampled resource amount and the maximum sampled resource amount when calculating the actual consumed resource amount of any resource.
  • the calculation node takes the average sampling resource amount and the maximum sampling resource amount as input parameters, and inputs them into the following formula (2):
  • u is the actual resource consumption of any resource of the job
  • w is the weight corresponding to the resource
  • u max is the maximum sampling resource of the resource of the job
  • u avg is the resource The average sampled amount of any resource for the job.
  • the target computing node may also quantify the actual resource consumption u of any resource, and use the quantified actual resource consumption u as the The final actual resource consumption of any resource.
  • Step 312 After the job is executed, the computing node sends the actual resource consumption of the job to the scheduling node.
  • the computing node After the execution of the job is completed, the computing node generates a job completion response of the job, and sends the job completion response to the scheduling node.
  • the job completion response indicates that the job has been executed, and the job completion response includes the actual resource consumption of the job.
  • Step 313 the scheduling node receives the actual resource consumption of the job.
  • the scheduling node receives the job completion response of the job. After receiving the job completion response of the job, it means that the job has been executed, and the scheduling node parses the job completion response from the job completion response. The actual resource consumption of this job.
  • the scheduling node if the scheduling node records the correspondence between the job category and the job (that is, the job belongs to a category), that is, the scheduling node knows the job category, then The scheduling node executes the following step 314. If the scheduling node does not record the correspondence between the job category and the job, the scheduling node determines the job category from multiple categories (that is, the above-mentioned step 305 The process shown), when the type of the job is determined, the scheduling node then executes the following step 314.
  • Step 314 the scheduling node updates the reference resource amount corresponding to the category based on the actual resource consumption of the job and the number of historical jobs in the category of the job.
  • the scheduling node will use the actual resource consumption of the any one resource of the job, the number of historical jobs of the category, and the The corresponding reference resource amount of any resource is used as an input parameter and input to the following formula (3):
  • baseline' (baseline*N+REQ)/(N+1)
  • baseline' is the updated reference resource amount of any resource.
  • Step 315 the scheduling node records the updated correspondence between the reference resource quantity and the job category.
  • the scheduling node modifies the reference resource quantity corresponding to the category in the second category relationship table in the memory and/or the database to the updated reference resource quantity, so that the subsequent scheduling node can directly read the first resource quantity from the memory.
  • the database involved in this application is implemented by a persistent storage medium, and the database can be located in the scheduling node or a storage node outside the scheduling node.
  • the scheduling node in the high-performance computing system receives the job to be processed, it predicts the amount of resources consumed in real time during the execution of the job according to the type of the job. Therefore, compared with the specified resource amount, the predicted resource amount is closer to the actual resource consumption of the job, and the scheduling node schedules the computing nodes in the high-performance computing system to execute the job according to the predicted resource amount During the process, the effective utilization of resources in the computing nodes can be improved, and the efficiency of job processing can be improved.
  • the scheduling node predicts the amount of resources consumed in real time during the execution of the job according to the type of the job, it can predict the amount of resources consumed in real time during the execution of the job according to the amount of reference resources corresponding to the type.
  • the quantity is obtained from the actual resource consumption of historical jobs in this category. Therefore, the predicted resource quantity predicted according to the reference resource quantity corresponding to this category is closer to the actual resource consumption quantity of the job.
  • the computing nodes in the high-performance computing system execute jobs, the effective utilization of resources in the computing nodes can be further improved.
  • the scheduling node after the scheduling node obtains the reference resource amount of the category according to the actual resource consumption of the historical jobs of the category, it can store the corresponding relationship between the reference resource quantity of the category and the category in the memory, so that the subsequent When predicting the predicted amount of resources for a job of this type, the scheduling node can read the amount of reference resources corresponding to this type from the memory, without calculating the amount of reference resources corresponding to this type in real time based on the actual resource consumption of historical jobs of this type , so that the reading efficiency of the reference resource amount can be improved.
  • FIG. 5 is a schematic diagram of a job processing flow provided by the present application.
  • the intelligent prediction module in the scheduling node loads the prediction parameters. If the prediction parameter is the first prediction parameter, it means that the scheduling node is working in the prediction mode.
  • the terminal submits a job to the scheduling node, and when the scheduling module in the scheduling node receives the job, it requests the intelligent prediction module to predict the predicted resource amount of the job.
  • the intelligent prediction module determines the category of the job, and inquires about the reference resource amount corresponding to the category of the job, based on the reference resource amount and the specified resource amount, calculates the predicted resource amount of the job (such as predicting the number of CPU cores required by the job, The size of memory, the size of network bandwidth), the intelligent prediction module returns the predicted resource amount of the job as the prediction result to the scheduling module.
  • the scheduling module queries computing nodes whose amount of idle resources (for example, the number of idle CPU cores and the size of free memory space) is greater than or equal to the amount of predicted resources, and the scheduling module schedules at least one computing node found to execute a job, for example, the job Send it to the queried computing nodes.
  • the agent module in the computing node When the agent module in the computing node receives the job issued by the scheduling node, it starts to execute the job, and periodically samples the amount of resources consumed by the job in real time, and counts the actual resource consumption of the job (such as the number of CPU cores actually consumed, memory size or network bandwidth), the proxy module returns the actual resource consumption of the job to the scheduling module.
  • the reference resource amount corresponding to the category of the job is updated by the intelligent prediction module based on the actual consumed resource amount of the job. Users can query the actual resource consumption of jobs through the terminal.
  • the scheduling module in FIG. 5 may be an integrated module of the job management module 11 and the resource scheduling module 12 in FIG. 2 .
  • the intelligent prediction module may be an integrated module of the resource consumption prediction module 13 and the reference amount management module 14 in FIG. 2 .
  • the process of the scheduling node predicting the resource amount of the job and the process of updating the reference resource amount are respectively executed by different threads of the scheduling node.
  • FIG. 6 a flow chart of different threads predicting resource amounts and updating reference resource amounts provided by the present application, wherein, initially, the first thread configuration module obtains a predictive parameter, if the predictive parameter is the first predictive parameter , then the first thread works in predictive mode, and the first thread waits for new jobs in predictive mode.
  • the first thread pair After receiving the job submitted by the terminal, the first thread pair determines the category of the job, reads the reference resource amount corresponding to the category from the memory, and according to the read reference resource amount and the specified resource amount of the job, Forecast the forecast resource amount for this job.
  • the first thread updates the specified resource amount in the job to the predicted resource amount of the job, and schedules computing nodes to execute the job based on the predicted resource amount.
  • the computing node returns the actual resource consumption of the job to the scheduling node, and the scheduling node sends the actual resource consumption of the job to the second thread.
  • the second thread receives the actual resource consumption of the job, it reads the job attributes and reference resource amounts corresponding to each category from the memory.
  • the second thread also reads unstated job attributes (that is, job attributes of newly completed jobs) from the memory.
  • the second thread determines the category of the job based on the job attributes of the newly completed job.
  • the second thread updates the reference resource amount of the job category based on the predicted resource amount of the job, or persistently stores the updated reference resource amount and the job attribute of the job in the hard disk, and stores the updated
  • the reference calculation resource amount is stored in the database, and then the updated reference resource amount is loaded into the memory from the database or the hard disk, so that the subsequent first thread can query the latest reference resource amount of this category.
  • Fig. 7 is a schematic structural diagram of a job processing device provided by the present application.
  • the device 700 may be a part of the scheduling node in the previous embodiments or Fig. 1-3 or Fig. 5-6, and is used to execute In the method of execution, the device 700 is configured as a scheduling node in a high-performance computing system, and the device 700 includes:
  • a receiving unit 701 configured to receive a job to be processed by the high-performance computing system
  • a determining unit 702 configured to determine the category of the job
  • a prediction unit 703 configured to predict a predicted resource amount of the job according to the category, where the predicted resource amount is the predicted resource amount consumed in real time during the execution of the job;
  • a scheduling unit 704, configured to schedule computing nodes in the high-performance computing system to execute the job according to the predicted amount of resources.
  • the apparatus 700 of the embodiment of the present invention may be implemented by a central processing unit (central processing unit, CPU), or by an application-specific integrated circuit (ASIC), or a programmable logic device (programmable logic device, PLD) implementation
  • the above-mentioned PLD can be a complex program logic device (complex programmable logical device, CPLD), field-programmable gate array (field-programmable gate array, FPGA), general array logic (generic array logic, GAL ) or any combination thereof.
  • the device 700 and its modules can also be software modules.
  • the prediction unit 703 is configured to: predict the amount of resources consumed in real time during the execution of the job according to the amount of reference resources corresponding to the category, to obtain the amount of predicted resources, and The reference resource amount is obtained based on the actual consumed resource amount of the historical jobs of the category.
  • the device 700 further includes an adjustment unit
  • the receiving unit 701 is also used to receive a mode adjustment instruction, the mode adjustment instruction is used to instruct to turn on the prediction mode, and the function of predicting the amount of resources required by the job is provided in the prediction mode;
  • the adjustment unit is further configured to enable the prediction mode according to the mode adjustment instruction
  • the prediction unit 703 is configured to predict, in the prediction mode, the amount of resources consumed in real time during the execution of the job according to the reference amount of resources corresponding to the category.
  • the prediction unit 703 is configured to: in the prediction mode, obtain the number of historical jobs of the category; if the number of historical jobs of the category is greater than or equal to the number threshold, Then, according to the reference resource amount corresponding to the category, the resource amount consumed in real time during the execution of the job is predicted.
  • the job includes a specified resource amount of the job
  • the prediction unit 703 is configured to: based on the specified resource amount, the reference resource amount corresponding to the category, and the history of the category
  • the number of jobs is to predict the average amount of resources consumed in real time during the execution of each job of the category.
  • the device 700 further includes:
  • an obtaining unit configured to obtain the average value of the actual resource consumption of the historical jobs of the category as the reference resource quantity of the category
  • the first recording unit is configured to record the corresponding relationship between the category and the amount of reference resources of the category.
  • the job includes a job attribute of the job, and the job attribute indicates an attribute of the job;
  • the determining unit 702 is configured to:
  • the category of the job is determined from a plurality of categories based on the job attribute of the job, and at least one attribute of each job of the same category is the same.
  • the job attributes include the user type of the user to which the job belongs and the queue identifier of the job queue where the job is located; the determining unit 702 is configured to: based on the multiple categories and job attributes According to the corresponding relationship between the multiple categories, the category corresponding to the job attribute among the multiple categories is determined as the category of the job, and each of the multiple categories corresponds to a job attribute.
  • the device 700 further includes an updating unit and a second recording unit;
  • the receiving unit is further configured to receive the actual resource consumption amount of the job from the computing node;
  • the updating unit is configured to update the reference resource amount based on the actual consumed resource amount and the number of historical jobs of the category;
  • the second recording unit is configured to record the updated correspondence between the reference resource amount and the category.
  • the second recording unit and the first recording unit may be the same unit or different units.
  • the device 700 corresponds to the scheduling node in the above method embodiment, and each module in the device 700 and the above-mentioned other operations and/or functions are to implement various steps and methods implemented by the scheduling node in the method embodiment, specifically For details, reference may be made to the foregoing method embodiments, and for the sake of brevity, details are not repeated here.
  • the device 700 when the device 700 is processing jobs, it only uses the division of the above-mentioned functional modules as an example for illustration. function modules to complete all or part of the functions described above.
  • the device 700 provided in the above-mentioned embodiment belongs to the same idea as the above-mentioned method embodiment, and its specific implementation process is detailed in the above-mentioned method embodiment, and will not be repeated here.
  • apparatus 700 may be equivalent to an execution component in the scheduling node 101 in the system 100 .
  • FIG. 8 is a schematic structural diagram of a computer device provided by the present application.
  • the computer device 800 may be a scheduling node involved in the content described in part in FIGS. 1-3 or 5-6.
  • the computer device 800 includes at least one processor 801 , a communication bus 802 , a memory 803 and at least one communication interface 804 .
  • Processor 801 can be a general central processing unit (central processing unit, CPU), network processor (Network Processor, NP), microprocessor, microcontroller (microcontroller unit, MCU), digital signal processor (digital signal processing , DSP), or artificial intelligence processors and other computing devices that run software, each computing device may include one or more cores for executing software instructions to perform calculations or processing.
  • the processor can be built into SoC (system on chip), or can be one or more integrated circuits for realizing the application scheme, for example, application-specific integrated circuit (application-specific integrated circuit, ASIC), programmable logic device (programmable logic device, PLD) or a combination thereof.
  • the aforementioned PLD may be a complex programmable logic device (complex programmable logic device, CPLD), a field-programmable gate array (field-programmable gate array, FPGA), a general array logic (generic array logic, GAL) or any combination thereof.
  • complex programmable logic device complex programmable logic device, CPLD
  • field-programmable gate array field-programmable gate array
  • GAL general array logic
  • the communication bus 802 is used to transfer information between the above-mentioned components.
  • the communication bus 802 can be divided into an address bus, a data bus, a control bus, and the like. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.
  • the memory 803 may include read-only memory and random-access memory, and provides instructions and data to the processor 801 .
  • Memory 803 may also include non-volatile random access memory.
  • the memory 803 may also store device type information.
  • Memory 203 may be volatile memory or nonvolatile memory, or may include both volatile and nonvolatile memory.
  • the non-volatile memory can be read-only memory (read-only memory, ROM), programmable read-only memory (programmable ROM, PROM), erasable programmable read-only memory (erasable PROM, EPROM), electrically programmable Erases programmable read-only memory (electrically EPROM, EEPROM) or flash memory.
  • Volatile memory can be random access memory (RAM), which acts as external cache memory.
  • RAM random access memory
  • SRAM static random access memory
  • DRAM dynamic random access memory
  • SDRAM synchronous dynamic random access memory
  • Double data rate synchronous dynamic random access memory double data date SDRAM, DDR SDRAM
  • enhanced SDRAM enhanced synchronous dynamic random access memory
  • SLDRAM synchronous connection dynamic random access memory
  • direct rambus RAM direct rambus RAM
  • the memory 803 may exist independently, and is connected to the processor 801 through the communication bus 802 .
  • the memory 803 can also be integrated with the processor 801.
  • the Communication interface 804 utilizes any transceiver-like device for communicating with other devices or a communication network.
  • the communication interface 804 includes a wired communication interface, and may also include a wireless communication interface.
  • the wired communication interface may be an Ethernet interface, for example.
  • the Ethernet interface can be an optical interface, an electrical interface or a combination thereof.
  • the wireless communication interface may be a wireless local area network (wireless local area networks, WLAN) interface, a cellular network communication interface or a combination thereof.
  • the processor 801 may include one or more CPUs, such as CPU0 and CPU1 shown in FIG. 8 .
  • a computer device may include multiple processors, such as a processor 801 and a processor 805 as shown in FIG. 8 .
  • processors can be a single-core processor (single-CPU) or a multi-core processor (multi-CPU).
  • a processor herein may refer to one or more devices, circuits, and/or processing cores for processing data such as computer program instructions.
  • the computer device may further include an output device 806 and an input device 807 .
  • Output device 806 is in communication with processor 801 and can display information in a variety of ways.
  • the output device 806 may be a liquid crystal display (liquid crystal display, LCD), a light emitting diode (light emitting diode, LED) display device, a cathode ray tube (cathode ray tube, CRT) display device, or a projector (projector), etc.
  • the input device 807 communicates with the processor 801 and can receive user input in various ways.
  • the input device 808 may be a mouse, a keyboard, a touch screen device, or a sensory device, among others.
  • the memory 803 is used to store the program code 810 for implementing the solution of the present application, and the processor 801 can execute the program code 810 stored in the memory 803 . That is, the computer device 800 can implement the method provided in the embodiment of FIG. 3 above through the processor 801 and the program code 810 in the memory 803 .
  • the above-mentioned embodiments may be implemented in whole or in part by software, hardware, firmware or other arbitrary combinations.
  • the above-described embodiments may be implemented in whole or in part in the form of computer program products.
  • the computer program product includes one or more computer instructions. When the computer program instructions are loaded or executed on the computer, the processes or functions according to the embodiments of the present application will be generated in whole or in part.
  • the computer may be a general purpose computer, a special purpose computer, a computer network, or other programmable devices.
  • the computer instructions may be stored in or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website, computer, server or data center Transmission to another website site, computer, server, or data center by wired (eg, coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (eg, infrared, wireless, microwave, etc.).
  • the computer-readable storage medium may be any available medium that can be accessed by a computer, or a data storage device such as a server or a data center that includes one or more sets of available media.
  • the available media may be magnetic media (eg, floppy disk, hard disk, magnetic tape), optical media (eg, DVD), or semiconductor media.
  • the semiconductor medium may be a solid state drive (SSD).
  • the embodiment of the present application also provides a computer program product or computer program, the computer program product or computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium, and the processor of the computer device reads from the computer-readable storage medium The computer instruction is read, and the processor executes the computer instruction, so that the computer device executes the above job processing method.
  • an embodiment of the present application also provides a device, which may specifically be a chip, a component or a module, and the device may include a connected processor and a memory; wherein the memory is used to store computer-executable instructions, and when the device is running, The processor can execute the computer-executable instructions stored in the memory, so that the chip executes the job processing methods in the above method embodiments.
  • the device, equipment, computer-readable storage medium, computer program product or chip provided in this embodiment are all used to execute the corresponding method provided above, therefore, the beneficial effects it can achieve can refer to the above-mentioned provided The beneficial effects of the corresponding method will not be repeated here.

Landscapes

  • Engineering & Computer Science (AREA)
  • Software Systems (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Management, Administration, Business Operations System, And Electronic Commerce (AREA)

Abstract

一种作业处理的方法，本方法由高性能计算系统中的调度节点在接收到待处理的作业后，根据作业的类别，预测该作业执行过程中实时消耗的资源量，由于同一类别的作业在执行过程中实时消耗的资源量比较接近，因此，相比指定资源量，预测出的资源量更接近作业的实际消耗资源量，则调度节点根据预测出的资源量，调度高性能计算系统中的计算节点执行该作业的过程中，能够提高计算节点中资源的有效利用率，提高作业处理的效率。

Description

作业处理的方法、装置、计算机设备以及存储介质
本申请要求于2021年06月30日提交的申请号为202110739273.8、发明名称为“动态调整作业诉求的调度方法和电子设备”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
本申请要求于2021年09月10日提交的申请号为202111062578.6、发明名称为“作业处理的方法、装置、计算机设备以及存储介质”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请涉及作业调度技术领域，特别涉及一种作业处理的方法、装置、计算机设备以及存储介质。
背景技术
高性能计算(high performance computing，HPC)是跨行业、跨应用领域的计算学科，广泛应用于大规模科学问题的计算和海量数据的处理，如气象预报、汽车仿真、生物制药、基因测序、核爆模拟，以及芯片设计制造等。有能力提供HPC服务的计算机集群，可以称为HPC系统，HPC系统凭借HPC性能，能够处理大量用户提交的作业，以完成各个用户的计算业务。
目前，作业处理的过程可以是：HPC系统中的调度节点支持多用户同时向HPC系统提交作业，每个用户在提交各自的作业时，还会指定各自的作业在执行过程中实时所需求的资源量。对于用户提交的某一作业，调度节点可以根据用户指定的该作业所需资源量，将该作业分配给HPC系统中具有充足空闲资源的计算节点，由该计算节点执行该作业。但是，一个作业的执行过程中所消耗资源量可能不断变化，而用户指定的该作业所需资源量往往是所消耗资源量的最大值(也可以称为峰值资源量)，但是，作业在执行过程中实际消耗资源量不会一直达到峰值资源量，从而导致该计算节点的资源未被充分利用，有效利用率低的问题。因此，如何，提供一种更高效的作业处理方法成为亟待解决的技术问题。
发明内容
本申请提供了一种作业处理的方法、装置、计算机设备以及存储介质，处理高性能计算系统中的作业的过程中，能够提高计算节点中资源的有效利用率，提高作业处理的效率。该技术方案如下：
第一方面，提供了一种作业处理的方法，该方法可以由高性能计算系统中的调度节点执行，在接收到高性能计算系统待处理的作业后，调度节点先确定作业的类别，然后再根据作业的类别预测作业的预测资源量，其中，预测资源量为预测的作业执行过程中实时消耗的资源量，之后，调度节点根据作业的预测资源量，调度高性能计算系统中的计算节点执行作业。
在本法中，由高性能计算系统中的调度节点在接收到待处理的作业后，根据作业的类别， 预测该作业执行过程中实时消耗的资源量，由于同一类别的作业在执行过程中实时消耗的资源量比较接近，因此，相比指定资源量，预测出的资源量更接近作业的实际消耗资源量，则调度节点根据预测出的资源量，调度高性能计算系统中的计算节点执行该作业的过程中，能够提高计算节点中资源的有效利用率，提高作业处理的效率。
在一种可能的实现方式中，调度节点根据作业的类别预测作业的预测资源量包括：调度节点根据作业的类别对应的参考资源量，对作业执行过程中实时消耗的资源量进行预测，得到预测资源量，其中，该参考资源量基于该类别的历史作业的实际消耗资源量得到。
基于上述可能的实现方式，调度节点在根据作业的类别，预测该作业执行过程中实时消耗的资源量时，可以根据该类别对应的参考资源量，对该作业执行过程中实时消耗的资源量进行预测，由于参考资源量由该类别中历史作业的实际消耗资源量得到，因此，根据该类别对应的参考资源量所预测出的预测资源量更接近作业的实际消耗资源量，则调度节点在基于预测资源量，调度高性能计算系统中的计算节点执行作业时，能够进一步提高计算节点中资源的有效利用率。
在一种可能的实现方式中，调度节点在确定作业的类别之前，所述方法还包括：调度节点接收模式调整指令，根据模式调整指令，开启预测模式，其中，模式调整指令用于指示开启预测模式，在预测模式下提供预测作业所需资源量的功能。
相应地，调度节点根据作业的类别对应的参考资源量，对作业执行过程中实时消耗的资源量进行预测，包括：在预测模式下，调度节点根据作业的类别对应的参考资源量，对作业执行过程中实时消耗的资源量进行预测。
基于上述可能的实现方式，预测模式为调度节点的一种工作模式，调度节点可以根据模式调整指令调整自己的工作模式，从而技术人员可以通过向调度节点下达模式调整指令，触发调度节点调整工作模式，以适应不同的应用场景。
在一种可能的实现方式中，在预测模式下，调度节点根据作业的类别对应的参考资源量，对作业执行过程中实时消耗的资源量进行预测，包括：在预测模式下，调度节点先获取作业的类别的历史作业个数，若该类别的历史作业个数大于或等于个数阈值，则调度节点再根据该类别对应的参考资源量，对该作业执行过程中实时消耗的资源量进行预测。
基于上述可能的实现方式，由于该类别的历史作业执行过程中所需的实时资源量是该类别的样本数据，而该类别对应的参考资源量是由该类别中的历史作业的实际消耗资源量来决定，那么，若该类别的历史作业个数大于或等于个数阈值，说明此时该类别的样本(即历史作业个数)数量足够，则认为此时该类别对应的参考资源量与该作业的实际消耗资源量之间的误差比较小，相应地，该调度节点根据该类别对应的参考资源量，预测出的该作业的预测资源量与该作业的实际消耗资源量误差比较小，从而进一步提高了该作业的预测资源量的精确度。
在一种可能的实现方式中，作业包括作业的指定资源量，调度节点根据作业的类别对应的参考资源量，对作业执行过程中实时消耗的资源量进行预测，包括：调度节点基于作业的指定资源量、该类别对应的参考资源量以及该类别的历史作业个数，对该类别的各个作业执行过程中实时消耗的平均资源量进行预测。
基于上述可能的实现方式，由于作业的指定资源量是用户根据经验，预测的作业在执行过程中实时需要的资源量，且参考资源量由该类别中历史作业的实际消耗资源量得到，因此， 基于指定资源量、该类别对应的参考资源量以及该类别的历史作业个数所预测出的预测资源量更接近作业的实际消耗资源量，则调度节点在基于预测资源量，调度高性能计算系统中的计算节点执行作业时，能够进一步提高计算节点中资源的有效利用率。
在一种可能的实现方式中，调度节点根据作业的类别预测述作业的预测资源量之前，所述方法还包括：调度节点将该类别的历史作业的实际消耗资源量的平均值，获取为该类别的参考资源量；并记录该类别与该类别的参考资源量之间的对应关系。
基于上述可能的实现方式，通过提前存储该类别与参考资源量之间的对应关系，以便在后续接收到该类别的作业后，在预测作业的预测资源量时，能够根据该对应关系，快速查询到该类别的对应的参考资源量，无需根据该类别的历史作业的实际消耗资源量，实时计算该类别对应的参考资源量，提高了参考资源量的获取效率。
在一种可能的实现方式中，作业包括作业的作业属性，作业属性指示作业的属性；调度节点确定作业的类别，包括：调度节点基于作业的作业属性，从多个类别中确定该作业的类别，同一类别的各个作业的至少一个属性相同。
基于上述可能的实现方式，由于同一类别的各个作业的至少一个属性相同，且作业的作业属性指示作业的属性，从而调度节点基于作业的作业属性，能够快速地，从多个类别中查询到作业的类别，提高了确定作业类别的效率。
在一种可能的实现方式中，作业属性包括作业所属用户的用户类型以及作业所在作业队列的队列标识；调度节点基于作业的作业属性，从多个类别中确定作业的类别包括：基于多个类别与作业属性之间的对应关系，将多个类别中与该作业的作业属性对应的类别，确定该述作业的类别，其中，该多个类别中的每个类别分别对应一个作业属性。
基于上述可能的实现方式，该调度节点基于多个类别与作业属性之间的对应关系，能够快速地从多个类别中，查询到该作业的作业属性对应的类别，进一步提高了确定作业类别的效率。
在一种可能的实现方式中，调度节点根据作业的预测资源量，调度高性能计算系统中的计算节点执行作业之后，所述方法还包括：调度节点从该计算节点接收作业的实际消耗资源量，基于实际消耗资源量以及该作业的类别的历史作业个数，对该类别对应的参考资源量进行更新，记录更新后的参考资源量与该作业的类别之间的对应关系。
基于上述可能的实现方式，通过存储该类别与最新参考资源量之间的对应关系，以便在后续接收到该类别的作业后，在预测作业的预测资源量时，能够根据该对应关系，快速查询到该类别的对应的最新参考资源量，无需根据该类别的历史作业的实际消耗资源量，实时计算该类别对应的参考资源量，提高了参考资源量的获取效率。
第二方面，提供了一种作业处理的装置，用于执行上述作业处理的方法。具体地，该作业处理的装置包括用于执行上述第一方面或上述第一方面的任一种可选方式提供的作业处理的方法的功能模块。
第三方面，提供一种计算机设备，该计算机设备包括处理器，所述处理器用于执行程序代码，使得计算机设备执行以实现如上述作业处理的方法所执行的操作。
第四方面，提供一种计算机可读存储介质，该存储介质中存储有至少一条程序代码，该程序代码由处理器读取以使计算机设备执行如上述作业处理的方法所执行的操作。
第五方面，提供了一种计算机程序产品或计算机程序，该计算机程序产品或计算机程序包括程序代码，该程序代码存储在计算机可读存储介质中，计算机设备的处理器从计算机可读存储介质读取该程序代码，处理器执行该程序代码，使得该计算机设备执行上述第一方面或者第一方面的各种可选实现方式中提供的方法。
本申请在上述各方面提供的实现方式的基础上，还可以进行进一步组合以提供更多实现方式。
附图说明
图1是本申请提供的一种HPC系统的示意图；
图2是本申请提供的一种调度节点与计算节点之间的交互示意图；
图3是本申请提供的一种作业处理方法的流程图；
图4是本申请提供的一种处理器资源占用示意图；
图5是本申请提供的一种作业处理流程的示意图；
图6是本申请提供的一种不同线程预测资源量以及更新参考资源量的流程图；
图7是本申请提供的一种作业处理的装置的结构示意图；
图8是本申请提供的一种计算机设备的结构示意图。
具体实施方式
为了便于了解，首先对本申请实施例中涉及的部分名词进行如下解释：
作业(job)：完成一项特定的计算业务所需要执行的一组程序实例的集合，通常对应于一台或多台计算机上的一组进程、容器或其他运行时实体，其中，一组程序实例包括至少一条程序代码。
历史作业：已经处理完成的作业，或者说已经执行完成的作业。
指定资源量：用户指定的作业在执行过程中实时需要的资源量，或者也可以理解为，用户预测的作业在执行过程中实时需要的资源量。
预测资源量：调度节点预测的作业在执行过程中实时消耗的资源量。
实际消耗资源量：作业在执行过程中实时消耗的资源量，也即使作业实际实时消耗的资源量，可以称为资源使用情况(resource usage，Rusage)。
参考资源量：同一类型下历史作业的实际消耗资源量的平均值，也可以称为同一类型的各个作业在执行过程中实时消耗的资源量的基准值(baseline)。
资源：计算机程序运行(或执行)时所需的处理器资源、内存资源、存储资源、网络资源中的至少一种。其中，处理器资源包括中央处理器(central processing unit，CPU)资源、图形处理器(graphics processing unit，GPU)资源、嵌入式神经网络处理器(neural-network processing units，NPU)资源中的至少一种，当然也可以包括其他类型的处理器资源。存储资源包括硬盘资源，当然也可以包括其他类型的存储资源。网络资源包括网络带宽，当然也可以包括其他类型的网络资源。
空闲资源：为还未使用的资源，或者也可以理解为处于空闲状态的资源。
资源量：为作业消耗的资源的数量，以处理器资源为CPU为例，CPU的资源量为作业消耗的CPU核数。以内存资源为例，作业消耗的内存的资源量为作业消耗的内存大小，例如 1GB或512MB。以存储资源为例，作业消耗的存储资源的资源量为作业消耗的硬盘大小，例如1GB或512MB，以网络资源为网络带宽为例，网络带宽的资源量为作业消耗的网络带宽的大小。
下面将结合附图对本申请实施方式作进一步地详细描述。
图1是本申请提供的一种HPC系统的示意图，参见图1，该系统100也可以称为HPC集群，该系统100包括调度节点101以及计算节点102，其中，该系统100中的计算节点101有至少一个。调度节点101或者计算节点102为一个或多个服务器所组成的服务器集群。
终端向调度节点101提交包括指定资源量的作业。该终端为用户设备，也可以称为终端站点、用户终端、用户装置，接入装置，订户站，订户单元，移动站，用户代理，用户装备、便携式终端、膝上型终端、台式终端等其他名称。例如，终端可以是个人(personal computer)计算机(例如笔记本电脑、平板电脑、台式电脑、超级本等)、具有网络产品界面设计(Website user interface，Web UI)功能的设备、智能手机、智能电视、智能可穿戴设备、人工智能(artificial intelligence，AI)产品智能汽车、智能仪器或物联网(internet of things，IoT)终端等。
调度节点101用于管理系统100中各个计算节点102资源的分配。例如，该调度节点101基于用户提交的作业的指定资源量，运行调度程序，查询系统100中空闲资源量大于或等于指定资源量的计算节点102。该调度节点101将该作业发给该计算节点102，由该计算节点102执行该作业。
本申请提供一种调度节点根据作业的类别，预测作业在执行过程中实时消耗的资源量，并根据预测出的资源量，调度计算节点执行作业的技术方案。由于同一类别的作业在执行过程中实时消耗的资源量比较接近，因此，相比指定资源量，预测出的资源量更接近作业的实际消耗资源量，从而能够提高计算节点中资源的有效利用率。
为了实现这一技术方案，本申请对调度节点进行改进，例如图2所示的本申请提供的一种调度节点与计算节点之间的交互示意图。调度节点101用于为终端提供主(master)服务以及智能预测服务，其中，主服务包括为终端提交的作业分配资源以及调度计算节点102执行终端提交的作业。智能预测服务为该调度节点101新增的服务，包括根据作业的类别，预测终端提交的作业的预测资源量。
在一种可能的实现方式中，该调度节点101包括作业管理模块11以及资源调度模块12，以提供该主服务。调度节点101还包括资源消耗预测模块13以及参考量管理模块14，以提供该智能预测服务。
在一种可能的实现方式中，该计算节点102用于提供代理(agent)服务，该代理服务包括执行作业、统计作业的实际消耗资源量。该代理服务由该计算节点102中的作业执行模块21来自提供。
当作业管理模块11接收到终端提交的作业后，请求资源消耗预测模块13预测该作业在执行过程中实时消耗的资源量。当接收到作业管理模块11的请求后，资源消耗预测模块13确定该作业的类别，并根据该作业的类别预测该作业的预测资源量，例如，资源消耗预测模块13根据该类别对应的参考资源量以及该作业的指定资源量，预测该作业的预测资源量。资源消耗预测模块13将该作业的预测资源量作为预测结果返回给作业管理模块11。作业管理模块11以该作业的预测资源量请求资源调度模块12为该作业分配资源。接收到作业管理模块11的请求后，该资源调度模块12基于该作业的预测资源量，为该作业分配资源(也即是 分配计算节点)，并向作业管理模块11下发分配结果。该作业管理模块11将该作业下发给分配结果所指示的计算节点102。由该计算节点102中的作业执行模块21来执行下发的作业，并在执行该作业的过程中监控该作业的实际消耗资源量，当作业完成后，向作业管理模块11返回该作业的实际消耗资源量。该作业管理模块11向参考量管理模块14发送该作业的实际消耗资源量。由该参考量管理模块14基于作业的实际消耗资源量，对该作业的类别所对应的参考资源量进行更新，以便资源消耗预测模块13从该参考量管理模块14获取该类别更新后的参考资源量，后续可以基于该类别更新后的参考资源量，预测该类别下新作业的预测资源量。
为了进一步说明调度节点通过预测的方式为作业调度计算节点执行作业的过程，参见图3所示的本申请提供的一种作业处理方法的流程图。
步骤301、终端向高性能计算系统中的调度节点发送作业。
其中，高性能计算系统用于为用户提供作业的提交、调度、执行以及监控等服务。终端向调度节点发送作业的过程也即是终端向调度节点提交作业的过程。
终端所发送的作业为该高性能计算系统中的任一待执行的作业，也即是高性能计算系统待处理的作业。在一种可能的实现方式中，该作业包括该作业的指定资源量，其中，该作业的指定资源量包括执行该作业所需的至少一种资源的指定资源量，例如处理器资源、内存资源、存储资源以及网络资源中的至少一种资源的指定资源量，处理器资源、内存资源、存储资源以及网络资源在前文有相关介绍，在此不再赘述。
在一种可能的实现方式中，该作业还包括该作业的作业属性，该作业属性指示该作业的属性。其中，该作业属性包括该作业所在作业队列的队列标识以及该作业所属用户的用户类型，也即是队列标识以及用户类型分别为作业的一种属性。同一用户类型的各个用户所提交的作业用于完成的计算业务比较类似，则同一用户类型的各个用户所提交的作业的实际消耗资源量具有相同的特征，例如，同一用户类型的各个用户所提交的作业在执行过程中实时消耗的处理器资源均比较多，如实时消耗的CPU的核数比较多。由于同一业务部门所负责的计算业务比较类似，可选地，该用户的用户类型指示该用户所在的业务部门。属于同一用户类型的各个用户的作业提交至调度节点中的同一作业队列，因此，同一作业队列中各个作业的实际消耗资源量具有相同的特征。
在一种可能的实现方式中，作业属性还包括该作业所属用户的用户标识，以指示哪个用户提交的作业。在一种可能的实现方式中，作业属性还包括该作业的作业标签(tag)，其中，作业标签包括该作业所实现的计算业务的业务类型、业务标识以及计算节点标签中的至少一个，该计算节点标签指示用户偏好的执行该作业的计算节点。该计算节点标签可以有至少一个，每个计算节点标签包括用户偏好的执行该作业的至少一个计算节点的标识。需要说明的是，该作业属性所包括的每一项内容为该作业的一个属性，例如用户类型为一个属性，队列标识为一个属性。
在一种可能的实现方式中，用户对该终端下发业务执行指令，该业务执行指令用于指示执行该作业，例如该作业执行指令包括该作业。当该终端接收到该业务执行指令后，基于该业务执行指令向该调度节点发送该作业。
步骤302、该调度节点接收该作业。
其中，该调度节点内维护多个作业队列，每个作业队列对应至少一个用户类型，每个作 业队列用于存储对应用户类型的用户所提交的作业，也即是不同用户类型的用户提供的作业可能被提交至同一作业队列。由于属于同一个用户类型的用户可以有至少一个，相应地，每个作业队列对应至少一个用户。
在一种可能的实现方式中，当该调度节点接收到该作业后，若该作业包括作业队列的队列标识，则该调度节点将该作业提交至该作业队列。若该作业不包括作业队列的队列标识或用户类型，则调度节点将该作业提交至默认作业队列，其中，该默认作业队列可以是作业所属用户的用户类型所对应的作业队列。例如，若该作业不包括队列标识，该调度节点基于用户类型与作业队列之间的对应关系，将该作业存储在该用户类型对应的作业队列。若再例如该作业不包括作业队列的队列标识以及用户类型，且包括用户的用户标识，该调度节点基于用户标识与用户类型之间的对应关系，确定该用户所属的用户类型，并将该作业存储在该用户类型对应的作业队列。该默认作业队列也可以是调度节点为该作业所属用户设置的专用作业队列，而不是该作业所属用户的用户类型所对应的作业队列，该作业不包括作业队列的队列，但包括用户标识，则该调度节点将该作业提交至该用户标识对应的专用专业队列。
当将该作业存储在作业队列后，该作业在该作业队列中排队等待该调度节点读取。在一种可能的实现方式中，该调度节点按照预设轮询规则，从该多个作业队列中轮询读取作业，当该作业位于某一作业队列的开头(即为该作业队列中的第一个作业)且轮询至该作业队列，则该调度节点从该作业队列读取出该作业。其中，该预设轮询规则包括按照作业队列的优先级从高到低的顺序依次轮询，每个作业队列对应一个优先级。优先级越高的作业队列中的作业所对应的计算业务的优先级越高，优先级越低的作业队列中的作业所对应的计算业务的优先级越低。或者，该预设轮询规则包括各个作业队列没有优先级，依次轮询各个作业队列。
以该作业为用户类型1中的用户1-1提交的作业A为例，若该作业A的作业属性包括1(即作业队列的队列标识)以及1-1(即用户类型)，该调度节点接收到该作业A后，将该作业A存储在作业队列1，当轮询至该作业队列1后，且该作业A位于该作业队列1的开头，则该调度节点从该作业队列1中读取该作业A。
在一种可能的实现方式中，该调度节点包括通信模块以及作业管理模块，由该通信模块接收该作业，由作业管理模块将该作业提交到作业队列以及从作业队列中读取作业。
步骤303、该调度节点接收模式调整指令，该模式调整指令用于指示开启预测模式，在该预测模式下提供预测作业所需资源量的功能。
其中，该预测模式是指通过预测作业的预测资源量为作业调度资源的一种工作模式，在一种可能的实现方式中，该模式调整指令包括预测参数，该预测参数用于指示调度节点在一种工作模式下为作业提供资源调度的功能，其中，预测模式为调度节点的一种工作模式。可选地，该预测参数为第一预测参数，该第一预测参数指示在预测模式下提供预测作业所需资源量的功能，则包括第一预测参数的模式调整指令用于指示开启预测模式，在该预测模式下提供预测作业所需资源量的功能。
在另一种可能的实现方式中，调度节点的工作模式除了包括预测模式之外，还包括非预测模式，其中，非预测模式是指通过作业的指定资源量为作业调度资源的一种工作模式。可选地，该预测参数为第二预测参数，该第二预测参数指示在非预测模式下提供资源调度的功能，则包括第一预测参数的模式调整指令用于指示开启非预测模式，在非预测模式下提供资源调度的功能。
为了区分开第一预测参数和第二预测参数，该第一预测参数和第二预测参数可以有不同的表示方式，例如，第一预测参数为“true”，第二预测参数为“false”。
在一种可能的实现方式中，技术人员在该调度节点的配置界面中，输入模式调整指令，并通过配置界面，向该调度节点下达输入的模式调度指令，相应地，该调度节点接收到该模式调度指令。
为了便于描述，将包括第一预测参数的模式调整指令记为第一模式调整指令，将包括第二预测参数的模式调整指令记为第二模式调整指令。图3所示的实施例以调度节点接收到的模式调整指令为第一模式调整指令为例进行说明。
步骤304、该调度节点根据接收到的模式调整指令，开启预测模式。
在一种可能的实现方式中，预测模式和非预测模式通过预测软件开启或关闭来实现，该预测软件用于提供预测作业所需资源量的功能。当预测软件开启时，该调度节点工作在预测模式，当该预测软件关闭时，该调度节点工作在非预测模式。
例如，若接收到的模式调整指令中预测参数为第一预测参数，说明该模式调整指令为第一模式调整指令，则该调度节点使能(enable)该预测软件的应用程序接口(application programming interface，API)，该调度节点工作在预测模式。若接收到的模式调整指令中预测参数为第二预测参数，说明该模式调整指令为第二模式调整指令，则该调度节点禁用(disable)该预测软件的API，该调度节点工作在非预测模式。
在另一种可能的实现方式中，调度节点在初始化时，已经开启预测模式，无需技术人员通过模式调整指令来开启。
需要说明的是，下述步骤305-308所示的过程为在预测模式下，调度节点获取作业的预测资源量的过程。而若接收到的模式调整指令为第二调整指令，则该调度节点在接收到作业后，不执行下述步骤305-308，执行下述步骤309。另外，调度节点也可以先执行步骤303-304，再执行步骤302，本申请对步骤302与步骤303-304的执行顺序不做限定。该调度节点的一种工作模式开启即可，无需重复开启。
步骤305、该调度节点确定该作业的类别。
其中，该调度节点内记录有多个类别与作业属性之间的对应关系，每个类别分别用一个类别标识来指示。每个类别分别对应一个作业属性，每个类别所对应的作业属性包括同一类别下各个作业相同的属性，或者包括同一类别下各个作业的所有属性。每个类别对应的作业属性包括队列标识、用户类型、用户标识以及作业标签中的至少一种。以类别1为例，作业1-2均属于类别1，作业1的作业属性包括队列标识1、用户类型1以及用户标识1，作业2的作业属性包括队列标识1、用户类型1以及用户标识2。其中，类别1对应的作业属性包括队列标识1以及用户类型1，此时，类别1对应的作业属性包括作业1和作业2相同的属性，如果新作业的作业属性包括队列标识1和用户类型1，则新作业的类别为类别1。或者，类别1对应的作业属性包括队列标识1、用户类型1以及用户标识1-10，此时，类别1对应的作业属性包括作业1和作业2的所有属性，如果新作业的作业属性包括队列标识1、用户类型1以及用户标识3，则新作业的类别为类别1。
在一种可能的实现方式中，同一类别对应至少一个作业队列，同一用户类型下的不同用户，可以将相同作业标签的作业提交至同一作业队列，将不同作业标签的作业分别提交至不同的作业队列，因此，不同用户提交不同作业标签的作业可能属于不同的类别。在一种可能 的实现方式中，同一个作业队列中的作业可以来自不同用户类型的用户，因此，不同类别对应的作业属性中的队列标识也可以相同。
例如，该调度节点内存储有一个第一类别关系表，该第一类别关系表中记录有多个类别与作业属性之间的对应关系，如下述表1所示。从表1可知，类别1对应的作业属性包括作业队列1、用户类型1、用户标识1-1至1-5以及作业标签1-A至1-C。根据类别1对应的作业属性可知，用户类型1包括5个用户，分别为用户1-1至1-5。用户类型1下的各个用户提交作业的作业标签有3种，分别为作业标签1-A至1-C。用户类型1下的各个作业提交均提交至作业队列1。用户类型1下的各个用户所提交的作业属于类别1。
表1
Figure PCTCN2022086057-appb-000001
在一种可能的实现方式中，该调度节点基于该作业的作业属性，从多个类别中确定作业的类别，同一类别的各个作业的至少一个属性相同。例如，该调度节点基于该多个类别与作业属性之间的对应关系，将该多个类别中与该作业的作业属性对应的类别，确定为该作业的类别。其中，对于该作业的类别，该类别的作业属性包括该作业的作业属性中的至少一个属性。
为了便于描述，将该作业的作业属性记为第一作业属性，将记录的多个类别所对应的作业属性记为第二作业属性。在一种可能的实现方式中，该调度节点在多个第二作业属性中，查询包括该第一作业属性内的属性的第二作业属性。若查询到至少一个第二作业属性包括该第一作业属性内的属性，该调度节点将该至少一个第二作业属性中包括该第一作业属性内最多属性的第二作业属性，作为该第一作业属性对应的第二作业属性，并将该第一作业属性对应的第二作业属性所对应的类型，作为该作业的类型。
例如，该作业为用户类型1中的用户1-1提交的作业A，类型与作业属性之间的对应关系如表1所示。若该作业A的作业属性包括作业队列1以及用户类型1，针对该作业A的作业属性中的作业队列1，该调度节点通过查询上述的表1，查询到表1中类别1和类别2对应的作业属性均包括作业队列1。然后，针对该作业A的作业属性中的用户类型1，表1中类别1对应的作业属性包括用户类型1，而不包括类别2对应的作业属性包括用户类型1，也即是表1中类别1对应的作业属性包括该作业A的作业属性内最多的属性，则该调度节点将类别1作为作业A的类别。
在一种可能的实现方式中，第一类别关系表存储在该调度节点的内存中，该调度节点可以从内存中读取该第一类别关系表，再执行本步骤304。
在另一种可能的实现方式中，当该作业的作业属性还包括作业的类别标识时，该调度节点无需基于作业类别与作业属性之间的对应关系，确定该作业的类别，而是基于作业属性中的类别标识，从该多个类别中确定该作业的类别。
步骤306、该调度节点获取记录的多个类别与参考资源量之间的对应关系。
其中，每个类别下的各个作业的实际消耗资源量具有相同的特征。例如同一类别下的各 个作业实时消耗的处理器资源比较多/少，或者实时消耗的存储资源比较多/少。
每个类别对应的参考计算量基于每个类别的历史作业的实际消耗资源量得到。每个类别分别对应一个参考资源量组，一个参考资源量组包括至少一种资源的参考资源量，例如处理器资源、内存资源、存储资源以及网络资源中的至少一种资源的参考资源量。该至少一种资源中任一种资源的参考资源量基于一个类别的历史作业的该任一种资源的实际消耗资源量得到。
在一种可能的实现方式中，该调度节点内存储有一个第二类别关系表，该第二类别关系表中记录有该多个类别与参考资源量之间的对应关系。如下述表2所示。从表2可知，类别1对应的参考资源量组包括处理器资源的参考资源量1、存储资源的参考资源量2以及网络资源的参考资源量3。
表2
Figure PCTCN2022086057-appb-000002
在一种可能的实现方式中，第二类别关系表存储在该调度节点的内存中，该调度节点可以从内存中读取该第二类别关系表，从而使得该调度节点从该类别关系表中，获取记录的多个类别与参考资源量之间的对应关系。
在一种可能的实现方式中，该第二类别关系表和第一类别关系表组合成一个类别关系表存储在内存中，该调度节点可以从内存中读取组合的类别关系表，从组合的类别关系表中获取记录的多个类别与参考资源量之间的对应关系。
在一种可能的实现方式中，每个类别的类别标识可以是每个类别对应的作业队列的队列标识，也可以是队列标识以外的用于标识类别的其他标识，在此，本申请实施例对类别的表示方式不做限定。
在一种可能的实现方式中，在执行本步骤306之前，该调度节点可以通过下述步骤A-B所示的过程，记录该多个类别与参考计算量之间的对应关系。
步骤A、对于每个类别，该调度节点将每个类别的历史作业的实际消耗资源量的平均值，确定为每个类别的参考资源量。
以一个类别与一种资源为例，该调度节点将该类别的历史作业的这一种资源的实际资源消耗量的平均值，获取为该类别的这一种种资源的参考资源量。
步骤B、该调度节点记录每个类别与每个类别的参考资源量之间的对应关系。
例如，该调度节点将每个类别的类别标识以及每个类别的参考资源量记录在第二类别关系表中，并将该第二类别关系表存储在数据库中，之后，该调度节点还可以将该第二类别关系表从数据库加载至内存中，以便后续能够调度节点能够直接从内存中快速读取到该第二类别关系表。或者，该调度节点也可以先将该第二类别关系表存储在内存中，之后，该调度节点还可以将该第二类别关系表存储在数据库中，以避免内存中的数据丢失。
步骤307、该调度节点基于记录的多个类别与参考资源量之间的对应关系，确定该作业的类别对应的参考资源量。
例如，该调度节点从第二类别关系表中获取该类别对应的参考资源量。
步骤308、该调度节点基于该指定资源量以及该作业的类别对应的参考资源量，对该作业执行过程中实时消耗的资源量进行预测，得到该作业的预测资源量。
其中，该作业的预测资源量包括至少一种资源的预测资源量，例如包括处理器资源、存储资源的预测资源量以及网络资源中的至少一种资源的预测资源量。
在一种可能的实现方式中，若预设的预测参数指示在预测模式下提供预测作业所需资源量的功能，则该调度节点基于该指定资源量以及该作业的类别对应的参考资源量，对该作业执行过程中实时消耗的资源量进行预测。也即是若预设的预测参数指示在预测模式下提供预测作业所需资源量的功能，则该调度节点执行本步骤307。
由于该类别的历史作业执行过程中所需的实时资源量是该类别的样本数据，而该类别对应的参考资源量是由该类别中的历史作业的实际消耗资源量来决定，若该类别的样本(即历史作业个数)数量足够，则认为基于足够样本的实际消耗资源量所得到的参考资源量更加接近新作业的实际消耗资源量。因此，在一种可能的实现方式中，在预测作业的预测资源量时，还会考虑历史作业的个数。
例如，该调度节点获取该作业的类别的历史作业个数，若该作业的类别的历史作业个数大于或等于个数阈值，则调度节点基于该指定资源量以及该作业的类别对应的参考资源量，对该作业执行过程中实时消耗的资源量进行预测。
若该类别的历史作业个数大于或等于个数阈值，说明此时该类别的样本(即历史作业个数)数量足够，则认为此时该类别对应的参考资源量与该作业的实际消耗资源量之间的误差比较小，则该调度节点执行本步骤308。若该类别的历史作业个数小于个数阈值，说明此时该类别的样本(即历史作业个数)数量不足，则认为此时该类别对应的参考资源量与该作业的实际消耗资源量之间误差比较大，则调度节点不执行本步骤308，而执行下述步骤309。
对于本步骤308，在一种可能的实现方式中，该调度节点基于该指定资源量、该作业的类别对应的参考资源量以及该类别的历史作业个数，对该类别的各个作业执行过程中实时消耗的平均资源量进行预测。其中，预测到的该类别的各个作业执行过程中实时消耗的平均资源量也即是该作业的预测资源量。
例如，对于该至少一种资源中任一种资源的指定资源量，调度节点基于该任一种资源的指定资源量、该类别对应的该任一种资源的参考资源量以及该类别的历史作业个数，对该作业执行过程中实时消耗的该任一种资源的资源量进行预测，得到该作业的该任一种资源的预测资源量。其中，该作业的该任一种资源的预测资源量也是预测到的该类别的各个作业执行过程中实时消耗的该任一种资源的平均资源量。
例如，该调度节点将该类别的历史作业个数、该任一种资源的指定资源量以及该类别对应的该任一种资源的参考资源量作为输入参数，输入下述公式(1)：
P＝(baseline*N+REQ)/(N+1)
其中，P为该作业的任一种资源的预测资源量，baseline为该作业的任一种资源的参考资源量，REQ为该作业的任一种资源的指定资源量，N为该作业的类别的历史作业个数，N大于个数阈值。该公式(1)也即是预测软件所实现的功能，例如该调度节点将该输入参数输 入该预测软件的API，预测软件基于输入参数，对公式(1)进行计算，并通过API向该调度节点输出该作业的任一种资源的预测资源量P。
当然，在一些实施例中，若对参考资源量与该作业的实际消耗资源量之间误差要求不高，则该调度节点可以直接采用公式(1)计算该任一种资源的预测资源量，此时，N也可以小于个数阈值。
需要说明的是，本步骤308为调度节点根据该类别对应的参考资源量，对该作业执行过程中实时消耗的资源量进行预测的一种实现方式。而在另一种可能的实现方式中，若该作业不包括作业的指定资源量，该计算机设备将该类别对应的参考资源量作为该作业的预测资源量。其中，该调度节点根据该类别对应的参考资源量，对该作业执行过程中实时消耗的资源量的过程，也即是调度节点根据该作业的类别，预测该作业的预测资源量的过程。
步骤309、该调度节点将该指定资源量作为该作业的预测资源量。
步骤310、该调度节点基于该预测资源量，调度该高性能计算系统中的计算节点执行该作业。
其中，该调度节点中记录有多个计算节点的空闲资源量，其中，该多个计算节点属于该高性能计算系统。一个计算节点的空闲资源量为当前时刻该计算节点的空闲资源的资源量。一个计算节点的空闲资源量包括至少一种资源的空闲资源量。
在一种可能的实现方式中，该调度节点基于该预测资源量以及记录的各个计算节点的空闲资源量，预测该高性能计算系统中执行该作业效果最佳的计算节点，并由预测出的计算节点来执行该作业。例如下述步骤3101-3105所示的过程。
步骤3101、该调度节点将该作业中的指定资源量更新为该作业的预测资源量。
需要说明的是，本步骤3101为可选步骤，例如若该调度节点执行了上述步骤308，则执行本步骤3101，若该调度节点没有执行上述步骤308，说明该作业的预测资源量为该作业的指定资源量，则该调度节点不执行本步骤3101。
步骤3102、该调度节点从多个计算节点中确定目标计算节点，该目标计算节点的空闲资源量大于或等于该作业的预测资源量。
其中，该目标计算节点是调度节点为该作业所调度的计算节点，或者也可以理解调度节点为该作业分配的计算节点，也即是调度节点预测出的该高性能计算系统中执行该作业效果最佳的计算节点。
在一种可能的实现方式中，该调度节点在资源量统计表中，查询是否存在空闲资源量大于或者等于该作业的预测计算资源量的计算节点，若存在，则该调度节点将该查询到的计算节点确定为目标计算节点。其中，该资源量统计表中记录有该多个计算节点的空闲资源量。
例如，对于该多个计算节点中的任一计算节点以及该至少一种资源的任一种资源，若该任一计算节点的该任一种资源的空闲资源量大于或等于该任一种资源的预测资源量，则该调度节点将该任一计算节点确定为目标计算节点。
需要说明的是，在一些实施例中，该调度节点也可以先执行本步骤3102，再执行本步骤3101，在此，本申请实施例对步骤3101和3102的执行顺序不作限定。
步骤3103、该调度节点向该目标计算节点发送更新后的作业。
步骤3104、该目标计算节点接收更新后的作业。
步骤3105、该目标计算节点执行更新后的该作业。
步骤311、在该作业的执行过程中，被调度的计算节点获取该作业的实际消耗资源量。
其中，被调度的计算节点也是目标计算节点。该作业的实际消耗资源量包括该作业的至少一种资源的实际消耗资源量。例如处理器资源、存储资源以及网络资源中至少一种资源的实际消耗资源量中的至少一种。
在一种可能的实现方式中，本步骤311由下述步骤3111-3113所示的过程实现。
步骤3111、对于至少一种资源中的任一种资源，该计算节点在执行该作业的过程中，每间隔预设时长，对该作业实时消耗的该任一资源的资源量进行采集，得到该任一种资源的多个采样资源量。
其中，该预设时长可以根据实际应用场景进行设置，在此，本申请实施例对该预设时长不作限定。任一种资源的每个采样资源量为该作业在一个时刻所消耗的任一资源的资源量。若该任一种资源为处理器资源中的CPU资源，该CPU资源的一个采样资源量为该作业在一个时刻所消耗的CPU核数，若该任一种资源为存储资源中的内存资源，该内存资源的一个采样资源量为该作业一个时刻所消耗的内存大小。若该任一种资源为网络带宽，该网络带宽的一个采样资源量为该作业在一个时刻所消耗的网络带宽的大小。
以该任一种资源为处理器资源中的CPU资源为例，该计算节点在执行该作业的过程中，每间隔预设时长，对该作业的CPU占用率进行周期性采样，得到多个CPU占用率，该计算节点将每个CPU占用率乘以100，得到作业的CPU资源的一个采样资源量(即CPU核数)。而CPU核数为整数值，因此，若一个CPU占用率乘以100所得数值为整数值，则该计算节点将该数值作为CPU资源的一个采样资源量。若一个CPU占用率乘以100所得数值不是整数值，则该计算节点对该数值的整数值加1，得到CPU资源的一个采样资源量。
为了进一步说明该量化过程，参见图4所示的本申请提供的一种处理器资源占用示意图。对于图4中的(1)，计算节点在采用2线程执行该作业的过程中，在一次采样时，该作业的CPU占用率为200％，则该作业的本次采样到的采样资源量大约为2个CPU核(即CPU的2个核)。对于图4中的(2)，计算节点在采用16线程执行某一作业的过程中，在一次采样时，该作业的CPU占用率为395.3％，则该作业的本次采样到的采样资源量大约为4个CPU核。对于图4中的(3)，计算节点为该作业绑定了2个CPU核，即用2个特定的CPU核为该作业提供计算服务，该计算节点在采用16线程执行该作业过程中，在一次采样时，该作业的CPU占用率一直为200％，则该作业的本次采样到的采样资源量大约2个CPU核。
步骤3112、该计算节点基于任一种资源的多个采样资源量，获取该任一种资源的平均采样资源量以及最大采样资源量。
其中，该任一种资源的平均采样资源量为任一种资源的多个采样资源量的平均值，任一种资源的最大采样资源量为任一种资源的多个采样资源量中的最大值。
步骤3113、该计算节点基于该平均采样资源量以及最大采样资源量，获取该作业的该任一种资源的实际消耗资源量。
其中，该任一种资源的实际消耗资源量大于等于该平均采样资源量且小于等于该最大采样资源量，也即是该任一种资源的实际消耗资源量位于该平均采样资源量与该最大采样资源量之间。
为了使得该任一种资源的实际消耗资源量能够取到该平均采样资源量与该最大采样资源量之间，技术人员可以在各个计算节点设置该任一种资源对应的权重，该权重为大于等于0 且小于等于1的任一数值。该权重指示在计算该任一种资源的实际消耗资源量时，对该平均采样资源量以及该最大采样资源量的偏好。
在一种可能的实现方式中，该计算节点将该平均采样资源量以及最大采样资源量作为输入参数，输入至下述公式(2)：
u＝w*(u max-u avg)+u avg
其中，u为该作业的该任一种资源的实际消耗资源量，w为该任一种资源对应的权重，u max为该作业的该任一种资源的最大采样资源量，u avg为该作业的该任一种资源的平均采样资源量。当该计算节点将该输入参数输入至公式(2)后，该计算节点对公式(2)进行计算，输出该任一种资源的实际消耗资源量u。
需要说明的是，当该任一种资源为处理器资源时，该目标计算节点还可以对该任一种资源的实际消耗资源量u进行量化，将量化后的实际消耗资源量u，作为该任一种资源最终的实际消耗资源量。
步骤312、当该作业执行完成后，该计算节点向该调度节点发送该作业的实际消耗资源量。
在一种可能的实现方式中，当该作业执行完成后，该计算节点生成作业的作业完成响应，并向该调度节点发送该作业完成响应。其中，该作业完成响应指示该作业已经执行完成，该作业完成响应包括该作业的实际消耗资源量。
步骤313、该调度节点接收该作业的实际消耗资源量。
在一种可能的实现方式中，该调度节点接收该作业的作业完成响应，当接收到该作业的作业完成响应后，则说明该作业已经执行完成，该调度节点从该作业完成响应中解析出该作业的实际消耗资源量。
在一种可能的实现方式中，若该调度节点中记录有该作业的类别与该作业之间的对应关系(即该作业属于一个类别)，也即是调度节点已知该作业的类别，则该调度节点执行下述步骤314，若该调度节点中没有记录该作业的类别与该作业之间的对应关系，则该调度节点从多个类别中确定该作业的类别(也即是上述步骤305所示的过程)，当确定出作业的类别后，该调度节点再执行下述步骤314。
步骤314、该调度节点基于该作业的实际消耗资源量以及该作业的类别的历史作业个数，对该类别对应的参考资源量进行更新。
在一种可能的实现方式中，对于该至少一种资源中的任一种资源，该调度节点将该作业的该任一种资源的实际消耗资源量、该类别的历史作业个数以及该类别对应的该任一种资源的参考资源量作为输入参数，输入至下述公式(3)：
baseline′＝(baseline*N+REQ)/(N+1)
其中，baseline′为该任一种资源的更新后的参考资源量。当该调度节点将该输入参数输入至公式(3)后，对该公式(3)进行计算，输出该任一种资源的更新后的参考资源量baseline′。
步骤315、该调度节点记录更新后的该参考资源量与该作业的类别之间的对应关系。
例如，该调度节点将内存和/或数据库中将第二类别关系表中该类别对应的参考资源量修改为更新后的参考资源量，以便后续能够调度节点能够直接从内存中读取到该第二类别关系表。且避免内存中的数据丢失。
需要说明的是，本申请所涉及的数据库由持久性存储介质来实现，该数据库可以位于该 调度节点内，也可以位于该调度节点以外的存储节点。
由上述描述可知，高性能计算系统中的调度节点在接收到待处理的作业后，根据作业的类别，预测该作业执行过程中实时消耗的资源量，由于同一类别的作业在执行过程中实时消耗的资源量比较接近，因此，相比指定资源量，预测出的资源量更接近作业的实际消耗资源量，则调度节点根据预测出的资源量，调度高性能计算系统中的计算节点执行该作业的过程中，能够提高计算节点中资源的有效利用率，提高作业处理的效率。进一步，调度节点在根据作业的类别，预测该作业执行过程中实时消耗的资源量时，可以根据该类别对应的参考资源量，对该作业执行过程中实时消耗的资源量进行预测，由于参考资源量由该类别中历史作业的实际消耗资源量得到，因此，根据该类别对应的参考资源量所预测出的预测资源量更接近作业的实际消耗资源量，则调度节点在基于预测资源量，调度高性能计算系统中的计算节点执行作业时，能够进一步提高计算节点中资源的有效利用率。进一步地，调度节点在根据该类别的历史作业的实际资源消耗量获取到该类别的参考资源量后，可以在内存中存储该类别的参考资源量和该类别之间的对应关系，以便后续在预测该类别的作业的预测资源量时，该调度节点可以从内存中读取该类别对应的参考资源量，无需根据该类别的历史作业的实际消耗资源量，实时计算该类别对应的参考资源量，从而能够提高参考资源量的读取效率。
为了进一步说明图3所示的过程，参见图5所示的本申请提供的一种作业处理流程的示意图。初始时，调度节点中的智能预测模块加载预测参数，若预测参数为第一预测参数说明该调度节点工作在预测模式。终端向调度节点提交作业，当调度节点中的调度模块接收到该作业后，请求智能预测模块预测作业的预测资源量。智能预测模块确定该作业的类别，并查询该作业的类别对应的参考资源量，基于该参考资源量以及指定资源量，计算该作业的预测资源量(如预测该作业所需要的CPU核数、内存的大小、网络带宽的大小)，智能预测模块将该作业的预测资源量作为预测结果返回调度模块。该调度模块查询空闲资源量(例如空闲的CPU核数以及空闲的内存空间大小)大于或等于预测资源量的计算节点，该调度模块调度查询到的至少一个计算节点执行作业，例如，将该作业下发至被查询到的计算节点。当计算节点中的代理模块接收到调度节点下发的作业后，启动执行作业，并该作业实时消耗的资源量进行周期性采样，统计作业的实际消耗资源量(例如实际消耗的CPU核数、内存大小或网络带宽的大小)，该代理模块将作业的实际消耗资源量返回给调度模块。由智能预测模块基于作业的实际消耗资源量更新该作业的类别所对应的参考资源量。用户可以通过终端查询作业的实际资源消耗量。其中，图5中的调度模块可以是图2中的作业管理模块11以及资源调度模块12的集成模块。智能预测模块可以是图2中资源消耗预测模块以13以及参考量管理模块14的集成模块。
在一种可能的实现方式，调度节点预测作业的资源量的过程以及更新参考资源量的过程分别由调度节点的不同线程来执行。例如图6所示的本申请提供的一种不同线程预测资源量以及更新参考资源量的流程图，其中，初始时，该第一线程配置模块获取预测参数，若该预测参数为第一预测参数，则该第一线程工作在预测模式，并第一线程在预测模式下等待新作业。当接收到终端提交的作业后，该第一线程对确定作业的类别，并从内存中读取该类别对应的参考资源量，并根据读取到的参考资源量以及该作业的指定资源量，预测该作业的预测资源量。该第一线程将作业中的指定资源量更新为该作业的预测资源量，并基于该预测资源量调度计算节点执行作业。当该作业执行结束，也即是作业完成后，计算节点向该调度节点 返回该作业的实际消耗资源量，调度节点将该作业的实际消耗资源量发送至第二线程。当第二线程接收到该作业的实际消耗资源量后，从内存中读取各个类别对应的作业属性以及参考资源量。该第二线程还从内存中读取未统计的作业属性(也即是新完成的作业的作业属性)。该第二线程基于新完成的该作业的作业属性，确定该作业的类别。之后，该第二线程基于该作业的预测资源量更新该作业的类别的参考资源量，或将更新后的参考资源量以及该作业的作业属性持久化存储在硬盘中，并将该更新后的参考计算资源量存储在数据库中，之后，内存并从数据库或者硬盘中加载更新后的参考资源量，以便后续第一线程查询该类别最新的参考资源量。
上文中结合图1至图6，详细描述了根据本发明本申请所提供的作业处理的方法，下面将结合图7至图8，描述根据本本申请所提供的作业处理的装置和计算机设备。
图7是本申请提供的一种作业处理的装置的结构示意图，所述装置700可以为前面各个实施例或图1-3或图5-6中的调度节点的部分，用于执行调度节点所执行的方法，所述装置700被配置为高性能计算系统中的调度节点，所述装置700包括：
接收单元701，用于接收所述高性能计算系统待处理的作业；
确定单元702，用于确定所述作业的类别；
预测单元703，用于根据所述类别预测所述作业的预测资源量，所述预测资源量为预测的所述作业执行过程中实时消耗的资源量；
调度单元704，用于根据所述预测资源量，调度所述高性能计算系统中的计算节点执行所述作业。
应理解的是，本发明本申请实施例的装置700可以通过中央处理单元(central processing unit，CPU)实现，也可以通过专用集成电路(application-specific integrated circuit，ASIC)实现，或可编程逻辑器件(programmable logic device，PLD)实现，上述PLD可以是复杂程序逻辑器件(complex programmable logical device，CPLD)，现场可编程门阵列(field-programmable gate array，FPGA)，通用阵列逻辑(generic array logic，GAL)或其任意组合。也可以通过软件实现图2至图6所示的作业处理方法时，装置700及其各个模块也可以为软件模块。
在一种可能的实现方式中，所述预测单元703用于：根据所述类别对应的参考资源量，对所述作业执行过程中实时消耗的资源量进行预测，得到所述预测资源量，所述参考资源量基于所述类别的历史作业的实际消耗资源量得到。
在一种可能的实现方式中，所述装置700还包括调整单元；
所述接收单元701，还用于接收模式调整指令，所述模式调整指令用于指示开启预测模式，在所述预测模式下提供预测作业所需资源量的功能；
所述调整单元，还用于根据所述模式调整指令，开启所述预测模式；
所述预测单元703，用于在所述预测模式下，根据所述类别对应的参考资源量，对所述作业执行过程中实时消耗的资源量进行预测。
在一种可能的实现方式中，所述预测单元703用于：在所述预测模式下，获取所述类别的历史作业个数；若所述类别的历史作业个数大于或等于个数阈值，则根据所述类别对应的参考资源量，对所述作业执行过程中实时消耗的资源量进行预测。
在一种可能的实现方式中，所述作业包括所述作业的指定资源量，所述预测单元703用 于：基于所述指定资源量、所述类别对应的参考资源量以及所述类别的历史作业个数，对所述类别的各个作业执行过程中实时消耗的平均资源量进行预测。
在一种可能的实现方式中，所述装置700还包括：
获取单元，用于将所述类别的历史作业的实际消耗资源量的平均值，获取为所述类别的参考资源量；
第一记录单元，用于记录所述类别与所述类别的参考资源量之间的对应关系。
在一种可能的实现方式中，所述作业包括所述作业的作业属性，所述作业属性指示所述作业的属性；所述确定单元702用于：
基于所述作业的作业属性，从多个类别中确定所述作业的类别，同一类别的各个作业的至少一个属性相同。
在一种可能的实现方式中，所述作业属性包括所述作业所属用户的用户类型以及所述作业所在作业队列的队列标识；所述确定单元702用于：基于所述多个类别与作业属性之间的对应关系，将所述多个类别中与所述作业属性对应的类别，确定为所述作业的类别，所述多个类别中的每个类别分别对应一个作业属性。
在一种可能的实现方式中，所述装置700还包括更新单元和第二记录单元；
所述接收单元，还用于从所述计算节点接收所述作业的实际消耗资源量；
所述更新单元，用于基于所述实际消耗资源量以及所述类别的历史作业个数，对所述参考资源量进行更新；
所述第二记录单元，用于记录更新后的所述参考资源量与所述类别之间的对应关系。
其中，所述第二记录单元和所述第一记录单元可以是同一个单元，也可以是不同的单元。
应理解，装置700对应于上述方法实施例中的调度节点，装置700中的各模块和上述其他操作和/或功能分别为了实现方法实施例中的调度节点所实施的各种步骤和方法，具体细节可参见上述方法实施例，为了简洁，在此不再赘述。
应理解，装置700在处理作业时，仅以上述各功能模块的划分进行举例说明，实际应用中，可以根据需要而将上述功能分配由不同的功能模块完成，即将装置700的内部结构划分成不同的功能模块，以完成以上描述的全部或者部分功能。另外，上述实施例提供的装置700与上述方法实施例属于同一构思，其具体实现过程详见上述方法实施例，这里不再赘述。
应理解，装置700可以相当于系统100中的调度节点101中的执行部件。
图8是本申请提供的一种计算机设备的结构示意图，该计算机设备800可以是图1-3或图5-6部分描述的内容中涉及的调度节点。该计算机设备800包括至少一个处理器801、通信总线802、存储器803以及至少一个通信接口804。
处理器801可以是一个通用中央处理器(central processing unit，CPU)、网络处理器(Network Processor，NP)、微处理器、微控制器(microcontroller unit，MCU)、数字信号处理器(digital signal processing，DSP)、或人工智能处理器等各类运行软件的计算设备，每种计算设备可包括一个或多个用于执行软件指令以进行运算或处理的核。该处理器可以内置于SoC(片上系统)、或者可以是一个或多个用于实现本申请方案的集成电路，例如，专用集成电路(application-specific integrated circuit，ASIC)，可编程逻辑器件(programmable logic device，PLD)或其组合。上述PLD可以是复杂可编程逻辑器件(complex programmable logic device，CPLD)，现场可编程逻辑门阵列(field-programmable gate array，FPGA)，通用阵列逻辑(generic  array logic，GAL)或其任意组合。
通信总线802用于在上述组件之间传送信息。通信总线802可以分为地址总线、数据总线、控制总线等。为便于表示，图中仅用一条粗线表示，但并不表示仅有一根总线或一种类型的总线。
存储器803可以包括只读存储器和随机存取存储器，并向处理器801提供指令和数据。存储器803还可以包括非易失性随机存取存储器。例如，存储器803还可以存储设备类型的信息。
存储器203可以是易失性存储器或非易失性存储器，或可包括易失性和非易失性存储器两者。其中，非易失性存储器可以是只读存储器(read-only memory，ROM)、可编程只读存储器(programmable ROM，PROM)、可擦除可编程只读存储器(erasable PROM，EPROM)、电可擦除可编程只读存储器(electrically EPROM，EEPROM)或闪存。易失性存储器可以是随机存取存储器(random access memory，RAM)，其用作外部高速缓存。通过示例性但不是限制性说明，许多形式的RAM可用，例如静态随机存取存储器(static RAM，SRAM)、动态随机存取存储器(DRAM)、同步动态随机存取存储器(synchronous DRAM，SDRAM)、双倍数据速率同步动态随机存取存储器(double data date SDRAM，DDR SDRAM)、增强型同步动态随机存取存储器(enhanced SDRAM，ESDRAM)、同步连接动态随机存取存储器(synchlink DRAM，SLDRAM)和直接内存总线随机存取存储器(direct rambus RAM，DR RAM)。
存储器803可以是独立存在，并通过通信总线802与处理器801相连接。存储器803也可以和处理器801集成在一起。
通信接口804使用任何收发器一类的装置，用于与其它设备或通信网络通信。通信接口804包括有线通信接口，还可以包括无线通信接口。其中，有线通信接口例如可以为以太网接口。以太网接口可以是光接口，电接口或其组合。无线通信接口可以为无线局域网(wireless local area networks，WLAN)接口，蜂窝网络通信接口或其组合等。
在具体实现中，作为一种实施例，处理器801可以包括一个或多个CPU，如图8中所示的CPU0和CPU1。
在具体实现中，作为一种实施例，计算机设备可以包括多个处理器，如图8中所示的处理器801和处理器805。这些处理器中的每一个可以是一个单核处理器(single-CPU)，也可以是一个多核处理器(multi-CPU)。这里的处理器可以指一个或多个设备、电路、和/或用于处理数据(如计算机程序指令)的处理核。
在具体实现中，作为一种实施例，计算机设备还可以包括输出设备806和输入设备807。输出设备806和处理器801通信，可以以多种方式来显示信息。例如，输出设备806可以是液晶显示器(liquid crystal display，LCD)、发光二级管(light emitting diode，LED)显示设备、阴极射线管(cathode ray tube，CRT)显示设备或投影仪(projector)等。输入设备807和处理器801通信，可以以多种方式接收用户的输入。例如，输入设备808可以是鼠标、键盘、触摸屏设备或传感设备等。
在一些实施例中，存储器803用于存储执行本申请方案的程序代码810，处理器801可以执行存储器803中存储的程序代码810。也即是，该计算机设备800可以通过处理器801以及存储器803中的程序代码810，来实现上文图3实施例提供的方法。
上述实施例，可以全部或部分地通过软件、硬件、固件或其他任意组合来实现。当使用软件实现时，上述实施例可以全部或部分地以计算机程序产品的形式实现。所述计算机程序产品包括一个或多个计算机指令。在计算机上加载或执行所述计算机程序指令时，全部或部分地产生按照本发明本申请实施例所述的流程或功能。所述计算机可以为通用计算机、专用计算机、计算机网络、或者其他可编程装置。所述计算机指令可以存储在计算机可读存储介质中，或者从一个计算机可读存储介质向另一个计算机可读存储介质传输，例如，所述计算机指令可以从一个网站站点、计算机、服务器或数据中心通过有线(例如同轴电缆、光纤、数字用户线(DSL))或无线(例如红外、无线、微波等)方式向另一个网站站点、计算机、服务器或数据中心进行传输。所述计算机可读存储介质可以是计算机能够存取的任何可用介质或者是包含一个或多个可用介质集合的服务器、数据中心等数据存储设备。所述可用介质可以是磁性介质(例如，软盘、硬盘、磁带)、光介质(例如，DVD)、或者半导体介质。半导体介质可以是固态硬盘(solid state drive，SSD)。
本申请实施例还提供了一种计算机程序产品或计算机程序，该计算机程序产品或计算机程序包括计算机指令，该计算机指令存储在计算机可读存储介质中，计算机设备的处理器从计算机可读存储介质读取该计算机指令，处理器执行该计算机指令，使得计算机设备执行上述作业处理的方法。
另外，本申请的实施例还提供一种装置，这个装置具体可以是芯片，组件或模块，该装置可包括相连的处理器和存储器；其中，存储器用于存储计算机执行指令，当装置运行时，处理器可执行存储器存储的计算机执行指令，以使芯片执行上述各方法实施例中作业处理的方法。
其中，本实施例提供的装置、设备、计算机可读存储介质、计算机程序产品或芯片均用于执行上文所提供的对应的方法，因此，其所能达到的有益效果可参考上文所提供的对应的方法中的有益效果，此处不再赘述。
以上所述仅为本申请的示例性实施例，并不用以限制本申请，凡在本申请的精神和原则之内，所作的任何修改、等同替换、改进等，均应包含在本申请的保护范围之内。

Claims (21)

  1. 一种作业处理的方法，其特征在于，所述方法由高性能计算系统中的调度节点执行，所述方法包括：
    接收所述高性能计算系统待处理的作业；
    确定所述作业的类别；
    根据所述类别预测所述作业的预测资源量，所述预测资源量为预测的所述作业执行过程中实时消耗的资源量；
    根据所述预测资源量，调度所述高性能计算系统中的计算节点执行所述作业。
  2. 根据权利要求1所述的方法，其特征在于，所述根据所述类别预测所述作业的预测资源量，包括：
    根据所述类别对应的参考资源量，对所述作业执行过程中实时消耗的资源量进行预测，得到所述预测资源量，所述参考资源量基于所述类别的历史作业的实际消耗资源量得到。
  3. 根据权利要求2所述的方法，其特征在于，所述确定所述作业的类别之前，所述方法还包括：
    接收模式调整指令，所述模式调整指令用于指示开启预测模式，在所述预测模式下提供预测作业所需资源量的功能；
    根据所述模式调整指令，开启所述预测模式；
    所述根据所述类别对应的参考资源量，对所述作业执行过程中实时消耗的资源量进行预测，包括：
    在所述预测模式下，根据所述类别对应的参考资源量，对所述作业执行过程中实时消耗的资源量进行预测。
  4. 根据权利要求3所述的方法，其特征在于，所述在所述预测模式下，根据所述类别对应的参考资源量，对所述作业执行过程中实时消耗的资源量进行预测，包括：
    在所述预测模式下，获取所述类别的历史作业个数；
    若所述类别的历史作业个数大于或等于个数阈值，则根据所述类别对应的参考资源量，对所述作业执行过程中实时消耗的资源量进行预测。
  5. 根据权利要求2-4中任一项所述的方法，其特征在于，所述作业包括所述作业的指定资源量，所述根据所述类别对应的参考资源量，对所述作业执行过程中实时消耗的资源量进行预测，包括：
    基于所述指定资源量、所述类别对应的参考资源量以及所述类别的历史作业个数，对所述类别的各个作业执行过程中实时消耗的平均资源量进行预测。
  6. 根据权利要求1-5中任一项所述的方法，其特征在于，所述根据所述类别预测所述作业 的预测资源量之前，所述方法还包括：
    将所述类别的历史作业的实际消耗资源量的平均值，获取为所述类别的参考资源量；
    记录所述类别与所述类别的参考资源量之间的对应关系。
  7. 根据权利要求1-6任一项所述的方法，其特征在于，所述作业包括所述作业的作业属性，所述作业属性指示所述作业的属性；所述确定所述作业的类别，包括：
    基于所述作业的作业属性，从多个类别中确定所述作业的类别，同一类别的各个作业的至少一个属性相同。
  8. 根据权利要求7所述的方法，其特征在于，所述作业属性包括所述作业所属用户的用户类型以及所述作业所在作业队列的队列标识；
    所述基于所述作业的作业属性，从多个类别中确定所述作业的类别包括：
    基于所述多个类别与作业属性之间的对应关系，将所述多个类别中与所述作业属性对应的类别，确定为所述作业的类别，所述多个类别中的每个类别分别对应一个作业属性。
  9. 根据权利要求2-8中任一项所述的方法，其特征在于，所述根据所述预测资源量，调度所述高性能计算系统中的计算节点执行所述作业之后，所述方法还包括：
    从所述计算节点接收所述作业的实际消耗资源量；
    基于所述实际消耗资源量以及所述类别的历史作业个数，对所述参考资源量进行更新；
    记录更新后的所述参考资源量与所述类别之间的对应关系。
  10. 一种作业处理的装置，其特征在于，所述装置被配置为高性能计算系统中的调度节点，所述装置包括：
    接收单元，用于接收所述高性能计算系统待处理的作业；
    确定单元，用于确定所述作业的类别；
    预测单元，用于根据所述类别预测所述作业的预测资源量，所述预测资源量为预测的所述作业执行过程中实时消耗的资源量；
    调度单元，用于根据所述预测资源量，调度所述高性能计算系统中的计算节点执行所述作业。
  11. 根据权利要求10所述的装置，其特征在于，所述预测单元用于：
    根据所述类别对应的参考资源量，对所述作业执行过程中实时消耗的资源量进行预测，得到所述预测资源量，所述参考资源量基于所述类别的历史作业的实际消耗资源量得到。
  12. 根据权利要求11所述的装置，其特征在于，所述装置还包括调整单元；
    所述接收单元，还用于接收模式调整指令，所述模式调整指令用于指示开启预测模式，在所述预测模式下提供预测作业所需资源量的功能；
    所述调整单元，还用于根据所述模式调整指令，开启所述预测模式；
    所述预测单元，用于在所述预测模式下，根据所述类别对应的参考资源量，对所述作业 执行过程中实时消耗的资源量进行预测。
  13. 根据权利要求12所述的装置，其特征在于，所述预测单元用于：
    在所述预测模式下，获取所述类别的历史作业个数；
    若所述类别的历史作业个数大于或等于个数阈值，则根据所述类别对应的参考资源量，对所述作业执行过程中实时消耗的资源量进行预测。
  14. 根据权利要求11-13中任一项所述的装置，其特征在于，所述作业包括所述作业的指定资源量，所述预测单元用于：
    基于所述指定资源量、所述类别对应的参考资源量以及所述类别的历史作业个数，对所述类别的各个作业执行过程中实时消耗的平均资源量进行预测。
  15. 根据权利要求10-14中任一项所述的装置，其特征在于，所述装置还包括：
    获取单元，用于将所述类别的历史作业的实际消耗资源量的平均值，获取为所述类别的参考资源量；
    第一记录单元，用于记录所述类别与所述类别的参考资源量之间的对应关系。
  16. 根据权利要求10-15任一项所述的装置，其特征在于，所述作业包括所述作业的作业属性，所述作业属性指示所述作业的属性；所述确定单元用于：
    基于所述作业的作业属性，从多个类别中确定所述作业的类别，同一类别的各个作业的至少一个属性相同。
  17. 根据权利要求16所述的装置，其特征在于，所述作业属性包括所述作业所属用户的用户类型以及所述作业所在作业队列的队列标识；所述确定单元用于：
    基于所述多个类别与作业属性之间的对应关系，将所述多个类别中与所述作业属性对应的类别，确定为所述作业的类别，所述多个类别中的每个类别分别对应一个作业属性。
  18. 根据权利要求11-17中任一项所述的装置，其特征在于，所述装置还包括更新单元和第二记录单元；
    所述接收单元，还用于从所述计算节点接收所述作业的实际消耗资源量；
    所述更新单元，用于基于所述实际消耗资源量以及所述类别的历史作业个数，对所述参考资源量进行更新；
    所述第二记录单元，用于记录更新后的所述参考资源量与所述类别之间的对应关系。
  19. 一种计算机设备，其特征在于，所述计算机设备包括处理器，所述处理器用于执行程序代码，使得所述计算机设备执行如权利要求1至权利要求9中任一项所述的方法。
  20. 一种计算机可读存储介质，其特征在于，所述存储介质中存储有至少一条程序代码，所述至少一条程序代码由处理器读取以使计算机设备执行如权利要求1至权利要求9中任一 项所述的方法。
  21. 一种计算机程序产品，其特征在于，所述计算机程序产品包括至少一条程序代码，所述至少一条程序代码由计算机设备中的处理器读取以使计算机设备执行如权利要求1至权利要求9中任一项所述的方法。
PCT/CN2022/086057 2021-06-30 2022-04-11 作业处理的方法、装置、计算机设备以及存储介质 WO2023273502A1 (zh)

Priority Applications (2)

Application Number Priority Date Filing Date Title
EP22831333.4A EP4357916A1 (en) 2021-06-30 2022-04-11 Job processing method and apparatus, computer device, and storage medium
US18/400,186 US20240231901A9 (en) 2021-06-30 2023-12-29 Job processing method and apparatus, computer device, and storage medium

Applications Claiming Priority (4)

Application Number Priority Date Filing Date Title
CN202110739273.8 2021-06-30
CN202110739273 2021-06-30
CN202111062578.6 2021-09-10
CN202111062578.6A CN115543552A (zh) 2021-06-30 2021-09-10 作业处理的方法、装置、计算机设备以及存储介质

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US18/400,186 Continuation US20240231901A9 (en) 2021-06-30 2023-12-29 Job processing method and apparatus, computer device, and storage medium

Publications (1)

Publication Number Publication Date
WO2023273502A1 true WO2023273502A1 (zh) 2023-01-05

Family

ID=84692453

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2022/086057 WO2023273502A1 (zh) 2021-06-30 2022-04-11 作业处理的方法、装置、计算机设备以及存储介质

Country Status (2)

Country Link
EP (1) EP4357916A1 (zh)
WO (1) WO2023273502A1 (zh)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117575654A (zh) * 2023-11-27 2024-02-20 数翊科技(北京)有限公司 数据加工作业的调度方法及装置

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101697141A (zh) * 2009-10-30 2010-04-21 清华大学 网格中基于历史数据建模的作业性能预测方法
CN104778079A (zh) * 2014-01-10 2015-07-15 国际商业机器公司 用于调度、执行的装置和方法以及分布式系统
CN110058936A (zh) * 2018-01-18 2019-07-26 伊姆西Ip控股有限责任公司 用于确定专用处理资源的资源量的方法、设备和计算机程序产品
US20210157642A1 (en) * 2019-11-21 2021-05-27 Accenture Global Solutions Limited Utilizing machine learning to concurrently optimize computing resources and licenses in a high-performance computing environment

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101697141A (zh) * 2009-10-30 2010-04-21 清华大学 网格中基于历史数据建模的作业性能预测方法
CN104778079A (zh) * 2014-01-10 2015-07-15 国际商业机器公司 用于调度、执行的装置和方法以及分布式系统
CN110058936A (zh) * 2018-01-18 2019-07-26 伊姆西Ip控股有限责任公司 用于确定专用处理资源的资源量的方法、设备和计算机程序产品
US20210157642A1 (en) * 2019-11-21 2021-05-27 Accenture Global Solutions Limited Utilizing machine learning to concurrently optimize computing resources and licenses in a high-performance computing environment

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
See also references of EP4357916A4

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117575654A (zh) * 2023-11-27 2024-02-20 数翊科技(北京)有限公司 数据加工作业的调度方法及装置
CN117575654B (zh) * 2023-11-27 2024-05-14 数翊科技(北京)有限公司 数据加工作业的调度方法及装置

Also Published As

Publication number Publication date
US20240134688A1 (en) 2024-04-25
EP4357916A4 (en) 2024-04-24
EP4357916A1 (en) 2024-04-24

Similar Documents

Publication Publication Date Title
US10198292B2 (en) Scheduling database queries based on elapsed time of queries
US20200396311A1 (en) Provisioning using pre-fetched data in serverless computing environments
TWI697787B (zh) 自適應資源分配方法及裝置
US20230179538A1 (en) Systems and methods for provision of a guaranteed batch
US9870269B1 (en) Job allocation in a clustered environment
US9973512B2 (en) Determining variable wait time in an asynchronous call-back system based on calculated average sub-queue wait time
WO2024016596A1 (zh) 容器集群调度的方法、装置、设备及存储介质
CN109564528B (zh) 分布式计算中计算资源分配的系统和方法
US20120297216A1 (en) Dynamically selecting active polling or timed waits
WO2020125396A1 (zh) 一种共享数据的处理方法、装置及服务器
Chen et al. Latency minimization for mobile edge computing networks
US20120324466A1 (en) Scheduling Execution Requests to Allow Partial Results
WO2023273502A1 (zh) 作业处理的方法、装置、计算机设备以及存储介质
WO2019024235A1 (zh) 电子装置、服务器分配控制方法和计算机可读存储介质
CN114155026A (zh) 一种资源分配方法、装置、服务器及存储介质
Liu et al. An Energy‐Saving Task Scheduling Model via Greedy Strategy under Cloud Environment
CN117331678B (zh) 面向异构算力联邦的多集群作业资源规格计算方法和系统
Shu-Jun et al. Optimization and research of hadoop platform based on fifo scheduler
Guo Ant colony optimization computing resource allocation algorithm based on cloud computing environment
Komarasamy et al. ScHeduling of jobs and Adaptive Resource Provisioning (SHARP) approach in cloud computing
WO2022151951A1 (zh) 任务调度方法及管理系统
US20240231901A9 (en) Job processing method and apparatus, computer device, and storage medium
Ru et al. Providing fairer resource allocation for multi-tenant cloud-based systems
CN115543552A (zh) 作业处理的方法、装置、计算机设备以及存储介质
CN114860449A (zh) 数据处理方法、装置、设备和存储介质

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22831333

Country of ref document: EP

Kind code of ref document: A1

WWE Wipo information: entry into national phase

Ref document number: 2022831333

Country of ref document: EP

NENP Non-entry into the national phase

Ref country code: DE

ENP Entry into the national phase

Ref document number: 2022831333

Country of ref document: EP

Effective date: 20240116