WO2016082693A1 - 一种用于在集群中调度计算任务的方法与设备 - Google Patents

一种用于在集群中调度计算任务的方法与设备 Download PDF

Info

Publication number
WO2016082693A1
WO2016082693A1 PCT/CN2015/094790 CN2015094790W WO2016082693A1 WO 2016082693 A1 WO2016082693 A1 WO 2016082693A1 CN 2015094790 W CN2015094790 W CN 2015094790W WO 2016082693 A1 WO2016082693 A1 WO 2016082693A1
Authority
WO
WIPO (PCT)
Prior art keywords
task
computing
cluster
information
tasks
Prior art date
Application number
PCT/CN2015/094790
Other languages
English (en)
French (fr)
Inventor
王奎
Original Assignee
阿里巴巴集团控股有限公司
王奎
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority to US15/526,789 priority Critical patent/US20180198855A1/en
Application filed by 阿里巴巴集团控股有限公司, 王奎 filed Critical 阿里巴巴集团控股有限公司
Publication of WO2016082693A1 publication Critical patent/WO2016082693A1/zh

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/10—Protocols in which an application is distributed across nodes in the network
    • H04L67/1001—Protocols in which an application is distributed across nodes in the network for accessing one among a plurality of replicated servers
    • H04L67/1004—Server selection for load balancing
    • H04L67/1008—Server selection for load balancing based on parameters of servers, e.g. available memory or workload
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/48—Program initiating; Program switching, e.g. by interrupt
    • G06F9/4806—Task transfer initiation or dispatching
    • G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
    • G06F9/4881—Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
    • G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
    • G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
    • G06F9/505—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals considering the load
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
    • G06F9/5061—Partitioning or combining of resources
    • G06F9/5066—Algorithms for mapping a plurality of inter-dependent sub-tasks onto a plurality of physical CPUs
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
    • G06F9/5083—Techniques for rebalancing the load in a distributed system
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/10—Protocols in which an application is distributed across nodes in the network
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/50—Network services
    • H04L67/53—Network services using third party service providers
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/50—Network services
    • H04L67/60—Scheduling or organising the servicing of application requests, e.g. requests for application data transmissions using the analysis and optimisation of the required network resources
    • H04L67/61—Scheduling or organising the servicing of application requests, e.g. requests for application data transmissions using the analysis and optimisation of the required network resources taking into account QoS or priority requirements

Definitions

  • the present application relates to the field of computers, and in particular, to a technique for scheduling computing tasks in a cluster.
  • the first is the absolute control method. That is, for a plurality of computing tasks corresponding to each cloud service node, the resource amount may be allocated according to the total resources of the cloud service node itself, and the maximum amount of resources that can be used for each computing task is a determined value.
  • the amount of resources actually needed by each computing task is greater than the determined value of the amount of resources allocated thereto, the requirements of the computing task are not met; and when the computing task requires a large amount of resources When the determined amount of resources is allocated, it will bring waste of resources.
  • the second is the relative control method. That is, for each computing task corresponding to each cloud service node, according to the total amount of resources of the cloud service node itself, the resource amount is allocated according to a certain proportion for each computing task. At this time, if a certain computing task is under pressure If it is too large, if the corresponding cloud service node is cut according to the weight, the abnormality of the computing task will affect the smooth progress of other computing tasks.
  • a method for scheduling computing tasks in a cluster comprising:
  • an apparatus for scheduling a computing task in a cluster comprising:
  • a first device configured to acquire multiple computing tasks to be scheduled in the cluster
  • the second device is configured to divide the plurality of computing tasks into a plurality of task subsets according to the task load information corresponding to the computing task, where each task subset corresponds to one cluster node in the cluster.
  • the present application divides each of the computing tasks into a task subset corresponding to a certain cluster node by using task load information corresponding to multiple computing tasks to be scheduled in the cluster, thereby The optimal partitioning combination is found for multiple computing tasks to be scheduled in the cluster, thereby maximizing the utilization of resources of the entire cluster and implementing a more reasonable system resource scheduling strategy.
  • FIG. 1 shows a schematic diagram of an apparatus for scheduling computing tasks in a cluster in accordance with an aspect of the present application
  • FIG. 2 shows a flow chart of a method for scheduling computing tasks in a cluster in accordance with another aspect of the present application
  • FIG. 3 is a schematic diagram showing a computing task load before scheduling of a cluster node M according to a preferred embodiment of the present application
  • FIG. 4 is a schematic diagram showing a computing task load before scheduling of a cluster node N according to a preferred embodiment of the present application
  • FIG. 5 is a schematic diagram showing a calculation task load of a cluster node M after scheduling according to a preferred embodiment of the present application
  • FIG. 6 shows a schematic diagram of computing task load after cluster node N scheduling according to a preferred embodiment of the present application.
  • the terminal, the device of the service network, and the trusted party each include one or more processors (CPUs), input/output interfaces, network interfaces, and memory.
  • processors CPUs
  • input/output interfaces network interfaces
  • memory volatile and non-volatile memory
  • the memory may include non-persistent memory, random access memory (RAM), and/or non-volatile memory in a computer readable medium, such as read only memory (ROM) or flash memory.
  • RAM random access memory
  • ROM read only memory
  • Memory is an example of a computer readable medium.
  • Computer readable media includes both permanent and non-persistent, removable and non-removable media.
  • Information storage can be implemented by any method or technology.
  • the information can be computer readable instructions, data structures, modules of programs, or other data.
  • Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read only memory. (ROM), electrically erasable programmable read only memory (EEPROM), flash memory or other memory technology, compact disk read only memory (CD-ROM), digital versatile disk (DVD) or other optical storage,
  • computer readable media does not include non-transitory computer readable media, such as modulated data signals and carrier waves.
  • the apparatus 1 shows a schematic diagram of an apparatus for scheduling computing tasks in a cluster in accordance with an aspect of the present application.
  • the apparatus 1 for scheduling computing tasks in a cluster includes a first device 101 and a second device 102.
  • the first device 101 acquires a plurality of computing tasks to be scheduled in the cluster; the second device 102 is configured to divide the plurality of computing tasks into multiple task subsets according to task load information corresponding to the computing tasks, where Each task subset corresponds to one cluster node in the cluster.
  • the first device 101 of the device 1 acquires a plurality of computing tasks to be scheduled in the cluster.
  • the cluster is preferably a cluster composed of a plurality of servers for performing cloud computing based on the Internet.
  • Each of the servers is a cluster section that provides users with services such as cloud computing. point.
  • the computing tasks include process services, thread services, and the like on the cluster nodes.
  • This solution needs to re-schedule multiple computing tasks under multiple cluster nodes in the cluster to optimize the allocation of cluster resources. Therefore, it is first necessary to determine multiple computing tasks waiting for scheduling. In this solution, the larger the resource pool is, the more computing tasks are determined to be scheduled, the higher the matching degree of the scheduling is to the system, and the better the scheduling accuracy is, the more significant the optimization effect of cluster resource allocation is. .
  • each of the computing tasks in the cluster for example, in a cloud resource system, has a backup computing task corresponding thereto, so that data information corresponding to the computing task is saved, and further, the The consistency of the data information is ensured by the synchronization mechanism between the computing task and the corresponding backup computing task.
  • disaster recovery arrangements on the computing tasks, it is possible to avoid losses caused by sudden changes in data information such as damage or loss.
  • the plurality of computing tasks to be scheduled in the cluster acquired by the first device may also be from third party devices other than the device 1. That is, a plurality of computing tasks to be scheduled are collected from the third-party device, and then the device 1 performs corresponding information processing, task scheduling generation, and execution operations.
  • the second device 102 of the device 1 divides the plurality of computing tasks into a plurality of task subsets according to task load information corresponding to the computing task, where each task subset corresponds to the cluster respectively A cluster node.
  • the task load information of the plurality of computing tasks waiting to be scheduled is first obtained.
  • the task load information includes various measurable indicator data information corresponding to the computing task, including but not limited to CPU usage, memory usage, network usage, memory usage, network card traffic, and the like, and attributes related to the computing task.
  • the task load information may also be a comprehensive measurable indicator data composed of a plurality of single and specific measurable indicator data information, for example, a single measurable indicator data information, CPU usage, memory usage, and network. Use a certain combination of calculations to generate a comprehensive indicator data, such as based on actual conditions Set a certain weight for CPU usage, memory usage, and network usage parameters to obtain the required comprehensive measurable indicator data.
  • computation tasks are divided into the same task subset.
  • the computing tasks A, B, C, and D to be scheduled belong to the cluster nodes 1, 2, 3, and 4 respectively; and the task load information of the computing tasks A, B, C, and D are analyzed, and based on the specific
  • the division operation finally selects the three tasks of the calculation tasks A, B, and C to be re-divided into a task subset, and corresponds to the cluster node 1, and the generation of the combination of the new calculation tasks is the utilization of the cluster resources.
  • the service capability of the cloud computing provided by the cluster resource presents a dynamic distribution and real-time change characteristics, and the task load information of the calculation data source corresponding to the division operation described in the solution can be well reflected.
  • the data dynamic change for example, the task load information may determine specific numerical information corresponding to a certain determinable indicator at a determined time period or time point.
  • first device and the second device may be different devices in the same device. Further, in a practical application, the first device and the second device may be deployed on different devices to complete the solution. In this case, if the application is applicable, it should also be included in the protection scope of the present application. It is hereby incorporated by reference.
  • the present application divides each of the computing tasks into a task subset corresponding to a certain cluster node by using task load information corresponding to a plurality of computing tasks to be scheduled in the cluster, thereby The plurality of computing tasks to be scheduled find the best partitioning combination, thereby maximizing the utilization of resources of the entire cluster and realizing a more reasonable system resource scheduling strategy.
  • the apparatus 1 for scheduling a computing task in a cluster further comprises a third device (not shown), the third device assigning the subset of tasks to the corresponding cluster node, and The computing tasks in the subset of tasks are performed in the cluster nodes.
  • the resource usage of the computing task is collected in real time, and the task scheduling is performed by the computing engine.
  • the system resources corresponding to the computing task are ready for storage.
  • the computing task can directly use the scheduled system resources.
  • the divided plurality of computing tasks are allocated to corresponding cluster nodes. And sharing the cluster resources by the cluster node.
  • the resources that are required to be consumed by each computing task re-divided under the cluster node are, for example, when the corresponding Internet user requests to perform a certain cloud computing task.
  • the required resources such as network card traffic, CPU, memory, etc. are all divided from the total amount of resources owned by the cluster nodes corresponding to the cloud computing task.
  • the execution of the computing task is completed based on the cluster node.
  • the cluster uses the control system to analyze and collect related data information of the cluster node and the computing task under the cluster node, and store related information in the control. In the system, this is used as the basic data for the information scheduling strategy.
  • first device, the second device, and the third device may be different devices in the same device. Further, in a practical application, the first device, the second device, and the third device may be deployed on different devices to complete the solution. In this case, if the application is applicable, it should also be included in the present application. The scope of protection of the application is incorporated herein by reference.
  • the second device 102 of the device 1 associates the multiple computing tasks according to the task load information corresponding to the computing task and the node resource threshold information corresponding to the cluster. Divided into a plurality of task subsets, wherein each task subset corresponds to one cluster node in the cluster.
  • the plurality of computing tasks to be scheduled in the cluster are divided into the same task subset, and each task subset corresponds to one cluster node in the cluster, and the partitioning operation needs to obtain the task load information of the to-be-scheduled computing task.
  • the node resource threshold information includes a maximum value allowed by the pressure load of the cluster node.
  • the node resource threshold information corresponds to the task load information.
  • the task load information includes various measurable indicator data information corresponding to the computing task
  • the node resource threshold information also includes the The total amount of resource load corresponding to each measurable indicator at the cluster node.
  • the task load information of the to-be-scheduled computing task information includes the network card traffic, and when the computing task partitioning operation is performed, the node resource threshold information corresponding to the set cluster correspondingly includes the network card traffic.
  • the sum of the values of the task load information of the respective computing tasks under the same cluster node with respect to the same measurable indicator data does not exceed the resources of the metric indicator corresponding to the cluster node.
  • Total load For example, the computing tasks E, F, and G are divided into the same cluster node.
  • the task load information includes the network card traffic
  • the sum of the network card traffic consumed by each of the computing tasks E, F, and G is a
  • the total network load corresponding to the NIC traffic of the cluster node is b
  • the NIC traffic value a cannot exceed b. Therefore, it is feasible to ensure that the operation of each computing task under the same cluster node by dividing the set to the same cluster node is feasible. , optimized. Further, preferably, an optimal range may be set for the node resource threshold information corresponding to the cluster, for example, the downward fluctuation does not exceed 10%, that is, the total network load corresponding to the network node traffic of the cluster node is b.
  • the setting of the node resource threshold information may be obtained by performing a stress test on the cluster node, and based on sampling and analyzing the specific operation conditions of each computing task under the cluster node.
  • the resource threshold information of the cluster node is the same when the configurations of the servers corresponding to the cluster nodes are the same, for example, the server software configuration, the hardware configuration, and the running environment are the same.
  • different configurations may be set for different servers that are cluster nodes based on the needs of the cluster resource allocation or the specific computing tasks, and the resource threshold information of different nodes in the cluster may be set differently.
  • the server configurations corresponding to all the cluster nodes are the same, so that the node resource threshold information corresponding to each node in the cluster is consistent.
  • the method for setting the threshold resource information of the foregoing node such as the stress test
  • the method for setting other node resource threshold information is applicable to the present application, and should also be included in the protection of the present application. It is within the scope and is hereby incorporated by reference.
  • the second device 102 of the device 1 performs a dividing operation according to the task load information corresponding to the computing task and the node resource threshold information corresponding to the cluster until the plurality of computing tasks are divided.
  • a plurality of task subsets wherein each of the task subsets respectively corresponds to one cluster node in the cluster; wherein the dividing operation comprises: taking the computing task to be divided into the plurality of computing tasks as a first computing task; determining one or more candidate task subsets, wherein the candidate task subset includes the first computing task and the plurality of At least one other computing task to be divided, wherein the cumulative information of the task load information of each of the computing tasks in the candidate task subset satisfies the node resource threshold information; from the one or more candidate tasks
  • the subset preferably determines the subset of tasks.
  • a dividing operation may be performed on the plurality of computing tasks.
  • the computing task to be divided is selected as the first computing task among the plurality of computing tasks to be scheduled, and the selected method may be random, or may be based on a certain regularity, for example, preferentially selecting the task load.
  • the calculation task corresponding to the increased load of the information it can be set that the selected first computing task corresponds to one cluster node 1. Then, for the first computing task, select one or more computing tasks that match the remaining plurality of computing tasks to be scheduled.
  • the condition to be satisfied by the matching includes that the accumulated information of the task load information corresponding to the first computing task and the one or more computing tasks matching the same cannot exceed the maximum value of the corresponding node resource threshold information.
  • the task load information corresponding to each of the computing tasks is set to a value corresponding to a certain metric indicator data at a certain determined time point.
  • the task load information is set to the network card traffic information, and the time point T in the time dimension is selected, and the node resource threshold corresponding to the network card traffic information of the cluster node 1 corresponding to the first computing task A is L,
  • the optimal range of the node resource threshold L is downwardly floating no more than 10%.
  • the computing task matching the matching task is found. If the network card traffic information of the first computing task A is L1 at the time point T, if the task B is matched for the task B, the corresponding network card traffic information is L2, and if the sum of L1 and L2 has exceeded the The node resource threshold L, if the computing task B does not match the first computing task A, the computing task B is discarded, and a new matching computing task is sought; if the sum of L1 and L2 is already at the node resource Within the optimal range of the threshold L, the computing task A and the computing task B satisfy the matching condition, and may correspond to a candidate task subset.
  • the continuation may be continued.
  • each of the determined computing tasks in the one or more candidate task subsets is corresponding to each time dimension under various task load information, such as various measurable indicator data
  • the accumulated information of the task load information must satisfy the corresponding node resource threshold information corresponding to each.
  • the metric data that the partitioning operation depends on may be multiple, and may even include comprehensive indicator data composed of multiple single measurable indicator data; at the same time, the acquired time dimension may be multiple, specifically There may be more than one time point, and the final partitioning result may have multiple possibilities based on different parameter changes.
  • one or more candidate task subsets including the first computing task and one or more of the other computing tasks may be present.
  • the determining the subset of tasks from the one or more candidate task subsets comprises: determining subset related information of the candidate task subsets; and according to the subset related information from the one
  • the subset of tasks is preferably determined in a plurality of candidate task subsets.
  • the subset related information includes a pulse ratio of the candidate task subset.
  • the subset related information includes a pulse ratio of the candidate task subset.
  • the sum of the data values L1, L2, and L3 corresponding to the network card traffic information of the computing tasks A, B, and C should not exceed the node resource threshold L.
  • the ratio of the maximum value of the corresponding L1, L2, and L3 to the average value of L1, L2, and L3 is the pulse value of the candidate task subset M at time T1, when the time dimension is hour.
  • the subset related information may further include: calculating a value corresponding to each computing task of the same candidate task subset at the same determined time point of the same measurable indicator data. And a difference between the node resource threshold information of the cluster node corresponding to the task subset. For example, for one candidate task subset N, including the first computing task A, the computing task D, and the computing task E, setting the task load information to network card traffic information, selecting a time dimension of hours, and the first The node resource threshold corresponding to the network card traffic information of the cluster node 1 corresponding to the task A is L.
  • the optimal range of the node resource threshold L is preferably not to be less than 10%.
  • the sum of the data values L1, L4, and L5 corresponding to the network card traffic information of the computing tasks A, D, and E should not exceed the node resource threshold L, and the corresponding difference is L-(L1+ L2+L3).
  • a certain seed set related information may be used for the candidate task subset for further screening.
  • the plurality of subset related information may also be used for comprehensive comparison at the same time.
  • the candidate task subset M and the candidate task subset N are respectively calculated for the corresponding pulse ratio and the difference, thereby obtaining an optimal select.
  • the priority of the pulse ratio is higher than the difference, for example, the optimal range of the threshold resource threshold L is preferably not more than 10% downward, and for the pulse ratio, It is additionally specified that it is preferably in a wider range of the node resource threshold L, for example, in the range of 80% to 95%, if the candidate task subset M corresponds to a pulse ratio in the range of 80% to 95%. If the candidate task subset N does not reach this range, the candidate task subset M is preferred regardless of the difference between the two sets of task subsets.
  • the optimal range of the node resource threshold L is 10%, and the pulse ratio corresponds to a wider preferred range of the node resource threshold L, for example, a range of 80% to 95%, which is merely an example, It can be flexibly arranged based on actual business needs.
  • subset related information includes a pulse ratio of the candidate task subset
  • the subset related information may further include: calculating each computing task under the same candidate task subset in the same
  • the difference between the sum of the corresponding values at the same determined time point of the measurable indicator data and the node resource threshold information of the cluster node corresponding to the task subset is only an example, and other subset related information may be applied to the application, It is intended to be included within the scope of this application and is hereby incorporated by reference.
  • the accumulated information of the task load information of each of the computing tasks in the candidate task subset in the device 1 satisfies the node resource threshold information, including: the candidate task subset
  • the accumulated information of the task load information of each of the computing tasks respectively satisfies the node resource threshold information by dimensions.
  • the task load information of the computing task can comprehensively and objectively reflect the resource cost requirement of the computing task. Therefore, when performing the dividing operation of the multiple computing tasks, the metric data according to the metric is multi-dimensional.
  • the metric indicator data letter may be derived from the following various types, such as CPU usage, memory usage, network usage, memory usage, network card traffic, and the like, and the attribute indicators related to the computing task are not only more A single, specific measurable indicator data information, or a plurality of comprehensive measurable indicator data composited by multiple indicators, for example, for a single measurable indicator data information, CPU usage, memory usage, and network usage are determined according to certain
  • the combined calculation generates a comprehensive indicator data, such as setting a certain weight for the CPU usage rate, the memory usage rate, and the network usage parameter based on actual conditions, and obtaining the required comprehensive measurable indicator data.
  • the diversification of the measurable indicator data corresponding to the task load information is to provide the most comprehensive basic data information for the division of the computing task, so that the optimal partitioning method can be found according to the purpose and expectation of the computing task. Therefore, the allocation and utilization of cluster resources is the most reasonable and more in line with actual business needs.
  • the time dimension on which the basis is based is also multi-dimensional, and the recorded task load information data can be recorded according to any required time unit such as year, month, day, hour, minute, second, and the like.
  • the data information in the appropriate group or groups of dimensions may be selected for use according to the specific division purpose of the plurality of computing tasks to be invoked.
  • the second device 102 of the device 1 calculates log information of a plurality of historical computing tasks according to task information of the cluster, and calculates tasks of the plurality of historical tasks.
  • the overhead information determines task load information of the computing task.
  • the task load information of the computing task to be scheduled is dynamically changed, so the task load information of the computing task to be scheduled obtained at any time is already a historical data.
  • the computing tasks such as Internet cloud computing tasks, are similar for the same type of computing tasks, especially for a series of computing tasks with similar or identical parameter conditions, and their computing execution. Therefore, a particular historical computing task is informative for subsequent computing tasks that match it. Further, if you use a more reasonable matching method, you can find a good job for the current computing task to be scheduled.
  • the matchable model history calculation task can calculate the task cost information of the task based on the history, such as pressure data corresponding to different measurable indicators in different time dimensions, to infer that the to-be-scheduled computing task may be required
  • the task overhead can also obtain the task load information required to divide the plurality of computing tasks.
  • the foregoing task calculation log information according to the cluster determines the task overhead information of the plurality of historical computing tasks; and determines the computing task according to the task overhead information of the plurality of historical computing tasks.
  • the task load information is only an example. Other task load information for determining the computing task, as applicable to the present application, is also included in the scope of protection of the present application, and is hereby incorporated by reference.
  • determining the task load information of the computing task according to the task cost information of the plurality of historical computing tasks comprises: filtering and calculating from the plurality of historical computing tasks according to the task related information of the computing task a preferred historical computing task matching the tasks; determining task load information of the computing tasks according to the task overhead information of the preferred historical computing tasks.
  • the task related information of the computing task includes various related information that can describe, locate a computing task execution condition, an execution situation, and the like, such as various parameters involved in the execution of the computing task, for example, Server software and hardware requirements.
  • the historical computing task corresponding to the computing task may be exactly the same dynamic computing task as the computing task, but the corresponding data has a regular change due to temporal changes; the corresponding history
  • the computing task and the computing task may also be two completely independent dynamic computing tasks, but because of the great similarity between the two, it is suitable to match.
  • determining the task load information of the computing task according to the task cost information of the plurality of historical computing tasks comprises: clustering the plurality of historical computing tasks according to task related information of the plurality of historical computing tasks Processing, determining task cost information corresponding to each computing task cluster; determining, according to the task related information of the computing task, a preferred computing task cluster that matches the computing task; and calculating the task clustering according to the preferred The task overhead information determines task load information of the computing task.
  • the preferred historical computing task matching the computing task is selected based on the plurality of historical computing tasks, and the task load information of the computing task can be determined objectively and accurately using the task cost information of the preferred historical computing task.
  • the plurality of historical computing tasks may be first performed, and clustering processing is performed based on the determined task related information, where the clustering process uses one or several metrics as a standard similarity. Sexuality, minimized between unified clusters, and maximized among different clusters.
  • the clustering algorithm makes the plurality of historical computing tasks aggregate into multiple categories. On the one hand, the information needed to be compared can be greatly reduced to A plurality of historical computing task clusters.
  • the task overhead information corresponding to the historical computing task clustering by the clustering process is a statistical analysis result, which is more universal and widely applicable, according to the preferred calculation.
  • the clustering criterion corresponding to the task clustering may find matching data information for the computing task, and determining the task load information of the computing task by using the matched task cost information of the matched preferred computing task cluster is more efficient and feasible.
  • FIG. 2 illustrates a flow diagram of a method for scheduling computing tasks in a cluster in accordance with another aspect of the present application.
  • the device 1 acquires a plurality of computing tasks to be scheduled in the cluster; in step S202, the device 1 is configured to calculate the multiple computing according to the task load information corresponding to the computing task.
  • the task is divided into a plurality of task subsets, wherein each task subset corresponds to one cluster node in the cluster.
  • the device 1 acquires a plurality of computing tasks to be scheduled in the cluster.
  • the cluster is preferably a cluster composed of a plurality of servers for performing cloud computing based on the Internet.
  • Each of the servers is a cluster node that provides services such as cloud computing to users.
  • the computing tasks include process services, thread services, and the like on the cluster nodes.
  • This solution needs to re-schedule multiple computing tasks under multiple cluster nodes in the cluster to optimize the allocation of cluster resources. Therefore, it is first necessary to determine multiple computing tasks waiting for scheduling. In this solution, the larger the resource pool is, the more computing tasks are determined to be scheduled, the higher the matching degree of the scheduling is to the system, and the better the scheduling accuracy is, the more significant the optimization effect of cluster resource allocation is. .
  • each of the computing tasks in the cluster for example, in a cloud resource system, has a backup computing task corresponding thereto, so that the data information corresponding to the computing task is obtained.
  • the computing task and the corresponding backup computing task ensure the consistency of the data information through a synchronization mechanism.
  • the plurality of computing tasks to be scheduled in the cluster that are acquired may also be deployed on third-party devices other than the device 1. That is, a plurality of computing tasks to be scheduled are collected from the third-party device, and then the device 1 performs corresponding information processing, task scheduling generation, and execution operations.
  • step S202 the device 1 divides the plurality of computing tasks into a plurality of task subsets according to task load information corresponding to the computing task, wherein each task subset corresponds to the cluster respectively A cluster node.
  • the task load information of the plurality of computing tasks waiting to be scheduled is first obtained.
  • the task load information includes various measurable indicator data information corresponding to the computing task, including but not limited to CPU usage, memory usage, network usage, memory usage, network card traffic, and the like, and attributes related to the computing task.
  • the task load information may also be a comprehensive measurable indicator data composed of a plurality of single and specific measurable indicator data information, for example, a single measurable indicator data information, CPU usage, memory usage, and network. Use a certain combination of calculations to generate a comprehensive indicator data, such as based on the actual situation, set a certain weight for the CPU usage, memory usage and network usage parameters, to obtain the required comprehensive measurable indicator data.
  • computation tasks are divided into the same task subset.
  • the computing tasks A, B, C, and D to be scheduled belong to the cluster nodes 1, 2, 3, and 4 respectively; and the task load information of the computing tasks A, B, C, and D are analyzed, and based on the specific Dividing operations, and finally selecting three tasks of computing tasks A, B, and C to be re-divided into one task
  • the subset, and corresponding to cluster node 1, the generation of this new combination of computing tasks is an optimization of the utilization of cluster resources.
  • the service capability of the cloud computing provided by the cluster resource presents a dynamic distribution and real-time change characteristics, and the task load information of the calculation data source corresponding to the division operation described in the solution can be well reflected.
  • the data dynamic change for example, the task load information may determine specific numerical information corresponding to a certain determinable indicator at a determined time period or time point.
  • step S201 and the step S202 can be implemented on the same device. Further, in an actual application, the operations corresponding to the step S201 and the step S202 may be performed on different devices, and the information should be included in the scope of protection of the present application, and is hereby incorporated by reference. Included here.
  • the present application divides each of the computing tasks into a task subset corresponding to a certain cluster node by using task load information corresponding to a plurality of computing tasks to be scheduled in the cluster, thereby The plurality of computing tasks to be scheduled find the best partitioning combination, thereby maximizing the utilization of resources of the entire cluster and realizing a more reasonable system resource scheduling strategy.
  • the method further includes step S203 (not shown), in step S203, the device 1 assigns the task subset to the corresponding cluster node, and performs the The computing task in the subset of tasks.
  • the resource usage of the computing task is collected in real time, and the task scheduling is performed by the computing engine.
  • the system resources corresponding to the computing task are ready for storage.
  • the computing task can directly use the scheduled system resources.
  • the divided plurality of computing tasks are allocated to the corresponding cluster nodes, and the cluster resources are shared by the cluster nodes.
  • each computing task obtained by re-division under the cluster nodes is executed.
  • the resources that need to be consumed, for example, when the corresponding Internet user requests to perform a certain cloud computing task the resources such as the network card traffic, the CPU, the memory, and the like that are required to be used from the cluster node corresponding to the cloud computing task are all resources. Divided into the total amount.
  • the execution of the computing task is completed based on the cluster node.
  • the cluster uses the control system to analyze and collect related data information of the cluster node and the computing task under the cluster node, and store related information in the control. In the system, this is used as the basic data for the information scheduling strategy.
  • steps S201, S202 and S203 can be implemented in the same device. Further, in an actual application, the operations corresponding to the step S201, the step S202, and the step S203 may be implemented on different devices, and the information should be included in the scope of protection of the present application, and is hereby incorporated by reference. Included here.
  • step S202 the device 1 divides the plurality of computing tasks into two groups according to the task load information corresponding to the computing task and the node resource threshold information corresponding to the cluster. A plurality of task subsets, wherein each task subset corresponds to one cluster node in the cluster respectively.
  • the plurality of computing tasks to be scheduled in the cluster are divided into the same task subset, and each task subset corresponds to one cluster node in the cluster, and the partitioning operation needs to obtain the task load information of the to-be-scheduled computing task.
  • the node resource threshold information includes a maximum value allowed by the pressure load of the cluster node.
  • the node resource threshold information corresponds to the task load information.
  • the task load information includes various measurable indicator data information corresponding to the computing task
  • the node resource threshold information also includes the The total amount of resource load corresponding to each measurable indicator at the cluster node.
  • the task load information of the to-be-scheduled computing task information includes the network card traffic, and when the computing task partitioning operation is performed, the node resource threshold information corresponding to the set cluster correspondingly includes the network card traffic.
  • the sum of the values of the task load information of each computing task under the same cluster node with respect to the same measurable indicator data does not exceed the total resource load of the metric indicator corresponding to the cluster node. the amount.
  • the computing tasks E, F, and G are divided into the same cluster node.
  • the task load information includes the network card traffic
  • the sum of the network card traffic consumed by each of the computing tasks E, F, and G is a
  • the total network load corresponding to the NIC traffic of the cluster node is b
  • the NIC traffic value a cannot exceed b. Therefore, it is feasible to ensure that the operation of each computing task under the same cluster node by dividing the set to the same cluster node is feasible. , optimized. Further, preferably, an optimal range may be set for the node resource threshold information corresponding to the cluster, for example, the downward fluctuation does not exceed 10%, that is, the total network load corresponding to the network node traffic of the cluster node is b.
  • the cluster node can be set that the sum of the computing tasks under the cluster node reaches the range of 0.9b to b is optimal, below 0.9b, the cluster node will have too many The resource is not utilized, which brings waste of resources, and exceeds the total amount of resource load b, and the cluster node is under excessive pressure.
  • the setting of the node resource threshold information may be obtained by performing a stress test on the cluster node, and based on sampling and analyzing the specific operation conditions of each computing task under the cluster node.
  • the resource threshold information of the cluster node is the same when the configurations of the servers corresponding to the cluster nodes are the same, for example, the server software configuration, the hardware configuration, and the running environment are the same.
  • different configurations may be set for different servers that are cluster nodes based on the needs of the cluster resource allocation or the specific computing tasks, and the resource threshold information of different nodes in the cluster may be set differently.
  • the server configurations corresponding to all the cluster nodes are the same, so that the node resource threshold information corresponding to each node in the cluster is consistent.
  • the method for setting the threshold resource information of the foregoing node such as the stress test
  • the method for setting other node resource threshold information is applicable to the present application, and should also be included in the protection of the present application. It is within the scope and is hereby incorporated by reference.
  • the device 1 performs a dividing operation according to the task load information corresponding to the computing task and the node resource threshold information corresponding to the cluster until the plurality of computing tasks are divided into multiple a task subset, wherein each task subset corresponds to one cluster node in the cluster; wherein the dividing operation comprises: taking the computing task to be divided into the first plurality of computing tasks as a first Computing a task; determining one or more candidate task subsets, wherein the candidate task subset includes the first computing task and at least one other of the plurality of computing tasks to be divided, the candidate The accumulated information of the task load information of each of the computing tasks in the task subset satisfies the node resource threshold information; and the task subset is preferably determined from the one or more candidate task subsets.
  • a dividing operation may be performed on the plurality of computing tasks.
  • the computing task to be divided is selected as the first computing task among the plurality of computing tasks to be scheduled, and the selected method may be random, or may be based on a certain regularity, for example, preferentially selecting the task load.
  • the calculation task corresponding to the increased load of the information it can be set that the selected first computing task corresponds to one cluster node 1. Subsequently, for the first computing task described One or more computing tasks are selected among the remaining plurality of computing tasks to be scheduled.
  • the condition to be satisfied by the matching includes that the accumulated information of the task load information corresponding to the first computing task and the one or more computing tasks matching the same cannot exceed the maximum value of the corresponding node resource threshold information.
  • the task load information corresponding to each of the computing tasks is set to a value corresponding to a certain metric indicator data at a certain determined time point.
  • the task load information is set to the network card traffic information, and the time point T in the time dimension is selected, and the node resource threshold corresponding to the network card traffic information of the cluster node 1 corresponding to the first computing task A is L,
  • the optimal range of the node resource threshold L is downwardly floating no more than 10%.
  • the computing task matching the matching task is found. If the network card traffic information of the first computing task A is L1 at the time point T, if the task B is matched for the task B, the corresponding network card traffic information is L2, and if the sum of L1 and L2 has exceeded the The node resource threshold L, if the computing task B does not match the first computing task A, the computing task B is discarded, and a new matching computing task is sought; if the sum of L1 and L2 is already at the node resource Within the optimal range of the threshold L, the computing task A and the computing task B satisfy the matching condition, and may correspond to a candidate task subset.
  • the continuation may be continued.
  • the determined computing tasks in the one or more candidate task subsets are all under the various task load information, such as the cumulative information of the task load information corresponding to each time dimension under various measurable indicator data.
  • the node resource threshold information corresponding to each is met.
  • the metric data that the partitioning operation depends on may be multiple, and may even include comprehensive indicator data composed of multiple single measurable indicator data; at the same time, the acquired time dimension may be multiple, specifically There may be more than one time point, and the final partitioning result may have multiple possibilities based on different parameter changes.
  • one or more candidate task subsets including the first computing task and one or more of the other computing tasks may be present.
  • the determining the subset of tasks from the one or more candidate task subsets comprises: determining subset related information of the candidate task subsets; and according to the subset related information from the one
  • the subset of tasks is preferably determined in a plurality of candidate task subsets.
  • the subset related information includes a pulse ratio of the candidate task subset.
  • the subset related information includes a pulse ratio of the candidate task subset.
  • the sum of the data values L1, L2, and L3 corresponding to the network card traffic information of the computing tasks A, B, and C should not exceed the node resource threshold L.
  • the ratio of the maximum value of the corresponding L1, L2, and L3 to the average value of L1, L2, and L3 is the pulse value of the candidate task subset M at time T1, when the time dimension is hour.
  • the subset related information may further include: calculating, by the respective candidate tasks in the same candidate task subset, a sum of values corresponding to the same determined time point of the same measurable indicator data corresponding to the task subset The difference between the node resource threshold information of the cluster node. For example, for one candidate task subset N, including the first computing task A, the computing task D, and the computing task E, setting the task load information to network card traffic information, selecting a time dimension of hours, and the first The node resource threshold corresponding to the network card traffic information of the cluster node 1 corresponding to the task A is L.
  • the optimal range of the node resource threshold L is preferably not to be less than 10%.
  • the sum of the data values L1, L4, and L5 corresponding to the network card traffic information of the computing tasks A, D, and E should not exceed the node resource threshold L, and the corresponding difference is L-(L1+ L2+L3).
  • a certain seed set related information may be used for the candidate task subset for further screening.
  • the plurality of subset related information may also be used for comprehensive comparison at the same time.
  • the candidate task subset M and the candidate task subset N are respectively calculated for the corresponding pulse ratio and the difference, thereby obtaining an optimal select.
  • the priority of the pulse ratio is higher than the difference, for example, the optimal range of the threshold resource threshold L is preferably not more than 10% downward, and for the pulse ratio, It is additionally specified that it is preferably in a wider range of the node resource threshold L, for example, in the range of 80% to 95%, if the candidate task subset M corresponds to a pulse ratio in the range of 80% to 95%. If the candidate task subset N does not reach this range, the candidate task subset M is preferred regardless of the difference between the two sets of task subsets.
  • the optimal range of the node resource threshold L is 10%, and the pulse ratio corresponds to a wider preferred range of the node resource threshold L, for example, a range of 80% to 95%, which is merely an example, It can be flexibly arranged based on actual business needs.
  • subset related information includes a pulse ratio of the candidate task subset
  • the subset related information may further include: calculating each computing task under the same candidate task subset in the same
  • the difference between the sum of the corresponding values at the same determined time point of the measurable indicator data and the node resource threshold information of the cluster node corresponding to the task subset is only an example, and other subset related information may be applied to the application, It is intended to be included within the scope of this application and is hereby incorporated by reference.
  • the accumulated information of the task load information of each of the computing tasks in the candidate task subset in the device 1 satisfies the node resource threshold information, and includes: task load information of each computing task in the candidate task subset The accumulated information satisfies the node resource threshold information by dimension.
  • the task load information of the computing task can comprehensively and objectively reflect the resource cost requirement of the computing task. Therefore, when performing the dividing operation of the multiple computing tasks, the metric data according to the metric is multi-dimensional.
  • the metric indicator data letter may be derived from the following various types, such as CPU usage, memory usage, network usage, memory usage, network card traffic, and the like, and the attribute indicators related to the computing task are not only more A single, specific measurable indicator data information, or a plurality of comprehensive measurable indicator data composited by multiple indicators, such as CPU usage and memory usage for a single measurable indicator data information The rate and network usage are calculated according to a certain combination to generate a comprehensive indicator data.
  • the CPU usage rate, the memory usage rate, and the network usage parameters are respectively set to a certain weight, and the required comprehensive measurable indicator data is obtained.
  • the diversification of the measurable indicator data corresponding to the task load information is to provide the most comprehensive basic data information for the division of the computing task, so that the optimal partitioning method can be found according to the purpose and expectation of the computing task. Therefore, the allocation and utilization of cluster resources is the most reasonable and more in line with actual business needs.
  • the time dimension on which the basis is based is also multi-dimensional, and the recorded task load information data can be recorded according to any required time unit such as year, month, day, hour, minute, second, and the like.
  • the data information in the appropriate group or groups of dimensions may be selected for use according to the specific division purpose of the plurality of computing tasks to be invoked.
  • step S202 the device 1 calculates log information of the plurality of historical computing tasks according to the task calculation log information of the cluster; and calculates task cost information of the task according to the plurality of historical calculation tasks. Determining task load information of the computing task.
  • the task load information of the computing task to be scheduled is dynamically changed, so the task load information of the computing task to be scheduled obtained at any time is already a historical data.
  • the computing tasks such as Internet cloud computing tasks
  • the computing tasks are similar for the same type of computing tasks, especially for a series of computing tasks with similar or identical parameter conditions, and their computing execution. Therefore, a particular historical computing task is informative for subsequent computing tasks that match it.
  • a well-matchable model history calculation task can be found for the current computing task to be scheduled, and the task overhead information of the task can be calculated based on the history, such as in different time dimensions.
  • the pressure data corresponding to the different measurable indicators is used to infer the task cost that may be required for the to-be-scheduled computing task, and the task load information required to divide the plurality of computing tasks may be obtained accordingly.
  • the foregoing task calculation log information according to the cluster determines the task overhead information of the plurality of historical computing tasks; and determines the computing task according to the task overhead information of the plurality of historical computing tasks.
  • the task load information is only an example. Other task load information for determining the computing task, as applicable to the present application, is also included in the scope of protection of the present application, and is hereby incorporated by reference.
  • determining the task load information of the computing task according to the task cost information of the plurality of historical computing tasks comprises: filtering and calculating from the plurality of historical computing tasks according to the task related information of the computing task a preferred historical computing task matching the tasks; determining task load information of the computing tasks according to the task overhead information of the preferred historical computing tasks.
  • the task related information of the computing task includes various related information that can describe, locate a computing task execution condition, an execution situation, and the like, such as various parameters involved in the execution of the computing task, for example, Server software and hardware requirements.
  • the historical computing task corresponding to the computing task may be exactly the same dynamic computing task as the computing task, but the corresponding data has a regular change due to temporal changes; the corresponding history
  • the computing task and the computing task may also be two completely independent dynamic computing tasks, but because of the great similarity between the two, it is suitable to match.
  • determining the task load information of the computing task according to the task cost information of the plurality of historical computing tasks comprises: clustering the plurality of historical computing tasks according to task related information of the plurality of historical computing tasks Processing, determining task cost information corresponding to each computing task cluster; determining, according to the task related information of the computing task, a preferred computing task cluster that matches the computing task; and calculating the task clustering according to the preferred The task overhead information determines task load information of the computing task.
  • the preferred historical computing task matching the computing task is selected based on the plurality of historical computing tasks, and the task load information of the computing task can be determined objectively and accurately using the task cost information of the preferred historical computing task.
  • the plurality of historical computing tasks may be first performed, and clustering processing is performed based on the determined task related information, where the clustering process uses one or several metrics as a standard similarity. Sexuality, minimized between unified clusters, and maximized among different clusters.
  • the clustering algorithm makes the plurality of historical computing tasks aggregate into multiple categories. On the one hand, the information needed to be compared can be greatly reduced to A plurality of historical computing task clusters.
  • the task overhead information corresponding to the historical computing task clustering by clustering processing is a statistical analysis result, which is more universal and widely applicable, according to
  • the clustering criterion corresponding to the preferred computing task cluster may find matching data information for the computing task, and determine task load information of the computing task by using task cost information corresponding to the matched preferred computing task cluster. More efficient and feasible.
  • FIG. 3 to FIG. 6 show a preferred embodiment of the present application. Based on the computing tasks to be scheduled by the two nodes M and N in the cluster, after the scheduling method of the present application is performed, the two clusters are re-divided. The calculation tasks under the point optimize the allocation of cluster resources.
  • FIG. 3 is a schematic diagram of a computing task load before scheduling of a cluster node M according to a preferred embodiment of the present application
  • FIG. 4 is a schematic diagram showing a computing task load before scheduling of a cluster node N according to a preferred embodiment of the present application
  • a schematic diagram of computing task load after cluster node M scheduling according to a preferred embodiment of the present application is shown
  • FIG. 6 is a schematic diagram showing computing task load after cluster node N scheduling according to a preferred embodiment of the present application.
  • FIG. 3 shows the task load of the computing tasks 1, 2, 3, 4 under the cluster node M before the partitioning operation
  • FIG. 4 shows the computing tasks 6, 7, 8, and 9 under the cluster node N.
  • the time dimension may select any dimension such as year, month, day, hour, and the like
  • the task load information includes various measurable indicator data information corresponding to the computing task, including but not limited to CPU usage and memory usage. , network usage, memory usage, network card traffic, and other attribute indicators related to the computing task. It can also be a comprehensive measurable indicator data composed of multiple single and specific measurable indicator data information.
  • each of the curves in FIG. 3 and FIG. 4 represents the pressure load condition of the corresponding computing task within one week.
  • FIG. 3 shows that the corresponding four computing tasks are in the low peak period of the business on the 2nd, 4th, and 6th day of the week, and the corresponding pressure load is relatively small, and the traffic peaks are at 1, 3, 5, and 7 in the week. Period, the corresponding pressure load is relatively large;
  • Figure 4 shows that the corresponding four computing tasks are in the peak period of business every week, 2, 4, and 6, the corresponding pressure load is relatively large, and in the week 1, 3, 5, 7 are in the low peak period of the business, and the corresponding pressure load is relatively small.
  • the computing tasks 1, 2, 3, 4 under the M cluster node and the computing tasks 6, 7, 8, and 9 under the cluster node N are used as scheduling tasks to be scheduled, and the two obtained by the dividing operation of the method are obtained.
  • cluster nodes M and N are at a certain time
  • the cluster resources are well balanced and complementary, which alleviates the excessive pressure of the cluster nodes at some time points, and the pressure is too small at other time points.
  • the specific division operation optimization effect is embodied in FIG.
  • the post-scheduled calculation tasks 2 and 4 have relatively small pressure loads on the 2nd, 4th, and 6th week, in the week 1.
  • the pressure load is relatively large
  • the calculation tasks 6 and 8 divided into the M cluster nodes have relatively large pressure loads on the 2nd, 4th, and 6th week, and the pressure load is 1, 3, 5, and 7 days.
  • the calculation tasks 5 and 7 have relatively large pressure loads on the 2nd, 4th, and 6th day of the week, and the pressure load is relatively small on the 1, 3, 5, and 7th days of the week, and is divided into N clusters.
  • the calculation tasks 1 and 3 of the node have relatively small pressure loads on days 2, 4, and 6 of the week, and the pressure loads are relatively large on days 1, 3, 5, and 7 of the week.
  • the accumulated value of the pressure load of each computing task under one cluster node after the scheduling is maintained under the node threshold information, based on the pressure load of each computing task.
  • the high and low balances have achieved optimal results in resource utilization.
  • a new task subset is obtained based on the rescheduling division of the plurality of computing tasks, and the task load information in the time dimension corresponding to the computing task under the computing task is stored as basic data in the control system corresponding to the cluster, as The historical computing task provides reference information data for the scheduling of the subsequent target computing tasks.

Abstract

本申请的目的是提供一种用于在集群中调度计算任务的方法与设备；获取集群中待调度的多个计算任务；根据所述计算任务所对应的任务负荷信息将所述多个计算任务划分为多个任务子集，其中，各任务子集分别对应于所述集群中一个集群节点。与现有技术相比，本申请通过基于集群中待调度的多个计算任务对应的任务负荷信息，将每个所述计算任务都划分到一个对应于某个集群节点的任务子集中，从而，为所述集群中待调度的多个计算任务找到了最佳匹配组合，从而最大化利用整个集群的资源，实现更加合理的系统资源调度策略。

Description

一种用于在集群中调度计算任务的方法与设备 技术领域
本申请涉及计算机领域，尤其涉及一种用于在集群中调度计算任务的技术。
背景技术
利用互联网实现云计算的过程中，在每一个云服务节点上存在多个计算任务，需要针对所述计算任务进行资源隔离，进而，相互隔离的计算任务基于所述云服务节点调用整个集群分配的各种资源。
目前，该领域采用的方案有：
一是绝对控制方法。即对每一个云服务节点对应的多个计算任务，可以依据所述云服务节点自身的资源总量进行资源量的分配，每个计算任务最多可以使用的资源量为确定数值。此种方法，在每个计算任务实际需要的资源量大于其所分配到的资源量的确定数值时，所述计算任务的需求得不到满足；而当所述计算任务需要的资源量大大小于所分配到的资源量的确定数值时，又会带来资源的浪费。
二是相对控制方法。即对每一个云服务节点对应的多个计算任务，可以依据所述云服务节点自身的资源总量、对每个计算任务按照一定比例进行资源量的分配，此时，若某个计算任务压力过大时，若所对应的云服务节点按照权重切割，则会由于该计算任务的异常情况影响到其他计算任务的顺利进行。
发明内容
本申请的目的是提供一种用于在集群中调度计算任务的方法与设备。
根据本申请的一个方面，提供了一种用于在集群中调度计算任务的方法，包括：
获取集群中待调度的多个计算任务；
根据所述计算任务所对应的任务负荷信息将所述多个计算任务划分为多个任务子集，其中，各任务子集分别对应于所述集群中一个集群节点。
根据本申请的另一方面，还提供了一种用于在集群中调度计算任务的设备，包括：
第一装置，用于获取集群中待调度的多个计算任务；
第二装置，用于根据所述计算任务所对应的任务负荷信息将所述多个计算任务划分为多个任务子集，其中，各任务子集分别对应于所述集群中一个集群节点。
与现有技术相比，本申请通过基于集群中待调度的多个计算任务对应的任务负荷信息，将每个所述计算任务都划分到一个对应于某个集群节点的任务子集中，从而，为所述集群中待调度的多个计算任务找到了最佳划分组合，从而最大化利用整个集群的资源，实现更加合理的系统资源调度策略。
附图说明
通过阅读参照以下附图所作的对非限制性实施例所作的详细描述，本申请的其它特征、目的和优点将会变得更明显：
图1示出根据本申请一个方面的一种用于在集群中调度计算任务的设备示意图；
图2示出根据本申请另一个方面的一种用于在集群中调度计算任务的方法流程图；
图3示出根据本申请一个优选实施例的集群节点M调度前的计算任务负荷示意图；
图4示出根据本申请一个优选实施例的集群节点N调度前的计算任务负荷示意图；
图5示出根据本申请一个优选实施例的集群节点M调度后的计算任务负荷示意图；
图6示出根据本申请一个优选实施例的集群节点N调度后的计算任务负荷示意图。
附图中相同或相似的附图标记代表相同或相似的部件。
具体实施方式
下面结合附图对本申请作进一步详细描述。
在本申请一个典型的配置中，终端、服务网络的设备和可信方均包括一个或多个处理器(CPU)、输入/输出接口、网络接口和内存。
内存可能包括计算机可读介质中的非永久性存储器，随机存取存储器(RAM)和/或非易失性内存等形式，如只读存储器(ROM)或闪存(flash RAM)。内存是计算机可读介质的示例。
计算机可读介质包括永久性和非永久性、可移动和非可移动媒体可以由任何方法或技术来实现信息存储。信息可以是计算机可读指令、数据结构、程序的模块或其他数据。计算机的存储介质的例子包括，但不限于相变内存(PRAM)、静态随机存取存储器(SRAM)、动态随机存取存储器(DRAM)、其他类型的随机存取存储器(RAM)、只读存储器(ROM)、电可擦除可编程只读存储器(EEPROM)、快闪记忆体或其他内存技术、只读光盘只读存储器(CD-ROM)、数字多功能光盘(DVD)或其他光学存储、磁盒式磁带，磁带磁盘存储或其他磁性存储设备或任何其他非传输介质，可用于存储可以被计算设备访问的信息。按照本文中的界定，计算机可读介质不包括非暂存电脑可读媒体(transitory media)，如调制的数据信号和载波。
图1示出根据本申请一个方面的一种用于在集群中调度计算任务的设备示意图。所述用于在集群中调度计算任务的设备1包括第一装置101和第二装置102。
其中，第一装置101获取集群中待调度的多个计算任务；第二装置102用于根据所述计算任务所对应的任务负荷信息将所述多个计算任务划分为多个任务子集，其中，各任务子集分别对应于所述集群中一个集群节点。
具体地，所述设备1的第一装置101获取集群中待调度的多个计算任务。在此，所述的集群，优选地为基于互联网用于进行云计算的多台服务器组成的集群。每一台所述服务器都是为用户提供云计算等服务的集群节 点。在所述每一台服务器上都有若干个正在进行的计算任务。所述的计算任务包括在集群节点上的进程服务、线程服务等。本方案需要对集群中多个集群节点下的多个计算任务进行重新调度划分，从而优化集群资源的分配，所以首先需要确定等待调度安排的多个计算任务。在本方案中，资源池越大，确定获取的待调度的计算任务越多，调度对于系统的匹配度就会越高，进而调度精确度就会越好，集群资源分配的优化效果就越显著。
此外，所述每一个计算任务在所述集群中，例如在云资源系统中，都有与之对应的的备份计算任务，从而使得所述计算任务对应的数据信息得以被保存，进一步，所述计算任务和与之对应的备份计算任务之间通过同步机制保障数据信息的一致性。在此，通过对计算任务进行灾备安排，可以避免由于数据信息的损毁、灭失等突发状况带来的损失。
在此，本领域技术人员应该能够理解，所述第一装置获取的集群中待调度的多个计算任务也可以来自于所述设备1之外的第三方设备。即从第三方设备中收集待调度的多个计算任务，再由所述设备1进行相应的信息处理、任务调度的生成并执行等操作。
接着，所述设备1的第二装置102根据所述计算任务所对应的任务负荷信息将所述多个计算任务划分为多个任务子集，其中，各任务子集分别对应于所述集群中一个集群节点。在此，若要为多个不同的计算任务进行最佳划分，以便使得集群资源利用最大化，首先要获得等待调度的多个计算任务的任务负荷信息。所述的任务负荷信息包括所述计算任务对应的各种可度量指标数据信息，包括但不限于CPU使用率、内存使用率、网络使用、内存使用、网卡流量等与所述计算任务相关的属性指标，这些指标数据能够直观反应所述计算任务对于具体某一种或几种相关集群资源的使用和消耗需要，如一个进程任务的运行需要的对应的CPU使用率、或是内存使用率情况等。所述计算任务消耗的某一类集群资源越多，则带给它所对应的集群节点的压力负荷也就越大。此外，所述的任务负荷信息也可以是多个单一、具体的可度量指标数据信息复合而成的一个综合可度量指标数据，例如对于单一可度量指标数据信息CPU使用率、内存使用率和网络使用按照一定的组合计算生成一个综合指标数据，如基于实际情况 为CPU使用率、内存使用率和网络使用参数分别设置一定的权重，得到需要的综合可度量指标数据。
例如，对所有的待调度的多个计算任务基于同一可度量指标在同一个时间段或是时间点的具体数值进行分析比较，基于一定的划分操作，就可以将满足最优划分条件的多个计算任务划分到同一个任务子集中。例如存在待调度的计算任务A、B、C、D分别属于集群节点1、2、3、4；通过对所述计算任务A、B、C、D各自的任务负荷信息进行分析，并基于具体的划分操作，最终选出计算任务A、B、C三个任务重新划分到一个任务子集中，并对应于集群节点1，这一新的计算任务的组合的生成是对集群资源的利用率的优化。在此，集群资源提供的云计算的服务能力呈现的是一种动态分布、实时变化的特点，而作为本方案中所述划分操作对应的计算数据来源的任务负荷信息即可以很好的反应这种数据动态变化，如，所述任务负荷信息可以确定到某一确定可度量指标在确定时间段或时间点对应的具体数值信息。
在此，本领域技术人员应能理解，所述第一装置和所述第二装置可以是同一设备中的不同装置。进一步，在实际应用中，将所述第一装置和所述第二装置部署在不同的设备上也可以完成本方案，此时如可适用本申请，也应包含在本申请的保护范围以内，并在此以引用方式包含于此。
在此，本申请通过基于集群中待调度的多个计算任务对应的任务负荷信息，将每个所述计算任务都划分到一个对应于某个集群节点的任务子集中，从而，为所述集群中待调度的多个计算任务找到了最佳划分组合，从而最大化利用整个集群的资源，实现更加合理的系统资源调度策略。优选地，所述用于在集群中调度计算任务的设备1还包括第三装置(未示出)，所述第三装置将所述任务子集分配至对应的所述集群节点，并在所述集群节点中执行所述任务子集中的所述计算任务。
具体地，通过实时采集所述计算任务的资源使用情况，并通过计算引擎，再进行任务调度，所述互联网用户在请求某一计算任务时，所述计算任务对应的系统资源已经储备就绪，所述计算任务即可直接使用调度后的系统资源。在此，经过划分的所述多个计算任务分配到对应的集群节点之 下，并通过所述集群节点分享着集群资源，此时，所述集群节点下重新划分得到的各个计算任务在执行时所需要消耗的资源，例如在相应互联网用户请求进行某一云计算任务时，所需要的使用的如网卡流量、CPU、内存等资源都要从所述云计算任务对应的集群节点所拥有的资源总量中划分出来。所以所述计算任务的执行是基于所述集群节点完成的。在此，优选地，在云计算任务中，所述集群会运用控制系统对所述集群节点、所述集群节点下的计算任务的相关数据信息进行分析采集，并将相关信息存储在所述控制系统中，以此作为信息调度策略制定的基础数据。
在此，本领域技术人员应能理解，所述第一装置、所述第二装置和所述第三装置可以是同一设备中的不同装置。进一步，在实际应用中，将所述第一装置、所述第二装置和所述第三装置部署在不同的设备上也可以完成本方案，此时如可适用本申请，也应包含在本申请的保护范围以内，并在此以引用方式包含于此。
在一个优选的实施例中，所述设备1的所述第二装置102根据所述计算任务所对应的任务负荷信息，并结合所述集群对应的节点资源阈值信息，将所述多个计算任务划分为多个任务子集，其中，各任务子集分别对应于所述集群中一个集群节点。
具体地，将集群中待调度的多个计算任务划分到同一任务子集中，每个任务子集与所述集群中一个集群节点相对应，划分操作需要获得所述待调度计算任务的任务负荷信息，同时，还需要设定集群对应的节点资源阈值信息，优选地，所述节点资源阈值信息包括所述集群节点的压力负荷所允许的最大值。进一步，所述节点资源阈值信息与所述任务负荷信息相对应，如当所述任务负荷信息包括所述计算任务对应的各种可度量指标数据信息，则所述节点资源阈值信息也包括所述各种可度量指标在所述集群节点分别对应的资源负荷总量。例如，所述待调度计算任务信息的任务负荷信息中包括网卡流量，则在进行计算任务划分操作时，所设定的集群对应的节点资源阈值信息就会对应包括所述网卡流量。在此，优选地，划分到同一个集群节点下的各个计算任务关于同一个可度量指标数据的任务负荷信息的数值之和要不超过所述可度量指标在所述集群节点对应的资源 负荷总量。例如，存在计算任务E、F、G划分到同一个集群节点之下，若所述任务负荷信息中包括网卡流量，计算任务E、F、G各自运行消耗的网卡流量之和为a，所述集群节点对应的网卡流量这一资源负荷总量为b，则网卡流量数值a不能超过b，由此，才能保证通过划分集合到同一集群节点下的各个计算任务在对应集群节点下的操作是可行的、优化的。进一步，优选地，对于所述所述集群对应的节点资源阈值信息还可以设置一个最优范围，例如向下浮动不超过10％，即上述集群节点对应的网卡流量这一资源负荷总量为b，则可以设定当所述集群节点下的各个计算任务之和达到0.9b～b范围之间是最优的效果，低于0.9b，所述集群节点会有过多资源未被利用，带来资源的浪费，而超过资源负荷总量为b，所述集群节点压力过大。
在此，所述节点资源阈值信息的设定可以是通过对所述集群节点进行的压力测试，并基于所述集群节点下各个计算任务的具体运行情况抽样采集分析而得的。理论上，当集群节点对应的服务器的配置一致，例如服务器软件配置、硬件配置和运行环境等条件一致，所述集群节点的资源阈值信息也是相同。在实际应用中，也可以基于集群资源分配的需要，或是具体计算任务的需要，为作为集群节点的不同服务器设定不同的配置，进而将集群下不同节点的资源阈值信息进行差别设置。在此，优选地，所有集群节点对应的服务器配置相同，从而，所述集群下各个节点对应的节点资源阈值信息一致。
本领域技术人员应能理解，上述节点资源阈值信息的设定方法，如压力测试等方法仅为举例，其他节点资源阈值信息的设定方法如可适用本申请，也应包含在本申请的保护范围以内，并在此以引用方式包含于此。
优选地，所述设备1的所述第二装置102根据所述计算任务所对应的任务负荷信息，并结合所述集群对应的节点资源阈值信息，执行划分操作直至将所述多个计算任务划分为多个任务子集，其中，各任务子集分别对应于所述集群中一个集群节点；其中，所述划分操作包括：在所述多个计算任务中取一待划分的所述计算任务作为第一计算任务；确定一个或多个候选任务子集，其中，所述候选任务子集包括所述第一计算任务及所述多 个计算任务中至少一个其他待划分的所述计算任务，所述候选任务子集中各所述计算任务的任务负荷信息的累计信息满足所述节点资源阈值信息；从所述一个或多个候选任务子集中优选确定所述任务子集。
具体地，为了将预先从属于集群下不同节点的多个计算任务重新调度到多个任务子集中，可以对所述多个计算任务执行划分操作。首先，在所述待调度的多个计算任务中选取一个待划分的所述计算任务作为第一计算任务，选取的方法可以是随机的，也可以依据一定的规律性，例如，优先选取任务负荷信息对应的负荷加大的计算任务。在此，可以设定所选中的第一计算任务对应于一个集群节点1。随后，为所述的第一计算任务在剩下的所述多个待调度的计算任务中选择与之相匹配的一个或多个计算任务。所述的相匹配所要满足的条件包括所述第一计算任务和与之相匹配的一个或多个计算任务对应的任务负荷信息的累计信息要不能超过对应节点资源阈值信息的最大值。在此，优选地，所述各个计算任务对应的任务负荷信息设定为某一个可度量指标数据在某一个确定时间点对应的数值。例如，设定所述任务负荷信息为网卡流量信息，选取时间维度上的时间点T时，并且所述第一计算任务A所对应的集群节点1对应于网卡流量信息的节点资源阈值为L，在此，可以优选节点资源阈值L的最优范围是向下浮动不超过10％。此时为所述的第一计算任务A，寻找到与其相匹配的计算任务。若在时间点T，所述第一计算任务A的网卡流量信息是L1，若此时为其匹配计算任务B，对应的网卡流量信息是L2，若此时L1与L2之和已经超过所述节点资源阈值L，则所述计算任务B与第一计算任务A并不匹配，则放弃所述计算任务B，寻找新的匹配计算任务；若此时L1与L2之和已经在所述节点资源阈值L的最优范围之内，说明所述计算任务A与所述计算任务B满足匹配条件，可以对应为一个候选任务子集。更优选地，若此时L1与L2之和满足小于节点资源阈值L的条件，但是其值在所述节点资源阈值L的最优范围之外，此时为了充分利用集群节点的资源，可以继续寻找一个或多个计算任务来与第一计算任务A和计算任务B匹配。进一步，所确定的所述一个或多个候选任务子集中各所述计算任务在各种任务负荷信息下，如各种可度量指标数据下的各个时间维度对应 的任务负荷信息的累计信息都要满足各自所对应的所述节点资源阈值信息。
进一步，实际运行中，划分操作所依赖的可度量指标数据可以有多种，甚至还可以包括多个单一可度量指标数据组成的综合指标数据；同时，所获取的时间维度可能为多个，具体的时间点也可能存在多个，进而最终的划分结果也会基于不同的参数变化存在多种可能。经过划分操作，得到同时包含第一计算任务和一个或多个其他所述计算任务的候选任务子集可能存在一个或多个。接下来，可以基于一定的信息，例如脉冲比值等数据，进一步进行优选判断。
更优选地，所述从所述一个或多个候选任务子集中优选确定所述任务子集包括：确定所述候选任务子集的子集相关信息；根据所述子集相关信息从所述一个或多个候选任务子集中优选确定所述任务子集。
具体地，当通过一定的划分操作，基于所述第一计算任务确定了多个候选任务子集时，需要基于所述任务子集的子集相关信息对多个候选任务子集进行更进一步的确定。在此，所述的子集相关信息包括候选任务子集的脉冲比值。例如，对于一个所述候选任务子集M，包括第一计算任务A、计算任务B和计算任务C，设定所述任务负荷信息为网卡流量信息，选取时间维度为小时，并且所述第一计算任务A所对应的集群节点1对应于网卡流量信息的节点资源阈值为L，在此，可以优选节点资源阈值L的最优范围是向下浮动不超过10％。作为候选任务子集，所述计算任务A、B、C的网卡流量信息对应的数据值L1、L2、L3之和应该不超过节点资源阈值L。在T1时，对应的L1、L2、L3中的最大值与L1、L2、L3的平均值的比值，即为所述候选任务子集M在时间T1点下的脉冲值，当时间维度为小时，则在各个时间点T1、T2、T3……，都分别对应着一个脉冲值，所述各个时间点对应的脉冲值组成了一个集合，则所述集合中最大值与最小值的比值即为所述脉冲比值。脉冲比值越小，对应的所述候选任务子集对资源的利用效果越优。
在此，所述子集相关信息还可以包括：计算同一个候选任务子集下各个计算任务在同一个可度量指标数据的同一个确定时间点下对应的数值 之和与所述任务子集对应的集群节点的节点资源阈值信息的差值。例如，对于一个所述候选任务子集N，包括第一计算任务A、计算任务D和计算任务E，设定所述任务负荷信息为网卡流量信息，选取时间维度为小时，并且所述第一计算任务A所对应的集群节点1对应于网卡流量信息的节点资源阈值为L，在此，可以优选节点资源阈值L的最优范围是向下浮动不超过10％。作为候选任务子集，所述计算任务A、D、E的网卡流量信息对应的数据值L1、L4、L5之和应该不超过节点资源阈值L,此时对应的差值是L-(L1+L2+L3)。所述差值越小，对应的所述候选任务子集对资源的利用效果越优。
在此，对于所述候选任务子集可以使用某一种子集相关信息进行进一步筛选。优选地，也可以同时利用多种子集相关信息进行综合比较，例如，对于上述候选任务子集M和候选任务子集N分别计算各自对应的所述脉冲比值和所述差值，从而得到最优选择。具体地，在实际应用中，可以优选所述脉冲比值的优先级高于所述差值，如，优选节点资源阈值L的最优范围是向下浮动不超过10％，同时对于脉冲比值，若额外规定其在节点资源阈值L的一个更广的优选范围，例如在80％～95％范围内都是可以优选地，则若候选任务子集M对应的脉冲比值在80％～95％范围内，而所述候选任务子集N达不到这个范围，则不论两组任务子集的所述差值情况如何，会优选候选任务子集M。在此，所述节点资源阈值L的最优范围10％、所述脉冲比值对应的其在节点资源阈值L的一个更广的优选范围，例如范围80％～95％，仅为举例，其都可以基于实际的业务需要灵活安排。
在此，本领域技术人员应能理解，上述子集相关信息包括候选任务子集的脉冲比值，所述的子集相关信息还可以包括：计算同一个候选任务子集下各个计算任务在同一个可度量指标数据的同一个确定时间点下对应的数值之和与所述任务子集对应的集群节点的节点资源阈值信息的差值仅为举例，其他子集相关信息如可适用本申请，也应包含在本申请的保护范围以内，并在此以引用方式包含于此。
优选地，所述设备1中所述候选任务子集中各所述计算任务的任务负荷信息的累计信息满足所述节点资源阈值信息包括：所述候选任务子集中 各所述计算任务的任务负荷信息的累计信息按维度分别满足所述节点资源阈值信息。
具体地，为了所述计算任务的任务负荷信息能够全面、客观的反映计算任务的资源开销需要，所以在进行所述多个计算任务的划分操作时，依据的可度量指标数据是多维度的，例如，所述的可度量指标数据信可以同时分别来源于下列多种，如CPU使用率、内存使用率、网络使用、内存使用、网卡流量等与所述计算任务相关的属性指标，不仅是多个单一、具体的可度量指标数据信息，还可以是多个指标复合而成的若干个综合可度量指标数据，例如对于单一可度量指标数据信息CPU使用率、内存使用率和网络使用按照一定的组合计算生成一个综合指标数据，如基于实际情况为CPU使用率、内存使用率和网络使用参数分别设置一定的权重，得到需要的综合可度量指标数据。所述任务负荷信息所对应的可度量指标数据的多样化是为了给所述计算任务的划分，提供最全面的基础数据信息，使得可以根据计算任务的目的、预期寻找到最理想的划分方法，从而使得集群资源的分配利用最合理、并更符合实际业务需要。同时，所依据的时间维度也是多维度的，记录的任务负荷信息数据可以依据以年、月、日、时、分、秒等任何需要的时间单位来记录。在此，可以依据具体的待调用的多个计算任务的划分目的，选取适当一组或多组维度下的数据信息进行利用。
在另一个优选实施例中，所述设备1的所述第二装置102根据所述集群的任务计算日志信息，确定多个历史计算任务的任务开销信息；根据所述多个历史计算任务的任务开销信息确定所述计算任务的任务负荷信息。
具体地，所述待调度的计算任务的任务负荷信息由于是动态变化的，所以实际上任何时候获得的所述待调度的计算任务的任务负荷信息都已经是一个历史数据。但是同时，所述计算任务，例如互联网云计算任务，对于同一类型的计算任务，特别是一系列参数条件相似或相同的计算任务，它的计算执行，所要消耗的集群资源的情况是相似的，所以特定的历史计算任务对于与其相匹配的后来的计算任务是具有参考性的。进一步，如果运用较合理的匹配方法，就可以为当前所要调度的计算任务找到很好 的可匹配的模型历史计算任务，就可以基于所述历史计算任务的任务开销信息，如在不同的时间维度下对应不同可度量指标的压力数据，去推断出所述待调度计算任务可能需要的任务开销，也就可以据此获得划分所述多个计算任务所需要的任务负荷信息。
在此，本领域技术人员应能理解，上述据所述集群的任务计算日志信息，确定多个历史计算任务的任务开销信息；根据所述多个历史计算任务的任务开销信息确定所述计算任务的任务负荷信息仅为举例，其他确定所述计算任务的任务负荷信息如可适用本申请，也应包含在本申请的保护范围以内，并在此以引用方式包含于此。
优选地，根据所述多个历史计算任务的任务开销信息确定所述计算任务的任务负荷信息包括：根据所述计算任务的任务相关信息，从所述多个历史计算任务中筛选与所述计算任务相匹配的优选历史计算任务；根据所述优选历史计算任务的任务开销信息确定所述计算任务的任务负荷信息。
具体地，所述计算任务的任务相关信息包括各种可以描述、定位一个计算任务执行条件、执行情况等多方面的相关信息，比如所述计算任务执行过程中涉及到的各种参数，例如对服务器各个软、硬件的要求。在此，与所述计算任务所对应的历史计算任务可能与所述计算任务完全就是同一个动态计算任务，只是由于时间上的变化，相应的数据产生了规律性的变化；所述对应的历史计算任务与所述计算任务也可能是完全两个独立的动态计算任务，但是因为两者之间具有极大的相似度，适宜匹配。此外，在寻找可与所述计算任务相匹配的历史计算任务过程中，可能存在多个具有一定匹配度的可匹配的历史计算任务，此时，可以基于精确的需要，依据所侧重的参数种类，筛选出最优选的历史计算任务。
优选地，根据所述多个历史计算任务的任务开销信息确定所述计算任务的任务负荷信息包括：通过根据所述多个历史计算任务的任务相关信息对所述多个历史计算任务进行聚类处理，确定每个计算任务聚类所对应的任务开销信息；根据所述计算任务的任务相关信息，确定与所述计算任务相匹配的优选计算任务聚类；根据所述优选计算任务聚类的任务开销信息确定所述计算任务的任务负荷信息。
具体地，基于所述众多历史计算任务选择与所述计算任务相匹配的优选历史计算任务，能够很客观、准确地利用所述优选历史计算任务的任务开销信息确定所述计算任务的任务负荷信息。除此之外，还可以首先对所述多个历史计算任务，基于所确定的任务相关信息进行聚类处理，在此，所述聚类处理中以某一种或几种度量为标准的相似性，在统一聚类间最小化，而在不同聚类间最大化，通过聚类算法使得所述多个历史计算任务聚集成多种类别，一方面使得需要需找比较的信息可以大大减少到若干个历史计算任务聚类，另一方面，通过聚类处理的所述历史计算任务聚类对应的任务开销信息是一种统计分析结果，更具有普遍性和广泛适用性，根据所述优选计算任务聚类对应的聚类标准可以为所述计算任务找到相匹配的数据信息，利用所匹配的优选计算任务聚类对应的任务开销信息确定所述计算任务的任务负荷信息也就更加高效可行。
图2示出根据本申请另一个方面的一种用于在集群中调度计算任务的方法流程图。
其中，在步骤S201中，所述设备1获取集群中待调度的多个计算任务；在步骤S202中，所述设备1用于根据所述计算任务所对应的任务负荷信息将所述多个计算任务划分为多个任务子集，其中，各任务子集分别对应于所述集群中一个集群节点。
具体地，在步骤S201中，所述设备1获取集群中待调度的多个计算任务。在此，所述的集群，优选地为基于互联网用于进行云计算的多台服务器组成的集群。每一台所述服务器都是为用户提供云计算等服务的集群节点。在所述每一台服务器上都有若干个正在进行的计算任务。所述的计算任务包括在集群节点上的进程服务、线程服务等。本方案需要对集群中多个集群节点下的多个计算任务进行重新调度划分，从而优化集群资源的分配，所以首先需要确定等待调度安排的多个计算任务。在本方案中，资源池越大，确定获取的待调度的计算任务越多，调度对于系统的匹配度就会越高，进而调度精确度就会越好，集群资源分配的优化效果就越显著。
此外，所述每一个计算任务在所述集群中，例如在云资源系统中，都有与之对应的的备份计算任务，从而使得所述计算任务对应的数据信息得 以被保存，进一步，所述计算任务和与之对应的备份计算任务之间通过同步机制保障数据信息的一致性。在此，通过对计算任务进行灾备安排，可以避免由于数据信息的损毁、灭失等突发状况带来的损失。
在此，本领域技术人员应该能够理解，所述步骤S201，获取的集群中待调度的多个计算任务，也可以部署在所述设备1之外的第三方设备。即从第三方设备中收集待调度的多个计算任务，再由所述设备1进行相应的信息处理、任务调度的生成并执行等操作。
接着，在步骤S202中，所述设备1根据所述计算任务所对应的任务负荷信息将所述多个计算任务划分为多个任务子集，其中，各任务子集分别对应于所述集群中一个集群节点。在此，若要为多个不同的计算任务进行最佳划分，以便使得集群资源利用最大化，首先要获得等待调度的多个计算任务的任务负荷信息。所述的任务负荷信息包括所述计算任务对应的各种可度量指标数据信息，包括但不限于CPU使用率、内存使用率、网络使用、内存使用、网卡流量等与所述计算任务相关的属性指标，这些指标数据能够直观反应所述计算任务对于具体某一种或几种相关集群资源的使用和消耗需要，如一个进程任务的运行需要的对应的CPU使用率、或是内存使用率情况等。所述计算任务消耗的某一类集群资源越多，则带给它所对应的集群节点的压力负荷也就越大。此外，所述的任务负荷信息也可以是多个单一、具体的可度量指标数据信息复合而成的一个综合可度量指标数据，例如对于单一可度量指标数据信息CPU使用率、内存使用率和网络使用按照一定的组合计算生成一个综合指标数据，如基于实际情况为CPU使用率、内存使用率和网络使用参数分别设置一定的权重，得到需要的综合可度量指标数据。
例如，对所有的待调度的多个计算任务基于同一可度量指标在同一个时间段或是时间点的具体数值进行分析比较，基于一定的划分操作，就可以将满足最优划分条件的多个计算任务划分到同一个任务子集中。例如存在待调度的计算任务A、B、C、D分别属于集群节点1、2、3、4；通过对所述计算任务A、B、C、D各自的任务负荷信息进行分析，并基于具体的划分操作，最终选出计算任务A、B、C三个任务重新划分到一个任务 子集中，并对应于集群节点1，这一新的计算任务的组合的生成是对集群资源的利用率的优化。在此，集群资源提供的云计算的服务能力呈现的是一种动态分布、实时变化的特点，而作为本方案中所述划分操作对应的计算数据来源的任务负荷信息即可以很好的反应这种数据动态变化，如，所述任务负荷信息可以确定到某一确定可度量指标在确定时间段或时间点对应的具体数值信息。
在此，本领域技术人员应能理解，所述步骤S201和所述步骤S202可以在同一设备上实现。进一步，在实际应用中，也可以将所述步骤S201和所述步骤S202所对应的操作部署在不同的设备上来完成，此时也应包含在本申请的保护范围以内，并在此以引用方式包含于此。
在此，本申请通过基于集群中待调度的多个计算任务对应的任务负荷信息，将每个所述计算任务都划分到一个对应于某个集群节点的任务子集中，从而，为所述集群中待调度的多个计算任务找到了最佳划分组合，从而最大化利用整个集群的资源，实现更加合理的系统资源调度策略。
优选地，所述方法还包括步骤S203(未示出)，在步骤S203中，所述设备1将所述任务子集分配至对应的所述集群节点，并在所述集群节点中执行所述任务子集中的所述计算任务。
具体地，通过实时采集所述计算任务的资源使用情况，并通过计算引擎，再进行任务调度，所述互联网用户在请求某一计算任务时，所述计算任务对应的系统资源已经储备就绪，所述计算任务即可直接使用调度后的系统资源。在此，经过划分的所述多个计算任务分配到对应的集群节点之下，并通过所述集群节点分享着集群资源，此时，所述集群节点下重新划分得到的各个计算任务在执行时所需要消耗的资源，例如在相应互联网用户请求进行某一云计算任务时，所需要的使用的如网卡流量、CPU、内存等资源都要从所述云计算任务对应的集群节点所拥有的资源总量中划分出来。所以所述计算任务的执行是基于所述集群节点完成的。在此，优选地，在云计算任务中，所述集群会运用控制系统对所述集群节点、所述集群节点下的计算任务的相关数据信息进行分析采集，并将相关信息存储在所述控制系统中，以此作为信息调度策略制定的基础数据。
在此，本领域技术人员应能理解，所述步骤S201、步骤S202和步骤S203可以在同一设备中实现。进一步，在实际应用中，也可以将所述步骤S201、步骤S202和步骤S203对应的操作部署在不同的设备上来实现，此时也应包含在本申请的保护范围以内，并在此以引用方式包含于此。
在一个优选的实施例中，在步骤S202中，所述设备1根据所述计算任务所对应的任务负荷信息，并结合所述集群对应的节点资源阈值信息，将所述多个计算任务划分为多个任务子集，其中，各任务子集分别对应于所述集群中一个集群节点。
具体地，将集群中待调度的多个计算任务划分到同一任务子集中，每个任务子集与所述集群中一个集群节点相对应，划分操作需要获得所述待调度计算任务的任务负荷信息，同时，还需要设定集群对应的节点资源阈值信息，优选地，所述节点资源阈值信息包括所述集群节点的压力负荷所允许的最大值。进一步，所述节点资源阈值信息与所述任务负荷信息相对应，如当所述任务负荷信息包括所述计算任务对应的各种可度量指标数据信息，则所述节点资源阈值信息也包括所述各种可度量指标在所述集群节点分别对应的资源负荷总量。例如，所述待调度计算任务信息的任务负荷信息中包括网卡流量，则在进行计算任务划分操作时，所设定的集群对应的节点资源阈值信息就会对应包括所述网卡流量。在此，优选地，划分到同一个集群节点下的各个计算任务关于同一个可度量指标数据的任务负荷信息的数值之和要不超过所述可度量指标在所述集群节点对应的资源负荷总量。例如，存在计算任务E、F、G划分到同一个集群节点之下，若所述任务负荷信息中包括网卡流量，计算任务E、F、G各自运行消耗的网卡流量之和为a，所述集群节点对应的网卡流量这一资源负荷总量为b，则网卡流量数值a不能超过b，由此，才能保证通过划分集合到同一集群节点下的各个计算任务在对应集群节点下的操作是可行的、优化的。进一步，优选地，对于所述所述集群对应的节点资源阈值信息还可以设置一个最优范围，例如向下浮动不超过10％，即上述集群节点对应的网卡流量这一资源负荷总量为b，则可以设定当所述集群节点下的各个计算任务之和达到0.9b～b范围之间是最优的效果，低于0.9b，所述集群节点会有过多 资源未被利用，带来资源的浪费，而超过资源负荷总量为b，所述集群节点压力过大。
在此，所述节点资源阈值信息的设定可以是通过对所述集群节点进行的压力测试，并基于所述集群节点下各个计算任务的具体运行情况抽样采集分析而得的。理论上，当集群节点对应的服务器的配置一致，例如服务器软件配置、硬件配置和运行环境等条件一致，所述集群节点的资源阈值信息也是相同。在实际应用中，也可以基于集群资源分配的需要，或是具体计算任务的需要，为作为集群节点的不同服务器设定不同的配置，进而将集群下不同节点的资源阈值信息进行差别设置。在此，优选地，所有集群节点对应的服务器配置相同，从而，所述集群下各个节点对应的节点资源阈值信息一致。
本领域技术人员应能理解，上述节点资源阈值信息的设定方法，如压力测试等方法仅为举例，其他节点资源阈值信息的设定方法如可适用本申请，也应包含在本申请的保护范围以内，并在此以引用方式包含于此。
优选地，在步骤S202中，所述设备1根据所述计算任务所对应的任务负荷信息，并结合所述集群对应的节点资源阈值信息，执行划分操作直至将所述多个计算任务划分为多个任务子集，其中，各任务子集分别对应于所述集群中一个集群节点；其中，所述划分操作包括：在所述多个计算任务中取一待划分的所述计算任务作为第一计算任务；确定一个或多个候选任务子集，其中，所述候选任务子集包括所述第一计算任务及所述多个计算任务中至少一个其他待划分的所述计算任务，所述候选任务子集中各所述计算任务的任务负荷信息的累计信息满足所述节点资源阈值信息；从所述一个或多个候选任务子集中优选确定所述任务子集。
具体地，为了将预先从属于集群下不同节点的多个计算任务重新调度到多个任务子集中，可以对所述多个计算任务执行划分操作。首先，在所述待调度的多个计算任务中选取一个待划分的所述计算任务作为第一计算任务，选取的方法可以是随机的，也可以依据一定的规律性，例如，优先选取任务负荷信息对应的负荷加大的计算任务。在此，可以设定所选中的第一计算任务对应于一个集群节点1。随后，为所述的第一计算任务在 剩下的所述多个待调度的计算任务中选择与之相匹配的一个或多个计算任务。所述的相匹配所要满足的条件包括所述第一计算任务和与之相匹配的一个或多个计算任务对应的任务负荷信息的累计信息要不能超过对应节点资源阈值信息的最大值。在此，优选地，所述各个计算任务对应的任务负荷信息设定为某一个可度量指标数据在某一个确定时间点对应的数值。例如，设定所述任务负荷信息为网卡流量信息，选取时间维度上的时间点T时，并且所述第一计算任务A所对应的集群节点1对应于网卡流量信息的节点资源阈值为L，在此，可以优选节点资源阈值L的最优范围是向下浮动不超过10％。此时为所述的第一计算任务A，寻找到与其相匹配的计算任务。若在时间点T，所述第一计算任务A的网卡流量信息是L1，若此时为其匹配计算任务B，对应的网卡流量信息是L2，若此时L1与L2之和已经超过所述节点资源阈值L，则所述计算任务B与第一计算任务A并不匹配，则放弃所述计算任务B，寻找新的匹配计算任务；若此时L1与L2之和已经在所述节点资源阈值L的最优范围之内，说明所述计算任务A与所述计算任务B满足匹配条件，可以对应为一个候选任务子集。更优选地，若此时L1与L2之和满足小于节点资源阈值L的条件，但是其值在所述节点资源阈值L的最优范围之外，此时为了充分利用集群节点的资源，可以继续寻找一个或多个计算任务来与第一计算任务A和计算任务B匹配。进一步，所确定的所述一个或多个候选任务子集中各所述计算任务在各种任务负荷信息下，如各种可度量指标数据下的各个时间维度对应的任务负荷信息的累计信息都要满足各自所对应的所述节点资源阈值信息。
进一步，实际运行中，划分操作所依赖的可度量指标数据可以有多种，甚至还可以包括多个单一可度量指标数据组成的综合指标数据；同时，所获取的时间维度可能为多个，具体的时间点也可能存在多个，进而最终的划分结果也会基于不同的参数变化存在多种可能。经过划分操作，得到同时包含第一计算任务和一个或多个其他所述计算任务的候选任务子集可能存在一个或多个。接下来，可以基于一定的信息，例如脉冲比值等数据，进一步进行优选判断。
更优选地，所述从所述一个或多个候选任务子集中优选确定所述任务子集包括：确定所述候选任务子集的子集相关信息；根据所述子集相关信息从所述一个或多个候选任务子集中优选确定所述任务子集。
具体地，当通过一定的划分操作，基于所述第一计算任务确定了多个候选任务子集时，需要基于所述任务子集的子集相关信息对多个候选任务子集进行更进一步的确定。在此，所述的子集相关信息包括候选任务子集的脉冲比值。例如，对于一个所述候选任务子集M，包括第一计算任务A、计算任务B和计算任务C，设定所述任务负荷信息为网卡流量信息，选取时间维度为小时，并且所述第一计算任务A所对应的集群节点1对应于网卡流量信息的节点资源阈值为L，在此，可以优选节点资源阈值L的最优范围是向下浮动不超过10％。作为候选任务子集，所述计算任务A、B、C的网卡流量信息对应的数据值L1、L2、L3之和应该不超过节点资源阈值L。在T1时，对应的L1、L2、L3中的最大值与L1、L2、L3的平均值的比值，即为所述候选任务子集M在时间T1点下的脉冲值，当时间维度为小时，则在各个时间点T1、T2、T3……，都分别对应着一个脉冲值，所述各个时间点对应的脉冲值组成了一个集合，则所述集合中最大值与最小值的比值即为所述脉冲比值。脉冲比值越小，对应的所述候选任务子集对资源的利用效果越优。
在此，所述子集相关信息还可以包括：计算同一个候选任务子集下各个计算任务在同一个可度量指标数据的同一个确定时间点下对应的数值之和与所述任务子集对应的集群节点的节点资源阈值信息的差值。例如，对于一个所述候选任务子集N，包括第一计算任务A、计算任务D和计算任务E，设定所述任务负荷信息为网卡流量信息，选取时间维度为小时，并且所述第一计算任务A所对应的集群节点1对应于网卡流量信息的节点资源阈值为L，在此，可以优选节点资源阈值L的最优范围是向下浮动不超过10％。作为候选任务子集，所述计算任务A、D、E的网卡流量信息对应的数据值L1、L4、L5之和应该不超过节点资源阈值L,此时对应的差值是L-(L1+L2+L3)。所述差值越小，对应的所述候选任务子集对资源的利用效果越优。
在此，对于所述候选任务子集可以使用某一种子集相关信息进行进一步筛选。优选地，也可以同时利用多种子集相关信息进行综合比较，例如，对于上述候选任务子集M和候选任务子集N分别计算各自对应的所述脉冲比值和所述差值，从而得到最优选择。具体地，在实际应用中，可以优选所述脉冲比值的优先级高于所述差值，如，优选节点资源阈值L的最优范围是向下浮动不超过10％，同时对于脉冲比值，若额外规定其在节点资源阈值L的一个更广的优选范围，例如在80％～95％范围内都是可以优选地，则若候选任务子集M对应的脉冲比值在80％～95％范围内，而所述候选任务子集N达不到这个范围，则不论两组任务子集的所述差值情况如何，会优选候选任务子集M。在此，所述节点资源阈值L的最优范围10％、所述脉冲比值对应的其在节点资源阈值L的一个更广的优选范围，例如范围80％～95％，仅为举例，其都可以基于实际的业务需要灵活安排。
在此，本领域技术人员应能理解，上述子集相关信息包括候选任务子集的脉冲比值，所述的子集相关信息还可以包括：计算同一个候选任务子集下各个计算任务在同一个可度量指标数据的同一个确定时间点下对应的数值之和与所述任务子集对应的集群节点的节点资源阈值信息的差值仅为举例，其他子集相关信息如可适用本申请，也应包含在本申请的保护范围以内，并在此以引用方式包含于此。
优选地，所述设备1中所述候选任务子集中各所述计算任务的任务负荷信息的累计信息满足所述节点资源阈值信息包括：所述候选任务子集中各所述计算任务的任务负荷信息的累计信息按维度分别满足所述节点资源阈值信息。
具体地，为了所述计算任务的任务负荷信息能够全面、客观的反映计算任务的资源开销需要，所以在进行所述多个计算任务的划分操作时，依据的可度量指标数据是多维度的，例如，所述的可度量指标数据信可以同时分别来源于下列多种，如CPU使用率、内存使用率、网络使用、内存使用、网卡流量等与所述计算任务相关的属性指标，不仅是多个单一、具体的可度量指标数据信息，还可以是多个指标复合而成的若干个综合可度量指标数据，例如对于单一可度量指标数据信息CPU使用率、内存使用 率和网络使用按照一定的组合计算生成一个综合指标数据，如基于实际情况为CPU使用率、内存使用率和网络使用参数分别设置一定的权重，得到需要的综合可度量指标数据。所述任务负荷信息所对应的可度量指标数据的多样化是为了给所述计算任务的划分，提供最全面的基础数据信息，使得可以根据计算任务的目的、预期寻找到最理想的划分方法，从而使得集群资源的分配利用最合理、并更符合实际业务需要。同时，所依据的时间维度也是多维度的，记录的任务负荷信息数据可以依据以年、月、日、时、分、秒等任何需要的时间单位来记录。在此，可以依据具体的待调用的多个计算任务的划分目的，选取适当一组或多组维度下的数据信息进行利用。
在另一个优选实施例中，在步骤S202中，所述设备1根据所述集群的任务计算日志信息，确定多个历史计算任务的任务开销信息；根据所述多个历史计算任务的任务开销信息确定所述计算任务的任务负荷信息。
具体地，所述待调度的计算任务的任务负荷信息由于是动态变化的，所以实际上任何时候获得的所述待调度的计算任务的任务负荷信息都已经是一个历史数据。但是同时，所述计算任务，例如互联网云计算任务，对于同一类型的计算任务，特别是一系列参数条件相似或相同的计算任务，它的计算执行，所要消耗的集群资源的情况是相似的，所以特定的历史计算任务对于与其相匹配的后来的计算任务是具有参考性的。进一步，如果运用较合理的匹配方法，就可以为当前所要调度的计算任务找到很好的可匹配的模型历史计算任务，就可以基于所述历史计算任务的任务开销信息，如在不同的时间维度下对应不同可度量指标的压力数据，去推断出所述待调度计算任务可能需要的任务开销，也就可以据此获得划分所述多个计算任务所需要的任务负荷信息。
在此，本领域技术人员应能理解，上述据所述集群的任务计算日志信息，确定多个历史计算任务的任务开销信息；根据所述多个历史计算任务的任务开销信息确定所述计算任务的任务负荷信息仅为举例，其他确定所述计算任务的任务负荷信息如可适用本申请，也应包含在本申请的保护范围以内，并在此以引用方式包含于此。
优选地，根据所述多个历史计算任务的任务开销信息确定所述计算任务的任务负荷信息包括：根据所述计算任务的任务相关信息，从所述多个历史计算任务中筛选与所述计算任务相匹配的优选历史计算任务；根据所述优选历史计算任务的任务开销信息确定所述计算任务的任务负荷信息。
具体地，所述计算任务的任务相关信息包括各种可以描述、定位一个计算任务执行条件、执行情况等多方面的相关信息，比如所述计算任务执行过程中涉及到的各种参数，例如对服务器各个软、硬件的要求。在此，与所述计算任务所对应的历史计算任务可能与所述计算任务完全就是同一个动态计算任务，只是由于时间上的变化，相应的数据产生了规律性的变化；所述对应的历史计算任务与所述计算任务也可能是完全两个独立的动态计算任务，但是因为两者之间具有极大的相似度，适宜匹配。此外，在寻找可与所述计算任务相匹配的历史计算任务过程中，可能存在多个具有一定匹配度的可匹配的历史计算任务，此时，可以基于精确的需要，依据所侧重的参数种类，筛选出最优选的历史计算任务。
优选地，根据所述多个历史计算任务的任务开销信息确定所述计算任务的任务负荷信息包括：通过根据所述多个历史计算任务的任务相关信息对所述多个历史计算任务进行聚类处理，确定每个计算任务聚类所对应的任务开销信息；根据所述计算任务的任务相关信息，确定与所述计算任务相匹配的优选计算任务聚类；根据所述优选计算任务聚类的任务开销信息确定所述计算任务的任务负荷信息。
具体地，基于所述众多历史计算任务选择与所述计算任务相匹配的优选历史计算任务，能够很客观、准确地利用所述优选历史计算任务的任务开销信息确定所述计算任务的任务负荷信息。除此之外，还可以首先对所述多个历史计算任务，基于所确定的任务相关信息进行聚类处理，在此，所述聚类处理中以某一种或几种度量为标准的相似性，在统一聚类间最小化，而在不同聚类间最大化，通过聚类算法使得所述多个历史计算任务聚集成多种类别，一方面使得需要需找比较的信息可以大大减少到若干个历史计算任务聚类，另一方面，通过聚类处理的所述历史计算任务聚类对应的任务开销信息是一种统计分析结果，更具有普遍性和广泛适用性，根据 所述优选计算任务聚类对应的聚类标准可以为所述计算任务找到相匹配的数据信息，利用所匹配的优选计算任务聚类对应的任务开销信息确定所述计算任务的任务负荷信息也就更加高效可行。
图3至图6示出了本申请一个优选实施例，基于所述集群中两个节点M和N各自待调度的计算任务，在进行了本申请的调度方法后，重新划分了两个集群几点下的计算任务，优化了集群资源的分配。
其中，图3示出根据本申请一个优选实施例的集群节点M调度前的计算任务负荷示意图；图4示出根据本申请一个优选实施例的集群节点N调度前的计算任务负荷示意图；图5示出根据本申请一个优选实施例的集群节点M调度后的计算任务负荷示意图；图6示出根据本申请一个优选实施例的集群节点N调度后的计算任务负荷示意图。
具体地，图3示出了集群节点M下的计算任务1、2、3、4在进行划分操作之前的任务负荷，图4示出了集群节点N下的计算任务6、7、8、9在进行划分操作之前的任务负荷。在此，时间维度可以选取年、月、日、时等等任意维度，所述任务负荷信息包括所述计算任务对应的各种可度量指标数据信息，包括但不限于CPU使用率、内存使用率、网络使用、内存使用、网卡流量等与所述计算任务相关的属性指标。也可以是多个单一、具体的可度量指标数据信息复合而成的一个综合可度量指标数据。在此，优选地，图3、图4中每一条曲线代表对应计算任务在一周内的压力负荷情况。其中，图3示出，对应的四个计算任务在一周2、4、6日均处于业务低峰期，对应的压力负荷相对较小，而在一周1、3、5、7均处于业务高峰期，对应的压力负荷相对较大；图4示出，对应的四个计算任务在每周2、4、6均处于业务高峰期，对应的压力负荷相对较大，而在周1、3、5、7均处于业务低峰期，对应的压力负荷相对叫小。进一步，以所述M集群节点下的计算任务1、2、3、4和集群节点N下的计算任务6、7、8、9作为待调度计算任务，通过本方法的划分操作，得到的两个优化后的任务子集，即图5示出的集群节点M调度后，对应由计算任务2、4、6、8组成的任务子集和图6示出的集群节点N调度后，对应由计算任务1、3、5、7组成的任务子集。通过优化划分，集群节点M和N的在一定的时间 点下，如图中所述一周的多个时间点下，集群资源得到了很好的平衡互补，缓解了集群节点在一些时间点下压力过大，而在另一些时间点下压力过小，带来的资源分配不足或资源浪费的问题。在此，通过本方案诉述的划分操作，具体的划分操作优化效果体现在图5中，其中，调度后计算任务2和4在一周2、4、6日压力负荷相对较小，在一周1、3、5、7日压力负荷相对较大，而划分到M集群节点的计算任务6和8在一周2、4、6日压力负荷相对较大，在一周1、3、5、7日压力负荷相对较小；同样，图6中，计算任务5和7在一周2、4、6日压力负荷相对较大，在一周1、3、5、7日压力负荷相对较小，而划分到N集群节点的计算任务1和3在一周2、4、6日压力负荷相对较小，在一周1、3、5、7日压力负荷相对较大。相比调度前的集群节点M和N的压力负荷情况，调度后一个集群节点下的各个计算任务的压力负荷的累计值维持在所述节点阈值信息之下的，基于各个计算任务的压力负荷的高低平衡达到了资源利用的优化结果。
进一步，基于所述多个计算任务的重新调度划分得到新的任务子集，将其下的计算任务对应的时间维度下的任务负荷信息作为基础数据存储在所述集群对应的控制系统中，作为历史计算任务，为之后的目标计算任务的调度提供可参考信息数据。
对于本领域技术人员而言，显然本申请不限于上述示范性实施例的细节，而且在不背离本申请的精神或基本特征的情况下，能够以其他的具体形式实现本申请。因此，无论从哪一点来看，均应将实施例看作是示范性的，而且是非限制性的，本申请的范围由所附权利要求而不是上述说明限定，因此旨在将落在权利要求的等同要件的含义和范围内的所有变化涵括在本申请内。不应将权利要求中的任何附图标记视为限制所涉及的权利要求。此外，显然“包括”一词不排除其他单元或步骤，单数不排除复数。装置权利要求中陈述的多个单元或装置也可以由一个单元或装置通过软件或者硬件来实现。第一，第二等词语用来表示名称，而并不表示任何特定的顺序。

Claims (18)

  1. 一种用于在集群中调度计算任务的方法，包括：
    获取集群中待调度的多个计算任务；
    根据所述计算任务所对应的任务负荷信息将所述多个计算任务划分为多个任务子集，其中，各任务子集分别对应于所述集群中一个集群节点。
  2. 根据权利要求1所述的方法，其中，该方法还包括：
    将所述任务子集分配至对应的所述集群节点，并在所述集群节点中执行所述任务子集中的所述计算任务。
  3. 根据权利要求1或2所述的方法，其中，所述根据所述计算任务所对应的任务负荷信息将所述多个计算任务划分为多个任务子集，其中，各任务子集分别对应于所述集群中一个集群节点包括：
    根据所述计算任务所对应的任务负荷信息，并结合所述集群对应的节点资源阈值信息，将所述多个计算任务划分为多个任务子集，其中，各任务子集分别对应于所述集群中一个集群节点。
  4. 根据权利要求3所述的方法，其中，所述根据所述计算任务所对应的任务负荷信息将所述多个计算任务划分为多个任务子集，其中，各任务子集分别对应于所述集群中一个集群节点包括：
    根据所述计算任务所对应的任务负荷信息，并结合所述集群对应的节点资源阈值信息，执行划分操作直至将所述多个计算任务划分为多个任务子集，其中，各任务子集分别对应于所述集群中一个集群节点；
    其中，所述划分操作包括：
    在所述多个计算任务中取一待划分的所述计算任务作为第一计算任务；
    确定一个或多个候选任务子集，其中，所述候选任务子集包括所述第一计算任务及所述多个计算任务中至少一个其他待划分的所述计算任务，所述候选任务子集中各所述计算任务的任务负荷信息的累计信息满足所述节点资源阈值信息；
    从所述一个或多个候选任务子集中优选确定所述任务子集。
  5. 根据权利要求4所述的方法，其中，所述从所述一个或多个候选任务子集中优选确定所述任务子集包括：
    确定所述候选任务子集的子集相关信息；
    根据所述子集相关信息从所述一个或多个候选任务子集中优选确定所述任务子集。
  6. 根据权利要求4或5所述的方法，其中，所述候选任务子集中各所述计算任务的任务负荷信息的累计信息满足所述节点资源阈值信息包括：
    所述候选任务子集中各所述计算任务的任务负荷信息的累计信息按维度分别满足所述节点资源阈值信息。
  7. 根据权利要求1至6中任一项所述的方法，其中，所述根据所述计算任务所对应的任务负荷信息将所述多个计算任务划分为多个任务子集，其中，各任务子集分别对应于所述集群中一个集群节点还包括：
    根据所述集群的任务计算日志信息，确定多个历史计算任务的任务销信息；
    根据所述多个历史计算任务的任务开销信息确定所述计算任务的任务负荷信息。
  8. 根据权利要求7所述的方法，其中，根据所述多个历史计算任务的任务开销信息确定所述计算任务的任务负荷信息包括：
    根据所述计算任务的任务相关信息，从所述多个历史计算任务中筛选与所述计算任务相匹配的优选历史计算任务；
    根据所述优选历史计算任务的任务开销信息确定所述计算任务的任务负荷信息。
  9. 根据权利要求7所述的方法，其中，根据所述多个历史计算任务的任务开销信息确定所述计算任务的任务负荷信息包括：
    通过根据所述多个历史计算任务的任务相关信息对所述多个历史计算任务进行聚类处理，确定每个计算任务聚类所对应的任务开销信息；
    根据所述计算任务的任务相关信息，确定与所述计算任务相匹配的优 选计算任务聚类；
    根据所述优选计算任务聚类的任务开销信息确定所述计算任务的任务负荷信息。
  10. 一种用于在集群中调度计算任务的设备，包括：
    第一装置，用于获取集群中待调度的多个计算任务；
    第二装置，用于根据所述计算任务所对应的任务负荷信息将所述多个计算任务划分为多个任务子集，其中，各任务子集分别对应于所述集群中一个集群节点。
  11. 根据权利要求10所述的设备，其中，该设备还包括：
    第三装置，用于将所述任务子集分配至对应的所述集群节点，并在所述集群节点中执行所述任务子集中的所述计算任务。
  12. 根据权利要求10或11所述的设备，其中，所述第二装置用于：
    根据所述计算任务所对应的任务负荷信息，并结合所述集群对应的节点资源阈值信息，将所述多个计算任务划分为多个任务子集，其中，各任务子集分别对应于所述集群中一个集群节点。
  13. 根据权利要求12所述的设备，其中，所述第二装置用于：
    根据所述计算任务所对应的任务负荷信息，并结合所述集群对应的节点资源阈值信息，执行划分操作直至将所述多个计算任务划分为多个任务子集，其中，各任务子集分别对应于所述集群中一个集群节点；
    其中，所述划分操作包括：
    在所述多个计算任务中取一待划分的所述计算任务作为第一计算任务；
    确定一个或多个候选任务子集，其中，所述候选任务子集包括所述第一计算任务及所述多个计算任务中至少一个其他待划分的所述计算任务，所述候选任务子集中各所述计算任务的任务负荷信息的累计信息满足所述节点资源阈值信息；
    从所述一个或多个候选任务子集中优选确定所述任务子集。
  14. 根据权利要求13所述的设备，其中，所述从所述一个或多个候选 任务子集中优选确定所述任务子集包括：
    确定所述候选任务子集的子集相关信息；
    根据所述子集相关信息从所述一个或多个候选任务子集中优选确定所述任务子集。
  15. 根据权利要求13或14所述的设备，其中，所述候选任务子集中各所述计算任务的任务负荷信息的累计信息满足所述节点资源阈值信息包括：
    所述候选任务子集中各所述计算任务的任务负荷信息的累计信息按维度分别满足所述节点资源阈值信息。
  16. 根据权利要求10至15中任一项所述的设备，其中，所述第二装置还用于：
    根据所述集群的任务计算日志信息，确定多个历史计算任务的任务开销信息；
    根据所述多个历史计算任务的任务开销信息确定所述计算任务的任务负荷信息。
  17. 根据权利要求16所述的设备，其中，根据所述多个历史计算任务的任务开销信息确定所述计算任务的任务负荷信息包括：
    根据所述计算任务的任务相关信息，从所述多个历史计算任务中筛选与所述计算任务相匹配的优选历史计算任务；
    根据所述优选历史计算任务的任务开销信息确定所述计算任务的任务负荷信息。
  18. 根据权利要求16所述的设备，其中，根据所述多个历史计算任务的任务开销信息确定所述计算任务的任务负荷信息包括：
    通过根据所述多个历史计算任务的任务相关信息对所述多个历史计算任务进行聚类处理，确定每个计算任务聚类所对应的任务开销信息；
    根据所述计算任务的任务相关信息，确定与所述计算任务相匹配的优选计算任务聚类；
    根据所述优选计算任务聚类的任务开销信息确定所述计算任务的任 务负荷信息。
PCT/CN2015/094790 2014-11-24 2015-11-17 一种用于在集群中调度计算任务的方法与设备 WO2016082693A1 (zh)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US15/526,789 US20180198855A1 (en) 2014-11-24 2014-11-17 Method and apparatus for scheduling calculation tasks among clusters

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201410681900.7 2014-11-24
CN201410681900.7A CN105700948A (zh) 2014-11-24 2014-11-24 一种用于在集群中调度计算任务的方法与设备

Publications (1)

Publication Number Publication Date
WO2016082693A1 true WO2016082693A1 (zh) 2016-06-02

Family

ID=56073586

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2015/094790 WO2016082693A1 (zh) 2014-11-24 2015-11-17 一种用于在集群中调度计算任务的方法与设备

Country Status (3)

Country Link
US (1) US20180198855A1 (zh)
CN (1) CN105700948A (zh)
WO (1) WO2016082693A1 (zh)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2018024076A1 (zh) * 2016-08-04 2018-02-08 中国银联股份有限公司 一种流速控制方法及装置
CN115114012A (zh) * 2021-08-12 2022-09-27 腾讯科技（深圳）有限公司 一种任务分配方法、装置、电子设备及存储介质

Families Citing this family (27)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP6743568B2 (ja) * 2016-08-09 2020-08-19 富士通株式会社 制御装置、情報処理システム、プログラム及び情報処理方法
CN106446959B (zh) * 2016-10-10 2019-06-07 北京邮电大学 一种云计算资源动态匹配方法及装置
CN108235134B (zh) * 2016-12-21 2019-12-13 杭州海康威视数字技术股份有限公司 图片分析方法及装置
CN108255820B (zh) * 2016-12-28 2022-03-04 阿里巴巴集团控股有限公司 分布式系统中数据入库的方法、装置以及电子设备
US10802880B2 (en) * 2017-09-19 2020-10-13 Huawei Technologies Co., Ltd. System and method for distributed resource requirement and allocation
CN109788013B (zh) * 2017-11-14 2022-02-25 阿里巴巴集团控股有限公司 分布式系统中作业资源分配方法、装置及设备
CN110046034B (zh) * 2018-01-15 2021-04-23 北京国双科技有限公司 任务获取方法及装置
CN108389121B (zh) * 2018-02-07 2021-06-22 平安普惠企业管理有限公司 贷款数据处理方法、装置、计算机设备和存储介质
CN110300130B (zh) * 2018-03-21 2022-04-29 中移(苏州)软件技术有限公司 一种资源调度方法、装置、电子设备及存储介质
CN110362388B (zh) * 2018-04-11 2021-08-31 中移(苏州)软件技术有限公司 一种资源调度方法及装置
US10733029B2 (en) * 2018-07-31 2020-08-04 Hewlett Packard Enterprise Development Lp Movement of services across clusters
CN109376005B (zh) * 2018-09-03 2021-10-29 福建星瑞格软件有限公司 大数据框架处理任务的资源管理方法
CN109766181A (zh) * 2018-12-06 2019-05-17 北京航空航天大学 一种基于深度学习的rms可调度性判定方法及装置
CN109739638A (zh) * 2018-12-06 2019-05-10 北京航空航天大学 一种基于深度学习的edf可调度性判定方法及装置
CN109754189A (zh) * 2019-01-07 2019-05-14 金邦达有限公司 一种制卡任务的分配方法、一种制卡任务的获取方法、计算机装置及计算机可读存储介质
CN110855762B (zh) * 2019-10-31 2022-07-22 云南电网有限责任公司信息中心 一种电网系统中异构集群节点的数据块分配方法
US11469943B2 (en) 2019-12-06 2022-10-11 Red Hat, Inc. Pre-scheduling for cloud resource provisioning
CN112948104B (zh) * 2019-12-11 2024-01-05 中盈优创资讯科技有限公司 负载均衡的数据采集方法及装置
CN111445101A (zh) * 2020-05-15 2020-07-24 广联达科技股份有限公司 一种云计算资源调度的方法、系统及介质
CN111737190B (zh) * 2020-07-03 2022-10-21 北京智芯微电子科技有限公司 嵌入式系统的动态软硬件协同方法、嵌入式系统
CN112148474A (zh) * 2020-08-20 2020-12-29 安徽中科龙安科技股份有限公司 面向负载均衡的龙芯大数据一体机自适应任务分割方法及系统
CN112286198B (zh) * 2020-11-04 2023-07-21 安徽仓擎机器人有限公司 一种港口agv车队与人工集卡混行调度系统与方法
CN112732401A (zh) * 2020-12-29 2021-04-30 深圳前海微众银行股份有限公司 虚拟机资源分配方法、系统、设备及介质
KR20220107621A (ko) * 2021-01-25 2022-08-02 삼성전자주식회사 가속기 자원 관리 방법 및 장치
CN115269206B (zh) * 2022-09-27 2023-01-10 湖南三湘银行股份有限公司 一种基于资源分配的数据处理方法及平台
CN115309613B (zh) * 2022-10-11 2022-12-06 中诚华隆计算机技术有限公司 一种由运行监控芯片选择辅助边缘节点的方法及系统
US11929932B1 (en) * 2023-03-06 2024-03-12 Capital One Services, Llc Systems and methods for balancing communication loads across computer networks based on real-time confirmation of network resource availability

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101075199A (zh) * 2006-05-18 2007-11-21 迈普（四川）通信技术有限公司 一种多cpu调度的方法
CN101248419B (zh) * 2005-08-22 2010-05-19 新日铁系统集成株式会社 信息处理系统
CN103164261A (zh) * 2011-12-15 2013-06-19 中国移动通信集团公司 多中心数据任务处理方法、装置及系统

Family Cites Families (15)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8108656B2 (en) * 2002-08-29 2012-01-31 Qst Holdings, Llc Task definition for specifying resource requirements
US20070233866A1 (en) * 2006-03-28 2007-10-04 Karen Appleby Method and system for dynamically allocating servers to compute-resources using capacity thresholds
US8239869B2 (en) * 2006-06-19 2012-08-07 Condusiv Technologies Corporation Method, system and apparatus for scheduling computer micro-jobs to execute at non-disruptive times and modifying a minimum wait time between the utilization windows for monitoring the resources
US9588809B2 (en) * 2006-10-10 2017-03-07 Invistasking LLC Resource-based scheduler
CN100570569C (zh) * 2008-06-13 2009-12-16 南京邮电大学 网格计算环境下的作业跨域控制方法
US8695009B2 (en) * 2011-04-18 2014-04-08 Microsoft Corporation Allocating tasks to machines in computing clusters
CN102521044B (zh) * 2011-12-30 2013-12-25 北京拓明科技有限公司 一种基于消息中间件的分布式任务调度方法及系统
US9104491B2 (en) * 2012-02-21 2015-08-11 Disney Enterprises, Inc. Batch scheduler management of speculative and non-speculative tasks based on conditions of tasks and compute resources
CN102622275A (zh) * 2012-04-19 2012-08-01 吴常国 一种云计算环境下负载均衡实现方法
US20130339977A1 (en) * 2012-06-19 2013-12-19 Jack B. Dennis Managing task load in a multiprocessing environment
US9141432B2 (en) * 2012-06-20 2015-09-22 International Business Machines Corporation Dynamic pending job queue length for job distribution within a grid environment
US9135076B2 (en) * 2012-09-28 2015-09-15 Caplan Software Development S.R.L. Automated capacity aware provisioning
CN103207920A (zh) * 2013-04-28 2013-07-17 北京航空航天大学 一种元数据并行采集系统
CN103475538B (zh) * 2013-09-02 2016-04-13 南京邮电大学 一种基于多接口的自适应的云服务测试方法
US9870269B1 (en) * 2013-09-05 2018-01-16 Amazon Technologies, Inc. Job allocation in a clustered environment

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101248419B (zh) * 2005-08-22 2010-05-19 新日铁系统集成株式会社 信息处理系统
CN101075199A (zh) * 2006-05-18 2007-11-21 迈普（四川）通信技术有限公司 一种多cpu调度的方法
CN103164261A (zh) * 2011-12-15 2013-06-19 中国移动通信集团公司 多中心数据任务处理方法、装置及系统

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2018024076A1 (zh) * 2016-08-04 2018-02-08 中国银联股份有限公司 一种流速控制方法及装置
CN115114012A (zh) * 2021-08-12 2022-09-27 腾讯科技（深圳）有限公司 一种任务分配方法、装置、电子设备及存储介质
CN115114012B (zh) * 2021-08-12 2023-04-21 腾讯科技（深圳）有限公司 一种任务分配方法、装置、电子设备及存储介质

Also Published As

Publication number Publication date
CN105700948A (zh) 2016-06-22
US20180198855A1 (en) 2018-07-12

Similar Documents

Publication Publication Date Title
WO2016082693A1 (zh) 一种用于在集群中调度计算任务的方法与设备
WO2021036936A1 (zh) 在分布式系统中资源及任务的分配方法、装置及系统
US9916183B2 (en) Scheduling mapreduce jobs in a cluster of dynamically available servers
US9575810B2 (en) Load balancing using improved component capacity estimation
US9396008B2 (en) System and method for continuous optimization of computing systems with automated assignment of virtual machines and physical machines to hosts
US8265973B2 (en) Analytic-based scaling of information technology resources
US20190268283A1 (en) Resource Demand Prediction for Distributed Service Network
US20140019964A1 (en) System and method for automated assignment of virtual machines and physical machines to hosts using interval analysis
US20140019965A1 (en) System and method for automated assignment of virtual machines and physical machines to hosts with right-sizing
US9870269B1 (en) Job allocation in a clustered environment
US20150381453A1 (en) Integrated global resource allocation and load balancing
US10425313B2 (en) Tuple traffic management
US11228489B2 (en) System and methods for auto-tuning big data workloads on cloud platforms
US20120198466A1 (en) Determining an allocation of resources for a job
US10901800B2 (en) Systems for parallel processing of datasets with dynamic skew compensation
US9423957B2 (en) Adaptive system provisioning
US20130318538A1 (en) Estimating a performance characteristic of a job using a performance model
US9875169B2 (en) Modeling real capacity consumption changes using process-level data
CN109189572B (zh) 一种资源预估方法及系统、电子设备和存储介质
US11016813B2 (en) Optimizing initiator allocation
Daradkeh et al. Google traces analysis for deep machine learning cloud elastic model
Ismaeel et al. Real-time energy-conserving vm-provisioning framework for cloud-data centers
Manjaly et al. Various approches to improve MapReduce performance in Hadoop
Pulamolu et al. Intra-Tenant resource sharing in yarn based on weighted arithmetic mean
Giannakouris et al. Cost-Effective, Workload-Adaptive Migration of Big Data Applications to the Cloud

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 15864094

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 15864094

Country of ref document: EP

Kind code of ref document: A1