WO2017162075A1

WO2017162075A1 - Task scheduling method and device

Info

Publication number: WO2017162075A1
Application number: PCT/CN2017/076709
Authority: WO
Inventors: 何乐; 黄俨; 史英杰; 张�杰; 张辰
Original assignee: 阿里巴巴集团控股有限公司
Priority date: 2016-03-25
Filing date: 2017-03-15
Publication date: 2017-09-28
Also published as: CN107229519B; TW201735596A; TWI718252B; CN107229519A

Abstract

Provided are a task scheduling method and device. Network resources between a default cluster of a task and an idle target cluster are determined, and the task is scheduled according to the determined network resources. The default cluster is a cluster storing task data required for the task to run. When a task is scheduled on the basis of a cluster load situation, excessive bandwidth occupation is mainly caused by the task being scheduled to a target cluster to run but still needing to read task data required for running from a default cluster. Therefore, using a means wherein a task is scheduled to a target cluster only when the network resources situation between the target cluster and a default cluster is relatively good solves inter-cluster excessive bandwidth occupation in the prior art.

Description

Task scheduling method and device

The present application claims the priority of the Chinese Patent Application Serial No. No. No. No. No. No. No.

Technical field

The present invention relates to computer technology, and in particular, to a task scheduling method and apparatus.

Background technique

In order to improve the stability of the system and the data processing capability and service capability of the network center, cluster technology is usually adopted. The emergence of clustering technology enables servers to be connected to each other to form a cluster. Multiple clusters are interconnected to form a distributed system. Each cluster in the distributed system runs a series of common applications.

Within a distributed system, the application is divided into multiple tasks, each task is assigned a cluster to run, the assigned cluster is used as the default cluster for the task, and the task is run on the default cluster, and the storage task is run. Required task data. It can be seen that in this case, when the running capacity required by the task does not match the running capability of the cluster, the load of each cluster is unbalanced.

In order to improve the operating efficiency of each cluster and maximize the operational efficiency of the distributed system, the distributed system can schedule the computing tasks based on the load conditions of the clusters, and run the computing tasks from the clusters that are scheduled. However, in actual operation, there is often a situation in which the bandwidth usage between the clusters is too high.

Summary of the invention

The invention provides a task scheduling method and device for solving the situation that the bandwidth occupation between clusters is too high in the prior art.

In order to achieve the above object, embodiments of the present invention adopt the following technical solutions:

In a first aspect, a task scheduling method is provided to determine network resources between a default cluster of a task and an idle target cluster; the default cluster is a cluster that stores task data required for the task to run;

The task is scheduled according to the network resource.

In a second aspect, a task scheduling apparatus is provided, including:

a determining module, configured to determine a network resource between a default cluster of the task and an idle target cluster; the default cluster is a cluster storing task data required for the task to run;

And a scheduling module, configured to schedule the task according to the network resource.

The task scheduling method and device provided by the embodiment of the present invention, after determining the network resource between the default cluster of the task and the idle target cluster, scheduling the task according to the determined network resource. The default cluster is a cluster that stores the task data required for the task to run. When the task is scheduled based on the cluster load, the bandwidth usage is too high because the task is scheduled to run to the target cluster. However, it still needs to read the task data required for running from the default cluster. Therefore, the method of scheduling tasks to the target cluster is solved only when the network resources between the target cluster and the default cluster are better. In the prior art, the bandwidth occupation between clusters is too high.

The above description is only an overview of the technical solutions of the present invention, and the above-described and other objects, features and advantages of the present invention can be more clearly understood. Specific embodiments of the invention are set forth below.

DRAWINGS

Various other advantages and benefits will become apparent to those skilled in the art from a The drawings are only for the purpose of illustrating the preferred embodiments and are not to be construed as limiting. Throughout the drawings, the same reference numerals are used to refer to the same parts. In the drawing:

1 is a schematic flowchart of a task scheduling method according to Embodiment 1 of the present invention;

2 is a schematic structural diagram of a network;

3 is a schematic flowchart of a task scheduling method according to Embodiment 2 of the present invention;

4 is a schematic structural diagram of a task scheduling apparatus according to Embodiment 3 of the present invention;

FIG. 5 is a schematic structural diagram of another task scheduling apparatus according to Embodiment 3 of the present invention.

detailed description

Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While the embodiments of the present invention have been shown in the drawings, the embodiments Rather, these embodiments are provided so that this disclosure will be more fully understood and the scope of the disclosure will be fully disclosed.

The task scheduling method and apparatus provided by the embodiments of the present invention are described in detail below with reference to the accompanying drawings.

Embodiment 1

FIG. 1 is a schematic flowchart of a task scheduling method according to Embodiment 1 of the present invention. The method provided in this embodiment may be performed by a task manager in a distributed system. As shown in FIG. 1 , the method includes:

Step 101: Determine network resources between a default cluster of the task and an idle target cluster.

The default cluster is a cluster that stores task data required for the task to run. The network resource includes at least one of network bandwidth and network bandwidth time delay product.

Specifically, a network model can be established, which is used to distinguish network structure relationships between different clusters. The network structure relationship mentioned herein may include the same core switch, the same region, and different locations. Among them, the same core switch means that the two clusters belong to the same core switch, and the same area means that the two clusters belong to the same area, and the off-site means that the two clusters belong to different areas. First, the idle target cluster can be determined based on the load balancing. If the default cluster and the target cluster are the same core switch, the level of the network resource is determined to be the first level, such as the priority; if the default cluster and the target cluster are the same region, Determine the level of the network resource as the second level, as in the general; if the default cluster and the target cluster are different, determine the level of the network resource as the third level, such as the difference.

Further, in the established network model, the inter-cluster distance can also be used to represent the network structure relationship between the clusters. The closer the distance is, the closer the network structure relationship is. The farther the distance is, the more distant the network structure relationship is, for example, the distance between clusters. At 20 o'clock, the inter-cluster is the same core switch; when the inter-cluster distance is 21, the inter-cluster is the same area; when the inter-cluster distance is 22, the inter-cluster is different.

For example, FIG. 2 is a schematic structural diagram of a network. For the network structure shown in FIG. 2, cluster 1 and cluster 2 belong to one core switch, and cluster 3 and cluster 4 belong to different switches, and cluster 1 and cluster 2 The cluster 3 and the cluster 4 belong to the area 1, and the cluster 5 belongs to the area 2 and is different from the cluster 1-4.

Therefore, the network model when the established cluster 1 is the default cluster is:

Cluster 1 and cluster 2 are the same core switch with a network distance of 1.

Cluster 1 and cluster 3 are in the same area, and the network distance is 2;

Cluster 1 and cluster 4 are in the same area, and the network distance is 2.

Cluster 1 and cluster 5 are offsite with a network distance of 4.

It should be noted that the area mentioned here does not refer to the area in the administrative area, but the area in the network. The network distance can be calculated by 2n. When the core switch is n=0, the same area is n=1, and when it is different, n=2.

Step 102: Schedule the task according to the determined network resource.

Specifically, the task is preferentially scheduled to a target cluster with the most network resources between the default cluster and the target cluster according to the network resources in at least an order.

When the task is scheduled based on the cluster load condition, the excessive bandwidth consumption occurs mainly because the task needs to be read from the default cluster to read the task data required for the operation, even though the task is scheduled to run to the target cluster. , using only the network resources between the target cluster and the default cluster is better, thus making The network resource between the target cluster and the default cluster can meet the requirements of the task, and then the task is dispatched to the target cluster. This solves the problem of excessive bandwidth usage between clusters in the prior art.

As a possible implementation manner, network resources can be divided into levels according to the target cluster with the most network resources between the default cluster and the target cluster. Before scheduling the task, determine the level of the network resource of the target cluster with the most network resources. If the level of the network resource of the target cluster with the most current network resources is superior, the task is scheduled to the target cluster; if the level of the network resource Generally, the task is scheduled to the target cluster according to the network resource occupancy scheduled for the task; if the level of the network resource is poor, the task is scheduled to the default cluster, and the task is not scheduled to the current network resource. The target cluster, unless the task needs to read dependent data from the target cluster.

Among them, the dependency data is the running result data generated by other tasks required for the task to run.

In this way, when the target cluster with the most network resources cannot meet the network resources required by the task, for example, when the target cluster with the largest network resource is the cross-region between the default cluster, the task is scheduled to be Excessive use of network resources caused by the target cluster.

Embodiment 2

FIG. 3 is a schematic flowchart of a task scheduling method according to Embodiment 2 of the present invention. As shown in FIG. 3, the method includes:

Step 201: Query whether the load of the default cluster of the task is idle. If it is idle, go to step 202. Otherwise, go to step 203.

Specifically, after obtaining the task to be scheduled, first query whether the load of the default cluster of the task is idle. If the task is idle, the task is run by the default cluster. This is because no matter which cluster the task runs on, the task needs to be defaulted. The cluster reads the task data required for the operation. Therefore, if the task is run on the default cluster, the bandwidth consumption caused by reading the task data can be effectively avoided, thereby avoiding the situation that the bandwidth usage is too high.

In step 202, the task is scheduled to the default cluster, and the process ends.

Specifically, the task is scheduled to run on the default cluster.

Step 203: Determine whether there is a target cluster in the cluster corresponding to the service unit to which the task belongs and the same core cluster as the core switch. If yes, go to step 204. Otherwise, go to step 202.

Specifically, the network model of the distributed system may be established in advance, and the cluster corresponding to each service unit is recorded in the network model, so that each service unit performs tasks in the service unit by using the corresponding clusters, thereby facilitating management of the service. At the same time, in the network model, the network distance is also used to describe the network relationship between the clusters. The network distance between the clusters is recorded as the network distance of 20, and the inter-cluster is the same area as the network distance. 21, the inter-cluster is off-site for a network distance of 22. As shown in FIG. 2, cluster 1 and cluster 2 belong to the same service unit 1, cluster 3 and cluster 4 belong to service unit 2, and cluster 5 belongs to service unit 3.

Based on this pre-established network model, in this step, the clusters to which the service unit to which the task belongs are corresponding, and in these clusters, the cluster with the network distance of 20 from the default cluster is first queried to schedule the tasks.

Therefore, through the pre-established network model, the target cluster is selected from the distributed system in a distributed system according to the network distance from near to far, thereby ensuring that the task is preferentially scheduled to a target cluster with better network resources. on.

Step 204: Determine whether the target cluster of the same core switch is idle. If yes, go to step 205. Otherwise, go to step 206.

Step 205: Schedule the task to a target cluster of the same core switch.

Step 206: Determine whether there is a target cluster in the same region as the default cluster in the cluster corresponding to the service unit to which the task belongs. If yes, go to step 207. Otherwise, go to step 202.

Specifically, based on the pre-established network model, query a cluster with a network distance of 2 from the default cluster.

Step 207: Determine whether the target cluster in the same area is idle. If yes, execute step 208; otherwise, perform step 202.

If the target clusters in the same region are overloaded, the task is scheduled to the default cluster that is also in the overload state. This is because, although there may be a remote target cluster, the tasks are scheduled to the off-site target cluster. When the network bandwidth is occupied, the task needs to be scheduled to the default cluster with less network resources to solve the problem of more network bandwidth usage.

Step 208: Determine whether the network bandwidth condition between the target cluster and the default cluster in the same area can meet the network overhead of the task. If yes, go to step 209; otherwise, go to step 202.

Specifically, it is assumed that the task only accesses one task data across the cluster, and the length of time for the task to access the task data can be obtained from the historical data, wherein the length of time is equal to the difference between the end time and the start time, that is, the interval. Assuming that the read data rate of the task is constant during this time, the network overhead caused by this task is: the ratio of the data volume of the task data to the length of time. The bandwidth between clusters is a fixed value. If only the task is running when the task accesses the task data period, that is, between the end time and the start time, the network overhead of the task can be satisfied as long as the network overhead is less than the bandwidth.

Step 209: Schedule the task to the target cluster in the same area, and the process ends.

According to the network bandwidth, at least in order, the task is preferentially scheduled to the target cluster with the most network bandwidth, that is, On the target cluster of the same core switch, if the same core switch is overloaded, the task is scheduled to the target cluster with the second most network bandwidth, that is, the target cluster in the same region, while performing load balancing. The network bandwidth occupation of the task is minimized, and the bandwidth occupation between the clusters in the prior art is solved.

Embodiment 3

4 is a schematic structural diagram of a task scheduling apparatus according to Embodiment 3 of the present invention. As shown in FIG. 3, the method includes: a determining module 31 and a scheduling module 32.

The determining module 31 is configured to determine network resources between the default cluster of the task and the idle target cluster.

The scheduling module 32 is configured to schedule tasks according to network resources between the default cluster and the target cluster.

Specifically, the scheduling module 32 is specifically configured to schedule the task to the target cluster with the most network resources.

Further, FIG. 5 is a schematic structural diagram of another task scheduling apparatus according to Embodiment 3 of the present invention. As shown in FIG. 5, on the basis of the task scheduling apparatus provided in FIG. 4, the determining module 31 includes: a relationship determining unit 311. And resource determination unit 312.

The relationship determining unit 311 is configured to determine a network structure relationship between the default cluster and the target cluster.

Among them, the network structure relationship includes the same core switch, the same geographical area and different places.

The resource determining unit 312 is configured to determine the network resource according to the network structure relationship.

Specifically, the resource determining unit 312 is specifically configured to: if the default cluster and the target cluster are the same core switch, determine that the level of the network resource is a first level; if the default cluster and the target cluster The level of the network resource is determined to be a second level. If the default cluster is different from the target cluster, the level of the network resource is determined to be a third level.

Further, the scheduling module 32 includes: a first scheduling unit 321, a second scheduling unit 322, and a third scheduling unit 323.

The first scheduling unit 321 is configured to schedule the task to the target cluster if the level of the network resource between the default cluster and the target cluster is a first level.

The second scheduling unit 322 is configured to: if the level of the network resource between the default cluster and the target cluster is a second level, determine, according to the network resource occupancy situation scheduled by the task, scheduling the task to the location The default cluster or the target cluster.

If the network resource is the network bandwidth, the second scheduling unit 322 is specifically configured to obtain the task from the history. a length of time for reading the task data in a single time; calculating a ratio of the data amount of the task data to the length of time, obtaining network overhead of the task; if the network overhead of the task is smaller than the default cluster and Dedicating the network bandwidth between the target clusters to the target cluster; if the network overhead of the task is not less than the network bandwidth between the default cluster and the target cluster, the task is Dispatched to the default cluster.

The third scheduling unit 323 is configured to schedule the task to the default cluster if the level of the network resource between the default cluster and the target cluster is a third level.

Further, the task scheduling device further includes:

The load balancing module 33 is configured to determine the target cluster based on a load balancing manner if the default cluster is in an overload state.

In this embodiment, after determining the network resource between the default cluster of the task and the idle target cluster, the task is scheduled according to the determined network resource. The default cluster is a cluster that stores the task data required for the task to run. When the task is scheduled based on the cluster load, the bandwidth usage is too high because the task is scheduled to run to the target cluster. However, it still needs to read the task data required for running from the default cluster. Therefore, the method of scheduling tasks to the target cluster is solved only when the network resources between the target cluster and the default cluster are better. In the prior art, the bandwidth occupation between clusters is too high.

One of ordinary skill in the art will appreciate that all or part of the steps to implement the various method embodiments described above may be accomplished by hardware associated with the program instructions. The aforementioned program can be stored in a computer readable storage medium. The program, when executed, performs the steps including the foregoing method embodiments; and the foregoing storage medium includes various media that can store program codes, such as a ROM, a RAM, a magnetic disk, or an optical disk.

Finally, it should be noted that the above embodiments are merely illustrative of the technical solutions of the present invention, and are not intended to be limiting; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art will understand that The technical solutions described in the foregoing embodiments may be modified, or some or all of the technical features may be equivalently replaced; and the modifications or substitutions do not deviate from the technical solutions of the embodiments of the present invention. range.

Claims

A task scheduling method, comprising:

Determining network resources between the default cluster of the task and the idle target cluster; the default cluster is a cluster storing task data required for the task to run;

The task is scheduled according to the network resource.
The task scheduling method according to claim 1, wherein the scheduling the task according to the network resource comprises:

The task is scheduled to the target cluster with the most network resources.
The task scheduling method according to claim 1, wherein the determining the network resource between the default cluster of the task and the idle target cluster comprises:

Determining a network structure relationship between the default cluster and the target cluster;

Determining the network resource according to the network structure relationship.
The task scheduling method according to claim 3, wherein the network structure relationship comprises a core switch, a same area, and a different place;

Determining, according to the network structure relationship, the network resource includes:

If the default cluster is the same core switch as the target cluster, determine that the level of the network resource is a first level;

If the default cluster and the target cluster are in the same area, determine that the level of the network resource is a second level;

If the default cluster is different from the target cluster, determine that the level of the network resource is a third level.
The task scheduling method according to claim 4, wherein the scheduling the task according to the network resource comprises:

If the level of the network resource between the default cluster and the target cluster is the first level, scheduling the task to the target cluster;

If the level of the network resource between the default cluster and the target cluster is a second level, determining, according to the network resource occupancy situation scheduled by the task, scheduling the task to the default cluster or the target cluster ;

If the level of network resources between the default cluster and the target cluster is a third level, the task is scheduled to the default cluster.
The task scheduling method according to claim 5, wherein the network resource is a network bandwidth, Determining, according to the network resource occupancy situation scheduled by the task, scheduling the task to the default cluster or the target cluster, including:

Obtaining, from the history record, a length of time for the task to read the task data in a single time;

Calculating a ratio of the data amount of the task data to the length of time, and obtaining network overhead of the task;

If the network overhead of the task is less than the network bandwidth between the default cluster and the target cluster, scheduling the task to the target cluster;

If the network overhead of the task is not less than the network bandwidth between the default cluster and the target cluster, the task is scheduled to the default cluster.
The task scheduling method according to claim 1, wherein before the determining the network resource between the default cluster of the task and the idle target cluster, the method further includes:

If the default cluster is in an overload state, the target cluster is determined based on a load balancing manner.
The task scheduling method according to any one of claims 1 to 5, wherein the network resource comprises at least one of a network bandwidth and a network bandwidth time delay product.
A task scheduling device, comprising:

a determining module, configured to determine a network resource between a default cluster of the task and an idle target cluster; the default cluster is a cluster storing task data required for the task to run;

And a scheduling module, configured to schedule the task according to the network resource.
A task scheduling apparatus according to claim 9, wherein:

The scheduling module is specifically configured to schedule the task to a target cluster with the most network resources.
The task scheduling apparatus according to claim 9, wherein the determining module comprises:

a relationship determining unit, configured to determine a network structure relationship between the default cluster and the target cluster;

And a resource determining unit, configured to determine the network resource according to the network structure relationship.
The task scheduling apparatus according to claim 11, wherein the network structure relationship comprises a core switch, a same area, and a different place;

The resource determining unit is specifically configured to: if the default cluster and the target cluster are the same core switch, determine that the level of the network resource is a first level; and between the default cluster and the target cluster For the same area, the level of the network resource is determined to be a second level; if the default cluster is different from the target cluster, the level of the network resource is determined to be a third level.
The task scheduling apparatus according to claim 12, wherein the scheduling module comprises:

a first scheduling unit, configured to: if a level of network resources between the default cluster and the target cluster is first Level, scheduling the task to the target cluster;

a second scheduling unit, configured to: if the level of the network resource between the default cluster and the target cluster is a second level, determine, according to the network resource occupancy situation scheduled by the task, scheduling the task to the Default cluster or the target cluster;

The third scheduling unit is configured to schedule the task to the default cluster if the level of the network resource between the default cluster and the target cluster is a third level.
The task scheduling apparatus according to claim 13, wherein the network resource is a network bandwidth;

The second scheduling unit is specifically configured to obtain, from a history record, a length of time for the task to read the task data in a single time; calculate a ratio of a data amount of the task data to the length of time, and obtain the task. Network overhead; if the network overhead of the task is less than the network bandwidth between the default cluster and the target cluster, scheduling the task to the target cluster; if the network overhead of the task is not less than the The network bandwidth between the default cluster and the target cluster is scheduled to the default cluster.
The task scheduling device according to claim 9, wherein the device further comprises:

The load balancing module is configured to determine the target cluster based on a load balancing manner if the default cluster is in an overload state.
The task scheduling apparatus according to any one of claims 9 to 13, wherein the network resource comprises at least one of a network bandwidth and a network bandwidth time delay product.