WO2017162075A1 - Procédé et dispositif de planification de tâche - Google Patents

Procédé et dispositif de planification de tâche Download PDF

Info

Publication number
WO2017162075A1
WO2017162075A1 PCT/CN2017/076709 CN2017076709W WO2017162075A1 WO 2017162075 A1 WO2017162075 A1 WO 2017162075A1 CN 2017076709 W CN2017076709 W CN 2017076709W WO 2017162075 A1 WO2017162075 A1 WO 2017162075A1
Authority
WO
WIPO (PCT)
Prior art keywords
task
cluster
network
default
scheduling
Prior art date
Application number
PCT/CN2017/076709
Other languages
English (en)
Chinese (zh)
Inventor
何乐
黄俨
史英杰
张�杰
张辰
Original Assignee
阿里巴巴集团控股有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 阿里巴巴集团控股有限公司 filed Critical 阿里巴巴集团控股有限公司
Publication of WO2017162075A1 publication Critical patent/WO2017162075A1/fr

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
    • G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
    • G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
    • G06F9/5038—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals considering the execution order of a plurality of tasks, e.g. taking priority or time dependency constraints into consideration
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
    • G06F9/5083—Techniques for rebalancing the load in a distributed system
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
    • Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management

Definitions

  • the present invention relates to computer technology, and in particular, to a task scheduling method and apparatus.
  • cluster technology In order to improve the stability of the system and the data processing capability and service capability of the network center, cluster technology is usually adopted.
  • clustering technology enables servers to be connected to each other to form a cluster. Multiple clusters are interconnected to form a distributed system. Each cluster in the distributed system runs a series of common applications.
  • the application is divided into multiple tasks, each task is assigned a cluster to run, the assigned cluster is used as the default cluster for the task, and the task is run on the default cluster, and the storage task is run.
  • Required task data It can be seen that in this case, when the running capacity required by the task does not match the running capability of the cluster, the load of each cluster is unbalanced.
  • the distributed system can schedule the computing tasks based on the load conditions of the clusters, and run the computing tasks from the clusters that are scheduled.
  • the bandwidth usage between the clusters is too high.
  • the invention provides a task scheduling method and device for solving the situation that the bandwidth occupation between clusters is too high in the prior art.
  • a task scheduling method is provided to determine network resources between a default cluster of a task and an idle target cluster; the default cluster is a cluster that stores task data required for the task to run;
  • the task is scheduled according to the network resource.
  • a task scheduling apparatus including:
  • a determining module configured to determine a network resource between a default cluster of the task and an idle target cluster; the default cluster is a cluster storing task data required for the task to run;
  • a scheduling module configured to schedule the task according to the network resource.
  • the task scheduling method and device provided by the embodiment of the present invention, after determining the network resource between the default cluster of the task and the idle target cluster, scheduling the task according to the determined network resource.
  • the default cluster is a cluster that stores the task data required for the task to run.
  • the bandwidth usage is too high because the task is scheduled to run to the target cluster.
  • it still needs to read the task data required for running from the default cluster. Therefore, the method of scheduling tasks to the target cluster is solved only when the network resources between the target cluster and the default cluster are better. In the prior art, the bandwidth occupation between clusters is too high.
  • FIG. 1 is a schematic flowchart of a task scheduling method according to Embodiment 1 of the present invention.
  • FIG. 2 is a schematic structural diagram of a network
  • FIG. 3 is a schematic flowchart of a task scheduling method according to Embodiment 2 of the present invention.
  • FIG. 4 is a schematic structural diagram of a task scheduling apparatus according to Embodiment 3 of the present invention.
  • FIG. 5 is a schematic structural diagram of another task scheduling apparatus according to Embodiment 3 of the present invention.
  • FIG. 1 is a schematic flowchart of a task scheduling method according to Embodiment 1 of the present invention.
  • the method provided in this embodiment may be performed by a task manager in a distributed system. As shown in FIG. 1 , the method includes:
  • Step 101 Determine network resources between a default cluster of the task and an idle target cluster.
  • the default cluster is a cluster that stores task data required for the task to run.
  • the network resource includes at least one of network bandwidth and network bandwidth time delay product.
  • a network model can be established, which is used to distinguish network structure relationships between different clusters.
  • the network structure relationship mentioned herein may include the same core switch, the same region, and different locations.
  • the same core switch means that the two clusters belong to the same core switch
  • the same area means that the two clusters belong to the same area
  • the off-site means that the two clusters belong to different areas.
  • the idle target cluster can be determined based on the load balancing.
  • the level of the network resource is determined to be the first level, such as the priority; if the default cluster and the target cluster are the same region, Determine the level of the network resource as the second level, as in the general; if the default cluster and the target cluster are different, determine the level of the network resource as the third level, such as the difference.
  • the inter-cluster distance can also be used to represent the network structure relationship between the clusters. The closer the distance is, the closer the network structure relationship is. The farther the distance is, the more distant the network structure relationship is, for example, the distance between clusters.
  • the inter-cluster is the same core switch; when the inter-cluster distance is 21, the inter-cluster is the same area; when the inter-cluster distance is 22, the inter-cluster is different.
  • FIG. 2 is a schematic structural diagram of a network.
  • cluster 1 and cluster 2 belong to one core switch, and cluster 3 and cluster 4 belong to different switches, and cluster 1 and cluster 2
  • the cluster 3 and the cluster 4 belong to the area 1
  • the cluster 5 belongs to the area 2 and is different from the cluster 1-4.
  • the network model when the established cluster 1 is the default cluster is:
  • Cluster 1 and cluster 2 are the same core switch with a network distance of 1.
  • Cluster 1 and cluster 3 are in the same area, and the network distance is 2;
  • Cluster 1 and cluster 4 are in the same area, and the network distance is 2.
  • Cluster 1 and cluster 5 are offsite with a network distance of 4.
  • the area mentioned here does not refer to the area in the administrative area, but the area in the network.
  • Step 102 Schedule the task according to the determined network resource.
  • the task is preferentially scheduled to a target cluster with the most network resources between the default cluster and the target cluster according to the network resources in at least an order.
  • the excessive bandwidth consumption occurs mainly because the task needs to be read from the default cluster to read the task data required for the operation, even though the task is scheduled to run to the target cluster.
  • using only the network resources between the target cluster and the default cluster is better, thus making The network resource between the target cluster and the default cluster can meet the requirements of the task, and then the task is dispatched to the target cluster. This solves the problem of excessive bandwidth usage between clusters in the prior art.
  • network resources can be divided into levels according to the target cluster with the most network resources between the default cluster and the target cluster.
  • the level of the network resource of the target cluster with the most network resources If the level of the network resource of the target cluster with the most current network resources is superior, the task is scheduled to the target cluster; if the level of the network resource Generally, the task is scheduled to the target cluster according to the network resource occupancy scheduled for the task; if the level of the network resource is poor, the task is scheduled to the default cluster, and the task is not scheduled to the current network resource.
  • the target cluster unless the task needs to read dependent data from the target cluster.
  • the dependency data is the running result data generated by other tasks required for the task to run.
  • the task is scheduled to be Excessive use of network resources caused by the target cluster.
  • FIG. 3 is a schematic flowchart of a task scheduling method according to Embodiment 2 of the present invention. As shown in FIG. 3, the method includes:
  • Step 201 Query whether the load of the default cluster of the task is idle. If it is idle, go to step 202. Otherwise, go to step 203.
  • the load of the default cluster of the task is idle. If the task is idle, the task is run by the default cluster. This is because no matter which cluster the task runs on, the task needs to be defaulted.
  • the cluster reads the task data required for the operation. Therefore, if the task is run on the default cluster, the bandwidth consumption caused by reading the task data can be effectively avoided, thereby avoiding the situation that the bandwidth usage is too high.
  • step 202 the task is scheduled to the default cluster, and the process ends.
  • the task is scheduled to run on the default cluster.
  • Step 203 Determine whether there is a target cluster in the cluster corresponding to the service unit to which the task belongs and the same core cluster as the core switch. If yes, go to step 204. Otherwise, go to step 202.
  • the network model of the distributed system may be established in advance, and the cluster corresponding to each service unit is recorded in the network model, so that each service unit performs tasks in the service unit by using the corresponding clusters, thereby facilitating management of the service.
  • the network distance is also used to describe the network relationship between the clusters.
  • the network distance between the clusters is recorded as the network distance of 20, and the inter-cluster is the same area as the network distance. 21, the inter-cluster is off-site for a network distance of 22.
  • cluster 1 and cluster 2 belong to the same service unit 1
  • cluster 3 and cluster 4 belong to service unit 2
  • cluster 5 belongs to service unit 3.
  • the clusters to which the service unit to which the task belongs are corresponding, and in these clusters, the cluster with the network distance of 20 from the default cluster is first queried to schedule the tasks.
  • the target cluster is selected from the distributed system in a distributed system according to the network distance from near to far, thereby ensuring that the task is preferentially scheduled to a target cluster with better network resources. on.
  • Step 204 Determine whether the target cluster of the same core switch is idle. If yes, go to step 205. Otherwise, go to step 206.
  • Step 205 Schedule the task to a target cluster of the same core switch.
  • Step 206 Determine whether there is a target cluster in the same region as the default cluster in the cluster corresponding to the service unit to which the task belongs. If yes, go to step 207. Otherwise, go to step 202.
  • Step 207 Determine whether the target cluster in the same area is idle. If yes, execute step 208; otherwise, perform step 202.
  • the task is scheduled to the default cluster that is also in the overload state. This is because, although there may be a remote target cluster, the tasks are scheduled to the off-site target cluster. When the network bandwidth is occupied, the task needs to be scheduled to the default cluster with less network resources to solve the problem of more network bandwidth usage.
  • Step 208 Determine whether the network bandwidth condition between the target cluster and the default cluster in the same area can meet the network overhead of the task. If yes, go to step 209; otherwise, go to step 202.
  • the task only accesses one task data across the cluster, and the length of time for the task to access the task data can be obtained from the historical data, wherein the length of time is equal to the difference between the end time and the start time, that is, the interval.
  • the network overhead caused by this task is: the ratio of the data volume of the task data to the length of time.
  • the bandwidth between clusters is a fixed value. If only the task is running when the task accesses the task data period, that is, between the end time and the start time, the network overhead of the task can be satisfied as long as the network overhead is less than the bandwidth.
  • Step 209 Schedule the task to the target cluster in the same area, and the process ends.
  • the task is preferentially scheduled to the target cluster with the most network bandwidth, that is, On the target cluster of the same core switch, if the same core switch is overloaded, the task is scheduled to the target cluster with the second most network bandwidth, that is, the target cluster in the same region, while performing load balancing.
  • the network bandwidth occupation of the task is minimized, and the bandwidth occupation between the clusters in the prior art is solved.
  • FIG. 4 is a schematic structural diagram of a task scheduling apparatus according to Embodiment 3 of the present invention. As shown in FIG. 3, the method includes: a determining module 31 and a scheduling module 32.
  • the determining module 31 is configured to determine network resources between the default cluster of the task and the idle target cluster.
  • the default cluster is a cluster that stores task data required for the task to run.
  • the network resource includes at least one of network bandwidth and network bandwidth time delay product.
  • the scheduling module 32 is configured to schedule tasks according to network resources between the default cluster and the target cluster.
  • the scheduling module 32 is specifically configured to schedule the task to the target cluster with the most network resources.
  • FIG. 5 is a schematic structural diagram of another task scheduling apparatus according to Embodiment 3 of the present invention.
  • the determining module 31 includes: a relationship determining unit 311. And resource determination unit 312.
  • the relationship determining unit 311 is configured to determine a network structure relationship between the default cluster and the target cluster.
  • the network structure relationship includes the same core switch, the same geographical area and different places.
  • the resource determining unit 312 is configured to determine the network resource according to the network structure relationship.
  • the resource determining unit 312 is specifically configured to: if the default cluster and the target cluster are the same core switch, determine that the level of the network resource is a first level; if the default cluster and the target cluster The level of the network resource is determined to be a second level. If the default cluster is different from the target cluster, the level of the network resource is determined to be a third level.
  • the scheduling module 32 includes: a first scheduling unit 321, a second scheduling unit 322, and a third scheduling unit 323.
  • the first scheduling unit 321 is configured to schedule the task to the target cluster if the level of the network resource between the default cluster and the target cluster is a first level.
  • the second scheduling unit 322 is configured to: if the level of the network resource between the default cluster and the target cluster is a second level, determine, according to the network resource occupancy situation scheduled by the task, scheduling the task to the location The default cluster or the target cluster.
  • the second scheduling unit 322 is specifically configured to obtain the task from the history. a length of time for reading the task data in a single time; calculating a ratio of the data amount of the task data to the length of time, obtaining network overhead of the task; if the network overhead of the task is smaller than the default cluster and Dedicating the network bandwidth between the target clusters to the target cluster; if the network overhead of the task is not less than the network bandwidth between the default cluster and the target cluster, the task is Dispatched to the default cluster.
  • the third scheduling unit 323 is configured to schedule the task to the default cluster if the level of the network resource between the default cluster and the target cluster is a third level.
  • the task scheduling device further includes:
  • the load balancing module 33 is configured to determine the target cluster based on a load balancing manner if the default cluster is in an overload state.
  • the task is scheduled according to the determined network resource.
  • the default cluster is a cluster that stores the task data required for the task to run.
  • the bandwidth usage is too high because the task is scheduled to run to the target cluster.
  • it still needs to read the task data required for running from the default cluster. Therefore, the method of scheduling tasks to the target cluster is solved only when the network resources between the target cluster and the default cluster are better. In the prior art, the bandwidth occupation between clusters is too high.
  • the aforementioned program can be stored in a computer readable storage medium.
  • the program when executed, performs the steps including the foregoing method embodiments; and the foregoing storage medium includes various media that can store program codes, such as a ROM, a RAM, a magnetic disk, or an optical disk.

Landscapes

  • Engineering & Computer Science (AREA)
  • Software Systems (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Data Exchanges In Wide-Area Networks (AREA)

Abstract

L'invention concerne un procédé et un dispositif de planification de tâche. Des ressources de réseau entre un groupe par défaut d'une tâche et un groupe cible inactif sont déterminées, et la tâche est planifiée selon les ressources de réseau déterminées. Le groupe par défaut est un groupe qui stocke des données de tâche requises pour l'exécution de la tâche. Lorsqu'une tâche est planifiée sur la base d'une situation de chargement de groupe, une mobilisation excessive de la bande passante est principalement provoquée par le fait que la tâche est planifiée et attribuée à un groupe cible pour son exécution, mais qu'elle doit encore lire d'un groupe par défaut des données de tâche requises pour s'exécuter. Par conséquent, le fait d'utiliser un moyen par lequel une tâche n'est planifiée et attribuée à un groupe cible que lorsque la situation des ressources de réseau entre le groupe cible et un groupe par défaut est relativement bonne résout le problème de la technique antérieure concernant la mobilisation excessive de la bande passante entre les groupes.
PCT/CN2017/076709 2016-03-25 2017-03-15 Procédé et dispositif de planification de tâche WO2017162075A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201610180450.2A CN107229519B (zh) 2016-03-25 2016-03-25 任务调度方法和装置
CN201610180450.2 2016-03-25

Publications (1)

Publication Number Publication Date
WO2017162075A1 true WO2017162075A1 (fr) 2017-09-28

Family

ID=59899248

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2017/076709 WO2017162075A1 (fr) 2016-03-25 2017-03-15 Procédé et dispositif de planification de tâche

Country Status (3)

Country Link
CN (1) CN107229519B (fr)
TW (1) TWI718252B (fr)
WO (1) WO2017162075A1 (fr)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112019581A (zh) * 2019-05-30 2020-12-01 华为技术有限公司 一种调度任务处理实体的方法及装置
CN113296913A (zh) * 2021-05-25 2021-08-24 未鲲(上海)科技服务有限公司 基于单集群的数据处理方法、装置、设备及存储介质

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103647820A (zh) * 2013-12-09 2014-03-19 华为数字技术（苏州）有限公司 用于分布式集群系统的仲裁方法及仲裁装置
US20140207736A1 (en) * 2013-01-18 2014-07-24 Microsoft Corporation Replication of assets across data centers
CN105391742A (zh) * 2015-12-18 2016-03-09 桂林电子科技大学 一种基于Hadoop的分布式入侵检测系统

Family Cites Families (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN100485625C (zh) * 2007-11-01 2009-05-06 北京工业大学 一种实时系统任务调度方法
EP2537106A4 (fr) * 2009-12-18 2013-10-02 Morningside Analytics Llc Système et procédé pour regroupement par attention et procédures analytiques et visualisations s'y rapportant
CN102143046B (zh) * 2010-08-25 2015-03-11 华为技术有限公司 负载均衡的方法、设备和系统
CN103605567B (zh) * 2013-10-29 2017-03-22 河海大学 面向实时性需求变化的云计算任务调度方法
US10216542B2 (en) * 2014-03-17 2019-02-26 Huawei Technologies Co., Ltd. Resource comparison based task scheduling method, apparatus, and device
US9367366B2 (en) * 2014-03-27 2016-06-14 Nec Corporation System and methods for collaborative query processing for large scale data processing with software defined networking

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20140207736A1 (en) * 2013-01-18 2014-07-24 Microsoft Corporation Replication of assets across data centers
CN103647820A (zh) * 2013-12-09 2014-03-19 华为数字技术（苏州）有限公司 用于分布式集群系统的仲裁方法及仲裁装置
CN105391742A (zh) * 2015-12-18 2016-03-09 桂林电子科技大学 一种基于Hadoop的分布式入侵检测系统

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112019581A (zh) * 2019-05-30 2020-12-01 华为技术有限公司 一种调度任务处理实体的方法及装置
WO2020238989A1 (fr) * 2019-05-30 2020-12-03 华为技术有限公司 Procédé et appareil permettant de planifier une entité de traitement de tâche
CN112019581B (zh) * 2019-05-30 2022-02-25 华为技术有限公司 一种调度任务处理实体的方法及装置
CN113296913A (zh) * 2021-05-25 2021-08-24 未鲲(上海)科技服务有限公司 基于单集群的数据处理方法、装置、设备及存储介质

Also Published As

Publication number Publication date
CN107229519B (zh) 2021-04-23
CN107229519A (zh) 2017-10-03
TW201735596A (zh) 2017-10-01
TWI718252B (zh) 2021-02-11

Similar Documents

Publication Publication Date Title
US20150295970A1 (en) Method and device for augmenting and releasing capacity of computing resources in real-time stream computing system
US20190324819A1 (en) Distributed-system task assignment method and apparatus
US8959219B2 (en) Dynamic rerouting of service requests between service endpoints for web services in a composite service
CN104391737B (zh) 云平台中负载平衡的优化方法
WO2016082693A1 (fr) Procédé et dispositif pour planifier des tâches de calcul dans un groupe
CN109960573B (zh) 一种基于智能感知的跨域计算任务调度方法及系统
US9870269B1 (en) Job allocation in a clustered environment
US20130055262A1 (en) Systems and methods of host-aware resource management involving cluster-based resource pools
CN104834569A (zh) 一种基于应用类型的集群资源调度方法及系统
CN103164279A (zh) 云计算资源分配方法和系统
TWI738721B (zh) 任務調度方法和裝置
WO2017028696A1 (fr) Procédé et dispositif pour surveiller la charge d'un système de stockage distribué
CN109861850B (zh) 一种基于sla的无状态云工作流负载均衡调度的方法
US20120089734A1 (en) Allocation of resources between web services in a composite service
US10148531B1 (en) Partitioned performance: adaptive predicted impact
KR101886072B1 (ko) 분산 인 메모리 환경에서 실시간 스트림 데이터 처리를 위한 동적 잡 스케쥴링 시스템 및 방법
US10142195B1 (en) Partitioned performance tracking core resource consumption independently
CN104407926A (zh) 一种云计算资源的调度方法
CN104462432A (zh) 自适应的分布式计算方法
KR20150030332A (ko) 데이터 분산 처리 시스템 및 이의 동작 방법
CN103746934A (zh) 一种cdn带宽平衡的方法、cdn控制中心及系统
CN102339233A (zh) 云计算集中管理平台
US20120102224A1 (en) Optimized consumption of third-party web services in a composite service
WO2017162075A1 (fr) Procédé et dispositif de planification de tâche
CN110096339B (zh) 一种基于系统负载实现的扩缩容配置推荐系统及方法

Legal Events

Date Code Title Description
NENP Non-entry into the national phase

Ref country code: DE

121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 17769352

Country of ref document: EP

Kind code of ref document: A1

122 Ep: pct application non-entry in european phase

Ref document number: 17769352

Country of ref document: EP

Kind code of ref document: A1