WO2020248226A1 - 一种基于负载预测的Hadoop计算任务初始分配方法 - Google Patents

一种基于负载预测的Hadoop计算任务初始分配方法 Download PDF

Info

Publication number
WO2020248226A1
WO2020248226A1 PCT/CN2019/091267 CN2019091267W WO2020248226A1 WO 2020248226 A1 WO2020248226 A1 WO 2020248226A1 CN 2019091267 W CN2019091267 W CN 2019091267W WO 2020248226 A1 WO2020248226 A1 WO 2020248226A1
Authority
WO
WIPO (PCT)
Prior art keywords
queue
resource
load
computing
node
Prior art date
Application number
PCT/CN2019/091267
Other languages
English (en)
French (fr)
Inventor
张斌
李薇
郭军
刘晨
侯帅
周杜凯
柳波
王嘉怡
王馨悦
张娅杰
张瀚铎
刘文凤
Original Assignee
东北大学
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 东北大学 filed Critical 东北大学
Publication of WO2020248226A1 publication Critical patent/WO2020248226A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
    • G06F9/5083—Techniques for rebalancing the load in a distributed system
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
    • Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management

Definitions

  • the present invention relates to the field of distributed, big data, and cloud computing, and in particular to a method for initial allocation of Hadoop computing tasks based on load prediction.
  • the resource manager is responsible for the computing resource scheduling of the entire cluster.
  • the resource manager tracks how many active nodes and resources are available in the cluster, and coordinates the application submitted by users to allocate these computing resources.
  • the node manager is responsible for reporting the computing resources on the computing node to the resource manager, starting the computing task on the computing node, and monitoring the computing node.
  • the node manager has many dynamically created resources Container, the size of the container depends on the amount of resources it contains, memory and the number of CPU cores.
  • the application controller is responsible for requesting the required computing resources from the resource manager and tracking the running status of the user's job in real time.
  • the application controller will start the computing task first, and then apply to the resource manager to open the subtask For the required computing resources, after the application, the corresponding computing task will be opened through the node manager.
  • the application controller will detect the execution of the computing task in real time, including opening subtasks for delayed tasks, and restarting failed tasks. Turn on etc.
  • the Hadoop cluster When a user submits a job, the Hadoop cluster will first run AppMaster for the job, and AppMaser will apply for computing resources from the resource manager and start the corresponding subtasks. This is the initial allocation of computing tasks. Rationalizing the initial allocation of computing tasks can optimize the Hadoop platform, balance the cluster load, and improve cluster performance and stability.
  • the current Hadoop platform only considers two computing resources, memory and CPU cores, when allocating computing tasks. When the memory and CPU cores of the computing node meet the requirements of the computing task, the computing task can be started on the computing node. Real-time control of the distribution of computing tasks on the computing node based on the load of the computing node will lead to unbalanced cluster load, reduce the overall performance and stability of the cluster, and extend the completion time of user jobs.
  • the method for initial allocation of Hadoop computing tasks based on load prediction of the present invention specifically includes three parts: AppMaster selection algorithm based on delayed scheduling strategy, and computing resource allocation algorithm based on BP neural network.
  • the AppMaster selection algorithm based on the delayed scheduling strategy uses the idea of delayed scheduling to select the computing node with low AppMaster failure rate for the job to start AppMster to ensure the normal operation of AppMster, thereby ensuring the stable operation of the entire job.
  • the resource manager dynamically adjusts the amount of computing resources allocated by the computing node according to the load of the computing node.
  • the AppMaster corresponding to the job will wait for the allocation of computing resources.
  • the computing node reports an idle container to the resource manager, the DRF algorithm will select the appropriate one.
  • the leaf queues and user jobs are allocated for containers.
  • a method for initial allocation of Hadoop computing tasks based on load prediction which specifically includes the following steps:
  • Step 1 Use the AppMaster selection method of the delayed scheduling strategy to turn on AppMaster, including:
  • Step 1.2 Determine the relationship between ⁇ 1 and the threshold ⁇ .
  • ⁇ 1 ⁇ ⁇ start AppMaster on the current computing node; when ⁇ 1 ⁇ , use formula (1) to measure the failure rate of computing node i running AppMaster
  • Step 1.3 Determine the failure rate And the magnitude of the threshold ⁇ , if Directly in the calculation
  • Step 2 Calculate the amount of resource allocation based on the nodes of the BP neural network, including:
  • Step 2.1 The resource manager also uses the serial mechanism to receive the heartbeat information sent by the computing node, which includes: the available computing resource information of the current computing node, the status information of the running container, and the predicted load of the computing node.
  • Step 2.2 The resource manager updates the sliding window, and the sliding window moves forward to ensure that the heartbeat information in the sliding window is the latest.
  • Step 2.3 Use the data in the sliding window ⁇ H L ,H L-1 ,...,H 3 ,H 2 ,H 1 > as the input of the BP neural network, and output the latest label label of the predicted load of the computing node, divided into H , M, and L are three categories, H represents that the computing node is a high-load node in the current cluster, M represents general, and L represents low load, as follows:
  • Step 2.3.1 Enter the load information in the sliding window.
  • Step 2.3.2 Calculate the result of the interneuron by formula (2).
  • b l-1 represents the bias term of layer l-1
  • f(*) represents the activation function, which is
  • Step 2.3.3 Calculate the probability of each category by formula (3) softmax function.
  • softmax(y) i represents the predicted probability of the i-th category
  • y i represents the original output of the i-th category of the BP neural network
  • Step 2.3.4 Use the category corresponding to the maximum probability as the load label label of the computing node.
  • Step 2.4 Calculate the number of containers allocated by the node by formula (4) according to the label label of the calculation node load.
  • H, M, L represent the label of load prediction
  • Load avg represents the average load of the cluster
  • ⁇ represents the maximum number of containers that can be allocated by a heartbeat computing node
  • Load represents the predicted load of the computing node.
  • Step 3 Use the user queue and user job selection method of the DRF (Dominant Resource Fairness) algorithm to open the subtasks, which specifically include:
  • Step 3.1 Traverse all sub-queues under the root node, and select user queues based on DRF. Specifically:
  • Step 3.1.1 Determine the main resource weight domin i of queue i , which specifically includes:
  • Step 3.1.1.1 Calculate the weight of memory and CPU core resources according to formula (5) and formula (6);
  • memory used represents the memory used by the queue
  • memory cluster represents the memory of the entire cluster
  • vcores used represents the number of cores that have been used in the queue
  • vcores cluster represents the number of cores in the entire cluster. Represents the ratio of the number of configured cores in the i-th queue to the total number of cores in the cluster, Represents the ratio of the configured memory of the i-th queue to the total memory of the cluster.
  • the dominant resource of the queue is the number of cores, Is the weight of the main resource of the queue, when When the queue’s dominant resource is memory, Is the weight of the main resource of the queue,
  • Step 3.1.2 Calculate the minimum share of the main resource according to formula (7)
  • I-th queue of the primary resource allocation coefficient minimum primary resource has been used up the domian used, domian minshare main minimum share of resources.
  • Step 3.1.3 When the minimum share of the main resource of the queue is less than 1, select The largest resource queue allocates resources; when the minimum share of primary resources of all queues is greater than or equal to 1, the queue with the largest primary resource weight is selected to allocate resources.
  • Step 3.1.4 Determine whether the queue is a leaf queue. If it is a leaf queue, go to step 3.2; if it is not a leaf queue, set the node as the root node and go to step 3.1.
  • Step 3.2 For the selected user queue, based on DRF, select the content in the user queue, that is, user jobs, including:
  • Step 3.2.1 Traverse all jobs, calculate the resource weight of each job, and compare with Take the largest as the main resource weight.
  • Step 3.2.2 Compare the main resource weights of different jobs: when there is a maximum main resource weight, select the highest main resource share to allocate resources; when there are multiple equal maximum main resource shares, select the job allocation with the earlier submission time Resources.
  • Step 3.3 According to the priority relationship: Map fail >Map>Reduce, select the task with the highest priority in the job to start the container, that is, start the subtask. Among them, Map fail is the failed task in the Map, and the Map is the waiting task for the Map. , Reduce is the task waiting in Reduce.
  • the present invention is a method for initial allocation of Hadoop computing tasks based on load prediction, AppMaster selection algorithm based on delayed scheduling strategy, using the idea of delayed scheduling to select computing nodes that meet the failure rate for the job as much as possible to start AppMster, which improves the operation of AppMaster Time stability ensures the normal operation of the job.
  • the load label classification of computing nodes is carried out through BP neural network, which reduces the amount of tasks allocated by high-load label computing nodes, increases the amount of tasks allocated by low-load label computing nodes, and improves the overall cluster Stability and performance.
  • Figure 1 IPO diagram of initial allocation of Hadoop computing tasks based on load prediction in an embodiment of the present invention
  • Figure 2 BP neural network model diagram of an embodiment of the present invention
  • FIG. 3 is a process diagram of DRF user queue selection and user job selection according to an embodiment of the present invention.
  • Figure 4 AppMaster failure times of the embodiment of the present invention
  • FIG. 7 job completion time of the embodiment of the present invention.
  • Figure 8 Hadoop cluster load variance of the embodiment of the present invention.
  • Figure 9 computes the number of downtimes of a node in an embodiment of the present invention.
  • the present invention is an initial allocation method of Hadoop computing tasks based on load prediction.
  • the initial allocation of Hadoop computing tasks based on load prediction is shown in Figure 1 IPO.
  • this article uses a delayed scheduling strategy to optimize the node selection to start the AppMaster process, ensuring the stability of the AppMaster runtime.
  • the resource manager will use the BP neural network to label the computing node load Perform real-time updates, allocate different numbers of computing resources to nodes with different load labels, and finally select user queues and user jobs through the DRF strategy to start subtasks.
  • the system is tested on the Hadoop platform with 20 homogeneous machines, of which one is master and 19 is slave. Three user queues a, b, and c are configured, occupying 30%, 30%, and 40% of the cluster respectively.
  • the computing resources of the Hadoop cluster are Hadoop version 2.6, Java version 1.7, operating system Centos7, compilation tool Maven, development tool Intelij, the number of nodes is 19, and the user queue is root.a, root.b, root.c .
  • CPU core number is 8 cores
  • CPU frequency is 2.2GHz
  • memory type is DDR3-1333ECC
  • memory capacity is 8GB
  • hard disk type is 15000 rpm SAS hard disk
  • hard disk capacity is 300GB
  • bandwidth is 1000Mbps.
  • a method for initial allocation of Hadoop computing tasks based on load prediction specifically includes the following steps:
  • Step 1 Use the AppMaster selection method of the delayed scheduling strategy to turn on AppMaster, including:
  • Step 1.2 Determine the relationship between ⁇ 1 and the threshold ⁇ .
  • ⁇ 1 ⁇ ⁇ start AppMaster on the current computing node; when ⁇ 1 ⁇ , use formula (1) to measure the failure rate of computing node i running AppMaster
  • Step 1.3 Determine the failure rate And the magnitude of the threshold ⁇ , if Directly in the calculation
  • Step 2 Calculate the amount of resource allocation based on the nodes of the BP neural network, as shown in Figure 2, which specifically includes:
  • Step 2.1 The resource manager also uses the serial mechanism to receive the heartbeat information sent by the computing node, which includes: the available computing resource information of the current computing node, the status information of the running container, and the predicted load of the computing node.
  • Step 2.2 The resource manager updates the sliding window, and the sliding window moves forward to ensure that the heartbeat information in the sliding window is the latest.
  • Step 2.3 Use the data in the sliding window ⁇ H L ,H L-1 ,...,H 3 ,H 2 ,H 1 > as the input of the BP neural network, and output the latest label label of the predicted load of the computing node, divided into H , M, and L are three categories, H represents that the computing node is a high-load node in the current cluster, M represents general, and L represents low load, as follows:
  • Step 2.3.1 Enter the load information in the sliding window.
  • Step 2.3.2 Calculate the result of the interneuron by formula (2).
  • b l-1 represents the bias term of layer l-1
  • f(*) represents the activation function, which is
  • Step 2.3.3 Calculate the probability of each category by formula (3) softmax function.
  • softmax(y) i represents the predicted probability of the i-th category
  • y i represents the original output of the i-th category of the BP neural network
  • Step 2.3.4 Use the category corresponding to the maximum probability as the load label label of the computing node.
  • Step 2.4 Calculate the number of containers allocated by the node by formula (4) according to the label label of the calculation node load.
  • H, M, L represent the label of load prediction
  • Load avg represents the average load of the cluster
  • ⁇ represents the maximum number of containers that can be allocated by a heartbeat computing node
  • Load represents the predicted load of the computing node.
  • Step 3 Use the user queue and user job selection method of the DRF (Dominant Resource Fairness) algorithm to start subtasks, as shown in Figure 3, which specifically include:
  • Step 3.1 Traverse all sub-queues under the root node, and select user queues based on DRF. Specifically:
  • Step 3.1.1 Determine the main resource weight domin i of queue i , which specifically includes:
  • Step 3.1.1.1 Calculate the weight of memory and CPU core resources according to formula (5) and formula (6);
  • memory used represents the memory used by the queue
  • memory cluster represents the memory of the entire cluster
  • vcores used represents the number of cores that have been used in the queue
  • vcores cluster represents the number of cores in the entire cluster. Represents the ratio of the number of configured cores in the i-th queue to the total number of cores in the cluster, Represents the ratio of the configured memory of the i-th queue to the total memory of the cluster.
  • the dominant resource of the queue is the number of cores, Is the weight of the main resource of the queue, when When the queue’s dominant resource is memory, Is the weight of the main resource of the queue,
  • Step 3.1.2 Calculate the minimum share of the main resource according to formula (7)
  • I-th queue of the primary resource allocation coefficient minimum primary resource has been used up the domian used, domian minshare main minimum share of resources.
  • Step 3.1.3 When the minimum share of the main resource of the queue is less than 1, select The largest resource queue allocates resources; when the minimum share of primary resources of all queues is greater than or equal to 1, the queue with the largest primary resource weight is selected to allocate resources.
  • Step 3.1.4 Determine whether the queue is a leaf queue. If it is a leaf queue, go to step 3.2; if it is not a leaf queue, set the node as the root node and go to step 3.1.
  • Step 3.2 For the selected user queue, based on DRF, select the content in the user queue, that is, user jobs, including:
  • Step 3.2.1 Traverse all jobs, calculate the resource weight of each job, and compare with Take the largest as the main resource weight.
  • Step 3.2.2 Compare the main resource weights of different jobs: when there is a maximum main resource weight, select the highest main resource share to allocate resources; when there are multiple equal maximum main resource shares, select the job allocation with the earlier submission time Resources.
  • Step 3.3 According to the priority relationship: Map fail >Map>Reduce, select the task with the highest priority in the job to start the container, that is, start the subtask. Among them, Map fail is the failed task in the Map, and the Map is the waiting task for the Map. , Reduce is the task waiting in Reduce.
  • the BP neural network has a higher accuracy index than other mainstream machine learning algorithms in the classification of the load label of the computing node, and can effectively and quickly judge the load label of the computing node without extending the scheduling time. Under the premise of ensuring the accuracy of the identification of the load label of the computing node.
  • the initial allocation method of Hadoop computing tasks based on load prediction proposed in this chapter is more balanced than the original initial allocation method of Hadoop, the cluster load is more balanced, the cluster stability and performance are higher, and the completion time of user jobs is less.

Landscapes

  • Engineering & Computer Science (AREA)
  • Software Systems (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Management, Administration, Business Operations System, And Electronic Commerce (AREA)
  • Multi Processors (AREA)
  • Debugging And Monitoring (AREA)

Abstract

本发明提出一种基于负载预测的Hadoop计算任务初始分配方法，包括：使用延迟调度策略的AppMaster选择方法，开启AppMaster；基于BP神经网络的节点，计算资源分配数量；使用DRF算法的用户队列和用户作业选择方法，开启子任务；本发明基于延迟调度策略的AppMaster选择算法，提高了AppMaster运行时的稳定性，保证了作业的正常运行。基于BP神经网络的节点计算资源分配算法，减少高负载标签计算节点分配的任务量，增加低负载标签计算节点分配的任务量，提高了集群整体的稳定性和性能。基于DRF的用户队列和用户作业选择算法，当作业所属的队列资源不够时，根据占主导地位的计算资源使用情况来选择叶子队列和用户作业，最终达到合理化计算任务初始分配，均衡集群负载，提高集群性能的目标。

Description

一种基于负载预测的Hadoop计算任务初始分配方法 技术领域
本发明涉本发明涉及分布式、大数据、云计算领域，具体涉及一种基于负载预测的Hadoop计算任务初始分配方法。
背景技术
目前，Hadoop平台中计算任务的初始分配由Yarn来完成，在Yarn框架下，有三个重要的角色，分别为资源管理器(ResourceManager)，节点管理器(NodeManager)和应用程序控制器(AppMaster)。
资源管理器负责整个集群的计算资源调度，资源管理器会追踪集群中有多少可用的活动节点和资源，协调用户提交的应用程序分配这些计算资源。
节点管理器作为资源管理器在各个计算节点上的代理，负责计算节点上计算资源向资源管理器的汇报，计算节点上计算任务的开启，计算节点的监控，节点管理器拥有许多动态创建的资源容器，容器的大小取决于它所包含的资源量内存和CPU核数。
应用程序控制器负责向资源管理器申请需要的计算资源和实时追踪用户作业的运行情况，当用户提交作业后，应用程序控制器会先于计算任务开启，随后会向资源管理器申请开启子任务需要的计算资源，申请后会通过节点管理器开启对应的计算任务，计算任务开启后，应用程序控制器开启后会实时检测计算任务的执行情况，包括对迟滞任务开启子任务，对失败任务重新开启等。
当用户提交作业时，Hadoop集群会为该作业先运行AppMaster，AppMaser会向资源管理器申请计算资源并且开启对应的子任务，这便是计算任务的初始分配。合理化计算任务的初始分配可以优化Hadoop平台，均衡集群负载，提高集群性能和稳定性。当前的Hadoop平台在分配计算任务时只考虑了内存和CPU核数两种计算资源，当计算节点的内存与CPU核数满足计算任务的需求时即可在该计算节点上开启计算任务，并没有实时根据计算节点的负载来控制计算节点上计算任务的分配量，这会导致集群的负载不均衡，集群整体性能和稳定性降低，延长了用户作业的完成时间。
发明内容
基于以上技术问题，本发明所述的一种基于负载预测的Hadoop计算任务初始分配方法，具体包括三部分：基于延迟调度策略的AppMaster选择算法部分，基于BP神经网络的计算资源分配算法部分，基于DRF的用户队列和用户作业选择算法部分。基于延迟调度策略的AppMaster选择算法，借助延迟调度的思想为作业选择AppMaster失效率低的计算节点进行 AppMster的开启，保证AppMster运行正常，从而保证整个作业的稳定运行。基于BP神经网络的计算资源分配算法，资源管理器根据计算节点的负载动态调整计算节点分配的计算资源量。基于DRF的用户队列和用户作业选择算法，当作业所属的队列资源不够时，作业对应的AppMaster会等待计算资源的分配，当计算节点向资源管理器汇报空闲的container时，会DRF算法选择合适的叶子队列和用户作业进行container的分配。
一种基于负载预测的Hadoop计算任务初始分配方法，具体包括如下步骤：
步骤1：使用延迟调度策略的AppMaster选择方法，开启AppMaster，具体包括：
步骤1.1：资源管理器接收用户作业提交，并初始化节点数量α 1＝0；
步骤1.2：判断α 1与阈值α大小关系，当α 1≥α时，在当前计算节点上开启AppMaster；当α 1<α时，通过公式(1)衡量计算节点i运行AppMaster的失效率
Figure PCTCN2019091267-appb-000001
Figure PCTCN2019091267-appb-000002
其中，
Figure PCTCN2019091267-appb-000003
代表第i个计算节点在历史日志中运行的AppMaster总数，
Figure PCTCN2019091267-appb-000004
代表第i个计算节点历史日志中失败AppMaster数量。
步骤1.3：判断失效率
Figure PCTCN2019091267-appb-000005
与阈值σ的大小关系，若
Figure PCTCN2019091267-appb-000006
则直接在该计
算节点上开启AppMaster；若
Figure PCTCN2019091267-appb-000007
则跳过本次节点，α 1加1，返回步骤1.1。
步骤2：基于BP神经网络的节点，计算资源分配数量，具体包括：
步骤2.1：资源管理器同时利用串行机制接收计算节点发送的心跳信息，其中包含：当前计算节点的可用计算资源信息、运行中container的状态信息、计算节点的预测负载。
步骤2.2：资源管理器更新滑动窗口，滑动窗口向前移动，保证滑动窗口内的心跳信息是最新传过来的；
步骤2.3：将滑动窗口内的数据<H L,H L-1,…,H 3,H 2,H 1>作为BP神经网络的输入，输出最新的计算节点预测负载的标签label，分为H,M,L三个类别，H代表该计算节点在当前集群下为高负载节点，M代表一般，L代表负载较低，具体如下：
步骤2.3.1：输入滑动窗口内的负载信息。
步骤2.3.2：通过公式(2)计算中间神经元的结果。
Figure PCTCN2019091267-appb-000008
其中，
Figure PCTCN2019091267-appb-000009
代表第l层第i个神经元的输出，b l-1代表l-1层的偏置项，
Figure PCTCN2019091267-appb-000010
代表第l层第i个神经元与第l-1层第j个神经元连接的权值，f(*)代表激活函数，为
Figure PCTCN2019091267-appb-000011
步骤2.3.3：通过公式(3)softmax函数计算每个类别概率。
Figure PCTCN2019091267-appb-000012
其中，softmax(y) i代表第i个类别的预测概率，y i代表BP神经网络第i个类别的原始输出；
步骤2.3.4：将概率的最大值对应的类别作为计算节点负载标签label。
步骤2.4：根据计算节点负载的标签label，通过公式(4)计算节点分配的container数量。
Figure PCTCN2019091267-appb-000013
其中，H,M,L代表负载预测的标签，Load avg代表集群的平均负载，τ代表一次心跳计算节点最多能分配的container数量，Load代表计算节点预测负载。
步骤3：使用DRF(Dominant Resource Fairness主导资源公平)算法的用户队列和用户作业选择方法，开启子任务，具体包括：
步骤3.1：遍历根节点下的所有子队列，基于DRF选择用户队列。具体包括：
步骤3.1.1：确定队列i主资源权值domin i，具体包括：
步骤3.1.1.1：根据公式(5)和公式(6)计算内存和CPU核数资源的权值；
Figure PCTCN2019091267-appb-000014
Figure PCTCN2019091267-appb-000015
其中，memory used代表了该队列已经使用掉的内存，memory cluster代表整个集群的内存，vcores used代表了该队列已经使用掉的核数，vcores cluster代表整个集群的核数，
Figure PCTCN2019091267-appb-000016
代表第i个队列配置核数占集群总核数的比值，
Figure PCTCN2019091267-appb-000017
代表第i个队列配置内存占集群总内存的比值。
3.1.1.2，判断内存资源的权值
Figure PCTCN2019091267-appb-000018
和CPU核数资源的权值
Figure PCTCN2019091267-appb-000019
大小，当
Figure PCTCN2019091267-appb-000020
时，则队列的主导资源为核数，
Figure PCTCN2019091267-appb-000021
为该队列主资源的权值，
Figure PCTCN2019091267-appb-000022
当
Figure PCTCN2019091267-appb-000023
时，则队列的主导资源为内存，
Figure PCTCN2019091267-appb-000024
为该队列主资源的权值，
Figure PCTCN2019091267-appb-000025
步骤3.1.2：根据公式(7)计算主资源最小份额
Figure PCTCN2019091267-appb-000026
Figure PCTCN2019091267-appb-000027
其中，
Figure PCTCN2019091267-appb-000028
第i个队列主资源最小分配额系数，domian used已经用掉的主资源，domian minshare最小的主资源份额。
步骤3.1.3：当存在队列主资源最小份额小于1时，选择
Figure PCTCN2019091267-appb-000029
最大的资源队列分配资源；当所有队列的主资源最小份额大于等于1时，选择主资源权值最大的队列分配资源。
步骤3.1.4：判断该队列是否为叶子队列，若为叶子队列，则调到步骤3.2；若不为叶子队列，则将该节点设置为根节点，转到步骤3.1。
步骤3.2：对于已经选择的用户队列，基于DRF对用户队列中的内容，即用户作业进行选择，具体包括：
步骤3.2.1：遍历所有作业，计算每个作业的资源权值，比较
Figure PCTCN2019091267-appb-000030
和
Figure PCTCN2019091267-appb-000031
将最大的作为主资源权值。
步骤3.2.2：比较不同作业的主资源权值：当存在一个最大主资源权值，选择主资源份额高作业分配资源；当存在多个相等最大主资源份额，选择提交作业时间早的作业分配资源。
步骤3.3：根据优先级关系：Map fail>Map>Reduce，选择作业中的优先级最高的任务进行container开启，即子任务开启，其中，Map fail为Map中失败的任务，Map为Map等待的任务，Reduce为Reduce中等待的任务。
有益技术效果：
本发明一种基于负载预测的Hadoop计算任务初始分配方法，基于延迟调度策略的AppMaster选择算法，利用延迟调度的思想尽可能地为作业选择满足失效率的计算节点进行AppMster的开启，提高了AppMaster运行时的稳定性，保证了作业的正常运行。基于BP神经网络的节点计算资源分配算法，通过BP神经网络对计算节点进行负载标签分类，减少高负载标签计算节点分配的任务量，增加低负载标签计算节点分配的任务量，提高了集群整体的稳定性和性能。基于DRF的用户队列和用户作业选择算法，当作业所属的队列资源不够时，根据占主导地位的计算资源使用情况来选择叶子队列和用户作业，最终达到合理化计算任务初始分配，均衡集群负载，提高集群性能的目标。
附图说明
图1本发明实施例的基于负载预测的Hadoop计算任务初始分配IPO图；
图2本发明实施例的BP神经网络模型图；
图3本发明实施例的DRF用户队列选择与用户作业选择过程图；
图4本发明实施例的AppMaster失效次数；
图5本发明实施例的任务失效次数；
图6本发明实施例的负载标签分类准确率；
图7本发明实施例的作业完成时间；
图8本发明实施例的Hadoop集群负载方差；
图9本发明实施例的计算节点宕机次数。
具体实施方式
下面结合附图和具体实施实例对发明做进一步说明，本发明为一种基于负载预测的Hadoop计算任务初始分配方法，基于负载预测的Hadoop计算任务初始分配IPO图1所示。当用户提交作业时本文采用延迟调度策略优化了开启AppMaster进程的节点选择，保障了AppMaster运行时的稳定性，其次，当AppMaster申请计算资源时，资源管理器会利用BP神经网络对计算节点负载标签进行实时更新，对不同负载标签的节点进行不同数量的计算资源分配，最后通过DRF策略选择用户队列和用户作业进行子任务的开启。
本系统在Hadoop平台上用20台同构的机器进行试验，其中1台为master，19台为slave，配置了三个用户队列a,b,c，分别占用集群30％，30％，40％的计算资源，Hadoop集群搭建的信息为Hadoop版本2.6，Java版本1.7，操作系统Centos7,编译工具Maven,开发工具Intelij,节点个数为19，用户队列为root.a,root.b,root.c。
本系统实施节点配置参数：CPU核数为8核，CPU主频2.2GHz,内存类型DDR3-1333ECC,内存容量8GB,硬盘类型为15000转SAS硬盘，硬盘容量300GB,带宽1000Mbps。
一种基于负载预测的Hadoop计算任务初始分配方法，如图1所示，具体包括如下步骤：
步骤1：使用延迟调度策略的AppMaster选择方法，开启AppMaster，具体包括：
步骤1.1：资源管理器接收用户作业提交，并初始化节点数量α 1＝0；
步骤1.2：判断α 1与阈值α大小关系，当α 1≥α时，在当前计算节点上开启AppMaster；当α 1<α时，通过公式(1)衡量计算节点i运行AppMaster的失效率
Figure PCTCN2019091267-appb-000032
Figure PCTCN2019091267-appb-000033
其中，
Figure PCTCN2019091267-appb-000034
代表第i个计算节点在历史日志中运行的AppMaster总数，
Figure PCTCN2019091267-appb-000035
代表第i个计算节点历史日志中失败AppMaster数量。
步骤1.3：判断失效率
Figure PCTCN2019091267-appb-000036
与阈值σ的大小关系，若
Figure PCTCN2019091267-appb-000037
则直接在该计
算节点上开启AppMaster；若
Figure PCTCN2019091267-appb-000038
则跳过本次节点，α 1加1，返回步骤1.1。
步骤2：基于BP神经网络的节点，计算资源分配数量，如图2所示，具体包括：
步骤2.1：资源管理器同时利用串行机制接收计算节点发送的心跳信息，其中包含：当前计算节点的可用计算资源信息、运行中container的状态信息、计算节点的预测负载。
步骤2.2：资源管理器更新滑动窗口，滑动窗口向前移动，保证滑动窗口内的心跳信息是最新传过来的；
步骤2.3：将滑动窗口内的数据<H L,H L-1,…,H 3,H 2,H 1>作为BP神经网络的输入，输出最新的计算节点预测负载的标签label，分为H,M,L三个类别，H代表该计算节点在当前集群下为高负载节点，M代表一般，L代表负载较低，具体如下：
步骤2.3.1：输入滑动窗口内的负载信息。
步骤2.3.2：通过公式(2)计算中间神经元的结果。
Figure PCTCN2019091267-appb-000039
其中，
Figure PCTCN2019091267-appb-000040
代表第l层第i个神经元的输出，b l-1代表l-1层的偏置项，
Figure PCTCN2019091267-appb-000041
代表第l层第i个神经元与第l-1层第j个神经元连接的权值，f(*)代表激活函数，为
Figure PCTCN2019091267-appb-000042
步骤2.3.3：通过公式(3)softmax函数计算每个类别概率。
Figure PCTCN2019091267-appb-000043
其中，softmax(y) i代表第i个类别的预测概率，y i代表BP神经网络第i个类别的原始输出；
步骤2.3.4：将概率的最大值对应的类别作为计算节点负载标签label。
步骤2.4：根据计算节点负载的标签label，通过公式(4)计算节点分配的container数量。
Figure PCTCN2019091267-appb-000044
其中，H,M,L代表负载预测的标签，Load avg代表集群的平均负载，τ代表一次心跳计算节点最多能分配的container数量，Load代表计算节点预测负载。
步骤3：使用DRF(Dominant Resource Fairness主导资源公平)算法的用户队列和用户作业选择方法，开启子任务，如图3所示，具体包括：
步骤3.1：遍历根节点下的所有子队列，基于DRF选择用户队列。具体包括：
步骤3.1.1：确定队列i主资源权值domin i，具体包括：
步骤3.1.1.1：根据公式(5)和公式(6)计算内存和CPU核数资源的权值；
Figure PCTCN2019091267-appb-000045
Figure PCTCN2019091267-appb-000046
其中，memory used代表了该队列已经使用掉的内存，memory cluster代表整个集群的内存，vcores used代表了该队列已经使用掉的核数，vcores cluster代表整个集群的核数，
Figure PCTCN2019091267-appb-000047
代表第i个队列配置核数占集群总核数的比值，
Figure PCTCN2019091267-appb-000048
代表第i个队列配置内存占集群总内存的比值。
3.1.1.2，判断内存资源的权值
Figure PCTCN2019091267-appb-000049
和CPU核数资源的权值
Figure PCTCN2019091267-appb-000050
大小，当
Figure PCTCN2019091267-appb-000051
时，则队列的主导资源为核数，
Figure PCTCN2019091267-appb-000052
为该队列主资源的权值，
Figure PCTCN2019091267-appb-000053
当
Figure PCTCN2019091267-appb-000054
时，则队列的主导资源为内存，
Figure PCTCN2019091267-appb-000055
为该队列主资源的权值，
Figure PCTCN2019091267-appb-000056
步骤3.1.2：根据公式(7)计算主资源最小份额
Figure PCTCN2019091267-appb-000057
Figure PCTCN2019091267-appb-000058
其中，
Figure PCTCN2019091267-appb-000059
第i个队列主资源最小分配额系数，domian used已经用掉的主资源，domian minshare最小的主资源份额。
步骤3.1.3：当存在队列主资源最小份额小于1时，选择
Figure PCTCN2019091267-appb-000060
最大的资源队列分配资源；当所有队列的主资源最小份额大于等于1时，选择主资源权值最大的队列分配资源。
步骤3.1.4：判断该队列是否为叶子队列，若为叶子队列，则调到步骤3.2；若不为叶子队列，则将该节点设置为根节点，转到步骤3.1。
步骤3.2：对于已经选择的用户队列，基于DRF对用户队列中的内容，即用户作业进行选择，具体包括：
步骤3.2.1：遍历所有作业，计算每个作业的资源权值，比较
Figure PCTCN2019091267-appb-000061
和
Figure PCTCN2019091267-appb-000062
将最大的作为主资源权值。
步骤3.2.2：比较不同作业的主资源权值：当存在一个最大主资源权值，选择主资源份额高作业分配资源；当存在多个相等最大主资源份额，选择提交作业时间早的作业分配资源。
步骤3.3：根据优先级关系：Map fail>Map>Reduce，选择作业中的优先级最高的任务进行container开启，即子任务开启，其中，Map fail为Map中失败的任务，Map为Map等待的任务，Reduce为Reduce中等待的任务。
将本发明所述方法和其他方法进行对比，结果如图4-9所示，得出以下结论：
(1)由图6可知，BP神经网络在计算节点负载标签的分类的准确率指标上高于其他的 主流机器学习算法，能够有效快速的对计算节点进行负载标签的判断，在没有延长调度时间的前提下保证了计算节点负载标签的识别准确率。
(2)AppMaster的失效率对任务完成时间影响较大。由图4和图7可知，本文提出的调度策略相较于Hadoop原生调度策略AppMaster失效的次数更少，从而减少用户作业的完成时间。
(3)根据图5和图7可知，作业量的规模变大时，负载对性能的影响开始显现，本文提出的计算任务初始分配方法考虑了负载的因素，在Fair-modified策略下的作业完成时间要明显短于其他调度器的完成时间。
(4)通过图5、8和9可知，本章提出的基于负载预测的Hadoop计算任务初始分配方法可以有效的提高集群的稳定性，任务的失效数相比于Hadoop原有的调度器有了明显的下降，从负载方差图来看，负载方差平均减少了50％多，并且由于负载过高而导致的计算节点宕机的数量也少于Hadoop中另外三款调度器。
综上所述，本章提出的基于负载预测的Hadoop计算任务初始分配方法相较于Hadoop原生的初始分配方法集群负载更加均衡，集群稳定性和性能更高，用户作业的完成时间更少。

Claims (3)

  1. 一种基于负载预测的Hadoop计算任务初始分配方法，其特征在于，具体步骤如下：
    步骤1：使用延迟调度策略的AppMaster选择方法，开启AppMaster，具体包括：
    步骤1.1：资源管理器接收用户作业提交，并初始化节点数量α 1＝0；
    步骤1.2：判断α 1与阈值α大小关系，当α 1≥α时，在当前计算节点上开启AppMaster；当α 1＜α时，通过公式(1)衡量计算节点i运行AppMaster的失效率
    Figure PCTCN2019091267-appb-100001
    Figure PCTCN2019091267-appb-100002
    其中，
    Figure PCTCN2019091267-appb-100003
    代表第i个计算节点在历史日志中运行的AppMaster总数，
    Figure PCTCN2019091267-appb-100004
    代表第i个计算节点历史日志中失败AppMaster数量；
    步骤1.3：判断失效率
    Figure PCTCN2019091267-appb-100005
    与阈值σ的大小关系，若
    Figure PCTCN2019091267-appb-100006
    则直接在该计算节点上开启AppMaster；若
    Figure PCTCN2019091267-appb-100007
    则跳过本次节点，α 1加1，返回步骤1.1；
    步骤2：基于BP神经网络的节点，计算资源分配数量；
    步骤3：使用DRF算法的用户队列和用户作业选择方法，开启子任务。
  2. 根据权利要求1所述基于负载预测的Hadoop计算任务初始分配方法，其特征在于，所述步骤2，具体包括：
    步骤2.1：资源管理器同时利用串行机制接收计算节点发送的心跳信息，其中包含：当前计算节点的可用计算资源信息、运行中container的状态信息、计算节点的预测负载；
    步骤2.2：资源管理器更新滑动窗口，滑动窗口向前移动，保证滑动窗口内的心跳信息是最新传过来的；
    步骤2.3：将滑动窗口内的数据<H L,H L-1,…,H 3,H 2,H 1>作为BP神经网络的输入，输出最新的计算节点预测负载的标签label，分为H,M,L三个类别，H代表该计算节点在当前集群下为高负载节点，M代表一般，L代表负载较低，具体如下：
    步骤2.3.1：输入滑动窗口内的负载信息；
    步骤2.3.2：通过公式(2)计算中间神经元的结果：
    Figure PCTCN2019091267-appb-100008
    其中，
    Figure PCTCN2019091267-appb-100009
    代表第l层第i个神经元的输出，b l-1代表l-1层的偏置项，
    Figure PCTCN2019091267-appb-100010
    代表第l层第i个神经元与第l-1层第j个神经元连接的权值，f(*)代表激活函数，为
    Figure PCTCN2019091267-appb-100011
    步骤2.3.3：通过公式(3)softmax函数计算每个类别概率：
    Figure PCTCN2019091267-appb-100012
    其中，softmax(y) i代表第i个类别的预测概率，y i代表BP神经网络第i个类别的原始输出；
    步骤2.3.4：将概率的最大值对应的类别作为计算节点负载标签label；
    步骤2.4：根据计算节点负载的标签label，通过公式(4)计算节点分配的container数量；
    Figure PCTCN2019091267-appb-100013
    其中，H,M,L代表负载预测的标签，Load avg代表集群的平均负载，τ代表一次心跳计算节点最多能分配的container数量，Load代表计算节点预测负载。
  3. 根据权利要求1所述基于负载预测的Hadoop计算任务初始分配方法，其特征在于，所述步骤3，具体包括：
    步骤3.1：遍历根节点下的所有子队列，基于DRF选择用户队列，具体包括步骤3.1.1～步骤3.1.4：
    步骤3.1.1：确定队列i主资源权值domin i，具体包括步骤3.1.1.1～步骤3.1.1.2：
    步骤3.1.1.1：根据公式(5)和公式(6)计算内存和CPU核数资源的权值；
    Figure PCTCN2019091267-appb-100014
    Figure PCTCN2019091267-appb-100015
    其中，memory used代表了该队列已经使用掉的内存，memory cluster代表整个集群的内存，vcores used代表了该队列已经使用掉的核数，vcores cluster代表整个集群的核数，
    Figure PCTCN2019091267-appb-100016
    代表第i个队列配置核数占集群总核数的比值，
    Figure PCTCN2019091267-appb-100017
    代表第i个队列配置内存占集群总内存的比值；
    3.1.1.2，判断内存资源的权值
    Figure PCTCN2019091267-appb-100018
    和CPU核数资源的权值
    Figure PCTCN2019091267-appb-100019
    大小，当
    Figure PCTCN2019091267-appb-100020
    时，则队列的主导资源为核数，
    Figure PCTCN2019091267-appb-100021
    为该队列主资源的权值，
    Figure PCTCN2019091267-appb-100022
    当
    Figure PCTCN2019091267-appb-100023
    时，则队列的主导资源为内存，
    Figure PCTCN2019091267-appb-100024
    为该队列主资源的权值，
    Figure PCTCN2019091267-appb-100025
    步骤3.1.2：根据公式(7)计算主资源最小份额
    Figure PCTCN2019091267-appb-100026
    Figure PCTCN2019091267-appb-100027
    其中，
    Figure PCTCN2019091267-appb-100028
    第i个队列主资源最小分配额系数，domian used已经用掉的主资源， domian minshare最小的主资源份额；
    步骤3.1.3：当存在队列主资源最小份额小于1时，选择
    Figure PCTCN2019091267-appb-100029
    最大的资源队列分配资源；当所有队列的主资源最小份额大于等于1时，选择主资源权值最大的队列分配资源；
    步骤3.1.4：判断该队列是否为叶子队列，若为叶子队列，则调到步骤3.2；若不为叶子队列，则将该节点设置为根节点，转到步骤3.1；
    步骤3.2：对于已经选择的用户队列，基于DRF对用户队列中的内容，即用户作业进行选择，具体包括：
    步骤3.2.1：遍历所有作业，计算每个作业的资源权值，比较
    Figure PCTCN2019091267-appb-100030
    和
    Figure PCTCN2019091267-appb-100031
    将最大的作为主资源权值；
    步骤3.2.2：比较不同作业的主资源权值：当存在一个最大主资源权值，选择主资源份额高作业分配资源；当存在多个相等最大主资源份额，选择提交作业时间早的作业分配资源；
    步骤3.3：根据优先级关系：Map fail＞Map＞Reduce，选择作业中的优先级最高的任务进行container开启，即子任务开启，其中，Map fail为Map中失败的任务，Map为Map等待的任务，Reduce为Reduce中等待的任务。
PCT/CN2019/091267 2019-06-13 2019-06-14 一种基于负载预测的Hadoop计算任务初始分配方法 WO2020248226A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201910510964.3A CN110262897B (zh) 2019-06-13 2019-06-13 一种基于负载预测的Hadoop计算任务初始分配方法
CN201910510964.3 2019-06-13

Publications (1)

Publication Number Publication Date
WO2020248226A1 true WO2020248226A1 (zh) 2020-12-17

Family

ID=67918048

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2019/091267 WO2020248226A1 (zh) 2019-06-13 2019-06-14 一种基于负载预测的Hadoop计算任务初始分配方法

Country Status (2)

Country Link
CN (1) CN110262897B (zh)
WO (1) WO2020248226A1 (zh)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114077486A (zh) * 2021-11-22 2022-02-22 内蒙古大学 一种MapReduce任务调度方法及系统

Families Citing this family (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112486687B (zh) * 2020-12-03 2022-09-27 重庆邮电大学 一种基于多任务学习时间序列的云平台工作负载预测方法
CN112527454A (zh) * 2020-12-04 2021-03-19 上海连尚网络科技有限公司 容器组调度方法、装置、电子设备和计算机可读介质
CN114064294B (zh) * 2021-11-29 2022-10-04 郑州轻工业大学 移动边缘计算环境下的动态资源分配方法和系统
CN113946430B (zh) * 2021-12-20 2022-05-06 北京并行科技股份有限公司 一种作业调度方法、计算设备及存储介质

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20160188594A1 (en) * 2014-12-31 2016-06-30 Cloudera, Inc. Resource management in a distributed computing environment
CN108182490A (zh) * 2017-12-27 2018-06-19 南京工程学院 一种大数据环境下的短期负荷预测方法
US10013289B2 (en) * 2016-04-28 2018-07-03 International Business Machines Corporation Performing automatic map reduce job optimization using a resource supply-demand based approach
CN108509280A (zh) * 2018-04-23 2018-09-07 南京大学 一种基于推送模型的分布式计算集群本地性调度方法
CN108681777A (zh) * 2018-05-07 2018-10-19 北京京东尚科信息技术有限公司 一种基于分布式系统的机器学习程序运行的方法和装置
CN109117254A (zh) * 2018-06-29 2019-01-01 国家电网公司 一种深度学习框架的调度方法及系统

Family Cites Families (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103399800B (zh) * 2013-08-07 2016-04-13 山东大学 基于Linux并行计算平台的动态负载均衡方法
WO2015187710A1 (en) * 2014-06-02 2015-12-10 Blackwatch International A generic template node for developing and deploying model software packages
CN107273209B (zh) * 2017-06-09 2020-11-03 北京工业大学 基于最小生成树聚类改进遗传算法的Hadoop任务调度方法
CN108170531B (zh) * 2017-12-26 2021-07-02 北京工业大学 一种基于深度信念网络的云数据中心请求流调度方法

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20160188594A1 (en) * 2014-12-31 2016-06-30 Cloudera, Inc. Resource management in a distributed computing environment
US10013289B2 (en) * 2016-04-28 2018-07-03 International Business Machines Corporation Performing automatic map reduce job optimization using a resource supply-demand based approach
CN108182490A (zh) * 2017-12-27 2018-06-19 南京工程学院 一种大数据环境下的短期负荷预测方法
CN108509280A (zh) * 2018-04-23 2018-09-07 南京大学 一种基于推送模型的分布式计算集群本地性调度方法
CN108681777A (zh) * 2018-05-07 2018-10-19 北京京东尚科信息技术有限公司 一种基于分布式系统的机器学习程序运行的方法和装置
CN109117254A (zh) * 2018-06-29 2019-01-01 国家电网公司 一种深度学习框架的调度方法及系统

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
DONG, CHUNTAO ET AL.: "Non-official translation: Research on HadoopP YARN Big Data Computing Framework and Resource Scheduling Mechanism Thereof", INFORMATION AND COMMUNICATIONS TECHNOLOGIES, no. 01, 15 February 2015 (2015-02-15), ISSN: 1674-1285, DOI: 20200115105446 *

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114077486A (zh) * 2021-11-22 2022-02-22 内蒙古大学 一种MapReduce任务调度方法及系统
CN114077486B (zh) * 2021-11-22 2024-03-29 内蒙古大学 一种MapReduce任务调度方法及系统

Also Published As

Publication number Publication date
CN110262897B (zh) 2023-01-31
CN110262897A (zh) 2019-09-20

Similar Documents

Publication Publication Date Title
WO2020248226A1 (zh) 一种基于负载预测的Hadoop计算任务初始分配方法
CN110096349B (zh) 一种基于集群节点负载状态预测的作业调度方法
WO2023184939A1 (zh) 基于深度强化学习的云数据中心自适应高效资源分配方法
US10089140B2 (en) Dynamically adaptive, resource aware system and method for scheduling
Hui et al. Improved strategies for dynamic load balancing
CN106790726B (zh) 一种基于Docker云平台的优先级队列动态反馈负载均衡资源调度方法
CN109324875B (zh) 一种基于强化学习的数据中心服务器功耗管理与优化方法
WO2017045211A1 (zh) 一种多服务质量约束的云计算任务调度方法
CN104168318B (zh) 一种资源服务系统及其资源分配方法
US9218213B2 (en) Dynamic placement of heterogeneous workloads
US8332873B2 (en) Dynamic application instance placement in data center environments
WO2016078178A1 (zh) 一种虚拟cpu调度方法
CN109120715A (zh) 一种云环境下动态负载均衡方法
Dhari et al. An efficient load balancing scheme for cloud computing
CN108845874B (zh) 资源的动态分配方法及服务器
CN113454614A (zh) 用于分布式计算中的资源划分的系统和方法
CN104298550A (zh) 一种面向Hadoop的动态调度方法
US11042419B2 (en) Cooperative scheduling method and system for computing resource and network resource of container cloud platform
CN108170531B (zh) 一种基于深度信念网络的云数据中心请求流调度方法
CN111190691A (zh) 适用于虚拟机的自动迁移方法、系统、装置及存储介质
CN108574600B (zh) 云计算服务器的功耗和资源竞争协同控制的服务质量保障方法
CN110221909A (zh) 一种基于负载预测的Hadoop计算任务推测执行方法
CN112306642A (zh) 一种基于稳定匹配博弈理论的工作流调度方法
CN115878260A (zh) 一种低碳自适应云主机任务调度系统
Gao et al. Deadline-aware preemptive job scheduling in hadoop yarn clusters

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19932341

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 19932341

Country of ref document: EP

Kind code of ref document: A1