CN113934525A - Hadoop cluster task scheduling method based on positive and negative feedback load scheduling algorithm - Google Patents

Hadoop cluster task scheduling method based on positive and negative feedback load scheduling algorithm Download PDF

Info

Publication number
CN113934525A
CN113934525A CN202111192598.5A CN202111192598A CN113934525A CN 113934525 A CN113934525 A CN 113934525A CN 202111192598 A CN202111192598 A CN 202111192598A CN 113934525 A CN113934525 A CN 113934525A
Authority
CN
China
Prior art keywords
task
node
calculation
positive
method based
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202111192598.5A
Other languages
Chinese (zh)
Inventor
岳正飞
郭强
杨融
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Positive Network Technology Co ltd
Original Assignee
Positive Network Technology Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Positive Network Technology Co ltd filed Critical Positive Network Technology Co ltd
Priority to CN202111192598.5A priority Critical patent/CN113934525A/en
Publication of CN113934525A publication Critical patent/CN113934525A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/48—Program initiating; Program switching, e.g. by interrupt
    • G06F9/4806—Task transfer initiation or dispatching
    • G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
    • G06F9/4881—Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10—File systems; File servers
    • G06F16/18—File system types
    • G06F16/182—Distributed file systems
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
    • G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
    • G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
    • G06F9/505—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals considering the load
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
    • G06F9/5083—Techniques for rebalancing the load in a distributed system

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Software Systems (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • Databases & Information Systems (AREA)
  • Multi Processors (AREA)

Abstract

The invention discloses a Hadoop cluster task scheduling method based on a positive and negative feedback load scheduling algorithm, which is applied to a Hadoop resource allocation center, and comprises the following steps: an idle computing node applies for a task execution request to a management node; the management node receives the task request and inquires whether a history record for running the task request exists; comparing and analyzing the historical records, if the task configuration meets the resource requirement, calculating the task in the calculation node, otherwise, re-matching the calculation node with a new task request; the method can dynamically distribute the computing tasks according to the resource use conditions of different servers and the performance differences of the servers in the cluster task scheduling, realize the efficient completion of the computing tasks and improve the use efficiency of the CPU and the resources.

Description

Hadoop cluster task scheduling method based on positive and negative feedback load scheduling algorithm
Technical Field
The invention belongs to the technical field of cluster task scheduling, and particularly relates to a Hadoop cluster task scheduling method based on a positive and negative feedback load scheduling algorithm.
Background
One of the core applications of the Hadoop platform is a job scheduling technology, the job scheduling technology allocates computing tasks to computing nodes as required for execution, and the allocation mode directly influences the resource utilization rate and the overall performance of the Hadoop platform system. Generally, a platform adopts a queue storage mode, namely a first-come first-allocation mode, for scheduling, but the scheduling mode lacks interactivity, cannot be allocated better according to needs, and realizes the maximization of resource utilization. If the allocation of the job scheduling algorithm is not reasonable, resource load task allocation of the cluster is unbalanced, and the operation efficiency of the cluster is affected. In order to optimize a job scheduling algorithm and solve the existing problems, Yahoo and Facebook companies respectively research Capacity Schedule and Fair Schedule algorithms to solve the problems existing in job scheduling, the two algorithms configure cluster computing nodes in a manual allocation mode through the task resource consumption condition and the cluster node operation condition of an administrator, and the method solves the problems existing in a first-come first-distributed mode to a certain extent, but does not realize the problem in a pure automatic mode. Therefore, by researching a resource scheduling framework (Yarn) and a distributed file system (HDFS) realized by Hadoop, the defects of the existing resource scheduling algorithm are improved, and the method has important significance for improving the overall performance of the cloud computing task and the utilization rate of resources.
The job scheduling algorithm is mainly applied to reasonably distribute a plurality of tasks on each job according to the requirements of the tasks, and the distributed nodes are guaranteed to be executed efficiently. The existing job scheduling algorithms have certain defects, for example, the scheduling algorithm according to the queue mode is generally used under the conditions that cluster nodes are similar in configuration and job tasks are the same in operation, and is only suitable for specific scenes with the same or basically similar jobs and tasks; although the calculation capacity scheduling algorithm can improve the system utilization rate and carry out automatic allocation according to the job execution efficiency, the hardware condition needs to be manually configured in advance, and automatic allocation of the machine cannot be realized. The existing scheduling method does not solve the problem in use fundamentally at present.
Chinese patent application No. 201710113656.8 discloses a distributed task scheduling system and method for improving efficiency and reliability of task scheduling execution, the system includes: the method comprises the following steps that a task submitting cluster, a task scheduling cluster and a task executing cluster are adopted; the task submitting cluster comprises a plurality of task client nodes, the task scheduling cluster comprises a plurality of task scheduling nodes, and the task executing cluster comprises a plurality of task executing nodes; the task client nodes in the task submitting cluster are used for submitting tasks to the task scheduling cluster; the task scheduling nodes in the task scheduling cluster are used for receiving the tasks, generating task allocation information of the tasks and sending the tasks to the task execution cluster according to the task allocation information; and the task execution nodes in the task execution cluster are used for executing the tasks and returning execution results to the task client nodes submitting the tasks. In the prior art, hardware conditions need to be manually configured in advance, automatic allocation of machines cannot be realized, so that resource distribution is uneven, the resource utilization rate is low, and the efficient operation stability of a cluster is influenced.
Disclosure of Invention
Aiming at the defects of the prior art, the invention aims to provide a Hadoop cluster task scheduling method based on a positive and negative feedback load scheduling algorithm, which can dynamically allocate computing tasks according to the resource use conditions of different servers and the performance difference of the servers in the cluster task scheduling, realize the efficient completion of the computing tasks and improve the use efficiency of a CPU and resources.
The invention provides the following technical scheme:
a Hadoop cluster task scheduling method based on a positive and negative feedback load scheduling algorithm is applied to a Hadoop resource allocation center, and comprises the following steps:
an idle computing node applies for a task execution request to a management node;
the management node receives the task request and inquires whether a history record for running the task request exists;
and comparing and analyzing the historical records, if the task configuration meets the resource requirement, calculating the task in the calculation node, and otherwise, re-matching the calculation node with a new task request.
Preferably, when the Node Manager of an idle Data Node computing Node in the Hadoop cluster applies for executing task resources to the Resource Manager of the management Node Name Node, the application information includes the memory of each computing Node, the disk I/O read-write rate, the network bandwidth, the size of the memory being used, and the disk margin.
Preferably, before the management Node receives the task request and inquires whether the history record for running the task request exists, the management Node arbitrarily pulls a calculation task of the Name Node calculation Node from a task queue initialized by the system or a CPU running queue and a disk load queue, and inquires the record of the completed task before the task is calculated.
Preferably, if the query history table does not have the relevant information of the running task, the Resource Manager in the management Node Name Node directly calculates the task at the Data Node calculation Node, and after the Name Node finishes the calculation task, the whole calculation task is stored in the history table according to the operation time consumption, the CPU utilization rate and the disk condition.
Preferably, if the query history list has the relevant information of the running task, the currently distributed task is compared with the history information, and the Resource Manager in the management Node Name Node directly calculates the task at the Data Node calculation Node through the analysis of the history running record and the task configuration according with the current Resource requirement.
Preferably, if the history records are compared to find that the current task is not in accordance with the initial configuration of the current task, the task is taken out and stored in a new queue, then the new task is taken out from the original queue again, and then the management node performs the query and comparison operation again until a reasonable task is matched.
Preferably, after matching reasonable tasks, the management Node Name Node completes the calculation task, and the whole calculation task is stored in the history table according to the operation time consumption, the CPU utilization rate and the disk condition.
Preferably, during task allocation, the Data Node computing Node applies for computing resources from the management Node, the application information carries detailed resources of the computing Node, including CPU utilization, disk and memory detailed information, and the Name Node management Node stores the application resources in a corresponding queue. The Name Node management Node takes out the task from the queue and randomly distributes the calculation task to the Data Node calculation Node.
Preferably, when detecting the resource, after the Data Node computing Node receives the task distributed by the Name Node management Node, the Data Node computing Node firstly inquires the historical operation record table to verify whether the operation related computing task can be satisfied.
Preferably, if the Data Node computing Node receives the task allocated by the Name Node management Node, if the condition is not met, the feedback is triggered, that is, the Data Node computing Node sends an operation failure signal to the management Node, requests a new computing task and carries the detailed resource information of the Node.
Preferably, the task is redistributed, the Name Node management Node rejoins the task into the queue after receiving the failure information of the computing Node, and marks the detailed information of the Node operation failure to wait for the management Node to redistribute the task.
In addition, by the technical scheme, the utilization rate of the server resources can be quantified, and the data quantity in the computing resources is assumed to be ID and the data output quantity is OD. The amount of data processed by computer resources changes, and assuming that the change rate is P, OD is P × ID, the time for executing one task is TCT, and the disk processing and memory frequency is IOR. Therefore it has the advantages of
Figure BDA0003301801060000051
Feedback logic is available. And if the IOR frequency is greater than the set threshold value, triggering a feedback mechanism, otherwise executing the calculation task at the node.
In addition, the Hadoop cluster task scheduling method based on the positive and negative feedback load scheduling algorithm adopts a cluster task scheduling system, and the system comprises a processor, a memory, a communication unit, an input unit and a programmable logic unit; the processor is a central processing unit.
Preferably, the cluster task scheduling system further comprises a communication bus, and the communication bus enables the task scheduling system to be used for scheduling the cluster tasksBy means of I2C, a communication bus; the processor, the memory, the communication unit, the input unit and the programmable logic unit are all connected through I2And C, the communication buses complete the communication among the communication buses.
The memory stores program instruction codes, the program instruction codes are computer operation instructions, and when the cluster task scheduling is carried out, the memory at least has computer program instructions for realizing the following functions: an idle computing node applies for a task execution request to a management node; the management node receives the task request and inquires whether a history record for running the task request exists; and comparing and analyzing the historical records, if the task configuration meets the resource requirement, calculating the task in the calculation node, and otherwise, re-matching the calculation node with a new task request.
In addition, when cluster task scheduling is carried out, task execution requests are applied to the management node through the idle computing nodes; the management node receives the task request and inquires whether a history record for running the task request exists; and comparing and analyzing the historical records, if the task configuration meets the resource requirement, calculating the task in the calculation node, otherwise, re-matching the calculation node with a new task request, ensuring the balance of resource allocation in the cluster, improving the resource utilization rate and ensuring the efficient and stable operation of the cluster.
Compared with the prior art, the invention has the following beneficial effects:
(1) the Hadoop cluster task scheduling method based on the positive and negative feedback load scheduling algorithm can dynamically allocate computing tasks according to resource use conditions of different servers and server performance differences in cluster task scheduling, achieves efficient completion of the computing tasks, and improves the use efficiency of a CPU and resources.
(2) The Hadoop cluster task scheduling method based on the positive and negative feedback load scheduling algorithm ensures balanced resource distribution in the cluster, improves the resource utilization rate and ensures efficient and stable operation of the cluster.
(3) The invention relates to a Hadoop cluster task scheduling method based on a positive and negative feedback load scheduling algorithm, which reclassifies scheduling tasks by examining the running condition and running state of the tasks, and solves the defect that hardware needs to be set in advance in the current algorithm by using a learning mode, thereby solving the problems of uneven resource distribution and low resource utilization rate caused by the fact that automatic allocation of a machine cannot be realized in the prior art, and improving the running efficiency of a Hadoop platform.
(4) The invention relates to a Hadoop cluster task scheduling method based on a positive and negative feedback load scheduling algorithm, which solves the problems that the resource distribution is unbalanced and cannot be distributed according to needs in the average distribution of tasks of the traditional Hadoop platform, and also solves the defect that the resource distribution cannot be realized automatically because manual configuration is needed in an optimization algorithm. The method really realizes full-automatic allocation according to needs according to cluster resources and operation task amount in the task scheduling process of the Hadoop cluster, fundamentally improves the rationality of task scheduling allocation, improves the calculation efficiency, optimizes the utilization rate of CPU resources, and can enable the Hadoop platform to be well suitable for various distributed storage and high-concurrency applications.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings that are required to be used in the embodiments will be briefly described below, it should be understood that the following drawings only illustrate some embodiments of the present invention and therefore should not be considered as limiting the scope, and for those skilled in the art, other related drawings can be obtained according to the drawings without inventive efforts.
FIG. 1 is a flow chart of the method of the present invention.
FIG. 2 is a flow chart of a cluster task scheduling algorithm of the present invention.
Fig. 3 is a block diagram of a cluster task scheduling system according to the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention more apparent, the technical solutions of the embodiments of the present invention will be described in detail and completely with reference to the accompanying drawings. It is to be understood that the described embodiments are only a few embodiments of the present invention, and not all embodiments. All other embodiments, which can be obtained by a person skilled in the art without any inventive step based on the embodiments of the present invention, are within the scope of the present invention.
Thus, the following detailed description of the embodiments of the present invention, presented in the figures, is not intended to limit the scope of the invention, as claimed, but is merely representative of selected embodiments of the invention. All other embodiments, which can be obtained by a person skilled in the art without any inventive step based on the embodiments of the present invention, are within the scope of the present invention.
The first embodiment is as follows:
as shown in fig. 1-2, a Hadoop cluster task scheduling method based on a positive and negative feedback load scheduling algorithm is applied to a Hadoop resource allocation center, and the cluster task scheduling method includes:
s1, the idle computing node applies for executing task request to the management node;
s2, the management node receives the task request and inquires whether the history record of the task request is available;
and S3, comparing and analyzing the historical records, if the task configuration meets the resource requirement, calculating the task in the computing node, otherwise, re-matching the computing node with the new task request.
When the Node Manager of the idle Data Node computing Node in the Hadoop cluster applies for executing task resources to the Resource Manager of the management Node Name Node, the application information comprises the memory of each computing Node, the disk I/O read-write rate, the network bandwidth, the size of the currently used memory and the disk allowance.
The management Node receives the task request, inquires whether a historical record for running the task request exists, the management Node arbitrarily pulls a calculation task of the Name Node calculation Node from a task queue initialized by a system or a CPU running queue and a disk load queue, and inquires the completed task record before the task is calculated.
If the query history list does not have the relevant information of the running task, the Resource Manager in the management Node Name Node directly calculates the task at the Data Node calculation Node, and after the Name Node finishes the calculation task, the whole calculation task is stored in the history list according to the operation time consumption, the CPU utilization rate and the disk condition.
If the query history list has the relevant information of the running task, comparing the currently distributed task with the history information, and calculating the task at the Data Node calculation Node directly by the Resource Manager in the management Node Name Node through the analysis of the history running record and the task configuration according with the current Resource requirement.
And if the historical records are compared to find that the task is not accordant with the initial configuration of the current task, taking out the task and storing the task into a new queue, then taking out the new task from the original queue again, and then carrying out the query and comparison operation again by the management node until a reasonable task is matched. After matching reasonable tasks, the management Node Name Node completes calculation tasks, and the whole calculation tasks are stored in a history record table according to operation time consumption, CPU utilization rate and disk conditions.
Example two:
on the basis of the first embodiment, during task allocation, the Data Node computing Node applies for computing resources from the management Node, the application information carries detailed resources of the computing Node, including CPU utilization, disk and memory detailed information, and the Name Node management Node stores the application resources in a corresponding queue. The Name Node management Node takes out the task from the queue and randomly distributes the calculation task to the Data Node calculation Node. When detecting the resources, after the Data Node computing Node receives the task distributed by the Name Node management Node, firstly, the historical operation record table is inquired, and whether the operation of the related computing task can be met is verified. If the Data Node computing Node receives the task distributed by the Name Node management Node, the feedback is triggered if the condition is not met, namely the Data Node computing Node sends an operation failure signal to the management Node to request a new computing task and carry the detailed resource information of the Node. And reallocating the tasks, namely, the management Node of the Name Node rejoins the tasks into the queue after receiving the failure information of the computing Node, and marks the detailed information of the operation failure of the Node to wait for the management Node to reallocate the tasks.
Example three:
as shown in fig. 3, on the basis of the first embodiment, a Hadoop cluster task scheduling method based on a positive-negative feedback load scheduling algorithm employs a cluster task scheduling system, which includes a processor, a memory, a communication unit, an input unit, and a programmable logic unit; the processor is a central processing unit. The cluster task scheduling system also comprises a communication bus, wherein the communication bus uses I2C, a communication bus; the processor, the memory, the communication unit, the input unit and the programmable logic unit are all connected through I2And C, the communication buses complete the communication among the communication buses.
The memory stores program instruction codes, the program instruction codes are computer operation instructions, and when the cluster task scheduling is carried out, the memory at least has computer program instructions for realizing the following functions: an idle computing node applies for a task execution request to a management node; the management node receives the task request and inquires whether a history record for running the task request exists; and comparing and analyzing the historical records, if the task configuration meets the resource requirement, calculating the task in the calculation node, and otherwise, re-matching the calculation node with a new task request.
Example four
On the basis of the first embodiment, in addition, by the technical solution, the utilization rate of the server resources can be quantified, and it is assumed that the data quantity in the computing resources is ID and the data output quantity is OD. The amount of data processed by computer resources changes, and assuming that the change rate is P, OD is P × ID, the time for executing one task is TCT, and the disk processing and memory frequency is IOR. Therefore it has the advantages of
Figure BDA0003301801060000111
Feedback logic is available. And if the IOR frequency is greater than the set threshold value, triggering a feedback mechanism, otherwise executing the calculation task at the node.
When cluster task scheduling is carried out, an idle computing node applies for a task execution request to a management node; the management node receives the task request and inquires whether a history record for running the task request exists; and comparing and analyzing the historical records, if the task configuration meets the resource requirement, calculating the task in the calculation node, otherwise, re-matching the calculation node with a new task request, ensuring the balance of resource allocation in the cluster, improving the resource utilization rate and ensuring the efficient and stable operation of the cluster.
The device obtained by the technical scheme is a Hadoop cluster task scheduling method based on a positive and negative feedback load scheduling algorithm, and can dynamically allocate computing tasks according to resource use conditions of different servers and server performance differences in cluster task scheduling, so that the computing tasks are efficiently completed, and the use efficiency of a CPU (Central processing Unit) and resources is improved. The method ensures the balance of resource distribution in the cluster, improves the resource utilization rate and ensures the efficient and stable operation of the cluster. The scheduling tasks are reclassified by examining the running conditions and running states of the tasks, and the learning mode is used for overcoming the defect that hardware needs to be set in advance in the existing algorithm, so that the problems that the machine automation distribution cannot be realized in the prior art, the resource distribution is uneven, the resource utilization rate is low are solved, and the running efficiency of the Hadoop platform is improved. The problem that the traditional Hadoop platform is unbalanced in resource distribution and incapable of distributing resources according to needs in the process of distributing tasks in an average mode is solved, and the defect that the resource distribution cannot be automated due to the fact that manual configuration is needed in an optimization algorithm is overcome. The method really realizes full-automatic allocation according to needs according to cluster resources and operation task amount in the task scheduling process of the Hadoop cluster, fundamentally improves the rationality of task scheduling allocation, improves the calculation efficiency, optimizes the utilization rate of CPU resources, and can enable the Hadoop platform to be well suitable for various distributed storage and high-concurrency applications.
The above description is only a preferred embodiment of the present invention, and is not intended to limit the present invention, and it will be apparent to those skilled in the art that various modifications and variations can be made in the present invention; any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims (7)

1. A Hadoop cluster task scheduling method based on a positive and negative feedback load scheduling algorithm is applied to a Hadoop resource allocation center and is characterized by comprising the following steps:
an idle computing node applies for a task execution request to a management node;
the management node receives the task request and inquires whether a history record for running the task request exists;
and comparing and analyzing the historical records, if the task configuration meets the resource requirement, calculating the task in the calculation node, and otherwise, re-matching the calculation node with a new task request.
2. The Hadoop cluster task scheduling method based on the positive and negative feedback load scheduling algorithm as claimed in claim 1, wherein when a Node Manager of an idle Data Node compute Node in a Hadoop cluster applies for executing task resources to a Resource Manager of a management Node Name Node, the application information includes memory of each compute Node, disk I/O read-write rate, network bandwidth, memory size being used, and disk margin.
3. The Hadoop cluster task scheduling method based on the positive and negative feedback load scheduling algorithm as claimed in claim 1, wherein the management Node receives the task request, queries whether there is a history record for running the task request, and arbitrarily pulls a computation task of the Name Node computation Node from a task queue initialized by the management Node or a CPU running queue and a disk load queue, and queries a completed task record before computing the task.
4. The Hadoop cluster task scheduling method based on the positive and negative feedback load scheduling algorithm as claimed in claim 3, wherein if the query history list has no related information of the running task, the Resource Manager in the management Node Name Node directly calculates the task at the Data Node calculation Node, and after the Name Node completes the calculation task, the whole calculation task is stored in the history list according to the operation time consumption, the CPU utilization rate and the disk condition.
5. The Hadoop cluster task scheduling method based on the positive and negative feedback load scheduling algorithm as claimed in claim 1, wherein if the query history list has the related information of the running task, the currently allocated task is compared with the history information, and through the analysis with the history running record and the task configuration meeting the current Resource requirement, the Resource Manager in the management Node Name Node directly calculates the task at the Data Node calculation Node.
6. The Hadoop cluster task scheduling method based on the positive and negative feedback load scheduling algorithm as claimed in claim 5, wherein if the historical records are compared to find that the task is not in accordance with the initialization configuration of the current task, the task is taken out and stored in a new queue, then the new task is taken out from the original queue again, and then the management node performs the query and comparison operation again until a reasonable task is matched.
7. The Hadoop cluster task scheduling method based on the positive and negative feedback load scheduling algorithm as claimed in claim 6, wherein after matching reasonable tasks, the management Node Name Node completes the calculation task, and stores the whole calculation task in a history table according to the time consumption of the operation, the CPU utilization rate and the disk condition.
CN202111192598.5A 2021-10-13 2021-10-13 Hadoop cluster task scheduling method based on positive and negative feedback load scheduling algorithm Pending CN113934525A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202111192598.5A CN113934525A (en) 2021-10-13 2021-10-13 Hadoop cluster task scheduling method based on positive and negative feedback load scheduling algorithm

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202111192598.5A CN113934525A (en) 2021-10-13 2021-10-13 Hadoop cluster task scheduling method based on positive and negative feedback load scheduling algorithm

Publications (1)

Publication Number Publication Date
CN113934525A true CN113934525A (en) 2022-01-14

Family

ID=79278684

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202111192598.5A Pending CN113934525A (en) 2021-10-13 2021-10-13 Hadoop cluster task scheduling method based on positive and negative feedback load scheduling algorithm

Country Status (1)

Country Link
CN (1) CN113934525A (en)

Similar Documents

Publication Publication Date Title
US11314551B2 (en) Resource allocation and scheduling for batch jobs
CN107066319B (en) Multi-dimensional scheduling system for heterogeneous resources
CN107038069B (en) Dynamic label matching DLMS scheduling method under Hadoop platform
CN107832153B (en) Hadoop cluster resource self-adaptive allocation method
CN103092698B (en) Cloud computing application automatic deployment system and method
US9104498B2 (en) Maximizing server utilization within a datacenter
CN109034396B (en) Method and apparatus for processing deep learning jobs in a distributed cluster
US8626890B2 (en) Connection pool use of runtime load balancing service performance advisories
US9323580B2 (en) Optimized resource management for map/reduce computing
US20130198755A1 (en) Apparatus and method for managing resources in cluster computing environment
CN114741207B (en) GPU resource scheduling method and system based on multi-dimensional combination parallelism
US9870269B1 (en) Job allocation in a clustered environment
CN109564528B (en) System and method for computing resource allocation in distributed computing
JP4970939B2 (en) Hierarchical management of dynamic resource allocation in multi-node systems
CN103761146B (en) A kind of method that MapReduce dynamically sets slots quantity
CN110221920B (en) Deployment method, device, storage medium and system
CN109257399B (en) Cloud platform application program management method, management platform and storage medium
CN104735095A (en) Method and device for job scheduling of cloud computing platform
CN111459641A (en) Cross-machine-room task scheduling and task processing method and device
KR101765725B1 (en) System and Method for connecting dynamic device on mass broadcasting Big Data Parallel Distributed Processing
CN114860387A (en) I/O virtualization method of HBA controller for virtualized storage application
CN111580951A (en) Task allocation method and resource management platform
CN105550025A (en) Distributed IaaS (Infrastructure as a Service) scheduling method and system
CN108287762B (en) Distributed computing interactive mode use resource optimization method and computer equipment
Wang et al. Improving utilization through dynamic VM resource allocation in hybrid cloud environment

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination