CN103761146B - A kind of method that MapReduce dynamically sets slots quantity - Google Patents
A kind of method that MapReduce dynamically sets slots quantity Download PDFInfo
- Publication number
- CN103761146B CN103761146B CN201410004521.4A CN201410004521A CN103761146B CN 103761146 B CN103761146 B CN 103761146B CN 201410004521 A CN201410004521 A CN 201410004521A CN 103761146 B CN103761146 B CN 103761146B
- Authority
- CN
- China
- Prior art keywords
- node
- tasktracker
- slots
- slot
- task
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 29
- 230000015654 memory Effects 0.000 claims abstract description 66
- 230000008569 process Effects 0.000 claims description 7
- 238000012544 monitoring process Methods 0.000 claims description 5
- 238000013461 design Methods 0.000 claims description 4
- 238000013467 fragmentation Methods 0.000 claims description 4
- 238000006062 fragmentation reaction Methods 0.000 claims description 4
- 230000008859 change Effects 0.000 claims description 2
- 230000001419 dependent effect Effects 0.000 claims 1
- 238000012545 processing Methods 0.000 description 5
- 230000000694 effects Effects 0.000 description 4
- 238000004364 calculation method Methods 0.000 description 2
- 230000007812 deficiency Effects 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 241000489861 Maximus Species 0.000 description 1
- 238000000429 assembly Methods 0.000 description 1
- 230000000712 assembly Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 230000000739 chaotic effect Effects 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 238000000151 deposition Methods 0.000 description 1
- 239000002360 explosive Substances 0.000 description 1
- 230000003862 health status Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000009434 installation Methods 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Landscapes
- Devices For Executing Special Programs (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The present invention provides a kind of method that MapReduce dynamically sets slots quantity, comprises the following steps：Node computing capability setting slots quantity first in cluster；Then slots quantity is suitably adjusted further according to the internal memory situation in each node.Compared to the prior art a kind of MapReduce dynamically sets the method for slots quantity, can lift mapreduce program runnabilities, optimizes the reasonable utilization of resource, practical, it is easy to promote.
Description
Technical field
The present invention relates to field of computer technology, specifically a kind of MapReduce dynamically sets the side of slots quantity
Method.
Background technology
Internet technology of today is continued to develop, and data are into explosive growth, and data scale is sharply increased on network, chaotic
Data in contain huge business opportunity, can be worth from the extracting data of magnanimity.But accompanying problem is that unit
Data-handling capacity can not meet the processing requirement of current mass data application, the Distributed Calculation based on large-scale calculations cluster
Main path as Future Data performance boost.Core technology MapReduce computation module for Hadoop is studied,
One kind is proposed for map, reduce count issue of default setting same number in each node in MapReduce
MapReduce dynamically sets the strategy of slots quantity.Set different according to the hardware configuration difference of different nodes in cluster
Map quantity and reduce quantity.
It is as follows for map numbers in mapreduce and the setting of reduce numbers at present：Map task quantity is
Mapred.tasktracker.map.tasks.maximu parameter value, but a TaskTracker can configure how many
Slot, or it is relevant with its physical environment.Each task is independently executed by the JVM newly started, when having multiple task
Multiple JVM are just had, each JVM consumes a part of internal memory, along with DataNode and TaskTracker memory consumption, machine
Device internal memory is possible will be not enough.So in addition to considering each new internal memory limitation for starting JVM of allotment, it must pay close attention to down on earth
Need it is how many it is new start JVM, that is, map slot and reduce slot number.They setting also with machine
Processor number is relevant.Specifically configure from must coming from the actual motion effect of cluster and analyze.Input Split's is big
It is small, determine that a Job possesses how many map.If however, the data volume of input is huge, then the block of acquiescence has several
Ten thousand even the Map Task of hundreds of thousands, the network transmission of cluster can be very big, and most serious is to Job Tracker scheduling, team
Row, internal memory can all bring very big pressure.Therefore the slots quantity for suitably conforming to machine computing capability is set.
In Hadoop, the resource on each TaskTraker is represented using slot, a slot represents the resource of fixation
Combination, when performing mapreduce programs, Map slot numbers and Reduce slot numbers on each TaskTracker are
By mapred.tasktracker.map.tasks.maximum and
Mapred.tasktracker.reduce.tasks.maximum is configured.The two parameters are postponed once matching somebody with somebody, it is impossible to dynamic
Modification.Because the stock number that the task of not same-action needs is different, the node hardware configuration in cluster is also not quite similar, therefore,
For the difference of node resource, propose that a kind of MapReduce dynamically sets the strategy of slots quantity.The strategy can be according to section
Point computing capability dynamically sets slot quantity, improves the performance that MapReduce programs are performed.
The content of the invention
The technical assignment of the present invention is to solve the deficiencies in the prior art dynamically to set slots numbers there is provided a kind of MapReduce
The method of amount.
The technical scheme is that realize in the following manner, a kind of MapReduce dynamically sets slots quantity
Method, its specifically set process as：
The quantity of CPU in clustered node is determined first, then according to the quantity of CPU core in each node by principal and subordinate's mould
Dynamically setting determines slots quantity to formula framework MapReduce：Made according to the resource situation of job queues and TaskTracker nodes
For the memory size of input, the wherein core amounts of TaskTracker resource situation including CPU and node, then further according to
The computing capability setting slots quantity of node；
JobTracker is run on MS master-slave pattern framework MapReduce host node, it is responsible for monitoring a group of planes, task
Scheduling；TaskTracker is run from node, it is responsible for monitor task execution, report on progress；
TaskTracker periodically sends heartbeat message to JobTracker, and the resource that this node is carried in the information is used
Situation；
When heartbeat is reached, the scheduling in host node occurs, if TaskTracker reports oneself available free resource,
JobTracker selects a task to be transmitted into node operation using dispatching algorithm.
Need to design two variables when setting slots quantity, one is map slot, and one is reduce slot：It is first
The code in TaskTracker is first changed, the core amounts for being CPU in node by map slot quantity initial setting, reduce
The half for the core amounts that slot quantity initial setting is CPU in node；Then in class method, according to slots quantity come
The size of application internal memory is determined, task total Memory Allocation size is equal to map slot quantity and single map in TaskTracker
The product of slot memory sizes along with resuce slot quantity and TaskTracker single reduce slot memory sizes it
Product；If task total Memory Allocation is small compared with the free memory of respective nodes in cluster, slots is set as the value；
If the free memory of task total Memory Allocation and respective nodes in cluster is small, map slot quantity or reduce are reduced
Slot quantity, alternate less slots quantity, until meeting in node untill memory conditions.
Produced beneficial effect is the present invention compared with prior art：
A kind of MapReduce of the present invention dynamically sets the method for slots quantity by analyzing Hadoop cluster interior joints
Computing capability, determine slots quantity using the CPU and internal memory situation of each node, then obtained rationally according to the quantity
Map quantity and reduce quantity, the strategy causes the performance of whole cluster processing MapReduce tasks to greatly promote, and excellent
Change the reasonable utilization of resource, it is practical, it is easy to promote.
Brief description of the drawings
Accompanying drawing 1 is the operation job execution flow charts of the present invention.
Accompanying drawing 2 is the flow chart of the setting slots quantity of the present invention.
Embodiment
The method work of slots quantity is dynamically set in detailed below to a kind of MapReduce of the present invention below in conjunction with the accompanying drawings
Explanation.
The present invention relates to the major issue that MapReduce in current big data Hadoop clusters is badly in need of solving, i.e. root
According to the problem of cluster interior joint hardware configuration is different, computing capability Different Dynamic sets map, reduce quantity.Pass through this method
The MapReduce of proposition dynamically sets the strategy of slots quantity, and the strategy can effectively solve dynamic setting slots quantity
Problem, and greatly promote the performance of whole cluster processing MapReduce tasks.
The present invention depends on MS master-slave pattern framework MapReduce, and the framework uses Master/Slave framework, and it leads
It is made up of following 4 parts：
1）Client.
2）JobTracker：JobTracke is responsible for monitoring resource and job scheduling.JobTracker monitoring is all
TaskTracker and job health status, once finding failure, is just transferred to other nodes by corresponding task；Meanwhile,
The information such as implementation progress, the resource usage amount of JobTracker meeting tracing tasks, and task dispatcher is told by these information,
And scheduler can select suitable task to use these resources when resource occurs idle.In Hadoop, task dispatcher
It is a pluggable module, user can design corresponding scheduler according to the need for oneself.
3）TaskTracker：TaskTracker can make resource on this node periodically by Heartbeat
JobTracker is reported to the operation progress of situation and task, while receiving the orders that send over of JobTracker and holding
The corresponding operation of row（Such as start new task, kill task dispatching）.TaskTracker uses " slot " equivalent to divide on this node
Stock number." slot " represents computing resource（CPU, internal memory etc.）.One Task gets after a slot fortune that just has an opportunity
OK, and the effect of Hadoop schedulers is exactly the idle slot on each TaskTracker is distributed into Task to use.
It is two kinds of Map slot and Reduce slot that slot, which is divided to, is used respectively for MapTask and Reduce Task.
TaskTracker passes through slot numbers（Configurable parameter）Limit Task concurrency.
4）Task：It is two kinds of Map Task and Reduce Task that Task, which is divided to, is started by TaskTracker.HDFS
Using the block of fixed size as base unit data storage, and for MapReduce, it is split that it, which handles unit,.
Split is a logical concept, and it only includes some metadata informations, such as data start, data length, data institute
In node etc..Its division methods are determined by user oneself completely.But should be noted that split number determine Map
Task number, because each split can only give a Map Task processing.
As shown in accompanying drawing 1, Fig. 2, a kind of MapReduce that the present invention is provided dynamically sets the method for slots quantity, the plan
Slots quantity is mainly slightly set according to computing capability in clustered node, node computing capability is according to CPU numbers and internal memory
Determined by two factors.The quantity of CPU in clustered node is determined first, then according to the number of CPU core in each node
Amount determines slots quantity, so can handle task according to different node computing capabilitys so that mapreduce tasks are higher
The execution of effect, improves performance.Internal memory factor in the strategy of MapReduce dynamic setting slots quantity, is according to slots quantity
To determine the size for applying for internal memory, the internal memory situation further according to node adjusts slots quantity accordingly, if in application process
Slots quantity can be then reduced when depositing deficiency know and reach memory conditions, otherwise be according to CPU quantity by slots quantity sets
The slots quantity of setting, finally determines map, reduce quantity according to slots quantity.Its is specific set process as：
The quantity of CPU in clustered node is determined first, then according to the quantity of CPU core in each node by principal and subordinate's mould
Dynamically setting determines slots quantity to formula framework MapReduce：Made according to the resource situation of job queues and TaskTracker nodes
For the memory size of input, the wherein core amounts of TaskTracker resource situation including CPU and node, then further according to
The computing capability setting slots quantity of node；
JobTracker is run on MS master-slave pattern framework MapReduce host node, it is responsible for monitoring a group of planes, task
Scheduling；TaskTracker is run from node, it is responsible for monitor task execution, report on progress；
TaskTracker periodically sends heartbeat message to JobTracker, and the resource that this node is carried in the information is used
Situation；
When heartbeat is reached, the scheduling in host node occurs, if TaskTracker reports oneself available free resource,
JobTracker selects a task to be transmitted into node operation using dispatching algorithm.
Need to design two variables when setting slots quantity, one is map slot, and one is reduce slot：It is first
The code in TaskTracker is first changed, the core amounts for being CPU in node by map slot quantity initial setting, reduce
The half for the core amounts that slot quantity initial setting is CPU in node；Then in class method, according to slots quantity come
The size of application internal memory is determined, task total Memory Allocation size is equal to map slot quantity and single map in TaskTracker
The product of slot memory sizes along with resuce slot quantity and TaskTracker single reduce slot memory sizes it
Product；If task total Memory Allocation is small compared with the free memory of respective nodes in cluster, slots is set as the value；
If the free memory of task total Memory Allocation and respective nodes in cluster is small, map slot quantity or reduce are reduced
Slot quantity, alternate less slots quantity, until meeting in node untill memory conditions.
The purpose of the present invention is dynamically to set slots quantity for distributed computing framework.The tactful thought is root
Slots quantity is dynamically set according to the computing capability difference of each node in Hadoop clusters.The CPU and internal memory possessed from node
Situation sets map quantity and reduce quantity, and the technical problem is the connection by CPU quantity in node and slots reasonable quantities
System gets up；Pass through the restriction constraint slots of internal memory quantity so that the disposal ability of cluster interior joint can be met so that task
It is more efficient.
In node in the contacting of CPU quantity and slots reasonable quantities, the CPU quantity of each node is counted, by slots numbers
Amount is arranged to the core quantity of CPU in node, because each core can individually handle a Task, and without waiting in map
Task or reduce Task can be performed quickly when performing.
In internal memory limits constraint, it can be determined to apply for the size of internal memory according to slots quantity, further according in node
Deposit situation and adjust slots quantity accordingly, when the low memory in application process if can reduce slots quantity, Zhi Daoda
The requirement limited to internal memory, on the contrary it is the slots quantity according to CPU quantity sets by slots quantity sets.
With reference to the accompanying drawings 1 and accompanying drawing 2, present disclosure is described in detail with an instantiation.
Distributed type assemblies environment is disposed first, and using the Hadoop group of planes with 11 nodes, one of node is made
For master, remaining ten use Xeon E5-2620@2.00GHz CPU as wherein 10 nodes of slave.,
Core quantity is 24,96GB internal memories, and 12*2T hard disks, operating system is centos6.3, and the configuration of another node is
Xeon E7- 8837@2.67GHz CPU, core quantity are 128,500GB internal memories, 5*2T hard disks, and operating system is
centos6.3.It is according to official document's installation hadoop components on centos6.3 in operating system.Then by hdfs,
Mapreduce services are opened.
Operation job execution flow charts are as shown in Figure 1, it is first determined MapReduce input file or catalogue should be
Exist on File system, if MapReduce depends on HDFS, first must upload to local file on HDFS.
Client can apply for that a Jobid is used as job identifier to JobTracker.Then MapReduce is accomplished by holding job
The necessary resource file of row is copied on HDFS.Next it is only operation job and submits process, data fragmentation is done to input file
(input split).Data fragmentation is and the burst in order to determine that the scope of its processing data before mapper is performed
Quantity determines map task quantity, one-to-one corresponding between them.This data fragmentation (split) is logic burst, record
It should access which block, and starting index and the information of data length on this block.Then initially it is turned into
Industry, JobTracker will be responsible for distributed tasks to TaskTracker, TaskTracker operationally can periodically to
JobTracker sends heartbeat request, report TaskTracker status data, TaskTracker upper task execution states and
Wish to obtain the task that can perform from JobTracker.And the map quantity run in real TaskTracker nodes and
Reduce quantity is determined by map slots and reduce slots quantity.Therefore, according to the calculating of respective nodes in cluster
Ability determines that map slots and reduce slots quantity is critically important in each node, directly affects the operation effect of task
Rate.
The flow chart of setting slots quantity obtains the CPU of each node in cluster core as shown in Figure 2, first
Quantity, the core quantity that map slot quantity initial setting is CPU in node, reduce slot quantity initial setting is node
The half of middle CPU core quantity；Then the free memory size in each node is obtained, in class method
In initializeMemoryManagement (), determined to apply for the size of internal memory according to slots quantity, task's
Total Memory Allocation size is equal to map slot quantity and the product of single map slot memory sizes in TaskTracker is added
Reduce slot quantity and the product of single reduce slot memory sizes in TaskTracker.If task total internal memory point
With small compared with the free memory of respective nodes in cluster, then map slots are set as to the core quantity of CPU in node,
Reduce slot quantity is the half of map slot quantity；Else if task total Memory Allocation and respective nodes in cluster
Free memory it is small, then reduce map slot quantity or reduce slot quantity, alternate less slots quantity, until meeting
In node untill memory conditions, at this moment map slots are set as meeting the map slots quantity of condition, reduce slot quantity
To meet the reduce slots quantity of condition.Then according to according in class method TaskTracker.initialize ()
Two TaskLauncher threads, are each responsible for starting Mapper and Reduce tasks, are needed in TaskLauncher
Incoming corresponding slots quantity is wanted, corresponding Task, such as map task or reduce task is then performed.After execution terminates,
The occupied resource of release.This method determines the computing capability of node with CPU core quantity and memory size, for
The node that CPU core quantity is more in node and internal memory is big sets larger map and reduce quantity, in some nodes
CPU core quantity is few and the relatively small number of node of internal memory sets less map and reduce quantity.In the cluster, use
Xeon E5-2620@2.00GHz CPU, core quantity are that map is set to by 10 nodes of 24,96GB internal memories
24, reduce are set to 12.The configuration of another node is that Xeon E7- 8837@2.67GHz CPU, core quantity are
128,500GB internal memories, it is 64 that map is set into 128, reduce.It is such that the map numbers set than each machine node are set
The tasks carrying efficiency high of amount and reduce quantity, while reaching the reasonable utilization of optimization resource.
Embodiments of the invention are the foregoing is only, within the spirit and principles of the invention, that is made is any
Modification, equivalent substitution, improvement etc., should be included in the scope of the protection.
Claims (2)
1. a kind of method of the dynamic setting slots quantity of MapReduce, it is characterised in that its specifically set process as：
The quantity of CPU in clustered node is determined first, then according to the quantity of CPU core in each node by master slave mode frame
Dynamically setting determines slots quantity to frame MapReduce：According to the resource situation of job queues and TaskTracker nodes as defeated
Enter, wherein TaskTracker resource situation includes CPU core amounts and the memory size of node, then further according to node
Computing capability setting slots quantity；
JobTracker is run on MS master-slave pattern framework MapReduce host node, it is responsible for monitoring a group of planes, task scheduling；
TaskTracker is run from node, it is responsible for monitor task execution, report on progress；
Determine whether MapReduce input file or catalogue exist on file system File system, work as MapReduce
During dependent on HDFS, first local file is uploaded on HDFS：Client applies for that a Jobid is used as to JobTracker
Job identifier；Then MapReduce copies to job execution resource files on HDFS；Then operation job submissions are carried out again
Process, data fragmentation is done to input file；
Initialization procedure, JobTracker is responsible for distributed tasks to TaskTracker, TaskTracker periodically to
JobTracker sends the resource service condition that this node is carried in heartbeat message, the information, i.e. TaskTracker state
The upper task of data, TaskTracker performs the task that state and hope obtain performing from JobTracker；
When heartbeat is reached, the scheduling in host node occurs, if TaskTracker reports oneself available free resource,
JobTracker selects a task to be transmitted into node operation using dispatching algorithm.
2. the method that a kind of MapReduce according to claim 1 dynamically sets slots quantity, it is characterised in that：Setting
Determine to need to design two variables during slots quantity, one is map slot, and one is reduce slot：Change first
Code in TaskTracker, the core amounts for being CPU in node by map slot quantity initial setting, reduce slot numbers
Measure the half for the core amounts that initial setting is CPU in node；Then in class method, Shen is determined according to slots quantity
Please internal memory size, task total Memory Allocation size is equal to map slot quantity and single map slot in TaskTracker
The product of memory size is along with resuce slot quantity and the product of single reduce slot memory sizes in TaskTracker；
If task total Memory Allocation is small compared with the free memory of respective nodes in cluster, slots is set as the value；If
The free memory of task total Memory Allocation and respective nodes in cluster is small, then reduces map slot quantity or reduce slot
Quantity, alternate less slots quantity, until meeting in node untill memory conditions；At this moment map slot are set as meeting bar
The map slot quantity of part, reduce slot quantity is the reduce slot quantity for the condition that meets；Then by class method
Two TaskLauncher threads, are each responsible for starting Mapper and Reduce tasks, are passed in TaskLauncher
Enter corresponding slots quantity, then perform corresponding Task, after execution terminates, discharge occupied resource.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410004521.4A CN103761146B (en) | 2014-01-06 | 2014-01-06 | A kind of method that MapReduce dynamically sets slots quantity |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410004521.4A CN103761146B (en) | 2014-01-06 | 2014-01-06 | A kind of method that MapReduce dynamically sets slots quantity |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN103761146A CN103761146A (en) | 2014-04-30 |
| CN103761146B true CN103761146B (en) | 2017-10-31 |
Family
ID=50528389
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201410004521.4A Active CN103761146B (en) | 2014-01-06 | 2014-01-06 | A kind of method that MapReduce dynamically sets slots quantity |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN103761146B (en) |
Families Citing this family (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104270412A (en) * | 2014-06-24 | 2015-01-07 | 南京邮电大学 | Three-level caching method based on Hadoop distributed file system |
| CN105607955A (en) * | 2015-12-23 | 2016-05-25 | 浪潮集团有限公司 | Calculation task distribution method and apparatus |
| CN105868070A (en) * | 2015-12-25 | 2016-08-17 | 乐视网信息技术（北京）股份有限公司 | Method and apparatus for determining resources consumed by tasks |
| CN105868025B (en) * | 2016-03-30 | 2019-05-10 | 华中科技大学 | A kind of system solving memory source keen competition in big data processing system |
| CN107203422B (en) * | 2016-08-28 | 2020-09-01 | 深圳晶泰科技有限公司 | Job scheduling method for high-performance computing cloud platform |
| CN108268316A (en) * | 2016-12-30 | 2018-07-10 | 北京国双科技有限公司 | The method and device of job scheduling |
| CN107766138A (en) * | 2017-10-20 | 2018-03-06 | 北京集奥聚合科技有限公司 | A kind of hadoop Mission Monitors method and system |
| CN108170530B (en) * | 2017-12-26 | 2021-08-17 | 北京工业大学 | Hadoop load balancing task scheduling method based on mixed element heuristic algorithm |
| CN110618865B (en) * | 2019-09-20 | 2022-07-05 | 中国银行股份有限公司 | Hadoop task scheduling method and device |
| CN111914007B (en) * | 2020-06-15 | 2024-02-02 | 武汉达梦数据库股份有限公司 | Method and device for hadoop cluster to run ETL flow |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102073546A (en) * | 2010-12-13 | 2011-05-25 | 北京航空航天大学 | Task-dynamic dispatching method under distributed computation mode in cloud computing environment |
| CN102541645A (en) * | 2012-01-04 | 2012-07-04 | 北京航空航天大学 | Dynamic adjustment method for node task slot based on node state feedbacks |
| CN102609303A (en) * | 2012-01-18 | 2012-07-25 | 华为技术有限公司 | Slow-task dispatching method and slow-task dispatching device of Map Reduce system |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20120304186A1 (en) * | 2011-05-26 | 2012-11-29 | International Business Machines Corporation | Scheduling Mapreduce Jobs in the Presence of Priority Classes |
| US9128763B2 (en) * | 2011-08-23 | 2015-09-08 | Infosys Limited | System and method for job scheduling optimization |
-
2014
- 2014-01-06 CN CN201410004521.4A patent/CN103761146B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102073546A (en) * | 2010-12-13 | 2011-05-25 | 北京航空航天大学 | Task-dynamic dispatching method under distributed computation mode in cloud computing environment |
| CN102541645A (en) * | 2012-01-04 | 2012-07-04 | 北京航空航天大学 | Dynamic adjustment method for node task slot based on node state feedbacks |
| CN102609303A (en) * | 2012-01-18 | 2012-07-25 | 华为技术有限公司 | Slow-task dispatching method and slow-task dispatching device of Map Reduce system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN103761146A (en) | 2014-04-30 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN103761146B (en) | A kind of method that MapReduce dynamically sets slots quantity | |
| CN104915407B (en) | A kind of resource regulating method based under Hadoop multi-job environment | |
| US20190324819A1 (en) | Distributed-system task assignment method and apparatus | |
| CN107832153B (en) | Hadoop cluster resource self-adaptive allocation method | |
| CN112162865B (en) | Scheduling method and device of server and server | |
| CN105934928B (en) | The dispatching method of user's request, device and system in distributed resource system | |
| US20130198755A1 (en) | Apparatus and method for managing resources in cluster computing environment | |
| CN109564528B (en) | System and method for computing resource allocation in distributed computing | |
| WO2019091387A1 (en) | Method and system for provisioning resources in cloud computing | |
| US20200174844A1 (en) | System and method for resource partitioning in distributed computing | |
| CN110597639B (en) | CPU distribution control method, device, server and storage medium | |
| US11496413B2 (en) | Allocating cloud computing resources in a cloud computing environment based on user predictability | |
| CN114138486A (en) | Containerized micro-service arranging method, system and medium for cloud edge heterogeneous environment | |
| US9535749B2 (en) | Methods for managing work load bursts and devices thereof | |
| CN111459641B (en) | Method and device for task scheduling and task processing across machine room | |
| WO2016061935A1 (en) | Resource scheduling method, device and computer storage medium | |
| CN106528288A (en) | Resource management method, device and system | |
| WO2016101799A1 (en) | Service allocation method and device based on distributed system | |
| CN104112049A (en) | P2P (peer-to-peer) architecture based cross-data-center MapReduce task scheduling system and P2P architecture based cross-data-center MapReduce task scheduling method | |
| Choi et al. | pHPA: A proactive autoscaling framework for microservice chain | |
| US20220300323A1 (en) | Job Scheduling Method and Job Scheduling Apparatus | |
| Li et al. | Endpoint-flexible coflow scheduling across geo-distributed datacenters | |
| CN113608869A (en) | Task scheduling method and device, electronic equipment and computer storage medium | |
| CN108958942A (en) | A kind of distributed system distribution multitask method, scheduler and computer equipment | |
| CN109729113A (en) | Manage method, server system and the computer program product of dedicated processes resource |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |