CN107832153A - A kind of Hadoop cluster resources self-adapting distribution method - Google Patents
A kind of Hadoop cluster resources self-adapting distribution method Download PDFInfo
- Publication number
- CN107832153A CN107832153A CN201711120624.7A CN201711120624A CN107832153A CN 107832153 A CN107832153 A CN 107832153A CN 201711120624 A CN201711120624 A CN 201711120624A CN 107832153 A CN107832153 A CN 107832153A
- Authority
- CN
- China
- Prior art keywords
- mrow
- msub
- node
- mfrac
- weights
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5083—Techniques for rebalancing the load in a distributed system
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1001—Protocols in which an application is distributed across nodes in the network for accessing one among a plurality of replicated servers
Abstract
The present invention provides a kind of Hadoop cluster resources self-adapting distribution method, and cluster operation can be made more efficient.Methods described includes：According to default homework type classifying rules, the type for the operation that user submits is determined, wherein, each operation can split into N number of task and realize Distributed Parallel Computing；If the type for the operation that user submits is the operation of CPU types, the operation of I/O types or important operation, the type for the operation then submitted according to user, it is determined that from the weights of node than row parameter, wherein, from the weights of node i than row parameter be equal to from the weights and cluster of node i all from the weights of node and ratio, be used to weigh the performance from node i from the weights of node i；It is each from the task requests of node distribution corresponding proportion according to each weights from node than row parameter.The present invention relates to big data and field of cloud calculation.
Description
Technical field
The present invention relates to big data and field of cloud calculation, particularly relates to a kind of Hadoop cluster resources self-adjusted block side
Method.
Background technology
With the extensive use of the prevalence of large-scale parallel distributed processing system(DPS), particularly group system, which kind of is taken
Efficient scheduling strategy balances the load of each node, and then improves the utilization rate of whole system resource, and oneself turns into grinding for people
Study carefully emphasis and focus.
In recent years, novel, tall and erect and effective load-balancing algorithm is as one of study hotspot of domestic and international research institution.
Wherein distributed heterogeneous cluster generally existing problem of load balancing, and Hadoop platform does not possess the energy of detection node performance in itself
Power, although the resource management system YARN of Hadoop clusters has the scheduling strategy for load imbalance, excessively simply simultaneously
It is not suitable for isomeric group complicated in reality, thus problem of load balancing is in the Hadoop platform built based on isomeric group
It is more prominent.
In the prior art, the Hadoop cluster task self-adapting dispatching methods based on node capacity of use, are not accounted for
Influence of the different work type to scheduling of resource, resource division is caused not enough to become more meticulous.
The content of the invention
It is existing to solve the technical problem to be solved in the present invention is to provide a kind of Hadoop cluster resources self-adapting distribution method
There is the problem of division of the resource present in technology not enough becomes more meticulous.
In order to solve the above technical problems, the embodiment of the present invention provides a kind of Hadoop cluster resources self-adapting distribution method,
Including：
According to default homework type classifying rules, the type for the operation that user submits is determined, wherein, each operation can
Split into N number of task and realize Distributed Parallel Computing；
If the type for the operation that user submits is the operation of CPU types, the operation of I/O types or important operation, submitted according to user
Operation type, it is determined that from the weights of node than row parameter, wherein, be equal to from the weights of node i than row parameter from node i
In weights and cluster all from the weights of node and ratio, be used to weigh the performance from node i from the weights of node i；
It is each from the task requests of node distribution corresponding proportion according to each weights from node than row parameter.
Further, described according to default homework type classifying rules, determining the type for the operation that user submits includes：
By way of label, judge that the operation that user submits is important operation or general job, wherein, the label bag
Include：It is important, general；
If the operation is general job, judge whether the size of the operation is less than default size threshold value, if small
In it is small operation then to judge the operation；
Otherwise, judge whether exceed default discrepancy threshold from the cpu resource and I/O resource differentializations of node in cluster,
If exceeding, the resource accounting that is consumed when executed according to operation, it is that the operation of CPU types or I/O types are made to judge the operation
Industry；
Otherwise, then according to the different loads degree in operation Map stages and Reduce stages, it is Map types to judge the operation
Operation or Reduce type operations.
Further, it is described to judge that the operation is that the operation of CPU types or I/O type operations include：
If the operation meets the first formula, I/O type operations are marked as, wherein, first formula is expressed as：
If the operation meets the second formula, CPU type operations are marked as, wherein, second formula is expressed as：
Wherein, n represents just to represent map end output data quantities and map ends in the quantity of executing tasks parallelly, ρ from node
The ratio of input data amount, MID represent map ends input data amount, and DIOR represents disk I/O transmission rate, and MTCT represents that Map appoints
The required time is completed in business.
Further, it is described to judge that the operation is the operation of Map types or Reduce type operations：
When the operation meets three formula, then judge that the operation for Reduce type operations, otherwise judges the work
Industry is Map type operations, wherein, the 3rd formula is expressed as：
Wherein, SmapThe data total amount that the expression Map stages input, SreduceThe data total amount that the expression Reduce stages input,
Td represents default proportion threshold value.
Further, it is described to be expressed as from the weights of node：
Wi=AY+BD+CF
Wherein, WiThe weights from node i are represented, Y represents the hardware performance from node i, and D represents the maneuverability from node i
Can, F represents node failure rate, and A, B, C are respectively Y, D, F coefficient.
Further, it is described to be expressed as from the hardware performance Y of node i：
Wherein, ScpuRepresent CPU frequency, SmemRepresent memory size, SnetRepresent network bandwidth, SdiskRepresent that disk is maximum
Read or write speed, avgcpu、avgmem、avgnet、avgdiskAverage CPU frequency, memory size, the Netowrk tape of cluster are represented respectively
Wide, disk maximum read or write speed, K1、K2、K3、K4All represent coefficient.
Further, it is expressed as from the runnability D of node i：
Wherein, tcmThe CPU types operation of unit-sized is expressed as in the run time in Map stages, tcrIt is expressed as unit-sized
The operation of CPU types in the run time in Reduce stages, tiomIt is expressed as operation of the I/O types operation in the Map stages of unit-sized
Time, tiorThe I/O types operation of unit-sized is expressed as in the run time in Reduce stages, avgcmIt is expressed as unit-sized
The operation of CPU types is in the average run time of Map stage clusters, avgcrThe CPU type operations of unit-sized are expressed as in Reduce
The average run time of stage cluster, avgiomThe I/O types operation of unit-sized is expressed as in the average of Map stage clusters
Run time, avgiorThe I/O types operation of unit-sized is expressed as in the average run time of Reduce stage clusters, G1、
G2、G3、G4All represent coefficient.
Further, the node failure rate F is expressed as：
Wherein, nnumRepresent that journal file reads the operation task number of each node, nfailRepresent time of each node operation failure
Number, tnumRepresent the average operation task number of whole cluster, tfailRepresent the number of tasks of average operation failure.
Further, if the operation is CPU type operations, COEFFICIENT K of the increase description from node cpu performance1、K2And G1、
G3Value；
If the operation is I/O type operations, COEFFICIENT K of the increase description from node I/O performances3、K4And G2、G4Value；
If the operation is important operation, increase node failure rate F coefficient C value.
Further, if the operation is Map type operations, priority scheduling is counted to data storage is more from node
Calculate；
If the operation is Reduce type operations, carried out more than priority scheduling to Map task output data quantities from node
Calculate.
The above-mentioned technical proposal of the present invention has the beneficial effect that：
In such scheme, according to default homework type classifying rules, the type for the operation that user submits is determined, wherein,
Each operation can split into N number of task and realize Distributed Parallel Computing, and N is positive integer；If the type for the operation that user submits
For the operation of CPU types, the operation of I/O types or important operation, then the type for the operation submitted according to user, it is determined that the weights ratio from node
Row parameter, wherein, from the weights of node i than row parameter be equal to from the weights and cluster of node i all from the weights of node and
Ratio, be used to weigh the performance from node i from the weights of node i；It is each according to each weights from node than row parameter
From the task requests of node distribution corresponding proportion, make each task requests for receiving corresponding proportion parameter from node.So, it is comprehensive
Joint behavior difference and influence of the operation different type to the job scheduling of whole cluster are considered, considers to save relative to original
The resource regulating method of point capacity variance, realizes the scheduling more to be become more meticulous to cluster resource, so that cluster operation is more
Efficiently.
Brief description of the drawings
Fig. 1 is the schematic flow sheet of Hadoop cluster resources self-adapting distribution method provided in an embodiment of the present invention；
Fig. 2 is the schematic flow sheet of the type for the operation that determination user provided in an embodiment of the present invention submits；
Fig. 3 is isomery Hadoop cluster schematic diagrames provided in an embodiment of the present invention；
Fig. 4 is the specific workflow figure of YARN after submission operation provided in an embodiment of the present invention；
Fig. 5 is π estimation program running situation schematic diagrames before improvement provided in an embodiment of the present invention；
Fig. 6 is π estimation program running situation schematic diagrames after improvement provided in an embodiment of the present invention；
Fig. 7 is WordCount program running situation schematic diagrames before improvement provided in an embodiment of the present invention；
Fig. 8 is WordCount program running situation schematic diagrames after improvement provided in an embodiment of the present invention；
Fig. 9 shows for π estimation programs before and after improvement provided in an embodiment of the present invention and the contrast of WordCount program runtimes
It is intended to.
Embodiment
To make the technical problem to be solved in the present invention, technical scheme and advantage clearer, below in conjunction with accompanying drawing and tool
Body embodiment is described in detail.
The problem of present invention not enough becomes more meticulous for the division of existing resource, there is provided a kind of Hadoop cluster resources are adaptive
Distribution method.
In order to more fully understand the Hadoop cluster resource self-adapting distribution methods described in the present embodiment, first to from node,
Host node, Map tasks, Reduce tasks are briefly described：
1st, from equivalent to one calculating point of node, there is network connection, can be sent out with independent process host node and explorer
The task of cloth.One server can arrange one from node, can also arrange multiple from node；
2nd, host node is responsible for job class, resource and task scheduling, and the execution of task is carried out from node；
3rd, each operation can split into N number of task and realize Distributed Parallel Computing, and N number of task can be at one from node
Complete, can also it is multiple it is different from node complete, N be more than or equal to 2；
4th, Map Charge-de-Mission is in the larger task of Map stage amounts of calculation；
5th, Reduce Charge-de-Mission is in the larger task of Reduce stage amounts of calculation.
As shown in figure 1, Hadoop cluster resources self-adapting distribution method provided in an embodiment of the present invention, including：
S101, according to default homework type classifying rules, the type for the operation that user submits is determined, wherein, Mei Gezuo
Industry can split into N number of task and realize Distributed Parallel Computing, and N is positive integer；
S102, if the type for the operation that user submits is the operation of CPU types, the operation of I/O types or important operation, according to user
The type of the operation of submission, it is determined that from the weights of node than row parameter, wherein, it is equal to from the weights of node i than row parameter from section
In point i weights and cluster all from the weights of node and ratio, be used to weigh the performance from node i from the weights of node i；
S103, it is each from the task requests of node distribution corresponding proportion according to each weights from node than row parameter.
Hadoop cluster resource self-adapting distribution methods described in the embodiment of the present invention, classify according to default homework type
Rule, the type for the operation that user submits is determined, wherein, each operation can split into N number of task and realize distributed parallel meter
Calculate, N is positive integer；If the type for the operation that user submits is the operation of CPU types, the operation of I/O types or important operation, according to user
The type of the operation of submission, it is determined that from the weights of node than row parameter, wherein, it is equal to from the weights of node i than row parameter from section
In point i weights and cluster all from the weights of node and ratio, be used to weigh the performance from node i from the weights of node i；
According to each weights from node than row parameter, it is each from the task requests of node distribution corresponding proportion, makes each from node
Receive the task requests of corresponding proportion parameter.So, joint behavior difference has been considered with operation different type to whole collection
The influence of the job scheduling of group, relative to the resource regulating method of original consideration node capacity difference, realize and cluster is provided
The scheduling that source more becomes more meticulous, so that cluster operation is more efficient.
In the embodiment of foregoing Hadoop cluster resources self-adapting distribution method, further, it is described according to
Default homework type classifying rules, determining the type for the operation that user submits includes：
By way of label, judge that the operation that user submits is important operation or general job, wherein, the label bag
Include：It is important, general；
If the operation is general job, judge whether the size of the operation is less than default size threshold value, if small
In it is small operation then to judge the operation；
Otherwise, judge whether exceed default discrepancy threshold from the cpu resource and I/O resource differentializations of node in cluster,
If exceeding, the resource accounting that is consumed when executed according to operation, it is that the operation of CPU types or I/O types are made to judge the operation
Industry；
Otherwise, then according to the different loads degree in operation Map stages and Reduce stages, it is Map types to judge the operation
Operation or Reduce type operations.
In the present embodiment, as shown in Fig. 2 the work that user submits can be determined according to default homework type classifying rules
The type of industry, specific steps can include：
The differentiation of A11, important operation and general aspects operation, this mode classification, which allows in practice, some operations
It is higher to the reliability requirement of cluster, thus to it is each from the reliability of node quantify after can be by important operation point
It is fitted on that reliability is higher to be calculated from node；Sorting technique is：By way of label, judge that the operation that user submits is
Important operation or general job, wherein, the label includes：It is important, general.
A12, if the operation is general job, the operation for continuing to submit user according to job size is classified.
By job size classify necessary, because if not making a distinction so but size operation is blended in a queue,
Then it is likely used only to situation about occurring is to obtain resource first after one big operation is submitted to be performed, and if had this when
Small operation is submitted, because can only be waited after above big operation is performed in same queue could apply arriving resource, this
The time that the small operation of sample waits can become very long, cause the Job execution efficiency of whole cluster and resource utilization low；Classification
Method is specially：If the operation is general job, judge whether the size of the operation is less than default size threshold value, if
It is less than, then it is small operation to judge the operation, otherwise, it is determined that the operation is big operation.
A13, if the operation is big operation, judge in cluster from the cpu resource and I/O resource differentializations of node whether
More than default discrepancy threshold, if exceeding, the resource accounting that is consumed when executed according to operation judges that the operation is
The operation of CPU types or I/O type operations；Wherein, the operation of CPU types is to need a large amount of even excess load integers and floating-point at full capacity of progress to transport
Calculate, mainly operated in internal memory, for example various scientific algorithms, large-scale data modeling etc., I/O type operations are to need
Hard disk or the read-write of other storage mediums are frequently carried out, for example various data centers, network storage and cloud deposit server.So divide
Class helps to realize the scheduling that becomes more meticulous of operation, and more efficient and rational utilizes cluster resource.
A14, if being no more than default discrepancy threshold from the cpu resource and I/O resource differentializations of node in cluster, according to
Operation Map stages and the different loads degree in Reduce stages, it is the operation of Map types or Reduce type operations to judge the operation.
According to step A11-A14, the classification work of the operation of user's submission can be completed, hereafter, can be according to homework type
In Fair Scheduler, establish corresponding queue respectively, that is, establish the operation of CPU types, the operation of I/O types, the operation of Map types,
The queues such as the operation of Reduce types, important operation, small operation, it is easy to by corresponding queue realize different operations targetedly
Scheduling.The configuration file of Fair Scheduler device is located in the fair-scheduler.xml files under class.path, and this path can
To be modified by yarn.scheduler.fair.allocation.file attributes.It can be configured in configuration file every
One queue.Each inner queue can still have different scheduling strategies, for example, first in first out scheduling strategy, weights training in rotation are adjusted
Spend strategy etc..Targetedly resource dispatching strategy is taken further according to different homework types, realizes the tune more to be become more meticulous to resource
Degree, improve the problem of load balancing of isomeric group (for example, Hadoop clusters), improve the resource utilization of whole isomeric group,
So as to lift the combination property of isomeric group.
In the embodiment of foregoing Hadoop cluster resources self-adapting distribution method, further, the judgement
The operation is that the operation of CPU types or I/O type operations include：
If the operation meets the first formula, I/O type operations are marked as, wherein, first formula is expressed as：
If the operation meets the second formula, CPU type operations are marked as, wherein, second formula is expressed as：
Wherein, n represents just to represent map end output data quantities and map ends in the quantity of executing tasks parallelly, ρ from node
The ratio of input data amount, MID represent map ends input data amount, and DIOR represents disk I/O transmission rate, and MTCT represents that Map appoints
The required time is completed in business.
In the present embodiment, ρ × MID=MOD, wherein, MOD represents map ends output data quantity.
In the embodiment of foregoing Hadoop cluster resources self-adapting distribution method, further, the judgement
The operation is the operation of Map types or Reduce type operations：
When the operation meets three formula, then judge that the operation for Reduce type operations, otherwise judges the work
Industry is Map type operations, wherein, the 3rd formula is expressed as：
Wherein, SmapThe data total amount that the expression Map stages input, SreduceThe data total amount that the expression Reduce stages input,
Td represents default proportion threshold value.
In the present embodiment,Wherein, K represents the number of server in cluster.
In the present embodiment, in Hadoop clusters, with weighing the different performances from node from node weights, specifically,
Weighed in terms of node hardware performance Y, runnability D, node failure rate F tri-.It can be expressed as from the weights of node：
Wi=AY+BD+CF
Wherein, WiThe weights from node i are represented, Y represents the hardware performance from node i, and D represents the maneuverability from node i
Can, F represents node failure rate, and A, B, C are respectively Y, D, F coefficient.
In the present embodiment, the CPU frequency from node, memory size, network bandwidth, most are mainly considered from node hardware performance
Big disk read-write speed etc., what these hardware parameters represented is the resource situation having in itself from node, is to weigh joint behavior
Basic index.It is described to be expressed as from node hardware performance：
Wherein, Y represents the hardware performance from node i, ScpuRepresent CPU frequency, SmemRepresent memory size, SnetRepresent net
Network bandwidth, SdiskRepresent disk maximum read or write speed, avgcpu、avgmem、avgnet、avgdiskThe average of cluster is represented respectively
CPU frequency, memory size, network bandwidth, disk maximum read or write speed, K1、K2、K3、K4All represent coefficient.
In the present embodiment, the hardware performance index of cluster is Static State Index, therefore directly can obtain hardware from node
The parameters of performance indications.
In the present embodiment, because only relying part point hardware performance index can not completely represent a performance from node,
Thus need to introduce running performance index when running more accurately to describe the dynamic property from node.Runnability is dynamic
Performance thus needs to obtain by caused daily record after reading operation operation.
In the present embodiment, it is expressed as from the runnability D of node i：
Wherein, tcmThe CPU types operation of unit-sized is expressed as in the run time in Map stages, tcrIt is expressed as unit-sized
The operation of CPU types in the run time in Reduce stages, tiomIt is expressed as operation of the I/O types operation in the Map stages of unit-sized
Time, tiorThe I/O types operation of unit-sized is expressed as in the run time in Reduce stages, avgcmIt is expressed as unit-sized
The operation of CPU types is in the average run time of Map stage clusters, avgcrThe CPU type operations of unit-sized are expressed as in Reduce
The average run time of stage cluster, avgiomThe I/O types operation of unit-sized is expressed as in the average of Map stage clusters
Run time, avgiorThe I/O types operation of unit-sized is expressed as in the average run time of Reduce stage clusters, G1、
G2、G3、G4All represent coefficient.
In the present embodiment, using two kinds of operation of CPU types, I/O types operation homework types in the run time conduct from node
Weigh the description indexes from node actual motion performance.
In the present embodiment, respectively transporting respectively using the operation of CPU types and the operation of I/O types for respectively taking 1GB data volumes and from node
OK, now, tcm1GB CPU types operation is expressed as in the run time in Map stages, tcrThe CPU type operations for being expressed as 1GB exist
The run time in Reduce stages, tiom1GB I/O types operation is expressed as in the run time in Map stages, tiorIt is expressed as 1GB
The operation of I/O types in the run time in Reduce stages, avgcmIt is expressed as 1GB CPU types operation being averaged in Map stage clusters
Run time, avgcr1GB CPU types operation is expressed as in the average run time of Reduce stage clusters, avgiomRepresent
For 1GB the operation of I/O types in the average run time of Map stage clusters, avgiorThe I/O type operations for being expressed as 1GB exist
The average run time of Reduce stage clusters.When collecting this index in order to avoid jitter as far as possible, it is ensured that
The credibility of data, 10 groups are gathered respectively per item data, is averaged after removing max min as index.
In the present embodiment, this performance indications of node failure rate are to weigh the ginseng chosen from the reliability of node
Number, it is contemplated that some operations in actual motion can be to having higher requirements from the reliability of node, thus this index is for whole
Have great importance in the individual description from joint behavior.
In the present embodiment, this performance indications of calculate node crash rate need to read the operation of each node by journal file
Number of tasks nnum, and wherein run the frequency n of failurefail, and calculate the average operation task number t of whole clusternum, and it is flat
Run the number of tasks t of failurefail, node failure rate F is represented by：
In the present embodiment, each coefficient in node weights calculation formula determines according to different work type, specifically：If
The operation is CPU type operations, then COEFFICIENT K of the increase description from node cpu performance1、K2And G1、G3Value；If the operation is
I/O type operations, then increase describe the COEFFICIENT K from node I/O performances3、K4And G2、G4Value；If the operation is important operation,
Then increase node failure rate F coefficient C value.So, joint behavior difference has been considered with operation different type to whole
The influence of the job scheduling of individual cluster, according to the different type of operation accordingly adjust weigh joint behavior formula in be
Number, relative to the dispatching method of original consideration joint behavior difference, the scheduling more to be become more meticulous to cluster resource is realized, from
And make cluster operation more efficient.
In the present embodiment, it is contemplated that consumption of the small operation to cluster resource is less, and run time is shorter, general individually to establish
One queue is used for running small operation.If the operation is Map type operations, priority scheduling is more from node to data storage
Calculated；If the operation is Reduce type operations, carried out more than priority scheduling to Map task output data quantities from node
Calculate so that scheduling of resource more becomes more meticulous.
In the present embodiment, in order to carry out the dispatching distribution of cluster resource using node weights, it is necessary to which node weights are changed
Likeness in form into weights than row parameter, in the present embodiment, node weights can be normalized, be limited to 0~1 scope
It is interior.Weights definition more such than row parameter, it is W from the weights of node i if having m in cluster from nodei, all from section in cluster
Point weights and be Wsum, then can be expressed as than row parameter P from the weights of node i：
In the present embodiment, if the type for the operation that user submits is the operation of CPU types, the operation of I/O types or important operation, adopt
It is scheduled with weights polling dispatching strategy, specifically：It is each corresponding from node distribution in cluster according to the performance from node
Weights than row parameter, so, host node can be each from node distribution phase according to each weights from node than row parameter
The task requests of ratio are answered, enable each task requests for receiving corresponding proportion parameter from node.
In the present embodiment, it is assumed that have one group is than row parameter from node S={ S0, S1 ..., Sn-1 }, the weights of initialization
0, according to the performance from node, for each from the corresponding weights of node distribution than row parameter, when dispatching first time, weighting
It is worth that maximum than row parameter, from node, by weights constantly successively decreasing than row parameter, to find suitable perform from node and appoint
Business, until end of polling(EOP), weights are returned as 0 than row parameter.If there are 2 times that 2 disposal abilities from node A and B, A are B,
Then A weights are 2 times of B than row parameter/weights, and the task requests number that A receives is also 2 times of B.That is, weights ratio
Row parameter/weights are high first to receive task requests from node, and weights are higher than row parameter/weights to join from node than weights than row
The more task requests of the low server process of number/weights, identical weights are than row parameter/weights from node processing same number
Task requests.
, can be from two angles by server as shown in figure 3, Fig. 3 is isomery Hadoop cluster schematic diagrames in the present embodiment
It is divided into host node (NameNode) and from two kinds of roles of node (DataNode)；Specifically：
First, from distributed file system (HDFS) angle, server is divided into host node and from node, is being distributed
In formula file system, the management of catalogue is critically important, and host node is exactly directory management person；
NameNode is host node, and metadata such as filename, document directory structure, the file attribute of storage file (generate
Time, number of copies, file permission), and DataNode where the block list of each file and block etc..Host node is one
Individual central server, the access of the name space (namespace) and client of file system to file is responsible for, safeguarded
File all in each file system tree and whole tree and catalogue, two document forms are permanently stored in this to these information
On local disk：Name control image file (Fsimage) and editor's daily record (Edit log).
DataNode in local file system storage file block number evidence, and block number evidence verification and.It can create, delete
Remove, mobile or Rename file, file content can not be changed after document creation, write-in and closing.One data block exists
DataNode is stored on disk with file, including two files, one be data in itself, one is that metadata includes data block
Length, the verification of block number evidence and, and timestamp.DataNode starts backward NameNode registrations, by rear, periodically (1
Hour) all block messages are reported to NameNode.Heartbeat be every 3 seconds once, heartbeat returning result with NameNode to should
DataNode order such as copy block data are to another machine, or delete some data block.If it exceeds 10 minutes do not receive
To some DataNode heartbeat, then it is assumed that the node is unavailable.
File operation, NameNode are responsible for the operation of file metadata, and DataNode is responsible for handling the read-write of file content
Request, with the related data flow of file content without NameNode, it can only inquire that it is contacted with that DataNode, otherwise
NameNode can turn into the bottleneck of system
Second, from the point of view of YARN angles, host node typically arranges explorer (ResourceManager), and the overall situation is born
Blame the monitoring, distribution and management of all resources, and arranged from node node manager (NodeManager) be responsible for each from
The maintenance of node.
Fig. 4 is YARN workflow schematic diagram, can be realized than row parameter to making according to obtained each weights from node
Weights polling dispatching during industry application resource, idiographic flow are as follows：
1) user submits application program into resource pipe platform YARN, including application manager
ApplicationMaster programs, the order for starting ApplicationMaster, user program etc..For the work often run
Industry can add label to it and directly be dispatched to respective queue, and its partial task can be run in advance for type pending job, collection
Relevant information is divided according to homework type sorting technique.
2) ResourceManager (explorer) is first Container (computing resource of the application assigned
Unit), and communicated with corresponding NodeManager (node manager), it is desirable to it is in this Container (computing resource list
Position) in start application program ApplicationMaster.
3) ApplicationMaster registers to ResourceManager first, and such user's can directly passes through
ResourceManager checks the running status of application program, and then it will be each task application resource, and monitor its fortune
Row state, until end of run, i.e. repeat step 4~7.
4) ApplicationMaster is applied by the way of weights poll by RPC agreements to ResourceManager
With get resource.
5) after ApplicationMaster applies to resource, just communicated with corresponding NodeManager, it is desirable to it
Startup task.
6) after NodeManager sets running environment (including environmental variance, JAR bags, binary program etc.) for task,
Task start order is write in a script, and by running the script startup task.
7) each task reports the state and progress of oneself by some RPC agreement to ApplicationMaster, to allow
ApplicationMaster grasps the running status of each task at any time, so as to restart task in mission failure.
8) after the completion of application program operation, ApplicationMaster is nullified to ResourceManager and closed certainly
Oneself.
As shown in Figure 5, using Hadoop original strategies (before improvement) the operation π value estimations based on polling dispatching algorithm
During program, due to failing to consider the difference of performance between node, ApplicationMaster passed through by the way of simple poll
Resource is applied for and got to RPC agreements to ResourceManager.So with the execution of operation, S1 joint behaviors are worst, perform
Efficiency most causes S1 later stages cpu busy percentage to be higher than other nodes slowly, and S3 joint behaviors are best, so with Job execution, CPU
Utilization rate is gradually less than other nodes, and so obviously and underuses cluster resource, causes the Job execution later stage to have node
Between load uneven, and then the execution efficiency for influenceing whole operation is relatively low.And the strategy after using improvement is (described in the present embodiment
Weights polling dispatching strategy) when, it can be seen that in job run, the load of each node is relative to more balanced, operation before improving
Efficiency is also more efficient, as shown in Figure 6.
In the present embodiment, the principle of the polling dispatching algorithm used before improvement is each time the task requests from user
Distribute in turn from node, since 1, until last, from node, then restarts to circulate.Polling dispatching algorithm is assumed
All process performances from node are all identical, are indifferent to each current connection number and response speed from node.When request services
When interval time change is bigger, polling dispatching algorithm easily causes from the laod unbalance between node.So such a scheduling is calculated
Method is suitable for having identical software and hardware configuration and average service request relative equilibrium from node from all in node group
Situation.
It is identical with π value estimation programs, line chart ordinate be CPU utilization rate, abscissa be with every 10 seconds be one between
Every unit.As can be seen from Figure 7 using the Hadoop original strategies operation WordCount (systems based on polling dispatching algorithm
Count the word frequency of occurrence of collection) program when, due to fail consider node between performance difference, ApplicationMaster
Applied by the way of simple poll by RPC agreements to ResourceManager and get resource.Cause to bear between node
Carry uneven, S1 nodes operationally load very low, are not utilized by cluster fully, and then the execution efficiency for influenceing whole operation is inclined
It is low.And in strategy (the weights polling dispatching strategy described in the present embodiment) after using improvement, it can be seen that in job run
When each node load relative to improve before it is more balanced, operational efficiency also more efficiently, as shown in Figure 8.
As seen from Figure 9, the scheduling strategy after improvement and the strategy before improvement compare, and are dispatched after finding application enhancements
Tactful π values estimation program run time is shorter, shortens 16.35% compared with before-improvement.WordCount programs are run, before contrast improves
Run time afterwards, find to use the tactful run time after improving shorter, shorten 14.65% compared with before-improvement.
It should be noted that herein, such as first and second or the like relational terms are used merely to a reality
Body or operation make a distinction with another entity or operation, and not necessarily require or imply and deposited between these entities or operation
In any this actual relation or order.
Described above is the preferred embodiment of the present invention, it is noted that for those skilled in the art
For, on the premise of principle of the present invention is not departed from, some improvements and modifications can also be made, these improvements and modifications
It should be regarded as protection scope of the present invention.
Claims (10)
- A kind of 1. Hadoop cluster resources self-adapting distribution method, it is characterised in that including：According to default homework type classifying rules, the type for the operation that user submits is determined, wherein, each operation can be split Distributed Parallel Computing is realized into N number of task；If the type for the operation that user submits is the operation of CPU types, the operation of I/O types or important operation, the work submitted according to user The type of industry, it is determined that from the weights of node than row parameter, wherein, the weights from node i are equal to than row parameter from the weights of node i With in cluster all from the weights of node and ratio, be used to weigh the performance from node i from the weights of node i；It is each from the task requests of node distribution corresponding proportion according to each weights from node than row parameter.
- 2. Hadoop cluster resources self-adapting distribution method according to claim 1, it is characterised in that described according to default Homework type classifying rules, determine that the type of operation that user submits includes：By way of label, judge that the operation that user submits is important operation or general job, wherein, the label includes： It is important, general；If the operation is general job, judge whether the size of the operation is less than default size threshold value, if being less than, It is small operation to judge the operation；Otherwise, judge whether exceed default discrepancy threshold from the cpu resource and I/O resource differentializations of node in cluster, if super Cross, then the resource accounting consumed when executed according to operation, it is the operation of CPU types or I/O type operations to judge the operation；Otherwise, then according to the different loads degree in operation Map stages and Reduce stages, it is Map type operations to judge the operation Or Reduce type operations.
- 3. Hadoop cluster resources self-adapting distribution method according to claim 3, it is characterised in that described in the judgement Operation is that the operation of CPU types or I/O type operations include：If the operation meets the first formula, I/O type operations are marked as, wherein, first formula is expressed as：<mrow> <mfrac> <mrow> <mi>n</mi> <mo>&times;</mo> <mrow> <mo>(</mo> <mn>1</mn> <mo>+</mo> <mi>&rho;</mi> <mo>)</mo> </mrow> <mi>M</mi> <mi>I</mi> <mi>D</mi> </mrow> <mrow> <mi>D</mi> <mi>I</mi> <mi>O</mi> <mi>R</mi> </mrow> </mfrac> <mo>></mo> <mi>M</mi> <mi>T</mi> <mi>C</mi> <mi>T</mi> </mrow>If the operation meets the second formula, CPU type operations are marked as, wherein, second formula is expressed as：<mrow> <mfrac> <mrow> <mi>n</mi> <mo>&times;</mo> <mrow> <mo>(</mo> <mn>1</mn> <mo>+</mo> <mi>&rho;</mi> <mo>)</mo> </mrow> <mi>M</mi> <mi>I</mi> <mi>D</mi> </mrow> <mrow> <mi>D</mi> <mi>I</mi> <mi>O</mi> <mi>R</mi> </mrow> </mfrac> <mo><</mo> <mi>M</mi> <mi>T</mi> <mi>C</mi> <mi>T</mi> </mrow>Wherein, n represents that ρ represents that map end output data quantities input with map ends just in the quantity of executing tasks parallelly from node The ratio of data volume, MID represent map ends input data amount, and DIOR represents disk I/O transmission rate, and MTCT represents that Map tasks are complete Into the required time.
- 4. Hadoop cluster resources self-adapting distribution method according to claim 3, it is characterised in that described in the judgement Operation is the operation of Map types or Reduce type operations：When the operation meets three formula, then judge that the operation for Reduce type operations, otherwise judges that the operation is Map type operations, wherein, the 3rd formula is expressed as：<mrow> <mfrac> <msub> <mi>S</mi> <mrow> <mi>r</mi> <mi>e</mi> <mi>d</mi> <mi>u</mi> <mi>c</mi> <mi>e</mi> </mrow> </msub> <msub> <mi>S</mi> <mrow> <mi>m</mi> <mi>a</mi> <mi>p</mi> </mrow> </msub> </mfrac> <mo>></mo> <mi>t</mi> <mi>d</mi> </mrow>Wherein, SmapThe data total amount that the expression Map stages input, SreduceThe data total amount that the expression Reduce stages input, td tables Show default proportion threshold value.
- 5. Hadoop cluster resources self-adapting distribution method according to claim 1, it is characterised in that described from node Weights are expressed as：Wi=AY+BD+CFWherein, WiThe weights from node i are represented, Y represents the hardware performance from node i, and D represents the runnability from node i, F tables Show node failure rate, A, B, C are respectively Y, D, F coefficient.
- 6. Hadoop cluster resources self-adapting distribution method according to claim 5, it is characterised in that described from node i Hardware performance Y be expressed as：<mrow> <mi>Y</mi> <mo>=</mo> <msub> <mi>K</mi> <mn>1</mn> </msub> <mfrac> <msub> <mi>S</mi> <mrow> <mi>c</mi> <mi>p</mi> <mi>u</mi> </mrow> </msub> <mrow> <msub> <mi>avg</mi> <mrow> <mi>c</mi> <mi>p</mi> <mi>u</mi> </mrow> </msub> </mrow> </mfrac> <mo>+</mo> <msub> <mi>K</mi> <mn>2</mn> </msub> <mfrac> <msub> <mi>S</mi> <mrow> <mi>m</mi> <mi>e</mi> <mi>m</mi> </mrow> </msub> <mrow> <msub> <mi>avg</mi> <mrow> <mi>m</mi> <mi>e</mi> <mi>m</mi> </mrow> </msub> </mrow> </mfrac> <mo>+</mo> <msub> <mi>K</mi> <mn>3</mn> </msub> <mfrac> <msub> <mi>S</mi> <mrow> <mi>n</mi> <mi>e</mi> <mi>t</mi> </mrow> </msub> <mrow> <msub> <mi>avg</mi> <mrow> <mi>n</mi> <mi>e</mi> <mi>t</mi> </mrow> </msub> </mrow> </mfrac> <mo>+</mo> <msub> <mi>K</mi> <mn>4</mn> </msub> <mfrac> <msub> <mi>S</mi> <mrow> <mi>d</mi> <mi>i</mi> <mi>s</mi> <mi>k</mi> </mrow> </msub> <mrow> <msub> <mi>avg</mi> <mrow> <mi>d</mi> <mi>i</mi> <mi>s</mi> <mi>k</mi> </mrow> </msub> </mrow> </mfrac> </mrow>Wherein, ScpuRepresent CPU frequency, SmemRepresent memory size, SnetRepresent network bandwidth, SdiskRepresent the maximum read-write speed of disk Degree, avgcpu、avgmem、avgnet、avgdiskAverage CPU frequency, memory size, network bandwidth, the disk of cluster are represented respectively Maximum read or write speed, K1、K2、K3、K4All represent coefficient.
- 7. Hadoop cluster resources self-adapting distribution method according to claim 6, it is characterised in that from the fortune of node i Row performance D is expressed as：<mrow> <mi>D</mi> <mo>=</mo> <msub> <mi>G</mi> <mn>1</mn> </msub> <mfrac> <mn>1</mn> <mrow> <msub> <mi>t</mi> <mrow> <mi>c</mi> <mi>m</mi> </mrow> </msub> <mo>/</mo> <msub> <mi>avg</mi> <mrow> <mi>c</mi> <mi>m</mi> </mrow> </msub> </mrow> </mfrac> <mo>+</mo> <msub> <mi>G</mi> <mn>2</mn> </msub> <mfrac> <mn>1</mn> <mrow> <msub> <mi>t</mi> <mrow> <mi>i</mi> <mi>o</mi> <mi>m</mi> </mrow> </msub> <mo>/</mo> <msub> <mi>avg</mi> <mrow> <mi>i</mi> <mi>o</mi> <mi>m</mi> </mrow> </msub> </mrow> </mfrac> <mo>+</mo> <msub> <mi>G</mi> <mn>3</mn> </msub> <mfrac> <mn>1</mn> <mrow> <msub> <mi>t</mi> <mrow> <mi>c</mi> <mi>r</mi> </mrow> </msub> <mo>/</mo> <msub> <mi>avg</mi> <mrow> <mi>c</mi> <mi>r</mi> </mrow> </msub> </mrow> </mfrac> <mo>+</mo> <msub> <mi>G</mi> <mn>4</mn> </msub> <mfrac> <mn>1</mn> <mrow> <msub> <mi>t</mi> <mrow> <mi>i</mi> <mi>o</mi> <mi>r</mi> </mrow> </msub> <mo>/</mo> <msub> <mi>avg</mi> <mrow> <mi>i</mi> <mi>o</mi> <mi>r</mi> </mrow> </msub> </mrow> </mfrac> </mrow>Wherein, tcmThe CPU types operation of unit-sized is expressed as in the run time in Map stages, tcrIt is expressed as the CPU of unit-sized Type operation is in the run time in Reduce stages, tiomRun time of the I/O types operation of unit-sized in the Map stages is expressed as, tiorThe I/O types operation of unit-sized is expressed as in the run time in Reduce stages, avgcmIt is expressed as the CPU types of unit-sized Operation is in the average run time of Map stage clusters, avgcrThe CPU types operation of unit-sized is expressed as in Reduce stage collection The average run time of group, avgiomThe I/O types operation of unit-sized is expressed as in the average operation of Map stage clusters Between, avgiorThe I/O types operation of unit-sized is expressed as in the average run time of Reduce stage clusters, G1、G2、G3、 G4All represent coefficient.
- 8. Hadoop cluster resources self-adapting distribution method according to claim 7, it is characterised in that the node failure Rate F is expressed as：<mrow> <mi>F</mi> <mo>=</mo> <mfrac> <mrow> <msub> <mi>n</mi> <mrow> <mi>f</mi> <mi>a</mi> <mi>i</mi> <mi>l</mi> </mrow> </msub> <mo>/</mo> <msub> <mi>n</mi> <mrow> <mi>n</mi> <mi>u</mi> <mi>m</mi> </mrow> </msub> </mrow> <mrow> <msub> <mi>t</mi> <mrow> <mi>f</mi> <mi>a</mi> <mi>i</mi> <mi>l</mi> </mrow> </msub> <mo>/</mo> <msub> <mi>t</mi> <mrow> <mi>n</mi> <mi>u</mi> <mi>m</mi> </mrow> </msub> </mrow> </mfrac> </mrow>Wherein, nnumRepresent that journal file reads the operation task number of each node, nfailThe number of each node operation failure is represented, tnumRepresent the average operation task number of whole cluster, tfailRepresent the number of tasks of average operation failure.
- 9. Hadoop cluster resources self-adapting distribution method according to claim 8, it is characterised in that if the operation is CPU type operations, then increase describe the COEFFICIENT K from node cpu performance1、K2And G1、G3Value；If the operation is I/O type operations, COEFFICIENT K of the increase description from node I/O performances3、K4And G2、G4Value；If the operation is important operation, increase node failure rate F coefficient C value.
- 10. Hadoop cluster resources self-adapting distribution method according to claim 2, it is characterised in that if the operation For Map type operations, then priority scheduling is calculated to data storage is more from node；If the operation is Reduce type operations, being calculated from node more than priority scheduling to Map task output data quantities.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711120624.7A CN107832153B (en) | 2017-11-14 | 2017-11-14 | Hadoop cluster resource self-adaptive allocation method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711120624.7A CN107832153B (en) | 2017-11-14 | 2017-11-14 | Hadoop cluster resource self-adaptive allocation method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN107832153A true CN107832153A (en) | 2018-03-23 |
| CN107832153B CN107832153B (en) | 2020-12-29 |
Family
ID=61655305
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201711120624.7A Active CN107832153B (en) | 2017-11-14 | 2017-11-14 | Hadoop cluster resource self-adaptive allocation method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107832153B (en) |
Cited By (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108551444A (en) * | 2018-03-30 | 2018-09-18 | 新华三信息安全技术有限公司 | A kind of log processing method, device and equipment |
| CN108958942A (en) * | 2018-07-18 | 2018-12-07 | 郑州云海信息技术有限公司 | A kind of distributed system distribution multitask method, scheduler and computer equipment |
| CN109309726A (en) * | 2018-10-25 | 2019-02-05 | 平安科技（深圳）有限公司 | Document generating method and system based on mass data |
| CN109947532A (en) * | 2019-03-01 | 2019-06-28 | 中山大学 | A kind of big data method for scheduling task in education cloud platform |
| CN110908796A (en) * | 2019-11-04 | 2020-03-24 | 北京理工大学 | Multi-operation merging and optimizing system and method in Gaia system |
| CN110971647A (en) * | 2018-09-30 | 2020-04-07 | 南京工程学院 | Node migration method of big data system |
| CN111459677A (en) * | 2020-04-01 | 2020-07-28 | 北京顺达同行科技有限公司 | Request distribution method and device, computer equipment and storage medium |
| CN111580950A (en) * | 2020-06-15 | 2020-08-25 | 四川中电启明星信息技术有限公司 | Self-adaptive feedback resource scheduling method for improving cloud reliability |
| CN111831418A (en) * | 2020-07-14 | 2020-10-27 | 华东师范大学 | Big data analysis job performance optimization method based on delay scheduling technology |
| CN112470148A (en) * | 2018-07-26 | 2021-03-09 | 罗布乐思公司 | MapReduce model for data tilting |
| CN112764906A (en) * | 2021-01-26 | 2021-05-07 | 浙江工业大学 | Cluster resource scheduling method based on user job type and node performance bias |
| CN113626098A (en) * | 2021-07-21 | 2021-11-09 | 长沙理工大学 | Data node dynamic configuration method based on information interaction |
| CN116302404A (en) * | 2023-02-16 | 2023-06-23 | 北京大学 | Resource decoupling data center-oriented server non-perception calculation scheduling method |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103793272A (en) * | 2013-12-27 | 2014-05-14 | 北京天融信软件有限公司 | Periodical task scheduling method and periodical task scheduling system |
| CN103902379A (en) * | 2012-12-25 | 2014-07-02 | 中国移动通信集团公司 | Task scheduling method and device and server cluster |
| US20150281114A1 (en) * | 2014-03-28 | 2015-10-01 | Akamai Technologies, Inc. | Systems and methods for allocating work for various types of services among nodes in a distributed computing system |
| CN107038069A (en) * | 2017-03-24 | 2017-08-11 | 北京工业大学 | Dynamic labels match DLMS dispatching methods under Hadoop platform |
-
2017
- 2017-11-14 CN CN201711120624.7A patent/CN107832153B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103902379A (en) * | 2012-12-25 | 2014-07-02 | 中国移动通信集团公司 | Task scheduling method and device and server cluster |
| CN103793272A (en) * | 2013-12-27 | 2014-05-14 | 北京天融信软件有限公司 | Periodical task scheduling method and periodical task scheduling system |
| US20150281114A1 (en) * | 2014-03-28 | 2015-10-01 | Akamai Technologies, Inc. | Systems and methods for allocating work for various types of services among nodes in a distributed computing system |
| CN107038069A (en) * | 2017-03-24 | 2017-08-11 | 北京工业大学 | Dynamic labels match DLMS dispatching methods under Hadoop platform |
Non-Patent Citations (3)
| Title |
|---|
| 胡丹: "基于负载均衡的Hadoop平台下作业调度算法研究", 《中国优秀硕士学位论文全文数据库 信息科技辑》 * |
| 郑晓薇等: "基于节点能力的Hadoop集群任务自适应调度方法", 《计算机研究与发展》 * |
| 陶韬: "云计算环境下基于MapReduce的资源调度模型和算法研究", 《中国优秀硕士学位论文全文数据库 信息科技辑》 * |
Cited By (17)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108551444A (en) * | 2018-03-30 | 2018-09-18 | 新华三信息安全技术有限公司 | A kind of log processing method, device and equipment |
| CN108958942A (en) * | 2018-07-18 | 2018-12-07 | 郑州云海信息技术有限公司 | A kind of distributed system distribution multitask method, scheduler and computer equipment |
| CN112470148A (en) * | 2018-07-26 | 2021-03-09 | 罗布乐思公司 | MapReduce model for data tilting |
| CN110971647A (en) * | 2018-09-30 | 2020-04-07 | 南京工程学院 | Node migration method of big data system |
| CN110971647B (en) * | 2018-09-30 | 2023-12-05 | 南京工程学院 | Node migration method of big data system |
| CN109309726A (en) * | 2018-10-25 | 2019-02-05 | 平安科技（深圳）有限公司 | Document generating method and system based on mass data |
| CN109947532A (en) * | 2019-03-01 | 2019-06-28 | 中山大学 | A kind of big data method for scheduling task in education cloud platform |
| CN109947532B (en) * | 2019-03-01 | 2023-06-09 | 中山大学 | Big data task scheduling method in education cloud platform |
| CN110908796A (en) * | 2019-11-04 | 2020-03-24 | 北京理工大学 | Multi-operation merging and optimizing system and method in Gaia system |
| CN111459677A (en) * | 2020-04-01 | 2020-07-28 | 北京顺达同行科技有限公司 | Request distribution method and device, computer equipment and storage medium |
| CN111580950A (en) * | 2020-06-15 | 2020-08-25 | 四川中电启明星信息技术有限公司 | Self-adaptive feedback resource scheduling method for improving cloud reliability |
| CN111831418A (en) * | 2020-07-14 | 2020-10-27 | 华东师范大学 | Big data analysis job performance optimization method based on delay scheduling technology |
| CN112764906A (en) * | 2021-01-26 | 2021-05-07 | 浙江工业大学 | Cluster resource scheduling method based on user job type and node performance bias |
| CN112764906B (en) * | 2021-01-26 | 2024-03-15 | 浙江工业大学 | Cluster resource scheduling method based on user job type and node performance bias |
| CN113626098A (en) * | 2021-07-21 | 2021-11-09 | 长沙理工大学 | Data node dynamic configuration method based on information interaction |
| CN116302404A (en) * | 2023-02-16 | 2023-06-23 | 北京大学 | Resource decoupling data center-oriented server non-perception calculation scheduling method |
| CN116302404B (en) * | 2023-02-16 | 2023-10-03 | 北京大学 | Resource decoupling data center-oriented server non-perception calculation scheduling method |
Also Published As
| Publication number | Publication date |
|---|---|
| CN107832153B (en) | 2020-12-29 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107832153A (en) | A kind of Hadoop cluster resources self-adapting distribution method | |
| US11487771B2 (en) | Per-node custom code engine for distributed query processing | |
| Hu et al. | Flutter: Scheduling tasks closer to data across geo-distributed datacenters | |
| Yao et al. | Fog resource provisioning in reliability-aware IoT networks | |
| CN109120715A (en) | Dynamic load balancing method under a kind of cloud environment | |
| Nghiem et al. | Towards efficient resource provisioning in MapReduce | |
| Kang et al. | Web service selection for resolving conflicting service requests | |
| US10642652B2 (en) | Best trade-off point on an elbow curve for optimal resource provisioning and performance efficiency | |
| CN107908672A (en) | Application report implementation method, equipment and storage medium based on Hadoop platform | |
| Taft et al. | P-store: An elastic database system with predictive provisioning | |
| CN103338228A (en) | Cloud calculating load balancing scheduling algorithm based on double-weighted least-connection algorithm | |
| CN105471985A (en) | Load balance method, cloud platform computing method and cloud platform | |
| CN104978236A (en) | HDFS load source and sink node selection method based on multiple measurement indexes | |
| CN108551489A (en) | A kind of application server load balancing method, system, device and storage medium | |
| CN104063501B (en) | copy balance method based on HDFS | |
| CN103218233A (en) | Data allocation strategy in hadoop heterogeneous cluster | |
| Gionis et al. | Piggybacking on social networks | |
| Javadpour et al. | Improving load balancing for data-duplication in big data cloud computing networks | |
| CN104657217B (en) | A kind of cloud environment method for scheduling task based on non-homogeneous grain-size classification | |
| Zacheilas et al. | Dynamic load balancing techniques for distributed complex event processing systems | |
| Liu et al. | Service reliability in an HC: Considering from the perspective of scheduling with load-dependent machine reliability | |
| Li et al. | Scalable and dynamic replica consistency maintenance for edge-cloud system | |
| WO2020040928A1 (en) | System for optimizing storage replication in a distributed data analysis system using historical data access patterns | |
| Canali et al. | Exploiting ensemble techniques for automatic virtual machine clustering in cloud systems | |
| CN110311933A (en) | A kind of method, apparatus, equipment and the storage medium of equilibrium service traffics |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |