CN109150759A - The gradual non-obstruction chance method for obligating resource of one kind and system - Google Patents
The gradual non-obstruction chance method for obligating resource of one kind and system Download PDFInfo
- Publication number
- CN109150759A CN109150759A CN201810988076.8A CN201810988076A CN109150759A CN 109150759 A CN109150759 A CN 109150759A CN 201810988076 A CN201810988076 A CN 201810988076A CN 109150759 A CN109150759 A CN 109150759A
- Authority
- CN
- China
- Prior art keywords
- node
- resource
- chance
- task
- resource reservation
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L47/00—Traffic control in data switching networks
- H04L47/70—Admission control; Resource allocation
- H04L47/72—Admission control; Resource allocation using reservation actions during connection setup
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L47/00—Traffic control in data switching networks
- H04L47/70—Admission control; Resource allocation
- H04L47/83—Admission control; Resource allocation based on usage prediction
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Computer And Data Communications (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
The invention belongs to network information processing technical fields, disclose the gradual non-obstruction chance method for obligating resource of one kind and system, the form combined using preselected node (Preselection) and preferred node (Priority), the node of resource reservation can be provided for big task by selecting first, Priority strategy is re-introduced into sort to filtered node priority, the resource reservation node that some node is current big task is locked, resource reservation is finally carried out to node using chance resource allocation policy.The present invention solves the node being reserved and no longer distributes resource until there is meeting reservation request for other tasks, this will will lead to cluster and generate a large amount of resource fragmentation, and cluster delay machine also becomes a Great possibility.
Description
Technical field
The invention belongs to network information processing technical fields, more particularly to a kind of gradual non-obstruction chance resource reservation side
Method and system.
Background technique
For Hadoop YARN resource scheduling system, it is pre- when some node is not able to satisfy mission requirements to will do it resource
It stays.
Currently, the prior art commonly used in the trade is such that incremental resource is reserved and disposable resource reservation.
It is preferably the resource on the reserved node of this application program, is answered until the idling-resource of accumulative release is met
Demand.The method of salary distribution of this resource is called incremental resource distribution.
Present node resource temporarily is abandoned, until the demand of the resource all promising policy application program of some node.It is this
The method of salary distribution is called disposable resource allocation.
However, two kinds of methods of salary distribution have deficiency, for incremental resource distribution, resource reservation mechanism will lead to resource
Waste, cluster resource utilization rate are low；Although and disposable resource allocation is when discovery resource is unable to satisfy some application demand
Time abandons resource request, still, this application be possible to cannot get forever oneself request resource thus be unable to run forever, i.e.,
Phenomenon hungry to death；
In conclusion problem of the existing technology is:
(1) current resource reservation mechanism is not to selecting which node is limited to execute resource reservation, Preserved node
Selection become uncontrollable, in fact it could happen that node is all reserved, leads to cluster delay machine.
(2) existing resource reservation mechanism does not make full use of the resource of reserved period, reserved node until
Appearance all no longer distributes resource before meeting reservation request for other tasks, and it is broken that this will will lead to a large amount of resource of cluster generation
Piece, resource are not fully utilized.
Solve the difficulty and meaning of above-mentioned technical problem:
Clustered node is being prevented all to be reserved, the case where only release does not distribute, leads to cluster delay machine occurs in cluster resource
On, it needs to meet the node for doing resource reservation by formulating corresponding strategy to select, whether selection is met most for strategy
Excellent node needs to do many experiments textual criticism.
On the Utilizing question to reserved resource, the present invention it is innovative in the way of non-block type resource reservation, this is right
Resource service can be provided for other application for node, the shape by monitoring other nodes can also be gone for application
Formula timely finds whether other nodes have the resource release for meeting and requesting.Solving cluster resource fragment problems and the utilization of resources
It is of great significance in rate problem.
Summary of the invention
In view of the problems of the existing technology, the present invention provides a kind of gradual non-obstruction chance method for obligating resource and
System.The present invention is by the method for obligating resource in optimization Hadoop Yarn when some node is not able to satisfy mission requirements, finally
It is excessively high to solve cluster resource fragment, the too low problem of cluster entirety resource utilization, to improve overall task execution efficiency.
The invention is realized in this way a kind of gradual non-obstruction chance method for obligating resource, using preselected node
The form that Preselection and preferred node Priority are combined；
It specifically includes:
When application program carries out resource bid to Hadoop Yarn cluster and is not being met, Yarn can be in cluster
A certain node carries out resource reservation:
It selects first and provides the node of resource reservation for big task: to all nodes of cluster according to Preselection strategy
Filter out the node that cannot do resource reservation；
Secondly Priority strategy is introduced to filtered node weight priority ranking, and it is current big for locking some node
The resource reservation node of task；
Finally using non-block type resource reservation algorithm bonding machine can resource allocation policy to choose some weight node and
Maximum node carries out resource reservation.
Further, Preselection strategy includes:
Average resource is filtered out in the node greater than 1.0*0.8*N as resource reservation node；It will full remaining foot
Input of the node of condition as the Priority stage；
It specifically includes:
One is created first for storing the empty array Pre_node [] of preselection pre-selection result；
Then all nodes are traversed, for all nodes for meeting node load and being less than N*0.8*1.0, are all added to Pre_
Inside node [] array；
The node inside Pre_node [] array is finally passed to the Priority stage.
Further, Priority strategy, using mapReduce program come node each in computing cluster all
The weight summation embodied under Prioritypolicy；With the policy weight of each node server and carry out sequencing selection synthesis
The most suitable node of performance；
Further, the formulation of PriorityPolicy optimal policy, comprising:
Preferential selection locality node: the resource of preferential selection and task in same node；
It selects the node more than small task: being calculated according to 100% resource load, node money is occupied to each node maximum task
Source situation is divided into 4 grades: container used in node maximum task accounts for node server total resources rate less than 20%, saves
Container used in the maximum task of point accounts for node server total resources rate up to 20%~40%, used in node maximum task
Container accounts for node server total resources rate up to 40%~60%, and container used in node maximum task accounts for node serve
Device total resources rate is more than 60%；
Select node server resource occupation amount and the smaller node of node server total resources ratio:
By the way that following Weight selected: node server resource occupation amount/node server total resources < 0.2 is arranged,
0.4 > node server resource occupation amount/node server total resources > 0.2,
0.6 > node server resource occupation amount/node server total resources > 0.4,
0.8 > node server resource occupation amount/node server total resources > 0.6,
Node server resource occupation amount/node server total resources > 0.8.
Further, the weight and statistical of each node are counted according to the PriorityPolicy optimal policy of formulation
Include:
The weight of each policy of each node is first write into file f ile.txt first；
Then using file.txt as the input of Map program, by row with tab t cutting, retain array index after cutting
The value for arriving array end for 1；Again with: cutting obtain input data；
The result of last map enters reduce program and is handled, and the map result of identical key enters the same reduce,
Weight is added again later, obtains processing data；
After processing by third step Reduce program, the corresponding weight number of node node, for by formulation
Priority policy preference policy screening after weight and, the node for selecting weight big is pre- as the resource of current task
Stay node；If there is the identical node of weight, according to the first choosing later of nodes heart beat communication.
Further, chance resource allocation policy specifically includes:
1) application takes non-blocking mode to carry out resource reservation in some node,
Some is allowed to apply the change in resources for monitoring other server nodes during some node carries out resource reservation, when
It was found that other a certain nodes release because certain reasons terminate the certain applications run on this node and have met this
When doing the resource request of the application of resource reservation, allows the resource reservation of application cancellation on this node to turn to this node and hold
Itself task of row.It specifically includes:
A dictionary nodes is defined first for storing all nodal informations, and each nodal information includes: name, res
(node resource total amount), remain_res (node surplus yield), wherein name=pre indicates that the Preselection stage is selected
The node selected, name=pri indicate the resource reservation node chosen in the Priority stage；
Then entire dictionary is traversed, it is pre- to carry out resource to this node if meeting nodes [i] [" name "]=pri
It stays, otherwise this node is monitored.
Finally if there is the node of some name=pre has discharged own resource in advance, and meets this and provided
The resource request of the reserved application in source no longer carries out resource reservation to pri node then dischargeing reserved resource first；Then
Resource request is done to the pre node.
2) utilization of chance resource, the resource for allowing to be reserved provide service under certain conditions for other application；Tool
Body includes:
Define 1 reserved container: some node is the node being reserved, and is reserved then it will generate one
The capacity of container, the container are the stock numbers that reserved required by task is wanted, after node resource release, by resource
Filling inside toward reserved container.When this reserve just this container is returned to after container is filled up it is pre-
Remain in office business.
Define 2 chance resources (opp.resource): some node being reserved gradually is discharging the pre- to fill of resource
When staying container, the resource before unfilled container is referred to as chance resource.
Since the distribution of resource is also required to the time, needed for chance resource reaches when reserved container stock number 80%,
This chance resource is closed, no longer provides resource for other application.
A first, it is determined that whether have in cluster can satisfy the node of application resource request, have, and return to resource, it is no then
It is walked into b；
B judge chance resource whether meet can distributive condition (chance resource meets task requests and chance resource is less than
The 80% of aggregate reservation stock number), meet and then enter c step, otherwise task resource distribution failure, finds a new node and carry out
Resource reservation；
C is calculated according to Preserved node Resource release time and is reserved Resource release time needed for residue；Worked as according to the node
Preceding cpu and memory information calculate this and apply the task execution time in present node；Time calculates to finish and walk into d；
If the d task execution time, which is less than needed for residue, reserves Resource release time, which is given
This new task is executed subsequently into e；Otherwise distribution failure finds a new node and carries out resource reservation；
E new task is finished, and chance resource is returned reserved container.
Another object of the present invention is to provide a kind of meters for realizing the gradual non-obstruction chance method for obligating resource
Calculation machine program.
Another object of the present invention is to provide a kind of letters for realizing the gradual non-obstruction chance method for obligating resource
Cease data processing terminal.
Another object of the present invention is to provide a kind of computer readable storage mediums, including instruction, when it is in computer
When upper operation, so that computer executes the gradual non-obstruction chance method for obligating resource.
Realize that the gradual non-obstruction chance method for obligating resource is progressive another object of the present invention is to provide a kind of
The non-obstruction chance resource reservation system of formula, the gradual non-obstruction chance resource reservation system specifically include:
Preselection policy unit provides the node of resource reservation for selecting for big task: to cluster
All nodes fall to do according to Preselection policy filtering the node of resource reservation；
Priority policy unit, for introducing Priority strategy to filtered node weight priority ranking, lock
Some fixed node is the resource reservation node of current big task；
Chance resource allocation policy unit, for using chance resource allocation policy to choosing some weight node and maximum
Node as resource reservation node, carry out resource reservation.
In conclusion advantages of the present invention and good effect are as follows:
The present invention proposes a kind of gradual non-obstruction chance resource reservation mechanism (Progressive Unblocking
Opportunity resources reservation, PUOR), using preselected node (Preselection) and preferred node
(Priority) form combined, the node of resource reservation can be provided for big task first by selecting, and be re-introduced into Priority
Strategy sorts to filtered node priority, locks the resource reservation node that some node is current big task, finally uses
Chance resource allocation policy carries out resource reservation to node.Relative to the resource reservation mechanism of Hadoop script, PUOR algorithm exists
7.22% is improved in resource utilization, is effectively reduced cluster resource fragment, is increased cluster throughput.
Detailed description of the invention
Fig. 1 be chosen in gradual non-obstruction chance method for obligating resource provided in an embodiment of the present invention some node as
Resource reservation node flow chart.
Fig. 2 is progress node resource reservations flow chart provided in an embodiment of the present invention.
Fig. 3 is provided in an embodiment of the present invention to take non-blocking mode to current application in PUOR resource reservation algorithm
Resource reservation figure is carried out in some node.
Fig. 4 is the concept map provided in an embodiment of the present invention that chance resource is proposed in PUOR resource reservation mechanism.
Fig. 5 is gradual non-obstruction chance resource reservation system schematic diagram provided in an embodiment of the present invention.
In figure: 1, Preselection policy unit；2, Priority policy unit；3, chance resource allocation policy list
Member.
Fig. 6 is that figure is compared in cluster cpu use provided in an embodiment of the present invention.
Fig. 7 is that figure is compared in cluster memory use provided in an embodiment of the present invention.
Specific embodiment
In order to make the objectives, technical solutions, and advantages of the present invention clearer, with reference to embodiments, to the present invention
It is further elaborated.It should be appreciated that the specific embodiments described herein are merely illustrative of the present invention, it is not used to
Limit the present invention.
One, below with reference to concrete analysis, the invention will be further described.
Gradual non-obstruction chance method for obligating resource provided in an embodiment of the present invention, comprising:
First part: the selection of Preserved node is done
This stage includes two steps: Preselection stage and Priority stage (such as Fig. 1).
First all nodes of cluster are fallen to do with the node of resource reservation according to Preselection policy filtering, then is led to
It crosses Priority Policy strategy to judge node weight, finally chooses some weight and maximum node as resource
Reserved node.
Preselection process:
In a multi-processor system, load mean value is what the quantity based on kernel determined, the server based on linux system
Its sensible load is CPU core number * 1.00.If indicating system nucleus number with N, load shows between 0.00~1.00*N
Although system load becoming, its system performance is all in flow.Load is equal to 1.00*N and illustrates that system has not had
There is remaining resource, if receiving new task again, this task is just waited in line, and whole system is in congestion state i.e..
In a practical situation, the present invention takes empirical value 1.0*0.8*N to measure the health standards of a server.If certain
A node load is close to 1.0*0.8*N, and remaining 20% resource is not able to satisfy the resource request of some big task but, when to this
Task carries out resource reservation in this node, then the load that will lead to following this node of a period of time will be from 1.0*0.8*N
Straight line decline, then disposably dispenses this bulk container when the resource container for meeting this big task occurs
Cause to load very fast rising and inevitable remote super 1.0*0.8*N again, there is maximum probability to be greater than 1.0*N.Such very fast variation of load, leads
The result of cause is exactly that server fluctuation is larger, in order to ensure the fluctuation of whole cluster average resource is gentle, therefore the present invention
Average resource be will filter out in the node of 1.0*0.8*N or more as resource reservation node.By full remaining sufficient condition
Input of the node as the Priority stage.
Priority process:
The present invention will formulate weight for each Prioritypolicy (preference policy).It is counted using mapReduce program
Calculate the weight summation that each node is embodied at all Prioritypolicy of the invention in cluster.With each node serve
The policy weight of device and carry out the most suitable node of sequencing selection comprehensive performance.
Formulation about PriorityPolicy optimal policy:
1, locality node is preferentially selected
In Hadoop cluster, the performance consumption of mobile data block is higher than the performance consumption of mobile computing task, therefore
The present invention this stage to pay the utmost attention to and task same node resource.For a task, server set
Node resource in group can be divided into three types:
2, the node more than small task is selected
If there is the resource container of a task requests 50G, and there are two total resources, resource occupation amount and resource releases
The all identical node A and B of rate, situation are as follows:
So present one needs the application of 50G resource because the preferential selection B node carries out resource reservation, because of A node
Just 40G, the task execution for needing to wait until to occupy 60Gcontainer complete release first dischargeing the container of 20G
Fall the resource request that resource is just able to satisfy new 50Gcontainer；And the small task of B node is more, discharges the rate of task more
Fastly, dischargeing two container just has 60G later, meets the needs of 50Gcontainer.Therefore the present invention is according to 100%
Resource load calculates, and occupies node resource situation to each node maximum task and is divided into 4 grades:
3, node server resource occupation amount and the smaller node of node server total resources ratio are selected
If what cluster executed is all small task, such as map task, then policy 2 is it is possible that all weights
Identical result.Then, Article 3 policy is supplemented.For example, if load is all 50% or more when cluster every server operation,
It is 92G that task maximum, which requests Container, and the physical memory of every server is 100G, then after server is scheduled
Cannot respond for a long time just becomes Great possibility.And the Container value of same average load and task requests, if
The physical memory of every server of the present invention is 200G, and the probability that server is reserved the response that but delays can then drop significantly
It is low.Can also reduce all nodes of cluster is reserved the probability for leading to cluster delay machine simultaneously.For this purpose, present invention setting weight is as follows:
Two, the weight and statistics of each node are counted below with reference to the PriorityPolicy optimal policy according to formulation
The invention will be further described for mode.
According to the PriorityPolicy optimal policy of formulation count each node weight and, statistical is as follows:
1, the weight of each policy of each node is first write into file first and is such as named as file.txt.
The content format of file.txt should are as follows:
2, using file.txt as the input of Map program, present invention design, with tab " t " cutting, retains cutting by row
Array index is 1 value for arriving array end afterwards；Following data can be obtained with ": " cutting again:
3, the result of map enters reduce program and is handled, and the map result of identical key enters the same reduce, it
Weight is added again afterwards, will then obtain following data:
After processing in this way Jing Guo third step Reduce program, present node node corresponding weight number, be through
Cross weight after the Priority policy preference policy screening that the present invention formulates and, the node for selecting weight big is as working as
The resource reservation node of preceding task.If there is the identical node of weight, according to the first choosing later of nodes heart beat communication, whose in period
Whom heartbeat selects first.
After the node selection of PUOR algorithm first stage, it will select and meet the section that current task does resource reservation
Point can effectively reduce because choosing at random node and do server cluster delay machine probability caused by resource reservation node.
Three, below with reference to it is gradual it is non-obstruction resource reservation and chance resource utilization the invention will be further described.
Second part: the utilization of gradual non-obstruction resource reservation and chance resource
PUOR algorithm second part is to carry out node resource reservations (such as Fig. 2).The present invention uses gradual non-obstruction resource
The method of salary distribution does resource reservation to node, the resource present invention for being used to execute the task requests discharged for Preserved node
It is defined as chance resource, the quantity of resource fragmentation is effectively reduced by resource of rationally improving the occasion.
1.1, gradual non-obstruction preservation algorithm
The resource reservation mechanism that Hadoop YARN is used, is all block type resource reservation, when some is applied in some node
When resource reservation occurs, whether the resource that will monitor node release always meets oneself request, during this this
Business cannot request resource to other nodes again, which can not provide resource to other application.For this case, the present invention
Non-blocking mode is taken to carry out resource reservation (such as Fig. 3) in some node current application in PUOR resource reservation algorithm.
Algorithm of the invention allow some apply some node carry out resource reservation during monitor other server nodes
Change in resources, discharged when finding other a certain nodes because certain reasons terminate operation certain applications on this node
When meeting the resource request of the application for having done resource reservation out, the resource reservation for allowing the application to cancel on this node turns
Itself task is executed to this node.
1.2, the utilization of chance resource
Hadoop YARN original future mechanism, when some node is that some application carries out resource reservation, until occurring
Before the resource container for meeting this application, this node all cannot provide resource container for other application, this causes very big
The wasting of resources.The present invention proposes the concept (such as Fig. 4) of chance resource, the resource for allowing to be reserved in PUOR resource reservation mechanism
Service is provided under certain conditions for other application.
。
As shown in figure 5, gradual non-obstruction chance resource reservation system provided in an embodiment of the present invention specifically includes:
Preselection policy unit 1 provides the node of resource reservation for selecting for big task: to all sections of cluster
Point falls to do according to Preselection policy filtering the node of resource reservation；
Priority policy unit 2, for introducing Priority strategy to filtered node weight priority ranking, lock
Some fixed node is the resource reservation node of current big task；
Chance resource allocation policy unit 3, for using chance resource allocation policy to choosing some weight node and most
Big node carries out resource reservation as the node of resource reservation.
Three, below with reference to experiment, the invention will be further described.
Experimental situation and data model:
Non- block type chance resource reservation mechanism performance is told herein to verify, and chooses 5 servers to build experiment
Group system, server are configured to 32 core CPU, 128GB memories.Wherein there are 1 Name Node, a Resource
Manager, 5 DataNode and 5 Node Manager.There are various types of tasks in true cluster environment, so
It chooses representational loading commissions and is conducive to accurate validation and assessment.Choose Sort herein, Tera Sort, TestDFSIO this
Three kinds of representative MapReduce tasks.Test program type is as shown in table 1 below.
It is test program title and corresponding types below:
| Test program | Type |
| Sort | Computation-intensive |
| TeraSort | Map stage computation-intensive, reduce stage I/O are intensive |
| TestDFSIO | I/O is intensive |
1 test program title of table and type
The function of Sort is exactly to be ranked up to input file by Key；TeraSort program is carried out input file by Key
Overall situation sequence, in addition TeraSort is directed to large batch of data；The read or write speed of TestDFSIO test HDFS.
Task is submitted to cluster using Hi Bench test suite, constructs 5 group tasks task-set not of uniform size, often
300 tasks of group, every kind of test program account for 1/3, and the data volume of each task is as shown in table 2
| Sort | TeraSort | TestDFSIO | |
| First group | 6.4GB | 64GB | 32GB |
| Second group | 6.4GB | 6.4GB | 32GB |
| Third group | 640MB | 6.4GB | 3.2GB |
| 4th group | 640MB | 6.4GB | 3.2GB |
| 5th group | 640MB | 640MB | 3.2GB |
2 test assignment size of table
Experimental result and analysis:
Object, two kinds of schedulers are all made of increment money to FIFO the and Capacity scheduler carried using Yarn as a comparison
The source method of salary distribution.
Figure is compared in Fig. 6 cluster cpu use.
Figure is compared in Fig. 7 cluster memory use.
In summary experimental result can be seen that requests when cluster had not only had big task resource, but also there are small task resources
When request, cluster can carry out resource reservation to the big task being not being met, and can effectively improve collection using PUOR preservation algorithm
Group's resource utilization.
In the above-described embodiments, can come wholly or partly by software, hardware, firmware or any combination thereof real
It is existing.When using entirely or partly realizing in the form of a computer program product, the computer program product include one or
Multiple computer instructions.When loading on computers or executing the computer program instructions, entirely or partly generate according to
Process described in the embodiment of the present invention or function.The computer can be general purpose computer, special purpose computer, computer network
Network or other programmable devices.The computer instruction may be stored in a computer readable storage medium, or from one
Computer readable storage medium is transmitted to another computer readable storage medium, for example, the computer instruction can be from one
A web-site, computer, server or data center pass through wired (such as coaxial cable, optical fiber, Digital Subscriber Line (DSL)
Or wireless (such as infrared, wireless, microwave etc.) mode is carried out to another web-site, computer, server or data center
Transmission).The computer-readable storage medium can be any usable medium or include one that computer can access
The data storage devices such as a or multiple usable mediums integrated server, data center.The usable medium can be magnetic Jie
Matter, (for example, floppy disk, hard disk, tape), optical medium (for example, DVD) or semiconductor medium (such as solid state hard disk Solid
State Disk (SSD)) etc..
The foregoing is merely illustrative of the preferred embodiments of the present invention, is not intended to limit the invention, all in essence of the invention
Made any modifications, equivalent replacements, and improvements etc., should all be included in the protection scope of the present invention within mind and principle.
Claims (10)
1. a kind of gradual non-obstruction chance method for obligating resource, which is characterized in that the gradual non-obstruction chance resource is pre-
Method is stayed first to select Preserved node using the form that preselected node Preselection and preferred node Priority are combined, then
By chance resource allocation policy to choose node carry out resource reservation；
It specifically includes:
When application program carries out resource bid to Hadoop Yarn cluster and is not being met, Yarn is to section a certain in cluster
Point carries out resource reservation:
It selects first and provides the node of resource reservation for big task: to all nodes of cluster according to Preselection policy filtering
Fall the node that cannot do resource reservation；
Secondly Priority strategy is introduced to filtered node weight priority ranking, and locking some node is current big task
Resource reservation node；
Finally using non-block type resource reservation algorithm bonding machine can resource allocation policy to choosing some weight node and maximum
Node carry out resource reservation.
2. gradual non-obstruction chance method for obligating resource as described in claim 1, which is characterized in that Preselection plan
It slightly include: to filter out average resource in the node greater than 1.0*0.8*N as resource reservation node；It will full remaining sufficient item
Input of the node of part as the Priority stage；
It specifically includes:
One is created first for storing the empty array Pre_node [] of preselection pre-selection result；
Then all nodes are traversed, for all nodes for meeting node load and being less than N*0.8*1.0, are all added to Pre_node
Inside [] array；
The node inside Pre_node [] array is finally passed to the Priority stage.
3. gradual non-obstruction chance method for obligating resource as described in claim 1, which is characterized in that Priority strategy packet
It includes: using mapReduce program come preselection stage node institute at all Prioritypolicy in computing cluster
The weight summation of embodiment；With the policy weight of each node server and carry out the most suitable node of sequencing selection comprehensive performance.
4. gradual non-obstruction chance method for obligating resource as claimed in claim 3, which is characterized in that
The formulation of PriorityPolicy optimal policy, comprising:
Preferential selection locality node: the resource of preferential selection and task in same node；
It selects the node more than small task: being calculated according to 100% resource load, node resource feelings are occupied to each node maximum task
Condition is divided into 4 grades: container used in node maximum task accounts for node server total resources rate less than 20%, and node is most
Container used in big task accounts for node server total resources rate up to 20%~40%, container used in node maximum task
Node server total resources rate is accounted for up to 40%~60%, container used in node maximum task accounts for node server total resources
Rate is more than 60%；
Select node server resource occupation amount and the smaller node of node server total resources ratio:
By the way that following Weight selected is arranged:
Node server resource occupation amount/node server total resources < 0.2,
0.4 > node server resource occupation amount/node server total resources > 0.2,
0.6 > node server resource occupation amount/node server total resources > 0.4,
0.8 > node server resource occupation amount/node server total resources > 0.6,
Node server resource occupation amount/node server total resources > 0.8.
5. gradual non-obstruction chance method for obligating resource as claimed in claim 3, which is characterized in that
According to the PriorityPolicy optimal policy of formulation count each node weight and, statistical includes:
The weight of each policy of each node is first write into file f ile.txt first；
Then using file.txt as the input of Map program, by row with tab " t " cutting, array index is after retaining cutting
1 arrives the value at array end；Input data is obtained with ": " cutting again；
The result of last map enters reduce program and is handled, and the map result of identical key enters the same reduce, later
Weight is added again, obtains processing data；
After processing by third step Reduce program, the corresponding weight number of node node, for by formulation
Priority policy preference policy screening after weight and, the node for selecting weight big is pre- as the resource of current task
Stay node；If there is the identical node of weight, according to the first choosing later of nodes heart beat communication.
6. gradual non-obstruction chance method for obligating resource as described in claim 1, which is characterized in that
Chance resource allocation policy specifically includes:
1) application takes non-blocking mode to carry out resource reservation in some node,
Allow some to apply the change in resources for monitoring other server nodes during some node carries out resource reservation, works as discovery
Other a certain nodes release because certain reasons terminate operation certain applications on this node and meet this and provided
When the resource request of the reserved application in source, the resource reservation for allowing the application to cancel on this node turns to the execution of this node certainly
Body task；
It specifically includes:
A dictionary nodes is defined first for storing all nodal informations, and each nodal information includes: name, res (node
Total resources), remain_res (node surplus yield), wherein name=pre indicates that the Preselection stage selects
Node, name=pri indicate the resource reservation node chosen in the Priority stage；
Then entire dictionary is traversed, resource reservation is carried out to this node if meeting nodes [i] [" name "]=pri, it is no
Then this node is monitored；
Finally if there is the node of some name=pre has discharged own resource in advance, and meets this to have done resource pre-
The resource request for the application stayed no longer carries out resource reservation to pri node then dischargeing reserved resource first；Then to this
Pre node does resource request；
2) utilization of chance resource, the resource for allowing to be reserved provide service under certain conditions for other application；
It specifically includes:
Needed for chance resource reaches when reserved container stock number 80%, this chance resource is closed, is no longer other
Using offer resource；
A has first, it is determined that whether there is the node that can satisfy application resource request in cluster, returns to resource, does not enter b then
Step；
B judge chance resource whether meet can distributive condition, meet then enter c walk, otherwise task resource distribution failure, find
One new node carries out resource reservation；
C is calculated according to Preserved node Resource release time and is reserved Resource release time needed for residue；It is current according to the node
Cpu and memory information calculate this and apply the task execution time in present node；Time calculates to finish and walk into d；
If the d task execution time, which is less than needed for residue, reserves Resource release time, the chance resource allocation is given to this
New task is executed subsequently into e；Otherwise distribution failure finds a new node and carries out resource reservation；
E new task is finished, and chance resource is returned reserved container.
7. a kind of computer journey for realizing gradual non-obstruction chance method for obligating resource described in claim 1~6 any one
Sequence.
8. a kind of information data for realizing gradual non-obstruction chance method for obligating resource described in claim 1~6 any one
Processing terminal.
9. a kind of computer readable storage medium, including instruction, when run on a computer, so that computer is executed as weighed
Benefit requires gradual non-obstruction chance method for obligating resource described in 1-6 any one.
10. the gradual non-gradual non-resistance of obstruction chance method for obligating resource described in a kind of realization claim 1~6 any one
Fill in chance resource reservation system, which is characterized in that the gradual non-obstruction chance resource reservation system specifically includes:
Preselection policy unit provides the node of resource reservation for selecting for big task: pressing to all nodes of cluster
Fall to do the node of resource reservation according to Preselection policy filtering；
Priority policy unit locks certain for introducing Priority strategy to filtered node weight priority ranking
A node is the resource reservation node of current big task；
Chance resource allocation policy unit, for using chance resource allocation policy to choosing some weight node and maximum section
Point carries out resource reservation as the node of resource reservation.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810988076.8A CN109150759B (en) | 2018-08-28 | 2018-08-28 | Progressive non-blocking opportunity resource reservation method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810988076.8A CN109150759B (en) | 2018-08-28 | 2018-08-28 | Progressive non-blocking opportunity resource reservation method and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109150759A true CN109150759A (en) | 2019-01-04 |
| CN109150759B CN109150759B (en) | 2022-05-03 |
Family
ID=64828669
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810988076.8A Active CN109150759B (en) | 2018-08-28 | 2018-08-28 | Progressive non-blocking opportunity resource reservation method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109150759B (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110784335A (en) * | 2019-09-19 | 2020-02-11 | 烽火通信科技股份有限公司 | Network element resource reservation system under cloud scene |
| CN112073532A (en) * | 2020-09-15 | 2020-12-11 | 北京字节跳动网络技术有限公司 | Resource allocation method and device |
| WO2022236816A1 (en) * | 2021-05-14 | 2022-11-17 | 华为技术有限公司 | Task allocation method and apparatus |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20060274650A1 (en) * | 2005-05-20 | 2006-12-07 | Satyam Tyagi | Avoiding unnecessary RSVP-based preemptions |
| CN107566443A (en) * | 2017-07-12 | 2018-01-09 | 郑州云海信息技术有限公司 | A kind of distributed resource scheduling method |
-
2018
- 2018-08-28 CN CN201810988076.8A patent/CN109150759B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20060274650A1 (en) * | 2005-05-20 | 2006-12-07 | Satyam Tyagi | Avoiding unnecessary RSVP-based preemptions |
| CN107566443A (en) * | 2017-07-12 | 2018-01-09 | 郑州云海信息技术有限公司 | A kind of distributed resource scheduling method |
Non-Patent Citations (3)
| Title |
|---|
| 李程等: "一种Hadoop YARN的资源调度机制", 《计算机与现代化》 * |
| 林起勋等: "集群资源管理及回填技术", 《科研信息化技术与应用》 * |
| 董春涛等: "Hadoop YARN大数据计算框架及其资源调度机制研究", 《信息通信技术》 * |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110784335A (en) * | 2019-09-19 | 2020-02-11 | 烽火通信科技股份有限公司 | Network element resource reservation system under cloud scene |
| CN110784335B (en) * | 2019-09-19 | 2022-09-02 | 烽火通信科技股份有限公司 | Network element resource reservation system under cloud scene |
| CN112073532A (en) * | 2020-09-15 | 2020-12-11 | 北京字节跳动网络技术有限公司 | Resource allocation method and device |
| CN112073532B (en) * | 2020-09-15 | 2022-09-09 | 北京火山引擎科技有限公司 | Resource allocation method and device |
| WO2022236816A1 (en) * | 2021-05-14 | 2022-11-17 | 华为技术有限公司 | Task allocation method and apparatus |
Also Published As
| Publication number | Publication date |
|---|---|
| CN109150759B (en) | 2022-05-03 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN103605567B (en) | Cloud computing task scheduling method facing real-time demand change | |
| CN107066332A (en) | Distributed system and its dispatching method and dispatching device | |
| CN103699446B (en) | Quantum-behaved particle swarm optimization (QPSO) algorithm based multi-objective dynamic workflow scheduling method | |
| CN110287245B (en) | Method and system for scheduling and executing distributed ETL (extract transform load) tasks | |
| WO2016082370A1 (en) | Distributed node intra-group task scheduling method and system | |
| CN102567086B (en) | Task scheduling method, equipment and system | |
| US9218213B2 (en) | Dynamic placement of heterogeneous workloads | |
| CN109150759A (en) | The gradual non-obstruction chance method for obligating resource of one kind and system | |
| CN103838621B (en) | Method and system for scheduling routine work and scheduling nodes | |
| CN105471985A (en) | Load balance method, cloud platform computing method and cloud platform | |
| CN103793272A (en) | Periodical task scheduling method and periodical task scheduling system | |
| US8171115B2 (en) | Resource equalization for inter- and intra- data center operations | |
| Wided et al. | Load balancing with Job Migration Algorithm for improving performance on grid computing: Experimental Results | |
| CN112817728B (en) | Task scheduling method, network device and storage medium | |
| CN105373426B (en) | A kind of car networking memory aware real time job dispatching method based on Hadoop | |
| CN109582448A (en) | A kind of edge calculations method for scheduling task towards criticality and timeliness | |
| CN103927231A (en) | Data-oriented processing energy consumption optimization dataset distribution method | |
| CN111176840B (en) | Distribution optimization method and device for distributed tasks, storage medium and electronic device | |
| CN104657217B (en) | A kind of cloud environment method for scheduling task based on non-homogeneous grain-size classification | |
| CN107203422A (en) | A kind of job scheduling method towards high-performance calculation cloud platform | |
| Squillante et al. | Threshold-based priority policies for parallel-server systems with affinity scheduling | |
| CN115629865B (en) | Deep learning inference task scheduling method based on edge calculation | |
| CN106201681B (en) | Method for scheduling task based on pre-release the Resources list under Hadoop platform | |
| CN111611076B (en) | Fair distribution method for mobile edge computing shared resources under task deployment constraint | |
| CN115640113A (en) | Multi-plane flexible scheduling method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |