CN109150759B - Progressive non-blocking opportunity resource reservation method and system - Google Patents
Progressive non-blocking opportunity resource reservation method and system Download PDFInfo
- Publication number
- CN109150759B CN109150759B CN201810988076.8A CN201810988076A CN109150759B CN 109150759 B CN109150759 B CN 109150759B CN 201810988076 A CN201810988076 A CN 201810988076A CN 109150759 B CN109150759 B CN 109150759B
- Authority
- CN
- China
- Prior art keywords
- node
- resource
- resource reservation
- nodes
- task
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Expired - Fee Related
Links
- 230000000903 blocking effect Effects 0.000 title claims abstract description 42
- 238000000034 method Methods 0.000 title claims abstract description 35
- 230000000750 progressive effect Effects 0.000 title claims abstract description 33
- 238000013468 resource allocation Methods 0.000 claims description 19
- 238000004422 calculation algorithm Methods 0.000 claims description 10
- 238000001914 filtration Methods 0.000 claims description 7
- 230000015654 memory Effects 0.000 claims description 7
- 238000012545 processing Methods 0.000 claims description 7
- 238000004364 calculation method Methods 0.000 claims description 3
- 238000004891 communication Methods 0.000 claims description 3
- 238000012544 monitoring process Methods 0.000 claims description 3
- 238000005457 optimization Methods 0.000 claims description 3
- 238000012216 screening Methods 0.000 claims description 2
- 239000012634 fragment Substances 0.000 abstract description 5
- 230000010365 information processing Effects 0.000 abstract description 2
- 230000007246 mechanism Effects 0.000 description 10
- 238000010586 diagram Methods 0.000 description 5
- 238000012360 testing method Methods 0.000 description 3
- 238000004590 computer program Methods 0.000 description 2
- 238000002474 experimental method Methods 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 230000008569 process Effects 0.000 description 2
- 239000002699 waste material Substances 0.000 description 2
- 238000004458 analytical method Methods 0.000 description 1
- 238000003556 assay Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000013499 data model Methods 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 230000003247 decreasing effect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000011156 evaluation Methods 0.000 description 1
- 239000000835 fiber Substances 0.000 description 1
- 238000009472 formulation Methods 0.000 description 1
- 238000013467 fragmentation Methods 0.000 description 1
- 238000006062 fragmentation reaction Methods 0.000 description 1
- 230000036541 health Effects 0.000 description 1
- 235000003642 hunger Nutrition 0.000 description 1
- 239000000203 mixture Substances 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000008092 positive effect Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 238000012163 sequencing technique Methods 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 230000037351 starvation Effects 0.000 description 1
- 238000007619 statistical method Methods 0.000 description 1
- 238000010998 test method Methods 0.000 description 1
- 238000012795 verification Methods 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L47/00—Traffic control in data switching networks
- H04L47/70—Admission control; Resource allocation
- H04L47/72—Admission control; Resource allocation using reservation actions during connection setup
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L47/00—Traffic control in data switching networks
- H04L47/70—Admission control; Resource allocation
- H04L47/83—Admission control; Resource allocation based on usage prediction
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
- Computer And Data Communications (AREA)
Abstract
The invention belongs to the technical field of network information processing, and discloses a progressive non-blocking opportunity resource reservation method and a system. The invention solves the problem that the reserved nodes do not allocate resources for other tasks until the reserved nodes meet the reservation request, which can cause a large amount of resource fragments to be generated by the cluster, and the cluster downtime also becomes a high-probability event.
Description
Technical Field
The invention belongs to the technical field of network information processing, and particularly relates to a progressive non-blocking opportunity resource reservation method and system.
Background
For the Hadoop YARN resource scheduling system, when a certain node cannot meet the task requirement, resource reservation is carried out.
Currently, the current state of the art commonly used in the industry is such that: incremental resource reservation and one-time resource reservation.
Resources on a node are preferentially reserved for the application until the cumulatively released free resources meet the application requirements. This way of allocating resources is called incremental resource allocation.
And temporarily abandoning the current node resources until the resources of a certain node meet the requirements of the application program at one time. This allocation formula is called one-time resource allocation.
However, both allocation methods are insufficient, and for incremental resource allocation, a resource reservation mechanism causes resource waste and the utilization rate of cluster resources is low; although the resource request is abandoned when the resource is found to be unable to meet the requirement of a certain application in the disposable resource allocation, the application may never obtain the resource requested by the application and thus never run, namely starvation phenomenon;
in summary, the problems of the prior art are as follows:
(1) the existing resource reservation mechanism does not limit which node is selected to execute resource reservation, the selection of reserved nodes becomes uncontrollable, and the nodes are all reserved possibly to cause cluster downtime.
(2) The existing resource reservation mechanism does not fully utilize the resources in the reservation period, and the reserved nodes do not allocate resources to other tasks until the reservation request is satisfied, which causes a large amount of resource fragments to be generated by the cluster and the resources are not fully utilized.
The difficulty and significance for solving the technical problems are as follows:
on the condition that cluster nodes are prevented from being reserved completely, cluster resources are only released and not distributed, and cluster downtime is caused, nodes which are in accordance with resource reservation need to be selected by formulating corresponding strategies, and whether the strategies are in accordance with the selection of optimal nodes needs to be proved by a large number of experiments.
On the aspect of utilizing reserved resources, the invention innovatively utilizes a non-blocking resource reservation mode, which can provide resource service for other applications for the node and can timely discover whether other nodes have resource release meeting the request or not in a mode of monitoring other nodes for the application. The method has great significance in solving the problems of cluster resource fragmentation and resource utilization rate.
Disclosure of Invention
Aiming at the problems in the prior art, the invention provides a progressive non-blocking opportunity resource reservation method and a progressive non-blocking opportunity resource reservation system. According to the invention, the resource reservation method in Hadoop Yarn when a certain node cannot meet the task requirement is optimized, and the problems of too high cluster resource fragments and too low utilization rate of the whole cluster resources are finally solved, so that the whole task execution efficiency is improved.
The invention is realized in this way, a progressive non-blocking opportunity resource reservation method, which adopts the form of combining Preselection and Priority of the preferred node;
the method specifically comprises the following steps:
when the application program applies for resources to the Hadoop Yarn cluster and cannot meet the requirements, the Yarn can reserve resources for a certain node in the cluster:
firstly, selecting nodes for providing resource reservation for a large task: filtering out nodes which can not be subjected to resource reservation from all the nodes of the cluster according to a Preselection strategy;
secondly, introducing a Priority strategy to sort the Priority of the filtered node weight, and locking a certain node as a resource reservation node of the current large task;
and finally, performing resource reservation on the selected certain weight node and the maximum node by adopting a non-blocking resource reservation algorithm and combining an opportunity resource allocation strategy.
Further, the Preselection strategy includes:
filtering out nodes with the average resource utilization rate larger than 1.0 x 0.8 x N as resource reservation nodes; taking the nodes meeting the residual satisfaction conditions as the input of Priority stages;
the method specifically comprises the following steps:
firstly, creating an empty array Pre _ node for storing preselection results;
then, traversing all nodes, and adding all the nodes meeting the condition that the node load is less than N0.8 1.0 into the Pre _ node [ ] array;
and finally, transmitting the nodes in the Pre _ node [ ] array into the Priority stage.
Further, in the Priority strategy, a mapReduce program is adopted to calculate the sum of the weights of each node in the cluster under all Priority policies; sorting and selecting the nodes with the most appropriate comprehensive performance according to the policy weight sum of each node server;
further, the making of the PriorityPolicy optimal strategy comprises the following steps:
preferentially selecting a local node: preferentially selecting resources on the same node with the task;
selecting nodes with multiple taskles: according to the calculation of 100% resource load, dividing the node resource occupation situation of the maximum task of each node into 4 grades: the total resource rate of the node server occupied by the container used by the maximum task of the node is less than 20%, the total resource rate of the node server occupied by the container used by the maximum task of the node reaches 20% -40%, the total resource rate of the node server occupied by the container used by the maximum task of the node reaches 40% -60%, and the total resource rate of the node server occupied by the container used by the maximum task of the node exceeds 60%;
selecting a node with a smaller ratio of the occupied node server resource amount to the total node server resource amount:
selecting by setting the following weight: node server resource occupancy/total node server resource amount <0.2,
0.4> node server resource occupancy/total node server resource amount >0.2,
0.6> node server resource occupancy/total node server resource amount >0.4,
0.8> node server resource occupancy/total node server resource amount >0.6,
the node server resource occupation amount/node server resource total amount is greater than 0.8.
Further, the weight sum of each node is counted according to the established priority policy optimal strategy, and the counting mode comprises the following steps:
firstly, writing the weight value of each policy of each node into file.
Secondly, using file.txt as the input of a Map program, cutting by a tab \ t according to rows, and reserving the subscript of the cut array as a value from 1 to the end of the array; cutting to obtain input data;
finally, the map result enters a reduce program for processing, the map results of the same key enter the same reduce, and then the weights are added to obtain processing data;
after the third step of Reduce program processing, selecting a node with a large weight as a resource reservation node of the current task for the weight sum obtained after screening the established Priority policy optimization strategy by using the weight number corresponding to the node; and if the nodes with the same weight value exist, selecting the nodes according to the sequence of the heartbeat communication of the nodes.
Further, the opportunity resource allocation policy specifically includes:
1) the application takes a non-blocking mode for resource reservation at a certain node,
and allowing an application to monitor resource changes of other server nodes during resource reservation of a certain node, and allowing the application to cancel resource reservation of the node to carry out self-task to the node when finding that the other certain node releases a resource request meeting the application which is already reserved due to the fact that certain applications running on the node are finished for some reasons. The method specifically comprises the following steps:
firstly, defining a dictionary nodes for storing all node information, wherein each node information comprises: name, res (total amount of node resources), and remaining _ res (amount of node remaining resources), where name — pre represents a node selected in the preference phase, and name — pri represents a resource reservation node selected in the Priority phase;
and traversing the whole dictionary, if nodes [ i ] [ "name" ] ═ pri is satisfied, reserving resources for the node, and otherwise, monitoring the node.
Finally, if some node with name equal to pre releases own resources in advance and meets the resource request of the application which has performed resource reservation, the reserved resources are released first, and the pri node is not reserved any more; a resource request is then made to the pre node.
2) The utilization of opportunity resources allows the reserved resources to provide services for other applications under certain conditions; the method specifically comprises the following steps:
definition 1 reservation container: if a node is a reserved node, it will generate a reserved container whose capacity is the amount of resources required by the reservation task, and after the node resource is released, fill the reserved container with the resources. When the reservation container is full, it is returned to the reservation task.
Define 2 opportunity resources (opp. resource): when a certain reserved node gradually releases resources to fill the reserved container, the resources before the container is not filled are all called as opportunistic resources.
Since the allocation of resources also requires time, when the opportunity resource reaches 80% of the amount of the required reservation coordinator resource, the opportunity resource is blocked, and no resource is provided for other applications.
Firstly, judging whether a cluster has nodes capable of meeting the application resource request, if so, returning the resource, and if not, entering the step b;
b, judging whether the opportunity resources meet allocable conditions (the opportunity resources meet the task request and the opportunity resources are less than 80% of the total reserved resource amount), if so, entering the step c, otherwise, failing to allocate the task resources, and searching a new node for resource reservation;
c, calculating the remaining required reserved resource release time according to the reserved node resource release time; calculating the task execution time of the application at the current node according to the current cpu and the memory information of the node; after the time is calculated, entering step d;
d, if the task execution time is less than the remaining required reserved resource release time, allocating the opportunity resource to the new task for execution and then entering e; otherwise, the allocation fails, and a new node is searched for resource reservation;
e, after the new task is executed, returning the opportunity resource to the reserved container.
It is a further object of the present invention to provide a computer program implementing said method of progressive non-blocking opportunistic resource reservation.
Another object of the present invention is to provide an information data processing terminal implementing the progressive non-blocking opportunistic resource reservation method.
It is another object of the present invention to provide a computer-readable storage medium comprising instructions which, when run on a computer, cause the computer to perform the method for progressive non-blocking opportunistic resource reservation.
Another object of the present invention is to provide a progressive non-blocking opportunistic resource reservation system for implementing the progressive non-blocking opportunistic resource reservation method, where the progressive non-blocking opportunistic resource reservation system specifically includes:
a Preselection policy unit to select a node providing resource reservation for the big task: to clusters
All nodes filter out nodes which can not be used for resource reservation according to a Preselection strategy;
the Priority strategy unit is used for introducing a Priority strategy to sort the Priority of the filtered node weight and lock a certain node as a resource reservation node of the current major task;
and the opportunity resource allocation strategy unit is used for adopting an opportunity resource allocation strategy to reserve resources for the selected certain weight node and the maximum node.
In summary, the advantages and positive effects of the invention are:
the invention provides a Progressive non-blocking Opportunity resource reservation mechanism (PUOR), which adopts a form of combining a Preselection node (preference) and a preference node (preference), firstly selects a node capable of providing resource reservation for a large task, then introduces a preference strategy to sort the Priority of the filtered nodes, locks a certain node as the resource reservation node of the current large task, and finally adopts an Opportunity resource allocation strategy to reserve the resource for the node. Compared with the original resource reservation mechanism of Hadoop, the resource utilization rate of the PUOR algorithm is improved by 7.22%, cluster resource fragments are effectively reduced, and the cluster throughput is increased.
Drawings
Fig. 1 is a flowchart of selecting a node as a resource reservation node in the method for reserving resources in a progressive non-blocking opportunity according to an embodiment of the present invention.
Fig. 2 is a flowchart for node resource reservation according to an embodiment of the present invention.
Fig. 3 is a resource reservation diagram at a node in a non-blocking mode for a current application in the PUOR resource reservation algorithm according to an embodiment of the present invention.
Fig. 4 is a conceptual diagram of proposing opportunistic resources in a PUOR resource reservation mechanism provided by an embodiment of the present invention.
Fig. 5 is a schematic diagram of a system for progressive non-blocking opportunistic resource reservation according to an embodiment of the present invention.
In the figure: 1. a Preselection policy unit; 2. a Priority policy unit; 3. an opportunistic resource allocation policy unit.
Fig. 6 is a comparison diagram of the use of the cluster cpu according to the embodiment of the present invention.
Fig. 7 is a diagram illustrating a comparison between the use of clustered memories according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is further described in detail with reference to the following embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
The present invention is further described below with reference to specific assays.
The progressive non-blocking opportunity resource reservation method provided by the embodiment of the invention comprises the following steps:
a first part: making a reservation node selection
This stage comprises two steps: preselection phase and Priority phase (see fig. 1).
Firstly, filtering out nodes which can not be subjected to resource reservation from all the nodes of the cluster according to a preference strategy, judging the weight values of the nodes through a Priority Policy strategy, and finally selecting a certain weight value and the maximum node as the nodes subjected to resource reservation.
Preselection process:
in a multiprocessor system, the load mean is determined based on the number of cores, and the reasonable load of the Linux system-based server is the number of CPU cores 1.00. If the number of the system cores is represented by N, the system load represented by the load between 0.00 and 1.00 x N is changed, but the system performance is in a smooth state. A load equal to 1.00 × N indicates that the system has no resources left, and if a new task is accepted, the task is queued and the entire system is in an imminent congestion state.
In practical cases, the invention takes an empirical value of 1.0 x 0.8 x N to measure the health criteria of a server. If a node is loaded near 1.0 x 0.8 x N and the remaining 20% of resources cannot satisfy the resource request of a large task, when the resource reservation is made for the task at the node, the load of the node will be decreased linearly from 1.0 x 0.8 x N in the future period of time, and then the large container is allocated once when the resource container satisfying the large task appears, and the load is increased extremely rapidly and is necessarily more than 1.0 x 0.8 x N, and the probability is more than 1.0 x N. The load is changed extremely quickly, the server fluctuation is large as a result, and in order to ensure that the average resource utilization rate of the whole cluster fluctuates smoothly, the node with the average resource utilization rate of more than 1.0 × 0.8 × N is filtered out to serve as the resource reservation node. Nodes that satisfy the remaining conditions are taken as inputs to the Priority stage.
Priority procedures:
the present invention will assign a weight to each priority policy (preferred policy). And calculating the weight sum of each node in the cluster under all Prioritypolicy of the invention by adopting a mapReduce program. And sorting and selecting the nodes with the most suitable comprehensive performance according to the policy weight sum of each node server.
For PriorityPolicy optimization strategy formulation:
1. prioritizing locality nodes
In a Hadoop cluster, the performance consumption of a mobile data block is higher than that of a mobile computing task, so the invention gives priority to resources at the same node as the task at this stage. With respect to a task, node resources in a server cluster can be divided into three types:
2. selecting nodes with small task multiple
If there is a resource container for task request 50G and there are two nodes a and B with the same total amount of resources, occupied amount of resources and released rate of resources, the following is the case:
then, an application requiring 50G resources performs resource reservation by preferentially selecting the node B, because the node a does not release 20G of containers until 40G, and the new resource request of 50Gcontainer can be satisfied until the task occupying 60Gcontainer completes the resource release after executing the task; and the node B has more small tasks and faster task release rate, and has 60G after two containers are released, thereby meeting the requirement of 50 Gcontainers. Therefore, the invention divides the node resource occupation situation of the maximum task of each node into 4 grades according to the calculation of 100 percent of resource load:
3. selecting nodes with smaller ratio of occupied node server resources to total node server resources
If the cluster is performing a small task, such as a map task, then policy 2 may have the result that all weights are the same. Thus, the third policy is supplemented. For example, if the runtime load of each server in the cluster is above 50%, the task maximum request Container is 92G, and the physical memory of each server is 100G, it is a high probability event that the server is not responded to for a long time after being scheduled. And if the physical memory of each server is 200G, the probability that the server is reserved but cannot respond in time is greatly reduced. And meanwhile, the probability of cluster downtime caused by the fact that all nodes of the cluster are reserved is reduced. Therefore, the invention sets the weight as follows:
secondly, the present invention will be further described below with reference to the statistical method of the sum of weights of each node according to the established priority policy optimal policy.
And counting the weight sum of each node according to the established PriorityPolicy optimal strategy in the following way:
1. first, the weight of each policy of each node is written into a file named file.
Txt should have a content format of:
2. txt is used as the input of a Map program, the invention designs to cut by a tab character "\ t" according to rows, and the subscript of an array after cutting is kept as a value from 1 to the end of the array; cutting can obtain the following data:
3. the map result enters a reduce program for processing, the map result with the same key enters the same reduce, and then the weights are added, so that the following data are obtained:
thus, after the third step of Reduce program processing, the weight number corresponding to the node is the sum of the weights screened by the Priority policy preference policy formulated by the invention, and the node with the larger weight is selected as the resource reservation node of the current task. If the nodes with the same weight value exist, the node is selected according to the communication sequence of the node heartbeats, and the node is selected according to the first heartbeat of the node in the cycle.
After the nodes in the first stage of the PUOR algorithm are selected, the nodes which accord with the current task and are used for resource reservation can be selected, and the downtime probability of a server cluster caused by randomly selecting the nodes as the resource reservation nodes can be effectively reduced.
The invention is further described below in connection with progressive non-blocking resource reservation and utilization of opportunistic resources.
A second part: progressive non-blocking resource reservation and utilization of opportunistic resources
The second part of the PUOR algorithm is to make node resource reservations (see fig. 2). The invention adopts a progressive non-blocking resource allocation mode to reserve resources for the nodes, defines the resources which are released by the reserved nodes and used for executing the task request as opportunity resources, and effectively reduces the quantity of resource fragments by reasonably utilizing the opportunity resources.
1.1 progressive non-blocking reservation algorithm
The resource reservation mechanisms adopted by the Hadoop YARN are all blocking resource reservation, when a certain application performs resource reservation at a certain node, whether the resource released by the node meets the request of the certain application is monitored all the time, in the process, the task can not request the resource from other nodes, and the node can not provide the resource for other applications. For this situation, the present invention adopts non-blocking mode for current application in the PUOR resource reservation algorithm to perform resource reservation at a certain node (as shown in fig. 3).
The algorithm of the invention allows an application to monitor the resource change of other server nodes during the resource reservation of a certain node, and allows the application to cancel the resource reservation of the node and turn to the node to execute the self task when finding that other certain node finishes the resource request of the application which is run on the node and releases the resource meeting the resource reservation for some reasons.
1.2 utilization of opportunistic resources
In the original reservation mechanism of the Hadoop YARN, when a certain node performs resource reservation for a certain application, the node cannot provide a resource container for other applications until a resource container meeting the application appears, which causes great resource waste. The invention provides the concept of opportunistic resources in a PUOR resource reservation mechanism (as shown in FIG. 4), and allows the reserved resources to provide services for other applications under certain conditions.
As shown in fig. 5, the system for reserving progressive non-blocking opportunistic resources provided by the embodiment of the present invention specifically includes:
preselection policy unit 1, configured to select a node providing resource reservation for a large task: filtering out nodes which can not be subjected to resource reservation from all the nodes of the cluster according to a Preselection strategy;
a Priority policy unit 2, configured to introduce a Priority policy to sort the Priority of the filtered node weights, and lock a certain node as a resource reservation node of the current large task;
and the opportunity resource allocation strategy unit 3 is used for adopting an opportunity resource allocation strategy to reserve resources for the selected certain weight node and the maximum node.
The invention is further described below in connection with experiments.
Experimental environment and data model:
in order to verify the performance of the non-blocking opportunistic resource reservation mechanism, 5 servers are selected to build an experimental cluster system, and the servers are configured to be 32-core CPUs and 128GB memories. Among them are 1 Name Node, one Resource Manager, 5 datanodes and 5 Node managers. Various types of tasks exist in the real cluster environment, so that the representative load task is selected to be beneficial to accurate verification and evaluation. Three representative MapReduce tasks of Sort, Tera Sort and TestDFSIO are selected. The test procedure types are shown in table 1 below.
The following are test program names and corresponding types:
| test program | Type (B) |
| Sort | Compute intensive |
| TeraSort | Map stage computation intensive, reduce stage I/O intensive |
| TestDFSIO | I/O intensive |
TABLE 1 test program name and type
The function of Sort is to Sort the input files according to Key; the TeraSort program carries out global sequencing on the input files according to Key, and in addition, TeraSort aims at large batch of data; TestDFSIO tests the read-write speed of HDFS.
Submitting tasks to the cluster by adopting a Hi Bench test tool set, constructing 5 task sets with different task sizes, wherein each task set comprises 300 tasks, each test program accounts for 1/3, and the data volume of each task is shown in Table 2
| Sort | TeraSort | TestDFSIO | |
| First group | 6.4GB | 64GB | 32GB |
| Second group | 6.4GB | 6.4GB | 32GB |
| Third group | 640MB | 6.4GB | 3.2GB |
| Fourth group | 640MB | 6.4GB | 3.2GB |
| Fifth group | 640MB | 640MB | 3.2GB |
TABLE 2 test task size
Experimental results and analysis:
and the Yarn FIFO and Capacity schedulers are used as comparison objects, and both schedulers adopt an incremental resource allocation mode.
Fig. 6 cluster cpu uses a comparison graph.
Fig. 7 is a comparison graph of the use of clustered memories.
The experimental results are synthesized, so that when the cluster has large task resource requests and small task resource requests, the cluster can reserve resources for the large tasks which cannot be met, and the utilization rate of the cluster resources can be effectively improved by adopting a PUOR reservation algorithm.
In the above embodiments, the implementation may be wholly or partially realized by software, hardware, firmware, or any combination thereof. When used in whole or in part, can be implemented in a computer program product that includes one or more computer instructions. When loaded or executed on a computer, cause the flow or functions according to embodiments of the invention to occur, in whole or in part. The computer may be a general purpose computer, a special purpose computer, a network of computers, or other programmable device. The computer instructions may be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another, for example, the computer instructions may be transmitted from one website site, computer, server, or data center to another website site, computer, server, or data center via wire (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL), or wireless (e.g., infrared, wireless, microwave, etc.)). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device, such as a server, a data center, etc., that includes one or more of the available media. The usable medium may be a magnetic medium (e.g., floppy Disk, hard Disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., Solid State Disk (SSD)), among others.
The above description is only for the purpose of illustrating the preferred embodiments of the present invention and is not to be construed as limiting the invention, and any modifications, equivalents and improvements made within the spirit and principle of the present invention are intended to be included within the scope of the present invention.
Claims (8)
1. A progressive non-blocking opportunity resource reservation method is characterized in that the progressive non-blocking opportunity resource reservation method adopts a mode of combining Preselection and Priority of a preferred node to select a reserved node first, and then performs resource reservation on the selected node through an opportunity resource allocation strategy;
the method specifically comprises the following steps:
when the application program applies for resources to the Hadoop Yarn cluster and cannot meet the requirements, the Yarn performs resource reservation on a certain node in the cluster:
firstly, selecting nodes for providing resource reservation for a large task: filtering out nodes which can not be subjected to resource reservation from all the nodes of the cluster according to a Preselection strategy;
secondly, introducing a Priority strategy to sort the Priority of the filtered node weight, and locking a certain node as a resource reservation node of the current large task;
finally, adopting a non-blocking resource reservation algorithm and combining an opportunity resource allocation strategy to perform resource reservation on a selected weight node and the maximum node;
the opportunity resource allocation strategy specifically includes:
1) the application takes a non-blocking mode for resource reservation at a certain node,
allowing an application to monitor resource changes of other server nodes during resource reservation of a certain node, and allowing the application to cancel resource reservation of the node and turn to the node to execute self tasks when finding that the other certain node finishes certain applications running on the node for some reasons and releases a resource request meeting the application which has already made resource reservation;
the method specifically comprises the following steps:
firstly, defining a dictionary nodes for storing all node information, wherein each node information comprises: name, res node resource total amount, and remaining resource amount of the remaining _ res node, where name pre represents a node selected in the preference phase, and name pri represents a resource reservation node selected in the Priority phase;
then traversing the whole dictionary, if satisfying nodes [ i ] [ "name" ] ═ pri, then reserving resources for the node, otherwise, monitoring the node;
finally, if some node with name equal to pre releases own resources in advance and meets the resource request of the application which has performed resource reservation, the reserved resources are released first, and the pri node is not reserved any more; then making a resource request to the pre node;
2) the utilization of opportunity resources allows the reserved resources to provide services for other applications under certain conditions;
the method specifically comprises the following steps:
when meeting resources reach 80% of the amount of the required reserved container resources, the meeting resources are forbidden and no resources are provided for other applications;
firstly, judging whether a cluster has nodes capable of meeting the application resource request, if so, returning the resource, and if not, entering the step b;
b, judging whether the opportunity resource meets allocable conditions, if so, entering the step c, otherwise, failing to allocate the task resource, and searching a new node for resource reservation;
c, calculating the remaining required reserved resource release time according to the reserved node resource release time; calculating the task execution time of the application at the current node according to the current cpu and the memory information of the node; after the time is calculated, entering step d;
d, if the task execution time is less than the remaining required reserved resource release time, allocating the opportunity resource to the new task for execution and then entering e; otherwise, the allocation fails, and a new node is searched for resource reservation;
e, after the new task is executed, returning the opportunity resource to the reserved container.
2. The method of progressive non-blocking opportunity resource reservation according to claim 1, wherein the Preselection policy comprises: filtering out nodes with the average resource utilization rate larger than 1.0 x 0.8 x N as resource reservation nodes; taking the nodes meeting the remaining conditions as the input of the Priority stage;
the method specifically comprises the following steps:
firstly, creating an empty array Pre _ node for storing preselection results;
then, traversing all nodes, and adding all the nodes meeting the condition that the node load is less than N0.8 1.0 into the Pre _ node [ ] array;
and finally, transmitting the nodes in the Pre _ node [ ] array into the Priority stage.
3. The method of progressive non-blocking opportunistic resource reservation according to claim 1, wherein the Priority policy comprises: calculating the weight sum of preselection stage nodes in the cluster under all priority policies by adopting a mapReduce program; and sorting and selecting the nodes with the most suitable comprehensive performance according to the policy weight sum of each node server.
4. The progressive non-blocking opportunistic resource reservation method of claim 3,
the establishment of the Priority strategy comprises the following steps:
preferentially selecting a local node: preferentially selecting resources on the same node with the task;
selecting nodes with multiple taskles: according to the calculation of 100% resource load, dividing the node resource occupation situation of the maximum task of each node into 4 grades: the total resource rate of the node server occupied by the container used by the maximum task of the node is less than 20%, the total resource rate of the node server occupied by the container used by the maximum task of the node reaches 20% -40%, the total resource rate of the node server occupied by the container used by the maximum task of the node reaches 40% -60%, and the total resource rate of the node server occupied by the container used by the maximum task of the node exceeds 60%;
selecting a node with a smaller ratio of the occupied node server resource amount to the total node server resource amount:
selecting by setting the following weight:
node server resource occupancy/total node server resource amount <0.2,
0.4> node server resource occupancy/total node server resource amount >0.2,
0.6> node server resource occupancy/total node server resource amount >0.4,
0.8> node server resource occupancy/total node server resource amount >0.6,
the node server resource occupation amount/node server resource total amount is greater than 0.8.
5. The progressive non-blocking opportunistic resource reservation method of claim 3,
and counting the weight sum of each node according to the established Priority strategy, wherein the counting mode comprises the following steps:
firstly, writing the weight value of each policy of each node into file.
Then, using file.txt as the input of a Map program, cutting by a tab character "\ t" according to a row, and keeping the index of the cut array as a value from 1 to the end of the array; then the following steps are carried out: "cut to get input data;
finally, the map result enters a reduce program for processing, the map results of the same key enter the same reduce, and then the weights are added to obtain processing data;
after the third step of Reduce program processing, selecting a node with a large weight as a resource reservation node of the current task for the weight sum obtained after screening the established Priority policy optimization strategy by using the weight number corresponding to the node; and if the nodes with the same weight value exist, selecting the nodes according to the sequence of the heartbeat communication of the nodes.
6. An information data processing terminal implementing the progressive non-blocking opportunistic resource reservation method of any one of claims 1 to 5.
7. A computer readable storage medium comprising instructions which, when run on a computer, cause the computer to perform the method of progressive non-blocking opportunistic resource reservation according to any one of claims 1-5.
8. A progressive non-blocking opportunistic resource reservation system implementing the progressive non-blocking opportunistic resource reservation method of any one of claims 1 to 5, characterized in that the progressive non-blocking opportunistic resource reservation system specifically comprises:
a Preselection policy unit to select a node providing resource reservation for the big task: filtering out nodes which can not be subjected to resource reservation from all the nodes of the cluster according to a Preselection strategy;
the Priority strategy unit is used for introducing a Priority strategy to sort the Priority of the filtered node weight and lock a certain node as a resource reservation node of the current major task;
and the opportunity resource allocation strategy unit is used for adopting an opportunity resource allocation strategy to reserve resources for the selected certain weight node and the maximum node.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810988076.8A CN109150759B (en) | 2018-08-28 | 2018-08-28 | Progressive non-blocking opportunity resource reservation method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810988076.8A CN109150759B (en) | 2018-08-28 | 2018-08-28 | Progressive non-blocking opportunity resource reservation method and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109150759A CN109150759A (en) | 2019-01-04 |
| CN109150759B true CN109150759B (en) | 2022-05-03 |
Family
ID=64828669
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810988076.8A Expired - Fee Related CN109150759B (en) | 2018-08-28 | 2018-08-28 | Progressive non-blocking opportunity resource reservation method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109150759B (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110784335B (en) * | 2019-09-19 | 2022-09-02 | 烽火通信科技股份有限公司 | Network element resource reservation system under cloud scene |
| CN112073532B (en) * | 2020-09-15 | 2022-09-09 | 北京火山引擎科技有限公司 | Resource allocation method and device |
| CN116724294A (en) * | 2021-05-14 | 2023-09-08 | 华为技术有限公司 | Task allocation method and device |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7636302B2 (en) * | 2005-05-20 | 2009-12-22 | Cisco Technology, Inc. | Avoiding unnecessary RSVP-based preemptions |
| CN107566443A (en) * | 2017-07-12 | 2018-01-09 | 郑州云海信息技术有限公司 | A kind of distributed resource scheduling method |
-
2018
- 2018-08-28 CN CN201810988076.8A patent/CN109150759B/en not_active Expired - Fee Related
Also Published As
| Publication number | Publication date |
|---|---|
| CN109150759A (en) | 2019-01-04 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| WO2021208546A1 (en) | Multi-dimensional resource scheduling method in kubernetes cluster architecture system | |
| CN110287245B (en) | Method and system for scheduling and executing distributed ETL (extract transform load) tasks | |
| JP2559915B2 (en) | Load balancing system | |
| CN109564528B (en) | System and method for computing resource allocation in distributed computing | |
| WO2017166803A1 (en) | Resource scheduling method and device | |
| CN109150759B (en) | Progressive non-blocking opportunity resource reservation method and system | |
| CN107430529B (en) | Load balancing device for large-scale memory database | |
| CN110058940B (en) | Data processing method and device in multi-thread environment | |
| US7920282B2 (en) | Job preempt set generation for resource management | |
| KR101886072B1 (en) | Dynamic job scheduling system and method for supporting real-time stream data processing in distributed in-memory environment | |
| CN116560860B (en) | Real-time optimization adjustment method for resource priority based on machine learning | |
| CN106502790A (en) | A kind of task distribution optimization method based on data distribution | |
| CN117519953B (en) | Separated memory management method for server-oriented non-perception calculation | |
| CN110178119B (en) | Method, device and storage system for processing service request | |
| CN110928649A (en) | Resource scheduling method and device | |
| CN109298949B (en) | Resource scheduling system of distributed file system | |
| CN105069019B (en) | Database update method free of discontinuities and system | |
| CN116260876A (en) | AI application scheduling method and device based on K8s and electronic equipment | |
| CN115878910A (en) | Line query method, device and storage medium | |
| CN114461356B (en) | Control method for number of processes of scheduler and IaaS cloud platform scheduling system | |
| CN114860449A (en) | Data processing method, device, equipment and storage medium | |
| CN112311695B (en) | On-chip bandwidth dynamic allocation method and system | |
| CN115509749A (en) | Task execution method and device, storage medium and electronic equipment | |
| CN112416538A (en) | Multilayer architecture and management method of distributed resource management framework | |
| CN111338782A (en) | Node allocation method based on competition perception and oriented to shared burst data caching |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20220503 |
|
| CF01 | Termination of patent right due to non-payment of annual fee |