CN112272203A - Cluster service node selection method, system, terminal and storage medium - Google Patents
Cluster service node selection method, system, terminal and storage medium Download PDFInfo
- Publication number
- CN112272203A CN112272203A CN202010988230.9A CN202010988230A CN112272203A CN 112272203 A CN112272203 A CN 112272203A CN 202010988230 A CN202010988230 A CN 202010988230A CN 112272203 A CN112272203 A CN 112272203A
- Authority
- CN
- China
- Prior art keywords
- task
- node
- resource
- amount
- preselected
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000010187 selection method Methods 0.000 title description 3
- 238000000034 method Methods 0.000 claims abstract description 25
- 238000012216 screening Methods 0.000 claims abstract description 21
- 238000004364 calculation method Methods 0.000 claims description 11
- 238000004458 analytical method Methods 0.000 claims description 3
- 238000004590 computer program Methods 0.000 claims description 3
- 239000002699 waste material Substances 0.000 abstract description 6
- 238000004891 communication Methods 0.000 description 5
- 238000010586 diagram Methods 0.000 description 5
- 238000007726 management method Methods 0.000 description 4
- 230000008878 coupling Effects 0.000 description 3
- 238000010168 coupling process Methods 0.000 description 3
- 238000005859 coupling reaction Methods 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 238000006467 substitution reaction Methods 0.000 description 3
- 230000006870 function Effects 0.000 description 2
- 238000013178 mathematical model Methods 0.000 description 2
- 230000007246 mechanism Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012544 monitoring process Methods 0.000 description 2
- 238000012545 processing Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000007812 deficiency Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000011156 evaluation Methods 0.000 description 1
- 230000008569 process Effects 0.000 description 1
- 238000013468 resource allocation Methods 0.000 description 1
- 238000012163 sequencing technique Methods 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/104—Peer-to-peer [P2P] networks
- H04L67/1074—Peer-to-peer [P2P] networks for supporting data block transmission mechanisms
Abstract
The invention provides a method, a system, a terminal and a storage medium for selecting a cluster service node, comprising the following steps: analyzing the required resource quantity of the task to be executed; collecting the total resource quantity of all nodes in the cluster, and screening the nodes with the total resource quantity not less than the required resource quantity as preselected nodes; acquiring the current task execution progress and task resource occupation condition of a preselected node; calculating the time required for the idle resource amount of the preselected node to reach the required resource amount according to the task execution progress and the task resource occupation condition; and screening out a preselected node with the shortest required time as an optimal node, and distributing the task to be executed to the optimal node. The invention can ensure that a new task can be preferentially scheduled to the node with shorter task remaining operation time under the condition that the overall resource quantity of different nodes is not large, thereby operating at the target node as early as possible, reducing the resource waste of the node and improving the overall operation efficiency of the cluster.
Description
Technical Field
The invention relates to the technical field of cluster resource allocation, in particular to a method, a system, a terminal and a storage medium for selecting a cluster service node.
Background
In YARN clusters, task scheduling typically employs incremental allocation rules. The increment allocation rule is mainly that in the stage of selecting the nodes for scheduling, the nodes with high resource idle rate are preferentially selected for scheduling. If the node free memory is 8G, the task applies for the memory 10G, the 8G on the node is scheduled by the task, and other tasks can not reuse the resource. The task will continue to wait until the idle resources on the node reach 10G.
Under the condition that the difference of each node resource in the cluster is not large, if only the size of the idle resource of the node is considered, the system defaults to schedule the task to the node with higher idle resource amount. However, if the amount of idle resources still cannot meet the operation requirement of the task, the submitted task still occupies all the idle resources of the node (other tasks cannot be utilized) in an incremental allocation manner, and resource waste is caused. Obviously, the node allocation at this time is not reasonable.
Disclosure of Invention
In view of the above-mentioned deficiencies of the prior art, the present invention provides a method, a system, a terminal and a storage medium for selecting a cluster service node, so as to solve the above-mentioned technical problems.
In a first aspect, the present invention provides a method for selecting a cluster service node, including:
analyzing the required resource quantity of the task to be executed;
collecting the total resource quantity of all nodes in the cluster, and screening the nodes with the total resource quantity not less than the required resource quantity as preselected nodes;
acquiring the current task execution progress and task resource occupation condition of a preselected node;
calculating the time required for the idle resource amount of the preselected node to reach the required resource amount according to the task execution progress and the task resource occupation condition;
and screening out a preselected node with the shortest required time as an optimal node, and distributing the task to be executed to the optimal node.
Further, the method further comprises:
acquiring the amount of idle resources of all nodes in a cluster;
judging whether an optimal node with the idle resource amount reaching the required resource amount exists or not:
and if so, distributing the task to be executed to the optimal node.
Further, the acquiring the existing task execution progress and the task resource occupation condition of the preselected node includes:
and acquiring the total execution time length, the executed time length and the occupied resource amount of the task of the existing task in the preselected node.
Further, the calculating the required time for the idle resource amount of the preselected node to reach the required resource amount according to the task execution progress and the task resource occupation condition includes:
calculating the remaining time length of each task according to the total execution time length and the executed time length of each task in the preselected nodes;
calculating the amount of idle resources according to the total resource amount of the preselected nodes and the resource occupation amount of each task;
calculating the task combination of which the sum of the resource occupation amount and the idle resource amount meets the required resource amount according to the resource occupation amount and the idle resource amount of each task;
taking the maximum value of the remaining time of each task in the task combination as the remaining time of the task combination;
and selecting the task combination with the shortest residual time length as the optimal combination of the preselected nodes, and outputting the residual time length of the optimal combination as the required time.
In a second aspect, the present invention provides a system for selecting a cluster service node, including:
the task analysis unit is configured for analyzing the required resource quantity of the task to be executed;
the resource acquisition unit is used for acquiring the total resource quantity of all nodes in the cluster and screening the nodes with the total resource quantity not lower than the required resource quantity as preselected nodes;
the progress acquisition unit is configured for acquiring the existing task execution progress and the task resource occupation condition of the preselected node;
the time calculation unit is configured to calculate the required time for the idle resource amount of the preselected node to reach the required resource amount according to the task execution progress and the task resource occupation condition;
and the node screening unit is configured to screen out a preselected node with the shortest required time as an optimal node and distribute the task to be executed to the optimal node.
Further, the system further comprises:
the idle acquisition unit is configured for acquiring the idle resource amount of all nodes in the cluster;
the idle screening unit is configured to judge whether an optimal node with the idle resource amount reaching the required resource amount exists or not;
and the idle allocation unit is configured to allocate the task to be executed to the optimal node if the optimal node with the idle resource amount reaching the required resource amount exists.
Further, the progress acquisition unit includes:
and the acquisition module is configured for acquiring the total execution time length, the executed time length and the occupied resource amount of the task of the existing task in the preselected node.
Further, the time calculation unit includes:
the residual calculation module is configured for calculating the residual time length of each task according to the total execution time length and the executed time length of each task in the preselected node;
the idle calculation module is configured for calculating the idle resource amount according to the total resource amount of the preselected nodes and the resource occupation amount of each task;
the task combination module is configured for calculating the task combination of which the sum of the resource occupation amount and the idle resource amount reaches the required resource amount according to the resource occupation amount and the idle resource amount of each task;
the combined time length module is configured to take the maximum value of the remaining time length of each task in the task combination as the remaining time length of the task combination;
and the combination screening module is configured to select the task combination with the shortest remaining time as the optimal combination of the preselected nodes, and output the remaining time of the optimal combination as the required time.
In a third aspect, a terminal is provided, including:
a processor, a memory, wherein,
the memory is used for storing a computer program which,
the processor is used for calling and running the computer program from the memory so as to make the terminal execute the method of the terminal.
In a fourth aspect, a computer storage medium is provided having stored therein instructions that, when executed on a computer, cause the computer to perform the method of the above aspects.
The beneficial effect of the invention is that,
according to the cluster service node selection method, the cluster service node selection system, the terminal and the storage medium, the scheduling mode of the YARN scheduler is improved, so that the task submitted in the cluster can select the resource node more reasonably, and the problem of resource waste caused by a single node allocation mechanism is avoided. Meanwhile, a decision model is established according to two factors of task duration and task resource usage to determine which node the new task is to be dispatched to finally, and the new task can be preferentially dispatched to the node with shorter task remaining operation duration under the condition that the overall resource amount of different nodes is not large, so that the new task can be operated at a target node as early as possible, the resource waste of the node is reduced, and the overall operation efficiency of the cluster is improved.
In addition, the invention has reliable design principle, simple structure and very wide application prospect.
Drawings
In order to more clearly illustrate the embodiments or technical solutions in the prior art of the present invention, the drawings used in the description of the embodiments or prior art will be briefly described below, and it is obvious for those skilled in the art that other drawings can be obtained based on these drawings without creative efforts.
FIG. 1 is a schematic flow diagram of a method of one embodiment of the invention.
FIG. 2 is a schematic block diagram of a system of one embodiment of the present invention.
Fig. 3 is a schematic structural diagram of a terminal according to an embodiment of the present invention.
Detailed Description
In order to make those skilled in the art better understand the technical solution of the present invention, the technical solution in the embodiment of the present invention will be clearly and completely described below with reference to the drawings in the embodiment of the present invention, and it is obvious that the described embodiment is only a part of the embodiment of the present invention, and not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The following explains key terms appearing in the present invention.
The YARN is a resource management system in Hadoop, is a universal resource management module, can perform resource management and scheduling for various application programs, and mainly comprises a resource manager, an ApplicationMaster and a node manager. YARN is not limited to MapReduce framework, but can be used by other frameworks, such as Spark, etc
Resource managers are responsible for the unified management and allocation of all resources in the cluster in YARN, and it receives resource report information from each node and allocates the information to each application program (actually, ApplicationMaster) according to a certain policy.
FIG. 1 is a schematic flow diagram of a method of one embodiment of the invention. The implementation subject of fig. 1 may be a system for selecting a cluster service node.
As shown in fig. 1, the method includes:
and 150, screening out a preselected node with the shortest required time as an optimal node, and distributing the task to be executed to the optimal node.
Specifically, the method for selecting a cluster service node includes:
and S1, analyzing the required resource quantity of the task to be executed.
And S2, collecting the total resource quantity of all nodes in the cluster, and screening the nodes with the total resource quantity not less than the required resource quantity as preselected nodes.
And acquiring attribute indexes of the cluster nodes through a back-end monitoring system, wherein the attribute indexes mainly comprise the total amount of resources owned by the nodes and the total amount of resources unoccupied on the nodes. For example: the back-end monitoring system collects information such as total resource amount (total memory amount is 64G, CPU total core number is 6) owned by the node1 node and node1 node resource idle rate (percentage of unused node resources in the total node resources) in real time.
And transmitting the indexes into a mathematical model for evaluation, and comparing the total resource amount of the nodes with the resource amount of the newly submitted task application to confirm whether the nodes are the pre-selected nodes. For example: the submitted task is network service, and the applied task resource is 10G memory. If the total resource amount of the node1 node of the cluster is 20G (greater than the application resource), the resource requirement of the new task is met, and the node1 node can be considered as a preselected node. If the total resource of the node1 node is 7G memory, the node is determined to be a non-preselected node.
All preselected nodes within the cluster are updated into a list of preselected nodes and the list is loaded into a Resource Manager (RM), for example: the list of preselected nodes including node1 node is loaded directly into the Resource Manager (RM).
And S3, acquiring the existing task execution progress and task resource occupation condition of the preselected node.
And acquiring the running time length of all tasks, the expected running time length, the resource usage and other data from the preselected nodes.
And S4, calculating the time required for the idle resource amount of the preselected node to reach the required resource amount according to the task execution progress and the task resource occupation condition.
And calculating and comparing the residual running time (the predicted running time minus the run-time) of all task combinations through a mathematical model to confirm the optimal task combination in the node (the sum of the use amount of the task resources in the combination and the idle resource amount of the node is greater than the application amount of the new task resources), and updating the combination into a scheduling list. The scheduling list containing the optimal combination of all the preselected nodes is then sent to the scheduler. For example:
(1) and collecting task duration indexes on the preselected nodes through a front-end system, wherein the task duration indexes mainly comprise the running time of a task, the time required by a user to estimate the task and the amount of task resources. For example, the running time of the hdfs service is 20h, the time required for the user to expect the task to be completed is 72h, and the occupied memory is 4G.
(2) And determining task combinations meeting the application amount of the new task resources according to the use amount of the task resources, wherein the sum of the resource amount occupied by each combination and the node idle resource amount is larger than the resource demand amount of the new task. For example, a certain combination of tasks for the node1 node includes two tasks, hdfs and mysql. The hdfs service memory uses 4G, the MySQL service memory uses 2G, and the node idle memory resource is 2G. The newly submitted task memory application amount is 8G, the task combination just meets the requirements of the new task, and the task combination is the target combination of the preselected nodes.
(3) And calculating and comparing the residual running time of all task combination tasks to determine the optimal task combination of the preselected nodes. For example, there are two task combinations on node1 (combination A includes tasks 1, 2 and combination B includes tasks 3, 4, 5). Through calculation, the estimated time length of the task 1 in the combined A is 6h, the running time length is 4h, the residual running time length of the task is 2h, and the residual running time length of the task 2 is calculated to be 3h in the same way. The remaining operating time of the combination A was taken to be a maximum of 3 h. And calculating the residual operation time length of the node B to be 4h according to the same mode, and selecting the combination (combination A) with the smaller residual operation time length in the two combinations of the node A and the node B as the optimal combination of the preselected nodes. And the remaining operation time of the optimal combination of each preselected node is the required time for the idle resource quantity of the preselected node to reach the resource quantity required by the task to be executed.
And S5, screening out the preselected node with the shortest required time as an optimal node, and distributing the task to be executed to the optimal node.
And acquiring the optimal combination of all the preselected nodes. For example, the cluster has three preselected nodes, node1, node2 and node3, and the optimal combination of the three nodes is A, B and C respectively.
And sequencing the obtained optimal combination, selecting the task combination with the shortest task residual time in the cluster, and determining the node to which the combination belongs as the optimal node. For example: the task remaining time of the node1 node optimal combination is 5 h; the remaining time length of the optimal combined task of the node3 node is 7 h; node2 node is then 8 h. Obviously, the task residual time of the node1 in the three nodes is the shortest, and the node1 node is determined to be the optimal node of the cluster.
The newly submitted task is assigned to the optimal node.
As shown in fig. 2, the system 200 includes:
a task analysis unit 210 configured to analyze a required resource amount of a task to be executed;
the resource acquisition unit 220 is configured to acquire the total resource amount of all nodes in the cluster, and screen the nodes with the total resource amount not less than the required resource amount as preselected nodes;
the progress acquisition unit 230 is configured to acquire an existing task execution progress and a task resource occupation condition of a preselected node;
a time calculating unit 240 configured to calculate a required time for the amount of idle resources of the preselected node to reach the amount of required resources according to the task execution progress and the task resource occupation condition;
the node screening unit 250 is configured to screen out a preselected node with the shortest required time as an optimal node, and allocate the task to be executed to the optimal node.
Optionally, as an embodiment of the present invention, the system further includes:
the idle acquisition unit is configured for acquiring the idle resource amount of all nodes in the cluster;
the idle screening unit is configured to judge whether an optimal node with the idle resource amount reaching the required resource amount exists or not;
and the idle allocation unit is configured to allocate the task to be executed to the optimal node if the optimal node with the idle resource amount reaching the required resource amount exists.
Optionally, as an embodiment of the present invention, the progress acquisition unit includes:
and the acquisition module is configured for acquiring the total execution time length, the executed time length and the occupied resource amount of the task of the existing task in the preselected node.
Optionally, as an embodiment of the present invention, the time calculating unit includes:
the residual calculation module is configured for calculating the residual time length of each task according to the total execution time length and the executed time length of each task in the preselected node;
the idle calculation module is configured for calculating the idle resource amount according to the total resource amount of the preselected nodes and the resource occupation amount of each task;
the task combination module is configured for calculating the task combination of which the sum of the resource occupation amount and the idle resource amount reaches the required resource amount according to the resource occupation amount and the idle resource amount of each task;
the combined time length module is configured to take the maximum value of the remaining time length of each task in the task combination as the remaining time length of the task combination;
and the combination screening module is configured to select the task combination with the shortest remaining time as the optimal combination of the preselected nodes, and output the remaining time of the optimal combination as the required time.
Fig. 3 is a schematic structural diagram of a terminal 300 according to an embodiment of the present invention, where the terminal 300 may be configured to execute the method for selecting a cluster service node according to the embodiment of the present invention.
Among them, the terminal 300 may include: a processor 310, a memory 320, and a communication unit 330. The components communicate via one or more buses, and those skilled in the art will appreciate that the architecture of the servers shown in the figures is not intended to be limiting, and may be a bus architecture, a star architecture, a combination of more or less components than those shown, or a different arrangement of components.
The memory 320 may be used for storing instructions executed by the processor 310, and the memory 320 may be implemented by any type of volatile or non-volatile storage terminal or combination thereof, such as Static Random Access Memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The executable instructions in memory 320, when executed by processor 310, enable terminal 300 to perform some or all of the steps in the method embodiments described below.
The processor 310 is a control center of the storage terminal, connects various parts of the entire electronic terminal using various interfaces and lines, and performs various functions of the electronic terminal and/or processes data by operating or executing software programs and/or modules stored in the memory 320 and calling data stored in the memory. The processor may be composed of an Integrated Circuit (IC), for example, a single packaged IC, or a plurality of packaged ICs connected with the same or different functions. For example, the processor 310 may include only a Central Processing Unit (CPU). In the embodiment of the present invention, the CPU may be a single operation core, or may include multiple operation cores.
A communication unit 330, configured to establish a communication channel so that the storage terminal can communicate with other terminals. And receiving user data sent by other terminals or sending the user data to other terminals.
The present invention also provides a computer storage medium, wherein the computer storage medium may store a program, and the program may include some or all of the steps in the embodiments provided by the present invention when executed. The storage medium may be a magnetic disk, an optical disk, a read-only memory (ROM) or a Random Access Memory (RAM).
Therefore, the invention ensures that the tasks submitted in the cluster can select the resource nodes more reasonably by improving the scheduling mode of the YARN scheduler, and avoids the problem of resource waste caused by a single node allocation mechanism. Meanwhile, a decision model is constructed according to two factors of task duration and task resource usage to determine which node to schedule the new task finally, and it is ensured that the new task can be scheduled to the node with the shorter task remaining operation duration preferentially under the condition that the overall resource amount of different nodes is not much different, so that the new task can be operated at a target node as early as possible, the resource waste of the node is reduced, and the overall operation efficiency of the cluster is improved.
Those skilled in the art will readily appreciate that the techniques of the embodiments of the present invention may be implemented as software plus a required general purpose hardware platform. Based on such understanding, the technical solutions in the embodiments of the present invention may be embodied in the form of a software product, where the computer software product is stored in a storage medium, such as a usb disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and the like, and the storage medium can store program codes, and includes instructions for enabling a computer terminal (which may be a personal computer, a server, or a second terminal, a network terminal, and the like) to perform all or part of the steps of the method in the embodiments of the present invention.
The same and similar parts in the various embodiments in this specification may be referred to each other. Especially, for the terminal embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant points can be referred to the description in the method embodiment.
In the embodiments provided in the present invention, it should be understood that the disclosed system and method can be implemented in other ways. For example, the above-described system embodiments are merely illustrative, and for example, the division of the units is only one logical functional division, and other divisions may be realized in practice, for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, systems or units, and may be in an electrical, mechanical or other form.
The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.
In addition, functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit.
Although the present invention has been described in detail by referring to the drawings in connection with the preferred embodiments, the present invention is not limited thereto. Various equivalent modifications or substitutions can be made on the embodiments of the present invention by those skilled in the art without departing from the spirit and scope of the present invention, and these modifications or substitutions are within the scope of the present invention/any person skilled in the art can easily conceive of the changes or substitutions within the technical scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.
Claims (10)
1. A method for selecting a cluster service node is characterized by comprising the following steps:
analyzing the required resource quantity of the task to be executed;
collecting the total resource quantity of all nodes in the cluster, and screening the nodes with the total resource quantity not less than the required resource quantity as preselected nodes;
acquiring the current task execution progress and task resource occupation condition of a preselected node;
calculating the time required for the idle resource amount of the preselected node to reach the required resource amount according to the task execution progress and the task resource occupation condition;
and screening out a preselected node with the shortest required time as an optimal node, and distributing the task to be executed to the optimal node.
2. The method of claim 1, further comprising:
acquiring the amount of idle resources of all nodes in a cluster;
judging whether an optimal node with the idle resource amount reaching the required resource amount exists or not:
and if so, distributing the task to be executed to the optimal node.
3. The method of claim 1, wherein the obtaining of the existing task execution progress and task resource occupation of the preselected node comprises:
and acquiring the total execution time length, the executed time length and the occupied resource amount of the task of the existing task in the preselected node.
4. The method according to claim 3, wherein the calculating the required time for the amount of free resources of the preselected node to reach the amount of required resources according to the task execution progress and the task resource occupation condition comprises:
calculating the remaining time length of each task according to the total execution time length and the executed time length of each task in the preselected nodes;
calculating the amount of idle resources according to the total resource amount of the preselected nodes and the resource occupation amount of each task;
calculating the task combination of which the sum of the resource occupation amount and the idle resource amount meets the required resource amount according to the resource occupation amount and the idle resource amount of each task;
taking the maximum value of the remaining time of each task in the task combination as the remaining time of the task combination;
and selecting the task combination with the shortest residual time length as the optimal combination of the preselected nodes, and outputting the residual time length of the optimal combination as the required time.
5. A cluster service node selection system, comprising:
the task analysis unit is configured for analyzing the required resource quantity of the task to be executed;
the resource acquisition unit is used for acquiring the total resource quantity of all nodes in the cluster and screening the nodes with the total resource quantity not lower than the required resource quantity as preselected nodes;
the progress acquisition unit is configured for acquiring the existing task execution progress and the task resource occupation condition of the preselected node;
the time calculation unit is configured to calculate the required time for the idle resource amount of the preselected node to reach the required resource amount according to the task execution progress and the task resource occupation condition;
and the node screening unit is configured to screen out a preselected node with the shortest required time as an optimal node and distribute the task to be executed to the optimal node.
6. The system of claim 5, further comprising:
the idle acquisition unit is configured for acquiring the idle resource amount of all nodes in the cluster;
the idle screening unit is configured to judge whether an optimal node with the idle resource amount reaching the required resource amount exists or not;
and the idle allocation unit is configured to allocate the task to be executed to the optimal node if the optimal node with the idle resource amount reaching the required resource amount exists.
7. The system of claim 5, wherein the progress acquisition unit comprises:
and the acquisition module is configured for acquiring the total execution time length, the executed time length and the occupied resource amount of the task of the existing task in the preselected node.
8. The system of claim 7, wherein the time calculation unit comprises:
the residual calculation module is configured for calculating the residual time length of each task according to the total execution time length and the executed time length of each task in the preselected node;
the idle calculation module is configured for calculating the idle resource amount according to the total resource amount of the preselected nodes and the resource occupation amount of each task;
the task combination module is configured for calculating the task combination of which the sum of the resource occupation amount and the idle resource amount reaches the required resource amount according to the resource occupation amount and the idle resource amount of each task;
the combined time length module is configured to take the maximum value of the remaining time length of each task in the task combination as the remaining time length of the task combination;
and the combination screening module is configured to select the task combination with the shortest remaining time as the optimal combination of the preselected nodes, and output the remaining time of the optimal combination as the required time.
9. A terminal, comprising:
a processor;
a memory for storing instructions for execution by the processor;
wherein the processor is configured to perform the method of any one of claims 1-4.
10. A computer-readable storage medium, in which a computer program is stored which, when being executed by a processor, carries out the method according to any one of claims 1-4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010988230.9A CN112272203B (en) | 2020-09-18 | 2020-09-18 | Cluster service node selection method, system, terminal and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010988230.9A CN112272203B (en) | 2020-09-18 | 2020-09-18 | Cluster service node selection method, system, terminal and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112272203A true CN112272203A (en) | 2021-01-26 |
| CN112272203B CN112272203B (en) | 2022-06-14 |
Family
ID=74349994
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010988230.9A Active CN112272203B (en) | 2020-09-18 | 2020-09-18 | Cluster service node selection method, system, terminal and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112272203B (en) |
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113220428A (en) * | 2021-04-23 | 2021-08-06 | 复旦大学 | Dynamic task scheduling algorithm for real-time requirements of cloud computing system |
| CN113535332A (en) * | 2021-08-11 | 2021-10-22 | 北京字节跳动网络技术有限公司 | Cluster resource scheduling method and device, computer equipment and storage medium |
| CN113742064A (en) * | 2021-08-06 | 2021-12-03 | 苏州浪潮智能科技有限公司 | Resource arrangement method, system, equipment and medium for server cluster |
| CN114143174A (en) * | 2021-11-30 | 2022-03-04 | 深信服科技股份有限公司 | Node repairing method, device, equipment and readable storage medium |
| CN114201306A (en) * | 2022-02-16 | 2022-03-18 | 长沙市规划勘测设计研究院 | Multi-dimensional geographic space entity distribution method and system based on load balancing technology |
| CN114363328A (en) * | 2021-11-19 | 2022-04-15 | 三维通信股份有限公司 | Task coordination method and system for server cluster |
| CN114490068A (en) * | 2022-01-28 | 2022-05-13 | 北京秒如科技有限公司 | Resource arrangement method of edge computing network |
| CN115562870A (en) * | 2022-10-25 | 2023-01-03 | 北京京航计算通讯研究所 | Method for constructing task node resources of cluster |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107045456A (en) * | 2016-02-05 | 2017-08-15 | 华为技术有限公司 | A kind of resource allocation methods and explorer |
| CN107682417A (en) * | 2017-09-19 | 2018-02-09 | 深圳市盛路物联通讯技术有限公司 | The method for allocating tasks and device of a kind of back end |
| CN111045808A (en) * | 2019-12-17 | 2020-04-21 | 广东工业大学 | Distributed network task scheduling method and device |
| CN111381950A (en) * | 2020-03-05 | 2020-07-07 | 南京大学 | Task scheduling method and system based on multiple copies for edge computing environment |
| CN111552550A (en) * | 2020-04-26 | 2020-08-18 | 星环信息科技（上海）有限公司 | Task scheduling method, device and medium based on GPU (graphics processing Unit) resources |
-
2020
- 2020-09-18 CN CN202010988230.9A patent/CN112272203B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107045456A (en) * | 2016-02-05 | 2017-08-15 | 华为技术有限公司 | A kind of resource allocation methods and explorer |
| CN107682417A (en) * | 2017-09-19 | 2018-02-09 | 深圳市盛路物联通讯技术有限公司 | The method for allocating tasks and device of a kind of back end |
| CN111045808A (en) * | 2019-12-17 | 2020-04-21 | 广东工业大学 | Distributed network task scheduling method and device |
| CN111381950A (en) * | 2020-03-05 | 2020-07-07 | 南京大学 | Task scheduling method and system based on multiple copies for edge computing environment |
| CN111552550A (en) * | 2020-04-26 | 2020-08-18 | 星环信息科技（上海）有限公司 | Task scheduling method, device and medium based on GPU (graphics processing Unit) resources |
Cited By (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113220428A (en) * | 2021-04-23 | 2021-08-06 | 复旦大学 | Dynamic task scheduling algorithm for real-time requirements of cloud computing system |
| CN113220428B (en) * | 2021-04-23 | 2022-06-21 | 复旦大学 | Dynamic task scheduling method for real-time requirements of cloud computing system |
| CN113742064A (en) * | 2021-08-06 | 2021-12-03 | 苏州浪潮智能科技有限公司 | Resource arrangement method, system, equipment and medium for server cluster |
| CN113535332A (en) * | 2021-08-11 | 2021-10-22 | 北京字节跳动网络技术有限公司 | Cluster resource scheduling method and device, computer equipment and storage medium |
| CN114363328A (en) * | 2021-11-19 | 2022-04-15 | 三维通信股份有限公司 | Task coordination method and system for server cluster |
| CN114363328B (en) * | 2021-11-19 | 2024-02-20 | 三维通信股份有限公司 | Task collaboration method and system for server cluster |
| CN114143174A (en) * | 2021-11-30 | 2022-03-04 | 深信服科技股份有限公司 | Node repairing method, device, equipment and readable storage medium |
| CN114490068A (en) * | 2022-01-28 | 2022-05-13 | 北京秒如科技有限公司 | Resource arrangement method of edge computing network |
| CN114201306A (en) * | 2022-02-16 | 2022-03-18 | 长沙市规划勘测设计研究院 | Multi-dimensional geographic space entity distribution method and system based on load balancing technology |
| CN115562870A (en) * | 2022-10-25 | 2023-01-03 | 北京京航计算通讯研究所 | Method for constructing task node resources of cluster |
| CN115562870B (en) * | 2022-10-25 | 2023-07-21 | 北京京航计算通讯研究所 | Task node resource construction method of cluster |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112272203B (en) | 2022-06-14 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112272203B (en) | Cluster service node selection method, system, terminal and storage medium | |
| CN109471727B (en) | Task processing method, device and system | |
| Jiang et al. | Optimal cloud resource auto-scaling for web applications | |
| CN111124687B (en) | CPU resource reservation method, device and related equipment | |
| Soualhia et al. | Predicting scheduling failures in the cloud: A case study with google clusters and hadoop on amazon EMR | |
| CN111966500A (en) | Resource scheduling method and device, electronic equipment and storage medium | |
| JPWO2007072544A1 (en) | Information processing apparatus, computer, resource allocation method, and resource allocation program | |
| CN110221920B (en) | Deployment method, device, storage medium and system | |
| CN107797863B (en) | Fine-grained resource matching method in cloud computing platform | |
| Imai et al. | Accurate resource prediction for hybrid IaaS clouds using workload-tailored elastic compute units | |
| CN112698952A (en) | Unified management method and device for computing resources, computer equipment and storage medium | |
| CN113467933A (en) | Thread pool optimization method, system, terminal and storage medium for distributed file system | |
| CN112015549B (en) | Method and system for selectively preempting scheduling nodes based on server cluster | |
| CN111338785A (en) | Resource scheduling method and device, electronic equipment and storage medium | |
| CN111475251A (en) | Cluster container scheduling method, system, terminal and storage medium | |
| CN110096339A (en) | A kind of scalable appearance configuration recommendation system and method realized based on system load | |
| CN115562846A (en) | Resource scheduling method and device and computing node | |
| US10503548B2 (en) | Resource and latency estimation-based scheduling in a distributed computing environment | |
| CN115421930B (en) | Task processing method, system, device, equipment and computer readable storage medium | |
| CN111324459A (en) | Calendar-based resource scheduling method and device, electronic equipment and storage medium | |
| CN113419836B (en) | Task processing method and device, electronic equipment and computer readable storage medium | |
| CN109257256A (en) | Apparatus monitoring method, device, computer equipment and storage medium | |
| CN112363826A (en) | Project resource comprehensive management system, method, terminal and storage medium | |
| CN111124854A (en) | Method, system, terminal and storage medium for distributing smoking test cases | |
| CN113886086A (en) | Cloud platform computing resource allocation method, system, terminal and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |