CN108279982B - Method, system and equipment for managing pbs resources and hadoop resources - Google Patents
Method, system and equipment for managing pbs resources and hadoop resources Download PDFInfo
- Publication number
- CN108279982B CN108279982B CN201810161510.5A CN201810161510A CN108279982B CN 108279982 B CN108279982 B CN 108279982B CN 201810161510 A CN201810161510 A CN 201810161510A CN 108279982 B CN108279982 B CN 108279982B
- Authority
- CN
- China
- Prior art keywords
- resource
- nodes
- resources
- starting
- node
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 37
- 238000007726 management method Methods 0.000 claims abstract description 33
- 238000004590 computer program Methods 0.000 claims description 9
- 230000008569 process Effects 0.000 claims description 8
- 238000009434 installation Methods 0.000 claims description 3
- 238000010586 diagram Methods 0.000 description 4
- 230000009471 action Effects 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 238000012986 modification Methods 0.000 description 3
- 238000004364 calculation method Methods 0.000 description 2
- 230000000750 progressive effect Effects 0.000 description 2
- 230000008859 change Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000013468 resource allocation Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5061—Partitioning or combining of resources
- G06F9/5072—Grid computing
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Mathematical Physics (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a method, a system and a device for managing pbs resources and hadoop resources and a computer readable storage medium, wherein the method comprises the following steps: deploying a first resource and a second resource in all nodes in a cluster; the first resource and the second resource are pbs resources or hadoop resources, and the first resource and the second resource are two resources with different types; starting first resources of N nodes and second resources of M nodes; n, M is a positive integer, N is the minimum number of nodes configuring the first resource, and M is the minimum number of nodes configuring the second resource; starting a first resource or a second resource of a free node in the cluster according to the priority configuration file; wherein, the idle node is a node which does not start resources. Therefore, the resource management method provided by the invention can dynamically start the corresponding resources of the idle nodes according to the task condition of each resource, and improves the utilization efficiency of cluster resources.
Description
Technical Field
The present invention relates to the field of distributed systems, and more particularly, to a method, system, and apparatus for managing pbs resources and hadoop resources, and a computer-readable storage medium.
Background
In a distributed system, pbs (a batch job and computer system resource management software package) is commonly used for a management system of tasks related to HPC (High Performance Computing, Chinese called High Performance computer group), and hadoop (a distributed system infrastructure developed by the Apache Foundation) is commonly used for a management system of tasks related to big data. When the HPC application and the big data application need to be operated and managed simultaneously by the same cluster, the pbs and the hadoop need to be operated simultaneously by the cluster, and the pbs and the hadoop are managed respectively because the pbs and the hadoop have no correlation in resource allocation, so that the same node resource is allocated to the pbs application and the hadoop application at the same time, the resource is allocated repeatedly, and the applications cannot operate efficiently.
In the prior art, two solutions exist, one is to install pbs and hadoop for all nodes of a cluster, automatically allocate tasks, and compete resources between the tasks, which may cause that neither HPC applications nor big data applications can fully exert the performance of a host, and cause that one of the applications cannot run at all or has extremely poor computing performance. And the other is that pbs is installed on part of nodes of the cluster, hadoop is installed on the other part of nodes, the cluster nodes are fixed and do not switch, so that when one of the nodes in the cluster has fewer tasks and the other node has more tasks, part of the nodes of the cluster are idle, some tasks do not have resources to run, and the utilization of the cluster resources is unreasonable.
Therefore, how to reasonably allocate pbs resources and hadoop resources of each node in the cluster and improve the utilization efficiency of the cluster resources is a problem to be solved by the technical personnel in the field.
Disclosure of Invention
The invention aims to provide a method, a system and equipment for managing pbs resources and hadoop resources and a computer readable storage medium, which are used for reasonably distributing the pbs resources and the hadoop resources of each node in a cluster and improving the utilization efficiency of the cluster resources.
In order to achieve the purpose, the invention provides a method for managing pbs resources and hadoop resources, which comprises the following steps:
deploying a first resource and a second resource in all nodes in a cluster; wherein the first resource and the second resource are the pbs resource or the hadoop resource, and the first resource and the second resource are two resources of different types;
starting first resources of N nodes and second resources of M nodes; n, M is a positive integer, N is the minimum number of nodes configuring the first resource, and M is the minimum number of nodes configuring the second resource;
starting a first resource or a second resource of an idle node in the cluster according to the priority configuration file; and the idle node is a node which does not start resources.
Before starting the first resource or the second resource of the idle node in the cluster according to the priority configuration file, the method further includes:
acquiring a priority configuration file at a preset time interval; the priority configuration file is resource configuration information and task information of all nodes in the cluster.
Wherein, the starting the first resource or the second resource of the idle node in the cluster according to the priority configuration file comprises:
when the task of the first resource is increased, selecting a first target node from the idle nodes, and starting the first resource of the first target node;
and when the task of the second resource is increased, selecting a second target node from the idle nodes, and starting the second resource of the second target node.
Wherein, when the idle node is not present in the cluster, the method further comprises:
judging whether redundant nodes exist in the nodes for starting the first resources and whether queuing tasks of the second resources exist; the redundant nodes are nodes which are not distributed with tasks;
if yes, selecting a third target node from the redundant nodes, and switching the starting resource of the third target node to be the second resource.
In order to achieve the above object, the present invention provides a pbs resource and hadoop resource management system, comprising:
the installation module is used for deploying the first resource and the second resource in all nodes in the cluster; wherein the first resource is any one of the pbs resource and the hadoop resource, and the second resource is one of the pbs resource and the hadoop resource except the first resource;
the first starting module is used for starting first resources of the N nodes and second resources of the M nodes; n, M is a positive integer, N is the minimum number of nodes configuring the first resource, and M is the minimum number of nodes configuring the second resource;
the second starting module is used for starting the first resource or the second resource of the idle node in the cluster according to the priority configuration file; and the idle node is a node which does not start resources.
Wherein, still include:
the acquisition module is used for acquiring the priority configuration file at a preset time interval; the priority configuration file is resource configuration information and task information of all nodes in the cluster.
Wherein, the second starting module specifically comprises:
a first starting unit, configured to select a first target node from the idle nodes and start a first resource of the first target node when a task of the first resource increases;
and the second starting unit is used for selecting a second target node from the idle nodes and starting the second resource of the second target node when the task of the second resource is increased.
Wherein, still include:
the judging module is used for judging whether redundant nodes exist in the nodes for starting the first resources and whether queuing tasks of the second resources exist; the redundant nodes are nodes which are not distributed with tasks; if yes, starting the working process of the switching module;
and the switching module is used for selecting a third target node from the redundant nodes and switching the starting resource of the third target node into the second resource.
In order to achieve the above object, the present invention provides a pbs resource and hadoop resource management device, comprising:
a memory for storing a computer program;
a processor for implementing the steps of the management method as described above when executing the computer program.
To achieve the above object, the present invention provides a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements the management method as described above.
According to the scheme, the method for managing the pbs resources and the hadoop resources comprises the following steps: deploying a first resource and a second resource in all nodes in a cluster; wherein the first resource and the second resource are the pbs resource or the hadoop resource, and the first resource and the second resource are two resources of different types; starting first resources of N nodes and second resources of M nodes; n, M is a positive integer, N is the minimum number of nodes configuring the first resource, and M is the minimum number of nodes configuring the second resource; starting a first resource or a second resource of an idle node in the cluster according to the priority configuration file; and the idle node is a node which does not start resources.
The pbs resource and hadoop resource management method provided by the invention starts two resources with the least number of nodes at the beginning, ensures the minimum used resources of pbs and hadoop, and starts a certain resource of an idle node according to the priority configuration file, namely when the task of a certain resource is increased, the node is allocated for the task, and the resource service of the node is started, so that the normal implementation of the task is ensured. Therefore, the resource management method provided by the invention can dynamically start the corresponding resources of the idle nodes according to the task condition of each resource, reasonably distribute the pbs resources and hadoop resources of each node in the cluster, and improve the utilization efficiency of the cluster resources. The invention also discloses a pbs resource and hadoop resource management system, equipment and a computer readable storage medium, which can also realize the technical effects.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.
FIG. 1 is a flow chart of a method for managing pbs resources and hadoop resources according to an embodiment of the present invention;
FIG. 2 is a flow chart of another method for managing pbs resources and hadoop resources disclosed by the embodiments of the present invention;
FIG. 3 is a block diagram of a pbs resource and hadoop resource management system according to an embodiment of the present invention;
fig. 4 is a structural diagram of a pbs resource and hadoop resource management device disclosed in the embodiment of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The embodiment of the invention discloses a method for managing pbs resources and hadoop resources, which reasonably distributes the pbs resources and the hadoop resources of each node in a cluster and improves the utilization efficiency of the cluster resources.
Referring to fig. 1, a flowchart of a method for managing pbs resources and hadoop resources disclosed in the embodiment of the present invention is shown in fig. 1, and includes:
s101: deploying a first resource and a second resource in all nodes in a cluster; wherein the first resource and the second resource are the pbs resource or the hadoop resource, and the first resource and the second resource are two resources of different types;
s102: starting first resources of N nodes and second resources of M nodes; n, M is a positive integer, N is the minimum number of nodes configuring the first resource, and M is the minimum number of nodes configuring the second resource;
in specific implementation, pbs resources and hadoop resources are deployed on all nodes in a cluster, but only one of the resources of a part of the nodes is started, that is, the pbs resources of N nodes are started, and the hadoop resources of M nodes are started, where N is the minimum number of nodes configuring the pbs resources in the cluster, M is the minimum number of nodes configuring the hadoop resources, values of N and M are not specifically limited herein, and those skilled in the art can flexibly set the values according to actual situations. For example, starting pbs resources of 5 nodes and hadoop resources of 10 nodes, the implementation code is as follows:
PBS_MIN＝5；
HADOOP_MIN＝10；
another form of the above implementation code is:
PBS_MIN＝node1-node5；
HADOOP_MIN＝node10-node20；
therefore, after some resources of the nodes are started, idle nodes exist in the cluster, namely the nodes which are not started with any resources, the idle nodes start the resources and are then flexibly set according to the priority configuration file.
It should be noted that the specific execution sequence of S101 and S102 is not limited here, that is, when a certain node needs to start a certain resource, the certain resource may be started directly after the node deploys two resources, or after all the two resources of all the nodes in the cluster are completely deployed, the certain resource of the node may be started again.
S103: starting a first resource or a second resource of an idle node in the cluster according to the priority configuration file; and the idle node is a node which does not start resources.
It should be noted that the priority configuration file is resource configuration information and task information of all nodes in the cluster, that is, the number of nodes that start each resource and the number of tasks of each resource in the cluster, where the number of nodes that start each resource includes the number of nodes that start the resource and do not process the task and the number of nodes that start the resource and process the task, and the number of tasks of each resource includes the number of tasks that are being executed by each resource and the number of tasks to be processed by each resource. The priority profile is for example:
PBS_PRIORITY＝70；
HADOOP_PRIORITY＝30；
PBS_JOB_PRIORITY＝70；
in a specific implementation, the processor selects a target node from the idle nodes according to the priority configuration file, that is, the resource type of the task to be processed, and starts the resource of the target node.
It should be understood that, before this step, a step of acquiring the priority configuration file at a preset time interval should be further included, that is, after the processor acquires the priority configuration file at the preset time interval, according to the priority configuration file, the pbs resource or hadoop resource of the idle node is started. The specific implementation codes are as follows:
the method for managing the pbs resources and the hadoop resources provided by the embodiment of the invention starts two resources with the least number of nodes at the beginning, ensures the minimum used resources of the pbs and the hadoop, and starts a certain resource of an idle node according to the priority configuration file, namely when the tasks of a certain resource are increased, the node is allocated to the task, and the resource service of the node is started, so that the normal implementation of the task is ensured. Therefore, the resource management method provided by the embodiment of the invention can dynamically start the corresponding resources of the idle node according to the task condition of each resource, reasonably distribute the pbs resources and hadoop resources of each node in the cluster, and improve the utilization efficiency of the cluster resources.
The embodiment of the invention discloses a method for managing pbs resources and hadoop resources, which further explains and optimizes the technical scheme compared with the previous embodiment. Specifically, the method comprises the following steps:
referring to fig. 2, a flowchart of another method for managing pbs resources and hadoop resources according to an embodiment of the present invention is shown in fig. 2, and includes:
s201: deploying a first resource and a second resource in all nodes in a cluster; wherein the first resource and the second resource are the pbs resource or the hadoop resource, and the first resource and the second resource are two resources of different types;
s202: starting first resources of N nodes and second resources of M nodes; n, M is a positive integer, N is the minimum number of nodes configuring the first resource, and M is the minimum number of nodes configuring the second resource;
s203: starting a first resource or a second resource of an idle node in the cluster according to the priority configuration file; the idle node is a node which does not start resources;
s204: when the idle node does not exist in the cluster, judging whether a redundant node exists in the nodes for starting the first resource and whether a queuing task of the second resource exists; the redundant nodes are nodes which are not distributed with tasks; if yes, entering S205;
in a specific implementation, when a queued task of a second resource exists, it is determined whether a redundant node exists in a node that starts the first resource, if yes, S205 is entered, and if no, after a node task in the system is completed, the starting resource of the node is switched to the second resource.
S205: and selecting a third target node from the redundant nodes, and switching the starting resource of the third target node to the second resource.
For example, if the first resource is a pbs resource, the second resource is a hadoop resource. In specific implementation, if the pbs node resource is more than the configured priority proportion of the pbs task and the hadoop has queued tasks, one node is removed from the pbs resource, and is switched to the hadoop resource, and the hadoop calculation task is preferentially performed. The specific codes are as follows:
and if the pbs node resource is equal to the configured pbs task priority ratio, not performing task switching operation. The specific codes are as follows:
else if((PBS_NUM-PBS_MIN)＝＝
Math.round(PBS_PRIORITY/HADOOP_PRIORITY*
(PBS_NUM-PBS_MIN+HADOOP_NUM-HADOOP_MIN)))
do nothing；
and if the pbs node resource is smaller than the configured priority ratio of the pbs task and the pbs has queued tasks, removing one node from the hadoop resource, switching to the pbs resource, and preferentially performing the pbs calculation task. The specific codes are as follows:
it should be noted that the above example only represents a specific application scenario, and the embodiment does not limit the specific types of the first resource and the second resource, that is, the first resource may be a hadoop resource, and the second resource may be a pbs resource.
Therefore, for a certain node, the starting resource of the node can change in real time according to the condition of queuing tasks of all resources in the cluster, and compared with the technical scheme that the starting resource of the node in the prior art is fixed, the condition that the task of a certain resource exists but only the starting node of another resource exists can not occur, so that the cluster resource is utilized more flexibly and reasonably.
In the following, a pbs resource and hadoop resource management system provided by an embodiment of the present invention is introduced, and a pbs resource and hadoop resource management system described below and a pbs resource and hadoop resource management method described above may be referred to each other.
Referring to fig. 3, a structure diagram of a pbs resource and hadoop resource management system according to an embodiment of the present invention is shown in fig. 3, and includes:
an installation module 301, configured to deploy a first resource and a second resource in all nodes in a cluster; wherein the first resource is any one of the pbs resource and the hadoop resource, and the second resource is one of the pbs resource and the hadoop resource except the first resource;
a first starting module 302, configured to start first resources of the N nodes and second resources of the M nodes; n, M is a positive integer, N is the minimum number of nodes configuring the first resource, and M is the minimum number of nodes configuring the second resource;
a second starting module 303, configured to start the first resource or the second resource of the idle node in the cluster according to the priority configuration file; and the idle node is a node which does not start resources.
The pbs resource and hadoop resource management system provided by the embodiment of the invention starts two resources with the least number of nodes at the beginning, ensures the minimum used resources of pbs and hadoop, and starts a certain resource of an idle node according to the priority configuration file, namely when the task of a certain resource is increased, the node is allocated to the task, and the resource service of the node is started, so that the normal implementation of the task is ensured. Therefore, the resource management system provided by the embodiment of the invention can dynamically start the corresponding resources of the idle node according to the task condition of each resource, reasonably distribute the pbs resources and hadoop resources of each node in the cluster, and improve the utilization efficiency of the cluster resources.
In addition to the above embodiments, as a preferred embodiment, the method further includes:
the acquisition module is used for acquiring the priority configuration file at a preset time interval; the priority configuration file is resource configuration information and task information of all nodes in the cluster.
On the basis of the foregoing embodiment, as a preferred implementation, the second starting module specifically includes:
a first starting unit, configured to select a first target node from the idle nodes and start a first resource of the first target node when a task of the first resource increases;
and the second starting unit is used for selecting a second target node from the idle nodes and starting the second resource of the second target node when the task of the second resource is increased.
In addition to the above embodiments, as a preferred embodiment, the method further includes:
the judging module is used for judging whether redundant nodes exist in the nodes for starting the first resources and whether queuing tasks of the second resources exist; the redundant nodes are nodes which are not distributed with tasks; if yes, starting the working process of the switching module;
and the switching module is used for selecting a third target node from the redundant nodes and switching the starting resource of the third target node into the second resource.
The present application further provides a pbs resource and hadoop resource management device, referring to fig. 4, a structural diagram of a pbs resource and hadoop resource management device provided in an embodiment of the present invention, as shown in fig. 4, includes:
a memory 401 for storing a computer program;
the processor 402, when executing the computer program, may implement the steps provided by the above embodiments. Of course, the management device may also include various network interfaces, power supplies, and other components.
The pbs resource and hadoop resource management device provided by the embodiment of the invention starts two resources with the least number of nodes at the beginning, ensures the minimum use of the pbs and hadoop resources, and starts a certain resource of an idle node according to the priority configuration file, namely when the task of a certain resource is increased, the node is allocated to the task, and the resource service of the node is started, so that the normal implementation of the task is ensured. Therefore, the resource management device provided by the embodiment of the invention can dynamically start the corresponding resources of the idle node according to the task condition of each resource, reasonably distribute the pbs resources and hadoop resources of each node in the cluster, and improve the utilization efficiency of the cluster resources.
The present application also provides a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, may implement the steps provided by the above-described embodiments. The storage medium may include: various media capable of storing program codes, such as a usb disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk.
The embodiments in the present description are described in a progressive manner, each embodiment focuses on differences from other embodiments, and the same and similar parts among the embodiments are referred to each other.
The previous description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the invention. Thus, the present invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
The embodiments are described in a progressive manner in the specification, each embodiment focuses on differences from other embodiments, and the same and similar parts among the embodiments are referred to each other. For the system disclosed by the embodiment, the description is relatively simple because the system corresponds to the method disclosed by the embodiment, and the relevant points can be referred to the method part for description. It should be noted that, for those skilled in the art, it is possible to make several improvements and modifications to the present application without departing from the principle of the present application, and such improvements and modifications also fall within the scope of the claims of the present application.
It is further noted that, in the present specification, relational terms such as first and second, and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.
Claims (10)
1. A method for managing pbs resources and hadoop resources is characterized by comprising the following steps:
deploying a first resource and a second resource in all nodes in a cluster; wherein the first resource and the second resource are the pbs resource or the hadoop resource, and the first resource and the second resource are two resources of different types;
starting first resources of N nodes and second resources of M nodes; n, M is a positive integer, N is the minimum number of nodes configuring the first resource, M is the minimum number of nodes configuring the second resource, and the N nodes and the M nodes are different nodes;
starting a first resource or a second resource of an idle node in the cluster according to the priority configuration file; and the idle node is a node which does not start resources.
2. The management method according to claim 1, wherein before starting the first resource or the second resource of the idle node in the cluster according to the priority profile, the method further comprises:
acquiring a priority configuration file at a preset time interval; the priority configuration file is resource configuration information and task information of all nodes in the cluster.
3. The management method according to claim 1, wherein the starting the first resource or the second resource of the idle node in the cluster according to the priority profile comprises:
when the task of the first resource is increased, selecting a first target node from the idle nodes, and starting the first resource of the first target node;
and when the task of the second resource is increased, selecting a second target node from the idle nodes, and starting the second resource of the second target node.
4. The method of managing as claimed in claim 3, wherein when the free node is not present in the cluster, the method further comprises:
judging whether redundant nodes exist in the nodes for starting the first resources and whether queuing tasks of the second resources exist; the redundant nodes are nodes which are not distributed with tasks;
if yes, selecting a third target node from the redundant nodes, and switching the starting resource of the third target node to be the second resource.
5. A pbs resource and hadoop resource management system, comprising:
the installation module is used for deploying the first resource and the second resource in all nodes in the cluster; wherein the first resource is any one of the pbs resource and the hadoop resource, and the second resource is one of the pbs resource and the hadoop resource except the first resource;
the first starting module is used for starting first resources of the N nodes and second resources of the M nodes; n, M is a positive integer, N is the minimum number of nodes configuring the first resource, M is the minimum number of nodes configuring the second resource, and the N nodes and the M nodes are different nodes;
the second starting module is used for starting the first resource or the second resource of the idle node in the cluster according to the priority configuration file; and the idle node is a node which does not start resources.
6. The management system of claim 5, further comprising:
the acquisition module is used for acquiring the priority configuration file at a preset time interval; the priority configuration file is resource configuration information and task information of all nodes in the cluster.
7. The management system according to claim 5, wherein the second starting module specifically includes:
a first starting unit, configured to select a first target node from the idle nodes and start a first resource of the first target node when a task of the first resource increases;
and the second starting unit is used for selecting a second target node from the idle nodes and starting the second resource of the second target node when the task of the second resource is increased.
8. The management system of claim 7, further comprising:
the judging module is used for judging whether redundant nodes exist in the nodes for starting the first resources and whether queuing tasks of the second resources exist; the redundant nodes are nodes which are not distributed with tasks; if yes, starting the working process of the switching module;
and the switching module is used for selecting a third target node from the redundant nodes and switching the starting resource of the third target node into the second resource.
9. A pbs resource and hadoop resource management device, comprising:
a memory for storing a computer program;
processor for implementing the steps of the management method according to any one of claims 1 to 4 when executing said computer program.
10. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the management method according to any one of claims 1 to 4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810161510.5A CN108279982B (en) | 2018-02-27 | 2018-02-27 | Method, system and equipment for managing pbs resources and hadoop resources |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810161510.5A CN108279982B (en) | 2018-02-27 | 2018-02-27 | Method, system and equipment for managing pbs resources and hadoop resources |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN108279982A CN108279982A (en) | 2018-07-13 |
| CN108279982B true CN108279982B (en) | 2021-11-09 |
Family
ID=62808748
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810161510.5A Active CN108279982B (en) | 2018-02-27 | 2018-02-27 | Method, system and equipment for managing pbs resources and hadoop resources |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108279982B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113680075A (en) * | 2021-08-24 | 2021-11-23 | 中国汽车技术研究中心有限公司 | Automobile race competition method, competition platform, electronic equipment and medium |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103345298A (en) * | 2013-07-16 | 2013-10-09 | 山东省计算中心 | Data center energy saving system and method based on virtual IT resource distribution technology |
| CN103365726A (en) * | 2013-07-08 | 2013-10-23 | 华中科技大学 | Resource management method and system facing GPU (Graphic Processing Unit) cluster |
| CN103645956A (en) * | 2013-12-18 | 2014-03-19 | 浪潮电子信息产业股份有限公司 | Intelligent cluster load management method |
| CN104572306A (en) * | 2015-01-28 | 2015-04-29 | 中国石油集团川庆钻探工程有限公司地球物理勘探公司 | Method for managing resources of computer cluster and resource manager |
| CN106933664A (en) * | 2017-03-09 | 2017-07-07 | 北京搜狐新媒体信息技术有限公司 | A kind of resource regulating method and device of Hadoop clusters |
| CN107135257A (en) * | 2017-04-28 | 2017-09-05 | 东方网力科技股份有限公司 | Task is distributed in a kind of node cluster method, node and system |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106708622B (en) * | 2016-07-18 | 2020-06-02 | 腾讯科技（深圳）有限公司 | Cluster resource processing method and system and resource processing cluster |
-
2018
- 2018-02-27 CN CN201810161510.5A patent/CN108279982B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103365726A (en) * | 2013-07-08 | 2013-10-23 | 华中科技大学 | Resource management method and system facing GPU (Graphic Processing Unit) cluster |
| CN103345298A (en) * | 2013-07-16 | 2013-10-09 | 山东省计算中心 | Data center energy saving system and method based on virtual IT resource distribution technology |
| CN103645956A (en) * | 2013-12-18 | 2014-03-19 | 浪潮电子信息产业股份有限公司 | Intelligent cluster load management method |
| CN104572306A (en) * | 2015-01-28 | 2015-04-29 | 中国石油集团川庆钻探工程有限公司地球物理勘探公司 | Method for managing resources of computer cluster and resource manager |
| CN106933664A (en) * | 2017-03-09 | 2017-07-07 | 北京搜狐新媒体信息技术有限公司 | A kind of resource regulating method and device of Hadoop clusters |
| CN107135257A (en) * | 2017-04-28 | 2017-09-05 | 东方网力科技股份有限公司 | Task is distributed in a kind of node cluster method, node and system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN108279982A (en) | 2018-07-13 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108965485B (en) | Container resource management method and device and cloud platform | |
| US20190319895A1 (en) | Resource Scheduling Method And Apparatus | |
| CN108337109B (en) | Resource allocation method and device and resource allocation system | |
| CN106293893B (en) | Job scheduling method and device and distributed system | |
| WO2021227999A1 (en) | Cloud computing service system and method | |
| CN111290838B (en) | Application access request processing method and device based on container cluster | |
| CN110806920A (en) | Mirror image deployment method and device | |
| US8028291B2 (en) | Method and computer program product for job selection and resource allocation of a massively parallel processor | |
| CN107291557B (en) | Service processing method and device | |
| CN111459681B (en) | Dynamic elastic telescoping method based on micro-service load | |
| CN112214288B (en) | Pod scheduling method, device, equipment and medium based on Kubernetes cluster | |
| CN113886069A (en) | Resource allocation method and device, electronic equipment and storage medium | |
| CN106775975B (en) | Process scheduling method and device | |
| CN108279982B (en) | Method, system and equipment for managing pbs resources and hadoop resources | |
| CN110795234A (en) | Resource scheduling method and device | |
| CN111240824A (en) | CPU resource scheduling method and electronic equipment | |
| CN111355751A (en) | Task scheduling method and device | |
| CN112398892B (en) | Service distribution method, device and equipment | |
| CN111143063B (en) | Task resource reservation method and device | |
| CN109189581B (en) | Job scheduling method and device | |
| CN112527490A (en) | Node resource control method and device, electronic equipment and storage medium | |
| CN113138842B (en) | Micro service cluster scheduling method, device and medium | |
| CN116149841A (en) | Processor resource dynamic superdivision method based on cloud database instance load | |
| CN105760215A (en) | Map-reduce model based job running method for distributed file system | |
| CN114328285A (en) | Heap memory allocation management method and device of embedded operating system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |