CN106648801A - Automatic node adding and deleting method for Hadoop cluster - Google Patents
Automatic node adding and deleting method for Hadoop cluster Download PDFInfo
- Publication number
- CN106648801A CN106648801A CN201611260873.1A CN201611260873A CN106648801A CN 106648801 A CN106648801 A CN 106648801A CN 201611260873 A CN201611260873 A CN 201611260873A CN 106648801 A CN106648801 A CN 106648801A
- Authority
- CN
- China
- Prior art keywords
- node
- hadoop
- component
- ambari
- restapi
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 53
- 240000000797 Hibiscus cannabinus Species 0.000 claims abstract description 25
- 235000002905 Rumex vesicarius Nutrition 0.000 claims abstract description 25
- 238000012217 deletion Methods 0.000 claims abstract description 22
- 230000037430 deletion Effects 0.000 claims abstract description 22
- 230000008569 process Effects 0.000 claims abstract description 19
- 238000005538 encapsulation Methods 0.000 claims description 6
- 238000009434 installation Methods 0.000 claims description 6
- 238000012545 processing Methods 0.000 abstract description 11
- 238000007792 addition Methods 0.000 abstract description 3
- 238000012423 maintenance Methods 0.000 description 3
- 238000012546 transfer Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 238000012790 confirmation Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/445—Program loading or initiating
- G06F9/44505—Configuring for program initiating, e.g. using registry, configuration files
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/445—Program loading or initiating
- G06F9/44521—Dynamic linking or loading; Link editing at or after load time, e.g. Java class loading
- G06F9/44526—Plug-ins; Add-ons
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/02—Protocols based on web technology, e.g. hypertext transfer protocol [HTTP]
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Databases & Information Systems (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Data Mining & Analysis (AREA)
- Computer And Data Communications (AREA)
Abstract
The invention discloses an automatic node adding and deleting method for a Hadoop cluster. According to the method, addition and deletion operations of the Hadoop cluster are realized by calling one or more Ambari RestApi. Multiple atomic operations needed to be performed on an Ambari interface originally are integrated, so that a user only needs to initiate a request once and the whole processing process is free from manual intervention; all processing stages are monitored and checked in the processing process, so that the processing success rate is ensured; and continuous operation of error breakpoints is supported.
Description
Technical field
The present invention relates to Hadoop Clusterings field, and in particular to a kind of Hadoop clusters add deletion of node automatically
Method.By being packaged to Ambari RestApi, it is possible to achieve the automatic addition of any type node in Hadoop clusters
And deletion, without the need for manually being intervened.
Background technology
Ambari is used as a kind of supply, management, the instrument for monitoring Hadoop clusters, there is provided easy-operating Hadoop directly perceived
Cluster is installed, management function.User only need to be by being clicked in WEB page, it is possible to is completed to create, is managed, configures
Operation, has provided the user one-stop service, simplifies the operation of user.However, due to Hadoop ecological complexity,
Increase node for Hadoop clusters on Ambari to remain a need for much being operated, even if these operations are manually added than pure
Hadoop nodes simplify many, but when safeguarding multiple clusters or batch addition, deletion of node at the same time, these operations still show
Obtain excessively loaded down with trivial details, operation maintenance personnel can feel unable to do what one wishes.
The content of the invention
The technical problem to be solved in the present invention is：The present invention is directed to problem above, there is provided a kind of Hadoop clusters add automatically
Plus the method for deletion of node.
The technical solution adopted in the present invention is：
A kind of method that Hadoop clusters add deletion of node automatically, methods described is by calling one or more Ambari
RestApi, realizes that addition, deletion, maintenance of Hadoop clusters etc. are operated.
Ambari RestApi be one group of Ambari provide meet connecing for REST specifications for operate Hadoop clusters
Mouthful.
REST is the abbreviation of Representational State Transfer, is that a kind of operation by Http requests is taken
The international interface specification of business end resource, its content mainly includes：
（1）Each URI represents a kind of resource of service end
（2）Arrange and transmit a form of expression of resource between client and service end.Such as：One section is described using XML format
The text of resource.
（3）Client passes through four HTTP verbs（GET、POST、PUT、DELETE）To specify the resource to service end to enter
Row is which type of operation.
Every API for meeting REST interface specifications is referred to as RestApi.
Methods described is packaged by the Amari RestApi logics higher to multiplicity, can significantly simplify tune
With the logic of Amari RestApi, user only needs to be concerned about the resource of operation itself.Such as：RestApi is called substantially to belong to
In Http requests are sent, it is required for building request header information, authentication information, request body information etc. when sending request every time；Simultaneously
After service end response is received, also head response, response body etc. are parsed accordingly.
Methods described is combined to multiple atomic operations by using program and layout and is packaged, and realizes a whole set of business
The automatic business processing of logic.
After being packaged by the logic higher to multiplicity, only it is a simplified and calls Amari RestApi to Hadoop
Cluster carries out the logic of atomic operation.And multiple atomic operations are combined by using program and layout, realize a whole set of
The automatic business processing of service logic（Such as increase Hadoop nodes newly）, it is the encapsulation in second aspect.
By being packaged to Ambari RestApi, user only needs to be input into the node IP that add or delete, you can
Complete the newly-increased of Hadoop nodes and delete.
Interface after Ambari RestApi encapsulation also can be called by other programs, will add or delete Hadoop sections
The operation of point is embedded in other service logics as a sub- service logic.
Addition Hadoop node flow processs are as follows：
A) user needs the Hadoop nodes for being ready to need addition in advance（Physical machine or virtual machine）, it is installed and activated
Ambari-agent；
B) user starts addition node-flow Cheng Qian, needs to select control node, and is input into the Hadoop node address to be added；Newly
Component meeting reference pair on node is installed according to mounted module condition on node；
C) flow process starts, and first checks for Ambari-server whether normal operations；
D) check whether Ambari-agent allows normally on destination node；
E) new node is registered in cluster；
F) mounted the component list on control node is read；
G) it is that new node adds component according to the component list of control node；
H) it is new node executive module installation operation；
I) component is started after component installation.
Delete Hadoop node flow processs as follows：
A) before flow process starts, user needs the node address to be deleted of input；
B) flow process starts, and first checks for Ambari-server whether normal operations；
C) the whether component for carrying out retired operation in need is checked on node（Such as DataNode, NodeManager, HBASE
Region Server etc.）, and carry out corresponding retired operation；If not carrying out retired operation, loss of data is may result in；
D) after the completion of retired operation, all component on Stop node；
E) all component on node is removed；
F) node is nullified from cluster；
G) inquire about in cluster because configuration change needs the component restarted and is restarted.
Methods described gets the information such as cluster, component, the node of Hadoop by calling one group of Ambari RestApi,
And be analyzed later confirm node on need component to be mounted.
Methods described gets the information such as cluster, component, the node of Hadoop by calling one group of Ambari RestApi,
And be analyzed later confirm node on mounted component.
Beneficial effects of the present invention are：
The present invention incorporates the multinomial atomic operation for needing to be carried out on Ambari interfaces originally, and user only need to initiate once please
Ask, middle whole processing procedure is without the need for manually being intervened；Processing procedure is monitored and verifies to each processing stage, it is ensured that
Process success rate；Support that wrong breakpoint is continuous to do.
Description of the drawings
Fig. 1 is newly-increased node schematic flow sheet；
Fig. 2 is deletion of node schematic flow sheet.
Specific embodiment
According to Figure of description, with reference to specific embodiment, the present invention is further described：
Embodiment 1：
A kind of method that Hadoop clusters add deletion of node automatically, methods described is by calling one or more Ambari
RestApi, realizes that addition, deletion, maintenance of Hadoop clusters etc. are operated.
Ambari RestApi be one group of Ambari provide meet connecing for REST specifications for operate Hadoop clusters
Mouthful.
REST is the abbreviation of Representational State Transfer, is that a kind of operation by Http requests is taken
The international interface specification of business end resource, its content mainly includes：
（1）Each URI represents a kind of resource of service end；
（2）Arrange and transmit a form of expression of resource between client and service end, such as：One section is described using XML format
The text of resource；
（3）Client passes through four HTTP verbs（GET、POST、PUT、DELETE）It is assorted to specify the resource to service end to carry out
The operation of sample.
Every API for meeting REST interface specifications is referred to as RestApi.
Embodiment 2
On the basis of embodiment 1, the present embodiment methods described is sealed by the Amari RestApi logics higher to multiplicity
Dress, can significantly simplify the logic for calling Amari RestApi, and user only needs to be concerned about the resource of operation itself.Such as：
Call RestApi substantially to belong to transmission Http requests, be required for building request header information, certification letter when sending request every time
Breath, request body information etc.；Simultaneously after service end response is received, also head response, response body etc. are parsed accordingly.
Embodiment 3
On the basis of embodiment 2, multiple atomic operations are combined by using program for the present embodiment methods described and layout is gone forward side by side
Row encapsulation, realizes the automatic business processing of a whole set of service logic.
After being packaged by the logic higher to multiplicity, only it is a simplified and calls Amari RestApi to Hadoop
Cluster carries out the logic of atomic operation.And multiple atomic operations are combined by using program and layout, realize a whole set of
The automatic business processing of service logic（Such as increase Hadoop nodes newly）, it is the encapsulation in second aspect.
Embodiment 4
On the basis of embodiment 3, by being packaged to Ambari RestApi, user only needs to input will be added the present embodiment
Or the node IP deleted, you can complete the newly-increased of Hadoop nodes and delete.
Embodiment 5
Interface behind the basis of embodiment 4, the present embodiment Ambari RestApi encapsulation also can be called by other programs,
The operation for adding or deleting Hadoop nodes is embedded in other service logics as a sub- service logic.
Embodiment 6
As shown in figure 1, on the basis of embodiment 4 or 5, the present embodiment addition Hadoop node flow processs are as follows：
A) user needs the Hadoop nodes for being ready to need addition in advance（Physical machine or virtual machine）, it is installed and activated
Ambari-agent；
B) user starts addition node-flow Cheng Qian, needs to select control node, and is input into the Hadoop node address to be added；Newly
Component meeting reference pair on node is installed according to mounted module condition on node；
C) flow process starts, and first checks for Ambari-server whether normal operations；
D) check whether Ambari-agent allows normally on destination node；
E) new node is registered in cluster；
F) mounted the component list on control node is read；
G) it is that new node adds component according to the component list of control node；
H) it is new node executive module installation operation；
I) component is started after component installation.
Embodiment 7
As shown in Fig. 2 on the basis of embodiment 4 or 5, it is as follows that the present embodiment deletes Hadoop node flow processs：
A) before flow process starts, user needs the node address to be deleted of input；
B) flow process starts, and first checks for Ambari-server whether normal operations；
C) the whether component for carrying out retired operation in need is checked on node（Such as DataNode, NodeManager, HBASE
Region Server etc.）, and carry out corresponding retired operation；If not carrying out retired operation, loss of data is may result in；
D) after the completion of retired operation, all component on Stop node；
E) all component on node is removed；
F) node is nullified from cluster；
G) inquire about in cluster because configuration change needs the component restarted and is restarted.
Embodiment 8
On the basis of embodiment 6, the present embodiment methods described gets Hadoop's by calling one group of Ambari RestApi
The information such as cluster, component, node, and be analyzed later confirm node on need component to be mounted.
Embodiment 9
On the basis of embodiment 7, the present embodiment methods described gets Hadoop's by calling one group of Ambari RestApi
The information such as cluster, component, node, and it is analyzed mounted component on confirmation node later.
Embodiment is merely to illustrate the present invention, and not limitation of the present invention, about the ordinary skill of technical field
Personnel, without departing from the spirit and scope of the present invention, can also make a variety of changes and modification, therefore all equivalents
Technical scheme fall within scope of the invention, the scope of patent protection of the present invention should be defined by the claims.
Claims (9)
1. a kind of method that Hadoop clusters add deletion of node automatically, it is characterised in that methods described by call one or
Multiple Ambari RestApi, realize addition, the deletion action of Hadoop clusters.
2. the method that a kind of Hadoop clusters according to claim 1 add deletion of node automatically, it is characterised in that described
Method is packaged by the Amari RestApi logics higher to multiplicity, and the logic of Amari RestApi is called in simplification,
User only needs to be concerned about the resource of operation itself.
3. the method that a kind of Hadoop clusters according to claim 2 add deletion of node automatically, it is characterised in that described
Method is combined to multiple atomic operations by using program and layout and is packaged, and realizes the automation of a whole set of service logic
Process.
4. the method that a kind of Hadoop clusters according to claim 3 add deletion of node automatically, it is characterised in that pass through
Ambari RestApi are packaged, user only needs to be input into the node IP that add or delete, you can complete Hadoop sections
The newly-increased and deletion of point.
5. the method that a kind of Hadoop clusters according to claim 4 add deletion of node automatically, it is characterised in that
Interface after Ambari RestApi encapsulation also can be called by other programs, will add or delete the operation of Hadoop nodes
It is embedded in other service logics as a sub- service logic.
6. the method that a kind of Hadoop clusters according to claim 4 or 5 add deletion of node automatically, it is characterised in that
Addition Hadoop node flow processs are as follows：
A) user is ready to need the Hadoop nodes of addition in advance（Physical machine or virtual machine）, it is installed and activated Ambari-
agent；
B) user starts addition node-flow Cheng Qian, needs to select control node, and is input into the Hadoop node address to be added；
C) flow process starts, and first checks for Ambari-server whether normal operations；
D) check whether Ambari-agent allows normally on destination node；
E) new node is registered in cluster；
F) mounted the component list on control node is read；
G) it is that new node adds component according to the component list of control node；
H) it is new node executive module installation operation；
I) component is started after component installation.
7. the method that a kind of Hadoop clusters according to claim 4 or 5 add deletion of node automatically, it is characterised in that
Delete Hadoop node flow processs as follows：
A) before flow process starts, the node address to be deleted of user input；
B) flow process starts, and first checks for Ambari-server whether normal operations；
C) the whether component for carrying out retired operation in need is checked on node（Such as DataNode, NodeManager, HBASE
Region Server etc.）, and carry out corresponding retired operation；
D) after the completion of retired operation, all component on Stop node；
E) all component on node is removed；
F) node is nullified from cluster；
G) inquire about in cluster because configuration change needs the component restarted and is restarted.
8. the method that a kind of Hadoop clusters according to claim 6 add deletion of node automatically, it is characterised in that described
Method gets cluster, component, the nodal information of Hadoop by calling one group of Ambari RestApi, and is analyzed later
Confirm to need component to be mounted on node.
9. the method that a kind of Hadoop clusters according to claim 7 add deletion of node automatically, it is characterised in that described
Method gets cluster, component, the nodal information of Hadoop by calling one group of Ambari RestApi, and is analyzed later
Confirm mounted component on node.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201611260873.1A CN106648801A (en) | 2016-12-30 | 2016-12-30 | Automatic node adding and deleting method for Hadoop cluster |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201611260873.1A CN106648801A (en) | 2016-12-30 | 2016-12-30 | Automatic node adding and deleting method for Hadoop cluster |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN106648801A true CN106648801A (en) | 2017-05-10 |
Family
ID=58838913
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201611260873.1A Pending CN106648801A (en) | 2016-12-30 | 2016-12-30 | Automatic node adding and deleting method for Hadoop cluster |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106648801A (en) |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107330113A (en) * | 2017-07-11 | 2017-11-07 | 郑州云海信息技术有限公司 | A kind of method serviced based on ambari databases Force Deletion |
| CN107704362A (en) * | 2017-09-25 | 2018-02-16 | 浪潮软件股份有限公司 | A kind of method and device based on Ambari monitoring big data components |
| CN109189594A (en) * | 2018-09-04 | 2019-01-11 | 山东浪潮云投信息科技有限公司 | The method of Hadoop cluster user is added under a kind of safe mode |
| CN111104068A (en) * | 2019-12-20 | 2020-05-05 | 北京浪潮数据技术有限公司 | Management method, system, equipment and storage medium of storage domain |
| CN112087506A (en) * | 2020-09-01 | 2020-12-15 | 北京字节跳动网络技术有限公司 | Cluster node management method and device and computer storage medium |
| US11329885B2 (en) | 2018-06-21 | 2022-05-10 | International Business Machines Corporation | Cluster creation using self-aware, self-joining cluster nodes |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1867113A (en) * | 2005-12-08 | 2006-11-22 | 华为技术有限公司 | Data management system and method for digital trunking system |
| CN103618634A (en) * | 2013-12-11 | 2014-03-05 | 上海爱数软件有限公司 | Method for automatically finding nodes in cluster |
-
2016
- 2016-12-30 CN CN201611260873.1A patent/CN106648801A/en active Pending
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1867113A (en) * | 2005-12-08 | 2006-11-22 | 华为技术有限公司 | Data management system and method for digital trunking system |
| CN103618634A (en) * | 2013-12-11 | 2014-03-05 | 上海爱数软件有限公司 | Method for automatically finding nodes in cluster |
Non-Patent Citations (1)
| Title |
|---|
| 任川: "基于云架构的存储资源管控技术研究", 《中国优秀硕士学位论文全文数据库 信息科技辑》 * |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107330113A (en) * | 2017-07-11 | 2017-11-07 | 郑州云海信息技术有限公司 | A kind of method serviced based on ambari databases Force Deletion |
| CN107704362A (en) * | 2017-09-25 | 2018-02-16 | 浪潮软件股份有限公司 | A kind of method and device based on Ambari monitoring big data components |
| US11329885B2 (en) | 2018-06-21 | 2022-05-10 | International Business Machines Corporation | Cluster creation using self-aware, self-joining cluster nodes |
| CN109189594A (en) * | 2018-09-04 | 2019-01-11 | 山东浪潮云投信息科技有限公司 | The method of Hadoop cluster user is added under a kind of safe mode |
| CN111104068A (en) * | 2019-12-20 | 2020-05-05 | 北京浪潮数据技术有限公司 | Management method, system, equipment and storage medium of storage domain |
| CN112087506A (en) * | 2020-09-01 | 2020-12-15 | 北京字节跳动网络技术有限公司 | Cluster node management method and device and computer storage medium |
| CN112087506B (en) * | 2020-09-01 | 2023-02-07 | 北京火山引擎科技有限公司 | Cluster node management method and device and computer storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106648801A (en) | Automatic node adding and deleting method for Hadoop cluster | |
| US8788565B2 (en) | Dynamic and distributed queueing and processing system | |
| US9152441B2 (en) | Systems and methods involving virtual machine host isolation over a network via a federated downstream cluster | |
| JP6332774B2 (en) | Network function virtualized NFV failure management apparatus, device, and method | |
| CN104199957B (en) | A kind of implementation method of Redis general-purpose proxies | |
| CN102197627B (en) | The improvement of flux of multicast convergence | |
| CN108989066B (en) | Equipment management method and device | |
| WO2007059665A1 (en) | A system and a method for the configuration of the network device | |
| TWI751139B (en) | Business processing system, business processing method and business update method | |
| CN113067883B (en) | Data transmission method, device, computer equipment and storage medium | |
| JP2017184259A (en) | Method for flow control under cooperation environment and communication with reliability | |
| CN104268241B (en) | Attribute display method in a kind of configuration database | |
| WO2019141089A1 (en) | Network alarm method, device, system and terminal | |
| CN102546839B (en) | Efficient and reliable software distribution method for large scale network | |
| US10942785B2 (en) | Integration of software applications with infrastructure | |
| US11838176B1 (en) | Provisioning and deploying RAN applications in a RAN system | |
| EP3467656B1 (en) | Guidance system for enterprise infrastructure change | |
| CN103914286A (en) | Management method and manager for tab page in user interface | |
| JP5436597B2 (en) | Virtual network infrastructure control system and method | |
| US20200412621A1 (en) | Health check automation for virtual network functions | |
| WO2013000231A1 (en) | Batch service processing method and device | |
| US11403093B1 (en) | Application modification with proxy service process | |
| CN114070784A (en) | Message processing method and device, electronic equipment and storage medium | |
| JP2017220240A (en) | Graphical policy interface for network control system | |
| CN103560976B (en) | A kind of method, apparatus and system that control data are sent |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20170510 |