CN106506233A - A kind of automatic deployment Hadoop clusters and the method for flexible working node - Google Patents
A kind of automatic deployment Hadoop clusters and the method for flexible working node Download PDFInfo
- Publication number
- CN106506233A CN106506233A CN201611089556.8A CN201611089556A CN106506233A CN 106506233 A CN106506233 A CN 106506233A CN 201611089556 A CN201611089556 A CN 201611089556A CN 106506233 A CN106506233 A CN 106506233A
- Authority
- CN
- China
- Prior art keywords
- hadoop
- working node
- virtual machine
- cluster
- service end
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/08—Configuration management of networks or network elements
- H04L41/0893—Assignment of logical groups to network elements
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The invention discloses a kind of method of automatic deployment Hadoop clusters and flexible working node, methods described is in virtualized environment, by the pattern of service end/agent side, the deployment template of Hadoop clusters is formulated, complete the deployment of Hadoop clusters and stretching for cluster working node.The inventive method is flexibly easy-to-use, and entirely deployment and telescopic process unmanned, are performed entirely automatically.Multiple different Hadoop clusters can be disposed in virtualized environment with simple and quick, and carry out online flexible process according to demand to the working node of Hadoop, and not affect the running status of cluster.With it, user can meet the big data of oneself with the limited physical environment resource of more efficient utilization calculates demand.
Description
Technical field
The present invention relates to computer virtual machine technical field, and in particular to a kind of automatic deployment Hadoop clusters and flexible work
The method for making node, a kind of method of automatic deployment Hadoop clusters and flexible working node in virtualized environment.
Background technology
Hadoop is often used for storage and the meter of mass data as a kind of efficient Distributed Calculation software frame
Calculate.Multiple Hadoop clusters can be built on limited physical environment come for users to use by Intel Virtualization Technology, to carry
The utilization rate of high physical environment.However, as the component added in the Hadoop ecospheres is more and more, deployment Hadoop clusters become
Thing into a unusual very complicated.If flexibly can quickly dispose Hadoop clusters energy in virtualized environment
Enough flexible expansion cluster scales, then the limited physical environment resource of utilization that user just can be more flexible and efficient, meet
The big data of user calculates demand.
Content of the invention
The technical problem to be solved in the present invention is：The present invention is directed to problem above, there is provided a kind of automatic deployment Hadoop collection
Group and the method for flexible working node.
The technical solution adopted in the present invention is：
A kind of automatic deployment Hadoop clusters and the method for flexible working node, methods described in virtualized environment, by clothes
The pattern of business end/agent side, formulates the deployment template of Hadoop clusters, completes deployment and the cluster working node of Hadoop clusters
Flexible.
The deployment of methods described Hadoop cluster realizes that flow process is as follows：
1）Agent side is built in virtual machine template or system image；
2）Formulate Hadoop clustered deploy(ment) templates：
By formulating Hadoop clustered deploy(ment) templates, the detailed configuration of a Hadoop cluster is defined, template file is uploaded to clothes
Business end；
3）Create virtual machine：
Service end creates different configuration of virtual machine in batches according to the definition in Hadoop clustered deploy(ment) templates；
4）Configuration Hadoop components are installed：
Complete when virtual machine starts, after all agent sides are all ready, service end is according to configuration in Hadoop clustered deploy(ment) templates
Component definition notifies the agent side in cluster on each virtual machine to install the Hadoop components specified and configured；
5）Start Hadoop components：
After Hadoop component installations, service end notifies the agent side on each virtual machine to start the Hadoop components in the machine.
The virtual machine calls the API of virtual platform by service end, using comprising agent side and Hadoop components
The virtual machine template in installation source or mirror image are created.
Methods described extension cluster working node is realized as follows：
When service end receives the working node request for extending certain cluster, it is new that service end calls the API of virtual platform to create
Virtual machine, and notify the Hadoop components required for the agent side installment work node on virtual machine and configured；Install
Into rear startup Hadoop components and access in former cluster.
Methods described reduction cluster working node is realized as follows：
When service end receives working node reduction request, service end notifies agent side on working node to be reduced by the node
Remove from cluster, and the Hadoop components on closed node；Hadoop components call the API of virtual platform after the completion of closing
Virtual machine is deleted.
Methods described is by preserving multiple Hadoop clustered deploy(ment)s template files, corresponding to create under different demands
Hadoop clusters.
The Hadoop clustered deploy(ment)s template file is write using Json forms.
Beneficial effects of the present invention are：
The inventive method is flexibly easy-to-use, and entirely deployment and telescopic process unmanned, are performed entirely automatically.Can with simple and quick
Multiple different Hadoop clusters are disposed in virtualized environment, and according to demand the working node of Hadoop is stretched online
Process, and do not affect the running status of cluster.With it, user can be with the limited physical environment resource of more efficient utilization
Demand is calculated to meet the big data of oneself.
Description of the drawings
Fig. 1 is formulation Hadoop template flow charts；
Fig. 2 is deployment Hadoop cluster flow charts；
Fig. 3 is extension cluster working node flow chart；
Fig. 4 is reduction cluster working node flow chart.
Specific embodiment
Below according to Figure of description, in conjunction with specific embodiment, the present invention is further described：
Embodiment 1：
A kind of automatic deployment Hadoop clusters and the method for flexible working node, methods described in virtualized environment, by clothes
The pattern of business end/agent side, formulates the deployment template of Hadoop clusters, completes deployment and the cluster working node of Hadoop clusters
Flexible.
Embodiment 2
As shown in Fig. 2 on the basis of embodiment 1, the deployment of the present embodiment methods described Hadoop cluster realizes that flow process is as follows：
1）Agent side is built in virtual machine template or system image, and is set to startup item of starting shooting；
2）Formulate Hadoop clustered deploy(ment) templates：
As shown in figure 1, by formulating Hadoop clustered deploy(ment) templates, the detailed configuration of a Hadoop cluster is defined, including：
Node group role, number of nodes, virtual machine configuration, node group need Hadoop components to be mounted and component Configuration etc.；Template
Files passe can create multiple Hadoop clusters based on the template definition on virtualized environment to service end；
3）Create virtual machine：
Service end creates different configuration of virtual machine in batches according to the definition in Hadoop clustered deploy(ment) templates；
4）Configuration Hadoop components are installed：
After the completion of virtual machine starts, agent side is notified to service end ready for sending, after all agent sides are all ready, service end
The agent side installation in cluster on each virtual machine is notified to specify according to the component definition configured in Hadoop clustered deploy(ment) templates
Hadoop components are simultaneously configured；
5）Start Hadoop components：
After Hadoop component installations, service end notifies the agent side on each virtual machine to start the Hadoop components in the machine.
Embodiment 3
On the basis of embodiment 1 or 2, virtual machine described in the present embodiment calls the API of virtual platform by service end, uses
The virtual machine template in source is installed comprising agent side and Hadoop components or mirror image is created.
Embodiment 4
As shown in figure 3, on the basis of embodiment 3, the present embodiment methods described extension cluster working node is realized as follows：
When service end receives the working node request for extending certain cluster, it is new that service end calls the API of virtual platform to create
Virtual machine, and notify the Hadoop components required for the agent side installment work node on virtual machine and configured；Install
Into rear startup Hadoop components and access in former cluster.
Embodiment 5
As shown in figure 4, on the basis of embodiment 3, the present embodiment methods described reduction cluster working node is realized as follows：
When service end receives working node reduction request, service end notifies agent side on working node to be reduced by the node
Remove from cluster, and the Hadoop components on closed node；Hadoop components call the API of virtual platform after the completion of closing
Virtual machine is deleted.
Embodiment 6
On the basis of embodiment 5, the present embodiment methods described passes through to preserve multiple Hadoop clustered deploy(ment)s template files, so as to
Corresponding Hadoop clusters are created under different demands.
Embodiment 7
On the basis of embodiment 6, described in the present embodiment, Hadoop clustered deploy(ment)s template file is write using Json forms, structure
Clearly should be readily appreciated that.
Embodiment is merely to illustrate the present invention, and not limitation of the present invention, about the ordinary skill of technical field
Personnel, without departing from the spirit and scope of the present invention, can also make a variety of changes and modification, therefore all equivalents
Technical scheme fall within scope of the invention, the scope of patent protection of the present invention should be defined by the claims.
Claims (7)
1. a kind of method of automatic deployment Hadoop clusters and flexible working node, it is characterised in that methods described is in virtualization
In environment, by the pattern of service end/agent side, the deployment template of Hadoop clusters is formulated, the deployment of Hadoop clusters is completed
Flexible with cluster working node.
2. the method for a kind of automatic deployment Hadoop clusters according to claim 1 and flexible working node, its feature exist
In the deployment of methods described Hadoop cluster realizes that flow process is as follows：
1）Agent side is built in virtual machine template or system image；
2）Formulate Hadoop clustered deploy(ment) templates：
By formulating Hadoop clustered deploy(ment) templates, the detailed configuration of a Hadoop cluster is defined, template file is uploaded to clothes
Business end；
3）Create virtual machine：
Service end creates different configuration of virtual machine in batches according to the definition in Hadoop clustered deploy(ment) templates；
4）Configuration Hadoop components are installed：
Complete when virtual machine starts, after all agent sides are all ready, service end is according to configuration in Hadoop clustered deploy(ment) templates
Component definition notifies the agent side in cluster on each virtual machine to install the Hadoop components specified and configured；
5）Start Hadoop components：
After Hadoop component installations, service end notifies the agent side on each virtual machine to start the Hadoop components in the machine.
3. the method for a kind of automatic deployment Hadoop clusters according to claim 1 and 2 and flexible working node, its feature
It is, the virtual machine calls the API of virtual platform by service end, using comprising agent side and the installation of Hadoop components
The virtual machine template in source or mirror image are created.
4. the method for a kind of automatic deployment Hadoop clusters according to claim 3 and flexible working node, its feature exist
In methods described extension cluster working node is realized as follows：
When service end receives the working node request for extending certain cluster, it is new that service end calls the API of virtual platform to create
Virtual machine, and notify the Hadoop components required for the agent side installment work node on virtual machine and configured；Install
Into rear startup Hadoop components and access in former cluster.
5. the method for a kind of automatic deployment Hadoop clusters according to claim 3 and flexible working node, its feature exist
In methods described reduction cluster working node is realized as follows：
When service end receives working node reduction request, service end notifies agent side on working node to be reduced by the node
Remove from cluster, and the Hadoop components on closed node；Hadoop components call the API of virtual platform after the completion of closing
Virtual machine is deleted.
6. the method for a kind of automatic deployment Hadoop clusters according to claim 5 and flexible working node, its feature exist
In methods described is by preserving multiple Hadoop clustered deploy(ment)s template files, corresponding to create under different demands
Hadoop clusters.
7. the method for a kind of automatic deployment Hadoop clusters according to claim 6 and flexible working node, its feature exist
In the Hadoop clustered deploy(ment)s template file is write using Json forms.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201611089556.8A CN106506233A (en) | 2016-12-01 | 2016-12-01 | A kind of automatic deployment Hadoop clusters and the method for flexible working node |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201611089556.8A CN106506233A (en) | 2016-12-01 | 2016-12-01 | A kind of automatic deployment Hadoop clusters and the method for flexible working node |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN106506233A true CN106506233A (en) | 2017-03-15 |
Family
ID=58329416
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201611089556.8A Pending CN106506233A (en) | 2016-12-01 | 2016-12-01 | A kind of automatic deployment Hadoop clusters and the method for flexible working node |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106506233A (en) |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107329804A (en) * | 2017-07-07 | 2017-11-07 | 郑州云海信息技术有限公司 | It is a kind of automatically created according to configuration, the method for upgrading virtual machine |
| CN108549580A (en) * | 2018-03-30 | 2018-09-18 | 平安科技（深圳）有限公司 | Methods and terminal device of the automatic deployment Kubernetes from node |
| CN108958745A (en) * | 2018-06-26 | 2018-12-07 | 郑州云海信息技术有限公司 | A kind of device and method in cloud platform deployment Spark cluster |
| CN109783198A (en) * | 2019-01-29 | 2019-05-21 | 中山大学 | A kind of fast automatic construction method of data experiment environment of large quantities |
| CN112822044A (en) * | 2020-12-30 | 2021-05-18 | 北京天融信网络安全技术有限公司 | Distributed cluster deployment method and device, electronic equipment and readable storage medium |
| US11329885B2 (en) | 2018-06-21 | 2022-05-10 | International Business Machines Corporation | Cluster creation using self-aware, self-joining cluster nodes |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20140281675A1 (en) * | 2000-03-16 | 2014-09-18 | Sony Computer Entertainment America Llc | Flexible failover policies in high availability computing systems |
| CN104065716A (en) * | 2014-06-18 | 2014-09-24 | 江苏物联网研究发展中心 | OpenStack based Hadoop service providing method |
| CN104580519A (en) * | 2015-01-29 | 2015-04-29 | 福建师范大学福清分校 | Method for rapid deployment of openstack cloud computing platform |
| CN105100180A (en) * | 2014-11-25 | 2015-11-25 | 航天恒星科技有限公司 | Cluster node dynamic loading method, device and system |
-
2016
- 2016-12-01 CN CN201611089556.8A patent/CN106506233A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20140281675A1 (en) * | 2000-03-16 | 2014-09-18 | Sony Computer Entertainment America Llc | Flexible failover policies in high availability computing systems |
| CN104065716A (en) * | 2014-06-18 | 2014-09-24 | 江苏物联网研究发展中心 | OpenStack based Hadoop service providing method |
| CN105100180A (en) * | 2014-11-25 | 2015-11-25 | 航天恒星科技有限公司 | Cluster node dynamic loading method, device and system |
| CN104580519A (en) * | 2015-01-29 | 2015-04-29 | 福建师范大学福清分校 | Method for rapid deployment of openstack cloud computing platform |
Cited By (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107329804A (en) * | 2017-07-07 | 2017-11-07 | 郑州云海信息技术有限公司 | It is a kind of automatically created according to configuration, the method for upgrading virtual machine |
| CN108549580A (en) * | 2018-03-30 | 2018-09-18 | 平安科技（深圳）有限公司 | Methods and terminal device of the automatic deployment Kubernetes from node |
| CN108549580B (en) * | 2018-03-30 | 2023-04-14 | 平安科技（深圳）有限公司 | Method for automatically deploying Kubernets slave nodes and terminal equipment |
| US11329885B2 (en) | 2018-06-21 | 2022-05-10 | International Business Machines Corporation | Cluster creation using self-aware, self-joining cluster nodes |
| CN108958745A (en) * | 2018-06-26 | 2018-12-07 | 郑州云海信息技术有限公司 | A kind of device and method in cloud platform deployment Spark cluster |
| CN108958745B (en) * | 2018-06-26 | 2021-11-26 | 郑州云海信息技术有限公司 | Device and method for deploying Spark cluster on cloud platform |
| CN109783198A (en) * | 2019-01-29 | 2019-05-21 | 中山大学 | A kind of fast automatic construction method of data experiment environment of large quantities |
| CN109783198B (en) * | 2019-01-29 | 2023-01-20 | 中山大学 | Rapid and automatic construction method for batch big data experiment environment |
| CN112822044A (en) * | 2020-12-30 | 2021-05-18 | 北京天融信网络安全技术有限公司 | Distributed cluster deployment method and device, electronic equipment and readable storage medium |
| CN112822044B (en) * | 2020-12-30 | 2022-12-20 | 北京天融信网络安全技术有限公司 | Distributed cluster deployment method and device, electronic equipment and readable storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106506233A (en) | A kind of automatic deployment Hadoop clusters and the method for flexible working node | |
| CN105391797B (en) | Cloud Server load-balancing method and device based on SDN | |
| CN104104720B (en) | A kind of server set group managing means and system | |
| Böhm et al. | xSim: The extreme-scale simulator | |
| CN104601492A (en) | Method and device for controlling business flow under NFV framework | |
| CN103793278B (en) | Automatic resource adjusting method on basis of operation and maintenance rules of virtual device | |
| CN102521055B (en) | Virtual machine resource allocating method and virtual machine resource allocating system | |
| CN112667362B (en) | Method and system for deploying Kubernetes virtual machine cluster on Kubernetes | |
| CN104008012B (en) | A kind of high-performance MapReduce implementation methods based on dynamic migration of virtual machine | |
| CN106293847B (en) | A kind of method of virtual platform supporting | |
| CN105308553B (en) | Dynamic provides storage | |
| CN103365726A (en) | Resource management method and system facing GPU (Graphic Processing Unit) cluster | |
| CN103136030A (en) | Virtual machine management system and method | |
| WO2019085104A1 (en) | Virtual machine deployment method, device, apparatus, and computer readable storage medium | |
| CN103561055A (en) | Web application automatic elastic extension method under cloud computing environment based on sessions | |
| CN104580194A (en) | Virtual resource management method and device oriented to video applications | |
| CN103780428A (en) | Centralized resource management method and system applied to cloud architecture | |
| CN109960579B (en) | Method and device for adjusting service container | |
| CN110532060A (en) | A kind of hybrid network environmental data collecting method and system | |
| CN106959950A (en) | The method and apparatus of migrating data between application cluster | |
| CN109739634A (en) | A kind of atomic task execution method and device | |
| CN109144666A (en) | A kind of method for processing resource and system across cloud platform | |
| CN112035063A (en) | Hard disk and file system thermal expansion method based on cloud platform | |
| CN110868330B (en) | Evaluation method, device and evaluation system for CPU resources which can be divided by cloud platform | |
| CN106406978A (en) | Automatic making device and method for private cloud virtual machine template |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20170315 |
|
| RJ01 | Rejection of invention patent application after publication |