CN106708573B - System and method for Hadoop cluster automatic installation - Google Patents
System and method for Hadoop cluster automatic installation Download PDFInfo
- Publication number
- CN106708573B CN106708573B CN201611180305.0A CN201611180305A CN106708573B CN 106708573 B CN106708573 B CN 106708573B CN 201611180305 A CN201611180305 A CN 201611180305A CN 106708573 B CN106708573 B CN 106708573B
- Authority
- CN
- China
- Prior art keywords
- installation
- hadoop
- physical machine
- application service
- node
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F8/00—Arrangements for software engineering
- G06F8/60—Software deployment
- G06F8/61—Installation
Abstract
The invention provides a system and a method for Hadoop cluster automatic installation, wherein the system comprises: the DHCP server is used for allocating and managing dynamic IP; the BOOT server is used for storing an operating system configuration template and a Hadoop application service configuration template; the YUM server is used for storing installation source files of an operating system and Hadoop application service; the physical machine node performs the following steps: acquiring a dynamic IP from a DHCP server; acquiring a corresponding installation source file from a YUM server according to a personalized Kickstart installation script; installing an operating system in a physical machine node according to the dynamic IP and the installation source file; and receiving a Hadoop application service installation instruction, and installing the Hadoop application service in the Hadoop cluster according to the Hadoop application service configuration template and the personalized Kickstart installation script.
Description
Technical Field
The invention relates to big data technology, in particular to a system and a method for Hadoop cluster automatic installation.
Background
Hadoop is a software framework developed by the Apache Foundation that enables distributed processing of large amounts of data. It is widely used in big data processing, thanks to its own advantages in data extraction, deformation and loading (ETL): the Hadoop distributed architecture places the big data processing engine as close as possible to the storage end, such as batch processing operation like ETL, and the result can be directly stored. MapReduce of Hadoop realizes the fragmentation of tasks, sends the fragmentation tasks (Map) to a plurality of nodes, and then loads (Reduce) the fragmentation tasks into a data warehouse in the form of data sets.
In Hadoop deployment practice, there are two issues: 1. various physical machine hardware and complex operating system configuration; 2. hadoop application services are rich, and the cluster structure coupling is high. In the traditional mode, the installation of the physical machine needs to be manually installed by an administrator, and the system configuration is carried out on one machine, so that the time is wasted, and the configuration process is complex and has high operability risk. And each service of Hadoop has different management interfaces, and an administrator must know all the interfaces to configure the service, so that the cluster structure complexity is high. In order to solve the problem of automatic installation of an operating system and a Hadoop application program, a set of standardized installation technology needs to be developed to realize a rapid deployment method of a physical machine cluster under a Hadoop platform.
Disclosure of Invention
The embodiment of the invention mainly aims to provide a system and a method for Hadoop cluster automatic installation, so as to solve the problems of time and labor waste caused by various hardware types and complex configuration in the existing installation mode.
In order to achieve the above object, an embodiment of the present invention provides a system for automatically installing a Hadoop cluster, where the system includes: the system comprises a DHCP server, a BOOT server, a YUM server, a Hadoop cluster and a physical machine node, wherein the DHCP server is used for distributing and managing dynamic IP; the BOOT server is used for storing an operating system configuration template and a Hadoop application service configuration template; the YUM server is used for storing installation source files of an operating system and Hadoop application service; the physical machine node performs the steps of: acquiring the dynamic IP from the DHCP server; acquiring a corresponding installation source file from the YUM server according to a personalized Kickstart installation script; installing an operating system in the physical machine node according to the dynamic IP and the installation source file; and receiving a Hadoop application service installation instruction, and installing the Hadoop application service in the Hadoop cluster according to the Hadoop application service configuration template and the personalized Kickstart installation script.
In one embodiment, after installing an operating system in the physical machine node according to the dynamic IP, the installation source file, the physical machine node further performs the following steps: replacing the dynamic IP with an application IP of the physical machine node.
In an embodiment, the types of the Hadoop application service configuration template at least include: a management node template, a service node template and an application node template.
In an embodiment, the personalized Kickstart installation script is an installation script that is generated by the BOOT server according to the type of the Hadoop application service configuration template and the physical machine node and corresponds to the type and the physical machine node.
In an embodiment, the configuration parameters of the os configuration template at least include: the host computer number, the host name, the cluster network IP address and the operating system version number of the physical computer node.
The embodiment of the invention also provides a method for automatically installing the Hadoop cluster, which is applied to the system and comprises the following steps: acquiring the dynamic IP from the DHCP server; acquiring a corresponding installation source file from the YUM server according to a personalized Kickstart installation script; installing an operating system in the physical machine node according to the dynamic IP and the installation source file; and receiving a Hadoop application service installation instruction, and installing the Hadoop application service in the Hadoop cluster according to the Hadoop application service configuration template and the personalized Kickstart installation script.
In one embodiment, after installing an operating system in the physical machine node according to the dynamic IP, installation source file, the method further comprises: replacing the dynamic IP with an application IP of the physical machine node.
In an embodiment, the types of the Hadoop application service configuration template at least include: a management node template, a service node template and an application node template.
In an embodiment, the personalized Kickstart installation script is an installation script that is generated by the BOOT server according to the type of the Hadoop application service configuration template and the physical machine node and corresponds to the type and the physical machine node.
In an embodiment, the configuration parameters of the os configuration template at least include: the host computer number, the host name, the cluster network IP address and the operating system version number of the physical computer node.
The embodiment of the invention has the advantages of solving the problems of time and labor waste in the installation process caused by various hardware and complex configuration, and being capable of quickly installing the operating system and the corresponding application program, thereby greatly saving the labor cost.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required to be used in the description of the embodiments will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art that other drawings can be obtained according to the drawings without inventive labor.
FIG. 1 is a schematic structural diagram of a system for Hadoop cluster automatic installation according to an embodiment of the present invention;
FIG. 2 is a flow diagram of a method for Hadoop cluster automatic installation according to an embodiment of the invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The embodiment of the invention provides a system and a method for automatically installing a Hadoop cluster. The present invention will be described in detail below with reference to the accompanying drawings.
An embodiment of the present invention provides a system for automatically installing a Hadoop cluster, as shown in fig. 1, the system for automatically installing a Hadoop cluster mainly includes: a DHCP server 101, a BOOT server 102, a YUM server 103, a Hadoop cluster 104 and a physical machine node 105.
The DHCP server 101 is configured to allocate and manage dynamic IP, and provide an installation network IP address before the physical machine node 105 accesses the Hadoop cluster 104.
The BOOT server 102 is used for storing an operating system configuration template and a Hadoop application service configuration template, a parameter configuration library is deployed in the BOOT server 102, and the parameter configuration library at least comprises various configuration parameters of the operating system configuration template and various configuration parameters of the Hadoop application service configuration template. Various configuration parameters of the operating system configuration template may be, for example, a host number (unique identifier) of the physical machine node 105, a host name, a cluster network IP address, an operating system version number, and the like; the various configuration parameters of the Hadoop application service configuration template at least comprise: three categories are divided according to the role of the physical machine node 105 in the cluster and the hardware configuration: the management node template, the service node template and the application node template are different in the types and the number of application services configured by the three types of templates.
The YUM server 103 is used for storing installation source files of operating systems and Hadoop application services of various versions.
As can be seen from the content shown in fig. 1, the following processes are mainly performed between the physical machine node 105 and the DHCP server 101, the BOOT server 102, and the YUM server 103 described above:
first, a dynamic IP is acquired from the DHCP server 101; specifically, the physical machine node 105 may be started from a PXE network card after accessing an installation network, and the DHCP server 101 in the installation network requests a dynamic IP.
Then, the corresponding installation source file is obtained from the YUM server 103 according to a personalized Kickstart installation script. In this process, the BOOT server 102 finds a corresponding configuration file according to the host number of the physical machine node 105, and loads an operating system kernel and a personalized Kickstart installation script in the memory of the physical machine node 105.
The personalized Kickstart installation script is an installation script which is generated by the BOOT server 102 according to the type of the Hadoop application service configuration template and the physical machine node 105 and corresponds to the type and the physical machine node 105. The personalized Kickstart installation script at least includes the os version number of the physical machine node 105. Further, the corresponding installation source file may be obtained from the YUM server 103 according to the os version number.
Next, the physical machine node 105 may complete automatic installation and system configuration of the operating system in the physical machine node 105 according to the dynamic IP and the installation source file.
Then, the physical machine node 105 can receive the Hadoop application service installation instruction, and install the Hadoop application service in the Hadoop cluster 104 according to the Hadoop application service configuration template and the personalized Kickstart installation script.
Specifically, after the operating system and the system configuration are completed, the physical machine node 105 is accessed to the Hadoop cluster 104 and receives the management of the CM5 node 106. And receiving a Hadoop application service installation instruction, acquiring corresponding Hadoop application service and parameter configuration configured in the personalized Kickstart installation script according to a Hadoop application configuration template in a BOOT server 102 parameter configuration library, and finishing customized installation of the Hadoop application service.
By the aid of the system for automatically installing the Hadoop cluster, a system capable of quickly deploying operating systems and application services in the Hadoop cluster 104 is constructed, the problem that operating systems of different versions are installed when multiple physical machine nodes 105 are connected into the Hadoop cluster 104 is solved, troubles of relevant configuration and application program deployment such as modification of MAC addresses and the like through one physical machine in the prior art are avoided, and accordingly installation safety, reliability and installation efficiency are greatly improved.
In practical application, after the operating system and the system configuration are completed by installing the physical machine node 105, a step of replacing the IP is further performed, and the application IP of the physical machine node 105 is used for replacing the dynamic IP used in the process of installing the operating system.
The embodiment of the invention also provides a method for automatically installing the Hadoop cluster, and in an embodiment, the method for automatically installing the Hadoop cluster can be applied to the system for automatically installing the Hadoop cluster in the embodiment.
As shown in fig. 2, the method for automatically installing the Hadoop cluster mainly includes the following steps:
step S201: acquiring a dynamic IP from the DHCP server 101;
step S202: acquiring a corresponding installation source file from the YUM server 103 according to a personalized Kickstart installation script;
step S203: installing an operating system in the physical machine node 105 according to the dynamic IP and the installation source file;
step S204: and receiving a Hadoop application service installation instruction, and installing the Hadoop application service in the Hadoop cluster 104 according to the Hadoop application service configuration template and the personalized Kickstart installation script.
Through the steps S201 to S204, the method for automatically installing the Hadoop cluster according to the embodiment of the present invention can implement a system for quickly deploying the operating system and the application service, solve the problem of installing operating systems of different versions in a plurality of physical machine nodes 105 accessing the Hadoop cluster 104, and avoid the trouble of manually modifying the related configuration such as the MAC address and the like and deploying the application program in the prior art, thereby greatly improving the safety, reliability and installation efficiency of installation.
Each step of the method for automatically installing the Hadoop cluster according to the embodiment of the present invention is further described below.
In the above step S201, the dynamic IP is acquired from the DHCP server 101. Specifically, the physical machine node 105 may be started from a PXE network card after accessing an installation network, and the DHCP server 101 in the installation network requests a dynamic IP.
In the step S202, the corresponding installation source file is obtained from the YUM server 103 according to a personalized Kickstart installation script.
In this step, the BOOT server 102 finds a corresponding configuration file according to the host number of the physical machine node 105, and loads an operating system kernel and a personalized Kickstart installation script in the memory of the physical machine node 105.
The personalized Kickstart installation script is an installation script which is generated by the BOOT server 102 according to the type of the Hadoop application service configuration template and the physical machine node 105 and corresponds to the type and the physical machine node 105. The personalized Kickstart installation script at least includes the os version number of the physical machine node 105. Further, the corresponding installation source file may be obtained from the YUM server 103 according to the os version number.
In step S203, the operating system is installed in the physical machine node 105 according to the dynamic IP and the installation source file. Specifically, the physical machine node 105 can complete automatic installation and system configuration of the operating system in the physical machine node 105 according to the dynamic IP and the installation source file.
And then, executing the step S204, receiving a Hadoop application service installation instruction, and installing the Hadoop application service in the Hadoop cluster 104 according to the Hadoop application service configuration template and the personalized Kickstart installation script. Specifically, after the operating system and the system configuration are completed, the physical machine node 105 is accessed to the Hadoop cluster 104 and receives the management of the CM5 node. And receiving a Hadoop application service installation instruction, acquiring corresponding Hadoop application service and parameter configuration configured in the personalized Kickstart installation script according to a Hadoop application configuration template in a BOOT server 102 parameter configuration library, and finishing customized installation of the Hadoop application service.
In practical application, after the operating system and the system configuration are completed by installing the physical machine node 105, a step of replacing the IP is further performed, and the application IP of the physical machine node 105 is used for replacing the dynamic IP used in the process of installing the operating system.
It will be understood by those skilled in the art that all or part of the steps in the method for implementing the above embodiments may be implemented by relevant hardware instructed by a program, and the program may be stored in a computer readable storage medium, such as ROM/RAM, magnetic disk, optical disk, etc.
The above-mentioned embodiments are intended to illustrate the objects, technical solutions and advantages of the present invention in further detail, and it should be understood that the above-mentioned embodiments are only exemplary embodiments of the present invention, and are not intended to limit the scope of the present invention, and any modifications, equivalent substitutions, improvements and the like made within the spirit and principle of the present invention should be included in the scope of the present invention.
Claims (8)
1. A system for automated Hadoop cluster installation, the system comprising: a DHCP server, a BOOT server, a YUM server, a Hadoop cluster, and a physical machine node, wherein,
the DHCP server is used for allocating and managing dynamic IP; the BOOT server is used for storing an operating system configuration template and a Hadoop application service configuration template; the YUM server is used for storing installation source files of an operating system and Hadoop application service;
the physical machine node performs the steps of:
acquiring the dynamic IP from the DHCP server;
acquiring a corresponding installation source file from the YUM server according to an individualized Kickstart installation script, wherein the individualized Kickstart installation script is an installation script which is generated by the BOOT server according to the type of the Hadoop application service configuration template and the physical machine node and corresponds to the type and the physical machine node;
installing an operating system in the physical machine node according to the dynamic IP and the installation source file;
and receiving a Hadoop application service installation instruction, and installing the Hadoop application service in the Hadoop cluster according to the Hadoop application service configuration template and the personalized Kickstart installation script.
2. The system for Hadoop cluster automated installation according to claim 1, wherein after installing an operating system in the physical machine node according to the dynamic IP, installation source files, the physical machine node further performs the steps of:
replacing the dynamic IP with an application IP of the physical machine node.
3. The system for Hadoop cluster automated installation according to claim 2, wherein the Hadoop application service configuration templates are of a type comprising at least: a management node template, a service node template and an application node template.
4. The system for Hadoop cluster automated installation according to claim 1, wherein the configuration parameters of the operating system configuration template comprise at least: the host computer number, the host name, the cluster network IP address and the operating system version number of the physical computer node.
5. A method for Hadoop cluster automatic installation, characterized in that the method is applied to the system as claimed in any one of claims 1-4, and the method comprises:
acquiring the dynamic IP from the DHCP server;
acquiring a corresponding installation source file from the YUM server according to an individualized Kickstart installation script, wherein the individualized Kickstart installation script is an installation script which is generated by the BOOT server according to the type of the Hadoop application service configuration template and the physical machine node and corresponds to the type and the physical machine node;
installing an operating system in the physical machine node according to the dynamic IP and the installation source file;
and receiving a Hadoop application service installation instruction, and installing the Hadoop application service in the Hadoop cluster according to the Hadoop application service configuration template and the personalized Kickstart installation script.
6. The method for Hadoop cluster automated installation according to claim 5, wherein after installing an operating system in the physical machine node according to the dynamic IP, installation source files, the method further comprises:
replacing the dynamic IP with an application IP of the physical machine node.
7. The method for Hadoop cluster automated installation according to claim 6, wherein the Hadoop application service configuration templates are of a type comprising at least: a management node template, a service node template and an application node template.
8. The method for Hadoop cluster automatic installation according to claim 5, wherein the configuration parameters of the operating system configuration template comprise at least: the host computer number, the host name, the cluster network IP address and the operating system version number of the physical computer node.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201611180305.0A CN106708573B (en) | 2016-12-19 | 2016-12-19 | System and method for Hadoop cluster automatic installation |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201611180305.0A CN106708573B (en) | 2016-12-19 | 2016-12-19 | System and method for Hadoop cluster automatic installation |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN106708573A CN106708573A (en) | 2017-05-24 |
| CN106708573B true CN106708573B (en) | 2020-12-18 |
Family
ID=58938100
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201611180305.0A Active CN106708573B (en) | 2016-12-19 | 2016-12-19 | System and method for Hadoop cluster automatic installation |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106708573B (en) |
Families Citing this family (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109558143B (en) * | 2017-09-22 | 2022-02-22 | 北京国双科技有限公司 | Method and device for deploying applications in cluster |
| CN108062225B (en) * | 2017-11-27 | 2021-08-27 | 国云科技股份有限公司 | Method for issuing application cluster by multi-template combination |
| CN108595190B (en) * | 2018-04-23 | 2020-06-19 | 平安科技（深圳）有限公司 | Report tool building method and device, computer device and storage medium |
| CN111061503B (en) * | 2018-10-16 | 2023-08-18 | 航天信息股份有限公司 | Cluster system configuration method and cluster system |
| CN110109684B (en) * | 2019-04-04 | 2023-06-09 | 平安科技（深圳）有限公司 | Block chain link point management proxy service installation method, electronic device and storage medium |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104317610A (en) * | 2014-10-11 | 2015-01-28 | 福建新大陆软件工程有限公司 | Method and device for automatic installation and deployment of hadoop platform |
| KR20150030332A (en) * | 2013-09-12 | 2015-03-20 | 삼성전자주식회사 | Distributed and parallel processing system on data and method of operating the same |
| CN105959134A (en) * | 2016-04-22 | 2016-09-21 | 汉柏科技有限公司 | Automatic installation deployment method based on PXE, and PXE server |
| CN106126303A (en) * | 2016-07-01 | 2016-11-16 | 合信息技术（北京）有限公司 | The installation method of (SuSE) Linux OS, Apparatus and system |
-
2016
- 2016-12-19 CN CN201611180305.0A patent/CN106708573B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR20150030332A (en) * | 2013-09-12 | 2015-03-20 | 삼성전자주식회사 | Distributed and parallel processing system on data and method of operating the same |
| CN104317610A (en) * | 2014-10-11 | 2015-01-28 | 福建新大陆软件工程有限公司 | Method and device for automatic installation and deployment of hadoop platform |
| CN105959134A (en) * | 2016-04-22 | 2016-09-21 | 汉柏科技有限公司 | Automatic installation deployment method based on PXE, and PXE server |
| CN106126303A (en) * | 2016-07-01 | 2016-11-16 | 合信息技术（北京）有限公司 | The installation method of (SuSE) Linux OS, Apparatus and system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN106708573A (en) | 2017-05-24 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106708573B (en) | System and method for Hadoop cluster automatic installation | |
| US10838747B2 (en) | Virtual appliances | |
| CN111666080A (en) | Micro-service cluster deployment method and device, computer equipment and storage medium | |
| CN110908859B (en) | Application service processing method, device, terminal and storage medium | |
| US10797952B1 (en) | Intelligent rollback analysis of configuration changes | |
| CN113296792B (en) | Storage method, device, equipment, storage medium and system | |
| CN103984575A (en) | Rapid deployment method of cluster Linux operation system in cloud-computing environment | |
| CN106549789B (en) | Method and system for realizing server installation | |
| CN103580908A (en) | Server configuration method and system | |
| US9847903B2 (en) | Method and apparatus for configuring a communication system | |
| WO2023160033A1 (en) | Virtual network card resource configuration method and apparatus, computer device, and medium | |
| EP3879875A1 (en) | Resource change method and device, apparatus, and storage medium | |
| CN113127150A (en) | Rapid deployment method and device of cloud native system, electronic equipment and storage medium | |
| CN108228197B (en) | Method and device for installing software in cluster | |
| US20220237090A1 (en) | Autonomous organization and role selection of homogenous workers | |
| CN113590169B (en) | Application deployment method, application deployment system, and computer-readable storage medium | |
| CN108255500B (en) | Cobbler-based operating system automatic installation method compatible with virtualization architecture | |
| CN111209125B (en) | Multi-process command line implementation method | |
| JP2015099547A (en) | Software update method, software update apparatus, software update program, and software update system | |
| CN112328363A (en) | Cloud hard disk mounting method and device | |
| EP3843361A1 (en) | Resource configuration method and apparatus, and storage medium | |
| US10884774B2 (en) | Virtual node deployments of cluster-based applications modified to exchange reference to file systems | |
| CN113467893B (en) | Automatic testing method and device for virtual machine, storage medium and processor | |
| CN107172161A (en) | A kind of method of rapid installing operating system, client and server cluster | |
| KR102156656B1 (en) | System and method for creating stack of virtual machine |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |