CN111459502A - Method for deploying big data service on Ambari-Blueprint cloud - Google Patents
Method for deploying big data service on Ambari-Blueprint cloud Download PDFInfo
- Publication number
- CN111459502A CN111459502A CN202010193746.4A CN202010193746A CN111459502A CN 111459502 A CN111459502 A CN 111459502A CN 202010193746 A CN202010193746 A CN 202010193746A CN 111459502 A CN111459502 A CN 111459502A
- Authority
- CN
- China
- Prior art keywords
- service
- components
- blueprint
- master
- ambari
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 22
- 238000004422 calculation algorithm Methods 0.000 claims abstract description 16
- 239000000306 component Substances 0.000 claims description 62
- 235000002905 Rumex vesicarius Nutrition 0.000 claims description 8
- 244000207667 Rumex vesicarius Species 0.000 claims description 7
- 230000008569 process Effects 0.000 claims description 7
- 239000008358 core component Substances 0.000 claims description 4
- 238000009434 installation Methods 0.000 claims description 3
- 238000012544 monitoring process Methods 0.000 abstract description 3
- 230000007613 environmental effect Effects 0.000 abstract description 2
- 238000013515 script Methods 0.000 description 5
- 244000062793 Sorghum vulgare Species 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 235000019713 millet Nutrition 0.000 description 2
- 240000000797 Hibiscus cannabinus Species 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 239000000463 material Substances 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F8/00—Arrangements for software engineering
- G06F8/60—Software deployment
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- General Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The invention provides a method for deploying big data service on Ambari-Blueprint cloud, which comprises the following steps: s1, acquiring hardware information, installed services and the number type of virtual machines required by a user; s2, acquiring corresponding component information of the purchased service; s3, judging the type of the component, and dividing the component into a master service component and a core service component according to the type of the component; s4, distributing the components to each host according to an algorithm strategy; s5, generating a blueprint file by the algorithm rule; s6, registering the blueprint to the target cluster; and S7, deploying all the components. The blueprint technology of the invention combines the environmental parameters of the host computer purchased by the user and the like to realize the distribution of the components by the equalization algorithm, thereby simplifying the complexity of single service deployment and the high cost of developing the monitoring system.
Description
Technical Field
The invention belongs to the technical field of big data, and particularly relates to a method for deploying big data service on an Ambari-Blueprint cloud.
Background
At present, version dependency among big data service components is strong, a script mode is adopted to deploy the big data components such as hadoop and spark respectively, and a deployment script needs to be developed for each service. The system is not only not easy to maintain, but also needs to be self-generated to complete the monitoring of the service assembly, and a large amount of manpower, material resources and financial resources of a company can be consumed. "false yellow millet cannot be drunk but also reaches thousand miles due to lack of profit, and yellow millet cannot be drunk but also reaches extremely river". Ambari is a very good "sharps". We should improve his integration into his own platform product.
Disclosure of Invention
In view of this, the invention aims to provide a method for deploying big data services on the Ambari-Blueprint cloud, so as to solve the problem that deployment scripts need to be developed for each service when a script mode is adopted to deploy big data components.
In order to achieve the purpose, the technical scheme of the invention is realized as follows:
the method for deploying the big data service on the Ambari-Blueprint cloud comprises the following steps:
s1, acquiring hardware information, installed services and the number type of virtual machines required by a user;
s2, acquiring corresponding component information of the purchased service;
s3, judging the type of the component, and dividing the component into a master service component and a core service component according to the type of the component;
s4, distributing the components to each host according to an algorithm strategy;
s5, generating a blueprint file by the algorithm rule;
s6, registering the blueprint to the target cluster;
and S7, deploying all the components.
Further, the hardware information in step S1 includes a host type, a network, a host IP, a user name, a password, and a service.
Further, the host types include: master node, core node;
the service comprises the following steps: the system comprises a master node, a slave node and a register node, wherein the master node is installed with the master node, and the master node is installed with the master node and the register node is installed with the master node, and the master node is installed with the master node, and the register node is installed with the master node, the master node and the register node.
Further, the hadoop service comprises hdfs service and yarn service;
the HDFS service comprises NAMENODE service, SECONDARY _ NAMENODE service, DATANODE service and HDFS _ C L IENT 'service, wherein the NAMENODE service and the SECONDARY _ NAMENODE service are master service components, and the DATANODE service and the HDFS _ C L IENT' service are core service components;
the YARN service comprises RESOURCEMANNAGER service, APP _ TIME L INE _ SERVER service, NODEMANAGER service and YARN _ C L IENT 'service, wherein the RESOURCEMANNAGER service, the APP _ TIME L INE _ SERVER service are master service components, and the NODEMANAGER service, the YARN _ C L IENT' service are core service components.
Further, the algorithm strategy in step S4 is as follows: the number difference of the components of each master is not more than 1 by adopting a balance algorithm, the number difference of the components of each corehosts is not more than 1, and if the components are high-availability installation, part of master components need to be installed on two master nodes; ZOOKEEPER _ SERVER in the core components is only installed on 3 core nodes, and other core components are respectively deployed on other core nodes.
Further, the process of registering the blueprint to the target cluster in step S6 is as follows: curl-k-H "X-Requested-By: ambari "-u admin: admin-X POST https: /'+ ambariServerhost +': 8443/api/v 1/blueprints/'+ blueprints name +'? valid _ topology ═ false-d @/json/blue.
Further, the process of deploying all components in step S7 is as follows: curl-k-H "X-Requested-By: ambari "-X POST-u admin: admin https: /'+ ambariServerhost +': 8443/api/v 1/clusterirs/'+ clusteriName +'.-d@./json/hostmap.json。
Compared with the prior art, the invention has the following advantages:
(1) the blueprint technology of the invention combines the environmental parameters of the host computer purchased by the user and the like to realize the distribution of the components by the equalization algorithm, thereby simplifying the complexity of single service deployment and the high cost of developing the monitoring system.
(2) The Ambari-Blueprint technology disclosed by the invention is combined with a specific balanced deployment strategy, so that the deployment of big data services on the cloud is well realized, and the batch service deployment can be realized without using an additional operation and maintenance tool and a complex development script.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate an embodiment of the invention and, together with the description, serve to explain the invention and not to limit the invention. In the drawings:
fig. 1 is a flowchart of a method for deploying big data services on Ambari-Blueprint based cloud according to an embodiment of the present invention;
fig. 2 is an exemplary diagram of a hostmap file according to an embodiment of the present invention;
FIG. 3 is a diagram illustrating a blueprint file according to an embodiment of the present invention; .
Detailed Description
It should be noted that the embodiments and features of the embodiments may be combined with each other without conflict.
In the description of the present invention, it is to be understood that the terms "center", "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", and the like, indicate orientations or positional relationships based on those shown in the drawings, and are used only for convenience in describing the present invention and for simplicity in description, and do not indicate or imply that the referenced devices or elements must have a particular orientation, be constructed and operated in a particular orientation, and thus, are not to be construed as limiting the present invention. Furthermore, the terms "first", "second", etc. are used for descriptive purposes only and are not to be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first," "second," etc. may explicitly or implicitly include one or more of that feature. In the description of the present invention, "a plurality" means two or more unless otherwise specified.
In the description of the present invention, it is to be noted that, unless otherwise explicitly specified or limited, the terms "mounted," "connected," and "connected" are to be construed broadly, e.g., as meaning either a fixed connection, a removable connection, or an integral connection; can be mechanically or electrically connected; they may be connected directly or indirectly through intervening media, or they may be interconnected between two elements. The specific meaning of the above terms in the present invention can be understood by those of ordinary skill in the art through specific situations.
The present invention will be described in detail below with reference to the embodiments with reference to the attached drawings.
As shown in fig. 1 to 3, the method for deploying big data service on Ambari-Blueprint based cloud includes the following steps:
s1, acquiring hardware information, installed services and the number type of virtual machines required by a user;
s2, acquiring corresponding component information of the purchased service;
s3, judging the type of the component, and dividing the component into a master service component and a core service component according to the type of the component;
s4, distributing the components to each host according to an algorithm strategy;
s5, generating a blueprint file by the algorithm rule;
s6, registering the blueprint to the target cluster;
and S7, deploying all the components.
The hardware information in the step S1 includes host type, network, host IP, user name, password, and service.
The host types include: master node, core node;
the service comprises the following steps: the system comprises a master node, a slave node and a register node, wherein the master node is installed with the master node, and the master node is installed with the master node and the register node is installed with the master node, and the master node is installed with the master node, and the register node is installed with the master node, the master node and the register node.
The hadoop service comprises hdfs service and yarn service;
the HDFS service comprises NAMENODE service, SECONDARY _ NAMENODE service, DATANODE service and HDFS _ C L IENT 'service, wherein the NAMENODE service and the SECONDARY _ NAMENODE service are master service components, and the DATANODE service and the HDFS _ C L IENT' service are core service components;
the YARN service comprises RESOURCEMANNAGER service, APP _ TIME L INE _ SERVER service, NODEMANAGER service and YARN _ C L IENT 'service, wherein the RESOURCEMANNAGER service, the APP _ TIME L INE _ SERVER service are master service components, and the NODEMANAGER service, the YARN _ C L IENT' service are core service components.
The algorithm strategy in the step S4 is as follows: the number difference of the components of each master is not more than 1 by adopting a balance algorithm, the number difference of the components of each master is not more than 1, if the components are high-availability installation, part of the master components need to be installed on two master nodes, zookeeper _ servers in the master components are only installed on 3 master nodes, and other master components are respectively deployed on other master nodes;
the master service component is allowed to be distributed on the master node only, and the core service component is allowed to be distributed on the core node only.
The process of registering blueprint to the target cluster in step S6 is as follows: curl-k-H "X-Requested-By: ambari "-u admin: admin-X POST https: /'+ ambariServerhost +': 8443/api/v 1/blueprints/'+ blueprints name +'? valid _ topology ═ false-d @/json/blue.
The process of deploying all components in the step S7 is as follows: curl-k-H "X-Requested-By: ambari' -XPOST-u admin: admin https: /'+ ambariServerhost +': 8443/api/v 1/clusterirs/'+ clusteriName +'.-d@./json/hostmap.json。
Json file configures that each host _ group in blueprint corresponds to a host ip address.
The specific implementation process comprises the following steps:
ambari provides a mechanism based on blueprint to distribute host positions of a cluster where components are located, and once the blueprint file exists, all service components can be deployed in the cluster;
and the background service program generates corresponding Buleprint configuration files Buleprint.json and hostmap.json according to information such as the host ip, the host type, the user name and the password transmitted from the foreground and the required service components and according to a certain rule, and finally registers the blueprint by the Rest API to realize service deployment.
The above description is only for the purpose of illustrating the preferred embodiments of the present invention and is not to be construed as limiting the invention, and any modifications, equivalents, improvements and the like that fall within the spirit and principle of the present invention are intended to be included therein.
Claims (7)
1. A method for deploying big data service on Ambari-Blueprint cloud is characterized by comprising the following steps:
s1, acquiring hardware information, installed services and the number type of virtual machines required by a user;
s2, acquiring corresponding component information of the purchased service;
s3, judging the type of the component, and dividing the component into a master service component and a core service component according to the type of the component;
s4, distributing the components to each host according to an algorithm strategy;
s5, generating a blueprint file by the algorithm rule;
s6, registering the blueprint to the target cluster;
and S7, deploying all the components.
2. The Ambari-Blueprint cloud based method of deploying big data services according to claim 1, characterized in that: the hardware information in the step S1 includes host type, network, host IP, user name, password, and service.
3. The Ambari-Blueprint cloud based method of deploying big data services according to claim 2, characterized in that: the host types include: master node, core node;
the service comprises the following steps: the system comprises a master node, a slave node and a register node, wherein the master node is installed with the master node, and the master node is installed with the master node and the register node is installed with the master node, and the master node is installed with the master node, and the register node is installed with the master node, the master node and the register node.
4. The Ambari-Blueprint cloud based method of deploying big data services according to claim 3, characterized in that: the hadoop service comprises hdfs service and yarn service;
the HDFS service comprises NAMENODE service, SECONDARY _ NAMENODE service, DATANODE service and HDFS _ C L IENT 'service, wherein the NAMENODE service and the SECONDARY _ NAMENODE service are master service components, and the DATANODE service and the HDFS _ C L IENT' service are core service components;
the YARN service comprises RESOURCEMANNAGER service, APP _ TIME L INE _ SERVER service, NODEMANAGER service and YARN _ C L IENT 'service, wherein the RESOURCEMANNAGER service, the APP _ TIME L INE _ SERVER service are master service components, and the NODEMANAGER service, the YARN _ C L IENT' service are core service components.
5. The Ambari-Blueprint cloud based method of deploying big data services according to claim 1, characterized in that: the algorithm strategy in the step S4 is as follows: the number difference of the components of each master is not more than 1 by adopting a balance algorithm, the number difference of the components of each corehosts is not more than 1, and if the components are high-availability installation, part of master components need to be installed on two master nodes; ZOOKEEPER _ SERVER in the core components is only installed on 3 core nodes, and other core components are respectively deployed on other core nodes.
6. The Ambari-Blueprint cloud based method of deploying big data services according to claim 1, characterized in that: the process of registering blueprint to the target cluster in step S6 is as follows: curl-k-H "X-Requested-By: ambari "-u admin: admin-X POST https: /'+ ambariServerhost +': 8443/api/v 1/blueprints/'+ blueprints name +'? valid _ topology ═ false-d @/json/blue.
7. The Ambari-Blueprint cloud based method of deploying big data services according to claim 1, characterized in that: the process of deploying all components in the step S7 is as follows: curl-k-H "X-Requested-By: ambari "-X POST-u admin: admin https: /'+ ambariServerhost +': 8443/api/v 1/clusterirs/'+ clusteriName +'.-d@./json/hostmap.json。
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010193746.4A CN111459502A (en) | 2020-03-18 | 2020-03-18 | Method for deploying big data service on Ambari-Blueprint cloud |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010193746.4A CN111459502A (en) | 2020-03-18 | 2020-03-18 | Method for deploying big data service on Ambari-Blueprint cloud |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111459502A true CN111459502A (en) | 2020-07-28 |
Family
ID=71676695
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010193746.4A Pending CN111459502A (en) | 2020-03-18 | 2020-03-18 | Method for deploying big data service on Ambari-Blueprint cloud |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111459502A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114039848A (en) * | 2021-09-30 | 2022-02-11 | 济南浪潮数据技术有限公司 | Method, device and equipment for realizing high availability of InCloudInsight management platform |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2011021087A (en) * | 2009-07-15 | 2011-02-03 | Lion Idemitsu Composites Co Ltd | Composite resin composition |
| CN106293818A (en) * | 2016-08-02 | 2017-01-04 | 浪潮电子信息产业股份有限公司 | Method for deleting installed service based on Ambari |
| CN109067599A (en) * | 2018-09-25 | 2018-12-21 | 山东浪潮云投信息科技有限公司 | A kind of method and device for disposing cluster |
| CN110209407A (en) * | 2019-06-12 | 2019-09-06 | 浪潮软件股份有限公司 | A kind of big data cluster automatically dispose system and method |
| CN110519100A (en) * | 2019-09-03 | 2019-11-29 | 浪潮云信息技术有限公司 | A kind of more cluster management methods, terminal and computer readable storage medium |
-
2020
- 2020-03-18 CN CN202010193746.4A patent/CN111459502A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2011021087A (en) * | 2009-07-15 | 2011-02-03 | Lion Idemitsu Composites Co Ltd | Composite resin composition |
| CN106293818A (en) * | 2016-08-02 | 2017-01-04 | 浪潮电子信息产业股份有限公司 | Method for deleting installed service based on Ambari |
| CN109067599A (en) * | 2018-09-25 | 2018-12-21 | 山东浪潮云投信息科技有限公司 | A kind of method and device for disposing cluster |
| CN110209407A (en) * | 2019-06-12 | 2019-09-06 | 浪潮软件股份有限公司 | A kind of big data cluster automatically dispose system and method |
| CN110519100A (en) * | 2019-09-03 | 2019-11-29 | 浪潮云信息技术有限公司 | A kind of more cluster management methods, terminal and computer readable storage medium |
Non-Patent Citations (2)
| Title |
|---|
| CISMAG: ""大数据集群管理系统设计与实现"", 《HTTPS://MP.WEIXIN.QQ.COM/S/WA_D7LRHRE02F3WY0NOCTA》 * |
| 高智勇等: "《COM+技术大全》", 31 August 2001, 北京：机械工业出版社 * |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114039848A (en) * | 2021-09-30 | 2022-02-11 | 济南浪潮数据技术有限公司 | Method, device and equipment for realizing high availability of InCloudInsight management platform |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10445069B2 (en) | System and method for generating an application structure for an application in a computerized organization | |
| EP2899947A1 (en) | Component oriented hybrid cloud operating system architecture and communication method thereof | |
| EP2388703A1 (en) | Techniques for evaluating and managing cloud networks | |
| CN114301972B (en) | Hierarchical deployment method and system for blockchain nodes based on cloud edge cooperation | |
| WO2019174000A1 (en) | Method and apparatus for service management | |
| CN110896506B (en) | Techniques for secure partitioning of an optical transmission system to provide multi-client management access and network management systems implementing the same | |
| US11509527B1 (en) | Assisted and context-driven network changes | |
| US7240109B2 (en) | Remote services system service module interface | |
| CN112434302B (en) | Multitask collaboration vulnerability platform and construction method and service method thereof | |
| US20190109866A1 (en) | Multi-verifier approach for attestation of nodes in a network | |
| WO2022267175A1 (en) | Information processing method and apparatus, and computer device and storage medium | |
| CN111641521A (en) | Ambari-based method for deploying big data clusters on cloud | |
| CN112286705A (en) | Kubernetes-based container web service interface aggregation system | |
| CN111459502A (en) | Method for deploying big data service on Ambari-Blueprint cloud | |
| US20030149889A1 (en) | Automatic communication and security reconfiguration for remote services | |
| US11622252B2 (en) | Methods and systems for management and control of communication network | |
| EP1333643A2 (en) | Remote services system data delivery mechanism | |
| US20240205068A1 (en) | Using generic key-value pairs to configure ran component attributes in a ran system | |
| CN114285859B (en) | Data processing method, device, equipment and storage medium for middle layer block chain service | |
| US20120254942A1 (en) | Connection destination determination device, connection destination determination method, and service collaboration system | |
| CN111683148B (en) | Service processing system and method, service publishing method | |
| CN115221237A (en) | Node deployment method, application chain deployment method, device, server and computer program | |
| CN112837156A (en) | Transaction method and device with fixed execution sequence | |
| US20030149870A1 (en) | Remote services wide area network connection anti-spoofing control | |
| CN109257436A (en) | A kind of load balancing configuration method and relevant apparatus |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20200728 |