CN110737445A - method for installing big data software based on intelligent planning - Google Patents
method for installing big data software based on intelligent planning Download PDFInfo
- Publication number
- CN110737445A CN110737445A CN201911013079.0A CN201911013079A CN110737445A CN 110737445 A CN110737445 A CN 110737445A CN 201911013079 A CN201911013079 A CN 201911013079A CN 110737445 A CN110737445 A CN 110737445A
- Authority
- CN
- China
- Prior art keywords
- service
- server
- big data
- resource
- data software
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F8/00—Arrangements for software engineering
- G06F8/60—Software deployment
- G06F8/61—Installation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
- G06F9/5038—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals considering the execution order of a plurality of tasks, e.g. taking priority or time dependency constraints into consideration
Abstract
The invention discloses methods for installing big data software based on intelligent planning, which are implemented by the existing manual configuration service, wherein the steps to be implemented include step of manually setting a server environment, such as closing selinux, a firewall and the like, the second step of manually configuring installation parameters, the third step of manually executing an installation command, and the fourth step of manually adjusting the installation parameters if resources do not meet service operation requirements.
Description
Technical Field
The invention relates to the field of big data, in particular to methods for installing big data software based on intelligent planning.
Background
CDH is the Apache Hadoop core element-scalable storage and distributed computing-as well as Web-based user interface and important enterprise functions.
The cloud is used as data center management tools, various data computing frameworks capable of running quickly and stably are provided, such as Apache Spark, Apache Impala is used as a high-performance SQL query engine for HDFS and HBase, a Hive data warehouse tool is also provided to help users analyze data, the users can also manage and install HBase distributed type NoSQL databases by the cloud, the cloud also comprises a native Hadoop search engine and a cloud Navigator optimizer to perform visual coordination and optimization on computing tasks on the Hadoop, the operation efficiency is improved, and various components provided in the cloud can enable the users to conveniently manage, configure and monitor the Hadoop and other all related components in visual UI interfaces and have -specified fault tolerance processing.
The existing installation of big data products such as Cloudera CDH and the like, services for manually configuring and installing a big data management system have the problems of complex configuration, easy error occurrence in operation, large time consumption for solving after failure and high requirement on operators.
Disclosure of Invention
The invention aims to provide methods for installing big data software based on intelligent planning, and solves the problems that the service of a big data management system is manually configured and installed, the configuration is complex, the operation is easy to make mistakes, a large amount of time is needed for solving after the failure, and the requirement on operators is high.
The technical scheme adopted by the invention is as follows:
method for installing big data software based on intelligent planning, which comprises the following steps:
s1, configuring the connection information of the server needing to install the big data software;
s2, installing resource monitoring service on each server according to the connection information configured in the step S1;
s3, starting installation planning service, and acquiring resource occupation requirements of each service in the big data software to be installed;
s4, matching each service with the server according to the resource occupation requirement of each service acquired in the step S3 and the resource use condition of each server acquired by the resource monitoring service in the step S2;
s5, outputting installation planning information according to the matching result of the step S4;
and S6, executing automatic installation by the automatic installation program according to the installation planning information.
The method is different from the prior art, and provides methods capable of intelligently planning and installing hadoop big data products according to software and hardware resource information, automatically configuring various services of big data software, reasonably allocating the resources and installing the big data products, reducing the operation and maintenance cost, and solves the problems that the services of a manually configured and installed big data management system are complex in configuration, easy to fail in operation, and high in requirement on operators due to the fact that a large amount of time is spent for solving the problems after failure.
And , the connection information of the server in the step S1 includes a server IP, a ssh service port, a root account and a root password.
, the step S3, after starting the installation planning service and acquiring the resource occupation requirements of each service in the big data software that needs to be installed, further includes:
s301, judging whether the server resources can meet the resource requirements of the big data service according to the resource monitoring service installed in the step S2, if so, entering the step S4, and if not, lifting the installation personnel and simultaneously suspending the installation plan.
, matching each service with the server in the step S4 according to the resource occupation requirement of each service acquired in the step S3 and the resource usage of each server acquired by the resource monitoring service in the step S2, and distributing each service to each server according to the resource occupation amount, so as to minimize the difference of the total amount of resources occupied by the big data software on each server.
, matching each service with the server in the step S4 according to the resource occupation requirement of each service acquired in the step S3 and the resource usage of each server acquired by the resource monitoring service in the step S2, and installing the service with strong dependency relationship on the corresponding server.
, matching each service with the server in the step S4 according to the resource occupation requirement of each service acquired in the step S3 and the resource usage of each server acquired by the resource monitoring service in the step S2, so that the difference of the total resource usage of each server after the big data software is installed is minimized.
, the number of the big data software is M, the number of the servers is N, M, N is positive integer, the method for minimizing the difference of the total resource usage amount of each server after the big data software is installed comprises the following steps:
s401, traversing M services of big data software, and sequencing the M services according to the size of resources occupied by each service from big to small;
s402, traversing the resource monitoring services of the N servers, and sequencing the servers according to the proportion of the used resources of the servers from small to large;
s403, selecting the first N services sequenced in the step S401, and matching the first N services with the N servers sequenced in the step S402;
s404, removing the service matched with the server from the sequence formed by sequencing in the step S401, calculating the proportion of the used resources of the server loaded with the matched service by using the resource monitoring service, and reordering the service according to the sequence from small to large of the proportion of the used resources;
s405, judging whether the service which is not matched with the server exists in the sequence formed by sequencing in the step S401, if so, turning to the step S403, and if not, completing the matching of the service and the server.
In summary, due to the adoption of the technical scheme, the invention has the beneficial effects that:
1. the method for installing big data software based on intelligent planning intelligently plans and installs, avoids readjusting and installing in subsequent installation, greatly prompts the installation efficiency, and improves the current installation time from 4 hours of manual installation to 45 minutes;
2. according to the method for installing the big data software based on the intelligent planning, the big data service is reasonably installed according to the software and hardware resource condition of the server, and the cluster resource waste caused by unreasonable resource use after periods of operation is avoided;
3. the method for installing big data software based on intelligent planning reduces the installation difficulty of operation and maintenance personnel and reduces the operation and maintenance cost.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are embodiments of the present invention, and other drawings can be obtained by those skilled in the art without creative efforts, wherein:
fig. 1 is a schematic view of the installation process of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention clearer, the present invention will be described in further detail with reference to fig. 1, the described embodiments should not be construed as limiting the present invention, and all other embodiments obtained by those skilled in the art without inventive labor fall within the scope of the present invention.
Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. The terminology used herein is for the purpose of describing embodiments of the invention only and is not intended to be limiting of the invention.
Before describing the embodiments of the present invention in further detail , terms and expressions used in the embodiments of the present invention will be described, and the terms and expressions used in the embodiments of the present invention will be explained as follows.
Example 1
method for installing big data software based on intelligent planning, which comprises the following steps:
s1, configuring the connection information of the server needing to install the big data software;
s2, installing resource monitoring service on each server according to the connection information configured in the step S1;
s3, starting installation planning service, and acquiring resource occupation requirements of each service in the big data software to be installed;
s4, matching each service with the server according to the resource occupation requirement of each service acquired in the step S3 and the resource use condition of each server acquired by the resource monitoring service in the step S2;
s5, outputting installation planning information according to the matching result of the step S4;
and S6, executing automatic installation by the automatic installation program according to the installation planning information.
The method is different from the prior art, and provides methods capable of intelligently planning and installing hadoop big data products according to software and hardware resource information, automatically configuring various services of big data software, reasonably allocating the resources and installing the big data products, reducing the operation and maintenance cost, and solves the problems that the services of a manually configured and installed big data management system are complex in configuration, easy to fail in operation, and high in requirement on operators due to the fact that a large amount of time is spent for solving the problems after failure.
Example 2
In this embodiment, based on step of embodiment 1, the connection information of the server in step S1 includes a server IP, a ssh service port, a root account, and a root password.
, the step S3, after starting the installation planning service and acquiring the resource occupation requirements of each service in the big data software that needs to be installed, further includes:
s301, judging whether the server resources can meet the resource requirements of the big data service according to the resource monitoring service installed in the step S2, if so, entering the step S4, and if not, lifting the installation personnel and simultaneously suspending the installation plan.
, matching each service with the server in the step S4 according to the resource occupation requirement of each service acquired in the step S3 and the resource usage of each server acquired by the resource monitoring service in the step S2, and distributing each service to each server according to the resource occupation amount, so as to minimize the difference of the total amount of resources occupied by the big data software on each server.
Example 3
Based on embodiment 1, , in the step S4, the services are matched with the servers according to the resource occupation requirement of each service acquired in the step S3 and the resource usage of each server acquired by the resource monitoring service in the step S2, and the services with strong dependency relationship are installed on the corresponding servers.
Example 4
In this embodiment, based on the step of embodiment 1, in the step S4, the matching between each service and the server is completed according to the resource occupation requirement of each service acquired in the step S3 and the resource usage of each server acquired by the resource monitoring service in the step S2, so that the difference of the total resource usage of each server after the large data software is installed is minimized.
, the number of the big data software is M, the number of the servers is N, M, N is positive integer, the method for minimizing the difference of the total resource usage amount of each server after the big data software is installed comprises the following steps:
s401, traversing M services of big data software, and sequencing the M services according to the size of resources occupied by each service from big to small;
s402, traversing the resource monitoring services of the N servers, and sequencing the servers according to the proportion of the used resources of the servers from small to large;
s403, selecting the first N services sequenced in the step S401, and matching the first N services with the N servers sequenced in the step S402;
s404, removing the service matched with the server from the sequence formed by sequencing in the step S401, calculating the proportion of the used resources of the server loaded with the matched service by using the resource monitoring service, and reordering the service according to the sequence from small to large of the proportion of the used resources;
s405, judging whether the service which is not matched with the server exists in the sequence formed by sequencing in the step S401, if so, turning to the step S403, and if not, completing the matching of the service and the server.
Example 4
The embodiment is a part of function codes of the scheme:
the code of the part is an installation strategy of more than CmptStrategDefault-5:
the above-described apparatus embodiments are merely illustrative, and for example, the flowcharts and block diagrams in the figures may illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods and computer program products according to various embodiments of the present invention.
In addition, each functional module in each embodiment of the present invention may be integrated in to form independent parts, or each module may exist separately, or two or more modules may be integrated to form independent parts.
It should be noted that, in this document, relational terms such as and second, etc. are used solely to distinguish entities or operations from another entities or operations without necessarily requiring or implying any actual such relationship or order between such entities or operations, and further, the terms "comprises", "comprising", or any other variation thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a series of elements is intended to exclude not only those elements but also those elements expressly listed or otherwise included in the same list, and other elements are not intended to exclude other elements from the list.
It should be noted that like reference numerals and letters refer to like elements in the following figures, and thus , once is defined in figures, it is not necessary to further define or interpret in the following figures.
The above description is only for the specific embodiments of the present invention, but the scope of the present invention is not limited thereto, and any person skilled in the art can easily conceive of the changes or substitutions within the technical scope of the present invention, and all the changes or substitutions should be covered within the scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.
Claims (7)
- The method for installing big data software based on intelligent planning is characterized by comprising the following steps of:s1, configuring the connection information of the server needing to install the big data software;s2, installing resource monitoring service on each server according to the connection information configured in the step S1;s3, starting installation planning service, and acquiring resource occupation requirements of each service in the big data software to be installed;s4, matching each service with the server according to the resource occupation requirement of each service acquired in the step S3 and the resource use condition of each server acquired by the resource monitoring service in the step S2;s5, outputting installation planning information according to the matching result of the step S4;and S6, executing automatic installation by the automatic installation program according to the installation planning information.
- 2. The method for installing big data software based on intelligent planning as claimed in claim 1, wherein the connection information of the server in step S1 includes server IP, ssh service port, root account and root password.
- 3. The method for installing big data software based on intelligent planning as claimed in claim 1, wherein the step S3 of starting installation planning services, after obtaining the resource occupation requirement of each service in the big data software to be installed, further comprises:s301, judging whether the server resources can meet the resource requirements of the big data service according to the resource monitoring service installed in the step S2, if so, entering the step S4, and if not, lifting the installation personnel and simultaneously suspending the installation plan.
- 4. The method for installing big data software based on intelligent planning as claimed in claim 1, wherein the step S4 is performed by matching each service with the server according to the resource occupation requirement of each service obtained in the step S3 and the resource usage of each server obtained by the resource monitoring service in the step S2, and the services are evenly distributed to each server according to the resource occupation amount, so that the difference of the total amount of resources occupied by the big data software on each server is minimized.
- 5. The method for installing big data software based on intelligent planning of claim 1, wherein the step S4 comprises matching each service with the server according to the resource occupation requirement of each service obtained in the step S3 and the resource usage of each server obtained by the resource monitoring service in the step S2, and installing the service with strong dependency relationship on the corresponding server.
- 6. The method for installing big data software based on intelligent planning as claimed in claim 1, wherein the matching between each service and server in step S4 is done according to the resource occupation requirement of each service obtained in step S3 and the resource usage of each server obtained by the resource monitoring service in step S2, so as to minimize the difference in the total resource usage of each server after installing big data software.
- 7. The method for installing big data software based on intelligent planning of claim 6, wherein the number of services of the big data software is M, the number of servers is N, M, N are positive integers, and the method for minimizing the difference in the total resource usage of each server after installing big data software comprises the following steps:s401, traversing M services of big data software, and sequencing the M services according to the size of resources occupied by each service from big to small;s402, traversing the resource monitoring services of the N servers, and sequencing the servers according to the proportion of the used resources of the servers from small to large;s403, selecting the first N services sequenced in the step S401, and matching the first N services with the N servers sequenced in the step S402;s404, removing the service matched with the server from the sequence formed by sequencing in the step S401, calculating the proportion of the used resources of the server loaded with the matched service by using the resource monitoring service, and reordering the service according to the sequence from small to large of the proportion of the used resources;s405, judging whether the service which is not matched with the server exists in the sequence formed by sequencing in the step S401, if so, turning to the step S403, and if not, completing the matching of the service and the server.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911013079.0A CN110737445A (en) | 2019-10-23 | 2019-10-23 | method for installing big data software based on intelligent planning |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911013079.0A CN110737445A (en) | 2019-10-23 | 2019-10-23 | method for installing big data software based on intelligent planning |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110737445A true CN110737445A (en) | 2020-01-31 |
Family
ID=69271012
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911013079.0A Pending CN110737445A (en) | 2019-10-23 | 2019-10-23 | method for installing big data software based on intelligent planning |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110737445A (en) |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2012048330A (en) * | 2010-08-25 | 2012-03-08 | Nippon Telegr & Teleph Corp <Ntt> | Cluster system and software deployment method |
| US20120180039A1 (en) * | 2011-01-11 | 2012-07-12 | International Business Machines Corporation | Automated Deployment of Applications with Tenant-Isolation Requirements |
| CN104317610A (en) * | 2014-10-11 | 2015-01-28 | 福建新大陆软件工程有限公司 | Method and device for automatic installation and deployment of hadoop platform |
| CN106325998A (en) * | 2015-06-30 | 2017-01-11 | 华为技术有限公司 | Method and device for deploying application based on cloud computing |
-
2019
- 2019-10-23 CN CN201911013079.0A patent/CN110737445A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2012048330A (en) * | 2010-08-25 | 2012-03-08 | Nippon Telegr & Teleph Corp <Ntt> | Cluster system and software deployment method |
| US20120180039A1 (en) * | 2011-01-11 | 2012-07-12 | International Business Machines Corporation | Automated Deployment of Applications with Tenant-Isolation Requirements |
| CN104317610A (en) * | 2014-10-11 | 2015-01-28 | 福建新大陆软件工程有限公司 | Method and device for automatic installation and deployment of hadoop platform |
| CN106325998A (en) * | 2015-06-30 | 2017-01-11 | 华为技术有限公司 | Method and device for deploying application based on cloud computing |
Non-Patent Citations (1)
| Title |
|---|
| 蒋少华 等: "基于Docker的计算机应用快速部署系统的设计", 《韶关学院学报》 * |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN103365708B (en) | Method and device for scheduling tasks | |
| US8978032B2 (en) | Host naming application programming interface | |
| US20140358844A1 (en) | Workflow controller compatibility | |
| CN110751358A (en) | Scheduling method for airport ground service personnel, electronic equipment and storage medium | |
| CN105119757A (en) | Method and system for operation and maintenance automation of enterprise servers | |
| US20110258620A1 (en) | Method and Apparatus for Making a BPM Application Available to Multiple Tenants | |
| CN110619014A (en) | ETL-based data extraction method | |
| EP3598297A1 (en) | Systems and methods for comparing computer scripts | |
| CN115391444A (en) | Heterogeneous data acquisition and interaction method, device, equipment and storage medium | |
| CN111932099A (en) | Marketing business management system and marketing business management method | |
| CN110134533B (en) | System and method capable of scheduling data in batches | |
| CN107329809A (en) | A kind of distributed transaction processing method and system towards multi-data source | |
| CN114066394A (en) | Implementation method of multi-tenant workflow engine and multi-tenant workflow engine system | |
| CN110737445A (en) | method for installing big data software based on intelligent planning | |
| CN113392154A (en) | Data system based on SOA framework and distributed deployment database | |
| CN104793928A (en) | Developing and running platform implementation method and system based on Java | |
| CN110019118A (en) | The system and method for database O&M efficiency are promoted based on DevOps | |
| CN109450670B (en) | Instruction conflict judgment method and system in artificial intelligence mode | |
| CN113722141B (en) | Method and device for determining delay reason of data task, electronic equipment and medium | |
| CN112988705B (en) | Data middlebox construction method for enterprise-level production | |
| CN107122246A (en) | Intelligent numerical simulation job management and feedback method | |
| US20200019910A1 (en) | Block-based prediction for manufacturing environments | |
| CN104778253B (en) | A kind of method and apparatus that data are provided | |
| CN111078777A (en) | Method for loading data based on dynamic increment of relational database | |
| CN112328683A (en) | Data transmission method, system and storage medium supporting database and library direct |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WD01 | Invention patent application deemed withdrawn after publication |
Application publication date: 20200131 |
|
| WD01 | Invention patent application deemed withdrawn after publication |