CN113127016B - Automatic deployment method, device, equipment and medium for Hdp big data platform - Google Patents

Automatic deployment method, device, equipment and medium for Hdp big data platform Download PDF

Info

Publication number
CN113127016B
CN113127016B CN202110485518.9A CN202110485518A CN113127016B CN 113127016 B CN113127016 B CN 113127016B CN 202110485518 A CN202110485518 A CN 202110485518A CN 113127016 B CN113127016 B CN 113127016B
Authority
CN
China
Prior art keywords
installation
script
configuration file
hdp
package
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202110485518.9A
Other languages
Chinese (zh)
Other versions
CN113127016A (en
Inventor
黄一格
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Ping An International Smart City Technology Co Ltd
Original Assignee
Ping An International Smart City Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Ping An International Smart City Technology Co Ltd filed Critical Ping An International Smart City Technology Co Ltd
Priority to CN202110485518.9A priority Critical patent/CN113127016B/en
Publication of CN113127016A publication Critical patent/CN113127016A/en
Application granted granted Critical
Publication of CN113127016B publication Critical patent/CN113127016B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F8/00—Arrangements for software engineering
    • G06F8/60—Software deployment
    • G06F8/61—Installation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/21—Design, administration or maintenance of databases
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/28—Databases characterised by their database models, e.g. relational or object models
    • G06F16/284—Relational databases
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/44—Arrangements for executing specific programs
    • G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
    • G06F9/45504—Abstract machines for programme code execution, e.g. Java virtual machine [JVM], interpreters, emulators
    • G06F9/45508—Runtime interpretation or emulation, e g. emulator loops, bytecode interpretation
    • G06F9/45512—Command shells
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
    • H04L41/08—Configuration management of networks or network elements
    • H04L41/0876—Aspects of the degree of configuration automation
    • H04L41/0886—Fully automatic configuration

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Software Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Databases & Information Systems (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • Automation & Control Theory (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Stored Programmes (AREA)

Abstract

The application relates to the technical field of big data, and discloses an automatic deployment method, device, equipment and medium of an Hdp big data platform, wherein the method comprises the following steps: acquiring and executing a script main program of an automatic deployment script; based on the execution of the script main program, calling an environment configuration script to obtain a server configuration completion result according to the environment configuration file, calling a basic software package installation script of the automatic deployment script according to the server configuration completion result to obtain a basic software installation completion result according to the environment configuration file, the basic software package installation configuration file and a basic software package set to be installed, and calling an Hdp cluster installation and starting script according to the basic software installation completion result to be used for carrying out Hdp cluster installation and service starting according to the environment configuration file, the Hdp cluster installation configuration file and the Hdp cluster software package set to obtain the target Hdp big data platform. The deployment efficiency and success rate are improved, and the difficulty of later maintenance is reduced.

Description

Automatic deployment method, device, equipment and medium for Hdp big data platform
Technical Field
The application relates to the technical field of big data, in particular to an automatic deployment method, an automatic deployment device, automatic deployment equipment and an automatic deployment medium for an Hdp big data platform.
Background
An Hdp big data platform (Hortonworks Data Platform, abbreviated as "Hdp") is one of the mainstream platforms that runs big data analysis programs. The platform needs to be deployed at a large number of different customer sites to perform data analysis on the customer data. The traditional deployment method is that deployment personnel conduct deployment step by step according to the official network deployment document. The traditional deployment method has the following defects: (1) The deployment efficiency is low, so that the deployment can be successfully performed in a plurality of days, and the project progress is greatly influenced; (2) The deployment process is easy to make mistakes, the levels of deployment personnel are uneven, and the probability of error in the deployment process is high; (3) The deployment mode of each deployment personnel is difficult to achieve absolute unification, and later maintenance is difficult.
Disclosure of Invention
The application mainly aims to provide an automatic deployment method, device, equipment and medium for an Hdp big data platform, and aims to solve the technical problems that deployment personnel are adopted to deploy the Hdp big data platform step by step according to a official network deployment document in the prior art, the deployment efficiency is low, errors are easy to occur in the deployment process, and later maintenance is difficult.
In order to achieve the above object, the present application provides an automated deployment method for an Hdp big data platform, the method comprising:
acquiring a main program operation instruction;
responding to the main program operation instruction, acquiring and executing a script main program of an automatic deployment script, wherein the script main program is used for indicating the deployment operation of an Hdp big data platform;
Calling an environment configuration script of the automatic deployment script based on the execution of the script main program, acquiring an environment configuration file of a target configuration file, and performing server configuration according to the environment configuration file to obtain a server configuration completion result;
Based on the execution of the script main program, calling a basic software package installation script of the automatic deployment script according to the server configuration completion result, obtaining a basic software package set to be installed and a basic software package installation configuration file of the target configuration file, and performing basic software installation according to the environment configuration file, the basic software package installation configuration file and the basic software package set to be installed to obtain a basic software installation completion result;
And calling an Hdp cluster installation and starting script of the automatic deployment script according to the result of the installation of the basic software based on the execution of the script main program, obtaining an Hdp cluster software package set and an Hdp cluster installation configuration file of the target configuration file, and carrying out the installation and service starting of the Hdp cluster according to the environment configuration file, the Hdp cluster installation configuration file and the Hdp cluster software package set to obtain the target Hdp big data platform.
The application also provides an automatic deployment device of the Hdp big data platform, which comprises:
the instruction acquisition module is used for acquiring a main program operation instruction;
The script acquisition module is used for responding to the main program operation instruction, acquiring and executing a script main program of the automatic deployment script, wherein the script main program is used for indicating the deployment operation of the Hdp big data platform;
The server configuration completion result determining module is used for calling an environment configuration script of the automatic deployment script based on the execution of the script main program, obtaining an environment configuration file of a target configuration file, and carrying out server configuration according to the environment configuration file to obtain a server configuration completion result;
The base software installation completion result determining module is used for calling a base software package installation script of the automatic deployment script according to the execution of the script main program and the server configuration completion result, obtaining a base software package set to be installed and a base software package installation configuration file of the target configuration file, and performing base software installation according to the environment configuration file, the base software package installation configuration file and the base software package set to be installed to obtain a base software installation completion result;
The target Hdp big data platform determining module is used for calling an Hdp cluster installation and starting script of the automatic deployment script according to the execution of the script main program and the result of the completion of the installation of the basic software to obtain an Hdp cluster software package set and an Hdp cluster installation configuration file of the target configuration file, and performing the installation and service starting of the Hdp cluster according to the environment configuration file, the Hdp cluster installation configuration file and the Hdp cluster software package set to obtain the target Hdp big data platform.
The application also proposes a computer device comprising a memory storing a computer program and a processor implementing the steps of any of the methods described above when the processor executes the computer program.
The application also proposes a computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, implements the steps of the method of any of the above.
The application relates to an automatic deployment method, a device, equipment and a medium for an Hdp big data platform, which are characterized in that firstly, a script main program of an automatic deployment script is obtained and executed in response to a main program operation instruction, the script main program is used for indicating the deployment operation of the Hdp big data platform, secondly, based on the execution of the script main program, an environment configuration script of the automatic deployment script is called, an environment configuration file of a target configuration file is obtained, server configuration is carried out according to the environment configuration file, a server configuration completion result is obtained, then, based on the execution of the script main program, a basic software package installation script of the automatic deployment script is called according to the server configuration completion result, a basic software package set to be installed and a basic software package installation configuration file of the target configuration file are obtained, basic software installation is carried out according to the environment configuration file, the basic software package installation configuration file and the basic software package set to be installed, obtaining a result of the completion of the basic software installation, finally calling an Hdp cluster installation and starting script of an automatic deployment script according to the result of the completion of the basic software installation, obtaining an Hdp cluster installation configuration file of an Hdp cluster software package set and a target configuration file, and carrying out the installation and service starting of an Hdp cluster according to the environment configuration file, the Hdp cluster installation configuration file and the Hdp cluster software package set to obtain a target Hdp big data platform, thereby realizing the automatic uniform deployment of the Hdp big data platform according to the operation instruction of the main program, avoiding the technical problems of low deployment efficiency, easy error in the deployment process and difficult later maintenance due to the adoption of deployment personnel to deploy according to a official network deployment document step by step in the prior art, the deployment efficiency is improved, and the success rate of deployment is improved; because the deployment paths and the deployment methods are consistent, the difficulty of maintenance and upgrading in the later stage is reduced.
Drawings
FIG. 1 is a flow chart of an automated deployment method of an Hdp big data platform according to an embodiment of the present application;
FIG. 2 is a schematic block diagram of an automated deployment apparatus for an Hdp big data platform according to an embodiment of the present application;
fig. 3 is a schematic block diagram of a computer device according to an embodiment of the present application.
The achievement of the objects, functional features and advantages of the present application will be further described with reference to the accompanying drawings, in conjunction with the embodiments.
Detailed Description
The present application will be described in further detail with reference to the drawings and examples, in order to make the objects, technical solutions and advantages of the present application more apparent. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the application.
In order to solve the technical problems that deployment staff deploys the Hdp big data platform step by step according to a official network deployment document in the prior art, deployment efficiency is low, errors are easy to occur in a deployment process, and later maintenance is difficult, the application provides an automatic deployment method of the Hdp big data platform, which is applied to the technical field of big data, and the method is further applied to the technical field of the big data platform. According to the automatic deployment method of the Hdp big data platform, server configuration, basic software installation and Hdp cluster installation and service starting are automatically carried out by responding to a main program operation instruction, so that unified deployment of the Hdp big data platform according to the main program operation instruction is realized, deployment personnel for deploying the Hdp big data platform step by step according to a official network deployment document in the prior art is avoided, the technical problems of low deployment efficiency, easy error in the deployment process and difficult later maintenance exist, the deployment difficulty is reduced, the deployment efficiency is improved, and the deployment success rate is improved; because the deployment paths and the deployment methods are consistent, the difficulty of maintenance and upgrading in the later stage is reduced.
Referring to fig. 1, in an embodiment of the present application, an automated deployment method for an Hdp big data platform is provided, where the method includes:
S1: acquiring a main program operation instruction;
S2: responding to the main program operation instruction, acquiring and executing a script main program of an automatic deployment script, wherein the script main program is used for indicating the deployment operation of an Hdp big data platform;
s3: calling an environment configuration script of the automatic deployment script based on the execution of the script main program, acquiring an environment configuration file of a target configuration file, and performing server configuration according to the environment configuration file to obtain a server configuration completion result;
S4: based on the execution of the script main program, calling a basic software package installation script of the automatic deployment script according to the server configuration completion result, obtaining a basic software package set to be installed and a basic software package installation configuration file of the target configuration file, and performing basic software installation according to the environment configuration file, the basic software package installation configuration file and the basic software package set to be installed to obtain a basic software installation completion result;
S5: and calling an Hdp cluster installation and starting script of the automatic deployment script according to the result of the installation of the basic software based on the execution of the script main program, obtaining an Hdp cluster software package set and an Hdp cluster installation configuration file of the target configuration file, and carrying out the installation and service starting of the Hdp cluster according to the environment configuration file, the Hdp cluster installation configuration file and the Hdp cluster software package set to obtain the target Hdp big data platform.
According to the embodiment, firstly, a script main program of an automatic deployment script is obtained and executed in response to a main program operation instruction, the script main program is used for indicating deployment operation of an Hdp big data platform, secondly, based on execution of the script main program, an environment configuration script of the automatic deployment script is called, an environment configuration file of a target configuration file is obtained, server configuration is carried out according to the environment configuration file, a server configuration completion result is obtained, then, based on execution of the script main program, a foundation software package installation script of the automatic deployment script is called according to the server configuration completion result, foundation software package installation configuration files of a foundation software package set to be installed and a foundation software package installation configuration file of the target configuration file are obtained, foundation software installation of the foundation software package installation configuration file is carried out according to the environment configuration file, the foundation software package installation configuration file and the foundation software package set to be installed, basic software installation completion result is obtained, finally, based on execution of the script main program, the environment configuration file installation completion result is called, the Hdp cluster installation and the target configuration file are carried out according to the Hdp cluster installation configuration file, the Hdp cluster installation and the Hdp cluster configuration file, the service deployment efficiency of the set is improved, the problem of the automatic deployment script is solved, the existing deployment platform is solved, the efficiency of the automatic deployment platform is easy to be improved, the problem of the deployment is solved, the existing deployment platform is easy to be deployed according to the existing in terms of the service deployment, the deployment platform is easy, the deployment of the system is easy, and the deployment of the service deployment platform is easy to be deployed, and the problem of the service deployment platform is easy to be deployed by the following a large-stage is due to the deployment stage to the service deployment stage has a high-stage to the service deployment efficiency, and has been achieved by the service deployment efficiency is achieved by the service deployment stage has a high service stage has a service deployment efficiency; because the deployment path and method are consistent, the difficulty of maintenance and upgrading in the later stage is reduced.
For S1, the main program operation instruction input by the user may be obtained, or the main program operation instruction input by the third party application system may be obtained.
The main program running instruction is an instruction of a script main program for running the automation deployment script.
The automation deployment script is a script developed by using Shell script language.
And S2, responding to the main program operation instruction, acquiring a script catalog, acquiring a script main program of the automatic deployment script according to the script catalog, and executing the script main program, so that the unified deployment operation of the Hdp big data platform is realized by executing the whole automatic deployment script through the execution of the script main program.
For example, the main script program is main.sh, and the scripts other than main.sh in the automatic deployment script can be executed only by running main.sh, so that the deployment operation of the Hdp big data platform can be realized by executing the scripts other than main.sh in the automatic deployment script, which is not particularly limited herein by way of example.
S3, acquiring a script catalog based on execution of the script main program, and acquiring an environment configuration script of the automatic deployment script according to the script catalog; and then, the logic of the environment configuration script is executed to acquire an environment configuration file of the target configuration file, server configuration is carried out according to the environment configuration file, a server configuration completion result is determined to be successful when all servers corresponding to the environment configuration file are successfully configured, and a server configuration completion result is determined to be failed when any server corresponding to the environment configuration file fails.
The script directory includes: script type, script name, preset script storage location.
The target configuration file is the configuration file of the Hdp big data platform deployment.
The environment configuration script is a script for configuring the installation environment of the Hdp big data platform.
For example, the environment configuration script is 1_init_env.sh, which is a script developed by using Shell script language, and is used for performing system name modification, password-free configuration and system parameter modification, and the example is not limited in detail herein.
The environment configuration file is a configuration file for configuring the installation environment of the Hdp big data platform. The environment configuration file includes: one or more pieces of environment configuration data, the environment configuration data comprising: machine IP address, machine name, cluster role, account number, password.
For example, the environment profile:
30.23.10.135,zhiniao-bigdata-1,Master,root,Mf3OJRzh；
30.23.10.136,zhiniao-bigdata-2,Slave,root,6P6Z4q6h；
30.23.10.137,zhiniao-bigdata-3,Slave,root,6zDPRhAA；
that is, the environment configuration file includes environment configuration data of 3 servers, wherein the first server is a Master server (Master server), the second server and the third server are Slave servers, and the server to be configured is determined to obtain a server set to be configured according to the environment configuration file, and the server set to be configured includes: the server corresponding to the machine IP address 30.23.10.135, the server corresponding to the machine IP address 30.23.10.136, and the server corresponding to the machine IP address 30.23.10.137 are not specifically limited.
It can be understood that when the server configuration completion result is failure, generating an environment configuration error signal according to the server configuration completion result, wherein the environment configuration error signal is used for suspending the automatic deployment of the Hdp big data platform and reminding a user of environment configuration errors.
For S4, when the server configuration completion result is successful, acquiring a script catalog based on the execution of the script main program, and acquiring a basic software package installation script of the automatic deployment script according to the script catalog; and then executing logic of the basic software package installation script, acquiring a basic software package installation configuration file of a basic software package set to be installed and the basic software package installation configuration file of the target configuration file, performing basic software installation on servers corresponding to the environment configuration file according to the basic software package installation configuration file and the basic software package set to be installed, determining that the basic software installation completion result is successful when all servers corresponding to the environment configuration file are successfully installed, and determining that the basic software installation completion result is failed when any server corresponding to the environment configuration file fails to be installed.
When the base software installation completion result is failure, generating a base software installation failure signal according to the base software installation completion result, wherein the base software installation failure signal is used for suspending the automatic deployment of the Hdp big data platform and reminding a user of the base software installation error.
The basic software package installation script includes: the system relies on RPM (Red-HAT PACKAGE MANAGER) package installation script, hdp (Hortonworks Data Platform) platform installation package installation script, mysql (relational database management system) installation package installation script, mysql initialization script, JDK (Java development kit) installation package installation script. The system relies on the RPM package installation script, i.e., the script that installs the system relies on the RPM package. The Hdp platform installation package installation script is a script for installing the Hdp platform installation package. The Mysql installation package installs the script, i.e., the script that installs the Mysql installation package. Mysql initialization script, i.e. the script that initializes the Mysql database. JDK installation package installation scripts, i.e., scripts that install JDK installation packages.
The base software package installation configuration file includes: the system relies on RPM package installation configuration files, hdp platform installation package installation configuration files, mysql initialization configuration files, and JDK installation package installation configuration files. The system relies on the RPM package installation profile, i.e., the installation system relies on the RPM package profile. The Hdp platform installation package installation configuration file is the configuration file for installing the Hdp platform installation package. The Mysql installation package installs the configuration file, i.e. the configuration file that installs the Mysql installation package. Mysql initialization profile, i.e. the profile that initializes the Mysql database. The JDK installation package installs the configuration file, i.e., installs the configuration file of the JDK installation package.
The set of base software packages to be installed includes: the target system depends on an RPM package, a target Hdp platform installation package, a target Mysql installation package, and a target JDK installation package. The target system relies on RPM packages, i.e., system RPM dependent installation packages. The target Hdp platform installation package is the installation package of the Hdp platform. The target Mysql installation package, i.e. the Mysql installation package. Target JDK installation packages, i.e., JDK installation packages.
For S5, when the installation completion result of the basic software is successful, acquiring a script catalog based on the execution of the script main program, and acquiring an Hdp cluster installation and starting script of the automatic deployment script according to the script catalog; and then logic of an Hdp cluster installation and starting script is executed, an Hdp cluster installation configuration file of the Hdp cluster software package set and the target configuration file is obtained, the server corresponding to the environment configuration file is used for carrying out the installation of the Hdp cluster according to the Hdp cluster installation configuration file and the Hdp cluster software package set, then service starting of the Hdp big data platform is carried out according to the environment configuration file, the Hdp cluster installation configuration file and the Hdp cluster software package set, and the Hdp big data platform which completes the service starting is used as the target Hdp big data platform.
The Hdp cluster installation and startup script includes: hdp cluster installation script, service start script. The Hdp cluster installation and startup script is a script developed by using Shell script language. The Hdp cluster installation script is the script for performing Hdp cluster installation. The service start script is the script for service start of the Hdp big data platform.
Hdp cluster installation profiles, i.e., profiles that do Ambari-server and Ambari-agent installations. Ambari is a Web-based tool supporting the provisioning, management and monitoring of Apache Hadoop clusters. Ambari-server is the server of Ambari. Ambari-agent is Ambari client.
In one embodiment, the step of calling the environment configuration script of the automation deployment script based on the execution of the script main program to obtain the environment configuration file of the target configuration file, and performing server configuration according to the environment configuration file to obtain a server configuration completion result includes:
s31: calling an environment configuration script of the automatic deployment script based on the execution of the script main program;
S32: and acquiring a configuration catalog by executing logic of the environment configuration script, acquiring the environment configuration file of the target configuration file according to the configuration catalog, determining a server to be configured according to the environment configuration file to obtain a server set to be configured, and respectively carrying out system name modification, password-free configuration and system parameter modification on each server in the server set to be configured according to the environment configuration file to obtain a server configuration completion result.
The embodiment realizes that the logic for executing the environment configuration script performs server configuration based on the execution of the script main program, provides a qualified installation environment for the subsequent Hdp big data platform, improves the deployment efficiency and success rate by automatically performing the server configuration, and reduces the difficulty of later maintenance.
And S31, acquiring a script catalog through execution of the script main program, and acquiring an environment configuration script of the automatic deployment script from a preset script storage position according to the script catalog.
For S32, acquiring a configuration catalog through logic of executing the environment configuration script, and acquiring the environment configuration file of the target configuration file from a preset configuration file storage position according to the configuration catalog; the method comprises the steps that logic of the environment configuration script is executed, a master server in the environment configuration file is used as a server to be configured, each slave server in the environment configuration file is used as a server to be configured, and all servers to be configured are used as a server set to be configured; and executing logic of the environment configuration script, taking any one server in the server set to be configured as a target server to be configured, and respectively carrying out system name modification, password-free configuration and system parameter modification on the target server to be configured according to environment configuration data corresponding to the target server to be configured in the environment configuration file to obtain a server configuration completion result.
The configuration catalog includes: profile type, profile name, preset profile storage location.
In one embodiment, the step of performing system name modification, password-free configuration and system parameter modification on each server in the server set to be configured according to the environment configuration file to obtain a server configuration completion result includes:
S321: and respectively carrying out system name modification, password-free configuration and system parameter modification on the servers in the server set to be configured according to each piece of environment configuration data in the environment configuration file by executing logic of the environment configuration script, acquiring a single-server environment configuration result corresponding to each server in the server set to be configured, determining that the server configuration completion result is failure when the single-server environment configuration result is failure, otherwise, determining that the server configuration completion result is success.
According to the embodiment, the system name modification, password-free configuration and system parameter modification are respectively carried out on each server in the server set to be configured according to the environment configuration file, a qualified installation environment is provided by carrying out the subsequent Hdp big data platform, the server configuration is carried out automatically, the deployment efficiency and success rate are improved, and the later maintenance difficulty is reduced.
For S321, by executing the logic of the environment configuration script, taking any one server in the server set to be configured as the server to be configured as the target server, and respectively performing system name modification, password-free configuration and system parameter modification on the server to be configured according to the environment configuration data corresponding to the server to be configured in the environment configuration file, to obtain a single server environment configuration result corresponding to the server to be configured; acquiring a single-server environment configuration result sent by each server in the server set to be configured by executing the logic of the environment configuration script; through logic of executing the environment configuration script, when the single-server environment configuration result is failure, determining that the server configuration completion result is failure; and through logic for executing the environment configuration script, when all the single-server environment configuration results are successful, determining that the server configuration completion result is successful.
For example, the environment configuration data is 30.23.10.135, zhiniao-bigdata-1, master, root, mf3ojrzh, the system name of the server with the machine IP address 30.23.10.135 in the server set to be configured is modified to zhiniao-bigdata-1, the account root and the password Mf3OJRzh are stored in the server with the machine IP address 30.23.10.135 to realize password-free login, and the system parameters of the server with the machine IP address 30.23.10.135 are modified by the system parameters of the environment configuration script, which is not limited in particular herein.
Optionally, the logic configured to execute the environment configuration script determines that the server configuration completion result is failure when the single server environment configuration result is failure, including: and through logic of executing the environment configuration script, when the single-server environment configuration result is failure or the duration of server configuration exceeds the first preset duration, determining that the server configuration completion result is failure. Therefore, the situation that the server configuration completion result cannot be determined when the server configuration duration is too long is avoided, the problem investigation is facilitated for deployment personnel in time, and the efficiency of automatic deployment of the Hdp big data platform is improved.
The first predetermined time period is a specific value.
In one embodiment, the step of installing the base software according to the environment configuration file, the base software package installation configuration file and the set of base software packages to be installed to obtain a base software installation completion result includes:
S41: and based on the execution of the script main program, carrying out system-dependent RPM package installation, hdp platform installation package installation, mysql installation package installation, initialization processing of a Mysql database and JDK installation package installation in sequence according to the environment configuration file, the basic software package installation configuration file and the basic software package set to be installed to obtain a basic software installation completion result.
According to the embodiment, the system-dependent RPM package installation, the Hdp platform installation package installation, the Mysql installation package installation, the initialization processing of the Mysql database and the JDK installation package installation are automatically carried out, and the deployment efficiency and success rate are improved and the later maintenance difficulty is reduced through the automatic basic software installation.
And S41, when the server configuration completion result is successful, based on the execution of the script main program, by executing logic of a basic software package installation script, according to the basic software package installation configuration file and the basic software package set to be installed, sequentially performing initialization processing of system-dependent RPM package installation, hdp platform installation package installation, mysql database installation and JDK installation package installation on the server corresponding to the environment configuration file, and when basic software (namely, a target system-dependent RPM package, a target Hdp platform installation package, a target Mysql installation package and a target JDK installation package) of all servers corresponding to the environment configuration file is successfully installed, determining that the basic software installation completion result is successful, and when any one of the basic software installation of any one of the servers corresponding to the environment configuration file fails, determining that the basic software installation completion result is failed.
In one embodiment, the step of sequentially performing system-dependent RPM package installation, hdp platform installation package installation, mysql installation package installation, initialization processing of a Mysql database, and JDK installation package installation according to the environment configuration file, the base software package installation configuration file, and the base software package set to be installed based on execution of the script main program to obtain the base software installation completion result includes:
s411: when the server configuration completion result is successful, calling a system dependent RPM package installation script of the automatic deployment script based on the execution of the script main program;
s412: acquiring the configuration catalog and the resource catalog by executing the system-dependent RPM package installation script, acquiring a system-dependent RPM package installation configuration file of the target configuration file according to the configuration catalog, acquiring a target system-dependent RPM package according to the resource catalog, performing system-dependent RPM package installation on a main server corresponding to the environment configuration file according to the system-dependent RPM package installation configuration file and the target system-dependent RPM package to obtain a main server system-dependent RPM package installation completion signal, performing system-dependent RPM package installation on all slave servers corresponding to the environment configuration file according to the main server corresponding to the environment configuration file based on the main server system-dependent RPM package installation completion signal, and obtaining a system-dependent RPM package installation result;
s413: when the system depends on the RPM package installation result to be successful, calling an Hdp platform installation package installation script of the automatic deployment script based on the execution of the script main program;
S414: acquiring the configuration catalog and the resource catalog by executing the Hdp platform installation package installation script, acquiring an Hdp platform installation package installation configuration file of the target configuration file according to the configuration catalog, acquiring a target Hdp platform installation package according to the resource catalog, performing Hdp platform installation package installation on the main server corresponding to the environment configuration file according to the Hdp platform installation package installation configuration file and the target Hdp platform installation package to obtain a main server Hdp platform installation package installation completion signal, and performing Hdp platform installation package installation on all the slave servers corresponding to the environment configuration file according to the main server corresponding to the environment configuration file based on the main server Hdp platform installation package completion signal to obtain an Hdp platform installation package installation result;
S415: when the result of the installation of the Hdp platform installation package is successful, calling a Mysql installation package installation script of the automatic deployment script based on the execution of the script main program;
S416: acquiring the configuration catalog and the resource catalog by executing the Mysql installation package installation script, acquiring a Mysql installation package installation configuration file of the target configuration file according to the configuration catalog, acquiring a target Mysql installation package according to the resource catalog, and performing Mysql installation package installation on the main server corresponding to the environment configuration file according to the Mysql installation package installation configuration file and the target Mysql installation package to obtain a Mysql installation package installation result;
S417: when the installation result of the Mysql installation package is successful, calling a Mysql initialization script of the automatic deployment script based on the execution of the script main program;
S418: performing initialization processing of a Mysql database on the main server corresponding to the environment configuration file by executing the Mysql initialization script to obtain a Mysql initialization result;
S419: when the Mysql initialization result is successful, calling a JDK installation package installation script of the automatic deployment script based on the execution of the script main program;
s4110: and acquiring the configuration catalog and the resource catalog by executing the JDK installation package installation script, acquiring a JDK installation package installation configuration file of the target configuration file according to the configuration catalog, acquiring a target JDK installation package according to the resource catalog, and installing the JDK installation package on all servers corresponding to the environment configuration file according to the JDK installation package installation configuration file and the target JDK installation package to obtain the installation completion result of the basic software.
According to the embodiment, the system-dependent RPM package installation, the Hdp platform installation package installation, the Mysql installation package installation, the initialization processing of the Mysql database and the JDK installation package installation are sequentially carried out, and the system-dependent RPM package installation, the Hdp platform installation package installation, the Mysql installation package installation, the initialization processing of the Mysql database and the installation of the JDK installation package are automatically carried out, so that the deployment efficiency and the success rate are improved, and the later maintenance difficulty is reduced.
For S411, when the server configuration is successful, it means that the installation environment of the Hdp big data platform has been successfully configured, and then the installation of the base software is performed to facilitate the improvement of the success rate of the installation of the base software, so that the script directory is obtained through the execution of the script main program, and the system-dependent RPM package installation script of the automated deployment script is obtained from the preset script storage location according to the script directory.
For S412, the configuration catalog and the resource catalog are obtained by executing the system-dependent RPM package installation script; acquiring a system-dependent RPM package installation configuration file of the target configuration file from a preset configuration file storage position according to the configuration catalog by executing the system-dependent RPM package installation script; acquiring a target system-dependent RPM package from a preset resource file storage position according to a resource catalog by executing the system-dependent RPM package installation script; installing the target system-dependent RPM package to a main server corresponding to the environment configuration file according to the system-dependent RPM package installation configuration file by executing the system-dependent RPM package installation script; acquiring a main server system dependent RPM package installation completion signal sent by the main server corresponding to the environment configuration file by executing the system dependent RPM package installation script; after receiving a system-dependent RPM package installation completion signal of a main server, calling a slave server corresponding to the environment configuration file to access the main server corresponding to the environment configuration file through an HTTP request (request message from a client to a server) according to the system-dependent RPM package installation configuration file, and automatically copying a target system-dependent RPM package to a local storage space of the slave server corresponding to the environment configuration file from the main server corresponding to the environment configuration file; calling a slave server corresponding to the environment configuration file to acquire a target system-dependent RPM package from a local storage space by executing the system-dependent RPM package installation script to install the system-dependent RPM package; and acquiring system-dependent RPM package installation completion signals sent by each slave server corresponding to the environment configuration file by executing the system-dependent RPM package installation script, and determining that the system-dependent RPM package installation result is successful when the system-dependent RPM package installation completion signals corresponding to all the slave servers corresponding to the environment configuration file are received.
The resource catalog includes: the method comprises the steps of resource file types, resource file names and preset resource file storage positions.
For example, the system-dependent RPM package installation script is 2_yum_install.sh, a script developed using the Shell scripting language, for installing system-dependent RPM packages at a master server and a slave server, and the examples are not specifically limited.
Optionally, when the duration of system-dependent RPM package installation exceeds a second preset duration, determining that the system-dependent RPM package installation result is failure by executing the system-dependent RPM package installation script. Therefore, the situation that the system-dependent RPM package installation result cannot be determined due to overlong time of system-dependent RPM package installation is avoided, the problem investigation is facilitated for deployment personnel in time, and the efficiency of automatic deployment of an Hdp big data platform is improved.
The second predetermined time period is a specific value.
For S413, when the system-dependent RPM package installation result is successful, it means that the master server and all the slave servers corresponding to the environment configuration file have successfully installed the target system-dependent RPM package, and at this time, a script directory may be obtained through execution of the script master program, and according to the script directory, an Hdp platform installation package installation script of the automation deployment script is obtained from a preset script storage location.
For S414, the configuration catalog and the resource catalog are obtained by executing the Hdp platform installation package installation script; acquiring an Hdp platform installation package installation configuration file of the target configuration file from a preset configuration file storage position according to the configuration catalog by executing the Hdp platform installation package installation script; acquiring a target Hdp platform installation package from a preset resource file storage position according to a resource catalog by executing the Hdp platform installation package installation script; installing the target Hdp platform installation package to a main server corresponding to the environment configuration file according to the Hdp platform installation package installation configuration file by executing the Hdp platform installation package installation script; acquiring a main server Hdp platform installation package installation completion signal sent by the main server corresponding to the environment configuration file by executing the Hdp platform installation package installation script; after receiving an Hdp platform installation package installation completion signal of a main server, calling a slave server corresponding to the environment configuration file to access the main server corresponding to the environment configuration file through an HTTP request according to the Hdp platform installation package installation configuration file, and automatically copying a target Hdp platform installation package from the main server corresponding to the environment configuration file to a local storage space of the slave server corresponding to the environment configuration file; calling a slave server corresponding to the environment configuration file to acquire a target Hdp platform installation package from a local storage space by executing the Hdp platform installation package installation script to install the Hdp platform installation package; and acquiring an Hdp platform installation package installation completion signal sent by each slave server corresponding to the environment configuration file by executing the Hdp platform installation package installation script, and determining that the Hdp platform installation package installation result is successful when the Hdp platform installation package installation completion signals corresponding to all the slave servers corresponding to the environment configuration file are received.
For example, the Hdp platform installation package installation script is 3_hdp_yum_install. Sh, which is a script developed by using Shell script language, and is used for installing Hdp platform installation packages on a master server and a slave server, and is not specifically limited herein by way of example.
Optionally, when the time for installing the Hdp platform installation package exceeds a third preset time, determining that the Hdp platform installation package installation result is failure by executing the Hdp platform installation package installation script. Therefore, the problem investigation is facilitated for deployment personnel in time, and the efficiency of automatic deployment of the high-data-rate Hdp platform is improved.
The third preset time period is a specific value.
For S415, when the result of the Hdp platform installation package installation is successful, it means that the target system-dependent RPM package and the target Hdp platform installation package have been successfully installed by the master server and all the slave servers corresponding to the environment configuration file, and at this time, a script directory may be obtained by execution of the script master program, and according to the script directory, a Mysql installation package installation script of the automation deployment script is obtained from a preset script storage location.
For S416, the configuration catalog and the resource catalog are obtained by executing the Mysql installation package installation script; acquiring Mysql installation package installation configuration files of the target configuration files from a preset configuration file storage position according to the configuration catalog by executing the Mysql installation package installation script; acquiring a target Mysql installation package from a preset resource file storage position according to a resource catalog by executing the Mysql installation package installation script; installing the target Mysql installation package to a main server corresponding to the environment configuration file according to the Mysql installation package installation configuration file by executing the Mysql installation package installation script; acquiring a main server Mysql installation package installation completion signal sent by the main server corresponding to the environment configuration file by executing the Mysql installation package installation script; and executing the Mysql installation package installation script, and determining that the Mysql installation package installation result is successful after receiving a Mysql installation package installation completion signal of the main server.
For example, the Mysql installation package installation script is 4_mysql_install. Sh, which is a script developed by using Shell script language, and is used for installing Mysql installation packages in the host server to obtain Mysql database, which is not specifically limited herein by way of example.
Optionally, by executing the Mysql installation package installation script, when the duration of Mysql installation package installation exceeds a fourth preset duration, determining that the Mysql installation package installation result is failure. Therefore, the situation that the Mysql installation package cannot be determined due to overlong installation time of the Mysql installation package is avoided, the problem investigation is facilitated for deployment personnel in time, and the efficiency of automatic deployment of the Hdp big data platform is improved.
The fourth preset time period is a specific value.
For S417, when the Mysql installation package installation result is successful, it means that the master server and all the slave servers corresponding to the environment configuration file have successfully installed the target system dependent RPM package and the target Hdp platform installation package, and the master server corresponding to the environment configuration file successfully installs the target Mysql installation package, at this time, a script directory may be obtained through execution of the script main program, and according to the script directory, a Mysql initialization script of the automation deployment script is obtained from a preset script storage location.
And for S418, performing Mysql database initialization processing on the main server corresponding to the environment configuration file by executing the Mysql initialization script, acquiring a Mysql database initialization completion signal sent by the main server corresponding to the environment configuration file, and determining that the Mysql initialization result is successful when the Mysql database initialization completion signal is received.
For example, the Mysql initialization script is 5_mysql_init.sh, which is a script developed using the Shell script language, and is used in the main server, and the example is not specifically limited herein.
Optionally, by executing the Mysql initialization script, when the duration of the initialization processing of the Mysql database exceeds a fifth preset duration, determining that the Mysql initialization result is failure. Therefore, the problem investigation by deployment personnel in time is facilitated, and the efficiency of automatic deployment of the Hdp big data platform is improved.
The fifth preset time period is a specific value.
For S419, when the Mysql initialization result is successful, it means that the master server and all the slave servers corresponding to the environment configuration file have successfully installed the target system dependent RPM package and the target Hdp platform installation package, and the master server corresponding to the environment configuration file successfully installs the target Mysql installation package and successfully initializes the target system dependent RPM package.
For S4110, the configuration directory and the resource directory are obtained by executing the JDK installation package installation script; acquiring a JDK installation package installation configuration file of the target configuration file from a preset configuration file storage position according to the configuration directory by executing the JDK installation package installation script; acquiring a target JDK installation package from a preset resource file storage position according to a resource directory by executing the JDK installation package installation script; installing the target JDK installation package to a main server corresponding to the environment configuration file according to the JDK installation package installation configuration file by executing the JDK installation package installation script; acquiring a main server JDK installation package installation completion signal sent by the main server corresponding to the environment configuration file by executing the JDK installation package installation script; after a host server JDK installation package installation completion signal is received, calling a slave server corresponding to the environment configuration file to access the host server corresponding to the environment configuration file through an HTTP request according to the JDK installation package installation configuration file by executing the JDK installation package installation script, and automatically copying a target JDK installation package from the host server corresponding to the environment configuration file to a local storage space of the slave server corresponding to the environment configuration file; the target JDK installation package is obtained from a local storage space by a slave server corresponding to the environment configuration file through executing the JDK installation package installation script to install the JDK installation package; and acquiring JDK installation package installation completion signals sent by each slave server corresponding to the environment configuration file by executing the JDK installation package installation script, and determining that the basic software installation completion result is successful when the JDK installation package installation completion signals corresponding to all the slave servers corresponding to the environment configuration file are received.
For example, the JDK installation package installation script is 6_java_install.sh, which is a script developed by using Shell script language, and is used for installing JDK installation packages in a master server and a slave server, and the example is not limited specifically.
Optionally, by executing the JDK installation package installation script, when the duration of JDK installation package installation exceeds a sixth preset duration, determining that the result of the basic software installation is failure. Therefore, the situation that the installation completion result of the basic software cannot be determined when the JDK installation package is installed for too long is avoided, the problem investigation is facilitated for deployment personnel in time, and the efficiency of automatic deployment of the Hdp big data platform is improved.
The sixth preset time period is a specific value.
In one embodiment, the step of calling an Hdp cluster installation and startup script of the automated deployment script according to the result of completing installation of the base software to obtain an Hdp cluster software package set and an Hdp cluster installation configuration file of the target configuration file, and performing Hdp cluster installation and service startup according to the environment configuration file, the Hdp cluster installation configuration file and the Hdp cluster software package set to obtain the target Hdp big data platform includes:
S51: when the result of the installation of the basic software is successful, calling the Hdp cluster installation and starting script of the automatic deployment script based on the execution of the script main program;
S52: acquiring the configuration catalog and the resource catalog by executing the Hdp cluster installation and starting script, acquiring the Hdp cluster installation configuration file according to the configuration catalog, acquiring the Hdp cluster software package set according to the resource catalog, performing ambari-server and ambari-agent installation on a main server corresponding to the environment configuration file according to the Hdp cluster installation configuration file and the Hdp cluster software package set to obtain an Hdp cluster main server installation completion signal, performing ambari-agent installation on each slave server corresponding to the environment configuration file according to the Hdp cluster installation configuration file and the Hdp cluster software package set to obtain an Hdp cluster slave server installation completion signal, and performing service starting of an Hdp large data platform according to the Hdp cluster slave server installation completion signal to obtain the target Hdp large data platform.
The embodiment realizes that the Hdp cluster is installed firstly and then service is started, and the efficiency and the success rate of deployment are improved and the difficulty of later maintenance is reduced by automating the installation and the service starting of the Hdp cluster.
For S51, when the result of the installation of the base software is successful, it means that the base software has been installed successfully, and at this time, the installation of the Hdp big data platform is performed again to facilitate the improvement of the success rate of the installation of the Hdp big data platform, so that, by executing the script main program, a script directory is obtained, and according to the script directory, the Hdp cluster installation and the startup script of the automated deployment script is obtained from a preset script storage location.
For S52, acquiring the configuration catalog and the resource catalog by executing the Hdp cluster installation and startup script; acquiring the Hdp cluster installation configuration file from a preset configuration file storage position according to the configuration catalog by executing the Hdp cluster installation and starting script; acquiring the set of the Hdp cluster software packages from a preset resource file storage position according to a resource catalog by executing the Hdp cluster installation and starting script; installing ambari-server installation packages and ambari-agent installation packages in the set of Hdp cluster software packages into a main server corresponding to the environment configuration file according to the Hdp cluster installation configuration file by executing the Hdp cluster installation script of the Hdp cluster installation and starting script; acquiring a main server ambari installation completion signal sent by a main server corresponding to the environment configuration file by executing the Hdp cluster installation script of the Hdp cluster installation and starting script; when a service start script of the Hdp cluster installation and start script is executed and a master server ambari installation completion signal is received, calling a slave server corresponding to the environment configuration file to access the master server corresponding to the environment configuration file through an HTTP request according to the Hdp cluster installation configuration file, and automatically copying ambari-agent installation packages from the master server corresponding to the environment configuration file to a local storage space of the slave server corresponding to the environment configuration file; invoking a corresponding slave server of the environment configuration file to acquire ambari-agent installation packages from a local storage space by executing the Hdp cluster installation and starting script to install the ambari-agent; acquiring a slave server ambari installation completion signal sent by each slave server corresponding to the environment configuration file by executing the Hdp cluster installation and starting script; and executing the Hdp cluster installation and starting script, when receiving the installation completion signals of all the slave servers ambari corresponding to the slave servers corresponding to the environment configuration file, starting the service of the Hdp big data platform according to the installation completion signals of the slave server ambari, and taking the Hdp big data platform with all the service started as the target Hdp big data platform.
For example, the Hdp cluster installation and startup script is 7_hdp_install.sh, which is a script developed by using Shell script language, and is used for performing Hdp cluster installation and service startup in the master server and the slave server, and is not specifically limited herein.
In one embodiment, before the step of obtaining the main program operation instruction, the method further includes:
s11: acquiring a configuration file storage request, wherein the configuration file storage request carries a configuration file to be stored and a configuration file storage path;
S12: responding to the configuration file storage request, and storing the configuration file to be stored in a preset configuration file storage position corresponding to the configuration file storage path;
S13: acquiring a resource file storage request, wherein the resource file storage request carries a resource file set to be stored and a resource file storage path;
S14: and responding to the resource file storage request, and storing the resource file set to be stored in a preset resource file storage position corresponding to the resource file storage path.
According to the embodiment, the configuration file is stored in the preset configuration file storage position after the main program operation instruction is acquired, and the resource file set is stored in the preset resource file storage position, so that the file does not need to be transmitted temporarily, the automation of the deployment of the Hdp big data platform is improved, and the time length required by the deployment of the Hdp big data platform is shortened.
For S11, a profile storage request input by the user may be obtained, or a profile storage request input by the third party application system may be obtained.
The configuration file storage request is a request for storing a configuration file in a preset configuration file storage location.
The configuration files to be stored are the configuration files to be stored for the deployment of the Hdp big data platform.
The profile storage path, i.e. the access path of the preset profile storage location.
And for S12, responding to the configuration file storage request, and storing the configuration file to be stored in a preset configuration file storage position corresponding to the configuration file storage path so as to be deployed and called by an Hdp big data platform.
For S13, the resource file storage request input by the user may be obtained, or the resource file storage request input by the third party application system may be obtained.
The resource file storage request is a request for storing a resource file set in a preset resource file storage position.
The resource file set to be stored is the resource file deployed by the Hdp big data platform needing to be stored. The resource files include installation packages, such as a system dependent RPM package, an Hdp platform installation package, a Mysql installation package, a JDK installation package.
The resource file storage path is an access path of a preset resource file storage position.
And for S14, responding to the resource file storage request, and storing the resource file set to be stored in a preset resource file storage position corresponding to the resource file storage path so as to be deployed and called by an Hdp big data platform.
Referring to fig. 2, the application further provides an automatic deployment device of the Hdp big data platform, which comprises:
an instruction acquisition module 100, configured to acquire a main program operation instruction;
The script acquisition module 200 is configured to respond to the main program operation instruction, acquire a script main program of an automatic deployment script, and execute the script main program, where the script main program is used to instruct deployment operation of an Hdp big data platform;
The server configuration completion result determining module 300 is configured to call an environment configuration script of the automation deployment script based on execution of the script main program, obtain an environment configuration file of a target configuration file, and perform server configuration according to the environment configuration file to obtain a server configuration completion result;
The base software installation completion result determining module 400 is configured to invoke a base software package installation script of the automated deployment script according to the server configuration completion result based on execution of the script main program, obtain a base software package set to be installed and a base software package installation configuration file of the target configuration file, and install base software according to the environment configuration file, the base software package installation configuration file and the base software package set to be installed to obtain a base software installation completion result;
The target Hdp big data platform determining module 500 is configured to invoke an Hdp cluster installation and startup script of the automated deployment script according to the result of the installation of the base software, obtain an Hdp cluster software package set and an Hdp cluster installation configuration file of the target configuration file, and perform Hdp cluster installation and service startup according to the environment configuration file, the Hdp cluster installation configuration file and the Hdp cluster software package set, thereby obtaining the target Hdp big data platform.
According to the embodiment, firstly, a script main program of an automatic deployment script is obtained and executed in response to a main program operation instruction, the script main program is used for indicating deployment operation of an Hdp big data platform, secondly, based on execution of the script main program, an environment configuration script of the automatic deployment script is called, an environment configuration file of a target configuration file is obtained, server configuration is carried out according to the environment configuration file, a server configuration completion result is obtained, then, based on execution of the script main program, a foundation software package installation script of the automatic deployment script is called according to the server configuration completion result, foundation software package installation configuration files of a foundation software package set to be installed and a foundation software package installation configuration file of the target configuration file are obtained, foundation software installation of the foundation software package installation configuration file is carried out according to the environment configuration file, the foundation software package installation configuration file and the foundation software package set to be installed, basic software installation completion result is obtained, finally, based on execution of the script main program, the environment configuration file installation completion result is called, the Hdp cluster installation and the target configuration file are carried out according to the Hdp cluster installation configuration file, the Hdp cluster installation and the Hdp cluster configuration file, the service deployment efficiency of the set is improved, the problem of the automatic deployment script is solved, the existing deployment platform is solved, the efficiency of the automatic deployment platform is easy to be improved, the problem of the deployment is solved, the existing deployment platform is easy to be deployed according to the existing in terms of the service deployment, the deployment platform is easy, the deployment of the system is easy, and the deployment of the service deployment platform is easy to be deployed, and the problem of the service deployment platform is easy to be deployed by the following a large-stage is due to the deployment stage to the service deployment stage has a high-stage to the service deployment efficiency, and has been achieved by the service deployment efficiency is achieved by the service deployment stage has a high service stage has a service deployment efficiency; because the deployment path and method are consistent, the difficulty of maintenance and upgrading in the later stage is reduced.
Referring to fig. 3, in an embodiment of the present application, there is further provided a computer device, which may be a server, and an internal structure thereof may be as shown in fig. 3. The computer device includes a processor, a memory, a network interface, and a database connected by a system bus. Wherein the computer is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The database of the computer equipment is used for storing data such as an automatic deployment method of an Hdp big data platform. The network interface of the computer device is used for communicating with an external terminal through a network connection. The computer program, when executed by the processor, implements an automated deployment method for Hdp big data platforms. The automatic deployment method of the Hdp big data platform comprises the following steps: acquiring a main program operation instruction; responding to the main program operation instruction, acquiring and executing a script main program of an automatic deployment script, wherein the script main program is used for indicating the deployment operation of an Hdp big data platform; calling an environment configuration script of the automatic deployment script based on the execution of the script main program, acquiring an environment configuration file of a target configuration file, and performing server configuration according to the environment configuration file to obtain a server configuration completion result; based on the execution of the script main program, calling a basic software package installation script of the automatic deployment script according to the server configuration completion result, obtaining a basic software package set to be installed and a basic software package installation configuration file of the target configuration file, and performing basic software installation according to the environment configuration file, the basic software package installation configuration file and the basic software package set to be installed to obtain a basic software installation completion result; and calling an Hdp cluster installation and starting script of the automatic deployment script according to the result of the installation of the basic software based on the execution of the script main program, obtaining an Hdp cluster software package set and an Hdp cluster installation configuration file of the target configuration file, and carrying out the installation and service starting of the Hdp cluster according to the environment configuration file, the Hdp cluster installation configuration file and the Hdp cluster software package set to obtain the target Hdp big data platform.
According to the embodiment, firstly, a script main program of an automatic deployment script is obtained and executed in response to a main program operation instruction, the script main program is used for indicating deployment operation of an Hdp big data platform, secondly, based on execution of the script main program, an environment configuration script of the automatic deployment script is called, an environment configuration file of a target configuration file is obtained, server configuration is carried out according to the environment configuration file, a server configuration completion result is obtained, then, based on execution of the script main program, a foundation software package installation script of the automatic deployment script is called according to the server configuration completion result, foundation software package installation configuration files of a foundation software package set to be installed and a foundation software package installation configuration file of the target configuration file are obtained, foundation software installation of the foundation software package installation configuration file is carried out according to the environment configuration file, the foundation software package installation configuration file and the foundation software package set to be installed, basic software installation completion result is obtained, finally, based on execution of the script main program, the environment configuration file installation completion result is called, the Hdp cluster installation and the target configuration file are carried out according to the Hdp cluster installation configuration file, the Hdp cluster installation and the Hdp cluster configuration file, the service deployment efficiency of the set is improved, the problem of the automatic deployment script is solved, the existing deployment platform is solved, the efficiency of the automatic deployment platform is easy to be improved, the problem of the deployment is solved, the existing deployment platform is easy to be deployed according to the existing in terms of the service deployment, the deployment platform is easy, the deployment of the system is easy, and the deployment of the service deployment platform is easy to be deployed, and the problem of the service deployment platform is easy to be deployed by the following a large-stage is due to the deployment stage to the service deployment stage has a high-stage to the service deployment efficiency, and has been achieved by the service deployment efficiency is achieved by the service deployment stage has a high service stage has a service deployment efficiency; because the deployment path and method are consistent, the difficulty of maintenance and upgrading in the later stage is reduced.
An embodiment of the present application further provides a computer readable storage medium having a computer program stored thereon, the computer program when executed by a processor implementing an automated deployment method for an Hdp big data platform, including the steps of: acquiring a main program operation instruction; responding to the main program operation instruction, acquiring and executing a script main program of an automatic deployment script, wherein the script main program is used for indicating the deployment operation of an Hdp big data platform; calling an environment configuration script of the automatic deployment script based on the execution of the script main program, acquiring an environment configuration file of a target configuration file, and performing server configuration according to the environment configuration file to obtain a server configuration completion result; based on the execution of the script main program, calling a basic software package installation script of the automatic deployment script according to the server configuration completion result, obtaining a basic software package set to be installed and a basic software package installation configuration file of the target configuration file, and performing basic software installation according to the environment configuration file, the basic software package installation configuration file and the basic software package set to be installed to obtain a basic software installation completion result; and calling an Hdp cluster installation and starting script of the automatic deployment script according to the result of the installation of the basic software based on the execution of the script main program, obtaining an Hdp cluster software package set and an Hdp cluster installation configuration file of the target configuration file, and carrying out the installation and service starting of the Hdp cluster according to the environment configuration file, the Hdp cluster installation configuration file and the Hdp cluster software package set to obtain the target Hdp big data platform.
According to the automatic deployment method of the executed Hdp big data platform, firstly, a script main program of an automatic deployment script is obtained and executed in response to a main program operation instruction, the script main program is used for indicating deployment operation of the Hdp big data platform, secondly, based on execution of the script main program, an environment configuration script of the automatic deployment script is called, an environment configuration file of a target configuration file is obtained, server configuration is carried out according to the environment configuration file, a server configuration completion result is obtained, then, based on execution of the script main program, a basic software package installation script of the automatic deployment script is called according to the server configuration completion result, a basic software package set to be installed and a basic software package installation configuration file of the target configuration file are obtained, basic software installation is carried out according to the environment configuration file, the basic software package installation configuration file and the basic software package set to be installed, so that basic software installation completion results are obtained, finally, based on execution of the script main program, hdp cluster installation and starting script of the automatic deployment script are called according to the basic software installation completion results, hdp cluster installation and the set of the target configuration file, hdp cluster installation configuration file and the Hdp cluster configuration file are obtained, the efficiency of the automatic deployment script is easy to be realized, the problem of the existing deployment platform is solved, the system is solved, the efficiency of the deployment platform is easy to be realized by implementing the configuration according to the configuration of the existing configuration platform, the configuration data of the Hdp big data, the Hdp deployment platform is easy, and the configuration stage is realized, the configuration is realized, and the efficiency of the Hdp deployment platform is easy is realized, the success rate of deployment is improved; because the deployment paths and the deployment methods are consistent, the difficulty of maintenance and upgrading in the later stage is reduced.
Those skilled in the art will appreciate that implementing all or part of the above described methods may be accomplished by way of a computer program stored on a non-transitory computer readable storage medium, which when executed, may comprise the steps of the embodiments of the methods described above. Any reference to memory, storage, database, or other medium provided by the present application and used in embodiments may include non-volatile and/or volatile memory. The nonvolatile memory can include Read Only Memory (ROM), programmable ROM (PROM), electrically Programmable ROM (EPROM), electrically Erasable Programmable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms such as Static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual speed data rate SDRAM (SSRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (SYNCHLINK) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), among others.
It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, apparatus, article, or method that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, apparatus, article, or method. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, apparatus, article, or method that comprises the element.
The foregoing description is only of the preferred embodiments of the present application and is not intended to limit the scope of the application, and all equivalent structures or equivalent processes using the descriptions and drawings of the present application or directly or indirectly applied to other related technical fields are included in the scope of the application.

Claims (6)

1. An automated deployment method of an Hdp big data platform, the method comprising:
acquiring a main program operation instruction;
responding to the main program operation instruction, acquiring and executing a script main program of an automatic deployment script, wherein the script main program is used for indicating the deployment operation of an Hdp big data platform;
Calling an environment configuration script of the automatic deployment script based on the execution of the script main program, acquiring an environment configuration file of a target configuration file, and performing server configuration according to the environment configuration file to obtain a server configuration completion result;
Based on the execution of the script main program, calling a basic software package installation script of the automatic deployment script according to the server configuration completion result, obtaining a basic software package set to be installed and a basic software package installation configuration file of the target configuration file, and performing basic software installation according to the environment configuration file, the basic software package installation configuration file and the basic software package set to be installed to obtain a basic software installation completion result;
Invoking an Hdp cluster installation and starting script of the automatic deployment script according to the result of the installation of the basic software based on the execution of the script main program, obtaining an Hdp cluster software package set and an Hdp cluster installation configuration file of the target configuration file, and performing Hdp cluster installation and service starting according to the environment configuration file, the Hdp cluster installation configuration file and the Hdp cluster software package set to obtain a target Hdp big data platform;
the step of calling the environment configuration script of the automatic deployment script based on the execution of the script main program to acquire the environment configuration file of the target configuration file, and performing server configuration according to the environment configuration file to acquire a server configuration completion result comprises the following steps:
Calling an environment configuration script of the automatic deployment script based on the execution of the script main program;
Acquiring a configuration catalog by executing logic of the environment configuration script, acquiring the environment configuration file of the target configuration file according to the configuration catalog, determining a server to be configured according to the environment configuration file to obtain a server set to be configured, and respectively carrying out system name modification, password-free configuration and system parameter modification on each server in the server set to be configured according to the environment configuration file to obtain a server configuration completion result;
the step of respectively carrying out system name modification, password-free configuration and system parameter modification on each server in the server set to be configured according to the environment configuration file to obtain the server configuration completion result comprises the following steps:
Performing system name modification, password-free configuration and system parameter modification on servers in the server set to be configured according to each piece of environment configuration data in the environment configuration file respectively by executing logic of the environment configuration script, acquiring single-server environment configuration results corresponding to each server in the server set to be configured, determining that the server configuration completion result is failure when the single-server environment configuration result is failure, otherwise, determining that the server configuration completion result is success;
The step of installing the base software according to the environment configuration file, the base software package installation configuration file and the base software package set to be installed to obtain a base software installation completion result comprises the following steps:
Based on the execution of the script main program, carrying out system-dependent RPM package installation, hdp platform installation package installation, mysql installation package installation, initialization processing of a Mysql database and JDK installation package installation in sequence according to the environment configuration file, the basic software package installation configuration file and the basic software package set to be installed to obtain a basic software installation completion result;
The step of calling an Hdp cluster installation and starting script of the automatic deployment script according to the result of the completion of the installation of the basic software to obtain an Hdp cluster software package set and an Hdp cluster installation configuration file of the target configuration file, and performing the installation and service starting of the Hdp cluster according to the environment configuration file, the Hdp cluster installation configuration file and the Hdp cluster software package set to obtain the target Hdp big data platform comprises the following steps:
when the result of the installation of the basic software is successful, calling the Hdp cluster installation and starting script of the automatic deployment script based on the execution of the script main program;
Acquiring the configuration catalog and the resource catalog by executing the Hdp cluster installation and starting script, acquiring the Hdp cluster installation configuration file according to the configuration catalog, acquiring the Hdp cluster software package set according to the resource catalog, performing ambari-server and ambari-agent installation on a main server corresponding to the environment configuration file according to the Hdp cluster installation configuration file and the Hdp cluster software package set to obtain an Hdp cluster main server installation completion signal, performing ambari-agent installation on each slave server corresponding to the environment configuration file according to the Hdp cluster installation configuration file and the Hdp cluster software package set to obtain an Hdp cluster slave server installation completion signal, and performing service starting of an Hdp large data platform according to the Hdp cluster slave server installation completion signal to obtain the target Hdp large data platform.
2. The automated deployment method of an Hdp big data platform according to claim 1, wherein the steps of sequentially performing an initialization process of a system-dependent RPM package installation, an Hdp platform installation package installation, a Mysql database, and a JDK installation package installation according to the environment configuration file, the base software package installation configuration file, and the set of base software packages to be installed based on execution of the script main program, and obtaining a result of completion of the base software installation include:
when the server configuration completion result is successful, calling a system dependent RPM package installation script of the automatic deployment script based on the execution of the script main program;
Acquiring the configuration catalog and the resource catalog by executing the system-dependent RPM package installation script, acquiring a system-dependent RPM package installation configuration file of the target configuration file according to the configuration catalog, acquiring a target system-dependent RPM package according to the resource catalog, performing system-dependent RPM package installation on a main server corresponding to the environment configuration file according to the system-dependent RPM package installation configuration file and the target system-dependent RPM package to obtain a main server system-dependent RPM package installation completion signal, performing system-dependent RPM package installation on all slave servers corresponding to the environment configuration file according to the main server corresponding to the environment configuration file based on the main server system-dependent RPM package installation completion signal, and obtaining a system-dependent RPM package installation result;
When the system depends on the RPM package installation result to be successful, calling an Hdp platform installation package installation script of the automatic deployment script based on the execution of the script main program;
Acquiring the configuration catalog and the resource catalog by executing the Hdp platform installation package installation script, acquiring an Hdp platform installation package installation configuration file of the target configuration file according to the configuration catalog, acquiring a target Hdp platform installation package according to the resource catalog, performing Hdp platform installation package installation on the main server corresponding to the environment configuration file according to the Hdp platform installation package installation configuration file and the target Hdp platform installation package to obtain a main server Hdp platform installation package installation completion signal, and performing Hdp platform installation package installation on all the slave servers corresponding to the environment configuration file according to the main server corresponding to the environment configuration file based on the main server Hdp platform installation package completion signal to obtain an Hdp platform installation package installation result;
when the result of the installation of the Hdp platform installation package is successful, calling a Mysql installation package installation script of the automatic deployment script based on the execution of the script main program;
acquiring the configuration catalog and the resource catalog by executing the Mysql installation package installation script, acquiring a Mysql installation package installation configuration file of the target configuration file according to the configuration catalog, acquiring a target Mysql installation package according to the resource catalog, and performing Mysql installation package installation on the main server corresponding to the environment configuration file according to the Mysql installation package installation configuration file and the target Mysql installation package to obtain a Mysql installation package installation result;
when the installation result of the Mysql installation package is successful, calling a Mysql initialization script of the automatic deployment script based on the execution of the script main program;
performing initialization processing of a Mysql database on the main server corresponding to the environment configuration file by executing the Mysql initialization script to obtain a Mysql initialization result;
when the Mysql initialization result is successful, calling a JDK installation package installation script of the automatic deployment script based on the execution of the script main program;
And acquiring the configuration catalog and the resource catalog by executing the JDK installation package installation script, acquiring a JDK installation package installation configuration file of the target configuration file according to the configuration catalog, acquiring a target JDK installation package according to the resource catalog, and installing the JDK installation package on all servers corresponding to the environment configuration file according to the JDK installation package installation configuration file and the target JDK installation package to obtain the installation completion result of the basic software.
3. An automated deployment method for an Hdp big data platform according to claim 1, further comprising, prior to the step of obtaining the main program running instruction:
acquiring a configuration file storage request, wherein the configuration file storage request carries a configuration file to be stored and a configuration file storage path;
responding to the configuration file storage request, and storing the configuration file to be stored in a preset configuration file storage position corresponding to the configuration file storage path;
Acquiring a resource file storage request, wherein the resource file storage request carries a resource file set to be stored and a resource file storage path;
And responding to the resource file storage request, and storing the resource file set to be stored in a preset resource file storage position corresponding to the resource file storage path.
4. An automated deployment apparatus for an Hdp big data platform, for performing the method of any of claims 1-3, the apparatus comprising:
the instruction acquisition module is used for acquiring a main program operation instruction;
The script acquisition module is used for responding to the main program operation instruction, acquiring and executing a script main program of the automatic deployment script, wherein the script main program is used for indicating the deployment operation of the Hdp big data platform;
The server configuration completion result determining module is used for calling an environment configuration script of the automatic deployment script based on the execution of the script main program, obtaining an environment configuration file of a target configuration file, and carrying out server configuration according to the environment configuration file to obtain a server configuration completion result;
The base software installation completion result determining module is used for calling a base software package installation script of the automatic deployment script according to the execution of the script main program and the server configuration completion result, obtaining a base software package set to be installed and a base software package installation configuration file of the target configuration file, and performing base software installation according to the environment configuration file, the base software package installation configuration file and the base software package set to be installed to obtain a base software installation completion result;
The target Hdp big data platform determining module is used for calling an Hdp cluster installation and starting script of the automatic deployment script according to the execution of the script main program and the result of the completion of the installation of the basic software to obtain an Hdp cluster software package set and an Hdp cluster installation configuration file of the target configuration file, and performing the installation and service starting of the Hdp cluster according to the environment configuration file, the Hdp cluster installation configuration file and the Hdp cluster software package set to obtain the target Hdp big data platform.
5. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that the processor implements the steps of the method of any of claims 1 to 3 when the computer program is executed.
6. A computer readable storage medium, on which a computer program is stored, characterized in that the computer program, when being executed by a processor, implements the steps of the method of any of claims 1 to 3.
CN202110485518.9A 2021-04-30 2021-04-30 Automatic deployment method, device, equipment and medium for Hdp big data platform Active CN113127016B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202110485518.9A CN113127016B (en) 2021-04-30 2021-04-30 Automatic deployment method, device, equipment and medium for Hdp big data platform

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202110485518.9A CN113127016B (en) 2021-04-30 2021-04-30 Automatic deployment method, device, equipment and medium for Hdp big data platform

Publications (2)

Publication Number Publication Date
CN113127016A CN113127016A (en) 2021-07-16
CN113127016B true CN113127016B (en) 2024-06-21

Family

ID=76781093

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202110485518.9A Active CN113127016B (en) 2021-04-30 2021-04-30 Automatic deployment method, device, equipment and medium for Hdp big data platform

Country Status (1)

Country Link
CN (1) CN113127016B (en)

Families Citing this family (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113568755B (en) * 2021-08-04 2023-11-17 上海易景信息科技有限公司 Distributed compiling system and distributed compiling method
CN115562758B (en) * 2022-11-10 2023-05-26 北京卓翼智能科技有限公司 Cloud simulation platform deployment method, system, storage medium and electronic equipment
CN116909584B (en) * 2023-05-06 2024-05-24 广东国地规划科技股份有限公司 Deployment method, device, equipment and storage medium of space-time big data engine

Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109710281A (en) * 2018-12-28 2019-05-03 中科曙光国际信息产业有限公司 The installation method and device of big data platform

Family Cites Families (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107463582B (en) * 2016-06-03 2021-11-12 中兴通讯股份有限公司 Distributed Hadoop cluster deployment method and device
CN109687987A (en) * 2017-10-19 2019-04-26 北京金山云网络技术有限公司 A kind of cloud platform dispositions method, device, electronic equipment and readable storage medium storing program for executing
CN110825420B (en) * 2019-11-26 2023-09-22 中国银行股份有限公司 Method, device, equipment and storage medium for updating configuration parameters of distributed cluster

Patent Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109710281A (en) * 2018-12-28 2019-05-03 中科曙光国际信息产业有限公司 The installation method and device of big data platform

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
基于Ambari的Hadoop集群部署实验的设计与实现;唐磊;;信息记录材料;20171101(第11期);全文 *

Also Published As

Publication number Publication date
CN113127016A (en) 2021-07-16

Similar Documents

Publication Publication Date Title
CN113127016B (en) Automatic deployment method, device, equipment and medium for Hdp big data platform
US11599348B2 (en) Container image building using shared resources
CN107729066B (en) Automatic construction and release method and device based on SVN (singular value decomposition) resource library
US8627311B2 (en) Systems, methods, and apparatus to deploy software
CN110908859B (en) Application service processing method, device, terminal and storage medium
US8151277B2 (en) Method and system for dynamic remote injection of in-process agents into virtual machine based applications
US8978015B2 (en) Self validating applications
CN110096424B (en) Test processing method and device, electronic equipment and storage medium
US10305731B2 (en) System and method for provisioning cloud services across heterogeneous environments using partitioned provisioning instructions stored on a configuration management server
CN106708573B (en) System and method for Hadoop cluster automatic installation
CN110618819B (en) Method, device and computer equipment for automatic batch deployment of WINPE (WINPE) system
CN112130871A (en) Method and device for remotely deploying middleware, computer equipment and storage medium
CN114721809A (en) Application deployment method and device of kubernets cluster
US11539612B2 (en) Testing virtualized network functions
CN113467893B (en) Automatic testing method and device for virtual machine, storage medium and processor
US11425203B2 (en) Commissioning a virtualized network function
CN115525545A (en) Docker-based automatic testing method, system, equipment and medium
CN111176671A (en) Method for automatically and continuously updating system, mobile terminal and storage medium
CN114356346A (en) Application program deployment method, device, storage medium and electronic equipment
CN112328267A (en) Multi-server installation method and device, terminal and storage medium
CN112256289A (en) Automatic deployment method, device and equipment
CN115328717A (en) Kernel testing method and system supporting various domestic platforms
CN114237649A (en) Code detection method and device, electronic equipment and program product
CN111176661A (en) Method and device for deploying python operating environment in batches based on distributed crawler
CN113312079A (en) Hot patch upgrading method and device, electronic equipment and storage medium

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant