CN115167878A - Cluster installation and deployment method, device and medium - Google Patents

Cluster installation and deployment method, device and medium Download PDF

Info

Publication number
CN115167878A
CN115167878A CN202210761543.XA CN202210761543A CN115167878A CN 115167878 A CN115167878 A CN 115167878A CN 202210761543 A CN202210761543 A CN 202210761543A CN 115167878 A CN115167878 A CN 115167878A
Authority
CN
China
Prior art keywords
installation
program
node
package
configuration
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202210761543.XA
Other languages
Chinese (zh)
Inventor
吴丹丹
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Suzhou Inspur Intelligent Technology Co Ltd
Original Assignee
Suzhou Inspur Intelligent Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Suzhou Inspur Intelligent Technology Co Ltd filed Critical Suzhou Inspur Intelligent Technology Co Ltd
Priority to CN202210761543.XA priority Critical patent/CN115167878A/en
Publication of CN115167878A publication Critical patent/CN115167878A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F8/00—Arrangements for software engineering
    • G06F8/60—Software deployment
    • G06F8/61—Installation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/21—Design, administration or maintenance of databases
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
    • H04L41/08—Configuration management of networks or network elements
    • H04L41/0803—Configuration setting
    • H04L41/0813—Configuration setting characterised by the conditions triggering a change of settings
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
    • H04L41/08—Configuration management of networks or network elements
    • H04L41/0803—Configuration setting
    • H04L41/084—Configuration by using pre-existing information, e.g. using templates or copying from other elements
    • H04L41/0846—Configuration by using pre-existing information, e.g. using templates or copying from other elements based on copy from other elements
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
    • H04L41/08—Configuration management of networks or network elements
    • H04L41/0876—Aspects of the degree of configuration automation
    • H04L41/0886—Fully automatic configuration
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
    • H04L41/08—Configuration management of networks or network elements
    • H04L41/0889—Techniques to speed-up the configuration process
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/10—Protocols in which an application is distributed across nodes in the network
    • H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/2866—Architectures; Arrangements
    • H04L67/30—Profiles
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/34—Network arrangements or protocols for supporting network services or applications involving the movement of software or configuration parameters 

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Theoretical Computer Science (AREA)
  • Databases & Information Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • Software Systems (AREA)
  • Computing Systems (AREA)
  • Automation & Control Theory (AREA)
  • Information Transfer Between Computers (AREA)

Abstract

The invention discloses a method, a device and a medium for installing and deploying a cluster, which are suitable for the technical field of big data storage. Storing the script file and the program installation package to a current node, wherein the current node is one; copying the program installation package to each node through a file copying instruction; establishing remote connection according to the IP of each node; installing and configuring each node through an installation program; and starting the nodes in batches through a processing program to realize installation and deployment. The method can realize unified installation and deployment configuration among the multi-state nodes only by a single node through an installation program and a processing program, automatically upload a program installation package through copying a file instruction, realize connection among all nodes through remote connection, automatically install and modify the configuration of the nodes, and further automatically start the nodes. The method avoids independent configuration and operation of each node, greatly simplifies the installation steps, saves the installation configuration time, and simultaneously solves the problem of confusion caused by the omission of the existing nodes.

Description

Cluster installation and deployment method, device and medium
Technical Field
The invention relates to the technical field of big data storage, in particular to a method, a device and a medium for installing and deploying a cluster.
Background
In recent years, with the development of large data business requirements and the increase of data magnitude, the query speed of the traditional database cannot meet the business requirements, and more enterprises choose to use a clickhouse cluster database for distributed storage of a large amount of data. The ZooKeeper is used by the ClickHouse to ensure the consistency of data when the ClickHouse is used for Distributed storage, and is mainly used for the synchronization of copy table data (ReplicatedMergeTree engine) and the operation of a Distributed table (Distributed).
In the process of using the ClickHouse and the ZooKeeper, the problems of deployment and installation are involved, many existing large data cluster services are huge, the number of database cluster servers is large, and in the traditional ClickHouse cluster installation and deployment mode, installation files need to be uploaded on each node, and relevant configuration is carried out in the/etc/metrika. Xml of each node. The existing deployment and installation mode is that nodes are manually deployed and installed one by one, in the process, the process is complicated and complicated, the cost of time is increased indirectly, and even a node confusion phenomenon caused by omission of a certain node and the like can be caused.
Therefore, it is a urgent need to solve by those skilled in the art to find an efficient method for installing and deploying a cluster.
Disclosure of Invention
The invention aims to provide a cluster installation and deployment method, a cluster installation and deployment device and a cluster installation and deployment medium, which avoid independent configuration and operation of each node, greatly simplify installation steps, save installation configuration time and simultaneously solve the problem of missing and chaos of the conventional nodes.
In order to solve the above technical problem, the present invention provides a cluster installation and deployment method, including:
acquiring a plurality of nodes under a cluster system and corresponding IP, script files and program installation packages;
storing the script file and the program installation package to a current node, wherein the number of the current node is one;
copying the program installation package to each node through a file copying instruction;
establishing remote connection according to the IP of each node;
installing and configuring each node through an installation program;
and starting the nodes in batches through a processing program to realize installation and deployment.
Preferably, the installing and configuring each node by an installation program includes:
acquiring the number of nodes, and setting cycle times according to the number;
installing a program rpm package of a program installation package at a current node, and taking the installation of the current node as a primary installation program rpm package circulation;
judging whether the cycle number of the rpm packet of the current installation program exceeds the cycle number;
if the cycle number of the current installation program rpm package does not exceed the cycle number, acquiring the next current node, and returning to the step of installing the program rpm package of the program installation package at the current node;
if the circulation times of the current installation program rpm package exceed the circulation times, returning to the current node, installing the ClickHouse program package of the program installation package at the current node, modifying the database configuration parameters of the current node, and taking the installation of the current node as the circulation of the ClickHouse program package for one-time installation;
judging whether the cycle times of the current ClickHouse program package is more than the cycle times;
if the cycle times of the current installation clickwouse program package do not exceed the cycle times, acquiring a next current node, and returning to the steps of installing the clickwouse program package of the program installation package at the current node and modifying the database configuration parameters of the current node;
if the cycle times of the current installation ClickHouse program package exceed the cycle times, returning to the current node, modifying the corresponding cluster configuration file at the current node, and taking the configuration of the current node as the cycle times of one-time configuration;
judging whether the current configuration cycle number exceeds the cycle number;
if the current configuration cycle number does not exceed the cycle number, acquiring the next current node, and returning to the step of modifying the corresponding cluster configuration file at the current node;
if the current configuration cycle number exceeds the cycle number, returning to the current node, installing the ZooKeeper and the corresponding configuration file at the current node, and taking the configuration of the current node as the cycle number of the primary configuration ZooKeeper;
judging whether the cycle number of the currently configured ZooKeeper exceeds the cycle number;
if the cycle number of the currently configured ZooKeeper does not exceed the cycle number, acquiring the next current node, and returning to the step of installing the ZooKeeper and the corresponding configuration file at the current node;
and if the cycle number of the currently configured ZooKeeper exceeds the cycle number, determining that the installation and the configuration of each node are finished.
Preferably, after the remote connection is established according to the IP of each node, before the installation and configuration of each node by the installation program, the method further includes:
writing the IP corresponding to each node into the host of each node;
and executing basic environment configuration for closing each node.
Preferably, the basic environment configuration includes at least firewall, SELinux configuration package.
Preferably, before installing and configuring each node through the installation program, the method further includes:
acquiring server accounts and passwords corresponding to the nodes;
and inputting the server account and the password into the installation and deployment system of the current node.
Preferably, the batch starting of each node through the processing program includes:
the ClickHouse program and the ZooKeeper program are started by the batch processing script file.
Preferably, the method further comprises the following steps:
and when the nodes are started in batch, outputting prompt information.
In order to solve the above technical problem, the present invention further provides a device for installing and deploying a cluster, including:
the acquisition module is used for acquiring a plurality of nodes under the cluster system and corresponding IP, script files and program installation packages;
the storage module is used for storing the script file and the program installation package to a current node, wherein the number of the current node is one;
the copying module is used for copying the program installation package to each node through a file copying instruction;
the establishing module is used for establishing remote connection according to the IP of each node;
the configuration module is used for installing and configuring each node through an installation program;
and the starting module is used for starting all the nodes in batches through the processing program so as to realize installation and deployment.
In order to solve the above technical problem, the present invention further provides a device for installing and deploying a cluster, including:
a memory for storing a computer program;
a processor for implementing the steps of the installation and deployment method of the cluster as described above when executing the computer program.
In order to solve the above technical problem, the present invention further provides a computer-readable storage medium, on which a computer program is stored, and the computer program, when executed by a processor, implements the steps of the installation and deployment method of the cluster as described above.
The invention provides a cluster installation and deployment method, which comprises the steps of obtaining a plurality of nodes under a cluster system and corresponding IP, script files and program installation packages; storing the script file and the program installation package to a current node, wherein the current node is one; copying the program installation package to each node through a file copying instruction; establishing remote connection according to the IP of each node; installing and configuring each node through an installation program; and starting the nodes in batches through a processing program to realize installation and deployment. The method can realize uniform installation, deployment and configuration among multi-state nodes only by a single node through an installation program and a processing program, automatically uploads a program installation package through a file copying instruction, realizes connection among all nodes through remote connection, automatically installs and modifies the configuration of the nodes, and further automatically starts. The method avoids independent configuration and operation of each node, greatly simplifies the installation steps, saves the installation configuration time, and simultaneously solves the problem of confusion caused by the omission of the existing nodes.
In addition, the invention also provides a device and a medium for installing and deploying the cluster, which have the same beneficial effects as the installation and deployment method of the cluster.
Drawings
In order to more clearly illustrate the embodiments of the present invention, the drawings required for the embodiments will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and that other drawings may be obtained by those skilled in the art without inventive effort.
Fig. 1 is a flowchart of a cluster installation and deployment method according to an embodiment of the present invention;
fig. 2 is a structural diagram of an installation and deployment apparatus of a cluster according to an embodiment of the present invention;
fig. 3 is a structural diagram of another installation and deployment apparatus for a cluster according to an embodiment of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments obtained by a person of ordinary skill in the art based on the embodiments of the present invention without any creative work belong to the protection scope of the present invention.
The core of the invention is to provide a method, a device and a medium for installing and deploying a cluster, which avoid the independent configuration and operation of each node, greatly simplify the installation steps, save the installation configuration time and simultaneously solve the phenomenon of missing and chaos of the existing nodes.
In order that those skilled in the art will better understand the disclosure, the invention will be described in further detail with reference to the accompanying drawings and specific embodiments.
It should be noted that the installation and deployment method of the cluster provided by the present invention is applicable to other distributed cluster databases besides the distributed cluster database clickwause and ZooKeeper, and is not limited specifically. The clickwouse is a column-wise storage Database (DBMS) of a Massively Parallel Processing architecture (MMP) for Online Analytical Processing (OLAP), and can generate an analysis data report in real time by using Structured Query Language (SQL) Query. The clickwouse can be used for user behavior analysis, integration of flow and batch, linear expansion and reliability guarantee and can support the board + reproduction originally. clickhouse does not go to hadoop ecology and adopts Local attached storage as storage.
ZooKeeper is a distributed, open-source distributed application coordination service, an important component of Hadoop and Hbase. It is a software that provides a consistent service for distributed applications, and the functions provided include: configuration maintenance, domain name service, distributed synchronization, group service, etc. The ZooKeeper aims to package complex and error-prone key services and provide a simple and easy-to-use interface and a system with high performance and stable functions for users.
Fig. 1 is a flowchart of a cluster installation and deployment method provided in an embodiment of the present invention, and as shown in fig. 1, the method includes:
s11: acquiring a plurality of nodes under a cluster system and corresponding IP, script files and program installation packages;
s12: storing the script file and the program installation package to a current node, wherein the current node is one;
s13: copying the program installation package to each node through a file copying instruction;
s14: establishing remote connection according to the IP of each node;
s15: installing and configuring each node through an installation program;
s16: and starting the nodes in batches through a processing program to realize installation and deployment.
Specifically, a plurality of nodes under the cluster system and corresponding Internet Protocol addresses (IPs), script files and program installation packages are acquired, and for the acquisition mode, the nodes and corresponding IPs are different from the acquisition mode of the script files and program installation packages, and the nodes and corresponding IPs are input by a user. The acquisition mode of the script file and the program installation package is to newly build a folder under a root directory in a node server to be installed, upload a compressed file containing the script file and the program installation package to the directory, and decompress the compressed file to obtain the script file and the program installation package.
The folder in the root directory does not have a special requirement, and may be any new folder, or may be any folder in the root directory, for example, a cfnet folder, as long as the script file and the program installation package are stored in the current node server to be installed, and this embodiment also corresponds to the above step S12 of storing the script file and the program installation package to the current node. It should be noted that the current node may be any one of the servers of the node to be installed, but the number of the current nodes is only one, and the installation and deployment method provided by the present invention is directed to a single node, so that the unified installation, deployment and configuration functions of a plurality of nodes can be realized. The script files and the program installation packages are corresponding to the click House cluster and the ZooKeeper distributed automatic installation deployment configuration scheme.
The program installation is copied to each node through a copy file instruction in step S13, where the copy file instruction can implement remote copy, from local to remote, or from remote to local. The copy file instruction may be a secure copy (scp) command or other commands. The Scp command is used for commands for remote copying files under Linux, and its analogous commands have cp, but cp cannot be across servers only for copying locally, and Scp transfer is encrypted. The speed may be slightly affected. When you change your server hard disk to a read only system you can be helped with scp to move files out. In addition, scp is also very resource-free and does not increase much system load, at which point rsync is far less than it. Although rsync is faster than scp, in the case of many small files, rsync causes the hard disk I/O to be very high, and scp basically does not affect the normal use of the system. The command format of the Scp command is: scp [ parameter ] [ original path ] [ target path ]; the function is to copy files and directories between Linux servers.
In step S14, the remote connection is established according to the IP of each node, and it should be noted that, according to the copy file instruction in step S13, the remote connection mode is selected. For example, in the scp command selected in step 13, in order to facilitate authentication, a Secure Shell protocol (SSH) with the same authentication method is used to connect and transmit data. The scp command is the most convenient and useful command in SSH, for example, to transfer files between two servers, i.e., two nodes, and is completely solved with only one scp command. The # scp server name:/home/ftp/pub/file1./; thus, the file/home/ftp/pub/file 1 on another server; the file/tmp/file 2 on the local machine can be sent to the/boot directory of another machine by using # scp/tmp/file2server name:/boot, and the whole transmission process is still encrypted by SSH.
SSH connection principle: the SSH service is a daemon process (demon), the background of the system monitors the connection of the client, the process name of the SSH service is sshd, and the SSH service is responsible for monitoring the request (IP 22 port) of the client in real time and comprises information such as public keys and the like. The SSH server consists of two parts: opennsh (providing SSH service) opennssl (providing encrypted program); the client of SSH can use XSHELL, securecht, mobaxterm, etc. tools to connect the working mechanism of SSH.
When the server is started, a secret key (768 bit public key) is generated, the local SSH client sends a connection request to the SSH server, the server checks data and an IP address sent by the connection point client, the secret key (768 bits) is sent to the client after the legitimacy is confirmed, at the moment, the local private key (256 bits) and the public key (768 bits) of the server are combined into a key pair (1024 bits) by the client, the key pair is sent back to the server, and connection is established through key-pair data transmission. SSH is a network protocol for encrypted login between computers. The SSH protocol can effectively prevent the problem of information leakage in the remote management process. The SSH client is suitable for a variety of platforms, such as Linux, AIX, solaris, digital UNIX, irix, among others, and can run SSH. SSH guarantees security because it encrypts all transmitted data using asymmetric encryption (RSA).
The SSH connection implementation process comprises the following steps:
(1) After receiving the login request of the user, the remote host sends the public key of the remote host to the user.
(2) The user uses the public key to encrypt the login password and send it back.
(3) The remote host decrypts the login password by using the private key of the remote host, and if the password is correct, the remote host agrees to login of the user.
It should be noted that, the SSH has two connection modes, one is a normal connection mode password access, and the other is a no-password connection, and for the no-password connection mode, for example, two servers, one is a server side, and the other is a client side, a directory is added at the server side: ssh: ssh, mkdir; ssh directory rights: chmod777.Ssh; and creating a public key and a private key at the client again: ssh-keygen; and finally copying the public key of the client to the server, and operating the command: ssh-copy-id root@10.0.0.20, after password input, ssh password-free access can be realized.
In the step S15, each node is installed and configured through an installation program; for the clickwause and ZooKeeper installation on each node, the installation needs to be performed according to the installation package and the configuration requirement, and the installation can be for program package installation, configuration modification of a database, modification of a cluster configuration file and the like. The installation configuration mode of each node is not specifically limited, and may be to simultaneously perform installation configuration on each node, or perform installation configuration of the next node after the installation configuration of one node is performed, which is not specifically limited in the present invention.
In step S16, each node is started in batch by the processing program to implement installation and deployment, and after the installation and deployment of each node are completed, the nodes can be started in batch, and it should be noted that the processing program is a certain script file and can implement batch processing, and the present invention does not limit the script file for implementing batch processing, and may be xcall. Sh/cfnet/zookeeper/bin/zkserver sh start; sh service clickwouse-server start.
The invention provides a method for installing and deploying a cluster, which comprises the steps of obtaining a plurality of nodes under a cluster system and corresponding IP, script files and program installation packages; storing the script file and the program installation package to a current node, wherein the current node is one; copying the program installation package to each node through a file copying instruction; establishing remote connection according to the IP of each node; installing and configuring each node through an installation program; and starting the nodes in batches through a processing program to realize installation and deployment. The method can realize uniform installation and deployment configuration among multi-state nodes only by a single node through an installation program and a processing program, automatically uploads a program installation package through copying a file instruction, realizes connection among all nodes through remote connection, automatically installs and modifies the configuration of the nodes, and further automatically starts. The method avoids independent configuration and operation of each node, greatly simplifies the installation steps, saves the installation configuration time, and simultaneously solves the problem of confusion caused by the omission of the existing nodes.
On the basis of the above embodiment, the installing and configuring each node in step S15 through an installation program, adopting a node-by-node installation and deployment manner, and seamlessly joining and processing a next node after installation and deployment of each node are completed, includes:
acquiring the number of nodes, and setting cycle times according to the number;
installing a program rpm package of a program installation package at a current node, and taking the installation of the current node as a primary installation program rpm package circulation;
judging whether the cycle number of the rpm packet of the current installation program exceeds the cycle number;
if the cycle number of the current installation program rpm package does not exceed the cycle number, acquiring the next current node, and returning to the step of installing the program rpm package of the program installation package at the current node;
if the circulation times of the current installation program rpm package exceed the circulation times, returning to the current node, installing the ClickHouse program package of the program installation package at the current node, modifying the database configuration parameters of the current node, and taking the installation of the current node as the circulation of the ClickHouse program package for one-time installation;
judging whether the cycle number of the current ClickHouse program package is larger than the cycle number or not;
if the cycle times of the current installation clickwouse program package do not exceed the cycle times, acquiring a next current node, and returning to the steps of installing the clickwouse program package of the program installation package at the current node and modifying the database configuration parameters of the current node;
if the cycle times of the current installation ClickHouse program package exceed the cycle times, returning to the current node, modifying the corresponding cluster configuration file at the current node, and taking the configuration of the current node as the cycle times of one-time configuration;
judging whether the current configuration cycle number exceeds the cycle number;
if the current configuration cycle number does not exceed the cycle number, acquiring the next current node, and returning to the step of modifying the corresponding cluster configuration file at the current node;
if the current configuration cycle number exceeds the cycle number, returning to the current node, installing the ZooKeeper and the corresponding configuration file at the current node, and taking the configuration of the current node as the cycle number of the primary configuration ZooKeeper;
judging whether the cycle number of the currently configured ZooKeeper exceeds the cycle number;
if the cycle number of the currently configured ZooKeeper does not exceed the cycle number, acquiring a next current node, and returning to the step of installing the ZooKeeper and a corresponding configuration file at the current node;
and if the cycle number of the currently configured ZooKeeper exceeds the cycle number, determining that the installation and the configuration of each node are finished.
Specifically, the installation and deployment of the ClickHouse and ZooKeeper cluster distributed systems relate to the installation of an rpm package, the ClickHouse program package, and the installation of a database configuration parameter, a ZooKeeper and a corresponding configuration file for modifying a current node, where the aforementioned configuration installation parameter may be implemented by first performing all installation on one node and then starting the installation of the next node, or may be implemented by first performing the installation of each node on each part, and the present invention is not particularly limited.
It should be noted that the order of partial installation of the installation and deployment files is not specifically limited, and may be that the clickwouse package is installed first, the database configuration parameters of the current node are modified, and then other installation files are installed.
The red hat Package Manager (RPM) is a management program for installing required software on a host by means of database management, the RPM Package is commonly used for installation upgrade uninstallation and query of software packages by red hat development, and linux implements and manages installation, viewing and deletion of software by RPM.
And installing the ClickHouse program package of the program installation package and modifying the database configuration parameters of the current node, and completing the installation configuration of the part by installing the ClickHouse program package on each node and modifying the database configuration parameters of the current node.
And modifying the corresponding cluster configuration file, and modifying the cluster configuration file metrika. Install ZooKeeper and corresponding configuration files, install ZooKeeper and modify configuration zo.cfg and myid files for each node.
The invention provides a method for installing and deploying a ClickHouse and a ZooKeeper cluster distributed system for each node in a circulating mode, which relates to the installation of an rpm (revolution speed) packet, a ClickHouse program packet, the modification of database configuration parameters of the current node, the installation of a ZooKeeper and a corresponding configuration file, ensures that the installation configuration of each node is processed in a seamless connection mode, realizes automatic installation, is responsible for installation configuration, is responsible for confirming the number of nodes, simply modifies the number of host cycles in an xcall program script (the number of cycles needs to be consistent with the number of input IP (Internet protocol) nodes), can realize the installation configuration deployment of a plurality of nodes, does not make any limit on the number of nodes, and greatly improves the working efficiency.
On the basis of the above embodiment, after the remote connection is established according to the IP of each node in step S14, before the installation and configuration of each node by the installation program in step S15, the method further includes:
writing the IP corresponding to each node into the host of each node;
and executing basic environment configuration for closing each node.
Specifically, after SSH connection is established, nodes IP are written into hosts of each node, and then basic environment configuration is closed, so that each node is not intercepted by installation of each node when being installed, and installation configuration efficiency is improved.
As a preferred example, the basic environment configuration at least includes a firewall and a SELinux configuration package. On the one hand, to avoid being prevented from adding defensive drivers, and on the other hand, may false alarm.
Security-Enhanced Linux (SELinux) is an implementation of the national Security administration (NAS) for mandatory access control, and under the restriction of this access control system, a process can only access files that are needed in his tasks. SELinux incorporates multiple levels of security or an optional multiple class strategy in a class enforcement server and employs a role-based control concept.
Many TYPEs (TYPE) are defined in SELinux, each process, file, device, etc. must identify that the TYPE process to which it belongs can only read files of the same TYPE, and if there is no related TYPE and SELinux does not allow reading, the files cannot be read. SELinux restricts the ability of a process to read files, limits the ability of the process to read devices, network connections, communication ports, and cross-process communication, and provides finer reading control.
The SELinux configuration package is characterized as follows:
(1) The access control is thorough, and the access to all files, directories and ports is set based on the strategy and can be set by an administrator;
(2) Only minimal rights are granted to the user. The user is divided into some roles and even a root user cannot perform related management if it does not have the role of sysadmin _ r. The corresponding domain for different role executions can be modified.
(3) When SELinux is started, all files and objects have security context.
Before each node is installed and deployed, xcall.sh is responsible for confirming the number of the nodes, and simply modifying the host cycle number in the xcall program script (the cycle number is required to be consistent with the number of input IP nodes), so that the installation configuration deployment of a plurality of nodes can be realized, no limitation is made on the number of the nodes, and the working efficiency is greatly improved.
On the basis of the above embodiment, before installing and configuring each node by the installation program in step S15, the method further includes:
acquiring a server account and a password corresponding to each node;
and inputting the server account and the password into the installation and deployment system of the current node.
Due to the security attribute of the SSH connection, a server account and a password need to be acquired and input to the installation and deployment system of the current node, and an SSH encryption form is adopted. And the automatic SSH is connected with each node server, inputs a server account number and a password and starts to execute automatic installation and deployment configuration.
The server account and the password corresponding to each node are obtained; and inputting the server account and the password into the installation and deployment system of the current node. The safety of connection transmission and installation deployment among the nodes is improved.
On the basis of the above embodiment, the batch starting of the nodes by the handler in step S16 further includes:
the ClickHouse program and the ZooKeeper program are started by the batch processing script file.
It should be noted that, the batch processing script uses xcall.sh script files, and is remotely copied to each node, and then sets the authority of each node, so as to perform batch startup. Sh script file realizes the functions: confirming the number of the nodes, realizing SSH connection of each node, and starting the ClickHouse and ZooKeeper programs of each node in batch. Sh script file throughout the whole method embodiment, its simple implementation process is as follows:
(1) Selecting a certain node server to be installed, newly building a folder cfnet under a root directory, uploading a compressed file containing a script and a program installation package to the directory, and decompressing;
(2) The decompressed file folder is opened, the xcall script is found, and the user needs to modify the script cycle times according to the node data so as to realize the installation of multiple nodes;
(3) According to the known node IP, the following operations are performed:
./xcall.sh/cfnet/ck_install_shell.19.62/install.sh IP1 IP2 IP3…IP*；
(4) The automatic SSH is connected with each node server, a server account and a password are input, and automatic installation deployment configuration is started to be executed;
(5) And after the installation configuration is finished, starting the click House and ZooKeeper programs of the multiple nodes through the xcall.
./xcall.sh/cfnet/zookeeper/bin/zkServer.sh start
./xcall.sh service clickhouse-server start
The ClickHouse program and the ZooKeeper program are started through the batch processing script file. And the xcall is responsible for confirming the number of nodes, simply modifying the host cycle number (the cycle number is required to be consistent with the number of input IP nodes) in the xcall program script, realizing the installation, configuration and deployment of a plurality of nodes, not limiting the number of the nodes at all, and greatly improving the working efficiency.
On the basis of the above embodiment, the method further includes:
and when the nodes are started in batch, outputting prompt information.
Specifically, when batch starting is performed, prompt information is output for reminding a worker that the current installation and configuration work is finished, and starting of the next stage is started, so that the subsequent starting processing of the worker is facilitated.
When the nodes are started in batch, the prompt information is output, so that the staff is reminded that the current installation and configuration work is finished.
On the basis that the embodiments corresponding to the installation and deployment of the cluster are described in detail, the invention further discloses a clock calibration device corresponding to the device, and fig. 2 is a structural diagram of an installation and deployment device of the cluster provided by the embodiment of the invention. As shown in fig. 2, the installation and deployment apparatus of a cluster includes:
the acquiring module 11 is configured to acquire multiple nodes in the cluster system, and corresponding IP, script files, and program installation packages;
the storage module 12 is configured to store the script file and the program installation package to a current node, where the current node is one;
a copy module 13, configured to copy the program installation package to each node through a copy file instruction;
an establishing module 14, configured to establish a remote connection according to the IP of each node;
a configuration module 15, configured to install and configure each node through an installation program;
and the starting module 16 is used for starting the nodes in batches through the processing program to realize installation and deployment.
Since the embodiment of the apparatus portion corresponds to the above-mentioned embodiment, the embodiment of the apparatus portion is described with reference to the embodiment of the method portion, and is not described again here.
The invention provides a device for installing and deploying a cluster, which comprises a plurality of nodes, corresponding IP (Internet protocol), script files and program installation packages, wherein the nodes, the corresponding IP, the script files and the program installation packages are obtained from a cluster system; storing the script file and the program installation package to a current node, wherein the current node is one; copying the program installation package to each node through a file copying instruction; establishing remote connection according to the IP of each node; installing and configuring each node through an installation program; and starting the nodes in batches through the processing program to realize installation and deployment. The device realizes that only a single node is needed through an installation program and a processing program, unified installation and deployment configuration among multi-state nodes can be realized, a program installation package is automatically uploaded through copying a file instruction, connection among all nodes is realized through remote connection, and the nodes are automatically installed and modified to be configured, so that automatic starting is carried out. The method avoids independent configuration and operation of each node, greatly simplifies the installation steps, saves the installation configuration time, and simultaneously solves the problem of confusion caused by the omission of the existing nodes.
Fig. 3 is a structural diagram of another cluster installation and deployment apparatus provided in an embodiment of the present invention, and as shown in fig. 3, the apparatus includes:
a memory 21 for storing a computer program;
a processor 22 for implementing the steps of the method of installation and deployment of the seed cluster when executing the computer program.
The installation and deployment device of the cluster provided in this embodiment may include, but is not limited to, a smart phone, a tablet computer, a notebook computer, or a desktop computer.
The processor 22 may include one or more processing cores, such as a 4-core processor, an 8-core processor, and so on. The Processor 22 may be implemented in hardware using at least one of a Digital Signal Processor (DSP), a Field-Programmable Gate Array (FPGA), and a Programmable Logic Array (PLA). The processor 22 may also include a main processor and a coprocessor, the main processor is a processor for Processing data in an awake state, and is also called a Central Processing Unit (CPU); a coprocessor is a low power processor for processing data in a standby state. In some embodiments, the processor 22 may be integrated with a Graphics Processing Unit (GPU) that is responsible for rendering and drawing the content that the display screen needs to display. In some embodiments, processor 22 may also include an Artificial Intelligence (AI) processor for processing computational operations related to machine learning.
Memory 21 may include one or more computer-readable storage media, which may be non-transitory. Memory 21 may also include high speed random access memory, as well as non-volatile memory, such as one or more magnetic disk storage devices, flash memory storage devices. In this embodiment, the memory 21 is at least used for storing the following computer program 201, wherein after being loaded and executed by the processor 22, the computer program can implement the relevant steps of the installation and deployment method of the cluster disclosed in any of the foregoing embodiments. In addition, the resources stored in the memory 21 may also include an operating system 202, data 203, and the like, and the storage manner may be a transient storage manner or a permanent storage manner. Operating system 202 may include, among other things, windows, unix, linux, etc. Data 203 may include, but is not limited to, data involved in the installation and deployment methods of the seed cluster, and the like.
In some embodiments, the installation and deployment apparatus of the seed cluster may further include a display screen 23, an input/output interface 24, a communication interface 25, a power supply 26, and a communication bus 27.
Those skilled in the art will appreciate that the configuration shown in fig. 3 does not constitute a limitation of the means for installation and deployment of the seed clusters and may include more or fewer components than those shown.
The processor 22 implements the installation and deployment method of the seed cluster provided by any of the above embodiments by calling instructions stored in the memory 21.
The invention provides a cluster installation and deployment device, which comprises a plurality of nodes, corresponding IP, script files and program installation packages, wherein the nodes are obtained from a cluster system; storing the script file and the program installation package to a current node, wherein the number of the current node is one; copying the program installation package to each node through a file copying instruction; establishing remote connection according to the IP of each node; installing and configuring each node through an installation program; and starting the nodes in batches through the processing program to realize installation and deployment. The device realizes that only a single node is needed through an installation program and a processing program, unified installation and deployment configuration among multi-state nodes can be realized, a program installation package is automatically uploaded through copying a file instruction, connection among all nodes is realized through remote connection, and the nodes are automatically installed and modified to be configured, so that automatic starting is carried out. The method avoids independent configuration and operation of each node, greatly simplifies the installation steps, saves the installation configuration time, and simultaneously solves the problem of confusion caused by the omission of the existing nodes.
Further, the present invention also provides a computer readable storage medium, on which a computer program is stored, which when executed by the processor 22, implements the steps of the installation and deployment method of the cluster as described above.
It is to be understood that if the method in the above embodiments is implemented in the form of software functional units and sold or used as a stand-alone product, it can be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and performs all or part of the steps of the methods according to the embodiments of the present invention, or all or part of the technical solution. And the aforementioned storage medium includes: various media capable of storing program codes, such as a usb disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk.
For the introduction of a computer-readable storage medium provided by the present invention, please refer to the above method embodiment, which is not described herein again, and has the same beneficial effects as the above cluster installation and deployment method.
The method, device and medium for installing and deploying various clusters provided by the present invention are described in detail above. The embodiments are described in a progressive manner in the specification, each embodiment focuses on differences from other embodiments, and the same and similar parts among the embodiments are referred to each other. The device disclosed by the embodiment corresponds to the method disclosed by the embodiment, so that the description is simple, and the relevant points can be referred to the method part for description. It should be noted that, for those skilled in the art, without departing from the principle of the present invention, it is possible to make various improvements and modifications to the present invention, and those improvements and modifications also fall within the scope of the claims of the present invention.
It is further noted that, in the present specification, relational terms such as first and second, and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising a … …" does not exclude the presence of another identical element in a process, method, article, or apparatus that comprises the element.

Claims (10)

1. A cluster installation and deployment method, comprising:
acquiring a plurality of nodes under a cluster system and corresponding IP, script files and program installation packages;
storing the script file and the program installation package to a current node, wherein the current node is one;
copying the program installation package to each node through a file copying instruction;
establishing remote connection according to the IP of each node;
installing and configuring each node through an installation program;
and starting the nodes in batches through a processing program to realize installation and deployment.
2. The cluster installation and deployment method of claim 1, wherein the installing and configuring each of the nodes by an installation program comprises:
acquiring the number of the nodes, and setting cycle times according to the number;
installing a program rpm package of the program installation package at the current node, and taking the installation of the current node as a cycle of installing the program rpm package once;
judging whether the cycle number of the current program rpm package is more than the cycle number;
if the cycle number of the currently installed program rpm package does not exceed the cycle number, acquiring a next current node, and returning to the step of installing the program rpm package of the program installation package at the current node;
if the circulation times of the program rpm package installation currently exceed the circulation times, returning to the current node, installing a ClickHouse program package of the program installation package at the current node, modifying the database configuration parameters of the current node, and taking the installation of the current node as one-time installation of the ClickHouse program package circulation;
judging whether the circulation times of the current installation of the ClickHouse program package exceed the circulation times;
if the cycle number of the current ClickHouse program package is not more than the cycle number, acquiring the next current node, and returning to the step of installing the ClickHouse program package of the program installation package at the current node and modifying the database configuration parameters of the current node;
if the cycle number of the current ClickHouse program package exceeds the cycle number, returning to the current node, modifying the corresponding cluster configuration file at the current node, and taking the configuration of the current node as the cycle number of one-time configuration;
judging whether the current configuration cycle number exceeds the cycle number;
if the current configuration cycle number does not exceed the cycle number, acquiring the next current node, and returning to the step of modifying the corresponding cluster configuration file at the current node;
if the current configuration cycle number exceeds the cycle number, returning to the current node, installing a ZooKeeper and a corresponding configuration file at the current node, and configuring the ZooKeeper cycle number by taking the configuration of the current node as one time;
judging whether the cycle number of the currently configured ZooKeeper exceeds the cycle number;
if the cycle number of the currently configured ZooKeeper does not exceed the cycle number, acquiring a next current node, and returning to the step of installing the ZooKeeper and a corresponding configuration file at the current node;
and if the cycle number of the currently configured ZooKeeper exceeds the cycle number, determining that the installation and the configuration of each node are finished.
3. The cluster installation and deployment method of claim 1, wherein after said establishing a remote connection according to the IP of each of said nodes, before said installing and configuring each of said nodes by an installation program, further comprising:
writing the IP corresponding to each node into the host of each node;
and executing basic environment configuration for closing each node.
4. The method according to claim 3, wherein the base environment configuration comprises at least firewall and SELinux configuration package.
5. The cluster installation and deployment method of claim 1, further comprising, before said installing and configuring each of said nodes with an installation program:
acquiring a server account and a password corresponding to each node;
and inputting the server account and the password into the installation and deployment system of the current node.
6. The cluster installation and deployment method of claim 2, wherein the batch startup of each node through a handler comprises:
and starting the ClickHouse program and the ZooKeeper program through the batch processing script file.
7. The cluster installation and deployment method of claim 6, further comprising:
and when the nodes are started in batch, outputting prompt information.
8. An installation and deployment apparatus for a cluster, comprising:
the acquisition module is used for acquiring a plurality of nodes under the cluster system, and corresponding IP, script files and program installation packages;
the storage module is used for storing the script file and the program installation package to a current node, wherein the current node is one;
the copying module is used for copying the program installation package to each node through a file copying instruction;
the establishing module is used for establishing remote connection according to the IP of each node;
the configuration module is used for installing and configuring each node through an installation program;
and the starting module is used for starting all the nodes in batches through a processing program so as to realize installation and deployment.
9. An installation and deployment apparatus for a cluster, comprising:
a memory for storing a computer program;
a processor for implementing the steps of the installation and deployment method of a cluster according to any one of claims 1 to 7 when executing said computer program.
10. A computer-readable storage medium, characterized in that a computer program is stored thereon, which computer program, when being executed by a processor, carries out the steps of the installation and deployment method of a cluster according to any one of claims 1 to 7.
CN202210761543.XA 2022-06-30 2022-06-30 Cluster installation and deployment method, device and medium Pending CN115167878A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202210761543.XA CN115167878A (en) 2022-06-30 2022-06-30 Cluster installation and deployment method, device and medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202210761543.XA CN115167878A (en) 2022-06-30 2022-06-30 Cluster installation and deployment method, device and medium

Publications (1)

Publication Number Publication Date
CN115167878A true CN115167878A (en) 2022-10-11

Family

ID=83488480

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202210761543.XA Pending CN115167878A (en) 2022-06-30 2022-06-30 Cluster installation and deployment method, device and medium

Country Status (1)

Country Link
CN (1) CN115167878A (en)

Similar Documents

Publication Publication Date Title
US11770381B2 (en) Managing security groups for data instances
CN109478149B (en) Access services in a hybrid cloud computing system
JP6140177B2 (en) Techniques for applying and sharing remote policies on mobile devices
US8254579B1 (en) Cryptographic key distribution using a trusted computing platform
CN109479062B (en) Usage tracking in hybrid cloud computing systems
EP2771803B1 (en) File fetch from a remote client device
EP3356978B1 (en) Applying rights management policies to protected files
CN113626133B (en) Virtual machine control method, device, equipment and computer readable storage medium
WO2015176461A1 (en) File access processing method, file access method, and device for distributed file system
CN114579250B (en) Method, device and storage medium for constructing virtual cluster
US12067121B2 (en) Trusted boot method and apparatus, electronic device, and readable storage medium
US11768692B2 (en) Systems and methods for automated application launching
US11636068B2 (en) Distributed file locking for a network file share
CN115167878A (en) Cluster installation and deployment method, device and medium
US11849037B1 (en) Cross-region replication of secrets
CN112287017A (en) OpenSSH-based Impala cluster visual management method
Goasguen et al. Kubernetes cookbook: building cloud native applications
Vetter et al. IBM Power Systems HMC Implementation and Usage Guide
US11593498B2 (en) Distribution of user specific data elements in a replication environment
US20240012665A1 (en) Optimized creation of identity information for provisioned virtual machines
US20230195493A1 (en) Virtual device enrollment and management
Marchioni WildFly Administration Guide: The ultimate and most up-to-date guide to manage WildFly appication server
Kefeng et al. Design and construction of the virtual cloud platform for the laboratory
Stanek Web Server Administration: The Personal Trainer for IIS 7.0 and IIS 7.5
CN118540315A (en) Communication method based on cloud computing technology and related equipment

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination