WO2016127756A1

WO2016127756A1 - Flexible deployment method for cluster and management system

Info

Publication number: WO2016127756A1
Application number: PCT/CN2016/070999
Authority: WO
Inventors: 唐平
Original assignee: 北京京东尚科信息技术有限公司
Priority date: 2015-02-15
Filing date: 2016-01-15
Publication date: 2016-08-18
Also published as: CN104679717A; CN104679717B

Abstract

A flexible deployment method for a cluster (20) and a management system. The method comprises: defining basic information of the cluster (20); selecting, from a resource pool (50) comprising machines intended to be used for generating the cluster (20), a machine intended to be used for generating the cluster (20); selecting a version of software to be used by the machine, and changing configuration information of the machine; selecting a service process to be deployed on the machine; and performing environment accuracy check on the machine according to detection script comprised in the machine, and if the environment accuracy check is successful, acquiring, from a cloud storage (30), a mirror file for configuring the machine, and enabling the to-be-deployed service process on the machine.

Description

Method and management system for cluster elastic deployment

Technical field

The present invention relates to the field of computer technologies, and in particular, to a method and a management system for cluster elastic deployment in a distributed system.

Background technique

In response to massive data storage, more and more companies are adopting open source databases such as HBase. However, as a distributed system, cluster deployment and operation and maintenance of HBase are very troublesome. For example, if the existing cluster is expanded or the machine with hardware and software failures is removed, it depends on the manual operation of the operation and maintenance personnel. These operations are not only repetitive, error-prone, but also make it difficult to manage them manually by increasing the number of clusters and the size of the cluster itself.

To simplify cluster management operations such as HBase, there are also some management tools in the industry, such as Cloudera Manager and Ambari. When deploying a cluster, users only need to select the version of the package that HBase depends on, configure the role of each machine through the tool interface, and adjust the configuration of HBase, Zookeeper, and Hadoop, and the specific cluster deployment operations are managed by these. The tool is complete.

Even if the management tools described above implement cluster management operations, the following drawbacks exist: management tools have many restrictions due to commercial authorization. For example, Cloudera Manager restricts cluster size to less than 100 machine nodes; it is not convenient for multi-cluster management, and management tools are targeted at Installation and management of a cluster, it is difficult to increase or decrease the flexibility of multiple clusters, etc.; lack of audit trails, do not make necessary records for the operation of machines in the production environment, can not help users to trace the causes and responsible persons in case of problems .

Summary of the invention

In view of this, the present invention provides a method and a management system for cluster elastic deployment. By introducing a resource pool and putting all the machines that are to be used for cluster deployment into the resource pool, the maintenance and management of existing resource pools and clusters can be simplified.

To achieve the above object, according to an aspect of the present invention, a method of cluster elastic deployment is provided.

The method for cluster elastic deployment of the present invention includes the steps of: defining basic information of a cluster; selecting, from a resource pool consisting of machines to be used for generating a cluster, a machine to be used to generate the cluster; Determining a version of the software of the machine and modifying configuration information of the machine; selecting a service process to be deployed by the machine; and performing an environmental correctness check on the machine according to a detection script included in the machine, wherein The environmental correctness check passes, the image file configuring the machine is obtained from the cloud storage, and the service process to be deployed is enabled on the machine.

Optionally, the basic information for defining the cluster includes: defining a cluster name, a cluster machine type, a use validity period, and a cluster size of the cluster.

Optionally, the performing an environmental correctness check on the machine includes: checking one or more of an operating system version, a JDK version, a host name, a firewall, and a time synchronization configuration of the machine.

Optionally, the method of cluster resilient deployment of the present invention further comprises the step of adding a machine to the cluster, the step comprising: selecting an idle machine from the resource pool and specifying a service process to be deployed thereon; Performing an environmental correctness check on the machine according to the detection script included in the machine, wherein if the environment correctness check passes, obtaining an image file for configuring the machine from the cloud storage, The configuration of the machine is updated and the service process to be deployed is enabled.

Optionally, the method of cluster elastic deployment of the present invention further comprises the step of dynamically deleting the machine to the cluster, the step comprising: selecting a machine that needs to be stopped and stopping the deployment a service process thereon; and detecting a working state of the cluster according to a detection script included in the machine, wherein if the cluster is working properly, resetting configuration information of the machine to an initial status.

Optionally, the method for cluster elastic deployment of the present invention further includes the step of dynamically modifying the configuration of the cluster, the step comprising: backing up an original configuration item of a machine for which configuration item modification is to be performed; a configuration item to be changed of the machine; and reviewing the legality of the modified configuration item according to the detection script included in the machine, wherein the modification is performed if the review of the legality of the configuration item is passed Subsequent configuration items are pushed to the machine.

According to another aspect of the present invention, there is provided a cluster resilient deployment management system comprising a central management server; a resource pool consisting of machines intended to be used to generate a cluster; and by the central management server, according to the present invention The cluster deployed by the cluster elastic deployment method.

Optionally, the machine in the resource pool that is to be used to generate the cluster can cooperate with the central management server to complete related operations of receiving a command and sending a response message.

Optionally, the machine in the resource pool that is to be used to generate the cluster includes a detection script.

Optionally, the central management server is further configured to acquire a cluster to which the machine belongs, a machine working state, and a machine type.

Optionally, the central management server records an operation log of the cluster and the resource pool.

According to the technical solution of the present invention, a cluster of elastic computing frameworks that can be dynamically managed is provided by introducing a resource pool in a cluster deployment and putting all the machines to be used for the cluster deployment into the resource pool.

DRAWINGS

The drawings are intended to provide a better understanding of the invention and are not intended to limit the invention. among them:

1 is a schematic diagram of a management system for describing a cluster elastic deployment of the present invention;

2A-2D are schematic diagrams for describing a method of cluster elastic deployment of the present invention.

detailed description

The exemplary embodiments of the present invention are described with reference to the accompanying drawings, and are in the For convenience of description, the following specific description is made by taking HBase as an example, but those skilled in the art should recognize that the method and management system of the present invention are not limited to HBase, but various changes can be made to the embodiments described herein. And modifications may be made without departing from the scope and spirit of the invention. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.

1 is a schematic diagram of a management system for describing a cluster elastic deployment of the present invention. The working mechanism of the cluster elastic deployment management system of the present invention will be described below with reference to FIG.

As shown in FIG. 1 , the cluster elastic deployment management system of the present invention includes a central management server 10 , a cluster 20 , a cloud storage 30 , and a database 40 , as in the prior art cluster management system. Cloud storage 30 is used to store functional resource files such as Hadoop, Zookeeper, HBase image files, and configuration files. The database 40 is used to store related metadata information such as machines, clusters, and status maintenance information. Cluster 20 includes multiple machines and is used to handle tasks from accessing business applications. The central management server 10 reads the files and information stored in the cloud storage 30 and the database 40, and maintains the list information of the machines in the cluster 20 according to the files and information to acquire the cluster to which the machine belongs, the machine working state, and the machine source, and then the cluster. 20 Perform configuration maintenance, image file management, machine management, and security control operations. The difference from the cluster management system of the prior art is that the cluster 20 in the cluster elastic deployment management system of the present invention is a cluster deployed through a flexible solution, and the management system of the cluster elastic deployment further includes The resource pool 50 consisting of the machines that generate the cluster, The heart management server 10 is also responsible for maintaining the machines in the resource pool 50. The central management server 10 can communicate with the cluster 20 and machines in the resource pool 50 via various communication protocols, such as by the TCP/IP protocol.

Each of the resources in the resource pool 50 is capable of cooperating with the central management server 10 to perform operations related to receiving commands and echoing response messages. Alternatively, each machine in the resource pool 50 may be equipped with an HAgent for cooperating with the central management server 10 to perform related operations of receiving commands and echoing response messages. Each machine in resource pool 50 may include a detection script for checking for machine environmental correctness checks and whether related services are functioning properly. The above environmental correctness check includes but is not limited to the check of the operating system version, JDK version, host name, firewall, and time synchronization configuration. The above related services include, but are not limited to, remote storage (RS) and data node (Datanode) service processes.

The central management server 10 is also responsible for maintaining the list information of the machines in the resource pool 50, obtaining the cluster to which the machine belongs, the machine working state, and the machine type (physical machine or virtual machine) to prevent the machine from being repeatedly deployed into multiple clusters. After the HAgent is installed on each machine in the resource pool 50, each machine is registered in the central management server 10.

For the created cluster 20, if the machine needs to be dynamically added, one or several joins the cluster from the machines of the resource pool 50 through the central management server 10, and then the service process to be deployed on the selected machine is enabled; if the cluster If there is an idle condition in the machine in 20, the partially idle machine is released back to the resource pool 50 by the central management server 10.

When the access service application needs to access the cluster 20, the user of the service obtains the configuration information of the corresponding used cluster, and then sends the configuration information to the access service application so that it can pass an interface such as an HBase application programming interface (API). The cluster 20 is accessed, and the associated access service application of the cluster 20 can be recorded in the central management server 10.

In addition, when the HAgent performs operations such as modifying the ulimit and the hosts requiring root privileges, the central management server 10 approves the permission of the root authority operation, and only the trial is performed. The batch can only be operated after it has passed. For permission approval, the preferred way is to put the root username and password into the HAgent configuration file. When the central management server 10 operates, the root user name and password are used to perform permission verification and log recording, thereby tracking the operator and preventing maliciousness. The operation provides the user with the necessary audit trails.

2A-2D are schematic diagrams for describing a method of cluster elastic deployment of the present invention. The core processes of the cluster elastic deployment method of the present invention include creating a cluster, a cluster dynamic adding machine, a cluster dynamic deleting machine, and a cluster dynamic modifying configuration, which will be described one by one with reference to FIGS. 2A-2D.

2A is a schematic diagram of a method of creating a cluster in accordance with an embodiment of the present invention. The method for creating a cluster according to an embodiment of the present invention may include: defining basic information of a cluster, including a cluster name, a cluster machine type (physical machine or virtual machine), a usage validity period, and a cluster size; In the resource pool composed of the clustered machines, select the machine to be used to generate the cluster; select the version of the software to be used for the machine such as Hadoop, HBase, Zookeeper and modify the configuration information of the machine; select the service to be deployed for each machine. Process; according to the detection script included in the machine, the machine is checked for environmental correctness; if the check is passed, the image file configured for the machine is obtained from the cloud storage, and the service process to be deployed is enabled through the HAgent. Optionally, if the check fails, leave the machine in the resource pool; or repeat the above selection to use the version of the software for the machine and modify the configuration information of the machine, select the service process to be deployed, and perform the environment on the machine. The steps to check for correctness.

2B is a schematic diagram of a method of dynamically adding a machine to a cluster according to an embodiment of the present invention. A method for dynamically adding a machine according to an embodiment of the present invention may include: selecting an idle machine from a resource pool and designating a service process to be deployed thereon; and performing an environment correct for the machine according to a detection script included in the machine Sex check; if the check passes, get the image file configured for the machine from the cloud storage, update its configuration, and enable the service process to be deployed. Optionally, if the check fails, repeat the step of checking the environmental correctness of the machine; or select another idle machine from the resource pool, and Repeat the above steps.

2C is a schematic diagram of a method of cluster dynamic deletion of a machine in accordance with an embodiment of the present invention. The method for dynamically deleting a machine according to an embodiment of the present invention may include: selecting, by an interface of the central management server, a machine that needs to be stopped and stopping a service process deployed thereon; working on the cluster according to a detection script included in the machine The status is detected; if the cluster is working properly, the configuration information of the machine is reset to the initial state, so that its configuration information is returned to the state when it is put into the resource pool, thereby completing the classification of the machine into the resource pool. operating. Optionally, if the cluster is not working properly, re-enable the stopped service process on that machine.

2D is a schematic diagram of a method for dynamically modifying a configuration of a cluster according to an embodiment of the present invention. The method for dynamically modifying a configuration of a cluster according to the embodiment of the present invention may include: backing up an original configuration item of a machine for which configuration item modification is to be performed; and modifying a configuration item to be changed through an interface of the central management server; The included detection script reviews the legality of the modified configuration item; if the review is passed, the modified configuration item is pushed to the machine. Optionally, if the review fails, the modified configuration item is overwritten in the machine with the original configuration item of the backup.

According to the technical solution of the embodiment of the present invention, by introducing a resource pool in a cluster deployment, all the machines to be used for the cluster deployment are put into the resource pool, and a cluster capable of dynamically managing the elastic computing framework is provided, which can be expanded. Strong performance, maintainability, high resource utilization and low labor costs.

The basic principles of the present invention have been described above in connection with the specific embodiments, but it should be noted that those skilled in the art can understand that all or any of the steps or components of the method and apparatus of the present invention may be in any computing device. (including a processor, a storage medium, etc.) or a network of computing devices implemented in hardware, firmware, software, or a combination thereof, which is used by those of ordinary skill in the art in view of the description of the present invention. Programming skills can be achieved.

Thus, the objects of the invention can also be achieved by running a program or a set of programs on any computing device. The computing device can be a well-known general purpose device. Accordingly, the object of the present invention can also be achieved by merely providing a program product comprising program code for implementing the method or apparatus. That is to say, such a program product also constitutes the present invention, and a storage medium storing such a program product also constitutes the present invention. It will be apparent that the storage medium may be any known storage medium or any storage medium developed in the future.

It should also be noted that in the apparatus and method of the present invention, it is apparent that the various components or steps may be decomposed and/or recombined. These decompositions and/or recombinations should be considered as equivalents to the invention. Also, the steps of performing the series of processes described above may naturally be performed in chronological order in the order illustrated, but need not necessarily be performed in chronological order. Certain steps may be performed in parallel or independently of one another.

The above specific embodiments do not constitute a limitation of the scope of the present invention. Those skilled in the art will appreciate that a wide variety of modifications, combinations, sub-combinations and substitutions can occur depending on design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and scope of the invention are intended to be included within the scope of the invention.

Claims

A method for cluster elastic deployment includes the steps of:

Define basic information about the cluster;

Selecting the machine to be used to generate the cluster from a pool of resources consisting of machines intended to be used to generate the cluster;

Selecting a version of software to be used for the machine and modifying configuration information for the machine;

Select the service process to be deployed by the machine; and

Performing an environmental correctness check on the machine according to the detection script included in the machine, wherein if the environmental correctness check passes, obtaining an image file for configuring the machine from the cloud storage, and Enable the service process to be deployed on the machine.
The method of claim 1, wherein the defining the basic information of the cluster comprises: defining a cluster name of the cluster, a cluster machine type, a usage validity period, and a cluster size.
The method of claim 1, wherein said performing an environmental correctness check on said machine comprises: one or more of an operating system version, a JDK version, a host name, a firewall, a time synchronization configuration of said machine Check the item.
The method of claim 1 further comprising the step of adding a machine to said cluster, said steps comprising:

Selecting an idle machine from the resource pool and specifying the service process to be deployed on it;

Performing an environmental correctness check on the machine according to the detection script included in the machine, wherein if the environment correctness check passes, obtaining an image file for configuring the machine from the cloud storage, The configuration of the machine is updated and the service process to be deployed is enabled.
The method of claim 1 further comprising the step of dynamically deleting the machine to said cluster, said steps comprising:

Select the machine that needs to be stopped and stop the service process deployed on it;

The cluster working state is detected according to a detection script included in the machine, wherein if the cluster can work normally, the configuration information of the machine is reset to an initial state.
The method of claim 1 further comprising the step of dynamically modifying said cluster, said steps comprising:

Back up the original configuration items of the machine for which you want to modify the configuration items;

Modifying the configuration item of the machine to be changed;

Examining the legality of the modified configuration item according to the detection script included in the machine, wherein if the review of the legality of the configuration item passes, the modified configuration item is pushed to the machine.
A management system for cluster elastic deployment, including:

Central management server;

Cloud storage

database;

a pool of resources consisting of the machines that are intended to be used to generate the cluster;

A cluster deployed by the central management server, according to the method of any of claims 1-6.
The management system of claim 7, wherein the machine in the resource pool to be used to generate the cluster is capable of collaborating with the central management server to perform related operations of receiving commands and echoing response messages.
The management system of claim 7, wherein the machine in the resource pool to be used to generate the cluster comprises a detection script.
The management system according to claim 7, wherein the central management server is further configured to acquire a cluster to which the machine belongs, a machine working state, and a machine type.
The management system according to claim 7, wherein said central management server records an operation log of said cluster and said resource pool.