CN111506555B - Druid cluster migration method and device and computer equipment - Google Patents

Druid cluster migration method and device and computer equipment Download PDF

Info

Publication number
CN111506555B
CN111506555B CN202010173251.5A CN202010173251A CN111506555B CN 111506555 B CN111506555 B CN 111506555B CN 202010173251 A CN202010173251 A CN 202010173251A CN 111506555 B CN111506555 B CN 111506555B
Authority
CN
China
Prior art keywords
cluster
druid
migrated
target
zookeeper
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202010173251.5A
Other languages
Chinese (zh)
Other versions
CN111506555A (en
Inventor
蔡劲
万书武
贺波
蒋英明
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Ping An Technology Shenzhen Co Ltd
Original Assignee
Ping An Technology Shenzhen Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Ping An Technology Shenzhen Co Ltd filed Critical Ping An Technology Shenzhen Co Ltd
Priority to CN202010173251.5A priority Critical patent/CN111506555B/en
Publication of CN111506555A publication Critical patent/CN111506555A/en
Priority to PCT/CN2020/119058 priority patent/WO2021179584A1/en
Application granted granted Critical
Publication of CN111506555B publication Critical patent/CN111506555B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/21—Design, administration or maintenance of databases
    • G06F16/214—Database migration support
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
    • Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management

Landscapes

  • Engineering & Computer Science (AREA)
  • Databases & Information Systems (AREA)
  • Theoretical Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Computing Systems (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
  • Stored Programmes (AREA)

Abstract

The application discloses a migration method, a migration device and computer equipment of a guide cluster, relates to the technical field of computers, and can solve the problems that state data cannot be identified, the migration period is long and the working efficiency is low when the migration of the guide cluster is carried out. The method comprises the following steps: acquiring a to-be-migrated Druid cluster needing cluster migration and a target Druid cluster conforming to configuration requirements; migrating the state data in the to-be-migrated guide cluster to the target guide cluster through a capacity-expansion Zookeeper cluster; after the migration of the state data is completed, the metadata in the to-be-migrated Druid cluster is migrated to the target Druid cluster; and after the metadata is migrated, switching the destination cluster service of the destination cluster into the destination cluster service of the destination cluster by adding a history service so as to realize the migration of the destination cluster. The method and the device are suitable for migration of the Druid cluster.

Description

Druid cluster migration method and device and computer equipment
Technical Field
The present invention relates to the field of computing technologies, and in particular, to a method and an apparatus for migrating a ruid cluster, and a computer device.
Background
A set of Druid cluster comprising a plurality of machines is built in the Hadoop cluster, and due to the increase of data volume, task and cluster performance requirements, the machine configuration of the previously built Duird cluster can not meet the current requirements at all, so that a new Druid cluster needs to be built by replacing machines with higher configuration to meet cluster performance and service requirements. It is therefore necessary to migrate the services and data of the drud cluster to the newly built drud cluster.
At present, when migration of a Druid cluster is performed, state data of a Druid in a Zookeeper is directly copied to a machine of a new Druid cluster in a copying mode, and data locally cached on the machine of the Druid cluster is remotely migrated to the same directory of the new Druid cluster or a histologic service is directly started to enable the machine to download the data from an HDFS to the local cache. However, the data migration method can easily make the state data of the migrated guide unrecognizable, and can make the period of data migration longer, thus affecting the working efficiency.
Disclosure of Invention
In view of this, the present application provides a migration method and apparatus of a guide cluster, and a computer device, which can solve the problems that when migration of the guide cluster is performed, state data cannot be identified, and the migration period is long and the working efficiency is low.
According to one aspect of the present application, there is provided a migration method of a ruid cluster, including:
acquiring a to-be-migrated Druid cluster needing cluster migration and a target Druid cluster conforming to configuration requirements;
migrating the state data in the to-be-migrated guide cluster to the target guide cluster through a capacity-expansion Zookeeper cluster;
after the migration of the state data is completed, the metadata in the to-be-migrated Druid cluster is migrated to the target Druid cluster;
and after the metadata is migrated, switching the destination cluster service of the destination cluster into the destination cluster service of the destination cluster by adding a history service so as to realize the migration of the destination cluster.
Correspondingly, the migration of the state data in the to-be-migrated guide cluster to the target guide cluster through the capacity-expansion Zookeeper cluster specifically includes:
acquiring a Zookeeper state of the to-be-migrated guide cluster;
and expanding the Zookeeper cluster of the target Druid cluster into the Zookeeper cluster of the to-be-migrated Druid cluster based on the Zookeeper state so as to migrate the state data in the to-be-migrated Druid cluster into the target Druid cluster.
Specifically, the expanding the Zookeeper cluster of the target Druid cluster into the Zookeeper cluster of the to-be-migrated Druid cluster based on the Zookeeper state so as to migrate the state data in the to-be-migrated Druid cluster to the target Druid cluster specifically includes:
extracting a preset number of target Zookeeper clusters from the target guide clusters;
starting a Zookeeper service of the target Zookeeper cluster;
closing the original Zookeeper cluster with the Zookeeper state being follower, and modifying corresponding configuration information and ports to be the same as those in the target Zookeeper cluster;
closing the original Zookeeper cluster with the Zookeeper state being a leader, and modifying corresponding configuration information and ports to be the same as those in the target Zookeeper cluster;
restarting the Zookeeper service in the to-be-migrated guide cluster, so as to reselect a leader in the Zookeeper state based on an election principle, and further loading the Zookeeper data in the to-be-migrated guide cluster to the Zookeeper of the target guide cluster.
Correspondingly, after completing the migration of the metadata, switching the destination cluster service of the destination cluster to the destination cluster service of the destination cluster by adding a history service, so as to realize the migration of the destination cluster, which specifically includes:
modifying a machine configuration $DRUID_HOME/conf/device/_common/common.runtime.properties of the target device cluster;
sequentially closing the device service of the device cluster to be migrated, and modifying the machine configuration $DRUID_HOME/conf/device/_common/. Runntime.properties of the device cluster to be migrated to be the same as that of the target device cluster;
sequentially starting the Druid service on the Druid cluster to be migrated, and starting the history service in the target Druid cluster so as to migrate the local cache data of the Druid cluster to be migrated to the target Druid cluster;
closing broker, coordinator, overlord, middleManager service of the to-be-migrated guide cluster;
and opening coordinator, overlord, middleManager, broker service of the target drud cluster.
Specifically, the closing broker, coordinator, overlord, middleManager service of the to-be-migrated guide cluster specifically includes:
using a command sh browser.sh stop to close a browser under a $DRUID_HOME/bin directory of a first machine in the to-be-migrated guide cluster;
using a command sh connector to close the connector under the $DRUID_HOME/bin directory of the second machine in the to-be-migrated guide cluster;
closing overlock by using a command sh overlock.sh stop under the $DRUID_HOME/bin directory of a third machine in the to-be-migrated guide cluster;
and using a command sh middleManager to close the middleManager under the $DRUID_HOME/bin directory of the fourth machine in the to-be-migrated guide cluster.
Correspondingly, the opening coordinator, overlord, middleManager, broker service of the target guide cluster specifically includes:
starting a browser under a $DRUID_HOME/bin directory of a first machine in the target guide cluster by using a command sh start_server.
Starting a connector under a $DRUID_HOME/bin directory of a second machine in the target guide cluster by using a command sh start_server.
Starting overlog under a $DRUID_HOME/bin directory of a third machine in the target guide cluster by using a command sh start_server.
The command sh start_server.sh middleManager is used to launch the middleManager under the $DRUID_HOME/bin directory of the fourth machine in the target guide cluster.
Correspondingly, after switching the drud cluster service of the to-be-migrated drud cluster to the drud cluster service of the target drud cluster by adding the histocal service so as to realize the migration of the to-be-migrated drud cluster, the method specifically further comprises:
and sequentially removing the Zookeeper of the to-be-migrated guide cluster and the history.
According to another aspect of the present application, there is provided a migration apparatus of a ruid cluster, including:
the acquisition module is used for acquiring the to-be-migrated Druid cluster needing cluster migration and a target Druid cluster meeting configuration requirements;
the migration module is used for migrating the state data in the to-be-migrated Druid cluster to the target Druid cluster through the capacity-expansion Zookeeper cluster;
the migration module is further configured to migrate metadata in the to-be-migrated Druid cluster to the target Druid cluster after completing migration of the state data;
and the switching module is used for switching the Druid cluster service of the Druid cluster to be migrated into the Druid cluster service of the target Druid cluster by adding the history service after completing the migration of the metadata so as to realize the migration of the Druid cluster to be migrated.
Correspondingly, the migration module is specifically configured to obtain a Zookeeper state of the to-be-migrated guide cluster;
and expanding the Zookeeper cluster of the target Druid cluster into the Zookeeper cluster of the to-be-migrated Druid cluster based on the Zookeeper state so as to migrate the state data in the to-be-migrated Druid cluster into the target Druid cluster.
The migration module is specifically configured to extract a preset number of target Zookeeper clusters from the target guide clusters;
starting a Zookeeper service of the target Zookeeper cluster;
closing the original Zookeeper cluster with the Zookeeper state being follower, and modifying corresponding configuration information and ports to be the same as those in the target Zookeeper cluster;
closing the original Zookeeper cluster with the Zookeeper state being a leader, and modifying corresponding configuration information and ports to be the same as those in the target Zookeeper cluster;
restarting the Zookeeper service in the to-be-migrated guide cluster, so as to reselect a leader in the Zookeeper state based on an election principle, and further loading the Zookeeper data in the to-be-migrated guide cluster to the Zookeeper of the target guide cluster.
Correspondingly, the switching module is specifically configured to modify a machine configuration $DRUID_HOME/conf/device/_common/common.run.properties of the target device cluster;
sequentially closing the device service of the device cluster to be migrated, and modifying the machine configuration $DRUID_HOME/conf/device/_common/. Runntime.properties of the device cluster to be migrated to be the same as that of the target device cluster;
sequentially starting the Druid service on the Druid cluster to be migrated, and starting the history service in the target Druid cluster so as to migrate the local cache data of the Druid cluster to be migrated to the target Druid cluster;
closing broker, coordinator, overlord, middleManager service of the to-be-migrated guide cluster;
and opening coordinator, overlord, middleManager, broker service of the target drud cluster.
The switching module is specifically configured to use a command sh maker, a sh stop to close the maker under a $DRUID_HOME/bin directory of a first machine in the to-be-migrated ruid cluster;
using a command sh connector to close the connector under the $DRUID_HOME/bin directory of the second machine in the to-be-migrated guide cluster;
closing overlock by using a command sh overlock.sh stop under the $DRUID_HOME/bin directory of a third machine in the to-be-migrated guide cluster;
and using a command sh middleManager to close the middleManager under the $DRUID_HOME/bin directory of the fourth machine in the to-be-migrated guide cluster.
Correspondingly, the switching module is specifically configured to start a browser by using a command sh start_server.
Starting a connector under a $DRUID_HOME/bin directory of a second machine in the target guide cluster by using a command sh start_server.
Starting overlog under a $DRUID_HOME/bin directory of a third machine in the target guide cluster by using a command sh start_server.
The command sh start_server.sh middleManager is used to launch the middleManager under the $DRUID_HOME/bin directory of the fourth machine in the target guide cluster.
Correspondingly, the device further comprises a removing module, wherein the removing module is used for sequentially removing the Zookeeper and the history of the to-be-migrated guide cluster.
According to still another aspect of the present application, there is provided a non-volatile readable storage medium having a computer program stored thereon, which when executed by a processor implements the above-described migration method of a ruid cluster.
According to still another aspect of the present application, there is provided a computer device, including a nonvolatile readable storage medium, a processor, and a computer program stored on the nonvolatile readable storage medium and executable on the processor, where the processor implements the migration method of the guide cluster described above when executing the program.
By means of the technical scheme, compared with the migration mode of the current guide cluster, the migration method, the migration device and the computer equipment of the guide cluster can migrate the state data in the guide cluster to be migrated to the target guide cluster through the capacity-expansion Zookeeper cluster when the cluster is migrated, so that zero risk migration of the Zookeeper data can be guaranteed, migration speed is further improved, and stability of service operation is guaranteed. By adding the history service so as to be based on the high availability of the history of the guide cluster, uninterrupted and unbroken migration of local data to the target guide cluster by a user without perception is realized, seamless butt migration of the data is realized, compatibility of the data and the cluster service is ensured, and the working efficiency is further improved.
Drawings
The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiments of the application and together with the description serve to explain the application and do not constitute an undue limitation to the present application. In the drawings:
fig. 1 shows a flow chart of a migration method of a ruid cluster according to an embodiment of the present application;
fig. 2 is a flow chart illustrating another migration method of a ruid cluster according to an embodiment of the present application;
fig. 3 is a schematic structural diagram of migration of a ruid cluster according to an embodiment of the present application;
fig. 4 is a schematic structural diagram of a migration apparatus of a ruid cluster according to an embodiment of the present application;
fig. 5 is a schematic structural diagram of another migration apparatus of a ruid cluster according to an embodiment of the present application.
Detailed Description
The present application will be described in detail hereinafter with reference to the accompanying drawings in conjunction with embodiments. It should be noted that, without conflict, the embodiments and features of the embodiments in the present application may be combined with each other.
The method aims at solving the problems of long migration period and low working efficiency when the Druid cluster is migrated at present. The embodiment of the application provides a migration method of a guide cluster, as shown in fig. 1, the method includes:
101. and acquiring the to-be-migrated Druid cluster needing cluster migration and a target Druid cluster meeting configuration requirements.
The configuration requirement means that the cluster information in the target drud cluster is corresponding to the cluster information in the to-be-migrated drud cluster, and the configuration in the target drud cluster is higher.
102. And migrating the state data in the to-be-migrated guide cluster to the target guide cluster through the capacity-expansion Zookeeper cluster.
For the embodiment, in a specific application scenario, in order to migrate a to-be-migrated guide cluster to a target guide cluster, firstly, migration of a Zookeeper cluster is required, and Zookeeper data is migrated from the to-be-migrated guide cluster to the target guide cluster, so that the original state of the Zookeeper cluster is maintained.
103. And after the migration of the state data is completed, the metadata in the to-be-migrated Druid cluster is migrated to the target Druid cluster.
For the embodiment, in a specific application scenario, since metadata of the to-be-migrated Druid cluster and the target Druid cluster are stored in the same Mysql library and table, and the two clusters use the same user and password. The metadata of the target drud cluster may seamlessly pair with the metadata of the to-be-migrated drud cluster.
104. After the metadata is migrated, the method comprises the steps of switching the Druid cluster service of the Druid cluster to be migrated into the Druid cluster service of the target Druid cluster by adding the history service so as to realize the migration of the Druid cluster to be migrated.
For this embodiment, in a specific application scenario, by adding a history service, the Druid service may migrate local data to the machine of the target Druid cluster.
By the migration method of the guide clusters in the embodiment, when the clusters are migrated, the state data in the guide clusters to be migrated can be migrated to the target guide clusters through the capacity-expansion Zookeeper clusters, so that zero risk migration of the Zookeeper data can be ensured, the migration speed is further improved, and the stability of service operation is ensured. By adding the history service so as to be based on the high availability of the history of the guide cluster, uninterrupted and unbroken migration of local data to the target guide cluster by a user without perception is realized, seamless butt migration of the data is realized, compatibility of the data and the cluster service is ensured, and the working efficiency is further improved.
Further, as a refinement and extension of the foregoing embodiment, in order to fully describe a specific implementation procedure in this embodiment, another migration method of a guide cluster is provided, as shown in fig. 2, where the method includes:
201. and acquiring the to-be-migrated Druid cluster needing cluster migration and a target Druid cluster meeting configuration requirements.
For example, to-be-migrated Druid cluster corresponds to Druid cluster a, and the target Druid cluster corresponds to Druid cluster B, in this application, as shown in fig. 3, it is intended to completely convert data in Druid cluster a into Druid cluster B, and replace Druid cluster a with Druid cluster B, where the cluster information of Druid cluster a and Druid cluster B corresponds to the same, but the configuration in Druid cluster B is higher.
202. And acquiring a Zookeeper state of the to-be-migrated guide cluster.
In a specific application scenario, the command zkServer.sh status may be used to check the status of the Zookeeper of the drive cluster A, and at this time, the one with the largest myid is selected as the leader, so that the A1 status is known as the follower, the A2 status is known as the follower, and the A3 status is known as the leader.
203. And expanding the Zookeeper cluster of the target guide cluster into the Zookeeper cluster of the guide cluster to be migrated based on the Zookeeper state so as to migrate the state data in the guide cluster to be migrated to the target guide cluster.
For the present implementation, in a specific application scenario, the embodiment step 203 may specifically include: extracting a preset number of target Zookeeper clusters from the target guide clusters; starting a Zookeeper service of a target Zookeeper cluster; closing an original Zookeeper cluster with a Zookeeper state being a follower, and modifying corresponding configuration information and ports to be the same as those in a target Zookeeper cluster; closing an original Zookeeper cluster with a Zookeeper state being a leader, and modifying corresponding configuration information and ports to be the same as those in a target Zookeeper cluster; restarting the Zookeeper service in the to-be-migrated Druid cluster so as to reselect the leader in the Zookeeper state based on the election principle, and further loading the Zookeeper data in the to-be-migrated Druid cluster onto the Zookeeper of the target Druid cluster.
In a specific application scenario, in order to facilitate selection of a leader, when a Zookeeper cluster is subjected to capacity expansion, the number of expanded Zookeeper clusters needs to be ensured to be an odd number. For example, in the scheme, the Zookeeper cluster can be expanded from 3 machines to 7 machines, and the machine IP of the newly added guide cluster B is B1, B2, B3, and B4. Taking the example of expanding the Zookeeper cluster from 3 machines to 7 machines, the specific capacity expansion processing steps are as follows: changing the configuration of the Zookeeper on the Druid cluster B into the IP of 7 Zookeeper in total of the Druid cluster B and the Druid cluster A, wherein the ports are the same; starting 4 ZooKeeper services of which the IP is B1, B2, B3 and B4, wherein the ZooKeeper states of the 4 machines are follower; skipping the A3 machine with the Zookeeper state of the guide cluster A as a leader, closing the Zookeeper by using a command zkServer.sh stop on A1 and A2, modifying the IP and the port of the configuration file to be the same as those on B1, and starting the Zookeeper service by using a zkServer.sh start command on A1 and A2; in the step, two folwers need to be stopped firstly, and because the leader data is up to date, the latest data in the Zookeeper can be ensured; and finally closing the Zookeeper service of A3 where the leader is located, changing the configuration file IP and the port to be the same as those of B1, and starting the Zookeeper service on A3 by using a zkServer.sh start command, wherein the Zookeeper needs to be reelected at the moment, and the leader of which B4 is the Zookeeper can be known by the election principle of the Zookeeper. In the step, the state data can be transferred, namely the Zookeeper data in the machine A is loaded onto the Zookeeper machine of the machine B, and when the machine data on the Zookeeper on the machine A is completely synchronous to the Zookeeper data on the machine B, the Zookeeper service of the machine A can be stopped, and the machine A is completely unloaded.
204. And after the migration of the state data is completed, the metadata in the to-be-migrated Druid cluster is migrated to the target Druid cluster.
For this embodiment, in a specific application scenario, since metadata of the to-be-migrated and target Druid clusters are stored in the same Mysql library and table, the B1, B2, B3, B4, B5, and B6 machines are wall-opened to the Mysql server, and the two clusters use the same user and password. The metadata of the target drud cluster may seamlessly pair with the metadata of the to-be-migrated drud cluster.
205. After the metadata is migrated, the method comprises the steps of switching the Druid cluster service of the Druid cluster to be migrated into the Druid cluster service of the target Druid cluster by adding the history service so as to realize the migration of the Druid cluster to be migrated.
For the present embodiment, in a specific application scenario, the embodiment step 205 may specifically include: modifying the machine configuration $DRUID_HOME/conf/drive/_common/common.runtime.properties of the target drive cluster; sequentially closing the guide services of the guide cluster to be migrated, and modifying the machine configuration $DRUID_HOME/conf/guide/_common/. Runntime.properties of the guide cluster to be migrated to be the same as that of the target guide cluster; the method comprises the steps of starting a guide service on a to-be-migrated guide cluster in sequence, and starting a histocal service in a target guide cluster so as to migrate local cache data of the to-be-migrated guide cluster to the target guide cluster; closing broker, coordinator, overlord, middleManager service of the to-be-migrated Druid cluster; the coordinator, overlord, middleManager, broker service of the target drud cluster is started.
For example, the configuration $DRUID_HOME/conf/device/_common/common.runtime.properties of 6 machines of the device cluster B may be modified, with device.zk.service.host=A1, A2, A3, B1, B2, B3, B4; sequentially closing the guide service of the guide cluster A, and enabling the guide.zk.service.host=A1, A2, A3, B1, B2, B3 and B4 in the $DRUID_HOME/conf/guide/_common/. Runntime.properties configuration file of 6 machines of the guide cluster A; then sequentially starting the guide service on the guide cluster A and starting the histocal service on the B5 and B6 machines; after B5B 6 is started, 4 history services are provided, tasks are operated later, partial data are cached on machines of B5 and B6, the data on A5 are redistributed and distributed on the rest machines by a guide cluster by turning off the service of A5, the operation is carried out for a period of time, after the data on A5 are distributed on the rest 3 machines uniformly, the history service on the A6 machine is turned off, and the data on the A6 machine are automatically distributed on the rest two machines, namely B5 and B6; sequentially closing broker, coordinator, overlord, middleManager services of the drive cluster A; the coordinator, overlord, middleManager, broker service of the guide cluster B is turned on in turn, and the user needs to make a corresponding IP to modify the service code.
Accordingly, closing broker, coordinator, overlord, middleManager service of the to-be-migrated guide cluster may specifically include: using a command sh browser.sh stop to close a browser under a $DRUID_HOME/bin directory of a first machine in a Druid cluster to be migrated; using a command sh connector to close the connector under the $DRUID_HOME/bin directory of the second machine in the guide cluster to be migrated; closing overword under $DRUID_HOME/bin directory of third machine in the to-be-migrated guide cluster by using command sh overword; the command sh middleManager is used under the $DRUID_HOME/bin directory of the fourth machine in the Druid cluster to be migrated.
For example, the command sh maker. Shstop may be used to turn off the maker under the $DRUID_HOME/bin directory on A1; using a command sh binder. Sh stop to close the binder under the $DRUID_HOME/bin directory on A2; closing overword using command sh overword. Sh stop under $DRUID_HOME/bin directory on A3; the command sh middleManager is used under the $ drud_home/bin directory on A4. Sh stop closes the middleManager.
Accordingly, the opening coordinator, overlord, middleManager, broker service of the target guide cluster may specifically include: starting a browser under a $DRUID_HOME/bin directory of a first machine in a target guide cluster by using a command sh start_server. Starting a corer under a $DRUID_HOME/bin directory of a second machine in the target guide cluster by using a command sh start_server. Starting overword under the $DRUID_HOME/bin directory of the third machine in the target guide cluster by using a command sh start_server. The command sh start_server.sh middleManager is used to launch the middleManager under the $DRUID_HOME/bin directory of the fourth machine in the target guide cluster.
For example, a command sh start server. Sh reader may be used to start a reader under the $DRUID_HOME/bin directory on B1; starting a binder under the $DRUID_HOME/bin directory on B2 by using a command sh start_server. Starting overword under the $DRUID_HOME/bin directory on B3 by using a command sh start_server. The command sh start_server.sh middleManager is used to launch the middleManager under the $DRUID_HOME/bin directory on B4.
206. And sequentially removing the Zookeeper and the histocal of the to-be-migrated guide cluster.
For the embodiment, in a specific application scenario, the specific steps of removing the Zookeeper of the to-be-migrated guide cluster may be: sequentially stopping the zookeeper services of the to-be-migrated Druid cluster (Druid cluster A), A1, A2 and A3; skipping the machine with the Zookeeper state of leader, stopping the Zookeeper service of the B1 machine, changing the IP of the Zookeeper cluster in the configuration file of the Zookeeper into the service of B1, B2, B3 and B4, restarting the service of the Zookeeper, and closing the Zookeeper service on the B4 machine after the B2 and B3 execute the operation in sequence, wherein the leader of the Zookeeper is the Zookeeper on the B3 machine with the second largest myd.
Accordingly, the specific steps of removing the history of the to-be-migrated guide cluster may be: under the/appcom/drive/bin directory on the A5, A6 machines, the sh history command is used to close the histocal service of the to-be-migrated drive cluster (drive cluster A). At this time, all data of the to-be-migrated Druid cluster is completely migrated into the target Druid cluster and no services of the to-be-migrated Druid cluster are relied on.
By the migration method of the guide clusters, when the clusters are migrated, the state data in the guide clusters to be migrated are migrated to the target guide clusters through the capacity-expansion Zookeeper clusters, zero risk migration of the Zookeeper data can be guaranteed, migration speed is further improved, and stability of service operation and safety of the data are guaranteed. By adding the history service so as to realize uninterrupted migration of local data to the target gui cluster without perception of a user based on high availability of the history of the gui cluster, seamless butt migration of the data is realized, compatibility of the data and the cluster service is ensured, the situation that the cluster runs for a period of time after migration and runs tasks or errors occur in related services of the gui is avoided, the problem that the gui cluster service needs to be migrated back to the original cluster is well avoided, and the working efficiency is further improved.
Further, as a specific embodiment of the method shown in fig. 1 and fig. 2, an embodiment of the present application provides a migration apparatus for a device cluster, as shown in fig. 4, where the apparatus includes: an acquisition module 31, a migration module 32, and a switching module 33.
The obtaining module 31 is configured to obtain a to-be-migrated guide cluster and a target guide cluster that meets a configuration requirement, where the to-be-migrated guide cluster needs to be migrated;
the migration module 32 is configured to migrate, through the capacity-expanding Zookeeper cluster, the state data in the to-be-migrated Druid cluster to the target Druid cluster;
the migration module 32 is further configured to migrate metadata in the to-be-migrated drud cluster to the target drud cluster after completing migration of the state data;
and the switching module 33 is configured to switch the destination cluster service of the destination cluster to the destination cluster service of the destination cluster by adding the history service after completing the migration of the metadata, so as to implement the migration of the destination cluster to be migrated.
In a specific application scenario, in order to implement migration of state data in the to-be-migrated guide cluster to the target guide cluster, the migration module 32 is specifically configured to obtain a Zookeeper state of the to-be-migrated guide cluster; and expanding the Zookeeper cluster of the target guide cluster into the Zookeeper cluster of the guide cluster to be migrated based on the Zookeeper state so as to migrate the state data in the guide cluster to be migrated to the target guide cluster.
Correspondingly, in order to expand the Zookeeper cluster of the target guide cluster into the Zookeeper cluster of the guide cluster to be migrated based on the Zookeeper state, the migration module 32 is specifically configured to extract a preset number of target Zookeeper clusters from the target guide cluster; starting a Zookeeper service of a target Zookeeper cluster; closing an original Zookeeper cluster with a Zookeeper state being a follower, and modifying corresponding configuration information and ports to be the same as those in a target Zookeeper cluster; closing an original Zookeeper cluster with a Zookeeper state being a leader, and modifying corresponding configuration information and ports to be the same as those in a target Zookeeper cluster; restarting the Zookeeper service in the to-be-migrated Druid cluster so as to reselect the leader in the Zookeeper state based on the election principle, and further loading the Zookeeper data in the to-be-migrated Druid cluster onto the Zookeeper of the target Druid cluster.
In a specific application scenario, in order to switch the Druid cluster service of the Druid cluster to be migrated to the Druid cluster service of the target Druid cluster by adding the history service, so as to implement migration of the Druid cluster to be migrated, a switching module 33 is specifically configured to modify a machine configuration $druid_home/conf/Druid/_common/common. Sequentially closing the guide services of the guide cluster to be migrated, and modifying the machine configuration $DRUID_HOME/conf/guide/_common/. Runntime.properties of the guide cluster to be migrated to be the same as that of the target guide cluster; the method comprises the steps of starting a guide service on a to-be-migrated guide cluster in sequence, and starting a histocal service in a target guide cluster so as to migrate local cache data of the to-be-migrated guide cluster to the target guide cluster; closing broker, coordinator, overlord, middleManager service of the to-be-migrated Druid cluster; the coordinator, overlord, middleManager, broker service of the target drud cluster is started.
Correspondingly, in order to close the broker, coordinator, overlord, middleManager service of the to-be-migrated guide cluster, the switching module 33 is specifically configured to use a command sh maker.sh stop to close the maker under the $DRUID_HOME/bin directory of the first machine in the to-be-migrated guide cluster; using a command short to close the connector under the $DRUID_HOME/bin directory of the second machine in the to-be-migrated guide cluster; closing overword under $DRUID_HOME/bin directory of third machine in the to-be-migrated guide cluster by using command sh overword; the command sh middleManager is used under the $DRUID_HOME/bin directory of the fourth machine in the Druid cluster to be migrated.
Correspondingly, in order to start the coordinator, overlord, middleManager, broker service of the target device cluster, the switching module 33 is specifically configured to start a browser using the command sh start_server. Starting a corer under a $DRUID_HOME/bin directory of a second machine in the target guide cluster by using a command sh start_server. Starting overword under the $DRUID_HOME/bin directory of the third machine in the target guide cluster by using a command sh start_server. The command sh start_server.sh middleManager is used to launch the middleManager under the $DRUID_HOME/bin directory of the fourth machine in the target guide cluster.
In a specific application scenario, in order to stop the service of the to-be-migrated guide cluster when the to-be-migrated guide cluster completes migration, as shown in fig. 5, the apparatus further includes: the module 34 is removed.
A removing module 34, configured to sequentially remove the Zookeeper and the history of the to-be-migrated guide cluster.
It should be noted that, for other corresponding descriptions of each functional unit related to the migration apparatus of the guide cluster provided in this embodiment, reference may be made to corresponding descriptions in fig. 1 to fig. 2, and no further description is given here.
Based on the above methods shown in fig. 1 and fig. 2, correspondingly, the embodiments of the present application further provide a storage medium, where a computer program is stored, and the program when executed by a processor implements the above method for migrating a guide cluster shown in fig. 1 and fig. 2.
Based on such understanding, the technical solution of the present application may be embodied in the form of a software product, which may be stored in a non-volatile storage medium (may be a CD-ROM, a U-disk, a mobile hard disk, etc.), and includes several instructions for causing a computer device (may be a personal computer, a server, or a network device, etc.) to perform the method of each implementation scenario of the present application.
Based on the methods shown in fig. 1 and fig. 2 and the virtual device embodiments shown in fig. 4 and fig. 5, in order to achieve the above objects, the embodiments of the present application further provide a computer device, which may specifically be a personal computer, a server, a network device, etc., where the entity device includes a storage medium and a processor; a storage medium storing a computer program; and a processor for executing a computer program to implement the migration method of the drud cluster shown in fig. 1 and fig. 2.
Optionally, the computer device may also include a user interface, a network interface, a camera, radio Frequency (RF) circuitry, sensors, audio circuitry, WI-FI modules, and the like. The user interface may include a Display screen (Display), an input unit such as a Keyboard (Keyboard), etc., and the optional user interface may also include a USB interface, a card reader interface, etc. The network interface may optionally include a standard wired interface, a wireless interface (e.g., bluetooth interface, WI-FI interface), etc.
It will be appreciated by those skilled in the art that the computer device structure provided in this embodiment is not limited to this physical device, and may include more or fewer components, or may combine certain components, or may be arranged in different components.
The non-volatile readable storage medium may also include an operating system, a network communication module, etc. The operating system is a program of physical device hardware and software resources for voice identification, supporting the execution of information handling programs and other software and/or programs. The network communication module is used for realizing communication among components in the nonvolatile readable storage medium and communication with other hardware and software in the entity device.
Through the description of the above embodiments, it can be clearly understood by those skilled in the art that the present application may be implemented by adding a necessary general hardware platform to software, or when performing cluster migration, the state data in the to-be-migrated guide cluster is migrated to the target guide cluster by expanding the Zookeeper cluster, so that zero risk migration of the Zookeeper data can be ensured, the migration speed is further improved, and the stability of service operation and the security of data are ensured. By adding the history service so as to realize uninterrupted migration of local data to the target gui cluster without perception of a user based on high availability of the history of the gui cluster, seamless butt migration of the data is realized, compatibility of the data and the cluster service is ensured, the situation that the cluster runs for a period of time after migration and runs tasks or errors occur in related services of the gui is avoided, the problem that the gui cluster service needs to be migrated back to the original cluster is well avoided, and the working efficiency is further improved.
Those skilled in the art will appreciate that the drawings are merely schematic illustrations of one preferred implementation scenario, and that the modules or flows in the drawings are not necessarily required to practice the present application. Those skilled in the art will appreciate that modules in an apparatus in an implementation scenario may be distributed in an apparatus in an implementation scenario according to an implementation scenario description, or that corresponding changes may be located in one or more apparatuses different from the implementation scenario. The modules of the implementation scenario may be combined into one module, or may be further split into a plurality of sub-modules.
The foregoing application serial numbers are merely for description, and do not represent advantages or disadvantages of the implementation scenario. The foregoing disclosure is merely a few specific implementations of the present application, but the present application is not limited thereto and any variations that can be considered by a person skilled in the art shall fall within the protection scope of the present application.

Claims (10)

1. The migration method of the drud cluster is characterized by comprising the following steps of:
acquiring a to-be-migrated Druid cluster needing cluster migration and a target Druid cluster conforming to configuration requirements;
migrating the state data in the to-be-migrated guide cluster to the target guide cluster through a capacity-expansion Zookeeper cluster;
after the migration of the state data is completed, the metadata in the to-be-migrated Druid cluster is migrated to the target Druid cluster;
and after the metadata is migrated, switching the destination cluster service of the destination cluster into the destination cluster service of the destination cluster by adding a history service so as to realize the migration of the destination cluster.
2. The method of claim 1, wherein the migration of the state data in the to-be-migrated guide cluster to the target guide cluster by the capacity-expanding Zookeeper cluster specifically includes:
acquiring a Zookeeper state of the to-be-migrated guide cluster;
and expanding the Zookeeper cluster of the target Druid cluster into the Zookeeper cluster of the to-be-migrated Druid cluster based on the Zookeeper state so as to migrate the state data in the to-be-migrated Druid cluster into the target Druid cluster.
3. The method according to claim 2, wherein the expanding the Zookeeper cluster of the target Druid cluster into the Zookeeper cluster of the to-be-migrated Druid cluster based on the Zookeeper state so as to migrate the state data in the to-be-migrated Druid cluster to the target Druid cluster specifically includes:
extracting a preset number of target Zookeeper clusters from the target guide clusters;
starting a Zookeeper service of the target Zookeeper cluster;
closing the original Zookeeper cluster with the Zookeeper state being follower, and modifying corresponding configuration information and ports to be the same as those in the target Zookeeper cluster;
closing the original Zookeeper cluster with the Zookeeper state being a leader, and modifying corresponding configuration information and ports to be the same as those in the target Zookeeper cluster;
restarting the Zookeeper service in the to-be-migrated guide cluster, so as to reselect a leader in the Zookeeper state based on an election principle, and further loading the Zookeeper data in the to-be-migrated guide cluster to the Zookeeper of the target guide cluster.
4. The method of claim 3, wherein after completing the migration of the metadata, switching the Druid cluster service of the Druid cluster to be migrated to the Druid cluster service of the target Druid cluster by adding a histocal service so as to realize the migration of the Druid cluster to be migrated, specifically comprising:
modifying a machine configuration $DRUID_HOME/conf/device/_common/common.runtime.properties of the target device cluster;
sequentially closing the device service of the device cluster to be migrated, and modifying the machine configuration $DRUID_HOME/conf/device/_common/. Runntime.properties of the device cluster to be migrated to be the same as that of the target device cluster;
sequentially starting the Druid service on the Druid cluster to be migrated, and starting the history service in the target Druid cluster so as to migrate the local cache data of the Druid cluster to be migrated to the target Druid cluster;
closing broker, coordinator, overlord, middleManager service of the to-be-migrated guide cluster;
and opening coordinator, overlord, middleManager, broker service of the target drud cluster.
5. The method of claim 4, wherein the closing broker, coordinator, overlord, middleManager service of the to-be-migrated Druid cluster specifically comprises:
using a command sh browser.sh stop to close a browser under a $DRUID_HOME/bin directory of a first machine in the to-be-migrated guide cluster;
using a command sh connector to close the connector under the $DRUID_HOME/bin directory of the second machine in the to-be-migrated guide cluster;
closing overlock by using a command sh overlock.sh stop under the $DRUID_HOME/bin directory of a third machine in the to-be-migrated guide cluster;
and using a command sh middleManager to close the middleManager under the $DRUID_HOME/bin directory of the fourth machine in the to-be-migrated guide cluster.
6. The method of claim 5, wherein the opening coordinator, overlord, middleManager, broker services of the target drud cluster specifically comprises:
starting a browser under a $DRUID_HOME/bin directory of a first machine in the target guide cluster by using a command sh start_server.
Starting a connector under a $DRUID_HOME/bin directory of a second machine in the target guide cluster by using a command sh start_server.
Starting overlog under a $DRUID_HOME/bin directory of a third machine in the target guide cluster by using a command sh start_server.
The command sh start_server.sh middleManager is used to launch the middleManager under the $DRUID_HOME/bin directory of the fourth machine in the target guide cluster.
7. The method of claim 1, wherein after switching the destination cluster service of the destination cluster to the destination cluster service of the destination cluster by adding a history service to implement migration of the destination cluster, specifically further comprising:
and sequentially removing the Zookeeper of the to-be-migrated guide cluster and the history.
8. A migration apparatus for a ruid cluster, comprising:
the acquisition module is used for acquiring the to-be-migrated Druid cluster needing cluster migration and a target Druid cluster meeting configuration requirements;
the migration module is used for migrating the state data in the to-be-migrated Druid cluster to the target Druid cluster through the capacity-expansion Zookeeper cluster;
the migration module is further configured to migrate metadata in the to-be-migrated Druid cluster to the target Druid cluster after completing migration of the state data;
and the switching module is used for switching the Druid cluster service of the Druid cluster to be migrated into the Druid cluster service of the target Druid cluster by adding the history service after completing the migration of the metadata so as to realize the migration of the Druid cluster to be migrated.
9. A non-transitory readable storage medium having stored thereon a computer program, characterized in that the program when executed by a processor implements the migration method of the guided cluster of any one of claims 1 to 7.
10. A computer device comprising a non-volatile readable storage medium, a processor and a computer program stored on the non-volatile readable storage medium and executable on the processor, characterized in that the processor implements the migration method of the ruid cluster of any of claims 1 to 7 when the program is executed by the processor.
CN202010173251.5A 2020-03-13 2020-03-13 Druid cluster migration method and device and computer equipment Active CN111506555B (en)

Priority Applications (2)

Application Number Priority Date Filing Date Title
CN202010173251.5A CN111506555B (en) 2020-03-13 2020-03-13 Druid cluster migration method and device and computer equipment
PCT/CN2020/119058 WO2021179584A1 (en) 2020-03-13 2020-09-29 Method and apparatus for migrating druid cluster, computer device and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202010173251.5A CN111506555B (en) 2020-03-13 2020-03-13 Druid cluster migration method and device and computer equipment

Publications (2)

Publication Number Publication Date
CN111506555A CN111506555A (en) 2020-08-07
CN111506555B true CN111506555B (en) 2023-05-23

Family

ID=71863859

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202010173251.5A Active CN111506555B (en) 2020-03-13 2020-03-13 Druid cluster migration method and device and computer equipment

Country Status (2)

Country Link
CN (1) CN111506555B (en)
WO (1) WO2021179584A1 (en)

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111506555B (en) * 2020-03-13 2023-05-23 平安科技（深圳）有限公司 Druid cluster migration method and device and computer equipment
CN112800029A (en) * 2021-01-29 2021-05-14 紫光云技术有限公司 Method for overall migration of ceph cluster

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105426439A (en) * 2015-11-05 2016-03-23 腾讯科技（深圳）有限公司 Metadata processing method and device
CN106959950A (en) * 2016-01-08 2017-07-18 北京京东尚科信息技术有限公司 The method and apparatus of migrating data between application cluster
CN109062666A (en) * 2018-07-27 2018-12-21 浪潮电子信息产业股份有限公司 Virtual machine cluster management method and related device

Family Cites Families (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7447940B2 (en) * 2005-11-15 2008-11-04 Bea Systems, Inc. System and method for providing singleton services in a cluster
CN103259863B (en) * 2013-05-17 2016-01-06 汉柏科技有限公司 Based on the system and method that the control zookeeper of cluster serves
US10353918B2 (en) * 2014-11-07 2019-07-16 Amobee, Inc. High availability and disaster recovery in large-scale data warehouse
US10628235B2 (en) * 2018-01-11 2020-04-21 Robin Systems, Inc. Accessing log files of a distributed computing system using a simulated file system
CN109376010B (en) * 2018-09-28 2020-11-27 上海思询信息科技有限公司 Method for realizing cross-cluster resource migration based on Openstack
CN111506555B (en) * 2020-03-13 2023-05-23 平安科技（深圳）有限公司 Druid cluster migration method and device and computer equipment

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105426439A (en) * 2015-11-05 2016-03-23 腾讯科技（深圳）有限公司 Metadata processing method and device
CN106959950A (en) * 2016-01-08 2017-07-18 北京京东尚科信息技术有限公司 The method and apparatus of migrating data between application cluster
CN109062666A (en) * 2018-07-27 2018-12-21 浪潮电子信息产业股份有限公司 Virtual machine cluster management method and related device

Also Published As

Publication number Publication date
CN111506555A (en) 2020-08-07
WO2021179584A1 (en) 2021-09-16

Similar Documents

Publication Publication Date Title
US9960963B2 (en) Dynamic client fail-over during a rolling patch installation based on temporal server conditions
US10031959B2 (en) Extensible application state preservation architecture
US9928059B1 (en) Automated deployment of a multi-version application in a network-based computing environment
CN106445951B (en) File transmission method and device
WO2014183213A1 (en) Dynamic rendering for software applications
CN107168780B (en) Virtual reality scene loading method and equipment and virtual reality equipment
TW201607280A (en) Methods and apparatus to support globalplatform usage on an embedded UICC
US20160008720A1 (en) Management device, management method, terminal device, control method, and program
US20140325043A1 (en) Network Switching Method, Version Upgrade Method, and Terminal Device
CN111506555B (en) Druid cluster migration method and device and computer equipment
US20170168807A1 (en) Method and electronic device for updating application program
AU2019261875B2 (en) Computer system providing SaaS application session state migration features and related methods
EP3016354A1 (en) Cloud-service-based data backup and operation method
CN105450759A (en) System mirror image management method and device
US20170163787A1 (en) Method and electronic device for upgrading or downgrading system
US20190121786A1 (en) Method for Data Migration and Terminal
US10795747B2 (en) File synchronizing service status monitoring and error handling
US10031764B2 (en) Managing executable files
CN102880478B (en) Oftware updating method
EP4026009A1 (en) Flexible datapath offload chaining
WO2018053838A1 (en) Load balancing method and related device
US10503430B2 (en) Method and device for clearing data and electronic device
CN107885543B (en) Function management method and device of application program and intelligent terminal
CN110673992B (en) Multi-mirror image fusion self-starting method and system
WO2017096889A1 (en) Method and device for upgrading and downgrading system

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant