CN111400065B - Pulsar message long-distance multi-live method and system for separating global zookeeper - Google Patents
Pulsar message long-distance multi-live method and system for separating global zookeeper Download PDFInfo
- Publication number
- CN111400065B CN111400065B CN202010175685.9A CN202010175685A CN111400065B CN 111400065 B CN111400065 B CN 111400065B CN 202010175685 A CN202010175685 A CN 202010175685A CN 111400065 B CN111400065 B CN 111400065B
- Authority
- CN
- China
- Prior art keywords
- cluster
- zookeeper
- global
- pulsar
- sub
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a remote multi-live method and a system for separating pulsear messages of global zookeeper, which comprise N pulsear clusters for separating global zookeeper, wherein each pulsear cluster for separating global zookeeper comprises: at least one local zookeeper sub-cluster, at least one global zookeeper sub-cluster; wherein all global zookeeper sub-clusters of the N separate global zookeeper clusters are interconnected; and each pulsar cluster of the split global zookeeper respectively creates a topic with the same name, so that the brokers in each pulsar cluster can communicate with brokers of other pulsar clusters.
Description
Technical Field
The embodiment of the invention relates to the technical field of data processing, in particular to a remote multi-live technology of a message queue, and particularly relates to a pulsar message remote multi-live method and system for separating global zookeeper.
Background
The remote multi-live of the message queue is a very important technology in the field of disaster prevention backup, and in order to realize that the service is not affected in the case of catastrophic failure, the remote multi-live of the message queue must be effectively realized. The existing allopatric multi-activity technology of the message queue is mainly a kafka cross-cluster synchronization technology based on Mirror marker.
Kafka is a linkenin sourced, high-performance distributed streaming platform, which is mainly used for publishing and subscribing streaming data, persisting the streaming data in a fault-tolerant manner, and processing the streaming data. For each topic, kafka maintains a log of partitions, each of which is an ordered, unchanging sequence of messages that are appended to the structured commit log. The messages in each partition have a unique Id, called offset, that uniquely identifies each message in the partition. The Kafka cluster persistently retains published messages using a configurable retention period, e.g., if the retention policy is set to two days, a record is available for use within two days after the record is published, and is then discarded to free space. The performance of Kafka is practically constant with respect to data size, so storing data for a long time is not an issue.
In the Kafka cluster, a producer is responsible for adding records to a commit log, a broker is responsible for copy between copies of respective topics, and a consumer is responsible for reading records from a specific offset and converting the records into messages, and the offset is moved to realize the function of sequential consumption.
In a Kafka cluster, each partition has one server that acts as a "leader" and zero or more servers that act as "followers". The leader handles all read and write requests to the partition, while the follower passively copies the leader. If the leader fails, one of the followers will automatically become the new leader. Each server acts as a leader for some of its partitions and as a follower for other partitions, so the load in the cluster is well balanced.
MirrorMarker is an official Kafka inter-cluster replication tool that implements mirroring functionality in another cluster by consuming a source cluster to a target cluster. Multiple mirror programs may be initiated simultaneously to increase the throughput of replication and improve fault tolerance. The underlying implementation principle of the MirrorMarker technology is to subscribe to the topic message of one kafka cluster and publish it to the topic of another kafka cluster. The two topics to be copied are independent of each other, and the other configurations may be completely different, including partition and repeater, etc., except that the same topic name is required. Since the MirrorMarker technology is based on topic level, whitelist is usually specified, which may be a simple topic name or a group of topics using regular expressions. Meanwhile, since some topic does not necessarily exist in the target kafka cluster, auto.
The disadvantages of the MirrorMarker are that: the MirrorMarker technology only provides a producer consumer tool based on configuration, and can realize message transfer among clusters, and the method has a plurality of defects, mainly comprising the following aspects:
single point of failure: the mirrormarker is a single process, and once the process cannot transmit messages due to resource limitation or network reasons, the messages are blocked or even synchronization is interrupted. Even if multiple processes can be deployed, load balancing and maintenance among the multiple processes is very difficult and costly.
Unidirectional replication: the mirrormarker can only be copied from source cluster to target cluster, and the one-way copied data is only suitable for main and standby scenes and cannot be suitable for double-living scenes.
Double configuration: the mirrormarker needs to configure the controller of the source cluster and the generator of the target cluster at the same time, and the dual-configuration maintenance and change cost is higher.
Topic grade: the synchronization strategy without the whole cluster level can only be realized through a regular expression, and whether synchronization is carried out or not is often difficult to determine when topic is newly added.
The delay is large: the mirrormarker consumes the source cluster in a consumer mode, the message is consumed after being serialized to the log file, and then the message is transmitted to the target cluster through the network and is consumed finally after being serialized to the log file, and the delay is often large and uncontrollable.
Disclosure of Invention
Aiming at the defects of the message queue remote multi-live technology in the prior art, the embodiment of the invention provides a pulsar message remote multi-live method and a pulsar message remote multi-live system for separating a global zookeeper.
In order to solve the above problem, an embodiment of the present invention provides a pulser message remote multi-active system of split global zookeeper, including N pulser clusters of split global zookeeper, where each pulser cluster of split global zookeeper includes: at least one local zookeeper sub-cluster, at least one global zookeeper sub-cluster;
wherein all global zookeeper sub-clusters of the N separate global zookeeper clusters are connected to each other;
the pulsar clusters of each split global zookeeper respectively create a topic with the same name, so that the brokers in each pulsar cluster can communicate with brokers of other pulsar clusters;
in some embodiments, the system further comprises a control end, wherein the control end is connected with each separated global zookeeper cluster; wherein the control end is used for executing at least one of the following operations:
maintaining a Pulsar cluster machine;
health detection of a Pulsar port number;
the Pulsar multi-cluster content and namespace have the functions of unified creation, updating and deletion;
the Pulsar multi-cluster topic has the functions of unified query, creation and deletion;
unified deleting function of pulser multi-cluster provider and subscription;
and the Pulsar multi-cluster load report unifies the query function.
In some embodiments, in the case that the network is unavailable, the pulser clusters of the N split global zookeeper perform data transmission respectively for their respective topics; after the network is available, synchronizing the respective topic messages of the pulser clusters of the N separate global zookeeper to other pulser clusters through geo-replication.
In some embodiments, the global zookeeper sub-cluster is a metadata storage system that performs ex-place backup of geo-replication.
In some embodiments, a local zookeeper sub-cluster employs a 2181 port, and the global zookeeper sub-cluster employs a 2184 port; wherein the number of nodes of the local zookeeper sub-cluster in each separated global zookeeper cluster is an odd number not less than 3, and the global zookeeper sub-cluster comprises not less than 5 nodes.
Meanwhile, an embodiment of the present invention further provides a data processing method using the pulsar message allopatric multi-active system that separates global zookeeper, including:
respectively initializing each separated global zookeeper cluster, namely executing initialization-cluster-metadata for each separated global zookeeper cluster for N times;
respectively executing clusterer create in each separated global zookeeper cluster to establish a global zookeeper sub-cluster in the cluster;
respectively creating the same rule content and named namespace for each separated global zookeeper cluster; n times in total; the method comprises the following steps:
creating a content: bin/pulsar-admin tents create CDC-c Cluster1, cluster2
Creating namespace: bin/pulsar-admin namespaces create CDC/ns-c Cluster1, cluster2
The same topic is created separately for each separate global zookeeper cluster.
The technical scheme of the invention has the following beneficial effects: the technical scheme provides a method and a system for separating the pulser cluster of the global zookeeper to realize message remote multi-live, and the method and the system can avoid the risk of message loss of the global zookeeper besides the characteristic of the global zookeeper cluster.
Drawings
FIG. 1 is a diagram of a related art architecture at pulsar;
FIG. 2 is a prior art pulsear ex-situ multi-live technique of global zookeeper;
FIG. 3 is a pulsar remote multi-live technique for separating global zookeeper according to an embodiment of the present invention.
Detailed Description
In order to make the technical problems, technical solutions and advantages of the present invention more apparent, the following detailed description is given with reference to the accompanying drawings and specific embodiments.
The following first explains various terms referred to in the present application:
record: record the elementary units of an information carrier describing a fact or state.
Log: a log for recording data or a record of data changes.
Message: messages, a base unit for data transfer between machines.
Stream: stream, a combination of messages that organizes one or more messages together in a time dimension.
Publish: publish, send out messages or store them.
Subscriber: subscription, the process of sequentially receiving messages from a message storage system.
Topic: the theme, the message carries on the logical and physical segregation in the form of theme, make the message handling simpler.
Producer: the producer, publishes the message to an application for a particular topic or group of topics.
Consumer: the consumer receives the application program for processing in sequence with one theme or a plurality of specific themes.
Zookeeper: a distributed, distributed application coordinates consistent services.
Partition: and the partitions divide the topic into different storage and processing units, so that the message throughput and performance are improved, and one topic can be composed of one or more partitions.
A Replicator: the copy, a storage unit that makes redundancy to the topic or the partition, increases redundancy of data, and may also improve performance of the consumer, and a partition may also be composed of one or more replicators.
A Leader: the leader, the copies are synchronized through a replication technique, and the copies used for interacting with the producer and the consumer are called leaders, and one fragment usually has only one leader.
Follower: the follower, which is not directly interactive with the producer and the consumer, is only used for improving the redundancy, and when the leader fails or is disconnected from the cluster, one copy is selected from the follower to become the leader, which is called leader election.
And (4) Broker: the server of the message, usually referred to as the server of topic, may persist the service of the message.
Cluster: the cluster is a server side consisting of one or more brokers, one cluster can support a plurality of brokers and also can support a plurality of topics, and the partitions and the copies of the topics are subjected to load balancing and copy synchronization in different brokers.
One of the related arts is the pulsar remote multi-live technology of global zookeeper. Pulsar is a yohoo open source technology, and is used for a multi-tenant high-performance solution for message delivery from a server to a server. Similar to kakfa, pulser also provides producer and consumer support, as well as message persistence. Unlike kafka, pulser can provide lower end-to-end delay while supporting the seamless geo-replication feature across clusters. In the Pulsar cluster, an additional offload function is provided for the message, and the message can be retained in a third-party storage engine, including hdfs, aws and the like. More importantly, pulsar provides a more reliable serialization guarantee than kafka, and supports a flush mechanism for messages that persist to storage hardware rather than to a file system.
In the related art, the principle of a pulsear ex-situ multi-live technology of a Global zookeeper is shown in fig. 1, and in the architecture diagram of the pulser, an internal component called Global relays and an external component called Global ZK form a pulser cluster together. Based on the global zookeeper technology, the pulsar can seamlessly realize the geo reproduction. Pulsar's brooker, unlike kafka, is a stateless proxy. The Global registers in the browser are specially used for performing the geo-replication, when the producer issues a message to the browser, the browser simultaneously acquires a plurality of target clusters needing to be subjected to the repeater from the Global browser, writes the message to other target clusters, and completes the cross-cluster synchronization of the message.
In FIG. 1, whenever P1, P2 and P3 producers publish messages to the T1 topics on Cluster-A, cluster-B and Cluster-C clusters, respectively, these messages are immediately replicated between clusters. After replication, the C1 and C2 users can use the messages in their respective clusters. Without the geographical replication, the C1 and C2 users would not be able to use the messages issued by the P3 producer.
In contrast to the MirrorMarker of kafka, geo-replication offers a number of advantages, such as:
the self-character of the Broker: kafka brookers are only responsible for messages of the local cluster, while brookers of pulser have provided inter-cluster replication functionality with geo replication itself.
No single point of failure: because the browser instances are distributed, the geo-replication is attached to the browsers and is also distributed, the defect of single point failure does not exist, once a certain browser instance crashes, the fault transfer also occurs, and the high availability is ensured.
Mutual replication: while the MirrorMarker of kafka can only be copied from source cluster to target cluster singly, the geo-copying of pulsar can support the mutual copying of two or more clusters, and seamlessly supports allopatric multi-activity.
Global zookeeper configuration: two clusters need to be configured by the MirrorMarker of kafka, and only one additional global zookeeper configuration is needed among a plurality of clusters of the geo-replication of pulsar, so that unified maintenance is easier.
Cluster level: unlike the karrka Mirror marker based topic replication, pulser's geo replication synchronizes based on cluster.
The delay is small: after the Global replicators in the pulser issue the messages, the Global replicators can immediately multiplex the cache data in the memory queue and issue the messages to other clusters, so that the read of the disk file is not needed, and the delay is smaller.
Although the geo-replication technology can solve the defects of many kafka mirrormarker, the geo-replication technology is based on the implementation of a global zookeeper, and has a plurality of defects, mainly including the following points:
deployment cost: the global zookeeper needs to be deployed across all the clusters, and the expansion and change costs are high.
And (3) state maintenance: in the global zookeeper cluster, the network is often unreliable, the network across the data center is less reliable, once a network fault occurs, the cluster state is very complex, and the processing is very difficult.
Low performance: the global zookeeper cluster generates great performance fluctuation due to the fact that writing data needs to guarantee cluster consistency across data centers, exponential amplification may be generated due to delay, performance loss of the pulsar cluster is also great, and great risk is brought.
Network interruption: once network interruption occurs, part of clusters cannot perform geo-replication, and even puslir cluster fails due to local global cookie failure, so that publication and subscription of messages cannot be performed.
Difficult to recover: once a network failure occurs in the global zookeeper, and a network partition is generated, the failed local pulsar cluster and the global zookeeper cluster are difficult to restore to a usable state, and if a global network branch occurs, all the clusters are not usable until the network partition problem disappears.
Due to the problems of the global zookeeper in the related technology, the remote multi-activity technology of the pulsar has great instability and risk in the production environment, the invention separates the global zookeeper in a targeted manner so as to avoid various unreliable risk points of the global zookeeper, so that the remote multi-activity technology of the pulsar is more stable, the risk is reduced, and the pulsar can be put into formal application in the production environment.
The geo reproduction provided by Pulsar is designed based on a global zookeeper, in order to overcome many problems brought by the global zookeeper, the invention adopts a technical scheme of separating the global zookeeper to realize the function of the global zookeeper, and in order to better understand the technical scheme of separating the global zookeeper, the invention only analyzes the technical principle and explains the technical scheme from the geo reproduction scene spanning two Pulsar clusters.
FIG. 2 is an architecture diagram illustrating a global zookeeper deployment; the global zookeeper deployment architecture as shown in fig. 2 includes two clusters: cluster A and Cluster B; of course, the specific number of the cluster is not limited in the embodiment of the present invention. As shown in FIG. 2, there are three node instances for the global zookeeper in Cluster A, zk1.A:2184, zk2.A:2184, and zk3.A:2184. There are two node instances for global zookeeper in Cluster B, zk1.B:2184, and zk3.B:2184, respectively.
As can be seen from the example shown in fig. 2, the global zookeeper of the two clusters Cluster a and Cluster B is unbalanced, where the global zookeeper requires at least (n + 1)/2 =3 node instances to provide services. The performance of the Cluster A is different after the Cluster B performs network partition, the Cluster A still survives 3 node instances and can perform normal message processing, and the Cluster B cannot meet the service providing condition because only two node instances of zk1.B:2184 and zk3.B:2184 are available, and the brooder in the Cluster B cannot be connected with the global browser.
In the pulser, each topic needs to be hosted by a broker corresponding to the topic, the broker corresponding to the topic becomes an ownershirp of the topic, and the following conditions must be satisfied when a client (including a producer and a consumer) of the pulser interacts with the pulser cluster:
specifying topic, default global topic;
positioning ownershirp of the topic through a lookup interface of the topic, and storing the ownershirp relation in the global zookeeper;
client performs send and receive operations of the message with ownershirp of the topic.
Global zookeeper is a metadata storage system for geo-replay, and once the Global zookeeper cannot be connected, the api of the Client cannot find the ownershirp of topic from the browser instance. The cluster is in an unavailable state due to the fact that the client-side message cannot be published and subscribed.
In the deployment architecture diagram of the global zookeeper as shown in fig. 2, in the zookeeper deployment architecture of pulser, 2181 port represents a local zookeeper cluster, 2184 port represents a global cluster, all zookeeper clusters must guarantee that the number of nodes is odd and at least three nodes, and the global zookeeper cluster has at least 5 nodes. The pulser cluster forbids automatic creating topic, the production environment is usually not started, and metadata pollution is avoided.
Where each topoc only needs to be created once in one of the clusters (e.g., clusterA and clusterb in fig. 2), and the topoc is stored in a global zookeeper sub-Cluster in that Cluster. A specific topic may be created by any local zookeeper sub-cluster in a cluster. After a Producer issues a message, the message is written into a topic stored in any local sub-cluster, and then a global copy global identifier of the message is synchronized into other clusters through a global browser sub-cluster; when network connection between two clusters is available, the cluster creating topoic synchronizes the global part to the second cluster through the global cookie sub-cluster by the global requester after the non-displaced cluster and the cluster creating topoic are connectable; and the second cluster creates a corresponding topic through the duplicate task, so that the second cluster can directly send the send and receive to the topic in the included sub-cluster.
Fig. 3 is a diagram illustrating an architecture of a cluster technique for splitting a global zookeeper according to an embodiment of the present invention.
In the embodiment of the invention, in the architecture of the separated global zookeeper's Pulsar message queue remote multi-active system, 2181 port represents a local zookeeper sub-cluster, 2184 port represents a global zookeeper sub-cluster, all the local zookeeper sub-clusters must ensure that the number of nodes is odd and at least three nodes, and the global zookeeper sub-cluster must ensure that no less than 5 nodes. The pulser cluster forbids automatic creating topic, the production environment cannot be started frequently, and metadata pollution is avoided. As shown in fig. 3, an exemplary embodiment of the present invention lists a Pulsar message queue offsite multi-live architecture for splitting a global zookeeper, which includes two clusters: cluster A and Cluster B. As shown in fig. 3, each cluster includes at least one local sub-cluster with 2181 ports: a local zookeeper sub-cluster, further comprising at least one global zookeeper sub-cluster with 2184 ports: a global zookeeper sub-cluster. Wherein, each Cluster is Cluster A and Cluster B
As shown in fig. 3, there is no global zookeeper function between ClusterA and clusterb. As shown in fig. 3, each cluster comprises at least one local zookeeper sub-cluster based on 2181 port and at least one global zookeeper sub-cluster based on 2184 port; wherein the global zookeeper sub-clusters of each Cluster (e.g. Cluster A and Cluster B in FIG. 3) are mutually conducted for data transmission; the system further comprises a control terminal admin, an interface of the control terminal admin is similar to the deployment architecture of the global zookeeper shown in fig. 2, and the control terminal admin can communicate with each Cluster (e.g., cluster a and Cluster B in fig. 3).
In the separated global zookeeper cluster of the embodiment of the invention, the same topic is established through the local zookeeper sub-clusters, and the metadata of the topic is transmitted to the global zookeeper sub-clusters in the cluster. Thus each cluster can send and receive to the topic individually even when the network between the clusters is unavailable.
In the embodiment of the present invention, when deploying a pulsar remote multi-live technical scheme based on a split global zookeeper, assuming that N clusters need to perform remote multi-live (N is greater than or equal to 2), the method of the embodiment of the present invention includes:
locally deploying separate global zookeeper clusters, wherein each separate global zookeeper cluster comprises N sub-clusters, wherein each separate global zookeeper cluster comprises at least one 2181 port-based local zookeeper sub-cluster and at least one 2184 port-based global zookeeper sub-cluster; and the global zookeeper sub-clusters of the N separate global zookeeper clusters can be accessed independently;
1. respectively initializing each separated global zookeeper cluster, namely executing initialization-cluster-metadata for each separated global zookeeper cluster for N times; initialization-cluster-metadata is a cluster initialization step, which is the prior art and is not described herein again;
2. respectively executing clusterer create in each separated global zookeeper cluster to establish 2184 port-based global zookeeper sub-clusters in the cluster, wherein each separated global zookeeper cluster is executed for N +1 times, and the total number of N is (N + 1);
3. respectively creating the same content and named namespace for each separated global zookeeper cluster; n times in total; the method comprises the following steps:
creating a content: bin/pulsar-admin tents create CDC-c Cluster1, cluster2
Creating namespace: bin/pulsar-admin namespaces create CDC/ns-c Cluster1, cluster2
4. Respectively establishing the same topic for each separated global zookeeper cluster; each separate global zookeeper cluster is executed once, for a total of N times.
It can be seen that deployment and topic creation costs where there are multiple clusters are N to N x N steps more times the solution of global zookeeper operates. In order to eliminate the complexity cost, the invention specially develops a set of pulsar multi-activity management system for cluster deployment and maintenance based on a separated zookeeper, and the system is developed and realized based on a pulsar java client.
The self-defined Pulsar multi-activity management system function comprises the following points:
pulsar cluster machine maintenance
Pulsar port number health detection
Function of unified creation, update and deletion of Pulsar multi-cluster content and namespace
Function of unified query, creation and deletion of Pulsar multi-cluster topic
Unified deleting function for pulser multi-cluster conner and subscription
Pulsar multi-cluster load report unified query function
After deployment is completed, the send message of any one cluster in the N clusters is copied to other clusters point to point through global replicators, all the N clusters can normally run even if network unreachable faults occur, and the message can be automatically synchronized to other target clusters needing synchronization when the network is reachable, so that continuous and stable running of all the clusters is ensured.
The technical scheme can avoid various instabilities of the global zookeeper, production accidents caused by the instabilities and even the risk of losing messages besides the characteristic of the global zookeeper cluster.
While the foregoing is directed to the preferred embodiment of the present invention, it will be understood by those skilled in the art that various changes and modifications may be made therein without departing from the spirit and scope of the invention as defined in the appended claims.
Claims (6)
1.A split global zookeeper's pulsear message remote multi-active system comprising N split global zookeeper clusters, wherein each split global zookeeper cluster comprises: at least one local zookeeper sub-cluster, at least one global zookeeper sub-cluster; each local zookeeper subgroup comprises an odd number of nodes which are not less than three; wherein each global zookeeper sub-cluster comprises at least five nodes;
wherein all global zookeeper sub-clusters of the N separate global zookeeper clusters are connected to each other;
and respectively establishing the same topic for bearing messages by each separated global zookeeper cluster, so that the brooker in each pulsar cluster can communicate with the brookers of other pulsar clusters.
2. The split global zookeeper' pulsear message placeshifting multi-active system of claim 1, further comprising a control terminal connecting each split global zookeeper cluster; wherein the control end is used for executing at least one of the following operations:
pulsar cluster machine maintenance;
pulse port number health detection;
the Pulsar multi-cluster content and namespace have the functions of unified creation, updating and deletion;
the pulser multi-cluster topic unified query, creation and deletion functions;
the pulser multi-cluster consumer and description unified deletion function;
and the Pulsar multi-cluster load report unifies the query function.
3. The split global zokeeper 'pulsear message ex-situ multi-active system as recited in claim 1, wherein when a network is unavailable, the N split global zokeeper' pulser clusters each perform data transfer for their respective topoics; after the network is available, synchronizing respective topic messages of the pulser clusters of the N separate global zookeeper to other pulser clusters through geo-replication.
4. The pulser message placeshifting multi-live system of split global zookeeper of claim 1, wherein the global zookeeper sub-cluster is a metadata storage system that performs placeshifting backup geo-replication.
5. The separate global zookeeper' pulsear message allopatric multi-active system of claim 1, wherein a local zookeeper sub-cluster employs a 2181 port, and the global zookeeper sub-cluster employs a 2184 port; wherein the number of nodes of the local zookeeper sub-cluster in each separated global zookeeper cluster is an odd number not less than 3, and the global zookeeper sub-cluster comprises not less than 5 nodes.
6. The data processing method of the pulser message placeshifting multi-active system of the split global zookeeper of any of claims 1-5, comprising:
initializing each separated global zookeeper cluster respectively, namely executing initialization-cluster-metadata for each separated global zookeeper cluster for N times in total;
respectively executing clusterer create in each separated global zookeeper cluster to establish a global zookeeper sub-cluster in the cluster;
respectively creating the same rule content and named namespace for each separated global zookeeper cluster; n times in total; the method comprises the following steps:
creating a content: bin/pulsar-admin tendents create CDC-cCluster1, cluster2
Creating namespace: bin/pulsar-admin namespaces create CDC/ns-cCluster1, cluster2
And respectively establishing the same topic for each separated global zookeeper cluster.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010175685.9A CN111400065B (en) | 2020-03-13 | 2020-03-13 | Pulsar message long-distance multi-live method and system for separating global zookeeper |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010175685.9A CN111400065B (en) | 2020-03-13 | 2020-03-13 | Pulsar message long-distance multi-live method and system for separating global zookeeper |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111400065A CN111400065A (en) | 2020-07-10 |
| CN111400065B true CN111400065B (en) | 2023-04-14 |
Family
ID=71432413
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010175685.9A Active CN111400065B (en) | 2020-03-13 | 2020-03-13 | Pulsar message long-distance multi-live method and system for separating global zookeeper |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111400065B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112231399A (en) * | 2020-09-25 | 2021-01-15 | 中国建设银行股份有限公司 | Method and device applied to graph database |
| CN113688111A (en) * | 2021-10-25 | 2021-11-23 | 深圳市明源云科技有限公司 | Cross-region message copying method, system, electronic equipment and readable storage medium |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104092767A (en) * | 2014-07-21 | 2014-10-08 | 北京邮电大学 | Posting/subscribing system for adding message queue models and working method thereof |
| CN106953901A (en) * | 2017-03-10 | 2017-07-14 | 重庆邮电大学 | A kind of trunked communication system and its method for improving message transmission performance |
| CN108280080A (en) * | 2017-01-06 | 2018-07-13 | 阿里巴巴集团控股有限公司 | A kind of method of data synchronization, device and electronic equipment |
Family Cites Families (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10261523B2 (en) * | 2015-08-17 | 2019-04-16 | Rainboxx, Inc. | Apparatus and method for controlling irrigation process by sending encoded acoustical messages along irrigation conduit |
| CN110019240B (en) * | 2017-12-29 | 2021-11-09 | 中国移动通信集团北京有限公司 | Service data interaction method, device and system |
| CN109684099B (en) * | 2018-11-23 | 2020-12-25 | 新华三大数据技术有限公司 | Message processing method and device |
| CN110515748B (en) * | 2019-08-28 | 2022-02-01 | 腾讯科技（深圳）有限公司 | Message processing method and related device |
-
2020
- 2020-03-13 CN CN202010175685.9A patent/CN111400065B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104092767A (en) * | 2014-07-21 | 2014-10-08 | 北京邮电大学 | Posting/subscribing system for adding message queue models and working method thereof |
| CN108280080A (en) * | 2017-01-06 | 2018-07-13 | 阿里巴巴集团控股有限公司 | A kind of method of data synchronization, device and electronic equipment |
| CN106953901A (en) * | 2017-03-10 | 2017-07-14 | 重庆邮电大学 | A kind of trunked communication system and its method for improving message transmission performance |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111400065A (en) | 2020-07-10 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109729129B (en) | Configuration modification method of storage cluster system, storage cluster and computer system | |
| AU2019236685B2 (en) | Distributed file system using consensus nodes | |
| US9811541B2 (en) | System and method for supporting lazy deserialization of session information in a server cluster | |
| US9846704B2 (en) | Distributed file system using consensus nodes | |
| US7870235B2 (en) | Highly scalable and highly available cluster system management scheme | |
| CN1893370B (en) | Server cluster recovery and maintenance method and system | |
| EP2643771B1 (en) | Real time database system | |
| US20070174660A1 (en) | System and method for enabling site failover in an application server environment | |
| US20090144338A1 (en) | Asynchronously replicated database system using dynamic mastership | |
| US20100023564A1 (en) | Synchronous replication for fault tolerance | |
| US20070061379A1 (en) | Method and apparatus for sequencing transactions globally in a distributed database cluster | |
| WO2023280127A1 (en) | Message communication method and apparatus | |
| US10423643B2 (en) | System and method for supporting resettable acknowledgements for synchronizing data in a distributed data grid | |
| US20120278817A1 (en) | Event distribution pattern for use with a distributed data grid | |
| CN111400065B (en) | Pulsar message long-distance multi-live method and system for separating global zookeeper | |
| CN110661841B (en) | Data consistency method for distributed service discovery cluster in micro-service architecture | |
| US7743381B1 (en) | Checkpoint service | |
| US20090144333A1 (en) | System for maintaining a database | |
| WO2015196692A1 (en) | Cloud computing system and processing method and apparatus for cloud computing system | |
| CN112231399A (en) | Method and device applied to graph database | |
| CN109218386B (en) | High-availability method for managing Hadoop namespace | |
| CN112202601B (en) | Application method of two physical node mongo clusters operated in duplicate set mode | |
| CN111309552A (en) | Service log acquisition system and method | |
| KR101491452B1 (en) | Method and apparatus for data replication using messaging channel | |
| JP5845298B2 (en) | Nodes and programs |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB03 | Change of inventor or designer information |
Inventor after: Zhang Shaofeng Inventor after: Chen Hewei Inventor after: Gao Weiguo Inventor before: Zhang Shaofeng Inventor before: Gao Weiguo |
|
| CB03 | Change of inventor or designer information | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |