WO2021037015A1

WO2021037015A1 - Full data synchronization device suitable for a/b network double clusters

Info

Publication number: WO2021037015A1
Application number: PCT/CN2020/111011
Authority: WO
Inventors: 王民安; 顾信安; 黎邓根
Original assignee: 湖南中车时代通信信号有限公司
Priority date: 2019-08-26
Filing date: 2020-08-25
Publication date: 2021-03-04
Also published as: CN110489491B; CN110489491A

Abstract

Provided is a full data synchronization device suitable for A/B network double clusters. According to the full data synchronization device, one end is connected with a first distributed service cluster, and the other end is connected with a second distributed service cluster; the data in the first distributed service cluster is read, and is compared with the corresponding data in the second distributed service cluster, and if the data does not exist in the second distributed service cluster or is not the latest data, the data is written into the second distributed service cluster; the data in the second distributed service cluster is read, and is compared with the corresponding data in the first distributed service cluster, and if the data does not exist in the first distributed service cluster or is not the latest data, the data is written into the first distributed service cluster. The present invention solves the problem of data inconsistency between two independent distributed service clusters, and meets the reliability requirement of the traditional rail transit industry for double clusters under the A/B network redundancy architecture.

Description

A full data synchronization device suitable for A/B network double clusters

This application claims the priority of a Chinese patent application filed with the Chinese Patent Office on August 26, 2019, the application number is 201910790758.2. The invention title is "A full data synchronization device suitable for A/B network dual clusters", all of which The content is incorporated in this application by reference.

Technical field

The invention relates to the technical field of clusters, and more specifically, to a full data synchronization device suitable for A/B network double clusters.

Background technique

At present, during the operation of rail transit, the monitoring and control of trains are very important. Therefore, many reliability designs have been carried out during the development of rail transit signal systems. In the urban rail transit signal system, the ATS system (Automatic Train Supervision), as the signal supervision and control end, is the most direct and convenient operation tool for dispatchers, and its importance is self-evident.

At present, most ATS systems in China have adopted the architectural requirements of A/B network. The so-called A/B network architecture requirements refer to the redundant deployment strategy of OCC (operating control center) dual hosts (active and standby). OCC, (single line) operation control center, is a management model with one line and one center. The so-called dual-system redundancy strategy refers to the existence of application server A and application server B in the system, one of which is the host and the other is the standby. The host synchronizes data to the standby in real time. When the host fails or is shut down, the standby The machine is automatically upgraded to the main machine. When the original main machine is repaired and restarted, it will be automatically upgraded to the standby machine. This classic redundancy strategy can ensure that the ATS function will not cause the main function of the ATS to fail due to the downtime of one of the servers.

In the above scheme, the data synchronization function as the bottom layer is more important, and the data synchronization between application server A and application server B must be real-time.

A cluster is a group of independent computers interconnected through a high-speed network. They form a group and are managed as a single system. When a client interacts with the cluster, the cluster is like an independent server. The cluster configuration is used to improve availability and scalability.

ZooKeeper is a software that provides consistent services for distributed applications. The functions provided include configuration maintenance, domain name services, distributed synchronization, group services, etc.

The ZooKeeper cluster can be used in the ATS system to work as an application server. Due to ZooKeeper itself, the ZooKeeper cluster only supports single network (A network or B network) work. In order to meet the A/B network redundancy architectural requirements in the rail transit industry, two independent ZooKeeper clusters must be established, A network is one Clusters (or network B is a cluster), the clusters do not communicate with each other due to the isolation of the A/B network, and the cluster client reads/writes the data of the two clusters in real time. If due to network or other reasons, one cluster writes successfully and another cluster fails to write, then there will be inconsistencies in the data of the two clusters, which will affect the function of the entire system.

In order to meet the redundant architecture of the traditional A/B network, data must be synchronized in two independent ZooKeeper clusters.

In the existing technical solutions, the underlying protocol of ZooKeeper can be modified to support the architectural requirements of the A/B network, and data consistency can be achieved within the protocol itself. However, the open source feature of ZooKeeper will result in poor version maintenance and a relatively large amount of modification.

Therefore, there is currently no device that can solve the data synchronization problem between two independent ZooKeeper clusters.

Summary of the invention

The purpose of the present invention is to provide a dual-cluster full data synchronization device, which solves the problem of data synchronization between two independent ZooKeeper clusters and improves the data reliability of the ZooKeeper cluster.

In order to achieve the above objective, the present invention provides a full data synchronization device, one end is connected to a first distributed service cluster, and one end is connected to a second distributed service cluster:

The full data synchronization device reads the data in the first distributed service cluster and compares it with the corresponding data in the second distributed service cluster. If the data does not exist in the second distributed service cluster or is not the latest data , Then write the data to the second distributed service cluster;

The full data synchronization device reads the data in the second distributed service cluster and compares it with the corresponding data in the first distributed service cluster. If the data does not exist in the first distributed service cluster or is not the latest data , The data is written into the first distributed service cluster.

In an embodiment, the data has the largest time stamp as the latest data.

In an embodiment, for the first distributed service cluster and/or the second distributed service cluster, the data is stored in units of distributed service nodes, and each distributed service node has a uniquely identifiable name.

In one embodiment, the full data synchronization device periodically traverses reading data from the first distributed service cluster and the second distributed service cluster, and performs data comparison, and the read data is based on each distributed service cluster. The storage is performed in units of service nodes.

In an embodiment, the distributed service node includes: a node name, a data value, and a timestamp, where the timestamp is a timestamp corresponding to the data value of the node.

In an embodiment, the full data synchronization device performs data comparison on the data of the same node name in the first distributed service cluster and the second distributed service cluster:

Determine whether the data values of the first distributed service cluster and the second distributed service cluster corresponding to the node are the same;

If they are the same, do nothing, if they are different, then further compare the timestamps;

Determine the size of the timestamp of the first distributed service cluster and the second distributed service cluster corresponding to the node;

If the timestamp of the first distributed service cluster is greater than the timestamp of the second distributed service cluster, the data value of the second distributed service cluster is overwritten with the data value of the first distributed service cluster;

If the timestamp of the second distributed service cluster is greater than the timestamp of the first distributed service cluster, the data value of the first distributed service cluster is overwritten with the data value of the second distributed service cluster.

In an embodiment, in the step of comparing the timestamps, if the timestamp of the first distributed service cluster is equal to the timestamp of the second distributed service cluster, a prompt is output and the operation is waited for input instructions.

In an embodiment, the data includes configuration files, memory data, and database data.

In an embodiment, the first distributed service cluster and/or the second distributed service cluster are composed of a leader server and several follower servers, and the user communicates with the first distributed service through the distributed service client. The cluster and/or the second distributed service cluster interact,

The follower server provides all read operations and returns the results to the distributed service client;

The leader server provides all write operations and replicates the written data to other follower servers in the cluster to update the state of the system.

In an embodiment, the first distributed service cluster and/or the second distributed service cluster stores all data in memory and files.

The present invention solves the problem of data inconsistency between two independent ZooKeeper clusters and improves the data reliability of ZooKeeper clusters. When one of the clusters is inaccessible, the same data can be obtained through the other cluster, which satisfies the requirements of the traditional rail transit industry. The reliability requirements for dual clusters under the A/B redundant network architecture.

Description of the drawings

The above and other features, properties and advantages of the present invention will become more apparent through the following description in conjunction with the drawings and embodiments. In the drawings, the same reference numerals always indicate the same features, among which:

Fig. 1 discloses a schematic diagram of the connection of a full data synchronization device according to an embodiment of the present invention;

FIG. 2 shows a schematic diagram of the working process of a full data synchronization device according to an embodiment of the present invention.

detailed description

In order to make the objectives, technical solutions, and advantages of the present invention clearer, the following further describes the present invention in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described here are only used to explain the invention and not to limit the invention.

The purpose of the present invention is to solve the problem of data inconsistency in two independent distributed service clusters in a dual-network environment by adopting a full data synchronization device, and to improve data reliability.

A distributed service cluster refers to a ZooKeeper cluster that uses ZooKeeper software. The ZooKeeper cluster in the dual-network environment refers to the use of two independent ZooKeeper clusters to realize the ATS system functions in the A network and the B network. The dual network environment is the redundant architecture of the A/B network.

Fig. 1 shows a schematic diagram of the connection of a full data synchronization device according to an embodiment of the present invention. In the embodiment shown in FIG. 1, the ZooKeeper cluster that implements the ATS system function of the A network is ZK cluster 1, and the ZooKeeper cluster that implements the ATS system function of the B network is ZK cluster 2.

In the embodiment shown in Figure 1, the ZooKeeper cluster is composed of a leader server and several follower servers. After users establish a session with the ZooKeeper cluster through the ZooKeeper client, they can interact with the ZooKeeper cluster through the client.

The follower server provides all read operations and returns the results to the client;

The leader server provides all write operations and copies the written data to other follower servers in the ZooKeeper cluster to update the state of the system.

When a ZooKeeper client in the ZooKeeper cluster modifies data, the leader server will synchronize the modified data to all servers in the ZooKeeper cluster, so that all clients on the ZooKeeper cluster can see The revised data.

When the leader server cannot be accessed due to a failure, all remaining follower servers begin to elect the leader. Through the election algorithm, a server that was originally a follower is finally upgraded to a leader. Once the original leader server is restored, it can only act as a follower server and compete for the leader's position in the next election.

The ZooKeeper cluster can synchronize data between the internal leader server and follower server. However, for two independent ZooKeeper clusters, the ZooKeeper underlying protocol does not support data synchronization between two independent ZooKeeper clusters.

The technical scheme adopted by the present invention is to establish a full data synchronization device between two independent ZooKeeper clusters to solve the data inconsistency of the two independent clusters and improve the reliability of the data.

In the embodiment shown in FIG. 1, the full data synchronization device 3 is designed for data consistency between ZK cluster 1 and ZK cluster 2 in a dual-network environment. One end of the full data synchronization device 3 is connected to ZK cluster 1, and the other end is connected to ZK cluster 2, to synchronize data between two ZooKeeper clusters in a dual-network environment.

In the embodiment shown in FIG. 1, the full data synchronization device 3 is a data comparison component, which implements the data synchronization function through the following steps:

Read the data in ZK cluster 1, the data includes configuration files, memory data, database data and other data, and compare it with the corresponding data in ZK cluster 2. If the data does not exist in ZK cluster 2 or is not the latest data , The data is written into ZK cluster 2, and the data corresponding to ZK cluster 2 is overwritten.

Read the data in ZK cluster 2. The data includes configuration files, memory data, database data and other data, and compare it with the corresponding data in ZK cluster 1. If the data does not exist in ZK cluster 1, it is not the latest data, Write data to ZK cluster 1, and the data corresponding to ZK cluster 1 is overwritten.

The data with the largest timestamp is the latest data, otherwise it will not be overwritten.

Optionally, the full data synchronization device is implemented in the form of hardware or software programs.

Both ZK cluster 1 and ZK cluster 2 are ZooKeeper clusters. The ZooKeeper cluster is a distributed small file system, and a single point of failure can be avoided through election algorithms and cluster replication.

The data of the distributed service cluster is stored in units of each distributed service node. The distributed service node includes: a node name, a data value, and a timestamp, where the timestamp is the timestamp corresponding to the data value of the node. In the embodiment shown in FIG. 1, the distributed service node refers to a ZooKeeper node.

The ZooKeeper cluster provides users with data storage services. The data storage is based on ZooKeeper nodes (znodes, data nodes). Each node has a name and can store data. These nodes form a file system-like namespace according to a hierarchical structure. .

Its data model is similar to a file system, and the hierarchical relationship before znodes is like the directory structure of a file system, but the ZooKeeper cluster stores all data in memory and files, and automatically sets response nodes to reduce elections Time-consuming, improve server throughput and reduce latency.

Because it is a file system, even if all ZooKeeper nodes are down, the data will not be lost. After restarting the server, the data can be restored.

In the embodiment shown in FIG. 1, the full data synchronization device 3 reads data from ZK cluster 1 and ZK cluster 2, and the data is compared with each ZooKeeper node as a unit. The data structure adopts the form of Key+Value.

The full data synchronization device 3 periodically reads the data of all ZooKeeper nodes in the ZK cluster 1 and ZK cluster 2 in the dual-network environment, and compares the data value of each node.

The data structure of the ZooKeeper node is Key=(Value, time), Key is the name of the node, the Key value of each node is different, Value is the data value of the node, including configuration files, memory data, database data and other data, time It is the timestamp corresponding to the Value data value of this node.

Figure 2 discloses a schematic diagram of the working process of a full data synchronization device according to an embodiment of the present invention. The following takes the data of the same node of the two clusters ZK cluster 1 and ZK cluster 2 as an example. With reference to Figure 2, the full data synchronization device is further explained 3 data synchronization work process.

Both ZK cluster 1 and ZK cluster 2 store node Key = (Value1, time1), Key = (Value2, time2) data respectively, where Key is the name of the node, Value1 is the data value of the node in ZK cluster 1, and time1 is The timestamp of this node in ZK cluster 1, Value2 is the data value of this node in ZK cluster 2, and time2 is the timestamp of this node in ZK cluster 2. The key must be the same, which is guaranteed by the application layer.

First, determine the values of Value1 and Value2.

If Value1=Value2, that is, Value1 and Value2 are the same, the full data synchronization device will not do any processing;

If Value1! = Value2, that is, Value1 and Value2 are different, and the latest data of the time stamp needs to be rewritten;

Further, determine the values of time1 and time2.

If time1>time2, the Value1 data of ZK cluster 1 is the latest reliable data, the full data synchronization device starts the write function, and rewrites the data value of ZK cluster 2, and Value2 is overwritten with the value of Value1;

If time1<time2, the Value2 data of ZK cluster 2 is the latest reliable data, and the full data synchronization device starts the write function, rewrites the data value of ZK cluster 1, and Value1 is overwritten with the value of Value2.

If time1=time2, output a prompt and wait for the input command to operate. Optionally, the instruction input operation is a manual intervention operation.

Since time is in milliseconds and the two clusters share the same clock source, it is almost impossible that the value is different but the time is the same. If it does happen, you can choose to output a prompt for manual processing.

The present invention solves the problem of data inconsistency between two independent ZooKeeper clusters and improves the data reliability of ZooKeeper clusters. When one of the clusters is inaccessible, the same data can be obtained through the other cluster, which satisfies the requirements of the traditional rail transit industry. Reliability requirements for dual clusters under A/B network redundancy architecture.

Although the above methods are illustrated and described as a series of actions to simplify the explanation, it should be understood and appreciated that these methods are not limited by the order of the actions, because according to one or more embodiments, some actions may occur in a different order And/or occur concurrently with other actions from what is illustrated and described herein or that are not illustrated and described herein but can be understood by those skilled in the art.

Those skilled in the art will further appreciate that the various illustrative logic blocks, modules, circuits, and algorithm steps described in conjunction with the embodiments disclosed herein can be implemented as electronic hardware, computer software, or a combination of the two. In order to clearly explain the interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps are generally described above in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the overall system. Technicians can implement the described functionality in different ways for each specific application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.

Various illustrative logic blocks, modules, and circuits described in conjunction with the embodiments disclosed herein can be used with general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA), or other A programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein are implemented or executed. A general-purpose processor may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors cooperating with a DSP core, or any other such configuration.

As shown in the present application and claims, unless the context clearly suggests exceptional circumstances, the words "a", "an", "an" and/or "the" do not specifically refer to the singular, but may also include the plural. Generally speaking, the terms "including" and "including" only suggest that the clearly identified steps and elements are included, and these steps and elements do not constitute an exclusive list, and the method or device may also include other steps or elements.

The above-mentioned embodiments are provided for those skilled in the art to implement or use the present invention. Those skilled in the art can make various modifications or changes to the above-mentioned embodiments without departing from the inventive idea of the present invention. The protection scope of the present invention is not limited by the above-mentioned embodiments, but should be the maximum scope that meets the innovative features mentioned in the claims.

Claims

A full data synchronization device, characterized in that one end is connected to a first distributed service cluster, and one end is connected to a second distributed service cluster:

The full data synchronization device reads the data in the first distributed service cluster and compares it with the corresponding data in the second distributed service cluster. If the data does not exist in the second distributed service cluster or is not the latest data , Then write the data to the second distributed service cluster;

The full data synchronization device reads the data in the second distributed service cluster and compares it with the corresponding data in the first distributed service cluster. If the data does not exist in the first distributed service cluster or is not the latest data , The data is written into the first distributed service cluster.
The device for synchronizing full data according to claim 1, wherein the data has a maximum time stamp as the latest data.
The full data synchronization device according to claim 1, wherein the first distributed service cluster and/or the second distributed service cluster are stored in units of distributed service nodes, and each distributed service node Have a uniquely identifiable name.
The full data synchronization device according to claim 1, wherein the full data synchronization device periodically traverses reading data from the first distributed service cluster and the second distributed service cluster, and performs data comparison, The read data is stored in units of each distributed service node.
The full data synchronization device according to claim 4, wherein the distributed service node comprises: a node name, a data value and a time stamp, wherein the time stamp is a time stamp corresponding to the data value of the node.
The full data synchronization device according to claim 5, wherein the full data synchronization device performs data comparison on the data of the same node name in the first distributed service cluster and the second distributed service cluster:

Determine whether the data values of the first distributed service cluster and the second distributed service cluster corresponding to the node are the same;

If they are the same, do nothing, if they are different, then further compare the timestamps;

Determine the size of the timestamp of the first distributed service cluster and the second distributed service cluster corresponding to the node;

If the timestamp of the first distributed service cluster is greater than the timestamp of the second distributed service cluster, the data value of the second distributed service cluster is overwritten with the data value of the first distributed service cluster;

If the timestamp of the second distributed service cluster is greater than the timestamp of the first distributed service cluster, the data value of the first distributed service cluster is overwritten with the data value of the second distributed service cluster.
The full data synchronization device according to claim 6, characterized in that, in the step of comparing the timestamps, if the timestamp of the first distributed service cluster is equal to the timestamp of the second distributed service cluster, output a prompt and wait Enter instructions to operate.
The full data synchronization device according to claim 1, wherein the data includes configuration files, memory data, and database data.
The full data synchronization device according to claim 1, wherein the first distributed service cluster and/or the second distributed service cluster is composed of a leader server and several follower servers, and users are distributed Service client interacts with the first distributed service cluster and/or the second distributed service cluster,

The follower server provides all read operations and returns the results to the distributed service client;

The leader server provides all write operations and replicates the written data to other follower servers in the cluster to update the state of the system.
The full data synchronization device according to claim 1, wherein the first distributed service cluster and/or the second distributed service cluster stores full data in memory and files.