CN112055054B - Multi-edge cluster data synchronization method and system based on multiple consistency protocols - Google Patents
Multi-edge cluster data synchronization method and system based on multiple consistency protocols Download PDFInfo
- Publication number
- CN112055054B CN112055054B CN202010788464.9A CN202010788464A CN112055054B CN 112055054 B CN112055054 B CN 112055054B CN 202010788464 A CN202010788464 A CN 202010788464A CN 112055054 B CN112055054 B CN 112055054B
- Authority
- CN
- China
- Prior art keywords
- data synchronization
- edge
- management
- edge cluster
- cluster
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1095—Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Abstract
The invention discloses a multi-edge cluster data synchronization method and a system based on multiple consistency protocols, wherein each edge cluster comprises at least one management node and at least one working node, and the method comprises the following steps: data synchronization is realized among management nodes in the edge cluster by adopting a strong consistency protocol; data synchronization between a management node and a working node in the edge cluster adopts a local area network weak consistency protocol; and the data synchronization among the management nodes of different edge clusters adopts a wide area network weak consistency protocol. The invention respectively uses different consistency protocols aiming at three ranges among the same-edge cluster management nodes, between the same-edge cluster management nodes and the working nodes and between different-edge cluster management nodes, thereby realizing high-efficiency data synchronization in and among the edge clusters of the edge multi-edge cluster, reducing delay and enhancing stability.
Description
Technical Field
The invention relates to the technical field of edge computing, in particular to a multi-edge cluster data synchronization method and system based on multiple consistency protocols.
Background
With the emergence of new services such as AR/VR, high-definition video, automatic driving and the like, the network is facing new challenges such as real-time computing capability, ultra-low time delay, ultra-large bandwidth and the like. The development of the edge computing industry is promoted, and a healthy ecological environment is constructed, so that the terminal user can obtain extreme experience brought by new services, and the terminal user can use the service more abundantly and safely and reliably. In recent years, various parties of industry chains including mobile operators, network equipment providers, application developers and content providers accelerate the progress of edge computing, and the technology is rapidly developed. Edge computing is derived from technical practices and commercial practices, and not only is an emerging technology and a deployment mode, but also is more important to realize the bottom layer opening of a network, so that the deep fusion of a mobile communication network, the internet and the internet of things is promoted.
The data synchronization mechanism of the traditional cloud computing data center is only suitable for managing a single or a small number of large-scale clusters, and cannot well perform data synchronization on small-scale edge clusters with dispersed layouts and huge numbers.
Disclosure of Invention
The embodiment of the invention aims to provide a multi-edge cluster data synchronization method and system based on multiple consistency protocols, so as to solve the problem that a cluster data synchronization mechanism of a traditional cloud computing data center is only suitable for a single or a small number of large-scale edge clusters and cannot efficiently synchronize data of small-scale edge clusters with scattered layouts and large numbers.
In order to achieve the above purpose, the technical solution adopted by the embodiment of the present invention is as follows:
in a first aspect, an embodiment of the present invention provides a method for data synchronization of multiple edge clusters based on multiple coherence protocols, where each edge cluster includes at least one management node and at least one working node, and the method includes:
data synchronization is realized among management nodes in the edge cluster by adopting a strong consistency protocol;
data synchronization between a management node and a working node in the edge cluster adopts a local area network weak consistency protocol;
and the data synchronization among the management nodes of different edge clusters adopts a wide area network weak consistency protocol.
Further, when the data synchronization is realized between the management nodes in the edge cluster by adopting a strong consistency protocol, the data synchronization is successful only when more than half of the management node data in the edge cluster are consistent, otherwise, the data synchronization is failed.
Further, the number of the management nodes is an odd number.
Further, data synchronization between the same edge cluster management node and the working node is realized by an infectious data synchronization method through a local area network weak consistency protocol.
Furthermore, data synchronization of management nodes among different edge clusters is realized by a wide area network weak consistency protocol in an infection type method, the wide area network weak consistency protocol does not block and tolerate delay, and requests among different edge clusters are allowed to be forwarded.
In a second aspect, an embodiment of the present invention further provides a system for synchronizing data of multiple edge clusters based on multiple coherence protocols, where the system includes: the edge cluster comprises at least one management node and at least one working node, and data synchronization is realized among the management nodes in the edge cluster by adopting a strong consistency protocol; the data synchronization between the management node and the working node in the edge cluster adopts a local area network weak consistency protocol; and the data synchronization among the management nodes of different edge clusters adopts a wide area network weak consistency protocol.
According to the technical scheme, the invention has the beneficial effects that: the invention realizes data synchronization between the management nodes in the edge cluster by adopting a strong consistency protocol; data synchronization between a management node and a working node in the edge cluster adopts a local area network weak consistency protocol; the data synchronization between the management nodes of different edge clusters adopts a wide area network weak consistency protocol, so that the data synchronization can be efficiently carried out on the scattered and numerous edge clusters, the data synchronization in the edge clusters and among the edge clusters can not be blocked while the stable operation of the management nodes in the edge clusters is ensured, and the request forwarding can be carried out before different edge clusters, thereby realizing the unified management of the edge multi-edge clusters.
Drawings
The accompanying drawings, which are included to provide a further understanding of the invention and are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and together with the description serve to explain the invention and not to limit the invention. In the drawings:
FIG. 1 is a schematic diagram of strong consistency data synchronization and election between edge and edge cluster management nodes in an embodiment of the present invention;
FIG. 2 is a schematic diagram of local area network weakly consistent data synchronization and member management between edge same-edge cluster management nodes and working nodes in the embodiment of the present invention;
fig. 3 is a schematic diagram of synchronization and request forwarding of weakly consistent data of a wide area network between different edge cluster management nodes in the embodiment of the present invention.
Detailed Description
To make the objects, technical solutions and advantages of the present application more clear, the technical solutions of the present application will be clearly and completely described below with reference to specific embodiments of the present application and the accompanying drawings. It should be apparent that the described embodiments are only a few embodiments of the present application, and not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
Referring to fig. 1 to fig. 3, an embodiment of the present invention provides a method for data synchronization of multiple edge clusters based on multiple coherence protocols, where each edge cluster includes at least one management node and at least one working node, and the method includes:
data synchronization is realized among management nodes in the edge cluster by adopting a strong consistency protocol;
data synchronization between a management node and a working node in the edge cluster adopts a local area network weak consistency protocol;
and the data synchronization among the management nodes of different edge clusters adopts a wide area network weak consistency protocol.
The management node and the working node can be a physical machine or a virtual machine, the management node is a control node of the edge cluster, and at least one management node in each edge cluster is responsible for the management and control of the whole edge cluster; the worker node is responsible for executing the workload from the management node.
In this embodiment, when the system is constructed, a plurality of edge clusters are first constructed using nodes, the edge clusters are usually located in different geographical areas, each edge cluster has at least one management node and a plurality of working nodes, and all the nodes constituting the edge clusters are required to conform to the authentication and authorization manner of the edge clusters. In order to ensure high availability of management nodes, a plurality of management nodes (generally an odd number, e.g., 1,3,5,7 \ 8230;) are deployed in an edge cluster to manage the edge cluster, where one of the management nodes is a primary management node and the others are backup management nodes, and the primary management node is generated by election of all the management nodes in the edge cluster through a Raft protocol.
Data synchronization within and among edge clusters of a plurality of edge clusters is divided into three levels: the data synchronization among the same edge cluster management nodes, the data synchronization between the same edge cluster management nodes and the working nodes and the data synchronization among different edge cluster management nodes.
A first layer: as shown in fig. 1, the spatial distance between the edge cluster management nodes is short, the network quality is good, the number of the management nodes is limited, and a strong consistency protocol is adopted to perform data synchronization: when more than half of the management nodes of the same edge cluster are consistent, the data synchronization is considered to be completed, the edge cluster can carry out the next work, otherwise, the edge cluster is always blocked. The method comprises the following specific steps:
before starting, all management node IP addresses and port information in the same edge cluster are determined, when each management node is started, all management node information is transmitted in through command line parameters or configuration files, after all management nodes are started, a main management node is selected to manage the edge cluster, and other management nodes are used as standby management nodes.
And (2) layer two: as shown in fig. 2, a local area network weak consistency protocol is used between the management node and the working node in the same edge cluster, and data synchronization is achieved through an "infection type" method, that is, a data change node selects a plurality of adjacent nodes for data synchronization, and then other work can be performed without blocking and waiting for all nodes to complete data synchronization. The method comprises the following specific steps:
(1) Setting working nodes of the same edge cluster according to the actual environment, and starting;
(2) When the IP address and the port of any management node are known, the management node can be added to form an edge cluster through a command line on a host of a working node.
And (3) layer three: as shown in fig. 3, a wide area network weak consistency protocol is used between different edge cluster management nodes, and data synchronization is achieved by an "infection type" method. The edge cluster needing data synchronization firstly randomly informs adjacent edge clusters of data synchronization, and other tasks can be executed without blocking after the notification is completed. The wide area network weak consistency protocol has higher tolerance to network delay, is suitable for the environment with large spatial distance of edge clusters, uncontrollable network quality and high network delay, and simultaneously supports the forwarding of access requests among different edge clusters. The method comprises the following specific steps:
and when the IP address and the port of any management node of other edge clusters are known among the management nodes of different edge clusters, the other edge cluster management nodes can be added through a command line to form a multi-edge cluster.
The embodiment of the invention also provides a system for synchronizing the data of the multi-edge cluster based on multiple consistency protocols, which comprises the following steps: the system comprises a plurality of edge clusters, a plurality of data processing units and a plurality of data transmission units, wherein each edge cluster comprises at least one management node and at least one working node, and data synchronization is realized among the management nodes in the edge clusters by adopting a strong consistency protocol; data synchronization between a management node and a working node in the edge cluster adopts a local area network weak consistency protocol; and the data synchronization among the management nodes of different edge clusters adopts a wide area network weak consistency protocol.
The above description is only for the purpose of illustrating the preferred embodiments of the present invention and should not be taken as limiting the scope of the present invention, which is intended to cover any modifications, equivalents, improvements, etc. within the spirit and scope of the present invention.
Claims (10)
1. A method for multi-edge cluster data synchronization based on multiple coherence protocols, wherein each edge cluster comprises at least one management node and at least one working node, the method comprising:
the data synchronization is realized between the management nodes in the edge cluster by adopting a strong consistency protocol;
data synchronization between a management node and a working node in the edge cluster adopts a local area network weak consistency protocol;
and the data synchronization among the management nodes of different edge clusters adopts a wide area network weak consistency protocol.
2. The method of claim 1, wherein when a strong consistency protocol is used between management nodes in the edge cluster to achieve data synchronization, the data synchronization is successful only when more than half of the management nodes in the edge cluster achieve consistency, otherwise, the data synchronization is failed.
3. The method of claim 1, wherein the number of management nodes is an odd number.
4. The method of claim 1, wherein the data synchronization between the edge-shared cluster management node and the working node is achieved by an epidemic data synchronization method through a local area network weak consistency protocol.
5. The method of claim 1, wherein the data synchronization between different edge clusters is achieved in an epidemic manner through a weak consistency protocol of wide area network, which does not block and tolerate delay and allows request forwarding between different edge clusters.
6. A system for multi-edge cluster data synchronization based on multiple coherence protocols, the system comprising: the system comprises a plurality of edge clusters, a plurality of data processing units and a plurality of data transmission units, wherein each edge cluster comprises at least one management node and at least one working node, and data synchronization is realized among the management nodes in the edge clusters by adopting a strong consistency protocol; data synchronization between a management node and a working node in the edge cluster adopts a local area network weak consistency protocol; and the data synchronization among the management nodes of different edge clusters adopts a wide area network weak consistency protocol.
7. The system of claim 6, wherein the number of management nodes is an odd number.
8. The system of claim 6, wherein when the data synchronization is implemented by using a strong consistency protocol between the management nodes in the edge cluster, the data synchronization is successful only when more than half of the management nodes in the edge cluster have reached consistency, otherwise the data synchronization is failed.
9. The system according to claim 6, wherein the data synchronization between the edge cluster management node and the working node is realized by an epidemic data synchronization method through a local area network weak consistency protocol.
10. The system of claim 6, wherein the data synchronization between different edge clusters is achieved in an epidemic manner through a weak consistency protocol of wide area network, which is non-blocking, delay tolerant and allows request forwarding between different edge clusters.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010788464.9A CN112055054B (en) | 2020-08-07 | 2020-08-07 | Multi-edge cluster data synchronization method and system based on multiple consistency protocols |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010788464.9A CN112055054B (en) | 2020-08-07 | 2020-08-07 | Multi-edge cluster data synchronization method and system based on multiple consistency protocols |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112055054A CN112055054A (en) | 2020-12-08 |
| CN112055054B true CN112055054B (en) | 2023-04-07 |
Family
ID=73601279
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010788464.9A Active CN112055054B (en) | 2020-08-07 | 2020-08-07 | Multi-edge cluster data synchronization method and system based on multiple consistency protocols |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112055054B (en) |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102436366A (en) * | 2011-08-22 | 2012-05-02 | 华为技术有限公司 | Data synchronization method and apparatus |
| CN102592420A (en) * | 2011-12-30 | 2012-07-18 | 北京理工大学 | Wireless sensor network testing node with microdrive and data synchronizing capacity |
| CN102681937A (en) * | 2012-05-15 | 2012-09-19 | 浪潮电子信息产业股份有限公司 | Correctness verifying method of cache consistency protocol |
| CN104360981A (en) * | 2014-11-12 | 2015-02-18 | 浪潮(北京)电子信息产业有限公司 | Design method of multi-core multiprocessor platform orientated Cache consistency protocol |
| CN104915756A (en) * | 2015-05-22 | 2015-09-16 | 电信科学技术第五研究所 | Data consistency cloud auditing system and implementation method |
| CN106254413A (en) * | 2015-06-09 | 2016-12-21 | 哈曼国际工业有限公司 | Data syn-chronization |
| CN109271433A (en) * | 2018-09-03 | 2019-01-25 | 中新网络信息安全股份有限公司 | A kind of company-data synchronous method |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8660132B2 (en) * | 2010-06-28 | 2014-02-25 | Avaya Inc. | Control plane packet processing and latency control |
| US8898728B2 (en) * | 2011-09-23 | 2014-11-25 | Oracle International Corporation | System and method of real-time change propagation and activation using a distributed object cache |
-
2020
- 2020-08-07 CN CN202010788464.9A patent/CN112055054B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102436366A (en) * | 2011-08-22 | 2012-05-02 | 华为技术有限公司 | Data synchronization method and apparatus |
| CN102592420A (en) * | 2011-12-30 | 2012-07-18 | 北京理工大学 | Wireless sensor network testing node with microdrive and data synchronizing capacity |
| CN102681937A (en) * | 2012-05-15 | 2012-09-19 | 浪潮电子信息产业股份有限公司 | Correctness verifying method of cache consistency protocol |
| CN104360981A (en) * | 2014-11-12 | 2015-02-18 | 浪潮(北京)电子信息产业有限公司 | Design method of multi-core multiprocessor platform orientated Cache consistency protocol |
| CN104915756A (en) * | 2015-05-22 | 2015-09-16 | 电信科学技术第五研究所 | Data consistency cloud auditing system and implementation method |
| CN106254413A (en) * | 2015-06-09 | 2016-12-21 | 哈曼国际工业有限公司 | Data syn-chronization |
| CN109271433A (en) * | 2018-09-03 | 2019-01-25 | 中新网络信息安全股份有限公司 | A kind of company-data synchronous method |
Non-Patent Citations (2)
| Title |
|---|
| "基于RDMA和NVM的大数据系统一致性协议研究";吴昊，陈康，武永卫，郑纬民;《大数据》;20190531;全文 * |
| "消息中间件元数据管理模块及发布订阅接口的设计与实现";姚思明;《中国优秀硕士学位论文全文数据库 (信息科技辑)》;20171231;全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112055054A (en) | 2020-12-08 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| EP2993838B1 (en) | Methods for setting a member identity of gateway device and corresponding management gateway devices | |
| Yang et al. | Blockchain-based secure distributed control for software defined optical networking | |
| WO2017036288A1 (en) | Network element upgrading method and device | |
| CN106656846B (en) | The construction method of cooperation layer in a kind of SDN architectural framework | |
| WO2019101020A1 (en) | Multi-terminal collaborative working method, terminal device and multi-terminal collaborative system | |
| CN111638957B (en) | Method for realizing cluster sharing type public cloud load balance | |
| CN103118148B (en) | A kind of ARP buffering updating method and equipment | |
| US9538440B2 (en) | Method and system of changing service points | |
| CN112636989B (en) | Method and device for federated learning communication | |
| WO2018121201A1 (en) | Distributed cluster service structure, node cooperation method and device, terminal and medium | |
| CN103631652A (en) | Method and system for achieving virtual machine migration | |
| CN111083177B (en) | Cross-domain collaborative interaction method based on collaborative gateway | |
| CN115955456A (en) | IPv 6-based enterprise campus network and networking method | |
| KR100742357B1 (en) | Intelligent distributed server system and method for employing as the same | |
| CN117296304A (en) | Managing data traffic for two simultaneous VPN connections using VPN clients | |
| CN107483628B (en) | DPDK-based one-way proxy method and system | |
| CN112055054B (en) | Multi-edge cluster data synchronization method and system based on multiple consistency protocols | |
| CN104601412A (en) | Independent test network system and establishing method thereof | |
| CN108200221A (en) | Rule synchronization method and device are converted in a kind of network address translation environment | |
| WO2023185428A1 (en) | Method for reconstructing key service capability of user plane function network element, and communication apparatus | |
| CN111880932A (en) | Data storage method and device based on multiple network ports | |
| CN112217680A (en) | Controller capability benchmark test method and device based on software-defined wide area network | |
| CN107968825B (en) | Message forwarding control method and device | |
| US20190036793A1 (en) | Network service implementation method, service controller, and communications system | |
| CN113630481B (en) | Automatic in-band control plane construction method and system in SDN |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |