CN110784532B - Bidirectional data synchronization method and system - Google Patents
Bidirectional data synchronization method and system Download PDFInfo
- Publication number
- CN110784532B CN110784532B CN201911022320.6A CN201911022320A CN110784532B CN 110784532 B CN110784532 B CN 110784532B CN 201911022320 A CN201911022320 A CN 201911022320A CN 110784532 B CN110784532 B CN 110784532B
- Authority
- CN
- China
- Prior art keywords
- data
- cloud platform
- instruction
- updating instruction
- bidirectional
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1095—Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/2365—Ensuring data consistency and integrity
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/54—Indexing scheme relating to G06F9/54
- G06F2209/548—Queue
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Databases & Information Systems (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Signal Processing (AREA)
- Computer Networks & Wireless Communication (AREA)
- Software Systems (AREA)
- Computing Systems (AREA)
- Computer Security & Cryptography (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a bidirectional data synchronization method and a bidirectional data synchronization system. The method comprises the following steps: acquiring a data updating instruction of a data source cloud platform; performing transaction processing on the data updating instruction; and synchronizing the data updating instruction subjected to the transaction processing to the cloud platform of the data destination through the interface service provided by the cloud platform of the data destination. The bidirectional data synchronization method and system provided by the invention can realize data decoupling between different cloud platforms and realize high availability of data across the cloud platforms.
Description
Technical Field
The invention relates to the technical field of cloud computing, in particular to a bidirectional data synchronization method and system.
Background
At present, in order to ensure high availability of application services, the application service capacity is dynamically allocated, and the enterprise market competitiveness is improved. Most enterprises choose to interface a plurality of cloud resource providers and deploy applications on cloud resources of different service providers. This ensures that "not all eggs are placed in one basket". However, there is no association between cloud resource servers. When one cloud resource fails, the application of the current platform is in an unavailable state, the normal operation of the service is affected, and huge loss is caused to enterprises.
The existing technical scheme is to ensure the data synchronism of the cross-cloud application by ensuring the public property of the database. This approach is the simplest and less costly way to achieve data synchronicity across cloud applications. The method ensures the universality of the database, namely, the storage service of the data is provided through a local machine room, the data is accessed to the cloud resources through a private line, and the same data source is deployed for the application access of different cloud service providers.
As shown in fig. 1, a solution for database public dual cloud data synchronization. As can be seen from fig. 1, this solution has the following disadvantages:
1. the data is highly coupled. If the local machine room and the cloud platform are linked into the backup barrier, the special line is interrupted.
Both platforms will lose service capability.
2. The scalability is poor. A single platform change may affect the use of another platform.
3. The resource utilization degree is low. The cloud resource platform has abundant ecosphere, and the existing cloud resources are not fully utilized due to single structure.
Disclosure of Invention
The technical problem to be solved by the invention is to provide a bidirectional data synchronization method and system, which can realize data decoupling between different cloud platforms and realize high availability of data across the cloud platforms.
In order to solve the above technical problem, the present invention provides a bidirectional data synchronization method, including: acquiring a data updating instruction of a data source cloud platform; performing transaction processing on the data updating instruction; and synchronizing the data updating instruction subjected to the transaction processing to the cloud platform of the data destination through the interface service provided by the cloud platform of the data destination.
In some embodiments, the data update instructions include: create instruction, delete instruction, update instruction, insert instruction.
In some embodiments, obtaining data update instructions of a data source cloud platform includes: and acquiring a data updating instruction of the data source cloud platform through the message queue of the data source cloud platform.
In some embodiments, the message queue comprises: redis message queues.
In some embodiments, the Redis message queue stores execution data for data update instructions that have been executed in a database of the data source cloud platform.
In some embodiments, the execution data of the data update instruction is stored item by item according to a time sequence of execution.
In some embodiments, transacting the data update instruction includes: analyzing the data updating instruction to obtain key parameters in the data updating instruction; and performing transaction recording on the data updating instruction according to the key parameters.
In some embodiments, synchronizing the data update instruction subjected to the transaction processing to the data destination cloud platform through an interface service provided by the data destination cloud platform includes: and synchronizing the data updating instruction which is not subjected to the synchronization processing to the cloud platform of the data destination according to the transaction record.
In addition, the present invention also provides a bidirectional data synchronization system, which includes: one or more processors; storage means for storing one or more programs which, when executed by the one or more processors, cause the one or more processors to implement the bidirectional data synchronization method according to the foregoing.
After adopting such design, the invention has at least the following advantages:
the invention not only ensures the independence of the applications on different cloud platforms, but also has the influence of zero change on the data structure of the application of the existing platform, and the only thing to do is the development and maintenance of the application of the migration module, thereby greatly reducing the cost of realizing the cross-cloud high availability of enterprises.
Drawings
The foregoing is only an overview of the technical solutions of the present invention, and in order to make the technical solutions of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and the detailed description.
FIG. 1 is a network architecture diagram of database commonality provided by the prior art;
fig. 2 is a diagram of an application network structure of a bidirectional data synchronization method according to an embodiment of the present invention;
FIG. 3 is a flow chart of a bidirectional data synchronization method provided by an embodiment of the invention;
fig. 4 is a structural diagram of a bidirectional data synchronization system according to an embodiment of the present invention.
Detailed Description
The preferred embodiments of the present invention will be described in conjunction with the accompanying drawings, and it will be understood that they are described herein for the purpose of illustration and explanation and not limitation.
Fig. 2 illustrates a network environment to which the bidirectional data synchronization method provided by the embodiment of the present invention is applied. Referring to fig. 2, in the two cloud platforms involved in the bidirectional data synchronization method, in addition to the respective deployed application services, respective background database systems and message queues are also respectively provided.
The background database system of each cloud platform is used for providing background data support for cloud services provided by the cloud platform. For example, each key data table and index required to be used in the cloud service providing process can be provided by a background database system.
And the respective background database systems of the cloud platforms are operated independently. Therefore, when the two cloud platforms are used as the master cloud platform and the slave cloud platform respectively, and the availability of the cloud service is ensured, the problem of data asynchronism exists between the background database system of the master cloud platform and the background database system of the slave cloud platform.
It can be understood that, in the process of respective independent operation, the background database of the main cloud platform and the background database of the standby cloud platform respectively execute respective data update instructions. A data update instruction as used herein refers to an instruction that changes the storage structure and storage contents of a database. Such instructions may be: create instruction, delete instruction, update instruction, insert instruction.
The message queue is used for storing the execution data of the data updating instruction executed in the database of the data source cloud platform. In other words, on each cloud platform, as long as the background database executes the data updating instruction once, a corresponding message is stored in the message queue belonging to the cloud platform. Typically, the message queue is referred to as a Redis message queue.
It should be noted that, in the cloud platform, the database and the message queue are both deployed in the cloud end.
In addition, in order to realize data synchronization between the two cloud platforms, a migration module is deployed in a local computer room. And the migration module executes data synchronization to the cloud platform on the other side according to the messages stored in the message queue on each cloud platform. At this time, the cloud platform as the message source is the data source cloud platform, and the other cloud platform is the data destination cloud platform. And, the migration module can realize bidirectional synchronization between the two cloud platforms. That is, the data source cloud platform may be a primary cloud platform or a backup cloud platform.
Fig. 3 shows a flow of a bidirectional data synchronization method provided by an embodiment of the present invention. Since the bidirectional data synchronization method is executed by a migration module deployed locally in the embodiment of the present invention, the flow illustrated in fig. 3 is the flow executed by the migration module. Referring to fig. 3, the bidirectional data synchronization method includes:
and S31, acquiring a data updating instruction of the data source cloud platform.
And S32, performing transaction processing on the data updating instruction.
And S33, synchronizing the data updating instruction subjected to the transaction processing to the cloud platform of the data destination through the interface service provided by the cloud platform of the data destination.
In the execution flow of the bidirectional data synchronization method shown in fig. 3, the transaction processing executed on the data update instruction mainly refers to the transaction record of the operation executed on the instruction. More specifically, the transaction processing includes: analyzing the data updating instruction to obtain key parameters in the data updating instruction; and performing transaction recording on the data updating instruction according to the key parameters.
Because the transaction record is carried out on the instruction execution, the data updating instruction subjected to the transaction processing is synchronized to the cloud platform of the data destination through the interface service provided by the cloud platform of the data destination, and the method comprises the following steps: and synchronizing the data updating instruction which is not subjected to the synchronization processing to the cloud platform of the data destination according to the transaction record.
The current dual-cloud data synchronization has great expansibility, such as: the multi-cloud synchronization can be achieved, and the synchronization is limited between two cloud platforms. The migration module can carry out continuous development and maintenance, and ensures high reliability of data.
Fig. 4 is a block diagram of the bi-directional data synchronization system of the present invention. Referring to fig. 4, the bidirectional data synchronization system includes: a Central Processing Unit (CPU)401 that can perform various appropriate actions and processes according to a program stored in a Read Only Memory (ROM) or a program loaded from a storage section 408 into a Random Access Memory (RAM) 403. In the RAM 403, various programs and data necessary for system operation are also stored. The CPU 401, ROM 402, and RAM 403 are connected to each other via a bus 404. An input/output (I/O) interface 405 is also connected to bus 404.
The following components are connected to the I/O interface 405: an input section 406 including a keyboard, a mouse, and the like; an output section 407 including a display device such as a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and the like, and a speaker; a storage section 408 including a hard disk and the like; and a communication section 409 including a network interface card such as a LAN card, a modem, or the like. The communication section 409 performs communication processing via a network such as the internet. A driver 410 is also connected to the I/O interface 405 as needed. A removable medium 411 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted on the drive 410 as necessary, so that a computer program read out therefrom is mounted into the storage section 408 as necessary.
The above description is only for the purpose of illustrating the preferred embodiments of the present invention and is not to be construed as limiting the present invention in any way, and it will be apparent to those skilled in the art that the above description of the present invention can be applied to various modifications, equivalent variations or modifications without departing from the spirit and scope of the present invention.
Claims (7)
1. A bi-directional data synchronization method, comprising:
acquiring a data updating instruction of a data source cloud platform;
performing transaction processing on the data updating instruction, wherein the transaction processing comprises the following steps: analyzing the data updating instruction to obtain key parameters in the data updating instruction; according to the key parameters, transaction recording is carried out on the data updating instruction;
synchronizing the data updating instruction subjected to the transaction processing to the cloud platform of the data destination through an interface service provided by the cloud platform of the data destination;
through interface service provided by a data destination cloud platform, a data update instruction subjected to transaction processing is synchronized to the data destination cloud platform, and the method comprises the following steps:
according to the transaction record, synchronizing the data updating instruction which is not subjected to synchronous processing to a cloud platform of a data destination;
the migration module is deployed in a local computer room, data synchronization is performed on the cloud platform on the other side according to the messages stored in the message queue on each cloud platform, the migration module can achieve bidirectional synchronization between the two cloud platforms, and the data source cloud platform can be a main cloud platform or a standby cloud platform.
2. The bidirectional data synchronization method of claim 1, wherein the data update instruction comprises: create instruction, delete instruction, update instruction, insert instruction.
3. The bidirectional data synchronization method of claim 1, wherein obtaining the data update instruction of the data source cloud platform comprises:
and acquiring a data updating instruction of the data source cloud platform through the message queue of the data source cloud platform.
4. A method for bi-directional data synchronization according to claim 3 wherein said message queue comprises: redis message queues.
5. The bidirectional data synchronization method of claim 4, wherein the Redis message queue stores therein execution data of data update instructions that have been executed in a database of the data source cloud platform.
6. The bi-directional data synchronization method of claim 5, wherein the execution data of the data update instruction is stored in a row by row according to a time sequence of execution.
7. A bi-directional data synchronization system, comprising:
one or more processors;
a storage device for storing one or more programs,
when executed by the one or more processors, cause the one or more processors to implement the bidirectional data synchronization method of any of claims 1-6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911022320.6A CN110784532B (en) | 2019-10-25 | 2019-10-25 | Bidirectional data synchronization method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911022320.6A CN110784532B (en) | 2019-10-25 | 2019-10-25 | Bidirectional data synchronization method and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110784532A CN110784532A (en) | 2020-02-11 |
| CN110784532B true CN110784532B (en) | 2021-09-07 |
Family
ID=69387764
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911022320.6A Active CN110784532B (en) | 2019-10-25 | 2019-10-25 | Bidirectional data synchronization method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110784532B (en) |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102629268A (en) * | 2012-03-09 | 2012-08-08 | 华为技术有限公司 | Data synchronization method, system and date access device |
| US8725684B1 (en) * | 2011-03-31 | 2014-05-13 | Amazon Technologies, Inc. | Synchronizing data stores |
| CN103793514A (en) * | 2014-02-11 | 2014-05-14 | 华为技术有限公司 | Database synchronization method and databases |
| CN106202365A (en) * | 2016-07-07 | 2016-12-07 | 帅斌鹏 | Method, system and the data-base cluster that database update synchronizes |
| CN109634978A (en) * | 2018-12-18 | 2019-04-16 | 钛马信息网络技术有限公司 | Reading data consistency system and method |
| CN109885617A (en) * | 2019-01-29 | 2019-06-14 | 中国工商银行股份有限公司 | The method of data synchronization and device of Distributed Heterogeneous Database system |
| CN109933630A (en) * | 2019-03-19 | 2019-06-25 | 武汉达梦数据库有限公司 | Database data real-time synchronization method and equipment |
| CN109947801A (en) * | 2019-02-25 | 2019-06-28 | 交通银行股份有限公司 | Database in phase system, method and device |
Family Cites Families (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1536791A (en) * | 2003-04-04 | 2004-10-13 | 北京圣世祺科技开发有限公司 | Automation of launching pad and remote monitoring system of technical equipment |
| CN102693324B (en) * | 2012-01-09 | 2015-03-18 | 西安电子科技大学 | Distributed database synchronization system, synchronization method and node management method |
| CN103631907B (en) * | 2013-11-26 | 2016-09-07 | 中国科学院信息工程研究所 | A kind of method and system that relational data is migrated to HBase |
| CN104268234B (en) * | 2014-09-26 | 2018-05-29 | 东软集团股份有限公司 | A kind of method of data synchronization and device based on SQL statement |
| CN105373599B (en) * | 2015-10-28 | 2017-05-31 | 北京汇商融通信息技术有限公司 | Data mover system based on various data storing platforms |
| CN108512693A (en) * | 2018-02-24 | 2018-09-07 | 国家计算机网络与信息安全管理中心 | A kind of trans-regional disaster recovery method and device |
| CN108932309B (en) * | 2018-06-15 | 2020-12-01 | 未鲲(上海)科技服务有限公司 | Cross-platform database management method, device, computer equipment and storage medium |
| CN108897895A (en) * | 2018-07-12 | 2018-11-27 | 浪潮软件集团有限公司 | Device and method for realizing cross-gateway Mysql database synchronization based on cloud message service platform |
| CN109669812A (en) * | 2018-11-27 | 2019-04-23 | 武汉烽火信息集成技术有限公司 | Cloud data center disaster tolerance system and method based on data distribution backup |
| CN109977168B (en) * | 2019-03-18 | 2021-11-05 | 武汉达梦数据库股份有限公司 | Database data synchronization method and device based on data page preloading |
-
2019
- 2019-10-25 CN CN201911022320.6A patent/CN110784532B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8725684B1 (en) * | 2011-03-31 | 2014-05-13 | Amazon Technologies, Inc. | Synchronizing data stores |
| CN102629268A (en) * | 2012-03-09 | 2012-08-08 | 华为技术有限公司 | Data synchronization method, system and date access device |
| CN103793514A (en) * | 2014-02-11 | 2014-05-14 | 华为技术有限公司 | Database synchronization method and databases |
| CN106202365A (en) * | 2016-07-07 | 2016-12-07 | 帅斌鹏 | Method, system and the data-base cluster that database update synchronizes |
| CN109634978A (en) * | 2018-12-18 | 2019-04-16 | 钛马信息网络技术有限公司 | Reading data consistency system and method |
| CN109885617A (en) * | 2019-01-29 | 2019-06-14 | 中国工商银行股份有限公司 | The method of data synchronization and device of Distributed Heterogeneous Database system |
| CN109947801A (en) * | 2019-02-25 | 2019-06-28 | 交通银行股份有限公司 | Database in phase system, method and device |
| CN109933630A (en) * | 2019-03-19 | 2019-06-25 | 武汉达梦数据库有限公司 | Database data real-time synchronization method and equipment |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110784532A (en) | 2020-02-11 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US5784617A (en) | Resource-capability-based method and system for handling service processor requests | |
| US8060892B2 (en) | Executing business logic extensions on a client computing system | |
| CN112069265B (en) | Synchronization method of configuration data, business data system, computer system and medium | |
| US20100023564A1 (en) | Synchronous replication for fault tolerance | |
| US8082344B2 (en) | Transaction manager virtualization | |
| CN110795503A (en) | Multi-cluster data synchronization method and related device of distributed storage system | |
| CN107729176B (en) | Disaster recovery method and disaster recovery system for configuration file management system | |
| CN109783151B (en) | Method and device for rule change | |
| CN109245908B (en) | Method and device for switching master cluster and slave cluster | |
| CN110727709A (en) | Cluster database system | |
| JP6405255B2 (en) | COMMUNICATION SYSTEM, QUEUE MANAGEMENT SERVER, AND COMMUNICATION METHOD | |
| CN112199427A (en) | Data processing method and system | |
| US8156374B1 (en) | Problem management for outsized queues | |
| CN113079098B (en) | Method, device, equipment and computer readable medium for updating route | |
| US11645109B2 (en) | Managing failures in edge computing environments | |
| CN113377863B (en) | Data synchronization method and device, electronic equipment and computer readable storage medium | |
| CN110784532B (en) | Bidirectional data synchronization method and system | |
| US10353735B2 (en) | Computing system including independent coupling facilities maintaining equivalency based on sequence values | |
| US10481963B1 (en) | Load-balancing for achieving transaction fault tolerance | |
| US11327785B2 (en) | Computing system including enhanced application performance based on last completed operation sequence value | |
| US10310921B2 (en) | Computing system including PLSO command to maintain equivalency between independent coupling facilities | |
| CN116361016B (en) | Message processing method and system of network controller | |
| CN109005059A (en) | A kind of system and method for realizing Redis automated back-up | |
| CN114745438B (en) | Cache data processing method, device and system of multi-data center and electronic equipment | |
| US11570110B2 (en) | Cloud system architecture and workload management |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |