CN112559548B - Data synchronization system and method for message middleware - Google Patents

Data synchronization system and method for message middleware Download PDF

Info

Publication number
CN112559548B
CN112559548B CN202011571203.8A CN202011571203A CN112559548B CN 112559548 B CN112559548 B CN 112559548B CN 202011571203 A CN202011571203 A CN 202011571203A CN 112559548 B CN112559548 B CN 112559548B
Authority
CN
China
Prior art keywords
data
user system
database
synchronization
module
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202011571203.8A
Other languages
Chinese (zh)
Other versions
CN112559548A (en
Inventor
黄建平
潘绍士
张琴
李健
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Zhejiang Rongxiang Digital Technology Holding Co ltd
Original Assignee
Zhejiang Rongxiang Digital Technology Holding Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Zhejiang Rongxiang Digital Technology Holding Co ltd filed Critical Zhejiang Rongxiang Digital Technology Holding Co ltd
Priority to CN202011571203.8A priority Critical patent/CN112559548B/en
Publication of CN112559548A publication Critical patent/CN112559548A/en
Application granted granted Critical
Publication of CN112559548B publication Critical patent/CN112559548B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/23—Updating
    • G06F16/2365—Ensuring data consistency and integrity
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/54—Interprogram communication
    • G06F9/546—Message passing systems or structures, e.g. queues
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F2209/00—Indexing scheme relating to G06F9/00
    • G06F2209/54—Indexing scheme relating to G06F9/54
    • G06F2209/548—Queue
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
    • Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management

Abstract

The invention provides a data synchronization system of a message middleware, which is used for data interaction between a first user system and a second user system, wherein the first user system and the second user system are two different systems which operate in the same way, the first user system and the second user system are respectively provided with a first database and a second database which are independent and are used for storing data, and the data synchronization system comprises a message module, a data processing module and a synchronization module; when the data synchronization fails, the invention can continuously synchronize until the synchronization is successful; the invention decides according to the last update time, thereby synchronizing the latest data to the other party and ensuring the integrity of the data.

Description

Data synchronization system and method for message middleware
Technical Field
The present invention relates to data synchronization technology, and in particular, to a system and method for synchronizing message middleware.
Background
With the rapid development of the information age, the development of big data and artificial intelligence is promoted, the data is taken as a resource, and is more and more valued by enterprises and units, and the economic and social values of data output are more and more obvious. The software or the system is modified at different clients, and the modified data cannot be synchronized due to the fact that the related servers are different or the database is down, so that the final accuracy of the data is affected, and therefore, how to ensure the data synchronization of the message middleware and improve the integrity of the data is a technical problem to be solved by a person skilled in the art; for example, chinese patent document CN111026774a discloses a data sequence synchronization method, apparatus, computer device and storage medium, by obtaining a change record table of a database, performing a modulo operation on a partition number of the message middleware with respect to a hash value of a unique field, determining that the value of the unique field does not exist, and obtaining a hash value of a table name of the change record table; the invention provides a data synchronization system and a data synchronization method of message middleware, which aim to solve the problem that intermediate data is lost due to untimely data synchronization.
Disclosure of Invention
The invention aims to provide a data synchronization system and a data synchronization method of message middleware, which are based on a message queue, and an arbitration database can synchronize updated or lost data to a data missing party in time so as to realize the integrity of the data.
In order to achieve the above object, the present invention provides the following technical solutions:
the invention provides a data synchronization system of a message middleware, which is used for data interaction between a first user system and a second user system, wherein the first user system and the second user system are two different systems which operate in the same way, the first user system and the second user system are respectively provided with a first database and a second database which are independent and are used for storing data and updating the data, the data synchronization system comprises a message module, a data processing module and a synchronization module, the message module contains a data updating arbitration service, the data updating arbitration service integrates the message middleware RocketMQ, the data updating of the first user system and/or the second user system is successfully delivered to the data updating arbitration service, the data updating failure is not delivered, and the flow is ended; the data update arbitration service sends updated data to the data processing module, the message middleware RocketMQ monitors consumption information, namely updated data, the data update arbitration service compares database data of the first user system and the second user system according to the updated data obtained when the message middleware consumes the information, the data update arbitration service sends the updated data to the synchronization module according to the result of the database data comparison, the synchronization module synchronizes the updated data to the first user system and/or the second user system, and the synchronization module feeds back to the data processing module for processing again until the synchronization of the updated data is successful.
Further, the first user system includes a first user information module, a first investment preference information module, and a first project information module;
the second user system includes a second user information module, a second investment preference information module, and a second project information module.
Further, each module included in the first user system and the second user system respectively performs data interaction with the data synchronization system.
Further, the message module is used for data update arbitration service, accurate and elegant synchronous data.
Further, the data processing module is used for processing the data acquisition of the first user system and/or the second user system database and the comparison of the data of the first user system and/or the second user system database.
Further, the database data acquisition is to monitor the consumption information by the message middleware RocketMQ and query information from the database of the other user system according to the database data updated by the first user system or the second user system data.
Further, the comparison of the database data includes a temporal comparison of the first user system and/or second user system database data updates and a comparison of how much of the first user system and/or second user system database data is.
Further, the synchronization module is configured to synchronize the user side with a large data amount or a short update time interval date to the database of another user side after the database data of the data processing module is compared.
Further, a data synchronization method of the message middleware comprises the following steps:
(1) User update data to the first user system or the second user system;
(2) Delivering the first user system or the second user system update data to a data update arbitration service;
(3) Message middleware RocketMQ monitors consumption messages;
(4) The data update arbitration service obtains updated data of the first user system and the second user system in the consumption message according to the message consumption of the message middleware;
(5) Comparing the updated database data of the first user system and the second user system;
(6) The data of the user system with more data quantity or short update time interval date is synchronized to the database of another user system; and (5) repeating the step (5) if the synchronization fails.
Further, judging whether the updated data is successful or not, checking a message consumption result record according to the RocketMQ management platform, if the message consumption is not complete, failing to synchronize the data, and if the message consumption is complete, completing the data synchronization successfully.
Further, the message queue monitors consumption and outputs consumption messages according to the first user system and the second user system;
the invention has the following beneficial effects: (1) The invention has no dependence on the task data synchronous party, and avoids the data loss caused by the data synchronous party; (2) When the data synchronization fails, the invention can continuously synchronize until the synchronization is successful; (3) The invention decides according to the last update time, thereby synchronizing the latest data to the other party and ensuring the integrity of the data.
Drawings
FIG. 1 is a schematic diagram of a system module according to the present invention.
Fig. 2 is a schematic diagram of a synchronization system module according to the present invention.
Fig. 3 is a block diagram of an embodiment of the present invention.
Fig. 4 is a flow chart of the method of the present invention.
Detailed Description
The following detailed description of the present invention is provided with reference to the accompanying drawings, and it should be noted that the examples are merely illustrative of the present invention and should not be construed as limiting the invention, and the scope of the present invention should be defined by the appended claims for better understanding and reproduction of the technical solutions of the present invention.
The invention is further described with reference to the drawings and detailed description which follow:
as shown in fig. 1-3, the present invention provides a data synchronization system of message middleware, the data synchronization system 2 is used for data interaction between a first user system 1 and a second user system 3, the first user system 1 and the second user system 3 are two different systems operating in the same mode, the first user system 1 and the second user system 3 are respectively provided with a first database 10 and a second database 30 which are independent and are used for storing data and updating data, the data synchronization system 2 comprises a message module 21, a data processing module 22 and a synchronization module 23, the message module 21 contains a data updating arbitration service, the data updating arbitration service integrates a message middleware, the data updating of the first user system and/or the second user system is successfully delivered to the data updating arbitration service, the data updating failure is not delivered to the data, and the flow is ended; the data update arbitration service sends updated data to the data processing module 22, the message middleware RocketMQ monitors consumption information, namely updated data, the data update arbitration service compares database data of the first user system 1 and the second user system 3 according to the updated data acquired when the message middleware consumes the information, the data update arbitration service delivers the updated data to the synchronization module 23 according to the comparison result of the database data, the synchronization module 23 synchronizes the updated data to the first user system 1 and/or the second user system 3, and the synchronization module 23 feeds back to the data processing module 22 for processing again until the synchronization of the updated data is successful.
Further, the first user system 1 comprises a first user information module 11, a first investment preference information module 12 and a first item information module 13;
the second user system 3 comprises a second user information module 31, a second investment preference information module 32 and a second item information module 33.
Further, each module included in the first user system 1 and the second user system 3 respectively performs data interaction with a data synchronization system.
Further, the message module 21 is used for data update arbitration service, accurate and elegant synchronization data.
Further, the data processing module 22 is configured to process the database data acquisition of the first user system 1 and/or the second user system 3 and the comparison of the database data of the first user system 1 and/or the second user system 3.
Further, the database data acquisition is to monitor the consumption information by the message middleware RocketMQ and query information from the database of the other user system according to the database data updated by the first user system 1 or the second user system 3.
Further, the comparison of the database data includes a time comparison of the update of the database data of the first user system 1 and/or the second user system 3 and a comparison of how much database data of the first user system 1 and/or the second user system 3.
Further, the synchronization module 23 is configured to synchronize the database of the data processing module 22 with the database of another user terminal after comparing the database data, which has a large data amount or a short update time interval and date.
As shown in fig. 4, a data synchronization method of message middleware includes the following steps:
(1) User update data to the first user system or the second user system;
(2) Delivering the first user system or the second user system update data to a data update arbitration service;
(3) Message middleware RocketMQ monitors consumption messages;
(4) The data update arbitration service obtains updated data of the first user system and the second user system in the consumption message according to the message consumption of the message middleware;
(5) Comparing the updated database data of the first user system and the second user system;
(6) The data of the user system with more data quantity or short update time interval date is synchronized to the database of another user system; and (5) repeating the step (5) if the synchronization fails.
Further, judging whether the updated data is successful or not, checking a message consumption result record according to the RocketMQ management platform, if the message consumption is not complete, failing to synchronize the data, and if the message consumption is complete, completing the data synchronization successfully;
the message queue monitors consumption and outputs consumption messages according to the first user system and the second user system;
the first user system is set as an APP system, the second user system is a PC system, and the data updated by the APP system is taken as an example, and the specific steps of data synchronization are as follows:
s1, an APP system updates user information;
s2, delivering updated user information to a data update arbitration service by the APP system;
s3, a message middleware RocketMQ monitors consumption messages;
s4, the data update arbitration service acquires database data of the APP system in the consumption message according to the message consumption of the message middleware;
s5, comparing the APP system database data with the database of the PC system,
s51, APP system database data exist, database data of a PC system do not exist, and the APP system database data are synchronized to the database of the PC system;
s52, the database data of the APP system does not exist, the database data of the PC system exists, and the database data of the PC system is synchronized to the database of the APP system;
s53, the database of the APP system is shorter than the database of the PC system in updating time interval and date, and the database of the APP system is synchronized to the database of the PC system;
s54, the database of the PC system is shorter than the database of the APP system in updating time interval and date, and the database of the PC system is synchronized to the database of the APP system;
s6, if the step S51, the step S52, the step S53 or the step S54 fail, the step S5 is repeated until the synchronization is successful.
The specific steps of data synchronization using the updated data of the PC system as an example are as follows:
s1, updating user information by a PC system;
s2, the PC system delivers the updated user information to a data update arbitration service;
s3, a message middleware RocketMQ monitors consumption messages;
s4, the data update arbitration service acquires database data of the PC system in the consumption message according to the message middleware;
s5, comparing the database data of the PC system with the database of the APP system,
s51, APP system database data exist, database data of a PC system do not exist, and the APP system database data are synchronized to the database of the PC system;
s52, the database data of the APP system does not exist, the database data of the PC system exists, and the database data of the PC system is synchronized to the database of the APP system;
s53, the database of the APP system is shorter than the database of the PC system in updating time interval and date, and the database of the APP system is synchronized to the database of the PC system;
s54, the database of the PC system is shorter than the database of the APP system in updating time interval and date, and the database of the PC system is synchronized to the database of the APP system;
s6, if the step S51, the step S52, the step S53 or the step S54 fail, the step S5 is repeated until the synchronization is successful.
While preferred embodiments of the present application have been described, additional variations and modifications in those embodiments may occur to those skilled in the art once they learn of the basic inventive concepts. It is therefore intended that the following claims be interpreted as including the preferred embodiments and all such alterations and modifications as fall within the scope of the application.

Claims (7)

1. A data synchronization system for message middleware, characterized in that,
the data synchronization system is used for data interaction between a first user system and a second user system, the first user system and the second user system are two different systems which operate in the same mode, the first user system and the second user system are respectively provided with a first database and a second database which are independent and used for storing data and updating the data, the data synchronization system comprises a message module, a data processing module and a synchronization module, the message module contains a data updating arbitration service, the data updating arbitration service integrates a message middleware RocketMQ, the data updating success of the first user system and/or the second user system is transmitted to the data updating arbitration service, the data updating failure is not transmitted to the data, and the flow is ended; the data update arbitration service sends updated data to the data processing module, the message middleware RocketMQ monitors consumption information, namely updated data, the data update arbitration service compares database data of the first user system and the second user system according to the updated data obtained when the message middleware consumes the information, the data update arbitration service sends the updated data to the synchronization module according to the result of the database data comparison, the synchronization module synchronizes the updated data to the first user system and/or the second user system, and the synchronization module feeds back to the data processing module for processing again until the synchronization of the updated data is successful;
the data processing module is used for processing the data acquisition of the first user system and/or the second user system database and the comparison of the data of the first user system and/or the second user system database;
the database data acquisition is to monitor the information of the database query of the other user system from the database data updated by the consumption message according to the data of the first user system or the second user system by the message middleware RocketMQ;
the comparison of the database data includes a temporal comparison of the first user system and/or second user system database data updates and a comparison of how much of the first user system and/or second user system database data is.
2. The message middleware data synchronization system of claim 1, wherein,
the first user system comprises a first user information module, a first investment preference information module and a first project information module;
the second user system includes a second user information module, a second investment preference information module, and a second project information module.
3. The message middleware data synchronization system of claim 2, wherein,
each module contained in the first user system and the second user system respectively performs data interaction with the data synchronization system.
4. The message middleware data synchronization system of claim 1, wherein,
the message module is used for data update arbitration service, accurate and elegant synchronous data.
5. The message middleware data synchronization system of claim 1, wherein,
and the synchronization module is used for synchronizing the user side with more data quantity or short update time interval date to the database of the other user side after the database data of the data processing module are compared.
6. A data synchronization method of message middleware is characterized in that,
the method comprises the following steps:
(1) User update data to the first user system or the second user system;
(2) Delivering the first user system or the second user system update data to a data update arbitration service;
(3) Message middleware RocketMQ monitors consumption messages;
(4) The data update arbitration service obtains updated data of the first user system and the second user system in the consumption message according to the message consumption of the message middleware;
(5) Comparing the updated database data of the first user system and the second user system;
(6) The data of the user system with more data quantity or short update time interval date is synchronized to the database of another user system; and (5) repeating the step (5) if the synchronization fails.
7. The method for data synchronization of message middleware as set forth in claim 6, wherein,
judging whether the updated data succeeds in checking the message consumption result record according to the RocketMQ management platform, if the message consumption is not complete, failing to synchronize the data, and if the message consumption is complete, the data synchronization is successful.
CN202011571203.8A 2020-12-27 2020-12-27 Data synchronization system and method for message middleware Active CN112559548B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202011571203.8A CN112559548B (en) 2020-12-27 2020-12-27 Data synchronization system and method for message middleware

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202011571203.8A CN112559548B (en) 2020-12-27 2020-12-27 Data synchronization system and method for message middleware

Publications (2)

Publication Number Publication Date
CN112559548A CN112559548A (en) 2021-03-26
CN112559548B true CN112559548B (en) 2023-06-16

Family

ID=75033302

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202011571203.8A Active CN112559548B (en) 2020-12-27 2020-12-27 Data synchronization system and method for message middleware

Country Status (1)

Country Link
CN (1) CN112559548B (en)

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106682140A (en) * 2016-12-20 2017-05-17 华北计算技术研究所（中国电子科技集团公司第十五研究所） Multi-system user incremental synchronization method based on timestamps and mapping strategies
CN110351662A (en) * 2018-04-04 2019-10-18 华为技术有限公司 A kind of method, platform and the device of the collaboration of end cloud
CN111597197A (en) * 2020-06-29 2020-08-28 腾讯科技（深圳）有限公司 Data reconciliation method and device between databases, storage medium and electronic equipment

Family Cites Families (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR100471567B1 (en) * 2000-07-29 2005-03-07 엘지전자 주식회사 Transaction Management Method For Data Synchronous In Dual System Environment
US11222001B2 (en) * 2013-03-15 2022-01-11 Sap Se Augmenting middleware communication services
US10447779B2 (en) * 2016-06-21 2019-10-15 Sap Se Synchronizing document replication in distributed systems
KR102333127B1 (en) * 2017-05-26 2021-11-30 삼성전자주식회사 Method for acquiring user information and an electronic device thereof
CN109408203B (en) * 2018-11-01 2019-10-18 无锡华云数据技术服务有限公司 A kind of implementation method, device, the computing system of queue message consistency
CN110309161B (en) * 2019-06-06 2021-10-29 新华三大数据技术有限公司 Data synchronization method and device and server
CN110704458B (en) * 2019-08-15 2023-04-18 平安科技（深圳）有限公司 Data synchronization method and device, computer equipment and storage medium

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106682140A (en) * 2016-12-20 2017-05-17 华北计算技术研究所（中国电子科技集团公司第十五研究所） Multi-system user incremental synchronization method based on timestamps and mapping strategies
CN110351662A (en) * 2018-04-04 2019-10-18 华为技术有限公司 A kind of method, platform and the device of the collaboration of end cloud
CN111597197A (en) * 2020-06-29 2020-08-28 腾讯科技（深圳）有限公司 Data reconciliation method and device between databases, storage medium and electronic equipment

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
A Method od Data Synchronization Based on Message Oriented Middleware and Xml in Distributed Heterogeneous Environments;Z.Y.Lu,Z.B.Guo;Proceedings of the 2015 International Conference on Artificial Intelligence and Industrial Engineering;全文 *
面向数据的实时网络中间件的设计与实现;汤学达等;网络安全技术与应用;全文 *

Also Published As

Publication number Publication date
CN112559548A (en) 2021-03-26

Similar Documents

Publication Publication Date Title
EP2919130A1 (en) Method and system for synchronizing distributed database
US11899689B2 (en) Blockchain-based data synchronization method, apparatus, and computer-readable storage medium
CN109388677B (en) Method, device and equipment for synchronizing data among clusters and storage medium thereof
CN115599870B (en) Data synchronization method based on fusion of stock data and incremental data of message queue
CN107231435B (en) Data synchronization monitoring method and system
CN111338834B (en) Data storage method and device
CN114092252A (en) Block chain transaction execution method, device, equipment and readable storage medium
CN113193947A (en) Method, apparatus, medium, and program product for implementing distributed global ordering
CN110750594B (en) Real-time cross-network database synchronization method based on mysql incremental log
CN110442564B (en) Incremental data verification method and storage medium
CN114416883A (en) Block chain light node data synchronization method, device, equipment and readable storage medium
CN112559548B (en) Data synchronization system and method for message middleware
CN111680106B (en) Method and device for synchronizing data of multiple application systems
US8843531B2 (en) Bookkeeping of download timestamps
CN110585724B (en) Method and device for updating form data in game client
CN109005246B (en) Data synchronization method, device and system
CN116701352A (en) Database data migration method and system
CN111966650B (en) Operation and maintenance big data sharing data table processing method and device and storage medium
US20090106781A1 (en) Remote call handling methods and systems
CN111522875B (en) Distributed system data copy consistency monitoring method for full data synchronization
CN114969206A (en) Data processing method, device, equipment and storage medium
CN107563942B (en) Logistics data batch processing method, logistics processing system and processing device
CN115033578A (en) Method for updating service data, related device and storage medium
CN113645309A (en) Processing method and system for multi-client data differentiation secondary caching and synchronization
CN112217712A (en) Message multi-terminal synchronization method and client

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant