CN105824865A - Distributed type database data synchronization system - Google Patents
Distributed type database data synchronization system Download PDFInfo
- Publication number
- CN105824865A CN105824865A CN201510905939.7A CN201510905939A CN105824865A CN 105824865 A CN105824865 A CN 105824865A CN 201510905939 A CN201510905939 A CN 201510905939A CN 105824865 A CN105824865 A CN 105824865A
- Authority
- CN
- China
- Prior art keywords
- data
- engine
- sql
- delta
- message
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
- G06F16/273—Asynchronous replication or reconciliation
Abstract
The invention discloses a distributed type database data synchronization system which comprises a management platform, a scanning engine and an execution engine, wherein the management platform is used for operating and controlling the scanning engine and the execution engine; the scanning engine is used for subscribing variation data of appointed fields in an appointing table according to configuration of the management platform of a user; the execution engine is used for synchronizing variation data scanned by the scanning engine to an appointed target database. By adopting the distributed type database data synchronization system, consumption of a great amount of log files transmitted among networks can be reduced, variation data which can be individually subscribed according to demands of users self are provided, differences of various database protocols are shielded, and data can be synchronized to multiple types of target databases such as SQL, DB2 and oracle.
Description
Technical field
The present invention relates to data synchronous system, particularly relate to data synchronous system between a kind of distributed data base.
Background technology
Owing to the transaction data amount of telecommunication system is the hugest; and single oracle database is difficult to bear the biggest data trade amount, so in order to improve performance, it will usually be deployed to distributed data base; i.e. dispose multiple data bases at different machine rooms, be allocated by province or districts and cities.Some so will certainly be caused to be present in the data property of there are differences of each database table.In order to ensure these tables concordance between disparate databases or can, in the data of certain DB Backup another one some table of data base, need a kind of data synchronization technology to reach between disparate databases to realize the purpose of data syn-chronization.
In the prior art, GoldenGate is a kind of based in the structural data copy mode of daily record, it is by resolving the online daily record of source database or the additions and deletions changeization of archive log acquisition data, again by these change application to target database, it is achieved source database is Tong Bu with target database, dual-active.This simultaneous techniques has problems with: first database journal needs to carry out network transmission by TCP/IP between networks, high to bandwidth requirement.Secondly, simultaneously supporting at source database and target database installation and deployment goldengate, complex operation are needed.Additionally, there is no good monitor supervision platform for whole data reproduction process, replicate and need the daily record artificially going to check source or destination end to carry out orientation problem if makeing mistakes.Again, it is impossible to do personalized synchronization, because some target database may have only to synchronize the data of wherein several tables rather than synchronize whole table data, and commercial version is independent charge relative to oracle, needs to pay extra-pay.
Summary of the invention
The technical problem to be solved in the present invention is, for the deficiencies in the prior art, it is provided that a kind of system that can realize data syn-chronization between disparate databases.
For solving above-mentioned technical problem, the present invention adopts the following technical scheme that.
Data synchronous system between a kind of distributed data base, it includes management platform, scanning engine and enforcement engine, wherein: described management platform is for manipulating scanning engine and enforcement engine；Described scanning engine, for managing the configuration of platform according to user, is subscribed to source database and is specified the delta data of specific field in table；Described enforcement engine is synchronized to the target database specified for the delta data scanned by scanning engine.
Further, under conditions of utilizing sql script file as interactive mode, after described scanning engine scans delta data, delta data is generated in order a rule executable sql statement and is saved in sql file, more described sql file is saved in user configured assigned catalogue.
Further, include temp directory, and when described scanning engine scans a collection of delta data, these delta datas are generated an independent sql file at temp directory in order, after described sql file generated is complete, move again to user configured assigned catalogue.
Further, described enforcement engine is timed to whether assigned catalogue scanning has executable sql file generated, if having, then reads the sql statement of sql file, is then performed to target database one by one by described sql statement, and then realize data syn-chronization.
Further, under conditions of utilizing message-oriented middleware as interactive mode, after described scanning engine scans delta data, in delta data being generated a rule executable sql statement and being sent to the queue of message-oriented middleware.
Further, described enforcement engine is timed in the queue of message-oriented middleware obtain message, if the message of getting, is then performed to target database one by one by the sql statement in message, and then realizes data syn-chronization.
Further, described target database is Mysql, DB2, SQLSERVER or oracle database.
The present invention having the beneficial effects that compared to existing technologies, first, the present invention only captures the delta data of table needing to synchronize, and each target database can the delta data of personalized some field subscribing to certain table as required, it might even be possible to synchronize some dml operation selectively；Secondly, between different types of data base, data syn-chronization is realized, as the popular databases such as DB2, MYSQL, SQLSERVER can be supported；Additionally, visual configuration can be carried out by configuration center, it is not necessary to sign in data base machine and configure, convenient and swift.The present invention is without carrying out any configuration in target database or installing other instruments, have only to dispose a data synchronous system as intermediate layer program, simultaneously without transmitting journal file between source database and target database, preferably achieve between disparate databases, realize data syn-chronization.
Accompanying drawing explanation
Fig. 1 is the composition frame chart of data synchronous system of the present invention.
Detailed description of the invention
With embodiment, the present invention is described in more detail below in conjunction with the accompanying drawings.
The invention discloses data synchronous system between a kind of distributed data base, as it is shown in figure 1, it includes management platform 1, scanning engine 2 and enforcement engine 3, wherein:
Described management platform 1 is for manipulating scanning engine 2 and enforcement engine 3；
Described scanning engine 2, for managing the configuration of platform 1 according to user, is subscribed to source database and is specified the delta data of specific field in table；
Described enforcement engine 3 for being synchronized to the target database specified by the delta data that scanning engine 2 scans.
The present invention, according to the difference of interactive mode, has a below embodiment:
Embodiment 1
Under conditions of utilizing sql script file as interactive mode, after described scanning engine 2 scans delta data, delta data is generated in order a rule executable sql statement and is saved in sql file, more described sql file is saved in user configured assigned catalogue.
In the present embodiment, system includes temp directory, and when described scanning engine 2 scans a collection of delta data, these delta datas are generated an independent sql file at temp directory in order, after described sql file generated is complete, move again to user configured assigned catalogue.Sql file can be avoided by the way the most not generated and to be just performed the problem that engine 3 performs, at the same time it can also be avoid adding file lock.
Described enforcement engine 3 is timed to whether assigned catalogue scanning has executable sql file generated, if having, then reads the sql statement of sql file, is then performed to target database one by one by described sql statement, and then realize data syn-chronization.
Embodiment 2
Under conditions of utilizing message-oriented middleware as interactive mode, after described scanning engine 2 scans delta data, in delta data being generated a rule executable sql statement and being sent to the queue of message-oriented middleware.Described enforcement engine 3 is timed in the queue of message-oriented middleware obtain message, if the message of getting, is then performed to target database one by one by the sql statement in message, and then realizes data syn-chronization.
Use message-oriented middleware can reach asynchronous by full decoupled to scanning engine and enforcement engine as interactive mode.Scanning engine and enforcement engine can be deployed to different machines, it is desirable to additionally dispose message-oriented middleware as support.
In above-described embodiment, described target database includes but not limited to Mysql, DB2, SQLSERVER or oracle database.
The present invention is in running, it is changed the capture of daily record by opening the feature list of the changedatacapture assembly of oracle source database and is saved in changtable, go the data of subscription changtable as subscriber by synchronization program, executable sql script file is generated after getting delta data, and directly perform these sql script files in different types of target database thus reach the purpose of data syn-chronization, the present invention solves a large amount of journal file and transmits the consumption brought between network, provide the delta data subscribed to oneself personalizedly and need, and shield the difference of various database protocol, support is synchronized to polytype target database, including MYSQL, DB2, oracle etc..Additionally, the present invention further provided with configuration center and Surveillance center, thus realize configuring simple and convenient, monitoring synchrodata amount, unsuccessfully the function such as reform.
The above is preferred embodiment of the present invention, is not limited to the present invention, all amendment, equivalent or improvement etc. made in the technical scope of the present invention, should be included in the range of the present invention protected.
Claims (7)
1. data synchronous system between a distributed data base, it is characterised in that include management platform (1), scanning engine (2) and enforcement engine (3), wherein:
Described management platform (1) is for manipulating scanning engine (2) and enforcement engine (3)；
Described scanning engine (2), for managing the configuration of platform (1) according to user, is subscribed to source database and is specified the delta data of specific field in table；
Described enforcement engine (3) for being synchronized to the target database specified by the delta data that scanning engine (2) scans.
2. data synchronous system between distributed data base as claimed in claim 1, it is characterized in that, under conditions of utilizing sql script file as interactive mode, after described scanning engine (2) scans delta data, delta data is generated in order a rule executable sql statement and is saved in sql file, more described sql file is saved in user configured assigned catalogue.
3. data synchronous system between distributed data base as claimed in claim 2, it is characterized in that, include temp directory, and when described scanning engine (2) scans a collection of delta data, these delta datas are generated an independent sql file at temp directory in order, after described sql file generated is complete, moves again to user configured assigned catalogue.
4. data synchronous system between distributed data base as claimed in claim 3, it is characterized in that, described enforcement engine (3) is timed to whether assigned catalogue scanning has executable sql file generated, if having, then read the sql statement of sql file, then described sql statement is performed to target database one by one, and then realize data syn-chronization.
5. data synchronous system between distributed data base as claimed in claim 1, it is characterized in that, under conditions of utilizing message-oriented middleware as interactive mode, after described scanning engine (2) scans delta data, in delta data being generated a rule executable sql statement and being sent to the queue of message-oriented middleware.
6. data synchronous system between distributed data base as claimed in claim 5, it is characterized in that, described enforcement engine (3) is timed in the queue of message-oriented middleware obtain message, if the message of getting, then the sql statement in message is performed to target database one by one, and then realize data syn-chronization.
7. data synchronous system between the distributed data base as described in any one of claim 1 to 6, it is characterised in that described target database is Mysql, DB2, SQLSERVER or oracle database.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510905939.7A CN105824865A (en) | 2015-12-09 | 2015-12-09 | Distributed type database data synchronization system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510905939.7A CN105824865A (en) | 2015-12-09 | 2015-12-09 | Distributed type database data synchronization system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN105824865A true CN105824865A (en) | 2016-08-03 |
Family
ID=56514652
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201510905939.7A Pending CN105824865A (en) | 2015-12-09 | 2015-12-09 | Distributed type database data synchronization system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN105824865A (en) |
Cited By (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106354865A (en) * | 2016-09-09 | 2017-01-25 | 北京奇虎科技有限公司 | Method, device and system for synchronizing master database and secondary database |
| CN106649756A (en) * | 2016-12-26 | 2017-05-10 | 北京奇虎科技有限公司 | Log synchronization method and device |
| CN107861997A (en) * | 2017-10-19 | 2018-03-30 | 金蝶软件（中国）有限公司 | ERP database recording method, device, computer equipment and storage medium |
| CN108228733A (en) * | 2017-12-12 | 2018-06-29 | 浪潮软件股份有限公司 | A kind of file syn chronizing system and method |
| CN109726046A (en) * | 2018-11-23 | 2019-05-07 | 网联清算有限公司 | Computer room switching method and switching device |
| CN110543513A (en) * | 2018-05-29 | 2019-12-06 | 中兴通讯股份有限公司 | incremental data synchronization method, equipment and storage medium for distributed database |
| CN111177254A (en) * | 2019-12-05 | 2020-05-19 | 武汉达梦数据库有限公司 | Method and device for data synchronization between heterogeneous relational databases |
| WO2021027363A1 (en) * | 2019-08-15 | 2021-02-18 | 平安科技（深圳）有限公司 | Data synchronization method and apparatus, computer device and storage medium |
| CN113641516A (en) * | 2021-08-09 | 2021-11-12 | 杭州安恒信息技术股份有限公司 | Scanning engine management and control method, system and computer readable storage medium |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20040024771A1 (en) * | 2002-08-01 | 2004-02-05 | Oracle International Corporation | Buffered message queue architecture for database management systems with transactional enqueue support |
| CN102929946A (en) * | 2012-09-27 | 2013-02-13 | 山东中创软件工程股份有限公司 | Data synchronization method, device and system |
| CN103020112A (en) * | 2012-11-06 | 2013-04-03 | 深圳中兴网信科技有限公司 | Heterogeneous database synchronizing system and heterogeneous database synchronizing method |
-
2015
- 2015-12-09 CN CN201510905939.7A patent/CN105824865A/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20040024771A1 (en) * | 2002-08-01 | 2004-02-05 | Oracle International Corporation | Buffered message queue architecture for database management systems with transactional enqueue support |
| CN102929946A (en) * | 2012-09-27 | 2013-02-13 | 山东中创软件工程股份有限公司 | Data synchronization method, device and system |
| CN103020112A (en) * | 2012-11-06 | 2013-04-03 | 深圳中兴网信科技有限公司 | Heterogeneous database synchronizing system and heterogeneous database synchronizing method |
Cited By (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106354865B (en) * | 2016-09-09 | 2020-05-15 | 北京奇虎科技有限公司 | Method, device and system for synchronizing master database and slave database |
| CN106354865A (en) * | 2016-09-09 | 2017-01-25 | 北京奇虎科技有限公司 | Method, device and system for synchronizing master database and secondary database |
| CN106649756A (en) * | 2016-12-26 | 2017-05-10 | 北京奇虎科技有限公司 | Log synchronization method and device |
| CN107861997A (en) * | 2017-10-19 | 2018-03-30 | 金蝶软件（中国）有限公司 | ERP database recording method, device, computer equipment and storage medium |
| CN107861997B (en) * | 2017-10-19 | 2021-12-03 | 金蝶软件（中国）有限公司 | ERP database recording method and device, computer equipment and storage medium |
| CN108228733A (en) * | 2017-12-12 | 2018-06-29 | 浪潮软件股份有限公司 | A kind of file syn chronizing system and method |
| CN110543513A (en) * | 2018-05-29 | 2019-12-06 | 中兴通讯股份有限公司 | incremental data synchronization method, equipment and storage medium for distributed database |
| CN110543513B (en) * | 2018-05-29 | 2024-03-15 | 金篆信科有限责任公司 | Incremental data synchronization method, equipment and storage medium for distributed database |
| CN109726046B (en) * | 2018-11-23 | 2021-01-08 | 网联清算有限公司 | Machine room switching method and device |
| CN109726046A (en) * | 2018-11-23 | 2019-05-07 | 网联清算有限公司 | Computer room switching method and switching device |
| WO2021027363A1 (en) * | 2019-08-15 | 2021-02-18 | 平安科技（深圳）有限公司 | Data synchronization method and apparatus, computer device and storage medium |
| CN111177254A (en) * | 2019-12-05 | 2020-05-19 | 武汉达梦数据库有限公司 | Method and device for data synchronization between heterogeneous relational databases |
| CN111177254B (en) * | 2019-12-05 | 2021-08-17 | 武汉达梦数据库股份有限公司 | Method and device for data synchronization between heterogeneous relational databases |
| CN113641516A (en) * | 2021-08-09 | 2021-11-12 | 杭州安恒信息技术股份有限公司 | Scanning engine management and control method, system and computer readable storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN105824865A (en) | Distributed type database data synchronization system | |
| CN109284334B (en) | Real-time database synchronization method and device, electronic equipment and storage medium | |
| US7899917B2 (en) | Synchronization framework for occasionally connected applications | |
| CN109413127B (en) | Data synchronization method and device | |
| CN102882985B (en) | Based on the file sharing method that cloud stores | |
| CN106503163B (en) | Global configuration multi-tenant dynamic data source system based on SaaS application | |
| US9043278B1 (en) | System and method for the merging of databases | |
| CN110009201B (en) | Electric power data link system and method based on block chain technology | |
| US20140268245A1 (en) | Client device state collection and network-based processing solution | |
| CN103177116A (en) | Distributed log handling and inquiring method based on two-stage index | |
| US10069941B2 (en) | Scalable event-based notifications | |
| WO2017114111A1 (en) | Asynchronous data synchronization method for distributed system | |
| US10019501B2 (en) | Data store synchronization utilizing synchronization logs | |
| CN104378234A (en) | Cross-data-center data transmission processing method and system | |
| CN110019539A (en) | A kind of method and apparatus that the data of data warehouse are synchronous | |
| CN103546572A (en) | Cloud storage device and multi-cloud storage networking system and method | |
| CN110096545A (en) | One kind being based on big data platform data processing domain architecting method | |
| CN114296836A (en) | Remote configuration system | |
| CN110099128B (en) | Authentication data synchronization method and device | |
| CN102624932A (en) | Index-based remote cloud data synchronizing method | |
| CN101720548B (en) | Represent the method and system of data in request entity access catalogue | |
| CN105069012B (en) | A kind of file access method and system | |
| CN111200623B (en) | Method and system for realizing terminal data synchronization based on distributed storage | |
| CN110633322A (en) | Resource information synchronization method and device, electronic equipment and storage medium | |
| WO2016082608A1 (en) | Service data query/search method, apparatus and system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20160803 |
|
| RJ01 | Rejection of invention patent application after publication |