CN111026808A - Heterogeneous database synchronization system and synchronization method based on domestic environment - Google Patents
Heterogeneous database synchronization system and synchronization method based on domestic environment Download PDFInfo
- Publication number
- CN111026808A CN111026808A CN201911211544.1A CN201911211544A CN111026808A CN 111026808 A CN111026808 A CN 111026808A CN 201911211544 A CN201911211544 A CN 201911211544A CN 111026808 A CN111026808 A CN 111026808A
- Authority
- CN
- China
- Prior art keywords
- module
- data
- database
- heterogeneous
- synchronization
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims description 21
- 230000005540 biological transmission Effects 0.000 claims abstract description 22
- 238000004519 manufacturing process Methods 0.000 claims abstract description 19
- 238000000605 extraction Methods 0.000 claims abstract description 10
- 238000012544 monitoring process Methods 0.000 claims abstract description 10
- 230000001360 synchronised effect Effects 0.000 claims abstract description 9
- 238000001914 filtration Methods 0.000 claims abstract description 8
- 230000008569 process Effects 0.000 claims description 10
- 230000003213 activating effect Effects 0.000 claims description 4
- 238000013075 data extraction Methods 0.000 claims description 4
- 230000008676 import Effects 0.000 claims description 4
- 239000000725 suspension Substances 0.000 claims description 4
- 238000012545 processing Methods 0.000 claims description 3
- 238000013507 mapping Methods 0.000 abstract description 4
- 238000002955 isolation Methods 0.000 abstract description 2
- 230000010076 replication Effects 0.000 description 4
- 238000006243 chemical reaction Methods 0.000 description 3
- 238000010586 diagram Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000011160 research Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computing Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a heterogeneous database synchronization system based on a domestic environment, which comprises: the system comprises an initialization module, a port monitoring module, a task management module, an acquisition and extraction module, a file writing module and other modules, data reading objects can be selected in batches through read-write personalized settings including read-write modes, frequency, speed and the like, the system automatically matches a table structure of a heterogeneous data source, a table which needs to be synchronized at a production end is selected through the task management module, the acquisition and extraction module, transmission configuration is completed, and the table at the production end is synchronously mapped to the target end through the file writing module and a channel transmission module, so that configuration of the target end is completed. The invention supports the mainstream database at home and abroad, simultaneously meets the data exchange under the isolation of the internal network and the external network of a user, realizes the real-time capture of database logs, and converts the non-domestic database sql sentences into the sql sentences required by the final generation target end according to the functions of filtering mapping and the like configured by the user.
Description
Technical Field
The invention relates to the technical field of databases, in particular to a heterogeneous database synchronization system and a synchronization method based on a domestic environment.
Background
At present, the internet is developed for a thousand days, the number of users is more and more, the application pressure of a large-scale network is more and more, the processing capacity of a single machine is limited, the highest value always exists, and the distributed environment is inevitable.
Because the system is continuously expanded and is built at different periods, heterogeneous databases are likely to appear, data cannot be shared among the heterogeneous databases, and otherwise, an information isolated island appears. Therefore, how to flexibly, effectively and economically process the information sharing and data synchronization of the heterogeneous databases becomes a much-concerned research direction.
In the prior art, a traditional heterogeneous database synchronization mode is realized based on a isomorphic replication technology of a database, each database developer provides a replication scheme of the heterogeneous database on the basis of respective database isomorphic replication, for example, a publisher and subscriber synchronization scheme provided by a relational database management system SQL server, and a relational database management system Oracle adopts a scheme of multi-master replication and materialized view to realize a data synchronization function. However, the existing technology has the problem that due to the low reliability of data synchronization between heterogeneous databases, database logs cannot be captured in real time, and according to functions such as filtering mapping configured by a user, non-domestic database sql statements are converted into sql statements required by a final generation target end.
Disclosure of Invention
The technical problem to be solved by the invention is to provide a heterogeneous database synchronization system and a synchronization method based on a domestic environment, so as to solve the problem that in the background art, due to the low reliability of data synchronization between heterogeneous databases, database logs cannot be captured in real time, and non-domestic database sql statements are converted into sql statements required by a final generation target end according to functions of filtering mapping and the like configured by a user.
The technical problem solved by the invention is realized by adopting the following technical scheme: a heterogeneous database synchronization system based on a domestic environment comprises: the initialization module is used for providing software starting parameters including synchronous data directories, data size of sending and receiving files, data file limit size and the like; the port monitoring module is used for providing module management of the process, including task starting, suspension, stopping and the like; the task management module is used for appointing capture of database types and versions, including full-quantity and increment capture, capture and filtration of database appointed tables, table names and fields and the like; the acquisition and extraction module is used for acquiring and extracting the original data and the logs; the file writing module writes the acquired data into a transmission channel directory; the channel transmission module is used for completing the transmission process of data in the safety channel; the data receiving module is used for receiving the data transmitted by the secure channel; the data checking module is used for checking the consistency of the data; and the data import module is used for importing the final data into the target database.
Preferably, the task management module supports all database field types, and capture of DML, DDL operations.
Preferably, the port listening module supports the start and pause of the task in the data extraction process.
A heterogeneous database synchronization method based on a domestic environment specifically comprises the following steps:
step 1, acquiring details of configuration information of a database at a production end and a database at a target end, and connecting the production end and the target end to an intermediate synchronization server;
step 2, through the read-write personalized setting, including the read-write mode, frequency, speed and the like, data reading objects can be selected in batches, the system automatically matches the table structure of the heterogeneous data source, a table which needs to be synchronized at the production end is selected through the task management module and the acquisition and extraction module, transmission configuration is completed, the table at the production end is synchronously mapped to the target end through the file writing module and the channel transmission module, and the configuration of the target end is completed;
step 3, activating the whole task through the port monitoring module, and completing the synchronization from the non-domestic database to the domestic database through the data receiving module, the data checking module and the data importing module;
and 4, extracting and converting different database versions and different database types through a first full amount mode and a later incremental mode.
Compared with the prior art, the invention has the following advantages: the invention supports domestic and foreign mainstream databases, including Oracle, Sqlserver, Mysql and other various versions to domestic dream database. Meanwhile, the data exchange under the isolation of the internal network and the external network of the user is met, wherein the heterogeneous data exchange comprises the following steps: heterogeneous database content conversion, heterogeneous table content conversion, database field reconstruction and the like; based on a transaction log mode, data is extracted and imported rapidly, namely the log can be captured once generated, and meanwhile, the capturing efficiency is high, and the whole database table does not need to be polled; the resource occupancy of the original system is extremely low, the performance of the original database server is almost not influenced, and the configuration of the existing database is not required to be changed in the deployment process; data synchronization between heterogeneous databases is supported, and when the table structure or service of a source database is changed, a capture module does not need to be redeployed or configured; all database field types are supported, and capturing of DML and DDL operations is realized; supporting table and field filtering, and mapping database names, table names and field names; supporting automatic conversion of heterogeneous data fields; the method does not need to be deployed to a production end, and after the first full synchronization, subsequent incremental synchronization can be carried out; and the starting and suspending of the task in the data extraction process are supported.
Drawings
Fig. 1 is a schematic block diagram of the present invention.
Detailed Description
In order to make the technical means, the creation features, the work flow and the using method of the present invention easily understand and understand the purpose and the efficacy, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, but not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Example 1
Fig. 1 shows a heterogeneous database synchronization system based on a domestic environment, which includes: the initialization module is used for providing software starting parameters including synchronous data directories, data size of sending and receiving files, data file limit size and the like; the port monitoring module is used for providing module management of the process, including task starting, suspension, stopping and the like; the task management module is used for appointing capture of database types and versions, including full-quantity and increment capture, capture and filtration of database appointed tables, table names and fields and the like; the acquisition and extraction module is used for acquiring and extracting the original data and the logs; the file writing module writes the acquired data into a transmission channel directory; the channel transmission module is used for completing the transmission process of data in the safety channel; the data receiving module is used for receiving the data transmitted by the secure channel; the data checking module is used for checking the consistency of the data; and the data import module is used for importing the final data into the target database.
A heterogeneous database synchronization method based on a domestic environment specifically comprises the following steps:
step 1, acquiring details of configuration information of a database at a production end and a database at a target end, and connecting the production end and the target end to an intermediate synchronization server;
step 2, through the read-write personalized setting, including the read-write mode, frequency, speed and the like, data reading objects can be selected in batches, the system automatically matches the table structure of the heterogeneous data source, a table which needs to be synchronized at the production end is selected through the task management module and the acquisition and extraction module, transmission configuration is completed, the table at the production end is synchronously mapped to the target end through the file writing module and the channel transmission module, and the configuration of the target end is completed;
step 3, activating the whole task through the port monitoring module, and completing the synchronization from the non-domestic database to the domestic database through the data receiving module, the data checking module and the data importing module;
and 4, extracting and converting different database versions and different database types through a first full amount mode and a later incremental mode.
Example 2
Fig. 1 shows a heterogeneous database synchronization system based on a domestic environment, which includes: the initialization module is used for providing software starting parameters including synchronous data directories, data size of sending and receiving files, data file limit size and the like; the port monitoring module is used for providing module management of the process, including task starting, suspension, stopping and the like; the task management module is used for appointing capture of database types and versions, including full-quantity and increment capture, capture and filtration of database appointed tables, table names and fields and the like; the acquisition and extraction module is used for acquiring and extracting the original data and the logs; the file writing module writes the acquired data into a transmission channel directory; the channel transmission module is used for completing the transmission process of data in the safety channel; the data receiving module is used for receiving the data transmitted by the secure channel; the data checking module is used for checking the consistency of the data; and the data import module is used for importing the final data into the target database.
Preferably, the task management module supports all database field types, and capture of DML, DDL operations.
Preferably, the port listening module supports the start and pause of the task in the data extraction process.
A heterogeneous database synchronization method based on a domestic environment specifically comprises the following steps:
step 1, acquiring details of configuration information of a database at a production end and a database at a target end, and connecting the production end and the target end to an intermediate synchronization server;
step 2, through the read-write personalized setting, including the read-write mode, frequency, speed and the like, data reading objects can be selected in batches, the system automatically matches the table structure of the heterogeneous data source, a table which needs to be synchronized at the production end is selected through the task management module and the acquisition and extraction module, transmission configuration is completed, the table at the production end is synchronously mapped to the target end through the file writing module and the channel transmission module, and the configuration of the target end is completed;
step 3, activating the whole task through the port monitoring module, and completing the synchronization from the non-domestic database to the domestic database through the data receiving module, the data checking module and the data importing module;
and 4, extracting and converting different database versions and different database types through a first full amount mode and a later incremental mode.
The foregoing shows and describes the general principles, essential features, and advantages of the invention. It will be understood by those skilled in the art that the present invention is not limited to the embodiments described above, which are described in the specification and illustrated only to illustrate the principle of the present invention, but that various changes and modifications may be made therein without departing from the spirit and scope of the present invention, which fall within the scope of the invention as claimed. The scope of the invention is defined by the appended claims and equivalents thereof.
Claims (4)
1. A heterogeneous database synchronization system based on a domestic environment is characterized in that: the system comprises an initialization module, a data processing module and a data processing module, wherein the initialization module is used for providing software starting parameters including synchronous data directories, data size of sending and receiving files, data file limit size and the like; the port monitoring module is used for providing module management of the process, including task starting, suspension, stopping and the like; the task management module is used for appointing capture of database types and versions, including full-quantity and increment capture, capture and filtration of database appointed tables, table names and fields and the like; the acquisition and extraction module is used for acquiring and extracting the original data and the logs; the file writing module writes the acquired data into a transmission channel directory; the channel transmission module is used for completing the transmission process of data in the safety channel; the data receiving module is used for receiving the data transmitted by the secure channel; the data checking module is used for checking the consistency of the data; and the data import module is used for importing the final data into the target database.
2. The heterogeneous database synchronization system based on the domestic environment of claim 1, wherein: the task management module supports all database field types and capture of DML and DDL operations.
3. The heterogeneous database synchronization system based on the domestic environment of claim 1, wherein: the port monitoring module supports the starting and suspending of tasks in the data extraction process.
4. A heterogeneous database synchronization method based on a domestic environment is characterized in that: the method specifically comprises the following steps of,
step 1, acquiring details of configuration information of a database at a production end and a database at a target end, and connecting the production end and the target end to an intermediate synchronization server;
step 2, through the read-write personalized setting, including the read-write mode, frequency, speed and the like, data reading objects can be selected in batches, the system automatically matches the table structure of the heterogeneous data source, a table which needs to be synchronized at the production end is selected through the task management module and the acquisition and extraction module, transmission configuration is completed, the table at the production end is synchronously mapped to the target end through the file writing module and the channel transmission module, and the configuration of the target end is completed;
step 3, activating the whole task through the port monitoring module, and completing the synchronization from the non-domestic database to the domestic database through the data receiving module, the data checking module and the data importing module;
and 4, extracting and converting different database versions and different database types through a first full amount mode and a later incremental mode.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911211544.1A CN111026808A (en) | 2019-12-02 | 2019-12-02 | Heterogeneous database synchronization system and synchronization method based on domestic environment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911211544.1A CN111026808A (en) | 2019-12-02 | 2019-12-02 | Heterogeneous database synchronization system and synchronization method based on domestic environment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111026808A true CN111026808A (en) | 2020-04-17 |
Family
ID=70207648
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911211544.1A Pending CN111026808A (en) | 2019-12-02 | 2019-12-02 | Heterogeneous database synchronization system and synchronization method based on domestic environment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111026808A (en) |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113051247A (en) * | 2021-03-18 | 2021-06-29 | 福建星瑞格软件有限公司 | Database migration method and system based on log synchronization |
| CN113204538A (en) * | 2021-04-27 | 2021-08-03 | 北京百度网讯科技有限公司 | Method, apparatus, device, medium and program product for data migration |
| CN113407641A (en) * | 2021-07-23 | 2021-09-17 | 中共四川省委组织部 | Data synchronization method based on domestic database |
| CN113656511A (en) * | 2021-10-20 | 2021-11-16 | 天津南大通用数据技术股份有限公司 | Heterogeneous database increment synchronization method and system based on source database non-outage |
| CN114500645A (en) * | 2021-12-30 | 2022-05-13 | 兰州乐智教育科技有限责任公司 | Data acquisition system and data acquisition method |
-
2019
- 2019-12-02 CN CN201911211544.1A patent/CN111026808A/en active Pending
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113051247A (en) * | 2021-03-18 | 2021-06-29 | 福建星瑞格软件有限公司 | Database migration method and system based on log synchronization |
| CN113204538A (en) * | 2021-04-27 | 2021-08-03 | 北京百度网讯科技有限公司 | Method, apparatus, device, medium and program product for data migration |
| CN113407641A (en) * | 2021-07-23 | 2021-09-17 | 中共四川省委组织部 | Data synchronization method based on domestic database |
| CN113656511A (en) * | 2021-10-20 | 2021-11-16 | 天津南大通用数据技术股份有限公司 | Heterogeneous database increment synchronization method and system based on source database non-outage |
| CN114500645A (en) * | 2021-12-30 | 2022-05-13 | 兰州乐智教育科技有限责任公司 | Data acquisition system and data acquisition method |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111026808A (en) | Heterogeneous database synchronization system and synchronization method based on domestic environment | |
| CN104967658B (en) | A kind of method of data synchronization on multi-terminal equipment | |
| CN105005618A (en) | Data synchronization method and system among heterogeneous databases | |
| CN107317838B (en) | Astronomical metadata filing method and system based on streaming data processing architecture | |
| CN108009258B (en) | Data acquisition and analysis platform capable of being configured online | |
| CN109753502B (en) | Data acquisition method based on NiFi | |
| CN102508886B (en) | Extensive makeup language (XML)-based method for synchronously updating increment of spatial data | |
| CN110968629A (en) | Cross-hierarchy heterogeneous data aggregation-based unified information resource management method and system | |
| CN108228755A (en) | The data of MySQL database based on daily record analytic technique to Hadoop platform synchronize clone method | |
| CN105701094A (en) | ETL data acquisition method and device | |
| CN108228756A (en) | Data based on the PG databases of daily record analytic technique to Hadoop platform synchronize clone method | |
| CN111143468B (en) | Multi-database data management method based on MPP distributed technology | |
| CN105095496A (en) | Method for monitoring MYSQL table space through ZABBIX | |
| CN114647716A (en) | Generalization data warehouse | |
| Hongqian et al. | Cloud-based data management system for automatic real-time data acquisition from large-scale laying-hen farms | |
| CN106250410A (en) | A kind of data processing method based on flume system and device thereof | |
| CN110895547A (en) | Multi-source heterogeneous database data synchronization system and method based on DB2 federal characteristics | |
| CN109271448A (en) | It is the data synchronous system and method for platform based on database | |
| CN116644136A (en) | Data acquisition method, device, equipment and medium for increment and full data | |
| CN112015795A (en) | System and method for large-data-volume ad hoc query | |
| CN112000684A (en) | Method and tool for realizing bidirectional synchronization of database of loop control based on TRXID | |
| CN112000732A (en) | Trigger-based mutual synchronization method between heterogeneous databases | |
| CN106855877A (en) | A kind of synchronous method for replicating of big data | |
| CN115757642A (en) | Data synchronization method and device based on filing log file | |
| CN113886472A (en) | Data access system, access method, computer equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WD01 | Invention patent application deemed withdrawn after publication |
Application publication date: 20200417 |
|
| WD01 | Invention patent application deemed withdrawn after publication |