CN105975502A - Method for realizing incremental data extract based on CDC (Change Data Capture) mode - Google Patents

Method for realizing incremental data extract based on CDC (Change Data Capture) mode Download PDF

Info

Publication number
CN105975502A
CN105975502A CN201610271011.2A CN201610271011A CN105975502A CN 105975502 A CN105975502 A CN 105975502A CN 201610271011 A CN201610271011 A CN 201610271011A CN 105975502 A CN105975502 A CN 105975502A
Authority
CN
China
Prior art keywords
data
extraction
source
cdc
log buffer
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN201610271011.2A
Other languages
Chinese (zh)
Inventor
游辉敏
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Nanjing Youce Information Technology Co Ltd
Original Assignee
Nanjing Youce Information Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Nanjing Youce Information Technology Co Ltd filed Critical Nanjing Youce Information Technology Co Ltd
Priority to CN201610271011.2A priority Critical patent/CN105975502A/en
Publication of CN105975502A publication Critical patent/CN105975502A/en
Pending legal-status Critical Current

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/25Integrating or interfacing systems involving database management systems
    • G06F16/254Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses

Landscapes

  • Engineering & Computer Science (AREA)
  • Databases & Information Systems (AREA)
  • Theoretical Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The invention discloses a method for realizing incremental data extract based on a CDC (Change Data Capture) mode and belongs to the application field of computer networks and database technologies. The method comprises the following steps of: firstly, providing a set of complete and feasible solution schemes for incremental data synchronism from a source database table to a target database table; secondly, carrying out incremental extract by using a CDC technology without modifying a source list data structure. According to the method, while the operating performance of a system is improved, the data transmission quantity is reduced and the network load is alleviated.

Description

A kind of method realizing increment extraction data based on CDC mode
Technical field
The invention belongs to computer network and database technology application, relate generally to oracle database Increment extraction and method of data synchronization, be specifically related to a kind of realize increment extraction data based on CDC mode Method.
Background technology
CDC (Change Data Capture changes data capture) is a kind of data increment treatment technology, its energy You is enough helped to identify the data changed after last fetched.Utilize CDC technology, source table is being carried out Data, and the data changed just can be extracted while the operation such as INSERT, UPDATE or DELETE It is stored in the change table of data base.Thus can capture the data changed, then utilize data Storehouse view by a kind of controlled in the way of be supplied to goal systems.And traditional increment extraction technology, generally have Former table adds timestamp with full table than Peer, need revise former list structure or use substantial amounts of algorithm Realize, simultaneously need to transmit more data, cause higher offered load, and direct and database engine The CDC technology being directly integrated is compared, and performance also has a certain distance.
Summary of the invention
In view of this, the invention provides a kind of method realizing increment extraction data based on CDC mode, Change monitoring device CDC table based on oracle database carries out incremental data extraction, mainly includes data manipulation The foundation of log buffer table, arrange with the timestamp in data manipulation log buffer table for according to coming periodically or Carry out data pick-up by hand, do corresponding increment processed and the delta data of extraction is supplied to target data Storehouse, all can set up data exchange node at source table and object table end and complete to operate accordingly, by unified number It is managed according to switching plane;This method purpose that the present invention provides is to realize high performance automatic extraction number Complete the data syn-chronization of source table and object table according to the incremental data in storehouse, alleviate offered load the most as far as possible.
The technical scheme is that a kind of method realizing increment extraction data based on CDC mode, including Following steps: step one, create identical table according to business scenario in the data source data base with datum target The database table of structure, then in source data base, create data manipulation log buffer table;
Step 2, creates dynamic sql rule, it is achieved from the CDC table of source, by table [TIMESTAMP $] field, as increment foundation, extracts incremental data;
Step 3, create Data Integration service, by data manipulation log buffer table data incrementally according to and Data variation mode updates in target;In the Data Integration service of data interchange platform, data source is data behaviour Make log buffer table, set change monitoring rule;
Step 4, by the Data Integration service arrangement that creates to the node at table place, source, timing extraction number According to;When database source table data send change, storage source table change automatically in data manipulation log buffer table Data;The timed task of data interchange platform can call the Data Integration service created, data at first When Operation Log cache table stores new delta data, the temporal information in configuration file can be read as depending on Extract according to by the delta data in data manipulation log buffer table, and store this operating time and be used as The foundation of extraction next time;
The data extracted are passed through the batch data bus transfer of data interchange platform to target by step 5 Node, destination node is by respective change information updating to object table and store.
Further, in described step 2, after extraction incremental data, also include setting increment foundation, and The data of extraction do not repeat every time.
Further, described step 3 kind sets change monitoring rule i.e. according in data manipulation log buffer table Data variation kind distinguish data manipulation species, simultaneously according to the dynamic sql rule set, it is intended that depend on According to field and initial value.
Further, described according to the data variation kind in data manipulation log buffer table distinguish data behaviour Making in species, described data variation kind includes I: insert, UU: amendment old value, the new value of UN: amendment, D: delete Remove.
Further, in the Data Integration service in described step 4, the mode of timing extraction data is for setting Timing operation task timing extraction data or on-demand manual triggering operation carry out extracted data.
Further, the foundation of data manipulation log buffer table described in described step one be ensure source table and The structure of object table is identical, sets up, at source table end, the data manipulation log buffer table that source table is corresponding, grasps in data Make log buffer table adds the timestamp field foundation as extraction.
The invention have the benefit that the present invention the method provides complete set feasible from source number Carry out the solution of incremental data synchronization to target database table according to storehouse table;Secondly, CDC technology is employed Carry out increment extraction, it is not necessary to amendment source list data structure, subtract while improve system performance Lack volume of transmitted data, alleviate offered load.
Accompanying drawing explanation
Fig. 1 is the structure composed figure that the CDC mode of the present invention realizes increment extraction data method;
Fig. 2 is this method that provides of the present invention interface schematic diagram when extracting incremental data;
Fig. 3 is that this method that the present invention provides is creating the interface schematic diagram one that Data Integration services;
Fig. 4 is that this method that the present invention provides is creating the interface schematic diagram two that Data Integration services.
Detailed description of the invention
Below in conjunction with the accompanying drawing in the embodiment of the present invention, the technical scheme in the embodiment of the present invention is carried out clearly Chu, be fully described by, it is clear that described embodiment be only a part of embodiment of the present invention rather than Whole embodiments.Based on the embodiment in the present invention, those of ordinary skill in the art are not making creation The every other embodiment obtained under property work premise, broadly falls into the scope of protection of the invention.
See Fig. 1, be the CDC mode that the present invention the provides structure composed figure that realizes increment extraction data method, This method that the present invention provides comprises the following steps: the first step, according to business scenario in data source and data The data base of target creates the database table of identical table structure, then in source data base, creates data manipulation Log buffer table.
Data source data database table structure is as follows:
CREATE TABLE INCR_TRANS_A
(
SNUM int NOT NULL PRIMARY KEY,
DATA_1 varchar (255),
DATA_2 varchar (255),
DATA_3 varchar (255),
DATA_4 varchar (255),
DATA_5 varchar (255),
DATE_F date NOT NULL
Create data base CDC monitoring table as follows:
A. creating CHANGE_SET for data source INCR_TRANS_A, source code is as follows:
B. creating CHANGE_TABLE for data source INCR_TRANS_A, source code is as follows:
Database source table (INCR_TRANS_A), object table (INCR_TRANS_B), data manipulation day After will cache table (INCR_TRANS_A_CDC) has created, when database source table (INCR_TRANS_A), after in table, data change, delta data can be stored by Oracel data base In data manipulation log buffer table (INCR_TRANS_A_CDC), CDC table can record data simultaneously Transformation period and data variation kind (I: insert, UU: amendment old value, the new value of UN: amendment, D: delete) etc. Information.
Second step, as in figure 2 it is shown, create dynamic sql rule, it is achieved from the CDC table of source, will [TIMESTAMP $] field in table, as increment foundation, extracts incremental data, sets increment foundation afterwards, Ensure that the data of extraction every time do not repeat extraction;Variable expression is: TIMESTAMP $ > To_date (' # ', ' yyyy-mm-dd hh24miss ') and TIMESTAMP $ <= To_date (' # ', ' yyyy-mm-dd hh24miss ')
3rd step, as shown in Figure 3 and Figure 4, creates Data Integration service, by data manipulation log buffer table Data incrementally foundation and data variation mode update in target.The Data Integration clothes of data interchange platform In business, data source is data manipulation log buffer table, sets change monitoring rule, i.e. according to data manipulation daily record Data variation kind (I: insert, UU: amendment old value, the new value of UN: amendment, D: delete) in cache table is come Distinguish data manipulation species, simultaneously according to the dynamic sql rule set, it is intended that according to field (timestamp) And initial value.
4th step, by the Data Integration service arrangement that creates to the node at table place, source, and sets timing Operation task carrys out timing extraction data.When database source table (INCR_TRANS_A) data send change, Data manipulation log buffer table stores source table delta data automatically.The timed task of data interchange platform starts Time can call the Data Integration service created, data manipulation log buffer table stores new delta data Time, the temporal information in configuration file can be read as according to by the change number in data manipulation log buffer table According to extracting, and store this operating time be used as next time extraction foundation.Data in this step are whole Conjunction service on-demand manual triggering operation can also carry out extracted data.
The data extracted are passed through the batch data bus transfer of data interchange platform to target by the 5th step Node, destination node is by respective change information updating to object table and store.
By above-mentioned operation, database source table data can synchronized update be in object table in time, accurately.
Specifically, the present invention is that change monitoring device CDC table based on oracle database carries out incremental data Extraction, mainly includes the foundation of data manipulation log buffer table, arranges with in data manipulation log buffer table Timestamp is for according to coming periodically or carrying out data pick-up by hand, doing corresponding increment processed by extraction Delta data is supplied to target database, all can set up data exchange node at source table and object table end and complete Corresponding operation, is managed by unified data interchange platform.
The foundation of data manipulation log buffer table is to ensure that source table is identical with the structure of object table, at source table end Set up the data manipulation log buffer table that source table is corresponding, data manipulation log buffer table adds timestamp Field is as the foundation of extraction.
In data interchange platform, set up the service of extracted data, be deployed to the data exchange node of source, hands The when of work or this service of clocked flip, the time catch cropping of last extraction can be read in from local profile For according to obtaining corresponding information from data manipulation log buffer table and being obtained by corresponding increment processed To delta data, the time of this extraction can be noted down after terminating as the foundation extracted next time.
The incremental data extracted by source data exchange node is total by the batch data of data interchange platform Line is transferred to the data exchange node at object table place, and completes the increment synchronization work of data.
The invention belongs to computer network and database technology application, it is provided that supplied complete set feasible Carry out the solution of incremental data synchronization to target database table from source database table;Secondly, employ CDC technology carries out increment extraction, it is not necessary to amendment source list data structure, is improving system performance Decrease volume of transmitted data simultaneously, alleviate offered load.
One of ordinary skill in the art will appreciate that all or part of flow process realizing in above-described embodiment method, Can be by computer program and complete to instruct relevant hardware, described program can be stored in a calculating In machine read/write memory medium, this program is upon execution, it may include such as the flow process of the embodiment of above-mentioned each method. Wherein, described storage medium can be magnetic disc, CD, read-only store-memory body (Read-Only Memory, Or random store-memory body (Random Access Memory, RAM) etc. ROM).
The above is the preferred embodiment of the present invention, it is noted that for the common skill of the art For art personnel, under the premise without departing from the principles of the invention, it is also possible to make some improvements and modifications, this A little improvements and modifications are also considered as protection scope of the present invention.

Claims (6)

1. the method realizing increment extraction data based on CDC mode, it is characterised in that include following Step: step one, creates identical table structure according to business scenario in the data source data base with datum target Database table, then in source data base create data manipulation log buffer table;
Step 2, creates dynamic sql rule, it is achieved from the CDC table of source, by table [TIMESTAMP $] field, as increment foundation, extracts incremental data;
Step 3, create Data Integration service, by data manipulation log buffer table data incrementally according to and Data variation mode updates in target;In the Data Integration service of data interchange platform, data source is data behaviour Make log buffer table, set change monitoring rule;
Step 4, by the Data Integration service arrangement that creates to the node at table place, source, timing extraction number According to;When database source table data send change, storage source table change automatically in data manipulation log buffer table Data;The timed task of data interchange platform can call the Data Integration service created, data at first When Operation Log cache table stores new delta data, the temporal information in configuration file can be read as depending on Extract according to by the delta data in data manipulation log buffer table, and store this operating time and be used as The foundation of extraction next time;
The data extracted are passed through the batch data bus transfer of data interchange platform to target by step 5 Node, destination node is by respective change information updating to object table and store.
A kind of method realizing increment extraction data based on CDC mode, its It is characterised by, in described step 2, after extraction incremental data, also includes setting increment foundation, and every time The data of extraction do not repeat.
A kind of method realizing increment extraction data based on CDC mode, its Being characterised by, described step 3 kind sets change monitoring rule i.e. according to the number in data manipulation log buffer table Data manipulation species are distinguished, simultaneously according to the dynamic sql rule set, it is intended that according to word according to change kind Section and initial value.
A kind of method realizing increment extraction data based on CDC mode, its It is characterised by, described distinguishes data manipulation kind according to the data variation kind in data manipulation log buffer table In not, described data variation kind includes I: insert, UU: amendment old value, the new value of UN: amendment, D: delete.
A kind of method realizing increment extraction data based on CDC mode, its Being characterised by, in the Data Integration service in described step 4, the mode of timing extraction data is for setting timing Operation task timing extraction data or on-demand manual triggering operation carry out extracted data.
A kind of method realizing increment extraction data based on CDC mode, its Being characterised by, the foundation of data manipulation log buffer table described in described step one is to ensure source table and target The structure of table is identical, sets up, at source table end, the data manipulation log buffer table that source table is corresponding, in data manipulation day Will cache table adds the timestamp field foundation as extraction.
CN201610271011.2A 2016-04-25 2016-04-25 Method for realizing incremental data extract based on CDC (Change Data Capture) mode Pending CN105975502A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201610271011.2A CN105975502A (en) 2016-04-25 2016-04-25 Method for realizing incremental data extract based on CDC (Change Data Capture) mode

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201610271011.2A CN105975502A (en) 2016-04-25 2016-04-25 Method for realizing incremental data extract based on CDC (Change Data Capture) mode

Publications (1)

Publication Number Publication Date
CN105975502A true CN105975502A (en) 2016-09-28

Family

ID=56994019

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201610271011.2A Pending CN105975502A (en) 2016-04-25 2016-04-25 Method for realizing incremental data extract based on CDC (Change Data Capture) mode

Country Status (1)

Country Link
CN (1) CN105975502A (en)

Cited By (12)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106383906A (en) * 2016-09-30 2017-02-08 杭州数梦工场科技有限公司 Method and system of optimizing data increment capture of Oracle database
CN108121730A (en) * 2016-11-30 2018-06-05 北京京东尚科信息技术有限公司 A kind of device and method by data update Fast synchronization to operation system
CN108197268A (en) * 2018-01-04 2018-06-22 国网福建省电力有限公司 Data center analyzes numeric field data Synchronous fluorimetry method for improving
CN109885448A (en) * 2018-12-29 2019-06-14 中兴智能交通股份有限公司 A kind of database data increment method for real-time monitoring
CN110019512A (en) * 2017-10-11 2019-07-16 北京国双科技有限公司 A kind of data processing method and device
CN110019477A (en) * 2017-12-27 2019-07-16 航天信息股份有限公司 A kind of method and system carrying out big data processing using HIVE backup table
CN110162461A (en) * 2019-04-16 2019-08-23 平安科技(深圳)有限公司 Data synchronize verification method, device and computer equipment
CN110222121A (en) * 2019-06-14 2019-09-10 浪潮软件股份有限公司 A kind of SQL Server database increment synchronization realization method and system based on CDC mode
CN111198911A (en) * 2018-11-19 2020-05-26 珠海格力电器股份有限公司 Data extraction increment interval acquisition method and data extraction method
CN112069256A (en) * 2020-08-27 2020-12-11 苏州浪潮智能科技有限公司 Data synchronization device on server cluster and synchronization method thereof
CN112527922A (en) * 2020-12-15 2021-03-19 神州数码信息系统有限公司 Data warehouse incremental processing method based on invariant model
WO2021174696A1 (en) * 2020-03-06 2021-09-10 平安科技(深圳)有限公司 Data updating method and apparatus, computer device, and storage medium

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101645072A (en) * 2009-08-25 2010-02-10 山东中创软件商用中间件股份有限公司 Changed data extracting method realized by being based on Oracle CDC technique
CN101923566A (en) * 2010-06-24 2010-12-22 浙江协同数据系统有限公司 Data increment extraction method based on trigger
CN102915336A (en) * 2012-09-18 2013-02-06 北京金和软件股份有限公司 Incremental data capturing and extraction method based on timestamps and logs
CN103823797A (en) * 2012-11-16 2014-05-28 镇江诺尼基智能技术有限公司 FTP (file transfer protocol) based real-time industry database data synchronization system

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101645072A (en) * 2009-08-25 2010-02-10 山东中创软件商用中间件股份有限公司 Changed data extracting method realized by being based on Oracle CDC technique
CN101923566A (en) * 2010-06-24 2010-12-22 浙江协同数据系统有限公司 Data increment extraction method based on trigger
CN102915336A (en) * 2012-09-18 2013-02-06 北京金和软件股份有限公司 Incremental data capturing and extraction method based on timestamps and logs
CN103823797A (en) * 2012-11-16 2014-05-28 镇江诺尼基智能技术有限公司 FTP (file transfer protocol) based real-time industry database data synchronization system

Non-Patent Citations (3)

* Cited by examiner, † Cited by third party
Title
吴秀君: ""基于ORACLE CDC技术实现数据自动同步"", 《电脑知识与技术》 *
李晓波 等: "《科学数据共享关键技术》", 30 November 2007 *
谭学清 等: "《商务智能》", 31 October 2006 *

Cited By (15)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106383906A (en) * 2016-09-30 2017-02-08 杭州数梦工场科技有限公司 Method and system of optimizing data increment capture of Oracle database
CN108121730A (en) * 2016-11-30 2018-06-05 北京京东尚科信息技术有限公司 A kind of device and method by data update Fast synchronization to operation system
CN108121730B (en) * 2016-11-30 2021-01-26 北京京东尚科信息技术有限公司 Device and method for quickly synchronizing data update to service system
CN110019512A (en) * 2017-10-11 2019-07-16 北京国双科技有限公司 A kind of data processing method and device
CN110019477A (en) * 2017-12-27 2019-07-16 航天信息股份有限公司 A kind of method and system carrying out big data processing using HIVE backup table
CN108197268A (en) * 2018-01-04 2018-06-22 国网福建省电力有限公司 Data center analyzes numeric field data Synchronous fluorimetry method for improving
CN111198911A (en) * 2018-11-19 2020-05-26 珠海格力电器股份有限公司 Data extraction increment interval acquisition method and data extraction method
CN109885448A (en) * 2018-12-29 2019-06-14 中兴智能交通股份有限公司 A kind of database data increment method for real-time monitoring
CN110162461A (en) * 2019-04-16 2019-08-23 平安科技(深圳)有限公司 Data synchronize verification method, device and computer equipment
CN110162461B (en) * 2019-04-16 2023-08-15 平安科技(深圳)有限公司 Data synchronization verification method and device and computer equipment
CN110222121A (en) * 2019-06-14 2019-09-10 浪潮软件股份有限公司 A kind of SQL Server database increment synchronization realization method and system based on CDC mode
WO2021174696A1 (en) * 2020-03-06 2021-09-10 平安科技(深圳)有限公司 Data updating method and apparatus, computer device, and storage medium
CN112069256A (en) * 2020-08-27 2020-12-11 苏州浪潮智能科技有限公司 Data synchronization device on server cluster and synchronization method thereof
CN112527922A (en) * 2020-12-15 2021-03-19 神州数码信息系统有限公司 Data warehouse incremental processing method based on invariant model
CN112527922B (en) * 2020-12-15 2023-11-24 神州数码信息系统有限公司 Data warehouse incremental processing method based on invariant model

Similar Documents

Publication Publication Date Title
CN105975502A (en) Method for realizing incremental data extract based on CDC (Change Data Capture) mode
CN102906751B (en) A kind of method of data storage, data query and device
CN110879813B (en) Binary log analysis-based MySQL database increment synchronization implementation method
CN103345521B (en) A kind of method and apparatus processing key assignments in Hash table database
CN105989076A (en) Data statistical method and device
CN105701218B (en) Realize that different terminals carry out the synchronous method of data on the database
CN106164865A (en) Affairs batch processing for the dependency perception that data replicate
CN101944114A (en) Data synchronization method between memory database and physical database
CN104636477B (en) The De-weight method of push list before a kind of information push
CN103970833A (en) Method for achieving two-way synchronous data circulation in heterogeneous database synchronizing system based on logs
CN106503158A (en) Method of data synchronization and device
CN108536745A (en) Tables of data extracting method, terminal, equipment and storage medium based on Shell
CN103327124A (en) File synchronization method, file synchronization method device, client, server side and equipment
CN103327123A (en) File synchronization method, client, server side and equipment
CN109522315A (en) A kind of data base processing method and system
CN105740410A (en) Data statistics method based on Hbase secondary index
CN106777387A (en) A kind of Internet of Things big data access method based on HBase
CN109873857A (en) Mobile terminal method of data synchronization and device under a kind of no network state
CN107870982A (en) Data processing method, system and computer-readable recording medium
CN104461929B (en) Distributed data cache method based on blocker
CN104504030A (en) Indexing method for power dispatching automation mass messages
KR20190063835A (en) System for processing real-time data modification of in-memory database
CN105447168A (en) Method for restoring and recombining fragmented files in MP4 format
CN112835918A (en) MySQL database increment synchronization implementation method
CN107403012A (en) A kind of method for interchanging data and device

Legal Events

Date Code Title Description
C06 Publication
PB01 Publication
C10 Entry into substantive examination
SE01 Entry into force of request for substantive examination
DD01 Delivery of document by public notice

Addressee: You Huimin

Document name: Notice of reexamination

DD01 Delivery of document by public notice
DD01 Delivery of document by public notice

Addressee: You Huimin

Document name: Notice of Closing the Reexamination Case

DD01 Delivery of document by public notice
RJ01 Rejection of invention patent application after publication

Application publication date: 20160928

RJ01 Rejection of invention patent application after publication