CN100562874C - A kind of increment data capturing method and system - Google Patents
A kind of increment data capturing method and system Download PDFInfo
- Publication number
- CN100562874C CN100562874C CNB2007101959824A CN200710195982A CN100562874C CN 100562874 C CN100562874 C CN 100562874C CN B2007101959824 A CNB2007101959824 A CN B2007101959824A CN 200710195982 A CN200710195982 A CN 200710195982A CN 100562874 C CN100562874 C CN 100562874C
- Authority
- CN
- China
- Prior art keywords
- snapshot
- data
- increment
- incremental
- cdc
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Expired - Fee Related
Links
Images
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a kind of increment data capturing method, comprising: the structural information according to object table is created the relational database snapshot table corresponding with object table; The snapshot that utilizes the snapshot table of described establishment to finish fixed time point target table extracts; Snapshot according to described different time points carries out the snapshot comparison, catches incremental data.Adopt snapshot table storage snapshot, do not need to safeguard the metadata information of incremental data, strengthened versatility.Further, the present invention has reduced the time of snapshot record comparison by creating the gathering index of snapshot table, has reduced the operand of computer system, has saved resource for computer system, has improved the efficient of catching incremental data.The invention also discloses a kind of increment data capturing system.
Description
Technical field
The present invention relates to database technical field, particularly relate to a kind of increment data capturing method and system in the integrated technical field of database data.
Background technology
(Change Data Capture CDC) is meant and catches the incremental data that produces in the data source, and offers the process of other enterprise application softwares increment data capturing.CDC is one of ETL in the data integration (ExtractTransform Load) process subject matter to be solved.CDC is used for catching that the Service Database data are inserted, deletion, upgrade operation, provides the increment data capturing service synchronously for enterprise application data database datas such as olap database, report database, data warehouse, business intelligence data storehouses.
Snapshot (Snapshot) comparison mode is one of main mode of increment data capturing in the prior art, obtains incremental data by the snapshot of comparing two time points.This mode highly versatile reads under the authority condition and can finish the increment data capturing process obtaining data source data.But snapshot comparison method of the prior art is to be storage snapshot in basis with the file, and obtains incremental data by two snapshot documents of design snapshot alignment algorithm comparison.The problem that this mode exists mainly contains:
1, higher based on the time and the space complexity of the snapshot alignment algorithm of file.
Need to travel through all records in two snapshot documents inevitably based on the snapshot alignment algorithm of file, this algorithm will be very consuming time under the bigger situation of snapshot data amount.Suppose to have two snapshot A and B, have n1 bar record among the snapshot A, have n2 bar record among the snapshot B, two record comparison time are t (ms), and the time of then finishing all snapshot record comparisons is with regard to needs: n1*n2*t (ms).Also comprise a large amount of IO operations in the algorithm, make consuming time even more serious; And if it is consuming time to reduce the minimizing of IO number of operations, just need take a large amount of memory headrooms.Under the situation limited at internal memory, that the snapshot data amount is bigger, algorithm is carried out efficient can be subjected to very big influence.
2, need the maintenance data base metadata information, limited the versatility of algorithm to a certain extent.
The data that the data pick-up of relational database inner structure form will be caused data of description to the data of this non-structure of file or semi-structured form are losing of metadata (data type, main external key etc.) just.Therefore, be that the increment data capturing of snapshot must need this part metadata of losing of additional maintenance with the file, increased the complexity of snapshot comparison increment extraction.And there is deviation in the definition to metadata between the distinct type data-base, and the data of different types storehouse is provided support just needs expansion algorithm, has therefore limited the versatility of algorithm to a certain extent.
3, do not provide the restore funcitons behind the incremental update of destination data storehouse.
After to destination data storehouse incremental update, do not provide the function that database recovery is arrived designated increments point update time.This function will be recovered to offer help to the incremental update historical query and the data of data in the destination data storehouse.
Summary of the invention
Technical matters to be solved by this invention provides a kind of increment data capturing method and system, does not need to safeguard metadata information, and improves the efficient based on snapshot mode increment data capturing, and the restore funcitons that incremental data is provided.
In order to address the above problem, the invention discloses a kind of increment data capturing method, comprising:
Structural information according to object table is created the relational database snapshot table corresponding with object table;
The snapshot that utilizes the snapshot table of described establishment to finish fixed time point target table extracts;
Snapshot according to described different time points carries out the snapshot comparison, catches incremental data.
Further, also comprise:
Create the gathering index of historical snapshot.
And, utilize the gathering index of described historical snapshot and current snapshot to compare, catch incremental data.
Further, also comprise:
Send described incremental data of catching to target database, incremental update target database.
Further, also comprise:
Create CDC formation and CDC table, described CDC formation is used to write down current incremental data set of catching, and described CDC table is used to write down the incremental data set that all time points are caught;
Generate the data of CDC table according to the data of CDC formation.
Further, also comprise:
From the CDC table, obtain the increment recovery data in the fixed time section;
Send the described increment recovery data of obtaining to target database;
Generate corresponding SQL statement according to described increment recovery data;
Carry out described SQL statement, finish the incremental data of target database fixed time point is recovered.
Wherein, utilize full the connection and the mode fulfillment database snapshot comparison of condition query and catching of incremental data in the relational database.
Wherein, described snapshot table be identical with object table or can complete compatible object table in the relational database list structure of all data.
Wherein, the step of described incremental update target database comprises:
Target database receives described increment and catches data;
The action type of catching data according to increment generates corresponding SQL statement;
Carry out described SQL statement and upgrade target database.
The invention also discloses a kind of increment data capturing system, comprising:
Snapshot table creating unit is used for creating the relational database snapshot table corresponding with object table according to the structural information of object table;
The snapshot extracting unit, the snapshot that is used to utilize the snapshot table to finish fixed time point target table extracts;
Capturing unit is used for carrying out the snapshot comparison according to the snapshot of different time points, catches incremental data.
Further, also comprise:
The index creation unit is used to create the gathering index of snapshot.
Described capturing unit is compared according to the gathering index and the current snapshot of historical snapshot, catches incremental data.
Further, also comprise:
First transmitting element is used to send described incremental data of catching to target database;
Updating block is used for incremental update target database.
Further, also comprise:
The formation creating unit is used to create the CDC formation, and described CDC formation is used to write down current incremental data set of catching;
The CDC table generates the unit, is used to generate the CDC table, and described CDC table is used to write down the incremental data set that all time points are caught.
Further, also comprise:
Acquiring unit is used for obtaining increment recovery data in the fixed time section from CDC table;
Second transmitting element is used to send the described increment recovery data of obtaining to target database;
The statement generation unit is used for generating corresponding SQL statement according to described increment recovery data;
Recovery unit is used to carry out described SQL statement, finishes the incremental data of target database fixed time point is recovered.
Compared with prior art, the present invention has the following advantages:
Because the present invention does not adopt the file storage snapshot, but adopt snapshot table storage snapshot, described snapshot table be identical with object table or can complete compatible object table in the relational database list structure of all data, therefore the problem that does not exist metadata information to lose does not need to safeguard the metadata information of incremental data.And owing to adopted the snapshot table storage snapshot of relational database list structure, therefore can avoid defining the process of efficient snapshot alignment algorithm, strengthen the versatility of described method by the full connected mode in the relational database.Adopt relational database list structure storage snapshot, reduced time and space complexity.
Further, the present invention has reduced the time of snapshot record comparison by creating the gathering index of snapshot table, has reduced the operand of computer system, has saved resource for computer system, has improved the efficient of catching incremental data.
Further, the present invention obtains the increment recovery data by CDC table, and the increment recovery data are sent to the mode that target database is carried out, and has realized the maintenance of incremental data and to the increment recovery function of target database.
Description of drawings
Fig. 1 is the process flow diagram of a kind of increment data capturing method first embodiment of the present invention;
Fig. 2 is the process flow diagram of a kind of increment data capturing method second embodiment of the present invention;
Fig. 3 is the process flow diagram of incremental update target database step in the second embodiment of the invention;
Fig. 4 is the process flow diagram that a kind of increment data capturing method the 3rd embodiment incremental data of the present invention is recovered;
Fig. 5 is the structured flowchart of a kind of increment data capturing of the present invention first embodiment of system;
Fig. 6 is the structured flowchart of a kind of increment data capturing of the present invention second embodiment of system;
Fig. 7 is the structured flowchart that a kind of increment data capturing of the present invention system the 3rd embodiment incremental data is recovered.
Embodiment
For above-mentioned purpose of the present invention, feature and advantage can be become apparent more, the present invention is further detailed explanation below in conjunction with the drawings and specific embodiments.
With reference to Fig. 1, show the process flow diagram of a kind of increment data capturing method first embodiment of the present invention, comprise step:
Described snapshot table be identical with object table or can complete compatible object table in the relational database list structure of all data.
The detailed process of creating described snapshot table is: at first obtain the structural information that increment is caught object table, then according to the SQL CREATE TABLE statement of described structural information tectonic relationship database table, in database, carry out described SQL statement at last, finish the establishment of described relational database snapshot table.
In order to finish snapshot comparison to different time points, need carry out snapshot to the data of a plurality of different time points of object table and extract, the data that described different time points is extracted are stored in respectively in the snapshot table, form a plurality of snapshot records of different time points.
Further, in order to reduce the time of snapshot comparison, after extracting generation object table data generation snapshot, the preferred mode of the present invention is to create the gathering index of historical snapshot.Index is a kind of structure that the row or the value of multiple row in the database table are sorted.In assembling index, the physical sequential of row is identical with logic (index) order of key assignments in the table.If certain index is not to assemble index, then the physical sequential of row and the logical order of key assignments do not match in the table.Compare with nonclustered index, assembling index provides faster data access queries speed.
Under the situation of use assembling index, if comprise n1 bar record in the snapshot, the number of times that then inquires this record need be inquired about n1 time at most, assembles index and use, no matter then record how many bars in the snapshot all only needs inquire about 1 time.
Utilize the mode of full connection in the relational database and condition query to realize the snapshot comparison and to inserting, deletion is upgraded the calculating of operation incremental data and is caught.
The full connection is a kind of of concatenation operation, and it is because of mismatch information dropout to take place when avoiding nature to connect.Suppose that certain row in the side table that participate in to connect does not mate with any delegation during the opposing party who is connected shows, this is about to mate a value and is the row of null value entirely.
Full concatenation operation among the embodiment of the present invention is that the identical snapshot of two structures that two moment are done equates to be that condition connects entirely with major key field, when having the major key that does not mate in two tables, the guild at this major key place and a value are complementary for the row of null value entirely like this.
Major key is to guarantee the unique field row of record in the relation database table.External key is the row of other table Major keys of setting recording for data in related other tables.The major key in the database table and the relation of other non-major keys are by the normal form constraint.Introduce normal form below: design pattern (normal form, database Normal Formula, the design pattern of database) is the set that meets a certain other relation schema of level.Constructs database must be followed certain rule.In relational database, this rule is exactly a normal form.Relation in the relational database must satisfy certain requirement, promptly satisfies different normal forms.Relational database has six kinds of normal forms at present: first normal form (1NF), second normal form (2NF), third normal form (3NF), fourth normal form (4nf (4NF), the 5th normal form (5NF) and the 6th normal form (6NF).The normal form that satisfies minimum requirements is first normal form (1NF).That further satisfies more requirements on the basis of first normal form is called second normal form (2NF), and all the other normal forms are analogized in proper order.In general, database only need satisfy third normal form (3NF) just.
Embodiment of the present invention is owing to adopt to assemble index and use full connected mode to catch incremental data, and the algorithm of computing velocity in compared to existing technology is greatly improved.
Adopt the raising of assembling computing velocity behind the index with implementation below.
Suppose two snapshot Table As and table B, comprise n1 bar record in the Table A, comprise n2 bar record among the table B, the time of a record of comparison is t (ms).File mode snapshot of the prior art comparison needs each bar record and all the n2 bars records among the table B among the question blank A respectively and compares, so the T.T. of snapshot comparison is exactly n1*n2*t (ms).Utilized the gathering index in the embodiment of the invention, by creating the gathering index of historical snapshot Table A, the record queries comparison time of the wall scroll of inquiry from B table record and A table has been reduced to t from n1*t, thus the T.T. of snapshot comparison be exactly (n1+n2) * t (ms), speed improves obvious.
With reference to Fig. 2, show the process flow diagram of a kind of increment data capturing method second embodiment of the present invention.
In second embodiment of a kind of increment data capturing method of the present invention, step 101 in comprising first embodiment, step 102, the step 103, also comprise step 104, send described incremental data of catching to target database, incremental update target database.
Can incremental data be sent to target database by dual mode among the embodiment of the present invention, a kind of is to be sent to target database immediately after catching new incremental data, another kind of mode is to create the CDC formation, described CDC formation is used to write down current incremental data set of catching, and by described formation incremental data is sent to target database.In embodiments of the present invention, what preferably adopt is second kind, safe because send incremental data by formation, can not cause losing of incremental data.
After creating the CDC formation, also comprise step: generate the CDC table according to the target database list structure, described CDC table is used to write down the incremental data set that all time points are caught.Described CDC table is used for safeguarding incremental data.CDC table is caught by increment and is added timestamp field, action type field and updating mark bit field on the basis of object table structural information and constitute.
The constructive process of CDC table is similar to the constructive process of snapshot table, that is: at first obtain the structural information that increment is caught object table, then according to the SQL CREATETABLE statement of described structural information tectonic relationship database table, in database, carry out described SQL statement at last, finish the establishment of described relational database CDC table.The CDC table is that with the difference of snapshot table the CDC table also comprises timestamp field, action type field and updating mark bit field.
With reference to Fig. 3, show the process flow diagram of incremental update target database step in the second embodiment of the invention, comprise step:
Step 301, target database receive described increment and catch data.
Step 302, the action type of catching data according to increment generate corresponding SQL statement.
Step 303, the described SQL statement of execution are upgraded target database.
Below the technical scheme of embodiment of the present invention is carried out more detailed introduction.At first do T table (object table) snapshot and store snapshot into A table (historical snapshot), create the gathering index of snapshot Table A at time point A.And then be T at time point B and show snapshot and store snapshot into the B table.
Described increment data capturing flow process is specially: do deletion constantly between T Table A time point and the B time point, insert, upgrade in the acquisition procedure of operation incremental data, at first create and safeguard the CDC formation of this time catching incremental data set, then equate to be connected entirely under the condition data snapshot A and B at A table major key and B table major key, and according to: 1) B table major key field is empty; 2) A table major key field is empty; 3) A, B two table major key fields equate but other non-major key fields do not wait three conditions to inquire deletion respectively, insert and upgrade three kinds of operation incremental data set.Then according to: 1) catch deletion action data collection, and add DELETE action type field and timestamp field value from snapshot A; 2) catch insertion service data collection from snapshot B, and add INSERT action type field and timestamp field value; 3) catch and upgrade the service data collection from A, B two tables, add UPDATE action type field value and timestamp field, and derive from the more new data of A table and add OLD operational word segment value, derive from the more new data of B table and add NEW operational word segment value; Described incremental data set of catching is inserted in the CDC table, is that the renewal operation incremental data of NEW is pressed into the CDC formation with deletion, insertion and operational word segment value.
The Data Update flow process of incremental update target database is: at first empty current A snapshot table and specify the A table to be next snapshot storage list, avoided the waste of Computer Storage resource like this, and be that up-to-date B snapshot table is created the pack index, make it become historical snapshot.Then, to comprise this increment catches the CDC formation of data set and is sent to target database, target database receives after the described formation principle by first in first out and reads data set in the formation in proper order, and generate corresponding SQL statement respectively for three types according to DELETE, INSERT, the UPDATE of data centralization action type field, then target database is carried out described SQL statement, incremental update target database successively by DELETE operation order formerly.
The snapshot table of embodiment of the present invention adopt identical with object table or can complete compatible object table in the relational database list structure of all data, so problem that does not exist metadata information to lose, do not need to safeguard the metadata information of incremental data, improved the versatility of this increment data capturing method.
And owing to adopted the snapshot table storage snapshot of relational database list structure, therefore can avoid defining the process of efficient snapshot alignment algorithm, strengthen the versatility of described method by the full connected mode in the relational database.Adopt relational database list structure storage snapshot, time and space complexity have all reduced.
Further, the present invention has reduced the time of snapshot record comparison by creating the gathering index of snapshot table, has reduced the operand of computer system, has saved resource for computer system, has improved the efficient of catching incremental data.
With reference to Fig. 4, show the process flow diagram that a kind of increment data capturing method the 3rd embodiment incremental data of the present invention is recovered, comprise step:
Step 401, from CDC table, obtain the increment recovery data in the fixed time.
Step 402, the described increment recovery data of obtaining of transmission are to target database.
Step 403, generate corresponding SQL statement according to the type of described increment recovery data.
Step 404, carry out described SQL statement, finish the incremental data of target database fixed time point is recovered.
It is to make up data by inquiry CDC table to recover formation and formation is sent to the mode that target database is carried out that incremental data is recovered, and incremental update target database returns to the state of designated increments point update time.
Below described technical scheme is made a more detailed description:
At first in catching the CDC time point of generation, all previous increment chooses certain time point that the target database recovering state need be arrived, inquire about then and be later than this appointment point release time all time points afterwards in the CDC table, and according to the backward traversing result collection of the sequencing of time point and create and recover formation, afterwards according to: 1) the data set retouching operation that is operating as insertion is deletion; 2) take out the data set and the retouching operation that are operating as deletion and be insertion; 3) take out to be operating as and upgrade and the updating mark bit field is the data set of OLD and data are pressed into the recovery formation, will recover formation at last and be sent to target database execution increment recovery.After the recovery formation is sent to target database, according to the data set in the principle access queue of first in first out, action type field value according to data centralization is judged deletion, insert, upgrade three kinds of operations and generate corresponding SQL statement, carry out described SQL statement in target database, finish the incremental data of target database fixed time point is recovered.
Embodiment of the present invention obtains the increment recovery data by CDC table, and the increment recovery data are sent to the mode that target database is carried out, and has realized the maintenance of incremental data and to the increment recovery function of target database.
With reference to Fig. 5, show the structured flowchart of a kind of increment data capturing of the present invention first embodiment of system, comprising:
Snapshot table creating unit 501, be used for creating the relational database snapshot table corresponding with object table according to the structural information of object table.
Capturing unit 503, the snapshot that is used for according to different time points carry out the snapshot comparison, catch incremental data.
The course of work and principle of work to described system is described in detail below:
Snapshot table creating unit 501 is created the relational database snapshot table corresponding with object table according to the structural information of object table.Described snapshot table be identical with object table or can complete compatible object table in the relational database list structure of all data.The snapshot that snapshot extracting unit 502 is finished fixed time point target table according to the snapshot table extracts, and forms a plurality of snapshot records of different time points.The mode of interior full connection of relational database and condition query of utilizing capturing unit 503 realizes the snapshot comparison and to inserting, deletes that renewal is operated the calculating of incremental data and caught.
Further, also comprise index creation unit 504, be used to create the gathering index of snapshot.After increasing described index creation unit 504, utilize described index creation unit to create the gathering index of historical snapshot, thus described capturing unit 503 can compare according to the focusing index and the current snapshot of historical snapshot, catch incremental data.Gathering search index visit snapshot has been adopted in utilization, thereby significantly reduces the time of snapshot comparison, has improved the efficient of increment data capturing.
With reference to Fig. 6, show the structured flowchart of a kind of increment data capturing of the present invention second embodiment of system, snapshot table creating unit 501, snapshot extracting unit 502, capturing unit 503 and the index creation unit 504 in comprising first embodiment, also comprise:
First transmitting element 601, be used to send described incremental data of catching to target database.
Updating block 602, be used for incremental update target database.
Described updating block 602 receives incremental data, generates corresponding SQL statement according to described incremental data, carries out described SQL statement incremental update target database.
Further, also comprise formation creating unit 603, be used to create the CDC formation, described CDC formation is used to write down the incremental data set of catching.Described incremental data of catching is sent to target database with the CDC formation, and security is higher.
The CDC table generates unit 604, is used for generating the CDC table according to described CDC formation.Described CDC table also comprises timestamp field, action type field and updating mark field except comprising data field.
With reference to Fig. 7, show the structured flowchart that a kind of increment data capturing of the present invention system the 3rd embodiment incremental data is recovered, comprising:
Acquiring unit 701, be used for obtaining increment recovery data in the fixed time from CDC table.
Second transmitting element 702, be used to send the described increment recovery data of obtaining to target database.
Each system embodiment of the present invention is corresponding with each method embodiment, and therefore, the part of not describing in detail in each system embodiment gets final product referring to the appropriate section among each method embodiment.
Need to prove, for aforesaid each method embodiment, for simple description, so it all is expressed as a series of combination of actions, but those skilled in the art should know, the present invention is not subjected to the restriction of described sequence of movement, because according to the present invention, some step can adopt other orders or carry out simultaneously.Secondly, those skilled in the art also should know, the embodiment described in the instructions all belongs to preferred embodiment, and related action and module might not be that the present invention is necessary.
In the above-described embodiments, the description of each embodiment is all emphasized particularly on different fields, do not have the part that describes in detail among certain embodiment, can be referring to the associated description of other embodiment.
More than to a kind of increment data capturing method provided by the present invention and system, be described in detail, used specific case herein principle of the present invention and embodiment are set forth, the explanation of above embodiment just is used for helping to understand method of the present invention and core concept thereof; Simultaneously, for one of ordinary skill in the art, according to thought of the present invention, the part that all can change in specific embodiments and applications, in sum, this description should not be construed as limitation of the present invention.
Claims (17)
1, a kind of increment data capturing method is characterized in that, comprising:
Structural information according to object table is created the relational database snapshot table corresponding with object table;
The snapshot that utilizes the snapshot table of described establishment to finish fixed time point target table extracts;
Snapshot according to described different time points carries out the snapshot comparison, catches incremental data.
2, increment data capturing method as claimed in claim 1 is characterized in that, also comprises:
Create the gathering index of historical snapshot.
3, increment data capturing method as claimed in claim 2 is characterized in that:
Utilize the gathering index and the current snapshot of described historical snapshot to compare, catch incremental data.
4, increment data capturing method as claimed in claim 3 is characterized in that, also comprises:
Send described incremental data of catching to target database, incremental update target database.
5, increment data capturing method as claimed in claim 4 is characterized in that, also comprises:
Create CDC formation and CDC table, described CDC formation is used to write down current incremental data set of catching, and described CDC table is used to write down the incremental data set that all time points are caught;
Generate the data of CDC table according to the data of CDC formation.
6, increment data capturing method as claimed in claim 5 is characterized in that, also comprises:
From the CDC table, obtain the increment recovery data in the fixed time section;
Send the described increment recovery data of obtaining to target database;
Generate corresponding SQL statement according to described increment recovery data;
Carry out described SQL statement, finish the incremental data of target database fixed time point is recovered.
7, as the arbitrary described increment data capturing method of claim 1 to 6, it is characterized in that:
Utilize full the connection and the mode fulfillment database snapshot comparison of condition query and catching of incremental data in the relational database.
8, as the arbitrary described increment data capturing method of claim 1 to 6, it is characterized in that:
Described snapshot table be identical with object table or can complete compatible object table in the relational database list structure of all data.
9, increment data capturing method as claimed in claim 4 is characterized in that, the step of described incremental update target database comprises:
Target database receives described increment and catches data;
The action type of catching data according to increment generates corresponding SQL statement;
Carry out described SQL statement and upgrade target database.
10, a kind of increment data capturing system is characterized in that, comprising:
Snapshot table creating unit is used for creating the relational database snapshot table corresponding with object table according to the structural information of object table;
The snapshot extracting unit, the snapshot that is used to utilize the snapshot table to finish fixed time point target table extracts;
Capturing unit is used for carrying out the snapshot comparison according to the snapshot of different time points, catches incremental data.
11, increment data capturing as claimed in claim 10 system is characterized in that, also comprises:
The index creation unit is used to create the gathering index of snapshot.
12, increment data capturing as claimed in claim 11 system is characterized in that:
Described capturing unit is compared according to the gathering index and the current snapshot of historical snapshot, catches incremental data.
13, increment data capturing as claimed in claim 12 system is characterized in that, also comprises:
First transmitting element is used to send described incremental data of catching to target database;
Updating block is used for incremental update target database.
14, increment data capturing as claimed in claim 13 system is characterized in that, also comprises:
The formation creating unit is used to create the CDC formation, and described CDC formation is used to write down current incremental data set of catching;
The CDC table generates the unit, is used to generate the CDC table, and described CDC table is used to write down the incremental data set that all time points are caught.
15, increment data capturing as claimed in claim 14 system is characterized in that, also comprises:
Acquiring unit is used for obtaining increment recovery data in the fixed time section from CDC table;
Second transmitting element is used to send the described increment recovery data of obtaining to target database;
The statement generation unit is used for generating corresponding SQL statement according to described increment recovery data;
Recovery unit is used to carry out described SQL statement, finishes the incremental data of target database fixed time point is recovered.
16, as the described increment data capturing of the arbitrary claim of claim 10 to 15 system, it is characterized in that:
Described snapshot table be identical with object table or can complete compatible object table in the relational database list structure of all data.
17, as the described increment data capturing of the arbitrary claim of claim 10 to 15 system, it is characterized in that:
Described capturing unit utilizes full the connection and the mode fulfillment database snapshot comparison of condition query and catching of incremental data in the relational database.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CNB2007101959824A CN100562874C (en) | 2007-12-14 | 2007-12-14 | A kind of increment data capturing method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CNB2007101959824A CN100562874C (en) | 2007-12-14 | 2007-12-14 | A kind of increment data capturing method and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN101183387A CN101183387A (en) | 2008-05-21 |
| CN100562874C true CN100562874C (en) | 2009-11-25 |
Family
ID=39448662
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CNB2007101959824A Expired - Fee Related CN100562874C (en) | 2007-12-14 | 2007-12-14 | A kind of increment data capturing method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN100562874C (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101923566A (en) * | 2010-06-24 | 2010-12-22 | 浙江协同数据系统有限公司 | Data increment extraction method based on trigger |
Families Citing this family (39)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101719131B (en) * | 2009-12-01 | 2013-08-07 | 金蝶软件（中国）有限公司 | Method, device and system for processing report data |
| CN102262544B (en) * | 2010-05-24 | 2016-06-08 | 腾讯科技（深圳）有限公司 | The method and apparatus of software upgrading |
| CN102411569A (en) * | 2010-09-20 | 2012-04-11 | 上海众融信息技术有限公司 | Database conversion and cleaning information processing method |
| CN102419754A (en) * | 2010-09-27 | 2012-04-18 | 金蝶软件（中国）有限公司 | Method for generating primary key number of data during extraction, transformation and loading (ETL), and counter |
| CN102521397B (en) * | 2011-12-23 | 2013-10-23 | 山东中创软件工程股份有限公司 | Data transmission method |
| US9256004B2 (en) * | 2012-04-10 | 2016-02-09 | Honeywell International Inc. | System and method for transmitting differential weather information to an in-flight aircraft |
| CN102929745B (en) * | 2012-09-26 | 2015-02-18 | 东软集团股份有限公司 | Status data updating method and device for virtual equipments |
| CN103034566B (en) * | 2012-12-06 | 2015-07-22 | 华为技术有限公司 | Method and device for restoring virtual machine |
| CN103149855B (en) * | 2013-03-22 | 2015-09-16 | 上海港务工程公司 | Marine guidance piling long distance control system |
| CN103455560A (en) * | 2013-08-12 | 2013-12-18 | 曙光信息产业股份有限公司 | Data query method and system |
| CN103699629A (en) * | 2013-12-20 | 2014-04-02 | 华为技术有限公司 | Method and device for processing chart |
| CN104298760B (en) * | 2014-10-23 | 2019-02-05 | 北京京东尚科信息技术有限公司 | A kind of data processing method and data processing equipment applied to data warehouse |
| CN104462568B (en) * | 2014-12-26 | 2018-07-31 | 山东中创软件商用中间件股份有限公司 | A kind of data account checking method, device and system |
| CN104572995A (en) * | 2015-01-06 | 2015-04-29 | 北京卫星环境工程研究所 | Fast distributed real-time data processing method |
| CN106528575B (en) * | 2015-09-14 | 2019-08-20 | 北京国双科技有限公司 | Data interconnection method and device |
| CN106250265A (en) * | 2016-07-18 | 2016-12-21 | 乐视控股（北京）有限公司 | Data back up method and system for object storage |
| CN106383906B (en) * | 2016-09-30 | 2020-12-11 | 杭州数梦工场科技有限公司 | Method and system for optimizing Oracle database data increment capture |
| CN107957918B (en) | 2016-10-14 | 2019-05-10 | 腾讯科技（深圳）有限公司 | Data reconstruction method and device |
| CN106682153A (en) * | 2016-12-23 | 2017-05-17 | 山东浪潮商用系统有限公司 | Data extraction tool on basis of data modeling and data increment implementation |
| CN107749914B (en) * | 2017-10-13 | 2020-10-30 | 北京小米移动软件有限公司 | Method and device for acquiring stuck data |
| CN110019266B (en) * | 2017-10-30 | 2021-06-29 | 北京国双科技有限公司 | SQL statement generation method and device |
| CN108492566B (en) * | 2018-04-23 | 2019-10-29 | 泰华智慧产业集团股份有限公司 | High fault-tolerant transportation card is made a slip of the tongue the method and system that car data extracts in real time |
| CN109871378A (en) * | 2019-02-21 | 2019-06-11 | 杭州市商务委员会（杭州市粮食局） | The data acquisition and processing (DAP) method and system of big data platform |
| CN110351386B (en) * | 2019-07-23 | 2022-09-16 | 华云工业互联网有限公司 | Increment synchronization method and device between different copies |
| CN110674146B (en) * | 2019-08-22 | 2022-01-25 | 视联动力信息技术股份有限公司 | Data synchronization method, synchronization end, end to be synchronized, equipment and storage medium |
| CN110865996B (en) * | 2019-11-07 | 2022-04-19 | 尚娱软件（深圳）有限公司 | Data storage method, device and computer readable storage medium |
| CN111104445A (en) * | 2019-12-06 | 2020-05-05 | 杭州数梦工场科技有限公司 | Data synchronization method, device and equipment |
| CN111026760A (en) * | 2019-12-11 | 2020-04-17 | 宏图智能物流股份有限公司 | CDC data acquisition method based on multidimensional service time |
| CN111309673B (en) * | 2020-02-12 | 2023-06-23 | 普信恒业科技发展（北京）有限公司 | Snapshot data generation method and device for incremental data |
| CN111858158B (en) * | 2020-06-19 | 2023-11-10 | 北京金山云网络技术有限公司 | Data processing method and device and electronic equipment |
| CN111782514A (en) * | 2020-06-23 | 2020-10-16 | 京东数字科技控股有限公司 | Test data comparison method and device |
| CN112256523B (en) * | 2020-09-23 | 2023-01-06 | 贝壳技术有限公司 | Service data processing method and device |
| CN112445759B (en) * | 2020-11-30 | 2024-04-16 | 中国人寿保险股份有限公司 | Method and device for copying data across clusters of distributed database and electronic equipment |
| CN112527922B (en) * | 2020-12-15 | 2023-11-24 | 神州数码信息系统有限公司 | Data warehouse incremental processing method based on invariant model |
| CN112559536B (en) * | 2021-02-20 | 2021-06-01 | 北京工业大数据创新中心有限公司 | Industrial equipment data processing method and system |
| CN112988473B (en) * | 2021-05-10 | 2021-11-23 | 南京云信达科技有限公司 | Backup data real-time recovery method and system |
| CN113407538B (en) * | 2021-06-17 | 2023-03-10 | 北京计算机技术及应用研究所 | Incremental acquisition method for data of multi-source heterogeneous relational database |
| CN113672679A (en) * | 2021-07-02 | 2021-11-19 | 杭州沃趣科技股份有限公司 | Method and system for restoring historical table structure |
| CN114020534B (en) * | 2021-11-05 | 2022-05-10 | 中远海运科技股份有限公司 | Shipping data snapshot generating method |
-
2007
- 2007-12-14 CN CNB2007101959824A patent/CN100562874C/en not_active Expired - Fee Related
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101923566A (en) * | 2010-06-24 | 2010-12-22 | 浙江协同数据系统有限公司 | Data increment extraction method based on trigger |
Also Published As
| Publication number | Publication date |
|---|---|
| CN101183387A (en) | 2008-05-21 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN100562874C (en) | A kind of increment data capturing method and system | |
| CN106227800B (en) | Storage method and management system for highly-associated big data | |
| CN102033954B (en) | Full text retrieval inquiry index method for extensible markup language document in relational database | |
| CN101727478B (en) | Method and system for dynamically building and populating data marts with data stored in repositories | |
| US9892185B2 (en) | Method and system for syncing data structures | |
| CN102521303B (en) | A kind of single-table multi-column sequence storage method for a column database | |
| CN107958057A (en) | A kind of code generating method and device for being used for Data Migration in heterogeneous database | |
| CN103460208A (en) | Methods and systems for loading data into a temporal data warehouse | |
| CN102893281A (en) | Information retrieval device, information retrieval method, computer program, and data structure | |
| WO2020192064A1 (en) | Incremental data consistency implementation method and device | |
| CN102819585B (en) | Method for controlling document of extensive makeup language (XML) database | |
| CN102646070A (en) | Space-time trajectory data storage method based on area | |
| CN103853820A (en) | Data processing method and data processing system | |
| CN101706803A (en) | Method for migrating production data of weaving workshop | |
| CN103823823A (en) | Denormalization strategy selection method based on frequent item set mining algorithm | |
| CN102456053A (en) | Method for mapping XML document to database | |
| CN105224527A (en) | Be applicable to the general ETL method of multiple object table update mode | |
| CN105279285A (en) | Synchronization system and method of relation-type database and non-relation-type database | |
| CN102346744A (en) | Device for processing materialized table in multi-tenancy (MT) application system | |
| CN102375827A (en) | Method for fast loading versioned electricity network model database | |
| CN107526746A (en) | The method and apparatus of management document index | |
| US9037553B2 (en) | System and method for efficient maintenance of indexes for XML files | |
| CN112231321A (en) | Oracle secondary index and index real-time synchronization method | |
| CN101963993B (en) | Method for fast searching database sheet table record | |
| AL-Msie'deen et al. | Detecting commonality and variability in use-case diagram variants |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant | ||
| CF01 | Termination of patent right due to non-payment of annual fee | ||
| CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20091125 Termination date: 20171214 |