CN107025242A - The clone method and device of data between database - Google Patents
The clone method and device of data between database Download PDFInfo
- Publication number
- CN107025242A CN107025242A CN201610073070.9A CN201610073070A CN107025242A CN 107025242 A CN107025242 A CN 107025242A CN 201610073070 A CN201610073070 A CN 201610073070A CN 107025242 A CN107025242 A CN 107025242A
- Authority
- CN
- China
- Prior art keywords
- sequence number
- log
- transaction sequence
- target database
- database
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Computing Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides the clone method and device of data between a kind of database, wherein, this method includes：Target database gathers the log recording in source database according to preset rules；Wherein, preset rules include：The Transaction Sequence number of log recording is more than the Transaction Sequence number corresponding to the data being synchronized in target database；Target database copies to log recording in the object table in target database；Wherein, target database and source database are all heterogeneous database.By the present invention, the problem of incremental data storehouse clone method in correlation technique depends on the type of concrete database is solved.
Description
Technical field
The present invention relates to the communications field, in particular to the clone method and device of data between a kind of database.
Background technology
In the application such as disaster tolerance, data backup, statistics, it is frequently encountered database and replicates problem, more than millions
Big table duplication, due to performance, typically using incremental replication.The database duplication side commonly used in correlation technique
The stream (Stream) that method has oracle database is replicated and GoldenGate, and both approaches are all based on source database day
The excavation of will file obtains synchronizing information, and the advantage of this method is transparent to application, have the disadvantage dependent on specific number
Need to take more machine resources according to the excavation in storehouse, and journal file, it is larger to the loading effects of source data.Other one
Plant the Materialized View that conventional database copy method is Oracle.Materialized View can also provide incremental refreshment and ensure data
Uniformity, but primary disadvantage is that can not be used in isomeric data lab environment, and data unanimously need between realizing different tables
A group refreshing is used, and organizing refreshing needs lock table, is not suitable for source database operation Transaction Processing (On-line
Transaction processing, abbreviation OLTP) type application and trading volume per second very high scene.
The problem of type of concrete database being depended on for the incremental data storehouse clone method in correlation technique, at present not yet
Propose effective solution.
The content of the invention
The invention provides the clone method and device of data between a kind of database, at least to solve the increment in correlation technique
The problem of database copy method depends on the type of concrete database.
According to an aspect of the invention, there is provided between a kind of database data clone method, including：Target database
The log recording in source database is gathered according to preset rules；Wherein, preset rules include：The transaction sequence of log recording
Number it is more than the Transaction Sequence number being synchronized to corresponding to data in target database；Target database replicates log recording
In object table into target database；Wherein, target database and source database are all heterogeneous database.
Further, before target database copies to log recording in the object table in target database, method is also
Including：Target database obtains the Transaction Sequence number corresponding to all log recordings of this collection of target database；According to
Pre-defined rule selects a Transaction Sequence number conduct from the Transaction Sequence number corresponding to all log recordings that this is gathered
The Transaction Sequence number of this collection.
Further, one is selected from the Transaction Sequence number corresponding to all log recordings that this is gathered according to pre-defined rule
The things sequence number that individual Transaction Sequence number is gathered as this includes：For the day where each log recording of this collection
Will table, target database obtains the maximum transaction sequence number in the corresponding Transaction Sequence number of all log recordings in log sheet；
The affairs that target database gathers the minimum Transaction Sequence number in the maximum transaction sequence number of all log sheets as this
Sequence number.
Further, after target database copies to log recording in the object table in target database, method is also
Including：Target database records the Transaction Sequence number of this collection.
Further, target database includes the object table that log recording is copied in target database：For each
Log sheet where log recording, target database obtains Transaction Sequence number in log sheet according to the corresponding major key of log sheet
Maximum log recording；Target database copies to the maximum log recording of Transaction Sequence number in log sheet in object table.
Further, the maximum log recording of Transaction Sequence number in log sheet is copied to object table and included by target database：
Target database searches the corresponding object table of major key according to the major key of log sheet；Target database is according to affairs in log sheet
The action type and the major key of log sheet of the maximum log recording of sequence number, are operated to object table.
Further, action type and daily record of the target database according to the maximum log recording of Transaction Sequence number in log sheet
The major key of table, carrying out operation to object table includes at least one of：When action type is deletion action, target data
Storehouse record corresponding with the major key of log sheet in the major key delete target table according to log sheet；It is newly-increased operation in action type
Or modification operation, and it is not present in object table with the major key of log sheet during corresponding record, target database is by daily record
The content of record is added in object table as new record；It is that newly-increased operation or modification are operated in action type, and
Existing with the major key of log sheet during corresponding record in object table, target database is with the content update target of log recording
Record corresponding with the major key of log sheet in table.
Further, target database includes the object table that log recording is copied in target database：Target data
Plus 1 with the Transaction Sequence number corresponding to the data that have been synchronized in target database in target database for starting point affairs sequence in storehouse
Row number, the Transaction Sequence number using this collection, will be every by the way of Transaction Sequence number is incremental as terminal Transaction Sequence number
Copied to successively in object table with the log recording corresponding to Transaction Sequence number in individual log sheet.
According to another aspect of the present invention there is provided a kind of reproducing unit of data between database, applied to target database
In, including：Acquisition module, for gathering the log recording in source database according to preset rules；Wherein, preset rules
Including：The Transaction Sequence number of log recording is more than the Transaction Sequence number corresponding to the data being synchronized in target database；
Replication module, for log recording to be copied in the object table in target database；Wherein, target database and source number
All it is heterogeneous database according to storehouse.
Further, device also includes：Acquisition module, all log recordings for obtaining this collection of target database
Corresponding Transaction Sequence number；Selecting module, for corresponding to all log recordings for being gathered according to pre-defined rule from this
Transaction Sequence number in the things sequence number that is gathered as this of one Transaction Sequence number of selection.
Further, selecting module includes：First acquisition unit, for where each log recording for gathering for this
Log sheet, obtain the maximum transaction sequence number in the corresponding Transaction Sequence number of all log recordings in log sheet；Compare list
Member, the transaction sequence gathered for the minimum Transaction Sequence number in the maximum transaction sequence number using all log sheets as this
Number.
Further, device also includes：Logging modle, the Transaction Sequence number for recording this collection.
Further, replication module includes：Second acquisition unit, for for the log sheet where each log recording,
According to the corresponding major key of log sheet, the maximum log recording of Transaction Sequence number in log sheet is obtained；First copied cells, is used
Copied in by the maximum log recording of Transaction Sequence number in log sheet in object table.
Further, the first copied cells includes：Subelement is searched, for the major key according to log sheet, major key pair is searched
The object table answered；Operate subelement, for the action type according to the maximum log recording of Transaction Sequence number in log sheet and
The major key of log sheet, is operated to object table.
Further, operation subelement includes：Unit is deleted, for when action type is deletion action, according to daily record
Record corresponding with the major key of log sheet in the major key delete target table of table；Adding device, for being newly-increased in action type
Operation or modification operation, and be not present in object table with the major key of log sheet during corresponding record, by log recording
Content is added in object table as new record；Updating block, for being newly-increased operation or modification behaviour in action type
Make, and in object table it is existing with the major key of log sheet during corresponding record, with the content update object table of log recording
In it is corresponding with the major key of log sheet record.
Further, replication module includes：Second copied cells, for be synchronized to target data in target database
It is starting point Transaction Sequence number that the Transaction Sequence number corresponding to data in storehouse, which adds 1, using the Transaction Sequence number of this collection as eventually
Point Transaction Sequence number, by the way of Transaction Sequence number is incremental, by each log sheet with the day corresponding to Transaction Sequence number
Will record is copied in object table successively.
By the present invention, the Transaction Sequence number collected from source data is more than using target database and has been synchronized to target
The corresponding log recording of Transaction Sequence number of the Transaction Sequence number corresponding to data in database copies to target database
In technological means, wherein, source database and target database are all heterogeneous database, that is, are realized between heterogeneous database
Incremental data clone method, and then cause incremental replication mode no longer to need to rely on the type of database, and then solve
The problem of incremental data storehouse clone method in correlation technique depends on the type of concrete database.
Brief description of the drawings
Accompanying drawing described herein is used for providing a further understanding of the present invention, constitutes the part of the application, the present invention
Schematic description and description be used for explain the present invention, do not constitute inappropriate limitation of the present invention.In the accompanying drawings：
Fig. 1 is the flow chart of the clone method of data between database according to embodiments of the present invention；
Fig. 2 is that the data manipulation for the source database being preferable to carry out according to the present invention indicates flow chart；
Fig. 3 is that the incremental data of target database according to the preferred embodiment of the invention replicates flow chart；
Fig. 4 is the full dose data duplication flow chart of target database according to the preferred embodiment of the invention；
Fig. 5 is the structured flowchart one of the reproducing unit of data between database according to embodiments of the present invention；
Fig. 6 is the structured flowchart two of the reproducing unit of data between database according to embodiments of the present invention；
Fig. 7 is the structure chart one of replication module 54 according to embodiments of the present invention；
Fig. 8 is the structure chart two of replication module 54 according to embodiments of the present invention.
Embodiment
Describe the present invention in detail below with reference to accompanying drawing and in conjunction with the embodiments.It should be noted that in the feelings not conflicted
Under condition, the feature in embodiment and embodiment in the application can be mutually combined.
It should be noted that term " first ", " second " in description and claims of this specification and above-mentioned accompanying drawing
Etc. being for distinguishing similar object, without for describing specific order or precedence.
In the present embodiment there is provided a kind of clone method of data between database, Fig. 1 is number according to embodiments of the present invention
According to the flow chart of the clone method of data between storehouse, as shown in figure 1, the flow comprises the following steps：
Step S102, target database gathers the log recording in source database according to preset rules；Wherein, rule are preset
Then include：The Transaction Sequence number of log recording is more than the Transaction Sequence number corresponding to the data being synchronized in target database；
Step S104, target database copies to log recording in the object table in target database；Wherein, target
Database and source database are all heterogeneous database.
By above-mentioned steps, the Transaction Sequence number collected from source data is more than using target database and has been synchronized to mesh
The corresponding log recording of Transaction Sequence number of the Transaction Sequence number corresponding to data in mark database copies to target data
Technological means in storehouse, wherein, source database and target database are all heterogeneous database, that is, realize heterogeneous database
Between incremental data clone method, and then cause incremental replication mode no longer to need to rely on the type of database, and then solve
The problem of incremental data storehouse clone method in correlation technique of having determined depends on the type of concrete database.
It should be noted that source database is the database where the log recording being replicated；Target database is object table
The database at place.One metadatabase can give multiple target database synchrodatas, and different target database can be with
Different replication strategies are configured with, such as：To the target database for disaster-tolerant backup, collection period can be set to point
Clock level；To the target database for statistics, collection period can be set to hour or day one-level.
In one embodiment of the invention, before step S104, the above method can also include：Target database
Obtain the Transaction Sequence number corresponding to all log recordings of this collection of target database；Adopted according to pre-defined rule from this
The affairs sequence that one Transaction Sequence number of selection is gathered as this in Transaction Sequence number corresponding to all log recordings of collection
Row number.Specifically, the Transaction Sequence number of this collection can be determined in the following manner：For each day of this collection
Log sheet where will record, target database is obtained in log sheet in the corresponding Transaction Sequence number of all log recordings most
Big Transaction Sequence number；Target database regard the minimum Transaction Sequence number in the maximum transaction sequence number of all log sheets as this
The Transaction Sequence number of secondary collection.
In one embodiment of the invention, above-mentioned steps S104 can have many forms, and for example the first is showed
Mode：For the log sheet where each log recording, target database obtains daily record according to the corresponding major key of log sheet
The maximum log recording of Transaction Sequence number in table；Target database answers the maximum log recording of Transaction Sequence number in log sheet
Make in object table；Second of manifestation mode：Target database in target database to be synchronized in target database
It is starting point Transaction Sequence number that Transaction Sequence number corresponding to data, which adds 1, and the Transaction Sequence number using this collection is terminal affairs
Sequence number, by the way of Transaction Sequence number is incremental, by each log sheet with the log recording corresponding to Transaction Sequence number
Copy to successively in object table.For the first manifestation mode by by the maximum log recording of Transaction Sequence number in log sheet
Copy in object table, to each change log recording, the mode replicated as a result of maximum transaction sequence number, i.e.,
Using final result is only replicated, eliminate the duplication of intermediate result, improve duplicating efficiency, the first manifestation mode due to
Intermediate steps are eliminated, it is necessary to which all more new records once submit (commit) to ensure the uniformity of data between table.It is right
A duplication can be chased after according to the order of things sequence number increments in second of manifestation mode, after each Transaction Sequence number has been handled
Can individually it submit, it is to avoid the data volume that the first manifestation mode is disposably submitted excessively may be caused rollback sections
Overflow problem.
Specifically, using which kind of manifestation mode, can be selected according to actual needs, can also both modes combine and make
With, however it is not limited to this.
For the first manifestation mode, the maximum log recording of Transaction Sequence number in log sheet is copied to mesh by target database
It can be accomplished by the following way in mark table：Target database searches the corresponding object table of major key according to the major key of log sheet；
Target database is according to the action type and the major key of log sheet of the maximum log recording of Transaction Sequence number in log sheet, to mesh
Mark table is operated.Specifically, target database is according to the operation class of the maximum log recording of Transaction Sequence number in log sheet
The major key of type and log sheet, carrying out operation to object table includes at least one of mode：It is deletion action in action type
When, target database record corresponding with the major key of log sheet in the major key delete target table according to log sheet；In operation class
Type is newly-increased operation or modification operation, and is not present in object table with the major key of log sheet during corresponding record, target
Database is added the content of log recording as new record in object table；Operate or repair to be newly-increased in action type
Change operation, and in object table it is existing with the major key of log sheet during corresponding record, target database is with log recording
Record corresponding with the major key of log sheet in content update object table.
The above method realizes the incremental crawler of change data by way of the additions and deletions of record sheet change log recording, while to every
Individual affairs distribute incremental Transaction Sequence number, it is ensured that when gathering different tables, keep the data consistency between table.
For second of manifestation mode, it can submit, can also handle a collection of after a complete Transaction Sequence number per treatment
Submitted after Transaction Sequence number.After all log sheets are disposed, the Transaction Sequence number of this collection recorded and gathered
Transaction Sequence number in.
In order to more fully understand the application, the application is further explained below in conjunction with preferred embodiment.
The invention provides a kind of clone method of data between preferred database, specifically include：The data behaviour of source database
Make sign algorithm, the incremental data replication strategy of target database, the full dose Data Replication Algorithm of target database.
(1) the data manipulation sign algorithm of source database：Fig. 2 is the number for the source database being preferable to carry out according to the present invention
Flow chart is indicated according to operation, as shown in Fig. 2 comprising the following steps：
Step S202, each needs synchronous table (being referred to as source table), a corresponding operation day is set up in source database
Will table.Operation Log table is except all row comprising source table, while including following three row：Affairs (transaction) sequence
Row number：For recording the corresponding Transaction Sequence number of the operation；Additions and deletions change operation flag：Record the corresponding database of the daily record
Operation, including increase, modification, three kinds of operations of deletion；Operating time：The time that the operation occurs；Determine the master of source table
Key；The source table being replicated must have unique major key；The major key uniquely identifies each record in the table of source, for searching
Log recording carries out other operations.Synchronous source table in need must comprising " Transaction Sequence number " row, for remembering
Record Transaction Sequence number corresponding when operating this to record；
Step S204, before each affairs (transaction) start, first to file Transaction Sequence number；
Step S206, when record is operated in source table, the Transaction Sequence number of application is filled up to the " thing of source table
Business sequence number " row, after source table has been operated, write log sheet by the corresponding Operation Log of operation note, write daily record
(commit) is submitted in same affairs, it is ensured that the data consistency of log sheet and source table with operation source table；
Step S208, source database, which is set, refreshes maximum duration interval, and the daily record more than most long refresh interval will be by
Delete, it is to avoid the excessive influence performance of log sheet.
(2) the incremental data replication strategy of target database：Fig. 3 is target data according to the preferred embodiment of the invention
The incremental data in storehouse replicates flow chart, as shown in figure 3, comprising the following steps：
Step S302, target database reads last time synchronous Transaction Sequence number, subsynchronous if first, together
It is 0 to walk Transaction Sequence number.
Step S304, mesh is collected by the log recording that all Transaction Sequence numbers in source database are more than SYN
Mark database (equivalent to step S104 in above-described embodiment).
Step S306, calculates the Transaction Sequence number of this collection：The transaction sequence for taking each log sheet gathered maximum
Number；The Transaction Sequence number that Transaction Sequence number minimum in all log sheet maximum sequence numbers is this collection is taken (equivalent to upper
State the Transaction Sequence number of this collection in embodiment).
Step S308, judges whether using the replication strategy once submitted, if it is, performing step S310；If not,
Perform step S312.
Step S310, Data Replication Algorithm 1 (equivalent to the first form of expression of step S104 in above-described embodiment)：
For each log sheet for having collected target database, for each Major key, take Transaction Sequence number in log sheet maximum
Record, be copied into object table.For each change record, replicated as a result of maximum transaction sequence number
Mode, eliminates the duplication of change process in the middle of the record, improves duplicating efficiency.During this replication strategy is due to eliminating
Intermediate step is, it is necessary to which all more new records once submit (commit) to ensure the uniformity of data between table.It is specific to replicate
Algorithm：Object table is searched according to the major key of log sheet, if the action type of log sheet is deleted to delete according to major key
Object table corresponding record；If the action type of log recording is newly-increased or modification, and the corresponding major key record of object table is not
In the presence of then according to the content of daily record in the newly-increased record of object table；If the action type of log recording is newly-increased or repaiied
Change, and the corresponding major key of object table records existing, then with the corresponding record of the content update object table of log sheet；It is all
Record renewal is disposable after finishing to submit (commit).
Step S312, Data Replication Algorithm 2 ((equivalent to second of form of expression of step S104 in above-described embodiment))：
The difference of this replication strategy and Data Replication Algorithm 1 is to be replicated one by one according to the incremental order of Transaction Sequence number, advantage is
Each Transaction Sequence number can be submitted individually (commit) after having handled, it is to avoid algorithm d) is due to the data volume once submitted
Rollback sections of overflow problems may excessively be caused.Specifically replication strategy is：
Step 1, the hyposynchronous Transaction Sequence number+1 of the above is starting point, searches for all log sheets, if any the record of hit,
Carry out corresponding data copy operation；
Step 2, Transaction Sequence number is incremental, searches for log sheet, the operation of repeat step 1；
Step 3, Transaction Sequence number, which is equal to after the Transaction Sequence number c) acquired in step, terminates circulation；
Step 4, it can be carried with being submitted after a complete Transaction Sequence number per treatment after can also handling a collection of Transaction Sequence number
Hand over (commit) replication task.
Step S314, after all log sheets are disposed, the Transaction Sequence number of this collection recorded the thing gathered
It is engaged in sequence number；
(3) the full dose Data Replication Algorithm of target database：Fig. 4 is target data according to the preferred embodiment of the invention
The full dose data duplication flow chart in storehouse, as shown in figure 4, comprising the following steps：
Step S402, after object table is emptied, by the data full dose of active table copy to target database；For big
The OLTP systems of portfolio, during collection, may have data continuous renewal, the data after full dose collection can not
Ensure the data consistency between table；Three follow-up steps are used for regaining the data consistency between object table；
Step S404, takes the maximum transaction sequence number of object table in target database；
Step S406, the maximum transaction sequence number of the log sheet of timing scan source database；
Step S408, when the source database log sheet maximum transaction sequence number of acquisition is more than the maximum that full dose gathers object table
During sequence number, start incremental replication；Detailed process is with " the incremental data replication strategy of target database "
After the completion of step S410, incremental crawler, target database realizes whole storehouse data consistency；
The Transaction Sequence number write-in of incremental crawler has been gathered in Transaction Sequence number.
Below using two source table table1 and table2 as example, the key component in three algorithms is illustrated in the way of pseudo-code.
Assuming that there is two tables to need duplication, respectively table1 and table2 in source database
Table 1：
table1
{
Pk/* major keys */
Col1/* row 1*/
Col2/* row 2*/
Transaction_seq/* Transaction Sequence numbers */
}
Table 2：
table2
{
Pk/* major keys */
Col1/* row 1*/
Col2/* row 2*/
Transaction_seq/* Transaction Sequence numbers */
}
Corresponding two log sheets：
table1_sync_log
{
Pk/* major keys */
Col1/* row 1*/
Col2/* row 2*/
Transaction_seq/* Transaction Sequence numbers */
operation_flag
operation_time
}
table2_sync_log
{
Pk/* major keys */
Col1/* row 1*/
Col2/* row 2*/
Transaction_seq/* Transaction Sequence numbers */
operation_flag
operation_time
}
The explanation of several main algorithms：
The data manipulation sign algorithm example of source database：
Update_Transaction(
i_table1_pk,
i_table1_col1,
i_table1_col2,
i_table2_pk,
i_table2_col1,
i_table2_col2,
)
Target database, which is obtained, replicates Transaction Sequence number algorithm example, i.e. step S306：
Target database replication strategy 1, i.e. step S310：
Target database replication strategy 2, i.e. step S312：
The embodiment of the present invention realizes the incremental crawler of change data by way of the additions and deletions will on some other day of record sheet, while to every
Individual affairs distribute incremental Transaction Sequence number, it is ensured that when gathering different tables, keep the data consistency between table.The present invention
The characteristics of be that source database is only responsible for database of record Operation Log and Transaction Sequence number, target database is responsible for gathering algorithm.
Final result is only replicated when another feature of the present invention is data duplication, the duplication of record modification intermediate result is eliminated, carries
High duplicating efficiency.For example, a record there occurs multiple change in data syn-chronization interim, there is many in log sheet
The corresponding operation note of bar, only renewal operation is done during target database replicate data according to the maximum daily record of Transaction Sequence number,
The renewal process of centre is have ignored, synchronous efficiency is improved.
Through the above description of the embodiments, those skilled in the art can be understood that according to above-described embodiment
Method the mode of required general hardware platform can be added to realize by software, naturally it is also possible to by hardware, but a lot
In the case of the former be more preferably embodiment.Understood based on such, technical scheme is substantially in other words to existing
The part for having technology to contribute can be embodied in the form of software product, and the computer software product is stored in one
In storage medium (such as ROM/RAM, magnetic disc, CD), including some instructions are make it that a station terminal equipment (can
To be mobile phone, computer, server, or network equipment etc.) perform method described in each embodiment of the invention.
A kind of reproducing unit of data between database is additionally provided in the present embodiment, and the device is used to realize above-described embodiment
And preferred embodiment, carried out repeating no more for explanation.As used below, term " module " can be real
The combination of the software and/or hardware of existing predetermined function.Although the device described by following examples preferably comes real with software
It is existing, but hardware, or the realization of the combination of software and hardware is also that may and be contemplated.
Fig. 5 is the structured flowchart one of the reproducing unit of data between database according to embodiments of the present invention, as shown in figure 5,
The device includes：
Acquisition module 52, for gathering the log recording in source database according to preset rules；Wherein, preset rules include：
The Transaction Sequence number of log recording is more than the Transaction Sequence number corresponding to the data being synchronized in target database；
Replication module 54, is connected with above-mentioned acquisition module 52, for the mesh for copying to log recording in target database
Mark in table；Wherein, target database and source database are all heterogeneous database.
By said apparatus, the incremental replication method between source database and target database is realized, and source database and mesh
Mark database is heterogeneous database, and then causes incremental replication mode no longer to need to rely on the type of database, is realized
Incremental data clone method between heterogeneous database, and then solve the incremental data storehouse clone method dependence in correlation technique
The problem of the type of concrete database.
It should be noted that source database is the database where the log recording being replicated；Target database is object table
The database at place.One metadatabase can give multiple target database synchrodatas, and different target database can be with
Different replication strategies are configured with, such as：To the target database for disaster-tolerant backup, collection period can be set to point
Clock level；To the target database for statistics, collection period can be set to hour or day one-level.
Fig. 6 is the structured flowchart two of the reproducing unit of data between database according to embodiments of the present invention, as shown in fig. 6,
Said apparatus also includes：
Acquisition module 62, for obtaining the Transaction Sequence number corresponding to all log recordings of this collection of target database；
Selecting module 64, is connected with above-mentioned acquisition module 62, for all daily records gathered according to pre-defined rule from this
The things sequence number that one Transaction Sequence number of selection is gathered as this in the corresponding Transaction Sequence number of record.
It should be noted that above-mentioned selecting module 64 can include：First acquisition unit 642, for for this collection
Each log recording where log sheet, obtain the maximum in the corresponding Transaction Sequence number of all log recordings in log sheet
Transaction Sequence number；Comparing unit 644, for the minimum Transaction Sequence number in the maximum transaction sequence number by all log sheets
It is used as the Transaction Sequence number of this collection.
Logging modle 66, the Transaction Sequence number for recording this collection.
In one embodiment of the invention, above-mentioned replication module 54 can have a variety of manifestation modes, and Fig. 7 is according to this
The structure chart one of the replication module 54 of inventive embodiments, as shown in fig. 7, above-mentioned replication module 54 can include：Second
Acquiring unit 542, for for the log sheet where each log recording, according to the corresponding major key of log sheet, obtains day
The maximum log recording of Transaction Sequence number in will table；First copied cells 544, for by Transaction Sequence number in log sheet most
Big log recording is copied in object table.First copied cells 544 can include：Subelement 5442 is searched, for root
According to the major key of log sheet, the corresponding object table of major key is searched；Subelement 5444 is operated, for according to affairs sequence in log sheet
The action type and the major key of log sheet of the maximum log recording of row number, are operated to object table.
It should be noted that aforesaid operations subelement 5444 can include deleting unit and/or adding device and/or update single
First (not shown)：Unit is deleted, for when action type is deletion action, being deleted according to the major key of log sheet
Record corresponding with the major key of log sheet in object table；Adding device, for being newly-increased operation or modification in action type
Operation, and being not present in object table with the major key of log sheet during corresponding record, using the content of log recording as new
Record addition is in object table；Updating block, for being operated or modification operation to be newly-increased in action type, and target
It is existing with the major key of log sheet during corresponding record in table, with the content update object table of log recording with log sheet
The corresponding record of major key.
By including the device of the replication module 54 shown in Fig. 7, realize the maximum day of Transaction Sequence number in log sheet
Will record is copied in object table, to each change log recording, the mode replicated by using maximum transaction sequence number,
I.e. using a final result is only replicated, eliminate the duplication of intermediate result, improve duplicating efficiency, and need it is all more
New record once submits (commit) to ensure the uniformity of data between table.
Fig. 8 is the structure chart two of replication module 54 according to embodiments of the present invention, as shown in figure 8, above-mentioned replication module
54 can include：Second copied cells 82, for be synchronized to the data institute in target database in target database
Corresponding Transaction Sequence number add 1 be starting point Transaction Sequence number, using this collection Transaction Sequence number as terminal Transaction Sequence number,
By the way of Transaction Sequence number is incremental, will successively it be replicated with the log recording corresponding to Transaction Sequence number in each log sheet
Into object table.
By including the device of the replication module 54 shown in Fig. 8, it can be chased after according to the order of things sequence number increments individual multiple
System, each Transaction Sequence number can be submitted individually after having handled, it is to avoid the data that the first manifestation mode is disposably submitted
Amount excessively may caused rollback sections of overflow problems
Specifically, the replication module 54 of which kind of structure is included in the reproducing unit of data between above-mentioned database, can basis
It is actually needed and is selected, can also be used in combination, however it is not limited to this.
It should be noted that above-mentioned modules can be by software or hardware to realize, for the latter, Ke Yitong
Cross in the following manner realization, but not limited to this：Above-mentioned modules are respectively positioned in same processor；Or, each above-mentioned mould
Block is located in multiple processors respectively.
Embodiments of the invention additionally provide a kind of storage medium.Alternatively, in the present embodiment, above-mentioned storage medium can
The program code for performing following steps to be arranged to storage to be used for：
S1, the log recording in source database is gathered according to preset rules；Wherein, preset rules include：Log recording
Transaction Sequence number is more than the Transaction Sequence number corresponding to the data being synchronized in target database；
S2, log recording is copied in the object table in target database；Wherein, target database and source database be all
For heterogeneous database.
Alternatively, in the present embodiment, above-mentioned storage medium can include but is not limited to：USB flash disk, read-only storage (ROM,
Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk,
Magnetic disc or CD etc. are various can be with the medium of store program codes.
Alternatively, the specific example in the present embodiment may be referred to showing described in above-described embodiment and optional embodiment
Example, the present embodiment will not be repeated here.
Obviously, those skilled in the art should be understood that above-mentioned each module of the invention or each step can be with general
Computing device realizes that they can be concentrated on single computing device, or is distributed in multiple computing devices and is constituted
Network on, alternatively, the program code that they can be can perform with computing device be realized, it is thus possible to by they
Storage is performed by computing device in the storage device, and in some cases, can be to be held different from order herein
They, are either fabricated to each integrated circuit modules or will be many in them by the shown or described step of row respectively
Individual module or step are fabricated to single integrated circuit module to realize.So, the present invention is not restricted to any specific hardware
Combined with software.
The preferred embodiments of the present invention are the foregoing is only, are not intended to limit the invention, for the technology of this area
For personnel, the present invention can have various modifications and variations.Within the spirit and principles of the invention, that is made is any
Modification, equivalent substitution, improvement etc., should be included in the scope of the protection.
Claims (16)
1. the clone method of data between a kind of database, it is characterised in that including：
Target database gathers the log recording in source database according to preset rules；Wherein, the preset rules bag
Include：The Transaction Sequence number of the log recording is more than the thing corresponding to the data being synchronized in the target database
Business sequence number；
The target database copies to the log recording in the object table in the target database；Wherein,
The target database and the source database are all heterogeneous database.
2. according to the method described in claim 1, it is characterised in that replicate the log recording in the target database
Before into the object table in the target database, methods described also includes：
The target database is obtained corresponding to all log recordings of this collection of the target database
The Transaction Sequence number；The affairs corresponding to all log recordings gathered according to pre-defined rule from this
The Transaction Sequence number for selecting a Transaction Sequence number to be gathered as this in sequence number.
3. method according to claim 2, it is characterised in that all days gathered according to pre-defined rule from this
The things sequence that one Transaction Sequence number of selection is gathered as this in the corresponding Transaction Sequence number of will record
Number include：
For the log sheet where each log recording of this collection, the target database obtains the day
Maximum transaction sequence number in will table in the corresponding Transaction Sequence number of all log recordings；
The target database makees the minimum Transaction Sequence number in the maximum transaction sequence number of all log sheets
For the Transaction Sequence number of this collection.
4. method according to claim 2, it is characterised in that replicate the log recording in the target database
After into the object table in the target database, methods described also includes：The target database record is described
The Transaction Sequence number of this collection.
5. method according to any one of claim 1 to 4, it is characterised in that the target database is by the day
The object table that will record is copied in the target database includes：
For the log sheet where each log recording, the target database is corresponding according to the log sheet
Major key, obtains the maximum log recording of Transaction Sequence number described in the log sheet；
The target database copies to the maximum log recording of Transaction Sequence number described in the log sheet described
In object table.
6. method according to claim 5, it is characterised in that the target database is by thing described in the log sheet
The maximum log recording of business sequence number, which copies to the object table, to be included：
The target database searches the corresponding object table of the major key according to the major key of the log sheet；
The operation class of the maximum log recording of target database Transaction Sequence number according to the log sheet
The major key of type and the log sheet, is operated to the object table.
7. method according to claim 6, it is characterised in that the target database is according to the log sheet
The action type and the major key of the log sheet of the maximum log recording of Transaction Sequence number, are grasped to the object table
Work includes at least one of：
When the action type is deletion action, the target database deletes institute according to the major key of the log sheet
State record corresponding with the major key of the log sheet in object table；
It is newly-increased operation or modification operation in the action type, and is not present and the day in the object table
During the corresponding record of the major key of will table, the target database adds the content of the log recording as new record
It is added in the object table；
It is that newly-increased operation or modification are operated in the action type, and the existing and day in the object table
During the corresponding record of the major key of will table, the target database is with object table described in the content update of the log recording
In it is corresponding with the major key of the log sheet record.
8. according to the method in claim 2 or 3, it is characterised in that the target database answers the log recording
The object table made in the target database includes：
The target database is right with the data institute being synchronized in the target database in the target database
It is starting point Transaction Sequence number that the Transaction Sequence number answered, which plus 1, and the Transaction Sequence number using this collection is terminal
Transaction Sequence number, will be right with Transaction Sequence number institute in each log sheet by the way of Transaction Sequence number is incremental
The log recording answered is copied in the object table successively.
9. the reproducing unit of data between a kind of database, applied in target database, it is characterised in that including：
Acquisition module, for gathering the log recording in source database according to preset rules；Wherein, the default rule
Then include：The Transaction Sequence number of the log recording is more than corresponding to the data being synchronized in the target database
Transaction Sequence number；
Replication module, for the log recording to be copied in the object table in the target database；Wherein,
The target database and the source database are all heterogeneous database.
10. device according to claim 9, it is characterised in that described device also includes：
Acquisition module, for obtaining the institute corresponding to all log recordings of this collection of the target database
State Transaction Sequence number；
Selecting module, for the thing corresponding to all log recordings for being gathered according to pre-defined rule from this
The things sequence number that one Transaction Sequence number of selection is gathered as this in business sequence number.
11. device according to claim 10, it is characterised in that the selecting module includes：
First acquisition unit, for the log sheet where each log recording for gathering for this, obtains institute
State the maximum transaction sequence number in the corresponding Transaction Sequence number of all log recordings in log sheet；
Comparing unit, for using the minimum Transaction Sequence number in the maximum transaction sequence number of all log sheets as
The Transaction Sequence number of this collection.
12. device according to claim 10, it is characterised in that described device also includes：Logging modle, for remembering
The Transaction Sequence number of record this collection.
13. the device according to any one of claim 9 to 12, it is characterised in that the replication module includes：
Second acquisition unit, for for the log sheet where each log recording, according to the log sheet pair
The major key answered, obtains the maximum log recording of Transaction Sequence number described in the log sheet；
First copied cells, for the maximum log recording of Transaction Sequence number described in the log sheet to be copied into institute
State in object table.
14. device according to claim 13, it is characterised in that first copied cells includes：
Subelement is searched, for the major key according to the log sheet, the corresponding object table of the major key is searched；
Subelement is operated, the operation class for the log recording of Transaction Sequence number maximum according to the log sheet
The major key of type and the log sheet, is operated to the object table.
15. device according to claim 14, it is characterised in that the operation subelement includes：
Unit is deleted, for when the action type is deletion action, institute to be deleted according to the major key of the log sheet
State record corresponding with the major key of the log sheet in object table；
Adding device, for being operated or modification operation to be newly-increased in the action type, and in the object table
During in the absence of record corresponding with the major key of the log sheet, add the content of the log recording as new record
It is added in the object table；
Updating block, for being operated or modification operation to be newly-increased in the action type, and in the object table
It is existing with the major key of the log sheet during corresponding record, with object table described in the content update of the log recording
In it is corresponding with the major key of the log sheet record.
16. the device according to claim 10 or 11, it is characterised in that the replication module includes：
Second copied cells, for be synchronized to the data institute in the target database in the target database
It is starting point Transaction Sequence number that corresponding Transaction Sequence number, which adds 1, using the Transaction Sequence number of this collection as eventually
Point Transaction Sequence number, by the way of the Transaction Sequence number is incremental, by each log sheet with the transaction sequence
The log recording corresponding to number is copied in the object table successively.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610073070.9A CN107025242A (en) | 2016-02-02 | 2016-02-02 | The clone method and device of data between database |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610073070.9A CN107025242A (en) | 2016-02-02 | 2016-02-02 | The clone method and device of data between database |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN107025242A true CN107025242A (en) | 2017-08-08 |
Family
ID=59525017
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610073070.9A Pending CN107025242A (en) | 2016-02-02 | 2016-02-02 | The clone method and device of data between database |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107025242A (en) |
Cited By (15)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108573042A (en) * | 2018-04-10 | 2018-09-25 | 平安科技（深圳）有限公司 | Report synchronous method, electronic equipment and computer readable storage medium |
| CN109241185A (en) * | 2018-08-27 | 2019-01-18 | 武汉达梦数据库有限公司 | A kind of method and data synchronization unit that data are synchronous |
| CN109271442A (en) * | 2018-07-27 | 2019-01-25 | 东软集团股份有限公司 | Database synchronization restoration methods, device, computer readable storage medium and electronic equipment |
| CN109271450A (en) * | 2018-10-10 | 2019-01-25 | 北京百度网讯科技有限公司 | Database synchronization method, device, server and storage medium |
| CN109408596A (en) * | 2018-11-06 | 2019-03-01 | 杭州通易科技有限公司 | A kind of dual-active database disaster tolerance system and method |
| CN109446196A (en) * | 2018-09-26 | 2019-03-08 | 努比亚技术有限公司 | Major key generates processing and determines that method, major key generates and application server |
| CN109542682A (en) * | 2018-11-16 | 2019-03-29 | 上海达梦数据库有限公司 | A kind of data back up method, device, equipment and storage medium |
| CN109684327A (en) * | 2018-11-23 | 2019-04-26 | 武汉达梦数据库有限公司 | A kind of data capture method of database external linkage |
| CN110209678A (en) * | 2018-02-07 | 2019-09-06 | 上海鼎茂信息技术有限公司 | Data replication performance monitoring method between a kind of multiple database |
| CN110647421A (en) * | 2018-06-27 | 2020-01-03 | 阿里巴巴集团控股有限公司 | Database processing method, device and system and electronic equipment |
| CN111198914A (en) * | 2019-12-12 | 2020-05-26 | 山西云时代技术有限公司 | Whole database real-time data acquisition method based on oracle database filing log |
| CN111666331A (en) * | 2019-03-06 | 2020-09-15 | 华为技术有限公司 | Method, device and system for copying data |
| CN113377502A (en) * | 2021-06-10 | 2021-09-10 | 上海达梦数据库有限公司 | Transaction processing method, device, server, database management system and medium |
| CN113836579A (en) * | 2021-09-26 | 2021-12-24 | 多点生活(成都)科技有限公司 | Data processing method and device, electronic equipment and storage medium |
| WO2022152085A1 (en) * | 2021-01-14 | 2022-07-21 | International Business Machines Corporation | Applying changes in a target database system |
-
2016
- 2016-02-02 CN CN201610073070.9A patent/CN107025242A/en active Pending
Cited By (23)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110209678A (en) * | 2018-02-07 | 2019-09-06 | 上海鼎茂信息技术有限公司 | Data replication performance monitoring method between a kind of multiple database |
| CN110209678B (en) * | 2018-02-07 | 2022-11-22 | 上海鼎茂信息技术有限公司 | Method for monitoring data copying performance among multiple databases |
| CN108573042B (en) * | 2018-04-10 | 2022-06-10 | 平安科技（深圳）有限公司 | Report synchronization method, electronic equipment and computer readable storage medium |
| CN108573042A (en) * | 2018-04-10 | 2018-09-25 | 平安科技（深圳）有限公司 | Report synchronous method, electronic equipment and computer readable storage medium |
| CN110647421B (en) * | 2018-06-27 | 2022-11-25 | 阿里巴巴集团控股有限公司 | Database processing method, device and system and electronic equipment |
| CN110647421A (en) * | 2018-06-27 | 2020-01-03 | 阿里巴巴集团控股有限公司 | Database processing method, device and system and electronic equipment |
| CN109271442A (en) * | 2018-07-27 | 2019-01-25 | 东软集团股份有限公司 | Database synchronization restoration methods, device, computer readable storage medium and electronic equipment |
| CN109241185B (en) * | 2018-08-27 | 2021-03-30 | 武汉达梦数据库股份有限公司 | Data synchronization method and data synchronization device |
| CN109241185A (en) * | 2018-08-27 | 2019-01-18 | 武汉达梦数据库有限公司 | A kind of method and data synchronization unit that data are synchronous |
| CN109446196A (en) * | 2018-09-26 | 2019-03-08 | 努比亚技术有限公司 | Major key generates processing and determines that method, major key generates and application server |
| CN109271450A (en) * | 2018-10-10 | 2019-01-25 | 北京百度网讯科技有限公司 | Database synchronization method, device, server and storage medium |
| CN109408596A (en) * | 2018-11-06 | 2019-03-01 | 杭州通易科技有限公司 | A kind of dual-active database disaster tolerance system and method |
| CN109542682A (en) * | 2018-11-16 | 2019-03-29 | 上海达梦数据库有限公司 | A kind of data back up method, device, equipment and storage medium |
| CN109684327A (en) * | 2018-11-23 | 2019-04-26 | 武汉达梦数据库有限公司 | A kind of data capture method of database external linkage |
| CN109684327B (en) * | 2018-11-23 | 2021-04-13 | 武汉达梦数据库有限公司 | Data acquisition method for external link of database |
| CN111666331A (en) * | 2019-03-06 | 2020-09-15 | 华为技术有限公司 | Method, device and system for copying data |
| CN111198914A (en) * | 2019-12-12 | 2020-05-26 | 山西云时代技术有限公司 | Whole database real-time data acquisition method based on oracle database filing log |
| WO2022152085A1 (en) * | 2021-01-14 | 2022-07-21 | International Business Machines Corporation | Applying changes in a target database system |
| GB2618022A (en) * | 2021-01-14 | 2023-10-25 | Ibm | Applying changes in a target database system |
| US11954119B2 (en) | 2021-01-14 | 2024-04-09 | International Business Machines Corporation | Applying changes in a target database system |
| CN113377502A (en) * | 2021-06-10 | 2021-09-10 | 上海达梦数据库有限公司 | Transaction processing method, device, server, database management system and medium |
| CN113836579A (en) * | 2021-09-26 | 2021-12-24 | 多点生活(成都)科技有限公司 | Data processing method and device, electronic equipment and storage medium |
| CN113836579B (en) * | 2021-09-26 | 2024-04-09 | 多点生活(成都)科技有限公司 | Data processing method and device, electronic equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107025242A (en) | The clone method and device of data between database | |
| US11500821B2 (en) | Synchronizing metadata in a data storage platform comprising multiple computer nodes | |
| US8078825B2 (en) | Composite hash and list partitioning of database tables | |
| CN106777225B (en) | Data migration method and system | |
| CN101334797B (en) | Distributed file systems and its data block consistency managing method | |
| US8626717B2 (en) | Database backup and restore with integrated index reorganization | |
| US11442961B2 (en) | Active transaction list synchronization method and apparatus | |
| CN111813760B (en) | Data migration method and device | |
| US6694337B1 (en) | Synchronizing databases | |
| Katz et al. | Database support for versions and alternatives of large design files | |
| CN103902410B (en) | The data backup accelerated method of cloud storage system | |
| US20110153568A1 (en) | High volume, high speed adaptive data replication | |
| CN105989194A (en) | Method and system of table data comparison | |
| EP2380090B1 (en) | Data integrity in a database environment through background synchronization | |
| US11176111B2 (en) | Distributed database management system with dynamically split B-tree indexes | |
| US11853581B2 (en) | Restoring a storage system using file relocation metadata | |
| CN111917834A (en) | Data synchronization method and device, storage medium and computer equipment | |
| US7200625B2 (en) | System and method to enhance availability of a relational database | |
| CN113656384B (en) | Data processing method, distributed database system, electronic device and storage medium | |
| EP1377906B1 (en) | Method and arrangements for node recovery | |
| CN113094442A (en) | Full data synchronization method, device, equipment and medium | |
| WO2023066222A1 (en) | Data processing method and apparatus, and electronic device, storage medium and program product | |
| CN104699720A (en) | Merging and storing method and system for massive data | |
| CN114153857A (en) | Data synchronization method, data synchronization apparatus, and computer-readable storage medium | |
| CN108027835A (en) | Apparatus and method for the storage for managing primary database and replica database |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| WD01 | Invention patent application deemed withdrawn after publication | ||
| WD01 | Invention patent application deemed withdrawn after publication |
Application publication date: 20170808 |