CN102081665A - Data synchronization method and device - Google Patents
Data synchronization method and device Download PDFInfo
- Publication number
- CN102081665A CN102081665A CN2011100240321A CN201110024032A CN102081665A CN 102081665 A CN102081665 A CN 102081665A CN 2011100240321 A CN2011100240321 A CN 2011100240321A CN 201110024032 A CN201110024032 A CN 201110024032A CN 102081665 A CN102081665 A CN 102081665A
- Authority
- CN
- China
- Prior art keywords
- data
- synchrodata
- storage area
- tables
- treatment
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a data synchronization method and device, and the method comprises the following steps: dividing storage regions occupied by data to be synchronized in a source data table into a plurality of groups; and starting a plurality of corresponding data synchronization processes in parallel, and using each data synchronization process to synchronize the data to be synchronized in all the storage regions in the corresponding group to a current target data table. By adopting the method, the data in a plurality of the storage regions can be simultaneously processed and submitted, the number of submissions can be reduced, and the submitting speed and efficiency of synchronous data can be further improved.
Description
The application be that Dec 10, application number in 2008 are 200810179394.6 the applying date, denomination of invention divides an application for the patent application of " a kind of method of data synchronization and device ".
Technical field
The application relates to computer realm, refers to a kind of method of data synchronization and device especially.
Background technology
In large-scale e-commerce website, often to from data warehouse, calculate Transaction Processing (the On-Line Transaction Processing of good data synchronously to the foreground, OLTP) in the inscriptions on bones or tortoise shells ORACLE database, for the application server inquiry on foreground.
Comprise one in the ORACLE database to cancel/rollback table space (undo tablespace), when a session is carried out data manipulation language (DML) (Data Manipulation Language to database, during renewal DML) and deletion (Update and Delete) operation, can adjourn the another one place to the raw data that influences data line and store away, this place is exactly to cancel/the rollback table space.Do not submit (Commit) to before in this session, during these data lines of other session access, find that data are modified, but do not have submitted, according to cancelling/the rollback address information of storage, can to cancel/obtain raw data in the rollback table space, but before this session is not submitted to or before rollback finished, shared roll-back segment can not discharge.
The method of the present synchrodata of using always, its synchronous architecture as shown in Figure 1.Among Fig. 1, the t1 table is a table in the OLTP ORACLE database of foreground; Data warehouse DW (data warehouse) is the data warehouse on backstage, and top service device (Application) obtains data from the t1 table.The logic of synchrodata following (all carrying out) in the Foreground Data storehouse to issue orders:
The first step: the data (delete from schema1.tablename) in the deletion subscriber's meter t1 table, the data of deletion temporarily are put into cancel/the rollback table space in.
Second step: from (the back-end data warehouse) subscriber's meter t2 table, obtain data and be inserted in the subscriber's meter t1 table (insert into schema1.tablename select*from schema2.tablename@lnk dw), the data of coming synchronously, do not submit to as yet also are temporary in and cancel/the rollback table space in.
The 3rd step: submit to (commit).
The 4th step: if rollback (rollback) occurs unusually.
Because foreground and data warehouse all are to adopt the ORACLE database, can directly connect (database link) and import/go out data by database; Employed lnk_dw is exactly that a database connects in second step of above-mentioned steps.
The method of data synchronization of prior art, because all operations of same data service all carries out in same table, be a table in the back-end data warehouse one the corresponding Foreground Data of table storehouse, when operation is carried out, will have mass data and need be placed to and cancel/the rollback table space.Cause prior art to have following shortcoming for above-mentioned reasons:
(1) in above-mentioned steps; the first step and second step will consume a large amount of cancelling/rollback table space; so here go wrong through regular meeting; if ruing out of all cancelled/the rollback table space; will have influence on other normal DML (Insert; Update and Delete) operation promptly has influence on professional normally carrying out.If in peak time, cancel/the rollback table space taken, waiting for all can appear in the DML statement that sends to database of application server so, and then the connection pool of application server end configuration is taken, because it is more and more to enter the request of application server, and these requests all will be waited in line because of can not get connection, thereby form vicious cycle, will soon run out of the memory source of application server etc., and then cause application server to crash.
(2) first step and second execution time in step long, within the so long time, if unusual, will enter for the 4th step, that is: rollback step.Thereby cause fault-tolerant ability poor, and after entering the rollback state, because data volume is huge, ORACLE does time of rollback will be very long, and then influenced the performance in production data storehouse.
(3) efficient of synchrodata is low, and this efficiency losses mainly is present in the deletion action of the first step.
(4) do not have the breakpoint transmission ability.
(5) can't monitor the progress of synchrodata.
Along with the synchrodata amount is increasing, the cancelling of foreground ORACLE database/rollback table space is easy to be taken, and it is more and more the situation that traffic affecting normally carries out to occur, and comes serious potential safety hazard for the production data library tape of enterprise.
Summary of the invention
The embodiment of the present application provides a kind of method of data synchronization and device, has solved the problem of blocking up when the data sync amount day by day increases, and has improved the speed and the efficient of data sync.
A kind of method of synchrodata comprises:
To treat in the source data table that the shared storage area of synchrodata is divided into some groups;
Several data synchronizing process of parallel starting correspondence are by treating that synchrodata is in the current goal tables of data in each the described storage area in the synchronously corresponding group of each data synchronizing process.
According to the application's said method, describedly carry out successively by the data in each the described storage area in the synchronously corresponding group of each data synchronizing process, specifically comprise:
Each data synchronizing process is handled the synchrodata for the treatment of in the storage area in the described corresponding group;
Synchronously after the treating synchrodata and successfully submit in the described storage area, judge whether to handle all storage areas in the described corresponding group;
When all storage areas in the described corresponding group that has been untreated, start the process for the treatment of synchrodata in the next storage area in the described corresponding group of handling.
The application's said method also comprises: when in synchronous some described storage areas treat the synchrodata failure time, restart the process for the treatment of synchrodata in this storage area synchronously;
When the number of times that restarts surpasses setting threshold, finish the interior process for the treatment of synchrodata of this storage area synchronously.
According to the application's said method, will treat in the source data table that the shared storage area of synchrodata was divided into before some groups, also comprised:
Determine some storage areas for the treatment of in the source data table that synchrodata is shared; And
Determine the storage address information for the treatment of synchrodata in each described storage area.
According to the application's said method, the described storage address information for the treatment of synchrodata in each described storage area of determining specifically comprises:
Determine lowest address and the maximum address for the treatment of the storage of synchrodata in each described storage area;
The address realm of determining that comprises lowest address and maximum address is stored in the middle table.
A kind of device of synchrodata comprises:
Grouped element is used for the source data table is treated that the shared storage area of synchrodata is divided into some groups;
Data synchronisation unit is used for several data synchronizing process of parallel starting correspondence, by treating that synchrodata is in the current goal tables of data in each the described storage area in the corresponding synchronously group of each data synchronizing process.
Put according to the above-mentioned of the application, described data synchronisation unit specifically comprises: subelement and judgment sub-unit are carried out in control;
Subelement is carried out in control, is used for controlling the synchrodata for the treatment of in the described corresponding storage area of organizing of each data synchronizing process processing; And during all storage areas in judgment sub-unit is judged the described corresponding group that has been untreated, start the process for the treatment of synchrodata in the next storage area in described corresponding group of handling;
Judgment sub-unit, be used for the treating synchrodata and successfully submit to of a synchronous described storage area after, judge whether to handle all storage areas in the described corresponding group.
Put according to the above-mentioned of the application, described data synchronisation unit also comprises:
Restart subelement, be used for when synchronous some described storage areas treat the synchrodata failure time, restart the process for the treatment of synchrodata in this storage area synchronously; And when the number of times that restarts surpasses setting threshold, finish the process for the treatment of synchrodata in synchronous this storage area.
The above-mentioned of the application put, and also comprises:
The storage area determining unit is used for definite source data table and treats some storage areas that synchrodata is shared;
The address information determining unit is used for determining the storage address information that each described storage area is treated synchrodata.
Method of data synchronization that the embodiment of the present application provides and device are by treating in the source data table that the shared storage area of synchrodata is divided into some groups; Several data synchronizing process of parallel starting correspondence are by treating that synchrodata is in the current goal tables of data in each the described storage area in the synchronously corresponding group of each data synchronizing process.Said method has adopted a plurality of process parallel processings, the data of a plurality of storage areas of single treatment, when submitting to, just can submit to the data of these a plurality of storage areas to give target matrix simultaneously, reduce the number of times of submitting to, thereby improve the submission speed and the efficient of synchrodata.
Description of drawings
Fig. 1 is the principle schematic of method of data synchronization in the prior art;
Fig. 2 is the principle schematic of method of data synchronization in the embodiment of the present application;
Fig. 3 is the process flow diagram of method of data synchronization in the embodiment of the present application;
Fig. 4 be in the embodiment of the present application in t2 table the principle schematic of synchrodata;
Fig. 5 be in the embodiment of the present application in t1 table the principle schematic of synchrodata;
Fig. 6 is the process flow diagram of method of data synchronization in the embodiment of the present application one;
Fig. 7 is the process flow diagram of method of data synchronization in the embodiment of the present application two;
Fig. 8 be in the embodiment of the present application in the current goal tables of data process flow diagram of synchrodata;
Fig. 9 is an oracle data of database storage format synoptic diagram in the embodiment of the present application;
Figure 10 is the process flow diagram that adopts multi-process parallel synchronous data in the embodiment of the present application;
Figure 11 is the structural representation of data synchronization unit in the embodiment of the present application;
Figure 12 is the structural representation of the device of multi-process synchrodata in the embodiment of the present application.
Embodiment
The method of data synchronization that the embodiment of the present application provides, its principle as shown in Figure 2.To same data service, set up in advance with a storage and remain the two corresponding mark tables of data of opening one's eyes wide of the source data table of synchrodata, target matrix 1 shown in Fig. 2 and target matrix 2.The target matrix that is to say each Zhang Yuan's tables of data correspondence has two, two open one's eyes wide the mark tables of data be used alternatingly.
Use the application server (Application) of data in the target matrix to visit that mark tables of data of opening one's eyes wide that top level view is pointed to by the top level view of setting up in advance (View).Wherein, the effect of top level view is equivalent to a pointer, makes server can pass through its visit data table.For example among Fig. 2, when top level view definite object tables of data 1, application server can be by the data in the top level view access destination tables of data 1, simultaneously, because target matrix 2 is in idle condition this moment, when remaining synchrodata, then can be in target matrix 2 synchrodata; When top level view definite object tables of data 2, application server can pass through the data in the top level view access destination tables of data 2, simultaneously, because target matrix 1 is in idle condition at this moment, when remaining synchrodata, then can be in target matrix 1 synchrodata.Thereby the process of the synchrodata of making and application server access data needn't be carried out in same table, can not disturb mutually.
What the embodiment of the present application provided carries out the method for data sync from the source data table to target matrix, detailed process as shown in Figure 3, execution in step is as follows:
Step S1: determine a current goal tables of data the tables of data from two marks of opening one's eyes wide.
Can be by calculating the two record numbers of data in the mark tables of data of opening one's eyes wide respectively, determine that the record number is that mark tables of data of opening one's eyes wide of zero is the current goal tables of data.
For example: if the record number of data is zero in the target matrix 1, illustrate that then target matrix 1 is empty table, determine that target matrix 1 is the current goal tables of data; If the record number of data is zero in the target matrix 2, illustrate that then target matrix 2 is empty table, determine that target matrix 2 is the current goal tables of data.
Especially, when two open one's eyes wide that the record number average of data is zero in the mark tables of data, can according to computation sequence determine one or optional wherein one as the current goal tables of data; When two open one's eyes wide that the record number average of data is non-vanishing in the mark tables of data, display abnormality, and the synchronous flow process of end data.
Step S2: from source data table synchrodata to the current goal tables of data of determining.
Especially, from the source data table to the current goal tables of data of determining during synchrodata, direct synchrodata in the current goal tables of data of determining (being the target matrix of original sky); Also the target data list deletion (drop) of original sky can be fallen, and rebulid a new target matrix, as the current goal tables of data, synchrodata in newly-established current goal tables of data.
For the situation of new destination tables of data, can adopt as the realization of issuing orders:
create?table?t2?as?select*from?schema2.tablename@lnk_dw；
The operation of this order is actually a composition operation, a DDL (creating the t2 table), and data are synchronous, but ORACLE can regard it as a DDL operation, and the DDL operation does not take basically to be cancelled/the rollback table space.Can adopt parallel prompting in addition, specific instructions is as follows:
create?table?t2?as?select/*+parallel(dw_t2，4)*/*fromschema2.tablename@lnk_dw?dw_t2；
Above in the order/*+parallel (dw_t2,4) */expression is to the scanning that dw_t2 shows, and adopts four processes to handle.
Adopt this kind scheme, can further reduce cancelling/the rollback table space taken may, and the efficient that can improve synchrodata to a certain extent, but the overlong time of whole DDL operation, the fault-tolerant ability of whole synchrodata is poor, and the progress of synchrodata can't be monitored.
Be described in detail again in the back from the detailed process of source data table synchrodata to the current goal tables of data.
Step S3: after the synchrodata success, application server turns to visit current goal tables of data.Specifically comprise:
After each synchrodata success, all need to upgrade top level view, make top level view point to the new target matrix that imports data, thereby guarantee the application server data of access synchronized in the target matrix smoothly.
Especially, after synchrodata success, can carry out emptying the two not operations of another mark tables of data of opening one's eyes wide of synchrodata in the mark tables of data of opening one's eyes wide.
Above-mentioned source data table and target matrix can all be the tables of data in the background data base warehouse, it also can all be the tables of data in the ORACLE database of foreground, can also one be the tables of data in the background data base warehouse, another be the tables of data in the ORACLE database of foreground.
Be that example is described in detail with tables of data (target matrix) synchrodata of tables of data (source data table) to the ORACLE database of foreground below from the back-end data warehouse.
In the Foreground Data storehouse, two tables of data are set up in same data service, be two tables of data (t1 table and t2 table) in the corresponding Foreground Data of the tables of data storehouse in the back-end data warehouse, two data forms are used alternatingly, be that the source data table is a tables of data in the back-end data warehouse, target matrix is two tables of data of setting up in advance in the ORACLE database of foreground corresponding with it.
Its principle as shown in Figure 4 and Figure 5.Wherein, Fig. 4 is application server (Application) visit t1 table, from the situation of back-end data warehouse synchrodata to the t2 of sky table; Fig. 5 is application server access t2 table, from the situation of back-end data warehouse synchrodata to the t1 of sky table.
Data warehouse DW represents the data warehouse on backstage among Fig. 4 and Fig. 5, t1 table and t2 table are two tables of data of corresponding same data service in the Foreground Data storehouse, two tables of data in the corresponding Foreground Data of tables of data among the back-end data warehouse DW storehouse (t1 table and t2 table), promptly can be from a table in back-end data warehouse in t1 table or t2 show synchrodata.
Application server points to t1 table or t2 table by top level view (view), determines the data in visit t1 table or the t2 table.Be that top level view is pointed to the t1 table, then application server is by the data in the top level view visit t1 table; Top level view is pointed to the t2 table, and then application server is by the data in the top level view visit t2 table.
To the ORACLE database of foreground, certain time interval (or saying the data sync cycle of setting) is arranged all usually from data warehouse DW synchrodata, for example: one day synchronously once.When wanting synchrodata, detect t1 table and t2 table; If t1 shows to empty, then synchrodata in t1 shows; If t2 shows to empty, then synchrodata in t2 shows.
Shown in Figure 4 is application server access t1 table, and the t2 table is for empty, and the situation of synchrodata in the t2 table treats that synchrodata finishes, and redefines top level view view, makes top level view point to the t2 table, carries out then as issues orders:
truncate?table?t1；
Data in the t1 table are emptied.
Especially,, use the truncate mode more a lot of soon, significantly reduced the time of deletion legacy data than delete mode because truncate (abreviation) mode is different with the principle of deletion (delete) mode.
When synchrodata in the t2 table and after redefining top level view, the data in the application server access t2 table, when need synchrodata next time again, synchrodata in then will showing to the t1 of sky.The application server access t2 table that is shown in Figure 5, the t1 table is sky, the situation of synchrodata in the t1 table.Treat that synchrodata finishes, redefine top level view view, make top level view point to the t1 table, carry out then as issue orders:
truncate?table?t2；
Data in the t2 table are emptied.
When from source data table (tables of data in back-end data warehouse) in the target matrix (tables of data the ORACLE database of foreground) during synchrodata, specifically can comprise two kinds of concrete implementations that the following example one and embodiment two are given.
Embodiment one:
What the embodiment of the present application one was provided carries out the method for synchrodata from source data table (tables of data in back-end data warehouse) in target matrix (tables of data the ORACLE database of foreground), show as target matrix 1 with t1, the t2 table describes for example as target matrix 2, its flow process as shown in Figure 6, execution in step is as follows:
Step S101: N1 counted in the record that calculates in the t1 table.
Especially, when handling, N1 counted in the record that can use the mode of rownum＜2 to calculate the t1 table, and specific instructions is as follows:
Select?count(*)into?N1?from?t1?where?rownum＜2。
Wherein, rownum represents data recording, and the mode of said rownum＜2 is specially: think in the tables of data data are arranged as long as detect when at least one record is arranged in the tables of data, and needn't continue to detect the more data record.
Step S102: judge in the t1 table whether data are arranged.Be specially: the record that calculates by judgement is counted N1 and whether is equalled zero to judge in the t1 table whether data are arranged.
If, execution in step S103 then; If not, execution in step S111 then.
Step S103: from the back-end data warehouse to t1 table synchrodata.
The detailed process of the synchrodata in the t1 table of the tables of data from the back-end data warehouse is referring to the detailed description of subsequent step.
Especially, when carrying out data sync, can adopt existing data sync mode, also can be according to the principle of ORACLE EXTENT, the mode that adopts segmentation to submit to.
Step S104: judge whether synchronous success.
If execution in step S105 then; If not, execution in step S108 then.
Step S105: create new top level view.
Create new top level view, point to the t1 table, make application server to visit data in the t1 table by top level view.
Step S106: execution empties operation to the t2 table.
Especially, when execution empties operation to the t2 table, preferably use above-mentioned truncate table order deletion legacy data fast, empty t2 and show.
Execution in step S107 then.
Step S107: the synchronous flow process of end data.
Step S108: display abnormality, and the synchronous flow process of end data.
Step S111: N2 counted in the record that calculates in the t2 table.
When handling, the record that also can use the mode of above-mentioned rownum＜2 to calculate the t2 table is counted N2, the specific instructions behavior:
Select?count(*)into?N2?from?t2?where?rownum＜2。
Step S112: judge in the t2 table whether data are arranged.Be specially: the record that calculates by judgement is counted N2 and whether is equalled zero to judge in the t2 table whether data are arranged.
If, execution in step S113 then; If not, illustrating in two tables all has data, occurs unusually, then execution in step S108.
Step S113: from the back-end data warehouse to t2 table synchrodata.
The detailed process of the synchrodata in the t2 table of the tables of data from the back-end data warehouse is referring to the detailed description of subsequent step.
Especially, when carrying out data sync, can adopt existing data sync mode, also can be according to the principle of ORACLE EXTENT, the mode that adopts segmentation to submit to.
Step S114: judge whether synchronous success.
If execution in step S115 then; If not, execution in step S108 then.
Step S115: create new top level view.
If top level view is then created in synchrodata success again, point to the table of firm synchrodata, i.e. t2 table makes application server to visit data in the t2 table by top level view.
Step S116: execution empties operation to the t1 table.
Especially, when execution empties operation to the t1 table, preferably use above-mentioned truncate table order deletion legacy data fast, empty t1 and show.
Execution in step S107 then.
Embodiment two:
What the embodiment of the present application two was provided works as from the process of source data table (tables of data in back-end data warehouse) synchrodata in target matrix (tables of data the ORACLE database of foreground), still show as target matrix 1 with t1, the t2 table describes for example as target matrix 2, its flow process as shown in Figure 7, execution in step is as follows:
Step S201: calculate the record that the record in the t1 table counts in N1 and the t2 table respectively and count N2.
When handling, the record that the record that still can use the mode of above-mentioned rownum＜2 to calculate the t1 table is counted in N1 and the t2 table is counted N2, the specific instructions behavior:
Select?count(*)into?N1?from?t1?where?rownum＜2；
Select?count(*)into?N2?from?t2?where?rownum＜2。
Step S202: judge in t1 table and the t2 table whether data are arranged.Specifically comprise: judge that record in the t1 table counts record in N1 and the t2 table and count N2 and whether equal zero.
If N1=0, N2 ≠ 0, then illustrating in the t1 table does not have data, in the t2 table data is arranged, then execution in step S203;
If N1 ≠ 0, N2=0, then illustrating in the t1 table has data, does not have data in the t2 table, then execution in step S213;
If N1=0, N2=0, then illustrating in t1 table and the t2 table does not all have data, then execution in step S223;
If N1 ≠ 0, N2 ≠ 0, then illustrating in t1 table and the t2 table all has data, then execution in step S233.
Step S203: from the back-end data warehouse to t1 table synchrodata.
From the detailed process of the tables of data in back-end data warehouse synchrodata to the t1 table detailed description referring to subsequent step.
Especially, when carrying out data sync, can adopt existing data sync mode, also can be according to the principle of ORACLE EXTENT, the mode that adopts segmentation to submit to.
Step S204: judge whether synchronous success.
If execution in step S205 then; If not, execution in step S233 then.
Step S205: create new top level view.
Common new top level view is pointed to the t1 table, makes application server to visit data in the t1 table by top level view.
Step S206: execution empties operation to the t2 table.
Especially, when execution empties operation to the t2 table, preferably use above-mentioned truncate table order deletion legacy data fast, empty t2 and show.
Execution in step S207 then.
Step S207: the synchronous flow process of end data.
Step S213: from the back-end data warehouse to t2 table synchrodata.
From the detailed process of the tables of data in back-end data warehouse synchrodata to the t2 table detailed description referring to subsequent step.
Especially, when carrying out data sync, can adopt existing data sync mode, also can be according to the principle of ORACLE EXTENT, the mode that adopts segmentation to submit to.
Step S214: judge whether synchronous success.
If execution in step S215 then; If not, execution in step S233 then.
Step S215: create new top level view.
If top level view is then created in synchrodata success again, point to the table of firm synchrodata, i.e. t2 table makes application server to visit data in the t2 table by top level view.
Step S216: execution empties operation to the t1 table.
Especially, when execution empties operation to the t1 table, preferably use above-mentioned truncate table order deletion legacy data fast, empty t1 and show.
Execution in step S207 then.
Step S223: from t1 table and t2 table, select a tables of data.
If selected t1 table, then execution in step S203-207; If selected t2 table, then execution in step S213-216 and step 207.
Step S233: display abnormality, and the synchronous flow process of end data.
In the foregoing description one the two record numbers of mark in the tables of data of opening one's eyes wide are judged one by one, when judgement obtains that the record number is zero in first target matrix, then do not needed to judge again the record number in another target matrix.Then be to calculate two record numbers in the target matrix simultaneously among the embodiment two, and to select wherein to write down number be a target matrix of zero.
Specifically describe below among the step S2 from the process of source data table synchrodata to the current goal tables of data of determining, as shown in Figure 8, comprising:
Step S31: determine some storage areas for the treatment of in the source data table that synchrodata is shared.
Especially, can in ORACLE data of database dictionary view, inquire and treat synchrodata shared storage area the source data table.
Step S32: determine the storage address information for the treatment of synchrodata in the above-mentioned storage area.Wherein, storage address information comprises the address realm of storing in each storage area for the treatment of synchrodata at least.
Treat synchrodata shared storage area in the source data table according to what ORACLE data of database dictionary view obtained, can calculate lowest address (e_min_rowid) and the maximum address (e_max_rowid) for the treatment of the synchrodata storage in each storage area.
Especially, the address realm of each region memory storage data of calculating can be saved in the middle table, for example: a tmp_rowid table.The list structure of this middle table can design as follows:
create?table?tmp_rowid
(
id?number；
ROWID_MIN?VARCHAR2(100)，
ROWID_MAX?VARCHAR2(100)
)；
When needs,, can obtain the address information for the treatment of synchrodata of each storage area storage in the source data table by inquiring about this middle table.
Step S33:, respectively the synchrodata for the treatment of in each storage area is synchronized in the current goal tables of data according to storage address information.Specifically can select one of following dual mode:
(1) according to storage address information, by a data synchronized process successively synchronously the data in each storage area in the current goal tables of data.
(2) storage area of determining is divided into N group, and starts N corresponding data synchronized process, according to storage address information, by the data of each storage area in synchronous successively one group of each data synchronizing process in the current goal tables of data.
If in synchronous some storage areas during data failure, restart the process of data in this storage area synchronously, when the number of times that restarts surpasses setting threshold, finish the process of data in this storage area synchronously.
Among the embodiment one among step S103, step S113 and the embodiment two step S203, step S213 carry out from back-end data depot data table and to t1 or t2 table, during synchrodata, adopt above-mentioned data synchronization process exactly.
Aforesaid way (1) is the mode of using always, and mode (2) is based on then that the characteristics of data storage in the characteristics of the embodiment of the present application method of data synchronization and the ORACLE database could realize.Make a concrete analysis of as follows:
At first, the above-mentioned method of data synchronization of the embodiment of the present application comes application server and concrete data isolation with a top level view, when from the data warehouse synchrodata to foreground ORACLE data of database table, the table that is using with operation system, promptly the table at the application server data place of calling is without any relation.Therefore, make from the data warehouse synchrodata to foreground ORACLE data of database table and the operation of upper layer application server calls data can not carried out same tables of data, therefore can not disturb mutually.
Secondly, data storage has following characteristics in the ORACLE database:
ORACLE data of database table is made up of a lot of storage areas (EXTENTS), and each EXTENT is made up of the certain continuous piece (block) of a plurality of sizes, and deposit data is in each piece.EXTENT is the least unit in tables of data space, and is geographically not necessarily continuous between EXTENT and the EXTENT.But in each EXTENT inside, be made up of a string continuous block, the physical address of inner each piece of each EXTENT is continuous.Its storage format as shown in Figure 9.
Under data synchronizing process and the non-interfering situation of application server access process, These characteristics in conjunction with data storage in the ORACLE database, thereby make synchrodata segmentation to submit to, promptly can adopt mode (2) to realize data synchronization process, so both can not influence the accuracy of the data that operation system visits, and can improve the speed of data sync again.
Below with an instantiation, describe the process that employing mode (2) realizes data sync in detail, can adopt mode that segmentation submits to (for example: synchrodata to current goal tables of data (for example: the t1 table on foreground or t2 table) the back-end data warehouse) from the source data table, as shown in figure 10, comprise the following steps:
Step S331: the storage area that synchrodata is shared is treated in inquiry.
The data dictionary view in the back-end data warehouse at query source tables of data place can get access to and treat synchrodata shared storage area in the source data table.
Step S332: determine and treat the address realm of synchrodata in the source data table in each storage area.
Can be by the middle table of inquiry storage, for example: the tmp_rowid table, determine to treat the address realm of synchrodata in the source data table in each storage area.
Step S333: storage is remained all storage areas of synchrodata be divided into the N group.
Next, we can carry out packet transaction to the data in the tmp_rowid table, and the mode of grouping is a lot, such as: can be divided into the N group according to sign ID (Identify) delivery.
Step S334: start data in the N group storage area that N data synchronized process mark off respectively synchronously in the current goal tables of data.
By the parallel processing of N data synchronized process, wherein, the storage area of the corresponding group of each process is carried out the process in the target matrix of the data sync in the storage area in the correspondence group, and each group is not disturbed mutually, carries out simultaneously.In synchronous each storage area, adopt parallel processing during data, improved the efficient of data sync.
Step S335a: handle one by one in each storage area in this group (for example: first group) and treat synchrodata.Be specially:
Handle the synchrodata for the treatment of of a storage area in this group, after obtaining synchronously successful affirmation message, start the process of the data in the next storage area in synchronous this group.
Step S336a: judge whether synchronous success;
That judges current storage area treats that whether synchrodata is by synchronous success.
If carry out corresponding step S338a; If not, then need rollback, carry out corresponding step S337a so that re-execute the process of the interior data of this storage area synchronously.
Step S337a: judge whether the rollback number of times surpasses setting threshold.Be specially:
Can set in advance the threshold value of rollback (retry) number of times, and the number of times of statistics retry, when number of retries greater than the threshold values of setting, retry no longer then, and withdraw from circulation immediately, expression is the data failure of this storage area synchronously.In data synchronization process, adopt retry strategy, improved the fault-tolerant ability of data sync.
If not, then return and re-execute step S335a, the data in synchronous again this storage area are in target matrix; If, execution in step S339a then.
Step S338a: submit the data of coming synchronously to.
If the success of the data sync of current storage area is then submitted the data of coming synchronously to target matrix.
Step S339a: judge whether all to finish, promptly judge whether to handle all storage areas in this group.
If illustrate that then intact this organized the corresponding data of all storage areas, execution in step S340 synchronously; Then explanation does not have to finish the corresponding data that this organizes all storage areas synchronously if not, returns step S335a, the data of the next storage area in continuing should organize synchronously.
In the process of parallel processing, can a counter variable v_done_num be set for every group, be used for every group of the corresponding expression number of the storage area of intact data synchronously, can calculate the speed of each process synchrodata very easily like this, and the total speed that calculates data sync.Can also realize easily the data synchronization efficiency is monitored based on above-mentioned statistics.
Step S335b-S339b is the process of the data in each storage area in synchronous one by one second group;
......；
Step S335x-S339x is the process of the data in each storage area in the synchronous one by one N group;
Step S335b-S339b ..., the processing procedure of step S335x-S339x is corresponding identical with step S335a-S339a, repeat no more herein.
Step S340: the synchronous flow process of end data.
Above-mentioned from source data table synchrodata process to target matrix, if adopted a plurality of process parallel processings, the data of a plurality of storage areas of single treatment (EXTENT), when submitting to, just can submit to the data of these a plurality of storage areas to give target matrix simultaneously, reduce the number of times of submitting to, thereby improved submission (commit) speed and the efficient of synchrodata.Wherein, specifically can handle what storage areas simultaneously according to the actual conditions adjustment.
Above-mentioned Fig. 8, Fig. 9, the given process of Figure 10 from source data table synchrodata to the current goal tables of data, can be used for other separately need be from the source data table to the process of target matrix synchrodata, and is not limited in the given data synchronization process of the embodiment of the present application.
Above-mentioned method of data synchronization according to the embodiment of the present application can make up a kind of data synchronization unit, as shown in figure 11, comprising: tables of data is set up module 10, determination module 20, data simultaneous module 30 and visit update module 40.
Tables of data is set up module 10, is used for setting up in advance with a storage remaining the two corresponding mark tables of data of opening one's eyes wide of the source data table of synchrodata.
Preferable, determination module 20 further can comprise: computing unit 201 and determining unit 202.
Determining unit 202 is used for determining that record number that computing unit 201 calculates is that zero the mark tables of data of opening one's eyes wide is the current goal tables of data.
Determining unit 202 also is used for two opening one's eyes wide the record number average of mark tables of data data when non-vanishing, display abnormality, and the synchronous flow process of end data when what computing unit 201 calculated.
Data simultaneous module 30 is used for the current goal tables of data synchrodata of determining to determination module 20 from the source data table.
Preferable, data simultaneous module further can comprise: storage area determining unit 301, address information determining unit 302 and data synchronisation unit 303.
Storage area determining unit 301 is used for root and determines that the source data table treats some storage areas that synchrodata is shared.
Address information determining unit 302 is used for determining the storage address information that each storage area is treated synchrodata.
Preferable, data synchronisation unit 303 further can comprise: the first data sync subelement 3031 and the second data sync subelement 3032.
The first data sync subelement 3031 is used for the storage address information determined according to address information determining unit 302, by a data synchronized process successively synchronously the data in each storage area in the current goal tables of data.
The second data sync subelement 3032 is used for storage area is divided into the N group, starts N corresponding data synchronized process; And the storage address information of determining according to address information determining unit 302, by the data of each storage area in the corresponding synchronously successively group of each data synchronizing process in the current goal tables of data.
Visit update module 40 is used for after the success of data simultaneous module 30 synchrodatas the current goal tables of data that application server turns to determination module 20 to determine.
Visit update module 40 specifically is used for: set up top level view in advance, make application server visit one of its sensing mark tables of data of opening one's eyes wide by this top level view; And after synchrodata is finished, upgrade described top level view, make it point to described current goal tables of data.
Above-mentioned data synchronization unit also comprises: tables of data empties module 50, is used for after data simultaneous module 30 synchrodatas successes, empties two not another the mark tables of data of opening one's eyes wide of synchrodata of opening one's eyes wide in the mark tables of data.
Data simultaneous module 30 in the above-mentioned data synchronization unit can be set to a device independent, that be used for synchrodata, and can be used for the process that any data shift separately.
According to above-mentioned Fig. 8 and shown in Figure 10 from the flow process of source data table to the target matrix synchrodata, can make up a kind of device of multi-process synchrodata, as shown in figure 12, comprising: grouped element 121 and data synchronisation unit 122.
Grouped element 121 is used for the source data table is treated that the shared storage area of synchrodata is divided into some groups.
Preferable, data synchronisation unit 122 further can comprise: subelement 1221 and judgment sub-unit 1222 are carried out in control.
Judgment sub-unit 1222, be used for the treating synchrodata and successfully submit to of a synchronous storage area after, judge whether to handle all storage areas in the corresponding group.
The device of above-mentioned synchrodata also comprises:
Storage area determining unit 123 is used for definite source data table and treats some storage areas that synchrodata is shared.
Address information determining unit 124 is used for determining the storage address information that each storage area is treated synchrodata.
Above-mentioned data synchronization unit can have multiple mode in specific implementation, for example: can also can realize by the mode of software and combination of hardware by the implementation of software.
Method of data synchronization that the embodiment of the present application provides and device, by setting up in advance and a storage remains the two corresponding mark tables of data of opening one's eyes wide of the source data table of synchrodata, the mark tables of data of opening one's eyes wide that application server visits by the top level view of setting up in advance that described top level view points to, pass through top level view, application layer does not need to be concerned about which data specifically open table from, has reduced and has used the complexity that realizes.
When to described target matrix, during synchrodata, at first determining two one of opening one's eyes wide mark tables of data hollow and be the current goal tables of data from described source data table; Then from source data table synchrodata to the current goal tables of data; After the synchrodata success, upgrade top level view, make it point to the current goal tables of data.In the said method, two open one's eyes wide the mark tables of data be used alternatingly, the tables of data mutually noninterfere of the tables of data of application server access and synchrodata routine access, avoided mass data transferred to cancel/the rollback table space in, effectively avoided cancelling/cause queuing and jam after the rollback space is taken, improved data sync speed and efficient again.
In the process of synchrodata, on the basis of allocation of space of fully showing and storage principle in conjunction with ORACLE, and utilize the inherent memory mechanism of ORACLE tables of data, data in each storage area are to target matrix synchronously, when data migtation, carry out at each storage area, realize pressing storage area parallel synchronous data, segmentation is submitted to, in the process of data sync, minimizing is cancelled/the taking for a long time of rollback table space foreground ORACLE database, make to cancel/the overall utilization rate of rollback table space significantly reduces, and reduces greatly to take and cancels/possibility of rollback table space, guarantees the safety in production data storehouse.When synchrodata, also can divide into groups to storage area, adopt the parallel mode migration data further to improve the efficient of migration data.
When the data failure in synchronous some storage areas, adopt rollback mechanism, improved system survivability greatly, simultaneously by rollback number of times controlling mechanism, avoided the unlimited rollback round-robin generation of waiting for death.And owing to operate at each storage area, and the table space of visiting with application server does not disturb, the original state of employed current data table is that sky does not have interfering data, therefore, when data sync is interrupted, can continue synchronous partial data of not coming synchronously, make the breakpoint transmission of synchrodata become possibility.
Simultaneously, finished the number of the storage area of data synchronously owing to be provided with the counter variable statistics, thereby can realize the synchronous progress of data, speed and efficient are monitored.
The above; it only is the preferable embodiment of the application; but the application's protection domain is not limited thereto; anyly be familiar with those skilled in the art in the technical scope that the application discloses; the variation that can expect easily, replace or be applied to other similar devices, all should be encompassed within the application's the protection domain.Therefore, the application's protection domain should be as the criterion with the protection domain of claims.
Claims (9)
1. the method for a synchrodata is characterized in that, comprising:
To treat in the source data table that the shared storage area of synchrodata is divided into some groups;
Several data synchronizing process of parallel starting correspondence are by treating that synchrodata is in the current goal tables of data in each the described storage area in the synchronously corresponding group of each data synchronizing process.
2. the method for claim 1 is characterized in that, is describedly carried out successively by the data in each the described storage area in the synchronously corresponding group of each data synchronizing process, specifically comprises:
Each data synchronizing process is handled the synchrodata for the treatment of in the storage area in the described corresponding group;
Synchronously after the treating synchrodata and successfully submit in the described storage area, judge whether to handle all storage areas in the described corresponding group;
When all storage areas in the described corresponding group that has been untreated, start the process for the treatment of synchrodata in the next storage area in the described corresponding group of handling.
3. method as claimed in claim 2 is characterized in that, also comprises: when in synchronous some described storage areas treat the synchrodata failure time, restart the process for the treatment of synchrodata in this storage area synchronously;
When the number of times that restarts surpasses setting threshold, finish the interior process for the treatment of synchrodata of this storage area synchronously.
4. as the arbitrary described method of claim 1-3, it is characterized in that, will treat in the source data table that the shared storage area of synchrodata was divided into before some groups, also comprised:
Determine some storage areas for the treatment of in the source data table that synchrodata is shared; And
Determine the storage address information for the treatment of synchrodata in each described storage area.
5. method as claimed in claim 4 is characterized in that, the described storage address information for the treatment of synchrodata in each described storage area of determining specifically comprises:
Determine lowest address and the maximum address for the treatment of the storage of synchrodata in each described storage area;
The address realm of determining that comprises lowest address and maximum address is stored in the middle table.
6. the device of a synchrodata is characterized in that, comprising:
Grouped element is used for the source data table is treated that the shared storage area of synchrodata is divided into some groups;
Data synchronisation unit is used for several data synchronizing process of parallel starting correspondence, by treating that synchrodata is in the current goal tables of data in each the described storage area in the corresponding synchronously group of each data synchronizing process.
7. device as claimed in claim 6 is characterized in that, described data synchronisation unit specifically comprises: subelement and judgment sub-unit are carried out in control;
Subelement is carried out in control, is used for controlling the synchrodata for the treatment of in the described corresponding storage area of organizing of each data synchronizing process processing; And during all storage areas in judgment sub-unit is judged the described corresponding group that has been untreated, start the process for the treatment of synchrodata in the next storage area in described corresponding group of handling;
Judgment sub-unit, be used for the treating synchrodata and successfully submit to of a synchronous described storage area after, judge whether to handle all storage areas in the described corresponding group.
8. device as claimed in claim 7 is characterized in that, described data synchronisation unit also comprises:
Restart subelement, be used for when synchronous some described storage areas treat the synchrodata failure time, restart the process for the treatment of synchrodata in this storage area synchronously; And when the number of times that restarts surpasses setting threshold, finish the process for the treatment of synchrodata in synchronous this storage area.
9. as the arbitrary described device of claim 6-8, it is characterized in that, also comprise:
The storage area determining unit is used for definite source data table and treats some storage areas that synchrodata is shared;
The address information determining unit is used for determining the storage address information that each described storage area is treated synchrodata.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN2011100240321A CN102081665A (en) | 2008-12-10 | 2008-12-10 | Data synchronization method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN2011100240321A CN102081665A (en) | 2008-12-10 | 2008-12-10 | Data synchronization method and device |
Related Parent Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CNA2008101793946A Division CN101419616A (en) | 2008-12-10 | 2008-12-10 | Data synchronization method and apparatus |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN102081665A true CN102081665A (en) | 2011-06-01 |
Family
ID=44087626
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN2011100240321A Pending CN102081665A (en) | 2008-12-10 | 2008-12-10 | Data synchronization method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN102081665A (en) |
Cited By (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104182441A (en) * | 2014-02-27 | 2014-12-03 | 无锡天脉聚源传媒科技有限公司 | Data sheet synchronization method and device |
| CN104750744A (en) * | 2013-12-31 | 2015-07-01 | 中国移动通信集团河南有限公司 | Method and device for synchronizing compressed data on basis of Oracle databases |
| CN106446090A (en) * | 2016-09-12 | 2017-02-22 | 浪潮软件股份有限公司 | Method for cross-database transmission of mass data |
| CN106484560A (en) * | 2015-08-25 | 2017-03-08 | 阿里巴巴集团控股有限公司 | Method of data synchronization and device |
| CN106844694A (en) * | 2017-01-24 | 2017-06-13 | 恒生电子股份有限公司 | For the method and apparatus of synchrodata |
| CN107295030A (en) * | 2016-03-30 | 2017-10-24 | 阿里巴巴集团控股有限公司 | A kind of method for writing data, device, data processing method, apparatus and system |
| CN107870960A (en) * | 2017-06-25 | 2018-04-03 | 平安科技（深圳）有限公司 | Data move system and method |
| CN109299116A (en) * | 2018-12-05 | 2019-02-01 | 浪潮电子信息产业股份有限公司 | Data synchronization method, device, equipment and readable storage medium |
| CN109492000A (en) * | 2018-10-09 | 2019-03-19 | 郑州云海信息技术有限公司 | A kind of clone method and system of database table |
| CN109522152A (en) * | 2018-09-06 | 2019-03-26 | 清华大学 | Electron microscopic data disaster recovery method, storage medium and device |
| CN109656999A (en) * | 2018-10-11 | 2019-04-19 | 平安科技（深圳）有限公司 | Method of data synchronization, equipment, storage medium and the device of big data quantity |
| CN111400104A (en) * | 2020-03-24 | 2020-07-10 | 杭州数梦工场科技有限公司 | Data synchronization method and device, electronic equipment and storage medium |
| CN114504828A (en) * | 2022-02-08 | 2022-05-17 | 北京趣玩天橙科技有限公司 | Method and system for realizing memory consistency through data rollback |
-
2008
- 2008-12-10 CN CN2011100240321A patent/CN102081665A/en active Pending
Cited By (19)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104750744B (en) * | 2013-12-31 | 2018-05-22 | 中国移动通信集团河南有限公司 | A kind of compressed data synchronous method and device based on oracle database |
| CN104750744A (en) * | 2013-12-31 | 2015-07-01 | 中国移动通信集团河南有限公司 | Method and device for synchronizing compressed data on basis of Oracle databases |
| CN104182441A (en) * | 2014-02-27 | 2014-12-03 | 无锡天脉聚源传媒科技有限公司 | Data sheet synchronization method and device |
| CN106484560A (en) * | 2015-08-25 | 2017-03-08 | 阿里巴巴集团控股有限公司 | Method of data synchronization and device |
| CN106484560B (en) * | 2015-08-25 | 2019-01-15 | 阿里巴巴集团控股有限公司 | Method of data synchronization and device |
| CN107295030A (en) * | 2016-03-30 | 2017-10-24 | 阿里巴巴集团控股有限公司 | A kind of method for writing data, device, data processing method, apparatus and system |
| CN106446090A (en) * | 2016-09-12 | 2017-02-22 | 浪潮软件股份有限公司 | Method for cross-database transmission of mass data |
| CN106844694A (en) * | 2017-01-24 | 2017-06-13 | 恒生电子股份有限公司 | For the method and apparatus of synchrodata |
| CN106844694B (en) * | 2017-01-24 | 2019-11-15 | 恒生电子股份有限公司 | Method and apparatus for synchrodata |
| CN107870960A (en) * | 2017-06-25 | 2018-04-03 | 平安科技（深圳）有限公司 | Data move system and method |
| CN109522152A (en) * | 2018-09-06 | 2019-03-26 | 清华大学 | Electron microscopic data disaster recovery method, storage medium and device |
| CN109522152B (en) * | 2018-09-06 | 2021-05-14 | 清华大学 | Disaster recovery method, storage medium and device for electron microscope data |
| CN109492000A (en) * | 2018-10-09 | 2019-03-19 | 郑州云海信息技术有限公司 | A kind of clone method and system of database table |
| CN109656999A (en) * | 2018-10-11 | 2019-04-19 | 平安科技（深圳）有限公司 | Method of data synchronization, equipment, storage medium and the device of big data quantity |
| CN109656999B (en) * | 2018-10-11 | 2024-03-15 | 平安科技（深圳）有限公司 | Method, device, storage medium and apparatus for synchronizing large data volume data |
| CN109299116A (en) * | 2018-12-05 | 2019-02-01 | 浪潮电子信息产业股份有限公司 | Data synchronization method, device, equipment and readable storage medium |
| CN111400104A (en) * | 2020-03-24 | 2020-07-10 | 杭州数梦工场科技有限公司 | Data synchronization method and device, electronic equipment and storage medium |
| CN111400104B (en) * | 2020-03-24 | 2021-09-28 | 杭州数梦工场科技有限公司 | Data synchronization method and device, electronic equipment and storage medium |
| CN114504828A (en) * | 2022-02-08 | 2022-05-17 | 北京趣玩天橙科技有限公司 | Method and system for realizing memory consistency through data rollback |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN101419616A (en) | Data synchronization method and apparatus | |
| CN102081665A (en) | Data synchronization method and device | |
| Huang et al. | X-Engine: An optimized storage engine for large-scale E-commerce transaction processing | |
| US10678808B2 (en) | Eager replication of uncommitted transactions | |
| EP3563268B1 (en) | Scalable database system for querying time-series data | |
| US9589041B2 (en) | Client and server integration for replicating data | |
| US7529752B2 (en) | Asymmetric streaming record data processor method and apparatus | |
| CN101334797B (en) | Distributed file systems and its data block consistency managing method | |
| US20090248617A1 (en) | Optimization technique for dealing with data skew on foreign key joins | |
| US20200142992A1 (en) | Dual-stack architecture that integrates relational database with blockchain | |
| US8386445B2 (en) | Reorganizing database tables | |
| Yang et al. | F1 Lightning: HTAP as a Service | |
| KR102038529B1 (en) | System for processing real-time data modification of in-memory database | |
| US20120290595A1 (en) | Super-records | |
| CN104834635A (en) | Data processing method and device | |
| US20150278327A1 (en) | Asynchronous global index maintenance during partition maintenance | |
| CN107480254B (en) | Online load balancing method suitable for distributed memory database | |
| US20130332465A1 (en) | Database management device and database management method | |
| TWI475409B (en) | Data synchronization method and device | |
| US11449521B2 (en) | Database management system | |
| US8290935B1 (en) | Method and system for optimizing database system queries | |
| CN107924399B (en) | Search processing system and method for processing search request with unknown data transmission amount to host | |
| JP2000020374A (en) | Replication control system | |
| US12105709B2 (en) | Blocked index join | |
| JPH0351912A (en) | Spool area return system for each data set |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| REG | Reference to a national code |
Ref country code: HK Ref legal event code: DE Ref document number: 1154679 Country of ref document: HK |
|
| C12 | Rejection of a patent application after its publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20110601 |
|
| REG | Reference to a national code |
Ref country code: HK Ref legal event code: WD Ref document number: 1154679 Country of ref document: HK |