CN104679841A - Consumption terminal data flow copying method and system - Google Patents
Consumption terminal data flow copying method and system Download PDFInfo
- Publication number
- CN104679841A CN104679841A CN201510072163.5A CN201510072163A CN104679841A CN 104679841 A CN104679841 A CN 104679841A CN 201510072163 A CN201510072163 A CN 201510072163A CN 104679841 A CN104679841 A CN 104679841A
- Authority
- CN
- China
- Prior art keywords
- statement
- copy
- perform
- batch
- target database
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Abstract
The invention discloses a consumption terminal data flow copying method and system. The method comprises the following steps: a copying step: pulling an accident log of a source database from a distributed type message system, and analyzing each source accident log into corresponding database statements to be used as copied statements; executing a data persistence step; a backtracking step: when a consumption terminal is abnormal, backtracking a position before a healthy point and pulling an accident log of the source database from the distributed type message system again, and analyzing the accident log into corresponding database statements to be used as copied statements; executing the data persistence step; the data persistence step: executing each copied statement in sequence in a target database corresponding to the consumption terminal. According to the method, when the abnormal condition occurs, the abnormal condition recovery of the database is realized by using a backtracking function of the distributed type message system, and the data persistence is carried out by using flexible conversion of the statements of the database, so that various conflict conditions are adapted to.
Description
Technical field
The present invention relates to database-related techniques field, particularly a kind of consumption end data stream clone method and system.
Background technology
In today of e-commerce development scale increasingly; large electricity Shang company have accumulated a large amount of businessman users; simultaneously; there are many independent software vendors (ISV) for these businessmans production software; these ISV need the data of electricity Shang company to provide service for businessman, so data opening uses to ISV by electricity Shang company become inevitable trend.
Data are open many forms: application programming interfaces call (API), data warehouse solution (historical data analysis), businessman's customer data base service etc., wherein database service refers to and the merchant data of ISV software directly will be used to be pushed to database.The features such as the service of this mode has in real time, accurate, stable.
The stream of database copies and refers to, by the altering event collected, analyze, stored data base running log obtains source database, is supplied to consumption end (destination end), consumes end by running log or event reduction, to reach the object copied.This mode has flexible configuration, supports heterogeneous database and to features such as network requirement are low, is widely used application under the scene such as database replication and disaster tolerance, be especially applicable to the scene that the real-time incremental of database is synchronous.
1.2.1 the technical scheme of prior art:
The replicated product of current comparative maturity has the Canal & Otter etc. of Replication, Alibaba of Mysql.
Mysql Replication based on binary log file (binlog) leader follower replication, the binary log file record data variation of master data base.Have three thread collaborative works to complete to copy, the Dump thread on master library be responsible for extract journal file, be responsible for the journal file of master library to be synchronized to this locality from the IO thread storehouse, from the SQL thread storehouse be responsible for by the journal file of IO thread synchronization apply from storehouse.
Canal & Otter is distributed data base synchro system.Canal simulates Mysql Slave interaction protocol, and pretend oneself for Slave, send Dump agreement to Master, Master pushes journal file Canal.Otter based on Canal, resolve Canal receive journal file, use distributed thought, multiple node cooperative work by the change application of master library to object library.
Prior art is all the streaming reproduction technologies based on journal file, is all the High Availabitity of database, copies in real time, the solution of the important application such as disaster tolerance.The abnormal conditions such as Mysql Replication technology application function and stability in production system is slightly weak, and the principal and subordinate of database is inconsistent happen occasionally, once comparatively difficult from the recovery in storehouse after abnormal; Canal & Otter is relatively powerful with Mysql Replication, and employ distributed thought consuming end, implementation is comparatively complicated, by not a duck soup in its flexible Application to oneself business.
At the consumption end that stream copies; different consumption ends is different to the demand of data; need and configure flexibly; prior art cannot do unified management and configuration to the consumption end of constantly change; often run into when reducing the event of source database: delay time is long, degraded performance; the problems such as consistance is difficult to ensure, abnormal restoring difficulty.Under data open service scene, need the source database event filtering to subscribing to, the complex operations such as conversion, prior art cannot be tackled flexibly.
Summary of the invention
Based on this, be necessary to be difficult to the technical matters of guarantee, abnormal restoring difficulty for prior art existing data base consistency(-tance) when consuming end data stream and copying, a kind of consumption end data stream clone method and system are provided.
A kind of consumption end data stream clone method, comprising:
Copy step, comprising: each described source event daily record, as source event daily record, is resolved to corresponding database statement as copy-statement by the event log pulling source database from distributed information system, performs persistence step;
Backtracking step, comprising: when consuming that end is abnormal to be occurred, tracing back to position before healthy point as backtracking position, from distributed information system, again pulling the event log of source database, described event log is resolved to corresponding database statement as copy-statement, perform persistence step;
Persistence step, comprising: in the target database that described consumption end is corresponding, perform each described copy-statement successively.
A kind of consumption end data stream dubbing system, comprising:
Replication module, for: each described source event daily record, as source event daily record, is resolved to corresponding database statement as copy-statement by the event log pulling source database from distributed information system, performs persistence module;
Backtracking module, for: when consuming that end is abnormal to be occurred, tracing back to position before healthy point as backtracking position, from distributed information system, again pulling the event log of source database, described event log is resolved to corresponding database statement as copy-statement, perform persistence module;
Persistence module, for: in the target database that described consumption end is corresponding, perform each described copy-statement successively.
The event log of source database is resolved to database statement by the present invention, and perform in target database, then when occurring extremely and utilize the position back track function fulfillment database backtracking of distributed system, usage data storehouse statement can carry out data insertion neatly, adapts to various conflict situations.Therefore, the present invention is very flexible and reliable.
Accompanying drawing explanation
Fig. 1 is the workflow diagram of a kind of data base management method of the present invention;
Fig. 2 is the system architecture diagram of preferred embodiment;
Fig. 3 is the workflow diagram of preferred embodiment;
Fig. 4 is the workflow diagram of the persistence operation of preferred embodiment;
Fig. 5 is a kind of construction module figure consuming end data stream dubbing system of the present invention.
Embodiment
Below in conjunction with the drawings and specific embodiments, the present invention will be further described in detail.
Be illustrated in figure 1 the workflow diagram of a kind of data base management method of the present invention, comprise:
Step S101, comprising: each described source event daily record, as source event daily record, is resolved to corresponding database statement as copy-statement by the event log pulling source database from distributed information system, performs step S103;
Step S102, comprising: when consuming that end is abnormal to be occurred, tracing back to position before healthy point as backtracking position, from distributed information system, again pulling the event log of source database, described event log is resolved to corresponding database statement as copy-statement, perform step S103;
Step S103, comprising: in the target database that described consumption end is corresponding, perform each described copy-statement successively.
Source event, in step S101, is resolved to corresponding database statement (being preferably the Structured Query Language (SQL) of standard, i.e. Structured Query Language, SQL) by the present invention.Then in target database, copy-statement is performed in step s 103.There is the faults such as the machine of delaying in the abnormal consumption end that refers to of consumption end, when occurring that consumption end is abnormal, recalls source database.A timing node before abnormal time of origin point can be selected when recalling as healthy point, and in the journal file of database, position is recalled in the position corresponding to this health point namely.The all daily records of source database after this backtracking position are again resolved to corresponding database statement and perform step S103 and copies.
What the present invention adopted is that database statement copies, simple journal file binary reproduction adopted compared to existing technology, and database statement of the present invention copies more flexible, and can adapt to more how different databases.For different databases, the form of its journal file has difference, if adopt the journal file binary reproduction of prior art, then only can copy between the database of same format, and the situation of source database and target database different-format cannot be adapted to.Present invention employs database statement translation source database, in target database, then perform the database statement through translation, make to copy more flexible between database, various different database format can be adapted to.
Wherein in an embodiment, described step S103, specifically comprises:
Initialization performs sub-step, comprising: be saved in queue successively by described copy-statement, and the copy-statement of continuous multiple execution identical data operation is organized preservation as batch, optimizes sub-step successively to each copy-statement execution analysis;
Analysis optimization sub-step, comprising: if performed copy-statement is in batch group, then all copy-statements in batch group are merged into batch statement, perform batch processing sub-step, otherwise performs wall scroll execution sub-step;
Batch processing sub-step, comprising: in described target database, perform described batch statement;
Wall scroll performs sub-step, comprising: in described target database, perform described current copy-statement.
The copy-statement of continuous multiple execution identical data operation merges as a batch statement by the present embodiment, then only needing by once performing, just can complete copying of multiple similar events in target database.
Data manipulation statement (SQL of standard) to be modified statement used to data in database.Conventional has: DML, DDL and DCL.DML (data manipulation language): they are SELECT, UPDATE, INSERT, DELETE, these 4 orders are used to operate the data of lane database.
DDL (data definition language):, main order has CREATE, ALTER, DROP etc., and DDL is mainly used in the structure of definition or change table (TABLE), data type, the link between table and constraint etc.
DCL (Data Control Language): be database controlling functions.Be used to the statement arranging or change database user or role-security, comprise (grant, deny, revoke etc.) statement.
For multiple database statement with identical data manipulation keyword, batch operation can be adopted once to perform.Thus avoid the repeatedly read-write of database, to improve execution efficiency.
Wherein in an embodiment:
Described wall scroll performs in sub-step, also comprises: if perform described current copy-statement failure in described target database, then:
If described current copy-statement is for inserting statement and running into major key conflict when performing described insertion statement in described target database, then described insertion statement is converted to and upgrades statement accordingly, and again in described target database, perform described renewal statement, if described current copy-statement is for upgrading statement, and do not find corresponding record when performing described renewal statement in described target database, then described renewal statement is converted to and inserts statement accordingly, and again in described target database, perform described insertion statement, if described current copy-statement is cancel statement, and do not find corresponding record when performing described cancel statement in described target database, then ignore described cancel statement.
When performing copy-statement, conflict may be there is, particularly when occurring extremely needing backtracking.Situation about managing conflict is needed to have several:
1. insert statement and run into major key conflict, find had identical major key to exist when namely inserting, this situation is because performed insertion statement to this major key in target database, therefore again inserts and causes major key conflict.For this kind of situation, insertion statement is converted to renewal statement.Conversion regime is that original insertion statement is converted to renewal statement, such as, SQL statement " INSERT INTO shows (field 1; major key field) VALUES (value 1, Major key) " is converted to " UPDATE shows SET field 1=value 1WHERE major key field=Major key (value 1) ".
2. upgrade statement and do not find corresponding record, upgrading statement is that in his-and-hers watches, existing record upgrades, when occurring this kind of situation, represent in target database there is not corresponding major key, its reason had performed the related commands such as deletion in target database to be deleted by this record, and upgrade again when performing backtracking.For this kind of situation, renewal statement is converted to insertion statement.Conversion regime is that original renewal statement is converted to insertion statement.Such as, SQL statement " UPDATE shows SET field 1=value 1WHERE major key field=Major key " is converted to " INSERT INTO shows (field 1, major key field) VALUES (value 1, Major key) ".
3. cancel statement does not find corresponding record, and this situation is because target database had deleted this record, but once more performs deletion when recalling, and for this kind of situation, adopts the mode ignoring described cancel statement to process.
Wherein in an embodiment:
Described batch processing sub-step, also comprises: if described batch statement is batch insertion, and run into major key conflict when described target database performs the insertion of described batch, then perform wall scroll and perform sub-step;
Described wall scroll performs sub-step, also comprise: if perform the success of described copy-statement in described target database, and described current copy-statement is in batch group, then record successful number of times, when described number of times exceedes predetermined threshold value, unenforced copy-statement in corresponding batch group is merged into batch statement, performs batch processing sub-step.
The present embodiment is optimized balance to batch processing and wall scroll process, when batch processing runs into conflict, then represent that this batch processing is the process to backtracking, be therefore converted into wall scroll and perform to change copy-statement for various collision event, to improve the fault-tolerance copied.
And when the successful number of times of wall scroll process reaches predetermined threshold value, then show that recalling the conflict caused has pass by, and now can be converted to batch processing to improve processing speed.
Wherein in an embodiment, also comprise:
At least one pushes the user configured filtering rule of node timing acquisition and consumption position, perform described step S101 and step S103, described filtering rule comprises the event rules for limiting source event daily record and the goal rule for limiting target database, in described step S101, described source event daily record meets described event rules, in described step S103, described target database meets described goal rule;
Described propelling movement node, when performing described step S101, pulls source event daily record after consumption position, and after the described step S103 of normal execution, upgrades the position that source event day aim in described distributed information system of consumption position for finally pulling;
The situation that node performs described step S101 and described step S103 is pushed by management node monitoring, if described step S101 and described step S103 performs occur that then management node selects new propelling movement node to perform described step S102 and described step S103 extremely.
In the present embodiment, user can input various filtering rule by the human-computer interaction interface provided by management node.Filtering rule refers to that user obtains the data of what type of source database and is saved in which target database.Therefore, push node timing according to this filtering rule, resolve the source database log event filtering and pull, perform target database.Management node is also monitored and is pushed the situation that node performs step S101 simultaneously, and when finding to occur abnormal, the position traced back to before healthy point re-executes.Be illustrated in figure 2 the system architecture diagram of preferred embodiment, comprise:
Distributed information system 21, management node 22, propelling movement node 23 and database 24.Wherein, distributed information system 21 manages the routine work of various propelling movement node 23; and state, the abnormal restoring function of the operation that management node 22 provides the configuration of filtering rule, management pushes node 23, monitoring propelling movement node; particularly; management node 22 provides user interface and timed task, and operation interface, for the configurating filtered rule of user, controls the operation pushing node 23; timed task is for monitoring the operation of propelling movement task, and abnormal transfer.Push the instruction of node 23 receiving management node 22, the source database in database 24 subscribed to, filter event, event is resolved to stsndard SQL and is persisted to the customer data base in database 24.Push node 23 and adopt multinode, one process, the realization of multithreading, that is: propelling movement task may operate on multiple physical node, the each port of every platform physical node runs a propelling movement process, the multiple thread of each propelling movement process initiation, and each thread is responsible for propelling movement target database.
The restarting of system, upgrade maintenance, abnormality processing can't long lasting effect to the consistance of target database, because the characteristic of the distributed information system that the present invention uses can be recalled by position-based, when occurring abnormal, the position of receiving before impact can be traced back to, by the object library that the event log after backtracking position is applied again, and the problem managed conflict.
Be illustrated in figure 3 the workflow diagram of preferred embodiment, comprise:
Step S301, pushes node initializing, reads the position pushed last time, loads filtering rule, opening timing task;
Step S302, the timing of propelling movement node pulls the event source database from distributed information system;
Step S303, pushes node and event is resolved to SQL statement;
Step S304, pushes node according to filtering rule, filters out only to the SQL statement of certain target database;
Step S305, pushes node and above-mentioned SQL statement is saved in persistence operation queue, and the SQL statement of adjacent execution identical data operation is preserved as a group;
Step S306, pushes node and performs persistence operation successively in target database.
Step S307, pushes node persistence and operate this position of rear renewal, so as to push next time and abnormal restoring or startup time from this position, pull message.
Be illustrated in figure 4 the workflow diagram of the persistence operation of preferred embodiment, comprise:
Step S401, judges whether persistence queue is empty, if be sky, terminate, otherwise selects SQL statement successively, perform step S402;
Step S402, if selected SQL statement in a group, then performs step S403, otherwise performs step S404;
Step S403, transfers batch statement to, performs in target database by SQL statement unenforced in same group, if run into conflict at the end of execution, then perform step S404, otherwise perform step S401;
Step S404, wall scroll performs SQL statement, if run succeeded, then performs step S406, otherwise performs step S405;
Step S405, is undertaken inserting/upgrade conversion by the SQL statement of conflict, performs step S404;
Step S406, if performed SQL is in a group, then performs step S407, otherwise, perform step S401;
Step S407, record monomer runs succeeded number of times, if wall scroll runs succeeded, number of times reaches threshold value, then perform step S403, otherwise, perform step S408;
Step S408, if in performed SQL place group, also has unenforced SQL statement, then selects next SQL statement in this group, perform step S404, otherwise perform step S401.
Be persisted to target database to perform by lot sequence.If batch runs into major key conflict, rollback affairs when inserting, change the mode that wall scroll inserts into.When wall scroll inserts and runs into major key conflict, statement is converted to renewal, re-executes; If wall scroll upgrades be not updated to record, renewal is converted to insertion, re-executes, ensure that all records of source perform object library with this conversion.
Wall scroll runs succeeded to change into after threshold values that number of times reaches certain again and is performed by the statement batch under this batch, and run into conflict and change wall scroll again into and perform, so conversion is until performing queue be sky.
Be illustrated in figure 5 a kind of construction module figure consuming end data stream dubbing system of the present invention, comprise:
Replication module 501, for: each described source event daily record, as source event daily record, is resolved to corresponding database statement as copy-statement by the event log pulling source database from distributed information system, performs persistence module 503;
Backtracking module 502, for: when consuming end and extremely occurring, trace back to position before healthy point as backtracking position, again from distributed information system, pull the event log of source database, described event log is resolved to corresponding database statement as copy-statement, perform persistence module 503;
Persistence module 503, for: in the target database that described consumption end is corresponding, perform each described copy-statement successively.
Wherein in an embodiment, described persistence module, specifically comprises:
Initialization implementation sub-module, for: described copy-statement is saved in queue successively, and the copy-statement of continuous multiple execution identical data operation is organized preservation as batch, successively submodule is optimized to each copy-statement execution analysis;
Analysis optimization submodule, for: if performed copy-statement is in batch group, then all copy-statements in batch group is merged into batch statement, perform batch processing submodule, otherwise perform wall scroll implementation sub-module;
Batch processing submodule, for: in described target database, perform described batch statement;
Wall scroll implementation sub-module, for: in described target database, perform described current copy-statement.
Wherein in an embodiment:
In described wall scroll implementation sub-module, also for: if perform the failure of described current copy-statement in described target database, then:
If described current copy-statement is for inserting statement and running into major key conflict when performing described insertion statement in described target database, then described insertion statement is converted to and upgrades statement accordingly, and again in described target database, perform described renewal statement, if described current copy-statement is for upgrading statement, and do not find corresponding record when performing described renewal statement in described target database, then described renewal statement is converted to and inserts statement accordingly, and again in described target database, perform described insertion statement, if described current copy-statement is cancel statement, and do not find corresponding record when performing described cancel statement in described target database, then ignore described cancel statement.
Wherein in an embodiment:
Described batch processing submodule, also for: insert if described batch statement is batch, and perform when described batch inserts in described target database and run into major key conflict, then perform wall scroll implementation sub-module;
Described wall scroll implementation sub-module, also for: if perform in described target database described copy-statement success, and described current copy-statement is in batch group, then record successful number of times, when described number of times exceedes predetermined threshold value, unenforced copy-statement in corresponding batch group is merged into batch statement, performs batch processing submodule.
Wherein in an embodiment, also comprise:
Rule control module, for: adopt and push node as consumption end, at least one pushes the user configured filtering rule of node timing acquisition and consumption position, perform described copy step and persistence step, described filtering rule comprises the event rules for limiting source event daily record and the goal rule for limiting target database, and in described copy step, described source event daily record meets described event rules, in described persistence step, described target database meets described goal rule;
Consumption location updating module, for: described propelling movement node is when performing described copy step, source event daily record is pulled after consumption position, and after the described persistence step of normal execution, upgrading consumption position is the position aimed at source event day in described distributed information system finally pulled;
Monitoring module, for: push by management node monitoring the situation that node performs described copy step and described persistence step, if described copy step and described persistence step perform occur that then management node selects new propelling movement node to perform described backtracking step and described persistence step extremely.
It should be pointed out that for the person of ordinary skill of the art, without departing from the inventive concept of the premise, can also make some distortion and improvement, these all belong to protection scope of the present invention.Therefore, the protection domain of patent of the present invention should be as the criterion with claims.
Claims (10)
1. consume an end data stream clone method, it is characterized in that, comprising:
Copy step, comprising: each described source event daily record, as source event daily record, is resolved to corresponding database statement as copy-statement by the event log pulling source database from distributed information system, performs persistence step;
Backtracking step, comprise: when consuming end and extremely occurring, the position before the healthy point of backtracking, pulls the event log of source database again from distributed information system, described event log is resolved to corresponding database statement as copy-statement, perform persistence step;
Persistence step, comprising: in the target database that described consumption end is corresponding, perform each described copy-statement successively.
2. consumption end data stream clone method according to claim 1, it is characterized in that, described persistence step, specifically comprises:
Initialization performs sub-step, comprising: be saved in queue successively by described copy-statement, and the copy-statement of continuous multiple execution identical data operation is organized preservation as batch, optimizes sub-step successively to each copy-statement execution analysis;
Analysis optimization sub-step, comprising: if performed copy-statement is in batch group, then all copy-statements in batch group are merged into batch statement, perform batch processing sub-step, otherwise performs wall scroll execution sub-step;
Batch processing sub-step, comprising: in described target database, perform described batch statement;
Wall scroll performs sub-step, comprising: in described target database, perform described current copy-statement.
3. consumption end data stream clone method according to claim 2, is characterized in that:
Described wall scroll performs in sub-step, also comprises: if perform described current copy-statement failure in described target database, then:
If described current copy-statement is for inserting statement and running into major key conflict when performing described insertion statement in described target database, then described insertion statement is converted to and upgrades statement accordingly, and again in described target database, perform described renewal statement, if described current copy-statement is for upgrading statement, and do not find corresponding record when performing described renewal statement in described target database, then described renewal statement is converted to and inserts statement accordingly, and again in described target database, perform described insertion statement, if described current copy-statement is cancel statement, and do not find corresponding record when performing described cancel statement in described target database, then ignore described cancel statement.
4. consumption end data stream clone method according to claim 3, is characterized in that:
Described batch processing sub-step, also comprises: if described batch statement is batch insertion, and run into major key conflict when described target database performs the insertion of described batch, then perform wall scroll and perform sub-step;
Described wall scroll performs sub-step, also comprise: if perform the success of described copy-statement in described target database, and described current copy-statement is in batch group, then record successful number of times, when described number of times exceedes predetermined threshold value, unenforced copy-statement in corresponding batch group is merged into batch statement, performs batch processing sub-step.
5. consumption end data stream clone method according to claim 1, is characterized in that, also comprise:
Adopt and push node as consumption end, at least one pushes the user configured filtering rule of node timing acquisition and consumption position, perform described copy step and persistence step, described filtering rule comprises the event rules for limiting source event daily record and the goal rule for limiting target database, in described copy step, described source event daily record meets described event rules, and in described persistence step, described target database meets described goal rule;
Described propelling movement node, when performing described copy step, pulls source event daily record after consumption position, and after the described persistence step of normal execution, upgrades the position that source event day aim in described distributed information system of consumption position for finally pulling;
The situation that node performs described copy step and described persistence step is pushed by management node monitoring, if described copy step and described persistence step perform occur that then management node selects new propelling movement node to perform described backtracking step and described persistence step extremely.
6. consume an end data stream dubbing system, it is characterized in that, comprising:
Replication module, for: each described source event daily record, as source event daily record, is resolved to corresponding database statement as copy-statement by the event log pulling source database from distributed information system, performs persistence module;
Backtracking module, for: when consuming that end is abnormal to be occurred, tracing back to position before healthy point as backtracking position, from distributed information system, again pulling the event log of source database, described event log is resolved to corresponding database statement as copy-statement, perform persistence module;
Persistence module, for: in the target database that described consumption end is corresponding, perform each described copy-statement successively.
7. consumption end data stream dubbing system according to claim 6, it is characterized in that, described persistence module, specifically comprises:
Initialization implementation sub-module, for: described copy-statement is saved in queue successively, and the copy-statement of continuous multiple execution identical data operation is organized preservation as batch, successively submodule is optimized to each copy-statement execution analysis;
Analysis optimization submodule, for: if performed copy-statement is in batch group, then all copy-statements in batch group is merged into batch statement, perform batch processing submodule, otherwise perform wall scroll implementation sub-module;
Batch processing submodule, for: in described target database, perform described batch statement;
Wall scroll implementation sub-module, for: in described target database, perform described current copy-statement.
8. consumption end data stream dubbing system according to claim 7, is characterized in that:
In described wall scroll implementation sub-module, also for: if perform the failure of described current copy-statement in described target database, then:
If described current copy-statement is for inserting statement and running into major key conflict when performing described insertion statement in described target database, then described insertion statement is converted to and upgrades statement accordingly, and again in described target database, perform described renewal statement, if described current copy-statement is for upgrading statement, and do not find corresponding record when performing described renewal statement in described target database, then described renewal statement is converted to and inserts statement accordingly, and again in described target database, perform described insertion statement, if described current copy-statement is cancel statement, and do not find corresponding record when performing described cancel statement in described target database, then ignore described cancel statement.
9. consumption end data stream dubbing system according to claim 8, is characterized in that:
Described batch processing submodule, also for: insert if described batch statement is batch, and perform when described batch inserts in described target database and run into major key conflict, then perform wall scroll implementation sub-module;
Described wall scroll implementation sub-module, also for: if perform in described target database described copy-statement success, and described current copy-statement is in batch group, then record successful number of times, when described number of times exceedes predetermined threshold value, unenforced copy-statement in corresponding batch group is merged into batch statement, performs batch processing submodule.
10. consumption end data stream dubbing system according to claim 6, is characterized in that, also comprise:
Rule control module, for: adopt and push node as consumption end, at least one user configured filtering rule of propelling movement node timing acquisition described and consumption position, perform described copy step and persistence step, described filtering rule comprises the event rules for limiting source event daily record and the goal rule for limiting target database, and in described copy step, described source event daily record meets described event rules, in described persistence step, described target database meets described goal rule;
Consumption location updating module, for: described propelling movement node is when performing described copy step, source event daily record is pulled after consumption position, and after the described persistence step of normal execution, upgrading consumption position is the position aimed at source event day in described distributed information system finally pulled;
Monitoring module, for: push by management node monitoring the situation that node performs described copy step and described persistence step, if described copy step and described persistence step perform occur that then management node selects new propelling movement node to perform described backtracking step and described persistence step extremely.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510072163.5A CN104679841B (en) | 2015-02-11 | 2015-02-11 | A kind of consumption terminal data flow replication method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510072163.5A CN104679841B (en) | 2015-02-11 | 2015-02-11 | A kind of consumption terminal data flow replication method and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN104679841A true CN104679841A (en) | 2015-06-03 |
| CN104679841B CN104679841B (en) | 2018-06-08 |
Family
ID=53314883
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201510072163.5A Active CN104679841B (en) | 2015-02-11 | 2015-02-11 | A kind of consumption terminal data flow replication method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN104679841B (en) |
Cited By (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105260486A (en) * | 2015-11-23 | 2016-01-20 | 郑州悉知信息科技股份有限公司 | Data processing method, device and system |
| CN106301972A (en) * | 2016-11-16 | 2017-01-04 | 北京集奥聚合科技有限公司 | A kind of monitoring data push away method and system in real time |
| CN106302780A (en) * | 2016-08-25 | 2017-01-04 | 北京京东尚科信息技术有限公司 | The method of cluster device bulk transfer data, Apparatus and system, server |
| CN106547801A (en) * | 2015-09-23 | 2017-03-29 | 北京奇虎科技有限公司 | Database data flash back method and apparatus |
| WO2017050176A1 (en) * | 2015-09-25 | 2017-03-30 | 阿里巴巴集团控股有限公司 | Data synchronization method and device |
| CN107016014A (en) * | 2016-09-30 | 2017-08-04 | 阿里巴巴集团控股有限公司 | The method of data synchronization and device of heterogeneous database |
| CN107506451A (en) * | 2017-08-28 | 2017-12-22 | 泰康保险集团股份有限公司 | abnormal information monitoring method and device for data interaction |
| CN108235069A (en) * | 2016-12-22 | 2018-06-29 | 北京国双科技有限公司 | The processing method and processing device of Web TV daily record |
| CN108351788A (en) * | 2015-11-18 | 2018-07-31 | 华为技术有限公司 | The data processing equipment of access to data flow is provided |
| CN109558452A (en) * | 2018-11-19 | 2019-04-02 | 武汉达梦数据库有限公司 | It is a kind of to inquire the synchronous method for building table handling |
| CN110750594A (en) * | 2019-09-30 | 2020-02-04 | 上海视云网络科技有限公司 | Mysql-based real-time cross-network database synchronization method for incremental logs |
| CN111737203A (en) * | 2020-06-09 | 2020-10-02 | 阿里巴巴集团控股有限公司 | Database history log backtracking method, device, system, equipment and storage medium |
| CN113254425A (en) * | 2021-06-24 | 2021-08-13 | 阿里云计算有限公司 | Method, apparatus, system, program and storage medium for database transaction retention |
| CN113688111A (en) * | 2021-10-25 | 2021-11-23 | 深圳市明源云科技有限公司 | Cross-region message copying method, system, electronic equipment and readable storage medium |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1701325A (en) * | 2002-08-01 | 2005-11-23 | 甲骨文国际公司 | Asynchronous information sharing system |
| CN1831810A (en) * | 2005-03-08 | 2006-09-13 | 中国科学院计算技术研究所 | Method and middleware for data base duplicating using middleware in group communication system |
| CN101706795A (en) * | 2009-11-30 | 2010-05-12 | 上海世范软件技术有限公司 | Method for synchronizing data of database in active/standby server |
| CN102262662A (en) * | 2011-07-22 | 2011-11-30 | 浪潮(北京)电子信息产业有限公司 | System, device and method for realizing database data migration in heterogeneous platform |
| CN103870357A (en) * | 2012-12-17 | 2014-06-18 | 中国移动通信集团河南有限公司 | Method and system for carrying out data replication |
| US20140379637A1 (en) * | 2013-06-25 | 2014-12-25 | Microsoft Corporation | Reverse replication to rollback corrupted files |
-
2015
- 2015-02-11 CN CN201510072163.5A patent/CN104679841B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1701325A (en) * | 2002-08-01 | 2005-11-23 | 甲骨文国际公司 | Asynchronous information sharing system |
| CN1831810A (en) * | 2005-03-08 | 2006-09-13 | 中国科学院计算技术研究所 | Method and middleware for data base duplicating using middleware in group communication system |
| CN101706795A (en) * | 2009-11-30 | 2010-05-12 | 上海世范软件技术有限公司 | Method for synchronizing data of database in active/standby server |
| CN102262662A (en) * | 2011-07-22 | 2011-11-30 | 浪潮(北京)电子信息产业有限公司 | System, device and method for realizing database data migration in heterogeneous platform |
| CN103870357A (en) * | 2012-12-17 | 2014-06-18 | 中国移动通信集团河南有限公司 | Method and system for carrying out data replication |
| US20140379637A1 (en) * | 2013-06-25 | 2014-12-25 | Microsoft Corporation | Reverse replication to rollback corrupted files |
Cited By (21)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106547801A (en) * | 2015-09-23 | 2017-03-29 | 北京奇虎科技有限公司 | Database data flash back method and apparatus |
| WO2017050176A1 (en) * | 2015-09-25 | 2017-03-30 | 阿里巴巴集团控股有限公司 | Data synchronization method and device |
| CN106557497A (en) * | 2015-09-25 | 2017-04-05 | 阿里巴巴集团控股有限公司 | A kind of method of data synchronization and device |
| US11016777B2 (en) | 2015-11-18 | 2021-05-25 | Huawei Technologies Co., Ltd. | Data processing device for providing access to a data stream |
| CN108351788A (en) * | 2015-11-18 | 2018-07-31 | 华为技术有限公司 | The data processing equipment of access to data flow is provided |
| CN105260486A (en) * | 2015-11-23 | 2016-01-20 | 郑州悉知信息科技股份有限公司 | Data processing method, device and system |
| CN106302780B (en) * | 2016-08-25 | 2020-07-31 | 北京京东尚科信息技术有限公司 | Method, device and system for batch data transmission of cluster equipment and server |
| CN106302780A (en) * | 2016-08-25 | 2017-01-04 | 北京京东尚科信息技术有限公司 | The method of cluster device bulk transfer data, Apparatus and system, server |
| CN107016014A (en) * | 2016-09-30 | 2017-08-04 | 阿里巴巴集团控股有限公司 | The method of data synchronization and device of heterogeneous database |
| CN107016014B (en) * | 2016-09-30 | 2020-08-04 | 阿里巴巴集团控股有限公司 | Data synchronization method and device for heterogeneous database |
| CN106301972A (en) * | 2016-11-16 | 2017-01-04 | 北京集奥聚合科技有限公司 | A kind of monitoring data push away method and system in real time |
| CN108235069A (en) * | 2016-12-22 | 2018-06-29 | 北京国双科技有限公司 | The processing method and processing device of Web TV daily record |
| CN107506451B (en) * | 2017-08-28 | 2020-11-03 | 泰康保险集团股份有限公司 | Abnormal information monitoring method and device for data interaction |
| CN107506451A (en) * | 2017-08-28 | 2017-12-22 | 泰康保险集团股份有限公司 | abnormal information monitoring method and device for data interaction |
| CN109558452A (en) * | 2018-11-19 | 2019-04-02 | 武汉达梦数据库有限公司 | It is a kind of to inquire the synchronous method for building table handling |
| CN109558452B (en) * | 2018-11-19 | 2020-12-08 | 武汉达梦数据库有限公司 | Synchronization method for query table building operation |
| CN110750594A (en) * | 2019-09-30 | 2020-02-04 | 上海视云网络科技有限公司 | Mysql-based real-time cross-network database synchronization method for incremental logs |
| CN110750594B (en) * | 2019-09-30 | 2023-05-30 | 上海视云网络科技有限公司 | Real-time cross-network database synchronization method based on mysql incremental log |
| CN111737203A (en) * | 2020-06-09 | 2020-10-02 | 阿里巴巴集团控股有限公司 | Database history log backtracking method, device, system, equipment and storage medium |
| CN113254425A (en) * | 2021-06-24 | 2021-08-13 | 阿里云计算有限公司 | Method, apparatus, system, program and storage medium for database transaction retention |
| CN113688111A (en) * | 2021-10-25 | 2021-11-23 | 深圳市明源云科技有限公司 | Cross-region message copying method, system, electronic equipment and readable storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN104679841B (en) | 2018-06-08 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN104679841A (en) | Consumption terminal data flow copying method and system | |
| CN104317843B (en) | A kind of data syn-chronization ETL system | |
| CN103455557B (en) | A kind of structural data synchronous method based on daily record | |
| CN108287835B (en) | Data cleaning method and device | |
| US9542468B2 (en) | Database management system and method for controlling synchronization between databases | |
| US10216589B2 (en) | Smart data replication recoverer | |
| CN110175213A (en) | A kind of oracle database synchronization system and method based on SCN mode | |
| CN106202365B (en) | Method and system for database update synchronization and database cluster | |
| CN103823797A (en) | FTP (file transfer protocol) based real-time industry database data synchronization system | |
| CN102142024A (en) | Logical data backup and rollback using incremental capture in a distributed database | |
| US9037905B2 (en) | Data processing failure recovery method, system and program | |
| CN102508886B (en) | Extensive makeup language (XML)-based method for synchronously updating increment of spatial data | |
| CN109634970A (en) | Table method of data synchronization, equipment, storage medium and device | |
| CN108228755A (en) | The data of MySQL database based on daily record analytic technique to Hadoop platform synchronize clone method | |
| CN115374102A (en) | Data processing method and system | |
| CN114048217A (en) | Incremental data synchronization method and device, electronic equipment and storage medium | |
| CN109672712A (en) | Method of data synchronization, device, super controller, domain controller and storage medium | |
| CN115438122A (en) | Data heterogeneous synchronization system | |
| CN103092866A (en) | Data monitoring method and monitoring device | |
| CN108228756A (en) | Data based on the PG databases of daily record analytic technique to Hadoop platform synchronize clone method | |
| CN114925073A (en) | Distributed database architecture supporting flexible dynamic fragmentation and implementation method thereof | |
| CN114281757A (en) | Database migration method and system and computer readable storage medium | |
| CN105468418A (en) | System and method for upgrading software of smart terminal cluster | |
| CN112905613A (en) | Data bidirectional synchronization method and device for heterogeneous database | |
| CN108196910B (en) | Software version switching method, device and system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |