CN106682141A - Data synchronization method based on business operation logs - Google Patents
Data synchronization method based on business operation logs Download PDFInfo
- Publication number
- CN106682141A CN106682141A CN201611184089.7A CN201611184089A CN106682141A CN 106682141 A CN106682141 A CN 106682141A CN 201611184089 A CN201611184089 A CN 201611184089A CN 106682141 A CN106682141 A CN 106682141A
- Authority
- CN
- China
- Prior art keywords
- data
- business
- business operation
- data synchronization
- synchronization
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/1805—Append-only file systems, e.g. using logs or journals to store data
- G06F16/1815—Journaling file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/178—Techniques for file synchronisation in file systems
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Debugging And Monitoring (AREA)
- Hardware Redundancy (AREA)
Abstract
The invention discloses a data synchronization method based on business operation logs. By using the method, the atomicity of business data and the integrity of data can be ensured, and according to the business data, a direct distribution mechanism instead of stepwise disposing and stepwise reporting is adopted, and therefore the speed of data synchronization is improved. The method uses the business operation logs as the basis of data synchronization, ensures the atomicity and coherence of the business data and the consistency and integrity of data to the maximum extent, and avoids the obvious disadvantage of using a data source completely reporting mode and a data source incrementally reporting mode. In the process of data transmission and distribution, the method abandons the necessary stepwise disposing and reporting modes of a hierarchical disposition system, the direct distribution mechanism is adopted, and there is no need to stepwise conduct file parsing and data operation, so that the data synchronization efficiency is improved. When the data synchronization fails, according to business operation log roll-back, the method prevents the data form data loss errors caused by the failed data synchronization, and the accuracy of data synchronization is improved.
Description
Technical field
The present invention relates to computer information technology field, and in particular to a kind of data syn-chronization side based on business operation daily record
Method.
Background technology
It is hierarchical application management mode in tree-like classification deployment system, data change need in time be reported and issued, to ensure
Data syn-chronizations at different levels are consistent；System is at different levels to have multiple data trnascriptions, it is necessary to using number in data synchronization technology capture data base
According to the change of object, so as to safeguarding the concordance of data bases at different levels and ensureing the high efficiency and reliability of data base.
The method of data synchronization that at present industry is present mainly has two classes：
(1) data source is all reported
It is by special messaging application that the content of the table of a certain moment source database is complete when data syn-chronization is needed
Portion replicates and reports to superior unit step by step, can be based on table with the target database that is changed to of complete copy source database, this method
Duplication, data syn-chronization amount is big, and higher to network bandwidth requirement, efficiency is low.
(2) data increment is reported
When data syn-chronization is needed, the data duplication of change is reported to step by step by higher level by special messaging application
Unit, from the point of view of the data volume of transmission, only uploads the data of change, is that a kind of increment is reported, and synchronous amount is little.But due to industry
The seriality that business is processed, process needs continuous synchronization, therefore this synchronous method cannot ensure the integrity of data.
Either all reported or data increment report method using data source, for being classified deployment system, now used
Mode be synchronous step by step, system is at different levels to need data receiver to transmit again after processing, the system higher for requirement of real-time
For, the mode transmission speed that reports step by step is slow, the cycle is long.Complicate in the face of a large amount of incremental datas and business operation, existing industry
There is obvious weak point in the data syn-chronization pattern for using.In sum, urgent need will a kind of transmission be more efficient, guarantee transmission number
According to more total solution.
The content of the invention
In view of this, the invention provides a kind of method of data synchronization based on business operation daily record, it can be ensured that business
The atomicity of data, the integrity of data, while business datum adopts direct distribution mechanisms, rather than stepwise disposal to report step by step,
Improve data syn-chronization speed.
The method of data synchronization based on business operation daily record of the present invention, comprises the steps：
Step one, the business operation of operation system minute book operation system while business operation is carried out to business object
Daily record, generates business operation journal file；
Step 2, operation system is transmitted the business operation journal file of generation according to deployment unit hierarchical relationship,
Using direct distribution mechanisms, business operation journal file is directly distributed and is transferred to the purpose business system related to the business operation
System；
Step 3, purpose operation system is received after business operation journal file, extracts and analyze business operation journal file
Described in business operation, according to business operation analysis result change purpose operation system data, realize data syn-chronization.
Further, if failure of data synchronization, data rewind, the i.e. data of all purposes operation system are carried out first
Before being restored to this operation system data change, step 2 and step 3 are then re-executed, carry out data syn-chronization.
Beneficial effect：
(1) the inventive method has ensured to greatest extent business number using business operation daily record as the foundation of data syn-chronization
According to atomicity, continuity, the concordance of data, integrity, it is to avoid using data source all report with data source increment
The obvious drawback of report mode.
(2) the inventive method is during data transfer distribution, has abandoned classification deployment system and must report step by step and issues
Mode, using directly distribute mechanism, without the need for carrying out document analysis and data manipulation step by step, improve data synchronization efficiency.
(3) the inventive method, according to business operation log rollback, ensures data factor data synchronization in failure of data synchronization
The loss of data mistake for unsuccessfully causing, improves the accuracy of data syn-chronization.
Description of the drawings
Fig. 1 is the inventive method flow chart.
Fig. 2 is the contrast of data transfer flow process.
Specific embodiment
Develop simultaneously below in conjunction with the accompanying drawings embodiment, describes the present invention.
The invention provides a kind of method of data synchronization based on business operation daily record, first recording business operation logs,
Business operation journal file is generated, then by business operation journal file synchronized upload to each purpose operation system, each purpose industry
Business operation string is analyzed and extracted to business system according to the business operation journal file for receiving, according to extraction business operation string weight
The data of handwritten copy operation system.
Flow chart of the present invention is as shown in figure 1, specifically include following steps：
Step one, recording business operation logs
Each operation system obtains data syn-chronization mapping message in specified listening port, confirms lock in time and uploads path.
When an operation system carries out business operation, the operation system carries out business operation according to service data manipulating instruction, safeguards
This DBMS, at the same record the system business operation daily record (change of business operation log recording to business object is operated,
For tracking the change procedure of traceability system business datum), preservation daily record is specified format file, generates business operation daily record text
Part.
Step 2, data transfer
As shown in Fig. 2 when lock in time reaches, by vertical application, according to systems communicate strategy by business operation
Journal file is transmitted according to deployment unit hierarchical relationship, and system, can be direct by journal file using the mechanism directly distributed
Distribution is transferred to related purpose operation system, without the need for carrying out document analysis and data manipulation step by step, improves efficiency of transmission, purpose industry
Business system at the appointed time reads synchronous documents.
Step 3, extraction data, it is at different levels according to business operation logredo data
One business operation daily record includes one group of data change vector, is a sequence of operation of data mode, record
Business operation string, root are analyzed and extracted to atomic operation to data base, purpose operation system according to the journal file for receiving
Purpose end data is rewritten according to analysis result.
Step 4, synchronization failure, data rewind
Once judging business operation journal file synchronization failure, rewrite data and suspend, system prompt synchronization failure, because of business
Operation may relate to multiple database tables, be to ensure that data are completely accurate, and system carries out data rewind, recovers to the last time just
True state, restoring data.When another subsynchronous, according to business operation logredo purpose end data, it is to avoid for fraction number
Recover whole data base according to synchronization failure.
In sum, presently preferred embodiments of the present invention is these are only, is not intended to limit protection scope of the present invention.
All any modification, equivalent substitution and improvements within the spirit and principles in the present invention, made etc., should be included in the present invention's
Within protection domain.
Claims (2)
1. a kind of method of data synchronization based on business operation daily record, it is characterised in that comprise the steps：
Step one, the business operation day of operation system minute book operation system while business operation is carried out to business object
Will, generates business operation journal file；
Step 2, operation system is transmitted the business operation journal file of generation according to deployment unit hierarchical relationship, adopts
Direct distribution mechanisms, business operation journal file is directly distributed and is transferred to the purpose operation system related to the business operation；
Step 3, purpose operation system is received after business operation journal file, is extracted and is analyzed in business operation journal file and remembers
The business operation of load, according to business operation analysis result purpose operation system data are changed, and realize data syn-chronization.
2. the method for data synchronization of business operation daily record is based on as claimed in claim 1, it is characterised in that if data syn-chronization
Failure, then carry out first data rewind, the i.e. data convert of all purposes operation system to this operation system data change,
Then step 2 and step 3 are re-executed, data syn-chronization is carried out.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201611184089.7A CN106682141B (en) | 2016-12-20 | 2016-12-20 | Data synchronization method based on service operation log |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201611184089.7A CN106682141B (en) | 2016-12-20 | 2016-12-20 | Data synchronization method based on service operation log |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN106682141A true CN106682141A (en) | 2017-05-17 |
| CN106682141B CN106682141B (en) | 2022-02-11 |
Family
ID=58870091
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201611184089.7A Active CN106682141B (en) | 2016-12-20 | 2016-12-20 | Data synchronization method based on service operation log |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106682141B (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108681558A (en) * | 2018-04-10 | 2018-10-19 | 阿里巴巴集团控股有限公司 | A kind of data rewind method, apparatus and terminal |
| CN108984337A (en) * | 2018-05-29 | 2018-12-11 | 杭州网易再顾科技有限公司 | A kind of restorative procedure of data synchronous abnormality, prosthetic device, medium and calculate equipment |
| CN114510495A (en) * | 2022-04-21 | 2022-05-17 | 北京安华金和科技有限公司 | Database service data consistency processing method and system |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101145945A (en) * | 2007-08-29 | 2008-03-19 | 中兴通讯股份有限公司 | A message bus system and control method in telecommunication network management |
| KR20130043823A (en) * | 2011-10-21 | 2013-05-02 | 엔에이치엔비즈니스플랫폼 주식회사 | Distributed storage system for maintaining data consistency based on log, and method for the same |
| CN103812703A (en) * | 2014-02-20 | 2014-05-21 | 浪潮集团有限公司 | Asynchronous log saving method based on cloud computing |
| CN103812929A (en) * | 2014-01-11 | 2014-05-21 | 浪潮电子信息产业股份有限公司 | Active-active method for cloud data center management platforms |
| US20150127901A1 (en) * | 2010-12-13 | 2015-05-07 | International Business Machines Corporation | Saving log data using a disk system as primary cache and a tape library as secondary cache |
| CN105183860A (en) * | 2015-09-10 | 2015-12-23 | 北京京东尚科信息技术有限公司 | Data synchronization method and system |
| CN105490868A (en) * | 2015-11-17 | 2016-04-13 | 世纪龙信息网络有限责任公司 | Bidirectional synchronous monitoring method and system for remote computer room data |
-
2016
- 2016-12-20 CN CN201611184089.7A patent/CN106682141B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101145945A (en) * | 2007-08-29 | 2008-03-19 | 中兴通讯股份有限公司 | A message bus system and control method in telecommunication network management |
| US20150127901A1 (en) * | 2010-12-13 | 2015-05-07 | International Business Machines Corporation | Saving log data using a disk system as primary cache and a tape library as secondary cache |
| KR20130043823A (en) * | 2011-10-21 | 2013-05-02 | 엔에이치엔비즈니스플랫폼 주식회사 | Distributed storage system for maintaining data consistency based on log, and method for the same |
| CN103812929A (en) * | 2014-01-11 | 2014-05-21 | 浪潮电子信息产业股份有限公司 | Active-active method for cloud data center management platforms |
| CN103812703A (en) * | 2014-02-20 | 2014-05-21 | 浪潮集团有限公司 | Asynchronous log saving method based on cloud computing |
| CN105183860A (en) * | 2015-09-10 | 2015-12-23 | 北京京东尚科信息技术有限公司 | Data synchronization method and system |
| CN105490868A (en) * | 2015-11-17 | 2016-04-13 | 世纪龙信息网络有限责任公司 | Bidirectional synchronous monitoring method and system for remote computer room data |
Non-Patent Citations (2)
| Title |
|---|
| ZHANG,JING: "A Data Synchronization Method Oriented to Custom Hierarchical Multi-node System", 《2015 IEEE INTERNATIONAL CONFERENCE ON COMPUTATIONAL INTELLIGENCE AND COMMUNICATION TECHNOLOGY CICT 2015》 * |
| 戴鹏飞: "山东省安全监督基础信息资源管理平台的设计与实现", 《中国优秀硕士学位论文全文数据库(电子期刊)》 * |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108681558A (en) * | 2018-04-10 | 2018-10-19 | 阿里巴巴集团控股有限公司 | A kind of data rewind method, apparatus and terminal |
| CN108681558B (en) * | 2018-04-10 | 2021-08-06 | 创新先进技术有限公司 | Data rollback method, device and terminal |
| CN108984337A (en) * | 2018-05-29 | 2018-12-11 | 杭州网易再顾科技有限公司 | A kind of restorative procedure of data synchronous abnormality, prosthetic device, medium and calculate equipment |
| CN108984337B (en) * | 2018-05-29 | 2021-04-16 | 杭州网易再顾科技有限公司 | Data synchronization abnormity repairing method, device, medium and computing equipment |
| CN114510495A (en) * | 2022-04-21 | 2022-05-17 | 北京安华金和科技有限公司 | Database service data consistency processing method and system |
| CN114510495B (en) * | 2022-04-21 | 2022-07-08 | 北京安华金和科技有限公司 | Database service data consistency processing method and system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN106682141B (en) | 2022-02-11 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10795911B2 (en) | Apparatus and method for replicating changed-data in source database management system to target database management system in real time | |
| CN110347746B (en) | Heterogeneous database synchronous data consistency checking method and device | |
| CN102317939B (en) | Asynchronous distributed garbage collection for replicated storage clusters | |
| CN107209704A (en) | Detect the write-in lost | |
| CN102968486B (en) | A kind of highly reliable file synchronisation method based on change journal | |
| EP2541413A1 (en) | Systems and Methods for Data Integrity Checking | |
| US20100257138A1 (en) | Data change ordering in multi-log based replication | |
| US20120136827A1 (en) | Periodic data replication | |
| CN101187888A (en) | Method for coping database data in heterogeneous environment | |
| CN104346454A (en) | Data consistency verification method based on Oracle database | |
| CN108810112B (en) | Node synchronization method and device for market supervision block chain system | |
| CN106599104A (en) | Mass data association method based on redis cluster | |
| TWI256556B (en) | Distributed concurrent version management system and method | |
| TW201140350A (en) | High throughput, reliable replication of transformed data in information systems | |
| CN103838847A (en) | Data organization method oriented to sea-cloud collaboration network computing network | |
| CN103795754A (en) | Method and system for data synchronization among multiple systems | |
| CN104252502A (en) | Method and device for carrying out data migration on database management platform | |
| CN106682141A (en) | Data synchronization method based on business operation logs | |
| CN109840166A (en) | Across the cluster object storage async backup methods, devices and systems of one kind | |
| CN108228726B (en) | Incremental transaction content acquisition method and storage medium for distribution network red and black images | |
| CN107423452A (en) | A kind of power network heterogeneous database synchronously replicates moving method | |
| CN113051347B (en) | Method, system, equipment and storage medium for synchronizing data between heterogeneous databases | |
| CN104616134A (en) | Statement analysis report pushing method and device | |
| CN110222121A (en) | A kind of SQL Server database increment synchronization realization method and system based on CDC mode | |
| CN113177088B (en) | Multi-scale simulation big data management system for material irradiation damage |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |