CN104486387A - Data synchronization processing method and system - Google Patents
Data synchronization processing method and system Download PDFInfo
- Publication number
- CN104486387A CN104486387A CN201410723701.8A CN201410723701A CN104486387A CN 104486387 A CN104486387 A CN 104486387A CN 201410723701 A CN201410723701 A CN 201410723701A CN 104486387 A CN104486387 A CN 104486387A
- Authority
- CN
- China
- Prior art keywords
- data
- node
- information
- bitmap
- rbtree
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a data synchronization processing method and a data synchronization processing system. The method comprises the following steps of when a data synchronization event is triggered to occur, acquiring the sum of setting numbers in a bitmap, and if the sum is not zero, traversing nodes, set as 1, in the bitmap; inserting the information of the nodes into leaf nodes of a red black tree, and performing synchronous data transmission, wherein the red black tree refers to a self-balancing binary search tree. According to the method and the system, data can be quickly and accurately synchronized; meanwhile, the problem of data inconsistency caused by writing conflicts during synchronization is avoided; moreover, an implementation mode of the red black tree has high insertion-deletion efficiency and has high worst-case time complexity.
Description
Technical field
The invention belongs to data syn-chronization field, particularly relate to a kind of data synchronizing processing method and system.
Background technology
Disaster tolerance system refers in strange land relatively far apart, set up two covers or the identical IT system of many cover functions, state of health monitoring and function switching can be carried out mutually, when place's system quits work because of unexpected (as fire, earthquake etc.), whole application system can be switched to another place, makes this systemic-function can continue normal work; Disaster tolerance technology is a part of the High availability of system, and disaster tolerance system emphasizes that process external environment is on the impact of system, the particularly catastrophic event impact on whole IT node, provides node level other System recover function more.
Remote copy is the important component part of data protection, data protection function when providing home site to lose efficacy; Remote copy provides a continuous print, nondestructive, independently solution is to tackle disaster recovery for main frame, and data backup or remote data move, user is met to cost, the different demands of performance and availability of data by disposing synchronous or asynchronous replication flexibly.
The remote copy function utilizing storage system powerful, the storage pool between different website or whole system can be allowed to carry out data copy, when local system fault or accident occur and cause source volume data to be damaged, user can twist in Resume service in a few minutes by means of the remote copy based on disk, if user wants storage application to switch back during local fault recovery, reverse sync can be realized fast between two volumes.
When main frame and standby host break down, or when network goes wrong, active and standby two end datas can be caused inconsistent, when problem is repaired, need to carry out data resynchronization, to reach the object of two ends data consistent, therefore how providing a kind of scheme improving data resynchronization accuracy and efficiency, is the problem needing solution badly.
Summary of the invention
The invention provides a kind of data synchronizing processing method and system, to solve the problem.
The invention provides a kind of data synchronizing processing method, comprise the following steps:
When trigger data synchronous event occurs, obtain the mid-figure place sum of bitmap, if not zero, then travel through in described bitmap the node being set to 1;
The information of described node is inserted the leaf node of RBTree, carry out synchronous data transmission; Wherein, described RBTree refers to self-balancing binary search tree.
The present invention also provides a kind of data synchronization processing system, comprises and triggers spider module, synchronization module; Wherein, described triggering spider module is connected with described synchronization module;
Described triggering spider module, when occurring for trigger data synchronous event, obtains bitmap mid-figure place sum, if not zero, then travel through in described bitmap be set to 1 node and the information of described node is sent to described synchronization module;
Described synchronization module, for the information of described node being inserted the leaf node of RBTree, carries out synchronous data transmission; Wherein, described RBTree refers to self-balancing binary search tree.
Compared to prior art, a kind of data synchronizing processing method and system is the invention provides according to provided by the invention, achieve the synchronous fast and accurately of data, the problem that the data caused by write conflict when simultaneously avoiding synchronous are inconsistent, and the implementation of RBTree, there is very high insertion deletion efficiency, and have the worst good time complexity.
Accompanying drawing explanation
Accompanying drawing described herein is used to provide a further understanding of the present invention, and form a application's part, schematic description and description of the present invention, for explaining the present invention, does not form inappropriate limitation of the present invention.In the accompanying drawings:
Figure 1 shows that the data synchronizing processing method flow chart of the embodiment of the present invention 1;
Figure 2 shows that the mode figure that the data bitmap information of the embodiment of the present invention 2 stores in metadata management central linear;
Figure 3 shows that the bitmap tree mode of the embodiment of the present invention 3 is stored to the schematic diagram at metadata management center;
Figure 4 shows that the data synchronization processing system construction drawing of the embodiment of the present invention 4.
Embodiment
Hereinafter also describe the present invention in detail with reference to accompanying drawing in conjunction with the embodiments.It should be noted that, when not conflicting, the embodiment in the application and the feature in embodiment can combine mutually.
Figure 1 shows that the data synchronizing processing method flow chart of the embodiment of the present invention 1, comprise the following steps:
Step 101: when trigger data synchronous event occurs, obtains the mid-figure place sum of bitmap, if not zero, then travels through in described bitmap the node being set to 1;
Trigger data synchronous event refers to link failure recovery.
A node in bitmap and bi t position, mark the data block of one piece of default size (such as: 4KB).
If described bit position " puts 1 ", then the data block of this bit bit representation was modified after link exception between local terminal and far-end; If this bi t position " sets to 0 ", then represent that local terminal is consistent with remote data.
Data bitmap information, in units of page, is stored to internal memory.Such as: page size is 4KB space, a page can store 2^15 bit, can describe 128M data.
During link failure, by the data bitmap information in internal memory in units of page according to the order of sequence linear memory to metadata management center;
During link-recovery, from described metadata management, in units of page, read institute's bit-map data information in the heart to page.
During link failure, the data bitmap information in internal memory is stored to metadata management center in the mode that bitmap is set;
During link-recovery, read institute's bit-map data information in described metadata management in the heart to page.
Described bitmap tree comprises host node, one or more first nodes, one or more two-level node; Described host node is connected with one or more first nodes respectively; Described first nodes is connected with corresponding two-level node respectively.
Host node stores the corresponding relation of first nodes and its metadata logical block, the corresponding relation of first nodes store secondary node and its metadata logical block, and two-level node stores institute's bit-map data information.
Data bitmap information has two kinds of modes to store at metadata management center:
One is linear memory, and all bitmap pages are stored to metadata management center in order, simple to operate, takies metadata space few, but poor expandability.When data volume increases, data bitmap information is in requisition for expansion, when there being multiple data bitmap information Coutinuous store, the metadata management center of linear memory cannot dilatation, can only again apply for metadata space, and the position of data bitmap information copy to new application, so often cause metadata space not to be utilized effectively.The mode that data bitmap information stores in metadata management central linear as shown in Figure 2, after data bitmap information is organized in units of page, is stored to metadata management center according to the order of sequence.
The second is that bitmap tree mode is stored to metadata management center, as shown in Figure 3, comprises host node, first nodes, two-level node; Wherein, host node is connected with one or more first nodes respectively, and first nodes is connected with corresponding two-level node.
The method obtaining the data bitmap information of first data block is as follows:
Step 1: host node obtains the information of first nodes 1 from its metadata logical block 5, wherein, host node stores the corresponding relation of first nodes and its metadata logical block;
Step 2: first nodes 1 obtains the information of two-level node 1 from its metadata logical block 7, wherein, the corresponding relation of first nodes store secondary node and its metadata logical block;
Step 3: two-level node 1 searches metadata logical block 9 corresponding to first data block, then reads metadata logical block 9, obtains data bitmap information.
Need additional metadata space to carry out memory node information with the mode storage bitmap data message of bitmap tree, advantage is convenient expansion, only needs when expanded distribute new metadata logical block and revise nodal information; Needing when creating remote copy to check that whether bitmap metadata space is sufficient, if metadata insufficient space, creating remote copy failure, and returning the prompting of metadata insufficient space.
Step 102: the leaf node information of described node being inserted RBTree, carries out synchronous data transmission; Wherein, described RBTree refers to self-balancing binary search tree.
Before the information of described node being inserted the leaf node of RBTree, also comprising: by comparing the nodal information in current operating information and RBTree, confirming there is not write conflict operation.
If there is write conflict operation, then skip described node.
If synchronously success, then delete RBTree interior joint information, bit position corresponding in bitmap " is set to 0 "; If not synchronous success, then delete RBTree interior joint information, skip described node.
RBTree is adopted to preserve the disk block information processed, RBTree is a kind of self-balancing binary search tree, and it is complicated, but its operation has good worst case running time, and be efficient in practice, and space complexity is less than hash table; Although Adelson-Velskii-Landis tree and RBTree are very similar, the deletion of Adelson-Velskii-Landis tree and update need multiple rotary to operate and constantly recall to root node, so when a large amount of deletion and update, the operating efficiency of Adelson-Velskii-Landis tree is not high; RBTree is the incomplete equilibrium binary tree that a kind of search efficiency is only second to Adelson-Velskii-Landis tree, it can carry out inserting with the time complexity of O (logn), deletion action, and its insertion and delete and need at most to rotate the balance that can keep setting twice or three times.Although the algorithm complex of the two is identical, in the worst cases, RBTree provides a kind of deletion real-time soon and inserts the algorithm of a node, and the efficient operation of obvious RBTree is more suitable for.
The leaf node of RBTree preserves original position and the size of current disk write operation, when disk write operation or simultaneous operation enters in remote copy flow process again time, information is inserted RBTree node; When operation returns, node is deleted from RBTree.Detect whether to have write conflict operation and current operating information and the node in RBTree are carried out inquiry and compare, see whether there is conflict.
Figure 4 shows that the data synchronization processing system construction drawing of the embodiment of the present invention 4, comprise and trigger spider module, synchronization module; Wherein, described triggering spider module is connected with described synchronization module;
Described triggering spider module, when occurring for trigger data synchronous event, obtains bitmap mid-figure place sum, if not zero, then travel through in described bitmap be set to 1 node and the information of described node is sent to described synchronization module;
Described synchronization module, for the information of described node being inserted the leaf node of RBTree, carries out synchronous data transmission; Wherein, described RBTree refers to self-balancing binary search tree.
Compared to prior art, a kind of data synchronizing processing method and system is the invention provides according to provided by the invention, achieve the synchronous fast and accurately of data, the problem that the data caused by write conflict when simultaneously avoiding synchronous are inconsistent, and the implementation of RBTree, there is very high insertion deletion efficiency, and have the worst good time complexity.
The foregoing is only the preferred embodiments of the present invention, be not limited to the present invention, for a person skilled in the art, the present invention can have various modifications and variations.Within the spirit and principles in the present invention all, any amendment done, equivalent replacement, improvement etc., all should be included within protection scope of the present invention.
Claims (10)
1. a data synchronizing processing method, is characterized in that, comprises the following steps:
When trigger data synchronous event occurs, obtain the mid-figure place sum of bitmap, if not zero, then travel through in described bitmap the node being set to 1;
The information of described node is inserted the leaf node of RBTree, carry out synchronous data transmission; Wherein, described RBTree refers to self-balancing binary search tree.
2. method according to claim 1, is characterized in that, trigger data synchronous event refers to link failure recovery.
3. method according to claim 1, it is characterized in that, a node in bitmap and bit position, the data block of mark one piece of default size, if described bit position " puts 1 ", then the data block of this bit bit representation was modified after link exception between local terminal and far-end; If described bit position " sets to 0 ", then represent that local terminal is consistent with remote data.
4. method according to claim 1, is characterized in that, data bitmap information, in units of page, is stored to internal memory.
5. method according to claim 4, is characterized in that, during link failure, by the data bitmap information in internal memory in units of page according to the order of sequence linear memory to metadata management center; During link-recovery, from described metadata management, in units of page, read institute's bit-map data information in the heart to page.
6. method according to claim 4, is characterized in that, during link failure, the data bitmap information in internal memory is stored to metadata management center in the mode that bitmap is set; During link-recovery, read institute's bit-map data information in described metadata management in the heart to page.
7. method according to claim 6, is characterized in that, described bitmap tree comprises host node, one or more first nodes, one or more two-level node; Described host node is connected with one or more first nodes respectively; Described first nodes is connected with corresponding two-level node respectively.
8. method according to claim 7, is characterized in that, host node stores the corresponding relation of first nodes and its metadata logical block, the corresponding relation of first nodes store secondary node and its metadata logical block, and two-level node stores institute's bit-map data information.
9. method according to claim 1, is characterized in that, before the information of described node being inserted the leaf node of RBTree, also comprises: by comparing the nodal information in current operating information and RBTree, confirms there is not write conflict operation.
10. a data synchronization processing system, is characterized in that, comprises and triggers spider module, synchronization module; Wherein, described triggering spider module is connected with described synchronization module;
Described triggering spider module, when occurring for trigger data synchronous event, obtains bitmap mid-figure place sum, if not zero, then travel through in described bitmap be set to 1 node and the information of described node is sent to described synchronization module;
Described synchronization module, for the information of described node being inserted the leaf node of RBTree, carries out synchronous data transmission; Wherein, described RBTree refers to self-balancing binary search tree.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410723701.8A CN104486387B (en) | 2014-12-02 | 2014-12-02 | A kind of data synchronizing processing method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410723701.8A CN104486387B (en) | 2014-12-02 | 2014-12-02 | A kind of data synchronizing processing method and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN104486387A true CN104486387A (en) | 2015-04-01 |
| CN104486387B CN104486387B (en) | 2018-03-27 |
Family
ID=52760891
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201410723701.8A Active CN104486387B (en) | 2014-12-02 | 2014-12-02 | A kind of data synchronizing processing method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN104486387B (en) |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105183898A (en) * | 2015-09-29 | 2015-12-23 | 北京奇艺世纪科技有限公司 | Data sync method and data sync system |
| CN105843750A (en) * | 2016-03-22 | 2016-08-10 | 深圳森格瑞通信有限公司 | Linear data structure query efficiency improving method |
| CN106371764A (en) * | 2016-08-23 | 2017-02-01 | 浪潮(北京)电子信息产业有限公司 | Virtual block device-based data processing method and apparatus |
| CN115242812A (en) * | 2022-07-25 | 2022-10-25 | 济南浪潮数据技术有限公司 | Node data synchronization method and device and computer readable storage medium |
| CN115801736A (en) * | 2023-01-09 | 2023-03-14 | 北京天际友盟信息技术有限公司 | IP address library construction loading method and device and IP address query method and device |
| US11960483B1 (en) | 2021-09-16 | 2024-04-16 | Wells Fargo Bank, N.A. | Constant time data structure for single and distributed networks |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102713826A (en) * | 2009-12-22 | 2012-10-03 | 苹果公司 | Methods and apparatuses to allocate file storage via tree representations of a bitmap |
| CN102761615A (en) * | 2012-06-29 | 2012-10-31 | 浪潮(北京)电子信息产业有限公司 | Method and device for realizing data synchronism of long-distance duplication system |
| CN104036141A (en) * | 2014-06-16 | 2014-09-10 | 上海大学 | Open computing language (OpenCL)-based red-black tree acceleration algorithm |
-
2014
- 2014-12-02 CN CN201410723701.8A patent/CN104486387B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102713826A (en) * | 2009-12-22 | 2012-10-03 | 苹果公司 | Methods and apparatuses to allocate file storage via tree representations of a bitmap |
| CN102761615A (en) * | 2012-06-29 | 2012-10-31 | 浪潮(北京)电子信息产业有限公司 | Method and device for realizing data synchronism of long-distance duplication system |
| CN104036141A (en) * | 2014-06-16 | 2014-09-10 | 上海大学 | Open computing language (OpenCL)-based red-black tree acceleration algorithm |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105183898A (en) * | 2015-09-29 | 2015-12-23 | 北京奇艺世纪科技有限公司 | Data sync method and data sync system |
| CN105843750A (en) * | 2016-03-22 | 2016-08-10 | 深圳森格瑞通信有限公司 | Linear data structure query efficiency improving method |
| CN106371764A (en) * | 2016-08-23 | 2017-02-01 | 浪潮(北京)电子信息产业有限公司 | Virtual block device-based data processing method and apparatus |
| US11960483B1 (en) | 2021-09-16 | 2024-04-16 | Wells Fargo Bank, N.A. | Constant time data structure for single and distributed networks |
| CN115242812A (en) * | 2022-07-25 | 2022-10-25 | 济南浪潮数据技术有限公司 | Node data synchronization method and device and computer readable storage medium |
| CN115801736A (en) * | 2023-01-09 | 2023-03-14 | 北京天际友盟信息技术有限公司 | IP address library construction loading method and device and IP address query method and device |
| CN115801736B (en) * | 2023-01-09 | 2023-04-18 | 北京天际友盟信息技术有限公司 | IP address library construction loading method and device and IP address query method and device |
Also Published As
| Publication number | Publication date |
|---|---|
| CN104486387B (en) | 2018-03-27 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US12117911B2 (en) | Remote data replication method and system | |
| CN104486387A (en) | Data synchronization processing method and system | |
| US9009402B2 (en) | Content addressable storage in legacy systems | |
| CN104077380B (en) | A kind of data de-duplication method, apparatus and system | |
| CN101539873B (en) | Data recovery method, data node and distributed file system | |
| US6948089B2 (en) | Apparatus and method for multiple generation remote backup and fast restore | |
| US8281093B1 (en) | Systems and methods for creating consolidated backups of snapshot hierarchies | |
| CN105376277B (en) | A kind of method of data synchronization and device | |
| US9483367B1 (en) | Data recovery in distributed storage environments | |
| CN111078667B (en) | Data migration method and related device | |
| CN104484480A (en) | Deduplication-based remote replication method and device | |
| CN106919465B (en) | Method and apparatus for multiple data protection in a storage system | |
| WO2013091212A1 (en) | Partition management method, device and system in distributed storage system | |
| CN104461773A (en) | Backup deduplication method of virtual machine | |
| CN103905220A (en) | Data synchronization processing method and system | |
| CN110990399B (en) | Reconstruction index method and device | |
| US9513996B2 (en) | Information processing apparatus, computer-readable recording medium having stored program for controlling information processing apparatus, and method for controlling information processing apparatus | |
| CN104516795A (en) | Data access method and system | |
| CN103500129A (en) | Back-up object sending and back-up method, production end, backup-for-disaster-recovery end and system | |
| CN104484402B (en) | A kind of method and device of deleting duplicated data | |
| CN108829813A (en) | A kind of File Snapshot method and system based on distributed memory system | |
| CN105511808B (en) | Data operation method, system and related device | |
| WO2017054643A1 (en) | Data recovery method and file server | |
| CN105939203B (en) | List item synchronous method and device | |
| CN103530205A (en) | Method and device for processing fault duplicate in multiple duplicates |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |