CN104486387B - A kind of data synchronizing processing method and system - Google Patents
A kind of data synchronizing processing method and system Download PDFInfo
- Publication number
- CN104486387B CN104486387B CN201410723701.8A CN201410723701A CN104486387B CN 104486387 B CN104486387 B CN 104486387B CN 201410723701 A CN201410723701 A CN 201410723701A CN 104486387 B CN104486387 B CN 104486387B
- Authority
- CN
- China
- Prior art keywords
- node
- data
- bitmap
- information
- rbtree
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 11
- 230000001360 synchronised effect Effects 0.000 claims abstract description 14
- 238000000034 method Methods 0.000 claims abstract description 11
- 238000012546 transfer Methods 0.000 claims abstract description 5
- 241000239290 Araneae Species 0.000 claims description 9
- 238000011084 recovery Methods 0.000 claims description 8
- 238000012217 deletion Methods 0.000 claims description 6
- 230000037430 deletion Effects 0.000 claims description 6
- 238000012545 processing Methods 0.000 claims description 4
- 230000006870 function Effects 0.000 description 5
- 238000003780 insertion Methods 0.000 description 4
- 230000037431 insertion Effects 0.000 description 4
- 238000005516 engineering process Methods 0.000 description 3
- 230000008859 change Effects 0.000 description 2
- 238000010276 construction Methods 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 241001269238 Data Species 0.000 description 1
- 230000009471 action Effects 0.000 description 1
- 230000008901 benefit Effects 0.000 description 1
- 230000007812 deficiency Effects 0.000 description 1
- 230000036541 health Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 230000008569 process Effects 0.000 description 1
- 230000010076 replication Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Abstract
The present invention provides a kind of data synchronizing processing method and system, the above method and comprised the following steps：Trigger data synchronous event occur when, obtain bitmap in set number sum, if not zero, then travel through be set in the bitmap 1 node；The information of the node is inserted to the leaf node of RBTree, synchronizes data transfer；Wherein, the RBTree refers to self-balancing binary search tree, the present invention realizes the fast and accurately synchronous of data, the problem of data are inconsistent as caused by write conflict when avoiding synchronization simultaneously, and the implementation of RBTree, there is very high insert and delete efficiency, and have the worst good time complexity.
Description
Technical field
The invention belongs to data syn-chronization field, more particularly to a kind of data synchronizing processing method and system.
Background technology
Disaster tolerance system refers in strange land relatively far apart, establishes two sets or more set function identical IT systems, from each other
State of health monitoring and function switch can be carried out, at one system because unexpected (such as fire, earthquake) be stopped when, entirely
Application system can be switched to another place so that the systemic-function can continue normal work；Disaster tolerance technology is that the height of system can
With a part of property technology, disaster tolerance system is more emphasized to handle influence of the external environment to system, particularly calamity
Influence of the event to whole IT nodes, there is provided the other system of node level recovers function.
Remote copy is the important component of data protection, there is provided data protection function during main website point failure；Remotely
Replicate provide one it is continuous, nondestructive, the solution of main frame independence is to tackle disaster recovery, data backup or remote
Range data migrates, and is either synchronously or asynchronously replicated by flexibly disposing to meet user to cost, performance and availability of data
Different demands.
Using the powerful remote copy function of storage system, the storage pool or whole system that can allow between different websites are carried out
Data copy, when causing source volume data to be damaged when local system failure or accident generation, user can be by means of based on magnetic
The remote replication volume of disk recovers service within a few minutes, if user wants storage application to switch back during local fault recovery,
Quickly reverse sync can be realized between two volumes.
When main frame and standby host break down, or network goes wrong, active and standby two end data can be caused inconsistent, when asking
, it is necessary to carry out data resynchronization when topic is repaired, to reach the consistent purpose of two end datas, therefore a kind of raising data how are provided
The problem of scheme of resynchronisation accuracy and efficiency is urgent need to resolve.
The content of the invention
The present invention provides a kind of data synchronizing processing method and system, to solve the above problems.
The present invention provides a kind of data synchronizing processing method, comprises the following steps：
When trigger data synchronous event occurs, set number sum in bitmap is obtained, if not zero, then travel through in the bitmap
It is set to 1 node；
The information of the node is inserted to the leaf node of RBTree, synchronizes data transfer；Wherein, the RBTree
Refer to self-balancing binary search tree.
The present invention also provides a kind of data synchronization processing system, including triggering spider module, synchronization module；Wherein, it is described
Triggering spider module is connected with the synchronization module；
The triggering spider module, when occurring for trigger data synchronous event, set number sum in bitmap is obtained, if not
Be zero, then travel through be set in the bitmap 1 node and the information of the node is sent to the synchronization module；
The synchronization module, for the information of the node to be inserted to the leaf node of RBTree, synchronize data biography
It is defeated；Wherein, the RBTree refers to self-balancing binary search tree.
Compared to prior art, a kind of data synchronizing processing method and system are provided according to the present invention provided by the invention,
The problem of data are inconsistent as caused by write conflict when realizing the fast and accurately synchronous of data, while avoiding synchronization, and
And the implementation of RBTree, there is very high insert and delete efficiency, and have the worst good time complexity.
Brief description of the drawings
Accompanying drawing described herein is used for providing a further understanding of the present invention, forms the part of the application, this hair
Bright schematic description and description is used to explain the present invention, does not form inappropriate limitation of the present invention.In the accompanying drawings：
Fig. 1 show the data synchronizing processing method flow chart of the embodiment of the present invention 1；
Fig. 2 show the mode figure that the bitmap data information of the embodiment of the present invention 2 stores in metadata management central linear；
The bitmap tree mode that Fig. 3 show the embodiment of the present invention 3 is stored to the schematic diagram at metadata management center；
Fig. 4 show the data synchronization processing system construction drawing of the embodiment of the present invention 4.
Embodiment
Describe the present invention in detail below with reference to accompanying drawing and in conjunction with the embodiments.It should be noted that do not conflicting
In the case of, the feature in embodiment and embodiment in the application can be mutually combined.
Fig. 1 show the data synchronizing processing method flow chart of the embodiment of the present invention 1, comprises the following steps：
Step 101：When trigger data synchronous event occurs, set number sum in bitmap is obtained, if not zero, then travel through institute
1 node is set in rheme figure；
Trigger data synchronous event refers to link failure recovery.
A node in bitmap is bi t positions, the default size of one piece of mark (such as：Data block 4KB).
If the bit positions " put 1 ", the data block that the bit positions represent is repaiied between local terminal and distal end after link exception
Correct one's mistakes；If bi t positions " setting to 0 ", then it represents that local terminal is consistent with remote data.
Bitmap data information is stored to internal memory in units of page.Such as：Page size is 4KB spaces, and a page can store
2^15 bit, 128M data can be described.
During link failure, by the bitmap data information in internal memory in units of page sequentially linear memory into metadata management
The heart；
During link-recovery, institute's bit-map data information is read to internal memory in units of page from the metadata management center
Page.
During link failure, the bitmap data information in internal memory is stored in a manner of bitmap tree to metadata management center；
Institute's bit-map data information is read during link-recovery, in the metadata management center to page.
The bitmap tree includes host node, one or more first nodes, one or more two-level nodes；The host node
It is connected respectively with one or more first nodes；The first nodes are connected with corresponding two-level node respectively.
Host node stores the corresponding relation of first nodes and its metadata logical block, and first nodes store two-level node and its
The corresponding relation of metadata logical block, two-level node storage institute bit-map data information.
Bitmap data information has two ways storage at metadata management center：
One kind is linear memory, and all bitmap pages store to metadata management center, simple to operate, occupancy member in order
Data space is few, but poor expandability.When data volume increases, bitmap data information is mutually in requisition for expansion, when there is multiple bitmaps
During data message Coutinuous store, the metadata management center of linear memory can not dilatation, metadata space can only be applied for again, and
Bitmap data information copy to the position newly applied, so metadata space is often caused to be utilized effectively.Position
The mode that diagram data information stores in metadata management central linear is as shown in Fig. 2 bitmap data information is carried out in units of page
After tissue, sequentially store to metadata management center.
Second be bitmap tree mode store to metadata management center, as shown in figure 3, including host node, first nodes,
Two-level node；Wherein, host node is connected with one or more first nodes respectively, first nodes and corresponding two-level node phase
Even.
The method for obtaining the bitmap data information of first data block is as follows：
Step 1：Host node obtains the information of first nodes 1 from its metadata logical block 5, wherein, host node storage one
The corresponding relation of level node and its metadata logical block；
Step 2：First nodes 1 obtain the information of two-level node 1 from its metadata logical block 7, wherein, first nodes are deposited
Store up the corresponding relation of two-level node and its metadata logical block；
Step 3：Two-level node 1 searches metadata logical block 9 corresponding to first data block, then reads metadata logic
Block 9, obtain bitmap data information.
Storage bitmap data message needs additional metadata space to carry out memory node information in a manner of bitmap tree, and advantage exists
Extended in convenient, only need to distribute new metadata logical block when expanded and change nodal information；When creating remote copy
Need to check whether bitmap metadata space is sufficient, remote copy failure is created if metadata insufficient space, and return to member
The prompting of data space deficiency.
Step 102：The information of the node is inserted to the leaf node of RBTree, synchronizes data transfer；Wherein, institute
State RBTree and refer to self-balancing binary search tree.
The information of the node is inserted before the leaf node of RBTree, in addition to：By comparing current operating information
With the nodal information in RBTree, confirm that write conflict, which is not present, to be operated.
If write conflict operation be present, the node is skipped.
If synchronous success, deletes RBTree interior joint information, bit positions " setting to 0 " will be corresponded in bitmap；If it is not synchronous into
Work(, then RBTree interior joint information is deleted, skips the node.
The disk block message handled is preserved using RBTree, RBTree is a kind of self-balancing binary search tree, it
It is complicated, but its operation has good worst case run time, and be efficient in practice, and space is answered
It is less than hash table on miscellaneous degree；Although Adelson-Velskii-Landis tree and RBTree are much like, the deletion of Adelson-Velskii-Landis tree and insertion operation need repeatedly
Rotation process and constantly recall to root node, thus it is a large amount of delete and insertion operation in the case of, the operating efficiency of Adelson-Velskii-Landis tree is not
It is high；RBTree is the incomplete equilibrium binary tree that a kind of search efficiency is only second to Adelson-Velskii-Landis tree, and it can be answered with O (logn) time
Miscellaneous degree is inserted, deletion action, and its insertion and deletion at most needs twice or rotated three times that tree can be kept
Balance.Although the algorithm complex of the two is identical, in the worst cases, RBTree provides a kind of fast deletion in real time and insertion
The algorithm of one node, it is clear that the efficient operation of RBTree is more suitable for.
The leaf node of RBTree preserves the original position and size of current disk write operation, when disk write operation or same again
When step is operated into remote copy flow, information is inserted into red-black tree node；When operation returns, node is deleted from RBTree
Remove.Detect whether write conflict operation and the node in current operating information and RBTree is subjected to inquiry comparison, see whether deposit
Conflicting.
Fig. 4 show the data synchronization processing system construction drawing of the embodiment of the present invention 4, including triggering spider module, synchronization
Module；Wherein, the triggering spider module is connected with the synchronization module；
The triggering spider module, when occurring for trigger data synchronous event, set number sum in bitmap is obtained, if not
Be zero, then travel through be set in the bitmap 1 node and the information of the node is sent to the synchronization module；
The synchronization module, for the information of the node to be inserted to the leaf node of RBTree, synchronize data biography
It is defeated；Wherein, the RBTree refers to self-balancing binary search tree.
Compared to prior art, a kind of data synchronizing processing method and system are provided according to the present invention provided by the invention,
The problem of data are inconsistent as caused by write conflict when realizing the fast and accurately synchronous of data, while avoiding synchronization, and
And the implementation of RBTree, there is very high insert and delete efficiency, and have the worst good time complexity.
The preferred embodiments of the present invention are the foregoing is only, are not intended to limit the invention, for the skill of this area
For art personnel, the present invention can have various modifications and variations.Within the spirit and principles of the invention, that is made any repaiies
Change, equivalent substitution, improvement etc., should be included in the scope of the protection.
Claims (10)
1. a kind of data synchronizing processing method, it is characterised in that comprise the following steps：
When trigger data synchronous event occurs, set number sum in bitmap is obtained, if not zero, then travel through and be set in the bitmap
For 1 node；
The information of the node is inserted to the leaf node of RBTree, synchronizes data transfer；Wherein, the RBTree refers to
Self-balancing binary search tree, the leaf node of RBTree preserves the original position and size of current disk write operation, when write magnetic disk
Operation or resynchronisation insert red-black tree node when being operated into remote copy flow, by information, when operation returns, by node from
Red-black tree deletion.
2. according to the method for claim 1, it is characterised in that trigger data synchronous event refers to link failure recovery.
3. according to the method for claim 1, it is characterised in that a node in bitmap is bit positions, and one piece of mark is default
The data block of size, if the bit positions " put 1 ", the data block that the bit positions represent is between local terminal and distal end after link exception
It is modified；If the bit positions " setting to 0 ", then it represents that local terminal is consistent with remote data.
4. according to the method for claim 1, it is characterised in that bitmap data information is stored to internal memory in units of page.
5. according to the method for claim 4, it is characterised in that during link failure, by the bitmap data information in internal memory with
Page for unit sequentially linear memory to metadata management center；During link-recovery, from the metadata management center using page as
Unit reads institute's bit-map data information to page.
6. according to the method for claim 4, it is characterised in that during link failure, by the bitmap data information in internal memory with
The mode of bitmap tree is stored to metadata management center；During link-recovery, the bitmap is read in the metadata management center
Data message is to page.
7. according to the method for claim 6, it is characterised in that the bitmap tree includes host node, one or more one-levels
Node, one or more two-level nodes；The host node is connected with one or more first nodes respectively；The first nodes point
It is not connected with corresponding two-level node.
8. according to the method for claim 7, it is characterised in that host node stores first nodes and its metadata logical block
Corresponding relation, the corresponding relation of first nodes storage two-level node and its metadata logical block, two-level node store the bitmap
Data message.
9. according to the method for claim 1, it is characterised in that the information of the node is inserted to the leaf node of RBTree
Before, in addition to：By comparing current operating information and the nodal information in RBTree, confirm that write conflict, which is not present, to be operated.
10. a kind of data synchronization processing system, it is characterised in that including triggering spider module, synchronization module；Wherein, it is described to touch
Hair spider module is connected with the synchronization module；
The triggering spider module, when occurring for trigger data synchronous event, set number sum in bitmap is obtained, if not
Zero, then travel through be set in the bitmap 1 node and the information of the node is sent to the synchronization module；
The synchronization module, for the information of the node to be inserted to the leaf node of RBTree, synchronize data transfer；Its
In, the RBTree refers to self-balancing binary search tree, and the leaf node of RBTree preserves the start bit of current disk write operation
Put and size, when disk write operation or resynchronisation are operated into remote copy flow, information is inserted into red-black tree node, grasped
When returning, by node from red-black tree deletion.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410723701.8A CN104486387B (en) | 2014-12-02 | 2014-12-02 | A kind of data synchronizing processing method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410723701.8A CN104486387B (en) | 2014-12-02 | 2014-12-02 | A kind of data synchronizing processing method and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN104486387A CN104486387A (en) | 2015-04-01 |
| CN104486387B true CN104486387B (en) | 2018-03-27 |
Family
ID=52760891
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201410723701.8A Active CN104486387B (en) | 2014-12-02 | 2014-12-02 | A kind of data synchronizing processing method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN104486387B (en) |
Families Citing this family (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105183898A (en) * | 2015-09-29 | 2015-12-23 | 北京奇艺世纪科技有限公司 | Data sync method and data sync system |
| CN105843750A (en) * | 2016-03-22 | 2016-08-10 | 深圳森格瑞通信有限公司 | Linear data structure query efficiency improving method |
| CN106371764A (en) * | 2016-08-23 | 2017-02-01 | 浪潮(北京)电子信息产业有限公司 | Virtual block device-based data processing method and apparatus |
| US11960483B1 (en) | 2021-09-16 | 2024-04-16 | Wells Fargo Bank, N.A. | Constant time data structure for single and distributed networks |
| CN115242812A (en) * | 2022-07-25 | 2022-10-25 | 济南浪潮数据技术有限公司 | Node data synchronization method and device and computer readable storage medium |
| CN115801736B (en) * | 2023-01-09 | 2023-04-18 | 北京天际友盟信息技术有限公司 | IP address library construction loading method and device and IP address query method and device |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102713826A (en) * | 2009-12-22 | 2012-10-03 | 苹果公司 | Methods and apparatuses to allocate file storage via tree representations of a bitmap |
| CN102761615A (en) * | 2012-06-29 | 2012-10-31 | 浪潮(北京)电子信息产业有限公司 | Method and device for realizing data synchronism of long-distance duplication system |
| CN104036141A (en) * | 2014-06-16 | 2014-09-10 | 上海大学 | Open computing language (OpenCL)-based red-black tree acceleration algorithm |
-
2014
- 2014-12-02 CN CN201410723701.8A patent/CN104486387B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102713826A (en) * | 2009-12-22 | 2012-10-03 | 苹果公司 | Methods and apparatuses to allocate file storage via tree representations of a bitmap |
| CN102761615A (en) * | 2012-06-29 | 2012-10-31 | 浪潮(北京)电子信息产业有限公司 | Method and device for realizing data synchronism of long-distance duplication system |
| CN104036141A (en) * | 2014-06-16 | 2014-09-10 | 上海大学 | Open computing language (OpenCL)-based red-black tree acceleration algorithm |
Also Published As
| Publication number | Publication date |
|---|---|
| CN104486387A (en) | 2015-04-01 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN104486387B (en) | A kind of data synchronizing processing method and system | |
| US11461202B2 (en) | Remote data replication method and system | |
| CN108376109B (en) | Apparatus and method for copying volume of source array to target array, storage medium | |
| US9009402B2 (en) | Content addressable storage in legacy systems | |
| US6948089B2 (en) | Apparatus and method for multiple generation remote backup and fast restore | |
| CN102891849B (en) | Service data synchronization method, data recovery method, data recovery device and network device | |
| US7191299B1 (en) | Method and system of providing periodic replication | |
| US7308545B1 (en) | Method and system of providing replication | |
| EP3062226B1 (en) | Data replication method and storage system | |
| CN104077380B (en) | A kind of data de-duplication method, apparatus and system | |
| CN110209726A (en) | Distributed experiment & measurement system system, method of data synchronization and storage medium | |
| CN107885622B (en) | Handling Virtual Data Mover (VDM) failover conditions | |
| CN111104069B (en) | Multi-region data processing method and device of distributed storage system and electronic equipment | |
| CN105955836A (en) | Cold-hot backup automatic walkthrough multifunction system | |
| CN107657027B (en) | Data storage method and device | |
| CN105376277A (en) | Data synchronization method and device | |
| CN101714169A (en) | Method and system for transmitting data on multiple nodes | |
| CN106873902B (en) | File storage system, data scheduling method and data node | |
| CN103842954A (en) | Data processing method, device and system in storage system | |
| CN103530205A (en) | Method and device for processing fault duplicate in multiple duplicates | |
| CN106294013A (en) | A kind of data disaster tolerance method and system between different backup territory | |
| CN103412802B (en) | Disaster tolerant data file accesses the method and device controlling list backup | |
| CN116389233A (en) | Container cloud management platform active-standby switching system, method and device and computer equipment | |
| CN112667698B (en) | MongoDB data synchronization method based on converged media platform | |
| WO2021208401A1 (en) | Continuous data protection system and method for modern applications |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |