CN110688071A - Data synchronization method and system for reducing data synchronization quantity - Google Patents
Data synchronization method and system for reducing data synchronization quantity Download PDFInfo
- Publication number
- CN110688071A CN110688071A CN201910934340.4A CN201910934340A CN110688071A CN 110688071 A CN110688071 A CN 110688071A CN 201910934340 A CN201910934340 A CN 201910934340A CN 110688071 A CN110688071 A CN 110688071A
- Authority
- CN
- China
- Prior art keywords
- bitmap
- data
- synchronization
- monitoring
- synchronous
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 37
- 238000012544 monitoring process Methods 0.000 claims abstract description 133
- 230000001360 synchronised effect Effects 0.000 claims abstract description 102
- 230000008859 change Effects 0.000 claims abstract description 55
- 230000008569 process Effects 0.000 claims description 11
- 238000010586 diagram Methods 0.000 description 8
- 238000005516 engineering process Methods 0.000 description 5
- 230000008901 benefit Effects 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000011084 recovery Methods 0.000 description 2
- 230000010076 replication Effects 0.000 description 2
- 238000000638 solvent extraction Methods 0.000 description 2
- 241000700605 Viruses Species 0.000 description 1
- 238000013475 authorization Methods 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 238000007796 conventional method Methods 0.000 description 1
- 230000007123 defense Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 239000002699 waste material Substances 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0653—Monitoring storage devices or systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/061—Improving I/O performance
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0638—Organizing or formatting or addressing of data
- G06F3/064—Management of blocks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0655—Vertical data movement, i.e. input-output transfer; data movement between one or more hosts and one or more storage devices
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/0671—In-line storage system
- G06F3/0683—Plurality of storage devices
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Signal Processing For Digital Recording And Reproducing (AREA)
Abstract
The invention discloses a data synchronization method and a system for reducing data synchronization quantity, wherein the method comprises the following steps: step S1, dividing the space of the source end disk, establishing a monitoring bitmap of the source end disk according to the dividing result, comparing the data of the corresponding blocks of the source end disk and the standby end disk, updating bitmap information according to the comparison result, generating a synchronous bitmap, and realizing primary data synchronization according to the synchronous bitmap; step S2, monitoring the change of the source end disk data, and updating the monitoring bitmap information according to the monitoring result; step S3, when synchronizing, acquiring the monitoring bitmap as the synchronous bitmap, determining the synchronous source end disk block according to the synchronous bitmap information, further dividing the source end disk block, creating a new synchronous bitmap according to the dividing result, reading the data of the corresponding blocks of the source end and the standby end according to the new synchronous bitmap for comparison, updating the new synchronous bitmap according to the comparison result, and synchronizing the data according to the new synchronous bitmap information.
Description
Technical Field
The present invention relates to the field of data synchronization, and in particular, to a data synchronization method and system for reducing data synchronization amount.
Background
With the rapid development of science and technology, computers have become an indispensable part of people's lives, and the importance of the computers is increasing, and the computers are related to various industries. With the development of computers, the amount of data stored by people using computers is also rapidly increasing, and with the continuous increase of mass data of servers, the volume of data becomes more and more huge, and meanwhile, the safety and importance degree of various data are more and more emphasized by people.
It is difficult to imagine data that may be several G's, but its value may not be measured in money, because the loss of such important data may be difficult to estimate, and natural disasters, as small as viruses, power failures, or even accidental operator errors, may affect the normal operation of the system, or even cause the entire system to completely crash, which may result in the loss of such important data. There are often problems in the collection, processing, storage, transmission and distribution of information, the most significant of which are system failures, data loss or corruption, and thus data backup and data recovery are the last means to protect data and the last line of defense against active information attacks.
Data backup is the basis of disaster recovery, and refers to a process of copying all or part of a data set from a hard disk or an array of an application host to another storage medium in order to prevent data loss caused by misoperation of a system or system failure. The essence of data backup is actually copying of data, and certainly, data backup is not just simple as data replication, but data replication is certainly the most important and fundamental step of data backup, and whatever data backup technology or method is based on data synchronization.
The traditional data backup technology generally synchronizes all data of a source-end disk to a standby-end disk at regular time or when the data of the source-end disk is monitored to change.
Because more and more data are generated by people, more and more attention is paid to data backup, but the data backup technology is difficult due to the fact that the data are too huge, and the data synchronization step is undoubtedly the step that the massive data have the greatest influence on the data backup, because the larger the data quantity to be backed up is, the more the data which needs to be synchronized is; the source data is changed in real time, so that data synchronization also requires that the backed-up data is updated in real time, because the accuracy of tracking the change of the data affects the performance of the machine, the higher the accuracy, the greater the effect, the less the data needs to be synchronized, the lower the relative accuracy, the less the effect on the machine, but the larger the data amount needs to be synchronized, the two aspects are considered comprehensively to find a balance point to determine the accuracy of monitoring the change of the data.
Therefore, the invention provides a data synchronization technology for reducing the data synchronization amount, so as to reduce the data synchronization amount of the change data and improve the data backup efficiency under the condition of having the smallest influence on the machine.
Disclosure of Invention
In order to overcome the above-mentioned shortcomings of the prior art, an object of the present invention is to provide a data synchronization method and system for reducing the data synchronization amount, so as to reduce the data synchronization amount of the change data under the condition of having as little influence on the machine as possible, so that the ratio of the actually changed data amount to the synchronized data amount is as close to 1 as possible, and the efficiency of data backup is improved.
To achieve the above object, the present invention provides a data synchronization method for reducing data synchronization amount, comprising the following steps:
step S1, dividing the space of the source end disk, establishing a monitoring bitmap of the source end disk according to the dividing result, comparing the data of the corresponding blocks of the source end disk and the standby end disk, updating bitmap information according to the comparison result, generating a synchronous bitmap, and realizing primary data synchronization according to the synchronous bitmap;
step S2, monitoring the change of the source end disk data, and updating the monitoring bitmap information according to the monitoring result;
step S3, when synchronizing, acquiring the monitoring bitmap as the synchronous bitmap, determining the synchronous source end disk block according to the synchronous bitmap information, further dividing the source end disk block, creating a new synchronous bitmap according to the dividing result, reading the data of the corresponding blocks of the source end and the standby end according to the new synchronous bitmap for comparison, updating the new synchronous bitmap according to the comparison result, and synchronizing the data according to the new synchronous bitmap information.
Preferably, the step S1 further includes:
step S100, dividing a source end disk space, establishing a monitoring bitmap of the source end disk according to a dividing result, and initializing the value of each bit of the monitoring bitmap, wherein each bit of the monitoring bitmap represents a certain continuous area of the source end disk space, and the value of each bit of the monitoring bitmap is used for indicating whether the data of a corresponding block of the source end disk changes;
step S101, comparing each block of the source end with the data of the corresponding block of the standby end respectively according to the division result;
step S102, if the comparison result is the same, setting the value of the corresponding bit of the monitoring bitmap corresponding to the current block of the source end disk to 0, if the comparison result is different, setting the value of the corresponding bit of the monitoring bitmap corresponding to the current block of the source end disk to 1, generating a synchronous bitmap, after the monitoring bitmap is updated, carrying out primary synchronization according to the generated synchronous bitmap, and after the block synchronization represented by each bit in the synchronization process is completed, setting the corresponding bit of the synchronous bitmap to 0, and after the monitoring bitmap is updated and the synchronous bitmap is generated, setting all the bits of the monitoring bitmap to 0. .
Preferably, in step S2, if a change of the disk data of the source end is monitored, the value of the corresponding bit of the monitoring bitmap of the disk block corresponding to the change of the data is set to 1, and if the change of the disk data of the source end is not monitored, the value of the corresponding bit of the monitoring bitmap of the corresponding disk block is set to 0.
Preferably, the step S3 further includes:
step S300, determining a source disk block with current data change according to the synchronous bitmap information, dividing the source disk block, creating a new synchronous bitmap according to a division result, and initializing the value of each bit of the new synchronous bitmap.
Step S301, reading data of corresponding blocks of the source end and the standby end in sequence according to each new synchronous bitmap for comparison;
step S302, updating new synchronous bitmap information according to the comparison result;
step S303, perform data synchronization according to the new synchronization bitmap information.
Preferably, after step S303, the method further includes:
and step S304, after the data synchronization is finished according to the new synchronization bitmap information, deleting the new synchronization bitmap, and setting the corresponding bit position of the synchronization bitmap to be 0.
Preferably, in step S300, all the bits of the new synchronization bitmap are initialized to 1.
Preferably, in step S302, if the comparison results are the same, the bit value in the corresponding new synchronization bitmap is set to 0, and if the comparison results are different, the bit value in the corresponding new synchronization bitmap is set to 1.
Preferably, in step S303, the data of the block of the source disk corresponding to the bit with the bit value of 1 in the new synchronization bitmap is acquired and synchronized to the corresponding block of the backup.
Preferably, after step S3, the method further includes the following steps:
when the data synchronization is completed based on all the new synchronization bitmaps, the process returns to step S2.
In order to achieve the above object, the present invention further provides a data synchronization system for reducing data synchronization amount, including:
the system comprises a monitoring bitmap creating and primary synchronizing unit, a primary synchronizing unit and a secondary synchronizing unit, wherein the monitoring bitmap creating and primary synchronizing unit is used for dividing the space of a source end disk, creating a monitoring bitmap of the source end disk according to a dividing result, comparing data of corresponding blocks of the source end disk and a standby end disk, updating bitmap information according to a comparison result, generating a synchronous bitmap, and realizing primary data synchronization according to the synchronous bitmap;
the monitoring updating unit is used for monitoring the change of the source end disk data and updating the monitoring bitmap information according to the monitoring result;
and the new synchronous bitmap creating and synchronizing unit is used for acquiring the monitoring bitmap as a synchronous bitmap during synchronization, determining a synchronous source end disk block according to synchronous bitmap information, further dividing the source end disk block, creating a new synchronous bitmap according to a dividing result, reading data of corresponding blocks of a source end and a standby end according to the new synchronous bitmap, comparing, updating the new synchronous bitmap according to a comparison result, and synchronizing data according to the new synchronous bitmap information.
Compared with the prior art, the data synchronization method and the data synchronization system for reducing the data synchronization quantity achieve the purpose of reducing the data quantity needing synchronization by partitioning the blocks of the difference data again and carrying out data comparison for one time before synchronization on the basis of the traditional data blocks monitoring the data difference.
Drawings
FIG. 1 is a flow chart illustrating the steps of a data synchronization method for reducing the amount of data synchronization according to the present invention;
fig. 2 is a schematic diagram of creating a bitmap for a source disk in an embodiment of the present invention.
FIG. 3 is a diagram illustrating creation of a new bitmap in an embodiment of the present invention;
FIG. 4 is a system architecture diagram of a data synchronization system for reducing the amount of data synchronization in accordance with the present invention;
FIG. 5 is a diagram illustrating a process of data synchronization and initial recording according to an embodiment of the present invention;
FIG. 6 is a diagram illustrating a data synchronization process for reducing the amount of data synchronization according to an embodiment of the present invention.
Detailed Description
Other advantages and capabilities of the present invention will be readily apparent to those skilled in the art from the present disclosure by describing the embodiments of the present invention with specific embodiments thereof in conjunction with the accompanying drawings. The invention is capable of other and different embodiments and its several details are capable of modification in various other respects, all without departing from the spirit and scope of the present invention.
FIG. 1 is a flowchart illustrating steps of a data synchronization method for reducing data synchronization amount according to the present invention. As shown in fig. 1, a data synchronization method for reducing the data synchronization amount of the present invention includes the following steps:
and step S1, dividing the space of the source end disk, establishing a monitoring bitmap (bitmap) of the source end disk according to the division result, comparing the data of corresponding blocks of the source end disk and the standby end disk, updating bitmap information according to the comparison result, generating a synchronous bitmap, and realizing primary data synchronization according to the synchronous bitmap.
In the invention, a bitmap (bitmap) is used to record whether the data of a source end disk and a spare end disk are consistent, each bit in the bitmap represents a certain continuous area on the source end disk, the size of the area is determined by the accuracy of tracking change data, and each bit represents whether the data of corresponding blocks of the source end and the spare end have difference through different values, for example, a value of 1 represents that the data of the corresponding blocks of the source end and the spare end have difference, and a value of 0 represents that the data of the corresponding blocks of the source end and the spare end are the same.
Specifically, step S1 further includes:
step S100, a source end disk space is divided, a monitoring bitmap (bitmap) of the source end disk is established according to a dividing result, and a value of each bit of the monitoring bitmap is initialized, wherein each bit of the monitoring bitmap represents a certain continuous area of the source end disk space, and the value of each bit of the monitoring bitmap is used for indicating whether block data corresponding to the source end disk changes or not. In the embodiment of the present invention, a source-side disk space is divided into N small disk spaces with equal space size on average, as shown in fig. 2, a left side of fig. 2 represents a physical layout of a source-side disk, a source side is assumed to be a physical layout of a disk of 810MB, the source-side disk is divided into 9 equal parts from space, each part represents 90MB of space, a right side represents an established bitmap of the source-side disk, each bit represents a block on the source-side disk, for example, a first bit of the monitoring bitmap represents a block of 0-90 MB on the source-side disk, a value of each bit of the monitoring bitmap is initialized to 1, and a value of each bit is used for representing whether data of a corresponding block of the source-side disk changes. In the invention, each bit of the monitoring bitmap is initialized to 1, so that a user can choose not to perform initialization comparison, and thus, full disk synchronization is directly performed.
Step S101, comparing each block of the source end with the data of the corresponding block of the standby end respectively according to the division result;
and S102, updating the monitoring bitmap according to the comparison result, generating a synchronous bitmap, and carrying out primary synchronization according to the synchronous bitmap. Specifically, if the comparison results are the same, setting the value of the corresponding bit of the monitoring bitmap corresponding to the current block of the source-end disk to 0, if the comparison results are different, setting the value of the corresponding bit of the monitoring bitmap corresponding to the current block of the source-end disk to 1, generating a synchronization bitmap according to the updated monitoring bitmap, after the monitoring bitmap is updated, performing primary synchronization according to the generated synchronization bitmap, and setting the corresponding bit of the synchronization bitmap to 0 every time data of one block is synchronized in the synchronization process. It should be noted that, after the synchronization bitmap is generated according to the updated monitoring bitmap, all bit positions of the monitoring bitmap are set to 0 at the same time, so as to facilitate subsequent monitoring.
Specifically, the invention firstly makes a data comparison before the initial synchronization operation, namely, reads the data of the corresponding block from the source end and the backup end for comparison, if the data are the same, only the value of the bit representing the corresponding block in the monitoring bitmap (bitmap) needs to be updated (namely, the value is updated to 0), and then the data synchronization is not needed, so if the backup end disk has backed up the source end disk before, the operation can greatly reduce the data amount of the initial synchronization.
That is, in the present invention, the monitoring bitmap used for monitoring and the synchronization bitmap used for synchronization are different, before synchronization, the synchronization program will acquire a currently monitored monitoring bitmap from the monitoring program, after the monitoring program receives a request, the monitoring program will upload the current bitmap to the synchronization program to form a synchronization bitmap, after uploading the current monitoring bitmap to the synchronization program, all the bit positions of the monitoring bitmap are set to 0, and the synchronization bitmap is updated according to the synchronization schedule in the synchronization process, the corresponding bit position of the block which is completed by synchronization is 0, and the synchronization program will complete the synchronization according to the synchronization bitmap.
And step S2, monitoring the change of the source disk data, and updating the monitoring bitmap according to the monitoring result.
That is, after the initial synchronization of step S1, all bit values on the monitoring bitmap (bitmap) are set to 0, and then the change of the source-side disk data needs to be tracked, when the disk data of the source side changes, the value of the corresponding bit position of the disk block in the monitoring bitmap (bitmap) corresponding to the data change is set to 1, and if the disk data of the source side does not change, the value of the corresponding bit position of the corresponding disk block in the monitoring bitmap (bitmap) is set to 0. It should be noted here that, if the value of the bit corresponding to the monitoring bitmap (bitmap) of the disk block corresponding to the data change is already 1, when the data change is monitored again, the value of the bit is still 1 and will not change.
Step S3, during synchronization, acquiring the monitoring bitmap as a synchronization bitmap, determining a synchronous source end disk block according to the synchronization bitmap information, further dividing the source end disk block, creating a new synchronization bitmap according to the division result, reading the data of the corresponding blocks of the source end and the standby end according to the new synchronization bitmap, comparing, updating the new synchronization bitmap information, performing data synchronization according to the new synchronization bitmap information, and after performing data synchronization according to all the new synchronization bitmaps, setting the corresponding bit position of the synchronization bitmap to 0. Similarly, after the monitoring bitmap is obtained as the synchronization bitmap, all bit positions of the monitoring bitmap are set to be 0 at the same time, so that subsequent monitoring is facilitated.
In the invention, the data change on the disk of the source end of the trace record is for data synchronization, when the data change is traced, the conventional method is to synchronize the data of the corresponding block according to the bitmap recorded by the change, but this may also generate a large amount of unnecessary data synchronization. Specifically, according to the synchronization bitmap information, when the value of one bit of the synchronization bitmap is 1, it indicates that the corresponding source disk block currently has data change and needs synchronization, but sometimes all data of the source disk block need not be updated, for example, assuming that 1bit of the synchronization bitmap represents 1MB of data, and a certain bit value thereof is 1, but perhaps only 4KB of data in the block represented by the 1bit of the value is different, but the data synchronization still needs to be performed on the entire block, i.e. 1MB of data, which is equivalent to that 1020KB of data synchronization is performing useless work, which would result in a large waste of read and write resources of the disk, so in step S3, the disk block represented by each source-side bit of 1bit is divided again based on the synchronization bitmap (bitmap) in order to reduce the data amount of data synchronization before synchronization, and (2) establishing a new bitmap based on the source-end disk block, namely, equivalently regarding the block represented by the bit as a small disk, further dividing the small disk, marking the block of the small disk by using a new synchronization bitmap (bitmap), comparing again, and synchronizing according to the new synchronization bitmap (bitmap), so that the synchronous data volume can be greatly reduced, and although the overhead of a certain CPU is increased, the data volume is not enough compared with the reduced synchronous data volume.
Specifically, step S3 further includes:
step S300, determining a source disk block with current data change according to the synchronous bitmap information, dividing the source disk block, creating a new synchronous bitmap according to a division result, and initializing the value of each bit of the new synchronous bitmap.
Specifically, the monitoring bitmap obtained in step S2 is used as a synchronization bitmap, a source disk block corresponding to a bit value 1 in the synchronization bitmap is obtained, and is divided to create a corresponding new synchronization bitmap, and all bits of the new synchronization bitmap are initialized to 1. That is, according to the update result of step S2, how many new synchronization bitmaps are created for how many bits of the synchronization bitmap have a value of 1. As shown in fig. 3, the left diagram is bitmap information updated according to the monitoring result, assuming that the source-side disk block corresponding to the position 1 is 360-450MB, the source-side disk block is divided into 360-450MB, and the source-side disk block is divided into 9 equal parts spatially, each part represents 10MB of space, the new bitmap created according to the source-side disk block is as the right diagram in fig. 3, and all the initialization positions are all 1.
Step S301, reading data of corresponding blocks of the source end and the standby end in sequence according to each new synchronous bitmap for comparison;
step S302, updating new synchronous bitmap information according to the comparison result. Specifically, if the comparison results are the same, the bit value in the corresponding new synchronization bitmap is set to 0, and if the comparison results are different, the bit value in the corresponding new synchronization bitmap is set to 1.
Step S303, perform data synchronization according to the new synchronization bitmap information, that is, acquire data of the block of the source end disk corresponding to the bit with the bit value of 1 in the new bitmap and synchronize the data to the block corresponding to the backup end.
And step S304, after the data synchronization is finished according to the new synchronization bitmap information, deleting the new synchronization bitmap, and setting the corresponding bit position of the monitoring bitmap to be 0.
That is, after the data synchronization is completed, the new synchronization bitmap can be deleted to release the computer resource, because the monitoring is based on the monitoring bitmap, the new synchronization bitmap is only for reducing the data amount synchronized during the synchronization, and as for the synchronization bitmap used as the reference, after the data in the corresponding bit block is synchronized, the corresponding bit position is 0.
This step is equivalent to increasing the accuracy of tracking the changed data, but this increased accuracy is not to the monoblock disk, but the partial block that has data change that is confirmed through the first round of tracking, this overhead of comparing the accuracy of increasing monoblock disk monitoring is almost negligible or reduced by many times, this is completely in the acceptable scope, secondly, because the comparison of data is accomplished through CPU operation, and CPU's arithmetic speed is far greater than the read-write speed of disk, so reduce a large amount of disk reads and writes through CPU operation, the speed of synchronous data can be greatly accelerated, thereby the efficiency of backup has been greatly improved.
Through the steps, the synchronization of data is completed once, certainly, the data backup cannot be performed only once, the data synchronization is needed as long as the data of the source end is changed, so the consistency of the data of the source end and the backup end can be ensured, so the steps S2 and S3 can be repeated ceaselessly, the data synchronization amount is reduced, a large number of disk read-write operations can be liberated, the use of cache is reduced due to the fact that the read-write data are few, namely, a certain memory space is liberated, and the most influence of the data backup on the source end and the backup end is just the disk read-write operation and the memory consumption, so the influence of the data backup on the machine performance can be greatly reduced through the invention.
FIG. 4 is a diagram of a system architecture of a data synchronization system for reducing the amount of data synchronization according to the present invention. As shown in fig. 4, a data synchronization system for reducing the amount of data synchronization according to the present invention includes:
the monitoring bitmap creating and primary synchronizing unit 401 is configured to divide a space of the source end disk, create a monitoring bitmap (bitmap) of the source end disk according to a division result, compare data of corresponding blocks of the source end disk and the standby end disk, update bitmap information according to a comparison result, generate a synchronization bitmap, and implement primary data synchronization according to the synchronization bitmap.
In the invention, a bitmap (bitmap) is used to record whether the data of a source end disk and a spare end disk are consistent, each bit in the bitmap represents a certain continuous area on the source end disk, the size of the area is determined by the accuracy of tracking change data, and each bit represents whether the data of corresponding blocks of the source end and the spare end have difference through different values, for example, a value of 1 represents that the data of the corresponding blocks of the source end and the spare end have difference, and a value of 0 represents that the data of the corresponding blocks of the source end and the spare end are the same.
Specifically, the monitoring bitmap creation and primary synchronization unit 401 further includes:
the monitoring bitmap creating module is used for dividing the space of the source end disk, creating a monitoring bitmap (bitmap) of the source end disk according to a dividing result, and initializing the value of each bit of the monitoring bitmap, wherein each bit of the monitoring bitmap represents a certain continuous area of the space of the source end disk, and the value of each bit of the monitoring bitmap is used for indicating whether the data of a corresponding block of the source end disk changes.
The data comparison module is used for comparing each block of the source end with the data of the corresponding block of the standby end according to the division result;
and the bitmap information updating synchronization module updates the monitoring bitmap according to the comparison result, generates a synchronization bitmap and carries out primary synchronization according to the synchronization bitmap. Specifically, if the comparison results are the same, setting the value of the corresponding bit of the monitoring bitmap corresponding to the current block of the source-end disk to be 0, if the comparison results are different, synchronizing the data of the current block of the source-end disk to the corresponding block of the standby end, simultaneously setting the value of the corresponding bit of the monitoring bitmap corresponding to the current block of the source-end disk to be 1, generating a synchronization bitmap according to the updated monitoring bitmap, after the monitoring bitmap is updated, performing initial synchronization according to the generated synchronization bitmap, and setting the corresponding bit of the synchronization bitmap to be 0 every time when the data synchronization of one block is completed in the synchronization process. It should be noted that, after the synchronization bitmap is generated according to the updated monitoring bitmap, all bit positions of the monitoring bitmap are set to 0 at the same time, so as to facilitate subsequent monitoring.
And a monitoring updating unit 402, configured to monitor changes in the source-end disk data, and update monitoring bitmap information according to the monitoring result.
That is to say, after the initial synchronization is performed, all bit values on the monitoring bitmap (bitmap) should be set to 0, and then the change of the source-side disk data needs to be tracked by using the monitoring update unit 402, when the disk data of the source side changes, the value of the corresponding bit position of the disk block in the bitmap (bitmap) corresponding to the data change is set to 1, and if the disk data of the source side does not change, the value of the corresponding bit position of the corresponding disk block in the bitmap (bitmap) corresponding to the disk block is set to 0. It should be noted here that, if the value of the bit corresponding to the bitmap (bitmap) of the disk block corresponding to the data change is already 1, when the data change is detected again, the value of the bit is still 1, and is not changed.
A new synchronization bitmap creating and synchronizing unit 403, configured to, during synchronization, acquire a monitoring bitmap as a synchronization bitmap, determine a synchronized source-end disk block according to the synchronization bitmap information, further divide the source-end disk block, create a new synchronization bitmap according to a division result, read data of a corresponding block of the source end and the standby end according to the new synchronization bitmap, compare the data, update new synchronization bitmap information, perform data synchronization according to the new synchronization bitmap information, and set a corresponding bit position of the synchronization bitmap to 0 after performing data synchronization according to all new synchronization bitmaps. Similarly, after the monitoring bitmap is obtained as the synchronization bitmap, all bit positions of the monitoring bitmap are set to be 0 at the same time, so that subsequent monitoring is facilitated.
Specifically, the new synchronization bitmap creation and synchronization unit 403 further includes:
and the new synchronous bitmap creating module is used for determining a source end disk block with current data change according to the bitmap information, dividing the source end disk block, creating a new bitmap according to a dividing result, and initializing the value of each bit of the new bitmap.
Specifically, the source disk block corresponding to the bit value 1 in the synchronization bitmap is obtained as the synchronization bitmap according to the monitoring bitmap obtained by the monitoring update unit 402, and is divided to create a corresponding new synchronization bitmap, and all bits of the new synchronization bitmap are initialized to 1. That is, according to the update result of the monitoring update unit 402, how many new bitmaps are created for how many bits of the synchronization bitmap have a value of 1.
The data comparison module is used for reading data of blocks corresponding to the source end and the standby end in sequence according to each new synchronous bitmap and comparing the data;
and the new bitmap information updating module updates the new synchronous bitmap information according to the comparison result. Specifically, if the comparison results are the same, the bit value in the corresponding new synchronization bitmap is set to 0, and if the comparison results are different, the bit value in the corresponding new synchronization bitmap is set to 1.
And the data synchronization module is used for performing data synchronization according to the new synchronization bitmap, namely acquiring data of a block of the source-end disk corresponding to a bit with a bit value of 1 in the new bitmap and synchronizing the data to a block corresponding to the standby end, deleting the new synchronization bitmap after completing the data synchronization of each new synchronization bitmap according to the information of the new synchronization bitmap, and setting the corresponding bit position of the basic synchronization bitmap as 0.
Examples
In this embodiment, the data synchronization method for reducing the data synchronization amount of the present invention includes the following steps:
And 2, performing data comparison before primary synchronization, namely comparing data of corresponding blocks read from the source-end disk and the standby-end disk, if the data of the corresponding blocks are the same, setting the value of a bit representing the corresponding block in the updated bitmap to be 0, if the data of the corresponding block are different, setting the value of the corresponding bit of the monitoring bitmap corresponding to the current block of the source-end disk to be 1 to form a synchronous bitmap, performing primary synchronization according to the updated synchronous bitmap after updating the bitmap, updating the synchronous bitmap in real time in the synchronization process, and setting all bit values of the synchronous bitmap to be 0 after the synchronization is completed. In this step, after the synchronization bitmap is formed according to the updated monitoring bitmap, all bit positions of the monitoring bitmap are set to 0 at the same time, so as to facilitate subsequent monitoring.
The change of the data of the disk is monitored mainly for data synchronous positioning, and since the whole disk cannot be backed up once every time of synchronization, it is necessary to monitor which blocks on the disk have changed in order to ensure that the data of the source end and the backup end are consistent after synchronization. The bitmap information is updated mainly for recording the result of monitoring data change, and is recorded on the bitmap, and the value of the corresponding bit of the block with data change is set to 1, which represents that the data of the block is changed.
And 4, establishing a new synchronous bitmap of the block with changed data based on the bitmap information in the step 3 during synchronization, comparing the data of the corresponding blocks of the source end disk and the standby end disk according to the new synchronous bitmaps, updating the new bitmaps, and synchronizing the data according to the new bitmaps.
In the present invention, a new bitmap of each block with changed data is created based on bitmap information, so as to reduce the overhead of data backup, which is equivalent to increasing the accuracy of tracking changed data, because if all the disks are doing so, the overhead of tracking the change of the positioning data is increased undoubtedly, and the accuracy of starting the method is the most suitable accuracy determined by balancing various aspects, and the overhead of data backup is increased undoubtedly without any authorization, while the problem does not exist in the block based on changed data, because the change of data is limited, and the increased overhead is negligible compared with the benefit of the change, although some overhead is increased somewhat. And performing data comparison according to the new bitmaps, wherein the purpose of updating the new bitmaps is to reduce the data synchronization amount, and the purpose of establishing the new bitmaps is to improve the accuracy of tracking the changed data, so that useless data synchronization can be reduced due to high accuracy, namely, the data synchronization amount is reduced.
As shown in fig. 6, the corresponding bit of the bitmap corresponding to the source-end disk block of 360-450MB is obtained as 1 according to the monitored bitmap information before synchronization, the source-end disk block is divided into 360-450MB, and is divided into 9 equal parts in space, each part represents 10MB of space, a new bitmap is created according to the bit, all the bit initializations are all 1, the data of the corresponding block of the source-end disk and the backup-end disk is read according to the block of the new bitmap for comparison, and the new bitmap is obtained according to the comparison result by updating, so that the data which originally needs to be synchronized with 90MB is changed into the data which only needs to be synchronized with 20MB, and the data synchronization amount is greatly reduced.
Therefore, one-time data synchronization is completed, improvement is performed on the basis of traditional data synchronization, useless data synchronization is greatly reduced, and meanwhile the influence of data backup on the performance of a machine is reduced.
Of course, data backup cannot be synchronized only once, and data synchronization is required only when the source data changes, so that the consistency of the source data and the backup data can be ensured, and the steps 3 and 4 can be repeated ceaselessly.
In summary, the data synchronization method and system for reducing data synchronization volume of the present invention achieve the purpose of reducing the data volume to be synchronized by partitioning the blocks of the difference data again and performing a data comparison before synchronization on the basis of the traditional data blocks monitoring data differences, and greatly reduce the data synchronization volume under the condition of increasing a small amount of CPU computation overhead on the basis of the traditional data backup, thereby greatly reducing the overhead of disk read-write and the consumption of memory, greatly reducing the influence of data backup on the machine performance, and greatly improving the efficiency of data synchronization.
The foregoing embodiments are merely illustrative of the principles and utilities of the present invention and are not intended to limit the invention. Modifications and variations can be made to the above-described embodiments by those skilled in the art without departing from the spirit and scope of the present invention. Therefore, the scope of the invention should be determined from the following claims.
Claims (10)
1. A data synchronization method for reducing data synchronization amount includes the following steps:
step S1, dividing the space of the source end disk, establishing a monitoring bitmap of the source end disk according to the dividing result, comparing the data of the corresponding blocks of the source end disk and the standby end disk, updating bitmap information according to the comparison result, generating a synchronous bitmap, and realizing primary data synchronization according to the synchronous bitmap;
step S2, monitoring the change of the source end disk data, and updating the monitoring bitmap information according to the monitoring result;
step S3, when synchronizing, acquiring the monitoring bitmap as the synchronous bitmap, determining the synchronous source end disk block according to the synchronous bitmap information, further dividing the source end disk block, creating a new synchronous bitmap according to the dividing result, reading the data of the corresponding blocks of the source end and the standby end according to the new synchronous bitmap for comparison, updating the new synchronous bitmap according to the comparison result, and synchronizing the data according to the new synchronous bitmap information.
2. The data synchronization method for reducing the data synchronization amount according to claim 1, wherein the step S1 further comprises:
step S100, dividing a source end disk space, establishing a monitoring bitmap of the source end disk according to a dividing result, and initializing the value of each bit of the monitoring bitmap, wherein each bit of the monitoring bitmap represents a certain continuous area of the source end disk space, and the value of each bit of the monitoring bitmap is used for indicating whether the data of a corresponding block of the source end disk changes;
step S101, comparing each block of the source end with the data of the corresponding block of the standby end respectively according to the division result;
step S102, if the comparison result is the same, setting the value of the corresponding bit of the monitoring bitmap corresponding to the current block of the source end disk to 0, if the comparison result is different, setting the value of the corresponding bit of the monitoring bitmap corresponding to the current block of the source end disk to 1, generating a synchronous bitmap, after the monitoring bitmap is updated, carrying out primary synchronization according to the generated synchronous bitmap, and after the block synchronization represented by each bit in the synchronization process is completed, setting the corresponding bit of the synchronous bitmap to 0, and after the monitoring bitmap is updated and the synchronous bitmap is generated, setting all the bits of the monitoring bitmap to 0.
3. A data synchronization method for reducing the amount of data synchronization as set forth in claim 2, wherein: in step S2, if the change of the disk data at the source end is monitored, the value of the corresponding bit of the monitoring bitmap of the disk block corresponding to the data change is set to 1, and if the change of the disk data at the source end is not monitored, the value of the corresponding bit of the monitoring bitmap of the corresponding disk block is set to 0.
4. The data synchronization method for reducing the data synchronization amount according to claim 2, wherein the step S3 further comprises:
step S300, determining a source disk block with current data change according to the synchronous bitmap information, dividing the source disk block, creating a new synchronous bitmap according to a division result, and initializing the value of each bit of the new synchronous bitmap.
Step S301, reading data of corresponding blocks of the source end and the standby end in sequence according to each new synchronous bitmap for comparison;
step S302, updating new synchronous bitmap information according to the comparison result;
step S303, perform data synchronization according to the new synchronization bitmap information.
5. The data synchronization method for reducing the data synchronization amount according to claim 4, further comprising, after the step S303:
and step S304, after the data synchronization is finished according to the new synchronization bitmap information, deleting the new synchronization bitmap, and setting the corresponding bit position of the synchronization bitmap to be 0.
6. A data synchronization method for reducing the amount of data synchronization as recited in claim 5, wherein: in step S300, all bits of the new synchronization bitmap are initialized to 1.
7. A data synchronization method for reducing the amount of data synchronization as recited in claim 5, wherein: in step S302, if the comparison results are the same, the bit value in the corresponding new synchronization bitmap is set to 0, and if the comparison results are different, the bit value in the corresponding new synchronization bitmap is set to 1.
8. A data synchronization method for reducing the amount of data synchronization as recited in claim 6, wherein: in step S303, the data of the block of the source disk corresponding to the bit with the bit value of 1 in the new synchronization bitmap is obtained and synchronized to the corresponding block of the backup.
9. The data synchronization method for reducing the data synchronization amount according to claim 1, further comprising, after the step S3, the steps of:
when the data synchronization is completed based on all the new synchronization bitmaps, the process returns to step S2.
10. A data synchronization system that reduces the amount of data synchronization, comprising:
the system comprises a monitoring bitmap creating and primary synchronizing unit, a primary synchronizing unit and a secondary synchronizing unit, wherein the monitoring bitmap creating and primary synchronizing unit is used for dividing the space of a source end disk, creating a monitoring bitmap of the source end disk according to a dividing result, comparing data of corresponding blocks of the source end disk and a standby end disk, updating bitmap information according to a comparison result, generating a synchronous bitmap, and realizing primary data synchronization according to the synchronous bitmap;
the monitoring updating unit is used for monitoring the change of the source end disk data and updating the monitoring bitmap information according to the monitoring result;
and the new synchronous bitmap creating and synchronizing unit is used for acquiring the monitoring bitmap as a synchronous bitmap during synchronization, determining a synchronous source end disk block according to synchronous bitmap information, further dividing the source end disk block, creating a new synchronous bitmap according to a dividing result, reading data of corresponding blocks of a source end and a standby end according to the new synchronous bitmap, comparing, updating the new synchronous bitmap according to a comparison result, and synchronizing data according to the new synchronous bitmap information.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910934340.4A CN110688071A (en) | 2019-09-29 | 2019-09-29 | Data synchronization method and system for reducing data synchronization quantity |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910934340.4A CN110688071A (en) | 2019-09-29 | 2019-09-29 | Data synchronization method and system for reducing data synchronization quantity |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110688071A true CN110688071A (en) | 2020-01-14 |
Family
ID=69111018
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910934340.4A Pending CN110688071A (en) | 2019-09-29 | 2019-09-29 | Data synchronization method and system for reducing data synchronization quantity |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110688071A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112769934A (en) * | 2021-01-07 | 2021-05-07 | 卓望数码技术（深圳）有限公司 | State synchronization method, system, electronic control device and storage medium |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101183383A (en) * | 2007-12-17 | 2008-05-21 | 中国科学院计算技术研究所 | Snapshot system and method of use thereof |
| CN103064766A (en) * | 2013-01-07 | 2013-04-24 | 天津火星科技有限公司 | Fast data synchronization method in continuous data protection technology |
| CN103885717A (en) * | 2012-12-19 | 2014-06-25 | 杭州宏杉科技有限公司 | Data replication method and data replication device |
| CN104615505A (en) * | 2015-02-11 | 2015-05-13 | 浪潮(北京)电子信息产业有限公司 | Method for remote duplicated record of data changes |
| CN107797770A (en) * | 2017-11-07 | 2018-03-13 | 深圳神州数码云科数据技术有限公司 | A kind of synchronous method and device of Disk State information |
-
2019
- 2019-09-29 CN CN201910934340.4A patent/CN110688071A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101183383A (en) * | 2007-12-17 | 2008-05-21 | 中国科学院计算技术研究所 | Snapshot system and method of use thereof |
| CN103885717A (en) * | 2012-12-19 | 2014-06-25 | 杭州宏杉科技有限公司 | Data replication method and data replication device |
| CN103064766A (en) * | 2013-01-07 | 2013-04-24 | 天津火星科技有限公司 | Fast data synchronization method in continuous data protection technology |
| CN104615505A (en) * | 2015-02-11 | 2015-05-13 | 浪潮(北京)电子信息产业有限公司 | Method for remote duplicated record of data changes |
| CN107797770A (en) * | 2017-11-07 | 2018-03-13 | 深圳神州数码云科数据技术有限公司 | A kind of synchronous method and device of Disk State information |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112769934A (en) * | 2021-01-07 | 2021-05-07 | 卓望数码技术（深圳）有限公司 | State synchronization method, system, electronic control device and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10268695B2 (en) | Snapshot creation | |
| US9495370B1 (en) | Data recovery point review in a continuous data protection system | |
| US9348827B1 (en) | File-based snapshots for block-based backups | |
| JP4419884B2 (en) | Data replication apparatus, method, program, and storage system | |
| US8825653B1 (en) | Characterizing and modeling virtual synthetic backup workloads | |
| US11093387B1 (en) | Garbage collection based on transmission object models | |
| JP6195614B2 (en) | System, method and computer program for cascading volume source cleaning (cascading volume source cleaning) | |
| US9075535B2 (en) | Source cleaning cascaded volumes using reference counts | |
| JP2005301497A (en) | Storage management system, restoration method and its program | |
| US20160154711A1 (en) | Flash copy for disaster recovery (dr) testing | |
| CN107562578B (en) | Snapshot creating method, device, equipment and storage medium for stored data | |
| CN109144416B (en) | Method and device for querying data | |
| CN105573859A (en) | Data recovery method and device of database | |
| CN104077380A (en) | Method and device for deleting duplicated data and system | |
| US9262290B2 (en) | Flash copy for disaster recovery (DR) testing | |
| WO2024113685A1 (en) | Data recovery method for raid array and related apparatus | |
| JP6281511B2 (en) | BACKUP CONTROL DEVICE, BACKUP CONTROL METHOD, AND PROGRAM | |
| EP2998903B1 (en) | System and method for robust full-drive encryption | |
| CN110187834B (en) | Data processing method and device for duplicate copies and electronic equipment | |
| CN111045865A (en) | Real-time synchronization method and system based on block replication | |
| US8595271B1 (en) | Systems and methods for performing file system checks | |
| CN110688071A (en) | Data synchronization method and system for reducing data synchronization quantity | |
| US10740189B2 (en) | Distributed storage system | |
| CN105138429B (en) | A kind of Copy on write Snapshot Method and system | |
| CN110019092B (en) | Data storage method, controller and system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20200114 |