CN102323908A - Method and system for data caching in large volume data synchronization process on disk - Google Patents
Method and system for data caching in large volume data synchronization process on disk Download PDFInfo
- Publication number
- CN102323908A CN102323908A CN201110221536A CN201110221536A CN102323908A CN 102323908 A CN102323908 A CN 102323908A CN 201110221536 A CN201110221536 A CN 201110221536A CN 201110221536 A CN201110221536 A CN 201110221536A CN 102323908 A CN102323908 A CN 102323908A
- Authority
- CN
- China
- Prior art keywords
- data
- data block
- block
- piece
- variation
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Signal Processing For Digital Recording And Reproducing (AREA)
- Memory System Of A Hierarchy Structure (AREA)
Abstract
The invention discloses a method and system for data caching in a large volume data synchronization process on a disk and relates to the field of disk data caching. The system comprises the following modules: a data block indexing module, a data block comparison module, a data caching module, wherein the data block indexing module is used for dividing the disk data into a plurality of data blocks and allocating an unique block index for each data block according to the positions of the divided data blocks; the data block comparison module is used for judging whether the block index of the currently varying data block is in a block index range of the synchronized data blocks, if so, confirming that the currently varying data block is an effectively varying data block; and the data caching module is used for caching the data blocks, which are confirmed as the effectively varying data block by the data block comparison module, as disk data block information. The technical scheme of the invention can be used for improving the data protection efficiency of a data management system, improving the reliability of data protection and ensuring the data integrity on the basis of preventing any loss of effective data when a huge amount of data varies on the disk.
Description
Technical field
The present invention relates to the data in magnetic disk cache field, data cache method and system in particularly a kind of data in magnetic disk synchronizing process.
Background technology
In data management system, data block is deposited the operation that is to use frequency the highest, because any data block of data in magnetic disk changes, will the data in magnetic disk piece be stored in backup area.But data in magnetic disk changes moment can be produced a lot of data blocks and change, and backup area will be deposited a lot of data blocks this moment, thereby possibly cause backup area can not in time backup to the variation of all data blocks.Change in order to tackle the valid data piece, can adopt the metadata cache technology in the data in magnetic disk synchronizing process.
The metadata cache technology that in the data in magnetic disk synchronizing process, adopts relates to delta data piece record; And traditional delta data piece record is meant; Changed the one piece of data piece, just deposited this data block in buffer area, the pointer that will point to this data block then is retained in the data block formation to be managed.And during system's long-play, the buffer zone in the data management system can arrive named cache district maximal value very soon, makes that system reliability and integrality can not get ensureing, system is just so unstable.
By on can know that how the maintenance data buffer memory is preserved effective data block and changed, and is the technology of the required innovation of current data management system.
Summary of the invention
Technical matters to be solved by this invention is data cache method and system in a kind of disk Large Volume Data synchronizing process to be provided, in time buffer memory valid data piece.
In order to address the above problem, the invention discloses a kind of data buffering system, comprising:
The data block index module is divided into a plurality of data blocks with data in magnetic disk, and is the unique piece index of each data block allocations according to the position of the data block of being divided;
The data block comparing module; Whether the piece index of judging the data block of current variation is in the piece index range of data in synchronization piece; If the piece index of the data block of current variation is in the piece index range of data in synchronization piece, the data block of then confirming current variation be the effective data block of variation;
Data cache module is with the data block of the definite effective variation of the said data block comparing module of data in magnetic disk block message mode buffer memory.
Preferably, said system also comprises the metadata cache filtering module;
Said data block comparing module, when the piece index of the data block of judging current variation was in outside the piece index range of data in synchronization piece, the data block of confirming current variation was the data block of invalid change;
Said metadata cache filtering module; When said data comparing module confirms that the data block of current variation is the data block that effectively changes; Call the data block of the said effective variation of said data cache module buffer memory; When said data comparing module confirms that the data block of current variation is the data block of invalid change, remove the data block of determined invalid change.
Preferably, in the said system, said data cache module, the piece index point of the data block of the said effective variation of buffer memory, and data block inquiry, deletion are provided and add operation.
Preferably; In the said system; Said metadata cache filtering module, when said data comparing module confirmed that the data block of current variation is the data block that effectively changes, also the data block with said effective variation sent to the backup server end; And after said backup server end backs up the data block of said effective variation, call said data cache module and remove the data block that this effectively changes.
The invention also discloses the data cache method in a kind of disk Large Volume Data synchronizing process, comprising:
Data in magnetic disk is divided into a plurality of data blocks, and is the unique piece index of each data block allocations according to the position of the data block of being divided;
In the data in magnetic disk synchronizing process; Whether the piece index of judging the data block of current variation is in the piece index range of data in synchronization piece; If the piece index of the data block of current variation is in the piece index range of data in synchronization piece; The data block of then confirming current variation is the data block that effectively changes, with the data block of the determined effective variation of data in magnetic disk block message mode buffer memory.
Preferably, said method also comprises:
If when the piece index of the data block of current variation is in outside the piece index range of data in synchronization piece, confirm that the data block of current variation is the data block of invalid change, remove the data block of determined invalid change.
Preferably, in the said method, refer to the data block of the determined effective variation of data in magnetic disk block message mode buffer memory:
The piece index point of the data block of the said effective variation of buffer memory, and data block inquiry, deletion are provided and add operation.
Preferably, said method also comprises:
When the data block of confirming current variation was the data block that effectively changes, also the data block with said effective variation sent to the backup server end, and after said backup server end backs up the data block of said effective variation, eliminated the data block that this effectively changes.
Technical scheme of the present invention writes down effective data block that changes, and removes the data block of invalid change in the synchronizing process, thereby the data block that the data block of the effective variation of buffer memory in time and transmission effectively change arrives backup server.Like this, when big data quantity on the disk changes, just can on the basis that valid data do not run off, improve the efficient of data protection in the data management system, improve the reliability of data protection, guarantee data integrity.
Description of drawings
Fig. 1 is a data buffering system structural representation in the present embodiment;
Fig. 2 is a metadata cache process flow diagram in the disk Large Volume Data synchronizing process in the present embodiment.
Embodiment
For the purpose, technical scheme and the advantage that make the application is clearer, technical scheme of the present invention is done further explain below in conjunction with accompanying drawing.Need to prove, under the situation of not conflicting, the combination each other arbitrarily of the application's embodiment and the characteristic among the embodiment.
At present, in disk Large Volume Data synchronizing process, data block takes place repeatedly to change, and the data pool that causes the data block change records is constantly increased, and causes protected data can not update to standby server.Therefore, constantly increase in order to solve the data block shelf space, the applicant proposes, and in the Large Volume Data synchronizing process, only writes down the data block that effectively changes, and the data block that is about to effectively variation is saved in the metadata cache zone and gets final product.
Embodiment 1
Present embodiment provides a kind of data buffering system based on above-mentioned thought, and its structure is as shown in Figure 1, comprises following each module.
Data block piece index module is divided into a plurality of data blocks with data in magnetic disk, and is the unique piece index of each data block allocations according to the position of the data block of being divided.
Particularly, data block piece index module is divided into the interface of plurality of data piece with the data block on the disk, and each data block is produced unique index of discerning indicates.
The data block comparing module; Whether the data block piece index of judging current variation is in the data in synchronization piece piece index range; If the data block piece index of current variation is in the data in synchronization piece piece index range, confirm that then this data block be the effective data block of variation.
Data cache module is with the data block of the definite effective variation of data in magnetic disk block message mode caching data block comparing module.
Particularly, data cache module is the effectively piece index point of the data block of variation of buffer memory, and data block inquiry, deletion is provided and adds operation.
Also have in some schemes, the data block comparing module when the piece index of the data block of judging current variation is in outside the piece index range of data in synchronization piece, confirms that then the data block of current variation is the data block of invalid change.Increase a data cache filter module this moment; When this module confirms that in the data comparing module data block of current variation is the data block that effectively changes; Call the data block that the data cache module buffer memory effectively changes; When the data comparing module confirms that the data block of current variation is the data block of invalid change, then remove the data block of determined invalid change.
Also have in the preferred version; The metadata cache filtering module; When the data comparing module confirms that the data block of current variation is the data block that effectively changes; The data block that also will effectively change sends to the backup server end, and after the data block that the backup of backup server end effectively changes, calls data cache module and remove the data block that this effectively changes.
Embodiment 2
Present embodiment is introduced a kind of method of in disk Large Volume Data synchronizing process, carrying out metadata cache.
At first data in magnetic disk is divided into a plurality of data blocks; And be the unique piece index of each data block allocations according to the position of the data block of being divided; Afterwards in the data in magnetic disk synchronizing process; Whether the piece index of judging the data block of current variation is in the piece index range of data in synchronization piece; If the piece index of the data block of current variation is in the piece index range of data in synchronization piece, the data block of then confirming current variation be the effective data block of variation, with the data block of the determined effective variation of data in magnetic disk block message mode buffer memory.
Wherein,, confirm that the data block of current variation is the data block of invalid change, remove the data block of determined invalid change if when the piece index of the data block of current variation is in outside the piece index range of data in synchronization piece.
And refer to the data block of the determined effective variation of data in magnetic disk block message mode buffer memory:
The piece index point of the data block of the said effective variation of buffer memory, and data block inquiry, deletion are provided and add operation.
Also have in some schemes; When the data block of confirming current variation is the data block that effectively changes; The data block that also will effectively change sends to the backup server end, and after the data block that the backup of backup server end effectively changes, eliminates the data block that this effectively changes.
Below in conjunction with the concrete method that specifies above-mentioned metadata cache of using.The course of work of the system that is promptly provided with embodiment 1 is the example explanation.Particularly, this metadata cache process is as shown in Figure 2, comprises the steps:
Step 1 when the data in magnetic disk piece changes, judges whether to be in the disk synchrodata stage, if, get into step 2, otherwise process ends;
Step 2 is obtained the piece index range of data in synchronization piece;
Step 3 is obtained the piece index of the data block of current variation;
Whether step 4, the piece index of judging the data block of current variation are in the piece index range of data in synchronization piece, if, get into step 5, otherwise, confirm that the data block of current variation is the data block of invalid change, and remove this data block;
Step 5, the data block of confirming current variation is for effective data block that changes, with the data block of the determined effective variation of data in magnetic disk block message mode buffer memory;
Step 6 sends to the backup server end with the data block that effectively changes, and after the data block that the backup of backup server end effectively changes, removes the data that this effectively changes.
One of ordinary skill in the art will appreciate that all or part of step in the said method can instruct related hardware to accomplish through program, said program can be stored in the computer-readable recording medium, like ROM (read-only memory), disk or CD etc.Alternatively, all or part of step of the foregoing description also can use one or more integrated circuit to realize.Correspondingly, each the module/unit in the foregoing description can adopt the form of hardware to realize, also can adopt the form of software function module to realize.The application is not restricted to the combination of the hardware and software of any particular form.
Can find out from the foregoing description; The present techniques scheme is when the timing disk data; Data block to the data in magnetic disk piece takes place effectively to change is carried out record; Thereby filter out backup server needed valid data in the time of the timing disk data, and be saved in backup server, thereby optimized the reliability and the integrality of data management system greatly.Run off thereby solved user's valid data, the data block change frequency is too much when having avoided synchronous, causes system to carry out the problem of timing disk data repeatedly.
The above is merely the application's preferred embodiments, is not the protection domain that is used to limit the application.All within the application's spirit and principle, any modification of being made, be equal to replacement, improvement etc., all should be included within the application's the protection domain.
Claims (8)
1. a data buffering system is characterized in that, this system comprises:
The data block index module is divided into a plurality of data blocks with data in magnetic disk, and is the unique piece index of each data block allocations according to the position of the data block of being divided;
The data block comparing module; Whether the piece index of judging the data block of current variation is in the piece index range of data in synchronization piece; If the piece index of the data block of current variation is in the piece index range of data in synchronization piece, the data block of then confirming current variation be the effective data block of variation;
Data cache module is with the data block of the definite effective variation of the said data block comparing module of data in magnetic disk block message mode buffer memory.
2. the system of claim 1 is characterized in that, this system also comprises the metadata cache filtering module;
Said data block comparing module, when the piece index of the data block of judging current variation was in outside the piece index range of data in synchronization piece, the data block of confirming current variation was the data block of invalid change;
Said metadata cache filtering module; When said data comparing module confirms that the data block of current variation is the data block that effectively changes; Call the data block of the said effective variation of said data cache module buffer memory; When said data comparing module confirms that the data block of current variation is the data block of invalid change, remove the data block of determined invalid change.
3. system as claimed in claim 2 is characterized in that,
Said data cache module, the piece index point of the data block of the said effective variation of buffer memory, and data block inquiry, deletion are provided and add operation.
4. like claim 2 or 3 described systems, it is characterized in that,
Said metadata cache filtering module; When said data comparing module confirms that the data block of current variation is the data block that effectively changes; Also the data block with said effective variation sends to the backup server end; And after said backup server end backs up the data block of said effective variation, call said data cache module and remove the data block that this effectively changes.
5. the data cache method in the disk Large Volume Data synchronizing process is characterized in that this method comprises:
Data in magnetic disk is divided into a plurality of data blocks, and is the unique piece index of each data block allocations according to the position of the data block of being divided;
In the data in magnetic disk synchronizing process; Whether the piece index of judging the data block of current variation is in the piece index range of data in synchronization piece; If the piece index of the data block of current variation is in the piece index range of data in synchronization piece; The data block of then confirming current variation is the data block that effectively changes, with the data block of the determined effective variation of data in magnetic disk block message mode buffer memory.
6. method as claimed in claim 5 is characterized in that, this method also comprises:
If when the piece index of the data block of current variation is in outside the piece index range of data in synchronization piece, confirm that the data block of current variation is the data block of invalid change, remove the data block of determined invalid change.
7. method as claimed in claim 6 is characterized in that,
Data block with the determined effective variation of data in magnetic disk block message mode buffer memory refers to:
The piece index point of the data block of the said effective variation of buffer memory, and data block inquiry, deletion are provided and add operation.
8. like claim 6 or 7 described methods, it is characterized in that this method also comprises:
When the data block of confirming current variation was the data block that effectively changes, also the data block with said effective variation sent to the backup server end, and after said backup server end backs up the data block of said effective variation, eliminated the data block that this effectively changes.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201110221536A CN102323908A (en) | 2011-08-03 | 2011-08-03 | Method and system for data caching in large volume data synchronization process on disk |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201110221536A CN102323908A (en) | 2011-08-03 | 2011-08-03 | Method and system for data caching in large volume data synchronization process on disk |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN102323908A true CN102323908A (en) | 2012-01-18 |
Family
ID=45451653
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201110221536A Pending CN102323908A (en) | 2011-08-03 | 2011-08-03 | Method and system for data caching in large volume data synchronization process on disk |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN102323908A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103369020A (en) * | 2012-03-27 | 2013-10-23 | Sk电信有限公司 | Cache synchronization system, cache synchronization method and apparatus thereof |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2004030090A (en) * | 2002-06-25 | 2004-01-29 | Hitachi Ltd | Cache memory management method |
| CN101178693A (en) * | 2007-12-14 | 2008-05-14 | 沈阳东软软件股份有限公司 | Data cache method and system |
| CN101789977A (en) * | 2010-02-08 | 2010-07-28 | 北京同有飞骥科技有限公司 | Teledata copying and de-emphasis method based on Hash coding |
-
2011
- 2011-08-03 CN CN201110221536A patent/CN102323908A/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2004030090A (en) * | 2002-06-25 | 2004-01-29 | Hitachi Ltd | Cache memory management method |
| CN101178693A (en) * | 2007-12-14 | 2008-05-14 | 沈阳东软软件股份有限公司 | Data cache method and system |
| CN101789977A (en) * | 2010-02-08 | 2010-07-28 | 北京同有飞骥科技有限公司 | Teledata copying and de-emphasis method based on Hash coding |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103369020A (en) * | 2012-03-27 | 2013-10-23 | Sk电信有限公司 | Cache synchronization system, cache synchronization method and apparatus thereof |
| CN103369020B (en) * | 2012-03-27 | 2017-03-01 | Sk电信有限公司 | Cache synchronization system, cache synchronization method and its device |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN103780638B (en) | Method of data synchronization and system | |
| CN106777225B (en) | Data migration method and system | |
| EP3680787B1 (en) | Method for synchronization between primary database and standby database, database system and device | |
| US20140089259A1 (en) | Operation method and apparatus for data storage system | |
| CN103138912B (en) | Method of data synchronization and system | |
| CN102752093B (en) | Based on the data processing method of distributed file system, equipment and system | |
| CN103246696A (en) | High-concurrency database access method and method applied to multi-server system | |
| CN103078933B (en) | A kind of method and apparatus determining data migration time | |
| CN104361030A (en) | Distributed cache architecture with task distribution function and cache method | |
| CN109710586B (en) | A kind of clustered node configuration file synchronous method and device | |
| EP2738712B1 (en) | Embedded multimedia card partitioned storage space adjustment method and terminal | |
| CN103713972A (en) | Incremental backup system and method | |
| CN107402870A (en) | The processing method and processing device of daily record section in a kind of meta data server | |
| CN108776690B (en) | Method for HDFS distributed and centralized mixed data storage system based on hierarchical governance | |
| CN103713973A (en) | Mixed storage backup method and system based on HDD and SSD | |
| CN1996305A (en) | Data saving and reading method and device therefore, and data transmission system | |
| CN109871295A (en) | A kind of data back up method, back-up device, electronic equipment and storage medium | |
| CN102231137A (en) | Data memory system and method | |
| CN104281673A (en) | Cache building system and method for database | |
| CN115146002A (en) | Cross-data-center data synchronization method and device | |
| CN109819013A (en) | A kind of block chain memory capacity optimization method based on cloud storage | |
| CN105760398A (en) | Log recording system and log record operating method | |
| CN110134541A (en) | Switching method, system, equipment and the storage medium of the configuration information of storage cluster | |
| CN102510390A (en) | Method and device for instructing data migration by hard disk temperature self-detection | |
| CN102323908A (en) | Method and system for data caching in large volume data synchronization process on disk |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C12 | Rejection of a patent application after its publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20120118 |