CN103150268A - Block-level data capture method in CDP (Continuous Data Protection) - Google Patents
Block-level data capture method in CDP (Continuous Data Protection) Download PDFInfo
- Publication number
- CN103150268A CN103150268A CN2013100667650A CN201310066765A CN103150268A CN 103150268 A CN103150268 A CN 103150268A CN 2013100667650 A CN2013100667650 A CN 2013100667650A CN 201310066765 A CN201310066765 A CN 201310066765A CN 103150268 A CN103150268 A CN 103150268A
- Authority
- CN
- China
- Prior art keywords
- cache
- data
- module
- user
- space
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Landscapes
- Memory System Of A Hierarchy Structure (AREA)
Abstract
The invention provides a block-level data capture method in CDP (Continuous Data Protection), which is characterized in that an inner-core space cache module and a user space cache module are respectively adopted in an inner-core cache space and a user cache space; two levels of cache mechanisms, including a static cache and a dynamic cache, and a two-level cache structure for data processing by using different cache modules are arranged in the user space cache module; under the premise of ensuring data reliability, as the user space cache module adopts the dynamic cache, the use of memory resources is reduced, the data capture performance and efficiency are improved, and the cache utilization ratio is improved; and through separating data capture and data transmission, and adopting the two-level cache method, including the inner-core space cache module and the user space cache module, read-write performance of a user is improved, expense of a memory device is lowered, and high efficiency and stability of a disaster recovery scheme are ensured.
Description
Technical field
The present invention relates to a kind of Computer Applied Technology field, relate in particular to the piece DBMS catching method in a kind of CDP.By the method that data capture and data transmission are separated from each other and the transmission policy that adopts the two-level cache of kernel state buffer memory and user's attitude buffer memory; can improve user's readwrite performance; reduce the expense of memory device; guarantee the reliability of data and the use that reduces memory source, a kind of method of good capture-data is provided for the CDP Data Protection Technologies.
Background technology
Disaster-tolerant backup field in present stage; the CDP technology is an important method of data protection; CDP can realize data protection by the mode of increment; its realization is mainly by catching or the variation of tracking data; data with these variations are placed on the place that is independent of outside production data simultaneously, can guarantee that like this user's data return to random time point in the past.The problem that the present invention is directed to the Trapped problems existence of piece DBMS in the CDP protection is analyzed and researched, and the efficient that how to improve data capture becomes point of penetration of the present invention and realizes a little.
In present stage, mainly there is following shortcoming in the implementation procedure for the data capture strategy of CDP piece DBMS protection:
1, traditional C DP realizes that the remote disaster tolerance scheme of data need to realize by the iSCSI agreement, and the shortcoming of this mode is in the management of client node realization to long-range CDP backup space, certain complicacy to be arranged; Simultaneously, the remote data transmission that they adopt and data capture are synchronous the execution, and this just involves transfer efficiency and the performance issue of data, and the piece DBMS remote transmission process in this mode can affect the performance of storage system;
2, traditional CDP data capture and transmission are all completed in the kernel spatial cache, because the situation of user's network and memory device is all Protean, this will cause the remote data transmission can greatly affect the performance of kernel, in the kernel spatial cache, limited address space is the size that can limit buffer memory simultaneously, so it is inadequate buffer memory to occur when high load capacity I/O, cause the situation of system kernel collapse.
Summary of the invention
The purpose of this invention is to provide the piece DBMS catching method in a kind of CDP.
the objective of the invention is to realize in the following manner, adopt respectively kernel spacing cache module and user's space cache module at kernel spatial cache and user cache space, be provided with again two kinds of other caching mechanisms of level of static cache and dynamic buffering in the user's space cache module, and use different cache modules to carry out the structure of the two-level cache of data processing, adopt dynamic buffering at the user's space cache module, reduce the use of memory source under the prerequisite that guarantees data reliability, performance and the efficient of data capture have been improved, improved the utilization factor of buffer memory, by data capture and data transmission being separated from each other and adopting the two-level cache method of kernel spacing cache module and user's space cache module, improve user's readwrite performance, reduce the expense of memory device, guarantee the high-effect and stable of Disaster Tolerant Scheme.The data capture step is as follows:
(1) user triggers write operation, send virtual I/O capture of labels module out in this patent to by the generic block layer, this I/O capture of labels module is by revising the member property of this data block, with the redirected equipment that can process this request of giving of the processing request of this module;
(2) be redirected write operation, at I/O capture of labels module definition a call back function, after the write operation that it hands down the upper strata is revised the attribute of the equipment of processing, request is redirected to real physical equipment, resubmits to the generic block mechanical floor and process;
(3) be redirected write operation, write operation by the generic block mechanical floor be redirected to revise the treatment facility property value the corresponding driver of physical equipment process;
(4) disk driver after being redirected is completed disk write operation;
(5) disk driver after being redirected returns to the flag information of I/O success or not to the upper strata;
(6) if step (5) report successfully, the data block of write operation is caught, and adds that metadata carries out the combination of data block;
(7) transmission of data blocks after making up is preserved to remote server, is used for the recovery of later data, and is failed skip.
In caching system, static cache is the storage space that distributes when task creation, static cache adopts the SSD hard disk as the buffer memory medium, buffer memory is the essential buffer memory of the normal operation of assurance task, size is selected and arranged to size by the user according to the service conditions of oneself, dynamic buffering is write or network speed reduces when causing static cache not enough by system's dynamic assignment in mass data burst, and when system is idle, the untapped dynamic buffering of administration module meeting the automatic recovery of system is saved memory source under the prerequisite that satisfies system's needs.The kernel spacing cache module is similar to register, in order to the data that temporary trapping module is caught, waits for being forwarded in user cache, just can discharge this part spatial cache after forwarding, then accept the new data of catching.
carry out catching of data under the generic block mechanical floor by the system kernel administration module, write operation requests to each system is processed, this module is a virtual unit, the I/O request can be done and catch processing, processing comprises lays the timestamp mark to the data of catching, record the address of those operations and the metadata informations such as size of write operation, then these information are preserved renewal, and then be redirected to physical storage devices and complete actual I/O operation, this module is equivalent to all data blocks are carried out the appointment of a treatment facility, reduced the busy property of processing of generic block mechanical floor with this, namely added the middleware of a virtual unit in the driver of generic block mechanical floor and physical equipment.
Between kernel spatial cache and user cache space, data interaction is divided into two parts, and a part is control information, uses netlink to carry out transmitted in both directions for the information of this part, reach administration module two spaces in fast not alternately; Another part is data message, and this part information is to the data copy in user cache space between the empty buffer memory of kernel. adopt mmap memory-mapped mechanism to realize that in conjunction with memcpy data copy mode data copy between kernel spatial cache and user cache spatial cache is to improve user data transmission speed and the mutual reliability of raising for this part information.
The invention has the beneficial effects as follows: catch and data transmission method by efficient piece DBMS, data capture and data transmission are separated from each other and adopt the two-level cache strategy of kernel state buffer memory and user's attitude buffer memory, can improve user's readwrite performance, reduce the expense of memory device, guarantee the high-effect and stable of Disaster Tolerant Scheme.
Description of drawings
Fig. 1 is software architecture implementation structure figure of the present invention;
Fig. 2 is the process flow diagram of data capture of the present invention, storage, transmission;
Fig. 3 is the realization flow figure of caching mechanism in the present invention.
Embodiment
With reference to accompanying drawing, the below will be described in further detail embodiment of the present invention.
of the present inventionly designed a kind of efficient method about catching of piece DBMS in CDP, to carry out catching of data under the generic block mechanical floor by the system kernel administration module, write operation requests to each system is processed, this module is a virtual unit, the I/O request can be done and catch processing, processing comprises lays the timestamp mark to the data of catching, record the address of those operations and the metadata informations such as size of write operation, then these information are preserved renewal, and then be redirected to physical storage devices and complete actual I/O operation, this module is equivalent to all data blocks are carried out the appointment of a treatment facility, this has just reduced the processing of generic block mechanical floor, reduced its busy property.This method has namely been added the middleware of a virtual unit in the driver of generic block mechanical floor and physical equipment.
In the caching system of native system, be mainly to have used kernel spacing cache module and user's space cache module.Be provided with again two kinds of other caching mechanisms of level of static cache and dynamic buffering in the user's space cache module, static cache is the storage space that distributes when task creation, this static cache we adopt the SSD hard disk as the buffer memory medium, this buffer memory is the normal operation of assurance task, be a substantially essential buffer memory, size has the user to select and arrange size according to the service conditions of oneself.Dynamic buffering is write or network speed reduces when causing static cache not enough by system's dynamic assignment in mass data burst, and when system is idle, the untapped dynamic buffering of administration module meeting the automatic recovery of system is saved memory source under the prerequisite that satisfies system's needs.The kernel spacing cache module is similar to register, and we keep in it the data that trapping module is caught, and waits for being forwarded in user cache, just can discharge this part spatial cache.Accept the new data of catching.
Main data capture step is as follows:
(1) user triggers write operation, send virtual I/O capture of labels module out in this patent to by the generic block layer, this I/O capture of labels module is by revising the member property of this data block, with the redirected equipment that can process this request of giving of the processing request of this module;
(2) be redirected write operation, at I/O capture of labels module definition a call back function, after the write operation that it hands down the upper strata is revised the attribute of the equipment of processing, request is redirected to real physical equipment, resubmits to the generic block mechanical floor and process;
(3) be redirected write operation, write operation by the generic block mechanical floor be redirected to revise the treatment facility property value the corresponding driver of physical equipment process;
(4) disk driver after being redirected is completed disk write operation;
(5) disk driver after being redirected returns to the flag information of I/O success or not to the upper strata;
(6) if step 5 report successfully, the data block of write operation is caught, and adds that metadata carries out the combination of data block;
(7) transmission of data blocks after making up is preserved to remote server, is used for the recovery of later data, and is failed skip.
In the caching system of native system, be mainly to have used kernel spacing cache module and user's space cache module.Be provided with again two kinds of other caching mechanisms of level of static cache and dynamic buffering in the user's space cache module, static cache is the storage space that distributes when task creation, this static cache adopts the SSD hard disk as the buffer memory medium, this buffer memory is the normal operation of assurance task, be a substantially essential buffer memory, size has the user to select and arrange size according to the service conditions of oneself.Dynamic buffering is write or network speed reduces when causing static cache not enough by system's dynamic assignment in mass data burst, and when system is idle, the untapped dynamic buffering of administration module meeting the automatic recovery of system is saved memory source under the prerequisite that satisfies system's needs.The kernel spacing cache module is similar to register, keeps in it the data that trapping module is caught, and waits for being forwarded in user cache, just can discharge this part spatial cache.Accept the new data of catching.
In native system, between kernel spatial cache and user cache space, data interaction is divided into two parts, and a part is control information, uses netlink to carry out transmitted in both directions for this part information the present invention, reaches the mutual fast of administration module in two spaces; Another part is data message, and this part is being mainly that the kernel spatial cache is to the data copy in user cache space. adopt mmap memory-mapped mechanism to realize data copy between kernel spatial cache and user cache spatial cache in conjunction with memcpy data copy mode in for this part information the present invention.
Except the described technical characterictic of instructions, be the known technology of those skilled in the art.
Claims (4)
1. the piece DBMS catching method in a CDP, it is characterized in that adopting respectively kernel spacing cache module and user's space cache module at kernel spatial cache and user cache space, be provided with again two kinds of other caching mechanisms of level of static cache and dynamic buffering in the user's space cache module, and use different cache modules to carry out the structure of the two-level cache of data processing, adopt dynamic buffering at the user's space cache module, reduce the use of memory source under the prerequisite that guarantees data reliability, performance and the efficient of data capture have been improved, improved the utilization factor of buffer memory, by data capture and data transmission being separated from each other and adopting the two-level cache method of kernel spacing cache module and user's space cache module, improve user's readwrite performance, reduce the expense of memory device, guarantee the high-effect and stable of Disaster Tolerant Scheme, the data capture step is as follows:
(1) user triggers write operation, send virtual I/O capture of labels module out in this patent to by the generic block layer, this I/O capture of labels module is by revising the member property of this data block, with the redirected equipment that can process this request of giving of the processing request of this module;
(2) be redirected write operation, at I/O capture of labels module definition a call back function, after the write operation that it hands down the upper strata is revised the attribute of the equipment of processing, request is redirected to real physical equipment, resubmits to the generic block mechanical floor and process;
(3) be redirected write operation, write operation by the generic block mechanical floor be redirected to revise the treatment facility property value the corresponding driver of physical equipment process;
(4) disk driver after being redirected is completed disk write operation;
(5) disk driver after being redirected returns to the flag information of I/O success or not to the upper strata;
(6) if step (5) report successfully, the data block of write operation is caught, and adds that metadata carries out the combination of data block;
(7) transmission of data blocks after making up is preserved to remote server, is used for the recovery of later data, and is failed skip.
2. method according to claim 1, it is characterized in that in caching system, static cache is the storage space that distributes when task creation, static cache adopts the SSD hard disk as the buffer memory medium, buffer memory is the essential buffer memory of the normal operation of assurance task, size is selected and arranged to size by the user according to the service conditions of oneself, dynamic buffering is write or network speed reduces when causing static cache not enough by system's dynamic assignment in mass data burst, and when system is idle, the untapped dynamic buffering of administration module meeting the automatic recovery of system, save memory source under the prerequisite that satisfies system's needs, the kernel spacing cache module is similar to register, the data of catching in order to keep in trapping module, wait is forwarded in user cache, just can discharge this part spatial cache after forwarding, accept again the new data of catching.
3. method according to claim 1, it is characterized in that carrying out under the generic block mechanical floor by the system kernel administration module catching of data, write operation requests to each system is processed, this module is a virtual unit, the I/O request can be done and catch processing, processing comprises lays the timestamp mark to the data of catching, record the address of those operations and the metadata informations such as size of write operation, then these information are preserved renewal, and then be redirected to physical storage devices and complete actual I/O operation, this module is equivalent to all data blocks are carried out the appointment of a treatment facility, reduced the busy property of processing of generic block mechanical floor with this, namely added the middleware of a virtual unit in the driver of generic block mechanical floor and physical equipment.
4. method according to claim 1, it is characterized in that: between kernel spatial cache and user cache space, data interaction is divided into two parts, a part is control information, uses netlink to carry out transmitted in both directions for the information of this part, reach administration module two spaces in fast not alternately; Another part is data message, and this part information is to the data copy in user cache space between the empty buffer memory of kernel. adopt mmap memory-mapped mechanism to realize that in conjunction with memcpy data copy mode data copy between kernel spatial cache and user cache spatial cache is to improve user data transmission speed and the mutual reliability of raising for this part information.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN2013100667650A CN103150268A (en) | 2013-03-04 | 2013-03-04 | Block-level data capture method in CDP (Continuous Data Protection) |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN2013100667650A CN103150268A (en) | 2013-03-04 | 2013-03-04 | Block-level data capture method in CDP (Continuous Data Protection) |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN103150268A true CN103150268A (en) | 2013-06-12 |
Family
ID=48548361
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN2013100667650A Pending CN103150268A (en) | 2013-03-04 | 2013-03-04 | Block-level data capture method in CDP (Continuous Data Protection) |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN103150268A (en) |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103916316A (en) * | 2014-04-11 | 2014-07-09 | 国家计算机网络与信息安全管理中心 | Linear speed capturing method of network data packages |
| CN108834086A (en) * | 2018-06-25 | 2018-11-16 | 平安科技（深圳）有限公司 | Method, apparatus, computer equipment and the storage medium that short message is sent |
| CN111897748A (en) * | 2019-05-05 | 2020-11-06 | 北京兆易创新科技股份有限公司 | Mapping relation storage method, reading method, device, equipment and medium |
| CN112003992A (en) * | 2020-08-14 | 2020-11-27 | 迅镭智能（广州）科技有限公司 | Transmission system and method based on scanning gun |
| CN112791413A (en) * | 2021-02-04 | 2021-05-14 | 网易（杭州）网络有限公司 | Game prop data processing method and device, processor and electronic device |
| CN114390098A (en) * | 2020-10-21 | 2022-04-22 | 北京金山云网络技术有限公司 | Data transmission method and device, electronic equipment and storage medium |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20040117560A1 (en) * | 2002-12-12 | 2004-06-17 | International Business Machines Corporation | Updating remote locked cache |
| CN101286127A (en) * | 2008-05-08 | 2008-10-15 | 华中科技大学 | Multi-fork diary memory continuous data protecting and restoration method |
-
2013
- 2013-03-04 CN CN2013100667650A patent/CN103150268A/en active Pending
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20040117560A1 (en) * | 2002-12-12 | 2004-06-17 | International Business Machines Corporation | Updating remote locked cache |
| CN101286127A (en) * | 2008-05-08 | 2008-10-15 | 华中科技大学 | Multi-fork diary memory continuous data protecting and restoration method |
Non-Patent Citations (1)
| Title |
|---|
| 李巍等: "一种基于块级的连续数据捕获方法研究", 《计算机研究与发展》 * |
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103916316A (en) * | 2014-04-11 | 2014-07-09 | 国家计算机网络与信息安全管理中心 | Linear speed capturing method of network data packages |
| CN108834086A (en) * | 2018-06-25 | 2018-11-16 | 平安科技（深圳）有限公司 | Method, apparatus, computer equipment and the storage medium that short message is sent |
| CN108834086B (en) * | 2018-06-25 | 2021-05-11 | 平安科技（深圳）有限公司 | Method and device for sending short message, computer equipment and storage medium |
| CN111897748A (en) * | 2019-05-05 | 2020-11-06 | 北京兆易创新科技股份有限公司 | Mapping relation storage method, reading method, device, equipment and medium |
| CN112003992A (en) * | 2020-08-14 | 2020-11-27 | 迅镭智能（广州）科技有限公司 | Transmission system and method based on scanning gun |
| CN114390098A (en) * | 2020-10-21 | 2022-04-22 | 北京金山云网络技术有限公司 | Data transmission method and device, electronic equipment and storage medium |
| CN112791413A (en) * | 2021-02-04 | 2021-05-14 | 网易（杭州）网络有限公司 | Game prop data processing method and device, processor and electronic device |
| CN112791413B (en) * | 2021-02-04 | 2024-02-23 | 网易（杭州）网络有限公司 | Game prop data processing method and device, processor and electronic device |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| JP6522812B2 (en) | Fast Crash Recovery for Distributed Database Systems | |
| US10956601B2 (en) | Fully managed account level blob data encryption in a distributed storage environment | |
| US10764045B2 (en) | Encrypting object index in a distributed storage environment | |
| TWI737395B (en) | Log-structured storage systems and method | |
| CN103268318B (en) | A kind of distributed key value database system of strong consistency and reading/writing method thereof | |
| US9378088B1 (en) | Method and system for reclamation of distributed dynamically generated erasure groups for data migration between high performance computing architectures and data storage using non-deterministic data addressing | |
| US10659225B2 (en) | Encrypting existing live unencrypted data using age-based garbage collection | |
| US8161321B2 (en) | Virtual machine-based on-demand parallel disaster recovery system and the method thereof | |
| US10810123B1 (en) | Flush strategy for using DRAM as cache media system and method | |
| CN103150268A (en) | Block-level data capture method in CDP (Continuous Data Protection) | |
| CN102868727B (en) | Method for realizing high availability of logical volume | |
| WO2019001521A1 (en) | Data storage method, storage device, client and system | |
| CN107832423B (en) | File reading and writing method for distributed file system | |
| CN103516549B (en) | A kind of file system metadata log mechanism based on shared object storage | |
| CN102662795A (en) | Metadata fault-tolerant recovery method in distributed storage system | |
| US10592165B1 (en) | Method, apparatus and computer program product for queueing I/O requests on mapped RAID | |
| US10235082B1 (en) | System and method for improving extent pool I/O performance by introducing disk level credits on mapped RAID | |
| CN106657356A (en) | Data writing method and device for cloud storage system, and cloud storage system | |
| CN104902009A (en) | Erasable encoding and chained type backup-based distributed storage system | |
| CN106686095A (en) | Data storage method and device based on erasure code technology | |
| CN103916459A (en) | Big data filing and storing system | |
| CN105516284A (en) | Clustered database distributed storage method and device | |
| CN102982182A (en) | Data storage planning method and device | |
| CN103297485A (en) | Distributed cache automatic management system and distributed cache automatic management method | |
| CN106325974A (en) | Virtualization IO performance optimization method and system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WD01 | Invention patent application deemed withdrawn after publication | ||
| WD01 | Invention patent application deemed withdrawn after publication |
Application publication date: 20130612 |