CN109614054B

CN109614054B - data reading method and system

Info

Publication number: CN109614054B
Application number: CN201811563362.6A
Authority: CN
Inventors: 郭勇; 陈鹏; 王子骏
Original assignee: GUANGZHOU DINGJIA COMPUTER TECHNOLOGY Co Ltd
Current assignee: GUANGZHOU DINGJIA COMPUTER TECHNOLOGY Co Ltd
Priority date: 2018-12-20
Filing date: 2018-12-20
Publication date: 2019-12-13
Anticipated expiration: 2038-12-20
Also published as: CN109614054A

Abstract

The invention relates to a data reading method and a data reading system, and belongs to the technical field of networks. The method comprises the following steps: obtaining incremental metadata; the incremental metadata is used for describing incremental data; acquiring a consistency mark of target data; the target data is incremental data described by the incremental metadata, and the consistency mark is used for describing the consistency of the target data and the incremental data written in a disk; and when the consistency of the target data and the incremental data written in the disk is determined according to the consistency mark, reading the corresponding incremental data from the disk. By the technical scheme, the problem that reading of the incremental data is wrong is solved. And determining whether the incremental data to be read is consistent with the incremental data written in the disk or not according to the consistency mark, and further reading the corresponding incremental data from the disk, so that the target backup machine can read the correct incremental data from the disk.

Description

Data reading method and system

Technical Field

The present invention relates to the field of network technologies, and in particular, to a method and a system for reading data, a computer device, and a storage medium.

background

Continuous Data Protection (CDP) is a real-time backup technique based on disk block devices. The backup target machine can package an IO (Input/Output) request into a monitoring record and send the monitoring record to the backup server. Unlike the timed backup method, the CDP backup technique has a protection granularity of IO level or one millionth of a second level for data. In addition, when a failure or disaster occurs, the user can precisely select a time Point for data Recovery, and data loss (RPO) can be reduced to the maximum extent.

In the process of implementing the invention, the inventor finds that at least the following problems exist in the prior art: for the CDP backup technology which adopts an asynchronous mode to read and send incremental data, the backup technology firstly constructs monitoring records and submits the monitoring records to a local disk, and then extracts the monitoring records from the disk and sends the monitoring records to a backup server for backup. Multiple IO writes may occur at the same location on the disk before being synchronized to the backup server, thereby causing the incremental data to be read or written to be in an inconsistent state with the actual incremental data in the disk. This may result in errors in the reading of the incremental data.

disclosure of Invention

Based on this, the embodiment of the invention provides a data reading method, a data reading system, computer equipment and a storage medium, which can improve the reading accuracy of incremental data.

The content of the embodiment of the invention is as follows:

a method for reading data, comprising the steps of: obtaining incremental metadata; the incremental metadata is used for describing incremental data; acquiring a consistency mark of target data; the target data is incremental data described by the incremental metadata, and the consistency mark is used for describing the consistency of the target data and the incremental data written in a disk; and when the consistency of the target data and the incremental data written in the disk is determined according to the consistency mark, reading the corresponding incremental data from the disk.

in one embodiment, after the step of obtaining the incremental metadata, the method further includes: adding the incremental metadata to an incremental record queue located in a memory; when it is determined that the target data is consistent with the incremental data written in the disk according to the consistency mark, reading the corresponding incremental data from the disk, including: when the consistency mark determines that the target data is consistent with the increment data written in the disk, extracting the increment metadata from the increment recording queue; and reading corresponding incremental data from the disk according to the extracted incremental metadata.

In one embodiment, the delta metadata includes: disk offset and data length; before the step of adding the incremental metadata to the incremental record queue in the memory, the method further includes: writing the incremental metadata into the sector position corresponding to the variable sector bitmap; the corresponding sector bit is determined according to the disk offset and the data length; when detecting that the changed sector bitmap has an overlapped sector position, adding newly written incremental metadata corresponding to the overlapped sector position into a conflict record queue; and the overlapped sector position is a sector position in which newly written increment metadata in the change sector bitmap is overlapped with the historical increment metadata.

in one embodiment, after the step of reading the corresponding incremental data from the disk, the method further includes: clearing the overlapping sector area and corresponding increment metadata in the conflict record queue; traversing conflict increment metadata in a conflict record queue, and judging whether an overlapped sector position corresponding to the conflict increment metadata exists in the change sector bitmap; when not present, determining that the conflicting delta metadata is read ahead; adding the conflicting delta metadata to an inconsistent record queue; wherein the inconsistent record queue is used for consistency checking before reading the incremental data from the disk.

In one embodiment, the step of reading corresponding delta data from the disk according to the extracted delta metadata includes: when the inconsistent record queue is empty, judging that the consistency mark is consistent; reading corresponding incremental data from a disk according to the extracted incremental metadata; when the inconsistent recording queue is not empty, judging that the consistency mark is inconsistent; emptying the inconsistent record queue and updating the consistency mark to be consistent; and reading corresponding incremental data from the disk according to the extracted incremental metadata.

In one embodiment, when the inconsistent recording queue is empty, before the step of reading corresponding incremental data from the disk according to the extracted incremental metadata, the method further includes: and updating the consistency time according to the current time.

In one embodiment, when the inconsistent record queue is not empty, the step of emptying the inconsistent record queue includes: and determining the time difference between the current time and the consistency time, and emptying the inconsistent record queue when the time difference is greater than a preset time threshold.

in one embodiment, before the step of obtaining the consistency flag of the target data, the method further includes: receiving a write IO request; the write IO request comprises disk offset; when the consistency marks are consistent, determining a write address of a disk according to disk offset in the write IO request, and writing corresponding incremental data into the write address; when the consistency marks are inconsistent, judging whether a sleep ending instruction is received or not; if so, determining a write address of the disk according to the disk offset in the write IO request, and writing corresponding incremental data into the write address.

Correspondingly, an embodiment of the present invention provides a data reading system, including: the filtering driving module is used for acquiring incremental metadata; the incremental metadata is used for describing incremental data; the mark acquisition module is used for acquiring a consistency mark of the target data; the target data is incremental data described by the incremental metadata, and the consistency mark is used for describing the consistency of the target data and the incremental data written in a disk; and the reading module is used for reading corresponding incremental data from the disk when the consistency mark determines that the target data is consistent with the incremental data written in the disk.

According to the data reading method and system, whether the incremental data to be read is consistent with the incremental data written in the disk or not is determined according to the consistency marks, and then the corresponding incremental data is read from the disk, so that the target backup machine can read the correct incremental data from the disk.

A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the following steps when executing the computer program: obtaining incremental metadata; the incremental metadata is used for describing incremental data; acquiring a consistency mark of target data; the target data is incremental data described by the incremental metadata, and the consistency mark is used for describing the consistency of the target data and the incremental data written in a disk; and when the consistency of the target data and the incremental data written in the disk is determined according to the consistency mark, reading the corresponding incremental data from the disk.

According to the computer equipment, whether the incremental data needing to be read is consistent with the incremental data written in the disk or not is determined according to the consistency marks, and then the corresponding incremental data is read from the disk, so that the target backup computer can read the correct incremental data from the disk.

A computer-readable storage medium, on which a computer program is stored which, when executed by a processor, carries out the steps of: obtaining incremental metadata; the incremental metadata is used for describing incremental data; acquiring a consistency mark of target data; the target data is incremental data described by the incremental metadata, and the consistency mark is used for describing the consistency of the target data and the incremental data written in a disk; and when the consistency of the target data and the incremental data written in the disk is determined according to the consistency mark, reading the corresponding incremental data from the disk.

The computer-readable storage medium determines whether the incremental data to be read is consistent with the incremental data written in the disk according to the consistency mark, and then reads the corresponding incremental data from the disk, so that the target backup machine can read the correct incremental data from the disk.

Drawings

FIG. 1 is a diagram of an exemplary embodiment of a data reading method;

FIG. 2 is a flow diagram illustrating a method for continuous data protection without local cache in one embodiment;

FIG. 3 is a schematic diagram of the operation of a filter driver in one embodiment;

FIG. 4 is a schematic diagram illustrating the principle of formation of a conflict record queue and an inconsistent record queue in one embodiment;

FIG. 5 is a flow diagram illustrating a method for reading incremental data according to one embodiment;

FIG. 6 is a flowchart illustrating a method for reading data according to an embodiment;

FIG. 7 is a flow chart illustrating a method for continuous data protection without local cache in another embodiment;

FIG. 8 is a schematic diagram illustrating incremental record conflicts and data consistency point checking in one embodiment;

FIG. 9 is a schematic diagram illustrating incremental recording continuity in one embodiment;

FIG. 10 is a block diagram of a continuous data protection system without a local cache, in one embodiment;

FIG. 11 is a block diagram of a system for reading incremental data in one embodiment;

FIG. 12 is a block diagram showing a configuration of a system for reading data in one embodiment;

FIG. 13 is a block diagram of a continuous data protection system without a local cache in another embodiment;

FIG. 14 shows an internal structure of a computer device according to an embodiment.

Detailed Description

In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

The continuous data protection method without the local cache provided by the application can be applied to the application environment shown in fig. 1. The application environment includes a backup target machine 101 and a backup server 102, and the backup target machine 101 and the backup server 102 can communicate via a network. The target backup machine 101 refers to a device that needs to perform data backup, and the backup server 102 is a server that performs backup of data on the target backup machine 101. The backup target machine 101 acquires the incremental metadata, reads the incremental data from the disk according to the incremental metadata, and sends the incremental data to the backup server 102 for backup. The backup target machine 101 may be a terminal, a PC (personal computer), a server, or the like, in which various data are stored. The servers, including the backup server 102, may be implemented as individual servers or as a cluster of servers. Of course, the backup of the data of the target backup machine 101 may be implemented by other hardware devices or virtual devices having a backup function, besides the backup server.

The embodiment of the invention provides a continuous data protection method without local cache, a system, computer equipment and a storage medium. The following are detailed below.

In one embodiment, as shown in FIG. 2, a method of continuous data protection without local caching is provided. Taking the application of the method to the target backup machine side in fig. 1 as an example for explanation, the method includes the following steps:

S201, acquiring a data backup request; obtaining incremental metadata according to the data backup request; wherein the delta metadata is used to describe delta data.

in an application environment of continuous data protection (hereinafter referred to as CDP), the data backup request may be triggered according to the write IO request. When the target backup machine intercepts the write IO request of the upper layer, it indicates that the data on the target backup machine will be written and the data in the disk will change, and it is necessary to backup the changed data to the backup server. The target backup machine may therefore translate the write IO request into a data backup request for the corresponding incremental data to trigger continuous data protection.

The CDP has an initialization process, in which the data of the monitoring disk is completely synchronized to the backup server, and after the initialization is completed, the real-time monitoring process is entered, and each write IO request response in the real-time monitoring process is an increment compared with the initialization synchronization, so that the data written in the target backup machine can be referred to as incremental data. Further, the write IO request may include description information of the incremental data, such as: disk offset, data length, time stamp, etc., which may be referred to as metadata. It should be noted that there may be multiple disks on the backup target machine, the target backup machine may only intercept a write IO request for a certain disk or certain blocks of disks, and these intercepted disks become the monitoring disks, that is, the monitoring disks refer to the disks that need to perform data backup.

The embodiment of the invention describes the incremental data through the incremental metadata (which may also be referred to as incremental metadata record or incremental record), and the incremental metadata may include metadata information such as disk offset, data length, and time stamp corresponding to the incremental data. It should be noted that the delta metadata is used to describe the delta data, and may not include the delta data itself.

Furthermore, the data backup request can be obtained through a kernel filter driver, the kernel filter driver intercepts and captures a write IO request sent by an upper layer in real time, converts the write IO request into a data backup request for corresponding incremental data, and generates corresponding incremental metadata according to disk offset, data length, time stamp and the like in the intercepted data backup request.

And S202, reading corresponding incremental data from the disk according to the incremental metadata.

The incremental data can be stored in a monitoring disk of the disk, and the incremental data can be read out from the disk when data backup is needed. The reading of the delta data and delta metadata may be accomplished by a kernel read routine. I.e., data read work in the target backup machine, may be performed by a kernel read routine.

S203, sending the read incremental data to a backup server to trigger the backup server to backup the incremental data.

The target backup machine reads and writes data in the monitoring disc according to the write IO request, and simultaneously sends corresponding incremental data to the backup server in real time to realize continuous data protection, so that time points can be accurately selected for data recovery, and data loss can be reduced to the maximum extent.

wherein the sending of the incremental data may be implemented by a kernel send routine; the kernel reading routine submits the read incremental data to a kernel sending routine, and the kernel sending routine sends the received incremental data to the backup server.

Continuous data protection can be divided into a synchronous mode and an asynchronous mode based on the disk-dropping time of the locally intercepted write IO request. In this embodiment, the kernel read routine and the kernel send routine may use an asynchronous mode for incremental data read and send.

The basic steps of the synchronization mode are as follows: 1. the upper layer submits a write IO request. 2. The kernel filter driver intercepts write IO requests. 3. The kernel send routine sends the monitoring record to the backup server. 4. And the kernel filtering driver submits the write IO request to a disk lower layer driver for data writing. The advantage of the synchronous mode is that no local caching is required. However, since the data corresponding to each write IO request is sent to the backup server before being submitted to the local disk, the IO performance of the local disk is seriously degraded, and the time-consuming overhead of each write IO request is increased by more than 2 times compared with the time-consuming overhead of the unfiltered drive interference. In addition, the synchronization mode highly depends on the network bandwidth between the backup target machine and the backup server, so the mode cannot be applied to disaster recovery in different places generally and is only applicable to scenes such as local backup, same-city backup and high availability; if the system is used for disaster recovery in different places, a private network must be erected.

As shown in fig. 3, the basic steps of the asynchronous mode are: and S1, submitting the write IO request by the upper layer. S2, the kernel filter driver intercepts the write IO request and constructs a monitoring record (the monitoring record comprises { offset, length, timestamp } + data, namely disk offset, data length, timestamp and data). And S3, submitting the monitoring record to a local cache disk by the kernel filter driver. And S4, submitting the write IO request to the disk lower layer driver by the kernel filter driver. And S5, the kernel reading routine extracts the incremental data from the local cache disk and sends the incremental data to the kernel sending routine. And S6, the kernel sending routine sends the incremental data to the backup server. The advantages of the asynchronous mode are: the influence on the IO performance of the monitoring disk is relatively small, the time consumption expense of each IO writing request is increased by about 1 time, and the time consumption expense is less than that of a synchronous mode; and the asynchronous mode smoothes the difference between the network bandwidth and the disk speed through the local cache disk, so the asynchronous mode can be used for remote data backup, can tolerate network delay to a greater extent and can be applied to remote disaster recovery. However, since the asynchronous mode needs to store data, an additional local cache disk is needed, and if there is no additional cache disk on the backup target machine, a continuous data protection scheme cannot be implemented or the cache disk needs to be provided by means of external storage such as FC, ISCSI, and the like. In addition, if the capacity of the cache disk is too small or the network bandwidth between the backup target machine and the backup server is far lower than the average data writing speed of the monitoring disk, the cache disk is easy to overflow.

in view of the above problems, the method for protecting continuous data without local cache provided in this embodiment has a small space occupied by incremental metadata, and the memory of the target backup machine is sufficient for storage. Compared with the traditional continuous data protection method based on local cache, the method does not need an additional disk as a cache disk, does not need external storage equipment such as FC (fiber channel), ISCSI (Internet Small computer System interface) and the like, and can implement continuous data protection on a backup target machine and a whole machine. In addition, compared with the continuous data protection in a synchronous mode, the method can tolerate a low-bandwidth network environment and support remote data disaster tolerance.

In one embodiment, after the step of obtaining the incremental metadata according to the data backup request, the method further includes: adding the incremental metadata to an incremental record queue located in a memory; the step of reading corresponding incremental data from the disk according to the incremental metadata includes: extracting the incremental metadata from the incremental record queue; and reading corresponding incremental data from the disk according to the extracted incremental metadata.

the increment record queue is used for recording the increment metadata and ensuring that the kernel reading routine reads the increment metadata in the increment record queue in sequence. And the kernel reading routine can read the increment metadata according to the sequence of adding the increment metadata to the increment record queue.

Because the memory resources are limited, if the data corresponding to the IO request is cached in the memory, the memory is easily overflowed. Thus, conventional CDP schemes require additional disks as cache disks for caching the monitored data.

In this embodiment, the incremental metadata is stored in an incremental record queue, and the incremental record queue is located in the memory of the target disk backup machine. The space occupied by the increment metadata is small (32 bytes of metadata information can represent 4K-512K data, and the space occupied is only 1/16384-1/128 of the traditional scheme), so that a sufficient amount of increment metadata can be cached in the memory. Therefore, the continuous data protection method provided by the embodiment of the invention can realize continuous data protection without local cache, and can effectively improve the convenience of continuous data protection.

In one embodiment, the incremental metadata is obtained according to the disk offset and the data length in the data backup request; before the step of adding the incremental metadata to the incremental record queue in the memory, the method further includes: writing the incremental metadata into the sector position corresponding to the variable sector bitmap; the corresponding sector bit is determined according to the disk offset and the data length; when detecting that the changed sector bitmap has an overlapped sector position, adding newly written incremental metadata corresponding to the overlapped sector position into a conflict record queue; and the overlapped sector position is a sector position overlapped by newly written increment metadata and historical increment metadata in the change sector bitmap.

the change sector bitmap is a bitmap which completely maps a monitor disk, and each bit (bit) of the bitmap represents a sector of the monitor disk. In an initial state, each bit of the bitmap is 0; when the sector bitmap is set, the start bit and the end bit of the bitmap can be located according to the disk offset and the length corresponding to the incremental metadata, and then the bit interval is set to be 1. Before setting to 1, it is also checked whether any bit in the bit section is already 1, if all bits are 1, it indicates that complete overlap occurs, and if some bits are 1, it indicates that partial overlap occurs. The complete overlap is mainly used for duplicate sector deduplication, and the embodiment of the invention mainly relates to the case of partial overlap. These overlapped sectors are the above-mentioned overlapped sectors, and refer to the same sector that is written many times.

the following describes the process of adding incremental metadata in the conflict record queue, according to the case in fig. 4 as an example: incremental metadata record (i.e., incremental metadata, 401 in FIG. 4) R2 is written on S1 of sector 402 (as dashed line with arrow in FIG. 4), after which incremental record (i.e., incremental metadata) R5 is also written on sector S1; delta record R4 was written to sector S6, after which delta record R6 was also written to sector S6; delta record R8 was written to sector S10, after which delta record R9 was also written to sector S10. That is, the sector area written by the subsequently generated delta record contains the sector written by the earlier delta record, i.e., sector overlap occurs (there is an overlapping sector area), when IO conflicts exist between delta records R5, R6, R9 and other delta records, wherein delta record R5 conflicts with delta record R2, delta record R6 conflicts with delta record R4, and delta record R9 conflicts with delta record R8. Delta records R4 and R8 that conflict with delta records R6 and R9 have not yet been read, at which point delta records R6 and R9 are in a conflict state and the corresponding delta metadata records are added to conflict record queue 403.

After constructing the delta metadata, the kernel filter driver may check whether the constructed delta metadata has a sector overlap with a previous earlier delta metadata record and then add the delta metadata record to the delta record queue. If there is overlap, the incremental metadata record is simultaneously added to the conflict record queue, indicating entry into a conflict state.

because the kernel reading routine and the kernel sending routine are used for reading and sending the incremental data in an asynchronous mode, multiple IO (input/output) writing and reading may occur at the same position on the disk before the incremental data is synchronized to the backup server, so that the incremental data corresponding to the incremental record is in an inconsistent state. In the embodiment, a change sector bitmap is introduced to store incremental metadata, which can indicate whether data at a certain address in a monitoring disc is written and read for multiple times, and indicate the position where the data conflicts with the data by overlapping sector bits. The kernel reading routine can know that the corresponding address is written for many times by overlapping the sector areas, and the later written data may cover the earlier written data, so that the data can not be read from the corresponding address or the read data is marked as conflict data or error data, and the accuracy of the read incremental data can be effectively ensured. In addition, such an operation mode can detect an inconsistent state of incremental data corresponding to the incremental record, which may affect the reading of disk data and possible data recovery.

In one embodiment, after the step of reading the corresponding delta data from the disk according to the delta metadata, the method further includes: clearing the overlapping sector area and corresponding increment metadata in the conflict record queue; traversing conflict increment metadata in a conflict record queue, and judging whether an overlapped sector position corresponding to the conflict increment metadata exists in the change sector bitmap; when not present, determining that the conflicting delta metadata is read ahead; adding the conflicting delta metadata to an inconsistent record queue; wherein the inconsistent record queue is used for consistency check before reading the incremental data from the disk; the consistency check includes determining consistency of the delta data that needs to be read with the delta data written in the disk.

The conflict of the increment metadata does not represent that the increment data is in an inconsistent state, and the conflict of the increment metadata is converted into the inconsistent state from the conflicting state only when the increment metadata in the sending overlapped sectors is read in advance. Taking the example according to the previous embodiment as an example, as shown in fig. 4, the incremental record R5 and the incremental record R2 conflict, when the kernel read routine reads the sector S1 according to the incremental record R2, the data contained in the sector S1 is actually written by the incremental record R5, that is, the data written by the incremental record R5 is read in advance, and the data corresponding to the incremental record R3 should be synchronized to the backup data in preference to the data of the incremental record R5, so that the incremental record R5 changes from the conflict state to the inconsistent state. Corresponding incremental metadata needs to be added to inconsistent record queue 404. When the inconsistent queue is empty, i.e., there is no inconsistent delta metadata, the backup data is in a disk level consistent state.

In addition, the write change sector bitmap may be incremental metadata or incremental data corresponding to the incremental metadata. Further, when two delta records are written to the same disk sector, e.g., delta record R1 writes sector S100 to data AAAA, delta record R20 writes sector S100 to data BBBB before the incremental record R1 is read. At this point in time the delta record R1 and the delta record R20 collide, and the read routine will also process the delta record R1 first, since the delta record R1 was written first. When the read routine reads data according to R1, it is the data written by R20 that is actually read, so the data of R20 is read in advance, so R20 transitions into the inconsistent record queue until R20 is read, and the data is restored to a consistent state.

After finishing continuous protection of data, if data recovery is needed; since the backed up incremental data does not correspond to the incremental metadata, the backed up data recorded as inconsistent may not be used for data recovery. For inconsistent backup data, the inconsistent backup data can be retrieved in other manners, for example, data in the inconsistent record queue is independently stored in a certain address interval of the backup server, and recovery of the inconsistent backup data can be achieved by reading the data in the address interval. The processing mode can ensure the accuracy of the recovered data.

In this embodiment, by setting the inconsistent recording queue to record the data read in advance, it can be determined whether the data in the disk is read in advance when the kernel reading routine performs data reading, that is, whether the data to be read is consistent with the data stored in the disk. The read-write state of the data in the disk can be acquired in real time, and when the incremental data is read in advance, the recovery of the data is controlled through the incremental metadata in the inconsistent recording queue, so that the accuracy of the recovered data can be ensured.

In one embodiment, the data backup request comprises a write IO request; before the step of adding the incremental metadata to the incremental record queue in the memory, the method further includes: and determining a write address of the disk according to the disk offset in the write IO request, and writing corresponding incremental data into the write address.

The incremental data writing can be realized by a disk underlying drive. And the lower-layer disk drive carries out addressing according to the disk offset corresponding to the write IO request, and then writes the data into a disk medium (a monitoring disk) by taking a sector as a unit, namely, the data is dropped. It should be noted that, in the embodiment of the present invention, writing the write IO request into the disk also includes writing corresponding incremental data into the disk.

In this embodiment, the write IO request can be stored in the disk to which the write IO request is written, and it is ensured that data is accurately written to an accurate disk position.

In one embodiment, the reading of the corresponding incremental data from the disk according to the incremental metadata is performed; sending the read incremental data to a backup server to trigger the backup server to backup the incremental data, wherein the step of sending the read incremental data to the backup server comprises the following steps: extracting first increment metadata from the increment record queue, and judging whether the corresponding increment data reaches the upper limit of single sending or not according to the first increment metadata; if not, merging the first increment metadata and the second increment metadata to obtain third increment metadata; wherein the second incremental metadata is adjacent to the first incremental metadata in an incremental record queue; reading corresponding incremental data from a disk according to the third incremental metadata; and adding the read incremental data into a sending queue to trigger the sending queue to send the read incremental data to a backup server for backup.

The process of merging the first incremental metadata and the second incremental metadata can be realized by the following steps: checking whether a next increment metadata (second increment metadata) of the first increment metadata in the increment recording queue is backward adjacent to the current reading position; if so, merging the first incremental metadata with the second incremental metadata. The merging here may mean that two pieces of incremental metadata are placed one behind the other, and when the incremental data is read, corresponding first incremental data and second incremental data are acquired according to the first incremental metadata and the second incremental metadata, respectively. Thereafter, both the first incremental data and the second incremental data are added to the transmit queue.

of course, the number of the second incremental metadata may be plural. That is, in the case where the length of the delta data does not reach the upper limit of a single transmission, a plurality of backward adjacent delta metadata may be merged into one delta metadata to be read and transmitted.

according to the embodiment, under the condition that the single sending upper limit is not exceeded, the adjacent incremental metadata are merged, so that data backup is performed, the data backup efficiency can be effectively improved, and the continuous protection of the data can be better realized.

In one embodiment, the step of reading corresponding delta data from the disk according to the delta metadata includes: and in the current reading period, when the number of the incremental metadata exceeds a set threshold, stopping reading the incremental data from the disk until the next reading period is reached.

When data writing is carried out, the kernel filter driver constructs an increment metadata and caches the increment metadata in an increment record queue in the memory, and then writes the increment metadata into the monitoring disk. And the kernel reading routine reads the incremental data from the monitoring disk according to the incremental metadata and submits the incremental data to the kernel sending routine to be sent to the backup server. If IO is frequent (namely IO requests are frequently received), the kernel reading routine and other system applications compete for disk resources, so that IO delay of other system applications is increased, and a write performance priority mechanism is introduced for the scheme: when IO is frequent, the working frequency of the kernel reading routine is reduced (for example, the kernel reading routine delays the reading action to the next reading period for processing), and other system applications are ensured to preferentially obtain the disk resources to complete the IO operation. The method for judging the write IO frequency comprises the following steps: judging whether the number of incremental metadata corresponding to the write IO request in each reading period exceeds a set threshold value or not; and if the write IO frequency exceeds the set threshold, determining that the IO is frequently written.

Additionally, if the kernel read routine is always working at a low frequency, the incremental metadata will increase, resulting in an incremental record queue overflow. Therefore, the length threshold of the incremental metadata record queue is limited on the basis of the priority of writing performance, and when the length threshold is reached, the kernel reading routine works at normal frequency, so that the incremental metadata records are ensured to be processed as soon as possible, and the incremental record queue is prevented from occupying excessive memory. For example, an overhead threshold of delta metadata in memory may be defined. Assuming that the overhead threshold is 50M, the buffer data size is 4 GB-512 GB. In this case, when the memory overhead of the incremental record queue reaches the overhead threshold, the incremental data is still read even though IO is frequent.

In one embodiment, as shown in fig. 5, there is provided a method for reading incremental data, including the steps of:

S501, obtaining incremental metadata; the delta metadata is used to describe the delta data.

S502, acquiring a writing state mark of the incremental metadata; the writing state mark is used for representing whether the corresponding incremental data is successfully written into the disk.

For the case that the incremental data is successfully written into the disk, the write status flag may be a write complete flag; for the case where the incremental data was not successfully written to disk (e.g., the case of not yet written or write failed), the write status flag may be a to-do flag.

Specifically, the writing status flag includes: writing a completion mark and a to-be-completed mark; reading corresponding incremental data from a disk according to the extracted incremental metadata and the write state mark, wherein the step comprises the following steps: when the write state mark corresponding to the incremental metadata is a write completion mark, reading corresponding incremental data from a disk according to the extracted incremental metadata; when the writing state mark corresponding to the increment metadata is a mark to be completed, judging whether the next reading period is reached; and if so, reading corresponding incremental data from the disk according to the extracted incremental metadata.

In addition, when the write status flag is a to-be-completed flag, the data may not be read again when the next read cycle arrives, for example: the write status flag can be detected in real time, and when the write status flag is detected to be changed from the pending flag to the write completion flag, the incremental data is read.

The kernel read routine reads data from the disk according to the incremental metadata, so that the intercepted write IO request needs to be submitted to the underlying disk drive before the incremental metadata is added to the incremental record queue, otherwise the read routine may read old disk data. That is, before the step of obtaining the write status flag of the incremental metadata, the method may further include: receiving a write IO request; the write IO request comprises disk offset; and determining a write address of the disk according to the disk offset in the write IO request, and writing corresponding incremental data into the write address.

And S503, when the writing state mark is a writing completion mark, reading corresponding incremental data from the disk according to the incremental metadata.

Some versions of kernels may merge IO requests at the lower level of the filter driver, so even if it is ensured that an IO request is submitted to the lower disk drive first, and then data is read according to the incremental metadata, it is still possible that a read IO request submitted by a read routine is responded to by the lower disk drive prior to a write IO request submitted previously. For this reason, each incremental metadata record needs to be added with a write status flag, and the write status flag is updated after the data corresponding to the write IO request falls to the disk. The reading routine needs to check whether the flag is updated to a completion state when reading data according to the incremental metadata record, and if not, the reading routine delays the processing until the next reading cycle. The method can ensure that the target backup machine reads correct incremental data from the disk and ensure the normal operation of continuous data protection.

In one embodiment, as shown in fig. 6, there is provided a data reading method, including the steps of:

S601, obtaining increment metadata; the delta metadata is used to describe the delta data.

S602, acquiring a consistency mark of target data; the target data is the incremental data described by the incremental metadata, and the consistency mark is used for describing the consistency of the target data and the incremental data written in the disk.

The target data may refer to incremental data that needs to be read from the disk. The consistency flag may be determined by information in the inconsistent record queue in the foregoing embodiment.

After the step of obtaining the incremental metadata, the method further comprises: adding the incremental metadata to an incremental record queue located in a memory; when it is determined that the target data is consistent with the incremental data written in the disk according to the consistency mark, reading the corresponding incremental data from the disk, including: when the consistency mark determines that the target data is consistent with the increment data written in the disk, extracting the increment metadata from the increment recording queue; and reading corresponding incremental data from the disk according to the extracted incremental metadata.

And S603, when the consistency between the target data and the incremental data written in the disk is determined according to the consistency marks, reading the corresponding incremental data from the disk.

The implementation of this step may include: when the inconsistent record queue is empty, judging that the consistency mark is consistent; reading corresponding incremental data from a disk according to the extracted incremental metadata; when the inconsistent recording queue is not empty, judging that the consistency mark is inconsistent; emptying the inconsistent record queue and updating the consistency mark to be consistent; and reading corresponding incremental data from the disk according to the extracted incremental metadata.

The purpose of clearing the inconsistent record queue is to prevent the target backup machine from being in an inconsistent state for a long time and to prevent the continuous data protection of the target backup machine from being influenced. After the inconsistent record queue is cleared, the incremental metadata originally recorded in the inconsistent record queue can be directly deleted, or can be sent to the backup server through a kernel sending routine so as to be stored in a certain independent storage space of the backup server.

According to the embodiment, whether the incremental data needing to be read is consistent with the incremental data written in the disk or not is determined according to the consistency mark, and then the corresponding incremental data is read from the disk, so that the target backup machine can read the correct incremental data from the disk.

In one embodiment, the step of obtaining the consistency flag of the target data further comprises: receiving a write IO request; the write IO request comprises disk offset; when the consistency marks are consistent, determining a write address of a disk according to disk offset in the write IO request, and writing corresponding incremental data into the write address; when the consistency marks are inconsistent, judging whether a sleep ending instruction is received or not; if so, determining a write address of the disk according to the disk offset in the write IO request, and writing corresponding incremental data into the write address. In this embodiment, only when the consistency flag is consistent, the write IO request is written.

The filter driver in this embodiment needs to check the consistency flag each time a write IO request is intercepted, and defers the write operation through a sleep action if a forced consistency period is entered. The purpose of suspending write operations is to reduce the speed of incremental metadata generation while maintaining a high rate of reads and sends, thereby causing the backup data to enter a consistent state as quickly as possible.

Of course, in other embodiments, before performing write IO request write and data read, if the consistency flag is determined to be inconsistent, in addition to waiting for the next write or read cycle to arrive, the status of the consistency flag may also be monitored in real time, and when the consistency flag becomes consistent, corresponding data write and read operations are performed.

in one embodiment, when the incremental record collision probability is high, the backup data may be in an inconsistent state for a long time. A mandatory coherence timeout mechanism is added for this purpose. That is, when the inconsistent record queue is empty, before the step of reading the corresponding incremental data from the disk according to the extracted incremental metadata, the method further includes: and updating the consistency time according to the current time. Further, when the inconsistent record queue is not empty, the step of emptying the inconsistent record queue includes: and determining the time difference between the current time and the consistency time, and emptying the inconsistent record queue when the time difference is greater than a preset time threshold. The data consistency is checked when the reading routine reads the data, and if the data is at a consistent point, the consistency time is updated to be the current time; if the current time is not consistent with the previous time, checking whether the current time is overtime or not, and if yes, setting a forced consistency mark. The embodiment can prevent backup data from being in an inconsistent state for a long time through a consistency timeout mechanism.

In one embodiment, as shown in fig. 7, a method for continuous data protection without local cache is provided, which includes the following steps:

S701, acquiring a plurality of data backup requests, and acquiring a plurality of corresponding initial incremental metadata according to the plurality of data backup requests; wherein the initial delta metadata is used to describe delta data.

When large blocks of continuous data are written into a disk, the kernel may split the large blocks of continuous data into a plurality of write IO requests, and incremental metadata corresponding to the write IO requests are combined into one reading and sending action, so that the real-time backup efficiency can be greatly improved. Therefore, when a plurality of write IO requests are received, the write IO requests can be converted into corresponding initial increment metadata and then corresponding increment metadata.

s702, merging the plurality of initial increment metadata to obtain increment metadata.

The initial incremental metadata may be derived from the disk offset and data length in the data backup request.

This step can be realized by the following steps: and merging the plurality of initial increment metadata according to the disk offset and the data length.

Further, the step of merging the plurality of initial incremental metadata to obtain the incremental metadata includes: merging the plurality of initial increment metadata to obtain merged initial increment metadata; acquiring the merging length of the merging initial increment metadata; and when the merging length is larger than a preset merging length threshold, ending merging the initial incremental metadata, and determining the merged initial incremental metadata as the incremental metadata. The merging length threshold may be set according to practical situations, and may be 1024 sectors, for example.

The specific implementation manner of this step may be: in order to avoid an excessive amount of data read and transmitted at a time, the merge length threshold is set to 1024 sectors. As shown in fig. 8, the incremental recording continuation case includes 6 scenes in which (1), (3), (4) are continued backward and (2), (5), (6) are continued forward. The present embodiment only considers the case of backward continuation, i.e., Offset_A＜＜Offset_BAnd Offset_A+Length_A＞＞Offset_BAnd Offset_B+Length_B-Offset_A＞＞T_LengthWherein, Offset_Aoffset of the disk recorded for the current increment_BDisk offset, Length, for the next incremental record_AFor the data length of the current delta record, Length B is the length of the next delta record, T_Lengthis a merge length threshold. Will be successively increased by an initial amountThe metadata is merged into one incremental metadata.

And S703, reading corresponding incremental data from the disk according to the incremental metadata.

For the implementation of this step, reference may be made to the corresponding step S202 implemented above, which is not described in detail.

S704, sending the read incremental data to a backup server to trigger the backup server to backup the incremental data.

For the implementation of this step, reference may be made to the corresponding step S203 implemented above, which is not described in detail.

In this embodiment, the initial incremental metadata corresponding to the multiple data backup requests are merged to obtain the incremental metadata, and the corresponding incremental data is read according to the incremental metadata obtained by merging and sent to the backup server for backup, so that the efficiency of real-time backup can be effectively improved.

In order to better understand the above method, an application example of the continuous data protection method without local cache according to the present invention is described in detail below as shown in fig. 9.

And S901, submitting a write IO request by an upper layer.

s902, the filter driver intercepts the write IO request, extracts information such as disk offset, data length and IO timestamp in the write IO request, and constructs an incremental metadata record (offset, length, timestamp).

and S903, submitting the write IO request to a disk lower layer drive by the filter driver, and writing the write IO request into the monitoring disk by the disk lower layer drive.

S904, the target backup machine completes the incremental metadata records, so that the disk offset and the data length corresponding to the incremental metadata records meet 4KB alignment. And setting a change sector bitmap according to the disk offset and the data length corresponding to the incremental metadata record, and checking whether an overlapped sector bit exists in the setting process.

S905, adding the incremental metadata records into an incremental record queue in the memory.

S906, if the overlapping sector location is detected in S904, adding the corresponding incremental metadata record to the conflict record queue.

S907, the kernel read routine extracts an incremental metadata record A from the incremental record queue.

s908, removing the overlapping sector position and the incremental metadata record A in the conflict record queue; when it is determined that the incremental metadata record A in the conflicting record queue is read ahead, the incremental metadata record A is added to the inconsistent record queue.

s909, the kernel reading routine checks whether the incremental data corresponding to the acquired incremental metadata record does not reach the upper limit of single transmission. And if the upper limit of single sending is not reached, acquiring an adjacent increment metadata record and merging the increment metadata records. After passing the data consistency check, the incremental data is read from the disk according to the merged incremental metadata record. And if the upper limit of single sending is reached, reading the incremental data from the disk according to the incremental metadata record after passing the data consistency check.

S910, adding the incremental data read from the disk into a sending queue by the kernel reading routine.

And S911, the kernel sending routine extracts the incremental data records from the sending queue and sends the incremental data records to the backup server.

In the embodiment, the space occupied by the incremental metadata is small, and the memory of the target backup machine is enough to store the incremental metadata, so that additional local cache is not needed, and the protection of continuous data is more convenient. Meanwhile, collision detection and data consistency point check are carried out by changing the sector bitmap, the collision record queue and the inconsistent record queue, and the orderly proceeding of continuous data protection can be ensured.

The steps involved in the continuous data protection method without the local cache, the incremental data reading method and the data reading method in the embodiment of the invention can be combined randomly. Embodiments of a step may be equally applicable to other embodiments that include the same step.

it should be noted that, for the sake of simplicity, the foregoing method embodiments are described as a series of acts or combinations, but those skilled in the art should understand that the present invention is not limited by the described order of acts, as some steps may be performed in other orders or simultaneously according to the present invention.

Based on the same idea as the continuous data protection method without the local cache in the above embodiment, the present invention further provides a continuous data protection system without the local cache, which can be used to execute the continuous data protection method without the local cache. For convenience of illustration, the schematic structural diagram of the embodiment of the continuous data protection system without local cache only shows the parts related to the embodiment of the present invention, and those skilled in the art will understand that the illustrated structure does not constitute a limitation to the system, and may include more or less components than those illustrated, or combine some components, or arrange different components.

In one embodiment, as shown in fig. 10, the continuous data protection system without local cache includes a filter driver module 1001, a read module 1002, and a send module 1003, which are described in detail as follows: a filter driver module 1001 configured to obtain a data backup request; obtaining incremental metadata according to the data backup request; wherein the delta metadata is used to describe delta data. A reading module 1002, configured to read corresponding incremental data from a disk according to the incremental metadata. And a sending module 1003, configured to send the read incremental data to a backup server, so as to trigger the backup server to backup the incremental data. In the embodiment, the space occupied by the incremental metadata is small, and the memory of the target backup machine is enough to store the incremental metadata, so that additional local cache is not needed, and the protection of continuous data is more convenient.

in one embodiment, further comprising: the first record queue adding module is used for adding the incremental metadata to an incremental record queue in the memory; a reading module 1002, comprising: a first metadata extraction sub-module, configured to extract the incremental metadata from the incremental record queue; and the first data reading sub-module is used for reading corresponding incremental data from the disk according to the extracted incremental metadata.

In one embodiment, the incremental metadata is obtained according to the disk offset and the data length in the data backup request; further comprising: the sector bitmap writing module is used for writing the incremental metadata into the sector position corresponding to the variable sector bitmap; the corresponding sector bit is determined according to the disk offset and the data length; a second recording queue adding module, configured to add, when it is detected that an overlapping sector bit exists in the changed sector bitmap, newly written incremental metadata corresponding to the overlapping sector bit to a conflict recording queue; and the overlapped sector position is a sector position overlapped by newly written increment metadata and historical increment metadata in the change sector bitmap.

In one embodiment, further comprising: a data clearing module for clearing the overlapping sector bit and the corresponding increment metadata in the conflict record queue; the overlap judging module is used for traversing conflict increment metadata in a conflict record queue and judging whether an overlap sector position corresponding to the conflict increment metadata exists in the change sector bitmap; an inconsistent queue adding module, configured to determine that the conflict incremental metadata is read in advance when the inconsistent queue adding module does not exist; adding the conflicting delta metadata to an inconsistent record queue; wherein the inconsistent record queue is used for consistency check before reading the incremental data from the disk; the consistency check includes determining consistency of the delta data that needs to be read with the delta data written in the disk.

in one embodiment, the data backup request comprises a write IO request; further comprising: and the first data writing module is used for determining a writing address of the disk according to the disk offset in the write IO request and writing corresponding incremental data into the writing address.

In one embodiment, further comprising: a sending upper limit judging module, configured to extract first incremental metadata from the incremental record queue, and judge whether corresponding incremental data reaches a single sending upper limit according to the first incremental metadata; the metadata merging module is used for merging the first incremental metadata and the second incremental metadata if the first incremental metadata and the second incremental metadata are not merged to obtain third incremental metadata; wherein the second incremental metadata is adjacent to the first incremental metadata in an incremental record queue; a first incremental data reading module, configured to read corresponding incremental data from a disk according to the third incremental metadata; and the data sending module is used for adding the read incremental data into a sending queue so as to trigger the sending queue to send the read incremental data to a backup server for backup.

in one embodiment, the reading module 1002 is further configured to, in the current reading cycle, stop reading the incremental data from the disk until a next reading cycle is reached when the number of the incremental metadata exceeds a set threshold.

In one embodiment, as illustrated in fig. 11, the incremental data reading system includes a filter driving module 1101, a flag obtaining module 1102, and a reading module 1103. The detailed description is as follows: a filter driver module 1101 for obtaining incremental metadata; wherein the delta metadata is used to describe delta data; a flag obtaining module 1102, configured to obtain a write status flag of the incremental metadata; the write state mark is used for representing whether the corresponding incremental metadata is successfully written into the disk; and a reading module 1103, configured to, when the write status flag is a write completion flag, read corresponding incremental data from the disk according to the incremental metadata. In this embodiment, by setting the write status flag, it can be characterized whether the corresponding incremental data is successfully written into the disk, so that the target backup machine can read the correct incremental data from the disk.

In one embodiment, further comprising: an IO request receiving module, configured to receive a write IO request; the write IO request comprises disk offset; and the second data writing module is used for determining a writing address of the disk according to the disk offset in the write IO request and writing corresponding incremental data into the writing address.

in one embodiment, further comprising: a third record queue adding module, configured to add the incremental metadata to an incremental record queue located in a memory; the reading module 1103 includes: the second metadata extraction sub-module is used for extracting the increment metadata from the increment record queue; and the second data reading submodule is used for reading corresponding incremental data from the disk according to the extracted incremental metadata and the write state mark.

In one embodiment, the write status flag includes: writing a completion mark and a to-be-completed mark; the reading module 1103 includes: the second incremental data reading module is used for reading corresponding incremental data from a magnetic disk according to the extracted incremental metadata when the write state mark corresponding to the incremental metadata is a write completion mark; a third incremental data reading module, configured to determine whether a next reading cycle is reached when the write status flag corresponding to the incremental metadata is a to-be-completed flag; and if so, reading corresponding incremental data from the disk according to the extracted incremental metadata.

In one embodiment, as depicted in FIG. 12, a system for reading data includes a filter driver module 1201, an indicia acquisition module 1202, and a read module 1203. The detailed description is as follows: a filter driver module 1201, configured to obtain incremental metadata; the incremental metadata is used for describing incremental data; a tag obtaining module 1202, configured to obtain a consistency tag of the target data; the target data is incremental data described by the incremental metadata, and the consistency mark is used for describing the consistency of the target data and the incremental data written in a disk; and a reading module 1203, configured to read corresponding incremental data from the disk when it is determined that the target data is consistent with the incremental data written in the disk according to the consistency flag. In this embodiment, whether the incremental data to be read is consistent with the incremental data written in the disk is determined according to the consistency flag, and then the corresponding incremental data is read from the disk, so that the target backup machine can read the correct incremental data from the disk.

in one embodiment, further comprising: a fourth record queue adding module, configured to add the incremental metadata to an incremental record queue located in a memory; the reading module 1203 includes: a third metadata extraction submodule, configured to extract the incremental metadata from the incremental record queue when it is determined, according to the consistency flag, that the target data is consistent with the incremental data written in the disk; and the third data reading sub-module is used for reading corresponding incremental data from the disk according to the extracted incremental metadata.

In one embodiment, the reading module 1203 includes: the fourth incremental data reading module is used for judging that the consistency marks are consistent when the inconsistent record queue is empty; reading corresponding incremental data from a disk according to the extracted incremental metadata; a fifth incremental data reading module, configured to determine that the consistency flag is inconsistent when the inconsistent record queue is not empty; emptying the inconsistent record queue and updating the consistency mark to be consistent; and reading corresponding incremental data from the disk according to the extracted incremental metadata.

in one embodiment, when the inconsistent recording queue is empty, the method further includes: and the consistency time updating module is used for updating the consistency time according to the current time.

In an embodiment, when the inconsistent recording queue is not empty, the fifth incremental data reading module is further configured to determine a time difference between the current time and the consistent time, and when the time difference is greater than a preset time threshold, empty the inconsistent recording queue.

In an embodiment, as shown in fig. 13, the continuous data protection system without local cache includes a filter driver module 1301, a merge module 1302, a read module 1303, and a filter driver module 1301, configured to obtain a plurality of data backup requests, and obtain a plurality of corresponding initial incremental metadata according to the plurality of data backup requests; wherein the initial delta metadata is used to describe delta data; a merging module 1302, configured to merge the multiple initial incremental metadata to obtain incremental metadata; a reading module 1303, configured to read corresponding incremental data from a disk according to the incremental metadata; and a sending module 1304, configured to send the read incremental data to a backup server, so as to trigger the backup server to backup the incremental data. In this embodiment, the initial incremental metadata corresponding to the multiple data backup requests are merged to obtain the incremental metadata, and the corresponding incremental data is read according to the incremental metadata obtained by merging and sent to the backup server for backup, so that the efficiency of real-time backup can be effectively improved.

in one embodiment, the initial incremental metadata is obtained according to a disk offset and a data length in the data backup request; the merging module 1302 is further configured to merge the plurality of initial incremental metadata according to the disk offset and the data length.

In one embodiment, the merge module 1302 includes: the initial metadata merging submodule is used for merging the plurality of initial incremental metadata to obtain merged initial incremental metadata; a merging length obtaining submodule, configured to obtain a merging length of the merged initial incremental metadata; and the metadata determining module is used for finishing the combination of the initial increment metadata when the combination length is larger than a preset combination length threshold value, and determining the combined initial increment metadata as the increment metadata.

It should be noted that, the continuous data protection system without a local cache of the present invention corresponds to the continuous data protection method without a local cache of the present invention one to one, and the technical features and the beneficial effects described in the above embodiments of the continuous data protection method without a local cache are all applicable to the embodiments of the continuous data protection system without a local cache.

In addition, in the above exemplary embodiment of the continuous data protection system without local cache, the logical division of each program module is only an example, and in practical applications, the above function distribution may be performed by different program modules according to needs, for example, due to configuration requirements of corresponding hardware or convenience of implementation of software, that is, the internal structure of the continuous data protection system without local cache is divided into different program modules to perform all or part of the above described functions.

In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as shown in fig. 14. The computer device includes a processor, a memory, a network interface, and a database connected by a system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device comprises a nonvolatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of an operating system and computer programs in the non-volatile storage medium. The database of the computer device is used to store incremental metadata, incremental data, and the like. The network interface of the computer device is used for communicating with an external terminal through a network connection. The computer program is executed by a processor to implement a method of continuous data protection without local caching.

Those skilled in the art will appreciate that the architecture shown in fig. 14 is merely a block diagram of some of the structures associated with the disclosed aspects and is not intended to limit the computing devices to which the disclosed aspects apply, as particular computing devices may include more or less components than those shown, or may combine certain components, or have a different arrangement of components.

In one embodiment, a computer device is provided, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, the processor implementing the following steps when executing the computer program: acquiring a data backup request; obtaining incremental metadata according to the data backup request; wherein the delta metadata is used to describe delta data; reading corresponding incremental data from a disk according to the incremental metadata; and sending the read incremental data to a backup server to trigger the backup server to backup the incremental data.

In one embodiment, a computer device is provided, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, the processor implementing the following steps when executing the computer program: obtaining incremental metadata; the incremental metadata is used for describing incremental data; acquiring a writing state mark of the incremental metadata; the writing state mark is used for representing whether the corresponding incremental data is successfully written into the disk; and when the writing state mark is a writing completion mark, reading corresponding incremental data from a disk according to the incremental metadata.

In one embodiment, a computer device is provided, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, the processor implementing the following steps when executing the computer program: obtaining incremental metadata; the incremental metadata is used for describing incremental data; acquiring a consistency mark of target data; the target data is incremental data described by the incremental metadata, and the consistency mark is used for describing the consistency of the target data and the incremental data written in a disk; and when the consistency of the target data and the incremental data written in the disk is determined according to the consistency mark, reading the corresponding incremental data from the disk.

In one embodiment, a computer device is provided, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, the processor implementing the following steps when executing the computer program: acquiring a plurality of data backup requests, and acquiring a plurality of corresponding initial incremental metadata according to the plurality of data backup requests; wherein the initial delta metadata is used to describe delta data; merging the plurality of initial increment metadata to obtain increment metadata; reading corresponding incremental data from a disk according to the incremental metadata; and sending the read incremental data to a backup server to trigger the backup server to backup the incremental data.

In one embodiment, a computer-readable storage medium is provided, having a computer program stored thereon, which when executed by a processor, performs the steps of: acquiring a data backup request; obtaining incremental metadata according to the data backup request; wherein the delta metadata is used to describe delta data; reading corresponding incremental data from a disk according to the incremental metadata; and sending the read incremental data to a backup server to trigger the backup server to backup the incremental data.

In one embodiment, a computer-readable storage medium is provided, having a computer program stored thereon, which when executed by a processor, performs the steps of: obtaining incremental metadata; the incremental metadata is used for describing incremental data; acquiring a writing state mark of the incremental metadata; the writing state mark is used for representing whether the corresponding incremental data is successfully written into the disk; and when the writing state mark is a writing completion mark, reading corresponding incremental data from a disk according to the incremental metadata.

In one embodiment, a computer-readable storage medium is provided, having a computer program stored thereon, which when executed by a processor, performs the steps of: obtaining incremental metadata; the incremental metadata is used for describing incremental data; acquiring a consistency mark of target data; the target data is incremental data described by the incremental metadata, and the consistency mark is used for describing the consistency of the target data and the incremental data written in a disk; and when the consistency of the target data and the incremental data written in the disk is determined according to the consistency mark, reading the corresponding incremental data from the disk.

In one embodiment, a computer-readable storage medium is provided, having a computer program stored thereon, which when executed by a processor, performs the steps of: acquiring a plurality of data backup requests, and acquiring a plurality of corresponding initial incremental metadata according to the plurality of data backup requests; wherein the initial delta metadata is used to describe delta data; merging the plurality of initial increment metadata to obtain increment metadata; reading corresponding incremental data from a disk according to the incremental metadata; and sending the read incremental data to a backup server to trigger the backup server to backup the incremental data.

It will be understood by those skilled in the art that all or part of the processes of the methods of the above embodiments may be implemented by a computer program, which is stored in a computer readable storage medium and sold or used as a stand-alone product. More specific examples (a non-exhaustive list) of the computer-readable medium would include the following: an electrical connection (electronic device) having one or more wires, a portable computer diskette (magnetic device), a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). Additionally, the computer-readable medium could even be paper or another suitable medium upon which the program is printed, as the program can be electronically captured, via for instance optical scanning of the paper or other medium, then compiled, interpreted or otherwise processed in a suitable manner if necessary, and then stored in a computer memory.

It should be understood that portions of the present invention may be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, the various steps or methods may be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, any one or combination of the following techniques, which are known in the art, may be used: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application specific integrated circuit having an appropriate combinational logic gate circuit, a Programmable Gate Array (PGA), a Field Programmable Gate Array (FPGA), or the like.

The terms "comprises" and "comprising," and any variations thereof, of embodiments of the present invention are intended to cover non-exclusive inclusions. For example, a process, method, system, article, or apparatus that comprises a list of steps or (module) elements is not limited to only those steps or elements but may alternatively include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.

the technical features of the embodiments described above may be arbitrarily combined, and for the sake of brevity, all possible combinations of the technical features in the embodiments described above are not described, but should be considered as being within the scope of the present specification as long as there is no contradiction between the combinations of the technical features.

The above-described examples merely represent several embodiments of the present invention and should not be construed as limiting the scope of the invention. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the inventive concept, which falls within the scope of the present invention. Therefore, the protection scope of the present patent shall be subject to the appended claims.

Claims

1. A method for reading data, comprising the steps of:

obtaining incremental metadata; the incremental metadata is used for describing incremental data;

Acquiring a consistency mark of target data; the target data is incremental data described by the incremental metadata, and the consistency mark is used for describing the consistency of the target data and the incremental data written in a disk;

When the consistency mark determines that the target data is consistent with the incremental data written in the disk, reading the corresponding incremental data from the disk;

After the step of obtaining the incremental metadata, the method further includes:

Adding the incremental metadata to an incremental record queue located in a memory;

when it is determined that the target data is consistent with the incremental data written in the disk according to the consistency mark, reading the corresponding incremental data from the disk, including:

When the consistency mark determines that the target data is consistent with the increment data written in the disk, extracting the increment metadata from the increment recording queue;

And reading corresponding incremental data from the disk according to the extracted incremental metadata.

2. The data reading method according to claim 1, wherein the incremental metadata includes: disk offset and data length;

Before the step of adding the incremental metadata to the incremental record queue in the memory, the method further includes:

writing the incremental metadata into the sector position corresponding to the variable sector bitmap; the corresponding sector bit is determined according to the disk offset and the data length;

When detecting that the changed sector bitmap has an overlapped sector position, adding newly written incremental metadata corresponding to the overlapped sector position into a conflict record queue; and the overlapped sector position is a sector position in which newly written increment metadata in the change sector bitmap is overlapped with the historical increment metadata.

3. The method for reading data according to claim 2, wherein the step of reading the corresponding incremental data from the disk is followed by further comprising:

Clearing the overlapping sector area and corresponding increment metadata in the conflict record queue;

Traversing conflict increment metadata in a conflict record queue, and judging whether an overlapped sector position corresponding to the conflict increment metadata exists in the change sector bitmap;

When not present, determining that the conflicting delta metadata is read ahead; adding the conflicting delta metadata to an inconsistent record queue; wherein the inconsistent record queue is used for consistency checking before reading the incremental data from the disk.

4. The method for reading data according to claim 3, wherein the step of reading corresponding delta data from the disk according to the extracted delta metadata comprises:

When the inconsistent record queue is empty, judging that the consistency mark is consistent; reading corresponding incremental data from a disk according to the extracted incremental metadata;

When the inconsistent recording queue is not empty, judging that the consistency mark is inconsistent; emptying the inconsistent record queue and updating the consistency mark to be consistent; and reading corresponding incremental data from the disk according to the extracted incremental metadata.

5. The data reading method according to claim 4,

when the inconsistent record queue is empty, before the step of reading the corresponding incremental data from the disk according to the extracted incremental metadata, the method further includes: updating the consistency time according to the current time;

and/or the presence of a gas in the gas,

when the inconsistent record queue is not empty, the step of emptying the inconsistent record queue includes:

And determining the time difference between the current time and the consistency time, and emptying the inconsistent record queue when the time difference is greater than a preset time threshold.

6. the method for reading data according to claim 4 or 5, wherein the step of obtaining the consistency flag of the target data is preceded by the step of:

Receiving a write IO request; the write IO request comprises disk offset;

When the consistency marks are consistent, determining a write address of a disk according to disk offset in the write IO request, and writing corresponding incremental data into the write address;

When the consistency marks are inconsistent, judging whether a sleep ending instruction is received or not; if so, determining a write address of the disk according to the disk offset in the write IO request, and writing corresponding incremental data into the write address.

7. A system for reading data, comprising:

the filtering driving module is used for acquiring incremental metadata; the incremental metadata is used for describing incremental data;

the mark acquisition module is used for acquiring a consistency mark of the target data; the target data is incremental data described by the incremental metadata, and the consistency mark is used for describing the consistency of the target data and the incremental data written in a disk;

The reading module is used for reading corresponding incremental data from the disk when the consistency between the target data and the incremental data written in the disk is determined according to the consistency mark;

Further comprising: the first record queue adding module is used for adding the incremental metadata to an incremental record queue in the memory; the reading module comprises: the metadata extraction sub-module is used for extracting the incremental metadata from the incremental record queue when the consistency mark determines that the target data is consistent with the incremental data written in the disk; and the data reading sub-module is used for reading corresponding incremental data from the disk according to the extracted incremental metadata.

8. The data reading system of claim 7, wherein the incremental metadata is derived from a disk offset and a data length in the data backup request;

Further comprising: the sector bitmap writing module is used for writing the incremental metadata into the sector position corresponding to the variable sector bitmap; the corresponding sector bit is determined according to the disk offset and the data length; a second recording queue adding module, configured to add, when it is detected that an overlapping sector bit exists in the changed sector bitmap, newly written incremental metadata corresponding to the overlapping sector bit to a conflict recording queue; and the overlapped sector position is a sector position overlapped by newly written increment metadata and historical increment metadata in the change sector bitmap.

9. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the steps of the method of any of claims 1 to 6 are implemented by the processor when executing the computer program.

10. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the method of any one of claims 1 to 6.