WO2017160318A1

WO2017160318A1 - Deduplicating blocks of data

Info

Publication number: WO2017160318A1
Application number: PCT/US2016/023260
Authority: WO
Inventors: Kevin Lloyd Jones
Original assignee: Hewlett Packard Enterprise Development LP
Current assignee: Hewlett Packard Enterprise Development LP
Priority date: 2016-03-18
Filing date: 2016-03-18
Publication date: 2017-09-21
Anticipated expiration: 2018-09-18

Abstract

Some examples provide a system including a processor and a non-transitory medium. The non-transitory medium comprises instructions stored thereon that, when executed, cause the processor to: receive a stream of data blocks, and receive localization metadata that indicates information about logical entities of the stream. The instructions, when executed, may further cause the processor to: reconstitute the logical entities from the stream of data blocks based on localization metadata and hashes derived from the data blocks, and deduplicate at least some of the data blocks of the reconstituted logical entities.

Description

Deduplicating Blocks of Data

BACKGROUND

[0001] A computing device may store data blocks on a storage device. Some of the data blocks may be duplicates of each other.

BRIEF DESCRIPTION OF THE DRAWINGS

[0002] Certain examples are described in the following detailed description and in reference to the drawings, in which:

[0003] FIG. 1 is a diagram of an example computing system that may perform deduplication of data blocks;

[0004] F!G. 2 is another diagram of an example computing system that may perform deduplication of data blocks;

[0005] FIG. 3 is a flowchart of an example method for performing deduplication of data blocks;

[0006] F!G. 4 is a flowchart of an example method for performing deduplication of data blocks; and

[0007] FIG. 5 is a block diagram of an example for performing

deduplication of data blocks.

DETAILED DESCRIPTION

[0008] Datacenters may comprise numerous computing devices. In some examples, each computing device may store a copy of a same file. In some examples, a computing device may repeatedly store different evolutions or versions of a same file, however the file versions may share many common data blocks. That is, there may exist a duplicate data blocks across multiple file versions, and/or duplicate data blocks of a same file on multiple computing devices. For example, a file common to a particular operating system file may be duplicated among the file systems of computing devices running a same operating system (OS) version. As another example, computing devices repeatedly change (e.g. append) and store a file. However, many data blocks of the repeatedly changing file may be identical. [0009] A computing device may support dedupiication capabilities, which may reduce the storage of duplicate blocks of data. More particularly a device such as a dedupiication appliance receives storage requests to write data to an address of a storage device. The dedupiication appliance determines whether the blocks of data to be stored (data blocks) are already stored on the storage device.

[0010] If the data blocks are already present on the storage device, the dedupiication appliance determines that the data blocks do not need to be written. Instead, the dedupiication device maintains a single copy of the data blocks and creates a record that maps to the address of the single copy of the data blocks already present on the storage device. In this manner, the dedupiication appliance avoids storing duplicate copies of data blocks, thereby reducing storage requirements. If the dedupiication appliance receives a read request for the stored data block, the dedupiication appliance reads the address stored in the pointer and retrieves the data blocks at the location referenced by the pointer.

[0011] However, a dedupiication appliance may not be readily able to identify data blocks from virtual machine (VM) backups for dedupiication. As an example, a backup agent that executes on a virtual machine may send a stream of data representing a VM backup. However, the stream may not identify particular files, data blocks or block boundaries as they are stored on the virtual machine. Additionally, the backup agent may send data blocks and files in an unordered fashion that does not correspond to a sequential ordering of blocks on the VM. Because the block sequencing of the backup stream may differ from the block sequencing of the dedupiication store (i.e. storage coupled with a dedupiication appliance), the dedupiication appliance may fail to detect duplicate blocks because the block matching algorithm of the dedupiication appliance degrades when blocks are mis-sequenced with respect to their storage sequence within the dedupiication store.

[0012] The techniques described herein improve dedupiication of such unordered backup streams. A backup agent executing on a source device (i.e. a computing device, such as a VM, to be backed up) generates and stores localization metadata. The localization metadata comprises a plurality of hash sets. Each hash set comprises a plurality of hash values that are generated in a sequence. The sequence of the hash values is based on each of the blocks of a logical entity (e.g. a file) of the source machines' file system.

[0013] During a backup, the backup agent executing on the source device may transmit the localization metadata to the dedupiication appliance discretely, or embedded within the backup stream. The localization metadata may have an embedded signature (e.g. magic data) to facilitate identification, extraction and reconstitution by the dedupiication appliance. The backup agent also transmits the unordered stream comprised of the source machine's file system blocks to the dedupiication appliance. The dedupiication appliance reconstitutes the logical entities of the source device based on the localization metadata to detect duplicate blocks and to sequentially store non-duplicate blocks.

[0014] FIG. 1 is a conceptual diagram of an example computing system that may perform dedupiication of data blocks. Computing system 100 is illustrated in FIG. 1 . Computing system 100 comprises a device 102, which may be a dedupiication appliance in some examples. Device 102 comprises a processor 104 and a non-transitory medium containing instructions stored thereon that, when executed, cause the processor to perform certain

functionality.

[0016] Processor 104 may comprise a central processing unit (CPU), graphics processing unit (GPU), application specific integrated circuit (ASIC), digital signal processor (DSP), field programmable gate array (FPGA) or the like. Processor 104 may comprise any combination of the aforementioned. Processor 104 may also comprise one or more virtual devices, such as virtual processors of one or more virtual machines. Medium 108 may comprise software, firmware, non-volatile memory, or the like. Medium 106 may also be any combination of the aforementioned types of media. Processor 104 executes the instructions on medium 106.

[0016] Processor 104 receives localization metadata 1 18 and a stream 108. Stream 108 comprises hashes data block 1 16 and hashes 1 10. A computing device transmitting stream 108 generates hashes 1 10 using a hashing algorithm for data blocks 1 16. In some examples, there may be a hash for each of data blocks 1 16. In some examples, one of hashes 1 10 may correspond to a plurality of data blocks 1 16.

[0017] Processor 104 may receive stream 108 in combination or discretely from localization metadata 106. Localization metadata 1 18 comprises information about logical entities in stream 108. Logical entities 1 12 may comprise files, data blocks, or the like. Based on localization metadata 1 18, hashes 1 10 and data blocks 1 16, processor 104 determines reconstituted logical entities 1 12.

[0018] Processor 104 determines deduplicated blocks 1 14 based on reconstituted logical entities 1 12. Processor 104 may determine deduplicated block 1 14 by comparing hashes or bit sequences of reconstituted logical entities 1 12 with hashes or bit sequences of a deduplication store.

[0019] FIG. 2 is another conceptual diagram of an example computing system that may perform deduplication of data blocks. FIG. 2 illustrates a computing system 200. Computing system 200 comprises a device 102, localization metadata 1 18, and stream 108 as in FIG. 1 .

[0020] In the example of FIG. 2, backup agent 216 of virtual machine /virtual snapshot 214 may generate stream 108. Stream 108 comprises blocks 204A-204E. Blocks 204A-204C correspond to hashes 206A-206C. Blocks 204D-204E correspond to partial hashes 206D-206E, That is, in some examples, backup agent 216 may generate full hashes, but may store partial (e.g. truncated hashes) in data blocks of virtual machine/virtual snapshot 214 to reduce the disk space used to store the hashes.

[0021] In the example of FIG. 2, processor 104 receives localization metadata 1 18 and stream 108. Localization metadata 1 18 comprises hash sets 202. Hash sets 202 comprise a first hash set, "Set 1 ", and a second hash set, "Set 2." Set 1 indicates an ordering of blocks of stream that comprise a logical entity. Set 1 indicates that the reconstituted entity comprises the hashes 206C and 206A in that order. [0022] Processor 104 determines and reconstitutes logical entities 1 12. Processor 104 attempts to dedupiicate data blocks 1 14 based on whether data blocks 1 14 correspond to deduplicated block 1 14 of deduplication store 224. Deduplication store 224 may comprise a storage appliance, a SAN, or the like. Deduplicated blocks 226 are blocks that are already present on deduplication store 224. Processor 104 determines whether data blocks 1 14 match deduplicated blocks 226 based on a block matching algorithm. If a match exists, processor 104 attempts to dedupiicate data blocks 226 by storing a reference to deduplicated blocks 226 rather than making a new copy of data blocks 1 14.

[0023] As indicated above, stream 108 may not be ordered based on logical entities of virtual machine / virtual snapshot 214. Some logical entities may be present in stream 108 that are not indicated by hash sets 202. In some examples, processor 104 may determine a logical entity that is not associated with hash sets 202, e.g. non-associated logical entity 212. Processor 104 may determine data blocks 228 of non-associated logical entity 212, and attempt to dedupiicate data blocks 228.

[0024] Processor 104 receives set 1 and based on set 1 , reconstitutes a logical entity by examining stream 108 for hashes 206C and 206A indicated by set 1 . Based on hashes 206C and 206A, processor 104 identifies the corresponding blocks 204A and 204C. Processor 104 may reorder and/or concatenate blocks 206A and 206C if necessary to construct at least one of reconstituted logical entities 1 12.

[0025] Processor 104 performs a similar process with respect to set 2 of hash sets 202. Processor 104 matches hash 206B, partial hash 206D and partial hash 206E with the corresponding hashes of stream 108 to determine the ordering of blocks 204B, 204D, and 204E. Processor 104 then reconstitutes a logical entity based on the ordering indicated in set 2 of hash sets 202.

[0026] FIG. 3 is a flowchart of an example method for assembling operating system volumes. FIG. 3 comprises method 300. Method 300 may be described below as being executed or performed by a system, for example, computing system 100 (FIG. 1 ), or computing system 200 (F!G. 2). In various examples, method 300 may be performed by hardware, software, fi mware, or any combination thereof. Other suitable systems and/or computing devices may be used as well. Method 300 may be implemented in the form of executable instructions stored on at least one machine-readable storage medium of the system and executed by at least one processor of the system. In various examples, the machine-readable storage medium is non-transitory. Alternatively or in addition, method 300 may be implemented in the form of electronic circuitry (e.g., hardware). In alternate examples of the present disclosure, one or more blocks of method 300 may be executed substantially concurrently or in a different order than shown in FIG. 3. In alternate examples of the present disclosure, method 300 may include more or fewer blocks than are shown in FIG. 3. In some examples, one or more of the blocks of method 300 may, at certain times, be ongoing and/or may repeat.

[0027] Method 300 may start at block 302 at which point a computing device, such as a backup agent executing on virtual machine/virtual machine snapshot 214, may identify logical entities stored on a storage device, e.g. a storage device of the VM/snapshot. At block 304, virtual machine/V!VI snapshot 214 may identify blocks of data (e.g. data blocks 1 16 or data blocks 204A- 204E) that comprise the logical entities. At block 306, virtual machine/VM snapshot 214 may generate hashes (e.g. hashes 206A-2G6C and partial hashes 206D-206E) for each of data blocks 204A-204E.

[0028] At block 308, virtual machine/VM snapshot 214 may determine hash sets (e.g. hash sets 202) that comprise at least some of the hashes. Each of each of the hash sets may be associated with one of the logical entities. Each of hash sets 202 may indicate data blocks of the associated logical entity. At block 310, virtual machineA M snapshot 214 may generate localization metadata, e.g. localization metadata 1 18. The localization metadata may comprise hash sets 202.

[0029] At block 312, virtual machineA M snapshot 214 may transmit, to a computing device, such as device 102, the localization metadata 1 18. At block 314, virtual machineA/M snapshot 214 may transmit, to the computing device, data blocks that comprise the logical entities. [0030] F!G, 4 is a flowchart of an example method for assembling operating system volumes, FIG, 4 comprises method 400. Method 400 may begin at block 402. At block 402, a computing device, such as virtual

machineA M snapshot 214, may execute a backup agent, such as backup agent 216 (illustrated in FIG. 2). At block 404, virtual machineA/M snapshot 214 may identify logical entities stored on a storage device. At block 406, virtual machineA M snapshot 214 may identify blocks of data that comprise the logical entities.

[0031] At block 408, virtual machineA/M snapshot 214 may generate hashes for each of the blocks of data. In some examples, at least some of the hashes, e.g. hashes 204A-2Q4E, may be partial hashes. In some examples a backup agent may generate the hashes. At block 410, virtual machineA/M snapshot 214 may determine hash sets (e.g. hash sets 202) that comprise at least some of the hashes. The hash sets may indicate a sequence of the associated blocks of the logical entity.

[0032] At block 412, virtual machineA/M snapshot 214 may generate localization metadata, e.g. localization metadata 1 18. The localization metadata comprises magic data, wherein the magic data comprises an identifier value of the localization metadata. In some examples, the backup agent may generate the localization metadata. The localization metadata may comprise hash sets 202.

[0033] At block 414, virtual machineA/M snapshot 214 may transmit, to a computing device, such as device 102, the localization metadata 1 18, In some examples, virtual machineA M snapshot 214 transmit the localization metadata separately from stream 108. For example, virtual machine/virtual machine snapshot 214 may transmit localization metadata 1 18 using a network socket, another type of separate network connection, or any other transport mechanism separate from that used to transmitting the data blocks. At block 416, virtual machineA M snapshot 214 may transmit, to the computing device, data blocks that comprise the logical entities.

[0034] F!G. 5 is a block diagram of an example for deduplicating data blocks. In the example of FIG. 5, system 500 includes a processor 510 and a machine-readable storage medium 520. Although the following descriptions refer to a single processor and a single machine-readable storage medium, the descriptions may also apply to a system with multiple processors and multiple machine-readable storage mediums. In such examples, the instructions may be distributed (e.g., stored) across multiple machine-readable storage mediums and the instructions may be distributed (e.g., executed by) across multiple processors.

[0036] Processor 510 may be one or more central processing units (CPUs), microprocessors, and/or other hardware devices suitable for retrieval and execution of instructions stored in machine-readable storage medium 520. In the particular example shown in FIG. 5, processor 510 may fetch, decode, and execute instructions 522, 524, 525, 528, 530, 532 to perform deduplication of data blocks.

[0036] As an alternative or in addition to retrieving and executing instructions, processor 510 may include one or more electronic circuits comprising a number of electronic components for performing the functionality of one or more of the instructions in machine-readable storage medium 520. With respect to the executable instruction representations (e.g., boxes) described and shown herein, it should be understood that part or all of the executable instructions and/or electronic circuits included within one box may, in alternate examples, be included in a different box shown in the figures or in a different box not shown.

[0037] Machine-readable storage medium 520 may be any electronic, magnetic, optical, or other physical storage device that stores executable instructions. Thus, machine-readable storage medium 520 may be, for example, Random Access Memory (RAM), an Electrically-Erasable

Programmable Read-Oniy Memory (EEPROM), non-volatile memory, a storage drive, an optical disc, and the like. Machine-readable storage medium 520 may be disposed within system 500, as shown in FIG. 5. Machine-readable medium 520 is non-transitory in various examples. In this situation, the executable instructions may be "installed" on the system 500. Alternatively, machine- readable storage medium 520 may be a portable, external or remote storage medium, for example, that allows system 500 to download the instructions from the portable/external/remote storage medium,

[0038] Referring to FIG. 5, stream receiving instructions 522 when executed by a processor, e.g. processor 510, may cause processor 510 to receive a stream of data blocks comprising hashes associated with the data blocks. Localization metadata receiving instructions 524 when executed, may cause processor 510 to receive localization metadata comprising sets of the hashes.

[0039] Data block identification instructions 526, when executed, may cause processor 510 to identify the data blocks based on the hashes of the stream. Logical entity identification instructions 528, when executed, cause processor 510 to identify data blocks of logical entities based on the hash sets.

[0040] Reconstitution instructions 530, when executed, may cause processor 510 to reconstitute the Iogicai entities from the stream based on the hashes and the hash sets. Deduplication instructions 532 when executed, may cause processor 510 to deduplicate at least some of the data blocks of the reconstituted logical entities.

Claims

1 . A deduplicating method comprising:

identifying logical entities stored on a storage device;

identifying blocks of data that comprise the logical entities;

generating hashes for each of the data blocks;

determining hash sets comprising at least some of the hashes, wherein each of the hash sets are associated with one of the logical entities,

wherein each of the hash sets indicates blocks of the associated logical entity;

generating localization metadata which comprises the hash sets;

transmitting, to a computing device, the localization metadata; and transmitting, to the computing device, the blocks of data that comprise the logical entities.

2 The method of claim 1 , comprising:

transmitting, to the computing device, the localization metadata separately from the blocks of data.

3. The method of claim 1 , wherein generating the hashes comprises generating the hashes by a backup agent executing on a virtual machine.

4. The method of claim 1 , wherein the method is performed by a virtual machine, the method comprising:

executing, by the virtual machine, a backup agent; and

generating, by the backup agent, the localization metadata.

5. The method of claim 1 ,

wherein each of the hash sets indicates a sequence of the associated blocks of the logical entity.

6. The method of claim 1 , wherein the localization metadata comprises magic data, wherein the magic data comprises an identifier value of the localization metadata.

7. The method of claim 1 , wherein at least some of the hashes are partial hashes,

8. A device comprising:

a processor; and

a non-transitory medium storing instructions thereon that, when executed, cause the processor to:

receive a stream of data blocks;

receive localization metadata that indicates information about logical entities of the stream;

reconstitute the logical entities from the stream of data blocks based on localization metadata and hashes derived from the data blocks; and

dedupiicate at least some of the data blocks of the reconstituted logical entities.

9. The device of claim 8, wherein the stream comprises data blocks of a virtual machine or a snapshot of a virtual machine, and

wherein the stream is not ordered based on logical entities.

10. The device of claim 8, wherein at least some of the hashes are partial hashes.

1 1 . The device of claim 8,

wherein the medium comprises instructions stored thereon that, when executed, cause the processor to:

determine a logical entity that is not associated with any of the hash sets; and

deduplicate data blocks of the determined logical entity that are not associated with any of the hash sets.

12. The device of claim 8, further comprising a dedupiication store,

wherein the localization metadata comprises hash sets, wherein the hash sets indicate an ordering of data blocks of the logical entities,

the medium further comprising instructions stored thereon that, when executed, cause the processor to:

deduplicate the at least some of the data blocks of the reconstituted logical entities responsive to determining that the at least some of the

reconstituted logical entities are stored in the dedupiication store.

13. A non-transitory machine-readable storage medium encoded with instructions, the instructions that, when executed, cause a processor to:

receive a stream of data blocks comprising hashes associated with the data blocks;

receive localization metadata comprising sets of the hashes;

identify the data blocks based on the hashes of the stream;

identify data blocks of logical entities based on the hash sets;

reconstitute the logical entities from the stream based on the hashes and the hash sets; and

deduplicate at least some of the data blocks of the reconstituted logical entities.

14. The non-transitory machine-readable storage medium of claim 13, wherein the processor to: receive the localization metadata prior to receiving the stream of blocks.

15. The non-transitory machine-readable storage medium of claim 13, wherein the localization metadata indicates an ordering of blocks of the logica entities of the stream.