WO2024046554A1

WO2024046554A1 - Parallel deduplication mechanism on sequential storage media

Info

Publication number: WO2024046554A1
Application number: PCT/EP2022/074201
Authority: WO
Inventors: Yair Toaff; Assaf Natanzon; Idan Zach; Aviv Kuvent; Michael Sternberg
Original assignee: Huawei Technologies Co., Ltd.
Priority date: 2022-08-31
Filing date: 2022-08-31
Publication date: 2024-03-07

Abstract

There is provided a device for management of storage-sets. Target data is divided into data segments and fingerprints are computed. A set of representatives is selected. A global index that maps between representatives' fingerprints and storage-sets is accessed, for identifying storage-set(s) that include multiple media likely storing a duplicate of segments. A media set specific deduplication mechanism is accessed for identifying duplicates of segments stored on multiple media of the identified storage-set. Non-duplicated segments of the target data are stored on the multiple media of the storage-set, optionally in parallel. For respective identified duplicated segments, the media set specific deduplication mechanism(s) of the media of the identified storage-set are updated for indicating the newly stored non-duplicated segments of the target data. The global index, a global file mapping, and an identification table are updated for enabling reading of the target data by reading the segments from the storage-set.

Description

PARALLEL DEDUPLICATION MECHANISM ON SEQUENTIAL STORAGE MEDIA

BACKGROUND

The present disclosure, in some embodiments thereof, relates to storage systems and, more specifically, but not exclusively, to management of deduplication based storage systems storing data on sequential storage media.

Deduplication (dedup) based storage systems eliminate redundant storage of same chunks of data, by storing a reference that points to a previously stored chunk of data, rather than storing another copy of the same chunk of data. However, even with dedup, the need for storage is reaching such volumes that disks are no longer big enough to accommodate needs. The focus for secondary storage and archiving is moving from disk with dedup to sequential storage media such as tape technologies due to the lower price of tape and/or long time of data retention. Since the seek time in tapes is relatively long (10s of seconds), most applications use tapes without any optimizations beside the built-in compression.

SUMMARY

It is an object of the present disclosure to provide a computing device, a system, a computer program product, and a method for management of sequential storage media accessed by multiple sequential access storage devices for deduplication storage of target data.

The foregoing and other objects are achieved by the features of the independent claims. Further implementation forms are apparent from the dependent claims, the description and the figures.

According to a first aspect, a device for deduplication storage of data, comprises: a processor configured for: accessing target data for storage on a plurality of storage-sets, each storage-set including a plurality of media of a plurality of sequential access storage devices, accessing a global index mapping between each of the storage-sets and data segments stored thereon, for identifying a storage-set that includes a plurality of media likely storing a duplicate of at least one segment of the target data, accessing a media set specific deduplication mechanism for identifying duplicates of segments of the target data stored on the plurality of media of the identified storage-set, storing non-duplicated segments of the target data on the plurality of media of the identified storage-set, updating the media set specific deduplication mechanisms of the plurality of media of the storage-set for indicating the identified duplicated segments and the stored non-duplicated segments of the target data, updating the global index for indicating mappings between the segments of the target data used during the accessing, updating a global file mapping for storing identifiers for the plurality of segments of the target data, and updating an identification table for storing metadata for each identifier.

According to a second aspect, a method for deduplication storage of data, comprises: accessing target data for storage on a plurality of storage-sets, each storage-set including a plurality of media of a plurality of sequential access storage devices, accessing a global index mapping between each of the storage-sets and data segments stored thereon, for identifying a storage-set that includes a plurality of media likely storing a duplicate of at least one segment of the target data, accessing a media set specific deduplication mechanism for identifying duplicates of segments of the target data stored on the plurality of media of the identified storage-set, storing non-duplicated segments of the target data on the plurality of media of the identified storage-set, updating the media set specific deduplication mechanisms of the plurality of media of the storage-set for indicating the identified duplicated segments and the stored nonduplicated segments of the target data, updating the global index for indicating mappings between the segments of the target data used during the accessing, updating a global file mapping for storing identifiers for the plurality of segments of the target data, and updating an identification table for storing metadata for each identifier.

According to a third aspect, a non-transitory medium storing program instructions for deduplication storage of data, which, when executed by at least one processor, cause the at least one processor to: access target data for storage on a plurality of storage-sets, each storage-set including a plurality of media of a plurality of sequential access storage devices, access a global index mapping between each of the storage-sets and data segments stored thereon, for identifying a storage-set that includes a plurality of media likely storing a duplicate of at least one segment of the target data, access a media set specific deduplication mechanism for identifying duplicates of segments of the target data stored on the plurality of media of the identified storage-set, store non-duplicated segments of the target data on the plurality of media of the identified storage-set, update the media set specific deduplication mechanisms of the plurality of media of the storage-set for indicating the identified duplicated segments and the stored non-duplicated segments of the target data, update the global index for indicating mappings between the segments of the target data used during the accessing, update a global file mapping for storing identifiers for the plurality of segments of the target data, and update an identification table for storing metadata for each identifier.

At least some embodiments described herein address the technical problem of increasing throughput of writing data for storage on media of sequential access storage devices, for example, tape cartridges of tape drives. Access to the data is slow due to the sequential access nature of the tape. At least some embodiments described herein improve throughput of writing data to media of sequential access storage devices (e.g., tape cartridges) by writing target data to multiple media of sequential access storage devices instead of writing the data to a single media.

In a further implementation form of the first, second, and third aspects, the nonduplicated segments of the target data are stored in parallel on the plurality of media of the identified storage-set.

Parallel storage of the target data on multiple media of the storage-set improves performance in comparison to storage of the target data on a single media. By using a set of multiple media, the data may be written to multiple media in parallel. This can increase write performance up to a factor of the number of media in the set, for example, for a set of 4 cartridges, performance is increased up to 4 times over a single cartridge.

In a further implementation form of the first, second, and third aspects, the processor is further configured for: dividing the target data into a plurality of data segments, computing a plurality of fingerprints for the plurality of data segments, selecting a representative subset of the plurality of fingerprints, wherein the global index is accessed for the representative subset for identifying the storage-set, wherein a media of the storage-set for deduplicated storage of a data segment is selected according to the fingerprint of the data segment.

The fingerprint links the data segment to the media it is stored on in a deterministic and repeatable manner.

In a further implementation form of the first, second, and third aspects, the media of the storage-set for deduplicated storage is selected by performing a modulo operation to find a remainder when the fingerprint is divided by the number of media in the storage-set, wherein the remainder indicates the selected media.

The modulo operation guarantees that a segment is only found on a single media of the storage-set, and always the same one. When a cryptographic hash (e.g., SHA1) or other function that generates random-like values is used to compute the fingerprint, the modulo operation spreads the segments substantially evenly among the media of the storage-set.

In a further implementation form of the first, second, and third aspects, each metadata stored on the identification table includes an indication of the storage-set on which the data segment is stored, offset of the media, size of the data segment, and fingerprint of each of the plurality of data segments, wherein the fingerprint is used for determining the media of the storage-set.

The same fingerprint used to search the indexes may be used to select which media to store the data segment on.

In a further implementation form of the first, second, and third aspects, a number of media of each storage-set is preconfigured and fixed.

The number of media for each storage-set may be selected to improve parallel read/write performance. Once the number of media per storage-set is selected, and the storageset has been written to, the number cannot be easily changed.

In a further implementation form of the first, second, and third aspects, each media is assigned to a single storage-set.

Assigning each media to only one storage-set improves performance, by eliminating complexity that would otherwise arise from different sets accessing the same media.

In a further implementation form of the first, second, and third aspects, the storing is done by appending the non-duplicated segments to the plurality of media of the storage-set. In a further implementation form of the first, second, and third aspects, the processor is further configured for: receiving a request to read the target data, accessing the global file map for identifiers of the plurality of data segments of the target data, accessing an identification table for obtaining metadata of each data segment of the plurality of data segments, planning an order for parallel reading the plurality of data segments according to the metadata, reading the plurality of data segments of the target data according to the order and in parallel, and creating the target data from the plurality of data segments.

In a further implementation form of the first, second, and third aspects, each metadata includes an indication of the storage-set on which the data segment is stored, offset of the media, size of the data segment, and fingerprint of the data segment, wherein the fingerprint is used for determining the media of the storage-set.

According to a fourth aspect, a device for restoration of data, comprises: a processor configured for: receiving a request to read target data stored on a storage-set of a plurality of storage-sets, each storage-set including a plurality of media of a plurality of sequential access storage devices, obtaining identifiers of a plurality of data segments of the target data from a global file mapping, obtaining metadata of each data segment of the plurality of data segments from an identification table, planning an order for parallel reading the plurality of data segments according to the metadata, reading the plurality of data segments of the target data from the plurality of media of the storage-set according to the order and in parallel, and creating the target data from the plurality of data segments.

According to a fifth aspect, a method for restoration of data, comprises: receiving a request to read target data stored on a storage-set of a plurality of storage-sets, each storage-set including a plurality of media of a plurality of sequential access storage devices, obtaining identifiers of a plurality of data segments of the target data from a global file map, obtaining metadata of each data segment of the plurality of data segments from an identification table, planning an order for parallel reading the plurality of data segments according to the metadata, reading the plurality of data segments of the target data according to the order and in parallel, and creating the target data from the plurality of data segments.

According to a sixth aspect, a non-transitory medium storing program instructions for restoration of data, which, when executed by at least one processor, cause the at least one processor to: receive a request to read target data stored on a storage-set of a plurality of storagesets, each storage-set including a plurality of media of a plurality of sequential access storage devices, obtain identifiers of a plurality of data segments of the target data from a global file map, obtain metadata of each data segment of the plurality of data segments from an identification table, planning an order for parallel reading the plurality of data segments according to the metadata, read the plurality of data segments of the target data according to the order and in parallel, and create the target data from the plurality of data segments.

At least some embodiments described herein address the technical problem of increasing throughput of reading data stored on media of sequential access storage devices, for example, tape cartridges of tape drives. Access to the data is slow due to the sequential access nature of the tape. At least some embodiments described herein improve throughput of reading data from media of sequential access storage devices (e.g., tape cartridges) by reading target data from multiple media of sequential access storage devices instead of reading the data from a single media.

In a further implementation form of the fourth, fifth, and sixth aspects, each metadata includes a fingerprint of the data segment, and wherein the processor is further configured for determining the media of the storage-set according to the fingerprint.

Parallel reading of the target data from multiple media of the storage-set improves performance in comparison to reading of the target data from a single media. By using a set of multiple media, the data may be read from the multiple media in parallel. This can increase read performance up to a factor of the number of media in the set, for example, for a set of 4 cartridges, performance is increased up to 4 times over a single cartridge.

In a further implementation form of the fourth, fifth, and sixth aspects, the processor is further configured for determining the media by performing a modulo operation to find a remainder when the fingerprint is divided by the number of media in the storage-set, wherein the remainder indicates the media.

In a further implementation form of the fourth, fifth, and sixth aspects, the processor is further configured for: for each of the plurality of media, arranging data segments for sequential reading according to location and offset of the data segment. Unless otherwise defined, all technical and/or scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the disclosure pertains. Although methods and materials similar or equivalent to those described herein can be used in the practice or testing of embodiments of the disclosure, exemplary methods and/or materials are described below. In case of conflict, the patent specification, including definitions, will control. In addition, the materials, methods, and examples are illustrative only and are not intended to be necessarily limiting.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

Some embodiments of the disclosure are herein described, by way of example only, with reference to the accompanying drawings. With specific reference now to the drawings in detail, it is stressed that the particulars shown are by way of example and for purposes of illustrative discussion of embodiments of the disclosure. In this regard, the description taken with the drawings makes apparent to those skilled in the art how embodiments of the disclosure may be practiced.

In the drawings:

FIG. l is a block diagram of components of a system for deduplication storage of target data on a storage-set of sequential storage media accessed by sequential access storage devices, in accordance with some embodiments;

FIG. 2 is a block diagram of another exemplary system for deduplication storage of target data on a storage-set of sequential storage media (e.g., tape), in accordance with some embodiments;

FIG. 3 is a flowchart of a method of deduplication storage of target data on a storageset of sequential storage media accessed by sequential access storage devices, in accordance with some embodiments;

FIG. 4 is a flowchart of a method of reading target data from a storage-set of sequential storage media accessed by sequential access storage devices, in accordance with some embodiments; and

FIG. 5 is a schematic depicting storage of data segments of a target data (e.g., file) on a storage-set of a plurality of media read by one or more sequential access storage devices, in accordance with some embodiments the present invention. DETAILED DESCRIPTION

An aspect of some embodiments relates to systems, methods, a computing device and/or apparatus, and/or computer program product (storing code instructions executable by one or more processors) for management of storage-sets of multiple sequential storage media (also referred to herein as “media”) of multiple sequential access storage devices for deduplication storage of target data thereon. The sequential access storage devices, which are not randomly accessible, may be, for example, a tape library with hundreds of tape drives and thousands of magnetic tape cartridges. The target data is divided into multiple data segments (also referred to herein as “segments”). Fingerprints are computed for the data segments, for example, by feeding the data segments into a hash function. A set of representatives is selected from the fingerprints. A global index that maps between representatives’ fingerprints and storage-sets is accessed, for identifying one or more storage-sets that include multiple media likely storing a duplicate of one or more segments of the target data. The global index may indicate a likelihood of a storage-set of multiple media storing the duplicated segment(s) rather than a certainty. A media set specific deduplication mechanism is accessed for identifying duplicates of segments of the target data stored on multiple media of the identified storage-set. The media set is treated as a single entity, until the actual IO is performed in read and/or write. Each respective media set specific deduplication mechanism(s) may indicate a certainty of whether a certain segment is stored on the respective storage-set. Non-duplicated segments of the target data are stored on the multiple media of the storage-set, optionally in parallel. For respective identified duplicated segments, the media set specific deduplication mechanism(s) of the media of the identified storage-set are updated for indicating the newly stored non-duplicated segments of the target data. The global index is updated with mappings between the segments of the target data used during the accessing. A global file mapping is updated by including identifiers of the segments of the target data. An identification table is updated by including metadata for each new segment identifier.

An aspect of some embodiments relates to systems, methods, a computing device and/or apparatus, and/or computer program product (storing code instructions executable by one or more processors) for restoration of data stored on a storage set that includes multiple media. A request to read target data is received. Identifiers of data segments of the target data are obtained from a global file map. Metadata of each data segment is obtained from an identification table. An order is planned for parallel reading the data segments according to the metadata. The data segments of the target data are read according to the order and in parallel. The target data is created from the data segments.

At least some embodiments described herein address the technical problem of increasing throughput of writing data for storage on media of sequential access storage devices, for example, tape cartridges of tape drives. Access to the data is slow due to the sequential access nature of the tape. At least some embodiments described herein improve throughput of writing data to media of sequential access storage devices (e.g., tape cartridges) by writing target data to multiple media of sequential access storage devices, in parallel (e.g., entirely in parallel, partially in parallel), instead of writing the data to a single media.

At least some implementations described herein improve a deduplication ratio for storage of large amounts of data on multiple data storage devices, optionally tape. For example, on a tape based data storage system that stores about 100 times the amount of data than a disk based storage system. The global index described herein enables implementing deduplication on multiple tapes arranged into storage-sets where each storage-set includes two or more tapes, for example, in comparison to other approaches that do not implement dedup on tapes, and/or other approaches with low dedup ratios.

At least some embodiments described herein address the technical problem of, and/or improve the technical field of, storage of large amounts of data, in particular on non-random accessible storage media and/or sequentially accessible media such as tapes. Management of data stored on tapes is technically challenging due to the size of tape storage. For example, while the largest secondary storage disk systems are in the petabyte (PB) scale, large tape libraries contain thousands of cartridges and reach the scale of hundreds of PB. This size is doubling itself every about 2-3 years as new linear tape-open (LTO) specifications are introduced, which doubles the size per cartridge (e.g., current version is LTO-9 18TB cartridge). Although different dedup systems exists, such solutions are designed for disk-based storage systems (i.e., randomly accessible data storage devices such as hard-disks and/or solid state drives (SSD)). Dedup designed for disk-based storage cannot be easily migrated to tape storage due to the larger size of the data being stored (e.g., at least about xlOO relative to disk storage) and/or the different ways of operating tape versus disk storage.

Existing approaches for management of data stored on tapes exist. In the most commonly used approach, no deduplication is used. When the data is written to the tape, the data is written in the original (i.e., non deduped) form. Even if the data is moved from a deduped form (e.g., on disk), the data is first inflated to the original non-dedup form, and stored in a nondedup format. In another example of an approach, the data is sent to the tape in a way that the data sent together is deduped only with other parts of this group. For such operation, the data is processes to find the matches inside the group, which are saved as a unit. For example, one such approach focuses on how this group of data may be saved in a way that reduces the restore time. In another example, a full backup and a series of incrementals are created and saved together as a unit (termed SILO). The limitation imposed by such approaches is that a closed group is required. Such closed group requires significant amount of computational preprocessing, and the dedup ratio is significantly reduced. For example, the SILO method with a monthly full (with the assumption of 2% daily change that partly falls on previous changes) will get a dedup ratio close to 1 : 1 since the incrementals have a low common blocks with the full (i.e., if they were not changed they were not included in the incremental). The main gain will be from deduping the full backups against each other. A year of monthly fulls and daily incrementals will obtain about a 1 : 12 ratio since the fulls are also much larger (in the regular cases).

Before explaining at least one embodiment of the disclosure in detail, it is to be understood that the disclosure is not necessarily limited in its application to the details of construction and the arrangement of the components and/or methods set forth in the following description and/or illustrated in the drawings and/or the Examples. The disclosure is capable of other embodiments or of being practiced or carried out in various ways. The present disclosure may be a system, a method, and/or a computer program product. The computer program product may include a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to carry out aspects of the present disclosure.

The computer readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device. The computer readable storage medium may be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing.

Computer readable program instructions described herein can be downloaded to respective computing/processing devices from a computer readable storage medium or to an external computer or external storage device via a network, for example, the Internet, a local area network, a wide area network and/or a wireless network.

The computer readable program instructions may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate arrays (FPGA), or programmable logic arrays (PLA) may execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present disclosure.

Aspects of the present disclosure are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the disclosure. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer readable program instructions. The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logical function(s). In some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems that perform the specified functions or acts or carry out combinations of special purpose hardware and computer instructions.

Reference is now made to FIG. 1, which is a block diagram of components of a system 100 for deduplication storage of target data 112A on a storage-set of sequential storage media 160 accessed by sequential access storage devices 150, in accordance with some embodiments. Reference is also made to FIG. 2, which is a block diagram of another exemplary system 200 for deduplication storage of target data on a storage-set of sequential storage media (e.g., tape) 252, in accordance with some embodiments. Reference is also made to FIG. 3, which is a flowchart of a method of deduplication storage of target data on a storage-set of sequential storage media accessed by sequential access storage devices, in accordance with some embodiments. Reference is also made to FIG. 4, which is a flowchart of a method of reading target data from a storage-set of sequential storage media accessed by sequential access storage devices, in accordance with some embodiments. Reference is also made to FIG. 5, which is a schematic depicting storage of data segments of a target data (e.g., file) on a storage-set of a plurality of media read by one or more sequential access storage devices, in accordance with some embodiments the present invention.

Systems 100 and/or 200 may implement the acts of the method described with reference to FIGs. 3-5, for example, by processor(s) 102 of a computing device 104 executing code instructions (e.g., code 106A) stored in a memory 106. Computing device 104 obtains target data 112A for deduplication storage on a storageset of sequential access storage device(s) 150, optionally on media 160 accessed by the sequential access storage device 150.

The sequential access storage device 150 and/or media 160 may be referred to herein as sequential access memory and/or non-random access memory, for example, a tape tier.

Media 160 may be referred to herein as sequential access storage media and/or nonrandom access storage media. The sequential access storage device read and/or write data sequentially, i.e., do not provide random access for reading and/or writing data in any order (e.g., as in random access memory). Media 160 may include cartridges of magnetic storage media such as tapes, sequential access memory (SAM), and/or optical storage media. The sequential access storage device may include a tape drive and/or optical drive for optical storage media.

Target data 112A may be obtained, for example, from client terminal(s) 112 and/or server(s) 110. Computing device 104 processes target data 112A, for example, performing the calculation for segmentation, computation of fingerprints (e.g., hashing), and/or maintaining the different data structures that are accessed, as described herein. A global index 116A is accessed to identify which storage-set of multiple media 160 likely stores duplicated segments of target data 112A, as described herein. For each identified storage-set, a respective target specific deduplication mechanism 150A is accessed to confirm the duplicated data segments of target data 112 A. Non-duplicated segments are stored on the multiple media of the storage-set 160, optionally in parallel. The global index 116A, a global file mapping 116B, the relevant media set specific deduplication mechanism(s) 150A and associated identification table 150B (e.g., metadata) are updated to indicate the presence of the duplicated segments and/or the newly stored non-duplicated segments, to enable reading the segments of the target data optionally in parallel.

Computing device 104 may be implemented as, for example, one of more of a computing cloud, a cloud network, a computer network, a virtual machine(s) (e.g., hypervisor, virtual server), a network node (e.g., switch, a virtual network, a router, a virtual router), a single computing device (e.g., client terminal), a group of computing devices arranged in parallel, a network server, a web server, a storage server, a local server, a remote server, a client terminal, a mobile device, and a stationary device.

Different architectures of system 100 may be implemented. For example, computing device 104 may act as a server, providing centralized storage to target data 112A obtained from different client terminals 112 and/or servers 110. In another example, computing device 104 may act as a local manager of media 160, for local storage of target data 112 A.

Computing device 104 includes one or more processor(s) 102, implemented as for example, central processing unit(s) (CPU), graphics processing unit(s) (GPU), field programmable gate array(s) (FPGA), digital signal processor(s) (DSP), application specific integrated circuit(s) (ASIC), customized circuit(s), processors for interfacing with other units, and/or specialized hardware accelerators. Processor(s) 102 may be implemented as a single processor, a multi-core processor, and/or a cluster of processors arranged for parallel processing (which may include homogenous and/or heterogeneous processor architectures). It is noted that processor(s) 102 may be designed to implement in hardware one or more features stored as code instructions 106 A.

Memory 106 stores code instructions executable by processor(s) 202, for example, a random access memory (RAM), dynamic random access memory (DRAM) and/or storage class memory (SCM), non-volatile memory, magnetic media, semiconductor memory devices, hard drive, removable storage, and optical media (e.g., DVD, CD-ROM). Memory 106 stores, for example, one or more of code 106A that implements one or more features and/or acts of the method described with reference to FIGs. 3-5 when executed by processor(s) 102. Memory may be selected to be large enough to store global index 116A.

Computing device 104 may include a data storage device 116 for storing data, for example, a queue accumulating target data 112A for storage on media 160, global index 116A, global file mapping 116B, and/or one or more media set specific deduplication mechanisms 150A which may each be associated with an identification table 150B, as described herein. Data storage device 116 may be implemented as, for example, as a fast tier of storage such as a solid state drive (SSD), a local hard-drive, virtual storage, a solid state device, a storage device, and/or as a remote server and/or computing cloud (e.g., accessed using a network connection). Computing device 104 is in communication with one or more sequential access storage devices 150 that access (i.e., read and/or write) on multiple media arranged into storage-sets 160. Sequential access storage devices 150 may be implemented as, for example, tape drives. Sequential access media 160 may be implemented, for example, as a tape library, tape cartridges (e.g., cartridges in drives, and/or cartridges in slots), for example, LTO-9 cartridges that store 18 terabytes (TB). Sequential access storage device 150 may include built-in compression, for example, an average of 1 :2.5 as in the LTO-9 specification.

Computing device 104 may include a network interface 118 for connecting to network 114, for example, one or more of, a network interface card, a wireless interface to connect to a wireless network, a physical interface for connecting to a cable for network connectivity, a virtual interface implemented in software, network communication software providing higher layers of network connectivity, and/or other implementations.

Network 114 may be implemented as, for example, the internet, a local area network, a virtual private network, a wireless network, a cellular network, a local bus, a point to point link (e.g., wired), and/or combinations of the aforementioned.

Computing device 104 may communicate with sequential access storage devices 150 (e.g., for reading data from media arranged into storage-sets 160 and/or storing data on media arranged into storage-sets 160), for example, via network interface 118 over network 114, and/or via another interface such as a virtual and/or physical interface, using wireless and/or physical cable links.

Computing device 104 may include and/or be in communication with one or more physical user interfaces 108 that include a mechanism for a user to enter data and/or view data. Exemplary user interfaces 108 include, for example, one or more of, a touchscreen, a display, a keyboard, a mouse, and voice activated software using speakers and microphone.

Referring now back to FIG. 1, components described with reference to system 200 may correspond to, be combined with, include, and/or may represent an exemplary implementation of components of system 100 described with reference to FIG. 1. System 200 may include a fast storage tier 202, which includes one or more fast data storage devices 204, for example, SSD(s). Fast storage tier 202 may store, for example, target data which is being processed for storage on a tape storage tier 250. It is noted that tapes represent an exemplary implementation of sequential access storage devices and/or sequential storage media, and tape drives and/or cartridges is not necessarily limiting, as other implementations of sequential access storage devices and/or sequential storage media. System 200 includes tape storage tier 250, which includes a global index 216A (e.g., global sparse index), as described herein. Global sparse index 216A may be stored on a fast data storage device (e.g., SSD(s) 206). Tape storage tier 250 includes multiple tape cartridges 252, where each tape cartridge 252 is read from and/or written to by a tape drive (not shown). Tape cartridges 252 are arranged in storage-sets 260, where each storage-set includes two or more tapes. Each storage-set 260 of multiple tape cartridges 252 (two are shown as an example, and it is to be understood there may be more) is associated with a media set specific deduplication mechanism, as described herein. Each storage-set 260 of tapes is associated with a sparse index 254 and a full index 256, as described herein. It is noted that other deduplication mechanisms described herein may be used, and that the sparse index and full index represent one not necessarily limiting example. Each set of sparse index 254 and full index 256 may be stored, for example, on fast data storage device 206.

Referring now back to FIG. 3, at 300, storage-sets may be configured, and/or a predefined configuration of the storage-sets is accessed.

The configuration of the storage sets may include defining a number of media of each storage-set. The number of media per storage-set may be, for example, 2, 3, 4, 5, or more. The number of media for each storage-set may be selected to improve parallel read/write performance. The number of media in each storage-set may be limited such as by a maximum value, to avoid too many media involved in the case of data recovery, which may be inefficient. Inventors discovered that a relatively lower number, for example, in the range of 2-4 provides optimal performance for tape cartridge storage systems.

Once the number of media per storage-set is defined, and the storage-set has been written to, the number cannot be easily changed.

The number may be defined as a global value, applicable to all storage-sets. Alternatively, the number may be defined per storage-set, or for a group of storage-sets. Each media may be assigned to a single storage-set. I.e., the same media is not necessarily assigned to two different storage-sets. Assigning each media to only one storageset improves performance, by eliminating complexity that would otherwise arise from different sets accessing the same media.

Using multiple media may improve memory utilization and/or reduce memory consumption by reducing cases of multiple media that store the same data segment in the global sparse index.

At 302, target data for storage on media of sequential access storage devices is obtained. The target data may be, for example, data for archiving.

At 303, the target data may be processed, by dividing into segments, computing fingerprints for the segments, and selecting representatives of the segments.

The target data may be divided into smaller parts, referred to herein as segmentation, i.e., segmenting the target data into data segments (target segments).

The target data may be segmented, for example, into constant size data segments and/or into variable size data segments. Constant size segmentation is the process of dividing the data to segment of the exact same size (e.g., 16KB, or 32KB, or other sizes). It is noted that constant segmentation is not widely used in deduplication based systems since a small insertion or deletion of a few bytes will change all the segments from the point of change on, and no deduplication will occur. Variable size segmentation is the process of dividing the data to segments according to a window of data. For example, a rolling hash function is used on a rolling window of 64-256 bytes. When the hash value of the rolling hash accommodates a certain condition, the segment is cut and a new segment is started. The condition may be selected in a way that the minimum/average/maximum sizes of the segments is the desired values.

Representations of the target data may be extracted. Optionally, samples of the data segments are identified, and the representations are computed for the sampled data segments.

As used herein, the term fingerprint (or data fingerprint) refers to representations extracted from the target data and/or from data stored on sequentially accessed media described herein. The fingerprints may be extracted, for example, by feeding data segments sampled from the target data into a strong hash function (e.g., SHA1 that outputs 20-bype crypto hash, or newer versions of secure hash algorithm (SHA) that output longer hash values). The fingerprints are used for fast comparison of the target data to fingerprints of the global index and/or index(es) of the media set specific deduplication mechanisms, i.e., rather than comparing the actual target data byte by byte. It is noted that the chance of a false positive comparison may be selected. For example, when using a strong hash, the chance of false positive is very low (for a 1PB storage with segments of ~16 kilobytes (KB) 2³⁶ segments are obtained, where the SHA1 signature is 20 bytes = 2160, which gives a false positive chance of 1 :2¹²⁴).

It is noted that the target data may be segmented and/or hashed with the same segmentation and/or hashing used by the media set specific deduplication mechanisms of the media. The segmentation and/or hashing may be skipped, for example, when the same values were used for both disk deduplication and media deduplication, but in common cases the segments of the disk deduplication may be smaller.

Incoming target data may be written into a fast tier (with or without dedup), for example, a fast data storage device. Target data may be accumulated on the fast data storage device, for example, in a buffer. Target data may be moved to the media, for example, according to a threshold and/or data retention policy defined for the fast data storage device. A certain amount of the accumulated target data may be moved to the media. For example, target data may be moved in chunks of hundreds of GB each time. For example, size of the data storage device that stores the target data is 10TB with a threshold rule that whenever the data passes the 8TB mark, 2 TB of target data is moved to the media. Alternatively, the target data may be written directly to the media. The fast tier (or for example, HDDs inside the tape tier) may be used as a staging area until enough target data is written.

At 304, a global index is accessed.

The global index maps between representatives’ fingerprints and storage-sets having media on which data segments corresponding to the representatives’ fingerprints are stored. The global index is used to efficiently search for matching fingerprints, which indicate the storageset with highest number of likely matching fingerprints, for further evaluation to determine actual matches. One or more of the target segments of the target data (e.g., fingerprints thereof) are searched (e.g., looked-up) in the global index to determine whether a duplicate of the respective target segment is likely stored on one or more of the storage-sets, i.e., a candidate match with one or more of the target fingerprints. The fingerprints of the target data that may be searched include: all of the fingerprints, a subset of the fingerprints selected using a same approach for selecting entries for the global index and/or entries of a sparse index of the media set specific deduplication mechanism, or combination thereof such as if an insufficient number of matches are found.

The global index enables a fast lookup of the target dataset segments to quickly identify which storage-set requires closer inspection to verify storage of a duplicate segment. For example, an operation without a global index would require accessing all indexes of all deduplication mechanisms of all media of all storage-sets, which is computationally intensive and/or inefficient in particular when looking for duplications of a large number of data segments and/or on a large number of media storing a large amount of data.

A search result indicating one or more storage-sets (each storage-set includes two or more media) likely storing matching data segments that are duplicates of the target data segments, is obtained. For example, a list of storage-sets is obtained, where each storage-sets has a grade indicating the likeliness of finding matching segments in it. The best matching storage-sets may be selected, for example, a predefined number of storage-sets with highest number of matches (indicating highest number of data segments that are duplicates of the target segments) is selected, such as the best matching 1, 2, 3, 4 or 5 (or other number) of storage-sets. Further searches of the media set specific deduplication mechanisms may be limited to the best matching media to avoid excessively long search times and/or improve efficiency of the processor performing the search.

The global index may be stored on a high performance storage device (e.g., fast memory, such as RAM, SSD, and the like) in communication with the processor that accesses the global index, for example, as described herein. The global index may be stored on the fast memory to improve memory utilization and/or system performance, by enabling fast lookups to quickly identify which storage-sets are relevant for further analysis, and/or to verify storage of a duplicate of the respective target segment. Optionally, the global index is implemented as a sparse index. The sparse index includes sparsely obtained data segments (e.g., fingerprints thereof) selected from each one of the storage-sets, i.e., a subset of the fingerprints of data segments stored on the storage-sets. For example, 2 fingerprint entries are selected for every 4 megabyte (MB) of data, or other values. The sparse design of the global index enables storing samples of data segments from all of data segments stored on all of the storage-sets, which may be very large, for quickly identifying which storage-sets likely store a duplicate.

The global index, optionally the sparse index, may be implemented as a key-value database (KVDB) and/or other searchable data structure that includes entries for all storage-sets storing relevant data entries. The search key may be, for example, the fingerprint (or portion thereof), or a weak hash created from the fingerprint. For each key, there may be multiple values indicating one or more storage-sets that likely store a duplicate of the data segment represented by the key, i.e., the same data segment may be stored on one or multiple storage-sets.

The sparse index may map a weak hash of the finger print of a target data segment to one or more storage-sets, with a certain probability of a false positive result that requires verification to determine whether the target data segment is actually stored on the mapped storage-sets. The weak hash may be implemented as a part of the strong hash (for example, bytes 10-17 out of the 20 bytes full hash). The weak hash may be used for detecting the area in the data stored on a storage-set that has a high probability to contain similar data. It is noted that the weak hash is not necessarily used for determining actual identification of the duplicated data since the false positive chance is not necessarily low enough to confirm presence of the duplicated data.

Confirmation of the storage of the target segment on the storage-sets is provided by the media set specific deduplication mechanism of the mapped storage-sets, as described herein. The relatively smaller size of the weak hash design enables quickly determining the storage-sets that most likely store duplicates of the target data. Utilization of the memory storing the global index is improved using the relatively smaller size of the weak hash.

The data segments represented in the global index are sampled from data segments represented in multiple media set specific deduplication mechanisms of the multiple media of each storage-set. Sampling segments of the data represented in the multiple media set specific deduplication mechanisms for inclusion in the global index enables efficiently covering the large amount of data stored in the multiple media of the storage-sets, while improving utilization of memory storing the global index.

The global index, which is independent of the implementation of the deduplication mechanism of each storage-sets, provides compatibility with different implementations of deduplication mechanisms of the different storage-sets.

The values of the segment sizes and/or resolution of the global index may be configurable.

An exemplary computation of the global index as a sparse index is now provided. The global index includes a subset of entries of the indexes of the media set specific deduplication mechanisms of the multiple media of the storage-sets. There are M entries per 10GB for example, 10GB / M is » 1GB / N). Each entry of the global index may include the following fields: [weak hash (8 - 16 bytes), media ID (4 bytes)].

At 306, a media set specific deduplication mechanism is accessed for each respective media of the storage- set(s) identified by the global index as likely storing one or more duplicates of the target data segments thereon.

Each respective media set specific deduplication mechanism may be stored on a high performance storage device in communication with the specific media of the storage-set, to improve memory utilization and/or system performance, by enabling fast lookups to quickly confirm which media actually store duplicates of the target data segments. For example, each media set specific deduplication mechanism may be stored on its own independent high performance data storage device, which may enable parallel access to multiple media set specific deduplication mechanisms. In another example, multiple media set specific deduplication mechanism may be stored on a common high performance data storage device.

An exemplary approach for searching the media set specific deduplication mechanism is now described. Representations of the target data (e.g., fingerprints) are searched in a sparse index of the media set specific deduplication mechanism. For example, all of the fingerprints are searched, a subset of fingerprints indicated by the global index are likely being stored by the media are searched, a subset of fingerprints selected using the same approach for selecting fingerprints for inclusion in the sparse index is searched, or combination thereof such as if an insufficient number of matches are found. The search result indicate which areas of the media likely store data segments that are duplicates of the target data segments. Entries of the full index corresponding to matches of the sparse index are read and compared to the target data segments (e.g., fingerprints thereof) to determine whether matches are present. The matches indicate duplicates of the target data segments that are stored on the media.

Exemplary implementations of media set specific deduplication mechanisms which may be searched are now described. It is noted that there may be different implementations of different media set specific deduplication mechanisms for different media. The media set specific deduplication mechanism may include a full index, which includes all representations (e.g., fingerprints) of all data segments stored on the media. The full index may map a strong hash of a data segment of data stored on the media (e.g., fingerprint) to an offset storage location on the media, for all data segments stored on the media. The full index may be stored on a fast data storage device. An exemplary calculation for the size of the full index is now provided. A number of entries per full media = 45TB/segment size(45KB) = ~1G entries. Each entry includes the following fields: [ID (8 bytes), offset in media (8 bytes), size (4bytes), hash (20bytes SHA1)]. Without optimizations each entry is 40bytes. The total size for a full media (e.g., cartridge) is 40GB.

Since the full index is usually too large for efficient and/or fast search, one or more of the following data structures (associated with the full index) may be accessed to prior to accessing the full index:

* A sparse index that includes a sample of representations that point to a location in the full index likely to include one or more fingerprints of the target data. The full index is then accessed for comparing the full fingerprints for confirming presence of a duplication of the target data segment. An exemplary calculation for the size of the sparse index is now provided. The sparse index for the media is in a ratio of N hashes (e.g., part of the fingerprints of the strong hash) for each GB of data. The hashes may be selected, for example, according to local minimum or maximum (this ratio may be adjusted as a tradeoff between accuracy and size). The number of entries in the sparse index is 45TB x N /1GB = 45K x N. Each entry includes the following fields: [weak hash (8 - 16 bytes), block id in tape (4 bytes)]. The total size of a sparse index per full media is <1MB x N.

The number of data segments represented in the sparse index may be a defined percentage of the number of data segments represented in the full index, for example, about 1 data segment in the sparse index per 100 data segments of the full index, or other values. The number of data segments represented in the global index may be a defined percentage of the number of data segments represented in the sparse index of the media, for example, about 1 data segment in the global index per 100-1000 data segments of the sparse index of the media.

The media sparse index may reside, for example, as an array in the fast disk and/or as a separated table in a global sparse indexes DB (e.g., using a fast data storage device).

* A weak hash table that includes mappings from a weak hash (e.g., part of the full fingerprint) of a data segment to a segment ID of the data segment in the full index. When a weak hash is found in the weak hash table, a range of IDs around the found ID may be loaded to a cache and the fingerprints are searched for in the cache. If no weak hash is found, the process may return to the weak hash table until all the unfound fingerprints has been searched for in the weak hash table.

* A Bloom filter that indicates whether a fingerprint is likely stored in the full index. The Bloom filter may give false positive answers.

At 308, a respective media of the storage-set is selected for deduplicated storage of each data segment.

The media of the storage-set for deduplicated storage of a data segment is selected according to the fingerprint of the data segment. The same fingerprint used to search the indexes may be used to select which media to store the data segment on. The fingerprint links the data segment to the media it is stored on in a deterministic and repeatable manner.

Optionally, the media of the storage-set is selected by performing a modulo operation to find a remainder. When the fingerprint is divided by the number of media in the storage-set, the remainder indicates the selected media. The modulo operation guarantees that a segment is only found on a single media of the storage-set, and always the same one. When a cryptographic hash (e.g., SHA1) or other function that generates random-like values is used to compute the fingerprint, the modulo operation spreads the segments substantially evenly among the media of the storage-set.

The combination of identified storage-sets may be selected to obtain a highest target deduplication ratio of the target data under defined constraints. The constraints for selection of the combination of identified storage-sets may be, for example, a division of the target data into predefined units such as files, and/or divisions of a same unit being written to a same storageset for restoration of the predefined unit from a single storage-set. Ensuring that the division of the same unit of data (e.g., file) is stored on a same storage-set allows allow for efficient restoration of the unit from a single storage-set. For example, ensuring that a same file is reconstructed from a single storage-set is more efficient, and allows faster access to the reconstructed file, in comparison to restoration of a file spanning over multiple different storagesets which is less efficient.

At 310, the non-duplicated target segments are stored on the selected media of the storage-set.

The non-duplicated segments of the target data may be stored in parallel on the multiple media of the identified storage-set.

The storing may be done by appending the non-duplicated segments to each of the media of the storage- set.

The non-duplicated target segments are the portion of the target data for which no duplicates (i.e., no matching data segments) were found on the selected storage-set by a search of the media set specific deduplication mechanisms. At 312, the media set specific deduplication mechanism(s) of the selected storage-set(s) is/are updated for indicating the identified duplicated segments of the target stored on the selected storage-set, and for indicating the newly stored non-duplicated segments of the target data.

The updating is done for the full index and associated data structure (e.g., sparse index, weak hash table, Bloom filter) of each media set specific deduplication mechanism associated with the selected storage-set(s).

To make the data readable again, the updating includes adding metadata that indicates how to construct the target data from the different segments on the storage-set(s), including the newly added non-duplicated segments and the previously existing duplicated data segments. The metadata may be a map of the data segments stored on the storage-set(s) that are assembled for reconstruction of the target data. The metadata may be stored on the fast tier.

An identification table of the media set specific deduplication mechanism is updated for storing metadata for each new segment identifier. Each metadata for each data segment stored on the identification table may include, for example, one or more of an indication of the storageset on which the data segment is stored, offset of the media where the data segment is stored, size of the data segment, and fingerprint. An identifier of the media within the storage-set may be included in the metadata, and/or computed using the fingerprint, for example, as the remainder of applying a modulo operation to the fingerprint divided by the number of media in the storage-set. The fingerprint that is included in the metadata is the fingerprint used for determining the media of the storage-set, for example, as described with reference to 308.

At 314, the global index is updated for indicating mappings between the segments of the target data used during the accessing.

At 316, a global file mapping for storing identifiers for the segments of the target data, is updated. The identifiers may be a code, for example, a number, letter, symbol, and/or combination thereof, that uniquely identifies each data segments. Referring now back to FIG. 4, at 402, a request is received to read target data. The target data is stored on a storage- set that includes multiple media (e.g., cartridges), as described herein, for example, with reference to FIG. 3.

At 404, identifiers of data segments of the target data are obtained, for example, from a global file map.

At 406, metadata of each data segment is obtained, for example, from an identification table.

At 408, when each metadata includes a fingerprint of the data segment, the media of the storage-set is determined according to the fingerprint. For example, by performing a modulo operation to find a remainder when the fingerprint is divided by the number of media in the storage-set. The remainder indicates the media.

At 410, An order for parallel reading the data segments is planned according to the metadata. For each media of the storage-set, data segments are arranged for sequential reading according to location and offset of the data segment.

At 412, the data segments of the target data are read from the multiple media of the storage-set according to the order and in parallel. Parallel reading of the target data from multiple media of the storage-set improves performance in comparison to reading of the target data from a single media. By using a set of multiple media, the data may be read from the multiple media in parallel. This can increase read performance up to a factor of the number of media in the set, for example, for a set of 4 cartridges, performance is increased up to 4 times over a single cartridge.

At 414, the target data is created from the data segments. The target data is created by arranging the data segments, which were obtained from the different media of the storage-set, in order.

Referring now back to FIG. 5, the data segments described with FIG. 5 may be stored there, for example, as described with reference to FIG. 3, and/or may be read therefrom, for example, as described with reference to FIG. 4. Segments metadata 502 may include the metadata associated with the stored target data, for example, a file. Segments metadata 502 may include, for example, an indication of IDs of data segments of the file. Segments metadata 502 may include metadata for each ID (i.e., for each data segment), for example, ID of the media storing the data segment 504, the fingerprint (FP) 506, offset in the media to start of the data segment, and/or size of the data segment.

In the example depicted in FIG. 5, there are seven data segments, having IDs 1 to 7 (504). Each data segment is associated with a fingerprint 506, which may be stored as metadata and/or computed (e.g., by feeding into a hash, as described herein). The data segments are stored on a storage-set, which includes a predefined number of media (e.g., tape cartridges) 508, for example, 4 cartridges 510A-D in the depicted example.

A mapping of each data segment to the cartridge of the storage-set storing that data segment 512 may be stored (e.g., as metadata), and/or dynamically computed. The mapping may be computed as a modulo operation that computes a remainder from dividing the value of the fingerprint by the number of cartridges in the storage-set. The remainder indicates the cartridge of the storage-set on which the data segment is stored. For example, for data segment having ID =1, the fingerprint is 1234. 1234 mod (%) 4 = 2, indicating that data segment ID=1 is stored on cartridge 2, as shown.

Other systems, methods, features, and advantages of the present disclosure will be or become apparent to one with skill in the art upon examination of the following drawings and detailed description. It is intended that all such additional systems, methods, features, and advantages be included within this description, be within the scope of the present disclosure, and be protected by the accompanying claims.

The descriptions of the various embodiments of the present disclosure have been presented for purposes of illustration, but are not intended to be exhaustive or limited to the embodiments disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terminology used herein was chosen to best explain the principles of the embodiments, the practical application or technical improvement over technologies found in the marketplace, or to enable others of ordinary skill in the art to understand the embodiments disclosed herein.

It is expected that during the life of a patent maturing from this application many relevant deduplication mechanisms will be developed and the scope of the term deduplication mechanism is intended to include all such new technologies a priori.

As used herein the term “about” refers to ± 10 %.

The terms "comprises", "comprising", "includes", "including", “having” and their conjugates mean "including but not limited to". This term encompasses the terms "consisting of' and "consisting essentially of'.

The phrase "consisting essentially of means that the composition or method may include additional ingredients and/or steps, but only if the additional ingredients and/or steps do not materially alter the basic and novel characteristics of the claimed composition or method.

As used herein, the singular form "a", "an" and "the" include plural references unless the context clearly dictates otherwise. For example, the term "a compound" or "at least one compound" may include a plurality of compounds, including mixtures thereof.

The word “exemplary” is used herein to mean “serving as an example, instance or illustration”. Any embodiment described as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments and/or to exclude the incorporation of features from other embodiments.

The word “optionally” is used herein to mean “is provided in some embodiments and not provided in other embodiments”. Any particular embodiment of the disclosure may include a plurality of “optional” features unless such features conflict.

Throughout this application, various embodiments of this disclosure may be presented in a range format. It should be understood that the description in range format is merely for convenience and brevity and should not be construed as an inflexible limitation on the scope of the disclosure. Accordingly, the description of a range should be considered to have specifically disclosed all the possible subranges as well as individual numerical values within that range. For example, description of a range such as from 1 to 6 should be considered to have specifically disclosed subranges such as from 1 to 3, from 1 to 4, from 1 to 5, from 2 to 4, from 2 to 6, from 3 to 6 etc., as well as individual numbers within that range, for example, 1, 2, 3, 4, 5, and 6. This applies regardless of the breadth of the range.

Whenever a numerical range is indicated herein, it is meant to include any cited numeral (fractional or integral) within the indicated range. The phrases “ranging/ranges between” a first indicate number and a second indicate number and “ranging/ranges from” a first indicate number “to” a second indicate number are used herein interchangeably and are meant to include the first and second indicated numbers and all the fractional and integral numerals therebetween.

It is appreciated that certain features of the disclosure, which are, for clarity, described in the context of separate embodiments, may also be provided in combination in a single embodiment. Conversely, various features of the disclosure, which are, for brevity, described in the context of a single embodiment, may also be provided separately or in any suitable subcombination or as suitable in any other described embodiment of the disclosure. Certain features described in the context of various embodiments are not to be considered essential features of those embodiments, unless the embodiment is inoperative without those elements.

It is the intent of the applicant(s) that all publications, patents and patent applications referred to in this specification are to be incorporated in their entirety by reference into the specification, as if each individual publication, patent or patent application was specifically and individually noted when referenced that it is to be incorporated herein by reference. In addition, citation or identification of any reference in this application shall not be construed as an admission that such reference is available as prior art to the present invention. To the extent that section headings are used, they should not be construed as necessarily limiting. In addition, any priority document(s) of this application is/are hereby incorporated herein by reference in its/their entirety.

Claims

1. A device (104) for deduplication storage of data, comprising: a processor (102) configured for: accessing target data (112A) for storage on a plurality of storage-sets (160), each storage-set including a plurality of media of a plurality of sequential access storage devices (150); accessing a global index (116A) mapping between each of the storage-sets and data segments stored thereon, for identifying a storage-set that includes a plurality of media likely storing a duplicate of at least one segment of the target data; accessing a media set specific deduplication mechanism (150A) for identifying duplicates of segments of the target data stored on the plurality of media of the identified storage-set; storing non-duplicated segments of the target data on the plurality of media of the identified storage-set; updating the media set specific deduplication mechanisms of the plurality of media of the storage-set for indicating the identified duplicated segments and the stored non-duplicated segments of the target data; updating the global index for indicating mappings between the segments of the target data used during the accessing; updating a global file mapping (116B) for storing identifiers for the plurality of segments of the target data; and updating an identification table (116C) for storing metadata for each identifier.

2. The device according to claim 1, wherein the non-duplicated segments of the target data are stored in parallel on the plurality of media of the identified storage-set.

3. The device according to any of the preceding claims, wherein the processor is further configured for: dividing the target data into a plurality of data segments (504); computing a plurality of fingerprints for the plurality of data segments (506); selecting a representative subset of the plurality of fingerprints; wherein the global index is accessed for the representative subset for identifying the storage-set; wherein a media of the storage-set for deduplicated storage of a data segment is selected according to the fingerprint of the data segment.

4. The device according to claim 3, wherein the media of the storage-set for deduplicated storage is selected by performing a modulo operation to find a remainder when the fingerprint is divided by the number of media in the storage-set (514), wherein the remainder indicates the selected media (510A-D).

5. The device according to claim 3 or claim 4, wherein each metadata stored on the identification table includes an indication of the storage-set on which the data segment is stored, offset of the media, size of the data segment, and fingerprint of each of the plurality of data segments, wherein the fingerprint is used for determining the media of the storage-set.

6. The device according to any of the preceding claims, wherein a number of media of each storage-set is preconfigured and fixed.

7. The device according to any of the preceding claims, wherein each media is assigned to a single storage-set.

8. The device according to any of the preceding claims, wherein the storing is done by appending the non-duplicated segments to the plurality of media of the storage-set.

9. The device according to any of the preceding claims, wherein the processor is further configured for: receiving a request to read the target data; accessing the global file map for identifiers of the plurality of data segments of the target data; accessing an identification table for obtaining metadata of each data segment of the plurality of data segments; planning an order for parallel reading the plurality of data segments according to the metadata; reading the plurality of data segments of the target data according to the order and in parallel; and creating the target data from the plurality of data segments.

10. The device according to claim 9, wherein each metadata includes an indication of the storage-set on which the data segment is stored, offset of the media, size of the data segment, and fingerprint of the data segment, wherein the fingerprint is used for determining the media of the storage-set.

11. A device (104) for restoration of data, comprising: a processor (102) configured for: receiving a request to read target data (112A) stored on a storage-set of a plurality of storage-sets (160), each storage-set including a plurality of media of a plurality of sequential access storage devices (150); obtaining identifiers of a plurality of data segments of the target data from a global file mapping (116B); obtaining metadata of each data segment of the plurality of data segments from an identification table (116C); planning an order for parallel reading the plurality of data segments according to the metadata; reading the plurality of data segments of the target data from the plurality of media of the storage-set according to the order and in parallel; and creating the target data from the plurality of data segments.

12. The device according to claim 11, wherein each metadata includes a fingerprint of the data segment, and wherein the processor is further configured for determining the media of the storage-set according to the fingerprint.

13. The device according to claim 11 or claim 12, wherein the processor is further configured for determining the media by performing a modulo operation to find a remainder when the fingerprint is divided by the number of media in the storage-set, wherein the remainder indicates the media.

14. The device according to claims 11-13, wherein the processor is further configured for: for each of the plurality of media, arranging data segments for sequential reading according to location and offset of the data segment. A method for deduplication storage of data, comprising: accessing target data for storage on a plurality of storage-sets, each storage-set including a plurality of media of a plurality of sequential access storage devices (302); accessing a global index mapping between each of the storage-sets and data segments stored thereon, for identifying a storage-set that includes a plurality of media likely storing a duplicate of at least one segment of the target data (304); accessing a media set specific deduplication mechanism for identifying duplicates of segments of the target data stored on the plurality of media of the identified storage-set (306); storing non-duplicated segments of the target data on the plurality of media of the identified storage-set (310); updating the media set specific deduplication mechanisms of the plurality of media of the storage-set for indicating the identified duplicated segments and the stored non-duplicated segments of the target data (312); updating the global index for indicating mappings between the segments of the target data used during the accessing (314); updating a global file mapping for storing identifiers for the plurality of segments of the target data (316); and updating an identification table for storing metadata for each identifier (318). A method for restoration of data, comprising: receiving a request to read target data stored on a storage-set of a plurality of storage-sets, each storage-set including a plurality of media of a plurality of sequential access storage devices (402); obtaining identifiers of a plurality of data segments of the target data from a global file map (404); obtaining metadata of each data segment of the plurality of data segments from an identification table (406); planning an order for parallel reading the plurality of data segments according to the metadata (410); reading the plurality of data segments of the target data according to the order and in parallel 412); and creating the target data from the plurality of data segments (414).

17. A non-transitory medium (106) storing program instructions (106A) for deduplication storage of data, which, when executed by at least one processor (102), cause the at least one processor to: access target data for storage on a plurality of storage-sets, each storage-set including a plurality of media of a plurality of sequential access storage devices; access a global index mapping between each of the storage-sets and data segments stored thereon, for identifying a storage-set that includes a plurality of media likely storing a duplicate of at least one segment of the target data; access a media set specific deduplication mechanism for identifying duplicates of segments of the target data stored on the plurality of media of the identified storage-set; store non-duplicated segments of the target data on the plurality of media of the identified storage-set; update the media set specific deduplication mechanisms of the plurality of media of the storage-set for indicating the identified duplicated segments and the stored non-duplicated segments of the target data; update the global index for indicating mappings between the segments of the target data used during the accessing; update a global file mapping for storing identifiers for the plurality of segments of the target data; and update an identification table for storing metadata for each identifier.

18. A non-transitory medium (106) storing program instructions (106 A) for restoration of data, which, when executed by at least one processor (102), cause the at least one processor to: receive a request to read target data stored on a storage-set of a plurality of storage-sets, each storage-set including a plurality of media of a plurality of sequential access storage devices; obtain identifiers of a plurality of data segments of the target data from a global file map; obtain metadata of each data segment of the plurality of data segments from an identification table; planning an order for parallel reading the plurality of data segments according to the metadata; read the plurality of data segments of the target data according to the order and in parallel; and create the target data from the plurality of data segments.