US20140279927A1 - Scalable graph modeling of metadata for deduplicated storage systems - Google Patents
Scalable graph modeling of metadata for deduplicated storage systems Download PDFInfo
- Publication number
- US20140279927A1 US20140279927A1 US13/832,043 US201313832043A US2014279927A1 US 20140279927 A1 US20140279927 A1 US 20140279927A1 US 201313832043 A US201313832043 A US 201313832043A US 2014279927 A1 US2014279927 A1 US 2014279927A1
- Authority
- US
- United States
- Prior art keywords
- data
- data objects
- graph
- shared
- objects
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000003860 storage Methods 0.000 title claims abstract description 75
- 238000000034 method Methods 0.000 claims abstract description 37
- 238000004590 computer program Methods 0.000 claims abstract description 15
- 230000004931 aggregating effect Effects 0.000 claims description 2
- 238000010586 diagram Methods 0.000 description 13
- 230000006870 function Effects 0.000 description 10
- 230000008569 process Effects 0.000 description 7
- 238000012545 processing Methods 0.000 description 6
- 230000008901 benefit Effects 0.000 description 5
- 238000007726 management method Methods 0.000 description 5
- 230000009467 reduction Effects 0.000 description 4
- 230000003287 optical effect Effects 0.000 description 3
- 238000013500 data storage Methods 0.000 description 2
- 239000000463 material Substances 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 239000013307 optical fiber Substances 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 238000011084 recovery Methods 0.000 description 2
- 230000008859 change Effects 0.000 description 1
- 239000003086 colorant Substances 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000013523 data management Methods 0.000 description 1
- 238000009826 distribution Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 230000005012 migration Effects 0.000 description 1
- 238000013508 migration Methods 0.000 description 1
- 230000003252 repetitive effect Effects 0.000 description 1
- 230000010076 replication Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 238000000638 solvent extraction Methods 0.000 description 1
Images
Classifications
-
- G06F17/30156—
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/174—Redundancy elimination performed by the file system
- G06F16/1748—De-duplication implemented within the file system, e.g. based on file segments
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/174—Redundancy elimination performed by the file system
- G06F16/1748—De-duplication implemented within the file system, e.g. based on file segments
- G06F16/1756—De-duplication implemented within the file system, e.g. based on file segments based on delta files
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
Definitions
- the present invention relates generally to storage systems and, more specifically, to scalable graph modeling of metadata for deduplicated storage systems.
- Data deduplication removes or minimizes the amount of redundant data in a storage system by keeping only unique instances of the data on storage. Redundant data is replaced with a pointer to the unique data copy. By reducing space requirements, deduplication reduces the need for new storage resources. Implementations of deduplication often lead to substantial savings, both in terms of new resource acquisition costs and management costs, thereby leading to a significant reduction in TCO. In backup environments, deduplication also lowers the network bandwidth requirements for remote backups, replication and disaster recovery by reducing the amount of transmitted data on the network.
- deduplication introduces challenges to storage management as storage objects (e.g., files) are no longer independent from each other due to content sharing between storage objects.
- storage objects e.g., files
- storage objects share content with each other, they cannot be managed independently because a management decision on one file may affect another file.
- individual “old” files can be migrated to a “colder” tier (e.g., from disks to tapes) without affecting other files.
- old files may share content with other files that are not necessarily old, so the migration of a candidate file needs to consider other files that share content with the candidate file, which complicates the storage management tasks.
- An embodiment includes a method, computer program product, and system for providing a scalable representation of metadata for deduplicated storage systems.
- the method includes identifying shared data segments that are contained in a plurality of data objects in a deduplicated storage system.
- a data object centric graph is generated.
- the generating includes creating vertices that represent the data objects and creating edges between the data objects.
- An edge connecting two data objects indicates that the two data objects contain at least one shared data segment in common.
- Each shared data segment between any two of the data objects is represented by at most one of the edges. At least one of the data objects is manipulated based on the data object centric graph.
- Another embodiment includes a method for providing scalable representation of metadata for deduplicated storage systems.
- the method includes identifying shared data segments that are contained in a plurality of data objects in a deduplicated storage system.
- a data segment centric bipartite graph is generated.
- the generating includes creating vertices that represent the data objects and the shared data segments, and creating edges between the data objects and the shared data segments.
- An edge between a data object and a shared data segment indicates that the data object contains the shared data segment.
- At least one of the data objects is manipulated based on the data segment centric bipartite graph.
- FIG. 1 illustrates a data segment centric graph that models the sharing of data segments between data objects in a storage system in accordance with an embodiment
- FIG. 2 illustrates a data object centric graph that models the sharing of data segments between data objects in a storage system in accordance with an embodiment
- FIG. 3 illustrates a process for creating the data segment centric graph shown in FIG. 1 and the data object centric graph shown in FIG. 2 in accordance with an embodiment
- FIG. 4 illustrates exemplary star and chain data object centric graphs that may be implemented in accordance with an embodiments.
- FIG. 5 illustrates a block diagram of a system for providing graph modeling of metadata for deduplicated storage systems in accordance with an embodiment.
- Embodiments described herein are directed to scalable graph modeling of metadata for deduplicated storage systems.
- the graph structures described herein illustrate content sharing between data objects (e.g., files) in deduplicated storage systems.
- Metadata that exposes sharing relationships between data objects is modeled in a deduplicated storage system as a graph in order to leverage the expressive power of graph representation yet with a reduced metadata size.
- Contemporary solutions rely on a single hash table that is keyed by hash values to indirectly expose sharing between data objects. This requires storage users to read through all hash values of a group of data objects each time they require a panorama of sharing among the group of data objects. This process is often time consuming and not scalable.
- embodiments described herein build up a graph initially from the hash table, and then update the graph incrementally later on so that the storage users do not need to rebuild the representation from scratch again and again. Furthermore, the size of the graph representation can be reduced significantly if storage users require visibility to only high level sharing information (e.g., the number of shared bytes) among data objects.
- Embodiments described herein identify two main ways to model data sharing as a graph: data segment centric and data object centric.
- data object refers to, but is not limited to: a file, a group of files, a block, a group of blocks, a volume, a group of volumes, all or a portion of the storage for a particular computer, or a user defined data structure.
- Data objects are made up of one or more fixed or variable sized data segments.
- data segment refers to, but is not limited to: a specified number of bytes, a block of bytes, a group of blocks, a segment containing more than one consecutive data segments (created, for example, by aggregating consecutive shared data segments), or a user defined grouping.
- shared data segment refers to a data segment that is contained in two or more data objects.
- the data segment centric graph represents both data segments 104 and data objects 102 as vertices.
- the data segments 104 are denoted as “S1”, “S2”, “S3”, “S4”, “S5”, “S6”, and “S7”, and the data objects 102 are denoted as “FILE 1”, “FILE 2”, and “FILE 3”.
- edges exist between data objects 102 and data segments 104 only (and not between data objects 102 ).
- data segment centric graph is a bipartite graph of the set of data objects 102 and the set of data segments 104 .
- An edge exists between a data object 102 and a data segment 104 when the data segment 104 exists in the data object 102 .
- data segments S1, S2 and S3 are contained in FILE 1; data segments S1, S3, S4, and S5 are contained in FILE 2; and data segments S2, S5, S6, and S7 are contained in FILE 3.
- the data segment centric graph also includes edge weights which are used to specify data segment offsets, where a data segment offset indicates the offset at which the data segment appears in the data object the edge connects to.
- the data segment centric graph shown in FIG. 1 includes detailed, data segment level, sharing information that may be used to aid in data placement or other data object manipulations.
- the data segment centric graph bipartite graph shown in FIG. 1 can be updated incrementally when the content of one or more data objects 102 is updated. When one data segment 104 changes, only the data objects 102 connected to it in the bipartite graph need to be updated.
- shared data segments 104 shown in FIG. 1 have spatial locality, so that instead of representing data segments 104 individually, a repetitive sequence of shared data segments 104 can be represented as a vertex in the bipartite graph, thus having the potential to reduce the size of the bipartite graph.
- FIG. 2 a data object centric graph that models the sharing of data segments between data objects in a storage system is illustrated in accordance with an embodiment.
- the data object centric graph shown in FIG. 2 has a lower memory footprint than the data segment centric graph shown in FIG. 1 .
- the data object centric graph shown in FIG. 2 represents only data objects 102 as vertices. Edges are placed between data objects 102 that share data segments 104 .
- edge weights 202 represent the amount of sharing. For example, an edge weight 202 value of “2” between the data object 102 labeled “FILE 1” and the data object 102 labeled “FILE 2” means that “FILE 1” and “FILE 2” share two data segments 104 .
- the data object centric graph summarizes the detailed information offered by the data segment centric model of FIG. 1 by indicating the amount of sharing between data objects 102 .
- the amount of sharing between data objects 102 is described by the graph and there is no visibility to the specific data segments 104 that are shared. Removing the detailed information results in a relatively small graph that is proportional in size to the order of the number of data objects 102 , which is typically orders of magnitude lower than the number of data segments.
- a data object centric graph such as that shown in FIG. 2
- the vertices are connected with a minimal set of edges in the following sense. For example, if “n” data objects 102 share a data segment 104 , then the number of edges between this set of data objects 102 is “n ⁇ 1.” Thus, each shared data segment 104 between any two of the data objects 102 is represented by at most one of the edges. The idea is to not have duplicate edges that model the sharing of the same content between data objects 102 . This enables an important property of embodiments of data object centric graph models described herein: the ability to estimate, as an upper bound, the deduplicated size of a set of data objects 102 through a simple graph traversal.
- the deduplicated size can be computed by a breadth-first search (BFS) traversal that sums the weights of the vertices (which represent data objects sizes) and subtracts the edge weights 202 (which represent the size of the duplicates). It is noted that for a non-partitioned graph component this is not an approximation but always produces the correct result. In other words, the deduplicated size of the set of data objects 102 is simply the sum of the vertex sizes minus the sum of the connecting edge weights.
- BFS breadth-first search
- identical data objects e.g., full file duplicates
- singleton data objects e.g., files that do not share any data segments with other files
- An important practical aspect of embodiments is the building of the data object centric graphs, particularly identifying the edges, which represent sharing dependencies, between vertices (i.e., data objects).
- the process takes as input deduplication metadata, particularly a data object segment map which identifies the list of data segments 104 that each data object 102 is composed of.
- embodiments produce a graph represented as a list of edges, called an edge list.
- One important requirement for building the graph is scalability, the graph should scale to handle petabyte scale systems. Therefore, scalability limiting constraints (such as requiring a large memory footprint) should be avoided.
- Block 302 is an embodiment of metadata stored in a database that maps data segments to data objects.
- the hash values (denoted as “H1”, “H2”, “H3”, “H4”, “H5”, “H6”, and “H7”) of data segments 104 are mapped to data objects 102 (denoted as “F1”, “F2”, and “F3”).
- H1 is the result when a hash operation is performed on the data segment 104 labeled “S1”
- H2 is the result when the hash operation is performed on the data segment 104 labeled “S2”
- the data shown in block 302 is referred to herein as a “data object to data segment map” and it may be used to generate the pair of data object and data segment.
- the data object to data segment map in block 302 is sorted based on hash values.
- the processing at block 304 groups together data objects 102 that share a common data segment 104 . In an embodiment, this is performed by sorting the data object to data segment map shown in block 302 by hash value.
- hash value sorted data object to data segment map output from block 304 is shown in block 306 .
- the contents of block 306 may be left in table form as shown in block 306 and/or the data in block 306 may be used to depict a graph such as that shown in FIG. 1 .
- edges are created between data objects 102 that share a data segment 104 using the hash value sorted data object/data segment map in block 306 as input.
- a simple scan is performed over the hash value sorted data object to data segment map in block 306 and the edges as shown in block 310 are created.
- any graph heuristic that satisfies a minimal set of edges requirement may be used by embodiments.
- Creating the edges at block 308 may produce more than one edge between any two vertices (e.g., when two data objects 102 sharing two different chunks 104 ).
- the processing at blocks 312 and 316 are performed to aggregate multiple edges between the same pair of nodes into 1 edge.
- the edges shown in block 310 are received and grouped by edge to generate the ordered edge list shown in block 314 . In an embodiment, this is performed by a sorting method, such as one or more of those described above with respect to block 304 .
- the edge weights 202 are aggregated to generate the data shown in block 318 . In an embodiment, at block 316 , a sum reduction is performed over the weights of similar edges via a single scan over the ordered edge list shown in block 314 .
- the resulting minimal edge data object centric graph may be left in table form as shown in block 318 and/or the data in block 318 may be used to depict a graph such as that shown in FIG. 2 .
- Embodiments of the graph generation process described herein do not require excessive memory requirements.
- the performance of the two major operations, hash and edge grouping, can benefit from additional physical memory, but can still be processed efficiently on disk.
- Graphs are well known for their expressive power and this can be leveraged by encoding sharing constraints among data objects 102 . For example, if two data objects 102 are supposed to be separated into two deduplication domains, these two vertices can be tagged as having different colors so that subsequent algorithms partitioning the graph can take into account this sharing constraints. More complicated sharing can also be applied to embodiments of the graphs. Any sharing constraint can be modeled as long as the constraint can be expressed by vertices and edges of the graph.
- a star graph heuristic is used where one of the data objects 102 is linked with every other data object 102 .
- a chain graph heuristic is used where the data objects 102 are linked to each other in a linked list.
- the master node (representing a data object 102 ) is connected with “n ⁇ 1” nodes, so it has a degree of “n ⁇ 1”, while the rest of the nodes (representing data objects 102 ) have a degree of 1.
- the two end nodes have degrees of 1 while the intermediary nodes have degrees of 2, so the distribution of node degrees is more balanced. Note that between these two heuristics there are many other ways to link the “n” data objects 102 by a minimal set of edges.
- FIG. 4 shows a block diagram of a table 402 summarizing content of nine files in a small folder, a data object centric graph with star connectivity 404 built from the data in the table 402 , and a data object centric graph with chain connectivity 406 built from the data in the table 402 .
- the table 402 summarizes the content of the nine files, each file is made up of a sequence of data segments made up of four bytes. The first data segment is different for each file, but several data segments are common between files.
- the last column of the table 402 shows the size of each file in bytes, the total size of the nine files in the directory is 148 bytes.
- trace data is collected by executing a file scanning program that traverses each file of the folder (or file system).
- the trace contains, for each file, a sequence of SHA-1 cryptographic hash function content hashes, one hash for each data segment contained in the file.
- fixed data segments of 4 bytes are used, however in real applications the data segments sizes will be much larger (e.g., 4 kilobytes, 8 kilobytes, or more).
- the star topology of the data object centric graph with star connectivity 404 includes a vertex for each file (v1, v2, v3 . . . v9) and edges represent sharing of content between vertices. To have a minimum number of edges in the graph, the shared content is represented only once. Representing the shared content only once provides the ability to determine the deduplicated size of the folder (or file system) as the sum of vertex sizes (raw file size) minus the sum of connecting edge weights.
- This topology is referred to as a star topology because for the “n” number of vertices sharing the same content, one is selected (i.e., the center of the star) to be connected with the other vertices.
- An alternative to the star topology is to use a chain topology as shown in the data object centric graph with chain connectivity 406 in FIG. 4 .
- Embodiments of the data segment and object centric graphs may be used to aid in determining data object groupings for backup and restore operations.
- the graphs can be used to determine the specific tape a data segment should be placed on for optimal data object restore performance.
- Another scenario is informing data segment placement in deduplicated disk-based pools in large shared-nothing commodity server clusters, where servers have asymmetric access to disks, for optimal data object restore performance, or to reduce failure propagation on disk pools (if a disk fails, data loss does not spread to other disks)
- the granularity of the data object is selected based on a granularity of a restore operation (also referred to herein as a “recovery operation”) associated with the deduplicated storage system. For example, if the granularity of a restore operation is a file, then each data object can be a file; if the granularity of a restore operation is a volume, each data object can represent a volume.
- a restore operation also referred to herein as a “recovery operation”
- FIG. 5 a block diagram of a system for providing graph modeling of metadata for deduplicated storage systems in accordance with an embodiment is generally shown.
- a host computer 502 is in communication with a storage controller 504 in a storage system 510 .
- the storage system 510 stores data in a storage array made up of “K” storage devices 506 (where K is greater than one): storage device one 506 a , storage device two 506 b , storage device three 506 c , through storage device K 506 d .
- data objects 102 and metadata about data segments 104 shared by the data objects 102 are stored on one or more of the storage devices 506 .
- the storage devices 506 in the storage array 508 are implemented by any storage devices known in the art such as, but not limited to: tapes/tape drives and/or disks/hard disk drives (HDDs).
- the storage controller 504 and the host computer may contain memory devices to store data, such as all or a portion of the metadata, graphs and other data described herein.
- the storage system 510 and/or the host computer 502 may implement a tiered hierarchy of memory and storage that includes memory devices (containing memory and/or cached memory), disks and tape devices.
- the storage controller 504 is part of the storage system 510 ; in another embodiment, the storage controller 504 is part of the host computer 502 .
- the term “processor” refers to one or more processors located in the host computer 502 and/or the storage controller 504 .
- Embodiments also provide a light weight, yet accurate, representation of sharing between data objects 102 .
- the representation of sharing is elastic in that it can be updated incrementally when data object 102 contents change.
- embodiments provide a flexible data structure that can incorporate sharing constraints among data objects 102 .
- the graph that represents sharing between data objects 102 is less than one percent of the size of a contemporary hash table representation of the metadata.
- aspects of the present invention may be embodied as a system, method, or computer program product. Accordingly, aspects of the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, micro-code, etc.) or an embodiment combining software and hardware aspects that may all generally be referred to herein as a “circuit,” “module” or “system.” Furthermore, aspects of the present invention may take the form of a computer program product embodied in one or more computer readable medium(s) having computer readable program code embodied thereon.
- the computer readable medium may be a computer readable signal medium or a computer readable storage medium.
- a computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing.
- a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
- a computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof.
- a computer readable signal medium may be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.
- Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, radio frequency (RF), etc., or any suitable combination of the foregoing.
- any appropriate medium including but not limited to wireless, wireline, optical fiber cable, radio frequency (RF), etc., or any suitable combination of the foregoing.
- Computer program code for carrying out operations for aspects of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages, such as the “C” programming language or similar programming languages.
- the program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server.
- the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet Service Provider).
- LAN local area network
- WAN wide area network
- Internet Service Provider for example, AT&T, MCI, Sprint, EarthLink, MSN, GTE, etc.
- These computer program instructions may also be stored in a computer readable medium that can direct a computer, other programmable data processing apparatus, or other devices to function in a particular manner, such that the instructions stored in the computer readable medium produce an article of manufacture including instructions which implement the function/act specified in the flowchart and/or block diagram block or blocks.
- the computer program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other devices to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks.
- each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s).
- the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved.
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
Description
- The present invention relates generally to storage systems and, more specifically, to scalable graph modeling of metadata for deduplicated storage systems.
- Digitization of large volumes of data and an increase in the richness of content of data have led to high demands for data storage capacity. One way to counter this increasing need for data storage capacity is to add additional hardware resources. However, in the storage domain, the addition of more storage often results in a disproportionate increase in the total cost of ownership (TCO). Though the cost of acquisition has retreated as a result of reductions in hardware costs, the cost of management (e.g., administration, power/energy) has increased. Many companies are attempting to provide a better solution by using data footprint reduction techniques such as deduplication.
- Data deduplication removes or minimizes the amount of redundant data in a storage system by keeping only unique instances of the data on storage. Redundant data is replaced with a pointer to the unique data copy. By reducing space requirements, deduplication reduces the need for new storage resources. Implementations of deduplication often lead to substantial savings, both in terms of new resource acquisition costs and management costs, thereby leading to a significant reduction in TCO. In backup environments, deduplication also lowers the network bandwidth requirements for remote backups, replication and disaster recovery by reducing the amount of transmitted data on the network.
- The use of deduplication introduces challenges to storage management as storage objects (e.g., files) are no longer independent from each other due to content sharing between storage objects. When storage objects share content with each other, they cannot be managed independently because a management decision on one file may affect another file. For example, in a traditional tiered storage system, individual “old” files can be migrated to a “colder” tier (e.g., from disks to tapes) without affecting other files. However, in a deduplicated tiered storage system, old files may share content with other files that are not necessarily old, so the migration of a candidate file needs to consider other files that share content with the candidate file, which complicates the storage management tasks.
- Understanding the sharing relationships between data objects in a deduplicated storage system is important in order to provide efficient data management, such as data placement and data retrieval.
- An embodiment includes a method, computer program product, and system for providing a scalable representation of metadata for deduplicated storage systems. The method includes identifying shared data segments that are contained in a plurality of data objects in a deduplicated storage system. A data object centric graph is generated. The generating includes creating vertices that represent the data objects and creating edges between the data objects. An edge connecting two data objects indicates that the two data objects contain at least one shared data segment in common. Each shared data segment between any two of the data objects is represented by at most one of the edges. At least one of the data objects is manipulated based on the data object centric graph.
- Another embodiment includes a method for providing scalable representation of metadata for deduplicated storage systems. The method includes identifying shared data segments that are contained in a plurality of data objects in a deduplicated storage system. A data segment centric bipartite graph is generated. The generating includes creating vertices that represent the data objects and the shared data segments, and creating edges between the data objects and the shared data segments. An edge between a data object and a shared data segment indicates that the data object contains the shared data segment. At least one of the data objects is manipulated based on the data segment centric bipartite graph.
- Additional features and advantages are realized through the techniques of the present invention. Other embodiments and aspects of the invention are described in detail herein and are considered a part of the claimed invention. For a better understanding of the invention with the advantages and the features, refer to the description and to the drawings.
- The subject matter which is regarded as the invention is particularly pointed out and distinctly claimed in the claims at the conclusion of the specification. The foregoing and other features, and advantages of the invention are apparent from the following detailed description taken in conjunction with the accompanying drawings in which:
-
FIG. 1 illustrates a data segment centric graph that models the sharing of data segments between data objects in a storage system in accordance with an embodiment; -
FIG. 2 illustrates a data object centric graph that models the sharing of data segments between data objects in a storage system in accordance with an embodiment; -
FIG. 3 illustrates a process for creating the data segment centric graph shown inFIG. 1 and the data object centric graph shown inFIG. 2 in accordance with an embodiment; -
FIG. 4 illustrates exemplary star and chain data object centric graphs that may be implemented in accordance with an embodiments; and -
FIG. 5 illustrates a block diagram of a system for providing graph modeling of metadata for deduplicated storage systems in accordance with an embodiment. - Embodiments described herein are directed to scalable graph modeling of metadata for deduplicated storage systems. The graph structures described herein illustrate content sharing between data objects (e.g., files) in deduplicated storage systems. Metadata that exposes sharing relationships between data objects is modeled in a deduplicated storage system as a graph in order to leverage the expressive power of graph representation yet with a reduced metadata size. Contemporary solutions rely on a single hash table that is keyed by hash values to indirectly expose sharing between data objects. This requires storage users to read through all hash values of a group of data objects each time they require a panorama of sharing among the group of data objects. This process is often time consuming and not scalable. In contrast to contemporary solutions, embodiments described herein build up a graph initially from the hash table, and then update the graph incrementally later on so that the storage users do not need to rebuild the representation from scratch again and again. Furthermore, the size of the graph representation can be reduced significantly if storage users require visibility to only high level sharing information (e.g., the number of shared bytes) among data objects.
- Embodiments described herein identify two main ways to model data sharing as a graph: data segment centric and data object centric. As used herein, the term “data object” refers to, but is not limited to: a file, a group of files, a block, a group of blocks, a volume, a group of volumes, all or a portion of the storage for a particular computer, or a user defined data structure. Data objects are made up of one or more fixed or variable sized data segments. As used herein, the term “data segment” refers to, but is not limited to: a specified number of bytes, a block of bytes, a group of blocks, a segment containing more than one consecutive data segments (created, for example, by aggregating consecutive shared data segments), or a user defined grouping. As used herein, the term “shared data segment” refers to a data segment that is contained in two or more data objects.
- Referring now to
FIG. 1 , a data segment centric graph that models the sharing of data segments between data objects in a storage system in accordance with an embodiment is generally shown. The data segment centric graph represents bothdata segments 104 anddata objects 102 as vertices. InFIG. 1 , thedata segments 104 are denoted as “S1”, “S2”, “S3”, “S4”, “S5”, “S6”, and “S7”, and thedata objects 102 are denoted as “FILE 1”, “FILE 2”, and “FILE 3”. As shown inFIG. 1 , edges exist betweendata objects 102 anddata segments 104 only (and not between data objects 102). Thus, the graph shown inFIG. 1 is a bipartite graph of the set ofdata objects 102 and the set ofdata segments 104. An edge exists between adata object 102 and adata segment 104 when thedata segment 104 exists in thedata object 102. As shown in the graph ofFIG. 1 : data segments S1, S2 and S3 are contained inFILE 1; data segments S1, S3, S4, and S5 are contained inFILE 2; and data segments S2, S5, S6, and S7 are contained inFILE 3. In an embodiment, the data segment centric graph also includes edge weights which are used to specify data segment offsets, where a data segment offset indicates the offset at which the data segment appears in the data object the edge connects to. The data segment centric graph shown inFIG. 1 includes detailed, data segment level, sharing information that may be used to aid in data placement or other data object manipulations. - There are many use cases where the incremental update of the graph is desired. For example, for online deduplication of active data, it is desirable to maintain an up-to-date deduplication graph to provide advisory information to improve deduplication performance (for example, by providing locality information of duplicate data segments). The data segment centric graph bipartite graph shown in
FIG. 1 can be updated incrementally when the content of one or more data objects 102 is updated. When onedata segment 104 changes, only the data objects 102 connected to it in the bipartite graph need to be updated. - Note that shared
data segments 104 shown inFIG. 1 have spatial locality, so that instead of representingdata segments 104 individually, a repetitive sequence of shareddata segments 104 can be represented as a vertex in the bipartite graph, thus having the potential to reduce the size of the bipartite graph. - Referring now to
FIG. 2 , a data object centric graph that models the sharing of data segments between data objects in a storage system is illustrated in accordance with an embodiment. The data object centric graph shown inFIG. 2 has a lower memory footprint than the data segment centric graph shown inFIG. 1 . The data object centric graph shown inFIG. 2 represents onlydata objects 102 as vertices. Edges are placed between data objects 102 that sharedata segments 104. In an embodiment, edge weights 202 represent the amount of sharing. For example, an edge weight 202 value of “2” between the data object 102 labeled “FILE 1” and the data object 102 labeled “FILE 2” means that “FILE 1” and “FILE 2” share twodata segments 104. The data object centric graph summarizes the detailed information offered by the data segment centric model ofFIG. 1 by indicating the amount of sharing between data objects 102. As shown inFIG. 2 , the amount of sharing between data objects 102 is described by the graph and there is no visibility to thespecific data segments 104 that are shared. Removing the detailed information results in a relatively small graph that is proportional in size to the order of the number of data objects 102, which is typically orders of magnitude lower than the number of data segments. - For a data object centric graph, such as that shown in
FIG. 2 , the vertices are connected with a minimal set of edges in the following sense. For example, if “n” data objects 102 share adata segment 104, then the number of edges between this set of data objects 102 is “n−1.” Thus, each shareddata segment 104 between any two of the data objects 102 is represented by at most one of the edges. The idea is to not have duplicate edges that model the sharing of the same content between data objects 102. This enables an important property of embodiments of data object centric graph models described herein: the ability to estimate, as an upper bound, the deduplicated size of a set ofdata objects 102 through a simple graph traversal. In particular, the deduplicated size can be computed by a breadth-first search (BFS) traversal that sums the weights of the vertices (which represent data objects sizes) and subtracts the edge weights 202 (which represent the size of the duplicates). It is noted that for a non-partitioned graph component this is not an approximation but always produces the correct result. In other words, the deduplicated size of the set of data objects 102 is simply the sum of the vertex sizes minus the sum of the connecting edge weights. - In an embodiment of the data object centric graph, identical data objects (e.g., full file duplicates) are represented only once and singleton data objects (e.g., files that do not share any data segments with other files) are not represented in the graph.
- An important practical aspect of embodiments is the building of the data object centric graphs, particularly identifying the edges, which represent sharing dependencies, between vertices (i.e., data objects). The process takes as input deduplication metadata, particularly a data object segment map which identifies the list of
data segments 104 that each data object 102 is composed of. As output, embodiments produce a graph represented as a list of edges, called an edge list. One important requirement for building the graph is scalability, the graph should scale to handle petabyte scale systems. Therefore, scalability limiting constraints (such as requiring a large memory footprint) should be avoided. - Turning now to
FIG. 3 , a process for creating the data segment centric graph shown inFIG. 1 and the data object centric graph shown inFIG. 2 in accordance with an embodiment is generally shown.Block 302 is an embodiment of metadata stored in a database that maps data segments to data objects. As shown inblock 302, the hash values (denoted as “H1”, “H2”, “H3”, “H4”, “H5”, “H6”, and “H7”) ofdata segments 104 are mapped to data objects 102 (denoted as “F1”, “F2”, and “F3”). H1 is the result when a hash operation is performed on thedata segment 104 labeled “S1”, H2 is the result when the hash operation is performed on thedata segment 104 labeled “S2”, and so on such that hash(S#)=H#. The data shown inblock 302 is referred to herein as a “data object to data segment map” and it may be used to generate the pair of data object and data segment. - At
block 304, the data object to data segment map inblock 302 is sorted based on hash values. The processing atblock 304 groups together data objects 102 that share acommon data segment 104. In an embodiment, this is performed by sorting the data object to data segment map shown inblock 302 by hash value. Several efficient and scalable disk-based sorting algorithms which do not demand a large memory footprint exist and may be utilized by embodiments, including, but not limited to: an ORDER BY SQL clause and a Linux® sorting tool. The hash value sorted data object to data segment map output fromblock 304 is shown inblock 306. The contents ofblock 306 may be left in table form as shown inblock 306 and/or the data inblock 306 may be used to depict a graph such as that shown inFIG. 1 . - At
block 308, edges are created between data objects 102 that share adata segment 104 using the hash value sorted data object/data segment map inblock 306 as input. In an embodiment, atblock 308, a simple scan is performed over the hash value sorted data object to data segment map inblock 306 and the edges as shown inblock 310 are created. For the sparse data object centric graph representation shown inFIG. 2 , any graph heuristic that satisfies a minimal set of edges requirement may be used by embodiments. Creating the edges atblock 308 may produce more than one edge between any two vertices (e.g., when twodata objects 102 sharing two different chunks 104). The processing atblocks - At
block 312, the edges shown inblock 310 are received and grouped by edge to generate the ordered edge list shown inblock 314. In an embodiment, this is performed by a sorting method, such as one or more of those described above with respect to block 304. Atblock 316, the edge weights 202 are aggregated to generate the data shown inblock 318. In an embodiment, atblock 316, a sum reduction is performed over the weights of similar edges via a single scan over the ordered edge list shown inblock 314. The resulting minimal edge data object centric graph may be left in table form as shown inblock 318 and/or the data inblock 318 may be used to depict a graph such as that shown inFIG. 2 . - Embodiments of the graph generation process described herein do not require excessive memory requirements. The performance of the two major operations, hash and edge grouping, can benefit from additional physical memory, but can still be processed efficiently on disk.
- Graphs are well known for their expressive power and this can be leveraged by encoding sharing constraints among data objects 102. For example, if two
data objects 102 are supposed to be separated into two deduplication domains, these two vertices can be tagged as having different colors so that subsequent algorithms partitioning the graph can take into account this sharing constraints. More complicated sharing can also be applied to embodiments of the graphs. Any sharing constraint can be modeled as long as the constraint can be expressed by vertices and edges of the graph. - In embodiments of the data object centric graph, a star graph heuristic is used where one of the data objects 102 is linked with every other data object 102. In other embodiments, a chain graph heuristic is used where the data objects 102 are linked to each other in a linked list. For the star topology, the master node (representing a data object 102) is connected with “n−1” nodes, so it has a degree of “n−1”, while the rest of the nodes (representing data objects 102) have a degree of 1. In the chain topology, on the other hand, the two end nodes have degrees of 1 while the intermediary nodes have degrees of 2, so the distribution of node degrees is more balanced. Note that between these two heuristics there are many other ways to link the “n” data objects 102 by a minimal set of edges.
- Turning now to
FIG. 4 , star and chain data object centric graphs that may be implemented in accordance with embodiments are generally shown.FIG. 4 shows a block diagram of a table 402 summarizing content of nine files in a small folder, a data object centric graph withstar connectivity 404 built from the data in the table 402, and a data object centric graph withchain connectivity 406 built from the data in the table 402. The table 402 summarizes the content of the nine files, each file is made up of a sequence of data segments made up of four bytes. The first data segment is different for each file, but several data segments are common between files. The last column of the table 402 shows the size of each file in bytes, the total size of the nine files in the directory is 148 bytes. - In an embodiment, to determine content sharing, for example between files in a folder (or in an entire file system), trace data is collected by executing a file scanning program that traverses each file of the folder (or file system). In an embodiment, the trace contains, for each file, a sequence of SHA-1 cryptographic hash function content hashes, one hash for each data segment contained in the file. In this example, fixed data segments of 4 bytes are used, however in real applications the data segments sizes will be much larger (e.g., 4 kilobytes, 8 kilobytes, or more).
- The star topology of the data object centric graph with
star connectivity 404 includes a vertex for each file (v1, v2, v3 . . . v9) and edges represent sharing of content between vertices. To have a minimum number of edges in the graph, the shared content is represented only once. Representing the shared content only once provides the ability to determine the deduplicated size of the folder (or file system) as the sum of vertex sizes (raw file size) minus the sum of connecting edge weights. This topology is referred to as a star topology because for the “n” number of vertices sharing the same content, one is selected (i.e., the center of the star) to be connected with the other vertices. An alternative to the star topology is to use a chain topology as shown in the data object centric graph withchain connectivity 406 inFIG. 4 . - Embodiments of the data segment and object centric graphs may be used to aid in determining data object groupings for backup and restore operations. For example, in the scenario where a deduplicated storage system is backed-up to a tape libraries, the graphs can be used to determine the specific tape a data segment should be placed on for optimal data object restore performance. Another scenario is informing data segment placement in deduplicated disk-based pools in large shared-nothing commodity server clusters, where servers have asymmetric access to disks, for optimal data object restore performance, or to reduce failure propagation on disk pools (if a disk fails, data loss does not spread to other disks)
- In embodiments, the granularity of the data object is selected based on a granularity of a restore operation (also referred to herein as a “recovery operation”) associated with the deduplicated storage system. For example, if the granularity of a restore operation is a file, then each data object can be a file; if the granularity of a restore operation is a volume, each data object can represent a volume.
- Turning now to
FIG. 5 , a block diagram of a system for providing graph modeling of metadata for deduplicated storage systems in accordance with an embodiment is generally shown. As shown inFIG. 5 , ahost computer 502 is in communication with astorage controller 504 in astorage system 510. Thestorage system 510 stores data in a storage array made up of “K” storage devices 506 (where K is greater than one): storage device one 506 a, storage device two 506 b, storage device three 506 c, throughstorage device K 506 d. In an embodiment, data objects 102 and metadata aboutdata segments 104 shared by the data objects 102 (i.e., the shared data segments) are stored on one or more of the storage devices 506. In an embodiment, the storage devices 506 in thestorage array 508 are implemented by any storage devices known in the art such as, but not limited to: tapes/tape drives and/or disks/hard disk drives (HDDs). Though not shown inFIG. 5 , one or both of thestorage controller 504 and the host computer may contain memory devices to store data, such as all or a portion of the metadata, graphs and other data described herein. In addition, thestorage system 510 and/or thehost computer 502 may implement a tiered hierarchy of memory and storage that includes memory devices (containing memory and/or cached memory), disks and tape devices. As shown inFIG. 5 , thestorage controller 504 is part of thestorage system 510; in another embodiment, thestorage controller 504 is part of thehost computer 502. As used herein, the term “processor” refers to one or more processors located in thehost computer 502 and/or thestorage controller 504. - Technical effects and benefits include providing a general graph based metadata layer that represents sharing between user defined data objects 102. Embodiments also provide a light weight, yet accurate, representation of sharing between data objects 102. The representation of sharing is elastic in that it can be updated incrementally when data object 102 contents change. In addition, embodiments provide a flexible data structure that can incorporate sharing constraints among data objects 102. In embodiments, the graph that represents sharing between data objects 102 is less than one percent of the size of a contemporary hash table representation of the metadata.
- The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used herein, the singular forms “a”, “an” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms “comprises” and/or “comprising,” when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.
- The corresponding structures, materials, acts, and equivalents of all means or step plus function elements in the claims below are intended to include any structure, material, or act for performing the function in combination with other claimed elements as specifically claimed. The description of the present invention has been presented for purposes of illustration and description, but is not intended to be exhaustive or limited to the invention in the form disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the invention. The embodiments were chosen and described in order to best explain the principles of the invention and the practical application, and to enable others of ordinary skill in the art to understand the invention for various embodiments with various modifications as are suited to the particular use contemplated.
- Further, as will be appreciated by one skilled in the art, aspects of the present invention may be embodied as a system, method, or computer program product. Accordingly, aspects of the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, micro-code, etc.) or an embodiment combining software and hardware aspects that may all generally be referred to herein as a “circuit,” “module” or “system.” Furthermore, aspects of the present invention may take the form of a computer program product embodied in one or more computer readable medium(s) having computer readable program code embodied thereon.
- Any combination of one or more computer readable medium(s) may be utilized. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
- A computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.
- Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, radio frequency (RF), etc., or any suitable combination of the foregoing.
- Computer program code for carrying out operations for aspects of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages, such as the “C” programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet Service Provider).
- Aspects of the present invention are described above with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks.
- These computer program instructions may also be stored in a computer readable medium that can direct a computer, other programmable data processing apparatus, or other devices to function in a particular manner, such that the instructions stored in the computer readable medium produce an article of manufacture including instructions which implement the function/act specified in the flowchart and/or block diagram block or blocks.
- The computer program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other devices to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks.
- The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems that perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
Claims (20)
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US13/832,043 US9195673B2 (en) | 2013-03-15 | 2013-03-15 | Scalable graph modeling of metadata for deduplicated storage systems |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US13/832,043 US9195673B2 (en) | 2013-03-15 | 2013-03-15 | Scalable graph modeling of metadata for deduplicated storage systems |
Publications (2)
Publication Number | Publication Date |
---|---|
US20140279927A1 true US20140279927A1 (en) | 2014-09-18 |
US9195673B2 US9195673B2 (en) | 2015-11-24 |
Family
ID=51532993
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US13/832,043 Expired - Fee Related US9195673B2 (en) | 2013-03-15 | 2013-03-15 | Scalable graph modeling of metadata for deduplicated storage systems |
Country Status (1)
Country | Link |
---|---|
US (1) | US9195673B2 (en) |
Cited By (71)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20150012698A1 (en) * | 2013-07-08 | 2015-01-08 | Dell Products L.P. | Restoring temporal locality in global and local deduplication storage systems |
US20150058297A1 (en) * | 2013-08-21 | 2015-02-26 | International Business Machines Corporation | Adding cooperative file coloring protocols in a data deduplication system |
US20160371352A1 (en) * | 2015-06-22 | 2016-12-22 | Microsoft Technology Licensing, Llc | Document storage for reuse of content within documents |
CN106844533A (en) * | 2016-12-29 | 2017-06-13 | 北京师范大学 | A kind of packet method for congregating and device |
US20170344586A1 (en) * | 2016-05-27 | 2017-11-30 | International Business Machines Corporation | De-Duplication Optimized Platform for Object Grouping |
US20170344598A1 (en) * | 2016-05-27 | 2017-11-30 | International Business Machines Corporation | De-Duplication Optimized Platform for Object Grouping |
US9911039B1 (en) * | 2015-09-08 | 2018-03-06 | National Technology & Engineering Solutions Of Sandia, Llc | Searches over graphs representing geospatial-temporal remote sensing data |
CN108132945A (en) * | 2016-12-01 | 2018-06-08 | 百度在线网络技术(北京)有限公司 | For generating the method and apparatus of report |
US10228957B2 (en) | 2017-01-20 | 2019-03-12 | International Business Machines Corporation | Online method handle deduplication |
US10242021B2 (en) | 2016-01-12 | 2019-03-26 | International Business Machines Corporation | Storing data deduplication metadata in a grid of processors |
CN109542854A (en) * | 2018-11-14 | 2019-03-29 | 网易(杭州)网络有限公司 | Data compression method, device, medium and electronic equipment |
US10255288B2 (en) | 2016-01-12 | 2019-04-09 | International Business Machines Corporation | Distributed data deduplication in a grid of processors |
US10261946B2 (en) | 2016-01-12 | 2019-04-16 | International Business Machines Corporation | Rebalancing distributed metadata |
US10324942B2 (en) * | 2013-07-26 | 2019-06-18 | Snap Inc. | Segment data visibility and management in a distributed database of time stamped records |
US10339183B2 (en) | 2015-06-22 | 2019-07-02 | Microsoft Technology Licensing, Llc | Document storage for reuse of content within documents |
US10394949B2 (en) | 2015-06-22 | 2019-08-27 | Microsoft Technology Licensing, Llc | Deconstructing documents into component blocks for reuse in productivity applications |
US10417258B2 (en) | 2013-12-19 | 2019-09-17 | Exposit Labs, Inc. | Interactive multi-dimensional nested table supporting scalable real-time querying of large data volumes |
US10769158B2 (en) * | 2018-05-15 | 2020-09-08 | National Technology & Engineering Solutions od Sandia, LLC | Computer processing through distance-based quality score method in geospatial-temporal semantic graphs |
US10795586B2 (en) | 2018-11-19 | 2020-10-06 | Alibaba Group Holding Limited | System and method for optimization of global data placement to mitigate wear-out of write cache and NAND flash |
US10831404B2 (en) | 2018-02-08 | 2020-11-10 | Alibaba Group Holding Limited | Method and system for facilitating high-capacity shared memory using DIMM from retired servers |
US10852948B2 (en) * | 2018-10-19 | 2020-12-01 | Alibaba Group Holding | System and method for data organization in shingled magnetic recording drive |
US10871921B2 (en) | 2018-07-30 | 2020-12-22 | Alibaba Group Holding Limited | Method and system for facilitating atomicity assurance on metadata and data bundled storage |
US10877898B2 (en) | 2017-11-16 | 2020-12-29 | Alibaba Group Holding Limited | Method and system for enhancing flash translation layer mapping flexibility for performance and lifespan improvements |
US10884926B2 (en) | 2017-06-16 | 2021-01-05 | Alibaba Group Holding Limited | Method and system for distributed storage using client-side global persistent cache |
US10891065B2 (en) | 2019-04-01 | 2021-01-12 | Alibaba Group Holding Limited | Method and system for online conversion of bad blocks for improvement of performance and longevity in a solid state drive |
US10891239B2 (en) | 2018-02-07 | 2021-01-12 | Alibaba Group Holding Limited | Method and system for operating NAND flash physical space to extend memory capacity |
US10908960B2 (en) | 2019-04-16 | 2021-02-02 | Alibaba Group Holding Limited | Resource allocation based on comprehensive I/O monitoring in a distributed storage system |
US10922234B2 (en) | 2019-04-11 | 2021-02-16 | Alibaba Group Holding Limited | Method and system for online recovery of logical-to-physical mapping table affected by noise sources in a solid state drive |
US10921992B2 (en) | 2018-06-25 | 2021-02-16 | Alibaba Group Holding Limited | Method and system for data placement in a hard disk drive based on access frequency for improved IOPS and utilization efficiency |
US10923156B1 (en) | 2020-02-19 | 2021-02-16 | Alibaba Group Holding Limited | Method and system for facilitating low-cost high-throughput storage for accessing large-size I/O blocks in a hard disk drive |
US10970212B2 (en) | 2019-02-15 | 2021-04-06 | Alibaba Group Holding Limited | Method and system for facilitating a distributed storage system with a total cost of ownership reduction for multiple available zones |
US10977122B2 (en) | 2018-12-31 | 2021-04-13 | Alibaba Group Holding Limited | System and method for facilitating differentiated error correction in high-density flash devices |
US10996886B2 (en) | 2018-08-02 | 2021-05-04 | Alibaba Group Holding Limited | Method and system for facilitating atomicity and latency assurance on variable sized I/O |
US11061735B2 (en) | 2019-01-02 | 2021-07-13 | Alibaba Group Holding Limited | System and method for offloading computation to storage nodes in distributed system |
US11061834B2 (en) | 2019-02-26 | 2021-07-13 | Alibaba Group Holding Limited | Method and system for facilitating an improved storage system by decoupling the controller from the storage medium |
US11068409B2 (en) | 2018-02-07 | 2021-07-20 | Alibaba Group Holding Limited | Method and system for user-space storage I/O stack with user-space flash translation layer |
US11074124B2 (en) | 2019-07-23 | 2021-07-27 | Alibaba Group Holding Limited | Method and system for enhancing throughput of big data analysis in a NAND-based read source storage |
US11126561B2 (en) | 2019-10-01 | 2021-09-21 | Alibaba Group Holding Limited | Method and system for organizing NAND blocks and placing data to facilitate high-throughput for random writes in a solid state drive |
US11132291B2 (en) | 2019-01-04 | 2021-09-28 | Alibaba Group Holding Limited | System and method of FPGA-executed flash translation layer in multiple solid state drives |
US11144250B2 (en) | 2020-03-13 | 2021-10-12 | Alibaba Group Holding Limited | Method and system for facilitating a persistent memory-centric system |
US11150986B2 (en) | 2020-02-26 | 2021-10-19 | Alibaba Group Holding Limited | Efficient compaction on log-structured distributed file system using erasure coding for resource consumption reduction |
US11169873B2 (en) | 2019-05-21 | 2021-11-09 | Alibaba Group Holding Limited | Method and system for extending lifespan and enhancing throughput in a high-density solid state drive |
US11188566B2 (en) | 2018-10-25 | 2021-11-30 | International Business Machines Corporation | Grouping of objects into clusters in an object-based storage environment |
US11200114B2 (en) | 2020-03-17 | 2021-12-14 | Alibaba Group Holding Limited | System and method for facilitating elastic error correction code in memory |
US11200337B2 (en) | 2019-02-11 | 2021-12-14 | Alibaba Group Holding Limited | System and method for user data isolation |
US11218165B2 (en) | 2020-05-15 | 2022-01-04 | Alibaba Group Holding Limited | Memory-mapped two-dimensional error correction code for multi-bit error tolerance in DRAM |
US11263132B2 (en) | 2020-06-11 | 2022-03-01 | Alibaba Group Holding Limited | Method and system for facilitating log-structure data organization |
US11281575B2 (en) | 2020-05-11 | 2022-03-22 | Alibaba Group Holding Limited | Method and system for facilitating data placement and control of physical addresses with multi-queue I/O blocks |
US11327929B2 (en) | 2018-09-17 | 2022-05-10 | Alibaba Group Holding Limited | Method and system for reduced data movement compression using in-storage computing and a customized file system |
US11354200B2 (en) | 2020-06-17 | 2022-06-07 | Alibaba Group Holding Limited | Method and system for facilitating data recovery and version rollback in a storage device |
US11354233B2 (en) | 2020-07-27 | 2022-06-07 | Alibaba Group Holding Limited | Method and system for facilitating fast crash recovery in a storage device |
US11372774B2 (en) | 2020-08-24 | 2022-06-28 | Alibaba Group Holding Limited | Method and system for a solid state drive with on-chip memory integration |
US11379155B2 (en) | 2018-05-24 | 2022-07-05 | Alibaba Group Holding Limited | System and method for flash storage management using multiple open page stripes |
US11379127B2 (en) | 2019-07-18 | 2022-07-05 | Alibaba Group Holding Limited | Method and system for enhancing a distributed storage system by decoupling computation and network tasks |
US11385833B2 (en) | 2020-04-20 | 2022-07-12 | Alibaba Group Holding Limited | Method and system for facilitating a light-weight garbage collection with a reduced utilization of resources |
US11416365B2 (en) | 2020-12-30 | 2022-08-16 | Alibaba Group Holding Limited | Method and system for open NAND block detection and correction in an open-channel SSD |
US11422931B2 (en) | 2020-06-17 | 2022-08-23 | Alibaba Group Holding Limited | Method and system for facilitating a physically isolated storage unit for multi-tenancy virtualization |
US11449455B2 (en) | 2020-01-15 | 2022-09-20 | Alibaba Group Holding Limited | Method and system for facilitating a high-capacity object storage system with configuration agility and mixed deployment flexibility |
US11461173B1 (en) | 2021-04-21 | 2022-10-04 | Alibaba Singapore Holding Private Limited | Method and system for facilitating efficient data compression based on error correction code and reorganization of data placement |
US11461262B2 (en) | 2020-05-13 | 2022-10-04 | Alibaba Group Holding Limited | Method and system for facilitating a converged computation and storage node in a distributed storage system |
US11476874B1 (en) | 2021-05-14 | 2022-10-18 | Alibaba Singapore Holding Private Limited | Method and system for facilitating a storage server with hybrid memory for journaling and data storage |
US11487465B2 (en) | 2020-12-11 | 2022-11-01 | Alibaba Group Holding Limited | Method and system for a local storage engine collaborating with a solid state drive controller |
US11494115B2 (en) | 2020-05-13 | 2022-11-08 | Alibaba Group Holding Limited | System method for facilitating memory media as file storage device based on real-time hashing by performing integrity check with a cyclical redundancy check (CRC) |
US11507499B2 (en) | 2020-05-19 | 2022-11-22 | Alibaba Group Holding Limited | System and method for facilitating mitigation of read/write amplification in data compression |
US11556277B2 (en) | 2020-05-19 | 2023-01-17 | Alibaba Group Holding Limited | System and method for facilitating improved performance in ordering key-value storage with input/output stack simplification |
US11617282B2 (en) | 2019-10-01 | 2023-03-28 | Alibaba Group Holding Limited | System and method for reshaping power budget of cabinet to facilitate improved deployment density of servers |
US20230139581A1 (en) * | 2021-10-29 | 2023-05-04 | Scality, S.A. | Data placement in large scale object storage system |
US11726699B2 (en) | 2021-03-30 | 2023-08-15 | Alibaba Singapore Holding Private Limited | Method and system for facilitating multi-stream sequential read performance improvement with reduced read amplification |
US11734115B2 (en) | 2020-12-28 | 2023-08-22 | Alibaba Group Holding Limited | Method and system for facilitating write latency reduction in a queue depth of one scenario |
US11816043B2 (en) | 2018-06-25 | 2023-11-14 | Alibaba Group Holding Limited | System and method for managing resources of a storage device and quantifying the cost of I/O requests |
US11893256B2 (en) | 2020-03-11 | 2024-02-06 | International Business Machines Corporation | Partitioning of deduplication domains in storage systems |
Families Citing this family (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US9753960B1 (en) * | 2013-03-20 | 2017-09-05 | Amdocs Software Systems Limited | System, method, and computer program for dynamically generating a visual representation of a subset of a graph for display, based on search criteria |
Citations (17)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US7143091B2 (en) * | 2002-02-04 | 2006-11-28 | Cataphorn, Inc. | Method and apparatus for sociological data mining |
US20090327625A1 (en) * | 2008-06-30 | 2009-12-31 | International Business Machines Corporation | Managing metadata for data blocks used in a deduplication system |
US20100018499A1 (en) * | 2008-07-24 | 2010-01-28 | Honda Motor Co., Ltd. | Fuel tuned variable valve timing |
US20100049735A1 (en) * | 2008-08-25 | 2010-02-25 | Hsu Windsor W | Method and apparatus for managing data objects of a data storage system |
US20100064166A1 (en) * | 2008-09-11 | 2010-03-11 | Nec Laboratories America, Inc. | Scalable secondary storage systems and methods |
US20100070698A1 (en) * | 2008-09-11 | 2010-03-18 | Nec Laboratories America, Inc. | Content addressable storage systems and methods employing searchable blocks |
US20100082700A1 (en) * | 2008-09-22 | 2010-04-01 | Riverbed Technology, Inc. | Storage system for data virtualization and deduplication |
US20100250501A1 (en) * | 2009-03-26 | 2010-09-30 | International Business Machines Corporation | Storage management through adaptive deduplication |
US20120109907A1 (en) * | 2010-10-30 | 2012-05-03 | International Business Machines Corporation | On-demand data deduplication |
US20120124046A1 (en) * | 2010-11-16 | 2012-05-17 | Actifio, Inc. | System and method for managing deduplicated copies of data using temporal relationships among copies |
US20120124013A1 (en) * | 2010-11-16 | 2012-05-17 | Actifio, Inc. | System and method for creating deduplicated copies of data storing non-lossy encodings of data directly in a content addressable store |
US20120124012A1 (en) * | 2010-11-16 | 2012-05-17 | Actifio, Inc. | System and method for creating deduplicated copies of data by tracking temporal relationships among copies and by ingesting difference data |
US20120124105A1 (en) * | 2010-11-16 | 2012-05-17 | Actifio, Inc. | System and method for improved garbage collection operations in a deduplicated store by tracking temporal relationships among copies |
US8463742B1 (en) * | 2010-09-17 | 2013-06-11 | Permabit Technology Corp. | Managing deduplication of stored data |
US20140006363A1 (en) * | 2012-06-29 | 2014-01-02 | International Business Machines Corporation | Optimized data placement for individual file accesses on deduplication-enabled sequential storage systems |
US8700578B1 (en) * | 2012-09-27 | 2014-04-15 | Emc Corporation | System and method for determining physical storage space of a deduplicated storage system |
US8719234B2 (en) * | 2012-01-25 | 2014-05-06 | International Business Machines Corporation | Handling rewrites in deduplication systems using data parsers |
Family Cites Families (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US8200640B2 (en) | 2009-06-15 | 2012-06-12 | Microsoft Corporation | Declarative framework for deduplication |
-
2013
- 2013-03-15 US US13/832,043 patent/US9195673B2/en not_active Expired - Fee Related
Patent Citations (23)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US7143091B2 (en) * | 2002-02-04 | 2006-11-28 | Cataphorn, Inc. | Method and apparatus for sociological data mining |
US20090327625A1 (en) * | 2008-06-30 | 2009-12-31 | International Business Machines Corporation | Managing metadata for data blocks used in a deduplication system |
US20100018499A1 (en) * | 2008-07-24 | 2010-01-28 | Honda Motor Co., Ltd. | Fuel tuned variable valve timing |
US8316064B2 (en) * | 2008-08-25 | 2012-11-20 | Emc Corporation | Method and apparatus for managing data objects of a data storage system |
US20100049735A1 (en) * | 2008-08-25 | 2010-02-25 | Hsu Windsor W | Method and apparatus for managing data objects of a data storage system |
US20130036104A1 (en) * | 2008-08-25 | 2013-02-07 | Hsu Windsor W | Method and apparatus for managing data objects of a data storage system |
US7992037B2 (en) * | 2008-09-11 | 2011-08-02 | Nec Laboratories America, Inc. | Scalable secondary storage systems and methods |
US20100070698A1 (en) * | 2008-09-11 | 2010-03-18 | Nec Laboratories America, Inc. | Content addressable storage systems and methods employing searchable blocks |
US20100064166A1 (en) * | 2008-09-11 | 2010-03-11 | Nec Laboratories America, Inc. | Scalable secondary storage systems and methods |
US20100088349A1 (en) * | 2008-09-22 | 2010-04-08 | Riverbed Technology, Inc. | Virtual file system stack for data deduplication |
US20100082700A1 (en) * | 2008-09-22 | 2010-04-01 | Riverbed Technology, Inc. | Storage system for data virtualization and deduplication |
US20100250501A1 (en) * | 2009-03-26 | 2010-09-30 | International Business Machines Corporation | Storage management through adaptive deduplication |
US8463742B1 (en) * | 2010-09-17 | 2013-06-11 | Permabit Technology Corp. | Managing deduplication of stored data |
US20120109907A1 (en) * | 2010-10-30 | 2012-05-03 | International Business Machines Corporation | On-demand data deduplication |
US20120124105A1 (en) * | 2010-11-16 | 2012-05-17 | Actifio, Inc. | System and method for improved garbage collection operations in a deduplicated store by tracking temporal relationships among copies |
US8299944B2 (en) * | 2010-11-16 | 2012-10-30 | Actifio, Inc. | System and method for creating deduplicated copies of data storing non-lossy encodings of data directly in a content addressable store |
US20120124012A1 (en) * | 2010-11-16 | 2012-05-17 | Actifio, Inc. | System and method for creating deduplicated copies of data by tracking temporal relationships among copies and by ingesting difference data |
US20120124013A1 (en) * | 2010-11-16 | 2012-05-17 | Actifio, Inc. | System and method for creating deduplicated copies of data storing non-lossy encodings of data directly in a content addressable store |
US20120124046A1 (en) * | 2010-11-16 | 2012-05-17 | Actifio, Inc. | System and method for managing deduplicated copies of data using temporal relationships among copies |
US8843489B2 (en) * | 2010-11-16 | 2014-09-23 | Actifio, Inc. | System and method for managing deduplicated copies of data using temporal relationships among copies |
US8719234B2 (en) * | 2012-01-25 | 2014-05-06 | International Business Machines Corporation | Handling rewrites in deduplication systems using data parsers |
US20140006363A1 (en) * | 2012-06-29 | 2014-01-02 | International Business Machines Corporation | Optimized data placement for individual file accesses on deduplication-enabled sequential storage systems |
US8700578B1 (en) * | 2012-09-27 | 2014-04-15 | Emc Corporation | System and method for determining physical storage space of a deduplicated storage system |
Cited By (78)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20150012698A1 (en) * | 2013-07-08 | 2015-01-08 | Dell Products L.P. | Restoring temporal locality in global and local deduplication storage systems |
US9201800B2 (en) * | 2013-07-08 | 2015-12-01 | Dell Products L.P. | Restoring temporal locality in global and local deduplication storage systems |
US10324942B2 (en) * | 2013-07-26 | 2019-06-18 | Snap Inc. | Segment data visibility and management in a distributed database of time stamped records |
US20150058297A1 (en) * | 2013-08-21 | 2015-02-26 | International Business Machines Corporation | Adding cooperative file coloring protocols in a data deduplication system |
US11048594B2 (en) | 2013-08-21 | 2021-06-29 | International Business Machines Corporation | Adding cooperative file coloring protocols in a data deduplication system |
US9830229B2 (en) * | 2013-08-21 | 2017-11-28 | International Business Machines Corporation | Adding cooperative file coloring protocols in a data deduplication system |
US10417258B2 (en) | 2013-12-19 | 2019-09-17 | Exposit Labs, Inc. | Interactive multi-dimensional nested table supporting scalable real-time querying of large data volumes |
US10740349B2 (en) * | 2015-06-22 | 2020-08-11 | Microsoft Technology Licensing, Llc | Document storage for reuse of content within documents |
US10394949B2 (en) | 2015-06-22 | 2019-08-27 | Microsoft Technology Licensing, Llc | Deconstructing documents into component blocks for reuse in productivity applications |
US20160371352A1 (en) * | 2015-06-22 | 2016-12-22 | Microsoft Technology Licensing, Llc | Document storage for reuse of content within documents |
US10339183B2 (en) | 2015-06-22 | 2019-07-02 | Microsoft Technology Licensing, Llc | Document storage for reuse of content within documents |
US9911039B1 (en) * | 2015-09-08 | 2018-03-06 | National Technology & Engineering Solutions Of Sandia, Llc | Searches over graphs representing geospatial-temporal remote sensing data |
US10242021B2 (en) | 2016-01-12 | 2019-03-26 | International Business Machines Corporation | Storing data deduplication metadata in a grid of processors |
US10255288B2 (en) | 2016-01-12 | 2019-04-09 | International Business Machines Corporation | Distributed data deduplication in a grid of processors |
US10261946B2 (en) | 2016-01-12 | 2019-04-16 | International Business Machines Corporation | Rebalancing distributed metadata |
US20170344598A1 (en) * | 2016-05-27 | 2017-11-30 | International Business Machines Corporation | De-Duplication Optimized Platform for Object Grouping |
US20170344586A1 (en) * | 2016-05-27 | 2017-11-30 | International Business Machines Corporation | De-Duplication Optimized Platform for Object Grouping |
CN108132945A (en) * | 2016-12-01 | 2018-06-08 | 百度在线网络技术(北京)有限公司 | For generating the method and apparatus of report |
CN106844533A (en) * | 2016-12-29 | 2017-06-13 | 北京师范大学 | A kind of packet method for congregating and device |
US10649797B2 (en) | 2017-01-20 | 2020-05-12 | International Business Machines Corporation | Online method handle deduplication |
US10228957B2 (en) | 2017-01-20 | 2019-03-12 | International Business Machines Corporation | Online method handle deduplication |
US10884926B2 (en) | 2017-06-16 | 2021-01-05 | Alibaba Group Holding Limited | Method and system for distributed storage using client-side global persistent cache |
US10877898B2 (en) | 2017-11-16 | 2020-12-29 | Alibaba Group Holding Limited | Method and system for enhancing flash translation layer mapping flexibility for performance and lifespan improvements |
US10891239B2 (en) | 2018-02-07 | 2021-01-12 | Alibaba Group Holding Limited | Method and system for operating NAND flash physical space to extend memory capacity |
US11068409B2 (en) | 2018-02-07 | 2021-07-20 | Alibaba Group Holding Limited | Method and system for user-space storage I/O stack with user-space flash translation layer |
US10831404B2 (en) | 2018-02-08 | 2020-11-10 | Alibaba Group Holding Limited | Method and system for facilitating high-capacity shared memory using DIMM from retired servers |
US10769158B2 (en) * | 2018-05-15 | 2020-09-08 | National Technology & Engineering Solutions od Sandia, LLC | Computer processing through distance-based quality score method in geospatial-temporal semantic graphs |
US11379155B2 (en) | 2018-05-24 | 2022-07-05 | Alibaba Group Holding Limited | System and method for flash storage management using multiple open page stripes |
US11816043B2 (en) | 2018-06-25 | 2023-11-14 | Alibaba Group Holding Limited | System and method for managing resources of a storage device and quantifying the cost of I/O requests |
US10921992B2 (en) | 2018-06-25 | 2021-02-16 | Alibaba Group Holding Limited | Method and system for data placement in a hard disk drive based on access frequency for improved IOPS and utilization efficiency |
US10871921B2 (en) | 2018-07-30 | 2020-12-22 | Alibaba Group Holding Limited | Method and system for facilitating atomicity assurance on metadata and data bundled storage |
US10996886B2 (en) | 2018-08-02 | 2021-05-04 | Alibaba Group Holding Limited | Method and system for facilitating atomicity and latency assurance on variable sized I/O |
US11327929B2 (en) | 2018-09-17 | 2022-05-10 | Alibaba Group Holding Limited | Method and system for reduced data movement compression using in-storage computing and a customized file system |
US10852948B2 (en) * | 2018-10-19 | 2020-12-01 | Alibaba Group Holding | System and method for data organization in shingled magnetic recording drive |
US11188566B2 (en) | 2018-10-25 | 2021-11-30 | International Business Machines Corporation | Grouping of objects into clusters in an object-based storage environment |
CN109542854A (en) * | 2018-11-14 | 2019-03-29 | 网易(杭州)网络有限公司 | Data compression method, device, medium and electronic equipment |
US10795586B2 (en) | 2018-11-19 | 2020-10-06 | Alibaba Group Holding Limited | System and method for optimization of global data placement to mitigate wear-out of write cache and NAND flash |
US10977122B2 (en) | 2018-12-31 | 2021-04-13 | Alibaba Group Holding Limited | System and method for facilitating differentiated error correction in high-density flash devices |
US11061735B2 (en) | 2019-01-02 | 2021-07-13 | Alibaba Group Holding Limited | System and method for offloading computation to storage nodes in distributed system |
US11768709B2 (en) | 2019-01-02 | 2023-09-26 | Alibaba Group Holding Limited | System and method for offloading computation to storage nodes in distributed system |
US11132291B2 (en) | 2019-01-04 | 2021-09-28 | Alibaba Group Holding Limited | System and method of FPGA-executed flash translation layer in multiple solid state drives |
US11200337B2 (en) | 2019-02-11 | 2021-12-14 | Alibaba Group Holding Limited | System and method for user data isolation |
US10970212B2 (en) | 2019-02-15 | 2021-04-06 | Alibaba Group Holding Limited | Method and system for facilitating a distributed storage system with a total cost of ownership reduction for multiple available zones |
US11061834B2 (en) | 2019-02-26 | 2021-07-13 | Alibaba Group Holding Limited | Method and system for facilitating an improved storage system by decoupling the controller from the storage medium |
US10891065B2 (en) | 2019-04-01 | 2021-01-12 | Alibaba Group Holding Limited | Method and system for online conversion of bad blocks for improvement of performance and longevity in a solid state drive |
US10922234B2 (en) | 2019-04-11 | 2021-02-16 | Alibaba Group Holding Limited | Method and system for online recovery of logical-to-physical mapping table affected by noise sources in a solid state drive |
US10908960B2 (en) | 2019-04-16 | 2021-02-02 | Alibaba Group Holding Limited | Resource allocation based on comprehensive I/O monitoring in a distributed storage system |
US11169873B2 (en) | 2019-05-21 | 2021-11-09 | Alibaba Group Holding Limited | Method and system for extending lifespan and enhancing throughput in a high-density solid state drive |
US11379127B2 (en) | 2019-07-18 | 2022-07-05 | Alibaba Group Holding Limited | Method and system for enhancing a distributed storage system by decoupling computation and network tasks |
US11074124B2 (en) | 2019-07-23 | 2021-07-27 | Alibaba Group Holding Limited | Method and system for enhancing throughput of big data analysis in a NAND-based read source storage |
US11126561B2 (en) | 2019-10-01 | 2021-09-21 | Alibaba Group Holding Limited | Method and system for organizing NAND blocks and placing data to facilitate high-throughput for random writes in a solid state drive |
US11617282B2 (en) | 2019-10-01 | 2023-03-28 | Alibaba Group Holding Limited | System and method for reshaping power budget of cabinet to facilitate improved deployment density of servers |
US11449455B2 (en) | 2020-01-15 | 2022-09-20 | Alibaba Group Holding Limited | Method and system for facilitating a high-capacity object storage system with configuration agility and mixed deployment flexibility |
US10923156B1 (en) | 2020-02-19 | 2021-02-16 | Alibaba Group Holding Limited | Method and system for facilitating low-cost high-throughput storage for accessing large-size I/O blocks in a hard disk drive |
US11150986B2 (en) | 2020-02-26 | 2021-10-19 | Alibaba Group Holding Limited | Efficient compaction on log-structured distributed file system using erasure coding for resource consumption reduction |
US11893256B2 (en) | 2020-03-11 | 2024-02-06 | International Business Machines Corporation | Partitioning of deduplication domains in storage systems |
US11144250B2 (en) | 2020-03-13 | 2021-10-12 | Alibaba Group Holding Limited | Method and system for facilitating a persistent memory-centric system |
US11200114B2 (en) | 2020-03-17 | 2021-12-14 | Alibaba Group Holding Limited | System and method for facilitating elastic error correction code in memory |
US11385833B2 (en) | 2020-04-20 | 2022-07-12 | Alibaba Group Holding Limited | Method and system for facilitating a light-weight garbage collection with a reduced utilization of resources |
US11281575B2 (en) | 2020-05-11 | 2022-03-22 | Alibaba Group Holding Limited | Method and system for facilitating data placement and control of physical addresses with multi-queue I/O blocks |
US11494115B2 (en) | 2020-05-13 | 2022-11-08 | Alibaba Group Holding Limited | System method for facilitating memory media as file storage device based on real-time hashing by performing integrity check with a cyclical redundancy check (CRC) |
US11461262B2 (en) | 2020-05-13 | 2022-10-04 | Alibaba Group Holding Limited | Method and system for facilitating a converged computation and storage node in a distributed storage system |
US11218165B2 (en) | 2020-05-15 | 2022-01-04 | Alibaba Group Holding Limited | Memory-mapped two-dimensional error correction code for multi-bit error tolerance in DRAM |
US11507499B2 (en) | 2020-05-19 | 2022-11-22 | Alibaba Group Holding Limited | System and method for facilitating mitigation of read/write amplification in data compression |
US11556277B2 (en) | 2020-05-19 | 2023-01-17 | Alibaba Group Holding Limited | System and method for facilitating improved performance in ordering key-value storage with input/output stack simplification |
US11263132B2 (en) | 2020-06-11 | 2022-03-01 | Alibaba Group Holding Limited | Method and system for facilitating log-structure data organization |
US11422931B2 (en) | 2020-06-17 | 2022-08-23 | Alibaba Group Holding Limited | Method and system for facilitating a physically isolated storage unit for multi-tenancy virtualization |
US11354200B2 (en) | 2020-06-17 | 2022-06-07 | Alibaba Group Holding Limited | Method and system for facilitating data recovery and version rollback in a storage device |
US11354233B2 (en) | 2020-07-27 | 2022-06-07 | Alibaba Group Holding Limited | Method and system for facilitating fast crash recovery in a storage device |
US11372774B2 (en) | 2020-08-24 | 2022-06-28 | Alibaba Group Holding Limited | Method and system for a solid state drive with on-chip memory integration |
US11487465B2 (en) | 2020-12-11 | 2022-11-01 | Alibaba Group Holding Limited | Method and system for a local storage engine collaborating with a solid state drive controller |
US11734115B2 (en) | 2020-12-28 | 2023-08-22 | Alibaba Group Holding Limited | Method and system for facilitating write latency reduction in a queue depth of one scenario |
US11416365B2 (en) | 2020-12-30 | 2022-08-16 | Alibaba Group Holding Limited | Method and system for open NAND block detection and correction in an open-channel SSD |
US11726699B2 (en) | 2021-03-30 | 2023-08-15 | Alibaba Singapore Holding Private Limited | Method and system for facilitating multi-stream sequential read performance improvement with reduced read amplification |
US11461173B1 (en) | 2021-04-21 | 2022-10-04 | Alibaba Singapore Holding Private Limited | Method and system for facilitating efficient data compression based on error correction code and reorganization of data placement |
US11476874B1 (en) | 2021-05-14 | 2022-10-18 | Alibaba Singapore Holding Private Limited | Method and system for facilitating a storage server with hybrid memory for journaling and data storage |
US20230139581A1 (en) * | 2021-10-29 | 2023-05-04 | Scality, S.A. | Data placement in large scale object storage system |
US11922042B2 (en) * | 2021-10-29 | 2024-03-05 | Scality, S.A. | Data placement in large scale object storage system |
Also Published As
Publication number | Publication date |
---|---|
US9195673B2 (en) | 2015-11-24 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US9195673B2 (en) | Scalable graph modeling of metadata for deduplicated storage systems | |
US11636031B2 (en) | Optimized inline deduplication | |
US12038927B2 (en) | Storage system having multiple tables for efficient searching | |
US10649988B1 (en) | Artificial intelligence and machine learning infrastructure | |
US10996875B2 (en) | Making more active use of a secondary storage system | |
US11403290B1 (en) | Managing an artificial intelligence infrastructure | |
US10613791B2 (en) | Portable snapshot replication between storage systems | |
US11966841B2 (en) | Search acceleration for artificial intelligence | |
US20210303523A1 (en) | Replicating a File System | |
US20210303528A1 (en) | Managing Directory-Tree Operations in File Storage | |
US12079162B2 (en) | Snapshot management in a storage system | |
US20230016822A1 (en) | Creating Batches Of Training Data For Machine Learning Workflows | |
US10108644B1 (en) | Method for minimizing storage requirements on fast/expensive arrays for data mobility and migration | |
US20210303511A1 (en) | Cloning a Managed Directory of a File System | |
US20220269601A1 (en) | Cost Effective Storage Management | |
US11422731B1 (en) | Metadata-based replication of a dataset | |
US20240004570A1 (en) | Storage cluster data structure expansion | |
JP2024528364A (en) | Generate a Dataset Using Approximate Baselines | |
US9678971B2 (en) | Packing deduplicated data in a self-contained deduplicated repository | |
CN106575296B (en) | Dynamic N-dimensional cube for hosted analytics | |
US20240111718A1 (en) | In-band file system access | |
US11861423B1 (en) | Accelerating artificial intelligence (‘AI’) workflows | |
US20230097507A1 (en) | Data backup optimization |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: INTERNATIONAL BUSINESS MACHINES CORPORATION, NEW Y Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:CONSTANTINESCU, MIHAIL CORNELIU;GHARAIBEH, ABDULLAH;LU, MAOHUA;SIGNING DATES FROM 20130311 TO 20130314;REEL/FRAME:030008/0822 |
|
STCF | Information on status: patent grant |
Free format text: PATENTED CASE |
|
MAFP | Maintenance fee payment |
Free format text: PAYMENT OF MAINTENANCE FEE, 4TH YEAR, LARGE ENTITY (ORIGINAL EVENT CODE: M1551); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY Year of fee payment: 4 |
|
FEPP | Fee payment procedure |
Free format text: MAINTENANCE FEE REMINDER MAILED (ORIGINAL EVENT CODE: REM.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY |
|
LAPS | Lapse for failure to pay maintenance fees |
Free format text: PATENT EXPIRED FOR FAILURE TO PAY MAINTENANCE FEES (ORIGINAL EVENT CODE: EXP.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY |
|
STCH | Information on status: patent discontinuation |
Free format text: PATENT EXPIRED DUE TO NONPAYMENT OF MAINTENANCE FEES UNDER 37 CFR 1.362 |
|
FP | Lapsed due to failure to pay maintenance fee |
Effective date: 20231124 |