US20190391917A1 - Shallow cache for content replication - Google Patents
Shallow cache for content replication Download PDFInfo
- Publication number
- US20190391917A1 US20190391917A1 US16/017,985 US201816017985A US2019391917A1 US 20190391917 A1 US20190391917 A1 US 20190391917A1 US 201816017985 A US201816017985 A US 201816017985A US 2019391917 A1 US2019391917 A1 US 2019391917A1
- Authority
- US
- United States
- Prior art keywords
- paths
- cache
- path
- hash
- content
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/13—File access structures, e.g. distributed indices
- G06F16/137—Hash-based
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F12/00—Accessing, addressing or allocating within memory systems or architectures
- G06F12/02—Addressing or allocation; Relocation
- G06F12/08—Addressing or allocation; Relocation in hierarchically structured memory systems, e.g. virtual memory systems
- G06F12/0802—Addressing of a memory level in which the access to the desired data or data block requires associative addressing means, e.g. caches
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/11—File system administration, e.g. details of archiving or snapshots
- G06F16/119—Details of migration of file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F12/00—Accessing, addressing or allocating within memory systems or architectures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
- G06F16/184—Distributed file systems implemented as replicated file system
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2212/00—Indexing scheme relating to accessing, addressing or allocation within memory systems or architectures
- G06F2212/27—Using a specific cache architecture
Definitions
- Computing systems store and manage hierarchies of content units or data items. Each content unit in such a hierarchy or storage space has a path or location. The paths are all in a same namespace, and the content units can be accessed by referring to their paths. Often, portions of one storage space need to be copied to another storage space. That is, a set of content units at respective paths in a source storage space may be need to be copied to a target storage space sharing the same namespace. For example, a directory in a remote filesystem may need to be copied to a directory in a local filesystem.
- same instances of a content unit might be stored in multiple paths in a storage space.
- filesystem files there may be multiple files containing the same content but at different full paths in the file system.
- the same file content “X” might be stored at “ ⁇ A ⁇ B” and at “ ⁇ A ⁇ C ⁇ D”.
- Files “B” and “D” store the same content, but at different locations and with different file names.
- the duplication of content units may have inefficiencies.
- the same content may be transferred from source to target multiple times. Referring to the example above, it would be redundant wasteful to copy both files “B” and “D” from source to target when they contain the same content.
- another inefficiency may occur when content units of a portion of source storage space to be transferred to target storage space already exist at the target storage space.
- Embodiments relate to efficiently replicating data from a source storage space to a target storage space.
- the storage spaces share a common namespace of paths where content units are stored.
- a shallow cache is maintained for the target storage space.
- Each entry in the cache includes a hash of a content unit in the target storage space and associated hierarchy paths in the target storage space where the corresponding content unit is stored.
- FIG. 1 shows a system for replicating a dataset of data items from a source storage space to a target storage space.
- FIG. 2 shows additional detail of a dataset.
- FIG. 3 shows a process for building a dataset.
- FIG. 4 shows a target storage space supplemented by a shallow cache.
- FIG. 5 shows a process for initializing the shallow cache.
- FIG. 6 shows a process for using a shallow cache to replicate a dataset to a target storage space.
- FIG. 7 shows an example of replication that initializes a shallow cache.
- FIG. 8 shows an example of using a shallow cache for replicating a dataset.
- FIG. 9 shows details of a computing device on which embodiments described herein may be implemented.
- FIG. 1 shows a system for replicating a dataset 100 of data items from a source storage space 100 to a target storage space 102 .
- the data items or content units may be any type of discrete unit of data such as files, memory chunks, media data, etc., so long as they are individually distinguishable and accessible. They may be of variable length or uniform size.
- the storage spaces 102 / 104 may be co-located on a same computing device or they may be on respective different computing devices connected by a network. Of note is that they share a same namespace.
- storage spaces generally organize their content units in hierarchical fashion. Content units in a storage space may be arranged or located by names, paths, identifiers of tree nodes, etc. The term “paths” will be used hereafter to refer to these types of hierarchical location descriptors.
- a common operation is to replicate a portion of a source storage space 102 to a target storage space 104 .
- this involves duplicating a dataset 100 that includes a set of paths in the source storage space 102 and corresponding content units that reside at the paths in the source storage space 102 .
- the dataset 100 may be an entire directory or a set of disperse files.
- the dataset 100 or other structuring of data for transfer may involve a software component such as a dataset builder 106 at the source side that collects the necessary data and information to allow a receiver to reconstruct the portion of source data that is being prepared. Any known packaging tools may be used.
- the dataset 100 may be a combination of indicia of the portion of source data, for instance a manifest of content unit paths.
- the dataset 100 may optionally include copies of content units.
- a manifest is used, and the target side may use the manifest to decide which content units to pull on an as-needed basis.
- the dataset builder 106 may perform a build process 108 that includes steps such as receiving a request to build the dataset and then building the dataset.
- the request may specify a set of paths, a path root or subtree, and so forth.
- building the dataset 100 includes identifying content items that have the same content, providing an identifier for the content (e.g., a hash), and adding the identifier or hash and the corresponding source paths that contain the same corresponding content.
- the dataset 100 is a list of each unique hash in the dataset, and each hash is accompanied by a list of full pathnames of files with the same hash.
- a data replicator process 110 performs a replication process 112 .
- the data replicator process 110 may include steps such as selecting the dataset 100 , providing a request for the dataset 100 to the source side, copying at least indicia of the content in the dataset, and then copying the relevant content units from the source storage space 102 and distributing them to the correct namespace locations or paths in the target storage space 104 .
- FIG. 1 The system shown in FIG. 1 is only an example. In different embodiments may perform the steps in different orders or at different places. It is possible for all of the building and replication to take place on the source side or the target side. Building and replication may be performed in a combined fashion.
- FIG. 2 shows additional detail of a dataset.
- the source storage space 102 stores content units 120 .
- the content units 120 may be stored in a hierarchy (not shown) using the same (or overlapping) namespace as that of the target storage space 104 .
- the dataset builder 106 may start with a list of paths of content units to be built or packaged.
- a hash function 122 is applied to each content unit 120 at each path in the list creating hashes 121 for the respective content units. Hashing might be performed in advance on all content units in the source storage space.
- the dataset builder 106 builds a manifest 124 by consolidating the hashes of the listed content units. Specifically, any content units that have the same hash contain the same content. Such hash is added to the manifest 124 along with the set of all paths in the source storage space 102 of the respective content units having the same hash. The result is that the manifest 124 represents content units in groups that have the same content (and hash value).
- the dataset 100 may include copies of the relevant content units 120 in a data section 125 . To minimize size, only one copy of content units having the same content is included.
- the target storage space 104 stores content units in an organized hierarchy of paths 126 .
- path “ ⁇ A ⁇ B” is the full path or location, in a namespace tree, of a content unit at node B.
- the paths 126 are full or absolute paths in the filesystem. With the example path “ ⁇ A ⁇ B”, a file named “B” is stored at the path.
- the paths 126 may also be relative paths sharing a common sub-root.
- the dataset replicator 110 once instructed which dataset 100 to replicate, accesses the manifest 124 . As described below, the dataset replicator 110 then determines which content units need to be transferred, either from the source storage space 102 or from the data section 125 , as the case may be.
- the list of paths and respective content units are reproduced in the target storage space 104 .
- Each path and its respective content unit that was included in the dataset 100 is duplicated in the namespace of the target storage space 104 .
- FIG. 3 shows a process for building a dataset.
- a request to form a dataset is received.
- the request includes a list of paths of content units or a reference to the same.
- a hash is computed or acquired for each content unit in the request list. Any known hash function may be used for computing hashes.
- a list of corresponding paths in the source data is compiled.
- the manifest 124 is built with each content unit in the request list being represented as at least one path associated with a corresponding hash.
- FIG. 4 shows the target storage space 104 supplemented by a shallow cache 160 .
- the shallow cache 160 does not need to cache content units. Rather, the shallow cache 160 tracks the pats in the target storage space 104 that contain the same content units. This may be in the form of a table or file of hashes, where each hash has an associated set of paths. The pathset of a hash in the shallow cache indicates each location in the target storage space 104 where the same content is possibly located. In some embodiments, the shallow cache need not be rigorously maintained. It may not be necessary that the shallow cache 160 continuously reflect the actual content of the target storage space 104 . For example, if a content unit listed in the shallow cache 160 is deleted, the shallow cache 160 need not be updated.
- the shallow cache 160 also does not need to be updated.
- the shallow cache 160 may be kept update through the underlying implementation of the target storage space 104 (i.e., storage update operations include cache updates) or by learning about accuracy of the shallow cache through its usage.
- the shallow cache 160 indicates at least some of the content units that are likely available in the target storage space 104 .
- the shallow cache 160 may be used to improve the efficiency of replicating a dataset from the source storage space to the target storage space by allowing content units being replicated to be obtained from the target storage space rather than from the source side.
- the shallow cache may be updated when replicating a dataset to the target storage space; new hashes of inbound content units may be added if they don't already exist, and if they do exist the corresponding pathset may be expanded to include any new locations where such content units are added.
- FIG. 5 shows a process for initializing the shallow cache 160 .
- the shallow cache 160 may be initialized in a number of ways. In one embodiment (not shown), a completed scan of all content units in the target storage space may be performed. Hashes of each content unit may be computed, and pathsets of same each unique hash constructed. For large content, this may be prohibitive. Another approach is to build up the shallow cache 160 with the content of manifests as respective datasets are added to the target storage space. At step 170 , a dataset being replicated is copied in its entirety from the source side. Since initially the shallow cache is empty, when this first dataset is replicated, it is unknown whether there are any local copies of the content units that are to be replicated. Therefore, all of the content units of the dataset are copied from the source.
- the content units from the source are added to the paths in the local storage space as indicated by the dataset's manifest.
- the hash-to-pathset mappings in the manifest are stored in the shallow cache 160 .
- the manifest itself is just stored as the initial shallow cache. If the storage space is implemented as a filesystem file, the shallow cache may be stored in a special directory and is initialized by storing the manifest in the special directory.
- FIG. 6 shows a process for using a shallow cache to replicate a dataset to a target storage space. It is assumed that the shallow cache has already been initialized and contains hashes and respective path lists of content units in the target storage space.
- a request is received to copy a dataset.
- the manifest of the dataset is copied or downloaded.
- the dataset is replicated to the target storage space using the following algorithm.
- the hashes in the manifest are iterated over. For each hash value H in the manifest, presence of H in the shallow cache is checked. If H is not present, then the source for H's content set to be the source storage space. However, if H is found in the shallow cache, then each of the paths in the path list associated with H in the shallow cache is checked to see if it exists and if a valid copy of the corresponding content unit is present at the path. The first found valid copy is then set to the source that is to be used to obtain the corresponding content unit.
- the content unit is copied therefrom to each path associated with H in the manifest.
- the content unit is obtained from the target storage space per the shallow cache, and otherwise is obtained from the source side. And, regardless of where the content unit is obtained from, once it is obtained it is copied to the paths for that content unit (hash) as indicated by the manifest.
- the shallow cache may also be updated. Any paths in the shallow cache that are found to not exist may be deleted from the shallow cache. If a content unit at a path in the shallow cache is found to have a hash that does not match the hash in the shallow cache, then that path in the target storage space is added to the cache (at an existing or new hash/entry, as the case may be).
- the manifest of the dataset being replicated is used to update the shallow cache.
- the details of this step will depend on how the shallow cache is implemented.
- the manifest may be added to shallow cache directory.
- the content of the manifest is logically merged with the shallow cache. This may involve adding to existing cache entries (hash-pathlist pairs) new paths where content units have been added.
- manifests or units of cache data may be periodically merged.
- FIG. 7 shows an example of replication that initializes a shallow cache.
- the manifest 124 includes two manifest entries for two respective content units.
- Content unit 1 200 is represented by a manifest entry that includes hash value hash 1 associated with paths p 1 and p 2 (“ ⁇ A ⁇ B”, and “ ⁇ A ⁇ B ⁇ C ⁇ D”, respectively).
- Content unit 2 202 is represented by a manifest entry that includes hash value hash 2 associated with path p 3 (“ ⁇ A ⁇ B ⁇ K ⁇ L ⁇ M ⁇ N ⁇ O”).
- the manifest or the hash-path data therefrom is the initial shallow cache data.
- FIG. 8 shows an example of using a shallow cache for replicating a dataset.
- the manifest 124 includes entries representing content unit 3 and content unit 4 .
- content unit 3 is represented by hash 3 in association with paths p 4 and p 5
- content unit 4 is represented by hash 1 in association with path p 6 .
- the manifest entry for content unit 3 204 is processed first.
- Hash 3 is checked for presence in the shallow cache 160 . Because it is not found, content unit 3 is copied from the source side to paths p 4 and p 5 , and hash 3 along with paths p 4 and p 5 are added to the shallow cache (which now reflects the added content).
- Hash 1 is searched for in the shallow cache and is found.
- a path in the corresponding cache entry is selected (e.g., randomly), in this example, path p 2 is selected from hash 1 's cache entry.
- the content unit stored at path p 2 in the target storage space is copied to new path p 6 (“ ⁇ Q ⁇ R ⁇ S ⁇ T ⁇ A ⁇ B ⁇ C”) as specified in the manifest (if other paths were specified, they would also be created and receive copies). Thus, a copy from the source side is avoided.
- path p 6 is added to the cache entry of hash 1 .
- path p 2 did not exist or did not contain a content unit hashed to hash 1 , then p 2 would have been deleted from the hash 1 cache entry and another path in the hash 1 cache entry (e.g., p 1 ) would have been tried. If no valid paths existed, the cache entry could be deleted and content unit 4 copied from the source side.
- new cache entries are added without concern for possible hash duplication, and deduplication is performed when convenient.
- the storage spaces may be realized in many forms. They may be filesystems, data structures in a same virtual or physical memory space, cloud storage services, database elements, and so forth. Replication may be between devices via a network, for instance, or between storage spaces on a same device.
- FIG. 9 shows details of a computing device 300 on which embodiments described above may be implemented.
- the technical disclosures herein will suffice for programmers to write software, and/or configure reconfigurable processing hardware (e.g., field-programmable gate arrays (FPGAs)), and/or design application-specific integrated circuits (ASICs), etc., to run on the computing device 300 (possibly via cloud APIs) to implement the embodiments described herein.
- reconfigurable processing hardware e.g., field-programmable gate arrays (FPGAs)
- ASICs design application-specific integrated circuits
- the computing device 300 may have one or more displays 322 , a network interface 324 (or several), as well as storage hardware 326 and processing hardware 328 , which may be a combination of any one or more: central processing units, graphics processing units, analog-to-digital converters, bus chips, FPGAs, ASICs, Application-specific Standard Products (ASSPs), or Complex Programmable Logic Devices (CPLDs), etc.
- the storage hardware 326 may be any combination of magnetic storage, static memory, volatile memory, non-volatile memory, optically or magnetically readable matter, etc.
- the meaning of the term “storage”, as used herein does not refer to signals or energy per se, but rather refers to physical apparatuses and states of matter.
- the hardware elements of the computing device 300 may cooperate in ways well understood in the art of machine computing.
- input devices may be integrated with or in communication with the computing device 300 .
- the computing device 300 may have any form-factor or may be used in any type of encompassing device.
- the computing device 300 may be in the form of a handheld device such as a smartphone, a tablet computer, a gaming device, a server, a rack-mounted or backplaned computer-on-a-board, a system-on-a-chip, or others.
- Embodiments and features discussed above can be realized in the form of information stored in volatile or non-volatile computer or device readable media. This is deemed to include at least media such as optical storage (e.g., compact-disk read-only memory (CD-ROM)), magnetic media, flash read-only memory (ROM), or any current or future means of storing digital information.
- the stored information can be in the form of machine executable instructions (e.g., compiled executable binary code), source code, bytecode, or any other information that can be used to enable or configure computing devices to perform the various embodiments discussed above.
- RAM random-access memory
- CPU central processing unit
- non-volatile media storing information that allows a program or executable to be loaded and executed.
- the embodiments and features can be performed on any type of computing device, including portable devices, workstations, servers, mobile wireless devices, and so on.
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
Description
- Computing systems store and manage hierarchies of content units or data items. Each content unit in such a hierarchy or storage space has a path or location. The paths are all in a same namespace, and the content units can be accessed by referring to their paths. Often, portions of one storage space need to be copied to another storage space. That is, a set of content units at respective paths in a source storage space may be need to be copied to a target storage space sharing the same namespace. For example, a directory in a remote filesystem may need to be copied to a directory in a local filesystem.
- Depending on the application or type of storage, same instances of a content unit might be stored in multiple paths in a storage space. In the case of filesystem files, there may be multiple files containing the same content but at different full paths in the file system. For instance, the same file content “X” might be stored at “\A\B” and at “\A\C\D”. Files “B” and “D” store the same content, but at different locations and with different file names.
- When duplicating a set of paths and respective content units from a source storage space to a target storage space, the duplication of content units may have inefficiencies. The same content may be transferred from source to target multiple times. Referring to the example above, it would be redundant wasteful to copy both files “B” and “D” from source to target when they contain the same content. As only the inventors have observed, another inefficiency may occur when content units of a portion of source storage space to be transferred to target storage space already exist at the target storage space. For instance, if content units at “\A\B” and “\A\C” are to be transferred perhaps as a package, if the target storage space already contains the same content of say “\A\B”, yet at a different local location such as “\D\E\F”, the inventors have observed that transferring the content unit at “\A\B” is potentially avoidable since the same content is already available in the target storage space (at “\D\E\F”).
- Discussed below are techniques related to using a shallow cache to efficiently transfer packages or sets of content units or data items from source storage spaces to target storage spaces by leverage existing content at the target storage spaces.
- The following summary is included only to introduce some concepts discussed in the Detailed Description below. This summary is not comprehensive and is not intended to delineate the scope of the claimed subject matter, which is set forth by the claims presented herewith.
- Embodiments relate to efficiently replicating data from a source storage space to a target storage space. The storage spaces share a common namespace of paths where content units are stored. A shallow cache is maintained for the target storage space. Each entry in the cache includes a hash of a content unit in the target storage space and associated hierarchy paths in the target storage space where the corresponding content unit is stored. When a set of content units in the source storage space is to be replicated at the target storage space, any content unit with a hash in the cache is replicated from one of the associated paths in the cache, thus avoiding having to replicate content from the source storage space.
- Many of the attendant features will be explained below with reference to the following detailed description considered in connection with the accompanying drawings.
- The present description will be better understood from the following detailed description read in light of the accompanying drawings, wherein like reference numerals are used to designate like parts in the accompanying description.
-
FIG. 1 shows a system for replicating a dataset of data items from a source storage space to a target storage space. -
FIG. 2 shows additional detail of a dataset. -
FIG. 3 shows a process for building a dataset. -
FIG. 4 shows a target storage space supplemented by a shallow cache. -
FIG. 5 shows a process for initializing the shallow cache. -
FIG. 6 shows a process for using a shallow cache to replicate a dataset to a target storage space. -
FIG. 7 shows an example of replication that initializes a shallow cache. -
FIG. 8 shows an example of using a shallow cache for replicating a dataset. -
FIG. 9 shows details of a computing device on which embodiments described herein may be implemented. -
FIG. 1 shows a system for replicating adataset 100 of data items from asource storage space 100 to atarget storage space 102. The data items or content units may be any type of discrete unit of data such as files, memory chunks, media data, etc., so long as they are individually distinguishable and accessible. They may be of variable length or uniform size. Thestorage spaces 102/104 may be co-located on a same computing device or they may be on respective different computing devices connected by a network. Of note is that they share a same namespace. Moreover, storage spaces generally organize their content units in hierarchical fashion. Content units in a storage space may be arranged or located by names, paths, identifiers of tree nodes, etc. The term “paths” will be used hereafter to refer to these types of hierarchical location descriptors. - A common operation is to replicate a portion of a
source storage space 102 to atarget storage space 104. In concept, this involves duplicating adataset 100 that includes a set of paths in thesource storage space 102 and corresponding content units that reside at the paths in thesource storage space 102. In the case of filesystems, thedataset 100 may be an entire directory or a set of disperse files. - In a typical system, the
dataset 100 or other structuring of data for transfer may involve a software component such as adataset builder 106 at the source side that collects the necessary data and information to allow a receiver to reconstruct the portion of source data that is being prepared. Any known packaging tools may be used. As described in detail later, thedataset 100 may be a combination of indicia of the portion of source data, for instance a manifest of content unit paths. Thedataset 100 may optionally include copies of content units. In one embodiment a manifest is used, and the target side may use the manifest to decide which content units to pull on an as-needed basis. - The
dataset builder 106 may perform abuild process 108 that includes steps such as receiving a request to build the dataset and then building the dataset. The request may specify a set of paths, a path root or subtree, and so forth. In one embodiment, as discussed in more detail below, building thedataset 100 includes identifying content items that have the same content, providing an identifier for the content (e.g., a hash), and adding the identifier or hash and the corresponding source paths that contain the same corresponding content. In the case of a filesystem, thedataset 100 is a list of each unique hash in the dataset, and each hash is accompanied by a list of full pathnames of files with the same hash. - In a pull scenario where the target side pulls data from the source side, a
data replicator process 110 or the like performs areplication process 112. Thedata replicator process 110 may include steps such as selecting thedataset 100, providing a request for thedataset 100 to the source side, copying at least indicia of the content in the dataset, and then copying the relevant content units from thesource storage space 102 and distributing them to the correct namespace locations or paths in thetarget storage space 104. - The system shown in
FIG. 1 is only an example. In different embodiments may perform the steps in different orders or at different places. It is possible for all of the building and replication to take place on the source side or the target side. Building and replication may be performed in a combined fashion. -
FIG. 2 shows additional detail of a dataset. As noted, thesource storage space 102 storescontent units 120. In one embodiment, thecontent units 120 may be stored in a hierarchy (not shown) using the same (or overlapping) namespace as that of thetarget storage space 104. Thedataset builder 106 may start with a list of paths of content units to be built or packaged. Ahash function 122 is applied to eachcontent unit 120 at each path in thelist creating hashes 121 for the respective content units. Hashing might be performed in advance on all content units in the source storage space. In either case, thedataset builder 106 builds amanifest 124 by consolidating the hashes of the listed content units. Specifically, any content units that have the same hash contain the same content. Such hash is added to themanifest 124 along with the set of all paths in thesource storage space 102 of the respective content units having the same hash. The result is that themanifest 124 represents content units in groups that have the same content (and hash value). - In one embodiment, the
dataset 100 may include copies of therelevant content units 120 in adata section 125. To minimize size, only one copy of content units having the same content is included. - The
target storage space 104 stores content units in an organized hierarchy ofpaths 126. In the example shown inFIG. 2 , path “\A\B” is the full path or location, in a namespace tree, of a content unit at node B. In the case of filesystems, thepaths 126 are full or absolute paths in the filesystem. With the example path “\A\B”, a file named “B” is stored at the path. Thepaths 126 may also be relative paths sharing a common sub-root. Thedataset replicator 110, once instructed which dataset 100 to replicate, accesses themanifest 124. As described below, thedataset replicator 110 then determines which content units need to be transferred, either from thesource storage space 102 or from thedata section 125, as the case may be. From whatever source, including copies in thetarget storage space 104, the list of paths and respective content units are reproduced in thetarget storage space 104. Each path and its respective content unit that was included in thedataset 100 is duplicated in the namespace of thetarget storage space 104. -
FIG. 3 shows a process for building a dataset. At step 140 a request to form a dataset is received. The request includes a list of paths of content units or a reference to the same. Atstep 142, a hash is computed or acquired for each content unit in the request list. Any known hash function may be used for computing hashes. Atstep 144, for each unique hash in the dataset, a list of corresponding paths in the source data is compiled. Atstep 146, themanifest 124 is built with each content unit in the request list being represented as at least one path associated with a corresponding hash. -
FIG. 4 shows thetarget storage space 104 supplemented by ashallow cache 160. Theshallow cache 160 does not need to cache content units. Rather, theshallow cache 160 tracks the pats in thetarget storage space 104 that contain the same content units. This may be in the form of a table or file of hashes, where each hash has an associated set of paths. The pathset of a hash in the shallow cache indicates each location in thetarget storage space 104 where the same content is possibly located. In some embodiments, the shallow cache need not be rigorously maintained. It may not be necessary that theshallow cache 160 continuously reflect the actual content of thetarget storage space 104. For example, if a content unit listed in theshallow cache 160 is deleted, theshallow cache 160 need not be updated. If content units are added, theshallow cache 160 also does not need to be updated. Alternatively, theshallow cache 160 may be kept update through the underlying implementation of the target storage space 104 (i.e., storage update operations include cache updates) or by learning about accuracy of the shallow cache through its usage. - In effect, the
shallow cache 160 indicates at least some of the content units that are likely available in thetarget storage space 104. As discussed below, theshallow cache 160 may be used to improve the efficiency of replicating a dataset from the source storage space to the target storage space by allowing content units being replicated to be obtained from the target storage space rather than from the source side. As discussed below, the shallow cache may be updated when replicating a dataset to the target storage space; new hashes of inbound content units may be added if they don't already exist, and if they do exist the corresponding pathset may be expanded to include any new locations where such content units are added. -
FIG. 5 shows a process for initializing theshallow cache 160. Theshallow cache 160 may be initialized in a number of ways. In one embodiment (not shown), a completed scan of all content units in the target storage space may be performed. Hashes of each content unit may be computed, and pathsets of same each unique hash constructed. For large content, this may be prohibitive. Another approach is to build up theshallow cache 160 with the content of manifests as respective datasets are added to the target storage space. Atstep 170, a dataset being replicated is copied in its entirety from the source side. Since initially the shallow cache is empty, when this first dataset is replicated, it is unknown whether there are any local copies of the content units that are to be replicated. Therefore, all of the content units of the dataset are copied from the source. Atstep 172, the content units from the source are added to the paths in the local storage space as indicated by the dataset's manifest. Atstep 174, the hash-to-pathset mappings in the manifest are stored in theshallow cache 160. In one embodiment, the manifest itself is just stored as the initial shallow cache. If the storage space is implemented as a filesystem file, the shallow cache may be stored in a special directory and is initialized by storing the manifest in the special directory. -
FIG. 6 shows a process for using a shallow cache to replicate a dataset to a target storage space. It is assumed that the shallow cache has already been initialized and contains hashes and respective path lists of content units in the target storage space. At step 190 a request is received to copy a dataset. Atstep 192 the manifest of the dataset is copied or downloaded. - At
step 194, the dataset is replicated to the target storage space using the following algorithm. The hashes in the manifest are iterated over. For each hash value H in the manifest, presence of H in the shallow cache is checked. If H is not present, then the source for H's content set to be the source storage space. However, if H is found in the shallow cache, then each of the paths in the path list associated with H in the shallow cache is checked to see if it exists and if a valid copy of the corresponding content unit is present at the path. The first found valid copy is then set to the source that is to be used to obtain the corresponding content unit. Finally, whichever source is to be used for the content unit represented by the current H (i.e., the source storage space or a path in the target storage space), the content unit is copied therefrom to each path associated with H in the manifest. In sum, for each unique unit of content to be replicated, if possible, the content unit is obtained from the target storage space per the shallow cache, and otherwise is obtained from the source side. And, regardless of where the content unit is obtained from, once it is obtained it is copied to the paths for that content unit (hash) as indicated by the manifest. - During
step 194 the shallow cache may also be updated. Any paths in the shallow cache that are found to not exist may be deleted from the shallow cache. If a content unit at a path in the shallow cache is found to have a hash that does not match the hash in the shallow cache, then that path in the target storage space is added to the cache (at an existing or new hash/entry, as the case may be). - At
step 196, the manifest of the dataset being replicated is used to update the shallow cache. The details of this step will depend on how the shallow cache is implemented. In a filesystem implementation, the manifest may be added to shallow cache directory. In another embodiment, the content of the manifest is logically merged with the shallow cache. This may involve adding to existing cache entries (hash-pathlist pairs) new paths where content units have been added. In another embodiment, manifests or units of cache data may be periodically merged. -
FIG. 7 shows an example of replication that initializes a shallow cache. Themanifest 124 includes two manifest entries for two respective content units.Content unit 1 200 is represented by a manifest entry that includes hash value hash1 associated with paths p1 and p2 (“\A\B”, and “\A\B\C\D”, respectively).Content unit 2 202 is represented by a manifest entry that includes hash value hash2 associated with path p3 (“\A\B\K\L\M\N\O”). As per the paths in the manifest,content unit 1 200 is replicated from the source side to paths p1 and p2 in the target storage space, andcontent unit 2 202 is replicated from the source side to path p3 in the target storage space. The manifest or the hash-path data therefrom is the initial shallow cache data. -
FIG. 8 shows an example of using a shallow cache for replicating a dataset. This example builds on the example shown inFIG. 7 . In this example, themanifest 124 includes entries representingcontent unit 3 andcontent unit 4. As shown in the manifest,content unit 3 is represented by hash3 in association with paths p4 and p5, andcontent unit 4 is represented by hash1 in association with path p6. The manifest entry forcontent unit 3 204 is processed first. Hash3 is checked for presence in theshallow cache 160. Because it is not found,content unit 3 is copied from the source side to paths p4 and p5, and hash3 along with paths p4 and p5 are added to the shallow cache (which now reflects the added content). - The manifest entry for
content unit 4 is processed next. Hash1 is searched for in the shallow cache and is found. A path in the corresponding cache entry is selected (e.g., randomly), in this example, path p2 is selected from hash1's cache entry. The content unit stored at path p2 in the target storage space is copied to new path p6 (“\Q\R\S\T\A\B\C”) as specified in the manifest (if other paths were specified, they would also be created and receive copies). Thus, a copy from the source side is avoided. In addition, path p6 is added to the cache entry of hash1 . If path p2 did not exist or did not contain a content unit hashed to hash1 , then p2 would have been deleted from the hash1 cache entry and another path in the hash1 cache entry (e.g., p1) would have been tried. If no valid paths existed, the cache entry could be deleted andcontent unit 4 copied from the source side. - As noted above, in one embodiment, new cache entries are added without concern for possible hash duplication, and deduplication is performed when convenient.
- As also noted above, the storage spaces may be realized in many forms. They may be filesystems, data structures in a same virtual or physical memory space, cloud storage services, database elements, and so forth. Replication may be between devices via a network, for instance, or between storage spaces on a same device.
-
FIG. 9 shows details of acomputing device 300 on which embodiments described above may be implemented. The technical disclosures herein will suffice for programmers to write software, and/or configure reconfigurable processing hardware (e.g., field-programmable gate arrays (FPGAs)), and/or design application-specific integrated circuits (ASICs), etc., to run on the computing device 300 (possibly via cloud APIs) to implement the embodiments described herein. - The
computing device 300 may have one ormore displays 322, a network interface 324 (or several), as well asstorage hardware 326 andprocessing hardware 328, which may be a combination of any one or more: central processing units, graphics processing units, analog-to-digital converters, bus chips, FPGAs, ASICs, Application-specific Standard Products (ASSPs), or Complex Programmable Logic Devices (CPLDs), etc. Thestorage hardware 326 may be any combination of magnetic storage, static memory, volatile memory, non-volatile memory, optically or magnetically readable matter, etc. The meaning of the term “storage”, as used herein does not refer to signals or energy per se, but rather refers to physical apparatuses and states of matter. The hardware elements of thecomputing device 300 may cooperate in ways well understood in the art of machine computing. In addition, input devices may be integrated with or in communication with thecomputing device 300. Thecomputing device 300 may have any form-factor or may be used in any type of encompassing device. Thecomputing device 300 may be in the form of a handheld device such as a smartphone, a tablet computer, a gaming device, a server, a rack-mounted or backplaned computer-on-a-board, a system-on-a-chip, or others. - Embodiments and features discussed above can be realized in the form of information stored in volatile or non-volatile computer or device readable media. This is deemed to include at least media such as optical storage (e.g., compact-disk read-only memory (CD-ROM)), magnetic media, flash read-only memory (ROM), or any current or future means of storing digital information. The stored information can be in the form of machine executable instructions (e.g., compiled executable binary code), source code, bytecode, or any other information that can be used to enable or configure computing devices to perform the various embodiments discussed above. This is also deemed to include at least volatile memory such as random-access memory (RAM) and/or virtual memory storing information such as central processing unit (CPU) instructions during execution of a program carrying out an embodiment, as well as non-volatile media storing information that allows a program or executable to be loaded and executed. The embodiments and features can be performed on any type of computing device, including portable devices, workstations, servers, mobile wireless devices, and so on.
Claims (20)
Priority Applications (4)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US16/017,985 US10534708B1 (en) | 2018-06-25 | 2018-06-25 | Shallow cache for content replication |
EP19734557.2A EP3811228A1 (en) | 2018-06-25 | 2019-06-10 | Shallow cache for content replication |
CN201980040778.6A CN112384905A (en) | 2018-06-25 | 2019-06-10 | Shallow cache for content replication |
PCT/US2019/036217 WO2020005512A1 (en) | 2018-06-25 | 2019-06-10 | Shallow cache for content replication |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US16/017,985 US10534708B1 (en) | 2018-06-25 | 2018-06-25 | Shallow cache for content replication |
Publications (2)
Publication Number | Publication Date |
---|---|
US20190391917A1 true US20190391917A1 (en) | 2019-12-26 |
US10534708B1 US10534708B1 (en) | 2020-01-14 |
Family
ID=67108140
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US16/017,985 Active US10534708B1 (en) | 2018-06-25 | 2018-06-25 | Shallow cache for content replication |
Country Status (4)
Country | Link |
---|---|
US (1) | US10534708B1 (en) |
EP (1) | EP3811228A1 (en) |
CN (1) | CN112384905A (en) |
WO (1) | WO2020005512A1 (en) |
Cited By (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US11138479B2 (en) * | 2019-06-26 | 2021-10-05 | Huazhong University Of Science And Technology | Method for valuation of image dark data based on similarity hashing |
US20230169090A1 (en) * | 2021-11-30 | 2023-06-01 | Snowflake Inc. | Replication group objects configuration in a network-based database system |
Family Cites Families (17)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
EP1595197A2 (en) | 2003-02-21 | 2005-11-16 | Caringo, Inc. | Additional hash functions in content-based addressing |
JP2005078612A (en) | 2003-09-04 | 2005-03-24 | Hitachi Ltd | File sharing system, and file transfer method between file sharing systems |
US7444389B2 (en) | 2003-12-09 | 2008-10-28 | Emc Corporation | Methods and apparatus for generating a content address to indicate data units written to a storage system proximate in time |
US20060117018A1 (en) | 2004-11-30 | 2006-06-01 | Microsoft Corporation | Method and system for caching remote files locally |
US9424266B2 (en) | 2007-10-01 | 2016-08-23 | Microsoft Technology Licensing, Llc | Efficient file hash identifier computation |
US9323758B1 (en) | 2009-12-22 | 2016-04-26 | Emc Corporation | Efficient migration of replicated files from a file server having a file de-duplication facility |
KR101840238B1 (en) * | 2010-03-08 | 2018-03-20 | 휴렛 팩커드 엔터프라이즈 디벨롭먼트 엘피 | Data storage apparatus and methods |
US9471696B2 (en) * | 2010-11-12 | 2016-10-18 | Yahoo! Inc. | Methods and systems for pathing analysis |
US9910904B2 (en) * | 2011-08-30 | 2018-03-06 | International Business Machines Corporation | Replication of data objects from a source server to a target server |
WO2013112124A1 (en) * | 2012-01-23 | 2013-08-01 | Empire Technology Development Llc | Managing addressable memory in heterogeneous multicore processors |
US9729659B2 (en) | 2013-03-14 | 2017-08-08 | Microsoft Technology Licensing, Llc | Caching content addressable data chunks for storage virtualization |
US20150227543A1 (en) | 2014-02-11 | 2015-08-13 | Atlantis Computing, Inc. | Method and apparatus for replication of files and file systems using a deduplication key space |
US9986029B2 (en) | 2014-03-19 | 2018-05-29 | Red Hat, Inc. | File replication using file content location identifiers |
US9864694B2 (en) * | 2015-05-04 | 2018-01-09 | Arm Limited | Tracking the content of a cache using a way tracker having entries with a cache miss indicator |
US9612967B1 (en) * | 2015-09-25 | 2017-04-04 | Dell Products, L.P. | Cache load balancing by reclaimable block migration |
US20180095985A1 (en) | 2016-09-30 | 2018-04-05 | Cubistolabs, Inc. | Physical Location Scrambler for Hashed Data De-Duplicating Content-Addressable Redundant Data Storage Clusters |
CN107153707B (en) * | 2017-05-12 | 2020-08-14 | 华中科技大学 | Hash table construction method and system for nonvolatile memory |
-
2018
- 2018-06-25 US US16/017,985 patent/US10534708B1/en active Active
-
2019
- 2019-06-10 CN CN201980040778.6A patent/CN112384905A/en active Pending
- 2019-06-10 WO PCT/US2019/036217 patent/WO2020005512A1/en unknown
- 2019-06-10 EP EP19734557.2A patent/EP3811228A1/en active Pending
Cited By (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US11138479B2 (en) * | 2019-06-26 | 2021-10-05 | Huazhong University Of Science And Technology | Method for valuation of image dark data based on similarity hashing |
US20230169090A1 (en) * | 2021-11-30 | 2023-06-01 | Snowflake Inc. | Replication group objects configuration in a network-based database system |
WO2023102390A1 (en) * | 2021-11-30 | 2023-06-08 | Snowflake Inc. | Replication group objects configuration |
US11748374B2 (en) * | 2021-11-30 | 2023-09-05 | Snowflake Inc. | Replication group objects configuration in a network-based database system |
Also Published As
Publication number | Publication date |
---|---|
CN112384905A (en) | 2021-02-19 |
WO2020005512A1 (en) | 2020-01-02 |
US10534708B1 (en) | 2020-01-14 |
EP3811228A1 (en) | 2021-04-28 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US10860457B1 (en) | Globally ordered event stream logging | |
US8683112B2 (en) | Asynchronous distributed object uploading for replicated content addressable storage clusters | |
US20220138163A1 (en) | Incremental virtual machine metadata extraction | |
EP2324440B1 (en) | Providing data structures for determining whether keys of an index are present in a storage system | |
US11210269B2 (en) | System and method for deduplicating container image storage data | |
US7769719B2 (en) | File system dump/restore by node numbering | |
US9659023B2 (en) | Maintaining and using a cache of child-to-parent mappings in a content-addressable storage system | |
AU2019213367A1 (en) | Rule-based, modular conflict resolution for composite documents | |
US11782878B2 (en) | Systems and methods for searching deduplicated data | |
US10534708B1 (en) | Shallow cache for content replication | |
US10241963B2 (en) | Hash-based synchronization of geospatial vector features | |
CN111522791A (en) | Distributed file repeating data deleting system and method | |
US9633023B2 (en) | Efficient DVCS storage system | |
US10606805B2 (en) | Object-level image query and retrieval | |
US20220222146A1 (en) | Versioned backup on an object addressable storage system | |
US20190354538A1 (en) | Ordered list management in a replicated environment | |
WO2023201002A1 (en) | Implementing graph search with in-structure metadata of a graph-organized file system | |
US20200249876A1 (en) | System and method for data storage management | |
CN113282551B (en) | Data processing method, system and equipment | |
CN111562936B (en) | Object history version management method and device based on Openstack-Swift | |
US20230147552A1 (en) | Methods and systems for ordering operations on a file system having a hierarchical namespace | |
US20240028466A1 (en) | Storing Namespace Metadata in a Key Value Store to Facilitate Space Efficient Point In Time Snapshots | |
CN111562936A (en) | Object history version management method and device based on Openstack-Swift | |
CN118159954A (en) | Method and system for ordering operations on a file system having a hierarchical namespace | |
Zacchiroli | So ware Heritage |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
FEPP | Fee payment procedure |
Free format text: ENTITY STATUS SET TO UNDISCOUNTED (ORIGINAL EVENT CODE: BIG.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY |
|
AS | Assignment |
Owner name: MICROSOFT TECHNOLOGY LICENSING, LLC, WASHINGTON Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:BARKER, ROSS NEAL;SUFARU, ADRIAN;SIGNING DATES FROM 20180625 TO 20180718;REEL/FRAME:046484/0733 |
|
STCF | Information on status: patent grant |
Free format text: PATENTED CASE |
|
MAFP | Maintenance fee payment |
Free format text: PAYMENT OF MAINTENANCE FEE, 4TH YEAR, LARGE ENTITY (ORIGINAL EVENT CODE: M1551); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY Year of fee payment: 4 |