US20040148306A1 - Hash file system and method for use in a commonality factoring system - Google Patents
Hash file system and method for use in a commonality factoring system Download PDFInfo
- Publication number
- US20040148306A1 US20040148306A1 US10/757,753 US75775304A US2004148306A1 US 20040148306 A1 US20040148306 A1 US 20040148306A1 US 75775304 A US75775304 A US 75775304A US 2004148306 A1 US2004148306 A1 US 2004148306A1
- Authority
- US
- United States
- Prior art keywords
- computer
- probabilistically unique
- unique identifier
- list
- data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Abandoned
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/14—Error detection or correction of the data by redundancy in operation
- G06F11/1402—Saving, restoring, recovering or retrying
- G06F11/1446—Point-in-time backing up or restoration of persistent data
- G06F11/1448—Management of the data involved in backup or backup restore
- G06F11/1453—Management of the data involved in backup or backup restore using de-duplication of the data
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/13—File access structures, e.g. distributed indices
- G06F16/137—Hash-based
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y10—TECHNICAL SUBJECTS COVERED BY FORMER USPC
- Y10S—TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y10S707/00—Data processing: database and file management or data structures
- Y10S707/99931—Database or file accessing
- Y10S707/99933—Query processing, i.e. searching
- Y10S707/99936—Pattern matching access
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y10—TECHNICAL SUBJECTS COVERED BY FORMER USPC
- Y10S—TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y10S707/00—Data processing: database and file management or data structures
- Y10S707/99931—Database or file accessing
- Y10S707/99937—Sorting
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y10—TECHNICAL SUBJECTS COVERED BY FORMER USPC
- Y10S—TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y10S707/00—Data processing: database and file management or data structures
- Y10S707/99951—File or database maintenance
- Y10S707/99952—Coherency, e.g. same view to multiple users
- Y10S707/99953—Recoverability
Definitions
- the present invention relates, in general, to the field of hash file systems and commonality factoring systems. More particularly, the present invention relates to a system and method for determining a correspondence between electronic files in a distributed computer data environment and particular applications therefor.
- RAID Redundant Array of Independent Disks
- I/O input/output
- MTBF mean time between failure
- RAID systems are difficult to scale because of physical limitations in the cabling and controllers.
- the availability of RAID systems is highly dependent on the functionality of the controllers themselves so that when a controller fails, the data stored behind the controller becomes unavailable.
- RAID systems require specialized, rather than commodity hardware, and so tend to be expensive solutions.
- NAS network-attached storage
- NAS network-attached storage
- NAS may provide transparent I/O operations using either hardware or software based RAID.
- NAS may also automate mirroring of data to one or more other NAS devices to further improve fault tolerance. Because NAS devices can be added to a network, they enable scaling of the total capacity of the storage available to a network.
- NAS devices are constrained in RAID applications to the abilities of the conventional RAID controllers. Also, NAS systems do not enable mirroring and parity across nodes, and so are a limited solution.
- a system and method for a computer file system that is based and organized upon hashes and/or strings of digits of certain, different, or changing lengths and which is capable of eliminating or screening redundant copies of the blocks of data (or parts of data blocks) from the system.
- hashes may be produced by a checksum generating program, engine or algorithm such as industry standard Message Digest 4 (“MD4”), MD5, Secure Hash Algorithm (“SHA”) or SHA-1 algorithms.
- hashes may be generated by a checksum program, engine, algorithm or other means that generates a probabilistically unique hash value for a block of data of indeterminate size based upon a non-linear probablistic mathematical algorithm or any industry standard technique for generating pseudo-random values from an input text of other data/numeric sequence.
- the system and method of the present invention may be utilized, in a particular application disclosed herein, to automatically factor out redundancies in data allowing potentially very large quantities of unfactored storage to be often reduced in size by several orders of magnitude.
- the system and method of the present invention would allow all computers, regardless of their particular hardware or software characteristics, to share data simply, efficiently and securely and to provide a uniquely advantageous means for effectuating the reading, writing or referencing of data.
- the system and method of the present invention is especially efficacious with respect to networked computers or computer systems but may also be applied to isolated data storage with comparable results.
- the hash file system of the present invention advantageously solves a number of problems that plague conventional storage architectures.
- the system and method of the present invention eliminates the need for managing a huge collection of directories and files, together with all the wasted system resources that inevitably occur with duplicates, and slightly different copies.
- the maintenance and storage of duplicate files plagues traditional corporate and private computer systems and generally requires painstaking human involvement to “cleanup disk space”.
- the hash file system of the present invention effectively eliminates this problem by eliminating the disk space used for copies and nearly entirely eliminating the disk space used in partial copies. For example, in a traditional computer system copying a gigabyte directory structure to a new location would require another gigabyte of storage.
- the hash file system of the present invention reduces the disk space used in this operation by up to a hundred thousand times or more.
- the hash file system of the present invention is designed to factor storage on a scale never previously attempted and in a first implementation, is capable of factoring 2 million petabytes of storage, with the ability to expand to much larger sizes. Existing file systems are incapable of managing data on such scales.
- the hash file system of the present invention may be utilized to provide inexpensive, global computer system data protection and backup. Its factoring function operates very efficiently on typical backup data sets because computer file systems rarely change more than a few percent of their overall storage between each backup operation. Further, the hash file system of the present invention can serve as the basis for an efficient messaging (e-mail) system.
- E-mail systems are fundamentally data copying mechanisms wherein an author writes a message and sends it to a list of recipients. An e-mail system implements this “sending” operation effectively by copying the data from one place to another. The author generally keeps copies of the messages he sends and the recipients each keep their own copies. These copies are often, in turn, attached in replies that are also kept (i.e. copies of copies).
- the commonality factoring feature of the present invention can eliminate this gross inefficiency while transparently allowing e-mail users to retain this familiar copy-oriented paradigm.
- the hash file system of the present invention allows for the reconstruction of complete snapshots of entire systems which can be kept, for example, for every hour of every day they exist or even continuously, with snapshots taken at even minute (or less) intervals depending on the system needs.
- conventional computer systems often provide limited versioning of files (i.e. Digital Equipment Corporation's VAX® VMS® file system)
- the hash file system of the present invention also provides significant advantages in this regard. Versioning in conventional systems presents both good and bad aspects. In the former instance, it helps prevent accidents, but, in the latter, it requires regular purging to reduce the disk space it consumes.
- the hash file system of the present invention provides versioning of files with little overhead through the factoring of identical copies or edited copies with little extra space. For example, saving one hundred revisions of a typical document typically requires about one hundred times the space of the original file. Using the hash file system disclosed herein, those revisions might require only three times the space of the original (depending on the document's size, the degree and type of editing, and external factors).
- the hash file system of the present invention can be used to efficiently distribute web content because the method of factoring commonality (hashing) also produces uniform distribution over all hash file system servers. This even distribution permits a large array of servers to function as a gigantic web server farm with an evenly distributed load.
- the hash file system of the present invention can be used as a network accelerator inasmuch as it can be used to reduce network traffic by sending proxies (hashes) for data instead of the data itself. A large percentage of current network traffic is redundant data moving between locations. Sending proxies for the data would allow effective local caching mechanisms to operate, possibly reducing the traffic on the Internet by several orders of magnitude.
- the hash file system and method of the present invention may be implemented using 160 bit hashsums as universal pointers. This differs from conventional file systems which use pointers assigned from a central authority (i.e. in Unix a 32 bit “inode” is assigned by the kernel's file systems in a lock-step operation to assure uniqueness).
- these 160 bit hashsums are assigned without a central authority (i.e. without locking, without synchronization) by a hashing algorithm.
- Known hashing algorithms produce probabilistically unique numbers that uniformly span a range of values. In the case of the hash function SHA-1, that range is between 0 and 10e 48 . This hashing operation is done by examining only the contents of the data being stored and, therefore, can be done in complete isolation, asynchronously, and without interlocking.
- Hashing is an operation that can be verified by any component of the system, eliminating the need for trusted operations across those components.
- the hash file system and method of the present invention disclosed herein is, therefore, functional to eliminate the critical bottleneck of conventional large scale distributed file systems, that is, a trusted encompassing central authority. It permits the construction of a large scale distributed file system with no limits on simultaneous read/write operations, that can operate without risk of incoherence and without the limitation of certain conventional bottlenecks.
- FIG. 1 is a high level illustration of a representative networked computer environment in which the system and method of the present invention may be implemented;
- FIG. 2 is a more detailed conceptual representation of a possible operating environment for utilization of the system and method of the present invention wherein files maintained on any number of computers or data centers may be stored in a decentralized computer system through an Internet connection to a number of Redundant Arrays of Independent Nodes (“RAIN”) racks located, for example, at geographically diverse locations;
- RAIN Redundant Arrays of Independent Nodes
- FIG. 3 is a logic flow chart depicting the steps in the entry of a computer file into the hash file system of the present invention wherein the hash value for the file is checked against hash values for files previously maintained in a set, or database;
- FIG. 4 is a further logic flow chart depicting the steps in the breakup of a file or other data sequence into hashed pieces resulting in the production of a number of data pieces as well as corresponding probabilistically unique hash values for each piece;
- FIG. 5 is another logic flow chart depicting the comparison of the hash values for each piece of a file to existing hash values in the set (or database), the production of records showing the equivalence of a single hash value for all file pieces with the hash values of the various pieces and whereupon new data pieces and corresponding new hash values are added to the set;
- FIG. 6 is yet another logic flow chart illustrating the steps in the comparison of file hash or directory list hash values to existing directory list hash values and the addition of new file or directory list hash values to the set directory list;
- FIG. 7 is a comparison of the pieces of a representative computer file with their corresponding hash values both before and after editing of a particular piece of the exemplary file;
- FIG. 8 is a conceptual representation of the fact that composite data which may be derived by means of the system and method of the present invention is effectively the same as the data represented explicitly but may instead be created by a “recipe” such as the concatenation of data represented by its corresponding hashes or the result of a function using the data represented by the hashes;
- FIG. 9 is another conceptual representation of how the hash file system and method of the present invention my be utilized to organize data to optimize the reutilization of redundant sequences through the use of hash values as pointers to the data they represent and wherein data may be represented either as explicit byte sequences (atomic data) or as groups of sequences (composites);
- FIG. 10 is a simplified diagram illustrative of a hash file system address translation function for an exemplary 160 bit hash value
- FIG. 11 is a simplified exemplary illustration of an index stripe splitting function for use with the system and method of the present invention.
- FIG. 12 is a simplified illustration of the overall functionality of the system and method of the present invention for use in the backup of data for a representative home computer having a number of program and document files on Day 1 and wherein one of the document files is edited on Day 2 together with the addition of a third document file;
- FIG. 13 illustrates the comparison of various pieces of a particular document file marked by a number of “sticky bytes” both before and following editing wherein one of the pieces is thereby changed while other pieces remain the same.
- hash file system that manages the correspondence of one or more block(s) of data (including but not limited to files, directories, drive images, software applications, digitized voice, and rich media content) together with one or more symbol(s) for that block of data, wherein the symbol may be a number, hash, checksum, binary sequence, or other identifier that is derived from the block of data itself and is statistically, probabilistically, or otherwise effectively unique to that block of data.
- block(s) of data including but not limited to files, directories, drive images, software applications, digitized voice, and rich media content
- symbol may be a number, hash, checksum, binary sequence, or other identifier that is derived from the block of data itself and is statistically, probabilistically, or otherwise effectively unique to that block of data.
- the system itself works on any computer system including, without limitation: personal computers; supercomputers; distributed or non-distributed networks; storage area networks (“SAN”) using IDE, SCSI or other disk buses; network attached storage (“NAS”) or other systems capable of storing and/or processing data.
- SAN storage area networks
- IDE IDE
- SCSI SCSI
- NAS network attached storage
- the symbol(s) may be derived using one or more hash or checksum generating engines, programs, or algorithms, including but not limited to MD4, MD5, SHA, SHA-1, or their derivatives. Further, the symbol(s) may comprise parts of variable or invariable length symbols derived using a hash or checksum generating engine, program, or algorithm, including but not limited to MD4, MD5, SHA, SHA-1, or other methods of generating probabilistically unique identifiers based on data content.
- FIG. 1 Further disclosed herein is a system and method for a hash file system wherein the symbols allow for the identification of redundant copies within the system and/or allow for the identification of copies within the system redundant with data presented to the system for filing and storage.
- the symbols allow for the elimination of, or allow for the screening of, redundant copies of the data and/or parts of the data in the system or in data and/or parts of data presented to the system, without loss of data integrity and can provide for the even distribution of data over available storage for the system.
- the system and method of the present invention as disclosed herein requires no central operating point and balances processing and/or input/output (“I/O”) load across all computers, supercomputers, or other devices capable of storing and/or processing data attached to the system.
- the screening of redundant copies of the data and/or parts of the data allows for the creation, repetitive creation, or retention of intelligent boundaries for screening other data in the system, future data presented to the system, or future data stored by the system.
- an exemplary internetwork environment 10 may include the Internet which comprises a global internetwork formed by logical and physical connection between multiple wide area networks (“WANs”) 14 and local area networks (“LANs”) 16 .
- An Internet backbone 12 represents the main lines and routers that carry the bulk of the data traffic.
- the backbone 12 is formed by the largest networks in the system that are operated by major Internet service providers (“ISPs”) such as GTE, MCI, Sprint, UUNet, and America Online, for example.
- ISPs major Internet service providers
- a “network” comprises a system of general purpose, usually switched physical connections that enable logical connections between processes operating on nodes 18 .
- the physical connections implemented by a network are typically independent of the logical connections that are established between processes using the network. In this manner, a heterogeneous set of processes ranging from file transfer, mail transfer, and the like can use the same physical network. Conversely, the network can be formed from a heterogeneous set of physical network technologies that are invisible to the logically connected processes using the network. Because the logical connection between processes implemented by a network is independent of the physical connection, internetworks are readily scaled to a virtually unlimited number of nodes over long distances.
- one or more of the nodes 18 may implement storage allocation management (“SAM”) processes that manage data storage across nodes 18 in a distributed, collaborative fashion.
- SAM processes preferably operate with little or no centralized control for the system as whole.
- SAM processes provide data distribution across nodes 18 and implement recovery in a fault-tolerant fashion across network nodes 18 in a manner similar to paradigms found in RAID storage subsystems.
- Disparate computing devices 102 including mainframe computers (e.g., VAX station 106 and IBM AS/400 station 116 ) as well as personal computer or workstation class devices such as IBM compatible device 108 , Macintosh device 110 and laptop computer 112 are readily interconnected via internetwork 10 and network 100 . Although not illustrated, mobile and other wireless devices may be coupled to the internetwork 10 .
- mainframe computers e.g., VAX station 106 and IBM AS/400 station 116
- personal computer or workstation class devices such as IBM compatible device 108 , Macintosh device 110 and laptop computer 112 are readily interconnected via internetwork 10 and network 100 .
- mobile and other wireless devices may be coupled to the internetwork 10 .
- internal networks 122 are conceptually akin to LANs 16 (FIG. 1) in that they enable logical connections across a more limited distance than WAN 14 .
- Internal networks 122 may be implemented using various LAN technologies including Ethernet, Fiber Distributed Data Interface (“FDDI”), Token Ring, Appletalk, Fibre Channel, and the like.
- FDDI Fiber Distributed Data Interface
- Token Ring Appletalk
- Fibre Channel Fibre Channel
- the RAIN elements 124 may comprise computers using commodity components such as Intel-based microprocessors mounted on a motherboard supporting a PCI bus and 256 megabytes of random access memory (“RAM”) housed in a conventional AT or ATX case.
- SCSI or IDE controllers may be implemented on the motherboard and/or by expansion cards connected to the PCI bus. Where the controllers are implemented only on the motherboard, a PCI expansion bus may be optionally used.
- the motherboard may implement two mastering EIDE channels and a PCI expansion card which is used to implement two additional mastering EIDE channels so that each RAIN element 124 includes up to four or more EIDE hard disks.
- each hard disk may comprise an 80 gigabyte hard disk for a total storage capacity of 320 gigabytes or more per RAIN element.
- the hard disk capacity and configuration within RAIN elements 124 can be readily increased or decreased to meet the needs of a particular application.
- the casing also houses supporting mechanisms such as power supplies and cooling devices (not shown).
- Each RAIN element 124 executes an operating system.
- the UNIX or UNIX variant operating system such as Linux may be used. It is contemplated, however, that other operating systems including DOS, Microsoft Windows, Apple Macintosh OS, OS/2, Microsoft Windows NT and the like may be equivalently substituted with predictable changes in performance.
- the operating system chosen forms a platform for executing application software and processes, and implements a file system for accessing mass storage via the hard disk controller(s).
- Various application software and processes can be implemented on each RAIN element 124 to provide network connectivity via a network interface using appropriate network protocols such as user datagram protocol (“UDP”), transmission control protocol (TCP), Internet protocol (IP) and the like.
- UDP user datagram protocol
- TCP transmission control protocol
- IP Internet protocol
- the process 200 begins by entry of a computer file data 202 (e.g. “File A”) into the hash file system (“HFS”) of the present invention upon which a hash function is performed at step 204 .
- the data 206 representing the hash of File A is then compared to the contents of a set containing hash file values at decision step 208 . If the data 206 is already in the set, then the file's hash value is added to a directory list at step 210 .
- the contents of the set 212 comprising hash values and corresponding data is provided in the form of existing hash values 214 for the comparison operation of decision step 208 .
- the hash value for File A is not currently in the set, the file is broken into hashed pieces (as will be more fully described hereinafter) at step 216 .
- the file data 302 is divided into pieces based on commonality with other pieces in the system or the likelihood of pieces being found to be in common in the future at step 304 .
- the results of the operation of step 304 upon the file data 302 is, in the representative example shown, the production of four file pieces 306 denominated A 1 through A 5 inclusively.
- Each of the file pieces 306 is then operated on at step 308 by placing it through individual hash function operations to assign a probabilistically unique number to each of the pieces 306 A 1 through A 5 .
- the results of the operation at step 308 is that each of the pieces 306 (A 1 through A 5 ) has an associated, probabilistically unique hash value 310 (shown as A 1 Hash through A 5 Hash respectively).
- the file division process of step 304 is described in greater detail hereinafter in conjunction with the unique “sticky byte” operation also disclosed herein.
- FIG. 5 another logic flow chart is shown depicting a comparison process 400 for the hash values 310 of each piece 306 of the file to those of existing hash values 214 maintained in the set 212 .
- the hash values 310 for each piece 306 of the file are compared to existing hash values 214 and new hash values 408 and corresponding new data pieces 406 are added to the set 212 .
- hash values 408 not previously present in the database set 212 are added together with their associated data pieces 406 .
- the process 400 also results in the production of records 404 showing the equivalence of a single hash value for all file pieces with the hash values 310 of the various pieces 306 .
- FIG. 6 yet another logic flow chart is shown illustrating a process 500 for the comparison of file hash or directory list hash values to existing directory list hash values and the addition of new file or directory list hash values to the database directory list.
- the process 500 operates on stored data 502 which comprises an accumulated list of file names, file meta-data (e.g. date, time, file length, file type etc.) and the file's hash value for each item in a directory.
- the hash function is run upon the contents of the directory list.
- Decision step 506 is operative to determine whether or not the hash value for the directory list is in the set 212 of existing hash values 214 .
- the process 500 returns to add another file hash or directory list hash to a directory list.
- the hash value for the directory list is not already in the database set 212 , the hash value and data for the directory list are added to the database 212 set at step 508 .
- a comparison 600 of the pieces 306 of a representative computer file (i.e. “File A”) with their corresponding hash values 310 is shown both before and after editing of a particular piece of the exemplary file.
- the record 404 contains the hash value of File A as well as the hash values 310 of each of the pieces of the file A 1 through A 5 .
- a representative edit of the File A may produce a change in the data for piece A 2 (now represented by A 2 - b ) of the file pieces 306 A along with a corresponding change in the hash value A 2 - b of the hash values 310 A.
- the edited file piece now produces an updated record 404 A which includes the modified hash value of File A and the modified hash value of piece A 2 - b.
- a conceptual representation 700 is shown illustrative of the fact that composite data (such as composite data 702 and 704 ) derived by means of the system and method of the present invention, is effectively the same as the data 706 represented explicitly but is instead created by a “recipe”, or formula.
- this recipe includes the concatenation of data represented by its corresponding hashes 708 or the result of a function using the data represented by the hashes.
- the data blocks 706 may be variable length quantities as shown and the hash values 708 are derived from their associated data blocks.
- the hash values 708 are a probabilistically unique identification of the corresponding data pieces but truly unique identifications can be used instead or intermixed therewith.
- the composite data 702 , 704 can also reference other composite data many levels deep while the hash values 708 for the composite data can be derived from the value of the data the recipe creates or the hash value of the recipe itself.
- FIG. 9 another conceptual representation 800 is shown of how the hash file system and method of the present invention may be utilized to organize data 802 to optimize the reutilization of redundant sequences through the use of hash values 806 as pointers to the data they represent and wherein data 802 may be represented either as explicit byte sequences (atomic data) 808 or as groups of sequences (composites) 804 .
- the representation 800 illustrates the tremendous commonality of recipes and data that gets reused at every level.
- the basic structure of the hash file system of the present invention is essentially that of a “tree” or “bush” wherein the hash values 806 are used instead of conventional pointers.
- the hash values 806 are used in the recipes to point to the data or another hash value that could also itself be a recipe.
- recipes can point to other recipes that point to still other recipes that ultimately point to some specific data that may, itself, point to other recipes that point to even more data, eventually getting down to nothing but data.
- FIG. 10 a simplified diagram 900 is shown illustrative of a hash file system address translation function for an exemplary 160 bit hash value 902 .
- the hash value 902 includes a data structure comprising a front portion 904 and a back portion 906 as shown and the diagram 900 illustrates a particular “0 of 1” operation that is used for enabling the use of the hash value 902 to go to the location of the particular node in the system that contains the corresponding data.
- the diagram 900 illustrates how the front portion 904 of the hash value 902 data structure may be used to indicate the hash prefix to stripe identification (“ID”) 908 and how that is, in turn, utilized to map the stripe ID to IP address and the ID class to IP address 910 .
- ID hash prefix to stripe identification
- the “S 2 ” indicates stripe 2 of index Node 37 912 .
- the index stripe 912 of Node 37 then indicates stripe 88 of data Node 73 indicated by the reference numeral 914 .
- a portion of the hash value 902 itself may be used to indicate which node in the system contains the relevant data
- another portion of the hash value 902 may be used to indicate which stripe of data at that particular node and yet another portion of the hash value 902 to indicate where within that stripe the data resides.
- FIG. 12 a simplified illustration 1100 of the overall functionality of the system and method of the present invention is shown for use, for example, in the backup of data for a representative home computer having a number of program and document files 1102 A and 1104 A on Day 1 and wherein the program files 1102 B remain the same on Day 2 while one of the document files 1104 B is edited on Day 2 (Y.doc) together with the addition of a third document file (Z.doc).
- the document files are represented by hash value H 6 1110 with the first document being represented by hash value H 3 1118 and the second by hash value H 4 1116 .
- the “program files H 5 ” and “my documents” H 10 indicated by numeral 1120 show that the “program files H 5 ” have not changed, but the “my document H 10 ” have.
- H 10 indicated by numeral 1122 shows the “X.doc” is still represented by hash value H 3 1118 while “Y.doc” is now represented by hash value H 8 at number 1124 .
- New document file “Z.doc” is now represented by hash value H 9 at numeral 1126 .
- Data sticky bytes are a unique, fully automated way to sub-divide computer files such that common elements may be found on multiple related and unrelated computers without the need for communication between the computers.
- the means in which data sticky points are found is completely mathematical in nature and performs equally well regardless of the data content of the files.
- all data objects may be indexed, stored and retrieved using, for example (but not limited to), an industry standard checksum such as: MD4, MD5, SHA, or SHA-1. In operation, if two files have the same checksum, it may be considered to be highly likely that they are the same file.
- data sticky points may be produced with a standard mathematical distribution and with standard deviations that are a small percentage of the target size.
- a data sticky point is a statistically infrequent arrangement of n bytes. In this case, an example is given with 32 bytes because of its ease in implementation in current microprocessor technology.
- a rolling hash of 32 bits could be generated for the file “f”.
- //f[i] is the ith byte of the file “f”.
- //scramble is a 256 entry array of integers with each //being 32 bits wide; //these integers are typically chosen to uniformly //span the range.
- sticky_bits (hash ⁇ 1) ⁇ circumflex over ( ) ⁇ hash; //sticks_bits is a variable which will have the //number of ones in the hash //that correspond to the number of trailing zeros in //the “hash”.
- number_of_bits count_ones(stick-bits); if(number_of_bits > t) output_sticky_point(i); ⁇
- ⁇ hash hash >> 1
- sticky_bits (hash ⁇ 1) ⁇ circumflex over ( ) ⁇ hash; //sticks_bits is a variable which will have the //number of ones that correspond to the number of //trailing zeros in the “hash”.
- hashing function utilized to implement the hash file system of the present invention requires a moderately complex computation, it is well within the capability of present day computer systems. Hashing functions are inherently probabilistic and any hashing function might possibly produce incorrect results when two different data objects happen to have the same hash value. However, the system and method herein disclosed mitigates this problem by using well known and researched hashing functions that reduce the probability of collision down to levels acceptable for reliable use (i.e. one chance in a trillion trillion), far less than the error rates otherwise tolerated in conventional computer hardware operations.
- Internet infrastructure encompasses a variety of hardware and software mechanisms, the term primarily refers to routers, router software, and physical links between these routers that function to transport data packets from one network node to another.
- a “digital sequence” may comprise, without limitation, computer program files, computer applications, data files, network packets, streaming data such as multimedia (including audio and video), telemetry data and any other form of data which can be represented by a digital or numeric sequence.
- the probabilistically unique identifiers produced by means of the hash file system and method of the present invention may also be used as URLs in network applications.
Abstract
A system and method for a computer file system that is based and organized upon hashes and/or strings of digits of certain, different, or changing lengths and which is capable of eliminating or screening redundant copies of aggregate blocks of data (or parts of data blocks) from the system. The hash file system of the present invention utilizes hash values for computer files or file pieces which may be produced by a checksum generating program, engine or algorithm such as industry standard MD4, MD5, SHA or SHA-1 algorithms. Alternatively, the hash values may be generated by a checksum program, engine, algorithm or other means that produces an effectively unique hash value for a block of data of indeterminate size based upon a non-linear probablistic mathematical algorithm.
Description
- The present invention claims priority from U.S. Provisional Patent Application Serial No. 60/183,762 for: “System and Method for Decentralized Data Storage” filed Feb. 18, 2000, the disclosure of which is herein specifically incorporated by this reference.
- A portion of the disclosure of this patent document may contain material which is subject to copyright protection. The copyright owner has no objection to the facsimile reproduction by anyone of the patent document of the patent disclosure as it appears in the United States Patent and Trademark Office patent file or records, but otherwise, reserves all copyright rights whatsoever. The following notice applies to the software and data and described below, inclusive of the drawing figures where applicable: Copyright© 2000, Undoo Technologies.
- The present invention relates, in general, to the field of hash file systems and commonality factoring systems. More particularly, the present invention relates to a system and method for determining a correspondence between electronic files in a distributed computer data environment and particular applications therefor.
- Economic, political, and social power are increasingly managed by data. Transactions and wealth are represented by data. Political power is analyzed and modified based on data. Human interactions and relationships are defined by data exchanges. Hence, the efficient distribution, storage, and management of data is expected to play an increasingly vital role in human society.
- The quantity of data that must be managed, in the form of computer programs, databases, files, and the like, increases exponentially. As computer processing power increases, operating system and application software becomes larger. Moreover, the desire to access larger data sets such as multimedia files and large databases further increases the quantity of data that is managed. This increasingly large data load must be transported between computing devices and stored in an accessible fashion. The exponential growth rate of data is expected to outpace the improvements in communication bandwidth and storage capacity, making data management using conventional methods even more urgent.
- Many factors must be balanced and often compromised in conventional data storage systems. Because the quantity of data is extremely large, there is continuing pressure to reduce the cost per bit of storage. Also, data management systems should be scaleable to contemplate not only current needs, but future needs as well. Preferably, storage systems are incrementally scaleable so that a user can purchase only the capacity needed at any particular time. High reliability and high availability are also considered as data users are increasingly intolerant of lost, damaged, and unavailable data. Unfortunately, conventional data management architectures must compromise these factors so that no one architecture provides a cost-effective, reliable, high availability, scaleable solution.
- Conventional RAID (Redundant Array of Independent Disks) systems are a way of storing the same data in different places (thus, redundantly) on multiple storage devices such as hard disks. By placing data on multiple disks, input/output (“I/O”) operations can overlap in a balanced way, improving performance. Since the use of multiple disks increases the mean time between failure (“MTBF”), storing data redundantly also increases fault-tolerance. A RAID system relies on a hardware or software controller to hide the complexities of the actual data management so that a RAID system appears to an operating system as a single logical hard disk. However, RAID systems are difficult to scale because of physical limitations in the cabling and controllers. Also, the availability of RAID systems is highly dependent on the functionality of the controllers themselves so that when a controller fails, the data stored behind the controller becomes unavailable. Moreover, RAID systems require specialized, rather than commodity hardware, and so tend to be expensive solutions.
- NAS (network-attached storage) refers to hard disk storage that is set up with its own network address rather than being attached to an application server. File requests are mapped to the NAS file server. NAS may provide transparent I/O operations using either hardware or software based RAID. NAS may also automate mirroring of data to one or more other NAS devices to further improve fault tolerance. Because NAS devices can be added to a network, they enable scaling of the total capacity of the storage available to a network. However, NAS devices are constrained in RAID applications to the abilities of the conventional RAID controllers. Also, NAS systems do not enable mirroring and parity across nodes, and so are a limited solution.
- In addition to data storage issues, data transport is rapidly evolving with improvements in wide area network (“WAN”) and internetworking technology. The Internet, for example, has created a globally networked environment with almost ubiquitous access. Despite rapid network infrastructure improvements, the rate of increase in the quantity of data that requires transport is expected to outpace improvements in available bandwidth.
- Philosophically, the way data is conventionally managed is inconsistent with the hardware devices and infrastructures that have been developed to manipulate and transport data. For example, computers are characteristically general-purpose machines that are readily programmed to perform a virtually unlimited variety of functions. In large part, however, computers are loaded with a fixed, slowly changing set of data that limit their general-purpose nature to make the machines special-purpose. Advances in processing speed, peripheral performance and data storage capacity are most dramatic in commodity computers. Yet many data storage solutions cannot take advantage of these advances because they are constrained rather than extended by the storage controllers upon which they are based. Similarly, the Internet was developed as a fault tolerant, multi-path interconnected network. However, network resources are conventionally implemented in specific network nodes such that failure of the node makes the resource unavailable despite the fault-tolerance of the network to which the node is connected. Continuing needs exist for high availability, high reliability, highly scaleable data storage solutions.
- Disclosed herein is a system and method for a computer file system that is based and organized upon hashes and/or strings of digits of certain, different, or changing lengths and which is capable of eliminating or screening redundant copies of the blocks of data (or parts of data blocks) from the system. Also disclosed herein is a system and method for a computer file system wherein hashes may be produced by a checksum generating program, engine or algorithm such as industry standard Message Digest 4 (“MD4”), MD5, Secure Hash Algorithm (“SHA”) or SHA-1 algorithms. Further disclosed herein is a system and method for a computer file system wherein hashes may be generated by a checksum program, engine, algorithm or other means that generates a probabilistically unique hash value for a block of data of indeterminate size based upon a non-linear probablistic mathematical algorithm or any industry standard technique for generating pseudo-random values from an input text of other data/numeric sequence.
- The system and method of the present invention may be utilized, in a particular application disclosed herein, to automatically factor out redundancies in data allowing potentially very large quantities of unfactored storage to be often reduced in size by several orders of magnitude. In this regard, the system and method of the present invention would allow all computers, regardless of their particular hardware or software characteristics, to share data simply, efficiently and securely and to provide a uniquely advantageous means for effectuating the reading, writing or referencing of data. The system and method of the present invention is especially efficacious with respect to networked computers or computer systems but may also be applied to isolated data storage with comparable results.
- The hash file system of the present invention advantageously solves a number of problems that plague conventional storage architectures. For example, the system and method of the present invention eliminates the need for managing a huge collection of directories and files, together with all the wasted system resources that inevitably occur with duplicates, and slightly different copies. The maintenance and storage of duplicate files plagues traditional corporate and private computer systems and generally requires painstaking human involvement to “cleanup disk space”. The hash file system of the present invention effectively eliminates this problem by eliminating the disk space used for copies and nearly entirely eliminating the disk space used in partial copies. For example, in a traditional computer system copying a gigabyte directory structure to a new location would require another gigabyte of storage. In particular applications, the hash file system of the present invention reduces the disk space used in this operation by up to a hundred thousand times or more.
- Currently, some file systems have mechanisms to eliminate copies, but none can accomplish this operation in a short amount of time which, in technical terms, means the system factors copies in O(l) (“on the order of constant time”) time, even as the system scales. This means a unit of time that is constant as opposed to other systems that would require O(N**2), O(N) or O(log(N)) time, meaning time is related to the amount of storage being factored. Factoring storage in non-constant time may be marginally satisfactory for systems where the amount of storage is small, but as a system grows to large sizes, even the most efficient non-constant factoring systems become untenable. The hash file system of the present invention is designed to factor storage on a scale never previously attempted and in a first implementation, is capable of factoring 2 million petabytes of storage, with the ability to expand to much larger sizes. Existing file systems are incapable of managing data on such scales.
- Moreover, the hash file system of the present invention may be utilized to provide inexpensive, global computer system data protection and backup. Its factoring function operates very efficiently on typical backup data sets because computer file systems rarely change more than a few percent of their overall storage between each backup operation. Further, the hash file system of the present invention can serve as the basis for an efficient messaging (e-mail) system. E-mail systems are fundamentally data copying mechanisms wherein an author writes a message and sends it to a list of recipients. An e-mail system implements this “sending” operation effectively by copying the data from one place to another. The author generally keeps copies of the messages he sends and the recipients each keep their own copies. These copies are often, in turn, attached in replies that are also kept (i.e. copies of copies). The commonality factoring feature of the present invention can eliminate this gross inefficiency while transparently allowing e-mail users to retain this familiar copy-oriented paradigm.
- Because, as previously noted, most data in computer systems rarely change, the hash file system of the present invention allows for the reconstruction of complete snapshots of entire systems which can be kept, for example, for every hour of every day they exist or even continuously, with snapshots taken at even minute (or less) intervals depending on the system needs. Further, since conventional computer systems often provide limited versioning of files (i.e. Digital Equipment Corporation's VAX® VMS® file system), the hash file system of the present invention also provides significant advantages in this regard. Versioning in conventional systems presents both good and bad aspects. In the former instance, it helps prevent accidents, but, in the latter, it requires regular purging to reduce the disk space it consumes. The hash file system of the present invention provides versioning of files with little overhead through the factoring of identical copies or edited copies with little extra space. For example, saving one hundred revisions of a typical document typically requires about one hundred times the space of the original file. Using the hash file system disclosed herein, those revisions might require only three times the space of the original (depending on the document's size, the degree and type of editing, and external factors).
- Still other potential applications of the hash file system of the present invention include web-serving. In this regard, the hash file system can be used to efficiently distribute web content because the method of factoring commonality (hashing) also produces uniform distribution over all hash file system servers. This even distribution permits a large array of servers to function as a gigantic web server farm with an evenly distributed load. In other applications, the hash file system of the present invention can be used as a network accelerator inasmuch as it can be used to reduce network traffic by sending proxies (hashes) for data instead of the data itself. A large percentage of current network traffic is redundant data moving between locations. Sending proxies for the data would allow effective local caching mechanisms to operate, possibly reducing the traffic on the Internet by several orders of magnitude.
- As particularly disclosed herein, the hash file system and method of the present invention may be implemented using 160 bit hashsums as universal pointers. This differs from conventional file systems which use pointers assigned from a central authority (i.e. in Unix a 32 bit “inode” is assigned by the kernel's file systems in a lock-step operation to assure uniqueness). In the hash file system of the present invention, these 160 bit hashsums are assigned without a central authority (i.e. without locking, without synchronization) by a hashing algorithm.
- Known hashing algorithms produce probabilistically unique numbers that uniformly span a range of values. In the case of the hash function SHA-1, that range is between 0 and 10e48. This hashing operation is done by examining only the contents of the data being stored and, therefore, can be done in complete isolation, asynchronously, and without interlocking.
- Hashing is an operation that can be verified by any component of the system, eliminating the need for trusted operations across those components. The hash file system and method of the present invention disclosed herein is, therefore, functional to eliminate the critical bottleneck of conventional large scale distributed file systems, that is, a trusted encompassing central authority. It permits the construction of a large scale distributed file system with no limits on simultaneous read/write operations, that can operate without risk of incoherence and without the limitation of certain conventional bottlenecks.
- The aforementioned and other features and objects of the present invention and the manner of attaining them will become more apparent, and the invention itself will be best understood by reference to the following description of a preferred embodiment taken in conjunction with the accompanying drawings, wherein:
- FIG. 1 is a high level illustration of a representative networked computer environment in which the system and method of the present invention may be implemented;
- FIG. 2 is a more detailed conceptual representation of a possible operating environment for utilization of the system and method of the present invention wherein files maintained on any number of computers or data centers may be stored in a decentralized computer system through an Internet connection to a number of Redundant Arrays of Independent Nodes (“RAIN”) racks located, for example, at geographically diverse locations;
- FIG. 3 is a logic flow chart depicting the steps in the entry of a computer file into the hash file system of the present invention wherein the hash value for the file is checked against hash values for files previously maintained in a set, or database;
- FIG. 4 is a further logic flow chart depicting the steps in the breakup of a file or other data sequence into hashed pieces resulting in the production of a number of data pieces as well as corresponding probabilistically unique hash values for each piece;
- FIG. 5 is another logic flow chart depicting the comparison of the hash values for each piece of a file to existing hash values in the set (or database), the production of records showing the equivalence of a single hash value for all file pieces with the hash values of the various pieces and whereupon new data pieces and corresponding new hash values are added to the set;
- FIG. 6 is yet another logic flow chart illustrating the steps in the comparison of file hash or directory list hash values to existing directory list hash values and the addition of new file or directory list hash values to the set directory list;
- FIG. 7 is a comparison of the pieces of a representative computer file with their corresponding hash values both before and after editing of a particular piece of the exemplary file;
- FIG. 8 is a conceptual representation of the fact that composite data which may be derived by means of the system and method of the present invention is effectively the same as the data represented explicitly but may instead be created by a “recipe” such as the concatenation of data represented by its corresponding hashes or the result of a function using the data represented by the hashes;
- FIG. 9 is another conceptual representation of how the hash file system and method of the present invention my be utilized to organize data to optimize the reutilization of redundant sequences through the use of hash values as pointers to the data they represent and wherein data may be represented either as explicit byte sequences (atomic data) or as groups of sequences (composites);
- FIG. 10 is a simplified diagram illustrative of a hash file system address translation function for an exemplary 160 bit hash value;
- FIG. 11 is a simplified exemplary illustration of an index stripe splitting function for use with the system and method of the present invention;
- FIG. 12 is a simplified illustration of the overall functionality of the system and method of the present invention for use in the backup of data for a representative home computer having a number of program and document files on
Day 1 and wherein one of the document files is edited onDay 2 together with the addition of a third document file; and - FIG. 13 illustrates the comparison of various pieces of a particular document file marked by a number of “sticky bytes” both before and following editing wherein one of the pieces is thereby changed while other pieces remain the same.
- In a particular implementation of the hash file system and method of the present invention as disclosed herein, its application is directed toward a high availability, high reliability data storage system that leverages rapid advances in commodity computing devices and the robust nature of internetwork technology such as the Internet. Particularly disclosed herein is a hash file system that manages the correspondence of one or more block(s) of data (including but not limited to files, directories, drive images, software applications, digitized voice, and rich media content) together with one or more symbol(s) for that block of data, wherein the symbol may be a number, hash, checksum, binary sequence, or other identifier that is derived from the block of data itself and is statistically, probabilistically, or otherwise effectively unique to that block of data. The system itself works on any computer system including, without limitation: personal computers; supercomputers; distributed or non-distributed networks; storage area networks (“SAN”) using IDE, SCSI or other disk buses; network attached storage (“NAS”) or other systems capable of storing and/or processing data.
- In a particular implementation of the hash file system disclosed herein, the symbol(s) may be derived using one or more hash or checksum generating engines, programs, or algorithms, including but not limited to MD4, MD5, SHA, SHA-1, or their derivatives. Further, the symbol(s) may comprise parts of variable or invariable length symbols derived using a hash or checksum generating engine, program, or algorithm, including but not limited to MD4, MD5, SHA, SHA-1, or other methods of generating probabilistically unique identifiers based on data content. In a particular implementation disclosed herein, file seeks, or lookups for retrieving data or checking on the existence/availability of data, may be accelerated by looking at all or a smaller portion of the symbol, with the symbol portion indicating or otherwise providing the routing information for finding, retrieving, or checking on the existence/availability of the data.
- Further disclosed herein is a system and method for a hash file system wherein the symbols allow for the identification of redundant copies within the system and/or allow for the identification of copies within the system redundant with data presented to the system for filing and storage. The symbols allow for the elimination of, or allow for the screening of, redundant copies of the data and/or parts of the data in the system or in data and/or parts of data presented to the system, without loss of data integrity and can provide for the even distribution of data over available storage for the system. The system and method of the present invention as disclosed herein requires no central operating point and balances processing and/or input/output (“I/O”) load across all computers, supercomputers, or other devices capable of storing and/or processing data attached to the system. The screening of redundant copies of the data and/or parts of the data provided herein allows for the creation, repetitive creation, or retention of intelligent boundaries for screening other data in the system, future data presented to the system, or future data stored by the system.
- The present invention is illustrated and described in terms of a distributed computing environment such as an enterprise computing system using public communication channels such as the Internet. However, an important feature of the present invention is that it is readily scaled upwardly and downwardly to meet the needs of a particular application. Accordingly, unless specified to the contrary the present invention is applicable to significantly larger, more complex network environments as well as small network environments such as conventional LAN systems.
- With reference now to FIG. 1, the present invention may be utilized in conjunction with a novel data storage system on a
network 10. In this figure, anexemplary internetwork environment 10 may include the Internet which comprises a global internetwork formed by logical and physical connection between multiple wide area networks (“WANs”) 14 and local area networks (“LANs”) 16. AnInternet backbone 12 represents the main lines and routers that carry the bulk of the data traffic. Thebackbone 12 is formed by the largest networks in the system that are operated by major Internet service providers (“ISPs”) such as GTE, MCI, Sprint, UUNet, and America Online, for example. While single connection lines are used to conveniently illustrateWANs 14 andLANs 16 connections to theInternet backbone 12, it should be understood that in reality, multi-path, routable physical connections exist betweenmultiple WANs 14 andLANs 16. This makesinternetwork 10 robust when faced with single or multiple failure points. - It is important to distinguish network connections from internal data pathways implemented between peripheral devices within a computer. A “network” comprises a system of general purpose, usually switched physical connections that enable logical connections between processes operating on
nodes 18. The physical connections implemented by a network are typically independent of the logical connections that are established between processes using the network. In this manner, a heterogeneous set of processes ranging from file transfer, mail transfer, and the like can use the same physical network. Conversely, the network can be formed from a heterogeneous set of physical network technologies that are invisible to the logically connected processes using the network. Because the logical connection between processes implemented by a network is independent of the physical connection, internetworks are readily scaled to a virtually unlimited number of nodes over long distances. - In contrast, internal data pathways such as a system bus, peripheral component interconnect (“PCI”) bus, Intelligent Drive Electronics (“IDE”) bus, small computer system interface (“SCSI”) bus, and the like define physical connections that implement special-purpose connections within a computer system. These connections implement physical connections between physical devices as opposed to logical connections between processes. These physical connections are characterized by limited distance between components, limited number of devices that can be coupled to the connection, and constrained format of devices that can be connected over the connection.
- In a particular implementation of the present invention, storage devices may be placed at
nodes 18. The storage at anynode 18 may comprise a single hard drive, or may comprise a managed storage system such as a conventional RAID device having multiple hard drives configured as a single logical volume. Significantly, the present invention manages redundancy operations across nodes, as opposed to within nodes, so that the specific configuration of the storage within any given node is less relevant. - Optionally, one or more of the
nodes 18 may implement storage allocation management (“SAM”) processes that manage data storage acrossnodes 18 in a distributed, collaborative fashion. SAM processes preferably operate with little or no centralized control for the system as whole. SAM processes provide data distribution acrossnodes 18 and implement recovery in a fault-tolerant fashion acrossnetwork nodes 18 in a manner similar to paradigms found in RAID storage subsystems. - However, because SAM processes operate across nodes rather than within a single node or within a single computer, they allow for greater fault tolerance and greater levels of storage efficiency than conventional RAID systems. For example, SAM processes can recover even where a
network node 18,LAN 16, orWAN 14 become unavailable. Moreover, even when a portion of theInternet backbone 12 becomes unavailable through failure or congestion, the SAM processes can recover using data distributed onnodes 18 that remain accessible. In this manner, the present invention leverages the robust nature of internetworks to provide unprecedented availability, reliability, fault tolerance and robustness. - With reference additionally now to FIG. 2, a more detailed conceptual view of an exemplary network computing environment in which the present invention is implemented is depicted. The
internetwork 10 of the preceding figure (orInternet 118 in this figure) enables aninterconnected network 100 of a heterogeneous set of computing devices andmechanisms 102 ranging from a supercomputer ordata center 104 to a hand-held or pen-baseddevice 114. While such devices have disparate data storage needs, they share an ability to retrieve data vianetwork 100 and operate on that data within their own resources.Disparate computing devices 102 including mainframe computers (e.g.,VAX station 106 and IBM AS/400 station 116) as well as personal computer or workstation class devices such as IBMcompatible device 108,Macintosh device 110 andlaptop computer 112 are readily interconnected viainternetwork 10 andnetwork 100. Although not illustrated, mobile and other wireless devices may be coupled to theinternetwork 10. - Internet-based
network 120 comprises a set of logical connections, some of which are made throughInternet 118, between a plurality ofinternal networks 122. Conceptually, Internet-basednetwork 120 is akin to a WAN 14 (FIG. 1) in that it enables logical connections between geographically distant nodes. Internet-basednetworks 120 may be implemented using theInternet 118 or other public and private WAN technologies including leased lines, Fibre Channel, and the like. - Similarly,
internal networks 122 are conceptually akin to LANs 16 (FIG. 1) in that they enable logical connections across a more limited distance thanWAN 14.Internal networks 122 may be implemented using various LAN technologies including Ethernet, Fiber Distributed Data Interface (“FDDI”), Token Ring, Appletalk, Fibre Channel, and the like. - Each
internal network 122 connects one or more redundant arrays of independent nodes (RAIN)elements 124 to implement RAIN nodes 18 (FIG. 1). EachRAIN element 124 comprises a processor, memory, and one or more mass storage devices such as hard disks.RAIN elements 124 also include hard disk controllers that may be conventional IDE or SCSI controllers, or may be managing controllers such as RAID controllers.RAIN elements 124 may be physically dispersed or co-located in one or more racks sharing resources such as cooling and power. Each node 18 (FIG. 1) is independent ofother nodes 18 in that failure or unavailability of onenode 18 does not affect availability ofother nodes 18, and data stored on onenode 18 may be reconstructed from data stored onother nodes 18. - In a particular exemplary implementation, the
RAIN elements 124 may comprise computers using commodity components such as Intel-based microprocessors mounted on a motherboard supporting a PCI bus and 256 megabytes of random access memory (“RAM”) housed in a conventional AT or ATX case. SCSI or IDE controllers may be implemented on the motherboard and/or by expansion cards connected to the PCI bus. Where the controllers are implemented only on the motherboard, a PCI expansion bus may be optionally used. In a particular implementation, the motherboard may implement two mastering EIDE channels and a PCI expansion card which is used to implement two additional mastering EIDE channels so that eachRAIN element 124 includes up to four or more EIDE hard disks. In the particular implementation, each hard disk may comprise an 80 gigabyte hard disk for a total storage capacity of 320 gigabytes or more per RAIN element. The hard disk capacity and configuration withinRAIN elements 124 can be readily increased or decreased to meet the needs of a particular application. The casing also houses supporting mechanisms such as power supplies and cooling devices (not shown). - Each
RAIN element 124 executes an operating system. In a particular implementation, the UNIX or UNIX variant operating system such as Linux may be used. It is contemplated, however, that other operating systems including DOS, Microsoft Windows, Apple Macintosh OS, OS/2, Microsoft Windows NT and the like may be equivalently substituted with predictable changes in performance. The operating system chosen forms a platform for executing application software and processes, and implements a file system for accessing mass storage via the hard disk controller(s). Various application software and processes can be implemented on eachRAIN element 124 to provide network connectivity via a network interface using appropriate network protocols such as user datagram protocol (“UDP”), transmission control protocol (TCP), Internet protocol (IP) and the like. - With reference additionally now to FIG. 3, a logic flow chart is shown depicting the steps in the entry of a computer file into the hash file system of the present invention and wherein the hash value for the file is checked against hash values for files previously maintained in a set, or database.
- The
process 200 begins by entry of a computer file data 202 (e.g. “File A”) into the hash file system (“HFS”) of the present invention upon which a hash function is performed atstep 204. Thedata 206 representing the hash of File A is then compared to the contents of a set containing hash file values atdecision step 208. If thedata 206 is already in the set, then the file's hash value is added to a directory list atstep 210. The contents of theset 212 comprising hash values and corresponding data is provided in the form of existinghash values 214 for the comparison operation ofdecision step 208. On the other hand, if the hash value for File A is not currently in the set, the file is broken into hashed pieces (as will be more fully described hereinafter) atstep 216. - With reference additionally now to FIG. 4, a further logic flow chart is provided depicting the steps in the
process 300 for breakup of a digital sequence (e.g. a file or other data sequence) into hashed pieces. Thisprocess 300 ultimately results in the production of a number of data pieces as well as corresponding probabilistically unique hash values for each piece. - The
file data 302 is divided into pieces based on commonality with other pieces in the system or the likelihood of pieces being found to be in common in the future atstep 304. The results of the operation ofstep 304 upon thefile data 302 is, in the representative example shown, the production of fourfile pieces 306 denominated A1 through A5 inclusively. Each of thefile pieces 306 is then operated on atstep 308 by placing it through individual hash function operations to assign a probabilistically unique number to each of thepieces 306 A1 through A5. The results of the operation atstep 308 is that each of the pieces 306 (A1 through A5) has an associated, probabilistically unique hash value 310 (shown as A1 Hash through A5 Hash respectively). The file division process ofstep 304 is described in greater detail hereinafter in conjunction with the unique “sticky byte” operation also disclosed herein. - With reference additionally now to FIG. 5, another logic flow chart is shown depicting a
comparison process 400 for the hash values 310 of eachpiece 306 of the file to those of existinghash values 214 maintained in theset 212. Particularly, atstep 402, the hash values 310 for eachpiece 306 of the file are compared to existinghash values 214 and new hash values 408 and correspondingnew data pieces 406 are added to theset 212. In this way, hash values 408 not previously present in the database set 212 are added together with their associateddata pieces 406. Theprocess 400 also results in the production ofrecords 404 showing the equivalence of a single hash value for all file pieces with the hash values 310 of thevarious pieces 306. - With reference additionally now to FIG. 6, yet another logic flow chart is shown illustrating a
process 500 for the comparison of file hash or directory list hash values to existing directory list hash values and the addition of new file or directory list hash values to the database directory list. Theprocess 500 operates on storeddata 502 which comprises an accumulated list of file names, file meta-data (e.g. date, time, file length, file type etc.) and the file's hash value for each item in a directory. Atstep 504, the hash function is run upon the contents of the directory list.Decision step 506 is operative to determine whether or not the hash value for the directory list is in theset 212 of existing hash values 214. If it is, then theprocess 500 returns to add another file hash or directory list hash to a directory list. Alternatively, if the hash value for the directory list is not already in the database set 212, the hash value and data for the directory list are added to thedatabase 212 set atstep 508. - With reference additionally now to FIG. 7, a
comparison 600 of thepieces 306 of a representative computer file (i.e. “File A”) with their corresponding hash values 310 is shown both before and after editing of a particular piece of the exemplary file. In this example, therecord 404 contains the hash value of File A as well as the hash values 310 of each of the pieces of the file A1 through A5. A representative edit of the File A may produce a change in the data for piece A2 (now represented by A2-b) of thefile pieces 306A along with a corresponding change in the hash value A2-b of the hash values 310A. The edited file piece now produces an updatedrecord 404A which includes the modified hash value of File A and the modified hash value of piece A2-b. - With reference additionally now to FIG. 8, a
conceptual representation 700 is shown illustrative of the fact that composite data (such ascomposite data 702 and 704) derived by means of the system and method of the present invention, is effectively the same as thedata 706 represented explicitly but is instead created by a “recipe”, or formula. In the example shown, this recipe includes the concatenation of data represented by its correspondinghashes 708 or the result of a function using the data represented by the hashes. The data blocks 706 may be variable length quantities as shown and the hash values 708 are derived from their associated data blocks. As previously stated, the hash values 708 are a probabilistically unique identification of the corresponding data pieces but truly unique identifications can be used instead or intermixed therewith. It should also be noted that thecomposite data - With reference additionally now to FIG. 9, another
conceptual representation 800 is shown of how the hash file system and method of the present invention may be utilized to organizedata 802 to optimize the reutilization of redundant sequences through the use ofhash values 806 as pointers to the data they represent and whereindata 802 may be represented either as explicit byte sequences (atomic data) 808 or as groups of sequences (composites) 804. - The
representation 800 illustrates the tremendous commonality of recipes and data that gets reused at every level. The basic structure of the hash file system of the present invention is essentially that of a “tree” or “bush” wherein the hash values 806 are used instead of conventional pointers. The hash values 806 are used in the recipes to point to the data or another hash value that could also itself be a recipe. In essence then, recipes can point to other recipes that point to still other recipes that ultimately point to some specific data that may, itself, point to other recipes that point to even more data, eventually getting down to nothing but data. - With reference additionally now to FIG. 10, a simplified diagram900 is shown illustrative of a hash file system address translation function for an exemplary 160
bit hash value 902. Thehash value 902 includes a data structure comprising afront portion 904 and aback portion 906 as shown and the diagram 900 illustrates a particular “0 of 1” operation that is used for enabling the use of thehash value 902 to go to the location of the particular node in the system that contains the corresponding data. - The diagram900 illustrates how the
front portion 904 of thehash value 902 data structure may be used to indicate the hash prefix to stripe identification (“ID”) 908 and how that is, in turn, utilized to map the stripe ID to IP address and the ID class toIP address 910. In this example, the “S2” indicatesstripe 2 ofindex Node 37 912. Theindex stripe 912 ofNode 37 then indicatesstripe 88 ofdata Node 73 indicated by thereference numeral 914. In operation then, a portion of thehash value 902 itself may be used to indicate which node in the system contains the relevant data, another portion of thehash value 902 may be used to indicate which stripe of data at that particular node and yet another portion of thehash value 902 to indicate where within that stripe the data resides. Through this three step process, it can rapidly be determined if the data represented by thehash value 902 is already present in the system. - With reference additionally now to FIG. 11, a simplified exemplary illustration of an index
stripe splitting function 1000 is shown for use with the system and method of the present invention. In this illustration, anexemplary function 1000 is shown that may be used to effectively split a stripe 1002 (S2) into two stripes 1004 (S2) and 1006 (S7) should one stripe become too full. In this example, the odd entries have been moved to stripe 1006 (S7) while the even ones remain instripe 1004. Thisfunction 1000 is one example of how stripe entries may be handled as the overall system grows in size and complexity. - With reference additionally now to FIG. 12, a
simplified illustration 1100 of the overall functionality of the system and method of the present invention is shown for use, for example, in the backup of data for a representative home computer having a number of program anddocument files Day 1 and wherein the program files 1102B remain the same onDay 2 while one of the document files 1104B is edited on Day 2 (Y.doc) together with the addition of a third document file (Z.doc). - The
illustration 1100 shows the details of how a computer file system may be broken into pieces and then listed as a series of recipes on a global data protection network (“gDPN”) to reconstruct the original data from the pieces. This very small computer system is shown in the form of a “snapshot” on “Day 1” and then subsequently on “Day 2”. On “Day 1”, the “program files H5” and “my documents H6” are illustrated by numeral 1106, with the former being represented by arecipe 1108 wherein a first executable file is represented by ahash value H1 1114 and a second represented by ahash value H2 1112. The document files are represented byhash value H6 1110 with the first document being represented byhash value H3 1118 and the second byhash value H4 1116. Thereafter on “Day 2”, the “program files H5” and “my documents” H10 indicated by numeral 1120 show that the “program files H5” have not changed, but the “my document H10” have. H10 indicated by numeral 1122 shows the “X.doc” is still represented byhash value H3 1118 while “Y.doc” is now represented by hash value H8 atnumber 1124. New document file “Z.doc” is now represented by hash value H9 at numeral 1126. - In this example, it can be seen that on
Day 2, some of the files have changed, while others have not. In the files that have changed, some of the pieces of them have not changed while other pieces have. Through the use of the hash file system of the present invention, a “snap shot” of the computer system can be made on Day 1 (producing the necessary recipes for reconstruction of the computer files as they exist then) and then onDay 2 through the reuse of some of the previous day's recipes together with the reformulation of others and the addition of new ones to describe the system at that time. In this manner, the computer system may be recreated in its entirety at any point in time on bothDay 1 orDay 2 as well as on any subsequent day. - With reference additionally now to FIG. 13, a
comparison 1200 of various pieces of a particular document file marked by a number of “sticky bytes” 1204 is shown both before (Day 1 1202A) and following editing (Day 2 1202B) wherein one of the pieces is thereby changed while other pieces remain the same. - For example, on
Day 1, file 1202A comprises variable length pieces 1206 (1.1), 1208 (1.2), 1210 (2.1), 1212 (2.), 1214 (2.3) and 1216 (3.1). OnDay 2,pieces piece 1212 has now been edited to producepiece 1212A (thus having a differing hash value). - Data sticky bytes (or “sticky points”) are a unique, fully automated way to sub-divide computer files such that common elements may be found on multiple related and unrelated computers without the need for communication between the computers. The means in which data sticky points are found is completely mathematical in nature and performs equally well regardless of the data content of the files. In the hash file system of the present invention, all data objects may be indexed, stored and retrieved using, for example (but not limited to), an industry standard checksum such as: MD4, MD5, SHA, or SHA-1. In operation, if two files have the same checksum, it may be considered to be highly likely that they are the same file. Using the system and method disclosed herein, data sticky points may be produced with a standard mathematical distribution and with standard deviations that are a small percentage of the target size.
- A data sticky point is a statistically infrequent arrangement of n bytes. In this case, an example is given with 32 bytes because of its ease in implementation in current microprocessor technology.
- A rolling hash of 32 bits could be generated for the file “f”.
//f[i] = is the ith byte of the file “f”. //scramble is a 256 entry array of integers with each //being 32 bits wide; //these integers are typically chosen to uniformly //span the range. int t=8 //target number of trailing zeros int hash = 0; int sticky_bits; for(int i=O; i<filesize; i++) hash = hash >> 1 |scramble[f[i]]; //At every byte in the file, hash represents the //rolling hash of the file. sticky_bits = (hash − 1){circumflex over ( )}hash; //sticks_bits is a variable which will have the //number of ones in the hash //that correspond to the number of trailing zeros in //the “hash”. number_of_bits = count_ones(stick-bits); if(number_of_bits > t) output_sticky_point(i); } - A sticky point is defined to be a rolling hash with at least the number of trailing zeros as the target number with the hash represented in binary. Statistically speaking, this algorithm will find points that are spaced at 2{circumflex over ( )}t, where t is the target number of trailing zeros. For this example where t=8, the algorithm will find, on average, sticky points that are spaced at 2{circumflex over ( )}8=256 bytes apart.
- A rolling hash of 32 bits may be generated for the f file where:
f[i] = is the ith byte of the file f. scramble is a 256 entry array of random elements with each being n bits wide; int t=8 // target number of trailing zeros int target_distance = 256; // 2 to the power of 8 int hash = 0; int sticky_bits; int distance = 0; int last_point = 0; for(int i=O; i<filesize; i++) { hash = hash >> 1| scramble[f[i]]; //At every byte in the file hash represents the //rolling hash of the file. sticky_bits = (hash − 1){circumflex over ( )}hash; //sticks_bits is a variable which will have the //number of ones that correspond to the number of //trailing zeros in the “hash”. number_of_bits = count_ones(stick_bits); distance = i-last_point; if(number_of_bits * distance/target_distance >t) last_point =i; output_sticky_point(i); } } - While the hashing function utilized to implement the hash file system of the present invention requires a moderately complex computation, it is well within the capability of present day computer systems. Hashing functions are inherently probabilistic and any hashing function might possibly produce incorrect results when two different data objects happen to have the same hash value. However, the system and method herein disclosed mitigates this problem by using well known and researched hashing functions that reduce the probability of collision down to levels acceptable for reliable use (i.e. one chance in a trillion trillion), far less than the error rates otherwise tolerated in conventional computer hardware operations.
- Although as used herein, the term “Internet infrastructure” encompasses a variety of hardware and software mechanisms, the term primarily refers to routers, router software, and physical links between these routers that function to transport data packets from one network node to another. As also used herein, a “digital sequence” may comprise, without limitation, computer program files, computer applications, data files, network packets, streaming data such as multimedia (including audio and video), telemetry data and any other form of data which can be represented by a digital or numeric sequence. The probabilistically unique identifiers produced by means of the hash file system and method of the present invention may also be used as URLs in network applications.
- While there have been described above the principles of the present invention in conjunction with specific implementations and applications of the system and method of the present invention, it is to be clearly understood that the foregoing description is made only by way of example and not as a limitation to the scope of the invention. Particularly, it is recognized that the teachings of the foregoing disclosure will suggest other modifications to those persons skilled in the relevant art. Such modifications may involve other features which are already known per se and which may be used instead of or in addition to features already described herein. Although claims have been formulated in this application to particular combinations of features, it should be understood that the scope of the disclosure herein also includes any novel feature or any novel combination of features disclosed either explicitly or implicitly or any generalization or modification thereof which would be apparent to persons skilled in the relevant art, whether or not such relates to the same invention as presently claimed in any claim and whether or not it mitigates any or all of the same technical problems as confronted by the present invention. The applicants hereby reserve the right to formulate new claims to such features and/or combinations of such features during the prosecution of the present application or of any further application derived therefrom.
Claims (40)
1. A method for managing data comprising:
producing a probabilistically unique identifier for a digital sequence; and
comparing said probabilistically unique identifier to a list of other identifiers with their corresponding digital sequences.
2. The method of claim 1 further comprising:
adding said probabilistically unique identifier to said list if said probabilistically unique identifier is not previously in said list.
3. The method of claim 1 further comprising:
removing said probabilistically unique identifier from said list if said probabilistically unique identifier is previously in said list.
4. The method of claim 2 further comprising:
adding said digital sequence corresponding to said probabilistically unique identifier to said list.
5. The method of claim 3 further comprising:
removing said digital sequence corresponding to said probabilistically unique identifier from said list.
6. The method of claim 4 further comprising:
adding a correspondence between said digital sequence and said probabilistically unique identifier for that sequence.
7. The method of claim 1 wherein said step of producing comprises:
hashing said digital sequence to produce said probabalistically unique identifier.
8. The method of claim 7 wherein said step of hashing is carried out by means of an industry standard digest algorithm.
9. The method of claim 8 wherein said step of hashing is carried out by one of an MD4, MD5, SHA or SHA-1 algorithm.
10. The method of claim 1 wherein said step of producing comprises:
generating a checksum for said digital sequence to produce said probabilistically unique identifier.
11. The method of claim 1 wherein said digital sequence is descriptive meta data of at least one other digital sequence.
12. The method of claim 1 wherein said digital sequence is descriptive meta data of at least one probabilistically unique identifier.
13. The method of claim 1 wherein said digital sequence describes a method that represents at least one digital sequence.
14. A method for managing data comprising:
dividing a digital sequence into a plurality of shorter digital sequences; and
producing probabilistically unique identifiers for each said plurality of shorter digital sequences; and
comparing said probabilistically unique identifiers to a list of other identifiers.
15. The method of claim 14 further comprising the step of:
dividing said digital sequence into a plurality of shorter digital sequences; and
producing a like plurality of probabilistically unique identifiers corresponding to each of said plurality of shorter digital sequences.
16. The method of claim 14 further comprising;
comparing each plurality of identifiers to said list.
17. The method of claim 14 wherein said step of dividing produces said shorter digital sequences having individually variable lengths.
18. The method of claim 14 wherein said step of dividing is based on the content of said digital sequence.
19. The method of claim 14 wherein said step of dividing is based on meta data describing said digital sequence.
20. The method of claim 14 wherein said step of dividing produces said shorter digital sequences having substantially invariable lengths.
21. The method of claim 14 wherein said step of producing said like plurality of probabilistically unique identifiers comprises:
individually hashing said shorter digital sequences to produce said like plurality of probabilistically unique identifiers.
22. The method of claim 14 further comprising the step of:
adding said plurality of shorter digital sequences and said corresponding like plurality of probabilistically unique identifiers to said list.
23. The method of claim 14 further comprising the step of:
removing said plurality of shorter digital sequences and said corresponding like plurality of probabilistically unique identifiers from said list.
24. The method of claim 9 further comprising the step of:
utilizing at least a portion of said probabilistically unique identifier as an indicator to a location in said list for said step of comparing.
25. A computer program product comprising:
a computer usable medium having computer readable code embodied therein for managing data, said computer program product comprising:
computer readable program code devices configured to cause a computer to effect producing a probabilistically unique identifier for a digital sequence; and
computer readable program code devices configured to cause a computer to effect comparing said probabilistically unique identifier to a list of other identifiers corresponding to other digital sequences.
26. The computer program product of claim 25 further comprising:
computer readable program code devices configured to cause a computer to effect adding said probabilistically unique identifier to said list if said probabilistically unique identifier is not previously in said list.
27. The computer program product of claim 26 further comprising:
computer readable program code devices configured to cause a computer to effect adding said corresponding digital sequence to said list.
28. The computer program product of claim 25 wherein said computer readable program code devices configured to cause said computer to effect producing comprises:
computer readable program code devices configured to cause a computer to effect hashing said digital sequence to produce said probabilistically unique identifier.
29. The computer program product of claim 28 wherein said computer readable program code devices configured to cause a computer to effect hashing is carried out by means of an industry standard digest algorithm.
30. The computer program product of claim 29 wherein said computer readable program code devices configured to cause a computer to effect hashing is carried out by one of an MD4, MD5, SHA or SHA-1 algorithm.
31. The computer program product of claim 25 wherein said computer readable program code devices configured to cause a computer to effect producing comprises:
computer readable program code devices configured to cause a computer to effect generating a checksum for said digital sequence to produce said probabilistically unique identifier.
32. The computer program product of claim 25 further comprising:
computer readable program code devices configured to cause a computer to effect creating a directory list containing said probabilistically unique identifier for said digital sequence.
33. The computer program product of claim 25 further comprising:
computer readable program code devices configured to cause a computer to effect dividing said digital sequence into a plurality of shorter digital sequences; and
computer readable program code devices configured to cause a computer to effect producing a like plurality of probabilistically unique identifiers corresponding to each of said plurality of shorter digital sequences.
34. The computer program product of claim 33 wherein said computer readable program code devices configured to cause a computer to effect dividing produces said shorter digital sequences having individually variable length.
35. The computer program product of claim 33 wherein said computer readable program code devices configured to cause a computer to effect dividing produces said shorter digital sequences having substantially invariable length.
36. The computer program product of claim 33 wherein said computer readable program code devices configured to cause a computer to effect producing said like plurality of probabilistically unique identifiers comprises:
computer readable program code devices configured to cause a computer to effect individually hashing said shorter digital sequences to produce said like plurality of probabilistically unique identifiers.
37. The computer program product of claim 33 further comprising:
computer readable program code devices configured to cause a computer to effect adding said plurality of shorter digital sequences and said corresponding like plurality of probabilistically unique identifiers to said list.
38. The computer program product of claim 25 further comprising:
computer readable program code devices configured to cause a computer to effect utilizing at least a portion of said probabilistically unique identifier as an index into a table of locations for said list for said step of comparing.
39. A method for managing data comprising:
producing a probabilistically unique identifier for a digital sequence; and
comparing said probabilistically unique identifier to a list of other identifiers corresponding to other digital sequences.
40. The method of claim 39 further comprising:
adding said probabilistically unique identifier to said list if said probabilistically unique identifier is not previously in said list.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US10/757,753 US20040148306A1 (en) | 2000-02-18 | 2004-01-14 | Hash file system and method for use in a commonality factoring system |
Applications Claiming Priority (3)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US18376200P | 2000-02-18 | 2000-02-18 | |
US09/777,150 US6704730B2 (en) | 2000-02-18 | 2001-02-05 | Hash file system and method for use in a commonality factoring system |
US10/757,753 US20040148306A1 (en) | 2000-02-18 | 2004-01-14 | Hash file system and method for use in a commonality factoring system |
Related Parent Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US09/777,150 Continuation US6704730B2 (en) | 2000-02-18 | 2001-02-05 | Hash file system and method for use in a commonality factoring system |
Publications (1)
Publication Number | Publication Date |
---|---|
US20040148306A1 true US20040148306A1 (en) | 2004-07-29 |
Family
ID=26879495
Family Applications (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US09/777,150 Expired - Lifetime US6704730B2 (en) | 2000-02-18 | 2001-02-05 | Hash file system and method for use in a commonality factoring system |
US10/757,753 Abandoned US20040148306A1 (en) | 2000-02-18 | 2004-01-14 | Hash file system and method for use in a commonality factoring system |
Family Applications Before (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US09/777,150 Expired - Lifetime US6704730B2 (en) | 2000-02-18 | 2001-02-05 | Hash file system and method for use in a commonality factoring system |
Country Status (1)
Country | Link |
---|---|
US (2) | US6704730B2 (en) |
Cited By (101)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20010034795A1 (en) * | 2000-02-18 | 2001-10-25 | Moulton Gregory Hagan | System and method for intelligent, globally distributed network storage |
US20030182291A1 (en) * | 2002-03-20 | 2003-09-25 | Sreenath Kurupati | Method and data structure for a low memory overhead database |
US20040225655A1 (en) * | 2000-11-06 | 2004-11-11 | Moulton Gregory Hagan | System and method for unorchestrated determination of data sequences using sticky factoring to determine breakpoints in digital sequences |
US20060031529A1 (en) * | 2004-06-03 | 2006-02-09 | Keith Robert O Jr | Virtual application manager |
US20060047716A1 (en) * | 2004-06-03 | 2006-03-02 | Keith Robert O Jr | Transaction based virtual file system optimized for high-latency network connections |
US20060047946A1 (en) * | 2004-07-09 | 2006-03-02 | Keith Robert O Jr | Distributed operating system management |
US20060224544A1 (en) * | 2005-03-04 | 2006-10-05 | Keith Robert O Jr | Pre-install compliance system |
US20060224545A1 (en) * | 2005-03-04 | 2006-10-05 | Keith Robert O Jr | Computer hardware and software diagnostic and report system |
US7143251B1 (en) * | 2003-06-30 | 2006-11-28 | Data Domain, Inc. | Data storage using identifiers |
US20070233633A1 (en) * | 2005-03-04 | 2007-10-04 | Keith Robert O Jr | Computer hardware and software diagnostic and report system |
US20070274315A1 (en) * | 2006-05-24 | 2007-11-29 | Keith Robert O | System for and method of securing a network utilizing credentials |
US20080077630A1 (en) * | 2006-09-22 | 2008-03-27 | Keith Robert O | Accelerated data transfer using common prior data segments |
US20080077724A1 (en) * | 2006-09-22 | 2008-03-27 | Parthasarathy Sarangam | Interrupt coalescing control scheme |
US7386616B1 (en) * | 2003-05-09 | 2008-06-10 | Google Inc. | System and method for providing load balanced processing |
US20080229037A1 (en) * | 2006-12-04 | 2008-09-18 | Alan Bunte | Systems and methods for creating copies of data, such as archive copies |
US20080243958A1 (en) * | 2006-12-22 | 2008-10-02 | Anand Prahlad | System and method for storing redundant information |
WO2008147516A1 (en) * | 2007-05-22 | 2008-12-04 | Network Appliance, Inc. | System and method for on-the-fly elimination of redundant date |
US20090007261A1 (en) * | 2007-06-29 | 2009-01-01 | Smith Mark A | Receiving data in a data store in a server computer system |
US20090051705A1 (en) * | 2007-08-20 | 2009-02-26 | Shih-Kuang Tsai | Method for automatically memorizing picture orientation |
US20090234892A1 (en) * | 2008-03-14 | 2009-09-17 | International Business Machines Corporation | Method and system for assuring integrity of deduplicated data |
US20090271454A1 (en) * | 2008-04-29 | 2009-10-29 | International Business Machines Corporation | Enhanced method and system for assuring integrity of deduplicated data |
US20090319585A1 (en) * | 2008-06-24 | 2009-12-24 | Parag Gokhale | Application-aware and remote single instance data management |
US20100005259A1 (en) * | 2008-07-03 | 2010-01-07 | Anand Prahlad | Continuous data protection over intermittent connections, such as continuous data backup for laptops or wireless devices |
US20100058002A1 (en) * | 2008-08-27 | 2010-03-04 | Netapp, Inc. | System and method for file system level compression using compression group descriptors |
US20100082672A1 (en) * | 2008-09-26 | 2010-04-01 | Rajiv Kottomtharayil | Systems and methods for managing single instancing data |
US20100169287A1 (en) * | 2008-11-26 | 2010-07-01 | Commvault Systems, Inc. | Systems and methods for byte-level or quasi byte-level single instancing |
US20100250549A1 (en) * | 2009-03-30 | 2010-09-30 | Muller Marcus S | Storing a variable number of instances of data objects |
US7840514B2 (en) | 2006-09-22 | 2010-11-23 | Maxsp Corporation | Secure virtual private network utilizing a diagnostics policy and diagnostics engine to establish a secure network connection |
US7844686B1 (en) | 2006-12-21 | 2010-11-30 | Maxsp Corporation | Warm standby appliance |
WO2010151813A1 (en) * | 2009-06-26 | 2010-12-29 | Simplivt Corporation | File system |
US20110022566A1 (en) * | 2009-06-26 | 2011-01-27 | Simplivt Corporation | File system |
US8099605B1 (en) | 2006-06-05 | 2012-01-17 | InventSec AB | Intelligent storage device for backup system |
US8131924B1 (en) | 2008-03-19 | 2012-03-06 | Netapp, Inc. | De-duplication of data stored on tape media |
US8175418B1 (en) | 2007-10-26 | 2012-05-08 | Maxsp Corporation | Method of and system for enhanced data storage |
US8307239B1 (en) | 2007-10-26 | 2012-11-06 | Maxsp Corporation | Disaster recovery appliance |
US8364652B2 (en) | 2010-09-30 | 2013-01-29 | Commvault Systems, Inc. | Content aligned block-based deduplication |
CN102915325A (en) * | 2012-08-11 | 2013-02-06 | 深圳市极限网络科技有限公司 | Md5 Hash list-based file decomposing and combining technique |
US20130086000A1 (en) * | 2011-09-29 | 2013-04-04 | International Business Machines Corporation | Managing back up operations for data |
US8423821B1 (en) | 2006-12-21 | 2013-04-16 | Maxsp Corporation | Virtual recovery server |
US8484162B2 (en) | 2008-06-24 | 2013-07-09 | Commvault Systems, Inc. | De-duplication systems and methods for application-specific data |
US8572340B2 (en) | 2010-09-30 | 2013-10-29 | Commvault Systems, Inc. | Systems and methods for retaining and using data block signatures in data protection operations |
US8578120B2 (en) | 2009-05-22 | 2013-11-05 | Commvault Systems, Inc. | Block-level single instancing |
US8589323B2 (en) | 2005-03-04 | 2013-11-19 | Maxsp Corporation | Computer hardware and software diagnostic and report system incorporating an expert system and agents |
US8645515B2 (en) | 2007-10-26 | 2014-02-04 | Maxsp Corporation | Environment manager |
US8667273B1 (en) | 2006-05-30 | 2014-03-04 | Leif Olov Billstrom | Intelligent file encryption and secure backup system |
US8898319B2 (en) | 2006-05-24 | 2014-11-25 | Maxsp Corporation | Applications and services as a bundle |
US8930306B1 (en) | 2009-07-08 | 2015-01-06 | Commvault Systems, Inc. | Synchronized data deduplication |
US8935492B2 (en) | 2010-09-30 | 2015-01-13 | Commvault Systems, Inc. | Archiving data objects using secondary copies |
US8954446B2 (en) | 2010-12-14 | 2015-02-10 | Comm Vault Systems, Inc. | Client-side repository in a networked deduplicated storage system |
US9020900B2 (en) | 2010-12-14 | 2015-04-28 | Commvault Systems, Inc. | Distributed deduplicated storage system |
US9020890B2 (en) | 2012-03-30 | 2015-04-28 | Commvault Systems, Inc. | Smart archiving and data previewing for mobile devices |
CN104731792A (en) * | 2013-12-19 | 2015-06-24 | 中国银联股份有限公司 | Method and system for verifying database consistency and method and system for positioning data difference |
US9098495B2 (en) | 2008-06-24 | 2015-08-04 | Commvault Systems, Inc. | Application-aware and remote single instance data management |
WO2015153259A1 (en) * | 2014-03-31 | 2015-10-08 | Amazon Technologies, Inc. | Namespace management in distributed storage systems |
US9218375B2 (en) | 2012-06-13 | 2015-12-22 | Commvault Systems, Inc. | Dedicated client-side signature generator in a networked storage system |
US20160042183A1 (en) * | 2013-03-28 | 2016-02-11 | Irdeto B.V. | Generating identifier |
US9357031B2 (en) | 2004-06-03 | 2016-05-31 | Microsoft Technology Licensing, Llc | Applications as a service |
US9575673B2 (en) | 2014-10-29 | 2017-02-21 | Commvault Systems, Inc. | Accessing a file system using tiered deduplication |
US9633056B2 (en) | 2014-03-17 | 2017-04-25 | Commvault Systems, Inc. | Maintaining a deduplication database |
US9633033B2 (en) | 2013-01-11 | 2017-04-25 | Commvault Systems, Inc. | High availability distributed deduplicated storage system |
US9633022B2 (en) | 2012-12-28 | 2017-04-25 | Commvault Systems, Inc. | Backup and restoration for a deduplicated file system |
US9646176B2 (en) | 2015-03-24 | 2017-05-09 | TmaxData Co., Ltd. | Method for encrypting database |
US9772787B2 (en) | 2014-03-31 | 2017-09-26 | Amazon Technologies, Inc. | File storage using variable stripe sizes |
US9779015B1 (en) | 2014-03-31 | 2017-10-03 | Amazon Technologies, Inc. | Oversubscribed storage extents with on-demand page allocation |
US9811522B2 (en) | 2013-08-21 | 2017-11-07 | Hewlett Packard Enterprise Development Lp | System and method for transforming a source virtual machine without copying of payload data |
US9875183B2 (en) | 2012-02-24 | 2018-01-23 | Hewlett Packard Enterprise Development Lp | Method and apparatus for content derived data placement in memory |
US10061663B2 (en) | 2015-12-30 | 2018-08-28 | Commvault Systems, Inc. | Rebuilding deduplication data in a distributed deduplication data storage system |
US10089337B2 (en) | 2015-05-20 | 2018-10-02 | Commvault Systems, Inc. | Predicting scale of data migration between production and archive storage systems, such as for enterprise customers having large and/or numerous files |
US10140312B2 (en) | 2016-03-25 | 2018-11-27 | Amazon Technologies, Inc. | Low latency distributed storage service |
US10148433B1 (en) | 2009-10-14 | 2018-12-04 | Digitalpersona, Inc. | Private key/public key resource protection scheme |
US10255340B2 (en) | 2011-06-23 | 2019-04-09 | Hewlett Packard Enterprise Development Lp | Method and apparatus for distributed configuration management |
US10264071B2 (en) | 2014-03-31 | 2019-04-16 | Amazon Technologies, Inc. | Session management in distributed storage systems |
US10303458B2 (en) | 2016-09-29 | 2019-05-28 | Hewlett Packard Enterprise Development Lp | Multi-platform installer |
US10324897B2 (en) | 2014-01-27 | 2019-06-18 | Commvault Systems, Inc. | Techniques for serving archived electronic mail |
US10339106B2 (en) | 2015-04-09 | 2019-07-02 | Commvault Systems, Inc. | Highly reusable deduplication database after disaster recovery |
US10372685B2 (en) | 2014-03-31 | 2019-08-06 | Amazon Technologies, Inc. | Scalable file storage service |
US10380072B2 (en) | 2014-03-17 | 2019-08-13 | Commvault Systems, Inc. | Managing deletions from a deduplication database |
US10474636B2 (en) | 2016-03-25 | 2019-11-12 | Amazon Technologies, Inc. | Block allocation for low latency file systems |
US10481824B2 (en) | 2015-05-26 | 2019-11-19 | Commvault Systems, Inc. | Replication using deduplicated secondary copy data |
US10545927B2 (en) | 2016-03-25 | 2020-01-28 | Amazon Technologies, Inc. | File system mode switching in a distributed storage service |
US10587454B2 (en) | 2018-01-30 | 2020-03-10 | Hewlett Packard Enterprise Development Lp | Object counts persistence for object stores |
US10860738B2 (en) | 2018-01-30 | 2020-12-08 | Hewlett Packard Enterprise Development Lp | Augmented metadata and signatures for objects in object stores |
US10884633B2 (en) | 2015-01-13 | 2021-01-05 | Hewlett Packard Enterprise Development Lp | System and method for optimized signature comparisons and data replication |
US10887176B2 (en) | 2017-03-30 | 2021-01-05 | Hewlett Packard Enterprise Development Lp | Predicting resource demand in computing environments |
US10997153B2 (en) | 2018-04-20 | 2021-05-04 | Hewlett Packard Enterprise Development Lp | Transaction encoding and transaction persistence according to type of persistent storage |
US11010300B2 (en) | 2017-05-04 | 2021-05-18 | Hewlett Packard Enterprise Development Lp | Optimized record lookups |
US11010258B2 (en) | 2018-11-27 | 2021-05-18 | Commvault Systems, Inc. | Generating backup copies through interoperability between components of a data storage management system and appliances for data storage and deduplication |
US11126755B2 (en) | 2018-01-30 | 2021-09-21 | Hewlett Packard Enterprise Development Lp | Object signatures in object stores |
US11243703B2 (en) | 2018-04-27 | 2022-02-08 | Hewlett Packard Enterprise Development Lp | Expandable index with pages to store object records |
US11249858B2 (en) | 2014-08-06 | 2022-02-15 | Commvault Systems, Inc. | Point-in-time backups of a production application made accessible over fibre channel and/or ISCSI as data sources to a remote application by representing the backups as pseudo-disks operating apart from the production application and its host |
US11294768B2 (en) | 2017-06-14 | 2022-04-05 | Commvault Systems, Inc. | Live browsing of backed up data residing on cloned disks |
US11314424B2 (en) | 2015-07-22 | 2022-04-26 | Commvault Systems, Inc. | Restore for block-level backups |
US11321195B2 (en) | 2017-02-27 | 2022-05-03 | Commvault Systems, Inc. | Hypervisor-independent reference copies of virtual machine payload data based on block-level pseudo-mount |
US11416341B2 (en) | 2014-08-06 | 2022-08-16 | Commvault Systems, Inc. | Systems and methods to reduce application downtime during a restore operation using a pseudo-storage device |
US11436038B2 (en) | 2016-03-09 | 2022-09-06 | Commvault Systems, Inc. | Hypervisor-independent block-level live browse for access to backed up virtual machine (VM) data and hypervisor-free file-level recovery (block- level pseudo-mount) |
US11442896B2 (en) | 2019-12-04 | 2022-09-13 | Commvault Systems, Inc. | Systems and methods for optimizing restoration of deduplicated data stored in cloud-based storage resources |
US11463264B2 (en) | 2019-05-08 | 2022-10-04 | Commvault Systems, Inc. | Use of data block signatures for monitoring in an information management system |
US11593217B2 (en) | 2008-09-26 | 2023-02-28 | Commvault Systems, Inc. | Systems and methods for managing single instancing data |
US11687424B2 (en) | 2020-05-28 | 2023-06-27 | Commvault Systems, Inc. | Automated media agent state management |
US11698727B2 (en) | 2018-12-14 | 2023-07-11 | Commvault Systems, Inc. | Performing secondary copy operations based on deduplication performance |
US11829251B2 (en) | 2019-04-10 | 2023-11-28 | Commvault Systems, Inc. | Restore using deduplicated secondary copy data |
Families Citing this family (175)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US6704730B2 (en) * | 2000-02-18 | 2004-03-09 | Avamar Technologies, Inc. | Hash file system and method for use in a commonality factoring system |
EP1311978A4 (en) * | 2000-07-20 | 2004-03-03 | Repliweb Inc | Focal point compression method and apparatus |
US7779117B2 (en) | 2002-05-31 | 2010-08-17 | Aol Inc. | Monitoring digital images |
US7197513B2 (en) * | 2000-12-08 | 2007-03-27 | Aol Llc | Distributed image storage architecture |
US7047420B2 (en) * | 2001-01-17 | 2006-05-16 | Microsoft Corporation | Exclusive encryption |
CN1316397C (en) * | 2001-02-12 | 2007-05-16 | Emc公司 | System and method of indexing unique electronic mail messages and uses for same |
US7478243B2 (en) * | 2001-03-21 | 2009-01-13 | Microsoft Corporation | On-disk file format for serverless distributed file system with signed manifest of file modifications |
US7043637B2 (en) * | 2001-03-21 | 2006-05-09 | Microsoft Corporation | On-disk file format for a serverless distributed file system |
US6981138B2 (en) * | 2001-03-26 | 2005-12-27 | Microsoft Corporation | Encrypted key cache |
US7062490B2 (en) | 2001-03-26 | 2006-06-13 | Microsoft Corporation | Serverless distributed file system |
US7310644B2 (en) * | 2001-06-06 | 2007-12-18 | Microsoft Corporation | Locating potentially identical objects across multiple computers |
US6988124B2 (en) * | 2001-06-06 | 2006-01-17 | Microsoft Corporation | Locating potentially identical objects across multiple computers based on stochastic partitioning of workload |
US7134041B2 (en) * | 2001-09-20 | 2006-11-07 | Evault, Inc. | Systems and methods for data backup over a network |
US8266113B2 (en) * | 2003-04-01 | 2012-09-11 | Cybersoft, Inc. | Methods, apparatus and articles of manufacture for computer file integrity and baseline maintenance |
US6795903B2 (en) * | 2002-01-17 | 2004-09-21 | Thomas Licensing S.A. | System and method for searching for duplicate data |
JP4154893B2 (en) * | 2002-01-23 | 2008-09-24 | 株式会社日立製作所 | Network storage virtualization method |
US7020665B2 (en) * | 2002-03-07 | 2006-03-28 | Microsoft Corporation | File availability in distributed file storage systems |
US20030188275A1 (en) * | 2002-03-27 | 2003-10-02 | Meares Lawrence G. | System and method of preventing the simulation of a circuit if a change to the circuit topology is detected |
US7152243B2 (en) * | 2002-06-27 | 2006-12-19 | Microsoft Corporation | Providing a secure hardware identifier (HWID) for use in connection with digital rights management (DRM) system |
US7120666B2 (en) * | 2002-10-30 | 2006-10-10 | Riverbed Technology, Inc. | Transaction accelerator for client-server communication systems |
US6667700B1 (en) | 2002-10-30 | 2003-12-23 | Nbt Technology, Inc. | Content-based segmentation scheme for data compression in storage and transmission including hierarchical segment representation |
US8176186B2 (en) | 2002-10-30 | 2012-05-08 | Riverbed Technology, Inc. | Transaction accelerator for client-server communications systems |
US7650416B2 (en) * | 2003-08-12 | 2010-01-19 | Riverbed Technology | Content delivery for client-server protocols with user affinities using connection end-point proxies |
US8364815B2 (en) | 2005-03-18 | 2013-01-29 | Riverbed Technology, Inc. | Reliability and availability of distributed servers |
JP2004171299A (en) * | 2002-11-20 | 2004-06-17 | Fujitsu Ltd | Network relay storage device and network relay method |
US6928526B1 (en) | 2002-12-20 | 2005-08-09 | Datadomain, Inc. | Efficient data storage system |
WO2004070568A2 (en) | 2003-01-31 | 2004-08-19 | Viair, Inc. | Asynchronous real-time retrieval of data |
US20040186859A1 (en) * | 2003-03-20 | 2004-09-23 | Sun Microsystems, Inc. | File access based on file digests |
CA2521814A1 (en) * | 2003-04-07 | 2004-10-28 | Synematics, Inc. | System and method for providing scalable management on commodity routers |
US7304994B2 (en) * | 2003-04-09 | 2007-12-04 | Nec Laboratories America, Inc. | Peer-to-peer system and method with prefix-based distributed hash table |
US9678967B2 (en) * | 2003-05-22 | 2017-06-13 | Callahan Cellular L.L.C. | Information source agent systems and methods for distributed data storage and management using content signatures |
US7424498B1 (en) | 2003-06-30 | 2008-09-09 | Data Domain, Inc. | Probabilistic summary data structure based encoding for garbage collection |
US7451168B1 (en) | 2003-06-30 | 2008-11-11 | Data Domain, Inc. | Incremental garbage collection of data in a secondary storage |
US8280926B2 (en) * | 2003-08-05 | 2012-10-02 | Sepaton, Inc. | Scalable de-duplication mechanism |
US7730222B2 (en) * | 2004-08-24 | 2010-06-01 | Symantec Operating System | Processing storage-related I/O requests using binary tree data structures |
US7577806B2 (en) * | 2003-09-23 | 2009-08-18 | Symantec Operating Corporation | Systems and methods for time dependent data storage and recovery |
US7904428B2 (en) * | 2003-09-23 | 2011-03-08 | Symantec Corporation | Methods and apparatus for recording write requests directed to a data store |
US7239581B2 (en) * | 2004-08-24 | 2007-07-03 | Symantec Operating Corporation | Systems and methods for synchronizing the internal clocks of a plurality of processor modules |
US7577807B2 (en) * | 2003-09-23 | 2009-08-18 | Symantec Operating Corporation | Methods and devices for restoring a portion of a data store |
US7631120B2 (en) * | 2004-08-24 | 2009-12-08 | Symantec Operating Corporation | Methods and apparatus for optimally selecting a storage buffer for the storage of data |
US7827362B2 (en) * | 2004-08-24 | 2010-11-02 | Symantec Corporation | Systems, apparatus, and methods for processing I/O requests |
US7725760B2 (en) * | 2003-09-23 | 2010-05-25 | Symantec Operating Corporation | Data storage system |
US7296008B2 (en) * | 2004-08-24 | 2007-11-13 | Symantec Operating Corporation | Generation and use of a time map for accessing a prior image of a storage device |
US7409587B2 (en) * | 2004-08-24 | 2008-08-05 | Symantec Operating Corporation | Recovering from storage transaction failures using checkpoints |
US7287133B2 (en) * | 2004-08-24 | 2007-10-23 | Symantec Operating Corporation | Systems and methods for providing a modification history for a location within a data store |
US7991748B2 (en) * | 2003-09-23 | 2011-08-02 | Symantec Corporation | Virtual data store creation and use |
AU2003279999A1 (en) * | 2003-10-21 | 2005-06-08 | Nielsen Media Research, Inc. | Methods and apparatus for fusing databases |
US7613775B2 (en) * | 2003-11-25 | 2009-11-03 | Freescale Semiconductor, Inc. | Network message filtering using hashing and pattern matching |
US8244677B2 (en) * | 2004-01-23 | 2012-08-14 | Elad Baron | Focal point compression method and apparatus |
US7574418B1 (en) | 2004-02-13 | 2009-08-11 | Data Domain, Inc. | Method and apparatus for storing composite data streams |
JP4521865B2 (en) * | 2004-02-27 | 2010-08-11 | 株式会社日立製作所 | Storage system, computer system, or storage area attribute setting method |
US7272654B1 (en) | 2004-03-04 | 2007-09-18 | Sandbox Networks, Inc. | Virtualizing network-attached-storage (NAS) with a compact table that stores lossy hashes of file names and parent handles rather than full names |
GB0411053D0 (en) * | 2004-05-18 | 2004-06-23 | Ricardo Uk Ltd | Data processing |
US20060031230A1 (en) * | 2004-07-21 | 2006-02-09 | Kumar Sinha M | Data storage systems |
JP4530763B2 (en) * | 2004-08-23 | 2010-08-25 | 富士機械製造株式会社 | Part data creation method and part data creation apparatus |
US7523098B2 (en) * | 2004-09-15 | 2009-04-21 | International Business Machines Corporation | Systems and methods for efficient data searching, storage and reduction |
US7689633B1 (en) * | 2004-09-15 | 2010-03-30 | Data Domain, Inc. | Network file system-based data storage system |
US8725705B2 (en) * | 2004-09-15 | 2014-05-13 | International Business Machines Corporation | Systems and methods for searching of storage data with reduced bandwidth requirements |
KR100714682B1 (en) * | 2004-12-02 | 2007-05-04 | 삼성전자주식회사 | File system path processing device and method thereof |
US7844691B2 (en) * | 2004-12-30 | 2010-11-30 | Xstor Systems, Inc. | Scalable distributed storage and delivery |
US9275052B2 (en) | 2005-01-19 | 2016-03-01 | Amazon Technologies, Inc. | Providing annotations of a digital work |
US8201021B1 (en) * | 2005-01-31 | 2012-06-12 | Symantec Corporation | Method and apparatus for efficiently creating backup files with less redundancy |
US7814129B2 (en) * | 2005-03-11 | 2010-10-12 | Ross Neil Williams | Method and apparatus for storing data with reduced redundancy using data clusters |
EP1866775B1 (en) * | 2005-03-11 | 2016-04-20 | Rocksoft Limited | Method for indexing in a reduced-redundancy storage system |
EP1866774A4 (en) * | 2005-03-11 | 2010-04-14 | Rocksoft Ltd | Method for storing data with reduced redundancy using data clusters |
US8356021B2 (en) * | 2005-03-11 | 2013-01-15 | Ross Neil Williams | Method and apparatus for indexing in a reduced-redundancy storage system |
US8051252B2 (en) | 2005-03-11 | 2011-11-01 | Ross Neil Williams | Method and apparatus for detecting the presence of subblocks in a reduced-redundancy storage system |
WO2006094367A1 (en) * | 2005-03-11 | 2006-09-14 | Rocksoft Limited | Method for detecting the presence of subblocks in a reduced-redundancy storage system |
US20060248194A1 (en) | 2005-03-18 | 2006-11-02 | Riverbed Technology, Inc. | Connection forwarding |
US7984018B2 (en) * | 2005-04-18 | 2011-07-19 | Microsoft Corporation | Efficient point-to-multipoint data reconciliation |
GB0510878D0 (en) * | 2005-05-27 | 2005-07-06 | Qinetiq Ltd | Digital evidence bag |
US8984636B2 (en) | 2005-07-29 | 2015-03-17 | Bit9, Inc. | Content extractor and analysis system |
US8272058B2 (en) | 2005-07-29 | 2012-09-18 | Bit 9, Inc. | Centralized timed analysis in a network security system |
US7895651B2 (en) | 2005-07-29 | 2011-02-22 | Bit 9, Inc. | Content tracking in a network security system |
WO2007016787A2 (en) * | 2005-08-09 | 2007-02-15 | Nexsan Technologies Canada Inc. | Data archiving system |
US7584338B1 (en) * | 2005-09-27 | 2009-09-01 | Data Domain, Inc. | Replication of deduplicated storage system |
US20070083378A1 (en) * | 2005-10-11 | 2007-04-12 | Microsoft Corporation | Secure application programming interface |
CN103927238B (en) * | 2005-10-14 | 2017-04-12 | 塞门铁克操作公司 | Technique For Timeline Compression In Data Store |
GB2431741B (en) * | 2005-10-27 | 2010-11-03 | Hewlett Packard Development Co | A method of digitally signing data and a data repository storing digitally signed data |
JP2007140887A (en) * | 2005-11-18 | 2007-06-07 | Hitachi Ltd | Storage system, disk array device, method of presenting volume, and method of verifying data consistency |
US20080276171A1 (en) * | 2005-11-29 | 2008-11-06 | Itzchak Sabo | Filing System |
US7457934B2 (en) * | 2006-03-22 | 2008-11-25 | Hitachi, Ltd. | Method and apparatus for reducing the amount of data in a storage system |
CA2648428C (en) * | 2006-04-07 | 2017-11-21 | Data Storage Group | Data compression and storage techniques |
US8832045B2 (en) | 2006-04-07 | 2014-09-09 | Data Storage Group, Inc. | Data compression and storage techniques |
US7814070B1 (en) | 2006-04-20 | 2010-10-12 | Datascout, Inc. | Surrogate hashing |
US7991206B1 (en) | 2007-07-02 | 2011-08-02 | Datascout, Inc. | Surrogate heuristic identification |
US8463000B1 (en) | 2007-07-02 | 2013-06-11 | Pinehill Technology, Llc | Content identification based on a search of a fingerprint database |
US8156132B1 (en) | 2007-07-02 | 2012-04-10 | Pinehill Technology, Llc | Systems for comparing image fingerprints |
US7840540B2 (en) | 2006-04-20 | 2010-11-23 | Datascout, Inc. | Surrogate hashing |
US20070250810A1 (en) * | 2006-04-20 | 2007-10-25 | Tittizer Abigail A | Systems and methods for managing data associated with computer code |
US9020964B1 (en) | 2006-04-20 | 2015-04-28 | Pinehill Technology, Llc | Generation of fingerprints for multimedia content based on vectors and histograms |
US7774385B1 (en) | 2007-07-02 | 2010-08-10 | Datascout, Inc. | Techniques for providing a surrogate heuristic identification interface |
US7801868B1 (en) | 2006-04-20 | 2010-09-21 | Datascout, Inc. | Surrogate hashing |
US8549022B1 (en) | 2007-07-02 | 2013-10-01 | Datascout, Inc. | Fingerprint generation of multimedia content based on a trigger point with the multimedia content |
KR101381551B1 (en) | 2006-05-05 | 2014-04-11 | 하이버 인크 | Group based complete and incremental computer file backup system, process and apparatus |
US8463843B2 (en) * | 2006-05-26 | 2013-06-11 | Riverbed Technology, Inc. | Throttling of predictive ACKs in an accelerated network communication system |
WO2007149687A2 (en) | 2006-05-30 | 2007-12-27 | Riverbed Technology, Inc. | Selecting proxies from among autodiscovered proxies |
US7921077B2 (en) | 2006-06-29 | 2011-04-05 | Netapp, Inc. | System and method for managing data deduplication of storage systems utilizing persistent consistency point images |
US7640262B1 (en) | 2006-06-30 | 2009-12-29 | Emc Corporation | Positional allocation |
US7673099B1 (en) | 2006-06-30 | 2010-03-02 | Emc Corporation | Affinity caching |
US7930559B1 (en) | 2006-06-30 | 2011-04-19 | Emc Corporation | Decoupled data stream and access structures |
US7720892B1 (en) | 2006-06-30 | 2010-05-18 | Emc Corporation | Bulk updates and tape synchronization |
US7725704B1 (en) | 2006-09-22 | 2010-05-25 | Emc Corporation | Techniques for performing a prioritized data restoration operation |
US8082231B1 (en) | 2006-09-22 | 2011-12-20 | Emc Corporation | Techniques using identifiers and signatures with data operations |
US7685171B1 (en) | 2006-09-22 | 2010-03-23 | Emc Corporation | Techniques for performing a restoration operation using device scanning |
US8069321B2 (en) | 2006-11-13 | 2011-11-29 | I365 Inc. | Secondary pools |
DE102006055964A1 (en) * | 2006-11-24 | 2008-05-29 | Bdt-Solutions Gmbh | Method and device for data backup |
JP2008146601A (en) * | 2006-12-13 | 2008-06-26 | Canon Inc | Information processor and information processing method |
US7733910B2 (en) * | 2006-12-29 | 2010-06-08 | Riverbed Technology, Inc. | Data segmentation using shift-varying predicate function fingerprinting |
US7921461B1 (en) * | 2007-01-16 | 2011-04-05 | Kaspersky Lab, Zao | System and method for rootkit detection and cure |
US7853750B2 (en) * | 2007-01-30 | 2010-12-14 | Netapp, Inc. | Method and an apparatus to store data patterns |
US8533310B2 (en) * | 2007-03-09 | 2013-09-10 | Riverbed Technology, Inc. | Method and apparatus for acceleration by prefetching associated objects |
WO2008138008A1 (en) * | 2007-05-08 | 2008-11-13 | Riverbed Technology, Inc | A hybrid segment-oriented file server and wan accelerator |
US8341210B1 (en) | 2007-05-21 | 2012-12-25 | Amazon Technologies, Inc. | Delivery of items for consumption by a user device |
US8775368B1 (en) | 2007-06-27 | 2014-07-08 | Emc Corporation | Fine grained tiered storage with thin provisioning |
US7949637B1 (en) | 2007-06-27 | 2011-05-24 | Emc Corporation | Storage management for fine grained tiered storage with thin provisioning |
US8028106B2 (en) | 2007-07-06 | 2011-09-27 | Proster Systems, Inc. | Hardware acceleration of commonality factoring with removable media |
US8095509B2 (en) * | 2007-08-11 | 2012-01-10 | Novell, Inc. | Techniques for retaining security restrictions with file versioning |
US7822939B1 (en) | 2007-09-25 | 2010-10-26 | Emc Corporation | Data de-duplication using thin provisioning |
US7870105B2 (en) * | 2007-11-20 | 2011-01-11 | Hitachi, Ltd. | Methods and apparatus for deduplication in storage system |
US8209334B1 (en) | 2007-12-28 | 2012-06-26 | Don Doerner | Method to direct data to a specific one of several repositories |
US8825971B1 (en) * | 2007-12-31 | 2014-09-02 | Emc Corporation | Age-out selection in hash caches |
US7949630B1 (en) * | 2007-12-31 | 2011-05-24 | Emc Corporation | Storage of data addresses with hashes in backup systems |
US8452736B2 (en) | 2008-03-05 | 2013-05-28 | Ca, Inc. | File change detection |
US8620877B2 (en) * | 2008-04-30 | 2013-12-31 | International Business Machines Corporation | Tunable data fingerprinting for optimizing data deduplication |
US9063947B2 (en) * | 2008-08-18 | 2015-06-23 | Hewlett-Packard Development Company, L.P. | Detecting duplicative hierarchical sets of files |
US8311985B2 (en) * | 2008-09-16 | 2012-11-13 | Quest Software, Inc. | Remote backup and restore system and method |
US8452731B2 (en) * | 2008-09-25 | 2013-05-28 | Quest Software, Inc. | Remote backup and restore |
US10089308B1 (en) | 2008-09-30 | 2018-10-02 | EMC IP Holding Company LLC | Method for using redundant data elimination to accelerate storage system scanning |
JP5468620B2 (en) * | 2008-12-18 | 2014-04-09 | コピウン,インク. | Method and apparatus for content-aware data partitioning and data deduplication |
US8566549B1 (en) | 2008-12-31 | 2013-10-22 | Emc Corporation | Synchronizing performance requirements across multiple storage platforms |
US8209290B1 (en) * | 2009-03-11 | 2012-06-26 | Symantec Corporation | Generic granular restore of application data from a volume image backup |
US8346736B2 (en) * | 2009-03-16 | 2013-01-01 | International Business Machines Corporation | Apparatus and method to deduplicate data |
US9160611B2 (en) * | 2009-04-22 | 2015-10-13 | Webroot Inc. | System and method for performing longest common prefix strings searches |
TWI389001B (en) * | 2009-06-01 | 2013-03-11 | Mstar Semiconductor Inc | File system and file system transforming method |
US20110055471A1 (en) * | 2009-08-28 | 2011-03-03 | Jonathan Thatcher | Apparatus, system, and method for improved data deduplication |
US8924439B2 (en) * | 2009-09-01 | 2014-12-30 | Verizon Patent And Licensing Inc. | System for and method of partial file hashing |
JP5595701B2 (en) * | 2009-09-16 | 2014-09-24 | 株式会社日立製作所 | File management method and storage system |
US8495312B2 (en) * | 2010-01-25 | 2013-07-23 | Sepaton, Inc. | System and method for identifying locations within data |
WO2011109558A1 (en) * | 2010-03-02 | 2011-09-09 | Renew Data Corp. | System and method for creating a de-duplicated data set and preserving its metadata |
US20120324182A1 (en) * | 2010-03-04 | 2012-12-20 | Nec Software Tohoku, Ltd. | Storage device |
EP2545472B1 (en) * | 2010-03-12 | 2020-05-06 | BlackBerry Limited | Distributed catalog, data store, and indexing |
CN102985911B (en) | 2010-03-16 | 2016-07-06 | 科派恩股份有限公司 | Telescopic in height and distributed data de-duplication |
US8370354B2 (en) * | 2010-06-30 | 2013-02-05 | International Business Machines Corporation | Acceleration of legacy to service oriented (L2SOA) architecture renovations |
WO2012027472A2 (en) | 2010-08-24 | 2012-03-01 | Copiun, Inc. | Constant access gateway and de-duplicated data cache server |
US9235593B2 (en) * | 2010-09-30 | 2016-01-12 | Emc Corporation | Transmitting filesystem changes over a network |
US8688651B2 (en) | 2011-01-25 | 2014-04-01 | Sepaton, Inc. | Dynamic deduplication |
CN102436402B (en) * | 2011-03-29 | 2014-12-10 | 奇智软件(北京)有限公司 | Module repairing method in software and software equipment |
US20120324236A1 (en) * | 2011-06-16 | 2012-12-20 | Microsoft Corporation | Trusted Snapshot Generation |
CA2890596C (en) | 2011-11-07 | 2017-01-03 | Nexgen Storage, Inc. | Primary data storage system with deduplication |
US20130232124A1 (en) * | 2012-03-05 | 2013-09-05 | Blaine D. Gaither | Deduplicating a file system |
JP2013222230A (en) | 2012-04-13 | 2013-10-28 | Hitachi-Lg Data Storage Inc | Information processing system |
US8984028B2 (en) | 2012-05-29 | 2015-03-17 | Recommind, Inc. | Systems and methods for storing data and eliminating redundancy |
US9405684B1 (en) | 2012-09-28 | 2016-08-02 | Emc Corporation | System and method for cache management |
US9766832B2 (en) | 2013-03-15 | 2017-09-19 | Hitachi Data Systems Corporation | Systems and methods of locating redundant data using patterns of matching fingerprints |
ES2731775T3 (en) | 2013-04-18 | 2019-11-19 | Risoftdev Inc | Data encryption system and procedures |
US9256611B2 (en) | 2013-06-06 | 2016-02-09 | Sepaton, Inc. | System and method for multi-scale navigation of data |
US9405479B1 (en) | 2013-08-26 | 2016-08-02 | Western Digital Technologies, Inc. | Faster file compression using sliding compression window and backward compound pointers |
US9678973B2 (en) | 2013-10-15 | 2017-06-13 | Hitachi Data Systems Corporation | Multi-node hybrid deduplication |
US10140136B2 (en) | 2013-11-07 | 2018-11-27 | Datrium, linc. | Distributed virtual array data storage system and method |
US10180948B2 (en) | 2013-11-07 | 2019-01-15 | Datrium, Inc. | Data storage with a distributed virtual array |
US9417955B2 (en) | 2014-08-21 | 2016-08-16 | Datrium, Inc. | Data reconstruction in distributed data storage system with key-based addressing |
US10514982B2 (en) | 2014-08-21 | 2019-12-24 | Datrium, Inc. | Alternate storage arrangement in a distributed data storage system with key-based addressing |
US9639268B2 (en) | 2014-08-21 | 2017-05-02 | Datrium, Inc. | Distributed data storage system with key-based addressing |
US9152500B1 (en) * | 2014-09-22 | 2015-10-06 | Storagecraft Technology Corporation | Hash collision recovery in a deduplication vault |
US9732593B2 (en) | 2014-11-05 | 2017-08-15 | Saudi Arabian Oil Company | Systems, methods, and computer medium to optimize storage for hydrocarbon reservoir simulation |
US9659047B2 (en) * | 2014-12-03 | 2017-05-23 | Netapp, Inc. | Data deduplication utilizing extent ID database |
US9639715B2 (en) | 2015-04-27 | 2017-05-02 | Microsoft Technology Licensing, Llc | Protecting user identifiable information in the transfer of telemetry data |
US10540504B2 (en) | 2015-05-12 | 2020-01-21 | Datrium, Inc. | Distributed data method for encrypting data |
US9866619B2 (en) | 2015-06-12 | 2018-01-09 | International Business Machines Corporation | Transmission of hierarchical data files based on content selection |
US11386067B2 (en) | 2015-12-15 | 2022-07-12 | Red Hat, Inc. | Data integrity checking in a distributed filesystem using object versioning |
WO2017182062A1 (en) | 2016-04-19 | 2017-10-26 | Huawei Technologies Co., Ltd. | Concurrent segmentation using vector processing |
JP6420489B2 (en) | 2016-04-19 | 2018-11-07 | 華為技術有限公司Huawei Technologies Co.,Ltd. | Vector processing for segmented hash calculation |
US10361997B2 (en) | 2016-12-29 | 2019-07-23 | Riverbed Technology, Inc. | Auto discovery between proxies in an IPv6 network |
KR102441299B1 (en) * | 2017-11-27 | 2022-09-08 | 스노우플레이크 인코포레이티드 | Batch data collection into database system |
Citations (14)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5754844A (en) * | 1995-12-14 | 1998-05-19 | Sun Microsystems, Inc. | Method and system for accessing chunks of data using matching of an access tab and hashing code to generate a suggested storage location |
US5915024A (en) * | 1996-06-18 | 1999-06-22 | Kabushiki Kaisha Toshiba | Electronic signature addition method, electronic signature verification method, and system and computer program product using these methods |
US5978791A (en) * | 1995-04-11 | 1999-11-02 | Kinetech, Inc. | Data processing system using substantially unique identifiers to identify data items, whereby identical data items have the same identifiers |
US5990810A (en) * | 1995-02-17 | 1999-11-23 | Williams; Ross Neil | Method for partitioning a block of data into subblocks and for storing and communcating such subblocks |
US6170744B1 (en) * | 1998-09-24 | 2001-01-09 | Payformance Corporation | Self-authenticating negotiable documents |
US6219423B1 (en) * | 1995-12-29 | 2001-04-17 | Intel Corporation | System and method for digitally signing a digital agreement between remotely located nodes |
US6377907B1 (en) * | 1999-11-17 | 2002-04-23 | Mci Worldcom, Inc. | System and method for collating UNIX performance metrics |
US20020194209A1 (en) * | 2001-03-21 | 2002-12-19 | Bolosky William J. | On-disk file format for a serverless distributed file system |
US20020194484A1 (en) * | 2001-03-21 | 2002-12-19 | Bolosky William J. | On-disk file format for serverless distributed file system with signed manifest of file modifications |
US6535894B1 (en) * | 2000-06-01 | 2003-03-18 | Sun Microsystems, Inc. | Apparatus and method for incremental updating of archive files |
US6553494B1 (en) * | 1999-07-21 | 2003-04-22 | Sensar, Inc. | Method and apparatus for applying and verifying a biometric-based digital signature to an electronic document |
US6671358B1 (en) * | 2001-04-25 | 2003-12-30 | Universal Identity Technologies, Inc. | Method and system for rewarding use of a universal identifier, and/or conducting a financial transaction |
US6704730B2 (en) * | 2000-02-18 | 2004-03-09 | Avamar Technologies, Inc. | Hash file system and method for use in a commonality factoring system |
US6826690B1 (en) * | 1999-11-08 | 2004-11-30 | International Business Machines Corporation | Using device certificates for automated authentication of communicating devices |
Family Cites Families (91)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US3668647A (en) | 1970-06-12 | 1972-06-06 | Ibm | File access system |
US4215402A (en) | 1978-10-23 | 1980-07-29 | International Business Machines Corporation | Hash index table hash generator apparatus |
US4404676A (en) * | 1981-03-30 | 1983-09-13 | Pioneer Electric Corporation | Partitioning method and apparatus using data-dependent boundary-marking code words |
US4887235A (en) | 1982-12-17 | 1989-12-12 | Symbolics, Inc. | Symbolic language data processing system |
US4649479A (en) | 1985-02-28 | 1987-03-10 | International Business Machines Corp. | Device driver and adapter binding technique |
US4901223A (en) | 1986-04-30 | 1990-02-13 | International Business Machines Corporation | Method and apparatus for application software control of echo response |
US4761785B1 (en) | 1986-06-12 | 1996-03-12 | Ibm | Parity spreading to enhance storage access |
US4887204A (en) | 1987-02-13 | 1989-12-12 | International Business Machines Corporation | System and method for accessing remote files in a distributed networking environment |
US4897781A (en) | 1987-02-13 | 1990-01-30 | International Business Machines Corporation | System and method for using cached data at a local node after re-opening a file at a remote node in a distributed networking environment |
US5005122A (en) | 1987-09-08 | 1991-04-02 | Digital Equipment Corporation | Arrangement with cooperating management server node and network service node |
US5109515A (en) | 1987-09-28 | 1992-04-28 | At&T Bell Laboratories | User and application program transparent resource sharing multiple computer interface architecture with kernel process level transfer of user requested services |
AU601328B2 (en) | 1988-05-26 | 1990-09-06 | Digital Equipment Corporation | Temporary state preservation for a distributed file service |
US5146568A (en) | 1988-09-06 | 1992-09-08 | Digital Equipment Corporation | Remote bootstrapping a node over communication link by initially requesting remote storage access program which emulates local disk to load other programs |
US4982324A (en) | 1988-12-19 | 1991-01-01 | International Business Machines Corporation | Method of and system for using device drivers to couple the communication and data storage of remote computer systems |
US5016009A (en) | 1989-01-13 | 1991-05-14 | Stac, Inc. | Data compression apparatus and method |
US5126739A (en) | 1989-01-13 | 1992-06-30 | Stac Electronics | Data compression apparatus and method |
US5089958A (en) | 1989-01-23 | 1992-02-18 | Vortex Systems, Inc. | Fault tolerant computer backup system |
US5018060A (en) | 1989-01-26 | 1991-05-21 | Ibm Corporation | Allocating data storage space of peripheral data storage devices using implied allocation based on user parameters |
US4929946A (en) * | 1989-02-09 | 1990-05-29 | Storage Technology Corporation | Adaptive data compression apparatus including run length encoding for a tape drive system |
JPH0833958B2 (en) | 1989-05-30 | 1996-03-29 | 沖電気工業株式会社 | Customer information processing system |
US5239637A (en) | 1989-06-30 | 1993-08-24 | Digital Equipment Corporation | Digital data management system for maintaining consistency of data in a shadow set |
US5454099A (en) | 1989-07-25 | 1995-09-26 | International Business Machines Corporation | CPU implemented method for backing up modified data sets in non-volatile store for recovery in the event of CPU failure |
US5133065A (en) | 1989-07-27 | 1992-07-21 | Personal Computer Peripherals Corporation | Backup computer program for networks |
US5163148A (en) | 1989-08-11 | 1992-11-10 | Digital Equipment Corporation | File backup system for producing a backup copy of a file which may be updated during backup |
US5163131A (en) | 1989-09-08 | 1992-11-10 | Auspex Systems, Inc. | Parallel i/o network file server architecture |
US5276867A (en) | 1989-12-19 | 1994-01-04 | Epoch Systems, Inc. | Digital data storage system with improved data migration |
US5276860A (en) | 1989-12-19 | 1994-01-04 | Epoch Systems, Inc. | Digital data processor with improved backup storage |
US5218695A (en) | 1990-02-05 | 1993-06-08 | Epoch Systems, Inc. | File server system having high-speed write execution |
US5307481A (en) | 1990-02-28 | 1994-04-26 | Hitachi, Ltd. | Highly reliable online system |
US5479654A (en) | 1990-04-26 | 1995-12-26 | Squibb Data Systems, Inc. | Apparatus and method for reconstructing a file from a difference signature and an original file |
EP0465018B1 (en) | 1990-06-29 | 1997-05-14 | Oracle Corporation | Method and apparatus for optimizing undo log usage |
US5239647A (en) | 1990-09-07 | 1993-08-24 | International Business Machines Corporation | Data storage hierarchy with shared storage level |
US5317728A (en) | 1990-09-07 | 1994-05-31 | International Business Machines Corporation | Storage management of a first file system using a second file system containing surrogate files and catalog management information |
US5210866A (en) | 1990-09-12 | 1993-05-11 | Storage Technology Corporation | Incremental disk backup system for a dynamically mapped data storage subsystem |
US5162986A (en) | 1990-10-19 | 1992-11-10 | Allen-Bradley Company, Inc. | Remote downloading and uploading of motion control program information to and from a motion control I/O module in a programmable controller |
US5155835A (en) | 1990-11-19 | 1992-10-13 | Storage Technology Corporation | Multilevel, hierarchical, dynamically mapped data storage subsystem |
US5278838A (en) | 1991-06-18 | 1994-01-11 | Ibm Corp. | Recovery from errors in a redundant array of disk drives |
US5239659A (en) | 1991-06-19 | 1993-08-24 | Storage Technology Corporation | Phantom duplex copy group apparatus for a disk drive array data storge subsystem |
US5347653A (en) | 1991-06-28 | 1994-09-13 | Digital Equipment Corporation | System for reconstructing prior versions of indexes using records indicating changes between successive versions of the indexes |
US5566330A (en) * | 1991-08-20 | 1996-10-15 | Powersoft Corporation | Method for forming a reusable and modifiable database interface object |
US5305389A (en) | 1991-08-30 | 1994-04-19 | Digital Equipment Corporation | Predictive cache system |
US5140321A (en) | 1991-09-04 | 1992-08-18 | Prime Computer, Inc. | Data compression/decompression method and apparatus |
US5325505A (en) | 1991-09-04 | 1994-06-28 | Storage Technology Corporation | Intelligent storage manager for data storage apparatus having simulation capability |
US5367698A (en) | 1991-10-31 | 1994-11-22 | Epoch Systems, Inc. | Network file migration system |
EP0541281B1 (en) | 1991-11-04 | 1998-04-29 | Commvault Systems, Inc. | Incremental-computer-file backup using signatures |
US5802264A (en) | 1991-11-15 | 1998-09-01 | Fujitsu Limited | Background data reconstruction in a storage device array system |
US5369758A (en) | 1991-11-15 | 1994-11-29 | Fujitsu Limited | Checking for proper locations of storage devices in a storage array |
US5452454A (en) | 1991-12-10 | 1995-09-19 | Digital Equipment Corporation | Generic remote boot for networked workstations by creating local bootable code image |
US5367637A (en) | 1992-03-26 | 1994-11-22 | International Business Machines Corporation | Self-tuning virtual storage management for dedicated real-time computer system |
US5263154A (en) | 1992-04-20 | 1993-11-16 | International Business Machines Corporation | Method and system for incremental time zero backup copying of data |
US5448718A (en) | 1992-04-20 | 1995-09-05 | International Business Machines Corporation | Method and system for time zero backup session security |
US5586322A (en) | 1992-06-11 | 1996-12-17 | Beck; Robert E. | Workgroup organized network manager with workstation comparison system |
US5403639A (en) | 1992-09-02 | 1995-04-04 | Storage Technology Corporation | File server having snapshot application data groups |
US5406279A (en) | 1992-09-02 | 1995-04-11 | Cirrus Logic, Inc. | General purpose, hash-based technique for single-pass lossless data compression |
US5497483A (en) | 1992-09-23 | 1996-03-05 | International Business Machines Corporation | Method and system for track transfer control during concurrent copy operations in a data processing storage subsystem |
US5530855A (en) | 1992-10-13 | 1996-06-25 | International Business Machines Corporation | Replicating a database by the sequential application of hierarchically sorted log records |
US5404508A (en) | 1992-12-03 | 1995-04-04 | Unisys Corporation | Data base backup and recovery system and method |
US5487160A (en) | 1992-12-04 | 1996-01-23 | At&T Global Information Solutions Company | Concurrent image backup for disk storage system |
US5404527A (en) | 1992-12-31 | 1995-04-04 | Unisys Corporation | System and method for remote program load |
US5544320A (en) | 1993-01-08 | 1996-08-06 | Konrad; Allan M. | Remote information service access system based on a client-server-service model |
US5659747A (en) | 1993-04-22 | 1997-08-19 | Microsoft Corporation | Multiple level undo/redo mechanism |
WO1995001599A1 (en) | 1993-07-01 | 1995-01-12 | Legent Corporation | System and method for distributed storage management on networked computer systems |
US5452440A (en) | 1993-07-16 | 1995-09-19 | Zitel Corporation | Method and structure for evaluating and enhancing the performance of cache memory systems |
US5521597A (en) * | 1993-08-02 | 1996-05-28 | Mircosoft Corporation | Data compression for network transport |
US5515502A (en) | 1993-09-30 | 1996-05-07 | Sybase, Inc. | Data backup system with methods for stripe affinity backup to multiple archive devices |
US5771354A (en) | 1993-11-04 | 1998-06-23 | Crawford; Christopher M. | Internet online backup system provides remote storage for customers using IDs and passwords which were interactively established when signing up for backup services |
JPH07234921A (en) * | 1993-12-28 | 1995-09-05 | Nikon Corp | Loading/unloading mechanism for recording medium |
US5835953A (en) | 1994-10-13 | 1998-11-10 | Vinca Corporation | Backup system that takes a snapshot of the locations in a mass storage device that has been identified for updating prior to updating |
US5574906A (en) | 1994-10-24 | 1996-11-12 | International Business Machines Corporation | System and method for reducing storage requirement in backup subsystems utilizing segmented compression and differencing |
US5659743A (en) | 1994-12-05 | 1997-08-19 | Legent Corporation | Method and apparatus for a pattern based spaced management system |
US5513314A (en) | 1995-01-27 | 1996-04-30 | Auspex Systems, Inc. | Fault tolerant NFS server system and mirroring protocol |
US5604862A (en) | 1995-03-14 | 1997-02-18 | Network Integrity, Inc. | Continuously-snapshotted protection of computer files |
US5802297A (en) | 1995-07-03 | 1998-09-01 | Sun Microsystems, Inc. | Client-server computer system and method utilizing a local client disk drive as a data cache |
JPH0937788A (en) * | 1995-07-31 | 1997-02-10 | Nitto Chem Ind Co Ltd | New nitrilase gane |
US5742811A (en) * | 1995-10-10 | 1998-04-21 | International Business Machines Corporation | Method and system for mining generalized sequential patterns in a large database |
US5778395A (en) | 1995-10-23 | 1998-07-07 | Stac, Inc. | System for backing up files from disk volumes on multiple nodes of a computer network |
US5933104A (en) * | 1995-11-22 | 1999-08-03 | Microsoft Corporation | Method and system for compression and decompression using variable-sized offset and length fields |
US5765173A (en) | 1996-01-11 | 1998-06-09 | Connected Corporation | High performance backup via selective file saving which can perform incremental backups and exclude files and uses a changed block signature list |
DE19602848A1 (en) * | 1996-01-26 | 1997-07-31 | Basf Ag | Process for the production of pigments |
US5831558A (en) | 1996-06-17 | 1998-11-03 | Digital Equipment Corporation | Method of compressing and decompressing data in a computer system by encoding data using a data dictionary |
US5850565A (en) | 1996-08-26 | 1998-12-15 | Novell, Inc. | Data compression method and apparatus |
US5794254A (en) | 1996-12-03 | 1998-08-11 | Fairbanks Systems Group | Incremental computer file backup using a two-step comparison of first two characters in the block and a signature with pre-stored character and signature sets |
JPH10171350A (en) * | 1996-12-10 | 1998-06-26 | Hitachi Ltd | Method and device for generating hash value |
US6044220A (en) * | 1997-02-25 | 2000-03-28 | Motorola, Inc. | Method and apparatus for operating a data processor to execute software written using a foreign instruction set |
US6016553A (en) | 1997-09-05 | 2000-01-18 | Wild File, Inc. | Method, software and apparatus for saving, using and recovering data |
JP3730385B2 (en) * | 1997-12-05 | 2006-01-05 | 株式会社東芝 | Data compression device |
US6029168A (en) | 1998-01-23 | 2000-02-22 | Tricord Systems, Inc. | Decentralized file mapping in a striped network file system in a distributed computing environment |
US6122754A (en) | 1998-05-22 | 2000-09-19 | International Business Machines Corporation | Method and system for data recovery using a distributed and scalable data structure |
US6307487B1 (en) | 1998-09-23 | 2001-10-23 | Digital Fountain, Inc. | Information additive code generator and decoder for communication systems |
US6320520B1 (en) | 1998-09-23 | 2001-11-20 | Digital Fountain | Information additive group code generator and decoder for communications systems |
JP3956638B2 (en) * | 2000-11-21 | 2007-08-08 | 日本精工株式会社 | Grease lubricated rolling bearing |
-
2001
- 2001-02-05 US US09/777,150 patent/US6704730B2/en not_active Expired - Lifetime
-
2004
- 2004-01-14 US US10/757,753 patent/US20040148306A1/en not_active Abandoned
Patent Citations (14)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5990810A (en) * | 1995-02-17 | 1999-11-23 | Williams; Ross Neil | Method for partitioning a block of data into subblocks and for storing and communcating such subblocks |
US5978791A (en) * | 1995-04-11 | 1999-11-02 | Kinetech, Inc. | Data processing system using substantially unique identifiers to identify data items, whereby identical data items have the same identifiers |
US5754844A (en) * | 1995-12-14 | 1998-05-19 | Sun Microsystems, Inc. | Method and system for accessing chunks of data using matching of an access tab and hashing code to generate a suggested storage location |
US6219423B1 (en) * | 1995-12-29 | 2001-04-17 | Intel Corporation | System and method for digitally signing a digital agreement between remotely located nodes |
US5915024A (en) * | 1996-06-18 | 1999-06-22 | Kabushiki Kaisha Toshiba | Electronic signature addition method, electronic signature verification method, and system and computer program product using these methods |
US6170744B1 (en) * | 1998-09-24 | 2001-01-09 | Payformance Corporation | Self-authenticating negotiable documents |
US6553494B1 (en) * | 1999-07-21 | 2003-04-22 | Sensar, Inc. | Method and apparatus for applying and verifying a biometric-based digital signature to an electronic document |
US6826690B1 (en) * | 1999-11-08 | 2004-11-30 | International Business Machines Corporation | Using device certificates for automated authentication of communicating devices |
US6377907B1 (en) * | 1999-11-17 | 2002-04-23 | Mci Worldcom, Inc. | System and method for collating UNIX performance metrics |
US6704730B2 (en) * | 2000-02-18 | 2004-03-09 | Avamar Technologies, Inc. | Hash file system and method for use in a commonality factoring system |
US6535894B1 (en) * | 2000-06-01 | 2003-03-18 | Sun Microsystems, Inc. | Apparatus and method for incremental updating of archive files |
US20020194484A1 (en) * | 2001-03-21 | 2002-12-19 | Bolosky William J. | On-disk file format for serverless distributed file system with signed manifest of file modifications |
US20020194209A1 (en) * | 2001-03-21 | 2002-12-19 | Bolosky William J. | On-disk file format for a serverless distributed file system |
US6671358B1 (en) * | 2001-04-25 | 2003-12-30 | Universal Identity Technologies, Inc. | Method and system for rewarding use of a universal identifier, and/or conducting a financial transaction |
Cited By (244)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20050120137A1 (en) * | 2000-02-18 | 2005-06-02 | Moulton Gregory H. | System and method for intelligent, globally distributed network storage |
US20010034795A1 (en) * | 2000-02-18 | 2001-10-25 | Moulton Gregory Hagan | System and method for intelligent, globally distributed network storage |
US20040225655A1 (en) * | 2000-11-06 | 2004-11-11 | Moulton Gregory Hagan | System and method for unorchestrated determination of data sequences using sticky factoring to determine breakpoints in digital sequences |
US7272602B2 (en) * | 2000-11-06 | 2007-09-18 | Emc Corporation | System and method for unorchestrated determination of data sequences using sticky byte factoring to determine breakpoints in digital sequences |
US20030182291A1 (en) * | 2002-03-20 | 2003-09-25 | Sreenath Kurupati | Method and data structure for a low memory overhead database |
US7467151B2 (en) | 2002-03-20 | 2008-12-16 | Intel Corporation | Method and data structure for a low memory overhead database |
US7058642B2 (en) * | 2002-03-20 | 2006-06-06 | Intel Corporation | Method and data structure for a low memory overhead database |
US20060122989A1 (en) * | 2002-03-20 | 2006-06-08 | Sreenath Kurupati | Method and data structure for a low memory overhead database |
US7386616B1 (en) * | 2003-05-09 | 2008-06-10 | Google Inc. | System and method for providing load balanced processing |
US7143251B1 (en) * | 2003-06-30 | 2006-11-28 | Data Domain, Inc. | Data storage using identifiers |
US9357031B2 (en) | 2004-06-03 | 2016-05-31 | Microsoft Technology Licensing, Llc | Applications as a service |
US20060047716A1 (en) * | 2004-06-03 | 2006-03-02 | Keith Robert O Jr | Transaction based virtual file system optimized for high-latency network connections |
US20060031529A1 (en) * | 2004-06-03 | 2006-02-09 | Keith Robert O Jr | Virtual application manager |
US8812613B2 (en) | 2004-06-03 | 2014-08-19 | Maxsp Corporation | Virtual application manager |
US7908339B2 (en) | 2004-06-03 | 2011-03-15 | Maxsp Corporation | Transaction based virtual file system optimized for high-latency network connections |
US9569194B2 (en) | 2004-06-03 | 2017-02-14 | Microsoft Technology Licensing, Llc | Virtual application manager |
US20060047946A1 (en) * | 2004-07-09 | 2006-03-02 | Keith Robert O Jr | Distributed operating system management |
US7664834B2 (en) | 2004-07-09 | 2010-02-16 | Maxsp Corporation | Distributed operating system management |
US20060224545A1 (en) * | 2005-03-04 | 2006-10-05 | Keith Robert O Jr | Computer hardware and software diagnostic and report system |
US20070233633A1 (en) * | 2005-03-04 | 2007-10-04 | Keith Robert O Jr | Computer hardware and software diagnostic and report system |
US8589323B2 (en) | 2005-03-04 | 2013-11-19 | Maxsp Corporation | Computer hardware and software diagnostic and report system incorporating an expert system and agents |
US8234238B2 (en) | 2005-03-04 | 2012-07-31 | Maxsp Corporation | Computer hardware and software diagnostic and report system |
US7624086B2 (en) | 2005-03-04 | 2009-11-24 | Maxsp Corporation | Pre-install compliance system |
US20060224544A1 (en) * | 2005-03-04 | 2006-10-05 | Keith Robert O Jr | Pre-install compliance system |
US7512584B2 (en) | 2005-03-04 | 2009-03-31 | Maxsp Corporation | Computer hardware and software diagnostic and report system |
US8811396B2 (en) | 2006-05-24 | 2014-08-19 | Maxsp Corporation | System for and method of securing a network utilizing credentials |
US9893961B2 (en) | 2006-05-24 | 2018-02-13 | Microsoft Technology Licensing, Llc | Applications and services as a bundle |
US9584480B2 (en) | 2006-05-24 | 2017-02-28 | Microsoft Technology Licensing, Llc | System for and method of securing a network utilizing credentials |
US8898319B2 (en) | 2006-05-24 | 2014-11-25 | Maxsp Corporation | Applications and services as a bundle |
US20070274315A1 (en) * | 2006-05-24 | 2007-11-29 | Keith Robert O | System for and method of securing a network utilizing credentials |
US10511495B2 (en) | 2006-05-24 | 2019-12-17 | Microsoft Technology Licensing, Llc | Applications and services as a bundle |
US9160735B2 (en) | 2006-05-24 | 2015-10-13 | Microsoft Technology Licensing, Llc | System for and method of securing a network utilizing credentials |
US9906418B2 (en) | 2006-05-24 | 2018-02-27 | Microsoft Technology Licensing, Llc | Applications and services as a bundle |
US8667273B1 (en) | 2006-05-30 | 2014-03-04 | Leif Olov Billstrom | Intelligent file encryption and secure backup system |
US8099605B1 (en) | 2006-06-05 | 2012-01-17 | InventSec AB | Intelligent storage device for backup system |
US7840514B2 (en) | 2006-09-22 | 2010-11-23 | Maxsp Corporation | Secure virtual private network utilizing a diagnostics policy and diagnostics engine to establish a secure network connection |
US8099378B2 (en) | 2006-09-22 | 2012-01-17 | Maxsp Corporation | Secure virtual private network utilizing a diagnostics policy and diagnostics engine to establish a secure network connection |
US20110047118A1 (en) * | 2006-09-22 | 2011-02-24 | Maxsp Corporation | Secure virtual private network utilizing a diagnostics policy and diagnostics engine to establish a secure network connection |
US9317506B2 (en) | 2006-09-22 | 2016-04-19 | Microsoft Technology Licensing, Llc | Accelerated data transfer using common prior data segments |
US20080077724A1 (en) * | 2006-09-22 | 2008-03-27 | Parthasarathy Sarangam | Interrupt coalescing control scheme |
US20080077630A1 (en) * | 2006-09-22 | 2008-03-27 | Keith Robert O | Accelerated data transfer using common prior data segments |
US8909881B2 (en) | 2006-11-28 | 2014-12-09 | Commvault Systems, Inc. | Systems and methods for creating copies of data, such as archive copies |
US8140786B2 (en) * | 2006-12-04 | 2012-03-20 | Commvault Systems, Inc. | Systems and methods for creating copies of data, such as archive copies |
US20080229037A1 (en) * | 2006-12-04 | 2008-09-18 | Alan Bunte | Systems and methods for creating copies of data, such as archive copies |
US8392677B2 (en) | 2006-12-04 | 2013-03-05 | Commvault Systems, Inc. | Systems and methods for creating copies of data, such as archive copies |
US9645900B2 (en) | 2006-12-21 | 2017-05-09 | Microsoft Technology Licensing, Llc | Warm standby appliance |
US7844686B1 (en) | 2006-12-21 | 2010-11-30 | Maxsp Corporation | Warm standby appliance |
US8423821B1 (en) | 2006-12-21 | 2013-04-16 | Maxsp Corporation | Virtual recovery server |
US8745171B1 (en) | 2006-12-21 | 2014-06-03 | Maxsp Corporation | Warm standby appliance |
US20080243958A1 (en) * | 2006-12-22 | 2008-10-02 | Anand Prahlad | System and method for storing redundant information |
US7840537B2 (en) | 2006-12-22 | 2010-11-23 | Commvault Systems, Inc. | System and method for storing redundant information |
US20130006946A1 (en) * | 2006-12-22 | 2013-01-03 | Commvault Systems, Inc. | System and method for storing redundant information |
US7953706B2 (en) | 2006-12-22 | 2011-05-31 | Commvault Systems, Inc. | System and method for storing redundant information |
US20080243879A1 (en) * | 2006-12-22 | 2008-10-02 | Parag Gokhale | System and method for storing redundant information |
US8712969B2 (en) * | 2006-12-22 | 2014-04-29 | Commvault Systems, Inc. | System and method for storing redundant information |
US20080243957A1 (en) * | 2006-12-22 | 2008-10-02 | Anand Prahlad | System and method for storing redundant information |
US10922006B2 (en) | 2006-12-22 | 2021-02-16 | Commvault Systems, Inc. | System and method for storing redundant information |
US8037028B2 (en) | 2006-12-22 | 2011-10-11 | Commvault Systems, Inc. | System and method for storing redundant information |
US8285683B2 (en) * | 2006-12-22 | 2012-10-09 | Commvault Systems, Inc. | System and method for storing redundant information |
US10061535B2 (en) | 2006-12-22 | 2018-08-28 | Commvault Systems, Inc. | System and method for storing redundant information |
WO2008147516A1 (en) * | 2007-05-22 | 2008-12-04 | Network Appliance, Inc. | System and method for on-the-fly elimination of redundant date |
US8315984B2 (en) | 2007-05-22 | 2012-11-20 | Netapp, Inc. | System and method for on-the-fly elimination of redundant data |
US20090007261A1 (en) * | 2007-06-29 | 2009-01-01 | Smith Mark A | Receiving data in a data store in a server computer system |
US8615798B2 (en) | 2007-06-29 | 2013-12-24 | International Business Machines Corporation | Optimizing a data deduplication system using client authentication information |
US20090051705A1 (en) * | 2007-08-20 | 2009-02-26 | Shih-Kuang Tsai | Method for automatically memorizing picture orientation |
US8422833B2 (en) | 2007-10-26 | 2013-04-16 | Maxsp Corporation | Method of and system for enhanced data storage |
US9092374B2 (en) | 2007-10-26 | 2015-07-28 | Maxsp Corporation | Method of and system for enhanced data storage |
US8307239B1 (en) | 2007-10-26 | 2012-11-06 | Maxsp Corporation | Disaster recovery appliance |
US8175418B1 (en) | 2007-10-26 | 2012-05-08 | Maxsp Corporation | Method of and system for enhanced data storage |
US9448858B2 (en) | 2007-10-26 | 2016-09-20 | Microsoft Technology Licensing, Llc | Environment manager |
US8645515B2 (en) | 2007-10-26 | 2014-02-04 | Maxsp Corporation | Environment manager |
US20090234892A1 (en) * | 2008-03-14 | 2009-09-17 | International Business Machines Corporation | Method and system for assuring integrity of deduplicated data |
US8280859B2 (en) | 2008-03-14 | 2012-10-02 | International Business Machines Corporation | Method and system for assuring integrity of deduplicated data |
US7814074B2 (en) * | 2008-03-14 | 2010-10-12 | International Business Machines Corporation | Method and system for assuring integrity of deduplicated data |
US20100299311A1 (en) * | 2008-03-14 | 2010-11-25 | International Business Machines Corporation | Method and system for assuring integrity of deduplicated data |
US8131924B1 (en) | 2008-03-19 | 2012-03-06 | Netapp, Inc. | De-duplication of data stored on tape media |
US20090271454A1 (en) * | 2008-04-29 | 2009-10-29 | International Business Machines Corporation | Enhanced method and system for assuring integrity of deduplicated data |
US8515909B2 (en) | 2008-04-29 | 2013-08-20 | International Business Machines Corporation | Enhanced method and system for assuring integrity of deduplicated data |
US9405763B2 (en) | 2008-06-24 | 2016-08-02 | Commvault Systems, Inc. | De-duplication systems and methods for application-specific data |
US8219524B2 (en) | 2008-06-24 | 2012-07-10 | Commvault Systems, Inc. | Application-aware and remote single instance data management |
US9971784B2 (en) | 2008-06-24 | 2018-05-15 | Commvault Systems, Inc. | Application-aware and remote single instance data management |
US8484162B2 (en) | 2008-06-24 | 2013-07-09 | Commvault Systems, Inc. | De-duplication systems and methods for application-specific data |
US9098495B2 (en) | 2008-06-24 | 2015-08-04 | Commvault Systems, Inc. | Application-aware and remote single instance data management |
US10884990B2 (en) | 2008-06-24 | 2021-01-05 | Commvault Systems, Inc. | Application-aware and remote single instance data management |
US20090319585A1 (en) * | 2008-06-24 | 2009-12-24 | Parag Gokhale | Application-aware and remote single instance data management |
US11016859B2 (en) | 2008-06-24 | 2021-05-25 | Commvault Systems, Inc. | De-duplication systems and methods for application-specific data |
US8380957B2 (en) | 2008-07-03 | 2013-02-19 | Commvault Systems, Inc. | Continuous data protection over intermittent connections, such as continuous data backup for laptops or wireless devices |
US8166263B2 (en) | 2008-07-03 | 2012-04-24 | Commvault Systems, Inc. | Continuous data protection over intermittent connections, such as continuous data backup for laptops or wireless devices |
US8612707B2 (en) | 2008-07-03 | 2013-12-17 | Commvault Systems, Inc. | Continuous data protection over intermittent connections, such as continuous data backup for laptops or wireless devices |
US20100005259A1 (en) * | 2008-07-03 | 2010-01-07 | Anand Prahlad | Continuous data protection over intermittent connections, such as continuous data backup for laptops or wireless devices |
US8838923B2 (en) | 2008-07-03 | 2014-09-16 | Commvault Systems, Inc. | Continuous data protection over intermittent connections, such as continuous data backup for laptops or wireless devices |
US20100058002A1 (en) * | 2008-08-27 | 2010-03-04 | Netapp, Inc. | System and method for file system level compression using compression group descriptors |
US8898117B2 (en) | 2008-08-27 | 2014-11-25 | Netapp, Inc. | System and method for file system level compression using compression group descriptors |
US8285687B2 (en) | 2008-08-27 | 2012-10-09 | Netapp, Inc. | System and method for file system level compression using compression group descriptors |
US9015181B2 (en) | 2008-09-26 | 2015-04-21 | Commvault Systems, Inc. | Systems and methods for managing single instancing data |
US11593217B2 (en) | 2008-09-26 | 2023-02-28 | Commvault Systems, Inc. | Systems and methods for managing single instancing data |
US20100082672A1 (en) * | 2008-09-26 | 2010-04-01 | Rajiv Kottomtharayil | Systems and methods for managing single instancing data |
US11016858B2 (en) | 2008-09-26 | 2021-05-25 | Commvault Systems, Inc. | Systems and methods for managing single instancing data |
US8412677B2 (en) | 2008-11-26 | 2013-04-02 | Commvault Systems, Inc. | Systems and methods for byte-level or quasi byte-level single instancing |
US8725687B2 (en) | 2008-11-26 | 2014-05-13 | Commvault Systems, Inc. | Systems and methods for byte-level or quasi byte-level single instancing |
US9158787B2 (en) | 2008-11-26 | 2015-10-13 | Commvault Systems, Inc | Systems and methods for byte-level or quasi byte-level single instancing |
US20100169287A1 (en) * | 2008-11-26 | 2010-07-01 | Commvault Systems, Inc. | Systems and methods for byte-level or quasi byte-level single instancing |
US11586648B2 (en) | 2009-03-30 | 2023-02-21 | Commvault Systems, Inc. | Storing a variable number of instances of data objects |
US20100250549A1 (en) * | 2009-03-30 | 2010-09-30 | Muller Marcus S | Storing a variable number of instances of data objects |
US10970304B2 (en) | 2009-03-30 | 2021-04-06 | Commvault Systems, Inc. | Storing a variable number of instances of data objects |
US9773025B2 (en) | 2009-03-30 | 2017-09-26 | Commvault Systems, Inc. | Storing a variable number of instances of data objects |
US8401996B2 (en) | 2009-03-30 | 2013-03-19 | Commvault Systems, Inc. | Storing a variable number of instances of data objects |
US11455212B2 (en) | 2009-05-22 | 2022-09-27 | Commvault Systems, Inc. | Block-level single instancing |
US9058117B2 (en) | 2009-05-22 | 2015-06-16 | Commvault Systems, Inc. | Block-level single instancing |
US8578120B2 (en) | 2009-05-22 | 2013-11-05 | Commvault Systems, Inc. | Block-level single instancing |
US10956274B2 (en) | 2009-05-22 | 2021-03-23 | Commvault Systems, Inc. | Block-level single instancing |
US11709739B2 (en) | 2009-05-22 | 2023-07-25 | Commvault Systems, Inc. | Block-level single instancing |
US10176113B2 (en) | 2009-06-26 | 2019-01-08 | Hewlett Packard Enterprise Development Lp | Scalable indexing |
US9367551B2 (en) | 2009-06-26 | 2016-06-14 | Simplivity Corporation | File system accessing an object store |
US9965483B2 (en) | 2009-06-26 | 2018-05-08 | Hewlett Packard Enterprise Company | File system |
US8880544B2 (en) | 2009-06-26 | 2014-11-04 | Simplivity Corporation | Method of adapting a uniform access indexing process to a non-uniform access memory, and computer system |
AU2010265954B2 (en) * | 2009-06-26 | 2016-01-28 | Hewlett Packard Enterprise Development Lp | File system |
US10474631B2 (en) * | 2009-06-26 | 2019-11-12 | Hewlett Packard Enterprise Company | Method and apparatus for content derived data placement in memory |
US8478799B2 (en) | 2009-06-26 | 2013-07-02 | Simplivity Corporation | Namespace file system accessing an object store |
WO2010151813A1 (en) * | 2009-06-26 | 2010-12-29 | Simplivt Corporation | File system |
US20110022566A1 (en) * | 2009-06-26 | 2011-01-27 | Simplivt Corporation | File system |
CN102483755A (en) * | 2009-06-26 | 2012-05-30 | 森普利维蒂公司 | File system |
US20100332846A1 (en) * | 2009-06-26 | 2010-12-30 | Simplivt Corporation | Scalable indexing |
US11288235B2 (en) | 2009-07-08 | 2022-03-29 | Commvault Systems, Inc. | Synchronized data deduplication |
US8930306B1 (en) | 2009-07-08 | 2015-01-06 | Commvault Systems, Inc. | Synchronized data deduplication |
US10540327B2 (en) | 2009-07-08 | 2020-01-21 | Commvault Systems, Inc. | Synchronized data deduplication |
US10148433B1 (en) | 2009-10-14 | 2018-12-04 | Digitalpersona, Inc. | Private key/public key resource protection scheme |
US9898225B2 (en) | 2010-09-30 | 2018-02-20 | Commvault Systems, Inc. | Content aligned block-based deduplication |
US8935492B2 (en) | 2010-09-30 | 2015-01-13 | Commvault Systems, Inc. | Archiving data objects using secondary copies |
US9619480B2 (en) | 2010-09-30 | 2017-04-11 | Commvault Systems, Inc. | Content aligned block-based deduplication |
US8364652B2 (en) | 2010-09-30 | 2013-01-29 | Commvault Systems, Inc. | Content aligned block-based deduplication |
US10762036B2 (en) | 2010-09-30 | 2020-09-01 | Commvault Systems, Inc. | Archiving data objects using secondary copies |
US11392538B2 (en) | 2010-09-30 | 2022-07-19 | Commvault Systems, Inc. | Archiving data objects using secondary copies |
US9639289B2 (en) | 2010-09-30 | 2017-05-02 | Commvault Systems, Inc. | Systems and methods for retaining and using data block signatures in data protection operations |
US9639563B2 (en) | 2010-09-30 | 2017-05-02 | Commvault Systems, Inc. | Archiving data objects using secondary copies |
US9262275B2 (en) | 2010-09-30 | 2016-02-16 | Commvault Systems, Inc. | Archiving data objects using secondary copies |
US9110602B2 (en) | 2010-09-30 | 2015-08-18 | Commvault Systems, Inc. | Content aligned block-based deduplication |
US8572340B2 (en) | 2010-09-30 | 2013-10-29 | Commvault Systems, Inc. | Systems and methods for retaining and using data block signatures in data protection operations |
US8577851B2 (en) | 2010-09-30 | 2013-11-05 | Commvault Systems, Inc. | Content aligned block-based deduplication |
US8578109B2 (en) | 2010-09-30 | 2013-11-05 | Commvault Systems, Inc. | Systems and methods for retaining and using data block signatures in data protection operations |
US11768800B2 (en) | 2010-09-30 | 2023-09-26 | Commvault Systems, Inc. | Archiving data objects using secondary copies |
US10126973B2 (en) | 2010-09-30 | 2018-11-13 | Commvault Systems, Inc. | Systems and methods for retaining and using data block signatures in data protection operations |
US9239687B2 (en) | 2010-09-30 | 2016-01-19 | Commvault Systems, Inc. | Systems and methods for retaining and using data block signatures in data protection operations |
US9104623B2 (en) | 2010-12-14 | 2015-08-11 | Commvault Systems, Inc. | Client-side repository in a networked deduplicated storage system |
US11169888B2 (en) | 2010-12-14 | 2021-11-09 | Commvault Systems, Inc. | Client-side repository in a networked deduplicated storage system |
US11422976B2 (en) | 2010-12-14 | 2022-08-23 | Commvault Systems, Inc. | Distributed deduplicated storage system |
US10191816B2 (en) | 2010-12-14 | 2019-01-29 | Commvault Systems, Inc. | Client-side repository in a networked deduplicated storage system |
US9898478B2 (en) | 2010-12-14 | 2018-02-20 | Commvault Systems, Inc. | Distributed deduplicated storage system |
US9116850B2 (en) | 2010-12-14 | 2015-08-25 | Commvault Systems, Inc. | Client-side repository in a networked deduplicated storage system |
US9020900B2 (en) | 2010-12-14 | 2015-04-28 | Commvault Systems, Inc. | Distributed deduplicated storage system |
US8954446B2 (en) | 2010-12-14 | 2015-02-10 | Comm Vault Systems, Inc. | Client-side repository in a networked deduplicated storage system |
US10740295B2 (en) | 2010-12-14 | 2020-08-11 | Commvault Systems, Inc. | Distributed deduplicated storage system |
US10255340B2 (en) | 2011-06-23 | 2019-04-09 | Hewlett Packard Enterprise Development Lp | Method and apparatus for distributed configuration management |
US9092377B2 (en) * | 2011-09-29 | 2015-07-28 | International Business Machines Corporation | Managing back up operations for data |
US9811426B2 (en) | 2011-09-29 | 2017-11-07 | International Business Machines Corporation | Managing back up operations for data |
US20130086000A1 (en) * | 2011-09-29 | 2013-04-04 | International Business Machines Corporation | Managing back up operations for data |
US9875183B2 (en) | 2012-02-24 | 2018-01-23 | Hewlett Packard Enterprise Development Lp | Method and apparatus for content derived data placement in memory |
US11042511B2 (en) | 2012-03-30 | 2021-06-22 | Commvault Systems, Inc. | Smart archiving and data previewing for mobile devices |
US11615059B2 (en) | 2012-03-30 | 2023-03-28 | Commvault Systems, Inc. | Smart archiving and data previewing for mobile devices |
US9020890B2 (en) | 2012-03-30 | 2015-04-28 | Commvault Systems, Inc. | Smart archiving and data previewing for mobile devices |
US9218374B2 (en) | 2012-06-13 | 2015-12-22 | Commvault Systems, Inc. | Collaborative restore in a networked storage system |
US9218375B2 (en) | 2012-06-13 | 2015-12-22 | Commvault Systems, Inc. | Dedicated client-side signature generator in a networked storage system |
US9251186B2 (en) | 2012-06-13 | 2016-02-02 | Commvault Systems, Inc. | Backup using a client-side signature repository in a networked storage system |
US9858156B2 (en) | 2012-06-13 | 2018-01-02 | Commvault Systems, Inc. | Dedicated client-side signature generator in a networked storage system |
US10176053B2 (en) | 2012-06-13 | 2019-01-08 | Commvault Systems, Inc. | Collaborative restore in a networked storage system |
US10387269B2 (en) | 2012-06-13 | 2019-08-20 | Commvault Systems, Inc. | Dedicated client-side signature generator in a networked storage system |
US10956275B2 (en) | 2012-06-13 | 2021-03-23 | Commvault Systems, Inc. | Collaborative restore in a networked storage system |
US9218376B2 (en) | 2012-06-13 | 2015-12-22 | Commvault Systems, Inc. | Intelligent data sourcing in a networked storage system |
CN102915325A (en) * | 2012-08-11 | 2013-02-06 | 深圳市极限网络科技有限公司 | Md5 Hash list-based file decomposing and combining technique |
US9959275B2 (en) | 2012-12-28 | 2018-05-01 | Commvault Systems, Inc. | Backup and restoration for a deduplicated file system |
US11080232B2 (en) | 2012-12-28 | 2021-08-03 | Commvault Systems, Inc. | Backup and restoration for a deduplicated file system |
US9633022B2 (en) | 2012-12-28 | 2017-04-25 | Commvault Systems, Inc. | Backup and restoration for a deduplicated file system |
US11157450B2 (en) | 2013-01-11 | 2021-10-26 | Commvault Systems, Inc. | High availability distributed deduplicated storage system |
US9633033B2 (en) | 2013-01-11 | 2017-04-25 | Commvault Systems, Inc. | High availability distributed deduplicated storage system |
US10229133B2 (en) | 2013-01-11 | 2019-03-12 | Commvault Systems, Inc. | High availability distributed deduplicated storage system |
US9665591B2 (en) | 2013-01-11 | 2017-05-30 | Commvault Systems, Inc. | High availability distributed deduplicated storage system |
US20160042183A1 (en) * | 2013-03-28 | 2016-02-11 | Irdeto B.V. | Generating identifier |
US10762038B2 (en) | 2013-08-21 | 2020-09-01 | Hewlett Packard Enterprise Development Lp | System and method for virtual machine conversion |
US9811522B2 (en) | 2013-08-21 | 2017-11-07 | Hewlett Packard Enterprise Development Lp | System and method for transforming a source virtual machine without copying of payload data |
CN104731792A (en) * | 2013-12-19 | 2015-06-24 | 中国银联股份有限公司 | Method and system for verifying database consistency and method and system for positioning data difference |
US11940952B2 (en) | 2014-01-27 | 2024-03-26 | Commvault Systems, Inc. | Techniques for serving archived electronic mail |
US10324897B2 (en) | 2014-01-27 | 2019-06-18 | Commvault Systems, Inc. | Techniques for serving archived electronic mail |
US10380072B2 (en) | 2014-03-17 | 2019-08-13 | Commvault Systems, Inc. | Managing deletions from a deduplication database |
US10445293B2 (en) | 2014-03-17 | 2019-10-15 | Commvault Systems, Inc. | Managing deletions from a deduplication database |
US11119984B2 (en) | 2014-03-17 | 2021-09-14 | Commvault Systems, Inc. | Managing deletions from a deduplication database |
US9633056B2 (en) | 2014-03-17 | 2017-04-25 | Commvault Systems, Inc. | Maintaining a deduplication database |
US11188504B2 (en) | 2014-03-17 | 2021-11-30 | Commvault Systems, Inc. | Managing deletions from a deduplication database |
US10264071B2 (en) | 2014-03-31 | 2019-04-16 | Amazon Technologies, Inc. | Session management in distributed storage systems |
US9779015B1 (en) | 2014-03-31 | 2017-10-03 | Amazon Technologies, Inc. | Oversubscribed storage extents with on-demand page allocation |
US9495478B2 (en) | 2014-03-31 | 2016-11-15 | Amazon Technologies, Inc. | Namespace management in distributed storage systems |
US9772787B2 (en) | 2014-03-31 | 2017-09-26 | Amazon Technologies, Inc. | File storage using variable stripe sizes |
KR101891425B1 (en) | 2014-03-31 | 2018-08-24 | 아마존 테크놀로지스, 인크. | Namespace management in distributed storage systems |
AU2015241298B2 (en) * | 2014-03-31 | 2018-03-15 | Amazon Technologies, Inc. | Namespace management in distributed storage systems |
US10372685B2 (en) | 2014-03-31 | 2019-08-06 | Amazon Technologies, Inc. | Scalable file storage service |
WO2015153259A1 (en) * | 2014-03-31 | 2015-10-08 | Amazon Technologies, Inc. | Namespace management in distributed storage systems |
US11249858B2 (en) | 2014-08-06 | 2022-02-15 | Commvault Systems, Inc. | Point-in-time backups of a production application made accessible over fibre channel and/or ISCSI as data sources to a remote application by representing the backups as pseudo-disks operating apart from the production application and its host |
US11416341B2 (en) | 2014-08-06 | 2022-08-16 | Commvault Systems, Inc. | Systems and methods to reduce application downtime during a restore operation using a pseudo-storage device |
US10474638B2 (en) | 2014-10-29 | 2019-11-12 | Commvault Systems, Inc. | Accessing a file system using tiered deduplication |
US11921675B2 (en) | 2014-10-29 | 2024-03-05 | Commvault Systems, Inc. | Accessing a file system using tiered deduplication |
US9575673B2 (en) | 2014-10-29 | 2017-02-21 | Commvault Systems, Inc. | Accessing a file system using tiered deduplication |
US9934238B2 (en) | 2014-10-29 | 2018-04-03 | Commvault Systems, Inc. | Accessing a file system using tiered deduplication |
US11113246B2 (en) | 2014-10-29 | 2021-09-07 | Commvault Systems, Inc. | Accessing a file system using tiered deduplication |
US10884633B2 (en) | 2015-01-13 | 2021-01-05 | Hewlett Packard Enterprise Development Lp | System and method for optimized signature comparisons and data replication |
US9646176B2 (en) | 2015-03-24 | 2017-05-09 | TmaxData Co., Ltd. | Method for encrypting database |
US11301420B2 (en) | 2015-04-09 | 2022-04-12 | Commvault Systems, Inc. | Highly reusable deduplication database after disaster recovery |
US10339106B2 (en) | 2015-04-09 | 2019-07-02 | Commvault Systems, Inc. | Highly reusable deduplication database after disaster recovery |
US10089337B2 (en) | 2015-05-20 | 2018-10-02 | Commvault Systems, Inc. | Predicting scale of data migration between production and archive storage systems, such as for enterprise customers having large and/or numerous files |
US10324914B2 (en) | 2015-05-20 | 2019-06-18 | Commvalut Systems, Inc. | Handling user queries against production and archive storage systems, such as for enterprise customers having large and/or numerous files |
US11281642B2 (en) | 2015-05-20 | 2022-03-22 | Commvault Systems, Inc. | Handling user queries against production and archive storage systems, such as for enterprise customers having large and/or numerous files |
US10977231B2 (en) | 2015-05-20 | 2021-04-13 | Commvault Systems, Inc. | Predicting scale of data migration |
US10481826B2 (en) | 2015-05-26 | 2019-11-19 | Commvault Systems, Inc. | Replication using deduplicated secondary copy data |
US10481825B2 (en) | 2015-05-26 | 2019-11-19 | Commvault Systems, Inc. | Replication using deduplicated secondary copy data |
US10481824B2 (en) | 2015-05-26 | 2019-11-19 | Commvault Systems, Inc. | Replication using deduplicated secondary copy data |
US11733877B2 (en) | 2015-07-22 | 2023-08-22 | Commvault Systems, Inc. | Restore for block-level backups |
US11314424B2 (en) | 2015-07-22 | 2022-04-26 | Commvault Systems, Inc. | Restore for block-level backups |
US10310953B2 (en) | 2015-12-30 | 2019-06-04 | Commvault Systems, Inc. | System for redirecting requests after a secondary storage computing device failure |
US10877856B2 (en) | 2015-12-30 | 2020-12-29 | Commvault Systems, Inc. | System for redirecting requests after a secondary storage computing device failure |
US10956286B2 (en) | 2015-12-30 | 2021-03-23 | Commvault Systems, Inc. | Deduplication replication in a distributed deduplication data storage system |
US10255143B2 (en) | 2015-12-30 | 2019-04-09 | Commvault Systems, Inc. | Deduplication replication in a distributed deduplication data storage system |
US10592357B2 (en) | 2015-12-30 | 2020-03-17 | Commvault Systems, Inc. | Distributed file system in a distributed deduplication data storage system |
US10061663B2 (en) | 2015-12-30 | 2018-08-28 | Commvault Systems, Inc. | Rebuilding deduplication data in a distributed deduplication data storage system |
US11436038B2 (en) | 2016-03-09 | 2022-09-06 | Commvault Systems, Inc. | Hypervisor-independent block-level live browse for access to backed up virtual machine (VM) data and hypervisor-free file-level recovery (block- level pseudo-mount) |
US10545927B2 (en) | 2016-03-25 | 2020-01-28 | Amazon Technologies, Inc. | File system mode switching in a distributed storage service |
US10474636B2 (en) | 2016-03-25 | 2019-11-12 | Amazon Technologies, Inc. | Block allocation for low latency file systems |
US10140312B2 (en) | 2016-03-25 | 2018-11-27 | Amazon Technologies, Inc. | Low latency distributed storage service |
US11061865B2 (en) | 2016-03-25 | 2021-07-13 | Amazon Technologies, Inc. | Block allocation for low latency file systems |
US10303458B2 (en) | 2016-09-29 | 2019-05-28 | Hewlett Packard Enterprise Development Lp | Multi-platform installer |
US11321195B2 (en) | 2017-02-27 | 2022-05-03 | Commvault Systems, Inc. | Hypervisor-independent reference copies of virtual machine payload data based on block-level pseudo-mount |
US10887176B2 (en) | 2017-03-30 | 2021-01-05 | Hewlett Packard Enterprise Development Lp | Predicting resource demand in computing environments |
US11010300B2 (en) | 2017-05-04 | 2021-05-18 | Hewlett Packard Enterprise Development Lp | Optimized record lookups |
US11294768B2 (en) | 2017-06-14 | 2022-04-05 | Commvault Systems, Inc. | Live browsing of backed up data residing on cloned disks |
US10862736B2 (en) | 2018-01-30 | 2020-12-08 | Hewlett Packard Enterprise Development Lp | Object counts persistence for object stores |
US10860738B2 (en) | 2018-01-30 | 2020-12-08 | Hewlett Packard Enterprise Development Lp | Augmented metadata and signatures for objects in object stores |
US10587454B2 (en) | 2018-01-30 | 2020-03-10 | Hewlett Packard Enterprise Development Lp | Object counts persistence for object stores |
US11126755B2 (en) | 2018-01-30 | 2021-09-21 | Hewlett Packard Enterprise Development Lp | Object signatures in object stores |
US10997153B2 (en) | 2018-04-20 | 2021-05-04 | Hewlett Packard Enterprise Development Lp | Transaction encoding and transaction persistence according to type of persistent storage |
US11243703B2 (en) | 2018-04-27 | 2022-02-08 | Hewlett Packard Enterprise Development Lp | Expandable index with pages to store object records |
US11681587B2 (en) | 2018-11-27 | 2023-06-20 | Commvault Systems, Inc. | Generating copies through interoperability between a data storage management system and appliances for data storage and deduplication |
US11010258B2 (en) | 2018-11-27 | 2021-05-18 | Commvault Systems, Inc. | Generating backup copies through interoperability between components of a data storage management system and appliances for data storage and deduplication |
US11698727B2 (en) | 2018-12-14 | 2023-07-11 | Commvault Systems, Inc. | Performing secondary copy operations based on deduplication performance |
US11829251B2 (en) | 2019-04-10 | 2023-11-28 | Commvault Systems, Inc. | Restore using deduplicated secondary copy data |
US11463264B2 (en) | 2019-05-08 | 2022-10-04 | Commvault Systems, Inc. | Use of data block signatures for monitoring in an information management system |
US11442896B2 (en) | 2019-12-04 | 2022-09-13 | Commvault Systems, Inc. | Systems and methods for optimizing restoration of deduplicated data stored in cloud-based storage resources |
US11687424B2 (en) | 2020-05-28 | 2023-06-27 | Commvault Systems, Inc. | Automated media agent state management |
Also Published As
Publication number | Publication date |
---|---|
US6704730B2 (en) | 2004-03-09 |
US20010037323A1 (en) | 2001-11-01 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US6704730B2 (en) | Hash file system and method for use in a commonality factoring system | |
AU2001238269B2 (en) | Hash file system and method for use in a commonality factoring system | |
AU2001238269A1 (en) | Hash file system and method for use in a commonality factoring system | |
US6810398B2 (en) | System and method for unorchestrated determination of data sequences using sticky byte factoring to determine breakpoints in digital sequences | |
US9967298B2 (en) | Appending to files via server-side chunking and manifest manipulation | |
AU2001296665A1 (en) | System for identifying common digital sequences | |
US8165221B2 (en) | System and method for sampling based elimination of duplicate data | |
US7457800B2 (en) | Storage system for randomly named blocks of data | |
EP2147437B1 (en) | Seeding replication | |
Phyu et al. | Efficient data deduplication scheme for scale-out distributed storage | |
Bhagwat | DOCTOR OF PHILOSOPHY in COMPUTER SCIENCE |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
STCB | Information on status: application discontinuation |
Free format text: ABANDONED -- FAILURE TO RESPOND TO AN OFFICE ACTION |